# [PATCH v5] git-p4: add config git-p4.pathEncoding 6 messages from 2015-09-02 to 2015-09-04. Participants: larsxschneider@gmail.com, Luke Diamand, Remi Galan Alfonso, Lars Schneider. Thread: https://gitlist.dev/t/40251 ## larsxschneider@gmail.com, 2015-09-02 20:39 Subject: [PATCH v5] git-p4: add config git-p4.pathEncoding Message-ID: <1441226355-61103-1-git-send-email-larsxschneider@gmail.com> URL: https://gitlist.dev/e/1441226355-61103-1-git-send-email-larsxschneider%40gmail.com ``` From: Lars Schneider Diff to v4: * add verbose output * reword help text * remove comment how to generate encoded sequence * use ascii characters in encoded sequence where possible * wrap 'cd' in TC in a subshell Thanks to Eric, Torsten, and Junio for feedback! Cheers, Lars Lars Schneider (1): git-p4: add config git-p4.pathEncoding Documentation/git-p4.txt | 7 +++++ git-p4.py | 11 ++++++++ t/t9822-git-p4-path-encoding.sh | 60 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 78 insertions(+) create mode 100755 t/t9822-git-p4-path-encoding.sh -- 1.9.5 (Apple Git-50.3) ``` ## larsxschneider@gmail.com, 2015-09-02 20:39 Subject: [PATCH v5] git-p4: add config git-p4.pathEncoding Message-ID: <1441226355-61103-2-git-send-email-larsxschneider@gmail.com> URL: https://gitlist.dev/e/1441226355-61103-2-git-send-email-larsxschneider%40gmail.com In-Reply-To: <1441226355-61103-1-git-send-email-larsxschneider@gmail.com> ``` From: Lars Schneider Perforce keeps the encoding of a path as given by the originating OS. Git expects paths encoded as UTF-8. Add a config to tell git-p4 what encoding Perforce had used for the paths. This encoding is used to transcode the paths to UTF-8. As an example, Perforce on Windows often uses “cp1252” to encode path names. Signed-off-by: Lars Schneider --- Documentation/git-p4.txt | 7 +++++ git-p4.py | 11 ++++++++ t/t9822-git-p4-path-encoding.sh | 60 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 78 insertions(+) create mode 100755 t/t9822-git-p4-path-encoding.sh diff --git a/Documentation/git-p4.txt b/Documentation/git-p4.txt index 82aa5d6..7b7a57d 100644 --- a/Documentation/git-p4.txt +++ b/Documentation/git-p4.txt @@ -510,6 +510,13 @@ git-p4.useClientSpec:: option '--use-client-spec'. See the "CLIENT SPEC" section above. This variable is a boolean, not the name of a p4 client. +git-p4.pathEncoding:: + Perforce keeps the encoding of a path as given by the originating OS. + Git expects paths encoded as UTF-8. Use this config to tell git-p4 + what encoding Perforce had used for the paths. This encoding is used + to transcode the paths to UTF-8. As an example, Perforce on Windows + often uses “cp1252” to encode path names. + Submit variables ~~~~~~~~~~~~~~~~ git-p4.detectRenames:: diff --git a/git-p4.py b/git-p4.py index 073f87b..b1ad86d 100755 --- a/git-p4.py +++ b/git-p4.py @@ -2213,6 +2213,17 @@ class P4Sync(Command, P4UserMap): text = regexp.sub(r'$\1$', text) contents = [ text ] + if gitConfig("git-p4.pathEncoding"): + relPath = relPath.decode(gitConfig("git-p4.pathEncoding")).encode('utf8', 'replace') + elif self.verbose: + try: + relPath.decode('ascii') + except: + print ( + "Path with Non-ASCII characters detected and no path encoding defined. " + "Please check the encoding: %s" % relPath + ) + self.gitStream.write("M %s inline %s\n" % (git_mode, relPath)) # total length... diff --git a/t/t9822-git-p4-path-encoding.sh b/t/t9822-git-p4-path-encoding.sh new file mode 100755 index 0000000..3646580 --- /dev/null +++ b/t/t9822-git-p4-path-encoding.sh @@ -0,0 +1,60 @@ +#!/bin/sh + +test_description='Clone repositories with non ASCII paths' + +. ./lib-git-p4.sh + +UTF8_ESCAPED="a-\303\244_o-\303\266_u-\303\274.txt" +ISO8859_ESCAPED="a-\344_o-\366_u-\374.txt" + +test_expect_success 'start p4d' ' + start_p4d +' + +test_expect_success 'Create a repo containing iso8859-1 encoded paths' ' + ( + cd "$cli" && + ISO8859="$(printf "$ISO8859_ESCAPED")" && + echo content123 >"$ISO8859" && + p4 add "$ISO8859" && + p4 submit -d "test commit" + ) +' + +test_expect_success 'Clone repo containing iso8859-1 encoded paths without git-p4.pathEncoding' ' + git p4 clone --destination="$git" //depot && + test_when_finished cleanup_git && + ( + cd "$git" && + UTF8="$(printf "$UTF8_ESCAPED")" && + echo $UTF8 >expect && + git -c core.quotepath=false ls-files >actual && + test_must_fail test_cmp expect actual + ) +' + +test_expect_success 'Clone repo containing iso8859-1 encoded paths with git-p4.pathEncoding' ' + + test_when_finished cleanup_git && + ( + cd "$git" && + git init . && + git config git-p4.pathEncoding iso8859-1 && + git p4 clone --use-client-spec --destination="$git" //depot && + UTF8="$(printf "$UTF8_ESCAPED")" && + echo $UTF8 >expect && + git -c core.quotepath=false ls-files >actual && + test_cmp expect actual && + cat >expect <<-\EOF && + content123 + EOF + cat $UTF8 >actual && + test_cmp expect actual + ) +' + +test_expect_success 'kill p4d' ' + kill_p4d +' + +test_done -- 1.9.5 (Apple Git-50.3) ``` ## Luke Diamand, 2015-09-02 21:32 Subject: Re: [PATCH v5] git-p4: add config git-p4.pathEncoding Message-ID: URL: https://gitlist.dev/e/CAE5ih7_Kje_Qrz7zKH4uSGziVpdp6VqsZf7hSMwQkvta97CG9w%40mail.gmail.com In-Reply-To: <1441226355-61103-2-git-send-email-larsxschneider@gmail.com> ``` On 2 September 2015 at 21:39, wrote: > From: Lars Schneider > > Perforce keeps the encoding of a path as given by the originating OS. > Git expects paths encoded as UTF-8. Add a config to tell git-p4 what > encoding Perforce had used for the paths. This encoding is used to > transcode the paths to UTF-8. As an example, Perforce on Windows often > uses “cp1252” to encode path names. Looks good to me, and the test passes. Note that git diff --check complains about a trailing whitespace. ``` ## Remi Galan Alfonso, 2015-09-03 08:26 Subject: Re: [PATCH v5] git-p4: add config git-p4.pathEncoding Message-ID: <983771583.1014619.1441268816360.JavaMail.zimbra@ensimag.grenoble-inp.fr> URL: https://gitlist.dev/e/983771583.1014619.1441268816360.JavaMail.zimbra%40ensimag.grenoble-inp.fr In-Reply-To: <1441226355-61103-2-git-send-email-larsxschneider@gmail.com> ``` Hi Lars, Lars Schneider writes: > +test_expect_success 'Clone repo containing iso8859-1 encoded paths with git-p4.pathEncoding' ' > + > + test_when_finished cleanup_git && > + ( > + cd "$git" && > + git init . && > + git config git-p4.pathEncoding iso8859-1 && Wouldn't 'test_config' be better here? Thanks, Rémi ``` ## Lars Schneider, 2015-09-03 15:20 Subject: Re: [PATCH v5] git-p4: add config git-p4.pathEncoding Message-ID: <5E77893F-8E13-4696-B7A6-46E10FE41989@gmail.com> URL: https://gitlist.dev/e/5E77893F-8E13-4696-B7A6-46E10FE41989%40gmail.com In-Reply-To: <983771583.1014619.1441268816360.JavaMail.zimbra@ensimag.grenoble-inp.fr> ``` On 03 Sep 2015, at 10:26, Remi Galan Alfonso wrote: > Hi Lars, > > Lars Schneider writes: > >> +test_expect_success 'Clone repo containing iso8859-1 encoded paths with git-p4.pathEncoding' ' >> + >> + test_when_finished cleanup_git && >> + ( >> + cd "$git" && >> + git init . && >> + git config git-p4.pathEncoding iso8859-1 && > > Wouldn't 'test_config' be better here? > If I got Eric’s explanation right in "[PATCH v1] t9821: use test_config” then we should not use “test_config” here? Thanks, Lars ``` ## Remi Galan Alfonso, 2015-09-04 08:30 Subject: Re: [PATCH v5] git-p4: add config git-p4.pathEncoding Message-ID: <1646294553.1067005.1441355413320.JavaMail.zimbra@ensimag.grenoble-inp.fr> URL: https://gitlist.dev/e/1646294553.1067005.1441355413320.JavaMail.zimbra%40ensimag.grenoble-inp.fr In-Reply-To: <5E77893F-8E13-4696-B7A6-46E10FE41989@gmail.com> ``` Lars Schneider writes: > > Hi Lars, > > > > Lars Schneider writes: > > > >> +test_expect_success 'Clone repo containing iso8859-1 encoded paths with git-p4.pathEncoding' ' > >> + > >> + test_when_finished cleanup_git && > >> + ( > >> + cd "$git" && > >> + git init . && > >> + git config git-p4.pathEncoding iso8859-1 && > > > > Wouldn't 'test_config' be better here? > > > If I got Eric’s explanation right in "[PATCH v1] t9821: use > test_config” then we should not use “test_config” here? After reading Eric's point of view, I agree with him about this. So, sorry for the noise, now I'll know better in the future when to use and not use 'test_config'. Thanks, Rémi ```