{"thread":{"id":"40258","subject":"[PATCH v6] git-p4: add config git-p4.pathEncoding","startedAt":"2015-09-03T09:14:06Z","lastAt":"2015-09-03T18:18:36Z","messageCount":5,"participants":["larsxschneider@gmail.com","Junio C Hamano","Lars Schneider"],"isPatch":true,"patchVersion":6,"patchTotal":null},"messages":[{"id":"269300","messageId":"1441271647-67824-1-git-send-email-larsxschneider@gmail.com","threadId":"40258","inReplyTo":null,"subject":"[PATCH v6] git-p4: add config git-p4.pathEncoding","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-09-03T09:14:06Z","receivedAt":"2015-09-03T09:14:06Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider <larsxschneider@gmail.com>\n\nDiff to v5:\n* use \"test_config\" (Thanks Remi! I am still learning all the tools...)\n* removed whitespaces (Thanks Luke! I added this to my \"generate patch\" script. Won't happen again :-)\n* added ACK from Luke (I interpreted \"Looks good to me\" that way. I hope this is OK.)\n\nThanks,\nLars\n\nLars Schneider (1):\n  git-p4: add config git-p4.pathEncoding\n\n Documentation/git-p4.txt        |  7 +++++\n git-p4.py                       | 11 ++++++++\n t/t9822-git-p4-path-encoding.sh | 60 +++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 78 insertions(+)\n create mode 100755 t/t9822-git-p4-path-encoding.sh\n\n--\n1.9.5 (Apple Git-50.3)\n"},{"id":"269301","messageId":"1441271647-67824-2-git-send-email-larsxschneider@gmail.com","threadId":"40258","inReplyTo":"1441271647-67824-1-git-send-email-larsxschneider@gmail.com","subject":"[PATCH v6] git-p4: add config git-p4.pathEncoding","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-09-03T09:14:07Z","receivedAt":"2015-09-03T09:14:07Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider <larsxschneider@gmail.com>\n\nPerforce keeps the encoding of a path as given by the originating OS.\nGit expects paths encoded as UTF-8. Add a config to tell git-p4 what\nencoding Perforce had used for the paths. This encoding is used to\ntranscode the paths to UTF-8. As an example, Perforce on Windows often\nuses “cp1252” to encode path names.\n\nSigned-off-by: Lars Schneider <larsxschneider@gmail.com>\nAcked-by: Luke Diamand <luke@diamand.org>\nSigned-off-by: Lars Schneider <larsxschneider@gmail.com>\n---\n Documentation/git-p4.txt        |  7 +++++\n git-p4.py                       | 11 ++++++++\n t/t9822-git-p4-path-encoding.sh | 60 +++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 78 insertions(+)\n create mode 100755 t/t9822-git-p4-path-encoding.sh\n\ndiff --git a/Documentation/git-p4.txt b/Documentation/git-p4.txt\nindex 82aa5d6..12a57d4 100644\n--- a/Documentation/git-p4.txt\n+++ b/Documentation/git-p4.txt\n@@ -510,6 +510,13 @@ git-p4.useClientSpec::\n \toption '--use-client-spec'.  See the \"CLIENT SPEC\" section above.\n \tThis variable is a boolean, not the name of a p4 client.\n \n+git-p4.pathEncoding::\n+\tPerforce keeps the encoding of a path as given by the originating OS.\n+\tGit expects paths encoded as UTF-8. Use this config to tell git-p4\n+\twhat encoding Perforce had used for the paths. This encoding is used\n+\tto transcode the paths to UTF-8. As an example, Perforce on Windows\n+\toften uses “cp1252” to encode path names.\n+\n Submit variables\n ~~~~~~~~~~~~~~~~\n git-p4.detectRenames::\ndiff --git a/git-p4.py b/git-p4.py\nindex 073f87b..b1ad86d 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -2213,6 +2213,17 @@ class P4Sync(Command, P4UserMap):\n             text = regexp.sub(r'$\\1$', text)\n             contents = [ text ]\n \n+        if gitConfig(\"git-p4.pathEncoding\"):\n+            relPath = relPath.decode(gitConfig(\"git-p4.pathEncoding\")).encode('utf8', 'replace')\n+        elif self.verbose:\n+            try:\n+                relPath.decode('ascii')\n+            except:\n+                print (\n+                    \"Path with Non-ASCII characters detected and no path encoding defined. \"\n+                    \"Please check the encoding: %s\" % relPath\n+                )\n+\n         self.gitStream.write(\"M %s inline %s\\n\" % (git_mode, relPath))\n \n         # total length...\ndiff --git a/t/t9822-git-p4-path-encoding.sh b/t/t9822-git-p4-path-encoding.sh\nnew file mode 100755\nindex 0000000..e507ad7\n--- /dev/null\n+++ b/t/t9822-git-p4-path-encoding.sh\n@@ -0,0 +1,60 @@\n+#!/bin/sh\n+\n+test_description='Clone repositories with non ASCII paths'\n+\n+. ./lib-git-p4.sh\n+\n+UTF8_ESCAPED=\"a-\\303\\244_o-\\303\\266_u-\\303\\274.txt\"\n+ISO8859_ESCAPED=\"a-\\344_o-\\366_u-\\374.txt\"\n+\n+test_expect_success 'start p4d' '\n+\tstart_p4d\n+'\n+\n+test_expect_success 'Create a repo containing iso8859-1 encoded paths' '\n+\t(\n+\t\tcd \"$cli\" &&\n+\t\tISO8859=\"$(printf \"$ISO8859_ESCAPED\")\" &&\n+\t\techo content123 >\"$ISO8859\" &&\n+\t\tp4 add \"$ISO8859\" &&\n+\t\tp4 submit -d \"test commit\"\n+\t)\n+'\n+\n+test_expect_success 'Clone repo containing iso8859-1 encoded paths without git-p4.pathEncoding' '\n+\tgit p4 clone --destination=\"$git\" //depot &&\n+\ttest_when_finished cleanup_git &&\n+\t(\n+\t\tcd \"$git\" &&\n+\t\tUTF8=\"$(printf \"$UTF8_ESCAPED\")\" &&\n+\t\techo $UTF8 >expect &&\n+\t\tgit -c core.quotepath=false ls-files >actual &&\n+\t\ttest_must_fail test_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'Clone repo containing iso8859-1 encoded paths with git-p4.pathEncoding' '\n+\n+\ttest_when_finished cleanup_git &&\n+\t(\n+\t\tcd \"$git\" &&\n+\t\tgit init . &&\n+\t\ttest_config git-p4.pathEncoding iso8859-1 &&\n+\t\tgit p4 clone --use-client-spec --destination=\"$git\" //depot &&\n+\t\tUTF8=\"$(printf \"$UTF8_ESCAPED\")\" &&\n+\t\techo $UTF8 >expect &&\n+\t\tgit -c core.quotepath=false ls-files >actual &&\n+\t\ttest_cmp expect actual &&\n+\t\tcat >expect <<-\\EOF &&\n+\t\tcontent123\n+\t\tEOF\n+\t\tcat $UTF8 >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'kill p4d' '\n+\tkill_p4d\n+'\n+\n+test_done\n-- \n1.9.5 (Apple Git-50.3)\n"},{"id":"269343","messageId":"xmqqa8t3l9km.fsf@gitster.mtv.corp.google.com","threadId":"40258","inReplyTo":"1441271647-67824-2-git-send-email-larsxschneider@gmail.com","subject":"Re: [PATCH v6] git-p4: add config git-p4.pathEncoding","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-09-03T17:03:05Z","receivedAt":"2015-09-03T17:03:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"larsxschneider@gmail.com writes:\n\n> +test_expect_success 'Clone repo containing iso8859-1 encoded paths without git-p4.pathEncoding' '\n> +\tgit p4 clone --destination=\"$git\" //depot &&\n> +\ttest_when_finished cleanup_git &&\n> +\t(\n> +\t\tcd \"$git\" &&\n> +\t\tUTF8=\"$(printf \"$UTF8_ESCAPED\")\" &&\n> +\t\techo $UTF8 >expect &&\n> +\t\tgit -c core.quotepath=false ls-files >actual &&\n> +\t\ttest_must_fail test_cmp expect actual\n\nI am not sure what this test wants to do.  It is not inconceivable\nthat future versions of \"git p4 clone\" becomes more intelligent to\ndetect the need for git-p4.pathEncoding and set it, so the only\neffect to insist the comparison fails is to block future advance in\nthat direction.\n\nBesides, \"test_must_fail test_cmp\" looks like a strange thing to\nsay.  \"! test_cmp expect actual\" perhaps.\n\nEven better, expect that \"expect\" and \"actual\" becomes the same, but\nmark the test itself to expect failure, to say \"it ought to work\nthis way in the ideal world, but we know the system currently does\nnot pass this test\".\n\nI'm tempted to suggest squashing the following in.  Thoughts?\n\n\n t/t9822-git-p4-path-encoding.sh | 17 ++++++++---------\n 1 file changed, 8 insertions(+), 9 deletions(-)\n\ndiff --git a/t/t9822-git-p4-path-encoding.sh b/t/t9822-git-p4-path-encoding.sh\nindex e507ad7..2d652d89 100755\n--- a/t/t9822-git-p4-path-encoding.sh\n+++ b/t/t9822-git-p4-path-encoding.sh\n@@ -21,15 +21,15 @@ test_expect_success 'Create a repo containing iso8859-1 encoded paths' '\n \t)\n '\n \n-test_expect_success 'Clone repo containing iso8859-1 encoded paths without git-p4.pathEncoding' '\n+test_expect_failure 'Clone auto-detects depot with iso8859-1 paths' '\n \tgit p4 clone --destination=\"$git\" //depot &&\n \ttest_when_finished cleanup_git &&\n \t(\n \t\tcd \"$git\" &&\n \t\tUTF8=\"$(printf \"$UTF8_ESCAPED\")\" &&\n-\t\techo $UTF8 >expect &&\n+\t\techo \"$UTF8\" >expect &&\n \t\tgit -c core.quotepath=false ls-files >actual &&\n-\t\ttest_must_fail test_cmp expect actual\n+\t\ttest_cmp expect actual\n \t)\n '\n \n@@ -39,16 +39,15 @@ test_expect_success 'Clone repo containing iso8859-1 encoded paths with git-p4.p\n \t(\n \t\tcd \"$git\" &&\n \t\tgit init . &&\n-\t\ttest_config git-p4.pathEncoding iso8859-1 &&\n+\t\tgit config git-p4.pathEncoding iso8859-1 &&\n \t\tgit p4 clone --use-client-spec --destination=\"$git\" //depot &&\n \t\tUTF8=\"$(printf \"$UTF8_ESCAPED\")\" &&\n-\t\techo $UTF8 >expect &&\n+\t\techo \"$UTF8\" >expect &&\n \t\tgit -c core.quotepath=false ls-files >actual &&\n \t\ttest_cmp expect actual &&\n-\t\tcat >expect <<-\\EOF &&\n-\t\tcontent123\n-\t\tEOF\n-\t\tcat $UTF8 >actual &&\n+\n+\t\techo content123 >expect &&\n+\t\tcat \"$UTF8\" >actual &&\n \t\ttest_cmp expect actual\n \t)\n '\n"},{"id":"269346","messageId":"9986CCDA-DB0B-4D7E-B758-668A46010188@gmail.com","threadId":"40258","inReplyTo":"xmqqa8t3l9km.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v6] git-p4: add config git-p4.pathEncoding","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-09-03T17:24:52Z","receivedAt":"2015-09-03T17:24:52Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\nOn 03 Sep 2015, at 19:03, Junio C Hamano <gitster@pobox.com> wrote:\n\n> larsxschneider@gmail.com writes:\n> \n>> +test_expect_success 'Clone repo containing iso8859-1 encoded paths without git-p4.pathEncoding' '\n>> +\tgit p4 clone --destination=\"$git\" //depot &&\n>> +\ttest_when_finished cleanup_git &&\n>> +\t(\n>> +\t\tcd \"$git\" &&\n>> +\t\tUTF8=\"$(printf \"$UTF8_ESCAPED\")\" &&\n>> +\t\techo $UTF8 >expect &&\n>> +\t\tgit -c core.quotepath=false ls-files >actual &&\n>> +\t\ttest_must_fail test_cmp expect actual\n> \n> I am not sure what this test wants to do.  It is not inconceivable\n> that future versions of \"git p4 clone\" becomes more intelligent to\n> detect the need for git-p4.pathEncoding and set it, so the only\n> effect to insist the comparison fails is to block future advance in\n> that direction.\n> \n> Besides, \"test_must_fail test_cmp\" looks like a strange thing to\n> say.  \"! test_cmp expect actual\" perhaps.\n> \n> Even better, expect that \"expect\" and \"actual\" becomes the same, but\n> mark the test itself to expect failure, to say \"it ought to work\n> this way in the ideal world, but we know the system currently does\n> not pass this test\".\n> \n> I'm tempted to suggest squashing the following in.  Thoughts?\nOK. The diff looks good to me. For some reason I can’t apply the patch though. git patch gives me \"fatal: corrupt patch at line 10”. Any idea? (I might do something stupid because I am not used to patches…)\n\nThanks,\nLars\n\n> \n> \n> t/t9822-git-p4-path-encoding.sh | 17 ++++++++---------\n> 1 file changed, 8 insertions(+), 9 deletions(-)\n> \n> diff --git a/t/t9822-git-p4-path-encoding.sh b/t/t9822-git-p4-path-encoding.sh\n> index e507ad7..2d652d89 100755\n> --- a/t/t9822-git-p4-path-encoding.sh\n> +++ b/t/t9822-git-p4-path-encoding.sh\n> @@ -21,15 +21,15 @@ test_expect_success 'Create a repo containing iso8859-1 encoded paths' '\n> \t)\n> '\n> \n> -test_expect_success 'Clone repo containing iso8859-1 encoded paths without git-p4.pathEncoding' '\n> +test_expect_failure 'Clone auto-detects depot with iso8859-1 paths' '\n> \tgit p4 clone --destination=\"$git\" //depot &&\n> \ttest_when_finished cleanup_git &&\n> \t(\n> \t\tcd \"$git\" &&\n> \t\tUTF8=\"$(printf \"$UTF8_ESCAPED\")\" &&\n> -\t\techo $UTF8 >expect &&\n> +\t\techo \"$UTF8\" >expect &&\n> \t\tgit -c core.quotepath=false ls-files >actual &&\n> -\t\ttest_must_fail test_cmp expect actual\n> +\t\ttest_cmp expect actual\n> \t)\n> '\n> \n> @@ -39,16 +39,15 @@ test_expect_success 'Clone repo containing iso8859-1 encoded paths with git-p4.p\n> \t(\n> \t\tcd \"$git\" &&\n> \t\tgit init . &&\n> -\t\ttest_config git-p4.pathEncoding iso8859-1 &&\n> +\t\tgit config git-p4.pathEncoding iso8859-1 &&\n> \t\tgit p4 clone --use-client-spec --destination=\"$git\" //depot &&\n> \t\tUTF8=\"$(printf \"$UTF8_ESCAPED\")\" &&\n> -\t\techo $UTF8 >expect &&\n> +\t\techo \"$UTF8\" >expect &&\n> \t\tgit -c core.quotepath=false ls-files >actual &&\n> \t\ttest_cmp expect actual &&\n> -\t\tcat >expect <<-\\EOF &&\n> -\t\tcontent123\n> -\t\tEOF\n> -\t\tcat $UTF8 >actual &&\n> +\n> +\t\techo content123 >expect &&\n> +\t\tcat \"$UTF8\" >actual &&\n> \t\ttest_cmp expect actual\n> \t)\n> '\n"},{"id":"269348","messageId":"xmqqr3mfjrib.fsf@gitster.mtv.corp.google.com","threadId":"40258","inReplyTo":"9986CCDA-DB0B-4D7E-B758-668A46010188@gmail.com","subject":"Re: [PATCH v6] git-p4: add config git-p4.pathEncoding","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-09-03T18:18:36Z","receivedAt":"2015-09-03T18:18:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lars Schneider <larsxschneider@gmail.com> writes:\n\n> On 03 Sep 2015, at 19:03, Junio C Hamano <gitster@pobox.com> wrote:\n>\n>> I'm tempted to suggest squashing the following in.  Thoughts?\n>\n> OK. The diff looks good to me. For some reason I can’t apply the\n> patch though. git patch gives me \"fatal: corrupt patch at line\n> 10”. Any idea? (I might do something stupid because I am not used to\n> patches…)\n\nSorry, but no idea.  I just saved the message I see on the list to a\nfile and did \"git am\" myself and didn't find anything.\n\nLine 9 of the patch is a removal of your overlong test_expect_success\nline, so if that were somehow line-wrapped by something between the\nlist and your \"git apply\" (e.g. with your MUA), it is very probable\nthat \"git apply\" would see there is something funny on line 10, but\nline 5 (a hunk header \"@@ -21,15 ...\") is also rather long, so...\n"}]}