{"thread":{"id":"40249","subject":"[PATCH v4] git-p4: add config git-p4.pathEncoding","startedAt":"2015-09-02T14:57:32Z","lastAt":"2015-09-02T18:13:20Z","messageCount":4,"participants":["larsxschneider@gmail.com","Junio C Hamano","Eric Sunshine"],"isPatch":true,"patchVersion":4,"patchTotal":null},"messages":[{"id":"269203","messageId":"1441205853-52065-1-git-send-email-larsxschneider@gmail.com","threadId":"40249","inReplyTo":null,"subject":"[PATCH v4] git-p4: add config git-p4.pathEncoding","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-09-02T14:57:32Z","receivedAt":"2015-09-02T14:57:32Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider <larsxschneider@gmail.com>\n\nDiff to v3:\n* add proper commit message\n* remove command line option \"--path-encoding\" and add config \"git-p4.pathEncoding\"\n* change TC number to 9822\n* escape UTF-8 characters in TC\n* change test encoding used in TC from cp1251 to ISO-8859-1\n* use static ISO-8859-1 encoded string in TC\n* check content of test file used in TC\n* shorten core.quotepath usage\n\nThanks to Torsten and Junio for feedback!\n\nCheers,\nLars\n\nLars Schneider (1):\n  git-p4: add config git-p4.pathEncoding\n\n Documentation/git-p4.txt        |  7 +++++\n git-p4.py                       |  3 ++\n t/t9822-git-p4-path-encoding.sh | 65 +++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 75 insertions(+)\n create mode 100755 t/t9822-git-p4-path-encoding.sh\n\n--\n1.9.5 (Apple Git-50.3)\n"},{"id":"269204","messageId":"1441205853-52065-2-git-send-email-larsxschneider@gmail.com","threadId":"40249","inReplyTo":"1441205853-52065-1-git-send-email-larsxschneider@gmail.com","subject":"[PATCH v4] git-p4: add config git-p4.pathEncoding","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-09-02T14:57:33Z","receivedAt":"2015-09-02T14:57:33Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider <larsxschneider@gmail.com>\n\nPerforce keeps the encoding of a path as given by the originating OS.\nGit expects paths encoded as UTF-8. Add a config to tell git-p4 what\nencoding Perforce had used for the paths. This encoding is used to\ntranscode the paths to UTF-8. As an example, Perforce on my Windows\nbox uses “cp1252” to encode path names.\n\nSigned-off-by: Lars Schneider <larsxschneider@gmail.com>\n---\n Documentation/git-p4.txt        |  7 +++++\n git-p4.py                       |  3 ++\n t/t9822-git-p4-path-encoding.sh | 65 +++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 75 insertions(+)\n create mode 100755 t/t9822-git-p4-path-encoding.sh\n\ndiff --git a/Documentation/git-p4.txt b/Documentation/git-p4.txt\nindex 82aa5d6..bf3adf9 100644\n--- a/Documentation/git-p4.txt\n+++ b/Documentation/git-p4.txt\n@@ -510,6 +510,13 @@ git-p4.useClientSpec::\n \toption '--use-client-spec'.  See the \"CLIENT SPEC\" section above.\n \tThis variable is a boolean, not the name of a p4 client.\n \n+git-p4.pathEncoding::\n+\tPerforce keeps the encoding of a path as given by the originating OS.\n+\tGit expects paths encoded as UTF-8. Use this config to tell git-p4 \n+\twhat encoding Perforce had used for the paths. This encoding is used\n+\tto transcode the paths to UTF-8. As an example, Perforce on my Windows\n+\tbox uses “cp1252” to encode path names.\n+\n Submit variables\n ~~~~~~~~~~~~~~~~\n git-p4.detectRenames::\ndiff --git a/git-p4.py b/git-p4.py\nindex 073f87b..706fcdc 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -2213,6 +2213,9 @@ class P4Sync(Command, P4UserMap):\n             text = regexp.sub(r'$\\1$', text)\n             contents = [ text ]\n \n+        if gitConfig(\"git-p4.pathEncoding\"):\n+            relPath = relPath.decode(gitConfig(\"git-p4.pathEncoding\")).encode('utf8', 'replace')\n+\n         self.gitStream.write(\"M %s inline %s\\n\" % (git_mode, relPath))\n \n         # total length...\ndiff --git a/t/t9822-git-p4-path-encoding.sh b/t/t9822-git-p4-path-encoding.sh\nnew file mode 100755\nindex 0000000..3a1779a\n--- /dev/null\n+++ b/t/t9822-git-p4-path-encoding.sh\n@@ -0,0 +1,65 @@\n+#!/bin/sh\n+\n+test_description='Clone repositories with non ASCII paths'\n+\n+. ./lib-git-p4.sh\n+\n+UTF8_ESCAPED=\"a-\\303\\244_o-\\303\\266_u-\\303\\274.txt\"\n+ISO8859_ESCAPED=\"\\141\\55\\344\\137\\157\\55\\366\\137\\165\\55\\374\\56\\164\\170\\164\"\n+\n+# You can generate the ISO8859_ESCAPED with the following command:\n+# printf \"$UTF8_ESCAPED\" | \\\n+# iconv -f utf-8 -t iso8859-1 | \\\n+# xxd -ps -u -c 1 | xargs bash -c 'for v; do echo \"ibase=16; obase=8; $v\" | bc; done' bash | \\\n+# tr \"\\n\" \"\\\\\"\n+\n+test_expect_success 'start p4d' '\n+\tstart_p4d\n+'\n+\n+test_expect_success 'Create a repo containing iso8859-1 encoded paths' '\n+\tcd \"$cli\" &&\n+\n+\tISO8859=\"$(printf \"$ISO8859_ESCAPED\")\" &&\n+\techo content123 >\"$ISO8859\" &&\n+\tp4 add \"$ISO8859\" &&\n+\tp4 submit -d \"test commit\"\n+'\n+\n+test_expect_success 'Clone repo containing iso8859-1 encoded paths without git-p4.pathEncoding' '\n+\tgit p4 clone --destination=\"$git\" //depot &&\n+\ttest_when_finished cleanup_git &&\n+\t(\n+\t\tcd \"$git\" &&\n+\t\tUTF8=\"$(printf \"$UTF8_ESCAPED\")\" &&\n+\t\techo $UTF8 >expect &&\n+\t\tgit -c core.quotepath=false ls-files >actual &&\n+\t\ttest_must_fail test_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'Clone repo containing iso8859-1 encoded paths with git-p4.pathEncoding' '\n+\n+\ttest_when_finished cleanup_git &&\n+\t(\n+\t\tcd \"$git\" &&\n+\t\tgit init . &&\n+\t\tgit config git-p4.pathEncoding iso8859-1 &&\n+\t\tgit p4 clone --use-client-spec --destination=\"$git\" //depot &&\n+\t\tUTF8=\"$(printf \"$UTF8_ESCAPED\")\" &&\n+\t\techo $UTF8 >expect &&\n+\t\tgit -c core.quotepath=false ls-files >actual &&\n+\t\ttest_cmp expect actual &&\n+\t\tcat >expect <<-\\EOF &&\n+\t\tcontent123\n+\t\tEOF\n+\t\tcat $UTF8 >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'kill p4d' '\n+\tkill_p4d\n+'\n+\n+test_done\n-- \n1.9.5 (Apple Git-50.3)\n"},{"id":"269218","messageId":"xmqqtwrcokaz.fsf@gitster.mtv.corp.google.com","threadId":"40249","inReplyTo":"1441205853-52065-2-git-send-email-larsxschneider@gmail.com","subject":"Re: [PATCH v4] git-p4: add config git-p4.pathEncoding","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-09-02T16:30:44Z","receivedAt":"2015-09-02T16:30:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"larsxschneider@gmail.com writes:\n\n> From: Lars Schneider <larsxschneider@gmail.com>\n>\n> Perforce keeps the encoding of a path as given by the originating OS.\n> Git expects paths encoded as UTF-8. Add a config to tell git-p4 what\n> encoding Perforce had used for the paths. This encoding is used to\n> transcode the paths to UTF-8. As an example, Perforce on my Windows\n> box uses “cp1252” to encode path names.\n>\n> Signed-off-by: Lars Schneider <larsxschneider@gmail.com>\n> ---\n\nThanks.\n\n> +git-p4.pathEncoding::\n> +\tPerforce keeps the encoding of a path as given by the originating OS.\n> +\tGit expects paths encoded as UTF-8. Use this config to tell git-p4 \n> +\twhat encoding Perforce had used for the paths. This encoding is used\n> +\tto transcode the paths to UTF-8. As an example, Perforce on my Windows\n> +\tbox uses “cp1252” to encode path names.\n> +\n\nThe log message is sort of personal statement, but \"my Windows\"\nsmells out of place here, as there is no clear writer in the\ndocumentation.  Perhaps rephrase it to:\n\n    For example, Perforce often uses \"cp1252\" to encode path names\n    on Windows box.\n\nor something?\n\n> diff --git a/t/t9822-git-p4-path-encoding.sh b/t/t9822-git-p4-path-encoding.sh\n> new file mode 100755\n> index 0000000..3a1779a\n> --- /dev/null\n> +++ b/t/t9822-git-p4-path-encoding.sh\n> @@ -0,0 +1,65 @@\n> +#!/bin/sh\n> +\n> +test_description='Clone repositories with non ASCII paths'\n> +\n> +. ./lib-git-p4.sh\n> +\n> +UTF8_ESCAPED=\"a-\\303\\244_o-\\303\\266_u-\\303\\274.txt\"\n> +ISO8859_ESCAPED=\"\\141\\55\\344\\137\\157\\55\\366\\137\\165\\55\\374\\56\\164\\170\\164\"\n> +\n> +# You can generate the ISO8859_ESCAPED with the following command:\n\nPlease don't.  You don't want to encode \".txt\" and other things that\nare ASCII, which you didn't encode in the original UTF8_ESCAPED.\n\n> +# printf \"$UTF8_ESCAPED\" | \\\n> +# iconv -f utf-8 -t iso8859-1 | \\\n> +# xxd -ps -u -c 1 | xargs bash -c 'for v; do echo \"ibase=16; obase=8; $v\" | bc; done' bash | \\\n> +# tr \"\\n\" \"\\\\\"\n\nBesides, you somehow came up with UTF8_ESCAPED with a procedure that\nis not documented (and it does avoid hiding obvious things like\n\".txt\" in the backslashes).  I do not think it is necessary or even\na good idea to give a procedure that is not very portable (xxd?\nbash?) and does not produce what we want to see, only for ISO8859.\n"},{"id":"269227","messageId":"CAPig+cSsZnoLNw+vBfHF7h7i=aDhNz0fD_Xd3zRv3RybcGKG2Q@mail.gmail.com","threadId":"40249","inReplyTo":"1441205853-52065-2-git-send-email-larsxschneider@gmail.com","subject":"Re: [PATCH v4] git-p4: add config git-p4.pathEncoding","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2015-09-02T18:13:20Z","receivedAt":"2015-09-02T18:13:20Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Wednesday, September 2, 2015, <larsxschneider@gmail.com> wrote:\n> Perforce keeps the encoding of a path as given by the originating OS.\n> Git expects paths encoded as UTF-8. Add a config to tell git-p4 what\n> encoding Perforce had used for the paths. This encoding is used to\n> transcode the paths to UTF-8. As an example, Perforce on my Windows\n> box uses “cp1252” to encode path names.\n>\n> Signed-off-by: Lars Schneider <larsxschneider@gmail.com>\n> ---\n> +test_expect_success 'start p4d' '\n> +       start_p4d\n> +'\n> +\n> +test_expect_success 'Create a repo containing iso8859-1 encoded paths' '\n> +       cd \"$cli\" &&\n\nTorsten, I think, mentioned previously that the 'cd' and code\nfollowing should be wrapped in a subshell.\n\n> +       ISO8859=\"$(printf \"$ISO8859_ESCAPED\")\" &&\n> +       echo content123 >\"$ISO8859\" &&\n> +       p4 add \"$ISO8859\" &&\n> +       p4 submit -d \"test commit\"\n\nAnd, it's odd that this test doesn't \"cd ..\", which means that\nsubsequent tests are running in directory $cli. Is that intentional?\nIf so, it probably ought to be done in a more explicit and clear\nfashion, perhaps by having each test do \"cd $cli/$git\" rather than\njust \"cd $git\".\n\n> +'\n> +\n> +test_expect_success 'Clone repo containing iso8859-1 encoded paths without git-p4.pathEncoding' '\n> +       git p4 clone --destination=\"$git\" //depot &&\n> +       test_when_finished cleanup_git &&\n> +       (\n> +               cd \"$git\" &&\n> +               UTF8=\"$(printf \"$UTF8_ESCAPED\")\" &&\n> +               echo $UTF8 >expect &&\n> +               git -c core.quotepath=false ls-files >actual &&\n> +               test_must_fail test_cmp expect actual\n> +       )\n> +'\n> +\n> +test_expect_success 'Clone repo containing iso8859-1 encoded paths with git-p4.pathEncoding' '\n> +\n> +       test_when_finished cleanup_git &&\n> +       (\n> +               cd \"$git\" &&\n> +               git init . &&\n> +               git config git-p4.pathEncoding iso8859-1 &&\n> +               git p4 clone --use-client-spec --destination=\"$git\" //depot &&\n> +               UTF8=\"$(printf \"$UTF8_ESCAPED\")\" &&\n> +               echo $UTF8 >expect &&\n> +               git -c core.quotepath=false ls-files >actual &&\n> +               test_cmp expect actual &&\n> +               cat >expect <<-\\EOF &&\n> +               content123\n> +               EOF\n> +               cat $UTF8 >actual &&\n> +               test_cmp expect actual\n> +       )\n> +'\n> +\n> +test_expect_success 'kill p4d' '\n> +       kill_p4d\n> +'\n> +\n> +test_done\n> --\n> 1.9.5 (Apple Git-50.3)\n"}]}