{"thread":{"id":"40234","subject":"[PATCH v3] git-p4: add \"--path-encoding\" option","startedAt":"2015-08-31T22:10:25Z","lastAt":"2015-09-01T19:06:27Z","messageCount":13,"participants":["larsxschneider@gmail.com","Junio C Hamano","Torsten Bögershausen","Lars Schneider"],"isPatch":true,"patchVersion":3,"patchTotal":null},"messages":[{"id":"269062","messageId":"1441059026-66814-1-git-send-email-larsxschneider@gmail.com","threadId":"40234","inReplyTo":null,"subject":"[PATCH v3] git-p4: add \"--path-encoding\" option","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-08-31T22:10:25Z","receivedAt":"2015-08-31T22:10:25Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nDiff to v2:\n* remove comment\n* use \"core.quotepath false\" to simplify test case\n\nUnit test passed on OS X and Linux. The unit test did also pass on v2 as \"printf $(git ls-files) >actual\" removes the LF.\n\nCheers,\nLars\n\nLars Schneider (1):\n git-p4: add \"--path-encoding\" option\n\n Documentation/git-p4.txt | 5 +++++\n git-p4.py | 6 ++++++\n t/t9821-git-p4-path-encoding.sh | 38 ++++++++++++++++++++++++++++++++++++++\n 3 files changed, 49 insertions(+)\n create mode 100755 t/t9821-git-p4-path-encoding.sh\n\n--\n1.9.5 (Apple Git-50.3)\n"},{"id":"269063","messageId":"1441059026-66814-2-git-send-email-larsxschneider@gmail.com","threadId":"40234","inReplyTo":"1441059026-66814-1-git-send-email-larsxschneider@gmail.com","subject":"[PATCH v3] git-p4: add \"--path-encoding\" option","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-08-31T22:10:26Z","receivedAt":"2015-08-31T22:10:26Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nSigned-off-by: Lars Schneider \n---\n Documentation/git-p4.txt | 5 +++++\n git-p4.py | 6 ++++++\n t/t9821-git-p4-path-encoding.sh | 38 ++++++++++++++++++++++++++++++++++++++\n 3 files changed, 49 insertions(+)\n create mode 100755 t/t9821-git-p4-path-encoding.sh\n\ndiff --git a/Documentation/git-p4.txt b/Documentation/git-p4.txt\nindex 82aa5d6..14bb79c 100644\n--- a/Documentation/git-p4.txt\n+++ b/Documentation/git-p4.txt\n@@ -252,6 +252,11 @@ Git repository:\n \tUse a client spec to find the list of interesting files in p4.\n \tSee the \"CLIENT SPEC\" section below.\n \n+--path-encoding ::\n+\tThe encoding to use when reading p4 client paths. With this option\n+\tnon ASCII paths are properly stored in Git. For example, the encoding\n+\t'cp1252' is often used on Windows systems.\n+\n -/ ::\n \tExclude selected depot paths when cloning or syncing.\n \ndiff --git a/git-p4.py b/git-p4.py\nindex 073f87b..2b3bfc4 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -1981,6 +1981,8 @@ class P4Sync(Command, P4UserMap):\n optparse.make_option(\"--silent\", dest=\"silent\", action=\"store_true\"),\n optparse.make_option(\"--detect-labels\", dest=\"detectLabels\", action=\"store_true\"),\n optparse.make_option(\"--import-labels\", dest=\"importLabels\", action=\"store_true\"),\n+ optparse.make_option(\"--path-encoding\", dest=\"pathEncoding\", type=\"string\",\n+ help=\"Encoding to use for paths\"),\n optparse.make_option(\"--import-local\", dest=\"importIntoRemotes\", action=\"store_false\",\n help=\"Import into refs/heads/ , not refs/remotes\"),\n optparse.make_option(\"--max-changes\", dest=\"maxChanges\",\n@@ -2025,6 +2027,7 @@ class P4Sync(Command, P4UserMap):\n self.clientSpecDirs = None\n self.tempBranches = []\n self.tempBranchLocation = \"git-p4-tmp\"\n+ self.pathEncoding = None\n \n if gitConfig(\"git-p4.syncFromOrigin\") == \"false\":\n self.syncWithOrigin = False\n@@ -2213,6 +2216,9 @@ class P4Sync(Command, P4UserMap):\n text = regexp.sub(r'$\\1$', text)\n contents = [ text ]\n \n+ if self.pathEncoding:\n+ relPath = relPath.decode(self.pathEncoding).encode('utf8', 'replace')\n+\n self.gitStream.write(\"M %s inline %s\\n\" % (git_mode, relPath))\n \n # total length...\ndiff --git a/t/t9821-git-p4-path-encoding.sh b/t/t9821-git-p4-path-encoding.sh\nnew file mode 100755\nindex 0000000..1626fc5\n--- /dev/null\n+++ b/t/t9821-git-p4-path-encoding.sh\n@@ -0,0 +1,38 @@\n+#!/bin/sh\n+\n+test_description='Clone repositories with non ASCII paths'\n+\n+. ./lib-git-p4.sh\n+\n+UTF8_ESCAPED=\"a-\\303\\244_o-\\303\\266_u-\\303\\274.txt\"\n+\n+test_expect_success 'start p4d' '\n+\tstart_p4d\n+'\n+\n+test_expect_success 'Create a repo containing iso8859-1 encoded paths' '\n+\tcd \"$cli\" &&\n+\n+\tISO8859=\"$(printf \"$UTF8_ESCAPED\" | iconv -f utf-8 -t iso8859-1)\" &&\n+\t>\"$ISO8859\" &&\n+\tp4 add \"$ISO8859\" &&\n+\tp4 submit -d \"test commit\"\n+'\n+\n+test_expect_success 'Clone repo containing iso8859-1 encoded paths' '\n+\tgit p4 clone --destination=\"$git\" --path-encoding=iso8859-1 //depot &&\n+\ttest_when_finished cleanup_git &&\n+\t(\n+\t\tcd \"$git\" &&\n+\t\tprintf \"$UTF8_ESCAPED\\n\" >expect &&\n+\t\ttest_config core.quotepath false &&\n+\t\tgit ls-files >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'kill p4d' '\n+\tkill_p4d\n+'\n+\n+test_done\n-- \n1.9.5 (Apple Git-50.3)\n"},{"id":"269065","messageId":"xmqqk2sbt5kv.fsf@gitster.mtv.corp.google.com","threadId":"40234","inReplyTo":"1441059026-66814-2-git-send-email-larsxschneider@gmail.com","subject":"Re: [PATCH v3] git-p4: add \"--path-encoding\" option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-08-31T23:13:04Z","receivedAt":"2015-08-31T23:13:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"larsxschneider@gmail.com writes:\n\n> From: Lars Schneider \n>\n\nHere is a space for you to describe what it does and why it is a\ngood idea to have it.\n\n> Signed-off-by: Lars Schneider \n> ---\n> Documentation/git-p4.txt | 5 +++++\n> git-p4.py | 6 ++++++\n> t/t9821-git-p4-path-encoding.sh | 38 ++++++++++++++++++++++++++++++++++++++\n\nI'll move this to 9822, as 9821 is taken by another git-p4 test,\nwhile queuing.\n\n> diff --git a/Documentation/git-p4.txt b/Documentation/git-p4.txt\n> index 82aa5d6..14bb79c 100644\n> --- a/Documentation/git-p4.txt\n> +++ b/Documentation/git-p4.txt\n> @@ -252,6 +252,11 @@ Git repository:\n> \tUse a client spec to find the list of interesting files in p4.\n> \tSee the \"CLIENT SPEC\" section below.\n> \n> +--path-encoding ::\n> +\tThe encoding to use when reading p4 client paths. With this option\n> +\tnon ASCII paths are properly stored in Git. For example, the encoding\n> +\t'cp1252' is often used on Windows systems.\n> +\n\nIs this something that needs to be consistently given while\ninteracting with the same P4 depot (in which case you may want to\nallow this to be given only once, record the value in the config and\nnever allow it to be updated once it is set, or something)?\n\nThanks.\n"},{"id":"269079","messageId":"55E52BA7.6020303@web.de","threadId":"40234","inReplyTo":"1441059026-66814-2-git-send-email-larsxschneider@gmail.com","subject":"Re: [PATCH v3] git-p4: add \"--path-encoding\" option","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2015-09-01T04:37:59Z","receivedAt":"2015-09-01T04:37:59Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"\n\nOn 01/09/15 00:10, larsxschneider@gmail.com wrote:\n> From: Lars Schneider \n>\n> Signed-off-by: Lars Schneider \n> ---\n> Documentation/git-p4.txt | 5 +++++\n> git-p4.py | 6 ++++++\n> t/t9821-git-p4-path-encoding.sh | 38 ++++++++++++++++++++++++++++++++++++++\n> 3 files changed, 49 insertions(+)\n> create mode 100755 t/t9821-git-p4-path-encoding.sh\n>\n> diff --git a/Documentation/git-p4.txt b/Documentation/git-p4.txt\n> index 82aa5d6..14bb79c 100644\n> --- a/Documentation/git-p4.txt\n> +++ b/Documentation/git-p4.txt\n> @@ -252,6 +252,11 @@ Git repository:\n> \tUse a client spec to find the list of interesting files in p4.\n> \tSee the \"CLIENT SPEC\" section below.\n> \n> +--path-encoding ::\n> +\tThe encoding to use when reading p4 client paths. With this option\n> +\tnon ASCII paths are properly stored in Git. For example, the encoding\n> +\t'cp1252' is often used on Windows systems.\n> +\n> -/ ::\n> \tExclude selected depot paths when cloning or syncing.\n> \n> diff --git a/git-p4.py b/git-p4.py\n> index 073f87b..2b3bfc4 100755\n> --- a/git-p4.py\n> +++ b/git-p4.py\n> @@ -1981,6 +1981,8 @@ class P4Sync(Command, P4UserMap):\n> optparse.make_option(\"--silent\", dest=\"silent\", action=\"store_true\"),\n> optparse.make_option(\"--detect-labels\", dest=\"detectLabels\", action=\"store_true\"),\n> optparse.make_option(\"--import-labels\", dest=\"importLabels\", action=\"store_true\"),\n> + optparse.make_option(\"--path-encoding\", dest=\"pathEncoding\", type=\"string\",\n> + help=\"Encoding to use for paths\"),\n> optparse.make_option(\"--import-local\", dest=\"importIntoRemotes\", action=\"store_false\",\n> help=\"Import into refs/heads/ , not refs/remotes\"),\n> optparse.make_option(\"--max-changes\", dest=\"maxChanges\",\n> @@ -2025,6 +2027,7 @@ class P4Sync(Command, P4UserMap):\n> self.clientSpecDirs = None\n> self.tempBranches = []\n> self.tempBranchLocation = \"git-p4-tmp\"\n> + self.pathEncoding = None\n> \n> if gitConfig(\"git-p4.syncFromOrigin\") == \"false\":\n> self.syncWithOrigin = False\n> @@ -2213,6 +2216,9 @@ class P4Sync(Command, P4UserMap):\n> text = regexp.sub(r'$\\1$', text)\n> contents = [ text ]\n> \n> + if self.pathEncoding:\n> + relPath = relPath.decode(self.pathEncoding).encode('utf8', 'replace')\n> +\n> self.gitStream.write(\"M %s inline %s\\n\" % (git_mode, relPath))\n> \n> # total length...\n> diff --git a/t/t9821-git-p4-path-encoding.sh b/t/t9821-git-p4-path-encoding.sh\n> new file mode 100755\n> index 0000000..1626fc5\n> --- /dev/null\n> +++ b/t/t9821-git-p4-path-encoding.sh\n> @@ -0,0 +1,38 @@\n> +#!/bin/sh\n> +\n> +test_description='Clone repositories with non ASCII paths'\n> +\n> +. ./lib-git-p4.sh\n> +\n> +UTF8_ESCAPED=\"a-\\303\\244_o-\\303\\266_u-\\303\\274.txt\"\n> +\n> +test_expect_success 'start p4d' '\n> +\tstart_p4d\n> +'\n> +\n> +test_expect_success 'Create a repo containing iso8859-1 encoded paths' '\n> +\tcd \"$cli\" &&\n> +\n> +\tISO8859=\"$(printf \"$UTF8_ESCAPED\" | iconv -f utf-8 -t iso8859-1)\" &&\n> +\t>\"$ISO8859\" &&\n> +\tp4 add \"$ISO8859\" &&\n> +\tp4 submit -d \"test commit\"\n> +'\nSorry for being persistant,\nbut you can't create files with names that are ISO-8859-1 encoded under Mac OS,\nwe end up like this:\n\n a-%E4_o-%F6_u-%FC.txt\n\n\n(And I'm still not convinced, that we need to call iconv each time we execute \nthe TC,\nfor a string that is always the same.\nThe string can be converted once, and embedded in the TC:\nThe following should work under Mac OS (but I don't have p4 to test it)\n\nISO8859_ESCAPED=\"a-\\303\\244_o-\\303\\266_u-\\303\\274.txt\"\n\nUTF8_ESCAPED=\"\\141\\055\\303\\203\\302\\244\\137\\157\\055\\303\\203\\302\\266\\137\\165\\055\\303\\203\\302\\274\\056\\164\\170\\164\"\n\nISO8859=$(printf \"$ISO8859_ESCAPED\")\n\n> +\n> +test_expect_success 'Clone repo containing iso8859-1 encoded paths' '\n> +\tgit p4 clone --destination=\"$git\" --path-encoding=iso8859-1 //depot &&\n> +\ttest_when_finished cleanup_git &&\n> +\t(\n> +\t\tcd \"$git\" &&\n> +\t\tprintf \"$UTF8_ESCAPED\\n\" >expect &&\n> +\t\ttest_config core.quotepath false &&\n> +\t\tgit ls-files >actual &&\n> +\t\ttest_cmp expect actual\n> +\t)\n> +'\n>\nThe ls-files can be written shorter (if we like short code)\n\n+\t\tgit -c core.quotepath=false ls-files >actual &&\n"},{"id":"269082","messageId":"55E5503E.2090101@web.de","threadId":"40234","inReplyTo":"55E52BA7.6020303@web.de","subject":"Fwd: Re: [PATCH v3] git-p4: add \"--path-encoding\" option","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2015-09-01T07:14:06Z","receivedAt":"2015-09-01T07:14:06Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"Sorry if this is possible re-sending\n\n\n-------- Forwarded Message --------\nSubject: \tRe: [PATCH v3] git-p4: add \"--path-encoding\" option\nDate: \tTue, 1 Sep 2015 06:37:59 +0200\nFrom: \tTorsten Bögershausen \nTo: \tlarsxschneider@gmail.com, git@vger.kernel.org\nCC: \tluke@diamand.org, gitster@pobox.com\n\n\n\nOn 01/09/15 00:10, larsxschneider@gmail.com wrote:\n> From: Lars Schneider \n>\n> Signed-off-by: Lars Schneider \n> ---\n> Documentation/git-p4.txt | 5 +++++\n> git-p4.py | 6 ++++++\n> t/t9821-git-p4-path-encoding.sh | 38 ++++++++++++++++++++++++++++++++++++++\n> 3 files changed, 49 insertions(+)\n> create mode 100755 t/t9821-git-p4-path-encoding.sh\n>\n> diff --git a/Documentation/git-p4.txt b/Documentation/git-p4.txt\n> index 82aa5d6..14bb79c 100644\n> --- a/Documentation/git-p4.txt\n> +++ b/Documentation/git-p4.txt\n> @@ -252,6 +252,11 @@ Git repository:\n> \tUse a client spec to find the list of interesting files in p4.\n> \tSee the \"CLIENT SPEC\" section below.\n>\n> +--path-encoding ::\n> +\tThe encoding to use when reading p4 client paths. With this option\n> +\tnon ASCII paths are properly stored in Git. For example, the encoding\n> +\t'cp1252' is often used on Windows systems.\n> +\n> -/ ::\n> \tExclude selected depot paths when cloning or syncing.\n>\n> diff --git a/git-p4.py b/git-p4.py\n> index 073f87b..2b3bfc4 100755\n> --- a/git-p4.py\n> +++ b/git-p4.py\n> @@ -1981,6 +1981,8 @@ class P4Sync(Command, P4UserMap):\n> optparse.make_option(\"--silent\", dest=\"silent\", action=\"store_true\"),\n> optparse.make_option(\"--detect-labels\", dest=\"detectLabels\", action=\"store_true\"),\n> optparse.make_option(\"--import-labels\", dest=\"importLabels\", action=\"store_true\"),\n> + optparse.make_option(\"--path-encoding\", dest=\"pathEncoding\", type=\"string\",\n> + help=\"Encoding to use for paths\"),\n> optparse.make_option(\"--import-local\", dest=\"importIntoRemotes\", action=\"store_false\",\n> help=\"Import into refs/heads/ , not refs/remotes\"),\n> optparse.make_option(\"--max-changes\", dest=\"maxChanges\",\n> @@ -2025,6 +2027,7 @@ class P4Sync(Command, P4UserMap):\n> self.clientSpecDirs = None\n> self.tempBranches = []\n> self.tempBranchLocation = \"git-p4-tmp\"\n> + self.pathEncoding = None\n>\n> if gitConfig(\"git-p4.syncFromOrigin\") == \"false\":\n> self.syncWithOrigin = False\n> @@ -2213,6 +2216,9 @@ class P4Sync(Command, P4UserMap):\n> text = regexp.sub(r'$\\1$', text)\n> contents = [ text ]\n>\n> + if self.pathEncoding:\n> + relPath = relPath.decode(self.pathEncoding).encode('utf8', 'replace')\n> +\n> self.gitStream.write(\"M %s inline %s\\n\" % (git_mode, relPath))\n>\n> # total length...\n> diff --git a/t/t9821-git-p4-path-encoding.sh b/t/t9821-git-p4-path-encoding.sh\n> new file mode 100755\n> index 0000000..1626fc5\n> --- /dev/null\n> +++ b/t/t9821-git-p4-path-encoding.sh\n> @@ -0,0 +1,38 @@\n> +#!/bin/sh\n> +\n> +test_description='Clone repositories with non ASCII paths'\n> +\n> +. ./lib-git-p4.sh\n> +\n> +UTF8_ESCAPED=\"a-\\303\\244_o-\\303\\266_u-\\303\\274.txt\"\n> +\n> +test_expect_success 'start p4d' '\n> +\tstart_p4d\n> +'\n> +\n> +test_expect_success 'Create a repo containing iso8859-1 encoded paths' '\n> +\tcd \"$cli\" &&\n> +\n> +\tISO8859=\"$(printf \"$UTF8_ESCAPED\" | iconv -f utf-8 -t iso8859-1)\" &&\n> +\t>\"$ISO8859\" &&\n> +\tp4 add \"$ISO8859\" &&\n> +\tp4 submit -d \"test commit\"\n> +'\nSorry for being persistant,\nbut you can't create files with names that are ISO-8859-1 encoded under Mac OS,\nwe end up like this:\n\n a-%E4_o-%F6_u-%FC.txt\n\n\n(And I'm still not convinced, that we need to call iconv each time we execute\nthe TC,\nfor a string that is always the same.\nThe string can be converted once, and embedded in the TC:\nThe following should work under Mac OS (but I don't have p4 to test it)\n\nISO8859_ESCAPED=\"a-\\303\\244_o-\\303\\266_u-\\303\\274.txt\"\n\nUTF8_ESCAPED=\"\\141\\055\\303\\203\\302\\244\\137\\157\\055\\303\\203\\302\\266\\137\\165\\055\\303\\203\\302\\274\\056\\164\\170\\164\"\n\nISO8859=$(printf \"$ISO8859_ESCAPED\")\n\n> +\n> +test_expect_success 'Clone repo containing iso8859-1 encoded paths' '\n> +\tgit p4 clone --destination=\"$git\" --path-encoding=iso8859-1 //depot &&\n> +\ttest_when_finished cleanup_git &&\n> +\t(\n> +\t\tcd \"$git\" &&\n> +\t\tprintf \"$UTF8_ESCAPED\\n\" >expect &&\n> +\t\ttest_config core.quotepath false &&\n> +\t\tgit ls-files >actual &&\n> +\t\ttest_cmp expect actual\n> +\t)\n> +'\n>\nThe ls-files can be written shorter (if we like short code)\n\n+\t\tgit -c core.quotepath=false ls-files >actual &&\n"},{"id":"269088","messageId":"6BD3F017-37E4-4677-BCCE-D84E7031B780@gmail.com","threadId":"40234","inReplyTo":"55E52BA7.6020303@web.de","subject":"Re: [PATCH v3] git-p4: add \"--path-encoding\" option","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-09-01T12:47:15Z","receivedAt":"2015-09-01T12:47:15Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\nOn 01 Sep 2015, at 06:37, Torsten Bögershausen wrote:\n\n> On 01/09/15 00:10, larsxschneider@gmail.com wrote:\n>> From: Lars Schneider \n>> \n>> Signed-off-by: Lars Schneider \n>> ---\n>> Documentation/git-p4.txt | 5 +++++\n>> git-p4.py | 6 ++++++\n>> t/t9821-git-p4-path-encoding.sh | 38 ++++++++++++++++++++++++++++++++++++++\n>> 3 files changed, 49 insertions(+)\n>> create mode 100755 t/t9821-git-p4-path-encoding.sh\n>> \n>> \n\n> \n>> +\n>> +test_expect_success 'Create a repo containing iso8859-1 encoded paths' '\n>> +\tcd \"$cli\" &&\n>> +\n>> +\tISO8859=\"$(printf \"$UTF8_ESCAPED\" | iconv -f utf-8 -t iso8859-1)\" &&\n>> +\t>\"$ISO8859\" &&\n>> +\tp4 add \"$ISO8859\" &&\n>> +\tp4 submit -d \"test commit\"\n>> +'\n> Sorry for being persistant,\n> but you can't create files with names that are ISO-8859-1 encoded under Mac OS,\n> we end up like this:\n> \n> a-%E4_o-%F6_u-%FC.txt\nYou are right. However, my goal is not to create a file with ISO-8859-1 characters in Mac OS. My goal is to create this file in Perforce and this approach seems to work.\n\n> \n> (And I'm still not convinced, that we need to call iconv each time we execute the TC,\n> for a string that is always the same.\n> The string can be converted once, and embedded in the TC:\n> The following should work under Mac OS (but I don't have p4 to test it)\n> \n> ISO8859_ESCAPED=\"a-\\303\\244_o-\\303\\266_u-\\303\\274.txt\"\n> \n> UTF8_ESCAPED=\"\\141\\055\\303\\203\\302\\244\\137\\157\\055\\303\\203\\302\\266\\137\\165\\055\\303\\203\\302\\274\\056\\164\\170\\164\"\n> \n> ISO8859=$(printf \"$ISO8859_ESCAPED”)\nOK. However, how about this?\n\nUTF8_ESCAPED=\"a-\\303\\244_o-\\303\\266_u-\\303\\274.txt\"\nISO8859_ESCAPED=\"\\141\\55\\344\\137\\157\\55\\366\\137\\165\\55\\374\\56\\164\\170\\164\"\n\n# You can generate the ISO8859_ESCAPED with the following command:\n# printf \"$UTF8_ESCAPED\" | \\\n# iconv -f utf-8 -t iso8859-1 | \\\n# xxd -ps -u -c 1 | xargs bash -c 'for v; do echo \"ibase=16; obase=8; $v\" | bc; done' bash | \\\n# tr \"\\n\" \"\\\\\"\n\n\n\n> \n>> +\n>> +test_expect_success 'Clone repo containing iso8859-1 encoded paths' '\n>> +\tgit p4 clone --destination=\"$git\" --path-encoding=iso8859-1 //depot &&\n>> +\ttest_when_finished cleanup_git &&\n>> +\t(\n>> +\t\tcd \"$git\" &&\n>> +\t\tprintf \"$UTF8_ESCAPED\\n\" >expect &&\n>> +\t\ttest_config core.quotepath false &&\n>> +\t\tgit ls-files >actual &&\n>> +\t\ttest_cmp expect actual\n>> +\t)\n>> +'\n>> \n> The ls-files can be written shorter (if we like short code)\n> \n> +\t\tgit -c core.quotepath=false ls-files >actual &&\nFixed. Thank you!\n"},{"id":"269091","messageId":"523D2C52-893E-4A54-9DD1-2DDBCEA29960@gmail.com","threadId":"40234","inReplyTo":"xmqqk2sbt5kv.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v3] git-p4: add \"--path-encoding\" option","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-09-01T13:42:39Z","receivedAt":"2015-09-01T13:42:39Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\nOn 01 Sep 2015, at 01:13, Junio C Hamano wrote:\n\n> larsxschneider@gmail.com writes:\n> \n>> From: Lars Schneider \n>> \n> \n> Here is a space for you to describe what it does and why it is a\n> good idea to have it.\nHow about this:\n\nPerforce keeps the encoding of a path as given by the originating OS. Git expects paths encoded as UTF-8. Add an option to tell git-p4 what encoding Perforce had used for the paths. This encoding is used to transcode the paths to UTF-8. As an example, Perforce on Windows uses “cp1252” to encode path names. \n\n\n> \n>> Signed-off-by: Lars Schneider \n>> ---\n>> Documentation/git-p4.txt | 5 +++++\n>> git-p4.py | 6 ++++++\n>> t/t9821-git-p4-path-encoding.sh | 38 ++++++++++++++++++++++++++++++++++++++\n> \n> I'll move this to 9822, as 9821 is taken by another git-p4 test,\n> while queuing.\nOK. I wasn’t sure how this is handled. Just for my understanding: As soon as a TC number is occupied in one of the official branches (master/next/pu/maint) then the next number should be used, right?\n\n> \n>> diff --git a/Documentation/git-p4.txt b/Documentation/git-p4.txt\n>> index 82aa5d6..14bb79c 100644\n>> --- a/Documentation/git-p4.txt\n>> +++ b/Documentation/git-p4.txt\n>> @@ -252,6 +252,11 @@ Git repository:\n>> \tUse a client spec to find the list of interesting files in p4.\n>> \tSee the \"CLIENT SPEC\" section below.\n>> \n>> +--path-encoding ::\n>> +\tThe encoding to use when reading p4 client paths. With this option\n>> +\tnon ASCII paths are properly stored in Git. For example, the encoding\n>> +\t'cp1252' is often used on Windows systems.\n>> +\n> \n> Is this something that needs to be consistently given while\n> interacting with the same P4 depot (in which case you may want to\n> allow this to be given only once, record the value in the config and\n> never allow it to be updated once it is set, or something)?\nGood idea! I will add this. However, I really wonder why nobody tripped over this rather obvious bug already.\n\n@Luke: Can you double check my fix?\n\n\nTwo general questions:\n\n(1) I saw that the path encoding fix is already on “pu”. What is more convenient for you? Do you want to get a [PATCH v4] with one commit including the changes mentioned above or do you want me to keep the v3 commit and add a v4 commit on top?\n\n(2) Is there a CI setup for Git somewhere accessible on the Internet? How about building and testing Git on Travis (https://travis-ci.org/git/git)?\n\nThanks,\nLars\n"},{"id":"269093","messageId":"55E5B798.40108@web.de","threadId":"40234","inReplyTo":"6BD3F017-37E4-4677-BCCE-D84E7031B780@gmail.com","subject":"Re: [PATCH v3] git-p4: add \"--path-encoding\" option","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2015-09-01T14:35:04Z","receivedAt":"2015-09-01T14:35:04Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 2015-09-01 14.47, Lars Schneider wrote:\n>>> +test_expect_success 'Create a repo containing iso8859-1 encoded paths' '\n>>> >> +\tcd \"$cli\" &&\n>>> >> +\n>>> >> +\tISO8859=\"$(printf \"$UTF8_ESCAPED\" | iconv -f utf-8 -t iso8859-1)\" &&\n>>> >> +\t>\"$ISO8859\" &&\n>>> >> +\tp4 add \"$ISO8859\" &&\n>>> >> +\tp4 submit -d \"test commit\"\n>>> >> +'\n>> > Sorry for being persistant,\n>> > but you can't create files with names that are ISO-8859-1 encoded under Mac OS,\n>> > we end up like this:\n>> > \n>> > a-%E4_o-%F6_u-%FC.txt\n> You are right. However, my goal is not to create a file with ISO-8859-1 characters in Mac OS. My goal is to create this file in Perforce and this approach seems to work.\n> \n>> > \nBut this line creates a file, doesn't it ?\n>\"$ISO8859\" &&\n\n(I just wonder how this works on you machine )\n\nAnd, may be, we could fill the file with some content, to be double-sure that\nthe file name conversion works with Perforce ?\n\nlike\necho content >\"$ISO8859\" &&\n\nand test the content later ?\n"},{"id":"269094","messageId":"3942C6D1-A758-4F50-9865-9D83C5CD9128@gmail.com","threadId":"40234","inReplyTo":"55E5B798.40108@web.de","subject":"Re: [PATCH v3] git-p4: add \"--path-encoding\" option","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-09-01T15:02:45Z","receivedAt":"2015-09-01T15:02:45Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\nOn 01 Sep 2015, at 16:35, Torsten Bögershausen wrote:\n\n> On 2015-09-01 14.47, Lars Schneider wrote:\n>>>> +test_expect_success 'Create a repo containing iso8859-1 encoded paths' '\n>>>>>> +\tcd \"$cli\" &&\n>>>>>> +\n>>>>>> +\tISO8859=\"$(printf \"$UTF8_ESCAPED\" | iconv -f utf-8 -t iso8859-1)\" &&\n>>>>>> +\t>\"$ISO8859\" &&\n>>>>>> +\tp4 add \"$ISO8859\" &&\n>>>>>> +\tp4 submit -d \"test commit\"\n>>>>>> +'\n>>>> Sorry for being persistant,\n>>>> but you can't create files with names that are ISO-8859-1 encoded under Mac OS,\n>>>> we end up like this:\n>>>> \n>>>> a-%E4_o-%F6_u-%FC.txt\n>> You are right. However, my goal is not to create a file with ISO-8859-1 characters in Mac OS. My goal is to create this file in Perforce and this approach seems to work.\n>> \n>>>> \n> But this line creates a file, doesn't it ?\n>> \"$ISO8859\" &&\n> \n> (I just wonder how this works on you machine )\nI tested it on OS X (HPFS) and Linux (ext4). Can you reproduce problems on your machine? If yes, what is your OS/filesystem?\n\n> And, may be, we could fill the file with some content, to be double-sure that\n> the file name conversion works with Perforce ?\n> \n> like\n> echo content >\"$ISO8859\" &&\n> \n> and test the content later ?\nOK, I will add this.\n\nThanks,\nLars\n"},{"id":"269108","messageId":"xmqqr3mirrau.fsf@gitster.mtv.corp.google.com","threadId":"40234","inReplyTo":"523D2C52-893E-4A54-9DD1-2DDBCEA29960@gmail.com","subject":"Re: [PATCH v3] git-p4: add \"--path-encoding\" option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-09-01T17:19:05Z","receivedAt":"2015-09-01T17:19:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lars Schneider writes:\n\n>> I'll move this to 9822, as 9821 is taken by another git-p4 test,\n>> while queuing.\n> OK. I wasn’t sure how this is handled. Just for my understanding: As\n> soon as a TC number is occupied in one of the official branches\n> (master/next/pu/maint) then the next number should be used, right?\n\nThat would help me great deal (as I'd be doing that if you didn't).\n\n> Two general questions:\n>\n> (1) I saw that the path encoding fix is already on “pu”. What is\n> more convenient for you? Do you want to get a [PATCH v4] with one\n> commit including the changes mentioned above or do you want me to keep\n> the v3 commit and add a v4 commit on top?\n\nThe convention is to do [PATCH v$n+1] (i.e. reroll) before the topic\nis merged to 'next' (i.e. while in 'pu').\n\nWhen further improvements are needed after the topic is merged to\n'next', we switch to incremental improvements.\n\n> (2) Is there a CI setup for Git somewhere accessible on the Internet?\n> How about building and testing Git on Travis\n> (https://travis-ci.org/git/git)?\n\nSorry but you are asking the right question to a wrong guy ;-)\n\nThere may be people who have already arranged something like that,\nwho can speak up.\n"},{"id":"269110","messageId":"xmqqmvx6rqk7.fsf@gitster.mtv.corp.google.com","threadId":"40234","inReplyTo":"523D2C52-893E-4A54-9DD1-2DDBCEA29960@gmail.com","subject":"Re: [PATCH v3] git-p4: add \"--path-encoding\" option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-09-01T17:35:04Z","receivedAt":"2015-09-01T17:35:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lars Schneider writes:\n\n> On 01 Sep 2015, at 01:13, Junio C Hamano wrote:\n>\n>> larsxschneider@gmail.com writes:\n>> \n>>> From: Lars Schneider \n>>> \n>> \n>> Here is a space for you to describe what it does and why it is a\n>> good idea to have it.\n> How about this:\n>\n> Perforce keeps the encoding of a path as given by the originating\n> OS. Git expects paths encoded as UTF-8. Add an option to tell git-p4\n> what encoding Perforce had used for the paths. This encoding is used\n> to transcode the paths to UTF-8. As an example, Perforce on Windows\n> uses “cp1252” to encode path names.\n\nVery readable. Does \"Perforce on Windows\" always use cp1252, or\nis it more correct to say \"often uses\" here?\n"},{"id":"269140","messageId":"1243E147-EFEC-4D0B-9872-6B9A8BBE7737@gmail.com","threadId":"40234","inReplyTo":"xmqqmvx6rqk7.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v3] git-p4: add \"--path-encoding\" option","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-09-01T18:55:13Z","receivedAt":"2015-09-01T18:55:13Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\nOn 01 Sep 2015, at 19:35, Junio C Hamano wrote:\n\n> Lars Schneider writes:\n> \n>> On 01 Sep 2015, at 01:13, Junio C Hamano wrote:\n>> \n>>> larsxschneider@gmail.com writes:\n>>> \n>>>> From: Lars Schneider \n>>>> \n>>> \n>>> Here is a space for you to describe what it does and why it is a\n>>> good idea to have it.\n>> How about this:\n>> \n>> Perforce keeps the encoding of a path as given by the originating\n>> OS. Git expects paths encoded as UTF-8. Add an option to tell git-p4\n>> what encoding Perforce had used for the paths. This encoding is used\n>> to transcode the paths to UTF-8. As an example, Perforce on Windows\n>> uses “cp1252” to encode path names.\n> \n> Very readable. Does \"Perforce on Windows\" always use cp1252, or\n> is it more correct to say \"often uses\" here?\nThank you! I don’t know if “always” or “often” is better. On my Windows test system it is “always”… but that’s not a valid sample size :-)\nI searched the Internet for clues around cp1252 and found that a similar patch was submitted to Mercurial just a month ago. The author seconds my cp1252 observation:\nhttp://mercurial.808500.n3.nabble.com/PATCH-stable-convert-use-original-local-encoding-when-converting-from-Perfoce-tp4025088p4025094.html\n\n- Lars"},{"id":"269143","messageId":"xmqq613urmbw.fsf@gitster.mtv.corp.google.com","threadId":"40234","inReplyTo":"1243E147-EFEC-4D0B-9872-6B9A8BBE7737@gmail.com","subject":"Re: [PATCH v3] git-p4: add \"--path-encoding\" option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-09-01T19:06:27Z","receivedAt":"2015-09-01T19:06:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lars Schneider writes:\n\n> I searched the Internet for clues around cp1252 and found that a\n> similar patch was submitted to Mercurial just a month ago. The author\n> seconds my cp1252 observation:\n> http://mercurial.808500.n3.nabble.com/PATCH-stable-convert-use-original-local-encoding-when-converting-from-Perfoce-tp4025088p4025094.html\n\nThanks for a pointer. I see this bit in that thread:\n\n If P4CHARSET is not set explicitly when connecting to a Unicode\n mode server, a default charset will be chosen based on the\n client's platform and/or code page.\n\nSo I would guess that it is reproducible for you as you are always\non a client with the same platform and/or code page, and it would be\nmore honest to phrase it as \"As an example, Perforce on my Windows\nbox uses...\" or something along that line.\n\nThanks.\n"}]}