{"thread":{"id":"35165","subject":"[PATCH] remote-hg: unquote C-style paths when exporting","startedAt":"2013-10-18T17:03:41Z","lastAt":"2013-10-23T15:53:15Z","messageCount":7,"participants":["Antoine Pelisse","Junio C Hamano","Felipe Contreras"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"229180","messageId":"1382115821-12586-1-git-send-email-apelisse@gmail.com","threadId":"35165","inReplyTo":null,"subject":"[PATCH] remote-hg: unquote C-style paths when exporting","fromName":"Antoine Pelisse","fromEmail":"apelisse@gmail.com","sentAt":"2013-10-18T17:03:41Z","receivedAt":"2013-10-18T17:03:41Z","isPatch":true,"sender":{"key":"apelisse@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1929644?v=4"},"body":"git-fast-import documentation says that paths can be C-style quoted.\nUnfortunately, the current remote-hg helper doesn't unquote quoted\npath and pass them as-is to Mercurial when the commit is created.\n\nThis result in the following situation:\n\n- clone a mercurial repository with git\n- Add a file with space: `mkdir dir/foo\\ bar`\n- Commit that new file, and push the change to mercurial\n- The mercurial repository as now a new directory named '\"dir', which\ncontains a file named 'foo bar\"'\n\nUse python ast.literal_eval to unquote the string if it starts with \".\nIt has been tested with quotes, spaces, and utf-8 encoded file-names.\n\nSigned-off-by: Antoine Pelisse \n---\n contrib/remote-helpers/git-remote-hg | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg\nindex 92d994e..0141949 100755\n--- a/contrib/remote-helpers/git-remote-hg\n+++ b/contrib/remote-helpers/git-remote-hg\n@@ -14,6 +14,7 @@\n \n from mercurial import hg, ui, bookmarks, context, encoding, node, error, extensions, discovery, util\n \n+import ast\n import re\n import sys\n import os\n@@ -742,6 +743,8 @@ def parse_commit(parser):\n f = { 'deleted' : True }\n else:\n die('Unknown file command: %s' % line)\n+ if path.startswith('\"'):\n+ path = ast.literal_eval(path)\n files[path] = f\n \n # only export the commits if we are on an internal proxy repo\n-- \n1.8.4.1.507.g9768648.dirty\n"},{"id":"229300","messageId":"xmqq4n89t8yw.fsf@gitster.dls.corp.google.com","threadId":"35165","inReplyTo":"1382115821-12586-1-git-send-email-apelisse@gmail.com","subject":"Re: [PATCH] remote-hg: unquote C-style paths when exporting","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-10-22T19:13:27Z","receivedAt":"2013-10-22T19:13:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Antoine Pelisse writes:\n\n> git-fast-import documentation says that paths can be C-style quoted.\n> Unfortunately, the current remote-hg helper doesn't unquote quoted\n> path and pass them as-is to Mercurial when the commit is created.\n>\n> This result in the following situation:\n>\n> - clone a mercurial repository with git\n> - Add a file with space: `mkdir dir/foo\\ bar`\n> - Commit that new file, and push the change to mercurial\n> - The mercurial repository as now a new directory named '\"dir', which\n> contains a file named 'foo bar\"'\n>\n> Use python ast.literal_eval to unquote the string if it starts with \".\n> It has been tested with quotes, spaces, and utf-8 encoded file-names.\n>\n> Signed-off-by: Antoine Pelisse \n> ---\n\nA path you read in fast-import input indeed needs to be unquoted\nwhen it begins with a dq, and I _think_ by using ast.literal_eval(),\nyou probably can correctly unquote any valid C-quoted string.\n\nBut it bothers me somewhat that what the patch does seems to be\noverly broad. Doesn't ast.literal_eval() take a lot more than just\nstrings?\n\n ast.literal_eval(node_or_string)\n\n Safely evaluate an expression node or a Unicode or Latin-1\n encoded string containing a Python expression. The string or\n node provided may only consist of the following Python literal\n structures: strings, numbers, tuples, lists, dicts, booleans,\n and None.\n\nAlso doesn't Python's double-quoted string have a lot more magic\nthan C-quoted string, e.g.\n\n\t$ python -i\n >>> import ast\n >>> not_cq_path = '\"abc\" \"def\"'\n >>> ast.literal_eval(not_cq_path)\n\t'abcdef'\n\n> contrib/remote-helpers/git-remote-hg | 3 +++\n> 1 file changed, 3 insertions(+)\n>\n> diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg\n> index 92d994e..0141949 100755\n> --- a/contrib/remote-helpers/git-remote-hg\n> +++ b/contrib/remote-helpers/git-remote-hg\n> @@ -14,6 +14,7 @@\n> \n> from mercurial import hg, ui, bookmarks, context, encoding, node, error, extensions, discovery, util\n> \n> +import ast\n> import re\n> import sys\n> import os\n> @@ -742,6 +743,8 @@ def parse_commit(parser):\n> f = { 'deleted' : True }\n> else:\n> die('Unknown file command: %s' % line)\n> + if path.startswith('\"'):\n> + path = ast.literal_eval(path)\n> files[path] = f\n> \n> # only export the commits if we are on an internal proxy repo\n"},{"id":"229306","messageId":"CALWbr2zsOYNN45d+qHDQ88eLj82iV4QxJ_9ro+RGk7upBJVATA@mail.gmail.com","threadId":"35165","inReplyTo":"xmqq4n89t8yw.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH] remote-hg: unquote C-style paths when exporting","fromName":"Antoine Pelisse","fromEmail":"apelisse@gmail.com","sentAt":"2013-10-22T20:49:23Z","receivedAt":"2013-10-22T20:49:23Z","isPatch":true,"sender":{"key":"apelisse@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1929644?v=4"},"body":"On Tue, Oct 22, 2013 at 9:13 PM, Junio C Hamano wrote:\n> Antoine Pelisse writes:\n>\n>> git-fast-import documentation says that paths can be C-style quoted.\n>> Unfortunately, the current remote-hg helper doesn't unquote quoted\n>> path and pass them as-is to Mercurial when the commit is created.\n>>\n>> This result in the following situation:\n>>\n>> - clone a mercurial repository with git\n>> - Add a file with space: `mkdir dir/foo\\ bar`\n\nNote to myself, mkdir doesn't create a \"file\"\n\n>> - Commit that new file, and push the change to mercurial\n>> - The mercurial repository as now a new directory named '\"dir', which\n>> contains a file named 'foo bar\"'\n>>\n>> Use python ast.literal_eval to unquote the string if it starts with \".\n>> It has been tested with quotes, spaces, and utf-8 encoded file-names.\n>>\n>> Signed-off-by: Antoine Pelisse \n>> ---\n>\n> A path you read in fast-import input indeed needs to be unquoted\n> when it begins with a dq, and I _think_ by using ast.literal_eval(),\n> you probably can correctly unquote any valid C-quoted string.\n>\n> But it bothers me somewhat that what the patch does seems to be\n> overly broad. Doesn't ast.literal_eval() take a lot more than just\n> strings?\n\nGood point\n\n> ast.literal_eval(node_or_string)\n>\n> Safely evaluate an expression node or a Unicode or Latin-1\n> encoded string containing a Python expression. The string or\n> node provided may only consist of the following Python literal\n> structures: strings, numbers, tuples, lists, dicts, booleans,\n> and None.\n\nFortunately, I don't believe any of the other type can start with a\ndq. So currently, I don't believe we can end-up with anything else but\na string. We could certainly check that this is always true though.\n\n> Also doesn't Python's double-quoted string have a lot more magic\n> than C-quoted string, e.g.\n>\n> $ python -i\n> >>> import ast\n> >>> not_cq_path = '\"abc\" \"def\"'\n> >>> ast.literal_eval(not_cq_path)\n> 'abcdef'\n\nIt is true that I have expected \"valid output\" from git-fast-export.\nAnd I don't have in mind any easy solution to detect that the output\nis broken, yet still accepted as a valid string by python. We could\nobviously write a unquote_c_style() equivalent in python if needed.\n\nThanks,\n"},{"id":"229310","messageId":"CAMP44s37-R0u4oLnuRfdghx-Tk3X9eer0MVTcAGmPZ3Bu32dug@mail.gmail.com","threadId":"35165","inReplyTo":"CALWbr2zsOYNN45d+qHDQ88eLj82iV4QxJ_9ro+RGk7upBJVATA@mail.gmail.com","subject":"Re: [PATCH] remote-hg: unquote C-style paths when exporting","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2013-10-23T00:45:22Z","receivedAt":"2013-10-23T00:45:22Z","isPatch":true,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"On Tue, Oct 22, 2013 at 3:49 PM, Antoine Pelisse wrote:\n\n> It is true that I have expected \"valid output\" from git-fast-export.\n> And I don't have in mind any easy solution to detect that the output\n> is broken, yet still accepted as a valid string by python. We could\n> obviously write a unquote_c_style() equivalent in python if needed.\n\nSomething like this?\n\ndef c_style_unescape(string):\n if string[0] == string[-1] == '\"':\n return string.decode('string-escape')[1:-1]\n return string\n\nIt's in git-remote-bzr.py.\n\n-- \nFelipe Contreras\n"},{"id":"229318","messageId":"CALWbr2zzT47e_B0moy0a5gpWfhberp9B3TEwkGFBBm19iGfQBw@mail.gmail.com","threadId":"35165","inReplyTo":"CAMP44s37-R0u4oLnuRfdghx-Tk3X9eer0MVTcAGmPZ3Bu32dug@mail.gmail.com","subject":"Re: [PATCH] remote-hg: unquote C-style paths when exporting","fromName":"Antoine Pelisse","fromEmail":"apelisse@gmail.com","sentAt":"2013-10-23T08:38:50Z","receivedAt":"2013-10-23T08:38:50Z","isPatch":true,"sender":{"key":"apelisse@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1929644?v=4"},"body":"On Wed, Oct 23, 2013 at 2:45 AM, Felipe Contreras\n wrote:\n> On Tue, Oct 22, 2013 at 3:49 PM, Antoine Pelisse wrote:\n>\n>> It is true that I have expected \"valid output\" from git-fast-export.\n>> And I don't have in mind any easy solution to detect that the output\n>> is broken, yet still accepted as a valid string by python. We could\n>> obviously write a unquote_c_style() equivalent in python if needed.\n>\n> Something like this?\n>\n> def c_style_unescape(string):\n> if string[0] == string[-1] == '\"':\n> return string.decode('string-escape')[1:-1]\n> return string\n>\n> It's in git-remote-bzr.py.\n\nYeah, that's certainly better,\n\nThanks,\n"},{"id":"229327","messageId":"xmqqa9i0rnzo.fsf_-_@gitster.dls.corp.google.com","threadId":"35165","inReplyTo":"CALWbr2zzT47e_B0moy0a5gpWfhberp9B3TEwkGFBBm19iGfQBw@mail.gmail.com","subject":"Re* [PATCH] remote-hg: unquote C-style paths when exporting","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-10-23T15:44:11Z","receivedAt":"2013-10-23T15:44:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Antoine Pelisse writes:\n\n>> def c_style_unescape(string):\n>> if string[0] == string[-1] == '\"':\n>> return string.decode('string-escape')[1:-1]\n>> return string\n>>\n>> It's in git-remote-bzr.py.\n>\n> Yeah, that's certainly better,\n>\n> Thanks,\n\nOK, so an amended one will look like this?\n\n-- >8 --\nFrom: Antoine Pelisse \nSubject: remote-hg: unquote C-style paths when exporting\n\ngit-fast-import documentation says that paths can be C-style quoted.\nUnfortunately, the current remote-hg helper doesn't unquote quoted\npath and pass them as-is to Mercurial when the commit is created.\n\nThis result in the following situation:\n\n- clone a mercurial repository with git\n- Add a file with space: `mkdir dir/foo\\ bar`\n- Commit that new file, and push the change to mercurial\n- The mercurial repository as now a new directory named '\"dir', which\ncontains a file named 'foo bar\"'\n\nUse python ast.literal_eval to unquote the string if it starts with \".\nIt has been tested with quotes, spaces, and utf-8 encoded file-names.\n\nSigned-off-by: Antoine Pelisse \nSigned-off-by: Junio C Hamano \n---\n\n contrib/remote-helpers/git-remote-hg | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg\nindex 0194c67..85abbed 100755\n--- a/contrib/remote-helpers/git-remote-hg\n+++ b/contrib/remote-helpers/git-remote-hg\n@@ -678,6 +678,11 @@ def get_merge_files(repo, p1, p2, files):\n f = { 'ctx' : repo[p1][e] }\n files[e] = f\n \n+def c_style_unescape(string):\n+ if string[0] == string[-1] == '\"':\n+ return string.decode('string-escape')[1:-1]\n+ return string\n+\n def parse_commit(parser):\n global marks, blob_marks, parsed_refs\n global mode\n@@ -720,6 +725,7 @@ def parse_commit(parser):\n f = { 'deleted' : True }\n else:\n die('Unknown file command: %s' % line)\n+ path = c_style_unescape(path).decode('utf-8')\n files[path] = f\n \n # only export the commits if we are on an internal proxy repo\n"},{"id":"229335","messageId":"CALWbr2z-+S_rNFEKNH2fYJQmn1z9SyO2Z4sNDoz=Kh8P8Y_ccQ@mail.gmail.com","threadId":"35165","inReplyTo":"xmqqa9i0rnzo.fsf_-_@gitster.dls.corp.google.com","subject":"Re: Re* [PATCH] remote-hg: unquote C-style paths when exporting","fromName":"Antoine Pelisse","fromEmail":"apelisse@gmail.com","sentAt":"2013-10-23T15:53:15Z","receivedAt":"2013-10-23T15:53:15Z","isPatch":true,"sender":{"key":"apelisse@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1929644?v=4"},"body":"On Wed, Oct 23, 2013 at 5:44 PM, Junio C Hamano wrote:\n> Antoine Pelisse writes:\n>\n>>> def c_style_unescape(string):\n>>> if string[0] == string[-1] == '\"':\n>>> return string.decode('string-escape')[1:-1]\n>>> return string\n>>>\n>>> It's in git-remote-bzr.py.\n>>\n>> Yeah, that's certainly better,\n>>\n>> Thanks,\n>\n> OK, so an amended one will look like this?\n\nThe commit message needs to be updated as well.\n\n> -- >8 --\n> From: Antoine Pelisse \n> Subject: remote-hg: unquote C-style paths when exporting\n>\n> git-fast-import documentation says that paths can be C-style quoted.\n> Unfortunately, the current remote-hg helper doesn't unquote quoted\n> path and pass them as-is to Mercurial when the commit is created.\n>\n> This result in the following situation:\n\ns/result/&s/\n\n> - clone a mercurial repository with git\n> - Add a file with space: `mkdir dir/foo\\ bar`\n\n- Add a file with space in a directory: `>dir/foo\\ bar`\n\n> - Commit that new file, and push the change to mercurial\n> - The mercurial repository as now a new directory named '\"dir', which\n> contains a file named 'foo bar\"'\n\nI'm so ashamed I'd rather not report this one: s/as/has/\n\n> Use python ast.literal_eval to unquote the string if it starts with \".\n\nUse python str.decode('string-escape') to unquote the string if it\nstarts and ends with \".\n\n> It has been tested with quotes, spaces, and utf-8 encoded file-names.\n>\n> Signed-off-by: Antoine Pelisse \n> Signed-off-by: Junio C Hamano \n> ---\n> contrib/remote-helpers/git-remote-hg | 6 ++++++\n> 1 file changed, 6 insertions(+)\n>\n> diff --git a/contrib/remote-helpers/git-remote-hg b/contrib/remote-helpers/git-remote-hg\n> index 0194c67..85abbed 100755\n> --- a/contrib/remote-helpers/git-remote-hg\n> +++ b/contrib/remote-helpers/git-remote-hg\n> @@ -678,6 +678,11 @@ def get_merge_files(repo, p1, p2, files):\n> f = { 'ctx' : repo[p1][e] }\n> files[e] = f\n>\n> +def c_style_unescape(string):\n> + if string[0] == string[-1] == '\"':\n> + return string.decode('string-escape')[1:-1]\n> + return string\n> +\n> def parse_commit(parser):\n> global marks, blob_marks, parsed_refs\n> global mode\n> @@ -720,6 +725,7 @@ def parse_commit(parser):\n> f = { 'deleted' : True }\n> else:\n> die('Unknown file command: %s' % line)\n> + path = c_style_unescape(path).decode('utf-8')\n> files[path] = f\n>\n> # only export the commits if we are on an internal proxy repo\n\nThat is consistent with git-remote-bzr,\n\nThanks\n"}]}