{"thread":{"id":"32239","subject":"[PATCH] git-remote-mediawiki: escape double quotes and LF in file names","startedAt":"2012-11-29T12:33:33Z","lastAt":"2012-12-02T02:27:14Z","messageCount":16,"participants":["Matthieu Moy","Junio C Hamano","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"204249","messageId":"1354192413-9959-1-git-send-email-Matthieu.Moy@imag.fr","threadId":"32239","inReplyTo":null,"subject":"[PATCH] git-remote-mediawiki: escape double quotes and LF in file names","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2012-11-29T12:33:33Z","receivedAt":"2012-11-29T12:33:33Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"A mediawiki page can contain, and even start with a \" character, we have\nto escape it when generating the fast-export stream. While we're there,\nalso escape newlines, but I don't think we can get them from MediaWiki\npages.\n\nSigned-off-by: Matthieu Moy \n---\n contrib/mw-to-git/git-remote-mediawiki | 15 ++++++++++++---\n 1 file changed, 12 insertions(+), 3 deletions(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\nindex 68555d4..e7a0e7b 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki\n+++ b/contrib/mw-to-git/git-remote-mediawiki\n@@ -711,6 +711,13 @@ sub fetch_mw_revisions {\n \treturn ($n, @revisions);\n }\n \n+sub fe_escape_path {\n+ my $path = shift;\n+ $path =~ s/\"/\\\\\"/g;\n+ $path =~ s/\\n/\\\\n/g;\n+ return $path;\n+}\n+\n sub import_file_revision {\n \tmy $commit = shift;\n \tmy %commit = %{$commit};\n@@ -738,15 +745,17 @@ sub import_file_revision {\n \t\tprint STDOUT \"from refs/mediawiki/$remotename/master^0\\n\";\n \t}\n \tif ($content ne DELETED_CONTENT) {\n-\t\tprint STDOUT \"M 644 inline $title.mw\\n\";\n+\t\tprint STDOUT \"M 644 inline \" .\n+\t\t fe_escape_path($title . \".mw\") . \"\\n\";\n \t\tliteral_data($content);\n \t\tif (%mediafile) {\n-\t\t\tprint STDOUT \"M 644 inline $mediafile{title}\\n\";\n+\t\t\tprint STDOUT \"M 644 inline \"\n+\t\t\t . fe_escape_path($mediafile{title}) . \"\\n\";\n \t\t\tliteral_data_raw($mediafile{content});\n \t\t}\n \t\tprint STDOUT \"\\n\\n\";\n \t} else {\n-\t\tprint STDOUT \"D $title.mw\\n\";\n+\t\tprint STDOUT \"D \" . fe_escape_path($title . \".mw\") . \"\\n\";\n \t}\n \n \t# mediawiki revision number in the git note\n-- \n1.8.0.319.g8abfee4\n"},{"id":"204261","messageId":"7vy5hkqsut.fsf@alter.siamese.dyndns.org","threadId":"32239","inReplyTo":"1354192413-9959-1-git-send-email-Matthieu.Moy@imag.fr","subject":"Re: [PATCH] git-remote-mediawiki: escape double quotes and LF in file names","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-11-29T16:25:14Z","receivedAt":"2012-11-29T16:25:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthieu Moy writes:\n\n> A mediawiki page can contain, and even start with a \" character, we have\n> to escape it when generating the fast-export stream. While we're there,\n> also escape newlines, but I don't think we can get them from MediaWiki\n> pages.\n>\n> Signed-off-by: Matthieu Moy \n> ---\n> contrib/mw-to-git/git-remote-mediawiki | 15 ++++++++++++---\n> 1 file changed, 12 insertions(+), 3 deletions(-)\n>\n> diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\n> index 68555d4..e7a0e7b 100755\n> --- a/contrib/mw-to-git/git-remote-mediawiki\n> +++ b/contrib/mw-to-git/git-remote-mediawiki\n> @@ -711,6 +711,13 @@ sub fetch_mw_revisions {\n> \treturn ($n, @revisions);\n> }\n> \n> +sub fe_escape_path {\n> + my $path = shift;\n> + $path =~ s/\"/\\\\\"/g;\n> + $path =~ s/\\n/\\\\n/g;\n> + return $path;\n> +}\n\nIs this sufficient?\n\nMy reading of the big comment at the beginning of fast-import.c is\nthat you would also want to quote each backslash; otherwise a\ncharacter (or an octal) after it will be taken as a C-style quoted\nspecial letter, no?\n\n> sub import_file_revision {\n> \tmy $commit = shift;\n> \tmy %commit = %{$commit};\n> @@ -738,15 +745,17 @@ sub import_file_revision {\n> \t\tprint STDOUT \"from refs/mediawiki/$remotename/master^0\\n\";\n> \t}\n> \tif ($content ne DELETED_CONTENT) {\n> -\t\tprint STDOUT \"M 644 inline $title.mw\\n\";\n> +\t\tprint STDOUT \"M 644 inline \" .\n> +\t\t fe_escape_path($title . \".mw\") . \"\\n\";\n> \t\tliteral_data($content);\n> \t\tif (%mediafile) {\n> -\t\t\tprint STDOUT \"M 644 inline $mediafile{title}\\n\";\n> +\t\t\tprint STDOUT \"M 644 inline \"\n> +\t\t\t . fe_escape_path($mediafile{title}) . \"\\n\";\n> \t\t\tliteral_data_raw($mediafile{content});\n> \t\t}\n> \t\tprint STDOUT \"\\n\\n\";\n> \t} else {\n> -\t\tprint STDOUT \"D $title.mw\\n\";\n> +\t\tprint STDOUT \"D \" . fe_escape_path($title . \".mw\") . \"\\n\";\n> \t}\n> \n> \t# mediawiki revision number in the git note\n"},{"id":"204264","messageId":"vpq624omjn4.fsf@grenoble-inp.fr","threadId":"32239","inReplyTo":"7vy5hkqsut.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH] git-remote-mediawiki: escape double quotes and LF in file names","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2012-11-29T16:57:51Z","receivedAt":"2012-11-29T16:57:51Z","isPatch":true,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Junio C Hamano writes:\n\n>> +sub fe_escape_path {\n>> + my $path = shift;\n>> + $path =~ s/\"/\\\\\"/g;\n>> + $path =~ s/\\n/\\\\n/g;\n>> + return $path;\n>> +}\n>\n> Is this sufficient?\n>\n> My reading of the big comment at the beginning of fast-import.c is\n> that you would also want to quote each backslash;\n\nNice catch, thanks.\n\nAlso, I was lacking the double-quotes around $path. In my defense, I had\nonly read the doc, not the code, and git-fast-import.txt was less\ncomplete than fast-import.c. New patch follows, fixing the doc too.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"204265","messageId":"1354208455-21228-1-git-send-email-Matthieu.Moy@imag.fr","threadId":"32239","inReplyTo":"vpq624omjn4.fsf@grenoble-inp.fr","subject":"[PATCH 1/2] git-fast-import.txt: improve documentation for quoted paths","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2012-11-29T17:00:54Z","receivedAt":"2012-11-29T17:00:54Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"The documentation mentionned only newlines and double quotes as\ncharacters needing escaping, but the backslash also needs it. Also, the\ndocumentation was not clearly saying that double quotes around the file\nname were required (double quotes in the examples could be interpreted as\npart of the sentence, not part of the actual string).\n\nSigned-off-by: Matthieu Moy \n---\n Documentation/git-fast-import.txt | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex 6603a7a..35b909c 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -558,8 +558,9 @@ A `` string must use UNIX-style directory separators (forward\n slash `/`), may contain any byte other than `LF`, and must not\n start with double quote (`\"`).\n \n-If an `LF` or double quote must be encoded into `` shell-style\n-quoting should be used, e.g. `\"path/with\\n and \\\" in it\"`.\n+If an `LF`, backslash or double quote must be encoded into ``\n+shell-style quoting should be used, and the complete name should be\n+surrounded with double quotes e.g. `\"path/with\\n, \\\\ and \\\" in it\"`.\n \n The value of `` must be in canonical form. That is it must not:\n \n-- \n1.8.0.319.g8abfee4\n"},{"id":"204266","messageId":"1354208455-21228-2-git-send-email-Matthieu.Moy@imag.fr","threadId":"32239","inReplyTo":"1354208455-21228-1-git-send-email-Matthieu.Moy@imag.fr","subject":"[PATCH 2/2 v2] git-remote-mediawiki: escape \", \\, and LF in file names","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2012-11-29T17:00:55Z","receivedAt":"2012-11-29T17:00:55Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"A mediawiki page can contain, and even start with a \" character, we have\nto escape it when generating the fast-export stream, as well as \\\ncharacter. While we're there, also escape newlines, but I don't think we\ncan get them from MediaWiki pages.\n\nSigned-off-by: Matthieu Moy \n---\n contrib/mw-to-git/git-remote-mediawiki | 16 +++++++++++++---\n contrib/mw-to-git/t/t9362-mw-to-git-utf8.sh | 26 ++++++++++++++++++++++++++\n 2 files changed, 39 insertions(+), 3 deletions(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\nindex 68555d4..094129d 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki\n+++ b/contrib/mw-to-git/git-remote-mediawiki\n@@ -711,6 +711,14 @@ sub fetch_mw_revisions {\n \treturn ($n, @revisions);\n }\n \n+sub fe_escape_path {\n+ my $path = shift;\n+ $path =~ s/\\\\/\\\\\\\\/g;\n+ $path =~ s/\"/\\\\\"/g;\n+ $path =~ s/\\n/\\\\n/g;\n+ return '\"' . $path . '\"';\n+}\n+\n sub import_file_revision {\n \tmy $commit = shift;\n \tmy %commit = %{$commit};\n@@ -738,15 +746,17 @@ sub import_file_revision {\n \t\tprint STDOUT \"from refs/mediawiki/$remotename/master^0\\n\";\n \t}\n \tif ($content ne DELETED_CONTENT) {\n-\t\tprint STDOUT \"M 644 inline $title.mw\\n\";\n+\t\tprint STDOUT \"M 644 inline \" .\n+\t\t fe_escape_path($title . \".mw\") . \"\\n\";\n \t\tliteral_data($content);\n \t\tif (%mediafile) {\n-\t\t\tprint STDOUT \"M 644 inline $mediafile{title}\\n\";\n+\t\t\tprint STDOUT \"M 644 inline \"\n+\t\t\t . fe_escape_path($mediafile{title}) . \"\\n\";\n \t\t\tliteral_data_raw($mediafile{content});\n \t\t}\n \t\tprint STDOUT \"\\n\\n\";\n \t} else {\n-\t\tprint STDOUT \"D $title.mw\\n\";\n+\t\tprint STDOUT \"D \" . fe_escape_path($title . \".mw\") . \"\\n\";\n \t}\n \n \t# mediawiki revision number in the git note\ndiff --git a/contrib/mw-to-git/t/t9362-mw-to-git-utf8.sh b/contrib/mw-to-git/t/t9362-mw-to-git-utf8.sh\nindex 246d47d..b6405ce 100755\n--- a/contrib/mw-to-git/t/t9362-mw-to-git-utf8.sh\n+++ b/contrib/mw-to-git/t/t9362-mw-to-git-utf8.sh\n@@ -318,4 +318,30 @@ test_expect_success 'git push with \\ in format control' '\n '\n \n \n+test_expect_success 'fast-import meta-characters in page name (mw -> git)' '\n+\twiki_reset &&\n+\twiki_editpage \\\"file\\\"_\\\\_foo \"expect to be called \\\"file\\\"_\\\\_foo\" false &&\n+\tgit clone mediawiki::'\"$WIKI_URL\"' mw_dir_21 &&\n+\ttest_path_is_file mw_dir_21/\\\"file\\\"_\\\\_foo.mw &&\n+\twiki_getallpage ref_page_21 &&\n+\ttest_diff_directories mw_dir_21 ref_page_21\n+'\n+\n+\n+test_expect_success 'fast-import meta-characters in page name (git -> mw) ' '\n+\twiki_reset &&\n+\tgit clone mediawiki::'\"$WIKI_URL\"' mw_dir_22 &&\n+\t(\n+\t\tcd mw_dir_22 &&\n+\t\techo \"this file is called \\\"file\\\"_\\\\_foo.mw\" >\\\"file\\\"_\\\\_foo &&\n+\t\tgit add . &&\n+\t\tgit commit -am \"file \\\"file\\\"_\\\\_foo\" &&\n+\t\tgit pull &&\n+\t\tgit push\n+\t) &&\n+\twiki_getallpage ref_page_22 &&\n+\ttest_diff_directories mw_dir_22 ref_page_22\n+'\n+\n+\n test_done\n-- \n1.8.0.319.g8abfee4\n"},{"id":"204272","messageId":"20121129181141.GA17309@sigill.intra.peff.net","threadId":"32239","inReplyTo":"1354208455-21228-1-git-send-email-Matthieu.Moy@imag.fr","subject":"Re: [PATCH 1/2] git-fast-import.txt: improve documentation for quoted paths","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-11-29T18:11:42Z","receivedAt":"2012-11-29T18:11:42Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Nov 29, 2012 at 06:00:54PM +0100, Matthieu Moy wrote:\n\n> The documentation mentionned only newlines and double quotes as\n\ns/nn/n/\n\n> diff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\n> index 6603a7a..35b909c 100644\n> --- a/Documentation/git-fast-import.txt\n> +++ b/Documentation/git-fast-import.txt\n> @@ -558,8 +558,9 @@ A `` string must use UNIX-style directory separators (forward\n> slash `/`), may contain any byte other than `LF`, and must not\n> start with double quote (`\"`).\n> \n> -If an `LF` or double quote must be encoded into `` shell-style\n> -quoting should be used, e.g. `\"path/with\\n and \\\" in it\"`.\n> +If an `LF`, backslash or double quote must be encoded into ``\n> +shell-style quoting should be used, and the complete name should be\n> +surrounded with double quotes e.g. `\"path/with\\n, \\\\ and \\\" in it\"`.\n\nI think the point of the original is that you do not _need_ to quote\nunless you have those two characters. IOW, you can do:\n\n M 100644 :1 file \\with \\backslashes\n\nand it will stay in the \"literal to the end of line\" code path because\nthe path does not begin with a double-quote. It is only when you trigger\nthe \"shell-style quoting\" code path is in effect that you must then\nfollow the rules of that quoting (which includes escaping backslashes).\nSo technically, your modification to the beginning of the sentence is\nnot correct.\n\nThat being said, I think what you have written is more helpful to an end\nuser. There is no harm in quoting when we do not have to, as fast-import\nimplementations must know how to unquote anyway (and we over-quote in\nfast-export in this case). And while the example above does work (and\nwas always designed to), it is sort of an unintuitive area that I would\nnot be surprised to see other fast-import implementations get wrong. As\na writer of a stream, it probably pays to be defensive and err on the\nside of quoting more.\n\nAs for the text itself, a few minor punctuation suggestions:\n\n> If an `LF`, backslash or double quote must be encoded\n ^\n missing comma as list delimiter\n\n> into `` shell-style quoting should be used, and the complete\n ^\n missing comma in if/then clause\n\nThis one was in the original as well, but it makes it harder to read and\nis worth fixing.\n\n> surrounded with double quotes e.g. `\"path/with\\n, \\\\ and \\\" in it\"`.\n\nShould the parenthetical be in parentheses (or a separate sentence)?\n\n-Peff\n"},{"id":"204278","messageId":"vpqhao8gsaj.fsf@grenoble-inp.fr","threadId":"32239","inReplyTo":"20121129181141.GA17309@sigill.intra.peff.net","subject":"Re: [PATCH 1/2] git-fast-import.txt: improve documentation for quoted paths","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2012-11-29T18:47:32Z","receivedAt":"2012-11-29T18:47:32Z","isPatch":true,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Jeff King writes:\n\n> So technically, your modification to the beginning of the sentence is\n> not correct.\n\nI'd say the resulting sentence is somehow incorrect, but not more than\nthe previous one (both say \"if ...\" without really telling what the\ncondition was).\n\n>> If an `LF`, backslash or double quote must be encoded\n> ^\n> missing comma as list delimiter\n\nGoogle tells me that my version was UK-correct but not US-correct. As\nfrench, I have no opinion on the subject, I take yours ;-).\n\nHow about this:\n\nA path can use the C-style string quoting (this is accepted in all\ncases and mandatory if the filename starts with double quote or\ncontains `LF`). In C-style quoting, `LF`, backslash, and double quote\ncharacters must be escaped by preceding them with a backslash. Also,\nthe complete name should be surrounded with double quotes (e.g.\n`\"path/with\\n, \\\\ and \\\" in it\"`).\n\nThis should be technically correct, and \"this is accepted in all cases\"\nshould encourrage people to use it.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"204280","messageId":"20121129185404.GC17309@sigill.intra.peff.net","threadId":"32239","inReplyTo":"vpqhao8gsaj.fsf@grenoble-inp.fr","subject":"Re: [PATCH 1/2] git-fast-import.txt: improve documentation for quoted paths","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-11-29T18:54:04Z","receivedAt":"2012-11-29T18:54:04Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Nov 29, 2012 at 07:47:32PM +0100, Matthieu Moy wrote:\n\n> Jeff King writes:\n> \n> > So technically, your modification to the beginning of the sentence is\n> > not correct.\n> \n> I'd say the resulting sentence is somehow incorrect, but not more than\n> the previous one (both say \"if ...\" without really telling what the\n> condition was).\n\nThat's fair. There is a lot left unsaid in the original. :)\n\n> >> If an `LF`, backslash or double quote must be encoded\n> > ^\n> > missing comma as list delimiter\n> \n> Google tells me that my version was UK-correct but not US-correct. As\n> french, I have no opinion on the subject, I take yours ;-).\n\nThanks, I had a vague recollection that it might be regional. I probably\nshould have looked it up myself.\n\n> How about this:\n> \n> A path can use the C-style string quoting (this is accepted in all\n> cases and mandatory if the filename starts with double quote or\n> contains `LF`). In C-style quoting, `LF`, backslash, and double quote\n> characters must be escaped by preceding them with a backslash. Also,\n> the complete name should be surrounded with double quotes (e.g.\n> `\"path/with\\n, \\\\ and \\\" in it\"`).\n> \n> This should be technically correct, and \"this is accepted in all cases\"\n> should encourrage people to use it.\n\nI think that is much better, but it reads a little more easily to me if\nwe rearrange the second sentence. To complete my English bikeshedding,\nhere is how I would have written the whole paragraph:\n\n A path can use C-style string quoting; this is accepted in all cases\n and mandatory if the filename starts with double quote or contains\n `LF`. In C-style quoting, the complete name should be surrounded with\n double quotes, and any `LF`, backslash, or double quote characters\n must be escaped by preceding them with a backslash (e.g.,\n `\"path/with\\n, \\\\ and \\\" in it\"`).\n\nFeel free to incorporate or ignore any of my tweaks from that version.\n\n-Peff\n"},{"id":"204283","messageId":"1354216293-21921-1-git-send-email-Matthieu.Moy@imag.fr","threadId":"32239","inReplyTo":"20121129185404.GC17309@sigill.intra.peff.net","subject":"[PATCH 1/2 v3] git-fast-import.txt: improve documentation for quoted paths","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2012-11-29T19:11:32Z","receivedAt":"2012-11-29T19:11:32Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"The documentation mentionned only newlines and double quotes as\ncharacters needing escaping, but the backslash also needs it. Also, the\ndocumentation was not clearly saying that double quotes around the file\nname were required (double quotes in the examples could be interpreted as\npart of the sentence, not part of the actual string).\n\nSigned-off-by: Matthieu Moy \n---\ncut-and-paste of Peff's version, adapted from mine.\n\n Documentation/git-fast-import.txt | 8 ++++++--\n 1 file changed, 6 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex 959e4d3..d1844ea 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -562,8 +562,12 @@ A `` string must use UNIX-style directory separators (forward\n slash `/`), may contain any byte other than `LF`, and must not\n start with double quote (`\"`).\n \n-If an `LF` or double quote must be encoded into `` shell-style\n-quoting should be used, e.g. `\"path/with\\n and \\\" in it\"`.\n+A path can use C-style string quoting; this is accepted in all cases\n+and mandatory if the filename starts with double quote or contains\n+`LF`. In C-style quoting, the complete name should be surrounded with\n+double quotes, and any `LF`, backslash, or double quote characters\n+must be escaped by preceding them with a backslash (e.g.,\n+`\"path/with\\n, \\\\ and \\\" in it\"`).\n \n The value of `` must be in canonical form. That is it must not:\n \n-- \n1.8.0.319.g8abfee4\n"},{"id":"204284","messageId":"1354216293-21921-2-git-send-email-Matthieu.Moy@imag.fr","threadId":"32239","inReplyTo":"1354216293-21921-1-git-send-email-Matthieu.Moy@imag.fr","subject":"[PATCH 2/2 v3] git-remote-mediawiki: escape \", \\, and LF in file names","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2012-11-29T19:11:33Z","receivedAt":"2012-11-29T19:11:33Z","isPatch":true,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"A mediawiki page can contain, and even start with a \" character, we have\nto escape it when generating the fast-export stream, as well as \\\ncharacter. While we're there, also escape newlines, but I don't think we\ncan get them from MediaWiki pages.\n\nSigned-off-by: Matthieu Moy \n---\n contrib/mw-to-git/git-remote-mediawiki | 16 +++++++++++++---\n contrib/mw-to-git/t/t9362-mw-to-git-utf8.sh | 26 ++++++++++++++++++++++++++\n 2 files changed, 39 insertions(+), 3 deletions(-)\n\ndiff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki\nindex 68555d4..094129d 100755\n--- a/contrib/mw-to-git/git-remote-mediawiki\n+++ b/contrib/mw-to-git/git-remote-mediawiki\n@@ -711,6 +711,14 @@ sub fetch_mw_revisions {\n \treturn ($n, @revisions);\n }\n \n+sub fe_escape_path {\n+ my $path = shift;\n+ $path =~ s/\\\\/\\\\\\\\/g;\n+ $path =~ s/\"/\\\\\"/g;\n+ $path =~ s/\\n/\\\\n/g;\n+ return '\"' . $path . '\"';\n+}\n+\n sub import_file_revision {\n \tmy $commit = shift;\n \tmy %commit = %{$commit};\n@@ -738,15 +746,17 @@ sub import_file_revision {\n \t\tprint STDOUT \"from refs/mediawiki/$remotename/master^0\\n\";\n \t}\n \tif ($content ne DELETED_CONTENT) {\n-\t\tprint STDOUT \"M 644 inline $title.mw\\n\";\n+\t\tprint STDOUT \"M 644 inline \" .\n+\t\t fe_escape_path($title . \".mw\") . \"\\n\";\n \t\tliteral_data($content);\n \t\tif (%mediafile) {\n-\t\t\tprint STDOUT \"M 644 inline $mediafile{title}\\n\";\n+\t\t\tprint STDOUT \"M 644 inline \"\n+\t\t\t . fe_escape_path($mediafile{title}) . \"\\n\";\n \t\t\tliteral_data_raw($mediafile{content});\n \t\t}\n \t\tprint STDOUT \"\\n\\n\";\n \t} else {\n-\t\tprint STDOUT \"D $title.mw\\n\";\n+\t\tprint STDOUT \"D \" . fe_escape_path($title . \".mw\") . \"\\n\";\n \t}\n \n \t# mediawiki revision number in the git note\ndiff --git a/contrib/mw-to-git/t/t9362-mw-to-git-utf8.sh b/contrib/mw-to-git/t/t9362-mw-to-git-utf8.sh\nindex 246d47d..b6405ce 100755\n--- a/contrib/mw-to-git/t/t9362-mw-to-git-utf8.sh\n+++ b/contrib/mw-to-git/t/t9362-mw-to-git-utf8.sh\n@@ -318,4 +318,30 @@ test_expect_success 'git push with \\ in format control' '\n '\n \n \n+test_expect_success 'fast-import meta-characters in page name (mw -> git)' '\n+\twiki_reset &&\n+\twiki_editpage \\\"file\\\"_\\\\_foo \"expect to be called \\\"file\\\"_\\\\_foo\" false &&\n+\tgit clone mediawiki::'\"$WIKI_URL\"' mw_dir_21 &&\n+\ttest_path_is_file mw_dir_21/\\\"file\\\"_\\\\_foo.mw &&\n+\twiki_getallpage ref_page_21 &&\n+\ttest_diff_directories mw_dir_21 ref_page_21\n+'\n+\n+\n+test_expect_success 'fast-import meta-characters in page name (git -> mw) ' '\n+\twiki_reset &&\n+\tgit clone mediawiki::'\"$WIKI_URL\"' mw_dir_22 &&\n+\t(\n+\t\tcd mw_dir_22 &&\n+\t\techo \"this file is called \\\"file\\\"_\\\\_foo.mw\" >\\\"file\\\"_\\\\_foo &&\n+\t\tgit add . &&\n+\t\tgit commit -am \"file \\\"file\\\"_\\\\_foo\" &&\n+\t\tgit pull &&\n+\t\tgit push\n+\t) &&\n+\twiki_getallpage ref_page_22 &&\n+\ttest_diff_directories mw_dir_22 ref_page_22\n+'\n+\n+\n test_done\n-- \n1.8.0.319.g8abfee4\n"},{"id":"204288","messageId":"7vvccop6dv.fsf@alter.siamese.dyndns.org","threadId":"32239","inReplyTo":"20121129181141.GA17309@sigill.intra.peff.net","subject":"Re: [PATCH 1/2] git-fast-import.txt: improve documentation for quoted paths","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-11-29T19:15:56Z","receivedAt":"2012-11-29T19:15:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> On Thu, Nov 29, 2012 at 06:00:54PM +0100, Matthieu Moy wrote:\n>\n>> The documentation mentionned only newlines and double quotes as\n>\n> s/nn/n/\n>\n>> diff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\n>> index 6603a7a..35b909c 100644\n>> --- a/Documentation/git-fast-import.txt\n>> +++ b/Documentation/git-fast-import.txt\n>> @@ -558,8 +558,9 @@ A `` string must use UNIX-style directory separators (forward\n>> slash `/`), may contain any byte other than `LF`, and must not\n>> start with double quote (`\"`).\n>> \n>> -If an `LF` or double quote must be encoded into `` shell-style\n>> -quoting should be used, e.g. `\"path/with\\n and \\\" in it\"`.\n>> +If an `LF`, backslash or double quote must be encoded into ``\n>> +shell-style quoting should be used, and the complete name should be\n>> +surrounded with double quotes e.g. `\"path/with\\n, \\\\ and \\\" in it\"`.\n\nThat \"shell-style\" contradicts with what fast-import.c says, though.\nIt claims to grok \\octal and described as C-style.\n"},{"id":"204289","messageId":"20121129191926.GA21439@sigill.intra.peff.net","threadId":"32239","inReplyTo":"7vvccop6dv.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 1/2] git-fast-import.txt: improve documentation for quoted paths","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-11-29T19:19:26Z","receivedAt":"2012-11-29T19:19:26Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Nov 29, 2012 at 11:15:56AM -0800, Junio C Hamano wrote:\n\n> >> -If an `LF` or double quote must be encoded into `` shell-style\n> >> -quoting should be used, e.g. `\"path/with\\n and \\\" in it\"`.\n> >> +If an `LF`, backslash or double quote must be encoded into ``\n> >> +shell-style quoting should be used, and the complete name should be\n> >> +surrounded with double quotes e.g. `\"path/with\\n, \\\\ and \\\" in it\"`.\n> \n> That \"shell-style\" contradicts with what fast-import.c says, though.\n> It claims to grok \\octal and described as C-style.\n\nYeah, I think it was just laziness by the original author to use\n\"shell-style\" to mean \"quotes and backslash escaping\" without thinking\ntoo hard about which escape sequences are available. Saying C-style is\nmore accurate (and Matthieu's more recent update does that).\n\n-Peff\n"},{"id":"204290","messageId":"7vr4ncp5kw.fsf@alter.siamese.dyndns.org","threadId":"32239","inReplyTo":"1354216293-21921-1-git-send-email-Matthieu.Moy@imag.fr","subject":"Re: [PATCH 1/2 v3] git-fast-import.txt: improve documentation for quoted paths","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-11-29T19:33:19Z","receivedAt":"2012-11-29T19:33:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthieu Moy writes:\n\n> The documentation mentionned only newlines and double quotes as\n> characters needing escaping, but the backslash also needs it. Also, the\n> documentation was not clearly saying that double quotes around the file\n> name were required (double quotes in the examples could be interpreted as\n> part of the sentence, not part of the actual string).\n>\n> Signed-off-by: Matthieu Moy \n> ---\n> cut-and-paste of Peff's version, adapted from mine.\n>\n> Documentation/git-fast-import.txt | 8 ++++++--\n> 1 file changed, 6 insertions(+), 2 deletions(-)\n>\n> diff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\n> index 959e4d3..d1844ea 100644\n> --- a/Documentation/git-fast-import.txt\n> +++ b/Documentation/git-fast-import.txt\n> @@ -562,8 +562,12 @@ A `` string must use UNIX-style directory separators (forward\n> slash `/`), may contain any byte other than `LF`, and must not\n> start with double quote (`\"`).\n> \n> -If an `LF` or double quote must be encoded into `` shell-style\n> -quoting should be used, e.g. `\"path/with\\n and \\\" in it\"`.\n> +A path can use C-style string quoting; this is accepted in all cases\n> +and mandatory if the filename starts with double quote or contains\n> +`LF`.\n\n... or backslash?\n\n> +In C-style quoting, the complete name should be surrounded with\n> +double quotes, and any `LF`, backslash, or double quote characters\n> +must be escaped by preceding them with a backslash (e.g.,\n> +`\"path/with\\n, \\\\ and \\\" in it\"`).\n> \n> The value of `` must be in canonical form. That is it must not:\n"},{"id":"204292","messageId":"20121129194637.GA22084@sigill.intra.peff.net","threadId":"32239","inReplyTo":"7vr4ncp5kw.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 1/2 v3] git-fast-import.txt: improve documentation for quoted paths","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-11-29T19:46:38Z","receivedAt":"2012-11-29T19:46:38Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Nov 29, 2012 at 11:33:19AM -0800, Junio C Hamano wrote:\n\n> > diff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\n> > index 959e4d3..d1844ea 100644\n> > --- a/Documentation/git-fast-import.txt\n> > +++ b/Documentation/git-fast-import.txt\n> > @@ -562,8 +562,12 @@ A `` string must use UNIX-style directory separators (forward\n> > slash `/`), may contain any byte other than `LF`, and must not\n> > start with double quote (`\"`).\n> > \n> > -If an `LF` or double quote must be encoded into `` shell-style\n> > -quoting should be used, e.g. `\"path/with\\n and \\\" in it\"`.\n> > +A path can use C-style string quoting; this is accepted in all cases\n> > +and mandatory if the filename starts with double quote or contains\n> > +`LF`.\n> \n> ... or backslash?\n\nNo, that was what we discussed elsewhere in the thread. It is OK to say:\n\n M 100644 :1 file \\with \\backslashes\n\nas de-quoting is triggered by the first character being double-quote.\n\n-Peff\n"},{"id":"204329","messageId":"vpqvccnctvd.fsf@grenoble-inp.fr","threadId":"32239","inReplyTo":"7vvccop6dv.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 1/2] git-fast-import.txt: improve documentation for quoted paths","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2012-11-30T09:39:18Z","receivedAt":"2012-11-30T09:39:18Z","isPatch":true,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Junio C Hamano writes:\n\n> That \"shell-style\" contradicts with what fast-import.c says, though.\n> It claims to grok \\octal and described as C-style.\n\nAs Peff mentionned, my last version is better, although still a bit\nincomplete. My new version documents things that _must_ be escaped, but\nnot what _can_.\n\nI'm reluctant to try being exhaustive in the .txt documentation if there\nis an exhaustive comment in the code. One option would be to actually\nturn the comment into an official specification by moving it to the .txt\nfile, but that goes far beyond the scope of my patch.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"204406","messageId":"7v8v9hmbnh.fsf@alter.siamese.dyndns.org","threadId":"32239","inReplyTo":"vpqvccnctvd.fsf@grenoble-inp.fr","subject":"Re: [PATCH 1/2] git-fast-import.txt: improve documentation for quoted paths","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-12-02T02:27:14Z","receivedAt":"2012-12-02T02:27:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthieu Moy writes:\n\n> Junio C Hamano writes:\n>\n>> That \"shell-style\" contradicts with what fast-import.c says, though.\n>> It claims to grok \\octal and described as C-style.\n>\n> As Peff mentionned, my last version is better, although still a bit\n> incomplete. My new version documents things that _must_ be escaped, but\n> not what _can_.\n\nYeah, I agree.\n"}]}