{"thread":{"id":"33271","subject":"Bug: `gitsubmodule` does not list modules with unicode characters","startedAt":"2013-03-23T16:28:46Z","lastAt":"2013-06-08T12:06:00Z","messageCount":5,"participants":["Ilya Kulakov","Jens Lehmann","Fredrik Gustafsson","Jonathan Nieder"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"212059","messageId":"loom.20130323T171809-46@post.gmane.org","threadId":"33271","inReplyTo":null,"subject":"Bug: `gitsubmodule` does not list modules with unicode characters","fromName":"Ilya Kulakov","fromEmail":"kulakov.ilya@gmail.com","sentAt":"2013-03-23T16:28:46Z","receivedAt":"2013-03-23T16:28:46Z","isPatch":false,"sender":{"key":"kulakov.ilya@gmail.com","avatar":null},"body":"The `git submodule` commands seem to ignore modules which paths contain\nunicode characters.\n\nConsider the following steps to reproduce the problem:\n\n 1. Create a directory with name that contains at least one unicode character\n (e.g. \"ûñïçödé-rèpø\")\n\n 2. Initialize git repository within this directory\n\n 3. Add this repository as a submodule to another repository so that\n unicode characters will appear in the path to the module\n (e.g. \"../ûñïçödé-rèpø\")\n\n 4. Check that .gitmodules file is updated and contains record\n about just added module\n\n 5. List submodules with using `git submodule` and find out\n that just added module is not listed\n"},{"id":"212150","messageId":"51500B34.30801@web.de","threadId":"33271","inReplyTo":"loom.20130323T171809-46@post.gmane.org","subject":"Re: Bug: `gitsubmodule` does not list modules with unicode characters","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2013-03-25T08:30:44Z","receivedAt":"2013-03-25T08:30:44Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 23.03.2013 17:28, schrieb Ilya Kulakov:\n> The `git submodule` commands seem to ignore modules which paths contain\n> unicode characters.\n> \n> Consider the following steps to reproduce the problem:\n> \n> 1. Create a directory with name that contains at least one unicode character\n> (e.g. \"ûñïçödé-rèpø\")\n> \n> 2. Initialize git repository within this directory\n> \n> 3. Add this repository as a submodule to another repository so that\n> unicode characters will appear in the path to the module\n> (e.g. \"../ûñïçödé-rèpø\")\n> \n> 4. Check that .gitmodules file is updated and contains record\n> about just added module\n> \n> 5. List submodules with using `git submodule` and find out\n> that just added module is not listed\n\nThanks for your report. It is known that git submodule does not behave\nvery well when path names contain special characters. I'll look into\nthat when I find some time to see if we can easily fix your problem.\n"},{"id":"219776","messageId":"20130608010556.GA28819@paksenarrion.iveqy.com","threadId":"33271","inReplyTo":"51500B34.30801@web.de","subject":"Re: Bug: `gitsubmodule` does not list modules with unicode characters","fromName":"Fredrik Gustafsson","fromEmail":"iveqy@iveqy.com","sentAt":"2013-06-08T01:05:56Z","receivedAt":"2013-06-08T01:05:56Z","isPatch":false,"sender":{"key":"iveqy@iveqy.com","avatar":"https://avatars.githubusercontent.com/u/761743?v=4"},"body":"On Mon, Mar 25, 2013 at 09:30:44AM +0100, Jens Lehmann wrote:\n> Am 23.03.2013 17:28, schrieb Ilya Kulakov:\n> > The `git submodule` commands seem to ignore modules which paths contain\n> > unicode characters.\n> > \n> > Consider the following steps to reproduce the problem:\n> > \n> > 1. Create a directory with name that contains at least one unicode character\n> > (e.g. \"ûñïçödé-rèpø\")\n> > \n> > 2. Initialize git repository within this directory\n> > \n> > 3. Add this repository as a submodule to another repository so that\n> > unicode characters will appear in the path to the module\n> > (e.g. \"../ûñïçödé-rèpø\")\n> > \n> > 4. Check that .gitmodules file is updated and contains record\n> > about just added module\n> > \n> > 5. List submodules with using `git submodule` and find out\n> > that just added module is not listed\n> \n> Thanks for your report. It is known that git submodule does not behave\n> very well when path names contain special characters. I'll look into\n> that when I find some time to see if we can easily fix your problem.\n\nI've looked into this a bit.\n\ngit ls-files will return all filenames \"c-style quoted\". Hence the\nfilename åäö will be returned as \"303245303244303266\". This is of course\nalso wrong as it should be \"\\303\\245\\303\\244\\303\\266\".\n\nHowever, if we tell git ls-files to use \\0 instead of \\n for line\ntermination. We get åäö returned. So how can the choose of line termination\neffect the encoding?\n\nLook in quote.c. The following patch will solve this particular problem\n(but break other usecases!)\n\ndiff --git a/quote.c b/quote.c\nindex 911229f..2870ca5 100644\n--- a/quote.c\n+++ b/quote.c\n@@ -284,7 +284,7 @@ void quote_two_c_style(struct strbuf *sb, const char *prefix, const char *path,\n void write_name_quoted(const char *name, FILE *fp, int terminator)\n {\n \tif (terminator) {\n-\t\tquote_c_style(name, NULL, fp, 0);\n+\t\tfputs(name, fp);\n \t} else {\n \t\tfputs(name, fp);\n \t}\n\nWhy don't we always print names quoted? IMHO the choose of line\ntermination should not do anything else than alter the line termination.\n\nHowever, an other solution would be to use git ls-files -z in\ngit-submodule.sh and then rewrite the perl-code to handle \\0 instead\nof \\n.\n\n(The same perl-code I wanted to throw away 13 months ago but\nJunio wanted to keep because perl can handle \\0 and eventually -z should\nbe used according to him. He was right.)\n\nHowever, a shortcut would be to the patch below. It will work as long as\nthere's no newline in the filename (is that really something we want to\nsupport? If not, let's throw away perl and stick with the sed solution\nbelow).\n\ndiff --git a/git-submodule.sh b/git-submodule.sh\nindex 79bfaac..31524d3 100755\n--- a/git-submodule.sh\n+++ b/git-submodule.sh\n@@ -113,9 +113,10 @@ resolve_relative_url ()\n module_list()\n {\n \t(\n-\t\tgit ls-files --error-unmatch --stage -- \"$@\" ||\n+\t\tgit ls-files --error-unmatch --stage -z -- \"$@\" ||\n \t\techo \"unmatched pathspec exists\"\n \t) |\n+\tsed -e 's/\\x00/\\n/g' |\n \tperl -e '\n \tmy %unmerged = ();\n \tmy ($null_sha1) = (\"0\" x 40);\n\n-- \nMed vänliga hälsningar\nFredrik Gustafsson\n\ntel: 0733-608274\ne-post: iveqy@iveqy.com\n"},{"id":"219790","messageId":"20130608091836.GA2925@elie.Belkin","threadId":"33271","inReplyTo":"20130608010556.GA28819@paksenarrion.iveqy.com","subject":"Re: Bug: `gitsubmodule` does not list modules with unicode characters","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2013-06-08T09:18:36Z","receivedAt":"2013-06-08T09:18:36Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Fredrik Gustafsson wrote:\n\n> I've looked into this a bit.\n\nThanks for investigating.\n\n[...]\n> Why don't we always print names quoted? IMHO the choose of line\n> termination should not do anything else than alter the line termination.\n>\n> However, an other solution would be to use git ls-files -z in\n> git-submodule.sh and then rewrite the perl-code to handle \\0 instead\n> of \\n.\n\nThe whole point of \"-z\" is that by using a terminator that is guaranteed\nnot to appear in filenames, it avoids the need to quote filenames.\nOtherwise at least \\n would need to be quoted.\n\nHow about something like this patch?\n\n-- >8 --\nSubject: ls-files doc: clarify purpose of \"-z\" option\n\nThe purpose of the \"-z\" option is to avoid quoting issues by using a\ndelimiter that implies a binary-clean parser and cannot appear in\nfilenames, and in that spirit, \"-z\" turns off C-style quoting. But\nwithout looking carefully through the entire manpage, it is too easy\nto miss that.\n\nReported-by: Fredrik Gustafsson \nSigned-off-by: Jonathan Nieder \n---\n Documentation/git-ls-files.txt | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-ls-files.txt b/Documentation/git-ls-files.txt\nindex c0856a6e..753c223f 100644\n--- a/Documentation/git-ls-files.txt\n+++ b/Documentation/git-ls-files.txt\n@@ -75,7 +75,9 @@ OPTIONS\n \tsucceed.\n \n -z::\n-\t\\0 line termination on output.\n+\tTerminate lines with NUL instead of LF.\n+\tThis avoids the need to quote filenames; see the Output section\n+\tbelow for details.\n \n -x ::\n --exclude=::\n-- \n1.8.3\n"},{"id":"219804","messageId":"20130608120600.GA2159@paksenarrion.iveqy.com","threadId":"33271","inReplyTo":"20130608091836.GA2925@elie.Belkin","subject":"Re: Bug: `gitsubmodule` does not list modules with unicode characters","fromName":"Fredrik Gustafsson","fromEmail":"iveqy@iveqy.com","sentAt":"2013-06-08T12:06:00Z","receivedAt":"2013-06-08T12:06:00Z","isPatch":false,"sender":{"key":"iveqy@iveqy.com","avatar":"https://avatars.githubusercontent.com/u/761743?v=4"},"body":"On Sat, Jun 08, 2013 at 02:18:36AM -0700, Jonathan Nieder wrote:\n> The whole point of \"-z\" is that by using a terminator that is guaranteed\n> not to appear in filenames, it avoids the need to quote filenames.\n> Otherwise at least \\n would need to be quoted.\n\nThanks, now I understand why.\n\n> \n> How about something like this patch?\n> \n> -- >8 --\n> Subject: ls-files doc: clarify purpose of \"-z\" option\n> \n> The purpose of the \"-z\" option is to avoid quoting issues by using a\n> delimiter that implies a binary-clean parser and cannot appear in\n> filenames, and in that spirit, \"-z\" turns off C-style quoting. But\n> without looking carefully through the entire manpage, it is too easy\n> to miss that.\n> \n> Reported-by: Fredrik Gustafsson \n> Signed-off-by: Jonathan Nieder \n> ---\n> Documentation/git-ls-files.txt | 4 +++-\n> 1 file changed, 3 insertions(+), 1 deletion(-)\n> \n> diff --git a/Documentation/git-ls-files.txt b/Documentation/git-ls-files.txt\n> index c0856a6e..753c223f 100644\n> --- a/Documentation/git-ls-files.txt\n> +++ b/Documentation/git-ls-files.txt\n> @@ -75,7 +75,9 @@ OPTIONS\n> \tsucceed.\n> \n> -z::\n> -\t\\0 line termination on output.\n> +\tTerminate lines with NUL instead of LF.\n> +\tThis avoids the need to quote filenames; see the Output section\n> +\tbelow for details.\n> \n> -x ::\n> --exclude=::\n> -- \n> 1.8.3\n> \n\nThat would be very helpfull. I would suggest to add something about\nunicode also (and maybe about the quotes that's added?). I'm a bit\nunsure about the formulating but how about something like this:\n\n\nFrom 114c34ea482873b39c02e63eeaf866c3e9ebfc14 Mon Sep 17 00:00:00 2001\nFrom: Jonathan Nieder \nDate: Sat, 8 Jun 2013 02:18:36 -0700\nSubject: [PATCH] Subject: ls-files doc: clarify purpose of \"-z\" option\n\nThe purpose of the \"-z\" option is to avoid quoting issues by using a\ndelimiter that implies a binary-clean parser and cannot appear in\nfilenames, and in that spirit, \"-z\" turns off C-style quoting. But\nwithout looking carefully through the entire manpage, it is too easy\nto miss that.\n\nReported-by: Fredrik Gustafsson \nSigned-off-by: Jonathan Nieder \n---\n Documentation/git-ls-files.txt | 7 +++++--\n 1 file changed, 5 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-ls-files.txt b/Documentation/git-ls-files.txt\nindex c0856a6..ef785ba 100644\n--- a/Documentation/git-ls-files.txt\n+++ b/Documentation/git-ls-files.txt\n@@ -75,7 +75,9 @@ OPTIONS\n \tsucceed.\n \n -z::\n-\t\\0 line termination on output.\n+\tTerminate lines with NUL instead of LF.\n+\tThis avoids the need to quote filenames; see the Output section\n+\tbelow for details.\n \n -x ::\n --exclude=::\n@@ -172,7 +174,8 @@ path. (see linkgit:git-read-tree[1] for more information on state)\n \n When `-z` option is not used, TAB, LF, and backslash characters\n in pathnames are represented as `\\t`, `\\n`, and `\\\\`,\n-respectively.\n+respectively. Multibyte characters are represented by they escaped\n+equivalents.\n \n \n Exclude Patterns\n-- \n1.8.1.5\n\n\n-- \nMed vänliga hälsningar\nFredrik Gustafsson\n\ntel: 0733-608274\ne-post: iveqy@iveqy.com\n"}]}