threads / patch / 41923

patchdiff: run arguments through precompose_argv

Subject: [PATCH] diff: run arguments through precompose_argv

## tl;dr

12 messages between Apr 4, 2016 and May 12, 2016. Diffs are folded; open one to read it.

replies: 11people: 4as markdown or json

Alexander Rinass· Apr 4, 2016, 20:38 UTC · lore

I found an issue where file paths containing decomposed unicode chars are not converted to precomposed unicode form when passed to the diff command family.

As a result, no diff is displayed.

With the help of Torsten Bögershausen and Junio C Hamano I came up with the following patch to fix the issue.

Alexander Rinass (1):
  diff: run arguments through precompose_argv
 builtin/diff-files.c         |  1 +
 builtin/diff-index.c         |  1 +
 builtin/diff-tree.c          |  2 ++
 builtin/diff.c               |  1 +
 t/t3910-mac-os-precompose.sh | 42 ++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 47 insertions(+)
-- 
2.7.2
Alexander Rinass· Apr 4, 2016, 20:38 UTC · re: Alexander Rinass · lore

File paths containing decomposed unicode chars passed to diff command are not converted to precomposed unicode form.

As a result, no diff is displayed when feeding such a file path to the diff command.

Opposite to most builtin commands, the diff builtin is missing the parse_options call, which internally runs arguments through the precompose_argv call, which ensures all arguments are in precomposed unicode form.

Fix the problem by adding a precompose_argv call directly, as a call to parse_options would require additional work to call it.

Also applies to diff-index, diff-files and diff-tree.
Signed-off-by: Alexander Rinass <alex@fournova.com>
Thanks-to: Torsten Bögershausen <tboegi@web.de>
Thanks-to: Junio C Hamano <gitster@pobox.com>
---
 builtin/diff-files.c         |  1 +
 builtin/diff-index.c         |  1 +
 builtin/diff-tree.c          |  2 ++
 builtin/diff.c               |  1 +
 t/t3910-mac-os-precompose.sh | 42 ++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 47 insertions(+)
Show changes to 5 files +47 −0

builtin/diff-files.c, builtin/diff-index.c, builtin/diff-tree.c, builtin/diff.c, t/t3910-mac-os-precompose.sh

diff --git a/builtin/diff-files.c b/builtin/diff-files.c
index 8ed2eb8..15c61fd 100644
--- a/builtin/diff-files.c
+++ b/builtin/diff-files.c
@@ -24,6 +24,7 @@ int cmd_diff_files(int argc, const char **argv, const char *prefix)
 	gitmodules_config();
 	git_config(git_diff_basic_config, NULL); /* no "diff" UI options */
 	rev.abbrev = 0;
+	precompose_argv(argc, argv);
 
 	argc = setup_revisions(argc, argv, &rev, NULL);
 	while (1 < argc && argv[1][0] == '-') {
diff --git a/builtin/diff-index.c b/builtin/diff-index.c
index d979824..1af373d 100644
--- a/builtin/diff-index.c
+++ b/builtin/diff-index.c
@@ -21,6 +21,7 @@ int cmd_diff_index(int argc, const char **argv, const char *prefix)
 	gitmodules_config();
 	git_config(git_diff_basic_config, NULL); /* no "diff" UI options */
 	rev.abbrev = 0;
+	precompose_argv(argc, argv);
 
 	argc = setup_revisions(argc, argv, &rev, NULL);
 	for (i = 1; i < argc; i++) {
diff --git a/builtin/diff-tree.c b/builtin/diff-tree.c
index 2a12b81..806dd7a 100644
--- a/builtin/diff-tree.c
+++ b/builtin/diff-tree.c
@@ -114,6 +114,8 @@ int cmd_diff_tree(int argc, const char **argv, const char *prefix)
 	opt->disable_stdin = 1;
 	memset(&s_r_opt, 0, sizeof(s_r_opt));
 	s_r_opt.tweak = diff_tree_tweak_rev;
+
+	precompose_argv(argc, argv);
 	argc = setup_revisions(argc, argv, opt, &s_r_opt);
 
 	while (--argc > 0) {
diff --git a/builtin/diff.c b/builtin/diff.c
index 52c98a9..d6b8f98 100644
--- a/builtin/diff.c
+++ b/builtin/diff.c
@@ -319,6 +319,7 @@ int cmd_diff(int argc, const char **argv, const char *prefix)
 	if (!no_index)
 		gitmodules_config();
 	git_config(git_diff_ui_config, NULL);
+	precompose_argv(argc, argv);
 
 	init_revisions(&rev, prefix);
 
diff --git a/t/t3910-mac-os-precompose.sh b/t/t3910-mac-os-precompose.sh
index 8319356..26dd5b7 100755
--- a/t/t3910-mac-os-precompose.sh
+++ b/t/t3910-mac-os-precompose.sh
@@ -49,12 +49,54 @@ test_expect_success "setup" '
 test_expect_success "setup case mac" '
 	git checkout -b mac_os
 '
+# This will test nfd2nfc in git diff
+test_expect_success "git diff f.Adiar" '
+	touch f.$Adiarnfc &&
+	git add f.$Adiarnfc &&
+	echo f.Adiarnfc >f.$Adiarnfc &&
+	git diff f.$Adiarnfd >expect &&
+	git diff f.$Adiarnfc >actual &&
+	test_cmp expect actual &&
+	git reset HEAD f.Adiarnfc &&
+	rm f.$Adiarnfc expect actual
+'
+# This will test nfd2nfc in git diff-files
+test_expect_success "git diff-files f.Adiar" '
+	touch f.$Adiarnfc &&
+	git add f.$Adiarnfc &&
+	echo f.Adiarnfc >f.$Adiarnfc &&
+	git diff-files f.$Adiarnfd >expect &&
+	git diff-files f.$Adiarnfc >actual &&
+	test_cmp expect actual &&
+	git reset HEAD f.Adiarnfc &&
+	rm f.$Adiarnfc expect actual
+'
+# This will test nfd2nfc in git diff-index
+test_expect_success "git diff-index f.Adiar" '
+	touch f.$Adiarnfc &&
+	git add f.$Adiarnfc &&
+	echo f.Adiarnfc >f.$Adiarnfc &&
+	git diff-index HEAD f.$Adiarnfd >expect &&
+	git diff-index HEAD f.$Adiarnfc >actual &&
+	test_cmp expect actual &&
+	git reset HEAD f.Adiarnfc &&
+	rm f.$Adiarnfc expect actual
+'
 # This will test nfd2nfc in readdir()
 test_expect_success "add file Adiarnfc" '
 	echo f.Adiarnfc >f.$Adiarnfc &&
 	git add f.$Adiarnfc &&
 	git commit -m "add f.$Adiarnfc"
 '
+# This will test nfd2nfc in git diff-tree
+test_expect_success "git diff-tree f.Adiar" '
+	echo f.Adiarnfc >>f.$Adiarnfc &&
+	git diff-tree HEAD f.$Adiarnfd >expect &&
+	git diff-tree HEAD f.$Adiarnfc >actual &&
+	test_cmp expect actual &&
+	git checkout f.$Adiarnfc &&
+	rm expect actual
+'
 # This will test nfd2nfc in git stage()
 test_expect_success "stage file d.Adiarnfd/f.Adiarnfd" '
 	mkdir d.$Adiarnfd &&
-- 
2.7.2
Junio C Hamano· Apr 5, 2016, 17:09 UTC · re: Alexander Rinass · lore

Re: [PATCH] diff: run arguments through precompose_argv

Alexander Rinass <alex@fournova.com> writes:
Show 15 quoted lines
> File paths containing decomposed unicode chars passed to diff
> command are not converted to precomposed unicode form.
>
> As a result, no diff is displayed when feeding such a file path to the
> diff command.
>
> Opposite to most builtin commands, the diff builtin is missing the
> parse_options call, which internally runs arguments through the
> precompose_argv call, which ensures all arguments are in precomposed
> unicode form.
>
> Fix the problem by adding a precompose_argv call directly, as a call to
> parse_options would require additional work to call it.
>
> Also applies to diff-index, diff-files and diff-tree.

Thanks. The log message talks about "such a file path", but precompose_argv() applies indiscriminately on any command line arguments, so things like -G<string> would also get the same treatment, which I think is what most users would want).

Will queue.
Show 119 quoted lines
> Signed-off-by: Alexander Rinass <alex@fournova.com>
> Thanks-to: Torsten Bögershausen <tboegi@web.de>
> Thanks-to: Junio C Hamano <gitster@pobox.com>
> ---
>  builtin/diff-files.c         |  1 +
>  builtin/diff-index.c         |  1 +
>  builtin/diff-tree.c          |  2 ++
>  builtin/diff.c               |  1 +
>  t/t3910-mac-os-precompose.sh | 42 ++++++++++++++++++++++++++++++++++++++++++
>  5 files changed, 47 insertions(+)
>
> diff --git a/builtin/diff-files.c b/builtin/diff-files.c
> index 8ed2eb8..15c61fd 100644
> --- a/builtin/diff-files.c
> +++ b/builtin/diff-files.c
> @@ -24,6 +24,7 @@ int cmd_diff_files(int argc, const char **argv, const char *prefix)
>  	gitmodules_config();
>  	git_config(git_diff_basic_config, NULL); /* no "diff" UI options */
>  	rev.abbrev = 0;
> +	precompose_argv(argc, argv);
>  
>  	argc = setup_revisions(argc, argv, &rev, NULL);
>  	while (1 < argc && argv[1][0] == '-') {
> diff --git a/builtin/diff-index.c b/builtin/diff-index.c
> index d979824..1af373d 100644
> --- a/builtin/diff-index.c
> +++ b/builtin/diff-index.c
> @@ -21,6 +21,7 @@ int cmd_diff_index(int argc, const char **argv, const char *prefix)
>  	gitmodules_config();
>  	git_config(git_diff_basic_config, NULL); /* no "diff" UI options */
>  	rev.abbrev = 0;
> +	precompose_argv(argc, argv);
>  
>  	argc = setup_revisions(argc, argv, &rev, NULL);
>  	for (i = 1; i < argc; i++) {
> diff --git a/builtin/diff-tree.c b/builtin/diff-tree.c
> index 2a12b81..806dd7a 100644
> --- a/builtin/diff-tree.c
> +++ b/builtin/diff-tree.c
> @@ -114,6 +114,8 @@ int cmd_diff_tree(int argc, const char **argv, const char *prefix)
>  	opt->disable_stdin = 1;
>  	memset(&s_r_opt, 0, sizeof(s_r_opt));
>  	s_r_opt.tweak = diff_tree_tweak_rev;
> +
> +	precompose_argv(argc, argv);
>  	argc = setup_revisions(argc, argv, opt, &s_r_opt);
>  
>  	while (--argc > 0) {
> diff --git a/builtin/diff.c b/builtin/diff.c
> index 52c98a9..d6b8f98 100644
> --- a/builtin/diff.c
> +++ b/builtin/diff.c
> @@ -319,6 +319,7 @@ int cmd_diff(int argc, const char **argv, const char *prefix)
>  	if (!no_index)
>  		gitmodules_config();
>  	git_config(git_diff_ui_config, NULL);
> +	precompose_argv(argc, argv);
>  
>  	init_revisions(&rev, prefix);
>  
> diff --git a/t/t3910-mac-os-precompose.sh b/t/t3910-mac-os-precompose.sh
> index 8319356..26dd5b7 100755
> --- a/t/t3910-mac-os-precompose.sh
> +++ b/t/t3910-mac-os-precompose.sh
> @@ -49,12 +49,54 @@ test_expect_success "setup" '
>  test_expect_success "setup case mac" '
>  	git checkout -b mac_os
>  '
> +# This will test nfd2nfc in git diff
> +test_expect_success "git diff f.Adiar" '
> +	touch f.$Adiarnfc &&
> +	git add f.$Adiarnfc &&
> +	echo f.Adiarnfc >f.$Adiarnfc &&
> +	git diff f.$Adiarnfd >expect &&
> +	git diff f.$Adiarnfc >actual &&
> +	test_cmp expect actual &&
> +	git reset HEAD f.Adiarnfc &&
> +	rm f.$Adiarnfc expect actual
> +'
> +# This will test nfd2nfc in git diff-files
> +test_expect_success "git diff-files f.Adiar" '
> +	touch f.$Adiarnfc &&
> +	git add f.$Adiarnfc &&
> +	echo f.Adiarnfc >f.$Adiarnfc &&
> +	git diff-files f.$Adiarnfd >expect &&
> +	git diff-files f.$Adiarnfc >actual &&
> +	test_cmp expect actual &&
> +	git reset HEAD f.Adiarnfc &&
> +	rm f.$Adiarnfc expect actual
> +'
> +# This will test nfd2nfc in git diff-index
> +test_expect_success "git diff-index f.Adiar" '
> +	touch f.$Adiarnfc &&
> +	git add f.$Adiarnfc &&
> +	echo f.Adiarnfc >f.$Adiarnfc &&
> +	git diff-index HEAD f.$Adiarnfd >expect &&
> +	git diff-index HEAD f.$Adiarnfc >actual &&
> +	test_cmp expect actual &&
> +	git reset HEAD f.Adiarnfc &&
> +	rm f.$Adiarnfc expect actual
> +'
>  # This will test nfd2nfc in readdir()
>  test_expect_success "add file Adiarnfc" '
>  	echo f.Adiarnfc >f.$Adiarnfc &&
>  	git add f.$Adiarnfc &&
>  	git commit -m "add f.$Adiarnfc"
>  '
> +# This will test nfd2nfc in git diff-tree
> +test_expect_success "git diff-tree f.Adiar" '
> +	echo f.Adiarnfc >>f.$Adiarnfc &&
> +	git diff-tree HEAD f.$Adiarnfd >expect &&
> +	git diff-tree HEAD f.$Adiarnfc >actual &&
> +	test_cmp expect actual &&
> +	git checkout f.$Adiarnfc &&
> +	rm expect actual
> +'
>  # This will test nfd2nfc in git stage()
>  test_expect_success "stage file d.Adiarnfd/f.Adiarnfd" '
>  	mkdir d.$Adiarnfd &&
Johannes Sixt· Apr 5, 2016, 19:15 UTC · re: Junio C Hamano · lore

Re: [PATCH] diff: run arguments through precompose_argv

Am 05.04.2016 um 19:09 schrieb Junio C Hamano:
>> Thanks-to: Torsten Bögershausen <tboegi@web.de>

I sense NFD disease: The combining diaresis should combine with the o, not the g. Here is a correct line to copy-and-paste if you like:

Thanks-to: Torsten Bögershausen <tboegi@web.de>
-- Hannes
Torsten Bögershausen· Apr 5, 2016, 19:27 UTC · re: Johannes Sixt · lore

Re: [PATCH] diff: run arguments through precompose_argv

On 05.04.16 21:15, Johannes Sixt wrote:
Show 8 quoted lines
> Am 05.04.2016 um 19:09 schrieb Junio C Hamano:
>>> Thanks-to: Torsten Bögershausen <tboegi@web.de>
> 
> I sense NFD disease: The combining diaresis should combine with the o, not the g. Here is a correct line to copy-and-paste if you like:
> 
> Thanks-to: Torsten Bögershausen <tboegi@web.de>
> 
> -- Hannes
Good eyes.
And thanks to Alexander for doing this patch
Junio C Hamano· Apr 5, 2016, 21:42 UTC · re: Torsten Bögershausen · lore

Re: [PATCH] diff: run arguments through precompose_argv

Torsten Bögershausen <tboegi@web.de> writes:
Show 13 quoted lines
> On 05.04.16 21:15, Johannes Sixt wrote:
>> Am 05.04.2016 um 19:09 schrieb Junio C Hamano:
>>>> Thanks-to: Torsten Bögershausen <tboegi@web.de>
>> 
>> I sense NFD disease: The combining diaresis should combine with the o, not the g. Here is a correct line to copy-and-paste if you like:
>> 
>> Thanks-to: Torsten Bögershausen <tboegi@web.de>
>> 
>> -- Hannes
>
> Good eyes.
>
> And thanks to Alexander for doing this patch
Yup.  It is funny that I wrote:
    Thanks.  The log message talks about "such a file path", but
    precompose_argv() applies indiscriminately on any command line
    arguments, so things like -G<string> would also get the same
    treatment, which I think is what most users would want.

but "git log --grep=" with your name would not have found the one that would have resulted from the original e-mail message applied.

Alexander Rinass· Apr 6, 2016, 06:51 UTC · re: Johannes Sixt · lore

Re: [PATCH] diff: run arguments through precompose_argv

Show 10 quoted lines
> On 05 Apr 2016, at 21:15, Johannes Sixt <j6t@kdbg.org> wrote:
> 
> Am 05.04.2016 um 19:09 schrieb Junio C Hamano:
>>> Thanks-to: Torsten Bögershausen <tboegi@web.de>
> 
> I sense NFD disease: The combining diaresis should combine with the o, not the g. Here is a correct line to copy-and-paste if you like:
> 
> Thanks-to: Torsten Bögershausen <tboegi@web.de>
> 
> -- Hannes
Thanks for reviewing and catching the NFD encoding error.
I will send in a patch v2 with the correct NFC encoding.
Would you also like me to alter the commit message as mentioned by Junio?
I could rewrite the sentence:

“As a result, no diff is displayed when feeding such a file path to the diff command.”

into simply saying:
“As a result, no diff is displayed.”

However, I don't read the original message as it would imply that only file paths are affected by the precompose_argv call.

Are there other suggestions on improving the commit message?
Torsten Bögershausen· Apr 6, 2016, 07:47 UTC · re: Alexander Rinass · lore

Re: [PATCH] diff: run arguments through precompose_argv

On 06.04.16 08:51, Alexander Rinass wrote:
Show 31 quoted lines
> 
>> On 05 Apr 2016, at 21:15, Johannes Sixt <j6t@kdbg.org> wrote:
>>
>> Am 05.04.2016 um 19:09 schrieb Junio C Hamano:
>>>> Thanks-to: Torsten Bögershausen <tboegi@web.de>
>>
>> I sense NFD disease: The combining diaresis should combine with the o, not the g. Here is a correct line to copy-and-paste if you like:
>>
>> Thanks-to: Torsten Bögershausen <tboegi@web.de>
>>
>> -- Hannes
> 
> Thanks for reviewing and catching the NFD encoding error.
> 
> I will send in a patch v2 with the correct NFC encoding.
> 
> Would you also like me to alter the commit message as mentioned by Junio?
> 
> I could rewrite the sentence:
> 
> “As a result, no diff is displayed when feeding such a file path to the
> diff command.”
> 
> into simply saying:
> 
> “As a result, no diff is displayed.”
> 
> However, I don't read the original message as it would imply that only
> file paths are affected by the precompose_argv call. 
> 
> Are there other suggestions on improving the commit message?
May be something like this, (but this is highly a personal taste question)

When running diff commands, file paths containing decomposed unicode code points are not converted to precomposed unicode form under Mac OS X.

As a result, no diff is displayed.

Opposite to most builtin commands, the diff builtin is missing the parse_options call, which internally runs arguments through the precompose_argv call, which ensures all arguments are in precomposed unicode form.

Fix the problem by adding a precompose_argv call to diff, diff-index, diff-files and diff-tree.
Signed-off-by: Alexander Rinass <alex@fournova.com>
Helped-By: Torsten Bögershausen <tboegi@web.de>
Helped-By: Junio C Hamano <gitster@pobox.com>
Junio C Hamano· Apr 6, 2016, 17:27 UTC · re: Torsten Bögershausen · lore

Re: [PATCH] diff: run arguments through precompose_argv

Torsten Bögershausen <tboegi@web.de> writes:
> May be something like this, (but this is highly a personal taste question)

My observation was not a suggestion to rewrite the log message, but primarily about describing the change in behaviour.

I didn't mean to nitpick the phrasing but let me think aloud about it, using this as a template. I rewrapped overlong lines from the original.

Show 5 quoted lines
> When running diff commands, file paths containing decomposed
> unicode code points are not converted to precomposed unicode form
> under Mac OS X.
>
> As a result, no diff is displayed.

Add ", but we normalize the paths in the index and the history to precomposed form on that platform." at the end of the first sentence, to make sure you mention both causes of "As a result" that follows. Also, these are better to be a single paragraph. I.e.

    When running diff commands, a pathspec containing decomposed
    unicode code points is not converted to precomposed unicode form
    under Mac OS X, but we normalize the paths in the index and the
    history to precomposed form on that platform.  As a result, the
    pathspec would not match and no diff is shown.
Show 7 quoted lines
> Opposite to most builtin commands, the diff builtin is missing the
> parse_options call, which internally runs arguments through the
> precompose_argv call, which ensures all arguments are in precomposed
> unicode form.
>
> Fix the problem by adding a precompose_argv call to diff,
> diff-index, diff-files and diff-tree.

Perhaps it is just me but I found "Opposite to most" harder to grok for some reason.

And the "backward compatibility note" is missing. I'd write these two paragraphs like this if I were doing this patch.

    Unlike many builtin commands, the "diff" family of commands do
    not use parse_options(), which is how other builtin commands
    indirectly call precompose_argv() to normalize argv[] into
    precomposed form on Mac OSX.  Teach these commands to call
    precompose_argv() themselves.
    Note that precomopose_argv() normalizes not just paths but all
    command line arguments, so things like "git diff -G $string"
    when $string has the decomposed form would first normalized into
    the precomposed form and would stop hitting the same string in
    the decomposed form in the diff output with this change.  It is
    not a problem per-se, as "log" family of commands already use
    parse_options() and call precompose_argv()--we can think of it
    as making the "diff" family of commands behave in a similar way
    as the commands in the "log" family.
Junio C Hamano· May 11, 2016, 22:08 UTC · re: Alexander Rinass · lore

Re: [PATCH] diff: run arguments through precompose_argv

Alexander Rinass <alex@fournova.com> writes:
Show 30 quoted lines
>> On 05 Apr 2016, at 21:15, Johannes Sixt <j6t@kdbg.org> wrote:
>> 
>> Am 05.04.2016 um 19:09 schrieb Junio C Hamano:
>>>> Thanks-to: Torsten Bögershausen <tboegi@web.de>
>> 
>> I sense NFD disease: The combining diaresis should combine with the o, not the g. Here is a correct line to copy-and-paste if you like:
>> 
>> Thanks-to: Torsten Bögershausen <tboegi@web.de>
>> 
>> -- Hannes
>
> Thanks for reviewing and catching the NFD encoding error.
>
> I will send in a patch v2 with the correct NFC encoding.
>
> Would you also like me to alter the commit message as mentioned by Junio?
>
> I could rewrite the sentence:
>
> “As a result, no diff is displayed when feeding such a file path to the
> diff command.”
>
> into simply saying:
>
> “As a result, no diff is displayed.”
>
> However, I don't read the original message as it would imply that only
> file paths are affected by the precompose_argv call. 
>
> Are there other suggestions on improving the commit message?

I think after this message there were a few suggestions, and then we heard nothing. Should we still be waiting for a response from you?

Thanks.
Alexander Rinass· May 12, 2016, 11:16 UTC · re: Junio C Hamano · lore

Re: [PATCH] diff: run arguments through precompose_argv

Show 39 quoted lines
> On 12 May 2016, at 00:08, Junio C Hamano <gitster@pobox.com> wrote:
> 
> Alexander Rinass <alex@fournova.com> writes:
> 
>>> On 05 Apr 2016, at 21:15, Johannes Sixt <j6t@kdbg.org> wrote:
>>> 
>>> Am 05.04.2016 um 19:09 schrieb Junio C Hamano:
>>>>> Thanks-to: Torsten Bögershausen <tboegi@web.de>
>>> 
>>> I sense NFD disease: The combining diaresis should combine with the o, not the g. Here is a correct line to copy-and-paste if you like:
>>> 
>>> Thanks-to: Torsten Bögershausen <tboegi@web.de>
>>> 
>>> -- Hannes
>> 
>> Thanks for reviewing and catching the NFD encoding error.
>> 
>> I will send in a patch v2 with the correct NFC encoding.
>> 
>> Would you also like me to alter the commit message as mentioned by Junio?
>> 
>> I could rewrite the sentence:
>> 
>> “As a result, no diff is displayed when feeding such a file path to the
>> diff command.”
>> 
>> into simply saying:
>> 
>> “As a result, no diff is displayed.”
>> 
>> However, I don't read the original message as it would imply that only
>> file paths are affected by the precompose_argv call. 
>> 
>> Are there other suggestions on improving the commit message?
> 
> I think after this message there were a few suggestions, and then we
> heard nothing.  Should we still be waiting for a response from you?
> 
> Thanks.
Sorry for not replying earlier. 
I will create a v2 patch until the weekend and send it to the mailing list.

← back to recent threads