# [Question] Unicode weirdness breaking tests on ZFS? 10 messages from 2021-11-17 to 2021-11-19. Participants: Derrick Stolee, Ævar Arnfjörð Bjarmason, Torsten Bögershausen, Torsten B��gershausen, Junio C Hamano. Thread: https://gitlist.dev/t/56927 ## Derrick Stolee, 2021-11-17 15:17 Subject: [Question] Unicode weirdness breaking tests on ZFS? Message-ID: <9393e572-0666-6485-df29-abad5e0d32a1@gmail.com> URL: https://gitlist.dev/e/9393e572-0666-6485-df29-abad5e0d32a1%40gmail.com ``` I recently had to pave my Linux machine, so I updated it to Ubuntu 21.10 and had the choice to start using the ZFS filesystem. I thought, "Why not?" but now I maybe see why. Running the Git test suite at the v2.34.0 tag on my machine results in these failures: t0050-filesystem.sh (Wstat: 0 Tests: 11 Failed: 0) TODO passed: 9-10 t0021-conversion.sh (Wstat: 256 Tests: 41 Failed: 1) Failed test: 31 Non-zero exit status: 1 t3910-mac-os-precompose.sh (Wstat: 256 Tests: 25 Failed: 10) Failed tests: 1, 4, 6, 8, 11-16 TODO passed: 23 Non-zero exit status: 1 These are all related to the UTF8_NFD_TO_NFC prereq. Zooming in on t0050, these tests are marked as "test_expect_failure" due to an assignment of $test_unicode using the UTF8_NFD_TO_NFC prereq: $test_unicode 'rename (silent unicode normalization)' ' git mv "$aumlcdiar" "$auml" && git commit -m rename ' $test_unicode 'merge (silent unicode normalization)' ' git reset --hard initial && git merge topic ' The prereq creates two files using unicode characters that could collapse to equivalent meanings: test_lazy_prereq UTF8_NFD_TO_NFC ' # check whether FS converts nfd unicode to nfc auml=$(printf "\303\244") aumlcdiar=$(printf "\141\314\210") >"$auml" && test -f "$aumlcdiar" ' What I see in that first test, the 'git mv' does change the index, but the filesystem thinks the files are the same. This may mean that our 'git add "$aumlcdiar"' from an earlier test is providing a non-equivalence in the index, and the 'git mv' changes the index without causing any issues in the filesystem. It reminds me as if we used 'git mv README readme' on a case- insensitive filesystem. Is this not a similar situation? What I'm trying to gather is that maybe this test is flawed? Or maybe something broke (or never worked?) in how we use 'git add' to not get the canonical unicode from the filesystem? The other tests all have similar interactions with 'git add'. I'm hoping that these are just test bugs, and not actually a functionality issue in Git. Yes, it is confusing that we can change the unicode of a file in the index without the filesystem understanding the difference, but that is very similar to how case-insensitive filesystems work and I don't know what else we would do here. These filesystem/unicode things are out of my expertise, so hopefully someone else has a clearer idea of what is going on. I'm happy to be a test bed, or even attempt producing patches to fix the issue once we have that clarity. Thanks, -Stolee ``` ## Ævar Arnfjörð Bjarmason, 2021-11-17 15:41 Subject: Re: [Question] Unicode weirdness breaking tests on ZFS? Message-ID: <211117.8635nu7pm3.gmgdl@evledraar.gmail.com> URL: https://gitlist.dev/e/211117.8635nu7pm3.gmgdl%40evledraar.gmail.com In-Reply-To: <9393e572-0666-6485-df29-abad5e0d32a1@gmail.com> ``` On Wed, Nov 17 2021, Derrick Stolee wrote: > I recently had to pave my Linux machine, so I updated it to Ubuntu > 21.10 and had the choice to start using the ZFS filesystem. I thought, > "Why not?" but now I maybe see why. > > Running the Git test suite at the v2.34.0 tag on my machine results in > these failures: > > t0050-filesystem.sh (Wstat: 0 Tests: 11 Failed: 0) > TODO passed: 9-10 > t0021-conversion.sh (Wstat: 256 Tests: 41 Failed: 1) > Failed test: 31 > Non-zero exit status: 1 > t3910-mac-os-precompose.sh (Wstat: 256 Tests: 25 Failed: 10) > Failed tests: 1, 4, 6, 8, 11-16 > TODO passed: 23 > Non-zero exit status: 1 > > These are all related to the UTF8_NFD_TO_NFC prereq. > > Zooming in on t0050, these tests are marked as "test_expect_failure" due > to an assignment of $test_unicode using the UTF8_NFD_TO_NFC prereq: > > > $test_unicode 'rename (silent unicode normalization)' ' > git mv "$aumlcdiar" "$auml" && > git commit -m rename > ' > > $test_unicode 'merge (silent unicode normalization)' ' > git reset --hard initial && > git merge topic > ' > > > The prereq creates two files using unicode characters that could > collapse to equivalent meanings: > > > test_lazy_prereq UTF8_NFD_TO_NFC ' > # check whether FS converts nfd unicode to nfc > auml=$(printf "\303\244") > aumlcdiar=$(printf "\141\314\210") > >"$auml" && > test -f "$aumlcdiar" > ' > > > What I see in that first test, the 'git mv' does change the > index, but the filesystem thinks the files are the same. This > may mean that our 'git add "$aumlcdiar"' from an earlier test > is providing a non-equivalence in the index, and the 'git mv' > changes the index without causing any issues in the filesystem. > > It reminds me as if we used 'git mv README readme' on a case- > insensitive filesystem. Is this not a similar situation? > > What I'm trying to gather is that maybe this test is flawed? > Or maybe something broke (or never worked?) in how we use > 'git add' to not get the canonical unicode from the filesystem? > > The other tests all have similar interactions with 'git add'. > I'm hoping that these are just test bugs, and not actually a > functionality issue in Git. Yes, it is confusing that we can > change the unicode of a file in the index without the filesystem > understanding the difference, but that is very similar to how > case-insensitive filesystems work and I don't know what else we > would do here. > > These filesystem/unicode things are out of my expertise, so > hopefully someone else has a clearer idea of what is going on. > I'm happy to be a test bed, or even attempt producing patches > to fix the issue once we have that clarity. I haven't used ZFS, but this points to non-POSIX behavior on the FS itself. It looks like tweaking the "normalization" property might change it, see: https://manpages.ubuntu.com/manpages/eoan/man8/zfs.8.html There's also "casesensitivity" and "utf8only". We probably don't want to invoke some ZFS command on every test to interrogate this, but if we can pass it down from GIT-BUILD-OPTIONS or similar then we could have a test prereq check this. Or perhaps it's as simple as changing the "UTF8_NFD_TO_NFC" prereq from doing a "test -f" to e.g. "echo *" and seeing what it gets back. Perhaps ZFS says "yes" to "it exists?" but when doing a readdir() it will canonicalize? ``` ## Torsten Bögershausen, 2021-11-17 16:12 Subject: Re: [Question] Unicode weirdness breaking tests on ZFS? Message-ID: <20211117161226.xcat77ewhf5inaif@tb-raspi4> URL: https://gitlist.dev/e/20211117161226.xcat77ewhf5inaif%40tb-raspi4 In-Reply-To: <9393e572-0666-6485-df29-abad5e0d32a1@gmail.com> ``` On Wed, Nov 17, 2021 at 10:17:53AM -0500, Derrick Stolee wrote: > I recently had to pave my Linux machine, so I updated it to Ubuntu > 21.10 and had the choice to start using the ZFS filesystem. I thought, > "Why not?" but now I maybe see why. > > Running the Git test suite at the v2.34.0 tag on my machine results in > these failures: > > t0050-filesystem.sh (Wstat: 0 Tests: 11 Failed: 0) > TODO passed: 9-10 > t0021-conversion.sh (Wstat: 256 Tests: 41 Failed: 1) > Failed test: 31 > Non-zero exit status: 1 > t3910-mac-os-precompose.sh (Wstat: 256 Tests: 25 Failed: 10) > Failed tests: 1, 4, 6, 8, 11-16 > TODO passed: 23 > Non-zero exit status: 1 > > These are all related to the UTF8_NFD_TO_NFC prereq. > > Zooming in on t0050, these tests are marked as "test_expect_failure" due > to an assignment of $test_unicode using the UTF8_NFD_TO_NFC prereq: > > > $test_unicode 'rename (silent unicode normalization)' ' > git mv "$aumlcdiar" "$auml" && > git commit -m rename > ' > > $test_unicode 'merge (silent unicode normalization)' ' > git reset --hard initial && > git merge topic > ' > > > The prereq creates two files using unicode characters that could > collapse to equivalent meanings: > > > test_lazy_prereq UTF8_NFD_TO_NFC ' > # check whether FS converts nfd unicode to nfc > auml=$(printf "\303\244") > aumlcdiar=$(printf "\141\314\210") > >"$auml" && > test -f "$aumlcdiar" > ' > > > What I see in that first test, the 'git mv' does change the > index, but the filesystem thinks the files are the same. This > may mean that our 'git add "$aumlcdiar"' from an earlier test > is providing a non-equivalence in the index, and the 'git mv' > changes the index without causing any issues in the filesystem. > > It reminds me as if we used 'git mv README readme' on a case- > insensitive filesystem. Is this not a similar situation? > > What I'm trying to gather is that maybe this test is flawed? > Or maybe something broke (or never worked?) in how we use > 'git add' to not get the canonical unicode from the filesystem? > > The other tests all have similar interactions with 'git add'. > I'm hoping that these are just test bugs, and not actually a > functionality issue in Git. Yes, it is confusing that we can > change the unicode of a file in the index without the filesystem > understanding the difference, but that is very similar to how > case-insensitive filesystems work and I don't know what else we > would do here. > > These filesystem/unicode things are out of my expertise, so > hopefully someone else has a clearer idea of what is going on. > I'm happy to be a test bed, or even attempt producing patches > to fix the issue once we have that clarity. > > Thanks, > -Stolee Interesting. The tests have always been working on HFS+, then we got APFS (and needed a small fix) and now ZFS. I'll can have a look - just installing in a virtual machine. ``` ## Torsten B��gershausen, 2021-11-17 17:06 Subject: Re: [Question] Unicode weirdness breaking tests on ZFS? Message-ID: <20211117170613.kyoe6ov2m5wi2i56@tb-raspi4> URL: https://gitlist.dev/e/20211117170613.kyoe6ov2m5wi2i56%40tb-raspi4 In-Reply-To: <20211117161226.xcat77ewhf5inaif@tb-raspi4> ``` On Wed, Nov 17, 2021 at 05:12:26PM +0100, Torsten B??gershausen wrote: > On Wed, Nov 17, 2021 at 10:17:53AM -0500, Derrick Stolee wrote: > > I recently had to pave my Linux machine, so I updated it to Ubuntu > > 21.10 and had the choice to start using the ZFS filesystem. I thought, > > "Why not?" but now I maybe see why. > > > > Running the Git test suite at the v2.34.0 tag on my machine results in > > these failures: > > > > t0050-filesystem.sh (Wstat: 0 Tests: 11 Failed: 0) > > TODO passed: 9-10 > > t0021-conversion.sh (Wstat: 256 Tests: 41 Failed: 1) > > Failed test: 31 > > Non-zero exit status: 1 > > t3910-mac-os-precompose.sh (Wstat: 256 Tests: 25 Failed: 10) > > Failed tests: 1, 4, 6, 8, 11-16 > > TODO passed: 23 > > Non-zero exit status: 1 > > > > These are all related to the UTF8_NFD_TO_NFC prereq. > > > > Zooming in on t0050, these tests are marked as "test_expect_failure" due > > to an assignment of $test_unicode using the UTF8_NFD_TO_NFC prereq: > > > > > > $test_unicode 'rename (silent unicode normalization)' ' > > git mv "$aumlcdiar" "$auml" && > > git commit -m rename > > ' > > > > $test_unicode 'merge (silent unicode normalization)' ' > > git reset --hard initial && > > git merge topic > > ' > > > > > > The prereq creates two files using unicode characters that could > > collapse to equivalent meanings: > > > > > > test_lazy_prereq UTF8_NFD_TO_NFC ' > > # check whether FS converts nfd unicode to nfc > > auml=$(printf "\303\244") > > aumlcdiar=$(printf "\141\314\210") > > >"$auml" && > > test -f "$aumlcdiar" > > ' > > > > > > What I see in that first test, the 'git mv' does change the > > index, but the filesystem thinks the files are the same. This > > may mean that our 'git add "$aumlcdiar"' from an earlier test > > is providing a non-equivalence in the index, and the 'git mv' > > changes the index without causing any issues in the filesystem. > > > > It reminds me as if we used 'git mv README readme' on a case- > > insensitive filesystem. Is this not a similar situation? > > > > What I'm trying to gather is that maybe this test is flawed? > > Or maybe something broke (or never worked?) in how we use > > 'git add' to not get the canonical unicode from the filesystem? > > > > The other tests all have similar interactions with 'git add'. > > I'm hoping that these are just test bugs, and not actually a > > functionality issue in Git. Yes, it is confusing that we can > > change the unicode of a file in the index without the filesystem > > understanding the difference, but that is very similar to how > > case-insensitive filesystems work and I don't know what else we > > would do here. > > > > These filesystem/unicode things are out of my expertise, so > > hopefully someone else has a clearer idea of what is going on. > > I'm happy to be a test bed, or even attempt producing patches > > to fix the issue once we have that clarity. > > > > Thanks, > > -Stolee > > Interesting. > The tests have always been working on HFS+, then we got > APFS (and needed a small fix) and now ZFS. > > I'll can have a look - just installing in a virtual machine. So, the virtual machine is up-and-running. I got 2 messages: ok 9 - rename (silent unicode normalization) # TODO known breakage vanished ok 10 - merge (silent unicode normalization) # TODO known breakage vanished Do you get the same ? ``` ## Torsten B��gershausen, 2021-11-17 17:39 Subject: Re: [Question] Unicode weirdness breaking tests on ZFS? Message-ID: <20211117173924.maporsti5cz2ixsu@tb-raspi4> URL: https://gitlist.dev/e/20211117173924.maporsti5cz2ixsu%40tb-raspi4 In-Reply-To: <20211117170613.kyoe6ov2m5wi2i56@tb-raspi4> ``` On Wed, Nov 17, 2021 at 06:06:13PM +0100, Torsten B??gershausen wrote: > On Wed, Nov 17, 2021 at 05:12:26PM +0100, Torsten B??gershausen wrote: > > On Wed, Nov 17, 2021 at 10:17:53AM -0500, Derrick Stolee wrote: > > > I recently had to pave my Linux machine, so I updated it to Ubuntu > > > 21.10 and had the choice to start using the ZFS filesystem. I thought, > > > "Why not?" but now I maybe see why. > > > > > > Running the Git test suite at the v2.34.0 tag on my machine results in > > > these failures: > > > > > > t0050-filesystem.sh (Wstat: 0 Tests: 11 Failed: 0) > > > TODO passed: 9-10 > > > t0021-conversion.sh (Wstat: 256 Tests: 41 Failed: 1) > > > Failed test: 31 > > > Non-zero exit status: 1 > > > t3910-mac-os-precompose.sh (Wstat: 256 Tests: 25 Failed: 10) > > > Failed tests: 1, 4, 6, 8, 11-16 > > > TODO passed: 23 > > > Non-zero exit status: 1 > > > > > > These are all related to the UTF8_NFD_TO_NFC prereq. > > > > > > Zooming in on t0050, these tests are marked as "test_expect_failure" due > > > to an assignment of $test_unicode using the UTF8_NFD_TO_NFC prereq: > > > > > > > > > $test_unicode 'rename (silent unicode normalization)' ' > > > git mv "$aumlcdiar" "$auml" && > > > git commit -m rename > > > ' > > > > > > $test_unicode 'merge (silent unicode normalization)' ' > > > git reset --hard initial && > > > git merge topic > > > ' > > > > > > > > > The prereq creates two files using unicode characters that could > > > collapse to equivalent meanings: > > > > > > > > > test_lazy_prereq UTF8_NFD_TO_NFC ' > > > # check whether FS converts nfd unicode to nfc > > > auml=$(printf "\303\244") > > > aumlcdiar=$(printf "\141\314\210") > > > >"$auml" && > > > test -f "$aumlcdiar" > > > ' > > > > > > > > > What I see in that first test, the 'git mv' does change the > > > index, but the filesystem thinks the files are the same. This > > > may mean that our 'git add "$aumlcdiar"' from an earlier test > > > is providing a non-equivalence in the index, and the 'git mv' > > > changes the index without causing any issues in the filesystem. > > > > > > It reminds me as if we used 'git mv README readme' on a case- > > > insensitive filesystem. Is this not a similar situation? > > > > > > What I'm trying to gather is that maybe this test is flawed? > > > Or maybe something broke (or never worked?) in how we use > > > 'git add' to not get the canonical unicode from the filesystem? > > > > > > The other tests all have similar interactions with 'git add'. > > > I'm hoping that these are just test bugs, and not actually a > > > functionality issue in Git. Yes, it is confusing that we can > > > change the unicode of a file in the index without the filesystem > > > understanding the difference, but that is very similar to how > > > case-insensitive filesystems work and I don't know what else we > > > would do here. > > > > > > These filesystem/unicode things are out of my expertise, so > > > hopefully someone else has a clearer idea of what is going on. > > > I'm happy to be a test bed, or even attempt producing patches > > > to fix the issue once we have that clarity. > > > > > > Thanks, > > > -Stolee > > > > Interesting. > > The tests have always been working on HFS+, then we got > > APFS (and needed a small fix) and now ZFS. > > > > I'll can have a look - just installing in a virtual machine. > > So, the virtual machine is up-and-running. > > I got 2 messages: > > ok 9 - rename (silent unicode normalization) # TODO known breakage vanished > ok 10 - merge (silent unicode normalization) # TODO known breakage vanished > > Do you get the same ? Now I am even more puzzled. running t0050 with -x gives this: Author: A U Thor 1 file changed, 0 insertions(+), 0 deletions(-) rename "a\314\210" => "\303\244" (100%) ok 9 - rename (silent unicode normalization) # TODO known breakage vanished ---------------- When I create a test Git, with one file in ä-decomposed, and rename into ä-precomposed, Git gives me: tb@Ubuntu2021:~/ttt$ git mv "$aumlcdiar" "$auml" fatal: destination exists, source=ä, destination=ä and in hex form: tb@Ubuntu2021:~/ttt$ git mv "$aumlcdiar" "$auml" 2>&1 | xxd 00000000: 6661 7461 6c3a 2064 6573 7469 6e61 7469 fatal: destinati 00000010: 6f6e 2065 7869 7374 732c 2073 6f75 7263 on exists, sourc 00000020: 653d 61cc 882c 2064 6573 7469 6e61 7469 e=a.., destinati 00000030: 6f6e 3dc3 a40a on=... ``` ## Derrick Stolee, 2021-11-17 18:29 Subject: Re: [Question] Unicode weirdness breaking tests on ZFS? Message-ID: <8a3d0d0e-cc82-b696-00f4-b71e6452e1bd@gmail.com> URL: https://gitlist.dev/e/8a3d0d0e-cc82-b696-00f4-b71e6452e1bd%40gmail.com In-Reply-To: <20211117173924.maporsti5cz2ixsu@tb-raspi4> ``` On 11/17/2021 12:39 PM, Torsten =?unknown-8bit?Q?B=C3=B6gershausen?= wrote: > On Wed, Nov 17, 2021 at 06:06:13PM +0100, Torsten B??gershausen wrote: >> On Wed, Nov 17, 2021 at 05:12:26PM +0100, Torsten B??gershausen wrote: >>> On Wed, Nov 17, 2021 at 10:17:53AM -0500, Derrick Stolee wrote: >>>> I recently had to pave my Linux machine, so I updated it to Ubuntu >>>> 21.10 and had the choice to start using the ZFS filesystem. I thought, >>>> "Why not?" but now I maybe see why. >>>> >>>> Running the Git test suite at the v2.34.0 tag on my machine results in >>>> these failures: >>>> >>>> t0050-filesystem.sh (Wstat: 0 Tests: 11 Failed: 0) >>>> TODO passed: 9-10 >>>> t0021-conversion.sh (Wstat: 256 Tests: 41 Failed: 1) >>>> Failed test: 31 >>>> Non-zero exit status: 1 >>>> t3910-mac-os-precompose.sh (Wstat: 256 Tests: 25 Failed: 10) >>>> Failed tests: 1, 4, 6, 8, 11-16 >>>> TODO passed: 23 >>>> Non-zero exit status: 1 >>>> >>>> These are all related to the UTF8_NFD_TO_NFC prereq. >>>> >>>> Zooming in on t0050, these tests are marked as "test_expect_failure" due >>>> to an assignment of $test_unicode using the UTF8_NFD_TO_NFC prereq: >>>> >>>> >>>> $test_unicode 'rename (silent unicode normalization)' ' >>>> git mv "$aumlcdiar" "$auml" && >>>> git commit -m rename >>>> ' >>>> >>>> $test_unicode 'merge (silent unicode normalization)' ' >>>> git reset --hard initial && >>>> git merge topic >>>> ' >>>> >>>> >>>> The prereq creates two files using unicode characters that could >>>> collapse to equivalent meanings: >>>> >>>> >>>> test_lazy_prereq UTF8_NFD_TO_NFC ' >>>> # check whether FS converts nfd unicode to nfc >>>> auml=$(printf "\303\244") >>>> aumlcdiar=$(printf "\141\314\210") >>>> >"$auml" && >>>> test -f "$aumlcdiar" >>>> ' >>>> >>>> >>>> What I see in that first test, the 'git mv' does change the >>>> index, but the filesystem thinks the files are the same. This >>>> may mean that our 'git add "$aumlcdiar"' from an earlier test >>>> is providing a non-equivalence in the index, and the 'git mv' >>>> changes the index without causing any issues in the filesystem. >>>> >>>> It reminds me as if we used 'git mv README readme' on a case- >>>> insensitive filesystem. Is this not a similar situation? >>>> >>>> What I'm trying to gather is that maybe this test is flawed? >>>> Or maybe something broke (or never worked?) in how we use >>>> 'git add' to not get the canonical unicode from the filesystem? >>>> >>>> The other tests all have similar interactions with 'git add'. >>>> I'm hoping that these are just test bugs, and not actually a >>>> functionality issue in Git. Yes, it is confusing that we can >>>> change the unicode of a file in the index without the filesystem >>>> understanding the difference, but that is very similar to how >>>> case-insensitive filesystems work and I don't know what else we >>>> would do here. >>>> >>>> These filesystem/unicode things are out of my expertise, so >>>> hopefully someone else has a clearer idea of what is going on. >>>> I'm happy to be a test bed, or even attempt producing patches >>>> to fix the issue once we have that clarity. >>>> >>>> Thanks, >>>> -Stolee >>> >>> Interesting. >>> The tests have always been working on HFS+, then we got >>> APFS (and needed a small fix) and now ZFS. >>> >>> I'll can have a look - just installing in a virtual machine. >> >> So, the virtual machine is up-and-running. >> >> I got 2 messages: >> >> ok 9 - rename (silent unicode normalization) # TODO known breakage vanished >> ok 10 - merge (silent unicode normalization) # TODO known breakage vanished >> >> Do you get the same ? Halfway, I see this: ok 9 - rename (silent unicode normalization) # TODO known breakage vanished not ok 10 - merge (silent unicode normalization) # TODO known breakage > Now I am even more puzzled. > running t0050 with -x gives this: > > Author: A U Thor > 1 file changed, 0 insertions(+), 0 deletions(-) > rename "a\314\210" => "\303\244" (100%) > ok 9 - rename (silent unicode normalization) # TODO known breakage vanished > > > ---------------- > When I create a test Git, with one file in ä-decomposed, > and rename into ä-precomposed, Git gives me: > > tb@Ubuntu2021:~/ttt$ git mv "$aumlcdiar" "$auml" > fatal: destination exists, source=ä, destination=ä > > and in hex form: > > tb@Ubuntu2021:~/ttt$ git mv "$aumlcdiar" "$auml" 2>&1 | xxd > 00000000: 6661 7461 6c3a 2064 6573 7469 6e61 7469 fatal: destinati > 00000010: 6f6e 2065 7869 7374 732c 2073 6f75 7263 on exists, sourc > 00000020: 653d 61cc 882c 2064 6573 7469 6e61 7469 e=a.., destinati > 00000030: 6f6e 3dc3 a40a on=... Interesting: does this "fatal" error not change the exit code? Oddly, I don't get that failure under -x: checking known breakage of 0050.9 'rename (silent unicode normalization)': git mv "$aumlcdiar" "$auml" && git commit -m rename + git mv ä ä + git commit -m rename [main 591d19c] rename Author: A U Thor 1 file changed, 0 insertions(+), 0 deletions(-) rename "a\314\210" => "\303\244" (100%) ok 9 - rename (silent unicode normalization) # TODO known breakage vanished checking known breakage of 0050.10 'merge (silent unicode normalization)': git reset --hard initial && git merge topic + git reset --hard initial error: unable to unlink old 'ä': No such file or directory fatal: Could not reset index file to revision 'initial'. error: last command exited with $?=128 not ok 10 - merge (silent unicode normalization) # TODO known breakage But notice that -x does make test 10 go back to failing. Thanks, -Stolee ``` ## Derrick Stolee, 2021-11-17 18:35 Subject: Re: [Question] Unicode weirdness breaking tests on ZFS? Message-ID: <5d1b5ce6-d9ff-1b2a-2ff2-19813c957a24@gmail.com> URL: https://gitlist.dev/e/5d1b5ce6-d9ff-1b2a-2ff2-19813c957a24%40gmail.com In-Reply-To: <8a3d0d0e-cc82-b696-00f4-b71e6452e1bd@gmail.com> ``` On 11/17/2021 1:29 PM, Derrick Stolee wrote: > On 11/17/2021 12:39 PM, Torsten =?unknown-8bit?Q?B=C3=B6gershausen?= wrote: >> On Wed, Nov 17, 2021 at 06:06:13PM +0100, Torsten B??gershausen wrote: >>> On Wed, Nov 17, 2021 at 05:12:26PM +0100, Torsten B??gershausen wrote: >>>> I'll can have a look - just installing in a virtual machine. >>> >>> So, the virtual machine is up-and-running. >>> >>> I got 2 messages: >>> >>> ok 9 - rename (silent unicode normalization) # TODO known breakage vanished >>> ok 10 - merge (silent unicode normalization) # TODO known breakage vanished >>> >>> Do you get the same ? > > Halfway, I see this: > > ok 9 - rename (silent unicode normalization) # TODO known breakage vanished > not ok 10 - merge (silent unicode normalization) # TODO known breakage Making this even more confusing, my original output shows both of the TODOs vanishing, but I can't make that happen only running this test. However, with "prove -j8 t00*.sh" I can get them to both vanish: Test Summary Report ------------------- t0050-filesystem.sh (Wstat: 0 Tests: 11 Failed: 0) TODO passed: 9-10 t0021-conversion.sh (Wstat: 256 Tests: 41 Failed: 1) Failed test: 31 Non-zero exit status: 1 Files=53, Tests=2896, 15 wallclock secs ( 0.59 usr 0.07 sys + 26.96 cusr 13.95 csys = 41.57 CPU) Result: FAIL ``` ## Torsten Bögershausen, 2021-11-19 15:44 Subject: Re: [Question] Unicode weirdness breaking tests on ZFS? Message-ID: <20211119154405.xhoqfh3vmljxbh5c@tb-raspi4> URL: https://gitlist.dev/e/20211119154405.xhoqfh3vmljxbh5c%40tb-raspi4 In-Reply-To: <5d1b5ce6-d9ff-1b2a-2ff2-19813c957a24@gmail.com> ``` On Wed, Nov 17, 2021 at 01:35:32PM -0500, Derrick Stolee wrote: > On 11/17/2021 1:29 PM, Derrick Stolee wrote: > > On 11/17/2021 12:39 PM, Torsten =?unknown-8bit?Q?B=C3=B6gershausen?= wrote: > >> On Wed, Nov 17, 2021 at 06:06:13PM +0100, Torsten B??gershausen wrote: > >>> On Wed, Nov 17, 2021 at 05:12:26PM +0100, Torsten B??gershausen wrote: > >>>> I'll can have a look - just installing in a virtual machine. > >>> > >>> So, the virtual machine is up-and-running. > >>> > >>> I got 2 messages: > >>> > >>> ok 9 - rename (silent unicode normalization) # TODO known breakage vanished > >>> ok 10 - merge (silent unicode normalization) # TODO known breakage vanished > >>> > >>> Do you get the same ? > > > > Halfway, I see this: > > > > ok 9 - rename (silent unicode normalization) # TODO known breakage vanished > > not ok 10 - merge (silent unicode normalization) # TODO known breakage > > Making this even more confusing, my original output shows both of > the TODOs vanishing, but I can't make that happen only running this > test. However, with "prove -j8 t00*.sh" I can get them to both > vanish: > > Test Summary Report > ------------------- > t0050-filesystem.sh (Wstat: 0 Tests: 11 Failed: 0) > TODO passed: 9-10 > t0021-conversion.sh (Wstat: 256 Tests: 41 Failed: 1) > Failed test: 31 > Non-zero exit status: 1 > Files=53, Tests=2896, 15 wallclock secs ( 0.59 usr 0.07 sys + 26.96 cusr 13.95 csys = 41.57 CPU) > Result: FAIL > Should we conclude that the underlying os/zfs is not stable ? Things don't seem to be reproducable What Git needs here in t0050 is that stat("ä") behaves the same as stat("a¨"), when either "ä" or "a¨" exist on disk. The same for open() and all other file system functions. ("ä" is the precomposed form "a¨" is the decomposed form, typically both render to the same glyph on the screen, and a hex dump or xxd will show what we had. I just use this notation here for illustration) Should we contact the zfs developers ? ``` ## Junio C Hamano, 2021-11-19 17:03 Subject: Re: [Question] Unicode weirdness breaking tests on ZFS? Message-ID: URL: https://gitlist.dev/e/xmqqmtm0qdqp.fsf%40gitster.g In-Reply-To: <20211119154405.xhoqfh3vmljxbh5c@tb-raspi4> ``` Torsten Bögershausen writes: > Should we conclude that the underlying os/zfs is not stable ? > Things don't seem to be reproducable > > What Git needs here in t0050 is that stat("ä") behaves the same as stat("a¨"), > when either "ä" or "a¨" exist on disk. > The same for open() and all other file system functions. We either need to see these two are treated as the same thing, or these two are treated as two distict filesystem entities, just like stat("a") and stat("b") are. What we absolutely need is the unification either always happens or never happens consistently. I wonder what readdir() is returning. After creat("ä") in an empty directory, does readdir() in there return "ä" or "a¨? And vice versa? Is this also inconsistent? > ("ä" is the precomposed form "a¨" is the decomposed form, > typically both render to the same glyph on the screen, > and a hex dump or xxd will show what we had. > I just use this notation here for illustration) > > Should we contact the zfs developers ? ``` ## Derrick Stolee, 2021-11-19 18:30 Subject: Re: [Question] Unicode weirdness breaking tests on ZFS? Message-ID: URL: https://gitlist.dev/e/e56b0227-14fc-26cf-7b98-fbf01f3c5cd7%40gmail.com In-Reply-To: ``` On 11/19/2021 12:03 PM, Junio C Hamano wrote: > Torsten Bögershausen writes: > >> Should we conclude that the underlying os/zfs is not stable ? >> Things don't seem to be reproducable >> >> What Git needs here in t0050 is that stat("ä") behaves the same as stat("a¨"), >> when either "ä" or "a¨" exist on disk. >> The same for open() and all other file system functions. > > We either need to see these two are treated as the same thing, or > these two are treated as two distict filesystem entities, just like > stat("a") and stat("b") are. What we absolutely need is the > unification either always happens or never happens consistently. > > I wonder what readdir() is returning. After creat("ä") in an empty > directory, does readdir() in there return "ä" or "a¨? And vice > versa? Is this also inconsistent? Following this suggestion, I added a test helper with this code: int cmd__create_and_read(int argc, const char **argv) { DIR *dir; struct dirent *de; if (strcmp(argv[0], "--nfc")) creat("\303\244", 0766); else if (strcmp(argv[0], "--nfd")) creat("\141\314\210", 0766); else die("select --nfc or --nfd"); dir = opendir("."); readdir(dir); while ((de = readdir(dir)) != NULL) printf("%s\n", de->d_name); return 0; } And then added this test: test_expect_success 'unicode stuff' ' mkdir nfc && ( cd nfc && test-tool create-and-read --nfc >../nfc.txt ) && mkdir nfd && ( cd nfd && test-tool create-and-read --nfd >../nfd.txt ) && test_cmp nfc.txt nfd.txt ' This test always passes for me, and is essentially doing a similar check that the prereq is doing, except that it actually writes both names to files instead of writing one and doing a read with the other. After changing the "$test_unicode" instances to instances of "test_expect_success", I ran t0050 under --stress and quickly got a failure on the 'merge (silent unicode normalization)' test: expecting success of 0050.11 'merge (silent unicode normalization)': git reset --hard initial && git merge topic + git reset --hard initial error: unable to unlink old 'ä': No such file or directory fatal: Could not reset index file to revision 'initial'. error: last command exited with $?=128 not ok 11 - merge (silent unicode normalization) Deleting that test gave mostly-consistent results, although I once got a failure on the "setup unicode normalization tests" tests with a similar error message: + git checkout -f main error: unable to unlink old 'ä': No such file or directory Switched to branch 'main' error: last command exited with $?=1 not ok 8 - setup unicode normalization tests >> ("ä" is the precomposed form "a¨" is the decomposed form, >> typically both render to the same glyph on the screen, >> and a hex dump or xxd will show what we had. >> I just use this notation here for illustration) >> >> Should we contact the zfs developers ? Hopefully someone has a good way to contact them, and I can start a thread at the appropriate place. To optimize for their time, what is our minimal reproduction steps? 1. Build Git at the v2.34.0 tag. 2. cd to t/ 3. ./t0050-filesystem.sh --stress Those instructions (given enough time) should get the repro on test 8, 'setup unicode normalization tests'. To get the faster stress, the same steps work except use the 'zfs-minimal' branch at my fork [1] because it changes the tests to expect success, and demonstrates the unpredictable tests more quickly. [1] https://github.com/derrickstolee/git/tree/zfs-minimal Thanks, -Stolee ```