{"thread":{"id":"56927","subject":"[Question] Unicode weirdness breaking tests on ZFS?","startedAt":"2021-11-17T15:18:04Z","lastAt":"2021-11-19T18:30:37Z","messageCount":10,"participants":["Derrick Stolee","Ævar Arnfjörð Bjarmason","Torsten Bögershausen","Torsten B��gershausen","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"441499","messageId":"9393e572-0666-6485-df29-abad5e0d32a1@gmail.com","threadId":"56927","inReplyTo":null,"subject":"[Question] Unicode weirdness breaking tests on ZFS?","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-11-17T15:17:53Z","receivedAt":"2021-11-17T15:18:04Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"I recently had to pave my Linux machine, so I updated it to Ubuntu\n21.10 and had the choice to start using the ZFS filesystem. I thought,\n\"Why not?\" but now I maybe see why.\n\nRunning the Git test suite at the v2.34.0 tag on my machine results in\nthese failures:\n\nt0050-filesystem.sh                   (Wstat: 0 Tests: 11 Failed: 0)\n  TODO passed:   9-10\nt0021-conversion.sh                   (Wstat: 256 Tests: 41 Failed: 1)\n  Failed test:  31\n  Non-zero exit status: 1\nt3910-mac-os-precompose.sh            (Wstat: 256 Tests: 25 Failed: 10)\n  Failed tests:  1, 4, 6, 8, 11-16\n  TODO passed:   23\n  Non-zero exit status: 1\n\nThese are all related to the UTF8_NFD_TO_NFC prereq.\n\nZooming in on t0050, these tests are marked as \"test_expect_failure\" due\nto an assignment of $test_unicode using the UTF8_NFD_TO_NFC prereq:\n\n\n$test_unicode 'rename (silent unicode normalization)' '\n\tgit mv \"$aumlcdiar\" \"$auml\" &&\n\tgit commit -m rename\n'\n\n$test_unicode 'merge (silent unicode normalization)' '\n\tgit reset --hard initial &&\n\tgit merge topic\n'\n\n\nThe prereq creates two files using unicode characters that could\ncollapse to equivalent meanings:\n\n\ntest_lazy_prereq UTF8_NFD_TO_NFC '\n\t# check whether FS converts nfd unicode to nfc\n\tauml=$(printf \"\\303\\244\")\n\taumlcdiar=$(printf \"\\141\\314\\210\")\n\t>\"$auml\" &&\n\ttest -f \"$aumlcdiar\"\n'\n\n\nWhat I see in that first test, the 'git mv' does change the\nindex, but the filesystem thinks the files are the same. This\nmay mean that our 'git add \"$aumlcdiar\"' from an earlier test\nis providing a non-equivalence in the index, and the 'git mv'\nchanges the index without causing any issues in the filesystem.\n\nIt reminds me as if we used 'git mv README readme' on a case-\ninsensitive filesystem. Is this not a similar situation?\n\nWhat I'm trying to gather is that maybe this test is flawed?\nOr maybe something broke (or never worked?) in how we use\n'git add' to not get the canonical unicode from the filesystem?\n\nThe other tests all have similar interactions with 'git add'.\nI'm hoping that these are just test bugs, and not actually a\nfunctionality issue in Git. Yes, it is confusing that we can\nchange the unicode of a file in the index without the filesystem\nunderstanding the difference, but that is very similar to how\ncase-insensitive filesystems work and I don't know what else we\nwould do here.\n\nThese filesystem/unicode things are out of my expertise, so\nhopefully someone else has a clearer idea of what is going on.\nI'm happy to be a test bed, or even attempt producing patches\nto fix the issue once we have that clarity.\n\nThanks,\n-Stolee\n"},{"id":"441503","messageId":"211117.8635nu7pm3.gmgdl@evledraar.gmail.com","threadId":"56927","inReplyTo":"9393e572-0666-6485-df29-abad5e0d32a1@gmail.com","subject":"Re: [Question] Unicode weirdness breaking tests on ZFS?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-17T15:41:47Z","receivedAt":"2021-11-17T15:45:58Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Nov 17 2021, Derrick Stolee wrote:\n\n> I recently had to pave my Linux machine, so I updated it to Ubuntu\n> 21.10 and had the choice to start using the ZFS filesystem. I thought,\n> \"Why not?\" but now I maybe see why.\n>\n> Running the Git test suite at the v2.34.0 tag on my machine results in\n> these failures:\n>\n> t0050-filesystem.sh                   (Wstat: 0 Tests: 11 Failed: 0)\n>   TODO passed:   9-10\n> t0021-conversion.sh                   (Wstat: 256 Tests: 41 Failed: 1)\n>   Failed test:  31\n>   Non-zero exit status: 1\n> t3910-mac-os-precompose.sh            (Wstat: 256 Tests: 25 Failed: 10)\n>   Failed tests:  1, 4, 6, 8, 11-16\n>   TODO passed:   23\n>   Non-zero exit status: 1\n>\n> These are all related to the UTF8_NFD_TO_NFC prereq.\n>\n> Zooming in on t0050, these tests are marked as \"test_expect_failure\" due\n> to an assignment of $test_unicode using the UTF8_NFD_TO_NFC prereq:\n>\n>\n> $test_unicode 'rename (silent unicode normalization)' '\n> \tgit mv \"$aumlcdiar\" \"$auml\" &&\n> \tgit commit -m rename\n> '\n>\n> $test_unicode 'merge (silent unicode normalization)' '\n> \tgit reset --hard initial &&\n> \tgit merge topic\n> '\n>\n>\n> The prereq creates two files using unicode characters that could\n> collapse to equivalent meanings:\n>\n>\n> test_lazy_prereq UTF8_NFD_TO_NFC '\n> \t# check whether FS converts nfd unicode to nfc\n> \tauml=$(printf \"\\303\\244\")\n> \taumlcdiar=$(printf \"\\141\\314\\210\")\n> \t>\"$auml\" &&\n> \ttest -f \"$aumlcdiar\"\n> '\n>\n>\n> What I see in that first test, the 'git mv' does change the\n> index, but the filesystem thinks the files are the same. This\n> may mean that our 'git add \"$aumlcdiar\"' from an earlier test\n> is providing a non-equivalence in the index, and the 'git mv'\n> changes the index without causing any issues in the filesystem.\n>\n> It reminds me as if we used 'git mv README readme' on a case-\n> insensitive filesystem. Is this not a similar situation?\n>\n> What I'm trying to gather is that maybe this test is flawed?\n> Or maybe something broke (or never worked?) in how we use\n> 'git add' to not get the canonical unicode from the filesystem?\n>\n> The other tests all have similar interactions with 'git add'.\n> I'm hoping that these are just test bugs, and not actually a\n> functionality issue in Git. Yes, it is confusing that we can\n> change the unicode of a file in the index without the filesystem\n> understanding the difference, but that is very similar to how\n> case-insensitive filesystems work and I don't know what else we\n> would do here.\n>\n> These filesystem/unicode things are out of my expertise, so\n> hopefully someone else has a clearer idea of what is going on.\n> I'm happy to be a test bed, or even attempt producing patches\n> to fix the issue once we have that clarity.\n\nI haven't used ZFS, but this points to non-POSIX behavior on the FS\nitself. It looks like tweaking the \"normalization\" property might change\nit, see: https://manpages.ubuntu.com/manpages/eoan/man8/zfs.8.html\n\nThere's also \"casesensitivity\" and \"utf8only\".\n\nWe probably don't want to invoke some ZFS command on every test to\ninterrogate this, but if we can pass it down from GIT-BUILD-OPTIONS or\nsimilar then we could have a test prereq check this.\n\nOr perhaps it's as simple as changing the \"UTF8_NFD_TO_NFC\" prereq from\ndoing a \"test -f\" to e.g. \"echo *\" and seeing what it gets back. Perhaps\nZFS says \"yes\" to \"it exists?\" but when doing a readdir() it will\ncanonicalize?\n\n"},{"id":"441506","messageId":"20211117161226.xcat77ewhf5inaif@tb-raspi4","threadId":"56927","inReplyTo":"9393e572-0666-6485-df29-abad5e0d32a1@gmail.com","subject":"Re: [Question] Unicode weirdness breaking tests on ZFS?","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2021-11-17T16:12:26Z","receivedAt":"2021-11-17T16:12:37Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Wed, Nov 17, 2021 at 10:17:53AM -0500, Derrick Stolee wrote:\n> I recently had to pave my Linux machine, so I updated it to Ubuntu\n> 21.10 and had the choice to start using the ZFS filesystem. I thought,\n> \"Why not?\" but now I maybe see why.\n>\n> Running the Git test suite at the v2.34.0 tag on my machine results in\n> these failures:\n>\n> t0050-filesystem.sh                   (Wstat: 0 Tests: 11 Failed: 0)\n>   TODO passed:   9-10\n> t0021-conversion.sh                   (Wstat: 256 Tests: 41 Failed: 1)\n>   Failed test:  31\n>   Non-zero exit status: 1\n> t3910-mac-os-precompose.sh            (Wstat: 256 Tests: 25 Failed: 10)\n>   Failed tests:  1, 4, 6, 8, 11-16\n>   TODO passed:   23\n>   Non-zero exit status: 1\n>\n> These are all related to the UTF8_NFD_TO_NFC prereq.\n>\n> Zooming in on t0050, these tests are marked as \"test_expect_failure\" due\n> to an assignment of $test_unicode using the UTF8_NFD_TO_NFC prereq:\n>\n>\n> $test_unicode 'rename (silent unicode normalization)' '\n> \tgit mv \"$aumlcdiar\" \"$auml\" &&\n> \tgit commit -m rename\n> '\n>\n> $test_unicode 'merge (silent unicode normalization)' '\n> \tgit reset --hard initial &&\n> \tgit merge topic\n> '\n>\n>\n> The prereq creates two files using unicode characters that could\n> collapse to equivalent meanings:\n>\n>\n> test_lazy_prereq UTF8_NFD_TO_NFC '\n> \t# check whether FS converts nfd unicode to nfc\n> \tauml=$(printf \"\\303\\244\")\n> \taumlcdiar=$(printf \"\\141\\314\\210\")\n> \t>\"$auml\" &&\n> \ttest -f \"$aumlcdiar\"\n> '\n>\n>\n> What I see in that first test, the 'git mv' does change the\n> index, but the filesystem thinks the files are the same. This\n> may mean that our 'git add \"$aumlcdiar\"' from an earlier test\n> is providing a non-equivalence in the index, and the 'git mv'\n> changes the index without causing any issues in the filesystem.\n>\n> It reminds me as if we used 'git mv README readme' on a case-\n> insensitive filesystem. Is this not a similar situation?\n>\n> What I'm trying to gather is that maybe this test is flawed?\n> Or maybe something broke (or never worked?) in how we use\n> 'git add' to not get the canonical unicode from the filesystem?\n>\n> The other tests all have similar interactions with 'git add'.\n> I'm hoping that these are just test bugs, and not actually a\n> functionality issue in Git. Yes, it is confusing that we can\n> change the unicode of a file in the index without the filesystem\n> understanding the difference, but that is very similar to how\n> case-insensitive filesystems work and I don't know what else we\n> would do here.\n>\n> These filesystem/unicode things are out of my expertise, so\n> hopefully someone else has a clearer idea of what is going on.\n> I'm happy to be a test bed, or even attempt producing patches\n> to fix the issue once we have that clarity.\n>\n> Thanks,\n> -Stolee\n\nInteresting.\nThe tests have always been working on HFS+, then we got\nAPFS (and needed a small fix) and now ZFS.\n\nI'll can have a look - just installing in a virtual machine.\n"},{"id":"441511","messageId":"20211117170613.kyoe6ov2m5wi2i56@tb-raspi4","threadId":"56927","inReplyTo":"20211117161226.xcat77ewhf5inaif@tb-raspi4","subject":"Re: [Question] Unicode weirdness breaking tests on ZFS?","fromName":"Torsten B��gershausen","fromEmail":"tboegi@web.de","sentAt":"2021-11-17T17:06:13Z","receivedAt":"2021-11-17T17:06:18Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Wed, Nov 17, 2021 at 05:12:26PM +0100, Torsten B??gershausen wrote:\n> On Wed, Nov 17, 2021 at 10:17:53AM -0500, Derrick Stolee wrote:\n> > I recently had to pave my Linux machine, so I updated it to Ubuntu\n> > 21.10 and had the choice to start using the ZFS filesystem. I thought,\n> > \"Why not?\" but now I maybe see why.\n> >\n> > Running the Git test suite at the v2.34.0 tag on my machine results in\n> > these failures:\n> >\n> > t0050-filesystem.sh                   (Wstat: 0 Tests: 11 Failed: 0)\n> >   TODO passed:   9-10\n> > t0021-conversion.sh                   (Wstat: 256 Tests: 41 Failed: 1)\n> >   Failed test:  31\n> >   Non-zero exit status: 1\n> > t3910-mac-os-precompose.sh            (Wstat: 256 Tests: 25 Failed: 10)\n> >   Failed tests:  1, 4, 6, 8, 11-16\n> >   TODO passed:   23\n> >   Non-zero exit status: 1\n> >\n> > These are all related to the UTF8_NFD_TO_NFC prereq.\n> >\n> > Zooming in on t0050, these tests are marked as \"test_expect_failure\" due\n> > to an assignment of $test_unicode using the UTF8_NFD_TO_NFC prereq:\n> >\n> >\n> > $test_unicode 'rename (silent unicode normalization)' '\n> > \tgit mv \"$aumlcdiar\" \"$auml\" &&\n> > \tgit commit -m rename\n> > '\n> >\n> > $test_unicode 'merge (silent unicode normalization)' '\n> > \tgit reset --hard initial &&\n> > \tgit merge topic\n> > '\n> >\n> >\n> > The prereq creates two files using unicode characters that could\n> > collapse to equivalent meanings:\n> >\n> >\n> > test_lazy_prereq UTF8_NFD_TO_NFC '\n> > \t# check whether FS converts nfd unicode to nfc\n> > \tauml=$(printf \"\\303\\244\")\n> > \taumlcdiar=$(printf \"\\141\\314\\210\")\n> > \t>\"$auml\" &&\n> > \ttest -f \"$aumlcdiar\"\n> > '\n> >\n> >\n> > What I see in that first test, the 'git mv' does change the\n> > index, but the filesystem thinks the files are the same. This\n> > may mean that our 'git add \"$aumlcdiar\"' from an earlier test\n> > is providing a non-equivalence in the index, and the 'git mv'\n> > changes the index without causing any issues in the filesystem.\n> >\n> > It reminds me as if we used 'git mv README readme' on a case-\n> > insensitive filesystem. Is this not a similar situation?\n> >\n> > What I'm trying to gather is that maybe this test is flawed?\n> > Or maybe something broke (or never worked?) in how we use\n> > 'git add' to not get the canonical unicode from the filesystem?\n> >\n> > The other tests all have similar interactions with 'git add'.\n> > I'm hoping that these are just test bugs, and not actually a\n> > functionality issue in Git. Yes, it is confusing that we can\n> > change the unicode of a file in the index without the filesystem\n> > understanding the difference, but that is very similar to how\n> > case-insensitive filesystems work and I don't know what else we\n> > would do here.\n> >\n> > These filesystem/unicode things are out of my expertise, so\n> > hopefully someone else has a clearer idea of what is going on.\n> > I'm happy to be a test bed, or even attempt producing patches\n> > to fix the issue once we have that clarity.\n> >\n> > Thanks,\n> > -Stolee\n>\n> Interesting.\n> The tests have always been working on HFS+, then we got\n> APFS (and needed a small fix) and now ZFS.\n>\n> I'll can have a look - just installing in a virtual machine.\n\nSo, the virtual machine is up-and-running.\n\nI got 2 messages:\n\nok 9 - rename (silent unicode normalization) # TODO known breakage vanished\nok 10 - merge (silent unicode normalization) # TODO known breakage vanished\n\nDo you get the same ?\n"},{"id":"441512","messageId":"20211117173924.maporsti5cz2ixsu@tb-raspi4","threadId":"56927","inReplyTo":"20211117170613.kyoe6ov2m5wi2i56@tb-raspi4","subject":"Re: [Question] Unicode weirdness breaking tests on ZFS?","fromName":"Torsten B��gershausen","fromEmail":"tboegi@web.de","sentAt":"2021-11-17T17:39:25Z","receivedAt":"2021-11-17T17:39:28Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Wed, Nov 17, 2021 at 06:06:13PM +0100, Torsten B??gershausen wrote:\n> On Wed, Nov 17, 2021 at 05:12:26PM +0100, Torsten B??gershausen wrote:\n> > On Wed, Nov 17, 2021 at 10:17:53AM -0500, Derrick Stolee wrote:\n> > > I recently had to pave my Linux machine, so I updated it to Ubuntu\n> > > 21.10 and had the choice to start using the ZFS filesystem. I thought,\n> > > \"Why not?\" but now I maybe see why.\n> > >\n> > > Running the Git test suite at the v2.34.0 tag on my machine results in\n> > > these failures:\n> > >\n> > > t0050-filesystem.sh                   (Wstat: 0 Tests: 11 Failed: 0)\n> > >   TODO passed:   9-10\n> > > t0021-conversion.sh                   (Wstat: 256 Tests: 41 Failed: 1)\n> > >   Failed test:  31\n> > >   Non-zero exit status: 1\n> > > t3910-mac-os-precompose.sh            (Wstat: 256 Tests: 25 Failed: 10)\n> > >   Failed tests:  1, 4, 6, 8, 11-16\n> > >   TODO passed:   23\n> > >   Non-zero exit status: 1\n> > >\n> > > These are all related to the UTF8_NFD_TO_NFC prereq.\n> > >\n> > > Zooming in on t0050, these tests are marked as \"test_expect_failure\" due\n> > > to an assignment of $test_unicode using the UTF8_NFD_TO_NFC prereq:\n> > >\n> > >\n> > > $test_unicode 'rename (silent unicode normalization)' '\n> > > \tgit mv \"$aumlcdiar\" \"$auml\" &&\n> > > \tgit commit -m rename\n> > > '\n> > >\n> > > $test_unicode 'merge (silent unicode normalization)' '\n> > > \tgit reset --hard initial &&\n> > > \tgit merge topic\n> > > '\n> > >\n> > >\n> > > The prereq creates two files using unicode characters that could\n> > > collapse to equivalent meanings:\n> > >\n> > >\n> > > test_lazy_prereq UTF8_NFD_TO_NFC '\n> > > \t# check whether FS converts nfd unicode to nfc\n> > > \tauml=$(printf \"\\303\\244\")\n> > > \taumlcdiar=$(printf \"\\141\\314\\210\")\n> > > \t>\"$auml\" &&\n> > > \ttest -f \"$aumlcdiar\"\n> > > '\n> > >\n> > >\n> > > What I see in that first test, the 'git mv' does change the\n> > > index, but the filesystem thinks the files are the same. This\n> > > may mean that our 'git add \"$aumlcdiar\"' from an earlier test\n> > > is providing a non-equivalence in the index, and the 'git mv'\n> > > changes the index without causing any issues in the filesystem.\n> > >\n> > > It reminds me as if we used 'git mv README readme' on a case-\n> > > insensitive filesystem. Is this not a similar situation?\n> > >\n> > > What I'm trying to gather is that maybe this test is flawed?\n> > > Or maybe something broke (or never worked?) in how we use\n> > > 'git add' to not get the canonical unicode from the filesystem?\n> > >\n> > > The other tests all have similar interactions with 'git add'.\n> > > I'm hoping that these are just test bugs, and not actually a\n> > > functionality issue in Git. Yes, it is confusing that we can\n> > > change the unicode of a file in the index without the filesystem\n> > > understanding the difference, but that is very similar to how\n> > > case-insensitive filesystems work and I don't know what else we\n> > > would do here.\n> > >\n> > > These filesystem/unicode things are out of my expertise, so\n> > > hopefully someone else has a clearer idea of what is going on.\n> > > I'm happy to be a test bed, or even attempt producing patches\n> > > to fix the issue once we have that clarity.\n> > >\n> > > Thanks,\n> > > -Stolee\n> >\n> > Interesting.\n> > The tests have always been working on HFS+, then we got\n> > APFS (and needed a small fix) and now ZFS.\n> >\n> > I'll can have a look - just installing in a virtual machine.\n>\n> So, the virtual machine is up-and-running.\n>\n> I got 2 messages:\n>\n> ok 9 - rename (silent unicode normalization) # TODO known breakage vanished\n> ok 10 - merge (silent unicode normalization) # TODO known breakage vanished\n>\n> Do you get the same ?\n\n\nNow I am even more puzzled.\nrunning t0050 with -x gives this:\n\n Author: A U Thor <author@example.com>\n  1 file changed, 0 insertions(+), 0 deletions(-)\n   rename \"a\\314\\210\" => \"\\303\\244\" (100%)\n   ok 9 - rename (silent unicode normalization) # TODO known breakage vanished\n\n\n----------------\nWhen I create a test Git, with one file in ä-decomposed,\nand rename into ä-precomposed, Git gives me:\n\ntb@Ubuntu2021:~/ttt$ git mv \"$aumlcdiar\" \"$auml\"\nfatal: destination exists, source=ä, destination=ä\n\nand in hex form:\n\ntb@Ubuntu2021:~/ttt$ git mv \"$aumlcdiar\" \"$auml\" 2>&1 | xxd\n00000000: 6661 7461 6c3a 2064 6573 7469 6e61 7469  fatal: destinati\n00000010: 6f6e 2065 7869 7374 732c 2073 6f75 7263  on exists, sourc\n00000020: 653d 61cc 882c 2064 6573 7469 6e61 7469  e=a.., destinati\n00000030: 6f6e 3dc3 a40a                           on=...\n\n"},{"id":"441516","messageId":"8a3d0d0e-cc82-b696-00f4-b71e6452e1bd@gmail.com","threadId":"56927","inReplyTo":"20211117173924.maporsti5cz2ixsu@tb-raspi4","subject":"Re: [Question] Unicode weirdness breaking tests on ZFS?","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-11-17T18:29:11Z","receivedAt":"2021-11-17T18:29:15Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/17/2021 12:39 PM, Torsten =?unknown-8bit?Q?B=C3=B6gershausen?= wrote:\n> On Wed, Nov 17, 2021 at 06:06:13PM +0100, Torsten B??gershausen wrote:\n>> On Wed, Nov 17, 2021 at 05:12:26PM +0100, Torsten B??gershausen wrote:\n>>> On Wed, Nov 17, 2021 at 10:17:53AM -0500, Derrick Stolee wrote:\n>>>> I recently had to pave my Linux machine, so I updated it to Ubuntu\n>>>> 21.10 and had the choice to start using the ZFS filesystem. I thought,\n>>>> \"Why not?\" but now I maybe see why.\n>>>>\n>>>> Running the Git test suite at the v2.34.0 tag on my machine results in\n>>>> these failures:\n>>>>\n>>>> t0050-filesystem.sh                   (Wstat: 0 Tests: 11 Failed: 0)\n>>>>   TODO passed:   9-10\n>>>> t0021-conversion.sh                   (Wstat: 256 Tests: 41 Failed: 1)\n>>>>   Failed test:  31\n>>>>   Non-zero exit status: 1\n>>>> t3910-mac-os-precompose.sh            (Wstat: 256 Tests: 25 Failed: 10)\n>>>>   Failed tests:  1, 4, 6, 8, 11-16\n>>>>   TODO passed:   23\n>>>>   Non-zero exit status: 1\n>>>>\n>>>> These are all related to the UTF8_NFD_TO_NFC prereq.\n>>>>\n>>>> Zooming in on t0050, these tests are marked as \"test_expect_failure\" due\n>>>> to an assignment of $test_unicode using the UTF8_NFD_TO_NFC prereq:\n>>>>\n>>>>\n>>>> $test_unicode 'rename (silent unicode normalization)' '\n>>>> \tgit mv \"$aumlcdiar\" \"$auml\" &&\n>>>> \tgit commit -m rename\n>>>> '\n>>>>\n>>>> $test_unicode 'merge (silent unicode normalization)' '\n>>>> \tgit reset --hard initial &&\n>>>> \tgit merge topic\n>>>> '\n>>>>\n>>>>\n>>>> The prereq creates two files using unicode characters that could\n>>>> collapse to equivalent meanings:\n>>>>\n>>>>\n>>>> test_lazy_prereq UTF8_NFD_TO_NFC '\n>>>> \t# check whether FS converts nfd unicode to nfc\n>>>> \tauml=$(printf \"\\303\\244\")\n>>>> \taumlcdiar=$(printf \"\\141\\314\\210\")\n>>>> \t>\"$auml\" &&\n>>>> \ttest -f \"$aumlcdiar\"\n>>>> '\n>>>>\n>>>>\n>>>> What I see in that first test, the 'git mv' does change the\n>>>> index, but the filesystem thinks the files are the same. This\n>>>> may mean that our 'git add \"$aumlcdiar\"' from an earlier test\n>>>> is providing a non-equivalence in the index, and the 'git mv'\n>>>> changes the index without causing any issues in the filesystem.\n>>>>\n>>>> It reminds me as if we used 'git mv README readme' on a case-\n>>>> insensitive filesystem. Is this not a similar situation?\n>>>>\n>>>> What I'm trying to gather is that maybe this test is flawed?\n>>>> Or maybe something broke (or never worked?) in how we use\n>>>> 'git add' to not get the canonical unicode from the filesystem?\n>>>>\n>>>> The other tests all have similar interactions with 'git add'.\n>>>> I'm hoping that these are just test bugs, and not actually a\n>>>> functionality issue in Git. Yes, it is confusing that we can\n>>>> change the unicode of a file in the index without the filesystem\n>>>> understanding the difference, but that is very similar to how\n>>>> case-insensitive filesystems work and I don't know what else we\n>>>> would do here.\n>>>>\n>>>> These filesystem/unicode things are out of my expertise, so\n>>>> hopefully someone else has a clearer idea of what is going on.\n>>>> I'm happy to be a test bed, or even attempt producing patches\n>>>> to fix the issue once we have that clarity.\n>>>>\n>>>> Thanks,\n>>>> -Stolee\n>>>\n>>> Interesting.\n>>> The tests have always been working on HFS+, then we got\n>>> APFS (and needed a small fix) and now ZFS.\n>>>\n>>> I'll can have a look - just installing in a virtual machine.\n>>\n>> So, the virtual machine is up-and-running.\n>>\n>> I got 2 messages:\n>>\n>> ok 9 - rename (silent unicode normalization) # TODO known breakage vanished\n>> ok 10 - merge (silent unicode normalization) # TODO known breakage vanished\n>>\n>> Do you get the same ?\n\nHalfway, I see this:\n\nok 9 - rename (silent unicode normalization) # TODO known breakage vanished\nnot ok 10 - merge (silent unicode normalization) # TODO known breakage\n\n> Now I am even more puzzled.\n> running t0050 with -x gives this:\n> \n>  Author: A U Thor <author@example.com>\n>   1 file changed, 0 insertions(+), 0 deletions(-)\n>    rename \"a\\314\\210\" => \"\\303\\244\" (100%)\n>    ok 9 - rename (silent unicode normalization) # TODO known breakage vanished\n> \n> \n> ----------------\n> When I create a test Git, with one file in ä-decomposed,\n> and rename into ä-precomposed, Git gives me:\n> \n> tb@Ubuntu2021:~/ttt$ git mv \"$aumlcdiar\" \"$auml\"\n> fatal: destination exists, source=ä, destination=ä\n> \n> and in hex form:\n> \n> tb@Ubuntu2021:~/ttt$ git mv \"$aumlcdiar\" \"$auml\" 2>&1 | xxd\n> 00000000: 6661 7461 6c3a 2064 6573 7469 6e61 7469  fatal: destinati\n> 00000010: 6f6e 2065 7869 7374 732c 2073 6f75 7263  on exists, sourc\n> 00000020: 653d 61cc 882c 2064 6573 7469 6e61 7469  e=a.., destinati\n> 00000030: 6f6e 3dc3 a40a                           on=...\n \nInteresting: does this \"fatal\" error not change the exit code? Oddly,\nI don't get that failure under -x:\n\nchecking known breakage of 0050.9 'rename (silent unicode normalization)': \n        git mv \"$aumlcdiar\" \"$auml\" &&\n        git commit -m rename\n\n+ git mv ä ä\n+ git commit -m rename\n[main 591d19c] rename\n Author: A U Thor <author@example.com>\n 1 file changed, 0 insertions(+), 0 deletions(-)\n rename \"a\\314\\210\" => \"\\303\\244\" (100%)\nok 9 - rename (silent unicode normalization) # TODO known breakage vanished\n\nchecking known breakage of 0050.10 'merge (silent unicode normalization)': \n        git reset --hard initial &&\n        git merge topic\n\n+ git reset --hard initial\nerror: unable to unlink old 'ä': No such file or directory\nfatal: Could not reset index file to revision 'initial'.\nerror: last command exited with $?=128\nnot ok 10 - merge (silent unicode normalization) # TODO known breakage\n\n\nBut notice that -x does make test 10 go back to failing.\n\nThanks,\n-Stolee\n"},{"id":"441517","messageId":"5d1b5ce6-d9ff-1b2a-2ff2-19813c957a24@gmail.com","threadId":"56927","inReplyTo":"8a3d0d0e-cc82-b696-00f4-b71e6452e1bd@gmail.com","subject":"Re: [Question] Unicode weirdness breaking tests on ZFS?","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-11-17T18:35:32Z","receivedAt":"2021-11-17T18:35:36Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/17/2021 1:29 PM, Derrick Stolee wrote:\n> On 11/17/2021 12:39 PM, Torsten =?unknown-8bit?Q?B=C3=B6gershausen?= wrote:\n>> On Wed, Nov 17, 2021 at 06:06:13PM +0100, Torsten B??gershausen wrote:\n>>> On Wed, Nov 17, 2021 at 05:12:26PM +0100, Torsten B??gershausen wrote:\n>>>> I'll can have a look - just installing in a virtual machine.\n>>>\n>>> So, the virtual machine is up-and-running.\n>>>\n>>> I got 2 messages:\n>>>\n>>> ok 9 - rename (silent unicode normalization) # TODO known breakage vanished\n>>> ok 10 - merge (silent unicode normalization) # TODO known breakage vanished\n>>>\n>>> Do you get the same ?\n> \n> Halfway, I see this:\n> \n> ok 9 - rename (silent unicode normalization) # TODO known breakage vanished\n> not ok 10 - merge (silent unicode normalization) # TODO known breakage\n\nMaking this even more confusing, my original output shows both of\nthe TODOs vanishing, but I can't make that happen only running this\ntest. However, with \"prove -j8 t00*.sh\" I can get them to both\nvanish:\n\nTest Summary Report\n-------------------\nt0050-filesystem.sh           (Wstat: 0 Tests: 11 Failed: 0)\n  TODO passed:   9-10\nt0021-conversion.sh           (Wstat: 256 Tests: 41 Failed: 1)\n  Failed test:  31\n  Non-zero exit status: 1\nFiles=53, Tests=2896, 15 wallclock secs ( 0.59 usr  0.07 sys + 26.96 cusr 13.95 csys = 41.57 CPU)\nResult: FAIL\n\n"},{"id":"441749","messageId":"20211119154405.xhoqfh3vmljxbh5c@tb-raspi4","threadId":"56927","inReplyTo":"5d1b5ce6-d9ff-1b2a-2ff2-19813c957a24@gmail.com","subject":"Re: [Question] Unicode weirdness breaking tests on ZFS?","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2021-11-19T15:44:05Z","receivedAt":"2021-11-19T15:44:08Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Wed, Nov 17, 2021 at 01:35:32PM -0500, Derrick Stolee wrote:\n> On 11/17/2021 1:29 PM, Derrick Stolee wrote:\n> > On 11/17/2021 12:39 PM, Torsten =?unknown-8bit?Q?B=C3=B6gershausen?= wrote:\n> >> On Wed, Nov 17, 2021 at 06:06:13PM +0100, Torsten B??gershausen wrote:\n> >>> On Wed, Nov 17, 2021 at 05:12:26PM +0100, Torsten B??gershausen wrote:\n> >>>> I'll can have a look - just installing in a virtual machine.\n> >>>\n> >>> So, the virtual machine is up-and-running.\n> >>>\n> >>> I got 2 messages:\n> >>>\n> >>> ok 9 - rename (silent unicode normalization) # TODO known breakage vanished\n> >>> ok 10 - merge (silent unicode normalization) # TODO known breakage vanished\n> >>>\n> >>> Do you get the same ?\n> >\n> > Halfway, I see this:\n> >\n> > ok 9 - rename (silent unicode normalization) # TODO known breakage vanished\n> > not ok 10 - merge (silent unicode normalization) # TODO known breakage\n>\n> Making this even more confusing, my original output shows both of\n> the TODOs vanishing, but I can't make that happen only running this\n> test. However, with \"prove -j8 t00*.sh\" I can get them to both\n> vanish:\n>\n> Test Summary Report\n> -------------------\n> t0050-filesystem.sh           (Wstat: 0 Tests: 11 Failed: 0)\n>   TODO passed:   9-10\n> t0021-conversion.sh           (Wstat: 256 Tests: 41 Failed: 1)\n>   Failed test:  31\n>   Non-zero exit status: 1\n> Files=53, Tests=2896, 15 wallclock secs ( 0.59 usr  0.07 sys + 26.96 cusr 13.95 csys = 41.57 CPU)\n> Result: FAIL\n>\n\nShould we conclude that the underlying os/zfs is not stable ?\nThings don't seem to be reproducable\n\nWhat Git needs here in t0050 is that stat(\"ä\") behaves the same as stat(\"a¨\"),\nwhen either \"ä\" or \"a¨\" exist on disk.\nThe same for open() and all other file system functions.\n(\"ä\" is the precomposed form \"a¨\" is the decomposed form,\n typically both render to the same glyph on the screen,\n and a hex dump or xxd will show what we had.\n I just use this notation here for illustration)\n\nShould we contact the zfs developers ?\n\n"},{"id":"441761","messageId":"xmqqmtm0qdqp.fsf@gitster.g","threadId":"56927","inReplyTo":"20211119154405.xhoqfh3vmljxbh5c@tb-raspi4","subject":"Re: [Question] Unicode weirdness breaking tests on ZFS?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-11-19T17:03:42Z","receivedAt":"2021-11-19T17:03:46Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Torsten Bögershausen <tboegi@web.de> writes:\n\n> Should we conclude that the underlying os/zfs is not stable ?\n> Things don't seem to be reproducable\n>\n> What Git needs here in t0050 is that stat(\"ä\") behaves the same as stat(\"a¨\"),\n> when either \"ä\" or \"a¨\" exist on disk.\n> The same for open() and all other file system functions.\n\nWe either need to see these two are treated as the same thing, or\nthese two are treated as two distict filesystem entities, just like\nstat(\"a\") and stat(\"b\") are.  What we absolutely need is the\nunification either always happens or never happens consistently.\n\nI wonder what readdir() is returning.  After creat(\"ä\") in an empty\ndirectory, does readdir() in there return \"ä\" or \"a¨?  And vice\nversa?  Is this also inconsistent?\n\n> (\"ä\" is the precomposed form \"a¨\" is the decomposed form,\n>  typically both render to the same glyph on the screen,\n>  and a hex dump or xxd will show what we had.\n>  I just use this notation here for illustration)\n>\n> Should we contact the zfs developers ?\n"},{"id":"441772","messageId":"e56b0227-14fc-26cf-7b98-fbf01f3c5cd7@gmail.com","threadId":"56927","inReplyTo":"xmqqmtm0qdqp.fsf@gitster.g","subject":"Re: [Question] Unicode weirdness breaking tests on ZFS?","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-11-19T18:30:33Z","receivedAt":"2021-11-19T18:30:37Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/19/2021 12:03 PM, Junio C Hamano wrote:\n> Torsten Bögershausen <tboegi@web.de> writes:\n> \n>> Should we conclude that the underlying os/zfs is not stable ?\n>> Things don't seem to be reproducable\n>>\n>> What Git needs here in t0050 is that stat(\"ä\") behaves the same as stat(\"a¨\"),\n>> when either \"ä\" or \"a¨\" exist on disk.\n>> The same for open() and all other file system functions.\n> \n> We either need to see these two are treated as the same thing, or\n> these two are treated as two distict filesystem entities, just like\n> stat(\"a\") and stat(\"b\") are.  What we absolutely need is the\n> unification either always happens or never happens consistently.\n> \n> I wonder what readdir() is returning.  After creat(\"ä\") in an empty\n> directory, does readdir() in there return \"ä\" or \"a¨?  And vice\n> versa?  Is this also inconsistent?\n\nFollowing this suggestion, I added a test helper with this code:\n\nint cmd__create_and_read(int argc, const char **argv)\n{\n\tDIR *dir;\n\tstruct dirent *de;\n\n\tif (strcmp(argv[0], \"--nfc\"))\n\t\tcreat(\"\\303\\244\", 0766);\n\telse if (strcmp(argv[0], \"--nfd\"))\n\t\tcreat(\"\\141\\314\\210\", 0766);\n\telse\n\t\tdie(\"select --nfc or --nfd\");\n\n\tdir = opendir(\".\");\n\treaddir(dir);\n\n\twhile ((de = readdir(dir)) != NULL)\n\t\tprintf(\"%s\\n\", de->d_name);\n\n\treturn 0;\n}\n\nAnd then added this test:\n\ntest_expect_success 'unicode stuff' '\n\tmkdir nfc &&\n\t(\n\t\tcd nfc &&\n\t\ttest-tool create-and-read --nfc >../nfc.txt\n\t) &&\n\n\tmkdir nfd &&\n\t(\n\t\tcd nfd &&\n\t\ttest-tool create-and-read --nfd >../nfd.txt\n\t) &&\n\n\ttest_cmp nfc.txt nfd.txt\n'\n\nThis test always passes for me, and is essentially doing\na similar check that the prereq is doing, except that it\nactually writes both names to files instead of writing\none and doing a read with the other.\n\nAfter changing the \"$test_unicode\" instances to instances of\n\"test_expect_success\", I ran t0050 under --stress and quickly\ngot a failure on the 'merge (silent unicode normalization)'\ntest:\n\n\nexpecting success of 0050.11 'merge (silent unicode normalization)': \n        git reset --hard initial &&\n        git merge topic\n\n+ git reset --hard initial\nerror: unable to unlink old 'ä': No such file or directory\nfatal: Could not reset index file to revision 'initial'.\nerror: last command exited with $?=128\nnot ok 11 - merge (silent unicode normalization)\n\n\nDeleting that test gave mostly-consistent results, although I once\ngot a failure on the \"setup unicode normalization tests\" tests with\na similar error message:\n\n+ git checkout -f main\nerror: unable to unlink old 'ä': No such file or directory\nSwitched to branch 'main'\nerror: last command exited with $?=1\nnot ok 8 - setup unicode normalization tests\n \n>> (\"ä\" is the precomposed form \"a¨\" is the decomposed form,\n>>  typically both render to the same glyph on the screen,\n>>  and a hex dump or xxd will show what we had.\n>>  I just use this notation here for illustration)\n>>\n>> Should we contact the zfs developers ?\n\nHopefully someone has a good way to contact them, and I\ncan start a thread at the appropriate place. To optimize\nfor their time, what is our minimal reproduction steps?\n\n1. Build Git at the v2.34.0 tag.\n2. cd to t/\n3. ./t0050-filesystem.sh --stress\n\nThose instructions (given enough time) should get the\nrepro on test 8, 'setup unicode normalization tests'.\n\nTo get the faster stress, the same steps work except\nuse the 'zfs-minimal' branch at my fork [1] because it\nchanges the tests to expect success, and demonstrates\nthe unpredictable tests more quickly.\n\n[1] https://github.com/derrickstolee/git/tree/zfs-minimal\n\nThanks,\n-Stolee\n"}]}