Re: [Question] Unicode weirdness breaking tests on ZFS?

From: Derrick Stolee <stolee@gmail.com>
Date: Nov 19, 2021, 18:30 UTC
Message-ID: <e56b0227-14fc-26cf-7b98-fbf01f3c5cd7@gmail.com>
In-Reply-To: <xmqqmtm0qdqp.fsf@gitster.g>

On 11/19/2021 12:03 PM, Junio C Hamano wrote:

Show 17 quoted lines

> Torsten Bögershausen <tboegi@web.de> writes:
> 
>> Should we conclude that the underlying os/zfs is not stable ?
>> Things don't seem to be reproducable
>>
>> What Git needs here in t0050 is that stat("ä") behaves the same as stat("a¨"),
>> when either "ä" or "a¨" exist on disk.
>> The same for open() and all other file system functions.
> 
> We either need to see these two are treated as the same thing, or
> these two are treated as two distict filesystem entities, just like
> stat("a") and stat("b") are.  What we absolutely need is the
> unification either always happens or never happens consistently.
> 
> I wonder what readdir() is returning.  After creat("ä") in an empty
> directory, does readdir() in there return "ä" or "a¨?  And vice
> versa?  Is this also inconsistent?

Following this suggestion, I added a test helper with this code:

int cmd__create_and_read(int argc, const char **argv)
{
	DIR *dir;
	struct dirent *de;

	if (strcmp(argv[0], "--nfc"))
		creat("\303\244", 0766);
	else if (strcmp(argv[0], "--nfd"))
		creat("\141\314\210", 0766);
	else
		die("select --nfc or --nfd");

	dir = opendir(".");
	readdir(dir);

	while ((de = readdir(dir)) != NULL)
		printf("%s\n", de->d_name);

	return 0;
}

And then added this test:

test_expect_success 'unicode stuff' '
	mkdir nfc &&
	(
		cd nfc &&
		test-tool create-and-read --nfc >../nfc.txt
	) &&

	mkdir nfd &&
	(
		cd nfd &&
		test-tool create-and-read --nfd >../nfd.txt
	) &&

	test_cmp nfc.txt nfd.txt
'

This test always passes for me, and is essentially doing a similar check that the prereq is doing, except that it actually writes both names to files instead of writing one and doing a read with the other.

After changing the "$test_unicode" instances to instances of "test_expect_success", I ran t0050 under --stress and quickly got a failure on the 'merge (silent unicode normalization)' test:

expecting success of 0050.11 'merge (silent unicode normalization)': 
        git reset --hard initial &&
        git merge topic

+ git reset --hard initial
error: unable to unlink old 'ä': No such file or directory
fatal: Could not reset index file to revision 'initial'.
error: last command exited with $?=128
not ok 11 - merge (silent unicode normalization)

Deleting that test gave mostly-consistent results, although I once got a failure on the "setup unicode normalization tests" tests with a similar error message:

+ git checkout -f main
error: unable to unlink old 'ä': No such file or directory
Switched to branch 'main'
error: last command exited with $?=1
not ok 8 - setup unicode normalization tests

Show 6 quoted lines

>> ("ä" is the precomposed form "a¨" is the decomposed form,
>>  typically both render to the same glyph on the screen,
>>  and a hex dump or xxd will show what we had.
>>  I just use this notation here for illustration)
>>
>> Should we contact the zfs developers ?

Hopefully someone has a good way to contact them, and I can start a thread at the appropriate place. To optimize for their time, what is our minimal reproduction steps?

1. Build Git at the v2.34.0 tag.
2. cd to t/
3. ./t0050-filesystem.sh --stress

Those instructions (given enough time) should get the repro on test 8, 'setup unicode normalization tests'.

To get the faster stress, the same steps work except use the 'zfs-minimal' branch at my fork [1] because it changes the tests to expect success, and demonstrates the unpredictable tests more quickly.

[1] https://github.com/derrickstolee/git/tree/zfs-minimal

Thanks, -Stolee