Re: [PATCH] Prevent git from rehashing 4GBi files

From: Jason Hatton <jhatton@globalfinishing.com>
Date: May 7, 2022, 02:15 UTC
Message-ID: <CY4PR16MB1655F83010A128D4ED67C7EDAFC49@CY4PR16MB1655.namprd16.prod.outlook.com>

Show 18 quoted lines

>Philip Oakley <philipoakley@iee.email> writes:
>
>>> This may treat non-zero multiple of 4GiB as "not racy", but has
>>> anybody double checked the concern Réne brought up earlier that a
>>> 4GiB file that was added and then got rewritten to 2GiB within the
>>> same second would suddenly start getting treated as not racy?
>> This is the pre-existing problem, that ~1in 2^31 size changes might not
>> get noticed for size change. The 0 byte / 4GiB change is an identical
>> issue, as is changing from 3 bytes to 4GiB+3 bytes, etc., so that's no
>> worse than before (well maybe twice as 'unlikely').
>
>OK, it added one more case to 2^32-1 existing cases, I guess.
>
>>> The patch (the firnal version of it anyway) needs to be accompanied
>>> by a handful of test additions to tickle corner cases like that.
>> They'd be protected by the EXPENSIVE prerequisite I would assume.
> 
> Oh, absolutely.  Thanks for spelling that out.

I have been testing out the patch a bit and have good and (mostly) bad news.

What works using a munge value of 1.

$ git add $ git status

Racy seems to work.

$ touch .git/index 4GiB # 4GiB is now racy $ git status # Git will rehash the racy file $ git status # Git cached the file. Second status is fast.

What doesn't work.

$ git checkout 4GiB $ fatal: packed object is corrupt!

Using a munge value of 1<<31 causes even more problems. The file hash in the index for 4GiB files (git ls-files -s --debug) are set to the zero file hash.

I looked up and down the code base and couldn't figure out how the munged value was leaking out of read-cache.c and breaking things. Most of the code I found tends to use stat and then convert that to a size_t, not using the munged unsigned int at all.

Maybe someone else will have better luck. This seems over my head :(

Thanks -- Jason