Re: NewHash alternatives and SHA benchmarks
- From
Philip Oakley <philipoakley@iee.email>
- Date
- Dec 23, 2019, 15:15 UTC
- Message-ID
- <5aff676a-c4b2-6448-f99c-ec6f54f25a1a@iee.email>
- In-Reply-To
- <9279D11D-2BC2-4800-8282-C2219F152269@mac.com>
On 23/12/2019 01:59, Michael Clark wrote:
> git doesn’t really support large files that well either so I’ll have to keep that in mind...
I presume you are thinking of the Git-for-Windows (GfW) implementation here, where 'sizeof(long)'=32 bits, so currently the GfW is limited to 2/4GB file sizes.
The limitation is partly due to the zlib library API which uses 'long' for stream sizes, though does have 'pointer*' file sizing, and hence follows the Microsoft 32->64 bit transition approach. However zlib is able to handle larger files when the deflation/inflation is done in chunks of less than 'size(long)'.
On the 'Linux' version the limit is a 64 bit address and file size, so probably not a real issue on those platforms ;-)
There has been a body of work on the change to using size_t for Git file sizes, but such a change is extensive and may not be acceptable because of the size of the code churn. I (and others!) had been looking at it https://github.com/git-for-windows/git/pull/2179.
Hope that helps clarify the large file support limitation on Windows.
Philip