Re: Problem with large files on different OSes

From: Christopher Jefferson <caj@cs.st-andrews.ac.uk>
Date: May 27, 2009, 17:22 UTC
Message-ID: <submission.1M9Mpl-0001CW-M0@mail.cs.st-andrews.ac.uk>
In-Reply-To: <alpine.LFD.2.01.0905270942580.3435@localhost.localdomain>

On 27 May 2009, at 17:59, Linus Torvalds wrote:

Show 7 quoted lines

>
>
> On Wed, 27 May 2009, Linus Torvalds wrote:
>>
>> I'll see if I can make us handle the "big file without diff" case  
>> better
>> by chunking.

> So "don't do it then" or "make sure you are 64-bit and have lots of
> memory if you do it" may well be the right solution.

Thank you for that description of the problem, I can see how hard it is.

Perhaps it might be useful to think about how to codify "don't do it then" in a reasonably simple, automatic way?

I've been trying to write a pre-commit hook (I think that's the right place?) which would refuse commits larger than some file size (512MB as a random number I decided), but am having trouble getting it to work right, and generally. Would such a thing be easy, and would that be the right place to put it?

While I wouldn't suggest this become default, providing such a hook, and describing why you might want to use it, would seem to avoid the accidental part of the problem.

Of course, people should really notice that they are submitting large files, but it's easy(ish) to commit some output file from a program, without realising the file ended up being the wrong side of 1GB.

Chris