Re: git packs

>
>
> On Sat, 10 Nov 2007, bob wrote:
>>
>> The reason that I ask is that I have been playing different
>> scenarios using git 1.5.3.5 under MacOSX 10.4.10 mostly
>> all day and every time that
>>
>> A) 	a file approaches or exceeds 2gig on an 'add', it
>> 	results in:
>> 		fatal: Out of memory? mmap failed: Cannot allocate memory
>
> Git wants to handle single files as one single entity, so single  
> big files
> really do end up being very painful. The costs of compressing them and
> generating deltas would probably get prohibitively high *anyway*,  
> but it
> does mean that if you have gigabyte files, you do want a 64-bit VM.
>
> I thought OS X could do 64 bits these days. Maybe not.
>
> Anyway, that explains the "cannot allocate memory". Git simply  
> wants to
> mmap the whole file. You don't have enough VM space for it.
>
> (And if you seriously want to work with multi-gigabyte files, git  
> probbaly
> isn't going to perform wonderfully well, even if it *should* work  
> fine if
> you just have a full 64-bit environment that allows the mmap).
>
>> B) 	the repository size less the .git subdirectory approaches
>> 	4gig on a 'fetch' it results in:
>>
>> 	Resolving 3356 deltas...
>> 	fatal: serious inflate inconsistency: -3 (unknown compression  
>> method)
>
> That sounds really broken. I'm not seeing what would cause that, apart
> from some really bad data corruption and/or broken zlib  
> implementation.
> But if the pack-file really is 2GB+ in size, I could imagine some sign
> issues cropping up.
>
> git will generally use "unsigned long" (which is probably just 32- 
> bit on
> your setup), but since git in those circumstances would be limited  
> by the
> size of the VM _anyway_, that's not really much of a limitation  
> (although
> probably broken on the crazy Windows "LLP64" model). But maybe we have
> some place where we use a signed thing, or zlib does, and I could  
> see that
> causing breakage.
>
> But that code-sequence really should never even come *close* to the  
> 31-bit
> limit, as long as the individual objects themselves aren't bigger  
> than the
> available VM space (and git currently assumes "unsigned long" is
> sufficiently big to cover the VM space, which is not technically  
> correct,
> but should be fine on OS X too).
>
> That said, we should use "off_t" in that function. I suspect we have a
> number of people (read: me) who have grown too used to living in a  
> 64-bit
> world..
>
>> I have been testing on my laptop which has a 32-bit Intel Core Duo.
>
> Ok, so you're 32-bit limited even if there is were to be some 64-bit
> support for OS X.
>
>> Also, I have run the same tests on a dual quad-core Intel processor
>> which is 64 bit, (but not sure that Apple uses the 64 bits in  
>> 10.4.10).  I
>> get the same results as above.
>
> I'm pretty sure OS X defaults to a 32-bit environment, but has at  
> least
> *some* 64-bit support. It would definitely need to be enabled  
> explicitly
> (since they made the *insane* decision to move over to Intel laptop  
> chips
> six months before they got 64-bit support! Somebody at Apple is a  
> total
> idiot, and should get fired).
>
> So it would be interesting to hear if a 64-bit build would make a
> difference.
>
>> The zlib is at the latest revision of 1.2.3 and gcc is at 4.0.1
>> which from what I can tell supports large files, because 'off_t'  
>> is 8 bytes
>> which is the size used for a 'stat' file size.
>
> See above: single files are size-limited, but with a large off_t like
> yours, you should be fine. Except we may have screwed up.
>
>> I am just wondering if these size limitations exist for MacOSX
>> or maybe I am doing something wrong (which is probably
>> the case).
>
> We *have* had issues with broken implementations of "pread()" on some
> systems.
>
> You could try setting NO_PREAD in the Makefile and compiling with the
> compatibility function.. That's the only thing that comes to mind  
> as being
> worth trying in that area.
>
> And if you have some script to generate the repository (ie you aren't
> using "live data", but are testing the limits of the system), if  
> you can
> make that available, so that people with non-OSX environments can  
> test,
> that would be interesting..
>
> I certainly have some 32-bit environments too (old linux boxes),  
> but I'm
> too lazy to write a test-case, so I was hoping you'd be using some  
> simple
> scripts that I could just test and see if I can see the behaviour you
> describe myself.
>
> That said, I have worked with a 3GB pack-file (one of the KDE trial
> repos). That worked fine. But git does tend to want a *lot* of  
> memory for
> really big repositories, so I suspect that if you actually work  
> with 2GB+
> pack-files, you'll be wanting a 64-bit environment just because  
> you'll be
> wanting more than 2GB of physical RAM in order to be able to access it
> efficiently.
>
> 			Linus