Re: Cygwin can't handle huge packfiles?

From: Junio C Hamano <junkio@cox.net>
Date: Apr 6, 2006, 04:13 UTC
Message-ID: <7vfykrwdcf.fsf@assigned-by-dhcp.cox.net>
In-Reply-To: <fa0b6e200604050624h13ebd8deg241ae98cef1f5a74@mail.gmail.com>

"Kees-Jan Dijkzeul" <k.j.dijkzeul@gmail.com> writes:

Show 12 quoted lines

> I'm trying to get Git to manage my companies source tree. We're
> writing software for digital TV sets. Anyway, the archive is about 5Gb
> in size and contains binaries, zip files, excel sheets meeting minutes
> and whatnot. So it doesn't compress very well. The 1.5Gb pack file
> hardly contains any history at all (five commits or so). On the flip
> side, for now I'll be the only one adding to the archive, so at least
> it will not grow that fast ;-)
>
> Anyway, to reconstitute the tree, I need very nearly the entire pack,
> so limiting the pack size won't do much good, as git will still try to
> allocate a total of 1.5Gb memory (which, unfortunately, isn't there
> :-)

Right now we LRU the pack files and evict older ones when we mmap too many, but the unit of eviction is the whole file, so it would not help the case like yours at all. It might be possible to mmap only part of a packfile, but it would involve fairly major surgery to sha1_file.c.