Re: large(25G) repository in git

From: Nicolas Pitre <nico@cam.org>
Date: Mar 25, 2009, 01:47 UTC
Message-ID: <alpine.LFD.2.00.0903242124470.26337@xanadu.home>
In-Reply-To: <49C9818A.9040507@brainfood.com>

On Tue, 24 Mar 2009, Adam Heath wrote:

Show 18 quoted lines

> Nicolas Pitre wrote:
> > On Tue, 24 Mar 2009, Adam Heath wrote:
> > 
> >> Sam Hocevar wrote:
> >>>    In your particular case, I would suggest setting pack.packSizeLimit
> >>> to something lower. This would reduce the time spent generating a new
> >>> pack file if the problem were to happen again.
> >> Yeah, saw that one, but *after* I had this problem.  The default, if
> >> not set, is unlimited, which in this case, is definately *not* what we
> >> want.
> > 
> > In your particular case, if the problem is actually what I think it is, 
> > the pack.packSizeLimit wouldn't have made any difference.  This setting 
> > affects local repacking only and has no effect what so ever on the push 
> > operation.
> 
> Ooh.  Care to enlighten those of us not blessed with git internal
> knowledge?

See my previous email for a likely explanation about your issue.

As to the pack.packSizeLimit setting: it is used when repacking only in order to avoid big packs on systems that might have issues dealing with large files. During a repack, if the currently produced pack is about to get over that limit, then the pack is closed and a new one is started. You therefore end up with many packs.

The transfer protocol used during a fetch or a push uses the pack format streamed over the network, but only one pack can be transferred that way. Maybe the reception of a pack during a network transfer should be split according to pack.packSizeLimit as well, but this is currently not implemented at all. No one complained about that either, so I'm guessing that splitting a large pack, if needed, by using 'git repack' after a clone/fetch is good enough.

Personally, I don't think actively splitting packs into smaller ones is that useful, unless you wish to archive them on a file system which cannot handle files larger than 2GB or the like.

> On another note, anyone have a goat I can buy, for the sacrifice?

Beware the wrath of Git...

Nicolas