Large repo and pack.packsizelimit

> I am using MSysgit 1.7.9 on WinXP 32bit and have a very large repo
> (10GB in .git; 20GB in source tree).  I had to set
> pack.packsizelimit=1024MB to prevent "out of memory" during repacking
> in git-gc and everything seemed to work fine.
> 
> When I tried to clone this repo an "out of memory" occured because the
> packs to be transferred by the git protocol are not limited by
> pack.packsizelimit.

> On Thu, May 03, 2012 at 01:57:58PM +0200, th.acker66@arcor.de wrote:
> 
> > I am using MSysgit 1.7.9 on WinXP 32bit and have a very large repo
> > (10GB in .git; 20GB in source tree).  I had to set
> > pack.packsizelimit=1024MB to prevent "out of memory" during repacking
> > in git-gc and everything seemed to work fine.
> > 
> > When I tried to clone this repo an "out of memory" occured because the
> > packs to be transferred by the git protocol are not limited by
> > pack.packsizelimit.
> 
> Yes, pack-objects respects pack.packsizelimit when creating local packs,
> but incoming packs from the network (which are processed by index-pack)
> are not split.
> 
> This should be fixed in git. Unfortunately, I don't know that it is as
> trivial as just splitting the incoming stream; we would also have to
> make sure that there were no cross-pack deltas in the result.

> > This should be fixed in git. Unfortunately, I don't know that it is as
> > trivial as just splitting the incoming stream; we would also have to
> > make sure that there were no cross-pack deltas in the result.
> 
> IMHO this is the wrong fix.  The pack size limit was created to deal 
> with storage media with limited capacity.  In this case, the repack 
> process should be told to limit its memory usage, and pack-index should 
> simply be taught to cope.

> On Tue, May 08, 2012 at 05:13:13PM -0400, Nicolas Pitre wrote:
> 
> > > This should be fixed in git. Unfortunately, I don't know that it is as
> > > trivial as just splitting the incoming stream; we would also have to
> > > make sure that there were no cross-pack deltas in the result.
> > 
> > IMHO this is the wrong fix.  The pack size limit was created to deal 
> > with storage media with limited capacity.  In this case, the repack 
> > process should be told to limit its memory usage, and pack-index should 
> > simply be taught to cope.
> 
> Hmm, you're right. I was thinking it helped to deal with memory
> addressing issues for 32-bit systems, but I guess
> core.packedGitWindowSize should be handling that. IOW, the 10G packfile
> should work just fine for normal access.
> 
> However, the OP did say he got an "out of memory" error during the
> clone. So maybe there is a problem to be fixed in index-pack there.

> 
> On Tue, 8 May 2012, Jeff King wrote:
> 
> > On Tue, May 08, 2012 at 05:13:13PM -0400, Nicolas Pitre wrote:
> > 
> > > > This should be fixed in git. Unfortunately, I don't know that it is as
> > > > trivial as just splitting the incoming stream; we would also have to
> > > > make sure that there were no cross-pack deltas in the result.
> > > 
> > > IMHO this is the wrong fix.  The pack size limit was created to deal 
> > > with storage media with limited capacity.  In this case, the repack 
> > > process should be told to limit its memory usage, and pack-index should 
> > > simply be taught to cope.
> > 
> > Hmm, you're right. I was thinking it helped to deal with memory
> > addressing issues for 32-bit systems, but I guess
> > core.packedGitWindowSize should be handling that. IOW, the 10G packfile
> > should work just fine for normal access.
> > 
> > However, the OP did say he got an "out of memory" error during the
> > clone. So maybe there is a problem to be fixed in index-pack there.
> 
> Was the OOM on the remote side (pack-objects) or on the local side 
> (index-pack) ?
> 
> Nicolas
> 

> To be exact I did the clone locally on the same machine and so the clone itself
> worked
> but I got the OOM during the first fetch. I "fixed" this by setting
> transfer.unpacklimit=100000
> which caused only loose objects to be transfered.
> So in this case I think the OOM was on the remote side. But there is another OOM
> if I try to repack locally.
> It seems to me that neither pack-objects nor index-pack respekt
> pack.packsizelimit and always
> try to pack all objects to be transferred resp. all local loose objects in one
> pack.
> I could live wth the transfer.unpacklimit=100000 but the local OOM stops me from
> using the cloned repo.

> > worked
> > but I got the OOM during the first fetch. I "fixed" this by setting
> > transfer.unpacklimit=100000
> > which caused only loose objects to be transfered.
> > So in this case I think the OOM was on the remote side. But there is another 

> > using the cloned repo.
> 
> I have some patches to make index-pack work better with large blobs
> but they're not ready yet. I think pack-objects works fine with large
> blobs as long as they are all in packs. Are there any loose objects on
> the source repo?
> 
> It's strange that you chose "256mb" as the upper limit for small
> objects in your first mail. Do you have a lot of >=10mb files? By
> default, files smaller than 512mb will be put in memory for delta. A
> lot of big (but smaller than 512mb) files can quickly consume all
> memory. If it's the case, maybe you can lower core.bigFileThreshold
> 
> Also maybe try remove the 1.2GB file from the source repo and see if
> it works better. That could give us some hints where the problem is.

> 
> Thomas <th.acker66 <at> arcor.de> writes:
> 
> > (1) all objects to be transferred to another repo 
> > (2) all loose objects when starting a local repack
> > Case (1) can be fixed by transfer.unpacklimit but there is no fix for (2).
> 
> Technically (1) is putting everything in a single pack to transfer, and it
> is only the receiving end that does the chopping.
> 
> For (2), you could do something like
> 
> 	keep=$( git rev-list --objects $some_rev |
> 	        git pack-objects --delta-base-offset pack ) &&
> 	mv pack-$keep.pack pack-$keep.idx .git/objects/pack/ &&
> 	echo "keep $some_rev" >.git/objects/pack/pack-$keep.keep
> 
> after finding a suitable $some_rev that is old enough so that it will be
> an ancestor of anything that matters in the future and gives small enough
> packfiles.  You may want to try doing the above multiple times, by picking
> a few strategic ranges, e.g.
> 
> 	for some_rev in v1.0 v1.0..v2.0 v2.0..v2.4 v2.4..v3.0
> 	do
> 		... the above four lines come here ...
> 	done
>