Re: Leaving large binaries out of the packfile

From: Shawn O. Pearce <spearce@spearce.org>
Date: Jun 11, 2010, 16:17 UTC
Message-ID: <20100611161744.GS14847@spearce.org>
In-Reply-To: <4C12566E.6020607@gnu.org>

Paolo Bonzini <bonzini@gnu.org> wrote:

Show 19 quoted lines

> On 06/10/2010 08:04 PM, Shawn O. Pearce wrote:
>> Joshua Jensen<jjensen@workspacewhiz.com>  wrote:
>>> Sometimes, 'git gc' runs out of memory.  I have to discover which file
>>> is causing the problem, so I can add it to .gitattributes with a
>>> '-delta' flag.  Mostly, though, the repacking takes forever, and I dread
>>> running the operation.
>>
>> If you have the list of big objects, you can put them into their
>> own pack file manually.  Feed their SHA-1 names on stdin to git
>> pack-objects, and save the resulting pack under .git/objects/pack.
>
> Do you know any simpler way than
>
> git log --pretty=format:%H | while read x; do
>   git ls-tree $x -- ChangeLog | awk '{print $3}'
> done | sort -u
>
> to do this?  I thought it would be nice to add --sha1-only to  
> git-ls-tree, but maybe I'm missing some other trick.

Maybe

  git rev-list --objects HEAD | grep ' ChangeLog'

pack-objects wants the output of rev-list --objects as input, file name and all. So its just a matter of selecting the right lines from its output.

Show 9 quoted lines

>> Assuming the pack was called pack-DEADC0FFEE.pack, create a file
>> called pack-DEADC0FFEE.keep in the same directory.  This will stop
>> Git from trying to repack the contents of that pack file.
>>
>> Now run `git gc` to remove those huge objects from the pack file
>> that contains all of the other stuff.
>
> That obviously wouldn't help if these large binaries are updated often,  
> however.

No, it doesn't. But you still could do this on a periodic basis. That way you only drag around a handful of recently created large binaries during a typical `git gc`, and not the entire project's history of them.

-- 
Shawn.