Re: pack operation is thrashing my server

From: Shawn O. Pearce <spearce@spearce.org>
Date: Aug 11, 2008, 15:01 UTC
Message-ID: <20080811150150.GC26363@spearce.org>
In-Reply-To: <a6b6acf60808110043t76dc0ae6l428c5da473d79c71@mail.gmail.com>

Ken Pratt <ken@kenpratt.net> wrote:

Show 16 quoted lines

> I just went as low as:
> 
> [core]
>         packedGitWindowSize = 1m
>         packedGitLimit = 4m
> [pack]
>         threads = 1
>         windowMemory = 4m
>         deltaCacheSize = 128k
> 
> And it didn't make a dent in memory usage. Server is still swapping
> within ~10 seconds of starting object compression.
> 
> I'm starting to think repacking is just not feasible on a 64-bit
> server with 256MB of RAM (which is a very popular configuration in the
> VPS market).

What is the largest object in that repository? Do you have a rough guess? You said earlier:

> The remote repository is bare, and is 180MB in size (says du), with
> 1824 objects.

That implies there is at least one really large object in that repository. The average of 101KB per object is not going to be a correct figure here as most commits and trees are _very_ tiny. It must be a large object. Those big objects are going to consume a lot of memory if they get inflated in memory.

You may very well be right that this particular repository of yours is simply not packable on a 64 bit system with only 256M. Packing takes a good chunk of memory as we maintain data about every single object, plus we need working space to unpack several objects at once so we can perform diffs to find deltas.

I'm not sure there are any more tunables you can try to tweak to reduce the memory usage further. The configuration above is pushed down about as low as it will go. For the most part the code is pretty good about not exploding memory usage.

You said earlier this was Git 1.5.6.4. I recently fixed a bug in the code that reads data from packs to prevent it from blowing out memory usage, but that bug fix was included in 1.5.6.4.

On the up side, packing should only be consuming huge memory like this when it needs to move loose objects into a pack file. I think Martin Langhoff suggested packing this on your laptop then using rsync over SSH to copy the pack file and .idx file to the server, so the server didn't have to spend time figuring out the deltas itself.

Even though the clone command will fire off git-pack-objects the pack-objects command will have a lot less work to do if the data it needs is already stored in existing pack files.

-- 
Shawn.