Re: pack-object poor performance (with large number of objects?)

From: Shawn Pearce <spearce@spearce.org>
Date: Oct 3, 2011, 16:05 UTC
Message-ID: <CAJo=hJtw+sYrP09zrDbZJNGHDYOeguQLkOe88FBYQDZrnaqsAw@mail.gmail.com>
In-Reply-To: <CAA01CspZijOO_xbR=OcaRaesTeSy=6RM4DR01-07qimVzxvJZA@mail.gmail.com>

On Mon, Oct 3, 2011 at 07:43, Piotr Krukowiecki <piotr.krukowiecki@gmail.com> wrote:

Show 12 quoted lines

> I'm having poor git gc (pack-object) performance. Please read below
> for details. What can I do to improve the performance/debug the reason
> for the slowness? Should I leave the process running over night, or
> should I stop it (for debugging)?
> CCing people who posted some patches/benchmarks for pack-objects recently.
>
> git gc was first run automatically by git svn clone. It found 1544673
> objects and worked for 50 minutes until I've killed it.
>
> Then I've run it by hand with --aggresive (because I've found on
> Internet it helped in some cases). It found 1742200 objects this time.
> At this moment it's been working for about 90 minutes.

Packing time depends on a number of factors. One of them is the number of unpacked objects to process. With 1.7 million objects, yes, its going to take some time. Another factor is how much RAM you have on your system. Packing requires a lot of memory, especially with the --aggressive flag, as the packer tries up to 250 different combinations of two objects searching for a good delta compression format, and all 250 of those are typically in-memory at once. If you have insufficient physical RAM, the system will swap, unless you decrease the window size.

> The large number of unpacked objects is probably caused by me - I've
> disabled auto gc when I was cloning from svn (I though it might speed
> up things if it didn't repack several times during clone, only once
> afterwards).

Yes, this the reason `git svn` runs GC during its import. If you defer all of the repacking work until the end, with everything loose, it can take a very, very long time to repack. If you repack as you go, the incremental repacks are less expensive than a full repack, and the entire process will go faster overall.

-- 
Shawn.