Re: [PATCH 0/2] optimizing pack access on "read only" fetch repos

From: Shawn Pearce <spearce@spearce.org>
Date: Feb 2, 2013, 10:07 UTC
Message-ID: <CAJo=hJvPz_EYFGzb8LLdW77T2MhcOqnUBtroLMWKkEWWKSTchQ@mail.gmail.com>
In-Reply-To: <20130201091431.GC30644@sigill.intra.peff.net>

On Fri, Feb 1, 2013 at 1:14 AM, Jeff King <peff@peff.net> wrote:

Show 17 quoted lines

> On Thu, Jan 31, 2013 at 08:47:37AM -0800, Shawn O. Pearce wrote:
>
>> >>  - System resource cost we incur by having to keep 50 file
>> >>    descriptors open and maintaining 50 mmap windows will reduce by
>> >>    50 fold.
>> >
>> > I wonder how measurable that is (and if it matters on Linux versus less
>> > efficient platforms).
>>
>> It does matter. We know it has a negative impact on JGit even on Linux
>> for example. You don't want 300 packs in a repository. 50 might be
>> tolerable. 300 is not.
>
> I'd love to see numbers if you have them. It's not that I don't believe
> it is slower, but knowing _how much_ is important when thinking about
> what kind of performance increase we are looking to get (which in turn
> impacts how much effort to put into the repacking).

Never done a formal experiment. Just working from memory where 4 years and 3 desks ago I got called because one of our Git servers was struggling to keep up with user git fetch commands. Turns out the repository had O(200) pack files. git gc that normally took only about 5 minutes took a hellva lot longer, like an hour or more.

The problem happened because the server was saving every push to a pack and never exploding incoming packs to loose objects. This meant the thin packs from the wire got delta bases appended to them. pack-objects was looking at O(50) different alternatives for most objects when trying to decide which one it should copy into the output pack... for either a fetch/clone client, or the gc I was trying to run to fix the repository.