From: Geert Bosch <bosch@adacore.com>
Date: Wed, 13 Aug 2008 03:58:22 GMT
Subject: Re: pack operation is thrashing my server
Message-ID: <70550C21-8358-4BEF-A7BA-3A41C1ACB346@adacore.com>
In-Reply-To: <20080813031503.GC5855@spearce.org>


On Aug 12, 2008, at 23:15, Shawn O. Pearce wrote:

> Geert Bosch <bosch@adacore.com> wrote:
>> I've always felt that keeping largish objects (say anything >1MB)
>> loose makes perfect sense. These objects are accessed infrequently,
>> often binary or otherwise poor candidates for the delta algorithm.
>
> Sadly this causes huge problems with streaming a pack because the
> loose object has to be inflated and then delfated again to fit into
> the pack stream.
Sure, but that really is not that much of an issue. For people
with large systems connected by very fast networks, the current
situation is probably fine, and spending a lot of effort for
packing often makes sense.

However, for a random repository of Joe User, all the effort spent
on packing will probably never be gained back. Most people just
suck content from upstream and at most maintain a couple of local
hacks on top of that. Little or nothing is ever pushed to other
systems.

Even when pushing to other systems, this often is just a handful of  
objects
though a slow line and compression/decompression speeds just don't  
matter
much.

> The new style loose object format was meant to fix this problem,
> and it did, but the code was difficult to manage so it was backed
> out of the tree.

One nice optimization we could do for those pesky binary large objects
(like PDF, JPG and GZIP-ed data), is to detect such files and revert
to compression level 0. This should be especially beneficial
since already compressed data takes most time to compress again.

   -Geert