Re: pack operation is thrashing my server

From: Nicolas Pitre <nico@cam.org>
Date: Aug 13, 2008, 14:35 UTC
Message-ID: <alpine.LFD.1.10.0808131024460.4352@xanadu.home>
In-Reply-To: <1EE44425-6910-4C37-9242-54D0078FC377@adacore.com>

On Tue, 12 Aug 2008, Geert Bosch wrote:

> I've always felt that keeping largish objects (say anything >1MB)
> loose makes perfect sense. These objects are accessed infrequently,
> often binary or otherwise poor candidates for the delta algorithm.

Or, as I suggested in the past, they can be grouped into a separate pack, or even occupy a pack of their own. As soon as you have more than one revision of such largish objects then you lose again by keeping them loose.

Show 11 quoted lines

> Many repositories are mostly well-behaved with large number of text
> files that aren't overly large and compress/diff well. However, often
> a few huge files creep in. These might be a 30 MB Word or PDF documents
> (with lots of images of course), a bunch of artwork, some random .tgz files
> with required tools or otherwise.
> 
> Regardless of their origin, the presence of such files in real-world SCMs
> is a given and can ruin performance, even if they're hardly ever accessed
> or updated. If we would leave such oddball objects loose, the pack would
> be much smaller, easier to generate, faster to use and there should be no
> memory usage issues.

You'll have memory usage issues whenever such objects are accessed, loose or not. However, once those big objects are packed once, they can be repacked (or streamed over the net) without really "accessing" them. Packed object data is simply copied into a new pack in that case which is less of an issue on memory usage, irrespective of the original pack size.

Nicolas