Re: [PATCH] Prevent megablobs from gunking up git packs

From: Shawn O. Pearce <spearce@spearce.org>
Date: May 22, 2007, 06:30 UTC
Message-ID: <20070522063050.GD11636@spearce.org>
In-Reply-To: <46528A48.9050903@gmail.com>

Dana How <danahow@gmail.com> wrote: ...

> 7 packfiles of 12.5GB total (--window=0 and --depth=0 were
> used due to runtime limits).  When using these packfiles,

...

Show 10 quoted lines

> Repacking the entire repository with a max-blob-size of 256KB
> resulted in a single 13.1MB packfile,  as well as 2853 loose
> objects totaling 15.4GB compressed and 100.08GB uncompressed,
> 11 files per objects/xx directory on average.  All was created
> in half the runtime of the previous yet with standard
> --window=10 and --depth=50 parameters.  The data in the
> packfile was 270MB uncompressed in 35976 blobs.  Operations
> such as "git-log --pretty=oneline" were about 30X faster
> on a cold cache and 2 to 3X faster otherwise.  Process sizes
> remained reasonable.

Can you give me details about your system? Is this a 64 bit binary? What is your core.packedGitWindowSize and core.packedGitLimit set to?

It sounds like the packed version was almost 3 GiB smaller, but was slower because we were mmap'ing far too much data at startup and that was making your OS page in things that you didn't really need to have.

Mind trying git-log with a smaller core.packedGitWindow{Size,Limit}? Perhaps its just as simple as our defaults are far far too high for your workload...

-- 
Shawn.