Re: [PATCH] Prevent megablobs from gunking up git packs
- From
- Dana How <danahow@gmail.com>
- Date
- May 22, 2007, 07:33 UTC
- Message-ID
- <56b7f5510705220033p43d20aabnbf86f1f6959d611a@mail.gmail.com>
- In-Reply-To
- <20070522063050.GD11636@spearce.org>
On 5/21/07, Shawn O. Pearce <spearce@spearce.org> wrote:
Show 7 quoted lines
> Dana How <danahow@gmail.com> wrote: > > ... Operations > > such as "git-log --pretty=oneline" were about 30X faster > > on a cold cache and 2 to 3X faster otherwise. Process sizes > > remained reasonable. > > Can you give me details about your system? Is this a 64 bit binary?
RHEL4/Nahant on an Opteron. Yes.
> What is your core.packedGitWindowSize and core.packedGitLimit set to?
I didn't change the default.
> It sounds like the packed version was almost 3 GiB smaller, but > was slower because we were mmap'ing far too much data at startup > and that was making your OS page in things that you didn't really > need to have.
The difference in size is because of the "Custom compression levels" patch -- now the loose objects use Z_BEST_SPEED, whereas the packs use Z_DEFAULT_COMPRESSION.
> Mind trying git-log with a smaller core.packedGitWindow{Size,Limit}?
> Perhaps its just as simple as our defaults are far far too high for
> your workload...I think that's a good idea and it should be easy to try tomorrow. It will improve the cold cache case definitely.
But we need to consider both *read* and *creation* performance. The portion of the repo I imported to git grows at about 500MB/week (compressed). Should I repack -a every week? Every month? In any case, should I use default window/depth, or 0/0? If default, run-times are prohibitive (in fact, I've always killed each attempt so the machine could be used for "real" work), and if 0/0, then I lose deltification on all objects.
These megablobs really are outliers and stress the "one size fits all" approach of packing in git. As a thought experiment, let's (1) pretend git-repack takes --max-blob-size= and --max-pack-size= , (2) pretend the patch doesn't add the repack.maxblobsize variable, and (3) do the following: % git-repack -a -d --max-blob-size=256 % git-repack --max-pack-size=2047 --window=0 --depth=0 The first step makes a digestible 13MB packfile, and the second puts all the megablobs in 6+ 2GB packfiles. Is there really any advantage to carrying out the second step? If I'm processing a 100MB+ blob, do I really care about an extra open(2) call?
Thanks,
-- Dana L. How danahow@gmail.com +1 650 804 5991 cell