Re: serious performance issues with images, audio files, and other "non-code" data

From: Jeff King <peff@peff.net>
Date: May 18, 2010, 19:19 UTC
Message-ID: <20100518191933.GB2383@coredump.intra.peff.net>
In-Reply-To: <4BF2E168.2020706@puckerupgames.com>

On Tue, May 18, 2010 at 02:50:16PM -0400, John wrote:

> I just compiled the latest git. It got worse!!

I think Sverre is right that this is simply that --aggressive got more so in the last few versions. But do note that aggressive implies that we should pack from scratch, not reusing previously found deltas (or accepting that we didn't find deltas previously).

So you might want "git gc --aggressive" the _first_ time you pack, or possibly even very occasionally. But if you are packing every day, you should just use "git gc", which will run much more quickly (and would probably have acceptable behavior even without the -delta attribute, as it would only have to look at _new_ objects).

It will have to write the whole 200M packfile out each time, though. From your timings that looks to take about 50 seconds or so (just looking at the difference between wall clock time and CPU time, which is presumably spent in I/O).

Packing nightly won't hurt, but is perhaps excessive. It sounds like you actually have a fairly normal workload.

Show 6 quoted lines

> >How big is your packed repo? Given the pattern you describe below, I am
> [...]
> The packed .git dir is 203 MB. Yes, we make frequent changes to these
> files, and push/pull frequently as well. Just a normal development
> pattern, though. It's all manually done -- i.e., there's no automated
> bot doing excessive git operations.

OK, that is not very big. Once packed, you really should not see performance issues.

Show 6 quoted lines

> culprit. Yes, network bandwidth is always a factor, but I guess my
> expectation is that git shouldn't transfer too much more info than
> the amount of recent changes. For example, if we change 10 files for
> a total of 10MB, then my admittedly naive expectation is that git
> will send that 10MB of changes, plus some small constant amount of
> meta info... not the whole repo every time. No?

Your assumption is correct. Git should transmit at _worst_ 10MB in such a scenario (i.e., often much less because of delta compression, but in your case of apparently-random media files, probably about 10MB).

I wasn't clear from your message: you indicated the changes you made, but are you still having performance problems, or are you still waiting to get data?

-Peff