Re: [PATCH] packfile.c: speed up loading lots of packfiles.

From: Jonathan Nieder <jrnieder@gmail.com>
Date: Dec 4, 2019, 04:23 UTC
Message-ID: <20191204042332.GG214771@google.com>
In-Reply-To: <20191203221730.GA28419@sigill.intra.peff.net>

Jeff King wrote:

> I agree that 10,000 packs is ridiculous, but we do see it (and worse)
> occasionally from people pushing in a loop before our scheduled
> maintenance kicks in.

On that subject: one thing Martin Fick (cc-ed) has suggested is moving more of the garbage collection work "inline" into the push path. That is, instead of letting someone push 10,000 packs in a loop, build concatenated packs ("exponential rollup") in the push path and don't return success and commit the packs into the object store until we're done. That way a reasonably small amortized cost is paid up front by the pusher instead of later by everyone.

Midx changes things a little: it might make sense to build concatenated idxes instead of packs, which would still avoid the same quadratic behavior.

Just a random thought, Jonathan