From: Jonathan Nieder <jrnieder@gmail.com>
Date: Wed, 04 Dec 2019 04:23:32 GMT
Subject: Re: [PATCH] packfile.c: speed up loading lots of packfiles.
Message-ID: <20191204042332.GG214771@google.com>
In-Reply-To: <20191203221730.GA28419@sigill.intra.peff.net>

Jeff King wrote:

> I agree that 10,000 packs is ridiculous, but we do see it (and worse)
> occasionally from people pushing in a loop before our scheduled
> maintenance kicks in.

On that subject: one thing Martin Fick (cc-ed) has suggested is moving
more of the garbage collection work "inline" into the push path.  That
is, instead of letting someone push 10,000 packs in a loop, build
concatenated packs ("exponential rollup") in the push path and don't
return success and commit the packs into the object store until we're
done.  That way a reasonably small amortized cost is paid up front by
the pusher instead of later by everyone.

Midx changes things a little: it might make sense to build
concatenated idxes instead of packs, which would still avoid the same
quadratic behavior.

Just a random thought,
Jonathan