Re: [External Mail]Re: why git is so slow for a tiny git push?

From: Jeff King <peff@peff.net>
Date: Oct 12, 2021, 21:46 UTC
Message-ID: <YWYCIndv/u67lNQU@coredump.intra.peff.net>
In-Reply-To: <87pmsak0hl.fsf@evledraar.gmail.com>

On Tue, Oct 12, 2021 at 12:06:04PM +0200, Ævar Arnfjörð Bjarmason wrote:

Show 5 quoted lines

> But more generally with these side-indexes it seems to me that the code
> involved might not be considering these sorts of edge cases, i.e. my
> understanding from you above is that if we have bitmaps anywhere we'll
> try to in-memory use them for all the objects in play? Or that otherwise
> having "partial" bitmaps leads to pathological behavior.

Sure, if there was an easy way to know beforehand whether the bitmap was going to help or run into these pathological cases, it would be nice to detect it. I don't know what that is (and I've given it quite a lot of thought over the past 8 years).

I suspect the most direction would be to teach the bitmap code to behave more like the regular traversal by just walking down to the UNINTERESTING commits. Right now it gets a complete bitmap for the commits we don't want, and then a bitmap for the ones we do want, and takes a set difference.

It could instead walk both sides in the usual way, filling in the bitmap for each, and then stop when it hits boundary commits. The bitmap for the boundary commit (if we don't have a full one on-disk) is filled in with what's in its tree. That means it's incomplete, and the result might include some extra objects (e.g., if boundary~100 had a blob that went away, but later came back in a descendant that isn't marked uninteresting). That's the same tradeoff the non-bitmap traversal makes.

It would be pretty major surgery to the bitmap code. I haven't actually tried it before.

-Peff