[PATCH 0/3] speeding up on-demand fetch for blobs in partial clone

From: Jeff King <peff@peff.net>
Date: Sep 6, 2022, 22:58 UTC
Message-ID: <YxfQi4qg8uJHs7Gp@coredump.intra.peff.net>
In-Reply-To: <YxeTsWrjpKo+JGfq@coredump.intra.peff.net>

On Tue, Sep 06, 2022 at 02:38:41PM -0400, Jeff King wrote:

Show 16 quoted lines

> On Mon, Sep 05, 2022 at 11:17:21AM +0000, 程洋 wrote:
> 
> > Sorry, I told you the wrong branch. It should be "android-t-preview-1"
> > git clone --filter=blob:none --no-local -b android-t-preview-1 grade-plugin
> > 
> > Can you try this one?
> 
> Yes, I see more slow-down there. There are many more blobs there, but I
> don't think it's really the number of them, but their sizes.
> 
> The problem is that both upload-pack and pack-objects are keen to call
> parse_object() on their inputs. For commits, etc, that is usually
> sensible; we have to parse the object to see what it points to. But for
> blobs, the only thing we do is inflate a ton of bytes in order to check
> the sha1. That's not really productive here; if there is a bit
> corruption, the client will notice it on the receiving side.

So here's a cleaned-up series which makes this a lot faster.

The special sauce is in patch 2, along with timings. The first one is just preparing, and the final one is a small cleanup it enables.

I based these directly on the current tip of master. There will be a small conflict in the test script when merging with the "rev-list --verify-objects" series I just sent in:

  https://lore.kernel.org/git/Yxe0k++LA%2FUfFLF%2F@coredump.intra.peff.net/

The resolution is to just keep the tests added by both sides.

  [1/3]: parse_object(): allow skipping hash check
  [2/3]: upload-pack: skip parse-object re-hashing of "want" objects
  [3/3]: parse_object(): check commit-graph when skip_hash set

 object.c        | 21 ++++++++++++++++++---
 object.h        |  6 ++++++
 revision.c      | 14 +++-----------
 t/t1450-fsck.sh | 20 ++++++++++++++++++++
 upload-pack.c   |  8 ++------
 5 files changed, 49 insertions(+), 20 deletions(-)

-Peff