Re: [External Mail]Re: Partial-clone cause big performance impact on server

From: Jeff King <peff@peff.net>
Date: Sep 1, 2022, 16:19 UTC
Message-ID: <YxDbfXyWzgokb1Bq@coredump.intra.peff.net>
In-Reply-To: <b0101e7e0e61496a92c2299454c2696a@xiaomi.com>

On Thu, Sep 01, 2022 at 06:53:01AM +0000, 程洋 wrote:

Show 10 quoted lines

> At first I also think it's some implementation bugs by jgit. However I
> can also reproduce it on cgit. Here is the steps, I'm not sure if you
> can reproduce too.
> 
> 1. Clone a repository from AOSP to local machine:  `git clone
>    "https://android.googlesource.com/platform/prebuilts/gradle-plugin"`
> 2. try to clone from localhost using cgit server.
>    `GIT_TRACE_PACKET=1 git clone --filter=blob:none -b master
>    user@localhost:/home/user/repositories/gradle-plugin `
> 3. During checkout phase, it also takes 15 seconds before actual downloading.

I don't see that at all. A few things on your reproduction:

  - you have to tell the local server repo that filters are OK:

      git -C gradle-plugin config uploadpack.allowfilter true

  - your example goes over localhost ssh. Is your server configured to
    allow passing the GIT_PROTOCOL environment variable? If not, you're
    using the v0 protocol. In which case you'll have to set a config
    option to allow clients to fetch objects that the server didn't
    advertise.

    If you do it with allowReachableSHA1InWant, like this:

      git -C gradle-plugin config uploadpack.allowReachableSHA1InWant true

    then there will be a short delay while it checks their
    reachability. That check happens via an external rev-list. I think
    it's not clever enough to use bitmaps, though it could. However, in
    this example, the delay only seems to be around 800ms for me (and of
    course we didn't generate bitmaps anyway, so that wouldn't matter).

    If you instead do:

      git -C gradle-plugin config uploadpack.allowAnySHA1InWant true

    then that reachability check goes away.

    But on modern servers, most of this should be moot anyway. A
    well-configured server should support protocol v2, which defaults to
    allowAnySHA1InWant.

    If you use --no-local to disable local-clone optimizations, then you
    can use --filter without having to go through ssh. That should use
    protocol version 2, as a "real" server would.

So all told, I think a more realistic reproduction is:

  $ git clone https://android.googlesource.com/platform/prebuilts/gradle-plugin
  $ git -C gradle-plugin config uploadpack.allowfilter true
  $ git clone --filter=blob:none --no-local -b master grade-plugin foo

which takes ~3s for me.

I do think upload-pack spends more time than it needs in this case, as it's keen to call parse_object() on oids that the client asks for. Which means we'll open up those blobs and check their sha1s before sending them out, which isn't strictly necessary.

All of this seems orthogonal to the original claim that "Counting objects" was taking a while, though. The delays here are all inside upload-pack, before it spawns pack-objects. And it's pack-objects that says "Counting objects".

-Peff