Re: [PATCH] mktree: learn about promised objects

From: Richard Oliver <roliver@roku.com>
Date: Jun 14, 2022, 16:33 UTC
Message-ID: <797af8c8-229f-538b-d122-8ea48067cc19@roku.com>
In-Reply-To: <0067c46a-7bfd-db9c-5156-16f032814464@github.com>

On 14/06/2022 15:14, Derrick Stolee wrote:

Show 14 quoted lines

> On 6/14/2022 9:36 AM, Richard Oliver wrote:
>> Do not use oid_object_info() to determine object type in mktree_line()
>> as this can cause promised objects to be dynamically faulted-in one at a
>> time which has poor performance. Instead, use a combination of
>> oid_object_info_extended() (with OBJECT_INFO_SKIP_FETCH_OBJECT option),
>> and the newly introduced promisor_object_type() to determine object type
>> before defaulting to fetch from remote.
> 
> Have you run some performance tests on this? It seems like this will
> scan all packed objects, which is probably much slower than just asking
> the remote for the object in most cases.
> 
> Thanks,
> -Stolee

Hi Stolee,

I've put together a synthetic experiment below (adding a new blob to anexisting tree) to show you the behaviour that we've been seeing. Our actual use-case (where we first encountered this behaviour) is updating submodules to a known hash. As you can see, the round-trip time of fetching objects one-by-one is very expensive.

Before, using system git (git version 2.32.0 (Apple Git-132)):

Show 30 quoted lines

> $ git init
> # Fetch a recent tree
> $ git fetch --filter=tree:0 --depth 1 https://github.com/git/git cdb48006b0ec7fe19794daf7b5363ab42d9d9371
> remote: Enumerating objects: 1, done.
> remote: Counting objects: 100% (1/1), done.
> remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
> Receiving objects: 100% (1/1), 13.12 KiB | 13.12 MiB/s, done.
> From https://github.com/git/git
>  * branch            cdb48006b0ec7fe19794daf7b5363ab42d9d9371 -> FETCH_HEAD
>
> $ NEW_BLOB=$(echo zzz | git hash-object --stdin -w)
>
> $ cat <(git ls-tree FETCH_HEAD) <(printf "100644 blob ${NEW_BLOB}\tzzz") | time git mktree
> remote: Enumerating objects: 1, done.
> remote: Counting objects: 100% (1/1), done.
> remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
> Receiving objects: 100% (1/1), 334 bytes | 334.00 KiB/s, done.
> remote: Enumerating objects: 1, done.
> remote: Counting objects: 100% (1/1), done.
> remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
> Receiving objects: 100% (1/1), 2.01 KiB | 2.01 MiB/s, done.
> remote: Enumerating objects: 1, done.
> remote: Counting objects: 100% (1/1), done.
> remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
> Receiving objects: 100% (1/1), 256 bytes | 256.00 KiB/s, done.
> # ...
> # SOME TIME LATER
> # ...
> e26c7ce7357b1649da7b4200d4e80d0b668db8d4
>       286.49 real        15.66 user        15.59 sys

Repeated experiment, but using modified mktree:

> $ cat <(git ls-tree FETCH_HEAD) <(printf "100644 blob ${NEW_BLOB}\tzzz") | time git mktree
> e26c7ce7357b1649da7b4200d4e80d0b668db8d4
>         0.06 real         0.01 user         0.03 sys

Did you have any other sort of performance test in mind? The remotes we typically deal with are geographically far away and deal with a high volume of traffic so we're keen to move behaviour to the client where it makes sense to do so.

Thanks, Richard