Re: Git fetch slow on local repository with 600k refs

From: Sean Allred <allred.sean@gmail.com>
Date: Mar 14, 2023, 14:29 UTC
Message-ID: <m0wn3jxu7g.fsf@epic96565.epic.com>
In-Reply-To: <e28a23e8eb044d26947462b8619e88bd@xiaomi.com>

程洋 <chengyang@xiaomi.com> writes:

Show 14 quoted lines

> We're holding a Gerrit server cluster. And uses pull-replication
> plugin to sync changes between master and slave.
>
> When a change is pushed to master, it notify the slave, and slave
> fetch it from master.
>
> But we found in a big repository with 600k refs. Fetch takes 5-10
> seconds even if fetching a 1 byte change. Here is the GIT_TRACE2_PERF
>
> I did an experiment to fetch a ref that my slave already have. And we
> can find git rev-list takes 2 seconds to perform. (I guess it try to
> find remote object from reachable objects of local refs one by one)
>
> Is there anyway to optimize such situation?

Do you need all those refs as refs -- or are you just looking to keep the commits?

We found a rather clever solution for the latter we're looking to upstream at some point to collect all refs into a single 'archive' ref that collects commits in fake merge commits (there's no actual conflict resolution happening -- we just use the same tree over and over). We make each commit message look like show-ref output. For example:

A single ref (refs/archive) pointing to commit (A), with contents

    tree <some arbitrary tree>
    parent <B> [... 500 other commits 'merged' in ...]
    author <system user>
    committer <system user>

    deadbeef0123456788... refs/tags/very/old/release-1
    deadbeef0123456789... refs/tags/very/old/release-2

When we want to pull a ref out of the archive, we have a process in place to do so. This keeps the total number of refs down and the fetch/push performance within acceptable limits.

-- Sean Allred