Re: Git Scaling: What factors most affect Git performance for a large repo?

From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date: Mar 2, 2015, 19:42 UTC
Message-ID: <CACBZZX4rjYoQZRkzrfBYp8jOFwGm64Eh2-6JpJJ=KMFNxCOkbg@mail.gmail.com>
In-Reply-To: <54EC7241.7000500@alum.mit.edu>

On Tue, Feb 24, 2015 at 1:44 PM, Michael Haggerty <mhagger@alum.mit.edu> wrote:

Show 30 quoted lines

> On 02/20/2015 03:25 PM, Ævar Arnfjörð Bjarmason wrote:
>> On Fri, Feb 20, 2015 at 1:09 PM, Ævar Arnfjörð Bjarmason
>> <avarab@gmail.com> wrote:
>>> On Fri, Feb 20, 2015 at 1:04 AM, Duy Nguyen <pclouds@gmail.com> wrote:
>>>> On Fri, Feb 20, 2015 at 6:29 AM, Ævar Arnfjörð Bjarmason
>>>> <avarab@gmail.com> wrote:
>>>>> Anecdotally I work on a repo at work (where I'm mostly "the Git guy") that's:
>>>>>
>>>>>  * Around 500k commits
>>>>>  * Around 100k tags
>>>>>  * Around 5k branches
>>>>>  * Around 500 commits/day, almost entirely to the same branch
>>>>>  * 1.5 GB .git checkout.
>>>>>  * Mostly text source, but some binaries (we're trying to cut down[1] on those)
>>>>
>>>> Would be nice if you could make an anonymized version of this repo
>>>> public. Working on a "real" large repo is better than an artificial
>>>> one.
>>>
>>> Yeah, I'll try to do that.
>>
>> tl;dr: After some more testing it turns out the performance issues we
>> have are almost entirely due to the number of refs. Some of these I
>> knew about and were obvious (e..g. git pull), but some aren't so
>> obvious (why does "git log" without "--all" slow down as a function of
>> the overall number of refs?).
>
> I'm assuming that you pack your references periodically. (If not, you
> should, because reading lots of loose references is very expensive for
> the commands that need to iterate over all references!)

Yes, as mentioned in another reply of mine, like this:

    git --git-dir={} gc &&
    git --git-dir={} pack-refs --all --prune &&
    git --git-dir={} repack -Ad --window=250 --depth=100
--write-bitmap-index --pack-kept-objects &&

Show 9 quoted lines

> On the other hand, packed refs also have a downside, namely that
> whenever even a single packed reference has to be read, the whole
> packed-refs file has to be read and parsed. One way that this can bite
> you, even with innocuous-seeming commands, is if you haven't disabled
> the use of replace references (i.e., using "git --no-replace-objects
> <CMD>" or GIT_NO_REPLACE_OBJECTS). In that case, almost any Git command
> has to read the "refs/replace/*" namespace, which, in turn, forces the
> whole packed-refs file to be read and parsed. This can take a
> significant amount of time if you have a very large number of references.

Interesting. I tried the rough benchmarks I posted above with GIT_NO_REPLACE_OBJECTS=1 and couldn't see any differences, although as mentioned in another reply --no-decorate had a big effect on git-log.

Show 9 quoted lines

> So try your experiments with replace references disabled. If that helps,
> consider disabling them on your server if you don't need them.
>
> Michael
>
> --
> Michael Haggerty
> mhagger@alum.mit.edu
>