Re: best practices against long git rebase times?

From: demerphq <demerphq@gmail.com>
Date: Dec 4, 2015, 17:33 UTC
Message-ID: <CANgJU+W2NWX4eK8M7GXu7+kmKsyEriau=_WXO8EEQZinCFTtRg@mail.gmail.com>
In-Reply-To: <20151204172851.GQ18913@serenity.lan>

On 4 December 2015 at 18:28, John Keeping <john@keeping.me.uk> wrote:

Show 38 quoted lines

> On Fri, Dec 04, 2015 at 06:09:33PM +0100, demerphq wrote:
>> On 4 December 2015 at 16:05, Andreas Krey <a.krey@gmx.de> wrote:
>> > Hi all,
>> >
>> > our workflow is pretty rebase-free for diverse reasons yet.
>> >
>> > One obstacle now appearing is that rebases simply take
>> > very long - once you might want to do a rebase there are
>> > several hundred commits on the remote branch, and our tree
>> > isn't small either.
>> >
>> > This produces rebase times in the minute range.
>> > I suppose this is because rebase tries to see
>> > if there are new commits in the destination
>> > branch that are identical to one of the local
>> > commits, to be able to skip them. (I didn't
>> > try to verify this hypothesis.)
>> >
>> > What can we do to make this faster?
>>
>> I bet you have a lot of refs; tags, or branches.
>>
>> git rebase performance along with many operations seems to scale
>> proportionately to the number of tags.
>>
>> At $work we create a tag every time we "roll out" a "server type".
>>
>> This produces many tags a day.
>>
>> Over time rebase, and many operations actually, start slowing down to
>> the point of painfulness.
>>
>> The workaround we ended up using was to set up a cron job and related
>> infra that removed old tags.
>>
>> Once we got rid of most of our old tags git became nice to use again.
>
> This is quite surprising.  Were you using packed or loose tags?

It didn't matter.

> It would be interesting to run git-rebase with GIT_TRACE_PERFORMANCE to
> see which subcommand is slow in this particular scenario.

These days it isn't that slow :-)

But I cc'ed Avar, he did the work on that, all I know is when he finished the tag remover I stopped cursing every time I rebased.

I believe I remember him saying that you can reproduce it using a public repo by taking the linux repo and creating a tag every 10 commits or so. Once you are done git in many operations will be nice and slow!

In all fairness however, I do believe that some of the recent changes to git helped, but I dont how much or which. What I do know is we still have the cron sweeper process cleaning refs. (It broke one of my repos that I set up with --reference just the other day).

Yves

-- 
perl -Mre=debug -e "/just|another|perl|hacker/"