threads / discuss / 7401

Git performance problems with many tags

Subject: Git performance problems with many tags

## tl;dr

3 messages between Mar 26, 2007 and Mar 26, 2007.

replies: 2people: 3as markdown or json

Tim Allen· Mar 26, 2007, 04:53 UTC · lore

I'm not subscribed, please Cc me on replies.

My company is considering switching from CVS to a more branch-friendly version-control tool, and so of course we've been playing with git. We imported our CVS repository into git with git-cvsimport, which worked well enough, and resulted in a tree about the same size as the official kernel repository: 454121 objects, 334977 deltas.

However, operations like 'git-fetch' take much, much longer in our repository than in the kernel repository: a git-fetch that pulls no updates in the kernel repository takes 1.7s, while our repository (fetching from one repository to a clone on the same local disk) takes about 20 seconds. After some experimentation, we discovered that deleting all the 5557 imported CVS tags made things fast again. (Interestingly, "git-fetch --no-tags" was not appreciably quicker, while the tags were still around)

I searched the mailing list archives for similar problems, and the closest thread I could find was this one:

    http://thread.gmane.org/gmane.comp.version-control.git/20682/

...however, that thread seems to have decided that large numbers of binary files were the problem, which is not the case in our repository.

Does git have known scalability problems with large numbers of tags? Is there anything we can do to mitigate this slowdown, apart from just not using git's tag feature at all? Are there any details I've overlooked or misunderstood?

Tim Allen

Shawn O. Pearce· Mar 26, 2007, 06:07 UTC · re: Tim Allen · lore

Re: Git performance problems with many tags

Tim Allen <tim@commsecure.com.au> wrote:

Show 6 quoted lines

> However, operations like 'git-fetch' take much, much longer in our
> repository than in the kernel repository: a git-fetch that pulls no
> updates in the kernel repository takes 1.7s, while our repository
> (fetching from one repository to a clone on the same local disk) takes
> about 20 seconds. After some experimentation, we discovered that
> deleting all the 5557 imported CVS tags made things fast again.

Yes. git-fetch in the current stable versions is a Bourne shell script. Its not very fast as it loops through the refs (tags) that the two ends have.

There is work in under development (and being tested) that improves this by converting some of the critical parts to C.

> (Interestingly, "git-fetch --no-tags" was not appreciably quicker, while
> the tags were still around)

Yes, because that swtich didn't bypass that section of the fetch code where the slowdown is occuring.

> Does git have known scalability problems with large numbers of tags?

Yup.

> Is
> there anything we can do to mitigate this slowdown, apart from just not
> using git's tag feature at all?

Upgrade to the upcoming 1.5.1 release. Junio recently tagged 1.5.1-rc1. You can get it by cloning git.git and building the 'master' branch. It is quite stable. I would encourage you to give it a try.

-- 
Shawn.

Junio C Hamano· Mar 26, 2007, 06:13 UTC · re: Tim Allen · lore

Re: Git performance problems with many tags

Tim Allen <tim@commsecure.com.au> writes:

Show 10 quoted lines

> However, operations like 'git-fetch' take much, much longer in our
> repository than in the kernel repository: a git-fetch that pulls no
> updates in the kernel repository takes 1.7s, while our repository
> (fetching from one repository to a clone on the same local disk) takes
> about 20 seconds.
> ...
> Does git have known scalability problems with large numbers of tags? Is
> there anything we can do to mitigate this slowdown, apart from just not
> using git's tag feature at all? Are there any details I've overlooked or
> misunderstood?

An optimization for "fetching between repositories with insane number of refs" problem was merged after v1.5.0, so if you try the current tip of 'master' (which happens to be at v1.5.1-rc2 right now), I suspect you may see significant improvements.