threads / discuss / 43175

Re: git fetch slow as molasses due to tag downloading

Subject: Re: git fetch slow as molasses due to tag downloading

## tl;dr

5 messages between Dec 14, 2006 and Dec 15, 2006.

replies: 4people: 4as markdown or json

Han-Wen Nienhuys· Dec 14, 2006, 15:40 UTC · lore

git fetch slow as molasses due to tag downloading

Hello,

just upgraded our autobuilder from 1.4.3 to 1.4.4.2.

Now, our standard download command comes to a complete halt. Judging from the "ps -ef" apparently, it does

   git-show-ref --verify --quiet -- [TAG]

This is done for every one of the 1500 tags that are in my repository. At approx 20 tags per second this takes an awful lot of time.

1. Is this necessary?

2. Is this efficient?  Wouldn't doing all tags in a single git-show-ref
invocation be potentially quicker?

-- 
 Han-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen

Johannes Schindelin· Dec 14, 2006, 16:05 UTC · re: Han-Wen Nienhuys · lore

Hi,

On Thu, 14 Dec 2006, Han-Wen Nienhuys wrote:

Show 11 quoted lines

> just upgraded our autobuilder from 1.4.3 to 1.4.4.2.
> 
> Now, our standard download command comes to a complete halt.  Judging
> >from the "ps -ef" apparently, it does 
> 
>    git-show-ref --verify --quiet -- [TAG]
> 
> This is done for every one of the 1500 tags that are in my repository. 
> At approx 20 tags per second this takes an awful lot of time. 
> 
> 1. Is this necessary?

Yes. The purpose is to check which tags have not yet been fetched.

> 2. Is this efficient?  Wouldn't doing all tags in a single git-show-ref 
> invocation be potentially quicker?

It is not efficient. But it cannot be solved like you propose, since it is inside a loop, and a "continue" is executed when the tag exists already.

IMHO this should be solved as a filter: "git-show-ref --stdin --show-invalid". Thus, git does not have to traverse _every_ ref for _every_ incoming tag.

Ciao, Dscho

Jakub Narebski· Dec 15, 2006, 09:58 UTC · re: Johannes Schindelin · lore

Johannes Schindelin wrote:

> IMHO this should be solved as a filter: "git-show-ref --stdin 
> --show-invalid". Thus, git does not have to traverse _every_ ref for 
> _every_ incoming tag.

Can you make it so? TIA.

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Junio C Hamano· Dec 15, 2006, 21:55 UTC · re: Johannes Schindelin · lore

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> IMHO this should be solved as a filter: "git-show-ref --stdin 
> --show-invalid". Thus, git does not have to traverse _every_ ref for 
> _every_ incoming tag.

That sounds like a too specialized hack to me. We should first speed up the general "--verify $ref" case; as you corrected me earlier it still has a useless loop.

If it is still too slow (which I suspect it could be the case, with fork+exec overhead), we should _also_ pursue the filter approach, but even then I think "filter out the valid ones" is a specialized hack, if you mean "show only the invalid one's names" by --show-invalid.

It would make sense to do

        $ git show-ref --show-invalid v1.0.0 v2.6.18
        f665776185ad074b236c00751d666da7d1977dbe refs/tags/v1.0.0
        - refs/tags/v2.6.18

(and its equivalent to take refs from --stdin) though.

Jakub Narebski· Dec 15, 2006, 22:27 UTC · re: Junio C Hamano · lore

Junio C Hamano wrote:

Show 23 quoted lines

> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> 
>> IMHO this should be solved as a filter: "git-show-ref --stdin 
>> --show-invalid". Thus, git does not have to traverse _every_ ref for 
>> _every_ incoming tag.
> 
> That sounds like a too specialized hack to me.  We should first
> speed up the general "--verify $ref" case; as you corrected me
> earlier it still has a useless loop.
> 
> If it is still too slow (which I suspect it could be the case,
> with fork+exec overhead), we should _also_ pursue the filter
> approach, but even then I think "filter out the valid ones" is a
> specialized hack, if you mean "show only the invalid one's
> names" by --show-invalid.
> 
> It would make sense to do
> 
>         $ git show-ref --show-invalid v1.0.0 v2.6.18
>         f665776185ad074b236c00751d666da7d1977dbe refs/tags/v1.0.0
>         - refs/tags/v2.6.18
> 
> (and its equivalent to take refs from --stdin) though.

Nice idea. And having '-' is probably better than using 00000000000000000000000000000000000000000 as for non-existing objects.

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git