Re: [RFC] Speeding up a null fetch

From: Julian Phillips <julian@quantumfyre.co.uk>
Date: Feb 12, 2007, 00:14 UTC
Message-ID: <Pine.LNX.4.64.0702120005530.5695@beast.quantumfyre.co.uk>
In-Reply-To: <Pine.LNX.4.63.0702120045320.22628@wbgn013.biozentrum.uni-wuerzburg.de>

On Mon, 12 Feb 2007, Johannes Schindelin wrote:

Show 13 quoted lines

> Hi,
>
> On Sun, 11 Feb 2007, Julian Phillips wrote:
>
>> An artifical test repository that has similar features (~25000 commits,
>> ~8000 tags, ~900 branches and a 2.5Gb packfile) when running locally
>> takes ~20m to clone and ~48m to fetch (with no new commits in the
>> original repository - i.e. the fetch does not update anything) with a
>> current code base (i.e. newer than 1.5.0-rc4).
>
> Ouch.
>
> I hope you packed the refs?

Unfortunately packing only makes things slower ... as it then becomes impossible to directly access a particular ref directly, which some of the calls to show-ref do.

Show 7 quoted lines

>
> BTW your patch
> - was not minimal (and therefore it takes longer than necessary to find
>  what you actually fixed),
> - it does not show where and how the call to show-ref is avoided (I
>  eventually understand that you avoid calling update_local_ref early, but
>  you sure could have made that easier), and

Ah yes, sorry. I seem to have managed to forget to include the paragraph explaining what I had done ... :$

(That'll teach me to trying doing too many things at once.)

> - it uses Pythong.
>
> Also, it touches a quite core part of git, which will hopefully be
> replaced by a builtin _after_ 1.5.0.

Indeed, I would never propose what I have done so far as a fix. I am definitely still in the investigation phase.

Show 7 quoted lines

>
>> However, this seems more band-aid than fix, and I wondered if someone
>> more familiar with the git internals could point me in the right
>> direction for a better fix, e.g. should I look at rewriting fetch in C?
>
> Look into the "pu" branch of git. There are the beginnings of a builtin
> (written in C) fetch.

Ah - this I didn't know. I shall have to have a play with that, I did notice that there is internal caching of the ref list that might magically solve the problem if fetch was a builtin (but I have a feeling that it won't be that simple).

>
> But this _will_ have to wait until after 1.5.0.

I hope so. 1.5 is looking very nice, and I really don't think that many people have such a stuipdly large repository ...

>
> Ciao,
> Dscho
>

-- 
Julian

  ---
You are in a maze of little twisting passages, all alike.