threads / discuss / 43072

Re: git-fetching from a big repository is slow

Subject: Re: git-fetching from a big repository is slow

## tl;dr

14 messages between Dec 14, 2006 and Dec 15, 2006.

replies: 13people: 7as markdown or json

Andy Parkins· Dec 14, 2006, 13:40 UTC · lore

git-fetching from a big repository is slow

Hello,

I've got a big repository. I've got two computers. One has the repository up-to-date (164M after repack); one is behind (30M ish).

I used git-fetch to try and update; and the sync took HOURS. I zipped the .git directory and transferred that and it took about 15 minutes to transfer.

Am I doing something wrong? The git-fetch was done with a git+ssh:// URL. The zip transfer with scp (so ssh shouldn't be a factor).

Andy
-- 
Dr Andy Parkins, M Eng (hons), MIEE
Andreas Ericsson· Dec 14, 2006, 13:53 UTC · re: Andy Parkins · lore
Andy Parkins wrote:
Show 12 quoted lines
> Hello,
> 
> I've got a big repository.  I've got two computers.  One has the repository 
> up-to-date (164M after repack); one is behind (30M ish).
> 
> I used git-fetch to try and update; and the sync took HOURS.  I zipped 
> the .git directory and transferred that and it took about 15 minutes to 
> transfer.
> 
> Am I doing something wrong?  The git-fetch was done with a git+ssh:// URL.  
> The zip transfer with scp (so ssh shouldn't be a factor).
> 

This seems to happen if your repository consists of many large binary files, especially many large binary files of several versions that do not deltify well against each other. Perhaps it's worth adding gzip compression detecion to git? I imagine more people than me are tracking gzipped/bzip2'ed content that pretty much never deltifies well against anything else.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Johannes Schindelin· Dec 14, 2006, 14:14 UTC · re: Andreas Ericsson · lore
Hi,
On Thu, 14 Dec 2006, Andreas Ericsson wrote:
Show 19 quoted lines
> Andy Parkins wrote:
> > Hello,
> > 
> > I've got a big repository.  I've got two computers.  One has the repository
> > up-to-date (164M after repack); one is behind (30M ish).
> > 
> > I used git-fetch to try and update; and the sync took HOURS.  I zipped the
> > .git directory and transferred that and it took about 15 minutes to
> > transfer.
> > 
> > Am I doing something wrong?  The git-fetch was done with a git+ssh:// URL.
> > The zip transfer with scp (so ssh shouldn't be a factor).
> > 
> 
> This seems to happen if your repository consists of many large binary files,
> especially many large binary files of several versions that do not deltify
> well against each other. Perhaps it's worth adding gzip compression detecion
> to git? I imagine more people than me are tracking gzipped/bzip2'ed content
> that pretty much never deltifies well against anything else.

Or we add something like the heuristics we discovered in another thread, where rename detection (which is related to delta candidate searching) is not started if the sizes differ drastically.

Ciao, Dscho

Andreas Ericsson· Dec 14, 2006, 15:06 UTC · re: Johannes Schindelin · lore
Johannes Schindelin wrote:
Show 27 quoted lines
> Hi,
> 
> On Thu, 14 Dec 2006, Andreas Ericsson wrote:
> 
>> Andy Parkins wrote:
>>> Hello,
>>>
>>> I've got a big repository.  I've got two computers.  One has the repository
>>> up-to-date (164M after repack); one is behind (30M ish).
>>>
>>> I used git-fetch to try and update; and the sync took HOURS.  I zipped the
>>> .git directory and transferred that and it took about 15 minutes to
>>> transfer.
>>>
>>> Am I doing something wrong?  The git-fetch was done with a git+ssh:// URL.
>>> The zip transfer with scp (so ssh shouldn't be a factor).
>>>
>> This seems to happen if your repository consists of many large binary files,
>> especially many large binary files of several versions that do not deltify
>> well against each other. Perhaps it's worth adding gzip compression detecion
>> to git? I imagine more people than me are tracking gzipped/bzip2'ed content
>> that pretty much never deltifies well against anything else.
> 
> Or we add something like the heuristics we discovered in another thread, 
> where rename detection (which is related to delta candidate searching) is 
> not started if the sizes differ drastically.
> 

It wouldn't work for this particular case though. In our distribution repository we have ~300 bzip2 compressed tarballs with an average size of 3MiB. 240 of those are between 2.5 and 4 MiB, so they don't drastically differ, but neither do they delta well.

One option would be to add some sort of config option to skip attempting deltas of files with a certain suffix. That way we could just tell it to ignore *.gz,*.tgz,*.bz2 and everything would work just as it does today, but a lot faster.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Geert Bosch· Dec 14, 2006, 19:05 UTC · re: Andreas Ericsson · lore
On Dec 14, 2006, at 10:06, Andreas Ericsson wrote:
Show 9 quoted lines
> It wouldn't work for this particular case though. In our  
> distribution repository we have ~300 bzip2 compressed tarballs with  
> an average size of 3MiB. 240 of those are between 2.5 and 4 MiB, so  
> they don't drastically differ, but neither do they delta well.
>
> One option would be to add some sort of config option to skip  
> attempting deltas of files with a certain suffix. That way we could  
> just tell it to ignore *.gz,*.tgz,*.bz2 and everything would work  
> just as it does today, but a lot faster.

Such special magic based on filenames is always a bad idea. Tomorrow somebody comes with .zip files (oh, and of course .ZIP), then it's .jpg's other compressed content. In the end git will be doing lots of magic and still perform badly on unknown compressed content.

There is a very simple way of detecting compressed files: just look at the size of the compressed blob and compare against the size of the expanded blob. If the compressed blob has a non-trivial size which is close to the expanded size, assume the file is not interesting as source or target for deltas.

Example:
    if (compressed_size > expanded_size / 4 * 3 + 1024) {
      /* don't try to deltify if blob doesn't compress well */
      return ...;
    }
Andreas Ericsson· Dec 14, 2006, 22:28 UTC · re: Geert Bosch · lore
Geert Bosch wrote:
Show 20 quoted lines
> 
> On Dec 14, 2006, at 10:06, Andreas Ericsson wrote:
> 
>> It wouldn't work for this particular case though. In our distribution 
>> repository we have ~300 bzip2 compressed tarballs with an average size 
>> of 3MiB. 240 of those are between 2.5 and 4 MiB, so they don't 
>> drastically differ, but neither do they delta well.
>>
>> One option would be to add some sort of config option to skip 
>> attempting deltas of files with a certain suffix. That way we could 
>> just tell it to ignore *.gz,*.tgz,*.bz2 and everything would work just 
>> as it does today, but a lot faster.
> 
> Such special magic based on filenames is always a bad idea. Tomorrow 
> somebody
> comes with .zip files (oh, and of course .ZIP), then it's .jpg's other
> compressed content. In the end git will be doing lots of magic and still 
> perform
> badly on unknown compressed content.
> 

Hence config option. People can tell git to skip trying to delta whatever they want. For this particular mothership repo, we only ever work against it when we're at the office, meaning resulting datasize is not an issue, but data computation can be a real bottle-neck.

Show 13 quoted lines
> There is a very simple way of detecting compressed files: just look at the
> size of the compressed blob and compare against the size of the expanded 
> blob.
> If the compressed blob has a non-trivial size which is close to the 
> expanded
> size, assume the file is not interesting as source or target for deltas.
> 
> Example:
>    if (compressed_size > expanded_size / 4 * 3 + 1024) {
>      /* don't try to deltify if blob doesn't compress well */
>      return ...;
>    }
> 

Many compression algorithms generate similar output for similar input. Most source-code projects change relatively little between releases, so they *could* delta well, it's just that in our repo they don't.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Andy Parkins· Dec 14, 2006, 15:18 UTC · re: Andreas Ericsson · lore
On Thursday 2006 December 14 13:53, Andreas Ericsson wrote:
> This seems to happen if your repository consists of many large binary
> files, especially many large binary files of several versions that do
> not deltify well against each other. Perhaps it's worth adding gzip

It's actually just every released patch to the linux kernel ever issued. Almost entirely ASCII and every revision (save the first) created by patching the previous.

Andy
-- 
Dr Andy Parkins, M Eng (hons), MIEE
Han-Wen Nienhuys· Dec 14, 2006, 15:45 UTC · re: Andy Parkins · lore
Andy Parkins escreveu:
Show 9 quoted lines
> On Thursday 2006 December 14 13:53, Andreas Ericsson wrote:
> 
>> This seems to happen if your repository consists of many large binary
>> files, especially many large binary files of several versions that do
>> not deltify well against each other. Perhaps it's worth adding gzip
> 
> It's actually just every released patch to the linux kernel ever issued.  
> Almost entirely ASCII and every revision (save the first) created by patching 
> the previous.

I just noticed that git-fetch now runs git-show-ref --verify on every tag it encounters. This seems to slow down fetch over here.

-- 
 Han-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen
Andy Parkins· Dec 14, 2006, 16:20 UTC · re: Han-Wen Nienhuys · lore
On Thursday 2006 December 14 15:45, Han-Wen Nienhuys wrote:
> I just noticed that git-fetch now runs git-show-ref --verify on every
> tag it encounters. This seems to slow down fetch over here.
There aren't any tags in this repository :-)
Andy
-- 
Dr Andy Parkins, M Eng (hons), MIEE
Johannes Schindelin· Dec 14, 2006, 16:34 UTC · re: Andy Parkins · lore
Hi,
On Thu, 14 Dec 2006, Andy Parkins wrote:
Show 6 quoted lines
> On Thursday 2006 December 14 15:45, Han-Wen Nienhuys wrote:
> 
> > I just noticed that git-fetch now runs git-show-ref --verify on every
> > tag it encounters. This seems to slow down fetch over here.
> 
> There aren't any tags in this repository :-)

git-show-ref traverses every single _local_ tag when called. This is to overcome the problem that tags can be packed now, so a simple file existence check is not sufficient.

It would be much faster, probably, if you pack the local refs. IIRC I once argued for automatically packing refs (and all refs), but this has not been picked up, and I do not really care about it either.

Ciao, Dscho

Junio C Hamano· Dec 14, 2006, 20:41 UTC · re: Johannes Schindelin · lore
Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> git-show-ref traverses every single _local_ tag when called. This is to 
> overcome the problem that tags can be packed now, so a simple file 
> existence check is not sufficient.

Is "traverses every single _local_ tag" a fact? It might go through every single _local_ (possibly stale) packed tag in memory but it should not traverse $GIT_DIR/refs/tags.

If I recall correctly, show-ref (1) first checks the filesystem "$GIT_DIR/$named_ref" and says Ok if found and valid; otherwise (2) checks packed refs (reads $GIT_DIR/packed-refs if not already). So that would be at most one open (which may fail in (1)) and one open+read (in (2)). Unless we are talking about fork+exec overhead, that "traverse" should be reasonably fast.

Where is the bottleneck?
Johannes Schindelin· Dec 14, 2006, 23:26 UTC · re: Junio C Hamano · lore
Hi,
On Thu, 14 Dec 2006, Junio C Hamano wrote:
Show 14 quoted lines
> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> 
> > git-show-ref traverses every single _local_ tag when called. This is to 
> > overcome the problem that tags can be packed now, so a simple file 
> > existence check is not sufficient.
> 
> Is "traverses every single _local_ tag" a fact?  It might go
> through every single _local_ (possibly stale) packed tag in
> memory but it should not traverse $GIT_DIR/refs/tags.
> 
> If I recall correctly, show-ref (1) first checks the filesystem
> "$GIT_DIR/$named_ref" and says Ok if found and valid; otherwise
> (2) checks packed refs (reads $GIT_DIR/packed-refs if not
> already).

If I read builtin-show-ref.c correctly, it _always_ calls for_each_ref(show_ref, NULL);

The only reason that the loop in for_each_ref can stop early is if show_ref returns something different than 0. But it does not! Every single return in show_ref() returns 0. It does not matter, though (see below).

Show 5 quoted lines
> So that would be at most one open (which may fail in (1)) and one 
> open+read (in (2)).  Unless we are talking about fork+exec overhead, 
> that "traverse" should be reasonably fast.
> 
> Where is the bottleneck?

The problem is that so many stat()s _do_ take time. Again, if I read the code correctly, it not only stat()s every loose ref, but also resolves the refs in get_ref_dir(), which is called from get_loose_refs(), which is unconditionally called in for_each_ref().

Even if the refs are packed, it takes quite _long_ (I confirmed this). And it is not at all necessary! Instead of a O(n^2) we can easily reduce this to O(n*log(n)), and we can reduce the n fork()&exec()s of git-show-ref by a single one.

Ciao, Dscho

Junio C Hamano· Dec 15, 2006, 00:38 UTC · re: Johannes Schindelin · lore
Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> If I read builtin-show-ref.c correctly, it _always_ calls 
> for_each_ref(show_ref, NULL);

Ok, that settles it. If there is a reason to have --verify, we should really special case it. There is no point in looping, because verify does not do the tail match (which could cause ambiguity) and its answer should be either "yes it is there" or "no there is no such ref".

Nicolas Pitre· Dec 14, 2006, 18:14 UTC · re: Andreas Ericsson · lore
On Thu, 14 Dec 2006, Andreas Ericsson wrote:
Show 19 quoted lines
> Andy Parkins wrote:
> > Hello,
> > 
> > I've got a big repository.  I've got two computers.  One has the repository
> > up-to-date (164M after repack); one is behind (30M ish).
> > 
> > I used git-fetch to try and update; and the sync took HOURS.  I zipped the
> > .git directory and transferred that and it took about 15 minutes to
> > transfer.
> > 
> > Am I doing something wrong?  The git-fetch was done with a git+ssh:// URL.
> > The zip transfer with scp (so ssh shouldn't be a factor).
> > 
> 
> This seems to happen if your repository consists of many large binary files,
> especially many large binary files of several versions that do not deltify
> well against each other. Perhaps it's worth adding gzip compression detecion
> to git? I imagine more people than me are tracking gzipped/bzip2'ed content
> that pretty much never deltifies well against anything else.

If your remote repository is fully packed in a single pack that should not have any impact on the transfer latency since no attempt to redeltify objects against each other is attempted by default when those objects are in the same pack.

← back to recent threads