# Re: git-fetching from a big repository is slow 14 messages from 2006-12-14 to 2006-12-15. Participants: Andreas Ericsson, Junio C Hamano, Geert Bosch, Andy Parkins, Johannes Schindelin, Nicolas Pitre, Han-Wen Nienhuys. Thread: https://gitlist.dev/t/43072 ## Andy Parkins, 2006-12-14 13:40 Subject: git-fetching from a big repository is slow Message-ID: <200612141340.43925.andyparkins@gmail.com> URL: https://gitlist.dev/e/200612141340.43925.andyparkins%40gmail.com ``` Hello, I've got a big repository. I've got two computers. One has the repository up-to-date (164M after repack); one is behind (30M ish). I used git-fetch to try and update; and the sync took HOURS. I zipped the .git directory and transferred that and it took about 15 minutes to transfer. Am I doing something wrong? The git-fetch was done with a git+ssh:// URL. The zip transfer with scp (so ssh shouldn't be a factor). Andy -- Dr Andy Parkins, M Eng (hons), MIEE ``` ## Andreas Ericsson, 2006-12-14 13:53 Subject: Re: git-fetching from a big repository is slow Message-ID: <4581573E.40803@op5.se> URL: https://gitlist.dev/e/4581573E.40803%40op5.se In-Reply-To: <200612141340.43925.andyparkins@gmail.com> ``` Andy Parkins wrote: > Hello, > > I've got a big repository. I've got two computers. One has the repository > up-to-date (164M after repack); one is behind (30M ish). > > I used git-fetch to try and update; and the sync took HOURS. I zipped > the .git directory and transferred that and it took about 15 minutes to > transfer. > > Am I doing something wrong? The git-fetch was done with a git+ssh:// URL. > The zip transfer with scp (so ssh shouldn't be a factor). > This seems to happen if your repository consists of many large binary files, especially many large binary files of several versions that do not deltify well against each other. Perhaps it's worth adding gzip compression detecion to git? I imagine more people than me are tracking gzipped/bzip2'ed content that pretty much never deltifies well against anything else. -- Andreas Ericsson andreas.ericsson@op5.se OP5 AB www.op5.se ``` ## Johannes Schindelin, 2006-12-14 14:14 Subject: Re: git-fetching from a big repository is slow Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.63.0612141513130.3635%40wbgn013.biozentrum.uni-wuerzburg.de In-Reply-To: <4581573E.40803@op5.se> ``` Hi, On Thu, 14 Dec 2006, Andreas Ericsson wrote: > Andy Parkins wrote: > > Hello, > > > > I've got a big repository. I've got two computers. One has the repository > > up-to-date (164M after repack); one is behind (30M ish). > > > > I used git-fetch to try and update; and the sync took HOURS. I zipped the > > .git directory and transferred that and it took about 15 minutes to > > transfer. > > > > Am I doing something wrong? The git-fetch was done with a git+ssh:// URL. > > The zip transfer with scp (so ssh shouldn't be a factor). > > > > This seems to happen if your repository consists of many large binary files, > especially many large binary files of several versions that do not deltify > well against each other. Perhaps it's worth adding gzip compression detecion > to git? I imagine more people than me are tracking gzipped/bzip2'ed content > that pretty much never deltifies well against anything else. Or we add something like the heuristics we discovered in another thread, where rename detection (which is related to delta candidate searching) is not started if the sizes differ drastically. Ciao, Dscho ``` ## Andreas Ericsson, 2006-12-14 15:06 Subject: Re: git-fetching from a big repository is slow Message-ID: <4581685D.1070407@op5.se> URL: https://gitlist.dev/e/4581685D.1070407%40op5.se In-Reply-To: ``` Johannes Schindelin wrote: > Hi, > > On Thu, 14 Dec 2006, Andreas Ericsson wrote: > >> Andy Parkins wrote: >>> Hello, >>> >>> I've got a big repository. I've got two computers. One has the repository >>> up-to-date (164M after repack); one is behind (30M ish). >>> >>> I used git-fetch to try and update; and the sync took HOURS. I zipped the >>> .git directory and transferred that and it took about 15 minutes to >>> transfer. >>> >>> Am I doing something wrong? The git-fetch was done with a git+ssh:// URL. >>> The zip transfer with scp (so ssh shouldn't be a factor). >>> >> This seems to happen if your repository consists of many large binary files, >> especially many large binary files of several versions that do not deltify >> well against each other. Perhaps it's worth adding gzip compression detecion >> to git? I imagine more people than me are tracking gzipped/bzip2'ed content >> that pretty much never deltifies well against anything else. > > Or we add something like the heuristics we discovered in another thread, > where rename detection (which is related to delta candidate searching) is > not started if the sizes differ drastically. > It wouldn't work for this particular case though. In our distribution repository we have ~300 bzip2 compressed tarballs with an average size of 3MiB. 240 of those are between 2.5 and 4 MiB, so they don't drastically differ, but neither do they delta well. One option would be to add some sort of config option to skip attempting deltas of files with a certain suffix. That way we could just tell it to ignore *.gz,*.tgz,*.bz2 and everything would work just as it does today, but a lot faster. -- Andreas Ericsson andreas.ericsson@op5.se OP5 AB www.op5.se ``` ## Andy Parkins, 2006-12-14 15:18 Subject: Re: git-fetching from a big repository is slow Message-ID: <200612141518.05770.andyparkins@gmail.com> URL: https://gitlist.dev/e/200612141518.05770.andyparkins%40gmail.com In-Reply-To: <4581573E.40803@op5.se> ``` On Thursday 2006 December 14 13:53, Andreas Ericsson wrote: > This seems to happen if your repository consists of many large binary > files, especially many large binary files of several versions that do > not deltify well against each other. Perhaps it's worth adding gzip It's actually just every released patch to the linux kernel ever issued. Almost entirely ASCII and every revision (save the first) created by patching the previous. Andy -- Dr Andy Parkins, M Eng (hons), MIEE ``` ## Han-Wen Nienhuys, 2006-12-14 15:45 Subject: Re: git-fetching from a big repository is slow Message-ID: <458171B7.1020702@xs4all.nl> URL: https://gitlist.dev/e/458171B7.1020702%40xs4all.nl In-Reply-To: <200612141518.05770.andyparkins@gmail.com> ``` Andy Parkins escreveu: > On Thursday 2006 December 14 13:53, Andreas Ericsson wrote: > >> This seems to happen if your repository consists of many large binary >> files, especially many large binary files of several versions that do >> not deltify well against each other. Perhaps it's worth adding gzip > > It's actually just every released patch to the linux kernel ever issued. > Almost entirely ASCII and every revision (save the first) created by patching > the previous. I just noticed that git-fetch now runs git-show-ref --verify on every tag it encounters. This seems to slow down fetch over here. -- Han-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen ``` ## Andy Parkins, 2006-12-14 16:20 Subject: Re: git-fetching from a big repository is slow Message-ID: <200612141620.17782.andyparkins@gmail.com> URL: https://gitlist.dev/e/200612141620.17782.andyparkins%40gmail.com In-Reply-To: <458171B7.1020702@xs4all.nl> ``` On Thursday 2006 December 14 15:45, Han-Wen Nienhuys wrote: > I just noticed that git-fetch now runs git-show-ref --verify on every > tag it encounters. This seems to slow down fetch over here. There aren't any tags in this repository :-) Andy -- Dr Andy Parkins, M Eng (hons), MIEE ``` ## Johannes Schindelin, 2006-12-14 16:34 Subject: Re: git-fetching from a big repository is slow Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.63.0612141732410.3635%40wbgn013.biozentrum.uni-wuerzburg.de In-Reply-To: <200612141620.17782.andyparkins@gmail.com> ``` Hi, On Thu, 14 Dec 2006, Andy Parkins wrote: > On Thursday 2006 December 14 15:45, Han-Wen Nienhuys wrote: > > > I just noticed that git-fetch now runs git-show-ref --verify on every > > tag it encounters. This seems to slow down fetch over here. > > There aren't any tags in this repository :-) git-show-ref traverses every single _local_ tag when called. This is to overcome the problem that tags can be packed now, so a simple file existence check is not sufficient. It would be much faster, probably, if you pack the local refs. IIRC I once argued for automatically packing refs (and all refs), but this has not been picked up, and I do not really care about it either. Ciao, Dscho ``` ## Nicolas Pitre, 2006-12-14 18:14 Subject: Re: git-fetching from a big repository is slow Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0612141311310.18171%40xanadu.home In-Reply-To: <4581573E.40803@op5.se> ``` On Thu, 14 Dec 2006, Andreas Ericsson wrote: > Andy Parkins wrote: > > Hello, > > > > I've got a big repository. I've got two computers. One has the repository > > up-to-date (164M after repack); one is behind (30M ish). > > > > I used git-fetch to try and update; and the sync took HOURS. I zipped the > > .git directory and transferred that and it took about 15 minutes to > > transfer. > > > > Am I doing something wrong? The git-fetch was done with a git+ssh:// URL. > > The zip transfer with scp (so ssh shouldn't be a factor). > > > > This seems to happen if your repository consists of many large binary files, > especially many large binary files of several versions that do not deltify > well against each other. Perhaps it's worth adding gzip compression detecion > to git? I imagine more people than me are tracking gzipped/bzip2'ed content > that pretty much never deltifies well against anything else. If your remote repository is fully packed in a single pack that should not have any impact on the transfer latency since no attempt to redeltify objects against each other is attempted by default when those objects are in the same pack. ``` ## Geert Bosch, 2006-12-14 19:05 Subject: Re: git-fetching from a big repository is slow Message-ID: URL: https://gitlist.dev/e/C287764F-6755-4291-A87A-3E8816E90B49%40adacore.com In-Reply-To: <4581685D.1070407@op5.se> ``` On Dec 14, 2006, at 10:06, Andreas Ericsson wrote: > It wouldn't work for this particular case though. In our > distribution repository we have ~300 bzip2 compressed tarballs with > an average size of 3MiB. 240 of those are between 2.5 and 4 MiB, so > they don't drastically differ, but neither do they delta well. > > One option would be to add some sort of config option to skip > attempting deltas of files with a certain suffix. That way we could > just tell it to ignore *.gz,*.tgz,*.bz2 and everything would work > just as it does today, but a lot faster. Such special magic based on filenames is always a bad idea. Tomorrow somebody comes with .zip files (oh, and of course .ZIP), then it's .jpg's other compressed content. In the end git will be doing lots of magic and still perform badly on unknown compressed content. There is a very simple way of detecting compressed files: just look at the size of the compressed blob and compare against the size of the expanded blob. If the compressed blob has a non-trivial size which is close to the expanded size, assume the file is not interesting as source or target for deltas. Example: if (compressed_size > expanded_size / 4 * 3 + 1024) { /* don't try to deltify if blob doesn't compress well */ return ...; } ``` ## Junio C Hamano, 2006-12-14 20:41 Subject: Re: git-fetching from a big repository is slow Message-ID: <7v8xha2osd.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7v8xha2osd.fsf%40assigned-by-dhcp.cox.net In-Reply-To: ``` Johannes Schindelin writes: > git-show-ref traverses every single _local_ tag when called. This is to > overcome the problem that tags can be packed now, so a simple file > existence check is not sufficient. Is "traverses every single _local_ tag" a fact? It might go through every single _local_ (possibly stale) packed tag in memory but it should not traverse $GIT_DIR/refs/tags. If I recall correctly, show-ref (1) first checks the filesystem "$GIT_DIR/$named_ref" and says Ok if found and valid; otherwise (2) checks packed refs (reads $GIT_DIR/packed-refs if not already). So that would be at most one open (which may fail in (1)) and one open+read (in (2)). Unless we are talking about fork+exec overhead, that "traverse" should be reasonably fast. Where is the bottleneck? ``` ## Andreas Ericsson, 2006-12-14 22:28 Subject: Re: git-fetching from a big repository is slow Message-ID: <4581D01E.9020806@op5.se> URL: https://gitlist.dev/e/4581D01E.9020806%40op5.se In-Reply-To: ``` Geert Bosch wrote: > > On Dec 14, 2006, at 10:06, Andreas Ericsson wrote: > >> It wouldn't work for this particular case though. In our distribution >> repository we have ~300 bzip2 compressed tarballs with an average size >> of 3MiB. 240 of those are between 2.5 and 4 MiB, so they don't >> drastically differ, but neither do they delta well. >> >> One option would be to add some sort of config option to skip >> attempting deltas of files with a certain suffix. That way we could >> just tell it to ignore *.gz,*.tgz,*.bz2 and everything would work just >> as it does today, but a lot faster. > > Such special magic based on filenames is always a bad idea. Tomorrow > somebody > comes with .zip files (oh, and of course .ZIP), then it's .jpg's other > compressed content. In the end git will be doing lots of magic and still > perform > badly on unknown compressed content. > Hence config option. People can tell git to skip trying to delta whatever they want. For this particular mothership repo, we only ever work against it when we're at the office, meaning resulting datasize is not an issue, but data computation can be a real bottle-neck. > There is a very simple way of detecting compressed files: just look at the > size of the compressed blob and compare against the size of the expanded > blob. > If the compressed blob has a non-trivial size which is close to the > expanded > size, assume the file is not interesting as source or target for deltas. > > Example: > if (compressed_size > expanded_size / 4 * 3 + 1024) { > /* don't try to deltify if blob doesn't compress well */ > return ...; > } > Many compression algorithms generate similar output for similar input. Most source-code projects change relatively little between releases, so they *could* delta well, it's just that in our repo they don't. -- Andreas Ericsson andreas.ericsson@op5.se OP5 AB www.op5.se ``` ## Johannes Schindelin, 2006-12-14 23:26 Subject: Re: git-fetching from a big repository is slow Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.63.0612150015330.3635%40wbgn013.biozentrum.uni-wuerzburg.de In-Reply-To: <7v8xha2osd.fsf@assigned-by-dhcp.cox.net> ``` Hi, On Thu, 14 Dec 2006, Junio C Hamano wrote: > Johannes Schindelin writes: > > > git-show-ref traverses every single _local_ tag when called. This is to > > overcome the problem that tags can be packed now, so a simple file > > existence check is not sufficient. > > Is "traverses every single _local_ tag" a fact? It might go > through every single _local_ (possibly stale) packed tag in > memory but it should not traverse $GIT_DIR/refs/tags. > > If I recall correctly, show-ref (1) first checks the filesystem > "$GIT_DIR/$named_ref" and says Ok if found and valid; otherwise > (2) checks packed refs (reads $GIT_DIR/packed-refs if not > already). If I read builtin-show-ref.c correctly, it _always_ calls for_each_ref(show_ref, NULL); The only reason that the loop in for_each_ref can stop early is if show_ref returns something different than 0. But it does not! Every single return in show_ref() returns 0. It does not matter, though (see below). > So that would be at most one open (which may fail in (1)) and one > open+read (in (2)). Unless we are talking about fork+exec overhead, > that "traverse" should be reasonably fast. > > Where is the bottleneck? The problem is that so many stat()s _do_ take time. Again, if I read the code correctly, it not only stat()s every loose ref, but also resolves the refs in get_ref_dir(), which is called from get_loose_refs(), which is unconditionally called in for_each_ref(). Even if the refs are packed, it takes quite _long_ (I confirmed this). And it is not at all necessary! Instead of a O(n^2) we can easily reduce this to O(n*log(n)), and we can reduce the n fork()&exec()s of git-show-ref by a single one. Ciao, Dscho ``` ## Junio C Hamano, 2006-12-15 00:38 Subject: Re: git-fetching from a big repository is slow Message-ID: <7vslfixaat.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7vslfixaat.fsf%40assigned-by-dhcp.cox.net In-Reply-To: ``` Johannes Schindelin writes: > If I read builtin-show-ref.c correctly, it _always_ calls > for_each_ref(show_ref, NULL); Ok, that settles it. If there is a reason to have --verify, we should really special case it. There is no point in looping, because verify does not do the tail match (which could cause ambiguity) and its answer should be either "yes it is there" or "no there is no such ref". ```