{"thread":{"id":"43072","subject":"Re: git-fetching from a big repository is slow","startedAt":"2006-12-14T13:40:36Z","lastAt":"2006-12-15T00:38:50Z","messageCount":14,"participants":["Andreas Ericsson","Junio C Hamano","Geert Bosch","Andy Parkins","Johannes Schindelin","Nicolas Pitre","Han-Wen Nienhuys"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"296468","messageId":"200612141340.43925.andyparkins@gmail.com","threadId":"43072","inReplyTo":null,"subject":"git-fetching from a big repository is slow","fromName":"Andy Parkins","fromEmail":"andyparkins@gmail.com","sentAt":"2006-12-14T13:40:36Z","receivedAt":"2006-12-14T13:40:36Z","isPatch":false,"sender":{"key":"andyparkins@gmail.com","avatar":null},"body":"Hello,\n\nI've got a big repository.  I've got two computers.  One has the repository \nup-to-date (164M after repack); one is behind (30M ish).\n\nI used git-fetch to try and update; and the sync took HOURS.  I zipped \nthe .git directory and transferred that and it took about 15 minutes to \ntransfer.\n\nAm I doing something wrong?  The git-fetch was done with a git+ssh:// URL.  \nThe zip transfer with scp (so ssh shouldn't be a factor).\n\n\n\nAndy\n-- \nDr Andy Parkins, M Eng (hons), MIEE\n"},{"id":"296165","messageId":"4581573E.40803@op5.se","threadId":"43072","inReplyTo":"200612141340.43925.andyparkins@gmail.com","subject":"Re: git-fetching from a big repository is slow","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2006-12-14T13:53:02Z","receivedAt":"2006-12-14T13:53:02Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Andy Parkins wrote:\n> Hello,\n> \n> I've got a big repository.  I've got two computers.  One has the repository \n> up-to-date (164M after repack); one is behind (30M ish).\n> \n> I used git-fetch to try and update; and the sync took HOURS.  I zipped \n> the .git directory and transferred that and it took about 15 minutes to \n> transfer.\n> \n> Am I doing something wrong?  The git-fetch was done with a git+ssh:// URL.  \n> The zip transfer with scp (so ssh shouldn't be a factor).\n> \n\nThis seems to happen if your repository consists of many large binary \nfiles, especially many large binary files of several versions that do \nnot deltify well against each other. Perhaps it's worth adding gzip \ncompression detecion to git? I imagine more people than me are tracking \ngzipped/bzip2'ed content that pretty much never deltifies well against \nanything else.\n\n-- \nAndreas Ericsson                   andreas.ericsson@op5.se\nOP5 AB                             www.op5.se\n"},{"id":"295897","messageId":"Pine.LNX.4.63.0612141513130.3635@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"43072","inReplyTo":"4581573E.40803@op5.se","subject":"Re: git-fetching from a big repository is slow","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-14T14:14:29Z","receivedAt":"2006-12-14T14:14:29Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 14 Dec 2006, Andreas Ericsson wrote:\n\n> Andy Parkins wrote:\n> > Hello,\n> > \n> > I've got a big repository.  I've got two computers.  One has the repository\n> > up-to-date (164M after repack); one is behind (30M ish).\n> > \n> > I used git-fetch to try and update; and the sync took HOURS.  I zipped the\n> > .git directory and transferred that and it took about 15 minutes to\n> > transfer.\n> > \n> > Am I doing something wrong?  The git-fetch was done with a git+ssh:// URL.\n> > The zip transfer with scp (so ssh shouldn't be a factor).\n> > \n> \n> This seems to happen if your repository consists of many large binary files,\n> especially many large binary files of several versions that do not deltify\n> well against each other. Perhaps it's worth adding gzip compression detecion\n> to git? I imagine more people than me are tracking gzipped/bzip2'ed content\n> that pretty much never deltifies well against anything else.\n\nOr we add something like the heuristics we discovered in another thread, \nwhere rename detection (which is related to delta candidate searching) is \nnot started if the sizes differ drastically.\n\nCiao,\nDscho\n"},{"id":"293809","messageId":"4581685D.1070407@op5.se","threadId":"43072","inReplyTo":"Pine.LNX.4.63.0612141513130.3635@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: git-fetching from a big repository is slow","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2006-12-14T15:06:05Z","receivedAt":"2006-12-14T15:06:05Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Johannes Schindelin wrote:\n> Hi,\n> \n> On Thu, 14 Dec 2006, Andreas Ericsson wrote:\n> \n>> Andy Parkins wrote:\n>>> Hello,\n>>>\n>>> I've got a big repository.  I've got two computers.  One has the repository\n>>> up-to-date (164M after repack); one is behind (30M ish).\n>>>\n>>> I used git-fetch to try and update; and the sync took HOURS.  I zipped the\n>>> .git directory and transferred that and it took about 15 minutes to\n>>> transfer.\n>>>\n>>> Am I doing something wrong?  The git-fetch was done with a git+ssh:// URL.\n>>> The zip transfer with scp (so ssh shouldn't be a factor).\n>>>\n>> This seems to happen if your repository consists of many large binary files,\n>> especially many large binary files of several versions that do not deltify\n>> well against each other. Perhaps it's worth adding gzip compression detecion\n>> to git? I imagine more people than me are tracking gzipped/bzip2'ed content\n>> that pretty much never deltifies well against anything else.\n> \n> Or we add something like the heuristics we discovered in another thread, \n> where rename detection (which is related to delta candidate searching) is \n> not started if the sizes differ drastically.\n> \n\nIt wouldn't work for this particular case though. In our distribution \nrepository we have ~300 bzip2 compressed tarballs with an average size \nof 3MiB. 240 of those are between 2.5 and 4 MiB, so they don't \ndrastically differ, but neither do they delta well.\n\nOne option would be to add some sort of config option to skip attempting \ndeltas of files with a certain suffix. That way we could just tell it to \nignore *.gz,*.tgz,*.bz2 and everything would work just as it does today, \nbut a lot faster.\n\n-- \nAndreas Ericsson                   andreas.ericsson@op5.se\nOP5 AB                             www.op5.se\n"},{"id":"294260","messageId":"200612141518.05770.andyparkins@gmail.com","threadId":"43072","inReplyTo":"4581573E.40803@op5.se","subject":"Re: git-fetching from a big repository is slow","fromName":"Andy Parkins","fromEmail":"andyparkins@gmail.com","sentAt":"2006-12-14T15:18:04Z","receivedAt":"2006-12-14T15:18:04Z","isPatch":false,"sender":{"key":"andyparkins@gmail.com","avatar":null},"body":"On Thursday 2006 December 14 13:53, Andreas Ericsson wrote:\n\n> This seems to happen if your repository consists of many large binary\n> files, especially many large binary files of several versions that do\n> not deltify well against each other. Perhaps it's worth adding gzip\n\nIt's actually just every released patch to the linux kernel ever issued.  \nAlmost entirely ASCII and every revision (save the first) created by patching \nthe previous.\n\n\nAndy\n\n-- \nDr Andy Parkins, M Eng (hons), MIEE\n"},{"id":"298418","messageId":"458171B7.1020702@xs4all.nl","threadId":"43072","inReplyTo":"200612141518.05770.andyparkins@gmail.com","subject":"Re: git-fetching from a big repository is slow","fromName":"Han-Wen Nienhuys","fromEmail":"hanwen@xs4all.nl","sentAt":"2006-12-14T15:45:59Z","receivedAt":"2006-12-14T15:45:59Z","isPatch":false,"sender":{"key":"hanwen@google.com","avatar":"https://avatars.githubusercontent.com/u/31547?v=4"},"body":"Andy Parkins escreveu:\n> On Thursday 2006 December 14 13:53, Andreas Ericsson wrote:\n> \n>> This seems to happen if your repository consists of many large binary\n>> files, especially many large binary files of several versions that do\n>> not deltify well against each other. Perhaps it's worth adding gzip\n> \n> It's actually just every released patch to the linux kernel ever issued.  \n> Almost entirely ASCII and every revision (save the first) created by patching \n> the previous.\n\nI just noticed that git-fetch now runs git-show-ref --verify on every\ntag it encounters. This seems to slow down fetch over here.\n\n-- \n Han-Wen Nienhuys - hanwen@xs4all.nl - http://www.xs4all.nl/~hanwen\n"},{"id":"297361","messageId":"200612141620.17782.andyparkins@gmail.com","threadId":"43072","inReplyTo":"458171B7.1020702@xs4all.nl","subject":"Re: git-fetching from a big repository is slow","fromName":"Andy Parkins","fromEmail":"andyparkins@gmail.com","sentAt":"2006-12-14T16:20:16Z","receivedAt":"2006-12-14T16:20:16Z","isPatch":false,"sender":{"key":"andyparkins@gmail.com","avatar":null},"body":"On Thursday 2006 December 14 15:45, Han-Wen Nienhuys wrote:\n\n> I just noticed that git-fetch now runs git-show-ref --verify on every\n> tag it encounters. This seems to slow down fetch over here.\n\nThere aren't any tags in this repository :-)\n\nAndy\n\n-- \nDr Andy Parkins, M Eng (hons), MIEE\n"},{"id":"295086","messageId":"Pine.LNX.4.63.0612141732410.3635@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"43072","inReplyTo":"200612141620.17782.andyparkins@gmail.com","subject":"Re: git-fetching from a big repository is slow","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-14T16:34:58Z","receivedAt":"2006-12-14T16:34:58Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 14 Dec 2006, Andy Parkins wrote:\n\n> On Thursday 2006 December 14 15:45, Han-Wen Nienhuys wrote:\n> \n> > I just noticed that git-fetch now runs git-show-ref --verify on every\n> > tag it encounters. This seems to slow down fetch over here.\n> \n> There aren't any tags in this repository :-)\n\ngit-show-ref traverses every single _local_ tag when called. This is to \novercome the problem that tags can be packed now, so a simple file \nexistence check is not sufficient.\n\nIt would be much faster, probably, if you pack the local refs. IIRC I once \nargued for automatically packing refs (and all refs), but this has not \nbeen picked up, and I do not really care about it either.\n\nCiao,\nDscho\n"},{"id":"298269","messageId":"Pine.LNX.4.64.0612141311310.18171@xanadu.home","threadId":"43072","inReplyTo":"4581573E.40803@op5.se","subject":"Re: git-fetching from a big repository is slow","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2006-12-14T18:14:07Z","receivedAt":"2006-12-14T18:14:07Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 14 Dec 2006, Andreas Ericsson wrote:\n\n> Andy Parkins wrote:\n> > Hello,\n> > \n> > I've got a big repository.  I've got two computers.  One has the repository\n> > up-to-date (164M after repack); one is behind (30M ish).\n> > \n> > I used git-fetch to try and update; and the sync took HOURS.  I zipped the\n> > .git directory and transferred that and it took about 15 minutes to\n> > transfer.\n> > \n> > Am I doing something wrong?  The git-fetch was done with a git+ssh:// URL.\n> > The zip transfer with scp (so ssh shouldn't be a factor).\n> > \n> \n> This seems to happen if your repository consists of many large binary files,\n> especially many large binary files of several versions that do not deltify\n> well against each other. Perhaps it's worth adding gzip compression detecion\n> to git? I imagine more people than me are tracking gzipped/bzip2'ed content\n> that pretty much never deltifies well against anything else.\n\nIf your remote repository is fully packed in a single pack that should \nnot have any impact on the transfer latency since no attempt to \nredeltify objects against each other is attempted by default when those \nobjects are in the same pack.\n\n\n"},{"id":"294087","messageId":"C287764F-6755-4291-A87A-3E8816E90B49@adacore.com","threadId":"43072","inReplyTo":"4581685D.1070407@op5.se","subject":"Re: git-fetching from a big repository is slow","fromName":"Geert Bosch","fromEmail":"bosch@adacore.com","sentAt":"2006-12-14T19:05:27Z","receivedAt":"2006-12-14T19:05:27Z","isPatch":false,"sender":{"key":"bosch@adacore.com","avatar":null},"body":"\nOn Dec 14, 2006, at 10:06, Andreas Ericsson wrote:\n\n> It wouldn't work for this particular case though. In our  \n> distribution repository we have ~300 bzip2 compressed tarballs with  \n> an average size of 3MiB. 240 of those are between 2.5 and 4 MiB, so  \n> they don't drastically differ, but neither do they delta well.\n>\n> One option would be to add some sort of config option to skip  \n> attempting deltas of files with a certain suffix. That way we could  \n> just tell it to ignore *.gz,*.tgz,*.bz2 and everything would work  \n> just as it does today, but a lot faster.\n\nSuch special magic based on filenames is always a bad idea. Tomorrow  \nsomebody\ncomes with .zip files (oh, and of course .ZIP), then it's .jpg's other\ncompressed content. In the end git will be doing lots of magic and  \nstill perform\nbadly on unknown compressed content.\n\nThere is a very simple way of detecting compressed files: just look  \nat the\nsize of the compressed blob and compare against the size of the  \nexpanded blob.\nIf the compressed blob has a non-trivial size which is close to the  \nexpanded\nsize, assume the file is not interesting as source or target for deltas.\n\nExample:\n    if (compressed_size > expanded_size / 4 * 3 + 1024) {\n      /* don't try to deltify if blob doesn't compress well */\n      return ...;\n    }\n\n"},{"id":"294543","messageId":"7v8xha2osd.fsf@assigned-by-dhcp.cox.net","threadId":"43072","inReplyTo":"Pine.LNX.4.63.0612141732410.3635@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: git-fetching from a big repository is slow","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-12-14T20:41:38Z","receivedAt":"2006-12-14T20:41:38Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> git-show-ref traverses every single _local_ tag when called. This is to \n> overcome the problem that tags can be packed now, so a simple file \n> existence check is not sufficient.\n\nIs \"traverses every single _local_ tag\" a fact?  It might go\nthrough every single _local_ (possibly stale) packed tag in\nmemory but it should not traverse $GIT_DIR/refs/tags.\n\nIf I recall correctly, show-ref (1) first checks the filesystem\n\"$GIT_DIR/$named_ref\" and says Ok if found and valid; otherwise\n(2) checks packed refs (reads $GIT_DIR/packed-refs if not\nalready).  So that would be at most one open (which may fail in\n(1)) and one open+read (in (2)).  Unless we are talking about\nfork+exec overhead, that \"traverse\" should be reasonably fast.\n\nWhere is the bottleneck?\n"},{"id":"298064","messageId":"4581D01E.9020806@op5.se","threadId":"43072","inReplyTo":"C287764F-6755-4291-A87A-3E8816E90B49@adacore.com","subject":"Re: git-fetching from a big repository is slow","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2006-12-14T22:28:46Z","receivedAt":"2006-12-14T22:28:46Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Geert Bosch wrote:\n> \n> On Dec 14, 2006, at 10:06, Andreas Ericsson wrote:\n> \n>> It wouldn't work for this particular case though. In our distribution \n>> repository we have ~300 bzip2 compressed tarballs with an average size \n>> of 3MiB. 240 of those are between 2.5 and 4 MiB, so they don't \n>> drastically differ, but neither do they delta well.\n>>\n>> One option would be to add some sort of config option to skip \n>> attempting deltas of files with a certain suffix. That way we could \n>> just tell it to ignore *.gz,*.tgz,*.bz2 and everything would work just \n>> as it does today, but a lot faster.\n> \n> Such special magic based on filenames is always a bad idea. Tomorrow \n> somebody\n> comes with .zip files (oh, and of course .ZIP), then it's .jpg's other\n> compressed content. In the end git will be doing lots of magic and still \n> perform\n> badly on unknown compressed content.\n> \n\nHence config option. People can tell git to skip trying to delta \nwhatever they want. For this particular mothership repo, we only ever \nwork against it when we're at the office, meaning resulting datasize is \nnot an issue, but data computation can be a real bottle-neck.\n\n> There is a very simple way of detecting compressed files: just look at the\n> size of the compressed blob and compare against the size of the expanded \n> blob.\n> If the compressed blob has a non-trivial size which is close to the \n> expanded\n> size, assume the file is not interesting as source or target for deltas.\n> \n> Example:\n>    if (compressed_size > expanded_size / 4 * 3 + 1024) {\n>      /* don't try to deltify if blob doesn't compress well */\n>      return ...;\n>    }\n> \n\nMany compression algorithms generate similar output for similar input. \nMost source-code projects change relatively little between releases, so \nthey *could* delta well, it's just that in our repo they don't.\n\n-- \nAndreas Ericsson                   andreas.ericsson@op5.se\nOP5 AB                             www.op5.se\n"},{"id":"297646","messageId":"Pine.LNX.4.63.0612150015330.3635@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"43072","inReplyTo":"7v8xha2osd.fsf@assigned-by-dhcp.cox.net","subject":"Re: git-fetching from a big repository is slow","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-14T23:26:24Z","receivedAt":"2006-12-14T23:26:24Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 14 Dec 2006, Junio C Hamano wrote:\n\n> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n> \n> > git-show-ref traverses every single _local_ tag when called. This is to \n> > overcome the problem that tags can be packed now, so a simple file \n> > existence check is not sufficient.\n> \n> Is \"traverses every single _local_ tag\" a fact?  It might go\n> through every single _local_ (possibly stale) packed tag in\n> memory but it should not traverse $GIT_DIR/refs/tags.\n> \n> If I recall correctly, show-ref (1) first checks the filesystem\n> \"$GIT_DIR/$named_ref\" and says Ok if found and valid; otherwise\n> (2) checks packed refs (reads $GIT_DIR/packed-refs if not\n> already).\n\nIf I read builtin-show-ref.c correctly, it _always_ calls \nfor_each_ref(show_ref, NULL);\n\nThe only reason that the loop in for_each_ref can stop early is if \nshow_ref returns something different than 0. But it does not! Every single \nreturn in show_ref() returns 0. It does not matter, though (see below).\n\n> So that would be at most one open (which may fail in (1)) and one \n> open+read (in (2)).  Unless we are talking about fork+exec overhead, \n> that \"traverse\" should be reasonably fast.\n> \n> Where is the bottleneck?\n\nThe problem is that so many stat()s _do_ take time. Again, if I read the \ncode correctly, it not only stat()s every loose ref, but also resolves the \nrefs in get_ref_dir(), which is called from get_loose_refs(), which is \nunconditionally called in for_each_ref().\n\nEven if the refs are packed, it takes quite _long_ (I confirmed this). And \nit is not at all necessary! Instead of a O(n^2) we can easily reduce this \nto O(n*log(n)), and we can reduce the n fork()&exec()s of git-show-ref by \na single one.\n\nCiao,\nDscho\n"},{"id":"293866","messageId":"7vslfixaat.fsf@assigned-by-dhcp.cox.net","threadId":"43072","inReplyTo":"Pine.LNX.4.63.0612150015330.3635@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: git-fetching from a big repository is slow","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-12-15T00:38:50Z","receivedAt":"2006-12-15T00:38:50Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> If I read builtin-show-ref.c correctly, it _always_ calls \n> for_each_ref(show_ref, NULL);\n\nOk, that settles it.  If there is a reason to have --verify, we\nshould really special case it.  There is no point in looping,\nbecause verify does not do the tail match (which could cause\nambiguity) and its answer should be either \"yes it is there\" or\n\"no there is no such ref\".\n"}]}