{"thread":{"id":"43166","subject":"Re: git-pull from git.git - no remote ref for pu or next?","startedAt":"2006-12-12T16:44:22Z","lastAt":"2006-12-12T21:26:12Z","messageCount":22,"participants":["Johannes Schindelin","Jakub Narebski","Randal L. Schwartz","Nicolas Pitre","Linus Torvalds","Eric Wong"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"298106","messageId":"863b7l83o9.fsf@blue.stonehenge.com","threadId":"43166","inReplyTo":null,"subject":"git-pull from git.git - no remote ref for pu or next?","fromName":"Randal L. Schwartz","fromEmail":"merlyn@stonehenge.com","sentAt":"2006-12-12T16:44:22Z","receivedAt":"2006-12-12T16:44:22Z","isPatch":false,"sender":{"key":"merlyn@stonehenge.com","avatar":"https://gravatar.com/avatar/dc528d210743ff0333e6213f9ee7b33b23f1b7bc1f3c5a8c2d819074ecd7ab19?d=mp&s=160"},"body":"\nI just got this on this morning's git-fetch:\n\n    error: no such remote ref refs/heads/pu\n    error: no such remote ref refs/heads/next\n    Fetch failure: git://git.kernel.org/pub/scm/git/git.git\n\nHere's my remotes/origin:\n\n    URL: git://git.kernel.org/pub/scm/git/git.git\n    Pull: master:origin\n    Pull: man:man\n    Pull: html:html\n    Pull: +pu:pu\n    Pull: +next:next\n\n-- \nRandal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095\n<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>\nPerl/Unix/security consulting, Technical writing, Comedy, etc. etc.\n"},{"id":"294835","messageId":"86y7pd6oz7.fsf@blue.stonehenge.com","threadId":"43166","inReplyTo":"863b7l83o9.fsf@blue.stonehenge.com","subject":"Re: git-pull from git.git - no remote ref for pu or next?","fromName":"Randal L. Schwartz","fromEmail":"merlyn@stonehenge.com","sentAt":"2006-12-12T16:47:08Z","receivedAt":"2006-12-12T16:47:08Z","isPatch":false,"sender":{"key":"merlyn@stonehenge.com","avatar":"https://gravatar.com/avatar/dc528d210743ff0333e6213f9ee7b33b23f1b7bc1f3c5a8c2d819074ecd7ab19?d=mp&s=160"},"body":">>>>> \"Randal\" == Randal L Schwartz <merlyn@stonehenge.com> writes:\n\nRandal> I just got this on this morning's git-fetch:\n\nRandal>     error: no such remote ref refs/heads/pu\nRandal>     error: no such remote ref refs/heads/next\nRandal>     Fetch failure: git://git.kernel.org/pub/scm/git/git.git\n\nRandal> Here's my remotes/origin:\n\nRandal>     URL: git://git.kernel.org/pub/scm/git/git.git\nRandal>     Pull: master:origin\nRandal>     Pull: man:man\nRandal>     Pull: html:html\nRandal>     Pull: +pu:pu\nRandal>     Pull: +next:next\n\nAnd then it mysteriously fixed itself a few minutes later.\nIs there some sort of publishing failure, or intermittent race condition?\n\nOr is this something unique to git.git?\n\nOr just bad electron spin or something?\n\n-- \nRandal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095\n<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>\nPerl/Unix/security consulting, Technical writing, Comedy, etc. etc.\n"},{"id":"294486","messageId":"Pine.LNX.4.63.0612121839200.2807@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"43166","inReplyTo":"86y7pd6oz7.fsf@blue.stonehenge.com","subject":"Re: git-pull from git.git - no remote ref for pu or next?","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-12T17:40:08Z","receivedAt":"2006-12-12T17:40:08Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 12 Dec 2006, Randal L. Schwartz wrote:\n\n> >>>>> \"Randal\" == Randal L Schwartz <merlyn@stonehenge.com> writes:\n> \n> Randal> I just got this on this morning's git-fetch:\n> \n> Randal>     error: no such remote ref refs/heads/pu\n> Randal>     error: no such remote ref refs/heads/next\n> Randal>     Fetch failure: git://git.kernel.org/pub/scm/git/git.git\n\nCongratulations!\n\nYou are experiencing the Thundering Herd Phenomenon we talked a lot about \nlately (the kernel.org mirroring thread).\n\nCiao,\nDscho\n"},{"id":"295051","messageId":"86odq96mfp.fsf@blue.stonehenge.com","threadId":"43166","inReplyTo":"Pine.LNX.4.63.0612121839200.2807@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: git-pull from git.git - no remote ref for pu or next?","fromName":"Randal L. Schwartz","fromEmail":"merlyn@stonehenge.com","sentAt":"2006-12-12T17:42:02Z","receivedAt":"2006-12-12T17:42:02Z","isPatch":false,"sender":{"key":"merlyn@stonehenge.com","avatar":"https://gravatar.com/avatar/dc528d210743ff0333e6213f9ee7b33b23f1b7bc1f3c5a8c2d819074ecd7ab19?d=mp&s=160"},"body":">>>>> \"Johannes\" == Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\nJohannes> Congratulations!\n\nJohannes> You are experiencing the Thundering Herd Phenomenon we talked a lot about \nJohannes> lately (the kernel.org mirroring thread).\n\nAs long as others are sharing my pain, I'll be OK. :)\n\n-- \nRandal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095\n<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>\nPerl/Unix/security consulting, Technical writing, Comedy, etc. etc.\n"},{"id":"297470","messageId":"Pine.LNX.4.64.0612120949230.3535@woody.osdl.org","threadId":"43166","inReplyTo":"86y7pd6oz7.fsf@blue.stonehenge.com","subject":"Re: git-pull from git.git - no remote ref for pu or next?","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-12-12T17:56:11Z","receivedAt":"2006-12-12T17:56:11Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 12 Dec 2006, Randal L. Schwartz wrote:\n\n> >>>>> \"Randal\" == Randal L Schwartz <merlyn@stonehenge.com> writes:\n> \n> Randal> I just got this on this morning's git-fetch:\n> \n> Randal>     error: no such remote ref refs/heads/pu\n> Randal>     error: no such remote ref refs/heads/next\n> Randal>     Fetch failure: git://git.kernel.org/pub/scm/git/git.git\n> \n> Randal> Here's my remotes/origin:\n> \n> Randal>     URL: git://git.kernel.org/pub/scm/git/git.git\n> Randal>     Pull: master:origin\n> Randal>     Pull: man:man\n> Randal>     Pull: html:html\n> Randal>     Pull: +pu:pu\n> Randal>     Pull: +next:next\n> \n> And then it mysteriously fixed itself a few minutes later.\n> Is there some sort of publishing failure, or intermittent race condition?\n\nIt's mirroring.\n\nThe way that kernel.org works is that there is one master site, which is \nnot actually running any public services at all. That's the one that \npeople who have write access can ssh into, and rather than run any public \nservices, it runs the security-conscious things, like the secure logins \nand the automated signing scripts.\n\nThe actual _public_ sites are just mirrors, with just rsync between the \nthings. All to keep the services on the master site minimal.\n\nBut because the public sites just mirror using rsync, and aren't really \naware of git repositories etc at that stage, what can happen is that a \nmirroring is on-going when Junio does a push, and then the changes to the \n\"refs/\" directory might get rsync'ed before the \"object/\" directory does, \nand you end up with the public sites having references to objects that \ndon't even _exist_ on those public sites any more.\n\nWhen they then run git-daemon, git-deamon will basically see a corrupt git \narchive, and not expose those \"broken\" refs at all. Which explains what \nyou see.\n\nAnd once the mirroring completes, the issue just goes away, which explains \nwhy it just magically works five minutes later.\n\nIf the public sites used git itself to synchronize git repositories, \nthey'd never see anything like this (because git itself will only write \nthe new refs after it has actually updated the data). But since the thing \nneeds mirroring for non-git uses too, and since rsync generally _works_ \napart from the slight race-condition issue, that's what it just uses.\n\n"},{"id":"297132","messageId":"Pine.LNX.4.63.0612121908100.2807@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"43166","inReplyTo":"Pine.LNX.4.64.0612120949230.3535@woody.osdl.org","subject":"Re: git-pull from git.git - no remote ref for pu or next?","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-12T18:09:39Z","receivedAt":"2006-12-12T18:09:39Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 12 Dec 2006, Linus Torvalds wrote:\n\n> But since the thing needs mirroring for non-git uses too, and since \n> rsync generally _works_ apart from the slight race-condition issue, \n\n... and git would probably change the pack structure (i.e. which objects \nare in which packs, or even loose) which would be too bad for all those \nHTTP leechers ...\n\n> that's what it just uses.\n\nCiao,\nDscho\n"},{"id":"296341","messageId":"Pine.LNX.4.64.0612121021270.3535@woody.osdl.org","threadId":"43166","inReplyTo":"Pine.LNX.4.63.0612121908100.2807@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: git-pull from git.git - no remote ref for pu or next?","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-12-12T18:23:36Z","receivedAt":"2006-12-12T18:23:36Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 12 Dec 2006, Johannes Schindelin wrote:\n> > rsync generally _works_ apart from the slight race-condition issue, \n> \n> ... and git would probably change the pack structure (i.e. which objects \n> are in which packs, or even loose) which would be too bad for all those \n> HTTP leechers ...\n\nWell, as it is, I end up repacking my git archives on kernel.org every two \nweeks or so anyway, so anybody who uses stupid protocols (rsync or http) \nwill end up downloading everything anew anyway.\n\nAnd kernel.org will probably start doing automatic repacking, since the \ncurrent situation just means that some people don't repack on their own, \nand have tens of thousands of loose objects.\n\nYou really don't want to use the non-native protocols unless you have to, \nor for projects that don't change.\n\n"},{"id":"297607","messageId":"Pine.LNX.4.64.0612121345180.18171@xanadu.home","threadId":"43166","inReplyTo":"Pine.LNX.4.64.0612120949230.3535@woody.osdl.org","subject":"Re: git-pull from git.git - no remote ref for pu or next?","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2006-12-12T18:51:43Z","receivedAt":"2006-12-12T18:51:43Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 12 Dec 2006, Linus Torvalds wrote:\n\n> If the public sites used git itself to synchronize git repositories, \n> they'd never see anything like this (because git itself will only write \n> the new refs after it has actually updated the data). But since the thing \n> needs mirroring for non-git uses too, and since rsync generally _works_ \n> apart from the slight race-condition issue, that's what it just uses.\n\nWouldn't it be a worthy goal to exclude git repos from the rsync \nmirroring and use git instead?  The current arrangement doesn't put git \nin good light for the general public not reading this mailing list wrt \ngit reliability, even if we know it is just a minor and temporary \nannoyance.\n\nA failure always makes you look bad regardless of its severity.\n\n\n"},{"id":"297152","messageId":"Pine.LNX.4.64.0612121352520.18171@xanadu.home","threadId":"43166","inReplyTo":"Pine.LNX.4.63.0612121908100.2807@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: git-pull from git.git - no remote ref for pu or next?","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2006-12-12T18:54:01Z","receivedAt":"2006-12-12T18:54:01Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 12 Dec 2006, Johannes Schindelin wrote:\n\n> Hi,\n> \n> On Tue, 12 Dec 2006, Linus Torvalds wrote:\n> \n> > But since the thing needs mirroring for non-git uses too, and since \n> > rsync generally _works_ apart from the slight race-condition issue, \n> \n> ... and git would probably change the pack structure (i.e. which objects \n> are in which packs, or even loose) which would be too bad for all those \n> HTTP leechers ...\n\nI don't see how that would be more of a concern than the current \nsituation with occasional repacks.\n\n\n"},{"id":"297726","messageId":"Pine.LNX.4.64.0612121102390.3535@woody.osdl.org","threadId":"43166","inReplyTo":"Pine.LNX.4.64.0612121345180.18171@xanadu.home","subject":"Re: git-pull from git.git - no remote ref for pu or next?","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-12-12T19:03:59Z","receivedAt":"2006-12-12T19:03:59Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 12 Dec 2006, Nicolas Pitre wrote:\n> \n> Wouldn't it be a worthy goal to exclude git repos from the rsync \n> mirroring and use git instead?\n\nWell, one of the problems is simply maintenance of kernel.org.\n\nIt's just _simpler_ to use rsync for everything.\n\nLook at the current gitweb caching discussion. Did anybody actually step \nup to be a gitweb maintainer on kernel.org?\n\nSame deal. Simplicity and lack of maintenance is sometimes not just a good \nidea, it's a requirement.\n\n"},{"id":"295671","messageId":"Pine.LNX.4.63.0612121956470.2807@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"43166","inReplyTo":"Pine.LNX.4.64.0612121352520.18171@xanadu.home","subject":"Re: git-pull from git.git - no remote ref for pu or next?","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-12T19:04:32Z","receivedAt":"2006-12-12T19:04:32Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 12 Dec 2006, Nicolas Pitre wrote:\n\n> On Tue, 12 Dec 2006, Johannes Schindelin wrote:\n> \n> > On Tue, 12 Dec 2006, Linus Torvalds wrote:\n> > \n> > > But since the thing needs mirroring for non-git uses too, and since \n> > > rsync generally _works_ apart from the slight race-condition issue, \n> > \n> > ... and git would probably change the pack structure (i.e. which objects \n> > are in which packs, or even loose) which would be too bad for all those \n> > HTTP leechers ...\n> \n> I don't see how that would be more of a concern than the current \n> situation with occasional repacks.\n\nOh well. I did not want to get bashed for something which is probably no \nproblem, but I suspected that the two mirror machines could get out of \nsync, which could well mean that the new packs would have to be downloaded \n_twice_. As I said, probably no problem.\n\nBut it would become a non-problem when the HTTP transport would learn to \nread and interpret the .idx files, basically constructing thin packs from \nparts of the .pack files (\"Content-Range:\" comes to mind)...\n\nCiao,\nDscho\n"},{"id":"295185","messageId":"Pine.LNX.4.64.0612121400530.18171@xanadu.home","threadId":"43166","inReplyTo":"Pine.LNX.4.64.0612121021270.3535@woody.osdl.org","subject":"Re: git-pull from git.git - no remote ref for pu or next?","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2006-12-12T19:07:04Z","receivedAt":"2006-12-12T19:07:04Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 12 Dec 2006, Linus Torvalds wrote:\n\n> And kernel.org will probably start doing automatic repacking, since the \n> current situation just means that some people don't repack on their own, \n> and have tens of thousands of loose objects.\n\nMaybe object sharing between repos could be a good idea too?  All kernel \nrepos are likely to contain a large subset of your own so they could \nhave a reference on it by default.  That would certainly allow for \nbetter caching and less IO on the server.\n\n\n"},{"id":"298138","messageId":"Pine.LNX.4.64.0612121111510.3535@woody.osdl.org","threadId":"43166","inReplyTo":"Pine.LNX.4.64.0612121400530.18171@xanadu.home","subject":"Re: git-pull from git.git - no remote ref for pu or next?","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-12-12T19:13:29Z","receivedAt":"2006-12-12T19:13:29Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 12 Dec 2006, Nicolas Pitre wrote:\n> \n> Maybe object sharing between repos could be a good idea too?\n\nWe often do. Many people have used \"git clone -s -l\", and as long as you \nrepack with the \"-l\" flag too, the object sharing actually increases over \ntime as the base repo gets repacked and the cloned repo keeps using the \ngrowing base pack..\n\n"},{"id":"294720","messageId":"elmv0c$kmd$1@sea.gmane.org","threadId":"43166","inReplyTo":"Pine.LNX.4.63.0612121956470.2807@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: git-pull from git.git - no remote ref for pu or next?","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-12-12T19:15:41Z","receivedAt":"2006-12-12T19:15:41Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Johannes Schindelin wrote:\n\n> But it would become a non-problem when the HTTP transport would learn to \n> read and interpret the .idx files, basically constructing thin packs from \n> parts of the .pack files (\"Content-Range:\" comes to mind)...\n\ncURL the CLI can do this with -r/--range option, so I think that curl\nthe library can do this too. Mind you, this is HTTP/1.1 extension\n(hmmm... I wonder if many sites run HTTP/1.0 only).\n\nSee also: http://www.linux.com/article.pl?sid=06/10/10/1824245\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n\n"},{"id":"296778","messageId":"elmv4t$kmd$2@sea.gmane.org","threadId":"43166","inReplyTo":"Pine.LNX.4.64.0612121021270.3535@woody.osdl.org","subject":"Re: git-pull from git.git - no remote ref for pu or next?","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-12-12T19:18:06Z","receivedAt":"2006-12-12T19:18:06Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Linus Torvalds wrote:\n\n> On Tue, 12 Dec 2006, Johannes Schindelin wrote:\n>> > rsync generally _works_ apart from the slight race-condition issue, \n>> \n>> ... and git would probably change the pack structure (i.e. which objects \n>> are in which packs, or even loose) which would be too bad for all those \n>> HTTP leechers ...\n> \n> Well, as it is, I end up repacking my git archives on kernel.org every two \n> weeks or so anyway, so anybody who uses stupid protocols (rsync or http) \n> will end up downloading everything anew anyway.\n\nWhat about \"logaritmic packs\" idea someone (Pasky?) posted on git mailing\nlist: pack from last week, pack from last month except last week, pack from\ntwo months, pack from four months, pack from last year...\n\n> And kernel.org will probably start doing automatic repacking, since the \n> current situation just means that some people don't repack on their own, \n> and have tens of thousands of loose objects.\n\n...with *.keep to keep archive packs it can be even automated somewhat.\n\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n\n"},{"id":"295719","messageId":"Pine.LNX.4.64.0612121420580.18171@xanadu.home","threadId":"43166","inReplyTo":"Pine.LNX.4.63.0612121956470.2807@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: git-pull from git.git - no remote ref for pu or next?","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2006-12-12T19:26:17Z","receivedAt":"2006-12-12T19:26:17Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 12 Dec 2006, Johannes Schindelin wrote:\n\n> On Tue, 12 Dec 2006, Nicolas Pitre wrote:\n> \n> > On Tue, 12 Dec 2006, Johannes Schindelin wrote:\n> > \n> > > On Tue, 12 Dec 2006, Linus Torvalds wrote:\n> > > \n> > > > But since the thing needs mirroring for non-git uses too, and since \n> > > > rsync generally _works_ apart from the slight race-condition issue, \n> > > \n> > > ... and git would probably change the pack structure (i.e. which objects \n> > > are in which packs, or even loose) which would be too bad for all those \n> > > HTTP leechers ...\n> > \n> > I don't see how that would be more of a concern than the current \n> > situation with occasional repacks.\n> \n> Oh well. I did not want to get bashed for something which is probably no \n> problem,\n\nSorry, far from me to sound as if I was bashing you.\n\n> but I suspected that the two mirror machines could get out of \n> sync, which could well mean that the new packs would have to be downloaded \n> _twice_. As I said, probably no problem.\n\nIn theory that should not happen since all mirrors would get the same \nupdates in the same steps.  But in practice if one mirror fails to get \nupdated for whatever reason then the next time around it could have a \nbigger pack instead of two smaller ones for the same set of objects.\n\n> But it would become a non-problem when the HTTP transport would learn to \n> read and interpret the .idx files, basically constructing thin packs from \n> parts of the .pack files (\"Content-Range:\" comes to mind)...\n\nWoooh.\n\n\n"},{"id":"293957","messageId":"Pine.LNX.4.63.0612122031450.2807@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"43166","inReplyTo":"Pine.LNX.4.64.0612121420580.18171@xanadu.home","subject":"Re: git-pull from git.git - no remote ref for pu or next?","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-12T19:32:39Z","receivedAt":"2006-12-12T19:32:39Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 12 Dec 2006, Nicolas Pitre wrote:\n\n> On Tue, 12 Dec 2006, Johannes Schindelin wrote:\n> \n> > But it would become a non-problem when the HTTP transport would learn \n> > to read and interpret the .idx files, basically constructing thin \n> > packs from parts of the .pack files (\"Content-Range:\" comes to \n> > mind)...\n> \n> Woooh.\n\nDoes that mean \"Yes, I'll do it\"? ;-)\n\nCiao,\nDscho\n"},{"id":"298384","messageId":"Pine.LNX.4.64.0612121144220.3535@woody.osdl.org","threadId":"43166","inReplyTo":"Pine.LNX.4.63.0612122031450.2807@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: git-pull from git.git - no remote ref for pu or next?","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-12-12T19:48:09Z","receivedAt":"2006-12-12T19:48:09Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 12 Dec 2006, Johannes Schindelin wrote:\n> On Tue, 12 Dec 2006, Nicolas Pitre wrote:\n> \n> > On Tue, 12 Dec 2006, Johannes Schindelin wrote:\n> > \n> > > But it would become a non-problem when the HTTP transport would learn \n> > > to read and interpret the .idx files, basically constructing thin \n> > > packs from parts of the .pack files (\"Content-Range:\" comes to \n> > > mind)...\n> > \n> > Woooh.\n> \n> Does that mean \"Yes, I'll do it\"? ;-)\n\nUmm. I hope it means \"Woooh, that's crazy talk\".\n\nYou do realize that then you need to teach the http-walker about walking \nthe delta chain all the way up? For big pulls, you're going to be a lot \n_slower_ than just downloading the whole dang thing, because the delta \nobjects are often just ~40 bytes, and you've now added a ping-pong latency \nfor each such small transfer.\n\nYou don't need to download many such small ranges, and suddenly the few \nhundred ping-pongs that got you a few tens of kB of data took longer than \njust downloading a big stream efficiently that got you everything.\n\nOne big reason the native git protocol is efficient is that it's largely a \nstreaming protocol (apart from the early ref-walking, but even that tries \nto stream as much as possible, rather than having a back-and-forth latency \nfor each query)\n\n"},{"id":"296785","messageId":"Pine.LNX.4.64.0612121445320.18171@xanadu.home","threadId":"43166","inReplyTo":"Pine.LNX.4.63.0612122031450.2807@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: git-pull from git.git - no remote ref for pu or next?","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2006-12-12T19:50:20Z","receivedAt":"2006-12-12T19:50:20Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 12 Dec 2006, Johannes Schindelin wrote:\n\n> Hi,\n> \n> On Tue, 12 Dec 2006, Nicolas Pitre wrote:\n> \n> > On Tue, 12 Dec 2006, Johannes Schindelin wrote:\n> > \n> > > But it would become a non-problem when the HTTP transport would learn \n> > > to read and interpret the .idx files, basically constructing thin \n> > > packs from parts of the .pack files (\"Content-Range:\" comes to \n> > > mind)...\n> > \n> > Woooh.\n> \n> Does that mean \"Yes, I'll do it\"? ;-)\n\nAbsolutely not.  ;-)  I know next to nothing about HTTP to start with.\nIt just looks like a crazy idea that might actually work.\n\n\n"},{"id":"293998","messageId":"eln1sr$1rm$1@sea.gmane.org","threadId":"43166","inReplyTo":"Pine.LNX.4.64.0612121144220.3535@woody.osdl.org","subject":"Re: git-pull from git.git - no remote ref for pu or next?","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-12-12T20:04:54Z","receivedAt":"2006-12-12T20:04:54Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Linus Torvalds wrote:\n\n> \n> \n> On Tue, 12 Dec 2006, Johannes Schindelin wrote:\n>> On Tue, 12 Dec 2006, Nicolas Pitre wrote:\n>> \n>>> On Tue, 12 Dec 2006, Johannes Schindelin wrote:\n>>> \n>>>> But it would become a non-problem when the HTTP transport would learn \n>>>> to read and interpret the .idx files, basically constructing thin \n>>>> packs from parts of the .pack files (\"Content-Range:\" comes to \n>>>> mind)...\n>>> \n>>> Woooh.\n>> \n>> Does that mean \"Yes, I'll do it\"? ;-)\n> \n> Umm. I hope it means \"Woooh, that's crazy talk\".\n> \n> You do realize that then you need to teach the http-walker about walking \n> the delta chain all the way up? For big pulls, you're going to be a lot \n> _slower_ than just downloading the whole dang thing, because the delta \n> objects are often just ~40 bytes, and you've now added a ping-pong latency \n> for each such small transfer.\n> \n> You don't need to download many such small ranges, and suddenly the few \n> hundred ping-pongs that got you a few tens of kB of data took longer than \n> just downloading a big stream efficiently that got you everything.\n\nWhile I think the problem is much better solved by having \"archive\" pack(s)\nand \"current\" pack, perhaps with always sownloading the whole delta it\nwould be feasible?\n\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n\n"},{"id":"295377","messageId":"Pine.LNX.4.63.0612122121190.2807@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"43166","inReplyTo":"Pine.LNX.4.64.0612121144220.3535@woody.osdl.org","subject":"Re: git-pull from git.git - no remote ref for pu or next?","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-12T20:26:22Z","receivedAt":"2006-12-12T20:26:22Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 12 Dec 2006, Linus Torvalds wrote:\n\n> On Tue, 12 Dec 2006, Johannes Schindelin wrote:\n> > On Tue, 12 Dec 2006, Nicolas Pitre wrote:\n> > \n> > > On Tue, 12 Dec 2006, Johannes Schindelin wrote:\n> > > \n> > > > But it would become a non-problem when the HTTP transport would learn \n> > > > to read and interpret the .idx files, basically constructing thin \n> > > > packs from parts of the .pack files (\"Content-Range:\" comes to \n> > > > mind)...\n> > > \n> > > Woooh.\n> > \n> > Does that mean \"Yes, I'll do it\"? ;-)\n> \n> Umm. I hope it means \"Woooh, that's crazy talk\".\n> \n> You do realize that then you need to teach the http-walker about walking \n> the delta chain all the way up? For big pulls, you're going to be a lot \n> _slower_ than just downloading the whole dang thing, because the delta \n> objects are often just ~40 bytes, and you've now added a ping-pong latency \n> for each such small transfer.\n\nTwo points:\n\n- For loose objects, the HTTP walker does exactly that. This is the normal \ncase for \"just a few objects since the last fetch\". It will _never_ be the \ncase for the initial clone!\n\n- Usually, the object fetching can be parallelized, because you want \nmultiple objects which are in disjunct delta chains. And for these, you \ncan say something like \"Content-Range: 15-31,64-79,108-135\" IIRC.\n\nYou could even fetch sensible chunks, i.e. cut only at multiples of 512 to \nmake the transport more efficient, and only fetch the parts which are \n_still_ missing.\n\nSo, a crazy idea, yes. But a feasible one. Just not crazy enough to be \ntempting for me (I use the git protocol whenever possible, too).\n\nCiao,\n"},{"id":"298736","messageId":"20061212212612.GA22497@localdomain","threadId":"43166","inReplyTo":"Pine.LNX.4.64.0612120949230.3535@woody.osdl.org","subject":"Re: git-pull from git.git - no remote ref for pu or next?","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2006-12-12T21:26:12Z","receivedAt":"2006-12-12T21:26:12Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Linus Torvalds <torvalds@osdl.org> wrote:\n> > And then it mysteriously fixed itself a few minutes later.\n> > Is there some sort of publishing failure, or intermittent race condition?\n> \n> But because the public sites just mirror using rsync, and aren't really \n> aware of git repositories etc at that stage, what can happen is that a \n> mirroring is on-going when Junio does a push, and then the changes to the \n> \"refs/\" directory might get rsync'ed before the \"object/\" directory does, \n> and you end up with the public sites having references to objects that \n> don't even _exist_ on those public sites any more.\n> \n> And once the mirroring completes, the issue just goes away, which explains \n> why it just magically works five minutes later.\n\nIf kernel.org isn't using it already, I've found the --delay-updates\noption of rsync works reasonably well and can cut down the\nrace-condition window.  It does use more memory and disk space, however.\natomic-rsync (a perl front-end distributed with the rsync source) takes\neven more disk space but works across an entire subdirectory all at once\n(with 2 renames)\n\n-- \n"}]}