{"thread":{"id":"20626","subject":"Continue git clone after interruption","startedAt":"2009-08-17T11:42:22Z","lastAt":"2009-08-23T10:37:09Z","messageCount":39,"participants":["Tomasz Kontusz","Johannes Schindelin","Shawn O. Pearce","Matthieu Moy","Nicolas Pitre","Jakub Narebski","Sitaram Chamarty","Nguyen Thai Ngoc Duy","=?VISCII?Q?Ren=E9_Scharfe?=","Sam Vilain"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"120865","messageId":"1250509342.2885.13.camel@cf-48","threadId":"20626","inReplyTo":null,"subject":"Continue git clone after interruption","fromName":"Tomasz Kontusz","fromEmail":"roverorna@gmail.com","sentAt":"2009-08-17T11:42:22Z","receivedAt":"2009-08-17T11:42:22Z","isPatch":false,"sender":{"key":"roverorna@gmail.com","avatar":null},"body":"Hi,\nis anybody working on making it possible to continue git clone after\ninterruption? It would be quite useful for people with bad internet\nconnection (I was downloading a big repo lately, and it was a bit\nfrustrating to start it over every time git stopped at ~90%).\n\nTomasz Kontusz\n"},{"id":"120874","messageId":"alpine.DEB.1.00.0908171430010.4991@intel-tinevez-2-302","threadId":"20626","inReplyTo":"1250509342.2885.13.camel@cf-48","subject":"Re: Continue git clone after interruption","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-08-17T12:31:51Z","receivedAt":"2009-08-17T12:31:51Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 17 Aug 2009, Tomasz Kontusz wrote:\n\n> is anybody working on making it possible to continue git clone after \n> interruption? It would be quite useful for people with bad internet \n> connection (I was downloading a big repo lately, and it was a bit \n> frustrating to start it over every time git stopped at ~90%).\n\nUnfortunately, we did not have enough GSoC slots for the project to allow \nrestartable clones.\n\nThere were discussions about how to implement this on the list, though.\n\nCiao,\nDscho\n"},{"id":"120899","messageId":"20090817152358.GV1033@spearce.org","threadId":"20626","inReplyTo":"alpine.DEB.1.00.0908171430010.4991@intel-tinevez-2-302","subject":"Re: Continue git clone after interruption","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-08-17T15:23:58Z","receivedAt":"2009-08-17T15:23:58Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Johannes Schindelin wrote:\n> On Mon, 17 Aug 2009, Tomasz Kontusz wrote:\n> \n> > is anybody working on making it possible to continue git clone after \n> > interruption? It would be quite useful for people with bad internet \n> > connection (I was downloading a big repo lately, and it was a bit \n> > frustrating to start it over every time git stopped at ~90%).\n> \n> Unfortunately, we did not have enough GSoC slots for the project to allow \n> restartable clones.\n> \n> There were discussions about how to implement this on the list, though.\n\nUnfortunately, those of us who know how the native protocol works\ncan't come to an agreement on how it might be restartable. If you\nreally read the archives on this topic, you'll see that Nico and I\ndisagree about how to do this. IIRC Nico's position is, it isn't\nreally possible to implement a restart.\n\n-- \nShawn.\n"},{"id":"121054","messageId":"vpqskfphe2k.fsf@bauges.imag.fr","threadId":"20626","inReplyTo":"alpine.DEB.1.00.0908171430010.4991@intel-tinevez-2-302","subject":"Re: Continue git clone after interruption","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2009-08-18T05:43:31Z","receivedAt":"2009-08-18T05:43:31Z","isPatch":false,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"Johannes Schindelin writes:\n\n> Hi,\n>\n> On Mon, 17 Aug 2009, Tomasz Kontusz wrote:\n>\n>> is anybody working on making it possible to continue git clone after \n>> interruption? It would be quite useful for people with bad internet \n>> connection (I was downloading a big repo lately, and it was a bit \n>> frustrating to start it over every time git stopped at ~90%).\n>\n> Unfortunately, we did not have enough GSoC slots for the project to allow \n> restartable clones.\n>\n> There were discussions about how to implement this on the list,\n> though.\n\nAnd a paragraph on the wiki:\n\nhttp://git.or.cz/gitwiki/SoC2009Ideas#RestartableClone\n\n-- \nMatthieu\n"},{"id":"121059","messageId":"1250578735.2885.40.camel@cf-48","threadId":"20626","inReplyTo":"vpqskfphe2k.fsf@bauges.imag.fr","subject":"Re: Continue git clone after interruption","fromName":"Tomasz Kontusz","fromEmail":"roverorna@gmail.com","sentAt":"2009-08-18T06:58:55Z","receivedAt":"2009-08-18T06:58:55Z","isPatch":false,"sender":{"key":"roverorna@gmail.com","avatar":null},"body":"Dnia 2009-08-18, wto o godzinie 07:43 +0200, Matthieu Moy pisze:\n> Johannes Schindelin writes:\n> \n> > Hi,\n> >\n> > On Mon, 17 Aug 2009, Tomasz Kontusz wrote:\n> >\n> >> is anybody working on making it possible to continue git clone after \n> >> interruption? It would be quite useful for people with bad internet \n> >> connection (I was downloading a big repo lately, and it was a bit \n> >> frustrating to start it over every time git stopped at ~90%).\n> >\n> > Unfortunately, we did not have enough GSoC slots for the project to allow \n> > restartable clones.\n> >\n> > There were discussions about how to implement this on the list,\n> > though.\n> \n> And a paragraph on the wiki:\n> \n> http://git.or.cz/gitwiki/SoC2009Ideas#RestartableClone\n\nOk, so it looks like it's not implementable without some kind of cache\nserver-side, so the server would know what the pack it was sending\nlooked like.\nBut here's my idea: make server send objects in different order (the\nnewest commit + whatever it points to first, then next one,then\nanother...). Then it would be possible to look at what we got, tell\nserver we have nothing, and want [the newest commit that was not\ncomplete]. I know the reason why it is sorted the way it is, but I think\nthat the way data is stored after clone is clients problem, so the\nclient should reorganize packs the way it wants.\n\nTomasz K.\n"},{"id":"121137","messageId":"alpine.LFD.2.00.0908181246470.6044@xanadu.home","threadId":"20626","inReplyTo":"1250578735.2885.40.camel@cf-48","subject":"Re: Continue git clone after interruption","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-08-18T17:56:16Z","receivedAt":"2009-08-18T17:56:16Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 18 Aug 2009, Tomasz Kontusz wrote:\n\n> Ok, so it looks like it's not implementable without some kind of cache\n> server-side, so the server would know what the pack it was sending\n> looked like.\n> But here's my idea: make server send objects in different order (the\n> newest commit + whatever it points to first, then next one,then\n> another...). Then it would be possible to look at what we got, tell\n> server we have nothing, and want [the newest commit that was not\n> complete]. I know the reason why it is sorted the way it is, but I think\n> that the way data is stored after clone is clients problem, so the\n> client should reorganize packs the way it wants.\n\nThat won't buy you much. You should realize that a pack is made of:\n\n1) Commit objects. Yes they're all put together at the front of the pack,\n but they roughly are the equivalent of:\n\n\tgit log --pretty=raw | gzip | wc -c\n\n For the Linux repo as of now that is around 32 MB.\n\n2) Tree andblob objects. Those are the bulk of the content for the top \n commit. The top commit is usually not delta compressed because we \n want fast access to the top commit, and that is used as the base for \n further delta compression for older commits. So the very first \n commit is whole at the front of the pack right after the commit \n objects. you can estimate the size of this data with:\n\n\tgit archive --format=tar HEAD | gzip | wc -c\n\n On the same Linux repo this is currently 75 MB.\n\n3) Delta objects. Those are making the rest of the pack, plus a couple \n tree/blob objects that were not found in the top commit and are \n different enough from any object in that top commit not to be \n represented as deltas. Still, the majority of objects for all the \n remaining commits are delta objects.\n\nSo... if we reorder objects, all that we can do is to spread commit \nobjects around so that the objects referenced by one commit are all seen \nbefore another commit object is included. That would cut on that \ninitial 32 MB.\n\nHowever you still have to get that 75 MB in order to at least be able to \nlook at _one_ commit. So you've only reduced your critical download \nsize from 107 MB to 75 MB. This is some improvement, of course, but not \nworth the bother IMHO. If we're to have restartable clone, it has to \nwork for any size.\n\nAnd that's where the real problem is. I don't think having servers to \ncache pack results for every fetch requests is sensible as that would be \nan immediate DoS attack vector.\n\nAnd because the object order in a pack is not defined by the protocol, \nwe cannot expect the server to necessarily always provide the same \nobject order either. For example, it is already undefined in which \norder you'll receive objects as threaded delta search is non \ndeterministic and two identical fetch requests may end up with slightly \ndifferent packing. Or load balancing may redirect your fetch requests \nto different git servers which might have different versions of zlib, or \neven git itself, affecting the object packing order and/or size.\n\nNow... What _could_ be done, though, is some extension to the \ngit-archive command. One thing that is well and strictly defined in git \nis the file path sort order. So given a commit SHA1, you should always \nget the same files in the same order from git-archive. For an initial \nclone, git could attempt fetching the top commit using the remote \ngit-archive service and locally reconstruct that top commit that way. \nif the transfer is interrupted in the middle, then the remote \ngit-archive could be told how to resume the transfer by telling it how \nmany files and how many bytes in the current file to skip. This way the \nserver doesn't need to perform any sort of caching and remains \nstateless.\n\nYou then end up with a pretty shallow repository. The clone process \ncould then fall back to the traditional native git transfer protocol to \ndeepen the history of that shallow repository. And then that special \npacking sort order to distribute commit objects would make sense since \neach commit would then have a fairly small set of new objects, and most \nof them would be deltas anyway, making the data size per commit really \nsmall and any interrupted transfer much less of an issue.\n\n\nNicolas\n"},{"id":"121143","messageId":"m3fxbpneqe.fsf@localhost.localdomain","threadId":"20626","inReplyTo":"alpine.LFD.2.00.0908181246470.6044@xanadu.home","subject":"Re: Continue git clone after interruption","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-08-18T18:45:44Z","receivedAt":"2009-08-18T18:45:44Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Nicolas Pitre writes:\n\n> On Tue, 18 Aug 2009, Tomasz Kontusz wrote:\n> \n> > Ok, so it looks like it's not implementable without some kind of cache\n> > server-side, so the server would know what the pack it was sending\n> > looked like.\n> > But here's my idea: make server send objects in different order (the\n> > newest commit + whatever it points to first, then next one,then\n> > another...). Then it would be possible to look at what we got, tell\n> > server we have nothing, and want [the newest commit that was not\n> > complete]. I know the reason why it is sorted the way it is, but I think\n> > that the way data is stored after clone is clients problem, so the\n> > client should reorganize packs the way it wants.\n> \n> That won't buy you much. You should realize that a pack is made of:\n> \n> 1) Commit objects. Yes they're all put together at the front of the pack,\n> but they roughly are the equivalent of:\n> \n> \tgit log --pretty=raw | gzip | wc -c\n> \n> For the Linux repo as of now that is around 32 MB.\n\nFor my clone of Git repository this gives 3.8 MB\n \n> 2) Tree and blob objects. Those are the bulk of the content for the top \n> commit. The top commit is usually not delta compressed because we \n> want fast access to the top commit, and that is used as the base for \n> further delta compression for older commits. So the very first \n> commit is whole at the front of the pack right after the commit \n> objects. you can estimate the size of this data with:\n> \n> \tgit archive --format=tar HEAD | gzip | wc -c\n> \n> On the same Linux repo this is currently 75 MB.\n\nOn the same Git repository this gives 2.5 MB\n\n> \n> 3) Delta objects. Those are making the rest of the pack, plus a couple \n> tree/blob objects that were not found in the top commit and are \n> different enough from any object in that top commit not to be \n> represented as deltas. Still, the majority of objects for all the \n> remaining commits are delta objects.\n\nYou forgot that delta chains are bound by pack.depth limit, which\ndefaults to 50. You would have then additional full objects.\n\nThe single packfile for this (just gc'ed) Git repository is 37 MB.\nMuch more than 3.8 MB + 2.5 MB = 6.3 MB.\n\n[cut]\n\nThere is another way which we can go to implement resumable clone.\nLet's git first try to clone whole repository (single pack; BTW what\nhappens if this pack is larger than file size limit for given\nfilesystem?). If it fails, client ask first for first half of of\nrepository (half as in bisect, but it is server that has to calculate\nit). If it downloads, it will ask server for the rest of repository.\nIf it fails, it would reduce size in half again, and ask about 1/4 of\nrepository in packfile first.\n\nThe only extension required is for server to support additional\ncapability, which enable for client to ask for appropriate 1/2^n part\nof repository (approximately), or 1/2^n between have and want.\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"121152","messageId":"alpine.LFD.2.00.0908181537360.6044@xanadu.home","threadId":"20626","inReplyTo":"m3fxbpneqe.fsf@localhost.localdomain","subject":"Re: Continue git clone after interruption","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-08-18T20:01:52Z","receivedAt":"2009-08-18T20:01:52Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 18 Aug 2009, Jakub Narebski wrote:\n\n> Nicolas Pitre writes:\n> \n> > On Tue, 18 Aug 2009, Tomasz Kontusz wrote:\n> > \n> > > Ok, so it looks like it's not implementable without some kind of cache\n> > > server-side, so the server would know what the pack it was sending\n> > > looked like.\n> > > But here's my idea: make server send objects in different order (the\n> > > newest commit + whatever it points to first, then next one,then\n> > > another...). Then it would be possible to look at what we got, tell\n> > > server we have nothing, and want [the newest commit that was not\n> > > complete]. I know the reason why it is sorted the way it is, but I think\n> > > that the way data is stored after clone is clients problem, so the\n> > > client should reorganize packs the way it wants.\n> > \n> > That won't buy you much. You should realize that a pack is made of:\n> > \n> > 1) Commit objects. Yes they're all put together at the front of the pack,\n> > but they roughly are the equivalent of:\n> > \n> > \tgit log --pretty=raw | gzip | wc -c\n> > \n> > For the Linux repo as of now that is around 32 MB.\n> \n> For my clone of Git repository this gives 3.8 MB\n> \n> > 2) Tree and blob objects. Those are the bulk of the content for the top \n> > commit. The top commit is usually not delta compressed because we \n> > want fast access to the top commit, and that is used as the base for \n> > further delta compression for older commits. So the very first \n> > commit is whole at the front of the pack right after the commit \n> > objects. you can estimate the size of this data with:\n> > \n> > \tgit archive --format=tar HEAD | gzip | wc -c\n> > \n> > On the same Linux repo this is currently 75 MB.\n> \n> On the same Git repository this gives 2.5 MB\n\nInteresting to see that the commit history is larger than the latest \nsource tree. Probably that would be the same with the Linux kernel as \nwell if all versions since the beginning with adequate commit logs were \nincluded in the repo.\n\n> > 3) Delta objects. Those are making the rest of the pack, plus a couple \n> > tree/blob objects that were not found in the top commit and are \n> > different enough from any object in that top commit not to be \n> > represented as deltas. Still, the majority of objects for all the \n> > remaining commits are delta objects.\n> \n> You forgot that delta chains are bound by pack.depth limit, which\n> defaults to 50. You would have then additional full objects.\n\nSure, but that's probably not significant. the delta chain depth is \nlimited, but not the width. A given base object can have unlimited \ndelta \"children\", and so on at each depth level.\n\n> The single packfile for this (just gc'ed) Git repository is 37 MB.\n> Much more than 3.8 MB + 2.5 MB = 6.3 MB.\n\nWhat I'm saying is that most of that 37 MB - 6.3 MB = 31 MB is likely to \nbe occupied by deltas.\n\n> [cut]\n> \n> There is another way which we can go to implement resumable clone.\n> Let's git first try to clone whole repository (single pack; BTW what\n> happens if this pack is larger than file size limit for given\n> filesystem?).\n\nWe currently fail. Seems that no one ever had a problem with that so \nfar. We'd have to split the pack stream into multiple packs on the \nreceiving end. But frankly, if you have a repository large enough to \nbust your filesystem's file size limit then maybe you should seriously \nreconsider your choice of development environment.\n\n> If it fails, client ask first for first half of of\n> repository (half as in bisect, but it is server that has to calculate\n> it). If it downloads, it will ask server for the rest of repository.\n> If it fails, it would reduce size in half again, and ask about 1/4 of\n> repository in packfile first.\n\nProblem people with slow links have won't be helped at all with this. \nWhat if the network connection gets broken only after 49% of the \ntransfer and that took 3 hours to download? You'll attempt a 25% size \ntransfer which would take 1.5 hour despite the fact that you already \nspent that much time downloading that first 1/4 of the repository \nalready. And yet what if you're unlucky and now the network craps on \nyou after 23% of that second attempt?\n\nI think it is better to \"prime\" the repository with the content of the \ntop commit in the most straight forward manner using git-archive which \nhas the potential to be fully restartable at any point with little \ncomplexity on the server side.\n\n\nNicolas\n"},{"id":"121162","messageId":"200908182302.10619.jnareb@gmail.com","threadId":"20626","inReplyTo":"alpine.LFD.2.00.0908181537360.6044@xanadu.home","subject":"Re: Continue git clone after interruption","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-08-18T21:02:09Z","receivedAt":"2009-08-18T21:02:09Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Tue, 18 Aug 2009, Nicolas Pitre wrote:\n> On Tue, 18 Aug 2009, Jakub Narebski wrote:\n>> Nicolas Pitre writes:\n\n>>> That won't buy you much. You should realize that a pack is made of:\n>>> \n>>> 1) Commit objects. Yes they're all put together at the front of the pack,\n>>> but they roughly are the equivalent of:\n>>> \n>>> \tgit log --pretty=raw | gzip | wc -c\n>>> \n>>> For the Linux repo as of now that is around 32 MB.\n>> \n>> For my clone of Git repository this gives 3.8 MB\n>> \n>>> 2) Tree and blob objects. Those are the bulk of the content for the top \n>>> commit. [...] You can estimate the size of this data with:\n>>> \n>>> \tgit archive --format=tar HEAD | gzip | wc -c\n>>> \n>>> On the same Linux repo this is currently 75 MB.\n>> \n>> On the same Git repository this gives 2.5 MB\n> \n> Interesting to see that the commit history is larger than the latest \n> source tree. Probably that would be the same with the Linux kernel as \n> well if all versions since the beginning with adequate commit logs were \n> included in the repo.\n\nNote that having reflog and/or patch management interface like StGit,\nand frequently reworking commits (e.g. using rebase) means more commit\nobjects in repository.\n\nAlso Git repository has 3 independent branches: 'man', 'html' and 'todo',\nfrom whose branches objects are not included in \"git archive HEAD\".\n\n> \n>>> 3) Delta objects. Those are making the rest of the pack, plus a couple \n>>> tree/blob objects that were not found in the top commit and are \n>>> different enough from any object in that top commit not to be \n>>> represented as deltas. Still, the majority of objects for all the \n>>> remaining commits are delta objects.\n>> \n>> You forgot that delta chains are bound by pack.depth limit, which\n>> defaults to 50. You would have then additional full objects.\n> \n> Sure, but that's probably not significant. the delta chain depth is \n> limited, but not the width. A given base object can have unlimited \n> delta \"children\", and so on at each depth level.\n\nYou can probably get number and size taken by delta and non-delta (base)\nobjects in the packfile somehow. Neither \"git verify-pack -v \"\nnor contrib/stats/packinfo.pl did help me arrive at this data.\n\n>> The single packfile for this (just gc'ed) Git repository is 37 MB.\n>> Much more than 3.8 MB + 2.5 MB = 6.3 MB.\n> \n> What I'm saying is that most of that 37 MB - 6.3 MB = 31 MB is likely to \n> be occupied by deltas.\n\nTrue.\n \n>> [cut]\n>> \n>> There is another way which we can go to implement resumable clone.\n>> Let's git first try to clone whole repository (single pack; BTW what\n>> happens if this pack is larger than file size limit for given\n>> filesystem?).\n> \n> We currently fail. Seems that no one ever had a problem with that so \n> far. We'd have to split the pack stream into multiple packs on the \n> receiving end. But frankly, if you have a repository large enough to \n> bust your filesystem's file size limit then maybe you should seriously \n> reconsider your choice of development environment.\n\nDo we fail gracefully (with an error message), or does git crash then?\n\nIf I remember correctly FAT28^W FAT32 has maximum file size of 2 GB.\nFAT is often used on SSD, on USB drive. Although if you have 2 GB\npackfile, you are doing something wrong, or UGFWIINI (Using Git For\nWhat It Is Not Intended).\n \n>> If it fails, client ask first for first half of of\n>> repository (half as in bisect, but it is server that has to calculate\n>> it). If it downloads, it will ask server for the rest of repository.\n>> If it fails, it would reduce size in half again, and ask about 1/4 of\n>> repository in packfile first.\n> \n> Problem people with slow links have won't be helped at all with this. \n> What if the network connection gets broken only after 49% of the \n> transfer and that took 3 hours to download? You'll attempt a 25% size \n> transfer which would take 1.5 hour despite the fact that you already \n> spent that much time downloading that first 1/4 of the repository \n> already. And yet what if you're unlucky and now the network craps on \n> you after 23% of that second attempt?\n\nA modification then.\n\nFirst try ordinary clone. If it fails because network is unreliable,\ncheck how much we did download, and ask server for packfile of slightly\nsmaller size; this means that we are asking server for approximate pack\nsize limit, not for bisect-like partitioning revision list.\n\n> I think it is better to \"prime\" the repository with the content of the \n> top commit in the most straight forward manner using git-archive which \n> has the potential to be fully restartable at any point with little \n> complexity on the server side.\n\nBut didn't it make fully restartable 2.5 MB part out of 37 MB packfile?\n\nA question about pack protocol negotiation. If clients presents some\nobjects as \"have\", server can and does assume that client has all \nprerequisites for such objects, e.g. for tree objects that it has\nall objects for files and directories inside tree; for commit it means\nall ancestors and all objects in snapshot (have top tree, and its \nprerequisites). Do I understand this correctly?\n\nIf we have partial packfile which crashed during downloading, can we\nextract from it some full objects (including blobs)? Can we pass\ntree and blob objects as \"have\" to server, and is it taken into account?\nPerhaps instead of separate step of resumable-downloading of top commit\nobjects (in snapshot), we can pass to server what we did download in\nfull?\n\n\nBTW. because of compression it might be more difficult to resume \narchive creation in the middle, I think...\n\n-- \nJakub Narebski\nPoland\n"},{"id":"121164","messageId":"alpine.LFD.2.00.0908181711350.6044@xanadu.home","threadId":"20626","inReplyTo":"200908182302.10619.jnareb@gmail.com","subject":"Re: Continue git clone after interruption","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-08-18T21:32:35Z","receivedAt":"2009-08-18T21:32:35Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 18 Aug 2009, Jakub Narebski wrote:\n\n> You can probably get number and size taken by delta and non-delta (base)\n> objects in the packfile somehow. Neither \"git verify-pack -v \"\n> nor contrib/stats/packinfo.pl did help me arrive at this data.\n\nDocumentation for verify-pack says:\n\n|When specifying the -v option the format used is:\n|\n| SHA1 type size size-in-pack-file offset-in-packfile\n|\n|for objects that are not deltified in the pack, and\n|\n| SHA1 type size size-in-packfile offset-in-packfile depth base-SHA1\n|\n|for objects that are deltified.\n\nSo a simple script should be able to give you the answer.\n\n> >> (BTW what happens if this pack is larger than file size limit for \n> >> given filesystem?).\n> > \n> > We currently fail. Seems that no one ever had a problem with that so \n> > far. We'd have to split the pack stream into multiple packs on the \n> > receiving end. But frankly, if you have a repository large enough to \n> > bust your filesystem's file size limit then maybe you should seriously \n> > reconsider your choice of development environment.\n> \n> Do we fail gracefully (with an error message), or does git crash then?\n\nIf the filesystem is imposing the limit, it will likely return an error \non the write() call and we'll die(). If the machine has a too small \noff_t for the received pack then we also die(\"pack too large for current \ndefinition of off_t\").\n\n> If I remember correctly FAT28^W FAT32 has maximum file size of 2 GB.\n> FAT is often used on SSD, on USB drive. Although if you have 2 GB\n> packfile, you are doing something wrong, or UGFWIINI (Using Git For\n> What It Is Not Intended).\n\nHopefully you're not performing a 'git clone' off of a FAT filesystem. \nFor physical transport you may repack with the appropriate switches.\n\n> >> If it fails, client ask first for first half of of\n> >> repository (half as in bisect, but it is server that has to calculate\n> >> it). If it downloads, it will ask server for the rest of repository.\n> >> If it fails, it would reduce size in half again, and ask about 1/4 of\n> >> repository in packfile first.\n> > \n> > Problem people with slow links have won't be helped at all with this. \n> > What if the network connection gets broken only after 49% of the \n> > transfer and that took 3 hours to download? You'll attempt a 25% size \n> > transfer which would take 1.5 hour despite the fact that you already \n> > spent that much time downloading that first 1/4 of the repository \n> > already. And yet what if you're unlucky and now the network craps on \n> > you after 23% of that second attempt?\n> \n> A modification then.\n> \n> First try ordinary clone. If it fails because network is unreliable,\n> check how much we did download, and ask server for packfile of slightly\n> smaller size; this means that we are asking server for approximate pack\n> size limit, not for bisect-like partitioning revision list.\n\nIf the download didn't reach past the critical point (75 MB in my linux \nrepo example) then you cannot validate the received data and you've \nwasted that much bandwidth.\n\n> > I think it is better to \"prime\" the repository with the content of the \n> > top commit in the most straight forward manner using git-archive which \n> > has the potential to be fully restartable at any point with little \n> > complexity on the server side.\n> \n> But didn't it make fully restartable 2.5 MB part out of 37 MB packfile?\n\nThe front of the pack is the critical point. If you get enough to \ncreate the top commit then further transfers can be done incrementally \nwith only the deltas between each commits.\n\n> A question about pack protocol negotiation. If clients presents some\n> objects as \"have\", server can and does assume that client has all \n> prerequisites for such objects, e.g. for tree objects that it has\n> all objects for files and directories inside tree; for commit it means\n> all ancestors and all objects in snapshot (have top tree, and its \n> prerequisites). Do I understand this correctly?\n\nThat works only for commits.\n\n> If we have partial packfile which crashed during downloading, can we\n> extract from it some full objects (including blobs)? Can we pass\n> tree and blob objects as \"have\" to server, and is it taken into account?\n\nNo.\n\n> Perhaps instead of separate step of resumable-downloading of top commit\n> objects (in snapshot), we can pass to server what we did download in\n> full?\n\nSee above.\n\n> BTW. because of compression it might be more difficult to resume \n> archive creation in the middle, I think...\n\nWhy so? the tar+gzip format is streamable.\n\n\nNicolas\n"},{"id":"121179","messageId":"alpine.DEB.1.00.0908190022090.8306@pacific.mpi-cbg.de","threadId":"20626","inReplyTo":"alpine.LFD.2.00.0908181537360.6044@xanadu.home","subject":"Re: Continue git clone after interruption","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-08-18T22:28:26Z","receivedAt":"2009-08-18T22:28:26Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 18 Aug 2009, Nicolas Pitre wrote:\n\n> On Tue, 18 Aug 2009, Jakub Narebski wrote:\n> \n> > There is another way which we can go to implement resumable clone. \n> > Let's git first try to clone whole repository (single pack; BTW what \n> > happens if this pack is larger than file size limit for given \n> > filesystem?).\n> \n> We currently fail. Seems that no one ever had a problem with that so \n> far.\n\nThey just went away, most probably.\n\nBut seriously, I miss a very important idea in this discussion: we control \nthe Git source code. So we _can_ add a upload_pack feature that a client \ncan ask for after the first failed attempt.\n\nCiao,\nDscho\n"},{"id":"121189","messageId":"alpine.LFD.2.00.0908181936130.6044@xanadu.home","threadId":"20626","inReplyTo":"alpine.DEB.1.00.0908190022090.8306@pacific.mpi-cbg.de","subject":"Re: Continue git clone after interruption","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-08-18T23:40:27Z","receivedAt":"2009-08-18T23:40:27Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 19 Aug 2009, Johannes Schindelin wrote:\n\n> Hi,\n> \n> On Tue, 18 Aug 2009, Nicolas Pitre wrote:\n> \n> > On Tue, 18 Aug 2009, Jakub Narebski wrote:\n> > \n> > > There is another way which we can go to implement resumable clone. \n> > > Let's git first try to clone whole repository (single pack; BTW what \n> > > happens if this pack is larger than file size limit for given \n> > > filesystem?).\n> > \n> > We currently fail. Seems that no one ever had a problem with that so \n> > far.\n> \n> They just went away, most probably.\n\nMost probably they simply don't exist. I would be highly surprised \notherwise.\n\n> But seriously, I miss a very important idea in this discussion: we control \n> the Git source code. So we _can_ add a upload_pack feature that a client \n> can ask for after the first failed attempt.\n\nIndeed. So what do you think about my proposal? It was included in my \nfirst reply to this thread.\n\n\nNicolas\n"},{"id":"121210","messageId":"2e24e5b90908182142n16201ed4ua41408878664e353@mail.gmail.com","threadId":"20626","inReplyTo":"m3fxbpneqe.fsf@localhost.localdomain","subject":"Re: Continue git clone after interruption","fromName":"Sitaram Chamarty","fromEmail":"sitaramc@gmail.com","sentAt":"2009-08-19T04:42:56Z","receivedAt":"2009-08-19T04:42:56Z","isPatch":false,"sender":{"key":"sitaramc@gmail.com","avatar":"https://avatars.githubusercontent.com/u/43316?v=4"},"body":"On Wed, Aug 19, 2009 at 12:15 AM, Jakub Narebski wrote:\n> There is another way which we can go to implement resumable clone.\n> Let's git first try to clone whole repository (single pack; BTW what\n> happens if this pack is larger than file size limit for given\n> filesystem?).  If it fails, client ask first for first half of of\n> repository (half as in bisect, but it is server that has to calculate\n> it).  If it downloads, it will ask server for the rest of repository.\n> If it fails, it would reduce size in half again, and ask about 1/4 of\n> repository in packfile first.\n\nHow about an extension where the user can *ask* for a clone of a\nparticular HEAD to be sent to him as a git bundle? Or particular\nrevisions (say once a week) were kept as a single file git-bundle,\nmade available over HTTP -- easily restartable with byte-range -- and\nanyone who has bandwidth problems first gets that, then changes the\norigin remote URL and does a \"pull\" to get uptodate?\n\nI've done this manually a few times when sneakernet bandwidth was\nbetter than the normal kind, heh, but it seems to me the lowest impact\nsolution.\n\nYes you'd need some extra space on the server, but you keep only one\nbundle, and maybe replace it every week by cron. Should work fine\nright now, as is, with a wee bit of manual work by the user, and a\nquick cron entry on the server\n"},{"id":"121218","messageId":"alpine.DEB.1.00.0908190921360.5594@intel-tinevez-2-302","threadId":"20626","inReplyTo":"alpine.LFD.2.00.0908181936130.6044@xanadu.home","subject":"Re: Continue git clone after interruption","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-08-19T07:35:19Z","receivedAt":"2009-08-19T07:35:19Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 18 Aug 2009, Nicolas Pitre wrote:\n\n> On Wed, 19 Aug 2009, Johannes Schindelin wrote:\n> \n> > But seriously, I miss a very important idea in this discussion: we \n> > control the Git source code. So we _can_ add a upload_pack feature \n> > that a client can ask for after the first failed attempt.\n> \n> Indeed. So what do you think about my proposal? It was included in my \n> first reply to this thread.\n\nDid you not talk about an extension of the archive protocol? That's not \nwhat I meant. The archive protocol can be disabled for completely \ndifferent reasons than to prevent restartable clones.\n\nBut you brought up an important point: shallow repositories.\n\nNow, the problem, of course, is that if you cannot even get a single ref \n(shallow'ed to depth 0 -- which reminds me: I think I promised to fix \nthat, but I did not do that yet) due to intermittent network failures, you \nare borked, as you said.\n\nBut here comes an idea: together with Nguy�n's sparse series, it is \nconceivable that we support a shallow & narrow clone via the upload-pack \nprotocol (also making mithro happy). The problem with narrow clones was \nnot the pack generation side, that is done by a rev-list that can be \nlimited to certain paths. The problem was that we end up with missing \ntree objects. However, if we can make a sparse checkout, we can avoid \nthe problem.\n\nNote: this is not well thought-through, but just a brainstorm-like answer \nto your ideas.\n\nCiao,\nDscho \"who should shut up now and get some work done instead ;-)\"\n"},{"id":"121224","messageId":"fcaeb9bf0908190125u3e1bcaaap27e72a5784f4f9db@mail.gmail.com","threadId":"20626","inReplyTo":"alpine.DEB.1.00.0908190921360.5594@intel-tinevez-2-302","subject":"Re: Continue git clone after interruption","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2009-08-19T08:25:22Z","receivedAt":"2009-08-19T08:25:22Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Aug 19, 2009 at 2:35 PM, Johannes\nSchindelin wrote:\n> But here comes an idea: together with Nguy要's sparse series, it is\n\nFWIW, you can write \"Nguyen\" instead. It might save you one copy/paste\n(I take it you don't have a Vietnamese IM ;-)\n\n> conceivable that we support a shallow & narrow clone via the upload-pack\n> protocol (also making mithro happy).  The problem with narrow clones was\n> not the pack generation side, that is done by a rev-list that can be\n> limited to certain paths.  The problem was that we end up with missing\n> tree objects.  However, if we can make a sparse checkout, we can avoid\n> the problem.\n\nBut then git-fsck, git-archive... will die?\n-- \nDuy\n"},{"id":"121232","messageId":"alpine.DEB.1.00.0908191150330.5594@intel-tinevez-2-302","threadId":"20626","inReplyTo":"fcaeb9bf0908190125u3e1bcaaap27e72a5784f4f9db@mail.gmail.com","subject":"Re: Continue git clone after interruption","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-08-19T09:52:10Z","receivedAt":"2009-08-19T09:52:10Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 19 Aug 2009, Nguyen Thai Ngoc Duy wrote:\n\n> On Wed, Aug 19, 2009 at 2:35 PM, Johannes\n> Schindelin wrote:\n> > But here comes an idea: together with Nguy要's sparse series, it is\n> \n> FWIW, you can write \"Nguyen\" instead. It might save you one copy/paste \n> (I take it you don't have a Vietnamese IM ;-)\n\nFWIW I originally wrote Nguyễn (not that Chinese(?) character)... I look \nit up everytime I want to write your name by searching my address book for \n\"pclouds\". ;-)\n\n> > conceivable that we support a shallow & narrow clone via the \n> > upload-pack protocol (also making mithro happy).  The problem with \n> > narrow clones was not the pack generation side, that is done by a \n> > rev-list that can be limited to certain paths.  The problem was that \n> > we end up with missing tree objects.  However, if we can make a sparse \n> > checkout, we can avoid the problem.\n> \n> But then git-fsck, git-archive... will die?\n\nOh, but they should be made aware of the narrow clone, just like for \nshallow clones.\n\nCiao,\nDscho\n"},{"id":"121233","messageId":"200908191153.36006.jnareb@gmail.com","threadId":"20626","inReplyTo":"2e24e5b90908182142n16201ed4ua41408878664e353@mail.gmail.com","subject":"Re: Continue git clone after interruption","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-08-19T09:53:33Z","receivedAt":"2009-08-19T09:53:33Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Wed, Aug 19, 2009, Sitaram Chamarty wrote:\n> On Wed, Aug 19, 2009 at 12:15 AM, Jakub Narebski wrote:\n\n> > There is another way which we can go to implement resumable clone.\n> > Let's git first try to clone whole repository (single pack; BTW what\n> > happens if this pack is larger than file size limit for given\n> > filesystem?).  If it fails, client ask first for first half of of\n> > repository (half as in bisect, but it is server that has to calculate\n> > it).  If it downloads, it will ask server for the rest of repository.\n> > If it fails, it would reduce size in half again, and ask about 1/4 of\n> > repository in packfile first.\n> \n> How about an extension where the user can *ask* for a clone of a\n> particular HEAD to be sent to him as a git bundle? Or particular\n> revisions (say once a week) were kept as a single file git-bundle,\n> made available over HTTP -- easily restartable with byte-range -- and\n> anyone who has bandwidth problems first gets that, then changes the\n> origin remote URL and does a \"pull\" to get uptodate?\n> \n> I've done this manually a few times when sneakernet bandwidth was\n> better than the normal kind, heh, but it seems to me the lowest impact\n> solution.\n> \n> Yes you'd need some extra space on the server, but you keep only one\n> bundle, and maybe replace it every week by cron. Should work fine\n> right now, as is, with a wee bit of manual work by the user, and a\n> quick cron entry on the server\n\nThis is a good idea, i think, and it can be implemented with various\namount of effort and changes to git, and various amount of seamless\nintegration.\n\n1. Simplest solution: social (homepage). Not integrated at all.\n\n On projects homepage, the one where there is described where project\n repository is and how to get it, you add a link to most recent bundle\n (perhaps in addition to most recent snapshot). This bundle would be\n served as a static file via HTTP (and perhaps also FTP) by (any) web\n server that supports resuming (range requests). Or you can make\n server generate bundles on demand, only when they are first requested.\n\n Most recent might mean latest tagged release, or it might mean daily\n snapshot^W bundle.\n\n This solution could be integrated into gitweb, either by generic \n 'latest bundle' link in project's README.html (or in site's \n GITWEB_HOMETEXT, default indextext.html), or by having gitweb\n generate those links (and perhaps bundles as well) by itself.\n\n2. Seamless solution: 'bundle' or 'bundles' capability. Requires \n changes to both server and client.\n\n If server supports (advertises) 'bundle' capability, it can serve\n list of bundles (as HTTP / FTP / rsync URLs) either at client request,\n or after (or before) list of refs if client requests 'bundle' \n capability.\n\n If client has support for 'bundles' capability, it terminates \n connection to sshd or git-daemon, and does ordinary resumable HTTP\n fetch using libcurl. After bundle is downloaded fully, it clones\n from bundle, and does git-fetch with the same server as before,\n which would then have less to transfer. Client has also to handle\n situation where bundle download is interrupted, and do not do cleanup,\n allowing for \"git clone --continue\".\n\n3. Seamless solution: GitTorrent or its simplification: git mirror-sync.\n\n I think that GitTorrent (see http://git.or.cz/gitwiki/SoC2009Ideas)\n or even its simplification git-mirror-sync would include restartable\n cloning. It is even among its intended features. Also this would\n help to download faster via mirrors which can have faster and better\n network connection.\n\n But this would be most work.\n\nYou can implement solution 1. even now...\n-- \nJakub Narebski\nPoland\n"},{"id":"121255","messageId":"200908191719.52974.jnareb@gmail.com","threadId":"20626","inReplyTo":"alpine.LFD.2.00.0908181711350.6044@xanadu.home","subject":"Re: Continue git clone after interruption","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-08-19T15:19:50Z","receivedAt":"2009-08-19T15:19:50Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Tue, 18 Aug 2009, Nicolas Pitre wrote:\n> On Tue, 18 Aug 2009, Jakub Narebski wrote:\n> \n>> You can probably get number and size taken by delta and non-delta (base)\n>> objects in the packfile somehow. Neither \"git verify-pack -v \"\n>> nor contrib/stats/packinfo.pl did help me arrive at this data.\n> \n> Documentation for verify-pack says:\n> \n> |When specifying the -v option the format used is:\n> |\n> | SHA1 type size size-in-pack-file offset-in-packfile\n> |\n> |for objects that are not deltified in the pack, and\n> |\n> | SHA1 type size size-in-packfile offset-in-packfile depth base-SHA1\n> |\n> |for objects that are deltified.\n> \n> So a simple script should be able to give you the answer.\n\nThanks.\n\nThere are 114937 objects in this packfile, including 56249 objects\nused as base (can be deltified or not). git-verify-pack -v shows\nthat all objects have total size-in-packfile of 33 MB (which agrees\nwith packfile size of 33 MB), with 17 MB size-in-packfile taken by\ndeltaified objects, and 16 MB taken by base objects.\n\n git verify-pack -v | \n grep -v \"^chain\" | \n grep -v \"objects/pack/pack-\" > verify-pack.out\n\n sum=0; bsum=0; dsum=0; \n while read sha1 type size packsize off depth base; do\n echo \"$sha1\" >> verify-pack.sha1.out\n sum=$(( $sum + $packsize ))\n if [ -n \"$base\" ]; then \n echo \"$sha1\" >> verify-pack.delta.out\n dsum=$(( $dsum + $packsize ))\n else\n echo \"$sha1\" >> verify-pack.base.out\n bsum=$(( $bsum + $packsize ))\n fi\n done < verify-pack.out\n echo \"sum=$sum; bsum=$bsum; dsum=$dsum\"\n \n>>>> (BTW what happens if this pack is larger than file size limit for \n>>>> given filesystem?).\n[...]\n\n>> If I remember correctly FAT28^W FAT32 has maximum file size of 2 GB.\n>> FAT is often used on SSD, on USB drive. Although if you have 2 GB\n>> packfile, you are doing something wrong, or UGFWIINI (Using Git For\n>> What It Is Not Intended).\n> \n> Hopefully you're not performing a 'git clone' off of a FAT filesystem. \n> For physical transport you may repack with the appropriate switches.\n\nNot off a FAT filesystem, but into a FAT filesystem.\n \n[...]\n\n>>> I think it is better to \"prime\" the repository with the content of the \n>>> top commit in the most straight forward manner using git-archive which \n>>> has the potential to be fully restartable at any point with little \n>>> complexity on the server side.\n>> \n>> But didn't it make fully restartable 2.5 MB part out of 37 MB packfile?\n> \n> The front of the pack is the critical point. If you get enough to \n> create the top commit then further transfers can be done incrementally \n> with only the deltas between each commits.\n\nHow? You have some objects that can be used as base; how to tell \ngit-daemon that we have them (but not theirs prerequisites), and how\nto generate incrementals?\n\n>> A question about pack protocol negotiation. If clients presents some\n>> objects as \"have\", server can and does assume that client has all \n>> prerequisites for such objects, e.g. for tree objects that it has\n>> all objects for files and directories inside tree; for commit it means\n>> all ancestors and all objects in snapshot (have top tree, and its \n>> prerequisites). Do I understand this correctly?\n> \n> That works only for commits.\n\nHmmmm... how do you intent for \"prefetch top objects restartable-y first\"\nto work, then?\n \n>> BTW. because of compression it might be more difficult to resume \n>> archive creation in the middle, I think...\n> \n> Why so? the tar+gzip format is streamable.\n\ngzip format uses sliding window in compression. \"cat a b | gzip\"\nis different from \"cat <(gzip a) <(gzip b)\".\n\nBut that doesn't matter. If we are interrupted in the middle, we can\nuncompress what we have to check how far did we get, and tell server\nto send the rest; this way server wouldn't have to even generate \n(but not send) what we get as partial transfer.\n\nP.S. What do you think about 'bundle' capability extension mentioned\n in a side sub-thread?\n-- \nJakub Narebski\nPoland\n"},{"id":"121271","messageId":"alpine.LFD.2.00.0908191122020.6044@xanadu.home","threadId":"20626","inReplyTo":"alpine.DEB.1.00.0908190921360.5594@intel-tinevez-2-302","subject":"Re: Continue git clone after interruption","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-08-19T17:21:19Z","receivedAt":"2009-08-19T17:21:19Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 19 Aug 2009, Johannes Schindelin wrote:\n\n> Hi,\n> \n> On Tue, 18 Aug 2009, Nicolas Pitre wrote:\n> \n> > On Wed, 19 Aug 2009, Johannes Schindelin wrote:\n> > \n> > > But seriously, I miss a very important idea in this discussion: we \n> > > control the Git source code. So we _can_ add a upload_pack feature \n> > > that a client can ask for after the first failed attempt.\n> > \n> > Indeed. So what do you think about my proposal? It was included in my \n> > first reply to this thread.\n> \n> Did you not talk about an extension of the archive protocol? That's not \n> what I meant. The archive protocol can be disabled for completely \n> different reasons than to prevent restartable clones.\n\nAnd those reasons are?\n\n> But you brought up an important point: shallow repositories.\n> \n> Now, the problem, of course, is that if you cannot even get a single ref \n> (shallow'ed to depth 0 -- which reminds me: I think I promised to fix \n> that, but I did not do that yet) due to intermittent network failures, you \n> are borked, as you said.\n\nExact.\n\n> But here comes an idea: together with Nguy�n's sparse series, it is \n> conceivable that we support a shallow & narrow clone via the upload-pack \n> protocol (also making mithro happy). The problem with narrow clones was \n> not the pack generation side, that is done by a rev-list that can be \n> limited to certain paths. The problem was that we end up with missing \n> tree objects. However, if we can make a sparse checkout, we can avoid \n> the problem.\n\nSure, if you can salvage as much as you can from a partial pack and \ncreate a shallow and narrow clone out of it then it should be possible \nto do some restartable clone. I still think this might be much less \ncomplex to achieve through git-archive, especially if some files i.e. \nobjects are large enough to expose themselves to network outage. It is \nlike the same issue as being able to fetch at least one revision but to \na lesser degree. You might be able to get that first revision through \nmultiple attempts by gathering missing objects on each attempt. But if \nyou encounter an object large enough you then might be unlucky enough \nnot to be able to transfer it all before the next network failure.\n\nWith a simple extension to git-archive, any object content could be \nresumed many times from any offset. Then, deepening the history should \nmake use of deltas through the pack protocol which should hopefully \nconsist of much smaller transfers and therefore less prone to network \noutage.\n\nThat could be sketched like this, supposing user runs\n\"git clone git://foo.bar/baz\":\n\n1) \"git ini baz\" etc. as usual.\n\n2) \"git ls-remote git://foo.bar/baz HEAD\" and store the result in\n .git/CLONE_HEAD so not to be confused by the remote HEAD possibly \n changing before we're done.\n\n3) \"git archive --remote=git://foo.bar/baz CLONE_HEAD\" and store the \n result locally. Keep track of how many files are received, and how \n many bytes for the currently received file.\n\n4) if network connection is broken, loop back to (3) adding\n --skip=${nr_files_received},${nr_bytes_in_curr_file_received} to\n the git-archive argument list. REmote server simply skips over \n specified number of files and bytes into the next file.\n\n5) Get content from remote commit object for CLONE_HEAD somehow. (?)\n\n6) \"git add . && git write-tree\" and make sure the top tree SHA1 matches \n the one in the commit from (5).\n\n7) \"git hash-object -w -t commit\" with data obtained in (5), and make \n sure it matches SHA1 from CLONE_HEAD.\n\n8) Update local HEAD with CLONE_HEAD and set it up as a shallow clone.\n Delete .git/CLONE_HEAD.\n\n9) Run \"git fetch\" with the --depth parameter to get more revisions.\n\nNotes:\n\n- This mode of operation should probably be optional, like by using \n --safe or --restartable with 'git clone'. And since this mode of \n operation is really meant for people with slow and unreliable network \n connections, they're unlikely to wish for the whole history to be \n fetched. Hence this mode could simply be triggered by the --depth \n parameter to 'git clone' which would provide a clear depth value to \n use in (9).\n\n- If the transfer is interrupted locally with ^C then it should be \n possible to resume it by noticing the presence of .git/CLONE_HEAD\n up front. DEtermining how many files to skip when resuming with \n git-archive can be done with $((`git ls-files -o | wc -l` - 1)) and\n $(git ls-files -o | tail -1 | wc -c).\n\n- That probably would be a good idea to have a tgz format to 'git \n archive' which might be simpler to deal with than the zip format.\n\n- Step (3) could be optimized in many ways, like by directly using \n hash-object and update-index, or by using a filter to pipe the result \n directly into fast-import.\n\n- So to say that the above should be pretty easy to implement even \n with a shell script. A builtin version could then be made if this \n proves to actually be useful. And the server remains stateless with \n no additional caching needed which would go against any attempt \n at making a busy server like git.kernel.org share as much of the \n object store between plenty of mostly identical repositoryes.\n\n> Note: this is not well thought-through, but just a brainstorm-like answer \n> to your ideas.\n\nAnd so is the above.\n\n\nNicolas\n"},{"id":"121279","messageId":"alpine.LFD.2.00.0908191326360.6044@xanadu.home","threadId":"20626","inReplyTo":"200908191719.52974.jnareb@gmail.com","subject":"Re: Continue git clone after interruption","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-08-19T19:04:59Z","receivedAt":"2009-08-19T19:04:59Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 19 Aug 2009, Jakub Narebski wrote:\n\n> There are 114937 objects in this packfile, including 56249 objects\n> used as base (can be deltified or not). git-verify-pack -v shows\n> that all objects have total size-in-packfile of 33 MB (which agrees\n> with packfile size of 33 MB), with 17 MB size-in-packfile taken by\n> deltaified objects, and 16 MB taken by base objects.\n> \n> git verify-pack -v | \n> grep -v \"^chain\" | \n> grep -v \"objects/pack/pack-\" > verify-pack.out\n> \n> sum=0; bsum=0; dsum=0; \n> while read sha1 type size packsize off depth base; do\n> echo \"$sha1\" >> verify-pack.sha1.out\n> sum=$(( $sum + $packsize ))\n> if [ -n \"$base\" ]; then \n> echo \"$sha1\" >> verify-pack.delta.out\n> dsum=$(( $dsum + $packsize ))\n> else\n> echo \"$sha1\" >> verify-pack.base.out\n> bsum=$(( $bsum + $packsize ))\n> fi\n> done < verify-pack.out\n> echo \"sum=$sum; bsum=$bsum; dsum=$dsum\"\n\nYour object classification is misleading. Because an object has no \nbase, that doesn't mean it is necessarily a base itself. You'd have to \nstore $base into a separate file and then sort it and remove duplicates \nto know the actual number of base objects. What you have right now is \nstrictly delta objects and non-delta objects. And base objects can \nthemselves be delta objects already of course.\n\nAlso... my git repo after 'git gc --aggressive' contains a pack which \nsize is 22 MB. Your script tells me:\n\nsum=22930254; bsum=14142012; dsum=8788242\n\nand:\n\n 29558 verify-pack.base.out\n 82043 verify-pack.delta.out\n 111601 verify-pack.out\n 111601 verify-pack.sha1.out\n\nmeaning that I have 111601 total objects, of which 29558 are non-deltas \noccupying 14 MB and 82043 are deltas occupying 8 MB. That certainly \nshows how deltas are space efficient. And with a minor modification to \nyour script, I know that 44985 objects are actually used as a delta \nbase. So, on average, each base is responsible for nearly 2 deltas.\n\n> >>>> (BTW what happens if this pack is larger than file size limit for \n> >>>> given filesystem?).\n> [...]\n> \n> >> If I remember correctly FAT28^W FAT32 has maximum file size of 2 GB.\n> >> FAT is often used on SSD, on USB drive. Although if you have 2 GB\n> >> packfile, you are doing something wrong, or UGFWIINI (Using Git For\n> >> What It Is Not Intended).\n> > \n> > Hopefully you're not performing a 'git clone' off of a FAT filesystem. \n> > For physical transport you may repack with the appropriate switches.\n> \n> Not off a FAT filesystem, but into a FAT filesystem.\n\nThat's what I meant, sorry. My point still stands.\n\n> > The front of the pack is the critical point. If you get enough to \n> > create the top commit then further transfers can be done incrementally \n> > with only the deltas between each commits.\n> \n> How? You have some objects that can be used as base; how to tell \n> git-daemon that we have them (but not theirs prerequisites), and how\n> to generate incrementals?\n\nJust the same as when you perform a fetch to update your local copy of a \nremote branch: you tell the remote about the commit you have and the one \nyou want, and git-repack will create delta objects for the commit you \nwant against similar objects from the commit you already have, and skip \nthose objects from the commit you want that are already included in the \ncommit you have.\n\n> >> A question about pack protocol negotiation. If clients presents some\n> >> objects as \"have\", server can and does assume that client has all \n> >> prerequisites for such objects, e.g. for tree objects that it has\n> >> all objects for files and directories inside tree; for commit it means\n> >> all ancestors and all objects in snapshot (have top tree, and its \n> >> prerequisites). Do I understand this correctly?\n> > \n> > That works only for commits.\n> \n> Hmmmm... how do you intent for \"prefetch top objects restartable-y first\"\n> to work, then?\n\nSee my latest reply to dscho (you were in CC already).\n\n> >> BTW. because of compression it might be more difficult to resume \n> >> archive creation in the middle, I think...\n> > \n> > Why so? the tar+gzip format is streamable.\n> \n> gzip format uses sliding window in compression. \"cat a b | gzip\"\n> is different from \"cat <(gzip a) <(gzip b)\".\n> \n> But that doesn't matter. If we are interrupted in the middle, we can\n> uncompress what we have to check how far did we get, and tell server\n> to send the rest; this way server wouldn't have to even generate \n> (but not send) what we get as partial transfer.\n\nYou got it.\n\n> P.S. What do you think about 'bundle' capability extension mentioned\n> in a side sub-thread?\n\nI don't like it. Reason is that it forces the server to be (somewhat) \nstateful by having to keep track of those bundles and cycle them, and it \ndoubles the disk usage by having one copy of the repository in the form \nof the original pack(s) and another copy as a bundle.\n\nOf course, the idea of having a cron job generating a bundle and \noffering it for download through HTTP or the like is fine if people are \nOK with that, and that requires zero modifications to git. But I don't \nthink that is a solution that scales.\n\nIf you think about git.kernel.org which has maybe hundreds of \nrepositories where the big majority of them are actually forks of Linus' \nown repository, then having all those forks reference Linus' repository \nis a big disk space saver (and IO too as the referenced repository is \nlikely to remain cached in memory). Having a bundle ready for each of \nthem will simply kill that space advantage, unless they all share the \nsame bundle.\n\nNow sharing that common bundle could be done of course, but that makes \nthings yet more complex while still wasting IO because some requests \nwill hit the common pack and some others will hit the bundle, making \nless efficient usage of the disk cache on the server.\n\nYet, that bundle would probably not contain the latest revision if it is \nonly periodically updated, even less so if it is shared between multiple \nrepositories as outlined above. And what people with slow/unreliable \nnetwork links are probably most interested in is the latest revision and \nmaybe a few older revisions, but probably not the whole repository as \nthat is simply too long to wait for. Hence having a big bundle is not \nflexible either with regards to the actual data transfer size.\n\nHence having a restartable git-archive service to create the top \nrevision with the ability to cheaply (in terms of network bandwidth) \ndeepen the history afterwards is probably the most straight forward way \nto achieve that. The server needs no be aware of separate bundles, etc. \nAnd the shared object store still works as usual with the same cached IO \nwhether the data is needed for a traditional fetch or a \"git archive\" \noperation.\n\nWhy \"git archive\"? Because its content is well defined. So if you give \nit a commit SHA1 you will always get the same stream of bytes (after \ndecompression) since the way git sort files is strictly defined. It is \ntherefore easy to tell a remote \"git archive\" instance that we want the \ncontent for commit xyz but that we already got n files already, and that \nthe last file we've got has m bytes. There is simply no confusion about \nwhat we've got already, unlike with a partial pack which might need \nyet-to-be-received objects in order to make sense of what has been \nalready received. The server simply has to skip that many files and \nresume the transfer at that point, independently of the compression or \neven the archive format.\n\n\nNicolas\n"},{"id":"121282","messageId":"200908192142.51384.jnareb@gmail.com","threadId":"20626","inReplyTo":"alpine.LFD.2.00.0908191326360.6044@xanadu.home","subject":"Re: Continue git clone after interruption","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-08-19T19:42:50Z","receivedAt":"2009-08-19T19:42:50Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Cc-ed Dscho, so he can easier participate in this subthread.\n\nOn Wed, 19 Aug 2009, Nicolas Pitre wrote:\n> On Wed, 19 Aug 2009, Jakub Narebski wrote:\n\n> > P.S. What do you think about 'bundle' capability extension mentioned\n> > in a side sub-thread?\n> \n> I don't like it. Reason is that it forces the server to be (somewhat) \n> stateful by having to keep track of those bundles and cycle them, and it \n> doubles the disk usage by having one copy of the repository in the form \n> of the original pack(s) and another copy as a bundle.\n\nI agree about problems with disk usage, but I disagree about server\nhaving to be stateful; server can just simply scan for bundles, and\noffer links to them if client requests 'bundles' capability, somewhere\naround initial git-ls-remote list of refs.\n\n> Of course, the idea of having a cron job generating a bundle and \n> offering it for download through HTTP or the like is fine if people are \n> OK with that, and that requires zero modifications to git. But I don't \n> think that is a solution that scales.\n\nWell, offering daily bundle in addition to daily snapshot could be\na good practice, at least until git acquires resumable fetch (resumable\nclone).\n\n> \n> If you think about git.kernel.org which has maybe hundreds of \n> repositories where the big majority of them are actually forks of Linus' \n> own repository, then having all those forks reference Linus' repository \n> is a big disk space saver (and IO too as the referenced repository is \n> likely to remain cached in memory). Having a bundle ready for each of \n> them will simply kill that space advantage, unless they all share the \n> same bundle.\n\nI am thinking about sharing the same bundle for related projects.\n\n> \n> Now sharing that common bundle could be done of course, but that makes \n> things yet more complex while still wasting IO because some requests \n> will hit the common pack and some others will hit the bundle, making \n> less efficient usage of the disk cache on the server.\n\nHmmm... true (unless bundles are on separate server).\n\n> \n> Yet, that bundle would probably not contain the latest revision if it is \n> only periodically updated, even less so if it is shared between multiple \n> repositories as outlined above. And what people with slow/unreliable \n> network links are probably most interested in is the latest revision and \n> maybe a few older revisions, but probably not the whole repository as \n> that is simply too long to wait for. Hence having a big bundle is not \n> flexible either with regards to the actual data transfer size.\n\nI agree that bundle would be useful for restartable clone, and not\nuseful for restartable fetch. Well, unless you count (non-existing)\nGitTorrent / git-mirror-sync as this solution... ;-)\n\n> \n> Hence having a restartable git-archive service to create the top \n> revision with the ability to cheaply (in terms of network bandwidth) \n> deepen the history afterwards is probably the most straight forward way \n> to achieve that. The server needs no be aware of separate bundles, etc. \n> And the shared object store still works as usual with the same cached IO \n> whether the data is needed for a traditional fetch or a \"git archive\" \n> operation.\n\nIt's the \"cheaply deepen history\" that I doubt would be easy. This is\nthe most difficult part, I think (see also below).\n\n> \n> Why \"git archive\"? Because its content is well defined. So if you give \n> it a commit SHA1 you will always get the same stream of bytes (after \n> decompression) since the way git sort files is strictly defined. It is \n> therefore easy to tell a remote \"git archive\" instance that we want the \n> content for commit xyz but that we already got n files already, and that \n> the last file we've got has m bytes. There is simply no confusion about \n> what we've got already, unlike with a partial pack which might need \n> yet-to-be-received objects in order to make sense of what has been \n> already received. The server simply has to skip that many files and \n> resume the transfer at that point, independently of the compression or \n> even the archive format.\n\nLet's reiterate it to check if I understand it correctly:\n\n\nAny \"restartable clone\" / \"resumable fetch\" solution must begin with\na file which is rock-solid stable wrt. reproductability given the same\nparameters. git-archive has this feature, packfile doesn't (so I guess\nthat bundle also doesn't, unless it was cached / saved on disk).\n\nIt would be useful if it was possible to generate part of this rock-solid\nfile for partial (range, resume) request, without need to generate \n(calculate) parts that client already downloaded. Otherwise server has\nto either waste disk space and IO for caching, or waste CPU (and IO)\non generating part which is not needed and dropping it to /dev/null.\ngit-archive you say has this feature.\n\nNext you need to tell server that you have those objects got using\nresumable download part (\"git archive HEAD\" in your proposal), and\nthat it can use them and do not include them in prepared file/pack.\n\"have\" is limited to commits, and \"have \" tells server that\nyou have and all its prerequisites (dependences). You can't \nuse \"have \" with git-archive solution. I don't know enough\nabout 'shallow' capability (and what it enables) to know whether\nit can be used for that. Can you elaborate?\n\nThen you have to finish clone / fetch. All solutions so far include\nsome kind of incremental improvements. My first proposal of bisect\nfetching 1/nth or predefined size pack is buttom-up solution, where\nwe build full clone from root commits up. You propose, from what\nI understand build full clone from top commit down, using deepening\nfrom shallow clone. In this step you either get full incremental\nor not; downloading incremental (from what I understand) is not\nresumable / they do not support partial fetch.\n\nDo I understand this correctly?\n-- \nJakub Narebski\nPoland\n"},{"id":"121289","messageId":"alpine.LFD.2.00.0908191552020.6044@xanadu.home","threadId":"20626","inReplyTo":"200908192142.51384.jnareb@gmail.com","subject":"Re: Continue git clone after interruption","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-08-19T21:13:59Z","receivedAt":"2009-08-19T21:13:59Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 19 Aug 2009, Jakub Narebski wrote:\n\n> Cc-ed Dscho, so he can easier participate in this subthread.\n> \n> On Wed, 19 Aug 2009, Nicolas Pitre wrote:\n> > On Wed, 19 Aug 2009, Jakub Narebski wrote:\n> \n> > > P.S. What do you think about 'bundle' capability extension mentioned\n> > > in a side sub-thread?\n> > \n> > I don't like it. Reason is that it forces the server to be (somewhat) \n> > stateful by having to keep track of those bundles and cycle them, and it \n> > doubles the disk usage by having one copy of the repository in the form \n> > of the original pack(s) and another copy as a bundle.\n> \n> I agree about problems with disk usage, but I disagree about server\n> having to be stateful; server can just simply scan for bundles, and\n> offer links to them if client requests 'bundles' capability, somewhere\n> around initial git-ls-remote list of refs.\n\nBut that's the client that has to deal with what the server wants to \noffer, instead of the server actually serving data as the client wants.\n\n> Well, offering daily bundle in addition to daily snapshot could be\n> a good practice, at least until git acquires resumable fetch (resumable\n> clone).\n\nOutside of Git: maybe. Through the git protocol: no. And what would \nthat bundle contain over the daily snapshot? The whole history? If so \nthat goes against the idea that people concerned by all this have slow \nlinks and probably aren't interested in the time to download it all. If \nthe bundle contains only the top revision then it has no advantage over \nthe snapshot. Somewhere in the middle? Sure, but then where to draw \nthe line? That's for the client to decide, not the server \nadministrator.\n\nAnd what if you start your slow transfer which breaks in the middle. \nThe next morning you want to restart it in the hope that you might \nresume the transfer of the bundle that is incomplete. But crap, the \nserver has updated its bundle and your half-bundle is now useless. \nYou've wasted your bandwidth for nothing.\n\n> > If you think about git.kernel.org which has maybe hundreds of \n> > repositories where the big majority of them are actually forks of Linus' \n> > own repository, then having all those forks reference Linus' repository \n> > is a big disk space saver (and IO too as the referenced repository is \n> > likely to remain cached in memory). Having a bundle ready for each of \n> > them will simply kill that space advantage, unless they all share the \n> > same bundle.\n> \n> I am thinking about sharing the same bundle for related projects.\n\n... meaning more administrative burden.\n\n> > Now sharing that common bundle could be done of course, but that makes \n> > things yet more complex while still wasting IO because some requests \n> > will hit the common pack and some others will hit the bundle, making \n> > less efficient usage of the disk cache on the server.\n> \n> Hmmm... true (unless bundles are on separate server).\n\n... meaning additional but avoidable costs.\n\n> > Yet, that bundle would probably not contain the latest revision if it is \n> > only periodically updated, even less so if it is shared between multiple \n> > repositories as outlined above. And what people with slow/unreliable \n> > network links are probably most interested in is the latest revision and \n> > maybe a few older revisions, but probably not the whole repository as \n> > that is simply too long to wait for. Hence having a big bundle is not \n> > flexible either with regards to the actual data transfer size.\n> \n> I agree that bundle would be useful for restartable clone, and not\n> useful for restartable fetch. Well, unless you count (non-existing)\n> GitTorrent / git-mirror-sync as this solution... ;-)\n\nI don't think fetches after a clone are such an issue. They are \ntypically transfers being orders of magnitude smaller than the initial \nclone. Same goes for fetches to deepen a shallow clone which are in \nfact fetches going back in history instead of forward. I still stands \nby my assertion that bundles are suboptimal for a restartable clone.\n\nAs for GitTorrent / git-mirror-sync... those are still vaporwares to me \nand I therefore have doubts about their actual feasability. So no, I \ndon't count on them.\n\n> > Hence having a restartable git-archive service to create the top \n> > revision with the ability to cheaply (in terms of network bandwidth) \n> > deepen the history afterwards is probably the most straight forward way \n> > to achieve that. The server needs no be aware of separate bundles, etc. \n> > And the shared object store still works as usual with the same cached IO \n> > whether the data is needed for a traditional fetch or a \"git archive\" \n> > operation.\n> \n> It's the \"cheaply deepen history\" that I doubt would be easy. This is\n> the most difficult part, I think (see also below).\n\nDon't think so. Try this:\n\n\tmkdir test\n\tcd test\n\tgit init\n\tgit fetch --depth=1 git://git.kernel.org/pub/scm/git/git.git\n\nREsult:\n\nremote: Counting objects: 1824, done.\nremote: Compressing objects: 100% (1575/1575), done.\nReceiving objects: 100% (1824/1824), 3.01 MiB | 975 KiB/s, done.\nremote: Total 1824 (delta 299), reused 1165 (delta 180)\nResolving deltas: 100% (299/299), done.\nFrom git://git.kernel.org/pub/scm/git/git\n * branch HEAD -> FETCH_HEAD\n\nYou'll get the very latest revision for HEAD, and only that. The size \nof the transfer will be roughly the size of a daily snapshot, except it \nis fully up to date. It is however non resumable in the event of a \nnetwork outage. My proposal is to replace this with a \"git archive\" \ncall. It won't get all branches, but for the purpose of initialising \none's repository that should be good enough. And the \"git archive\" can \nbe fully resumable as I explained.\n\nNow to deepen that history. Let's say you want 10 more revisions going \nback then you simply perform the fetch again with a --depth=10. Right \nnow it doesn't seem to work optimally, but the pack that is then being \nsent could be made of deltas against objects found in the commits we \nalready have. Currently it seems that a pack that also includes those \nobjects we already have in addition to those we want is created, which \nis IMHO a flaw in the shallow support that shouldn't be too hard to fix. \nEach level of deepening should then be as small as standard fetches \ngoing forward when updating the repository with new revisions.\n\n> > Why \"git archive\"? Because its content is well defined. So if you give \n> > it a commit SHA1 you will always get the same stream of bytes (after \n> > decompression) since the way git sort files is strictly defined. It is \n> > therefore easy to tell a remote \"git archive\" instance that we want the \n> > content for commit xyz but that we already got n files already, and that \n> > the last file we've got has m bytes. There is simply no confusion about \n> > what we've got already, unlike with a partial pack which might need \n> > yet-to-be-received objects in order to make sense of what has been \n> > already received. The server simply has to skip that many files and \n> > resume the transfer at that point, independently of the compression or \n> > even the archive format.\n> \n> Let's reiterate it to check if I understand it correctly:\n> \n> Any \"restartable clone\" / \"resumable fetch\" solution must begin with\n> a file which is rock-solid stable wrt. reproductability given the same\n> parameters. git-archive has this feature, packfile doesn't (so I guess\n> that bundle also doesn't, unless it was cached / saved on disk).\n\nRight.\n\n> It would be useful if it was possible to generate part of this rock-solid\n> file for partial (range, resume) request, without need to generate \n> (calculate) parts that client already downloaded. Otherwise server has\n> to either waste disk space and IO for caching, or waste CPU (and IO)\n> on generating part which is not needed and dropping it to /dev/null.\n> git-archive you say has this feature.\n\n\"Could easily have\" is more appropriate.\n\n> Next you need to tell server that you have those objects got using\n> resumable download part (\"git archive HEAD\" in your proposal), and\n> that it can use them and do not include them in prepared file/pack.\n> \"have\" is limited to commits, and \"have \" tells server that\n> you have and all its prerequisites (dependences). You can't \n> use \"have \" with git-archive solution. I don't know enough\n> about 'shallow' capability (and what it enables) to know whether\n> it can be used for that. Can you elaborate?\n\nSee above, or Documentation/technical/shallow.txt.\n\n> Then you have to finish clone / fetch. All solutions so far include\n> some kind of incremental improvements. My first proposal of bisect\n> fetching 1/nth or predefined size pack is buttom-up solution, where\n> we build full clone from root commits up. You propose, from what\n> I understand build full clone from top commit down, using deepening\n> from shallow clone. In this step you either get full incremental\n> or not; downloading incremental (from what I understand) is not\n> resumable / they do not support partial fetch.\n\nRight. However, like I said, the incremental part should be much \nsmaller and therefore less susceptible to network troubles.\n\n\nNicolas\n"},{"id":"121291","messageId":"4A8C7B7A.6070700@lsrfire.ath.cx","threadId":"20626","inReplyTo":"alpine.LFD.2.00.0908191122020.6044@xanadu.home","subject":"Re: Continue git clone after interruption","fromName":"=?VISCII?Q?Ren=E9_Scharfe?=","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2009-08-19T22:23:54Z","receivedAt":"2009-08-19T22:23:54Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Nicolas Pitre schrieb:\n> 3) \"git archive --remote=git://foo.bar/baz CLONE_HEAD\" and store the \n> result locally. Keep track of how many files are received, and how \n> many bytes for the currently received file.\n> \n> 4) if network connection is broken, loop back to (3) adding\n> --skip=${nr_files_received},${nr_bytes_in_curr_file_received} to\n> the git-archive argument list. REmote server simply skips over \n> specified number of files and bytes into the next file.\n> \n> 5) Get content from remote commit object for CLONE_HEAD somehow. (?)\n\n[...]\n\n> - That probably would be a good idea to have a tgz format to 'git\n> archive' which might be simpler to deal with than the zip format.\n\nAdding support for the tgz format would be useful anyway, I guess, and\nis easy to implement.\n\nAnd adding support for cpio (and cpio.gz) and writing an extractor for\nit should be simpler than writing a tar extractor alone.\n\nOne needs to take a closer look at the limits of the chosen archive\nformat (file name length, supported file types and attributes, etc.) to\nmake sure any archive can be turned back into the same git tree.\n\nThe commit object could be sent as the first (fake) file of the archive.\n\nYou'd need a way to turn off the effect of the attributes export-subst\nand export-ignore.\n\nCurrently, convert_to_working_tree() is used on the contents of all\nfiles in an archive. You'd need a way to turn that off, too.\n\nAdding a new format type is probably the easiest way to bundle the\nspecial requirements of the previous three paragraphs.\n\nRen�\n"},{"id":"121304","messageId":"1250727998.5867.185.camel@maia.lan","threadId":"20626","inReplyTo":"alpine.LFD.2.00.0908191552020.6044@xanadu.home","subject":"Re: Continue git clone after interruption","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2009-08-20T00:26:38Z","receivedAt":"2009-08-20T00:26:38Z","isPatch":false,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"On Wed, 2009-08-19 at 17:13 -0400, Nicolas Pitre wrote:\n> > It's the \"cheaply deepen history\" that I doubt would be easy. This is\n> > the most difficult part, I think (see also below).\n> \n> Don't think so. Try this:\n> \n> \tmkdir test\n> \tcd test\n> \tgit init\n> \tgit fetch --depth=1 git://git.kernel.org/pub/scm/git/git.git\n> \n> REsult:\n> \n> remote: Counting objects: 1824, done.\n> remote: Compressing objects: 100% (1575/1575), done.\n> Receiving objects: 100% (1824/1824), 3.01 MiB | 975 KiB/s, done.\n> remote: Total 1824 (delta 299), reused 1165 (delta 180)\n> Resolving deltas: 100% (299/299), done.\n> From git://git.kernel.org/pub/scm/git/git\n> * branch HEAD -> FETCH_HEAD\n> \n> You'll get the very latest revision for HEAD, and only that. The size \n> of the transfer will be roughly the size of a daily snapshot, except it \n> is fully up to date. It is however non resumable in the event of a \n> network outage. My proposal is to replace this with a \"git archive\" \n> call. It won't get all branches, but for the purpose of initialising \n> one's repository that should be good enough. And the \"git archive\" can \n> be fully resumable as I explained.\n> \n> Now to deepen that history. Let's say you want 10 more revisions going \n> back then you simply perform the fetch again with a --depth=10. Right \n> now it doesn't seem to work optimally, but the pack that is then being \n> sent could be made of deltas against objects found in the commits we \n> already have. Currently it seems that a pack that also includes those \n> objects we already have in addition to those we want is created, which \n> is IMHO a flaw in the shallow support that shouldn't be too hard to fix. \n> Each level of deepening should then be as small as standard fetches \n> going forward when updating the repository with new revisions.\n\nNicholas, apart from starting with most recent commits and working\nbackwards, this is very similar to the \"bundle slicing\" idea defined in\nGitTorrent. What the GitTorrent research project has so far achieved is\ndefining a slicing algorithm, and figuring out how well slicing works,\nin terms of wasted bandwidth.\n\nIf you do it right, then you can support download spreading across\nmirrors, too. Eg, given a starting point, a 'slice size' - which I\nbased on uncompressed object size but could as well be based on commit\ncount - and a slice number to fetch, you should be able to look up in\nthe revision list index the revisions to select and then make a thin\npack corresponding to those commits. Currently creating this index is\nthe slowest part of creating bundle fragments in my Perl implementation.\n\nOnce Nick Edelen's project is mergeable, we have a mechanism for being\nable to relatively quickly draw a manifest of objects for these slices.\n\nSo how much bandwidth is lost?\n\nEg, for git.git, taking the complete object list, slicing it into 1024k\n(uncompressed) bundle slices, and making thin packs from those slices we\nget:\n\nGenerating index...\nLength is 1291327524, 1232 blocks\nSlice #0: 1050390 => 120406 (11%)\nSlice #1: 1058162 => 124978 (11%)\nSlice #2: 1049858 => 104363 (9%)\n...\nSlice #51: 1105090 => 43140 (3%)\nSlice #52: 1091282 => 45367 (4%)\nSlice #53: 1067675 => 39792 (3%)\n...\nSlice #211: 1086238 => 25451 (2%)\nSlice #212: 1055705 => 31294 (2%)\nSlice #213: 1059460 => 7767 (0%)\n...\nSlice #1129: 1109209 => 38182 (3%)\nSlice #1130: 1125925 => 29829 (2%)\nSlice #1131: 1120203 => 14446 (1%)\nFinal slice: 623055 => 49345\nOverall compressed: 39585851\nCalculating Repository bundle size...\nCounting objects: 107369, done.\nCompressing objects: 100% (28059/28059), done.\nWriting objects: 100% (107369/107369), 29.20 MiB | 48321 KiB/s, done.\nTotal 107369 (delta 78185), reused 106770 (delta 77609)\nBundle size: 30638967\nOverall inefficiency: 29%\n\nIn the above output, the first figure is the complete un-delta'd,\nuncompressed size of the slice - that is, the size of all of the new\nobjects that the commit introduces. The second figure is the full size\nof a thin pack with those objects in it. ie the above tells me that in\ngit.git there are 1.2GB of uncompressed objects. Each slice ends up\nvarying in size between about 10k and 200k, but most of the slices end\nup between 15k and 50k.\n\nActually the test script was thrown off by a loose root and that added\nabout 3MB to the compressed size, so the overall inefficiency with this\nblock size is actually more like 20%. I think I am running into the\nflaw you mention above, too, especially when I do a larger block size\nrun:\n\nGenerating index...\nLength is 1291327524, 62 blocks\nSlice #0: 21000218 => 1316165 (6%)\nSlice #1: 20988208 => 1107636 (5%)\n...\nSlice #59: 21102776 => 1387722 (6%)\nSlice #60: 20974960 => 876648 (4%)\nFinal slice: 6715954 => 261218\nOverall compressed: 50071857\nCalculating Repository bundle size...\nCounting objects: 107369, done.\nCompressing objects: 100% (28059/28059), done.\nWriting objects: 100% (107369/107369), 29.20 MiB | 48353 KiB/s, done.\nTotal 107369 (delta 78185), reused 106770 (delta 77609)\nBundle size: 30638967\nOverall inefficiency: 63%\n\nSomehow we made larger packs but the total packed size was larger.\n\nTrying with 100MB \"blocks\" I get:\n\nGenerating index...\nLength is 1291327524, 13 blocks\nSlice #0: 104952661 => 4846553 (4%)\nSlice #1: 104898188 => 2830056 (2%)\nSlice #2: 105007998 => 2856535 (2%)\nSlice #3: 104909972 => 2583402 (2%)\nSlice #4: 104909440 => 2187708 (2%)\nSlice #5: 104859786 => 2555686 (2%)\nSlice #6: 104873317 => 2358914 (2%)\nSlice #7: 104881597 => 2183894 (2%)\nSlice #8: 104863418 => 3555224 (3%)\nSlice #9: 104896599 => 3192564 (3%)\nSlice #10: 104876697 => 3895707 (3%)\nSlice #11: 104903491 => 3731555 (3%)\nFinal slice: 32494360 => 1270887\nOverall compressed: 38048685\nCalculating Repository bundle size...\nCounting objects: 107369, done.\nCompressing objects: 100% (28059/28059), done.\nWriting objects: 100% (107369/107369), 29.20 MiB | 48040 KiB/s, done.\nTotal 107369 (delta 78185), reused 106770 (delta 77609)\nBundle size: 30638967\nOverall inefficiency: 24%\n\nIn the above, we broke the git.git download into 13 partial downloads of\na few meg each, at the expense of an extra 24% of download.\n\nAnyway I've hopefully got more to add to this but this will do for a\nstarting point.\n\nSam\n"},{"id":"121318","messageId":"200908200937.05412.jnareb@gmail.com","threadId":"20626","inReplyTo":"alpine.LFD.2.00.0908191552020.6044@xanadu.home","subject":"Re: Continue git clone after interruption","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-08-20T07:37:04Z","receivedAt":"2009-08-20T07:37:04Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Wed, 19 Aug 2009, Nicolas Pitre wrote:\n> On Wed, 19 Aug 2009, Jakub Narebski wrote:\n> > \n> > On Wed, 19 Aug 2009, Nicolas Pitre wrote:\n> > > On Wed, 19 Aug 2009, Jakub Narebski wrote:\n\n[...]\n> > > Yet, that bundle would probably not contain the latest revision if it is \n> > > only periodically updated, even less so if it is shared between multiple \n> > > repositories as outlined above. And what people with slow/unreliable \n> > > network links are probably most interested in is the latest revision and \n> > > maybe a few older revisions, but probably not the whole repository as \n> > > that is simply too long to wait for. Hence having a big bundle is not \n> > > flexible either with regards to the actual data transfer size.\n> > \n> > I agree that bundle would be useful for restartable clone, and not\n> > useful for restartable fetch. Well, unless you count (non-existing)\n> > GitTorrent / git-mirror-sync as this solution... ;-)\n> \n> I don't think fetches after a clone are such an issue. They are \n> typically transfers being orders of magnitude smaller than the initial \n> clone. Same goes for fetches to deepen a shallow clone which are in \n> fact fetches going back in history instead of forward. I still stands \n> by my assertion that bundles are suboptimal for a restartable clone.\n\nThey are good as a workaround for lack of resumable clone in a pinch,\nbut I agree that because a) like packfile they cannot guarantee that\nfor the same arguments (endpoints) they would generate the same file,\nb) you can't generate currently resume pat of bundle only (and it would\nbe probably difficult to add it) they are not optimal solution.\n\n> As for GitTorrent / git-mirror-sync... those are still vaporwares to me \n> and I therefore have doubts about their actual feasability. So no, I \n> don't count on them.\n\nWell... there is _some_ code.\n\n> > > Hence having a restartable git-archive service to create the top \n> > > revision with the ability to cheaply (in terms of network bandwidth) \n> > > deepen the history afterwards is probably the most straight forward way \n> > > to achieve that. The server needs no be aware of separate bundles, etc. \n> > > And the shared object store still works as usual with the same cached IO \n> > > whether the data is needed for a traditional fetch or a \"git archive\" \n> > > operation.\n> > \n> > It's the \"cheaply deepen history\" that I doubt would be easy. This is\n> > the most difficult part, I think (see also below).\n> \n> Don't think so. Try this:\n> \n> \tmkdir test\n> \tcd test\n> \tgit init\n> \tgit fetch --depth=1 git://git.kernel.org/pub/scm/git/git.git\n> \n> Result:\n> \n> remote: Counting objects: 1824, done.\n> remote: Compressing objects: 100% (1575/1575), done.\n> Receiving objects: 100% (1824/1824), 3.01 MiB | 975 KiB/s, done.\n> remote: Total 1824 (delta 299), reused 1165 (delta 180)\n> Resolving deltas: 100% (299/299), done.\n> From git://git.kernel.org/pub/scm/git/git\n> * branch HEAD -> FETCH_HEAD\n> \n> You'll get the very latest revision for HEAD, and only that. The size \n> of the transfer will be roughly the size of a daily snapshot, except it \n> is fully up to date. It is however non resumable in the event of a \n> network outage. My proposal is to replace this with a \"git archive\" \n> call. It won't get all branches, but for the purpose of initialising \n> one's repository that should be good enough. And the \"git archive\" can \n> be fully resumable as I explained.\n\nIt is however only 2.5 MB out of 37 MB that are resumable, which is 7%\n(well, that of course depends on repository). Not that much that is\nresumable.\n\n> Now to deepen that history. Let's say you want 10 more revisions going \n> back then you simply perform the fetch again with a --depth=10. Right \n> now it doesn't seem to work optimally, but the pack that is then being \n> sent could be made of deltas against objects found in the commits we \n> already have. Currently it seems that a pack that also includes those \n> objects we already have in addition to those we want is created, which \n> is IMHO a flaw in the shallow support that shouldn't be too hard to fix. \n> Each level of deepening should then be as small as standard fetches \n> going forward when updating the repository with new revisions.\n\nYou would have the same (or at least quite similar) problems with \ndeepening part (the 'incrementals' transfer part) as you found with my\nfirst proposal of server bisection / division of rev-list, and serving\n1/Nth of revisions (where N is selected so packfile is reasonable) to\nclient as incrementals. Yours is top-down, mine was bottom-up approach\nto sending series of smaller packs. The problem is how to select size\nof incrementals, and that incrementals are all-or-nothing (but see also\ncomment below).\n\nIn proposal using git-archive and shallow clone deepening as incrementals\nyou have this small seed (how small it depends on repository: 50% - 5%)\nwhich is resumable. And presumably with deepening you can somehow make\nsome use from incomplete packfile, only part of which was transferred \nbefore network error / disconnect. And even tell server about objects\nwhich you managed to extract from *.pack.part.\n\n> > > Why \"git archive\"? Because its content is well defined. So if you give \n> > > it a commit SHA1 you will always get the same stream of bytes (after \n> > > decompression) since the way git sort files is strictly defined. It is \n> > > therefore easy to tell a remote \"git archive\" instance that we want the \n> > > content for commit xyz but that we already got n files already, and that \n> > > the last file we've got has m bytes. There is simply no confusion about \n> > > what we've got already, unlike with a partial pack which might need \n> > > yet-to-be-received objects in order to make sense of what has been \n> > > already received. The server simply has to skip that many files and \n> > > resume the transfer at that point, independently of the compression or \n> > > even the archive format.\n> > \n> > Let's reiterate it to check if I understand it correctly:\n> > \n> > Any \"restartable clone\" / \"resumable fetch\" solution must begin with\n> > a file which is rock-solid stable wrt. reproductability given the same\n> > parameters. git-archive has this feature, packfile doesn't (so I guess\n> > that bundle also doesn't, unless it was cached / saved on disk).\n> \n> Right.\n\n*NEW IDEA*\n\nAnother solution would be to try to come up with some sort of stable\nsorting of objects so that packfile generated for the same parameters\n(endpoints) would be always byte-for-byte the same. But that might be\ndifficult, or even impossible.\n\nWell, we could send the list of objects in pack in order used later by\npack creation to client (non-resumable but small part), and if packfile\ntransport was interrupted in the middle client would compare list of \ncomplete objects in part of packfile against this manifest, and sent\nrequest to server with *sorted* list of object it doesn't have yet.\nServer would probably have to check validity of objects list first (the\nobject list might be needed to be more than just object list; it might\nneed to specify topology of deltas, i.e. which objects are base for which\nones). Then it would generate rest of packfile.\n \n> > It would be useful if it was possible to generate part of this rock-solid\n> > file for partial (range, resume) request, without need to generate \n> > (calculate) parts that client already downloaded. Otherwise server has\n> > to either waste disk space and IO for caching, or waste CPU (and IO)\n> > on generating part which is not needed and dropping it to /dev/null.\n> > git-archive you say has this feature.\n> \n> \"Could easily have\" is more appropriate.\n\nO.K. And I can see how this can be easy done.\n\n> > Next you need to tell server that you have those objects got using\n> > resumable download part (\"git archive HEAD\" in your proposal), and\n> > that it can use them and do not include them in prepared file/pack.\n> > \"have\" is limited to commits, and \"have \" tells server that\n> > you have and all its prerequisites (dependences). You can't \n> > use \"have \" with git-archive solution. I don't know enough\n> > about 'shallow' capability (and what it enables) to know whether\n> > it can be used for that. Can you elaborate?\n> \n> See above, or Documentation/technical/shallow.txt.\n \nDocumentation/technical/shallow.txt doesn't cover \"shallow\", \"unshallow\"\nand \"deepen\" commands from 'shallow' capability extension to git pack\nprotocol (http://git-scm.com/gitserver.txt).\n\n> > Then you have to finish clone / fetch. All solutions so far include\n> > some kind of incremental improvements. My first proposal of bisect\n> > fetching 1/nth or predefined size pack is buttom-up solution, where\n> > we build full clone from root commits up. You propose, from what\n> > I understand build full clone from top commit down, using deepening\n> > from shallow clone. In this step you either get full incremental\n> > or not; downloading incremental (from what I understand) is not\n> > resumable / they do not support partial fetch.\n> \n> Right. However, like I said, the incremental part should be much \n> smaller and therefore less susceptible to network troubles.\n\nIf you have 7% total pack size of git-archive resumable part, how small\ndo you plan to have those incremental deepening? Besides in my 1/Nth\nproposal those bottom-up packs werealso meant to be sufficiently small\nto avoid network troubles.\n\n\nP.S. As you can see implementing resumable clone isn't easy...\n\n-- \nJakub Narebski\nPoland\n"},{"id":"121319","messageId":"fcaeb9bf0908200048v4c918a16o6c0e0fb545742f2@mail.gmail.com","threadId":"20626","inReplyTo":"200908200937.05412.jnareb@gmail.com","subject":"Re: Continue git clone after interruption","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2009-08-20T07:48:29Z","receivedAt":"2009-08-20T07:48:29Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Aug 20, 2009 at 2:37 PM, Jakub Narebski wrote:\n> *NEW IDEA*\n>\n> Another solution would be to try to come up with some sort of stable\n> sorting of objects so that packfile generated for the same parameters\n> (endpoints) would be always byte-for-byte the same.  But that might be\n> difficult, or even impossible.\n\nIsn't it the idea of commit reels [1] from git-torrent?\n\n[1] http://gittorrent.utsl.gen.nz/rfc.html#org-reels\n-- \nDuy\n"},{"id":"121321","messageId":"200908201023.09380.jnareb@gmail.com","threadId":"20626","inReplyTo":"fcaeb9bf0908200048v4c918a16o6c0e0fb545742f2@mail.gmail.com","subject":"Re: Continue git clone after interruption","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-08-20T08:23:08Z","receivedAt":"2009-08-20T08:23:08Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Dnia czwartek 20. sierpnia 2009 09:48, Nguyen Thai Ngoc Duy napisał:\n> On Thu, Aug 20, 2009 at 2:37 PM, Jakub Narebski wrote:\n\n> > *NEW IDEA*\n> >\n> > Another solution would be to try to come up with some sort of stable\n> > sorting of objects so that packfile generated for the same parameters\n> > (endpoints) would be always byte-for-byte the same.  But that might be\n> > difficult, or even impossible.\n> \n> Isn't it the idea of commit reels [1] from git-torrent?\n> \n> [1] http://gittorrent.utsl.gen.nz/rfc.html#org-reels\n\nWell, I didn't thought that this idea didn't occur to anybody else.\nWhat I meant here was that it is a new idea mentioned in this subthread.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"121356","messageId":"alpine.LFD.2.00.0908201358010.6044@xanadu.home","threadId":"20626","inReplyTo":"200908200937.05412.jnareb@gmail.com","subject":"Re: Continue git clone after interruption","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-08-20T18:41:25Z","receivedAt":"2009-08-20T18:41:25Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 20 Aug 2009, Jakub Narebski wrote:\n\n> On Wed, 19 Aug 2009, Nicolas Pitre wrote:\n> > You'll get the very latest revision for HEAD, and only that. The size \n> > of the transfer will be roughly the size of a daily snapshot, except it \n> > is fully up to date. It is however non resumable in the event of a \n> > network outage. My proposal is to replace this with a \"git archive\" \n> > call. It won't get all branches, but for the purpose of initialising \n> > one's repository that should be good enough. And the \"git archive\" can \n> > be fully resumable as I explained.\n> \n> It is however only 2.5 MB out of 37 MB that are resumable, which is 7%\n> (well, that of course depends on repository). Not that much that is\n> resumable.\n\nTake the Linux kernel then. It is more like 75 MB.\n\n> > Now to deepen that history. Let's say you want 10 more revisions going \n> > back then you simply perform the fetch again with a --depth=10. Right \n> > now it doesn't seem to work optimally, but the pack that is then being \n> > sent could be made of deltas against objects found in the commits we \n> > already have. Currently it seems that a pack that also includes those \n> > objects we already have in addition to those we want is created, which \n> > is IMHO a flaw in the shallow support that shouldn't be too hard to fix. \n> > Each level of deepening should then be as small as standard fetches \n> > going forward when updating the repository with new revisions.\n> \n> You would have the same (or at least quite similar) problems with \n> deepening part (the 'incrementals' transfer part) as you found with my\n> first proposal of server bisection / division of rev-list, and serving\n> 1/Nth of revisions (where N is selected so packfile is reasonable) to\n> client as incrementals. Yours is top-down, mine was bottom-up approach\n> to sending series of smaller packs. The problem is how to select size\n> of incrementals, and that incrementals are all-or-nothing (but see also\n> comment below).\n\nYes and no. Combined with a slight reordering of commit objects, it \ncould be possible to receive a partial pack and still be able to extract \na bunch of full revisions. The biggest issue is to be able to transfer \nrevision x (75 MB for Linux), but revision x-1 usually requires only a \nfew kilobytes, revision x-2 a few other kilobytes, etc. Remember that \nyou are likely to have only a few deltas from one revision to another, \nwhich is not the case for the very first revision you get. A special \nmode to pack-object could place commit objects only after all the \nobjects needed to create that revision. So once you get a commit object \non the receiving end, you could assume that all objects reachable from \nthat commit are already received, or you had them locally already.\n\n> In proposal using git-archive and shallow clone deepening as incrementals\n> you have this small seed (how small it depends on repository: 50% - 5%)\n> which is resumable. And presumably with deepening you can somehow make\n> some use from incomplete packfile, only part of which was transferred \n> before network error / disconnect. And even tell server about objects\n> which you managed to extract from *.pack.part.\n\nyes. And at that point resuming the transfer is just another case of \nshallow repository deepening.\n\n> *NEW IDEA*\n> \n> Another solution would be to try to come up with some sort of stable\n> sorting of objects so that packfile generated for the same parameters\n> (endpoints) would be always byte-for-byte the same. But that might be\n> difficult, or even impossible.\n\nAnd I don't want to commit to that either. Having some flexibility in \nobject ordering makes it possible to improve on the packing heuristics. \nWe certainly should avoid imposing strong restrictions like that for \nlittle gain. Even the deltas are likely to be different from one \nrequest to another when using threads as one thread might be getting \nmore CPU time than another slightly modifying the outcome.\n\n> Well, we could send the list of objects in pack in order used later by\n> pack creation to client (non-resumable but small part), and if packfile\n> transport was interrupted in the middle client would compare list of \n> complete objects in part of packfile against this manifest, and sent\n> request to server with *sorted* list of object it doesn't have yet.\n\nWell... actually that's one of the item for pack V4. Lots of SHA1s are \nduplicated in tree and commit objects, in addition to the pack index \nfile. With pack v4 all those SHA1s would be stored only once in a table \nand objects would index that table instead.\n\nStill, that is not _that_ small though. Just look at the size of the \npack index file for the Linux repository to give you an idea.\n\n> Server would probably have to check validity of objects list first (the\n> object list might be needed to be more than just object list; it might\n> need to specify topology of deltas, i.e. which objects are base for which\n> ones). Then it would generate rest of packfile.\n\nI'm afraid that has the looks of something adding lots of complexity to \na piece of git that is already quite complex already, namely \npack-objects. And there is already only a few individuals with their \nbrain around it.\n\n> > > It would be useful if it was possible to generate part of this rock-solid\n> > > file for partial (range, resume) request, without need to generate \n> > > (calculate) parts that client already downloaded. Otherwise server has\n> > > to either waste disk space and IO for caching, or waste CPU (and IO)\n> > > on generating part which is not needed and dropping it to /dev/null.\n> > > git-archive you say has this feature.\n> > \n> > \"Could easily have\" is more appropriate.\n> \n> O.K. And I can see how this can be easy done.\n> \n> > > Next you need to tell server that you have those objects got using\n> > > resumable download part (\"git archive HEAD\" in your proposal), and\n> > > that it can use them and do not include them in prepared file/pack.\n> > > \"have\" is limited to commits, and \"have \" tells server that\n> > > you have and all its prerequisites (dependences). You can't \n> > > use \"have \" with git-archive solution. I don't know enough\n> > > about 'shallow' capability (and what it enables) to know whether\n> > > it can be used for that. Can you elaborate?\n> > \n> > See above, or Documentation/technical/shallow.txt.\n> \n> Documentation/technical/shallow.txt doesn't cover \"shallow\", \"unshallow\"\n> and \"deepen\" commands from 'shallow' capability extension to git pack\n> protocol (http://git-scm.com/gitserver.txt).\n\n404 Not Found\n\nMaybe that should be committed to git in Documentation/technical/ as \nwell?\n\n> > > Then you have to finish clone / fetch. All solutions so far include\n> > > some kind of incremental improvements. My first proposal of bisect\n> > > fetching 1/nth or predefined size pack is buttom-up solution, where\n> > > we build full clone from root commits up. You propose, from what\n> > > I understand build full clone from top commit down, using deepening\n> > > from shallow clone. In this step you either get full incremental\n> > > or not; downloading incremental (from what I understand) is not\n> > > resumable / they do not support partial fetch.\n> > \n> > Right. However, like I said, the incremental part should be much \n> > smaller and therefore less susceptible to network troubles.\n> \n> If you have 7% total pack size of git-archive resumable part, how small\n> do you plan to have those incremental deepening? Besides in my 1/Nth\n> proposal those bottom-up packs werealso meant to be sufficiently small\n> to avoid network troubles.\n\nTwo issues here: 1) people with slow links might not be interested in a \ndeep history as it costs them time. 2) Extra revisions should typically \nrequire only a few KB each, therefore we might manage to ask for the \nfull history after the initial revision is downloaded and salvage as \nmuch as we can if a network outage is encountered. There is no need for \narbitrary size, unless the user decides arbitrarily to get only 10 more \nrevisions, or 100 more, etc.\n\n> P.S. As you can see implementing resumable clone isn't easy...\n\nI've been saying that all along for quite a while now. ;-)\n\n\nNicolas\n"},{"id":"121362","messageId":"1250809062.5963.24.camel@maia.lan","threadId":"20626","inReplyTo":"200908200937.05412.jnareb@gmail.com","subject":"Re: Continue git clone after interruption","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2009-08-20T22:57:42Z","receivedAt":"2009-08-20T22:57:42Z","isPatch":false,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"On Thu, 2009-08-20 at 09:37 +0200, Jakub Narebski wrote:\n> You would have the same (or at least quite similar) problems with \n> deepening part (the 'incrementals' transfer part) as you found with my\n> first proposal of server bisection / division of rev-list, and serving\n> 1/Nth of revisions (where N is selected so packfile is reasonable) to\n> client as incrementals. Yours is top-down, mine was bottom-up approach\n> to sending series of smaller packs. The problem is how to select size\n> of incrementals, and that incrementals are all-or-nothing (but see also\n> comment below).\n\nI've defined a way to do this which doesn't have the complexity of\nbisect in GitTorrent, making the compromise that you can't guarantee\neach chunk is exactly the same size... I'll have a crack at doing it\nbased on the rev-cache code in C instead of the horrendously slow\nPerl/Berkeley solution I have at the moment to see how well it fares.\n\n> Another solution would be to try to come up with some sort of stable\n> sorting of objects so that packfile generated for the same parameters\n> (endpoints) would be always byte-for-byte the same. But that might be\n> difficult, or even impossible.\n\ndelta compression is not repeatable enough for this.\n\nThis was an assumption made by the first version of GitTorrent, that\nthis would be an appropriate solution. \n\nSo, first you have to sort the objects - that's fine, --date-order is a\ngood starting point, then I reasoned that interleaving new objects for\neach commit with commit objects would be a useful sort order. You also\nneed to tie-break for commits with the same commit date; I just used the\nSHA-1 of the commit for that. Finally, when making packs to avoid\nexcessive transfer you have to try to make sure that they are \"thin\"\npacks.\n\nCurrently, thin packs can only work starting at the beginning of history\nand working forward, which is opposite to what happens most of the time\nin packs. I think this is the source of much of the inefficiency caused\nby chopping up the object lists mentioned in my other e-mail. It might\nbe possible, if you could also know which earlier objects were using\nthis object as a delta base, to try delta'ing against all those objects\nand see which one results in the smallest delta.\n\n> Well, we could send the list of objects in pack in order used later by\n> pack creation to client (non-resumable but small part), and if packfile\n> transport was interrupted in the middle client would compare list of \n> complete objects in part of packfile against this manifest, and sent\n> request to server with *sorted* list of object it doesn't have yet.\n> Server would probably have to check validity of objects list first (the\n> object list might be needed to be more than just object list; it might\n> need to specify topology of deltas, i.e. which objects are base for which\n> ones). Then it would generate rest of packfile.\n\nMmm. It's a bit chatty, that. Object lists add another 10-20% on,\nwhich I think should be avoidable if the thin pack problem, plus the\nproblem of some objects ending up in more than one of the thin packs\nthat are created, should be reduced to very little.\n\nSam\n"},{"id":"121409","messageId":"200908211207.38555.jnareb@gmail.com","threadId":"20626","inReplyTo":"alpine.LFD.2.00.0908201358010.6044@xanadu.home","subject":"Re: Continue git clone after interruption","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-08-21T10:07:37Z","receivedAt":"2009-08-21T10:07:37Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Thu, 20 Aug 2009, Nicolas Pitre wrote:\n> On Thu, 20 Aug 2009, Jakub Narebski wrote:\n>> On Wed, 19 Aug 2009, Nicolas Pitre wrote:\n\n>>> You'll get the very latest revision for HEAD, and only that. The size \n>>> of the transfer will be roughly the size of a daily snapshot, except it \n>>> is fully up to date. It is however non resumable in the event of a \n>>> network outage. My proposal is to replace this with a \"git archive\" \n>>> call. It won't get all branches, but for the purpose of initialising \n>>> one's repository that should be good enough. And the \"git archive\" can \n>>> be fully resumable as I explained.\n>> \n>> It is however only 2.5 MB out of 37 MB that are resumable, which is 7%\n>> (well, that of course depends on repository). Not that much that is\n>> resumable.\n> \n> Take the Linux kernel then. It is more like 75 MB.\n\nAh... good example.\n\nOn the other hand Linux is fairly large project in terms of LoC, but\nit had its history cut when moving to Git, so the ratio of git-archive\nof HEAD to the size of packfile is overemphasized here.\n\n>>> Now to deepen that history. Let's say you want 10 more revisions going \n>>> back then you simply perform the fetch again with a --depth=10. Right \n>>> now it doesn't seem to work optimally, but the pack that is then being \n>>> sent could be made of deltas against objects found in the commits we \n>>> already have. Currently it seems that a pack that also includes those \n>>> objects we already have in addition to those we want is created, which \n>>> is IMHO a flaw in the shallow support that shouldn't be too hard to fix. \n>>> Each level of deepening should then be as small as standard fetches \n>>> going forward when updating the repository with new revisions.\n>> \n>> You would have the same (or at least quite similar) problems with \n>> deepening part (the 'incrementals' transfer part) as you found with my\n>> first proposal of server bisection / division of rev-list, and serving\n>> 1/Nth of revisions (where N is selected so packfile is reasonable) to\n>> client as incrementals. Yours is top-down, mine was bottom-up approach\n>> to sending series of smaller packs. The problem is how to select size\n>> of incrementals, and that incrementals are all-or-nothing (but see also\n>> comment below).\n> \n> Yes and no. Combined with a slight reordering of commit objects, it \n> could be possible to receive a partial pack and still be able to extract \n> a bunch of full revisions. The biggest issue is to be able to transfer \n> revision x (75 MB for Linux), but revision x-1 usually requires only a \n> few kilobytes, revision x-2 a few other kilobytes, etc. Remember that \n> you are likely to have only a few deltas from one revision to another, \n> which is not the case for the very first revision you get.\n\nLet's reiterate again, to be sure that I understand this correctly:\n\n\nYou make use here of a few facts:\n\n1. Objects in packfile are _usually_ sorted in recency order, with most\n recent commits, and most recent versions of trees and tags being in\n the front of pack file, and being base objects for a large set of \n objects. Note the \"usually\" part; it is not set in stone as for RCS\n (and CVS) reverse delta based repository format.\n\n2. There is support in git pack format to do 'deepening' of shallow\n clone, which means that git can generate incrementals in top-down\n order, _similar to how objects are ordered in packfile_.\n\n3. git-archive output is stable. _git-archive can be made resumable_\n (with range/partial requests), and can be made so it can create\n single-head depth 0 shallow clone.\n\nAlso, with top-down deepening order even if you don't use \n'git clone --continue' but 'git clone --skip' (or something), you\nwould have got usable shallow clone. In the most extreme case when\nyou are able to get only the fully resumable part, i.e. git-archive\npart (with top commit), you would have single-branch depth 0\nshallow clone (not very usable, but still better than nothing).\n\n> A special \n> mode to pack-object could place commit objects only after all the \n> objects needed to create that revision. So once you get a commit object \n> on the receiving end, you could assume that all objects reachable from \n> that commit are already received, or you had them locally already.\n\nYes, with such mode (which I think wouldn't reduce / interfere with\nability for upload-pack to pack more tightly by reordering objects\nand choosing different deltas) it would be easy to do a salvage of\na partially completed / transferred packfile. Even if there is no\nextension to tell git server which objects we have (\"have\" is only\nabout commits), if there is at least one commit object in received\npart of packfile, we can try to continue from later (from more);\nthere is less left to download.\n\n> \n>> In proposal using git-archive and shallow clone deepening as incrementals\n>> you have this small seed (how small it depends on repository: 50% - 5%)\n>> which is resumable. And presumably with deepening you can somehow make\n>> some use from incomplete packfile, only part of which was transferred \n>> before network error / disconnect. And even tell server about objects\n>> which you managed to extract from *.pack.part.\n> \n> yes. And at that point resuming the transfer is just another case of \n> shallow repository deepening.\n\nAlso for deepening top-down incrementals like in your proposal you can\nhave 'salvage' operation which tries to use something out of partially\ntransferred packfile (partially downloaded incremental). It is not, \nI think, the case with my earlier 'server bisect' bottom-up incrementals\nidea.\n\n> \n>> *NEW IDEA*\n>> \n>> Another solution would be to try to come up with some sort of stable\n>> sorting of objects so that packfile generated for the same parameters\n>> (endpoints) would be always byte-for-byte the same. But that might be\n>> difficult, or even impossible.\n> \n> And I don't want to commit to that either. Having some flexibility in \n> object ordering makes it possible to improve on the packing heuristics. \n> We certainly should avoid imposing strong restrictions like that for \n> little gain. Even the deltas are likely to be different from one \n> request to another when using threads as one thread might be getting \n> more CPU time than another slightly modifying the outcome.\n\nRight.\n\n>> Well, we could send the list of objects in pack in order used later by\n>> pack creation to client (non-resumable but small part), and if packfile\n>> transport was interrupted in the middle client would compare list of \n>> complete objects in part of packfile against this manifest, and sent\n>> request to server with *sorted* list of object it doesn't have yet.\n> \n> Well... actually that's one of the item for pack V4. Lots of SHA1s are \n> duplicated in tree and commit objects, in addition to the pack index \n> file. With pack v4 all those SHA1s would be stored only once in a table \n> and objects would index that table instead.\n> \n> Still, that is not _that_ small though. Just look at the size of the \n> pack index file for the Linux repository to give you an idea.\n\nWell, as such plan (map) of a packfile wouldn't be much smaller than\npack index, if pack index file is large enough (or connection crappy\nenough) that it couldn't be transferred without interruption.\n\nNevertheless 34 MB index for largest 310 MB packfile in Linux kernel\nhttp://www.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git/objects/pack/\nisn't something very large. And objects-list / plan of packfile would\nbe of comparable size.\n \n\nI was envisioning packfile plan (packfile map) as something like that\n\n sha1 TERMINATOR\n\nfor objects that are not deltified in the packfile, and\n\n sha1 SEPARATOR base-sha1 TERMINATOR\n\nfor objects that are deltified (or something like that; we could use\npkt-line format instead).\n\n>> Server would probably have to check validity of objects list first (the\n>> object list might be needed to be more than just object list; it might\n>> need to specify topology of deltas, i.e. which objects are base for which\n>> ones). Then it would generate rest of packfile.\n> \n> I'm afraid that has the looks of something adding lots of complexity to \n> a piece of git that is already quite complex already, namely \n> pack-objects. And there is already only a few individuals with their \n> brain around it.\n\nWell, with complexity or server CPU/IO because one should not trust \nclient (unfortunately), with 'packfile plan' transfer being non-resumable,\nand also with requiring packv4 or a temporary file or memory to send \npackfile plan (packfile map)... I think we can scrape that half-baked\nidea.\n\n>>>> [...] I don't know enough\n>>>> about 'shallow' capability (and what it enables) to know whether\n>>>> it can be used for that. Can you elaborate?\n>>> \n>>> See above, or Documentation/technical/shallow.txt.\n>> \n>> Documentation/technical/shallow.txt doesn't cover \"shallow\", \"unshallow\"\n>> and \"deepen\" commands from 'shallow' capability extension to git pack\n>> protocol (http://git-scm.com/gitserver.txt).\n> \n> 404 Not Found\n> \n> Maybe that should be committed to git in Documentation/technical/ as \n> well?\n\nThis was plain text RFC for the Git Packfile Protocol, generated from\nrfc2629 XML sources at http://github.com/schacon/gitserver-rfc\n\n\nScott, what happened to http://git-scm.com/gitserver.txt? \n\nAnd could you create 'rfc' or 'text' branch in gitserver-rfc\nrepository, with processed plain/text output, similar to 'man' and\n'html' branches in git.git repository? TIA.\n\n\n_Some_ description of pack protocol can be found in git mailing list\narchives\n http://thread.gmane.org/gmane.comp.version-control.git/118956\nin \"The Git Community Book\"\n http://book.git-scm.com/7_transfer_protocols.html\n http://github.com/schacon/gitbook/blob/master/text/54_Transfer_Protocols/0_Transfer_Protocols.markdown\nand in \"Pro Git\"\n http://progit.org/book/ch9-6.html\n http://github.com/progit/progit/blob/master/en/09-git-internals/01-chapter9.markdown\n\nThe description in Documentation/technical/pack-protocol.txt is very\nbrief, and Documentation/technical/shallow.txt doesn't cover 'shallow'\ncapability of git pack protocol.\n \n>>>> Then you have to finish clone / fetch. All solutions so far include\n>>>> some kind of incremental improvements. My first proposal of bisect\n>>>> fetching 1/nth or predefined size pack is buttom-up solution, where\n>>>> we build full clone from root commits up. You propose, from what\n>>>> I understand build full clone from top commit down, using deepening\n>>>> from shallow clone. In this step you either get full incremental\n>>>> or not; downloading incremental (from what I understand) is not\n>>>> resumable / they do not support partial fetch.\n>>> \n>>> Right. However, like I said, the incremental part should be much \n>>> smaller and therefore less susceptible to network troubles.\n>> \n>> If you have 7% total pack size of git-archive resumable part, how small\n>> do you plan to have those incremental deepening? Besides in my 1/Nth\n>> proposal those bottom-up packs werealso meant to be sufficiently small\n>> to avoid network troubles.\n> \n> Two issues here: 1) people with slow links might not be interested in a \n> deep history as it costs them time. 2) Extra revisions should typically \n> require only a few KB each, therefore we might manage to ask for the \n> full history after the initial revision is downloaded and salvage as \n> much as we can if a network outage is encountered. There is no need for \n> arbitrary size, unless the user decides arbitrarily to get only 10 more \n> revisions, or 100 more, etc.\n\nThose two features of your proposal: \n1.) It is possible salvage of partially transferred packfiles (so there\n is no requirement to guess accurately what size should they be),\n2.) After completing initial git-archive transfer, you can convert \n incomplete clone into functioning repository. It would be shallow\n clone, and can be missing some branches and tags, but you can work\n with it if network connection fails completely.\nmake it very compelling.\n\n>> P.S. As you can see implementing resumable clone isn't easy...\n> \n> I've been saying that all along for quite a while now. ;-)\n\nWell, on the other hand side we have example of how long it took to\ncome to current implementation of git submodules. But if finally\ngot done.\n\n\nThe git-archive + deepening approach you proposed can be split into\nsmaller individual improvements. You don't need to implement it all\nat once.\n\n1. Improve deepening of shallow clone. This means sending only required\n objects, and being able to use as a base objects that other side has\n and send thin pack.\n\n2. Add support for resuming (range request) of git-archive. It is up\n to client to translate size of partial transfer of compressed file\n into range request of original (uncompressed) archive.\n\n3. Create new git-archive pseudoformat, used to transfer single commit\n (with commit object and original branch name in some extended header,\n similar to how commit ID is stored in extended pax header or ZIP\n comment). It would imply not using export-* gitattributes.\n\n4. Implement alternate ordering of objects in packfile, so commit object\n is put immediately after all its prerequisites.\n\n5. Implement 'salvage' operation, which given partially transferred \n packfile would deepen shallow clone, or advance tracking branches,\n ensuring that repository would pass fsck after this operation.\n\n Probably requires 4; might be not possible or much harder to salvage\n anything with current ordering of objects in packfile.\n\n6. Implement resumable clone (\"git clone --keep []\",\n \"git clone --resume\" / \"git clone --continue\", \"git clone --abort\",\n \"git clone --make-shallow\" / \"git clone --salvage\").\n\n Requires 1-5.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"121410","messageId":"vpqtz01cvj2.fsf@bauges.imag.fr","threadId":"20626","inReplyTo":"200908211207.38555.jnareb@gmail.com","subject":"Re: Continue git clone after interruption","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2009-08-21T10:26:41Z","receivedAt":"2009-08-21T10:26:41Z","isPatch":false,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"Jakub Narebski writes:\n\n> On the other hand Linux is fairly large project in terms of LoC, but\n> it had its history cut when moving to Git, so the ratio of git-archive\n> of HEAD to the size of packfile is overemphasized here.\n\nEmacs can be a good example if you want a project with a loooong\nhistory.\n\nemacs.git$ git ll | wc -l \n100651\nemacs.git$ du -sh emacs.tar.gz .git/objects/pack/pack-144583582d53e273028966c6de2b3fb2fe3504bc.pack \n29M\temacs.tar.gz\n138M\t.git/objects/pack/pack-144583582d53e273028966c6de2b3fb2fe3504bc.pack\n\n(from git://git.savannah.gnu.org/emacs.git )\n\n-- \nMatthieu\n"},{"id":"121461","messageId":"alpine.LFD.2.00.0908211614220.6044@xanadu.home","threadId":"20626","inReplyTo":"200908211207.38555.jnareb@gmail.com","subject":"Re: Continue git clone after interruption","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-08-21T21:07:51Z","receivedAt":"2009-08-21T21:07:51Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 21 Aug 2009, Jakub Narebski wrote:\n\n> On Thu, 20 Aug 2009, Nicolas Pitre wrote:\n> > On Thu, 20 Aug 2009, Jakub Narebski wrote:\n> >> It is however only 2.5 MB out of 37 MB that are resumable, which is 7%\n> >> (well, that of course depends on repository). Not that much that is\n> >> resumable.\n> > \n> > Take the Linux kernel then. It is more like 75 MB.\n> \n> Ah... good example.\n> \n> On the other hand Linux is fairly large project in terms of LoC, but\n> it had its history cut when moving to Git, so the ratio of git-archive\n> of HEAD to the size of packfile is overemphasized here.\n\nThat doesn't matter. You still need that amount of data up front to do \nanything. And I doubt people with slow links will want the full history \nanyway, regardless if it goes backward 4 years or 18 years back.\n\n> You make use here of a few facts:\n> \n> 1. Objects in packfile are _usually_ sorted in recency order, with most\n> recent commits, and most recent versions of trees and tags being in\n> the front of pack file, and being base objects for a large set of \n> objects. Note the \"usually\" part; it is not set in stone as for RCS\n> (and CVS) reverse delta based repository format.\n\nExact. In theory the object order could be totally random and the pack \nwould still be valid. The only restriction at the moment has to do with \nOFS_DELTA objects as the reference to the base object is encoded as a \ndownward offset from the beginning of that OFS_DELTA object. Hence the \nbase object has to appear first. In the case of REF_DELTA objects, the \nbase can be located anywhere in the pack (or anywhere else outside of \nthe pack in the thin pack case).\n\n> 2. There is support in git pack format to do 'deepening' of shallow\n> clone, which means that git can generate incrementals in top-down\n> order, _similar to how objects are ordered in packfile_.\n\nWell... the pack format was not meant for that \"support\". The fact that \nthe typical object order used by pack-objects when serving fetch request \nis amenable to incremental top-down updates is rather coincidental and \nnot really planned.\n\n> 3. git-archive output is stable. _git-archive can be made resumable_\n> (with range/partial requests), and can be made so it can create\n> single-head depth 0 shallow clone.\n> \n> Also, with top-down deepening order even if you don't use \n> 'git clone --continue' but 'git clone --skip' (or something), you\n> would have got usable shallow clone. In the most extreme case when\n> you are able to get only the fully resumable part, i.e. git-archive\n> part (with top commit), you would have single-branch depth 0\n> shallow clone (not very usable, but still better than nothing).\n\nRight.\n\n> > A special \n> > mode to pack-object could place commit objects only after all the \n> > objects needed to create that revision. So once you get a commit object \n> > on the receiving end, you could assume that all objects reachable from \n> > that commit are already received, or you had them locally already.\n> \n> Yes, with such mode (which I think wouldn't reduce / interfere with\n> ability for upload-pack to pack more tightly by reordering objects\n> and choosing different deltas) it would be easy to do a salvage of\n> a partially completed / transferred packfile. Even if there is no\n> extension to tell git server which objects we have (\"have\" is only\n> about commits), if there is at least one commit object in received\n> part of packfile, we can try to continue from later (from more);\n> there is less left to download.\n\nExact. Suffice to set the last received commit(s) (after validation) as \none of the shallow points.\n\n> >> Documentation/technical/shallow.txt doesn't cover \"shallow\", \"unshallow\"\n> >> and \"deepen\" commands from 'shallow' capability extension to git pack\n> >> protocol (http://git-scm.com/gitserver.txt).\n> > \n> > 404 Not Found\n> > \n> > Maybe that should be committed to git in Documentation/technical/ as \n> > well?\n> \n> This was plain text RFC for the Git Packfile Protocol, generated from\n> rfc2629 XML sources at http://github.com/schacon/gitserver-rfc\n\nI suggest you track it down and prod/propose a version for merging in \nthe git repository.\n\n> The description in Documentation/technical/pack-protocol.txt is very\n> brief, and Documentation/technical/shallow.txt doesn't cover 'shallow'\n> capability of git pack protocol.\n\nYeah. I finally had a look directly at the code to understand how it \nworks.\n\n> >> P.S. As you can see implementing resumable clone isn't easy...\n> > \n> > I've been saying that all along for quite a while now. ;-)\n> \n> Well, on the other hand side we have example of how long it took to\n> come to current implementation of git submodules. But if finally\n> got done.\n\nIn this case there is still no new line of code what so ever. Thinking \nit through is what takes time.\n\n> The git-archive + deepening approach you proposed can be split into\n> smaller individual improvements. You don't need to implement it all\n> at once.\n> \n> 1. Improve deepening of shallow clone. This means sending only required\n> objects, and being able to use as a base objects that other side has\n> and send thin pack.\n\nYes. And now that I understand how shallow clones are implemented, I \nProbably will fix that flaw soon. Won't be hard at all.\n\n> 2. Add support for resuming (range request) of git-archive. It is up\n> to client to translate size of partial transfer of compressed file\n> into range request of original (uncompressed) archive.\n> \n> 3. Create new git-archive pseudoformat, used to transfer single commit\n> (with commit object and original branch name in some extended header,\n> similar to how commit ID is stored in extended pax header or ZIP\n> comment). It would imply not using export-* gitattributes.\n\nThe format I was envisioning is really simple:\n\nFirst the size of the raw commit object data content in decimal, \nfollowed by a 0 byte, followed by the actual content of the commit \nobject, followed by a 0 byte. (Note: this could be the exact same \ncontent as the canonical commit object data with the \"commit\" prefix, \nbut as all the rest are all blob content this would be redundant.)\n\nThen, for each file:\n\n - The file mode in octal notation just as in tree objects\n - a space\n - the size of the file in decimal\n - a tab\n - the full path of the file\n - a 0 byte\n - the file content as found in the corresponding blob\n - a 0 byte\n\nAnd finally some kind of marker to indicate the end of the stream.\n\nPut the lot through zlib and you're done.\n\n> 4. Implement alternate ordering of objects in packfile, so commit object\n> is put immediately after all its prerequisites.\n\nThat would require some changes in the object enumeration code which is \nan area of the code I don't know well.\n\n> 5. Implement 'salvage' operation, which given partially transferred \n> packfile would deepen shallow clone, or advance tracking branches,\n> ensuring that repository would pass fsck after this operation.\n> \n> Probably requires 4; might be not possible or much harder to salvage\n> anything with current ordering of objects in packfile.\n> \n> 6. Implement resumable clone (\"git clone --keep []\",\n> \"git clone --resume\" / \"git clone --continue\", \"git clone --abort\",\n> \"git clone --make-shallow\" / \"git clone --salvage\").\n\nRight. This is all doable fairly easily.\n\n\nNicolas\n"},{"id":"121469","messageId":"200908212341.33324.jnareb@gmail.com","threadId":"20626","inReplyTo":"alpine.LFD.2.00.0908211614220.6044@xanadu.home","subject":"Re: Continue git clone after interruption","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-08-21T21:41:30Z","receivedAt":"2009-08-21T21:41:30Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Fri, 21 Aug 2009, Nicolas Pitre wrote:\n> On Fri, 21 Aug 2009, Jakub Narebski wrote:\n>> On Thu, 20 Aug 2009, Nicolas Pitre wrote:\n>>> On Thu, 20 Aug 2009, Jakub Narebski wrote:\n\n>>>> It is however only 2.5 MB out of 37 MB that are resumable, which is 7%\n>>>> (well, that of course depends on repository). Not that much that is\n>>>> resumable.\n>>> \n>>> Take the Linux kernel then. It is more like 75 MB.\n>> \n>> Ah... good example.\n>> \n>> On the other hand Linux is fairly large project in terms of LoC, but\n>> it had its history cut when moving to Git, so the ratio of git-archive\n>> of HEAD to the size of packfile is overemphasized here.\n> \n> That doesn't matter. You still need that amount of data up front to do \n> anything. And I doubt people with slow links will want the full history \n> anyway, regardless if it goes backward 4 years or 18 years back.\n\nOn the other hand unreliable link doesn't need to mean unreasonably\nslow link.\n\nHopefully GitTorrent / git-mirror-sync would finally come out of \nvapourware and wouldn't share the fate of Duke Nukem Forever ;-),\nand we would have this as an alternative to clone large repositories.\nWell, supposedly there is some code, and last year GSoC project at\nleast shook the dust out of initial design and made it simplier, IIUC.\n \n>> You make use here of a few facts:\n[...]\n\n>> 2. There is support in git pack format to do 'deepening' of shallow\n>> clone, which means that git can generate incrementals in top-down\n>> order, _similar to how objects are ordered in packfile_.\n> \n> Well... the pack format was not meant for that \"support\". The fact that \n> the typical object order used by pack-objects when serving fetch request \n> is amenable to incremental top-down updates is rather coincidental and \n> not really planned.\n\nOoops. I meant \"git pack PROTOCOL\" here, not \"git pack _format_\".\nthe one about want/have/shallow/deepen exchange.\n \n[...]\n>>> A special \n>>> mode to pack-object could place commit objects only after all the \n>>> objects needed to create that revision. So once you get a commit object \n>>> on the receiving end, you could assume that all objects reachable from \n>>> that commit are already received, or you had them locally already.\n>> \n>> Yes, with such mode (which I think wouldn't reduce / interfere with\n>> ability for upload-pack to pack more tightly by reordering objects\n>> and choosing different deltas) it would be easy to do a salvage of\n>> a partially completed / transferred packfile. Even if there is no\n>> extension to tell git server which objects we have (\"have\" is only\n>> about commits), if there is at least one commit object in received\n>> part of packfile, we can try to continue from later (from more);\n>> there is less left to download.\n> \n> Exact. Suffice to set the last received commit(s) (after validation) as \n> one of the shallow points.\n\nAssuming that received commit is full (has all prerequisites), and\nis connected to the rest of body of partially [shallow] cloned \nrepository.\n\n>>>> Documentation/technical/shallow.txt doesn't cover \"shallow\", \"unshallow\"\n>>>> and \"deepen\" commands from 'shallow' capability extension to git pack\n>>>> protocol (http://git-scm.com/gitserver.txt).\n>>> \n>>> 404 Not Found\n>>> \n>>> Maybe that should be committed to git in Documentation/technical/ as \n>>> well?\n>> \n>> This was plain text RFC for the Git Packfile Protocol, generated from\n>> rfc2629 XML sources at http://github.com/schacon/gitserver-rfc\n> \n> I suggest you track it down and prod/propose a version for merging in \n> the git repository.\n\nScott Chacon was (and is) CC-ed.\n \nI don't know if you remember mentioned discussion about pack protocol, \nstemming from the fact that some of git (re)implementations (Dulwich,\nJGit) failed to implement it properly, where properly = same as \ngit-core, i.e. the original implementation in C... because there were\nnot enough documentation.\n\n\n>>>> P.S. As you can see implementing resumable clone isn't easy...\n>>> \n>>> I've been saying that all along for quite a while now. ;-)\n>> \n>> Well, on the other hand side we have example of how long it took to\n>> come to current implementation of git submodules. But if finally\n>> got done.\n> \n> In this case there is still no new line of code what so ever. Thinking \n> it through is what takes time.\n\nMeasure twice, cut once :-)\n\nIn this case I think design upfront is a good solution.\n \n>> The git-archive + deepening approach you proposed can be split into\n>> smaller individual improvements. You don't need to implement it all\n>> at once.\n[...]\n\n>> 3. Create new git-archive pseudoformat, used to transfer single commit\n>> (with commit object and original branch name in some extended header,\n>> similar to how commit ID is stored in extended pax header or ZIP\n>> comment). It would imply not using export-* gitattributes.\n> \n> The format I was envisioning is really simple:\n> \n> First the size of the raw commit object data content in decimal, \n> followed by a 0 byte, followed by the actual content of the commit \n> object, followed by a 0 byte. (Note: this could be the exact same \n> content as the canonical commit object data with the \"commit\" prefix, \n> but as all the rest are all blob content this would be redundant.)\n> \n> Then, for each file:\n> \n> - The file mode in octal notation just as in tree objects\n> - a space\n> - the size of the file in decimal\n> - a tab\n> - the full path of the file\n> - a 0 byte\n> - the file content as found in the corresponding blob\n> - a 0 byte\n> \n> And finally some kind of marker to indicate the end of the stream.\n> \n> Put the lot through zlib and you're done.\n\nSo you don't want to just tack commit object (as extended pax header,\nor a comment - if it is at all possible) to the existing 'tar' and\n'zip' archive formats. Probably better to design format from scratch.\n \n>> 4. Implement alternate ordering of objects in packfile, so commit object\n>> is put immediately after all its prerequisites.\n> \n> That would require some changes in the object enumeration code which is \n> an area of the code I don't know well.\n\nOh.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"121478","messageId":"1250896025.19039.7.camel@maia.lan","threadId":"20626","inReplyTo":"alpine.LFD.2.00.0908211614220.6044@xanadu.home","subject":"Re: Continue git clone after interruption","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2009-08-21T23:07:05Z","receivedAt":"2009-08-21T23:07:05Z","isPatch":false,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"On Fri, 2009-08-21 at 17:07 -0400, Nicolas Pitre wrote:\n> > 2. There is support in git pack format to do 'deepening' of shallow\n> > clone, which means that git can generate incrementals in top-down\n> > order, _similar to how objects are ordered in packfile_.\n> \n> Well... the pack format was not meant for that \"support\". The fact\n> that \n> the typical object order used by pack-objects when serving fetch\n> request \n> is amenable to incremental top-down updates is rather coincidental\n> and \n> not really planned.\n\nMmm. And the problem with 'thin' packs is that they normally allow\ndeltas the other way.\n\nI think the first step here would be to allow thin pack generation to\naccept a bounded range of commits, any of the objects within which may\nbe used as delta base candidates. That way, these \"top down\" thin packs\ncan be generated. Currently of course it just uses the --not and makes\n\"bottom up\" thin packs.\n\n> > Another solution would be to try to come up with some sort of stable\n> > sorting of objects so that packfile generated for the same\n> > parameters (endpoints) would be always byte-for-byte the same. But\n> > that might be difficult, or even impossible.\n>\n> And I don't want to commit to that either. Having some flexibility\n> in object ordering makes it possible to improve on the packing\n> heuristics.\n\nYou don't have to lose that for storage. It's only for generating the\nthin packs that it matters; also, the restriction is relaxed when it\ncomes to objects which are all being sent in the same pack, which can\nfreely delta amongst themselves in any direction.\n\nWhat did you think about the bundle slicing stuff?\n\nSam\n"},{"id":"121482","messageId":"alpine.LFD.2.00.0908212010440.6044@xanadu.home","threadId":"20626","inReplyTo":"200908212341.33324.jnareb@gmail.com","subject":"Re: Continue git clone after interruption","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-08-22T00:59:54Z","receivedAt":"2009-08-22T00:59:54Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 21 Aug 2009, Jakub Narebski wrote:\n\n> On Fri, 21 Aug 2009, Nicolas Pitre wrote:\n> > On Fri, 21 Aug 2009, Jakub Narebski wrote:\n> >> On Thu, 20 Aug 2009, Nicolas Pitre wrote:\n> >>> On Thu, 20 Aug 2009, Jakub Narebski wrote:\n> \n> >>>> It is however only 2.5 MB out of 37 MB that are resumable, which is 7%\n> >>>> (well, that of course depends on repository). Not that much that is\n> >>>> resumable.\n> >>> \n> >>> Take the Linux kernel then. It is more like 75 MB.\n> >> \n> >> Ah... good example.\n> >> \n> >> On the other hand Linux is fairly large project in terms of LoC, but\n> >> it had its history cut when moving to Git, so the ratio of git-archive\n> >> of HEAD to the size of packfile is overemphasized here.\n> > \n> > That doesn't matter. You still need that amount of data up front to do \n> > anything. And I doubt people with slow links will want the full history \n> > anyway, regardless if it goes backward 4 years or 18 years back.\n> \n> On the other hand unreliable link doesn't need to mean unreasonably\n> slow link.\n\nIn my experience speed and reliability are more or less tied together.\nAnd the slower is your link, the longer your transfer will last, the \ngreater are the chances for you to have troubles.\n\n> Hopefully GitTorrent / git-mirror-sync would finally come out of \n> vapourware and wouldn't share the fate of Duke Nukem Forever ;-),\n> and we would have this as an alternative to clone large repositories.\n\nWell... Maybe.\n\n> Well, supposedly there is some code, and last year GSoC project at\n> least shook the dust out of initial design and made it simplier, IIUC.\n\nThe BitTorrent protocol is a nifty thing (although I doubt the \nintertainment industry think so). But its efficiency relies on the fact \nthat many many people are expected to download the same stuff at the \nsame time. I have some doubts about the availability of the right \nconditions in the context of git for a BitTorrent-like protocol to work \nwell in practice. But this is Open Source and no one has to wait for me \nor anyone else to be convinced before attempting it and showing results \nto the world.\n\n> >> This was plain text RFC for the Git Packfile Protocol, generated from\n> >> rfc2629 XML sources at http://github.com/schacon/gitserver-rfc\n> > \n> > I suggest you track it down and prod/propose a version for merging in \n> > the git repository.\n> \n> Scott Chacon was (and is) CC-ed.\n\nHe might not have followed all our exchange so deeply in this thread \nthough. So maybe another thread with him in the To: field might be \nrequired to get his attention.\n\n> I don't know if you remember mentioned discussion about pack protocol, \n> stemming from the fact that some of git (re)implementations (Dulwich,\n> JGit) failed to implement it properly, where properly = same as \n> git-core, i.e. the original implementation in C... because there were\n> not enough documentation.\n\nYes I followed the discussion. I still think that, since that \ndocumentation exists now, that would be a good idea to have a copy \nincluded in the git sources.\n\n> > The format I was envisioning is really simple:\n> > \n> > First the size of the raw commit object data content in decimal, \n> > followed by a 0 byte, followed by the actual content of the commit \n> > object, followed by a 0 byte. (Note: this could be the exact same \n> > content as the canonical commit object data with the \"commit\" prefix, \n> > but as all the rest are all blob content this would be redundant.)\n> > \n> > Then, for each file:\n> > \n> > - The file mode in octal notation just as in tree objects\n> > - a space\n> > - the size of the file in decimal\n> > - a tab\n> > - the full path of the file\n> > - a 0 byte\n> > - the file content as found in the corresponding blob\n> > - a 0 byte\n> > \n> > And finally some kind of marker to indicate the end of the stream.\n> > \n> > Put the lot through zlib and you're done.\n> \n> So you don't want to just tack commit object (as extended pax header,\n> or a comment - if it is at all possible) to the existing 'tar' and\n> 'zip' archive formats. Probably better to design format from scratch.\n\nAs René Scharfe mentioned, the existing archive formats have limitations \nand complexities that we might simply avoid altogether by creating a \nsimpler format that is more likely to never fail to faithfully reproduce \na git revision content. Maybe the git-fast-import format could do it \neven better, and maybe not. That's an implementation detail that needs \nto be worked out once one is ready to get real with actual coding.\n\n\nNicolas\n"},{"id":"121487","messageId":"alpine.LFD.2.00.0908212324130.6044@xanadu.home","threadId":"20626","inReplyTo":"1250896025.19039.7.camel@maia.lan","subject":"Re: Continue git clone after interruption","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-08-22T03:37:31Z","receivedAt":"2009-08-22T03:37:31Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sat, 22 Aug 2009, Sam Vilain wrote:\n\n> On Fri, 2009-08-21 at 17:07 -0400, Nicolas Pitre wrote:\n> > > 2. There is support in git pack format to do 'deepening' of shallow\n> > > clone, which means that git can generate incrementals in top-down\n> > > order, _similar to how objects are ordered in packfile_.\n> > \n> > Well... the pack format was not meant for that \"support\". The fact\n> > that \n> > the typical object order used by pack-objects when serving fetch\n> > request \n> > is amenable to incremental top-down updates is rather coincidental\n> > and \n> > not really planned.\n> \n> Mmm. And the problem with 'thin' packs is that they normally allow\n> deltas the other way.\n\nSure. The pack format is flexible.\n\n> I think the first step here would be to allow thin pack generation to\n> accept a bounded range of commits, any of the objects within which may\n> be used as delta base candidates. That way, these \"top down\" thin packs\n> can be generated. Currently of course it just uses the --not and makes\n> \"bottom up\" thin packs.\n\nThe pack is still almost top-down. It's only the missing delta base \nthat are in the other direction, refering to objects you have locally \nand therefore older.\n\n> > > Another solution would be to try to come up with some sort of stable\n> > > sorting of objects so that packfile generated for the same\n> > > parameters (endpoints) would be always byte-for-byte the same. But\n> > > that might be difficult, or even impossible.\n> >\n> > And I don't want to commit to that either. Having some flexibility\n> > in object ordering makes it possible to improve on the packing\n> > heuristics.\n> \n> You don't have to lose that for storage. It's only for generating the\n> thin packs that it matters;\n\nWhat matters?\n\n> also, the restriction is relaxed when it\n> comes to objects which are all being sent in the same pack, which can\n> freely delta amongst themselves in any direction.\n\nThat's always the case within a pack, but only for REF_DELTA objects. \nThe OFS_DELTA objects have to be ordered. And yes, having deltas across \npacks is disallowed to avoid cycles and to keep the database robust. \nThe only exception is for thin packs, but those are never created on \ndisk. Thin packs are only used for transport and quickly \"fixed\" upon \nreception by appending the missing objects to them so they are not \n\"thin\" anymore.\n\n> What did you think about the bundle slicing stuff?\n\nIf I didn't comment on it already, then I probably missed it and have no \nidea.\n\n\nNicolas\n"},{"id":"121502","messageId":"1250920259.3644.11.camel@maia.lan","threadId":"20626","inReplyTo":"alpine.LFD.2.00.0908212324130.6044@xanadu.home","subject":"Re: Continue git clone after interruption","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2009-08-22T05:50:59Z","receivedAt":"2009-08-22T05:50:59Z","isPatch":false,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"On Fri, 2009-08-21 at 23:37 -0400, Nicolas Pitre wrote:\n> What did you think about the bundle slicing stuff?\n> \n> If I didn't comment on it already, then I probably missed it and have no \n> idea.\n\nI really tire of repeating myself for your sole benefit. Please show\nsome consideration for other people in the conversation by trying to\nlisten. Thank-you.\n\n> > I think the first step here would be to allow thin pack generation to\n> > accept a bounded range of commits, any of the objects within which may\n> > be used as delta base candidates. That way, these \"top down\" thin packs\n> > can be generated. Currently of course it just uses the --not and makes\n> > \"bottom up\" thin packs.\n> \n> The pack is still almost top-down. It's only the missing delta base \n> that are in the other direction, refering to objects you have locally \n> and therefore older.\n\nOk, but right now there's no way to specify that you want a thin pack,\nwhere the allowable base objects are *newer* than the commit range you\nwish to include.\n\nWhat I said in my other e-mail where I showed how well it works taking\na given bundle, and slicing it into a series of thin packs, was that it\nseems to add a bit of extra size to the resultant packs - best I got for\nslicing up the entire git.git run was about 20%. If this can be\nreduced to under 10% (say), then sending bundle slices would be quite\nreasonable by default for the benefit of making large fetches\nrestartable, or even spreadable across multiple mirrors.\n\nThe object sorting stuff is something of a distraction; it's required\nfor download spreading but not for the case at hand now.\n\nSam\n"},{"id":"121508","messageId":"alpine.LFD.2.00.0908220155240.6044@xanadu.home","threadId":"20626","inReplyTo":"1250920259.3644.11.camel@maia.lan","subject":"Re: Continue git clone after interruption","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-08-22T08:13:45Z","receivedAt":"2009-08-22T08:13:45Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sat, 22 Aug 2009, Sam Vilain wrote:\n\n> On Fri, 2009-08-21 at 23:37 -0400, Nicolas Pitre wrote:\n> > > What did you think about the bundle slicing stuff?\n> > \n> > If I didn't comment on it already, then I probably missed it and have no \n> > idea.\n> \n> I really tire of repeating myself for your sole benefit. Please show\n> some consideration for other people in the conversation by trying to\n> listen. Thank-you.\n\nI'm sorry but I have way too many emails to consider reading. This is \nlike ethernet: not a reliable transport, and lost packets means you have \nto retransmit. Cut and paste does wonders, or even a link to previous \npost.\n\n> > > I think the first step here would be to allow thin pack generation to\n> > > accept a bounded range of commits, any of the objects within which may\n> > > be used as delta base candidates. That way, these \"top down\" thin packs\n> > > can be generated. Currently of course it just uses the --not and makes\n> > > \"bottom up\" thin packs.\n> > \n> > The pack is still almost top-down. It's only the missing delta base \n> > that are in the other direction, refering to objects you have locally \n> > and therefore older.\n> \n> Ok, but right now there's no way to specify that you want a thin pack,\n> where the allowable base objects are *newer* than the commit range you\n> wish to include.\n\nSure you can. Try this:\n\n\t( echo \"-$(git rev-parse v1.6.4)\"; \\\n\t git rev-list --objects v1.6.2..v1.6.3 ) | \\\n\t\tgit pack-objects --progress --stdout > foo.pack\n\nThat'll give you a thin pack for the _new_ objects that _appeared_ \nbetween v1.6.2 and v1.6.3, but which external delta base objects are \nfound in v1.6.4.\n\nIf you want _all_ the objects that are referenced from commits between \nv1.6.2 and v1.6.3 then you just have to list them all for v1.6.2 in \naddition to the rest:\n\n\t( echo \"-$(git rev-parse v1.6.4)\"; \\\n\t git rev-list --objects v1.6.2..v1.6.3; \\\n\t git ls-tree -t -r v1.6.2 | cut -d' ' -f 3- | tr \"\\t\" \" \"; ) | \\\n\t\tgit pack-objects --progress --stdout > foo.pack\n\n> What I said in my other e-mail where I showed how well it works taking\n> a given bundle, and slicing it into a series of thin packs, was that it\n> seems to add a bit of extra size to the resultant packs - best I got for\n> slicing up the entire git.git run was about 20%. If this can be\n> reduced to under 10% (say), then sending bundle slices would be quite\n> reasonable by default for the benefit of making large fetches\n> restartable, or even spreadable across multiple mirrors.\n\nIn theory you could have about no overhead. That all depends how you \nslice the pack. If you want a pack to contain a fixed number of commits \n(such that all objects introduced by a given commit are all in the same \npack) then you are of course putting a constraint on the possible delta \nmatches and compression result might be suboptimal. In comparison, with \na single big pack a given blob can delta against a blob from a \ncompletely distant commit in the history graph if that provides a better \ncompression ratio.\n\nIf you slice your pack according to a size treshold, then you might \nconsider the --max-pack-size= argument to pack-objects. This currently \ndoesn't produce thin pack as delta objects whose base are stored in a \ndifferent pack than their base because of a pack split are simply not \nstored as delta. Only a few line of code would need to be modified in \norder to store those deltas nevertheless and turn those packs into thin \npacks, preserving the optimal delta match. Of course cross pack delta \nreference have to be REF_DELTA objects with headers about 16 to 17 bytes \nlarger than those of OFS_DELTA objects, so you will still have some \noverhead.\n\n> The object sorting stuff is something of a distraction; it's required\n> for download spreading but not for the case at hand now.\n\nWell, the idea of spreading small packs has its drawbacks. You still \nmight need to get a sizeable portion of them to get at least one usable \ncommit. And ideally you want the top commit in priority, which pretty \nmuch impose an ordering on the packs you're likely to want first, unlike \nwith BitTorrent where you don't care as you normally want all \nthe blocks anyway.\n\nIf the goal is to make for faster downloads, then you could simply make \na bundle, copy it on multiple server, and slice your download across \nthose servers. This has the disadvantage of being static data that \ndoubles the disk (and cache) usage. That doesn't work too well with \nshallow clones though.\n\nIf you were envisioning _clients_ à la BitTorrent putting up pack slices \ninstead, then in that case the slices have to be well defined entities, \nlike packs containing objects for known range of commits, but then we're \nback to the delta inefficiency I mentioned above. And again this might \nwork only if a lot of people are interested in the same repository at \nthe same time, and of course most people have no big insentive to \"seed\" \nonce they got their copy. So I'm not sure if that might work that well \nin practice.\n\nThis certainly still looks like a pretty cool project. But it is not \nall the cool stuff that works well in real conditions I'm afraid. Just \nmy opinion of course.\n\n\nNicolas\n"},{"id":"121560","messageId":"1251023829.8115.23.camel@maia.lan","threadId":"20626","inReplyTo":"alpine.LFD.2.00.0908220155240.6044@xanadu.home","subject":"Re: Continue git clone after interruption","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2009-08-23T10:37:09Z","receivedAt":"2009-08-23T10:37:09Z","isPatch":false,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"On Sat, 2009-08-22 at 04:13 -0400, Nicolas Pitre wrote:\n> > Ok, but right now there's no way to specify that you want a thin pack,\n> > where the allowable base objects are *newer* than the commit range you\n> > wish to include.\n> \n> Sure you can. Try this:\n> \n> \t( echo \"-$(git rev-parse v1.6.4)\"; \\\n> \t git rev-list --objects v1.6.2..v1.6.3 ) | \\\n> \t\tgit pack-objects --progress --stdout > foo.pack\n> \n> That'll give you a thin pack for the _new_ objects that _appeared_ \n> between v1.6.2 and v1.6.3, but which external delta base objects are \n> found in v1.6.4.\n\nAha. I guess I had made an assumption about where that '-' lets\npack-objects find deltas from that aren't true.\n\n> > What I said in my other e-mail where I showed how well it works taking\n> > a given bundle, and slicing it into a series of thin packs, was that it\n> > seems to add a bit of extra size to the resultant packs - best I got for\n> > slicing up the entire git.git run was about 20%. If this can be\n> > reduced to under 10% (say), then sending bundle slices would be quite\n> > reasonable by default for the benefit of making large fetches\n> > restartable, or even spreadable across multiple mirrors.\n> \n> In theory you could have about no overhead. That all depends how you \n> slice the pack. If you want a pack to contain a fixed number of commits \n> (such that all objects introduced by a given commit are all in the same \n> pack) then you are of course putting a constraint on the possible delta \n> matches and compression result might be suboptimal. In comparison, with \n> a single big pack a given blob can delta against a blob from a \n> completely distant commit in the history graph if that provides a better \n> compression ratio.\n [...]\n> If you were envisioning _clients_ à la BitTorrent putting up pack slices \n> instead, then in that case the slices have to be well defined entities, \n> like packs containing objects for known range of commits, but then we're \n> back to the delta inefficiency I mentioned above.\n\nI'll do some more experiments to try to quantify this in light of this\nnew information; I still think that if the overhead is marginal there\nare significant wins to this approach.\n\n> And again this might \n> work only if a lot of people are interested in the same repository at \n> the same time, and of course most people have no big insentive to \"seed\" \n> once they got their copy. So I'm not sure if that might work that well \n> in practice.\n\nThrow away terms like \"seeding\" and replace with \"mirroring\". Sites\nwhich currently house mirrors could potentially be helping serve git\nrepos, too. Popular projects could have many mirrors and on the edges\nof the internet, git servers could mirror many projects for users in\ntheir country.\n\nSam\n"}]}