{"thread":{"id":"28803","subject":"New Feature wanted: Is it possible to let git clone continue last break point?","startedAt":"2011-10-31T02:28:04Z","lastAt":"2011-11-05T10:00:15Z","messageCount":18,"participants":["netroby","Tay Ray Chuan","Jonathan Nieder","Jakub Narebski","Michael Schubert","Jeff King","Junio C Hamano","Shawn Pearce","Clemens Buchacher","Johannes Sixt","Nguyen Thai Ngoc Duy"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"178547","messageId":"CAEZo+gcj5q2UYnak1+1UG7pPzoeaUr=QLsiCiNXbC_n+JQbKQQ@mail.gmail.com","threadId":"28803","inReplyTo":"CAEZo+gfKVY-YgMjd=bEYzRV4-460kqDik-yVcQ9Xs=DoCZOMDg@mail.gmail.com","subject":"New Feature wanted: Is it possible to let git clone continue last break point?","fromName":"netroby","fromEmail":"hufeng1987@gmail.com","sentAt":"2011-10-31T02:28:04Z","receivedAt":"2011-10-31T02:28:04Z","isPatch":false,"sender":{"key":"hufeng1987@gmail.com","avatar":null},"body":"Is it possible to let git clone continue last break point.\nwhen we git clone very large project from the web,  we may face some\ninterupt, then we must clone it from zero .\n\nit is bad feeling for low  connection  speed users.\n\nplease help us out.\n\nwe need git clone continue last break point\n\nnetroby\n----------------------------------\nhttp://www.netroby.com\n"},{"id":"178548","messageId":"CALUzUxpDVq7GbgEJkgJ5pCw9XouOBZEsD=U4GYyHivX4ji_8FA@mail.gmail.com","threadId":"28803","inReplyTo":"CAEZo+gcj5q2UYnak1+1UG7pPzoeaUr=QLsiCiNXbC_n+JQbKQQ@mail.gmail.com","subject":"Re: New Feature wanted: Is it possible to let git clone continue last break point?","fromName":"Tay Ray Chuan","fromEmail":"rctay89@gmail.com","sentAt":"2011-10-31T04:00:08Z","receivedAt":"2011-10-31T04:00:08Z","isPatch":false,"sender":{"key":"rctay89@gmail.com","avatar":"https://avatars.githubusercontent.com/u/61553?v=4"},"body":"This is a hard problem that hasn't been solved. Year after year, it's\na GSoC proposal...\n\nWhat you can do is use --depth 1 with your git-clone; then \"extend\"\nthe depth incrementally.\n-- Cheers,Ray Chuan\n\nOn Mon, Oct 31, 2011 at 10:28 AM, netroby wrote:\n> Is it possible to let git clone continue last break point.\n> when we git clone very large project from the web,  we may face some\n> interupt, then we must clone it from zero .\n>\n> it is bad feeling for low  connection  speed users.\n>\n> please help us out.\n>\n> we need git clone continue last break point\n>\n> netroby\n> ----------------------------------\n> http://www.netroby.com\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n>\n"},{"id":"178554","messageId":"20111031090717.GA24978@elie.hsd1.il.comcast.net","threadId":"28803","inReplyTo":"CAEZo+gcj5q2UYnak1+1UG7pPzoeaUr=QLsiCiNXbC_n+JQbKQQ@mail.gmail.com","subject":"Re: New Feature wanted: Is it possible to let git clone continue last break point?","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-10-31T09:07:18Z","receivedAt":"2011-10-31T09:07:18Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nnetroby wrote:\n\n> Is it possible to let git clone continue last break point.\n> when we git clone very large project from the web,  we may face some\n> interupt, then we must clone it from zero .\n\nYou might find [1] useful as a stopgap (thanks, Tomas!).\n\nSomething like Jeff's \"priming the well with a server-specified\nbundle\" proposal[2] might be a good way to make the same trick\ntransparent to clients in the future.\n\nEven with that, later fetches, which grab a pack generated on the fly\nto only contain the objects not already fetched, are generally not\nresumable. Overcoming that would presumably require larger protocol\nchanges, and I don't know of anyone working on it. (My workaround\nwhen in a setup where this mattered was to use the old-fashioned\n\"dumb\" http protocol. It worked fine.)\n\nHope that helps,\nJonathan\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/181380\n[2] http://thread.gmane.org/gmane.comp.version-control.git/164569/focus=164701\n http://thread.gmane.org/gmane.comp.version-control.git/168906/focus=168912\n"},{"id":"178555","messageId":"m3obwx4j7s.fsf@localhost.localdomain","threadId":"28803","inReplyTo":"CAEZo+gcj5q2UYnak1+1UG7pPzoeaUr=QLsiCiNXbC_n+JQbKQQ@mail.gmail.com","subject":"Re: New Feature wanted: Is it possible to let git clone continue last break point?","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2011-10-31T09:14:01Z","receivedAt":"2011-10-31T09:14:01Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"netroby writes:\n\n> Is it possible to let git clone continue last break point.\n> when we git clone very large project from the web,  we may face some\n> interupt, then we must clone it from zero .\n> \n> it is bad feeling for low  connection  speed users.\n> \n> please help us out.\n> \n> we need git clone continue last break point\n\nResuming \"git clone\" is not currently possible in Git, and it would be\ndifficult to add such feature to Git; there were several attempts and\nneither succeeded.\n\nWhat you can do is generate a starter bundle out of your repository\n(using \"git bundle\"), and serve this file via HTTP / FTP / BitTorrent,\ni.e. some resumable transport. Then you \"git clone \",\nfix up configuration, and fetch the rest since bundle creation.\n\nThough this is possible only if it is your project... or can ask\nproject administrator to provide bundle.\n\n-- \nJakub Narębski\n"},{"id":"178556","messageId":"CAEZo+gfvmeQW2pyGF3tH-Zo1Eac0REZAZmYMpBdY2+gvvHvKPg@mail.gmail.com","threadId":"28803","inReplyTo":"20111031090717.GA24978@elie.hsd1.il.comcast.net","subject":"Re: New Feature wanted: Is it possible to let git clone continue last break point?","fromName":"netroby","fromEmail":"hufeng1987@gmail.com","sentAt":"2011-10-31T09:16:01Z","receivedAt":"2011-10-31T09:16:01Z","isPatch":false,"sender":{"key":"hufeng1987@gmail.com","avatar":null},"body":"the example :\n\ni want to clone the freebsd and linux kernel git repo , to view their\nsource code.\n\ngit://github.com/freebsd/freebsd.git\n\ngit://github.com/torvalds/linux.git\n\n\nthey are big project, so they are huge.\n\nthanks for your tips. it will let me have a try .\n\n\nI am current using 256K Adsl , so it is very not stable when clone in progress.\n\nnetroby\n----------------------------------\nhttp://www.netroby.com\n\n\n\nOn Mon, Oct 31, 2011 at 17:07, Jonathan Nieder wrote:\n> Hi,\n>\n> netroby wrote:\n>\n>> Is it possible to let git clone continue last break point.\n>> when we git clone very large project from the web,  we may face some\n>> interupt, then we must clone it from zero .\n>\n> You might find [1] useful as a stopgap (thanks, Tomas!).\n>\n> Something like Jeff's \"priming the well with a server-specified\n> bundle\" proposal[2] might be a good way to make the same trick\n> transparent to clients in the future.\n>\n> Even with that, later fetches, which grab a pack generated on the fly\n> to only contain the objects not already fetched, are generally not\n> resumable.  Overcoming that would presumably require larger protocol\n> changes, and I don't know of anyone working on it.  (My workaround\n> when in a setup where this mattered was to use the old-fashioned\n> \"dumb\" http protocol.  It worked fine.)\n>\n> Hope that helps,\n> Jonathan\n>\n> [1] http://thread.gmane.org/gmane.comp.version-control.git/181380\n> [2] http://thread.gmane.org/gmane.comp.version-control.git/164569/focus=164701\n>    http://thread.gmane.org/gmane.comp.version-control.git/168906/focus=168912\n>\n"},{"id":"178560","messageId":"4EAE9941.2080609@elegosoft.com","threadId":"28803","inReplyTo":"m3obwx4j7s.fsf@localhost.localdomain","subject":"Re: New Feature wanted: Is it possible to let git clone continue last break point?","fromName":"Michael Schubert","fromEmail":"mschub@elegosoft.com","sentAt":"2011-10-31T12:49:05Z","receivedAt":"2011-10-31T12:49:05Z","isPatch":false,"sender":{"key":"mschub@elegosoft.com","avatar":null},"body":"On 10/31/2011 10:14 AM, Jakub Narebski wrote:\n> netroby writes:\n> \n>> Is it possible to let git clone continue last break point.\n>> when we git clone very large project from the web, we may face some\n>> interupt, then we must clone it from zero .\n>>\n>> it is bad feeling for low connection speed users.\n>>\n>> please help us out.\n>>\n>> we need git clone continue last break point\n> \n> Resuming \"git clone\" is not currently possible in Git, and it would be\n> difficult to add such feature to Git; there were several attempts and\n> neither succeeded.\n> \n> What you can do is generate a starter bundle out of your repository\n> (using \"git bundle\"), and serve this file via HTTP / FTP / BitTorrent,\n> i.e. some resumable transport. Then you \"git clone \",\n> fix up configuration, and fetch the rest since bundle creation.\n\nThere's also a \"git bundler service\":\n\nhttp://comments.gmane.org/gmane.comp.version-control.git/181380\n"},{"id":"178727","messageId":"20111102220614.GB14108@sigill.intra.peff.net","threadId":"28803","inReplyTo":"20111031090717.GA24978@elie.hsd1.il.comcast.net","subject":"Re: New Feature wanted: Is it possible to let git clone continue last break point?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-11-02T22:06:14Z","receivedAt":"2011-11-02T22:06:14Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Oct 31, 2011 at 04:07:18AM -0500, Jonathan Nieder wrote:\n\n> Something like Jeff's \"priming the well with a server-specified\n> bundle\" proposal[2] might be a good way to make the same trick\n> transparent to clients in the future.\n\nYes, that is one of the use cases I hope to address. But it will require\nthe publisher specifying a mirror location (it's possible we could add\nsome kind of automagic \"hit a bundler service first\" config option,\nthough I fear that the existing small-time bundler services would\ncrumble under the load).\n\nSo in the general case (and in the meantime), you may have to learn to\nmanually prime the repo using a bundle.\n\nI haven't started on the patches for communicating mirror sites between\nthe server and client, but I did just write some patches to handle \"git\nfetch http://host/path/to/file.bundle\" automatically, which is the first\nstep. They need a few finishing touches and some testing, though.\n\n> Even with that, later fetches, which grab a pack generated on the fly\n> to only contain the objects not already fetched, are generally not\n> resumable. Overcoming that would presumably require larger protocol\n> changes, and I don't know of anyone working on it. (My workaround\n> when in a setup where this mattered was to use the old-fashioned\n> \"dumb\" http protocol. It worked fine.)\n\nMy goal was for the mirror communication between client and server to be\nsomething like:\n\n - if you don't have object XXXXXX, then prime with URL\n http://host/bundle1\n\n - if you don't have object YYYYYY, then prime with URL\n http://host/bundle2\n\nand so forth. A cloning client would grab the first bundle, then the\nsecond, and then hit the real repo via the git protocol. A client who\nhad previously cloned might have XXX, but would now grab bundle2, and\nthen hit the real repo.\n\nSo depending on how often the server side feels like creating new\nbundles, you would get most of the changes via bundles, and then only\nbe getting a small number of objects via git.\n\nThe downside of cumulative fetching is that the bundles can only serve\nwell-known checkpoints. So if you have a timeline like this:\n\n t0: server publishes bundle/mirror config with one line (the XXX bit\n above)\n\n t1: you clone, getting the whole bundle. No waste, because you had\n nothing in the first place, and you needed everything.\n\n t2: you fetch again, getting N commits worth of history via the git\n protocol\n\n t3: server decides a lot of new objects (let's say M commits worth)\n have accumulated, and generates a new line (the YYY line).\n\n t4: you fetch, see that you don't yet have YYY, and grab the second\n bundle\n\nBut in t4 you grabbed a bundle containing M commits, when you already\nhad the first N of them. So you actually wasted bandwidth getting\nobjects you already had. The only benefit is that you grabbed a static\nfile, which is resumable.\n\nSo I suspect there is some black magic involved in deciding when to\ncreate a new bundle, and at what tip. If you create a bundle once a\nmonth, but include only commits up to a week ago, then people pulling\nweekly will never grab the bundle, but people pulling less frequently\nwill get the whole month as a bundle.\n\nA secondary issue is also that in a scheme like this, your mirror list\nwill grow without bound. So you'd want to periodically repack everything\ninto a single bundle. But then people who are fetching wouldn't want\nthat, as it is just an exacerbated version of the same problem above.\n\nWhich is all a roundabout way of saying that the git protocol is really\nthe sane way to do efficient transfers. An alternative, much simpler\nscheme would be for the server to just say:\n\n - if you have nothing, then prime with URL http://host/bundle\n\nAnd then _only_ clone would bother with checking mirrors. People doing\nfetch would be expected to do it often enough that not being resumable\nisn't a big deal.\n\n-Peff\n"},{"id":"178732","messageId":"7vwrbigna7.fsf@alter.siamese.dyndns.org","threadId":"28803","inReplyTo":"20111102220614.GB14108@sigill.intra.peff.net","subject":"Re: New Feature wanted: Is it possible to let git clone continue last break point?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-11-02T22:41:36Z","receivedAt":"2011-11-02T22:41:36Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> Which is all a roundabout way of saying that the git protocol is really\n> the sane way to do efficient transfers. An alternative, much simpler\n> scheme would be for the server to just say:\n>\n> - if you have nothing, then prime with URL http://host/bundle\n>\n> And then _only_ clone would bother with checking mirrors. People doing\n> fetch would be expected to do it often enough that not being resumable\n> isn't a big deal.\n\nI think that is a sensible place to start.\n\nA more fancy conditional \"If you have X then fetch this, if you have Y\nfetch that, ...\" sounds nice but depending on what branch you are fetching\nthe answer has to be different. If we were to do that, the natural place\nfor the server to give the redirect instruction to the client is after the\nclient finishes saying \"want\", and before the client starts saying \"have\".\n"},{"id":"178734","messageId":"20111102232735.GA17466@sigill.intra.peff.net","threadId":"28803","inReplyTo":"7vwrbigna7.fsf@alter.siamese.dyndns.org","subject":"Re: New Feature wanted: Is it possible to let git clone continue last break point?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-11-02T23:27:35Z","receivedAt":"2011-11-02T23:27:35Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Nov 02, 2011 at 03:41:36PM -0700, Junio C Hamano wrote:\n\n> Jeff King writes:\n> \n> > Which is all a roundabout way of saying that the git protocol is really\n> > the sane way to do efficient transfers. An alternative, much simpler\n> > scheme would be for the server to just say:\n> >\n> > - if you have nothing, then prime with URL http://host/bundle\n> >\n> > And then _only_ clone would bother with checking mirrors. People doing\n> > fetch would be expected to do it often enough that not being resumable\n> > isn't a big deal.\n> \n> I think that is a sensible place to start.\n\nOK. That had been my original intent, but somebody (you?) mentioned the\n\"if you have X\" thing at the GitTogether, which got me thinking.\n\nI don't mind starting slow, as long as we don't paint ourselves into a\ncorner for future expansion. I'll try to design the data format for\nspecifying the mirror locations with that extension in mind.\n\nEven if the bundle thing ends up too wasteful, it may still be useful to\noffer a \"if you don't have X, go see Y\" type of mirror when \"Y\" is\nsomething efficient, like git:// at a faster host (i.e., the \"I built 3\ncommits on top of Linus\" case).\n\n> A more fancy conditional \"If you have X then fetch this, if you have Y\n> fetch that, ...\" sounds nice but depending on what branch you are fetching\n> the answer has to be different. If we were to do that, the natural place\n> for the server to give the redirect instruction to the client is after the\n> client finishes saying \"want\", and before the client starts saying \"have\".\n\nAgreed. I was really trying to avoid protocol extensions, though, at\nleast for an initial version. I'd like to see how far we can get doing\nthe simplest thing.\n\n-Peff\n"},{"id":"178740","messageId":"CAJo=hJtt8vjB5oU+tEabN2AS7c-24bMHNwQSoWtZYtjjrR3d7Q@mail.gmail.com","threadId":"28803","inReplyTo":"20111102232735.GA17466@sigill.intra.peff.net","subject":"Re: New Feature wanted: Is it possible to let git clone continue last break point?","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2011-11-03T00:06:53Z","receivedAt":"2011-11-03T00:06:53Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Wed, Nov 2, 2011 at 16:27, Jeff King wrote:\n> On Wed, Nov 02, 2011 at 03:41:36PM -0700, Junio C Hamano wrote:\n>> Jeff King writes:\n>>\n>> > Which is all a roundabout way of saying that the git protocol is really\n>> > the sane way to do efficient transfers. An alternative, much simpler\n>> > scheme would be for the server to just say:\n>> >\n>> >   - if you have nothing, then prime with URL http://host/bundle\n>> >\n>> > And then _only_ clone would bother with checking mirrors. People doing\n>> > fetch would be expected to do it often enough that not being resumable\n>> > isn't a big deal.\n>>\n>> I think that is a sensible place to start.\n\nYup, I agree. The \"repo\" tool used by Android does this in Python\nright now[1]. Its a simple hack, if the protocol is HTTP or HTTPS the\nclient first tries to download $URL/clone.bundle. My servers have\nrules that trap on */clone.bundle and issue an HTTP 302 Found response\nto direct the client to a CDN. Works. :-)\n\n[1] http://code.google.com/p/git-repo/source/detail?r=f322b9abb4cadc67b991baf6ba1b9f2fbd5d7812&name=stable\n\n> OK. That had been my original intent, but somebody (you?) mentioned the\n> \"if you have X\" thing at the GitTogether, which got me thinking.\n>\n> I don't mind starting slow, as long as we don't paint ourselves into a\n> corner for future expansion. I'll try to design the data format for\n> specifying the mirror locations with that extension in mind.\n\nRight. Aside from the fact that $URL/clone.bundle is perhaps a bad way\nto decide on the URL to actually fetch (and isn't supportable over\ngit:// or ssh://)... we should start with the clone case and worry\nabout incremental updates later.\n\n> Even if the bundle thing ends up too wasteful, it may still be useful to\n> offer a \"if you don't have X, go see Y\" type of mirror when \"Y\" is\n> something efficient, like git:// at a faster host (i.e., the \"I built 3\n> commits on top of Linus\" case).\n\nActually, I really think the bundle thing is wasteful. Its a ton of\nadditional disk. Hosts like kernel.org want to use sendfile() when\npossible to handle bulk transfers. git:// is not efficient for them\nbecause we don't have sendfile() capability.\n\nIts also expensive for kernel.org to create each Git repository twice\non disk. The disk is cheap. Its the kernel buffer cache that is damned\nexpensive. Assume for a minute that Linus' kernel repository is a\npopular thing to access. If 400M of that history is available in a\nnormal pack file on disk, and again 400M is available as a \"clone\nbundle thingy\", kernel.org now has to eat 800M of disk buffer cache\nfor that one Git repository, because both of those files are going to\nbe hot.\n\nI think I messed up with \"repo\" using a Git bundle file as its data\nsource. What we should have done was a bog standard pack file. Then\nthe client can download the pack file into the .git/objects/pack\ndirectory and just generate the index, reusing the entire dumb\nprotocol transport logic. It also allows the server to pass out the\nsame file the server retains for the repository itself, and thus makes\nthe disk buffer cache only 400M for Linus' repository.\n\n> Agreed. I was really trying to avoid protocol extensions, though, at\n> least for an initial version. I'd like to see how far we can get doing\n> the simplest thing.\n\nOne (maybe dumb idea I had) was making the $GIT_DIR/objects/info/packs\nfile contain other lines to list reference tips at the time the pack\nwas made. The client just needs the SHA-1s, it doesn't necessarily\nneed the branch names themselves. A client could initialize itself by\ngetting this set of references, creating temporary dummy references at\nthose SHA-1s, and downloading the corresponding pack file, indexing\nit, then resuming with a normal fetch.\n\nThen we wind up with a git:// or ssh:// protocol extension that\nenables sendfile() on an entire pack, and to provide the matching\nobjects/info/packs data to help a client over git:// or ssh://\ninitialize off the existing pack files.\n\n\nObviously there is the existing security feature that over git:// or\nssh:// (or even smart HTTP), a deleted or rewound reference stops\nexposing the content in the repository that isn't reachable from the\nother reference tips. The repository owner / server administrator will\nhave to make a choice here, either the existing packs are not exposed\nas available via sendfile() until after GC can be run to rebuild them\naround the right content set, or they are exposed and the time to\nexpunge/hide an unreferenced object is expanded until the GC completes\n(rather than being immediate after the reference updates).\n\nBut either way, I like the idea of coupling the \"resumable pack\ndownload\" to the *existing* pack files, because this is easy to deal\nwith. If you do have a rewind/delete and need to expunge content,\nusers/administrators already know how to run `git gc --expire=now` to\naccomplish a full erase. Adding another thing with bundle files\nsomewhere else that may or may not contain the data you want to erase\nand remembering to clean that up is not a good idea.\n"},{"id":"178750","messageId":"20111103024248.GA9492@sigill.intra.peff.net","threadId":"28803","inReplyTo":"CAJo=hJtt8vjB5oU+tEabN2AS7c-24bMHNwQSoWtZYtjjrR3d7Q@mail.gmail.com","subject":"Re: New Feature wanted: Is it possible to let git clone continue last break point?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-11-03T02:42:48Z","receivedAt":"2011-11-03T02:42:48Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Nov 02, 2011 at 05:06:53PM -0700, Shawn O. Pearce wrote:\n\n> Yup, I agree. The \"repo\" tool used by Android does this in Python\n> right now[1]. Its a simple hack, if the protocol is HTTP or HTTPS the\n> client first tries to download $URL/clone.bundle. My servers have\n> rules that trap on */clone.bundle and issue an HTTP 302 Found response\n> to direct the client to a CDN. Works. :-)\n\nI thought of doing something like that, but I wanted to be able to make\ncross-domain links. The \"302 to a CDN\" thing is a clever hack, but it\nrequires more control of the webserver than some users might have. And\nof course it doesn't work for the \"redirect to git:// on a different\nserver\" trick. Or redirect from \"git://\".\n\nMy thought of having it in \"refs/mirrors\" is only slightly less hacky,\nbut I think covers all of those cases. :)\n\n> > Even if the bundle thing ends up too wasteful, it may still be useful to\n> > offer a \"if you don't have X, go see Y\" type of mirror when \"Y\" is\n> > something efficient, like git:// at a faster host (i.e., the \"I built 3\n> > commits on top of Linus\" case).\n> \n> Actually, I really think the bundle thing is wasteful. Its a ton of\n> additional disk. Hosts like kernel.org want to use sendfile() when\n> possible to handle bulk transfers. git:// is not efficient for them\n> because we don't have sendfile() capability.\n\nI didn't quite parse this. You say it is wasteful, but then indicate\nthat it can use sendfile(), which is a good thing.\n\nHowever, I do agree with this:\n\n> Its also expensive for kernel.org to create each Git repository twice\n> on disk. The disk is cheap. Its the kernel buffer cache that is damned\n> expensive. Assume for a minute that Linus' kernel repository is a\n> popular thing to access. If 400M of that history is available in a\n> normal pack file on disk, and again 400M is available as a \"clone\n> bundle thingy\", kernel.org now has to eat 800M of disk buffer cache\n> for that one Git repository, because both of those files are going to\n> be hot.\n\nDoubling the disk cache required is evil and ugly. I was hoping it\nwouldn't matter because the bundle would be hosted on some far-away CDN\nserver anyway, though. But that is highly dependent on your setup. And\nit's really just glossing over the fact that you have twice as many\nservers. ;)\n\n> I think I messed up with \"repo\" using a Git bundle file as its data\n> source. What we should have done was a bog standard pack file. Then\n> the client can download the pack file into the .git/objects/pack\n> directory and just generate the index, reusing the entire dumb\n> protocol transport logic. It also allows the server to pass out the\n> same file the server retains for the repository itself, and thus makes\n> the disk buffer cache only 400M for Linus' repository.\n\nThat would be cool, but what about ref tips? The pack is just a big blob\nof objects, but we need ref tips to advertise to the server when we come\nback via the smart protocol. We can make a guess about them, obviously,\nbut it would be nice to communicate them. I guess the mirror data could\ninclude the tips and a pointer to a pack file.\n\nAnother issue with packs is that they generally aren't supposed to be\n--thin on disk, whereas bundles can be. So I could point you to a\nsuccession of bundles. Which is maybe a feature, or maybe just makes\nthings insanely complex[1].\n\n> One (maybe dumb idea I had) was making the $GIT_DIR/objects/info/packs\n> file contain other lines to list reference tips at the time the pack\n> was made.\n\nSo yeah, that's another solution to the ref tip thingy, and that would\nwork. I don't think it would make a big difference whether the tips were\nin the \"mirror\" file, or alongside the packfile. The latter I guess\nmight make administration easier. The \"real\" repo points its mirror one\ntime to a static pack store, and then the client goes and grabs whatever\nit can from that store.\n\n> Then we wind up with a git:// or ssh:// protocol extension that\n> enables sendfile() on an entire pack, and to provide the matching\n> objects/info/packs data to help a client over git:// or ssh://\n> initialize off the existing pack files.\n\nI think we can get around this by pointing git:// clients, either via\nprotocol extension or via a magic ref, to an http pack store. Sure, it's\nan extra TCP connection, but that's not a big deal compared to doing an\ninitial clone of most repos.\n\nSo the sendfile() stuff would always happen over http.\n\n> But either way, I like the idea of coupling the \"resumable pack\n> download\" to the *existing* pack files, because this is easy to deal\n> with.\n\nYeah, I'm liking that idea. In reference to my [1] above, what I've\nstarted with is making:\n\n git fetch http://host/foo.bundle\n\nwork automatically. And it does work. But it actually spools the bundle\nto disk and then unpacks from it, rather than placing it right into the\nobjects/pack directory. I did this because:\n\n 1. We have to feed it to \"index-pack --fix-thin\", because bundles can\n be thin. So they're not suitable for sticking right into the pack\n directory.\n\n 2. We could feed it straight to an index-pack pipe, but then we don't\n have a byte-for-byte file on disk to resume an interrupted\n transfer.\n\nBut spooling sucks, of course. It means we use twice as much disk space\nduring the index-pack as we would otherwise need to, not to mention the\nlatency of not starting the index-pack until we get the whole file.\n\nPulling down a non-thin packfile makes the problem go away. We can spool\nit right into objects/pack, index it on the fly, and if all is well,\nmove it into its final filename. If the transfer is interrupted, you\ndrop what's been indexed so far, finish the transfer, and then re-start\nthe indexing from scratch (actually, the \"on the fly\" would probably\ninvolve teaching index-pack to be clever about incrementally reading a\npartially written file, but it should be possible).\n\n-Peff\n"},{"id":"178757","messageId":"CAJo=hJt2kU10r5rq23qgimtW8dmzu-N92vjO_hNBbVVsKSpDmg@mail.gmail.com","threadId":"28803","inReplyTo":"20111103024248.GA9492@sigill.intra.peff.net","subject":"Re: New Feature wanted: Is it possible to let git clone continue last break point?","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2011-11-03T04:19:03Z","receivedAt":"2011-11-03T04:19:03Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Wed, Nov 2, 2011 at 19:42, Jeff King wrote:\n> On Wed, Nov 02, 2011 at 05:06:53PM -0700, Shawn O. Pearce wrote:\n>\n> I thought of doing something like that, but I wanted to be able to make\n> cross-domain links. The \"302 to a CDN\" thing is a clever hack, but it\n> requires more control of the webserver than some users might have. And\n> of course it doesn't work for the \"redirect to git:// on a different\n> server\" trick. Or redirect from \"git://\".\n\nI agree. Later I said I regret this being a bundle file. I also regret\nit being this $URL/clone.bundle thing. Its a reasonable quick hack in\nPython for repo. Its cheap for my servers to respond 404 Not Found or\n302 Found, and cheap to use the CDN. But it isn't the right solution\nfor git-core.\n\nIt has given us some useful information already in the context of\nandroid.googlesource.com. It appears to work quite well for\ndistributing the large Android operating system. So the notion of\nmaking packs available from another URL than the main repository, and\ndoing it as primarily a pack and not the native Git protocol, with a\nfollow-up incremental fetch to bring the client current seems to work.\n :-)\n\n> My thought of having it in \"refs/mirrors\" is only slightly less hacky,\n> but I think covers all of those cases. :)\n\nRight, but this would have been a bit more work for me to code in Python. :-)\n\nLong term this may be a better approach, because it does allow the\nuser to control the redirect without having full control over their\nHTTP server. It also supports redirections across protocols like you\nnoted above. So its probably the direction we will see git-core take.\n\n>> Actually, I really think the bundle thing is wasteful.... sendfile() capability.\n>\n> I didn't quite parse this. You say it is wasteful, but then indicate\n> that it can use sendfile(), which is a good thing.\n\nApparently I was babbling. Based on what else you say, we agree. That\nis good enough for me.\n\n> However, I do agree with this:\n>\n>> Its also expensive for kernel.org to create each Git repository twice\n>> on disk. The disk is cheap. Its the kernel buffer cache that is damned\n>> expensive.\n>\n> Doubling the disk cache required is evil and ugly. I was hoping it\n> wouldn't matter because the bundle would be hosted on some far-away CDN\n> server anyway, though. But that is highly dependent on your setup. And\n> it's really just glossing over the fact that you have twice as many\n> servers. ;)\n\nRight. :-)\n\nIn my opinion this is the important part. We shouldn't double the disk\nusage required to support this. Most users can't afford the extra disk\ncache or the extra server required to make this work well. But they\ncan use sendfile() on the server they have and get a lot of\nimprovement in clone speed due to lower system load, plus resumable\nclone for the relatively stable history part.\n\n> Another issue with packs is that they generally aren't supposed to be\n> --thin on disk, whereas bundles can be. So I could point you to a\n> succession of bundles. Which is maybe a feature, or maybe just makes\n> things insanely complex[1].\n\nActually we can store --thin on disk safely. Don't laugh until you\nfinish reading it through.\n\nTo build an incremental pack we modify pack-objects to construct a\ncompleted thin pack on disk. Build up the list of objects that you\nwant in the thin pack, as though it were thin. Use REF_DELTA format to\nreference objects that are not in this set but are delta bases. Copy\nthe necessary delta bases from the base pack over to the thin pack, at\nthe end just like it would be if received over the wire. The pack is\nnow self-contained like its supposed to be, but the tail of it is\nredundant information.\n\nIf you cache alongside of the pack the \"thin\" object count, the cut\noffset of the thin vs. completed bases, and the SHA-1 of the \"thin\"\npack, you can serve the \"thin\" pack by copying the header, then the\nregion of the file up to the cut point, and the final SHA-1. And there\nare no pack file format changes involved.\n\n:-)\n\nObviously this has some downside. Using REF_DELTA instead of OFS_DELTA\nfor the relatively small number of references from the \"thin\" part to\nthe completed part at the tail isn't a big disk space overhead. The\nbig overhead is storing the boundary data that served as delta bases\nat the tail of this incremental pack. But we already do that when you\ntransfer this section of data over the network and it was more than\n100 objects.\n\nSo I think we can get away with doing this. The serving repository is\nin no worse state than if the owner had just pushed all of that\nincremental stuff into the serving repository and it completed as a\nthin pack. With only 2 packs in the serving repository (e.g. the\nhistorical stuff that is stable, and the incremental current thin pack\n+ completed bases), git gc --auto wouldn't even kick in to GC this\nthing for a while *anyway*. So we already probably have a ton of\nrepositories in the wild that exhibit this disk layout and space\nusage, and nobody has complained about it.\n\nFor a server admin or repository owner who cares about his user's\nresumable clone support, carrying around a historical pack and a\nsingle new incremental pack for say 2-3 months before repacking the\nentire thing down to 1 new historical pack... the disk space and\nadditional completed base data is an acceptable cost. We already do\nit.\n\nClients can figure out whether or not they should use an incremental\npack download vs the native Git protocol if the incremental pack does\nlike a bundle does and stores the base information alongside of it.\nActually you don't want the base (the ^ lines in a bundle), but the\nimmediate child of those. If the client has any of those children,\nthere is some chance the client has other objects in the pack and\nshould favor native protocol. But if the client has none of those base\nchildren, but does have the base, it may be more efficient to download\nthe entire pack to bring the client current.\n\nThe problem with incremental pack updates is balancing the number of\nround-trip requests against the update rate of the repository against\nthe polling frequency of the client. Its not an easy thing to solve.\n\nHowever, we may be able to do better if the server can do a reasonably\nfast concat of these thin pack slices together by writing a new object\nheader and computing the SHA-1 trailer as it goes. Instead of\ncomputing actual graph connectivity, just concat packs together\nbetween the base children and the requested tips. This probably\nrequires that the client ask for every branch (e.g. the typical\nrefs/heads/*:refs/remotes/origin/* refspec) and that branches didn't\nrewind. But I think this is so common its perhaps worthwhile to look\ninto optimizing. But note we can do this in the native protocol at the\nserver side without telling the client anything, or changing the\nprotocol. It just isn't resumable without a bit more glue to have a\nstate marker available to the client. Nor does it work on a CDN\nwithout giving the client more information. :-)\n\n> So the sendfile() stuff would always happen over http.\n\nI'm OK with that. I was just saying we may be able to also support\nsendfile() over git:// if the repository owner / git-daemon owner\nwants us to. Or if not sendfile(), a simple read-write loop that\ndoesn't have to look at the data, since the client will validate it\nall.\n\n> Yeah, I'm liking that idea. In reference to my [1] above, what I've\n> started with is making:\n>\n>  git fetch http://host/foo.bundle\n\nThis should work, whether or not we use it for resumable clone. Its\njust nice to have that tiny bit of extra glue to make it easy to pull\na bundle. So I'd like this too. :-)\n\n> Pulling down a non-thin packfile makes the problem go away. We can spool\n> it right into objects/pack, index it on the fly, and if all is well,\n> move it into its final filename. If the transfer is interrupted, you\n> drop what's been indexed so far, finish the transfer, and then re-start\n> the indexing from scratch (actually, the \"on the fly\" would probably\n> involve teaching index-pack to be clever about incrementally reading a\n> partially written file, but it should be possible).\n\nI wonder if we can teach index-pack to work with a thin pack on disk\nand complete that by appending to the file, in addition to the\nstreaming from stdin it supports. Seems like that should be possible.\nSo then you could save a thin pack to a temp file on disk, and thus\ncould split a bundle header from its pack content, saving them into\ntwo different temp files, allowing index-pack to avoid copying the\npack portion if its non-thin, or if its a huge thin pack.\n\nI did think about doing this in \"repo\" and decided it was complex, and\nnot worth the effort. So we spool. 2G+ bundles. Its not the most\npleasant user experience. If I had more time, I would have tried to\nsplit the bundle header from the pack and written the pack directly\noff for index-pack to read from disk.\n"},{"id":"178815","messageId":"20111104085633.GA13924@ecki","threadId":"28803","inReplyTo":"CAJo=hJt2kU10r5rq23qgimtW8dmzu-N92vjO_hNBbVVsKSpDmg@mail.gmail.com","subject":"Re: New Feature wanted: Is it possible to let git clone continue last break point?","fromName":"Clemens Buchacher","fromEmail":"drizzd@aon.at","sentAt":"2011-11-04T08:56:33Z","receivedAt":"2011-11-04T08:56:33Z","isPatch":false,"sender":{"key":"drizzd@gmx.net","avatar":"https://avatars.githubusercontent.com/u/59082?v=4"},"body":"On Wed, Nov 02, 2011 at 09:19:03PM -0700, Shawn Pearce wrote:\n> \n> [...] But they can use sendfile() on the server they have and get\n> a lot of improvement in clone speed due to lower system load,\n> plus resumable clone for the relatively stable history part.\n\nSetting aside the system load issue for now, couldn't we simply do\nthe following?\n\n1. Figure out HAVE's and WANT's [1], based on which an ad-hoc pack\n will be made and sent to the client.\n2. Cache the information on disk (not the pack but the information\n to re-create it), and give the client a 'ticket number' which\n corresponds to that ad-hoc pack.\n3. Start downloading the packfile\n\nWhen the connection drops, we can resume like this:\n\n1. Send the previously received 'ticket number', and the amount of\n previously received data.\n2. Re-generate the pack from the HAVE's and WANT's cached under\n 'ticket number'. (This may fail if the repo state has changed\n such that previously accessible refs are now inaccessible.)\n3. Resume download of that pack.\n\nThe upside of this approach is that it would work automatically,\nwithout any manual setup by the server admin. All the previously\ndiscussed ideas skip the step where we figure out the HAVE's and\nWANT's. And to me that implies that we manually prepare a packfile\nsomewhere on disk, which contains what the user usually WANT's and\nis allowed to have (think per-branch access control). Even if we\ndisregard access control, wouldn't that at least require the server\nto create a \"clean\" pack which does not contain any objects from\nthe reflog?\n\nThe whole mirror thing could be pursued independently of the resume\ncapability, and if each git repo is capable of resuming the mirrors\ncan be plain git clones as well.\n\nJust my 2 cents,\nClemens\n"},{"id":"178817","messageId":"4EB3B1E7.7080507@viscovery.net","threadId":"28803","inReplyTo":"20111104085633.GA13924@ecki","subject":"Re: New Feature wanted: Is it possible to let git clone continue last break point?","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2011-11-04T09:35:35Z","receivedAt":"2011-11-04T09:35:35Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 11/4/2011 9:56, schrieb Clemens Buchacher:\n> Cache ... not the pack but the information\n> to re-create it...\n\nIt has been discussed. It doesn't work. Because with threaded pack\ngeneration, the resulting pack is not deterministic.\n\n-- Hannes\n"},{"id":"178825","messageId":"CAJo=hJtsiEEHA33CQn1MCvb7vFv7uEF+U292YgBa7EWv7P8Jng@mail.gmail.com","threadId":"28803","inReplyTo":"4EB3B1E7.7080507@viscovery.net","subject":"Re: New Feature wanted: Is it possible to let git clone continue last break point?","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2011-11-04T14:22:20Z","receivedAt":"2011-11-04T14:22:20Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Fri, Nov 4, 2011 at 02:35, Johannes Sixt wrote:\n> Am 11/4/2011 9:56, schrieb Clemens Buchacher:\n>> Cache ... not the pack but the information\n>>    to re-create it...\n>\n> It has been discussed. It doesn't work. Because with threaded pack\n> generation, the resulting pack is not deterministic.\n\nThe information to create a pack for a repository with 2M objects\n(e.g. Linux kernel tree) is *at least* 152M of data. This is just a\nfirst order approximation of what it takes to write out the 2M SHA-1s,\nalong with say a 4 byte length so you can find given an offset\nprovided by the client roughly where to resumse in the object stream.\nThis is like 25% of the pack size itself. Ouch.\n\nThis data is still insufficient to resume from. A correct solution\nwould allow you to resume in the middle of an object, which means we\nalso need to store some sort of indicator of which representation was\nchosen from an existing pack file for object reuse. Which adds more\ndata to the stream. And then there is the not so simple problem of how\nto resume in the middle of an object that was being recompressed on\nthe fly, such as a large loose object.\n\nBy the time you get done with all of that, your \"ticket\" might as well\nbe the name of a pack file. And your \"resume information\" is just a\npack file itself. Which would be very expensive to recreate.\n"},{"id":"178834","messageId":"m3k47f51f0.fsf@localhost.localdomain","threadId":"28803","inReplyTo":"CAJo=hJtsiEEHA33CQn1MCvb7vFv7uEF+U292YgBa7EWv7P8Jng@mail.gmail.com","subject":"Re: New Feature wanted: Is it possible to let git clone continue last break point?","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2011-11-04T15:55:32Z","receivedAt":"2011-11-04T15:55:32Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Shawn Pearce writes:\n> On Fri, Nov 4, 2011 at 02:35, Johannes Sixt wrote:\n> > Am 11/4/2011 9:56, schrieb Clemens Buchacher:\n\n> > > Cache ... not the pack but the information\n> > >    to re-create it...\n> >\n> > It has been discussed. It doesn't work. Because with threaded pack\n> > generation, the resulting pack is not deterministic.\n> \n> The information to create a pack for a repository with 2M objects\n> (e.g. Linux kernel tree) is *at least* 152M of data. This is just a\n> first order approximation of what it takes to write out the 2M SHA-1s,\n> along with say a 4 byte length so you can find given an offset\n> provided by the client roughly where to resumse in the object stream.\n> This is like 25% of the pack size itself. Ouch.\n\nWell, perhaps caching a few most popular packs in some kind of cache\n(packfile is saved to disk as it is streamed if we detect that it will\nbe large), indexing by WANT / HAVE?\n \n> This data is still insufficient to resume from. A correct solution\n> would allow you to resume in the middle of an object, which means we\n> also need to store some sort of indicator of which representation was\n> chosen from an existing pack file for object reuse. Which adds more\n> data to the stream. And then there is the not so simple problem of how\n> to resume in the middle of an object that was being recompressed on\n> the fly, such as a large loose object.\n\nWell, so you wouldn't be able to just concatenate packs^W received\ndata. Still it should be possible to \"repair\" halfway downloaded\npartial pack...\n \nJust my 2 eurocents^W groszy.\n-- \nJakub Narębski\n"},{"id":"178836","messageId":"CACsJy8C1JxoCsCoGW8kOVEtOGUjAUGL0G68qLEZ484yRJGgAkA@mail.gmail.com","threadId":"28803","inReplyTo":"CAJo=hJtsiEEHA33CQn1MCvb7vFv7uEF+U292YgBa7EWv7P8Jng@mail.gmail.com","subject":"Re: New Feature wanted: Is it possible to let git clone continue last break point?","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2011-11-04T16:05:07Z","receivedAt":"2011-11-04T16:05:07Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"2011/11/4 Shawn Pearce :\n> By the time you get done with all of that, your \"ticket\" might as well\n> be the name of a pack file. And your \"resume information\" is just a\n> pack file itself. Which would be very expensive to recreate.\n\nI'll deal with initial clone case only here. Can we make git protocol\nsend multiple packs, then send on-disk packs one by one together with\npack SHA1? This way we do not need to recreate anything. If new packs\nare created during cloning, git client should be able to construct\n\"have\" list from good packs and fetch updates from server again.\n-- \nDuy\n"},{"id":"178894","messageId":"20111105100015.GB17752@ecki","threadId":"28803","inReplyTo":"CAJo=hJtsiEEHA33CQn1MCvb7vFv7uEF+U292YgBa7EWv7P8Jng@mail.gmail.com","subject":"Re: New Feature wanted: Is it possible to let git clone continue last break point?","fromName":"Clemens Buchacher","fromEmail":"drizzd@aon.at","sentAt":"2011-11-05T10:00:15Z","receivedAt":"2011-11-05T10:00:15Z","isPatch":false,"sender":{"key":"drizzd@gmx.net","avatar":"https://avatars.githubusercontent.com/u/59082?v=4"},"body":"On Fri, Nov 04, 2011 at 07:22:20AM -0700, Shawn Pearce wrote:\n> On Fri, Nov 4, 2011 at 02:35, Johannes Sixt wrote:\n> > Am 11/4/2011 9:56, schrieb Clemens Buchacher:\n> >> Cache ... not the pack but the information\n> >>    to re-create it...\n> >\n> > It has been discussed. It doesn't work. Because with threaded pack\n> > generation, the resulting pack is not deterministic.\n\nSo let the client disable it, if they'd rather have a resumeable\nfetch than a fast one.\n\nSorry if I'm being obstinate here. But I don't understand the\nproblem and I can't find an explanation in related discussions.\n\n> The information to create a pack for a repository with 2M objects\n> (e.g. Linux kernel tree) is *at least* 152M of data. This is just a\n> first order approximation of what it takes to write out the 2M SHA-1s,\n> along with say a 4 byte length so you can find given an offset\n> provided by the client roughly where to resumse in the object stream.\n> This is like 25% of the pack size itself. Ouch.\n\nSorry, I should not have said HAVEs. All we need is the common\ncommits, and the sha1s of the WANTed branch heads at the time of\nthe initial fetch. That shouldn't be more than 10 or so in typical\ncases.\n\n> This data is still insufficient to resume from. A correct solution\n> would allow you to resume in the middle of an object, which means we\n> also need to store some sort of indicator of which representation was\n> chosen from an existing pack file for object reuse. Which adds more\n> data to the stream. And then there is the not so simple problem of how\n> to resume in the middle of an object that was being recompressed on\n> the fly, such as a large loose object.\n\nHow often does the \"representation chosen from an existing pack\nfile for object reuse\" change? Long term determinism is a problem,\nyes. But I see no reason why it should not work for this short-term\ncase. So long as the pack is created by one particular git and libz\nversion, and for this particular consecutive run of fetches, we do\nnot need to store anything about the pack. The client downloads n\nMB of data until the drop. To resume, the client says it already\nhas n MB of data.\n\nNo?\n\nClemens\n"}]}