{"thread":{"id":"26710","subject":"Re: GSoC resumable clone","startedAt":"2011-03-11T15:17:31Z","lastAt":"2011-03-11T21:43:46Z","messageCount":7,"participants":["Shawn Pearce","Jeff King","Nguyen Thai Ngoc Duy","Ilari Liusvaara"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"163227","messageId":"AANLkTinrgqLhZK=fQ_+gUanT-zy9Mcbw-y3o7nYV9A-m@mail.gmail.com","threadId":"26710","inReplyTo":null,"subject":"Re: GSoC resumable clone","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2011-03-11T15:17:31Z","receivedAt":"2011-03-11T15:17:31Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Fri, Mar 11, 2011 at 06:10, Alexander Miseler wrote:\n> On 11.03.2011 14:48, Nguyen Thai Ngoc Duy wrote:\n>>> On Fri, Mar 11, 2011 at 01:18:45PM +0100, Alexander Miseler wrote:\n>>>>\n>>>> Resumable clone\n>>\n>> A simpler way to restartable clone is to facilitate bundles (Nicolas'\n>> idea). Some glue is needed to teach git-fetch/git-daemon to use the\n>> bundles, and git-push to automatically create bundles periodically (or\n>> a new command that can be run from cron). I think this way fit in GSoC\n>> scope better.\n\nI think the cached bundle idea is horrifically stupid in the face of\nthe subsequent cached pack idea. JGit already implements cached packs,\nand it works very well. The feature just needs to be back-ported to\nbuiltin/pack-objects.c, along with some minor edits to my RFC patch to\ngit-repack.sh to be able to construct the cached pack.\n\nUnlike a cached bundle, the cached pack doesn't eat up useless disk\nspace on the server. Its still the only copy of the object content,\nwhich keeps server disk usage (and buffer cache usage) lower.\n\nA protocol extension in the fetch-pack/upload-pack protocol is\nrequired to allow pack-objects to delimit the early thin-pack from the\nlater cached pack, as well as supply the cached-pack's identity. A\nclient who breaks the connection after the leading thin-pack has been\nreceived could restart by downloading the cached pack from a specific\nstarting byte.\n\nWithout waiting for pack v4, cached packs can shave a full minute of\nserver CPU time during a clone of the linux-2.6 kernel. That's nothing\nto laugh at, its a full CPU minute. These days a full CPU minute is a\nlot of computational work. It also is pretty backwards compatible with\nthe current network protocol, even ancient Git clients can still use\nthe cached pack during an initial clone, saving a lot of server\nresources.\n\nWith cached packs, organizations like Gentoo wouldn't need to\nimplement bizarre hacks in their upload-pack binary to prevent clones\nover git:// from their servers.\n\nIts also well within GSoC size scope. I think the hard part is\nunderstanding enough of how the revision walker works inside of\npack-objects in order to construct the leading thin-pack.\n\n>> [1] The idea of my work above was mentioned elsewhere, history is cut\n>> down by path. Each file/dir's history a very long chain of deltas. We\n>> can stream deltas (in parallel if needed) over the wire, resuming\n>> where the chain stops last time.\n>\n> This may all be aiming to short. IMHO the best solution would be some\n> generic way for the client to specify exactly what it wants to get and to\n> get just that. This would lay the groundwork for:\n> - lazy clones\n> - sparse clones\n> - resumable cloning\n> - resumable fetching\n\nJunio and I would like see narrow checkout code re-implemented to\nsupport obtaining only a subset of the paths from the remote.\n\nOnce that is implemented, a client on a really bad network connection\ncould do a resumable clone by grabbing a shallow clone of depth 1\nalong no paths, partition the root tree up, then extend its paths\ngrabbing subdirectories until the root commit is fully expanded. Then\nit can walk back increasing its depth until it runs into the cached\npack... where it can then do byte range requests.\n\nThis won't be pretty. And given that the leading thin-pack for a\ncached pack can be less than 2% of the entire data transfer, may not\nbe necessary for a resumable clone. IMHO if you cannot get 2% of the\ndata transfer before your connection breaks, maybe you should ask for\nthe data on DVD via post, because your network sucks.\n\n-- \nShawn.\n"},{"id":"163229","messageId":"20110311153752.GA30329@sigill.intra.peff.net","threadId":"26710","inReplyTo":"AANLkTinrgqLhZK=fQ_+gUanT-zy9Mcbw-y3o7nYV9A-m@mail.gmail.com","subject":"Re: GSoC resumable clone","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-03-11T15:37:52Z","receivedAt":"2011-03-11T15:37:52Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 11, 2011 at 07:17:31AM -0800, Shawn O. Pearce wrote:\n\n> >> A simpler way to restartable clone is to facilitate bundles (Nicolas'\n> >> idea). Some glue is needed to teach git-fetch/git-daemon to use the\n> >> bundles, and git-push to automatically create bundles periodically (or\n> >> a new command that can be run from cron). I think this way fit in GSoC\n> >> scope better.\n> \n> I think the cached bundle idea is horrifically stupid in the face of\n> the subsequent cached pack idea. JGit already implements cached packs,\n> and it works very well. The feature just needs to be back-ported to\n> builtin/pack-objects.c, along with some minor edits to my RFC patch to\n> git-repack.sh to be able to construct the cached pack.\n\nI think there is room for both ideas. The cached bundle idea is not just\n\"here, download this bundle first\". It is \"here, download this _other\nthing_ first, which might be a bundle, another git repo, a torrent,\netc\".\n\nSo yeah, cached packs are a way better solution if you are just going to\nhave an extra bundle on the same machine. But that's just one use case.\nThe ability for my server to say \"go hit kernel.org first, and then come\nback to me to pick up the deltas\" is also valuable. Similarly, the\nability to serve an initial bundle off a torrent is useful for extremely\nlarge projects.\n\n-Peff\n"},{"id":"163230","messageId":"AANLkTi=D5kYh-w-6CKuCf39fnH1SYT5HxJ31tdq2vb9d@mail.gmail.com","threadId":"26710","inReplyTo":"20110311153752.GA30329@sigill.intra.peff.net","subject":"Re: GSoC resumable clone","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2011-03-11T15:41:14Z","receivedAt":"2011-03-11T15:41:14Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Fri, Mar 11, 2011 at 07:37, Jeff King wrote:\n> On Fri, Mar 11, 2011 at 07:17:31AM -0800, Shawn O. Pearce wrote:\n>\n>> >> A simpler way to restartable clone is to facilitate bundles (Nicolas'\n>> >> idea). Some glue is needed to teach git-fetch/git-daemon to use the\n>> >> bundles, and git-push to automatically create bundles periodically (or\n>> >> a new command that can be run from cron). I think this way fit in GSoC\n>> >> scope better.\n>>\n>> I think the cached bundle idea is horrifically stupid in the face of\n>> the subsequent cached pack idea. JGit already implements cached packs,\n>> and it works very well. The feature just needs to be back-ported to\n>> builtin/pack-objects.c, along with some minor edits to my RFC patch to\n>> git-repack.sh to be able to construct the cached pack.\n>\n> I think there is room for both ideas. The cached bundle idea is not just\n> \"here, download this bundle first\". It is \"here, download this _other\n> thing_ first, which might be a bundle, another git repo, a torrent,\n> etc\".\n\nFair enough. Though I wouldn't limit this to bundles. Instead I would\nsuggest supporting any valid Git URLs, and then extend our URL syntax\nto support bundles over http://, rsync://, and torrent.\n\n> So yeah, cached packs are a way better solution if you are just going to\n> have an extra bundle on the same machine. But that's just one use case.\n> The ability for my server to say \"go hit kernel.org first, and then come\n> back to me to pick up the deltas\" is also valuable. Similarly, the\n> ability to serve an initial bundle off a torrent is useful for extremely\n> large projects.\n\nIf we support any URL and don't assume the URL is a bundle, you can\npoint traffic at kernel.org to for example grab Linus' primary\nrepository first, even if he doesn't have a bundle.\n\n-- \nShawn.\n"},{"id":"163231","messageId":"AANLkTinDBm-Q3MA62r_TRqt-qxmPuDKPACr4bEPnJ6vz@mail.gmail.com","threadId":"26710","inReplyTo":"AANLkTinrgqLhZK=fQ_+gUanT-zy9Mcbw-y3o7nYV9A-m@mail.gmail.com","subject":"Re: GSoC resumable clone","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2011-03-11T15:42:35Z","receivedAt":"2011-03-11T15:42:35Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Mar 11, 2011 at 10:17 PM, Shawn Pearce wrote:\n> I think the cached bundle idea is horrifically stupid in the face of\n> the subsequent cached pack idea. JGit already implements cached packs,\n> and it works very well. The feature just needs to be back-ported to\n> builtin/pack-objects.c, along with some minor edits to my RFC patch to\n> git-repack.sh to be able to construct the cached pack.\n> ...\n\nI wonder why I missed it. Probably to recent and has not be carved to\nmy mind yet.\n\n> Junio and I would like see narrow checkout code re-implemented to\n> support obtaining only a subset of the paths from the remote.\n\nI'm close to finishing negative pathspecs (for extending narrow\nclones). I'll get there.\n\n> Once that is implemented, a client on a really bad network connection\n> could do a resumable clone by grabbing a shallow clone of depth 1\n> along no paths, partition the root tree up, then extend its paths\n> grabbing subdirectories until the root commit is fully expanded. Then\n> it can walk back increasing its depth until it runs into the cached\n> pack... where it can then do byte range requests.\n\nYes. But then it'll cost server's processing power more. Partitioning\nby path reduces chances of reusing deltas a lot.\n-- \nDuy\n"},{"id":"163233","messageId":"20110311154822.GA30605@sigill.intra.peff.net","threadId":"26710","inReplyTo":"AANLkTi=D5kYh-w-6CKuCf39fnH1SYT5HxJ31tdq2vb9d@mail.gmail.com","subject":"Re: GSoC resumable clone","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-03-11T15:48:22Z","receivedAt":"2011-03-11T15:48:22Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 11, 2011 at 07:41:14AM -0800, Shawn O. Pearce wrote:\n\n> > I think there is room for both ideas. The cached bundle idea is not just\n> > \"here, download this bundle first\". It is \"here, download this _other\n> > thing_ first, which might be a bundle, another git repo, a torrent,\n> > etc\".\n> \n> Fair enough. Though I wouldn't limit this to bundles. Instead I would\n> suggest supporting any valid Git URLs, and then extend our URL syntax\n> to support bundles over http://, rsync://, and torrent.\n\nSorry, I didn't mean to imply that it was limited to bundles. It would\nsupport arbitrary URLs or schemes. See this thread for some past\ndiscussion:\n\n http://article.gmane.org/gmane.comp.version-control.git/164700\n\n> If we support any URL and don't assume the URL is a bundle, you can\n> point traffic at kernel.org to for example grab Linus' primary\n> repository first, even if he doesn't have a bundle.\n\nExactly.\n\n-Peff\n"},{"id":"163244","messageId":"20110311205041.GA10210@LK-Perkele-VI.localdomain","threadId":"26710","inReplyTo":"20110311154822.GA30605@sigill.intra.peff.net","subject":"Re: GSoC resumable clone","fromName":"Ilari Liusvaara","fromEmail":"ilari.liusvaara@elisanet.fi","sentAt":"2011-03-11T20:50:41Z","receivedAt":"2011-03-11T20:50:41Z","isPatch":false,"sender":{"key":"ilari.liusvaara@elisanet.fi","avatar":null},"body":"On Fri, Mar 11, 2011 at 10:48:22AM -0500, Jeff King wrote:\n> On Fri, Mar 11, 2011 at 07:41:14AM -0800, Shawn O. Pearce wrote:\n> \n> > Fair enough. Though I wouldn't limit this to bundles. Instead I would\n> > suggest supporting any valid Git URLs, and then extend our URL syntax\n> > to support bundles over http://, rsync://, and torrent.\n> \n> Sorry, I didn't mean to imply that it was limited to bundles. It would\n> support arbitrary URLs or schemes. See this thread for some past\n> discussion:\n\nSecurity pitfall: You need a way to restrict URL schemes that can\nbe specified from the remote. Some URL schemes are wildly unsafe\nto use that way (or just don't make sense).\n\nThe URL schemes where it is safe and makes sense are (at least):\n- git://\n- ssh:// (and the scp syntax)\n- http://\n- ftp://\n- https://\n- ftps://\n- rsync://\n- file:// (?)\n\nNew capabilities perhaps? This would allow allowing it on\nper-remote-helper basis if that remote helper is deemed safe to\nbe able to receive arbitrary URLs from untrusted sources.\n\n-Ilari\n"},{"id":"163248","messageId":"20110311214346.GA19235@sigill.intra.peff.net","threadId":"26710","inReplyTo":"20110311205041.GA10210@LK-Perkele-VI.localdomain","subject":"Re: GSoC resumable clone","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-03-11T21:43:46Z","receivedAt":"2011-03-11T21:43:46Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 11, 2011 at 10:50:41PM +0200, Ilari Liusvaara wrote:\n\n> > Sorry, I didn't mean to imply that it was limited to bundles. It would\n> > support arbitrary URLs or schemes. See this thread for some past\n> > discussion:\n> \n> Security pitfall: You need a way to restrict URL schemes that can\n> be specified from the remote. Some URL schemes are wildly unsafe\n> to use that way (or just don't make sense).\n\nDid you mean on the server end? Or the client?\n\nIf the server, then I think no, it's a client decision. If on the client\nend, then yes, but it's one of many criteria.\n\nThe server end provides specially-formed refs that mention alternate\nlocations. The client decides which of those locations, if any, meet\nits criteria for mirroring, including but not limited to:\n\n 1. Whether the client supports the protocol in question (not everybody\n will be able to torrent, for example).\n\n 2. Whether the client's network allows it (e.g., restrictive proxies).\n\n 3. Whether it meets the client's security requirements (e.g., we\n probably shouldn't accept file:// URLs at all).\n\nBut it's clear to me that the security decision is only one of many\ncriteria, and that the client is in a much better place to make those\ndecisions. And that some of those decisions are going to have to be\nconfigurable by the user.\n\nSo yes, I agree we shouldn't blindly follow URLs.\n\n-Peff\n"}]}