{"thread":{"id":"41386","subject":"RFC: Resumable clone based on hybrid \"smart\" and \"dumb\" HTTP","startedAt":"2016-02-10T18:59:42Z","lastAt":"2016-02-16T18:34:21Z","messageCount":24,"participants":["Shawn Pearce","Stefan Beller","Junio C Hamano","Jonathan Nieder","Jeff King","Duy Nguyen","Blake Burkhart"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"277883","messageId":"CAJo=hJtHgE_vye_1sPTDsvJ0X=Cs72HKLgRH8btpW-pMrDdk9g@mail.gmail.com","threadId":"41386","inReplyTo":null,"subject":"RFC: Resumable clone based on hybrid \"smart\" and \"dumb\" HTTP","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2016-02-10T18:59:42Z","receivedAt":"2016-02-10T18:59:42Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Sorry, no code, only words today. Previously people have proposed a\nfew different options for resumable clone, like \"clone.bundle\"\n(currently used by Android as a hack) or \"skip start of regenerated\npack\". Here is another option.\n\nWe could implement resumable clone by making a bit of a hybrid of the\nsmart and dumb HTTP protocol.\n\n\nOverview\n--------\nDuring `git gc` break out the refs/heads/* namespace into its own\npackfile, and pack the remaining references into an \"other\" pack file.\nThis is already done by JGit's DfsGarbageCollector and normal\nfilesystem based GC, we have many years of experience with this\napproach working well.\n\nThe GC process saves the roots used for the \"head\" pack to a new\npack-$HASH.info file, which is written as a sibling to the existing\n.pack, .idx, and .bitmap files produced by GC. The .info file contains\nmetadata about the pack in a text format such as:\n\n  pack-name objects/pack/pack-$HASH.pack\n  pack-trailer $FOOTER_HASH\n  pack-size $LENGTH_IN_BYTES\n\n  ff4ea6004fb48146330d663d64a71e7774f059f9 refs/heads/master\n  ced85554d3d871f66dc4442631320a96a5d702f5 refs/heads/next\n\nIdeally $HASH = $FOOTER_HASH here so that any repack with the same\nobject set but different compression settings yields a different\npack-name in the info file.\n\n\nDuring `git clone` HTTP clients first fetch:\n\n  https://host/repo/info/clone\n\nThis returns the most recent pack-*.info file available in the\nrepository, assuming the client has read permission on all branches\ncontained in the info file. If there is no info file present or the\nclient has read restrictions that requires eliding content, the URL\n404s. On 404 the client tries a normal smart HTTP ref advertisement\n(/info/refs?service=git-upload-pack).\n\nWith the info file in hand the client writes this to a temporary file\nunder $GIT_DIR and then uses dumb HTTP byte range requests to download\nthe pack identified by pack-name:\n\n  https://host/repo/$pack-name\naka\n  https://host/repo/objects/pack/pack-$HASH.pack\n\nSmart HTTP servers that have info/clone support will need to return\nthis pack as-is to clients, if the client still has read permission\nover all branches in the associated pack-*.info file.\n\nClients store the file data locally and check the last 20 bytes\nmatches $FOOTER_HASH from the info file before running git-index-pack\nto verify the contents and build the local .idx file. If the footer\nhash does not match or any resume attempt for the pack-name URL fails,\nthe client deletes everything and starts over from the beginning.\n\n\nPost Clone Fetch\n----------------\nAfter the pack file is downloaded and indexed the client creates local\ntracking references matching the info file it obtained at the start.\nOnce the references are created the client issues a standard `git\nfetch` with the origin server to become current.\n\nSince the client is working off the server’s last GC pack, the client\nwill be very close to the server, and the final fetch will be very\nsmall, however this depends on the frequency of the server’s GCs. With\na volume based GC trigger such as `git gc` in a post-receive hook,\nclients will be very close to the server after this clone.\n\n\nRedirects\n---------\nClients should follow HTTP redirects, allowing servers to position\nboth the info/clone file and the pack-name on CDNs. Clients should not\nremember the redirected locations and instead always use the base\nrepository URL during resume. This allows servers to embed short-lived\nauthentication tokens into the redirect URL for processing at the edge\nCDN.\n\n\nParallel Download\n-----------------\nServers may optionally include a header indicating they are willing to\nperform parallel download of a file iff the client is well connected:\n\n  parallel-download 5 8388608\n\nin the header of the info/clone file indicates the server will allow\nclients to send up to 5 parallel byte-range requests on 8 MiB\nalignments.\n\nClients should start 1 HTTP download, track incoming bandwidth, then\nadd a parallel download for a later segment of the file. If bandwidth\nincreases with the additional stream, clients may try adding a 3rd\nstream, etc. until the maximum of 5 streams is reached.\n\n\nRacing with GC\n--------------\nServers may choose to keep a pack file around several hours after GC\nto permit clients to continue to resume and download a repository they\nstarted cloning before the GC began.\n\nBy sending the most recent pack-*.info file to new clients, servers\ncan choose to keep older packs on disk for a time window (e.g. 4\nhours) that older sessions can continue to retrieve before the file is\nunlinked from the filesystem and its disk is reclaimed.\n\nThis is an optional service that is subject to disk constraints on the\nserver. Unlike clone.bundle, the server does not need 2x disk capacity\nfor all repositories; the additional disk required depends on GC rate\nand size of repositories.\n\n\nEnd user experience\n-------------------\nFrom the end-user side experience, I think we want the following to happen:\n\n1. When initiating a `git clone`, the user does not have to do anything special.\n\n2. A `git clone` eventually calls into the transport layer, and\ngit-remote-curl will probe for the info/clone URL; if the resource\nfails to load, everything goes through the traditional codepath.\n\n3. When git-remote-curl detects the support of dumb clone, it does the\n\"retry until successfully download the pack data fully\" dance\ninternally, tentatively updates the remote tracking refs, and then\npretends as if it was asked to do an incremental fetch. If this\nsucceeds without any die(), everybody is happy.\n\n4. If the above step 3. has to die() for some reason (including\nimpatience hitting ^C), leave the $GIT_DIR, downloaded .info file and\npartially downloaded .pack file. Tell the user that the cloning can be\nresumed and how.\n\n5. `git clone --resume` would take the leftover $GIT_DIR and friends\nfrom step 4. `git clone --resume`, when it is given a $GIT_DIR that\nlacks .info, will refuse to start. When `--resume` is given, there is\nno other option acceptable (perhaps other than naming which $GIT_DIR\nthe partial result lives).\n\n\n... Thoughts?\n"},{"id":"277885","messageId":"CAJo=hJuRxoe6tXe65ci-A35c_PWJEP7KEPFu5Ocn147HwVuo3A@mail.gmail.com","threadId":"41386","inReplyTo":"CAJo=hJtHgE_vye_1sPTDsvJ0X=Cs72HKLgRH8btpW-pMrDdk9g@mail.gmail.com","subject":"Re: RFC: Resumable clone based on hybrid \"smart\" and \"dumb\" HTTP","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2016-02-10T20:11:46Z","receivedAt":"2016-02-10T20:11:46Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Wed, Feb 10, 2016 at 10:59 AM, Shawn Pearce <spearce@spearce.org> wrote:\n>\n> ... Thoughts?\n\nSeveral of us at $DAY_JOB talked about this more today and thought a\nvariation makes more sense:\n\n1. Clients attempting clone ask for /info/refs?service=git-upload-pack\nlike they do today.\n\n2. Servers that support resumable clone include a \"resumable\"\ncapability in the advertisement.\n\n3. Updated clients on clone request GET /info/refs?service=git-resumable-clone.\n\n4. The server may return a 302 Redirect to its current \"mostly whole\"\npack file. This can be more flexible than \"refs/heads/*\", it just\nneeds to be a mostly complete pack file that contains a complete graph\nfrom any arbitrary roots.\n\n5. Clients fetch the file using standard HTTP GET, possibly with\nbyte-ranges to resume.\n\n6. Once stored and indexed with .idx, clients run `git fsck\n--lost-found` to discover the roots of the pack it downloaded. These\nare saved as temporary references.\n\n7. Client runs incremental fetch, and then deletes the temporary\nreferences from 6.\n\n\nAn advantage to this process is its much more flexible for the server.\nThere is no additional pack-*.info file required. GC can organize\npacks anyway it wants, etc.\n\nTo make step 4 really resume well, clients may need to save the first\nLocation header it gets back from\n/info/refs?service=git-resumable-clone and use that on resume. Servers\nare likely to embed the pack SHA-1 in the Location header, and the\nclient wants to use this on subsequent GET attempts to abort early if\nthe server has deleted the pack the client is trying to obtain.\n"},{"id":"277886","messageId":"CAGZ79kZMvxa5Np4GbShv_A6NZwVAqff94+d8MFTZwrZS+2CqeQ@mail.gmail.com","threadId":"41386","inReplyTo":"CAJo=hJuRxoe6tXe65ci-A35c_PWJEP7KEPFu5Ocn147HwVuo3A@mail.gmail.com","subject":"Re: RFC: Resumable clone based on hybrid \"smart\" and \"dumb\" HTTP","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-02-10T20:23:06Z","receivedAt":"2016-02-10T20:23:06Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Feb 10, 2016 at 12:11 PM, Shawn Pearce <spearce@spearce.org> wrote:\n> On Wed, Feb 10, 2016 at 10:59 AM, Shawn Pearce <spearce@spearce.org> wrote:\n>>\n>> ... Thoughts?\n>\n> Several of us at $DAY_JOB talked about this more today and thought a\n> variation makes more sense:\n>\n> 1. Clients attempting clone ask for /info/refs?service=git-upload-pack\n> like they do today.\n>\n> 2. Servers that support resumable clone include a \"resumable\"\n> capability in the advertisement.\n\nlike \"resumable-token=hash\" similar to a push cert advertisement?\n\n>\n> 3. Updated clients on clone request GET /info/refs?service=git-resumable-clone.\n\nOr just in the non-http case, they would terminate after the ls-remote\n(including capability advertisement) was done and connect again to\na different service such as git-upload-stale-pack with the resumable\ntoken to identify the pack.\n\n>\n> 4. The server may return a 302 Redirect to its current \"mostly whole\"\n> pack file. This can be more flexible than \"refs/heads/*\", it just\n> needs to be a mostly complete pack file that contains a complete graph\n> from any arbitrary roots.\n>\n> 5. Clients fetch the file using standard HTTP GET, possibly with\n> byte-ranges to resume.\n\nIn the non-http case the git-upload-stale-pack would be rsync with the\nresume token to determine the file name of the pack,\nsuch that we have resumeability.\n\n>\n> 6. Once stored and indexed with .idx, clients run `git fsck\n> --lost-found` to discover the roots of the pack it downloaded. These\n> are saved as temporary references.\n\njrn:\n> I suspect we can do even faster by making index-pack do the work\n\n    index-pack --check-self-contained-and-connected\n\n>\n> 7. Client runs incremental fetch, and then deletes the temporary\n> references from 6.\n>\n>\n> An advantage to this process is its much more flexible for the server.\n> There is no additional pack-*.info file required. GC can organize\n> packs anyway it wants, etc.\n>\n> To make step 4 really resume well, clients may need to save the first\n> Location header it gets back from\n> /info/refs?service=git-resumable-clone and use that on resume. Servers\n> are likely to embed the pack SHA-1 in the Location header, and the\n> client wants to use this on subsequent GET attempts to abort early if\n> the server has deleted the pack the client is trying to obtain.\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n"},{"id":"277887","messageId":"xmqqh9hgz3km.fsf@gitster.mtv.corp.google.com","threadId":"41386","inReplyTo":"CAGZ79kZMvxa5Np4GbShv_A6NZwVAqff94+d8MFTZwrZS+2CqeQ@mail.gmail.com","subject":"Re: RFC: Resumable clone based on hybrid \"smart\" and \"dumb\" HTTP","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-02-10T20:57:13Z","receivedAt":"2016-02-10T20:57:13Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller <sbeller@google.com> writes:\n\n>> 6. Once stored and indexed with .idx, clients run `git fsck\n>> --lost-found` to discover the roots of the pack it downloaded. These\n>> are saved as temporary references.\n>\n> jrn:\n>> I suspect we can do even faster by making index-pack do the work\n\nI somehow doubt it.  Both index-pack and lost-found need to trace\n\"object A depends on object B\", but the similarity ends there.\nindex-pack traces the delta dependency, which does not have any\nrelation with the reachability, which is what lost-found needs to\ncompute.\n\n>> 7. Client runs incremental fetch, and then deletes the temporary\n>> references from 6.\n>>\n>>\n>> An advantage to this process is its much more flexible for the server.\n>> There is no additional pack-*.info file required. GC can organize\n>> packs anyway it wants, etc.\n\nI am not quite sure if that is an advantage, though.  The second\nmessage proposes that the lost-found computation to be done by the\nclient using *.pack, but any client, given the same *.pack, will\ncompute the same result, so if the result is computed on the server\nside just once when the *.pack is prepared and downloaded to the\nclient, it would give us a better overall resource utilization.  And\nin essence, that was what the *.info file in the first message was.\n\nThe original proposal explicitly specified that the initial bulk\ntransfer pack must cover all heads and no tags.  For the purpose of\noffloading the clone traffic and making it resumable, there is no\nneed for such a restriction, and I do like the fact that the second\none loosened the requirement.\n"},{"id":"277888","messageId":"20160210210115.GA10155@google.com","threadId":"41386","inReplyTo":"CAGZ79kZMvxa5Np4GbShv_A6NZwVAqff94+d8MFTZwrZS+2CqeQ@mail.gmail.com","subject":"Re: RFC: Resumable clone based on hybrid \"smart\" and \"dumb\" HTTP","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2016-02-10T21:01:15Z","receivedAt":"2016-02-10T21:01:15Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Stefan Beller wrote:\n> On Wed, Feb 10, 2016 at 12:11 PM, Shawn Pearce <spearce@spearce.org> wrote:\n\n>> Several of us at $DAY_JOB talked about this more today and thought a\n>> variation makes more sense:\n>>\n>> 1. Clients attempting clone ask for /info/refs?service=git-upload-pack\n>> like they do today.\n>>\n>> 2. Servers that support resumable clone include a \"resumable\"\n>> capability in the advertisement.\n>\n> like \"resumable-token=hash\" similar to a push cert advertisement?\n\nIt could just be the string 'resumable'.\n\nBut I wonder if it would be possible to save a round-trip by getting the\n302 response in the initial request.  If the client requests\n\n\t/info/refs?service=git-upload-pack&want_resumable=true\n\nthen allow the server to make a 302 in response to its current mostly\nwhole pack.  Current clients would never send such a request because the\ncurrent protocol requires that for smart clients\n\n\tThe request MUST contain exactly one query parameter,\n\t`service=$servicename`, where `$servicename` MUST be the service\n\tname the client wishes to contact to complete the operation.\n\tThe request MUST NOT contain additional query parameters.\n\nCurrent http-backend ignores extra query parameters.  I haven't\nchecked other smart http server implementations, though.\n\n>> 3. Updated clients on clone request GET /info/refs?service=git-resumable-clone.\n>\n> Or just in the non-http case, they would terminate after the ls-remote\n> (including capability advertisement) was done and connect again to\n> a different service such as git-upload-stale-pack with the resumable\n> token to identify the pack.\n\nHTTP supports range requests and existing CDNs speak HTTP, so I\nsuspect it would work better if the git-resumable-clone service\nprinted an HTTP URL from which to grab the packfile.\n\nI think the details are something that could be figured out after\ntrying out the idea with http first, though.\n\n[...]\n>> 5. Clients fetch the file using standard HTTP GET, possibly with\n>> byte-ranges to resume.\n>\n> In the non-http case the git-upload-stale-pack would be rsync with the\n> resume token to determine the file name of the pack,\n> such that we have resumeability.\n\nHow do I tunnel rsync over git protocol?\n\nSo I think in the non-http case the git-resumable-clone service would\nhave to print a URL to be served using a possibly different protocol\n(e.g., a signed https URL for getting the file from a service like S3,\nor an rsync URL for getting the file using the same ssh creds that\nwere used for the initial request).\n\n[...]\n>> 6. Once stored and indexed with .idx, clients run `git fsck\n>> --lost-found` to discover the roots of the pack it downloaded. These\n>> are saved as temporary references.\n>\n> jrn:\n> > I suspect we can do even faster by making index-pack do the work\n>\n>     index-pack --check-self-contained-and-connected\n\n--strict + --check-self-contained-and-connected check that the pack\nis self-contained.  In the process they mark each object that is\nreachable from another object in the pack with FLAG_LINK.\n\nThe objects not marked with FLAG_LINK are the roots.\n\n[...]\n>> To make step 4 really resume well, clients may need to save the first\n>> Location header it gets back from\n>> /info/refs?service=git-resumable-clone and use that on resume. Servers\n>> are likely to embed the pack SHA-1 in the Location header, and the\n>> client wants to use this on subsequent GET attempts to abort early if\n>> the server has deleted the pack the client is trying to obtain.\n\nYes.\n\nI really like this design.  I'm tempted to implement it (since it\nlacks a bunch of the downsides of clone.bundle).\n\nThanks,\nJonathan\n"},{"id":"277889","messageId":"xmqqd1s4z334.fsf@gitster.mtv.corp.google.com","threadId":"41386","inReplyTo":"20160210210115.GA10155@google.com","subject":"Re: RFC: Resumable clone based on hybrid \"smart\" and \"dumb\" HTTP","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-02-10T21:07:43Z","receivedAt":"2016-02-10T21:07:43Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder <jrnieder@gmail.com> writes:\n\n> --strict + --check-self-contained-and-connected check that the pack\n> is self-contained.  In the process they mark each object that is\n> reachable from another object in the pack with FLAG_LINK.\n>\n> The objects not marked with FLAG_LINK are the roots.\n\nAhh, OK, if we already have the code that does it, I have no\nobjection.  At some point before finalizing the pack on the disk we\nwould want to run fsck on all objects anyway, so having to pass\n\"--strict\" to find objects that lack FLAG_LINK does not sound too\nbig a downside.\n\nThanks.\n"},{"id":"277894","messageId":"20160210212207.GB10155@google.com","threadId":"41386","inReplyTo":"xmqqh9hgz3km.fsf@gitster.mtv.corp.google.com","subject":"Re: RFC: Resumable clone based on hybrid \"smart\" and \"dumb\" HTTP","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2016-02-10T21:22:07Z","receivedAt":"2016-02-10T21:22:07Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Junio C Hamano wrote:\n\n> I somehow doubt it.  Both index-pack and lost-found need to trace\n> \"object A depends on object B\", but the similarity ends there.\n\nYou already responded about this in another side-thread, so the\nfollowing is mostly redundant. :)  I think this message can still be\nuseful to future readers since it more explicitly goes through the\ntrade-offs.\n\nThe relevant similarity is that both index-pack and lost-found need to\ninflate all commit, tag, and tree objects.\n\nThis is the same idea that led to the introduction of \"git index-pack\n--strict\".\n\n[...]\n> I am not quite sure if that is an advantage, though.  The second\n> message proposes that the lost-found computation to be done by the\n> client using *.pack, but any client, given the same *.pack, will\n> compute the same result, so if the result is computed on the server\n> side just once when the *.pack is prepared and downloaded to the\n> client, it would give us a better overall resource utilization.  And\n> in essence, that was what the *.info file in the first message was.\n\nAdvantages of not providing the list of roots:\n 1. only need one round-trip to serve the packfile as-is\n 2. less data sent over the wire (not important unless the list of roots\n    is long)\n 3. can be enabled on the server for existing repositories without an\n    extra step of generating .info files\n\nAdvantage of providing the list of roots:\n- speedup because the client does not have to compute the list of roots\n\nFor a client that is already iterating over all objects and inspecting\nFLAG_LINK, the advantage (3) seems compelling enough to prefer the\nprotocol that doesn't sent a list of roots.\n\nExcept when people pass --depth, \"git clone\" sets\n'check_self_contained_and_connected = 1'.  That means clients that\nalready iterate over all objects and inspect FLAG_LINK are the usual\ncase.\n\nThanks for your thoughtfulness,\nJonathan\n"},{"id":"277896","messageId":"20160210214945.GA5853@sigill.intra.peff.net","threadId":"41386","inReplyTo":"CAJo=hJuRxoe6tXe65ci-A35c_PWJEP7KEPFu5Ocn147HwVuo3A@mail.gmail.com","subject":"Re: RFC: Resumable clone based on hybrid \"smart\" and \"dumb\" HTTP","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-02-10T21:49:46Z","receivedAt":"2016-02-10T21:49:46Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Feb 10, 2016 at 12:11:46PM -0800, Shawn Pearce wrote:\n\n> On Wed, Feb 10, 2016 at 10:59 AM, Shawn Pearce <spearce@spearce.org> wrote:\n> >\n> > ... Thoughts?\n> \n> Several of us at $DAY_JOB talked about this more today and thought a\n> variation makes more sense:\n> \n> 1. Clients attempting clone ask for /info/refs?service=git-upload-pack\n> like they do today.\n> \n> 2. Servers that support resumable clone include a \"resumable\"\n> capability in the advertisement.\n\nBecause the magic happens in the git protocol, that would mean this does\nnot have to be limited to git-over-http. It could be \"resumable=<url>\"\nto point the client anywhere (the same server over a different protocol,\nanother server, etc).\n\n> 3. Updated clients on clone request GET /info/refs?service=git-resumable-clone.\n> \n> 4. The server may return a 302 Redirect to its current \"mostly whole\"\n> pack file. This can be more flexible than \"refs/heads/*\", it just\n> needs to be a mostly complete pack file that contains a complete graph\n> from any arbitrary roots.\n\nAnd with \"resumable=<url>\", the client does not have to hit the server\nto do a redirect; it can go straight to the final URL, saving a\nround-trip.\n\n> 5. Clients fetch the file using standard HTTP GET, possibly with\n> byte-ranges to resume.\n> \n> 6. Once stored and indexed with .idx, clients run `git fsck\n> --lost-found` to discover the roots of the pack it downloaded. These\n> are saved as temporary references.\n\nClients do not have to _just_ fetch a packfile. They could get a bundle\nfile that contains the roots along with the packfile. I know that one of\nyour goals is not duplicating the storage of the packfile on the server,\nbut it would not be hard for the server to store the packfile and the\nbundle header separately, and concatenate them on the fly.\n\nRight now the clients can't clone from bundles directly via HTTP. I\nwrote patches for that ages ago, but got stuck on this very issue\n(basically that I had to spool the bundle and then clone from it, which\ntemporarily doubled the client's disk space requirements). One\nalternative would be to amend the bundle format so that rather than a\nsingle file, you get a bundle header whose end says \"...and my matching\npackfile is 1234-abcd\". And then the client knows that they can fetch\nthat separately from the same source.\n\nIt's an extra HTTP request, but it makes the code for client _and_\nserver way simpler. So the whole thing is basically then:\n\n  0. During gc, server generates pack-1234abcd.pack. It writes matching\n     tips into pack-1234abcd.info, which is essentially a bundle file\n     whose final line says \"pack-1234abcd.pack\".\n\n  1. Client contacts server via any git protocol. Server says\n     \"resumable=<url>\". Let's says that <url> is\n     https://example.com/repo/clones/1234abcd.bundle.\n\n  2. Client goes to <url>. They see that they are fetching a bundle,\n     and know not to do the usual smart-http or dumb-http protocols.\n     They can fetch the bundle header resumably (though it's tiny, so it\n     doesn't really matter).\n\n  3. After finishing the bundle header, they see they need to grab the\n     packfile. Based on the bundle header's URL and the filename\n     contained within it, they know to get\n     https://example.com/repo/clones/pack-1234abcd.pack\". This is\n     resumable, too.\n\n  4. Client clones from bundled pack as normal; no root-finding magic\n     required.\n\n  5. Client runs incremental fetch against original repo from step 1.\n\nAnd you'll notice, too, that all of the bundle-http magic kicks in\nduring step 2 because the client sees they're grabbing a bundle. Which\nmeans that the <url> in step 1 doesn't _have_ to be a bundle. It can be\n\"go fetch from kernel.org, then come back to me\".\n\n> An advantage to this process is its much more flexible for the server.\n> There is no additional pack-*.info file required. GC can organize\n> packs anyway it wants, etc.\n\nYes, it's much better than your original email, at least for GitHub\nservers. We're not very flexible with GC tricks, because we need bitmaps\nto work, and because we get a lot of benefit from sharing the object\nstorage for forks of a single repository.\n\n> To make step 4 really resume well, clients may need to save the first\n> Location header it gets back from\n> /info/refs?service=git-resumable-clone and use that on resume. Servers\n> are likely to embed the pack SHA-1 in the Location header, and the\n> client wants to use this on subsequent GET attempts to abort early if\n> the server has deleted the pack the client is trying to obtain.\n\nYou could possibly do away with this trick if the server hands out a\nunique URL in its \"resumable\" header. Though I imagine it might be\nconvenient for server admins to always point to a generic url, and\nput the logic in the HTTP layer.\n\nOTOH, if you do the \"split bundle\" thing I mentioned above, then this\nhappens for free. The client caches the bundle header it grabs in my\nstep 2, and then that contains the unique pack name to fetch in step 3.\n\n-Peff\n"},{"id":"277898","messageId":"20160210220334.GB5853@sigill.intra.peff.net","threadId":"41386","inReplyTo":"20160210212207.GB10155@google.com","subject":"Re: RFC: Resumable clone based on hybrid \"smart\" and \"dumb\" HTTP","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-02-10T22:03:35Z","receivedAt":"2016-02-10T22:03:35Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Feb 10, 2016 at 01:22:07PM -0800, Jonathan Nieder wrote:\n\n> > I am not quite sure if that is an advantage, though.  The second\n> > message proposes that the lost-found computation to be done by the\n> > client using *.pack, but any client, given the same *.pack, will\n> > compute the same result, so if the result is computed on the server\n> > side just once when the *.pack is prepared and downloaded to the\n> > client, it would give us a better overall resource utilization.  And\n> > in essence, that was what the *.info file in the first message was.\n> \n> Advantages of not providing the list of roots:\n>  1. only need one round-trip to serve the packfile as-is\n>  2. less data sent over the wire (not important unless the list of roots\n>     is long)\n>  3. can be enabled on the server for existing repositories without an\n>     extra step of generating .info files\n> \n> Advantage of providing the list of roots:\n> - speedup because the client does not have to compute the list of roots\n> \n> For a client that is already iterating over all objects and inspecting\n> FLAG_LINK, the advantage (3) seems compelling enough to prefer the\n> protocol that doesn't sent a list of roots.\n\nI'm not sure how compelling (3) is, since we are relying on the server\nto make certain packing choices. I guess a stock \"git repack -ad\" would\ndo in a pinch; it should at least contain all needed objects, but it's\ngoing to potentially have extra cruft objects (from reflogs, for\nexample).\n\nI outlined some alternatives to Shawn's proposal elsewhere in the\nthread. I think it's a useful feature for this redirect to not just be\n\"go fetch this packfile\", but \"go clone from here and come back to me\".\nThat opens up a lot of flexibility.\n\nIt does make \"go fetch this packfile without roots\" a little harder, but\nI think it's still do-able. Right now when git hits an http URL, we pass\nthe smart-http \"?service=\" magic, and we look at the response to figure\nout whether we got:\n\n  1. A smart-http server.\n\n  2. A dumb-http server.\n\n  N. Something else, in which case we die.\n\nThe alternative I outlined elsewhere (and the patches I posted long\nago) basically adds:\n\n  3. If it's a bundle, fetch the bundle and then clone from that.\n\nBut we could also do:\n\n  4. If it's a packfile, fetch the packfile and then do the\n     find-the-roots magic.\n\n> Except when people pass --depth, \"git clone\" sets\n> 'check_self_contained_and_connected = 1'.  That means clients that\n> already iterate over all objects and inspect FLAG_LINK are the usual\n> case.\n\nSomewhat related, but I've wondered if we could do something similar\neven for non-clone cases. That is, `index-pack` could tell us the set of\nreferenced but missing objects, and we could verify that each of those\nis reachable (we _could_ just have it verify that we have the object at\nall; traditionally we only guaranteed that reachable objects were kept,\nbut these days we keep anything reachable from another object we are\nkeeping, so if you have X, you should always have X^, etc).\n\nAnyway, that's quite a tangent from this topic.\n\n-Peff\n"},{"id":"277899","messageId":"20160210221758.GC10155@google.com","threadId":"41386","inReplyTo":"20160210214945.GA5853@sigill.intra.peff.net","subject":"Re: RFC: Resumable clone based on hybrid \"smart\" and \"dumb\" HTTP","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2016-02-10T22:17:58Z","receivedAt":"2016-02-10T22:17:58Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jeff King wrote:\n> On Wed, Feb 10, 2016 at 12:11:46PM -0800, Shawn Pearce wrote:\n\n>> Several of us at $DAY_JOB talked about this more today and thought a\n>> variation makes more sense:\n>>\n>> 1. Clients attempting clone ask for /info/refs?service=git-upload-pack\n>> like they do today.\n>>\n>> 2. Servers that support resumable clone include a \"resumable\"\n>> capability in the advertisement.\n>\n> Because the magic happens in the git protocol, that would mean this does\n> not have to be limited to git-over-http. It could be \"resumable=<url>\"\n> to point the client anywhere (the same server over a different protocol,\n> another server, etc).\n\nThanks for bringing this up.  A worry with putting the URL in the\ncapabilities line is that it makes it easy to run into the 1000-byte\nlimit.  It's been a while since v1.8.3-rc0~148^2~6 (pkt-line: provide\na LARGE_PACKET_MAX static buffer, 2013-02-20) but we still can't\nrely on clients having that applied.\n\n(I also haven't checked whether current versions of git are able to\nhandle longer capability strings with that patch applied.)\n\nAnother nice thing about using a 302 is that you can set cookies\nduring the redirect, which might make authenticated access easier.\n(That said, authenticated access through e.g. signed URLs can work\nfine without that.)\n\n[...]\n> Clients do not have to _just_ fetch a packfile. They could get a bundle\n> file that contains the roots along with the packfile. I know that one of\n> your goals is not duplicating the storage of the packfile on the server,\n> but it would not be hard for the server to store the packfile and the\n> bundle header separately, and concatenate them on the fly.\n\nDoesn't that prevent using a git-unaware file transfer service to\nserve the files?\n\nIt also means the client can't use the downloaded file as-is --- they\nneed to separate the root list from the packfile (that's not a big\ndeal; just some added complication to switch files at the appropriate\nmoment during download if you want to avoid temporarily using twice\nthe space).\n\nThat said, both these problems are avoided by serving the 'split\nbundle' you described as-is instead of concatenating.\n\n[...]\n> And you'll notice, too, that all of the bundle-http magic kicks in\n> during step 2 because the client sees they're grabbing a bundle. Which\n> means that the <url> in step 1 doesn't _have_ to be a bundle. It can be\n> \"go fetch from kernel.org, then come back to me\".\n\nI think that use case brings in complications that make it not\nnecessarily worth it.  In this example, if kernel.org is serving pack\nfiles, why shouldn't I point directly at the advertised pack CDN URL\ninstead of adding an extra hop that puts added load on kernel.org\nservers?\n\nAllowing an arbitrary \"fetch from here first\" capability is very\nflexible.  I guess my fear comes from not knowing what the flexibility\nbuys beyond aesthetics.  (My motivation comes from the example of\nalternates: it is pretty and very flexible and ended up as a support\nand maintenance headache instead of being widely useful.  I think what\nyou are proposing is more harmless but I'd still want to have an\nexample of what it's used for before going in that direction.)\n\nThanks,\nJonathan\n"},{"id":"277901","messageId":"xmqqwpqcxk87.fsf@gitster.mtv.corp.google.com","threadId":"41386","inReplyTo":"20160210214945.GA5853@sigill.intra.peff.net","subject":"Re: RFC: Resumable clone based on hybrid \"smart\" and \"dumb\" HTTP","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-02-10T22:40:24Z","receivedAt":"2016-02-10T22:40:24Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> Clients do not have to _just_ fetch a packfile. They could get a bundle\n> file that contains the roots along with the packfile. I know that one of\n> your goals is not duplicating the storage of the packfile on the server,\n> but it would not be hard for the server to store the packfile and the\n> bundle header separately, and concatenate them on the fly.\n\nYeah, I was wondering about that.  Just storing them as separate\ntwo-file pair on the server side and serving it as if it is a single\nfile to the client does not sound like a rocket science, and a\nreasonable webserver should be able to serve even a byte-range\nrequest out of such a thing.\n\nOf course, updating the clients to understand such a two-file pair\nas a valid bundle is a good thing to do, and I like that step 0. you\noutlined below.\n\n>   0. During gc, server generates pack-1234abcd.pack. It writes matching\n>      tips into pack-1234abcd.info, which is essentially a bundle file\n>      whose final line says \"pack-1234abcd.pack\".\n>\n>   1. Client contacts server via any git protocol. Server says\n>      \"resumable=<url>\". Let's says that <url> is\n>      https://example.com/repo/clones/1234abcd.bundle.\n>\n>   2. Client goes to <url>. They see that they are fetching a bundle,\n>      and know not to do the usual smart-http or dumb-http protocols.\n>      They can fetch the bundle header resumably (though it's tiny, so it\n>      doesn't really matter).\n> ...\n"},{"id":"277902","messageId":"20160210230307.GA6633@sigill.intra.peff.net","threadId":"41386","inReplyTo":"20160210221758.GC10155@google.com","subject":"Re: RFC: Resumable clone based on hybrid \"smart\" and \"dumb\" HTTP","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-02-10T23:03:07Z","receivedAt":"2016-02-10T23:03:07Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Feb 10, 2016 at 02:17:58PM -0800, Jonathan Nieder wrote:\n\n> > Because the magic happens in the git protocol, that would mean this does\n> > not have to be limited to git-over-http. It could be \"resumable=<url>\"\n> > to point the client anywhere (the same server over a different protocol,\n> > another server, etc).\n> \n> Thanks for bringing this up.  A worry with putting the URL in the\n> capabilities line is that it makes it easy to run into the 1000-byte\n> limit.  It's been a while since v1.8.3-rc0~148^2~6 (pkt-line: provide\n> a LARGE_PACKET_MAX static buffer, 2013-02-20) but we still can't\n> rely on clients having that applied.\n\nI hadn't considered that, but I'm not sure how much of a problem it is\nin practice. The first line is 40-hex sha1, space, HEAD, NUL, then\ncapabilities. The current capabilities string from github.com on a\nsample repo (which has a rather large agent string, and a normal-sized\nsymref pointer for HEAD) is 188 bytes. So that's still over 750 bytes\navailable for a URL. The space isn't infinite, and we may want to add\nmore capabilities later. But I'd think that devoting even 256 bytes to a\nurl would be reasonable, and leave us with fair room to grow.\n\n> Another nice thing about using a 302 is that you can set cookies\n> during the redirect, which might make authenticated access easier.\n> (That said, authenticated access through e.g. signed URLs can work\n> fine without that.)\n\nYeah, I can see there are advantages to assuming all the world is HTTP,\nbut it just doesn't seem that practical to me.\n\n> > Clients do not have to _just_ fetch a packfile. They could get a bundle\n> > file that contains the roots along with the packfile. I know that one of\n> > your goals is not duplicating the storage of the packfile on the server,\n> > but it would not be hard for the server to store the packfile and the\n> > bundle header separately, and concatenate them on the fly.\n> \n> Doesn't that prevent using a git-unaware file transfer service to\n> serve the files?\n\nSort of. They would just need to serve the combined file. But _if_ you\nhave a git-unaware service (rsync?) _and_ its hitting the same storage\n(so you could in theory not store the packfile twice), _and_ it cannot\nbe taught to do any kind of concatenation, then yes, it would be a\nproblem.\n\nI do think I favor the \"split bundle\" anyway, though, just for\nsimplicity on both ends.\n\n> > And you'll notice, too, that all of the bundle-http magic kicks in\n> > during step 2 because the client sees they're grabbing a bundle. Which\n> > means that the <url> in step 1 doesn't _have_ to be a bundle. It can be\n> > \"go fetch from kernel.org, then come back to me\".\n> \n> I think that use case brings in complications that make it not\n> necessarily worth it.  In this example, if kernel.org is serving pack\n> files, why shouldn't I point directly at the advertised pack CDN URL\n> instead of adding an extra hop that puts added load on kernel.org\n> servers?\n\nSure, that would be more efficient if kernel.org is providing such a CDN\nURL. But they aren't now, because it doesn't exist yet, and this feature\ncan be used by you without having to coordinate their use of the\nfeature. And you can replace kernel.org in my example with any other\nserver that happens to be preferable to fetching from you, for whatever\nreason.\n\n> My motivation comes from the example of\n> alternates: it is pretty and very flexible and ended up as a support\n> and maintenance headache instead of being widely useful.\n\nI guess one man's trash is another's treasure. Alternates are in wide\nuse at GitHub, and are pretty much what make our data model feasible.\nI'm pretty sure that git.or.cz uses them for similar purposes, too.\n\n> I think what\n> you are proposing is more harmless but I'd still want to have an\n> example of what it's used for before going in that direction.\n\nUnless there is a big immediate downside, I'd rather err in the opposite\ndirection: keep orthogonal concerns separate (e.g., redirecting to X\nversus protocol details of X).\n\n-Peff\n"},{"id":"277905","messageId":"xmqqfuwzykr5.fsf@gitster.mtv.corp.google.com","threadId":"41386","inReplyTo":"20160210210115.GA10155@google.com","subject":"Re: RFC: Resumable clone based on hybrid \"smart\" and \"dumb\" HTTP","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-02-11T03:43:42Z","receivedAt":"2016-02-11T03:43:42Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder <jrnieder@gmail.com> writes:\n\n> I really like this design.  I'm tempted to implement it (since it\n> lacks a bunch of the downsides of clone.bundle).\n\nJust to see people are not stepping on each others toe, implementing\nslightly different components in parallel based on the same theme to\nsolve the same problem, it may be beneficial to have a list of a bit\nmore detailed breakdown of the necessary parts.\n\nI think possible small first steps include:\n\n * A new \"--split-header\" option to \"git bundle\" that allows the\n   command to write out two files;\n\n * An update to the \"bundle\" transport so that it understand the new\n   split bundle format (i.e. when you have base.bndl that refers to\n   pack-deadbeef.pack, you can still say \"git clone base.bndl\" and\n   \"git ls-remote base.bndl\");\n\n * A new \"--bundle-header\" option to \"git index-pack\", which makes\n   it to write out the bundle header file that references the\n   resulting packfile in addition to the pack .idx file (this should\n   also be able to \"fix\" thin pack, and also by keeping track of the\n   actual \"foreign\" objects not just number of them, compute and\n   record the list of prerequisite objects in the resulting bundle\n   header file).\n\nAm I on the right track?  Assuming I am, anything I missed?\n"},{"id":"277948","messageId":"CAJo=hJt=hYf27bTcuyZEsn1w5CAXC=BoFfCcLGQ7ZmXe-g4SKg@mail.gmail.com","threadId":"41386","inReplyTo":"xmqqfuwzykr5.fsf@gitster.mtv.corp.google.com","subject":"Re: RFC: Resumable clone based on hybrid \"smart\" and \"dumb\" HTTP","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2016-02-11T18:04:32Z","receivedAt":"2016-02-11T18:04:32Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Wed, Feb 10, 2016 at 7:43 PM, Junio C Hamano <gitster@pobox.com> wrote:\n> Jonathan Nieder <jrnieder@gmail.com> writes:\n>\n>> I really like this design.  I'm tempted to implement it (since it\n>> lacks a bunch of the downsides of clone.bundle).\n>\n> Just to see people are not stepping on each others toe, implementing\n> slightly different components in parallel based on the same theme to\n> solve the same problem, it may be beneficial to have a list of a bit\n> more detailed breakdown of the necessary parts.\n>\n> I think possible small first steps include:\n>\n>  * A new \"--split-header\" option to \"git bundle\" that allows the\n>    command to write out two files;\n>\n>  * An update to the \"bundle\" transport so that it understand the new\n>    split bundle format (i.e. when you have base.bndl that refers to\n>    pack-deadbeef.pack, you can still say \"git clone base.bndl\" and\n>    \"git ls-remote base.bndl\");\n>\n>  * A new \"--bundle-header\" option to \"git index-pack\", which makes\n>    it to write out the bundle header file that references the\n>    resulting packfile in addition to the pack .idx file (this should\n>    also be able to \"fix\" thin pack, and also by keeping track of the\n>    actual \"foreign\" objects not just number of them, compute and\n>    record the list of prerequisite objects in the resulting bundle\n>    header file).\n>\n> Am I on the right track?  Assuming I am, anything I missed?\n\n--bundle-header option for pack-objects so that repack/gc can create\nthe matching .info file?\n\nI waffled on the URL in the capabilities line, but I think Peff\nconvinced me earlier in the thread that its reasonably safe to do. So\nthis split bundle with header URL in the capabilities sounds like a\ngood approach, its just a matter of implementation. :)\n"},{"id":"277964","messageId":"xmqq4mdfvspl.fsf@gitster.mtv.corp.google.com","threadId":"41386","inReplyTo":"20160210214945.GA5853@sigill.intra.peff.net","subject":"Re: RFC: Resumable clone based on hybrid \"smart\" and \"dumb\" HTTP","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-02-11T21:32:22Z","receivedAt":"2016-02-11T21:32:22Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> ... One\n> alternative would be to amend the bundle format so that rather than a\n> single file, you get a bundle header whose end says \"...and my matching\n> packfile is 1234-abcd\". And then the client knows that they can fetch\n> that separately from the same source.\n\nI would imagine that we would introduce bundle v3 format for this.\n\nIt may want to say \"my matching packfiles are these\" to accomodate a\nset of packs split at max-pack-size, but I am perfectly fine to say\nyou must create a single pack when you use a bundle with separate\nheader to keep things simpler.\n\n> It's an extra HTTP request, but it makes the code for client _and_\n> server way simpler. So the whole thing is basically then:\n>\n>   0. During gc, server generates pack-1234abcd.pack. It writes matching\n>      tips into pack-1234abcd.info, which is essentially a bundle file\n>      whose final line says \"pack-1234abcd.pack\".\n\nOK.\n\n>   1. Client contacts server via any git protocol. Server says\n>      \"resumable=<url>\". Let's says that <url> is\n>      https://example.com/repo/clones/1234abcd.bundle.\n>\n>   2. Client goes to <url>. They see that they are fetching a bundle,\n>      and know not to do the usual smart-http or dumb-http protocols.\n>      They can fetch the bundle header resumably (though it's tiny, so it\n>      doesn't really matter).\n\nMight be in megabytes range, though, with many refs.  It still is\ntiny, though ;-).\n\n>   3. After finishing the bundle header, they see they need to grab the\n>      packfile. Based on the bundle header's URL and the filename\n>      contained within it, they know to get\n>      https://example.com/repo/clones/pack-1234abcd.pack\". This is\n>      resumable, too.\n\nOK.\n\n>   4. Client clones from bundled pack as normal; no root-finding magic\n>      required.\n\nI like this part the most.\n\n>   5. Client runs incremental fetch against original repo from step 1.\n>\n> And you'll notice, too, that all of the bundle-http magic kicks in\n> during step 2 because the client sees they're grabbing a bundle. Which\n> means that the <url> in step 1 doesn't _have_ to be a bundle. It can be\n> \"go fetch from kernel.org, then come back to me\".\n\nOr it could be a packfile (and the client discovers roots), as you\nmentioned in a separate message.  I personally do not think it buys\nus much, as long as we do a bundle represented as a header and a\nseparate pack.\n"},{"id":"277965","messageId":"20160211214653.GA835@sigill.intra.peff.net","threadId":"41386","inReplyTo":"xmqq4mdfvspl.fsf@gitster.mtv.corp.google.com","subject":"Re: RFC: Resumable clone based on hybrid \"smart\" and \"dumb\" HTTP","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-02-11T21:46:53Z","receivedAt":"2016-02-11T21:46:53Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Feb 11, 2016 at 01:32:22PM -0800, Junio C Hamano wrote:\n\n> > ... One\n> > alternative would be to amend the bundle format so that rather than a\n> > single file, you get a bundle header whose end says \"...and my matching\n> > packfile is 1234-abcd\". And then the client knows that they can fetch\n> > that separately from the same source.\n> \n> I would imagine that we would introduce bundle v3 format for this.\n\nYeah, I think so. And in fact, the \"here are my packfiles...\" bit should\nprobably be in the v3 header.\n\n> It may want to say \"my matching packfiles are these\" to accomodate a\n> set of packs split at max-pack-size, but I am perfectly fine to say\n> you must create a single pack when you use a bundle with separate\n> header to keep things simpler.\n\nInteresting. My initial thought is that one could replace \"git bundle\ncreate foo.bundle --all && split foo.bundle\" with this (for storing or\ntransferring a bundle somewhere that cannot handle the whole thing in\none go).  It has the advantage that you do not need to recreate the full\nbundle to extract the data.\n\nBut I think the negatives of splitting across packs would outweigh that.\nYou cannot have cross-pack deltas, so your total size would be much\nlarger (in general, I have yet to see a case where max-pack-size is\nbeneficial, beyond the obvious \"your filesystem cannot store files\nlarger than N bytes\").\n\nSo I don't think it would be helpful for normal bundle use.\n\nIt _could_ be helpful in the context we're talking about here, though.\nIf I create the split-bundle so that people can resumable-clone from me,\nthey can only clone up to that bundle's creation point (and\nincrementally fetch the rest). But with a single pack, I can't update\nthe split-bundle without doing a full repack. With multiple packs, I\ncould regenerate the split-bundle header and just mention the new pack.\n\nIt wouldn't be as _efficient_ as a full repack of course, but it may be\na good, cheap interim solution between repacks.\n\n> >   2. Client goes to <url>. They see that they are fetching a bundle,\n> >      and know not to do the usual smart-http or dumb-http protocols.\n> >      They can fetch the bundle header resumably (though it's tiny, so it\n> >      doesn't really matter).\n> \n> Might be in megabytes range, though, with many refs.  It still is\n> tiny, though ;-).\n\nYes, but it's the same amount we already spew for a ref advertisement,\nwhich isn't resumable, either. :)\n\nI think I'd probably make this a straight fetch in the first iteration,\nand we can worry about making it resumable later on if people actually\ncare.\n\n> > And you'll notice, too, that all of the bundle-http magic kicks in\n> > during step 2 because the client sees they're grabbing a bundle. Which\n> > means that the <url> in step 1 doesn't _have_ to be a bundle. It can be\n> > \"go fetch from kernel.org, then come back to me\".\n> \n> Or it could be a packfile (and the client discovers roots), as you\n> mentioned in a separate message.  I personally do not think it buys\n> us much, as long as we do a bundle represented as a header and a\n> separate pack.\n\nYeah, I think I agree, and if it were just me, I'd implement the bundle\npart and call it done. But the important thing to me is that we haven't\neliminated the possibility of doing the pure-pack thing on top, if we\nchoose to.\n\n-Peff\n"},{"id":"277980","messageId":"CACsJy8DLYjXW1VBsxLYwcpjFV2s3MKqw9FL_2S7dMXpZigc2kA@mail.gmail.com","threadId":"41386","inReplyTo":"20160210210115.GA10155@google.com","subject":"Re: RFC: Resumable clone based on hybrid \"smart\" and \"dumb\" HTTP","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-02-11T23:53:42Z","receivedAt":"2016-02-11T23:53:42Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Feb 11, 2016 at 4:01 AM, Jonathan Nieder <jrnieder@gmail.com> wrote:\n> ....\n>\n> I really like this design.  I'm tempted to implement it (since it\n> lacks a bunch of the downsides of clone.bundle).\n\nA bit disappointed we still do not address resumable fetch. But I'm\nhappy that it makes people excited and one of them even tempted to\nimplement. This thread sounds like we'll see resumable clone before\n2017 :)\n-- \nDuy\n"},{"id":"278062","messageId":"CAP3OtXjyy+7cSi0S17FSsx8gnP1rKboeSAz_Hz1ZNVDuqcGASw@mail.gmail.com","threadId":"41386","inReplyTo":"20160210214945.GA5853@sigill.intra.peff.net","subject":"Re: RFC: Resumable clone based on hybrid \"smart\" and \"dumb\" HTTP","fromName":"Blake Burkhart","fromEmail":"bburky@bburky.com","sentAt":"2016-02-13T01:40:43Z","receivedAt":"2016-02-13T01:40:43Z","isPatch":false,"sender":{"key":"bburky@bburky.com","avatar":"https://avatars.githubusercontent.com/u/140526?v=4"},"body":"On Wed, Feb 10, 2016 at 3:49 PM, Jeff King <peff@peff.net> wrote:\n>> 2. Servers that support resumable clone include a \"resumable\"\n>> capability in the advertisement.\n>\n> Because the magic happens in the git protocol, that would mean this does\n> not have to be limited to git-over-http. It could be \"resumable=<url>\"\n> to point the client anywhere (the same server over a different protocol,\n> another server, etc).\n\nI'd like to call this out as a possible security issue before it gets\nimplemented. Allowing the server to instruct the client what protocol\nto use is a security risk. This sounds like a fine feature, just do it\ncarefully.\n\nI reported a similar issue was discussed off list which eventually\nbecame CVE-2015-7545. Basically, git-submodule allowed a repository to\nspecify any protocol via .gitmodules, causing the client to fetch an\narbitrary URL using a protocol of the attacker's choice. Sadly, the\nexistence of git-remote-ext allows easily executing arbitrary shell\ncommands if the server can tell the client to use it. Furthermore,\nit's possible the client has some insecure or sensitive custom git\nremote helpers installed.\n\nTo address this GIT_ALLOW_PROTOCOL was introduced, and git-submodule\nnow uses it as of 33cfccb. This environment variable specifies a\ndefault whitelist of protocols. Whoever implements this should\nprobably make use of GIT_ALLOW_PROTOCOL to limit resumable clones to\nthe same default whitelist that git-submodule now uses.\n\n-- \nBlake Burkhart\n"},{"id":"278064","messageId":"xmqq37sxtcyk.fsf@gitster.mtv.corp.google.com","threadId":"41386","inReplyTo":"CACsJy8DLYjXW1VBsxLYwcpjFV2s3MKqw9FL_2S7dMXpZigc2kA@mail.gmail.com","subject":"Re: RFC: Resumable clone based on hybrid \"smart\" and \"dumb\" HTTP","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-02-13T05:07:47Z","receivedAt":"2016-02-13T05:07:47Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen <pclouds@gmail.com> writes:\n\n> On Thu, Feb 11, 2016 at 4:01 AM, Jonathan Nieder <jrnieder@gmail.com> wrote:\n>> ....\n>>\n>> I really like this design.  I'm tempted to implement it (since it\n>> lacks a bunch of the downsides of clone.bundle).\n>\n> A bit disappointed we still do not address resumable fetch.\n\nAs you already said, the initial \"clone\" and subsequent incremental\n\"fetch\" are orthogonal issues.\n\nBecause the proposed update to \"clone\" has much larger payoff than\nthe proposed change to \"fetch\", i.e.\n\n * The amount of data that gets transferred is much larger, hence\n   the chance of network timing out in a poor network environment is\n   much higher, need for resuming much larger.\n\n * Not only the approach makes \"clone\" resumable and helping\n   clients, it helps the server offload bulk transfer out to CDN.\n\nand it has much smaller damage to the existing code, i.e.\n\n * We do not have to pessimize the packing process, only to discard\n   the bulk of bytes that were generated, like the proposed approach\n   for \"fetch\".\n\n * The area new code is needed is well isolated and the switch to\n   new protocol happens very early in the exchange without sharing\n   code to existing codepath; these properties make it less risky to\n   introduce regression.\n\nit is simply a good taste to give \"clone\" higher priority between\nthe two.\n\nSo I do not think there is much reason to feel disappointed.\n"},{"id":"278074","messageId":"20160213170044.GB30144@sigill.intra.peff.net","threadId":"41386","inReplyTo":"CAP3OtXjyy+7cSi0S17FSsx8gnP1rKboeSAz_Hz1ZNVDuqcGASw@mail.gmail.com","subject":"Re: RFC: Resumable clone based on hybrid \"smart\" and \"dumb\" HTTP","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-02-13T17:00:44Z","receivedAt":"2016-02-13T17:00:44Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Feb 12, 2016 at 07:40:43PM -0600, Blake Burkhart wrote:\n\n> On Wed, Feb 10, 2016 at 3:49 PM, Jeff King <peff@peff.net> wrote:\n> >> 2. Servers that support resumable clone include a \"resumable\"\n> >> capability in the advertisement.\n> >\n> > Because the magic happens in the git protocol, that would mean this does\n> > not have to be limited to git-over-http. It could be \"resumable=<url>\"\n> > to point the client anywhere (the same server over a different protocol,\n> > another server, etc).\n> \n> I'd like to call this out as a possible security issue before it gets\n> implemented. Allowing the server to instruct the client what protocol\n> to use is a security risk. This sounds like a fine feature, just do it\n> carefully.\n\nThanks for mentioning this. I agree it's a potential issue, and we\nshould use the same solution as submodules, as you pointed out:\n\n> To address this GIT_ALLOW_PROTOCOL was introduced, and git-submodule\n> now uses it as of 33cfccb. This environment variable specifies a\n> default whitelist of protocols. Whoever implements this should\n> probably make use of GIT_ALLOW_PROTOCOL to limit resumable clones to\n> the same default whitelist that git-submodule now uses.\n\n-Peff\n"},{"id":"278091","messageId":"CAJo=hJv-GWZOsv31iekW+AdNazLGQ=XYD=UXMO+RuB15baTsow@mail.gmail.com","threadId":"41386","inReplyTo":"20160210214945.GA5853@sigill.intra.peff.net","subject":"Re: RFC: Resumable clone based on hybrid \"smart\" and \"dumb\" HTTP","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2016-02-14T02:14:31Z","receivedAt":"2016-02-14T02:14:31Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Wed, Feb 10, 2016 at 1:49 PM, Jeff King <peff@peff.net> wrote:\n> On Wed, Feb 10, 2016 at 12:11:46PM -0800, Shawn Pearce wrote:\n>\n>> On Wed, Feb 10, 2016 at 10:59 AM, Shawn Pearce <spearce@spearce.org> wrote:\n>> >\n>> > ... Thoughts?\n>>\n>> Several of us at $DAY_JOB talked about this more today and thought a\n>> variation makes more sense:\n>>\n>> 1. Clients attempting clone ask for /info/refs?service=git-upload-pack\n>> like they do today.\n>>\n>> 2. Servers that support resumable clone include a \"resumable\"\n>> capability in the advertisement.\n>\n> Because the magic happens in the git protocol, that would mean this does\n> not have to be limited to git-over-http. It could be \"resumable=<url>\"\n> to point the client anywhere (the same server over a different protocol,\n> another server, etc).\n>\n>> 3. Updated clients on clone request GET /info/refs?service=git-resumable-clone.\n>>\n>> 4. The server may return a 302 Redirect to its current \"mostly whole\"\n>> pack file. This can be more flexible than \"refs/heads/*\", it just\n>> needs to be a mostly complete pack file that contains a complete graph\n>> from any arbitrary roots.\n>\n> And with \"resumable=<url>\", the client does not have to hit the server\n> to do a redirect; it can go straight to the final URL, saving a\n> round-trip.\n\nIt occurred to me today that to make the above (\"resumable=<url>\") as\nefficient as possible, we should allow HTTP clients to include\n&resumable=1 in the GET /info/refs URL as a hint to the server that if\nit serves a resumable=<url> it can shrink the ref advertisement to 1\nline containing the capabilities.\n\nClients are going to follow the returned <url> to get a bundle header,\nwhich contains the references. And then incremental fetch from the\nserver after downloading the pack. So returning references with the\nresumable URL during a clone is an unnecessary waste of bandwidth.\n\nWe could also consider allowing resumable=<url> to be relative to the\nrepository, so that on HTTP schemes a server could just reply\nresumable=pack-HASH.info or something short and not worry about\noverflowing the capabilities line.\n"},{"id":"278116","messageId":"20160214170525.GB10219@sigill.intra.peff.net","threadId":"41386","inReplyTo":"CAJo=hJv-GWZOsv31iekW+AdNazLGQ=XYD=UXMO+RuB15baTsow@mail.gmail.com","subject":"Re: RFC: Resumable clone based on hybrid \"smart\" and \"dumb\" HTTP","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-02-14T17:05:25Z","receivedAt":"2016-02-14T17:05:25Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Feb 13, 2016 at 06:14:31PM -0800, Shawn Pearce wrote:\n\n> > And with \"resumable=<url>\", the client does not have to hit the server\n> > to do a redirect; it can go straight to the final URL, saving a\n> > round-trip.\n> \n> It occurred to me today that to make the above (\"resumable=<url>\") as\n> efficient as possible, we should allow HTTP clients to include\n> &resumable=1 in the GET /info/refs URL as a hint to the server that if\n> it serves a resumable=<url> it can shrink the ref advertisement to 1\n> line containing the capabilities.\n\nI'm slightly wary of this. The client advertising \"resumable=1\" does not\nmean \"I will definitely do a resumable clone\". It means \"I support the\nresumable keyword, and if you share a resumable URL with me, I _might_\nuse it, depending on things that are none of your business\" (e.g., if it\ndoes not like the server URL's protocol).\n\nIt is recoverable by having the client re-contact the server without the\nresumable flag, so it could still be a net win if the client will select\nthe resumable URL a majority of the time.\n\nI'm also not happy about having an HTTP-only feature in the protocol. I\nliked Stefan's proposal for the \"v2\" protocol that would let the two\nsides exchange capabilities before the ref advertisement. Then the\nclient, having seen the server's resumable URL, knows whether or not\nto proceed with the advertisement.\n\n> Clients are going to follow the returned <url> to get a bundle header,\n> which contains the references. And then incremental fetch from the\n> server after downloading the pack. So returning references with the\n> resumable URL during a clone is an unnecessary waste of bandwidth.\n\nIf the bundle is up to date, the client can skip the follow-up\nincremental fetch, as it knows that it has everything needed for the\noriginal ref advertisement it got. Whether that's a net win depends on\nhow up-to-date the bundles are.\n\nIf \"C\" is the cost to contact the server at all and \"A\" is the cost of\nthe advertisement, then a \"hit\" with this scheme means the overhead is\nC+A (we contact the server only once). A \"miss\" means we have do the\nfollowup fetch anyway, and we pay 2C+2A (paying the advertisement cost\nboth times). Whereas with your scheme, we pay 2C+A always; two contacts,\nbut only the second has an advertisement.\n\nSo it depends on the relative cost of C and A, and how often we expect\nit to kick in.\n\nIn practice, I suspect it's mostly dominated by the cost of the actual\nclone objects anyway, but maybe that is different for Gerrit. I hear\nrefs/changes/ can get pretty big. :)\n\nBut if that is the case, then \"C\" is almost certainly negligible\ncompared to \"A\".\n\n> We could also consider allowing resumable=<url> to be relative to the\n> repository, so that on HTTP schemes a server could just reply\n> resumable=pack-HASH.info or something short and not worry about\n> overflowing the capabilities line.\n\nI think that's orthogonal, but yeah, it might be a nice feature for\nadmins setting up the server side config.\n\n-Peff\n"},{"id":"278117","messageId":"CAJo=hJuvHpWPQGV1gwgwOATy+8OO0Smt-BG9xb3q7eeJSypdaA@mail.gmail.com","threadId":"41386","inReplyTo":"20160214170525.GB10219@sigill.intra.peff.net","subject":"Re: RFC: Resumable clone based on hybrid \"smart\" and \"dumb\" HTTP","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2016-02-14T17:56:16Z","receivedAt":"2016-02-14T17:56:16Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Sun, Feb 14, 2016 at 9:05 AM, Jeff King <peff@peff.net> wrote:\n> On Sat, Feb 13, 2016 at 06:14:31PM -0800, Shawn Pearce wrote:\n>\n>> > And with \"resumable=<url>\", the client does not have to hit the server\n>> > to do a redirect; it can go straight to the final URL, saving a\n>> > round-trip.\n>>\n>> It occurred to me today that to make the above (\"resumable=<url>\") as\n>> efficient as possible, we should allow HTTP clients to include\n>> &resumable=1 in the GET /info/refs URL as a hint to the server that if\n>> it serves a resumable=<url> it can shrink the ref advertisement to 1\n>> line containing the capabilities.\n>\n> I'm slightly wary of this. The client advertising \"resumable=1\" does not\n> mean \"I will definitely do a resumable clone\". It means \"I support the\n> resumable keyword, and if you share a resumable URL with me, I _might_\n> use it, depending on things that are none of your business\" (e.g., if it\n> does not like the server URL's protocol).\n>\n> It is recoverable by having the client re-contact the server without the\n> resumable flag, so it could still be a net win if the client will select\n> the resumable URL a majority of the time.\n\nThat was my thinking; if the client starts out with &resumable=1 and\nthen doesn't like the URL the server offers in the advertisement it\ncan drop the query parameter and relist the references for a standard\ndynamic clone.\n\n> I'm also not happy about having an HTTP-only feature in the protocol. I\n> liked Stefan's proposal for the \"v2\" protocol that would let the two\n> sides exchange capabilities before the ref advertisement. Then the\n> client, having seen the server's resumable URL, knows whether or not\n> to proceed with the advertisement.\n\nOk, that's fair. Lets forget the \"&resumable=1\" for now so that the\nvarious protocols are more identical. Later if \"v2\" makes enough\nprogress we can rely on its capability exchange before advertisement\nto get this potential bandwidth savings.\n\n>> Clients are going to follow the returned <url> to get a bundle header,\n>> which contains the references. And then incremental fetch from the\n>> server after downloading the pack. So returning references with the\n>> resumable URL during a clone is an unnecessary waste of bandwidth.\n>\n> If the bundle is up to date, the client can skip the follow-up\n> incremental fetch, as it knows that it has everything needed for the\n> original ref advertisement it got. Whether that's a net win depends on\n> how up-to-date the bundles are.\n\nWell, that comes down to \"was the repository repacked since last\npush\". If yes the pack contains everything and its pack-*.info / split\nbundle header thing contains the same contents as packed-refs.\n\nI think in practice clients aren't going to bother with the\nimplementation detail of saving the initial ref advertisement aside,\ngrabbing the bundle header, and comparing them to see if it needs a\nsubsequent fetch. They are just going to discard the ref\nadvertisement, retrieve the pack, anchor the objects using the bundle\nheader, and then reconnect.\n\nRemember this isn't just about holding the refs in memory. To be\nresumable the client process may have to be restarted, so that initial\nref advertisement has to be written out to disk to persist.\n\n> If \"C\" is the cost to contact the server at all and \"A\" is the cost of\n> the advertisement, then a \"hit\" with this scheme means the overhead is\n> C+A (we contact the server only once). A \"miss\" means we have do the\n> followup fetch anyway, and we pay 2C+2A (paying the advertisement cost\n> both times). Whereas with your scheme, we pay 2C+A always; two contacts,\n> but only the second has an advertisement.\n>\n> So it depends on the relative cost of C and A, and how often we expect\n> it to kick in.\n>\n> In practice, I suspect it's mostly dominated by the cost of the actual\n> clone objects anyway, but maybe that is different for Gerrit. I hear\n> refs/changes/ can get pretty big. :)\n\nI hear refs/pulls/ can also get big. :)\n\nBut yes, some Gerrit advertisements are not small.\n"},{"id":"278367","messageId":"CAGZ79kaDM+sATm73t1pkdc7PqYeZVDgZyYaL-0b8VS2HB9x93g@mail.gmail.com","threadId":"41386","inReplyTo":"20160214170525.GB10219@sigill.intra.peff.net","subject":"Re: RFC: Resumable clone based on hybrid \"smart\" and \"dumb\" HTTP","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-02-16T18:34:21Z","receivedAt":"2016-02-16T18:34:21Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Sun, Feb 14, 2016 at 9:05 AM, Jeff King <peff@peff.net> wrote:\n>\n> I'm also not happy about having an HTTP-only feature in the protocol. I\n> liked Stefan's proposal for the \"v2\" protocol that would let the two\n> sides exchange capabilities before the ref advertisement. Then the\n> client, having seen the server's resumable URL, knows whether or not\n> to proceed with the advertisement.\n\nI completely stalled that side project, so if anyone wants to pick it up,\nfeel free.\n\nThe server side was complete (and reviewed and people liked it).\nThe only missing part was the client side to be completed. (There were\nsome patches back and forth and it worked mostly, the design seemed to\nbe accepted and well). The code is at\nhttps://github.com/stefanbeller/git/commits/protocol2-10\n\nThanks,\nStefan\n"}]}