{"thread":{"id":"45007","subject":"Git clonebundles","startedAt":"2017-01-31T07:01:29Z","lastAt":"2017-02-08T14:31:07Z","messageCount":8,"participants":["Stefan Saasen","Shawn Pearce","Christian Couder","Junio C Hamano","Johannes Schindelin","Stefan Beller"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"310589","messageId":"CADoxLGPFgF7W4XJzt0X+xFJDoN6RmfFGx_96MO9GPSSOjDK0EQ@mail.gmail.com","threadId":"45007","inReplyTo":null,"subject":"Git clonebundles","fromName":"Stefan Saasen","fromEmail":"ssaasen@atlassian.com","sentAt":"2017-01-31T07:00:34Z","receivedAt":"2017-01-31T07:01:29Z","isPatch":false,"sender":{"key":"ssaasen@atlassian.com","avatar":null},"body":"Hi all,\n\nBitbucket recently added support for Mercurial’s clonebundle extension\n(http://gregoryszorc.com/blog/2015/10/22/cloning-improvements-in-mercurial-3.6/).\nMercurial’s clone bundles allow the Mercurial client to seed a repository using\na bundle file instead of dynamically generating a bundle for the client.\n\n\nMercurial clonebundles?\n~~~~~~~~~~~~~~~~~~~~~~~\n\nWith Mercurial clonebundles the high level clone sequence looks like this:\n\n1. The command \"hg clone URL\" attempts to clone the repository at URL.\n2. If a bundle file exists for the repository, the existence of the file\n`clonebundles.manifest` causes the server to advertise the `clonebundle`\ncapability (capabilities lookup is the first command the client issues).\n3. In the above case the client then executes the command \"clonebundles\".\n4. The manifest file will be returned.\n5. The client then selects a bundle file to download from the list of URLs\nadvertised in the manifests file, to seed the repository.\n6. To update the repository the last step involves fetching the latest changes.\n\n\nWhy is this useful?\n~~~~~~~~~~~~~~~~~~~\n\nThe fact that clone bundles can be distributed as static files enables us to\nuse static file servers for bundle distribution. Users have also reported\nlatency improvements for clone operations of popular Mercurial repositories.\nAdditionally this significantly reduces the resource usage of clone operations,\nas clone operations are reduced to simpler fetches to resolve the delta between\nthe current repository and the downloaded bundle state.\n\n\nclonebundles for git?\n~~~~~~~~~~~~~~~~~~~~~\n\nWe recently looked into how this concept could be translated to git. This is\nnot a new idea and has been discussed before (more on that later) but our\nsuccess with the Mercurial clonebundle rollout prompted us to revisit this\ntopic.\n\nWe believe that bringing a similar concept to git could have the following\nbenefits:\n\n* Improved clone times for users that clone large git repositories, especially\n if bundle file distribution leverages global CDNs.\n* Improved scalability of git for managing large popular repositories.\n Offloading a significant portion of the clone resource usage to CDNs or static\n file hosts.\n\n\nOur current proof-of-concept to explore this space, closely follows\nthe approach from Mercurial outlined above.\n\n* An `/info/bundle` path returns a bundle manifest (over HTTP)\n* The bundle manifest contains a simple list of URLs with some additional meta\n data that allows the client to select a suitable bundle download URL\n* The bundle download URL points to a bundle file generated using `git bundle\n create` including all the relevant refs as a self contained repository seed.\n* The client probes the target URL with a `GET` request to $URL/info/bundle and\n downloads the bundle file if present.\n* The repository will be created based on the downloaded bundle (downloading a\n static file allows resumable downloads or parallel downloads of chunks if the\n file/web server supports range requests).\n* A `git fetch` and the appropriate checkout then updates the \"cloned\"\n repository to match the latest upstream state.\n\nThe proof-of-concept was built as an external binary `git-clone2` that\nmimics the behaviour of the `git clone` command, so unfortunately I\ncan't provide any patches to git to demonstrate the behaviour.\n\n\nUltimately our proof-of-concept is built around a few core ideas:\n\n* Re-use the existing bundle format as a single-file, self-contained\nrepository representation.\n* Introduce a bundle manifest (accessible at `$URL/info/bundle`) that allows\n the client to resolve a suitable bundle download URL.\n* Teach the `git clone` command to accept and prefer seeding a repository using\n a static bundle file that is advertised in a bundle manifest.\n* Re-use as much as possible of the existing commands and in particular the\n `git bundle` machinery to seed the repository and to create the static bundle\n file.\n* We accept additional storage requirements for the bundle files in addition to\n the actual repository content in pack-files or loose objects.\nHosting providers\n or system administrators are free to decide how many bundles to advertise and\n how frequently the bundles are updated.\n* It targets the \"seed from a bundle file\" use case, with resumable clones just\n being a potential side-effect.\n\n\nSome of the problems that need to be solved with an approach like this are:\n\n* Bundle advertisement/bundle negotiation: We considered advertising a\n new capability \"clonebundle\" as part of the rev advertisement\ncapabilities list.\n This would allow clients that support clonebundles to abort the clone attempt\n and resolve a suitable bundle URL from a bundle manifest at `$URL/info/bundle`\n instead. For HTTP this would amount to an early termination when\nretrieving the\n ref-advertisement.\n Note: We didn't pursue this for our proof-of-concept so we didn't\nexplore whether\n this is feasible.\n* Uniform approach for the supported transports: Our proof-of-concept\nonly supports HTTP as\n a transport. Ideally the clonebundle capability could be supported by all\n available transports (of which at least ssh would be highly desirable).\n* Bundle manifest and bundle download: It is unclear whose responsibility it is\n to generate the bundle manifest with the bundle download URLs. Most likely the\n bundle files will be served using a webserver or CDN, so download\nURL generation\n should not be a core git responsibility. For hosting purpose we envision that\n the bundle manifest might contain dynamic download URLs with personalised\n access tokens with expiry.\n* Bundle generation: Similar to the above it is unclear how bundle\ngeneration is handled.\n For hosting purposes, the operator would likely want to influence\nwhen and how bundles are generated.\n\n\n\nPrior art\n~~~~~~~~~\n\nOur proof-of-concept is built on top of ideas that have been\ncirculating for a while. We are aware of a number of proposed changes\nin this space:\n\n\n* Jeff King's work on network bundles:\nhttps://github.com/peff/git/commit/17e2409df37edd0c49ef7d35f47a7695f9608900\n* Nguyễn Thái Ngọc Duy's work on \"[PATCH 0/8] Resumable clone\nrevisited, proof of concept\":\nhttps://www.spinics.net/lists/git/msg267260.html\n* Resumable clone work by Kevin Wern:\nhttps://public-inbox.org/git/1473984742-12516-1-git-send-email-kevin.m.wern@gmail.com/\n\n\nWhilst the above mentioned proposals/proposed changes are in a similar\nspace, I would be interest to understand whether there is any\nconsensus on the general idea of supporting static bundle files as a\nmechanism to seed a repository?\nI would also appreciate any pointers to other discussions in this area.\n\n\nBest regards,\nStefan Saasen & Erik van Zijst; Atlassian Bitbucket\n"},{"id":"310858","messageId":"CAJo=hJsS6FmL9iNScaXqkWJumALfGr8Od5QkbfZ+ZG3osxkp7Q@mail.gmail.com","threadId":"45007","inReplyTo":"CADoxLGPFgF7W4XJzt0X+xFJDoN6RmfFGx_96MO9GPSSOjDK0EQ@mail.gmail.com","subject":"Re: Git clonebundles","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2017-02-04T17:39:20Z","receivedAt":"2017-02-04T17:39:47Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Mon, Jan 30, 2017 at 11:00 PM, Stefan Saasen wrote:\n>\n> Bitbucket recently added support for Mercurial’s clonebundle extension\n> (http://gregoryszorc.com/blog/2015/10/22/cloning-improvements-in-mercurial-3.6/).\n> Mercurial’s clone bundles allow the Mercurial client to seed a repository using\n> a bundle file instead of dynamically generating a bundle for the client.\n...\n> Prior art\n> ~~~~~~~~~\n>\n> Our proof-of-concept is built on top of ideas that have been\n> circulating for a while. We are aware of a number of proposed changes\n> in this space:\n>\n>\n> * Jeff King's work on network bundles:\n> https://github.com/peff/git/commit/17e2409df37edd0c49ef7d35f47a7695f9608900\n> * Nguyễn Thái Ngọc Duy's work on \"[PATCH 0/8] Resumable clone\n> revisited, proof of concept\":\n> https://www.spinics.net/lists/git/msg267260.html\n> * Resumable clone work by Kevin Wern:\n> https://public-inbox.org/git/1473984742-12516-1-git-send-email-kevin.m.wern@gmail.com/\n\nI think you missed the most common deployment of prior art, which is\nAndroid using the git-repo tool[1]. The git-repo tool has had\nclone.bundle support since Sep 2011[2] and the Android Git servers\nhave been answering /clone.bundle requests[3] since just before that.\nThe bundle files are generated with `git bundle create` on a regular\nschedule by cron.\n\n[1] https://gerrit.googlesource.com/git-repo/+/04071c1c72437a930db017bd4c562ad06087986a/project.py#2091\n[2] https://gerrit.googlesource.com/git-repo/+/f322b9abb4cadc67b991baf6ba1b9f2fbd5d7812\n[3] https://android.googlesource.com/platform/frameworks/base/clone.bundle\n\n\n> Whilst the above mentioned proposals/proposed changes are in a similar\n> space, I would be interest to understand whether there is any\n> consensus on the general idea of supporting static bundle files as a\n> mechanism to seed a repository?\n\nI don't think we have a consensus on how to advertise a bundle file is\navailable, which is why there are so many instances of prior art. In\n2011 I just threw together /clone.bundle on HTTP because it was easy\nto make the Python wrapper ask for the file and handle 404 gracefully\nas not found and fall back to `git clone`.\n"},{"id":"310877","messageId":"CAP8UFD2ffeHr0-z3xPFFODqRTTgVnrrWcYRbASpUOrp0QGnp6g@mail.gmail.com","threadId":"45007","inReplyTo":"CAJo=hJsS6FmL9iNScaXqkWJumALfGr8Od5QkbfZ+ZG3osxkp7Q@mail.gmail.com","subject":"Re: Git clonebundles","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-02-05T16:37:06Z","receivedAt":"2017-02-05T16:37:14Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Sat, Feb 4, 2017 at 6:39 PM, Shawn Pearce wrote:\n> On Mon, Jan 30, 2017 at 11:00 PM, Stefan Saasen wrote:\n>>\n>> Bitbucket recently added support for Mercurial’s clonebundle extension\n>> (http://gregoryszorc.com/blog/2015/10/22/cloning-improvements-in-mercurial-3.6/).\n>> Mercurial’s clone bundles allow the Mercurial client to seed a repository using\n>> a bundle file instead of dynamically generating a bundle for the client.\n> ...\n>> Prior art\n>> ~~~~~~~~~\n>>\n>> Our proof-of-concept is built on top of ideas that have been\n>> circulating for a while. We are aware of a number of proposed changes\n>> in this space:\n>>\n>>\n>> * Jeff King's work on network bundles:\n>> https://github.com/peff/git/commit/17e2409df37edd0c49ef7d35f47a7695f9608900\n>> * Nguyễn Thái Ngọc Duy's work on \"[PATCH 0/8] Resumable clone\n>> revisited, proof of concept\":\n>> https://www.spinics.net/lists/git/msg267260.html\n>> * Resumable clone work by Kevin Wern:\n>> https://public-inbox.org/git/1473984742-12516-1-git-send-email-kevin.m.wern@gmail.com/\n>\n> I think you missed the most common deployment of prior art, which is\n> Android using the git-repo tool[1]. The git-repo tool has had\n> clone.bundle support since Sep 2011[2] and the Android Git servers\n> have been answering /clone.bundle requests[3] since just before that.\n> The bundle files are generated with `git bundle create` on a regular\n> schedule by cron.\n>\n> [1] https://gerrit.googlesource.com/git-repo/+/04071c1c72437a930db017bd4c562ad06087986a/project.py#2091\n> [2] https://gerrit.googlesource.com/git-repo/+/f322b9abb4cadc67b991baf6ba1b9f2fbd5d7812\n> [3] https://android.googlesource.com/platform/frameworks/base/clone.bundle\n\nThere is also Junio's work on Bundle v3 that was unfortunately\nrecently discarded.\nLook for \"jc/bundle\" in:\n\nhttp://public-inbox.org/git/xmqq4m0cry60.fsf@gitster.mtv.corp.google.com/\n\nand previous \"What's cooking in git.git\" emails.\n\nI am also working on adding external object database support using\nprevious work by Peff:\n\nhttp://public-inbox.org/git/20161130210420.15982-1-chriscool@tuxfamily.org/\n\nthat could be extended to support clone bundles.\n\n[...]\n"},{"id":"310948","messageId":"xmqq4m070xua.fsf@gitster.mtv.corp.google.com","threadId":"45007","inReplyTo":"CAP8UFD2ffeHr0-z3xPFFODqRTTgVnrrWcYRbASpUOrp0QGnp6g@mail.gmail.com","subject":"Re: Git clonebundles","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-02-06T22:16:13Z","receivedAt":"2017-02-06T22:16:27Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> There is also Junio's work on Bundle v3 that was unfortunately\n> recently discarded.\n> Look for \"jc/bundle\" in:\n>\n> http://public-inbox.org/git/xmqq4m0cry60.fsf@gitster.mtv.corp.google.com/\n>\n> and previous \"What's cooking in git.git\" emails.\n\nIf people think it might be useful to have it around to experiment,\nI can resurrect and keep that in 'pu' (or rather 'jch'), as long as\nit does not overlap and conflict with other topics in flight. Let\nme try that in today's integration cycle.\n"},{"id":"310982","messageId":"alpine.DEB.2.20.1702071303370.3496@virtualbox","threadId":"45007","inReplyTo":"xmqq4m070xua.fsf@gitster.mtv.corp.google.com","subject":"Re: Git clonebundles","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-02-07T12:04:50Z","receivedAt":"2017-02-07T12:05:15Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Mon, 6 Feb 2017, Junio C Hamano wrote:\n\n> Christian Couder writes:\n> \n> > There is also Junio's work on Bundle v3 that was unfortunately\n> > recently discarded. Look for \"jc/bundle\" in:\n> >\n> > http://public-inbox.org/git/xmqq4m0cry60.fsf@gitster.mtv.corp.google.com/\n> >\n> > and previous \"What's cooking in git.git\" emails.\n> \n> If people think it might be useful to have it around to experiment, I\n> can resurrect and keep that in 'pu' (or rather 'jch'), as long as it\n> does not overlap and conflict with other topics in flight. Let me try\n> that in today's integration cycle.\n\nI would like to remind you of my suggestion to make this more publicly\nvisible and substantially easier to play with, by adding it as an\nexperimental feature (possibly guarded via an explicit opt-in config\nsetting).\n\nCiao,\nJohannes\n"},{"id":"310990","messageId":"CAGZ79kYPx9_z9x86BwC4bW4D9fPNwzjvPsW8mAsnkk94wtmQSw@mail.gmail.com","threadId":"45007","inReplyTo":"alpine.DEB.2.20.1702071303370.3496@virtualbox","subject":"Re: Git clonebundles","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-02-07T15:34:04Z","receivedAt":"2017-02-07T15:34:10Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Feb 7, 2017 at 4:04 AM, Johannes Schindelin\n wrote:\n> Hi Junio,\n>\n> On Mon, 6 Feb 2017, Junio C Hamano wrote:\n>\n>> Christian Couder writes:\n>>\n>> > There is also Junio's work on Bundle v3 that was unfortunately\n>> > recently discarded. Look for \"jc/bundle\" in:\n>> >\n>> > http://public-inbox.org/git/xmqq4m0cry60.fsf@gitster.mtv.corp.google.com/\n>> >\n>> > and previous \"What's cooking in git.git\" emails.\n>>\n>> If people think it might be useful to have it around to experiment, I\n>> can resurrect and keep that in 'pu' (or rather 'jch'), as long as it\n>> does not overlap and conflict with other topics in flight. Let me try\n>> that in today's integration cycle.\n>\n> I would like to remind you of my suggestion to make this more publicly\n> visible and substantially easier to play with, by adding it as an\n> experimental feature (possibly guarded via an explicit opt-in config\n> setting).\n>\n> Ciao,\n> Johannes\n\nFor making this more publicly visible, I want to look into publishing\nthe cooking reports on the git-scm.com. Maybe we can have a \"dev\"\nsection there, that has\n* a \"getting started\" section\n linking to\n Documentation/SubmittingPatches\n How to setup your travis\n* \"current state of development\" section\n e.g. the cooking reports, the\n release calender, description of the workflow\n (which branches do exist and serve which purpose),\n\nMost of the static information is already covered quite\nwell in Documentation/ so there is definitively overlap,\nhence lots of links to the ground truth.\n\nThe dynamic information however (release calender,\ncooking reports) are not described well enough in\nDocumentation/ so I think we'd want to focus on these\nin that dev section.\n"},{"id":"311007","messageId":"xmqqzihxyb66.fsf@gitster.mtv.corp.google.com","threadId":"45007","inReplyTo":"alpine.DEB.2.20.1702071303370.3496@virtualbox","subject":"Re: Git clonebundles","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-02-07T20:54:09Z","receivedAt":"2017-02-07T20:54:27Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n>> If people think it might be useful to have it around to experiment, I\n>> can resurrect and keep that in 'pu' (or rather 'jch'), as long as it\n>> does not overlap and conflict with other topics in flight. Let me try\n>> that in today's integration cycle.\n>\n> I would like to remind you of my suggestion to make this more publicly\n> visible and substantially easier to play with, by adding it as an\n> experimental feature (possibly guarded via an explicit opt-in config\n> setting).\n\nI do not understand why you want to give this topic undue prominence\novver any other random topic that cook in 'pu' and later merged down\nto 'next' and then 'master' only after they turn out to be useful\n(or at least harmless).\n\nIf there were somebody who is the champion of that topic, advocating\nthat any clone-bundle solution must be based on this topic, it would\nbe different. Even though I am not opposed to the topic myself, I\nam not that somebody. That is why I kept it around to wait to see\nif somebody finds it potentially useful and then discarded it after\nseeing no such person stepped up.\n\nThat champion of the topic would spend the necessaly engineering\neffort to document it as experimental, to make sure that there is a\nreasonable upgrade/transition route if the \"v3\" format turns out to\nbe not very useful, etc. by rerolling the patches or following-up on\nthem to advance it from 'pu' down to 'next' and to 'master' just\nlike any other topic.\n\nJudging from the tone of his message (i.e. \"unfortunately\" in it),\nChristian may want to be one, or somebody else may want to be one.\n"},{"id":"311061","messageId":"alpine.DEB.2.20.1702081521290.3496@virtualbox","threadId":"45007","inReplyTo":"xmqqzihxyb66.fsf@gitster.mtv.corp.google.com","subject":"Re: Git clonebundles","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-02-08T14:28:58Z","receivedAt":"2017-02-08T14:31:07Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Tue, 7 Feb 2017, Junio C Hamano wrote:\n\n> Johannes Schindelin writes:\n> \n> >> If people think it might be useful to have it around to experiment, I\n> >> can resurrect and keep that in 'pu' (or rather 'jch'), as long as it\n> >> does not overlap and conflict with other topics in flight. Let me try\n> >> that in today's integration cycle.\n> >\n> > I would like to remind you of my suggestion to make this more publicly\n> > visible and substantially easier to play with, by adding it as an\n> > experimental feature (possibly guarded via an explicit opt-in config\n> > setting).\n> \n> I do not understand why you want to give this topic undue prominence\n> ovver any other random topic that cook in 'pu' [...]\n\nSince you ask so nicely for an explanation: clonebundles got a really\nlively and active discussion at the Contributors' Summit. So it is not\nyour run of the mill typo fix, the bundle issue is something that clearly\nreceives a lot of interest in particular from developers who are\nunfamiliar with the idiosynchracies of the code Git development.\n\nAnd I got the very distinct impression that Git would benefit a lot from\nthese developers, *in particular* since they come with fresh perspectives.\n\nNow, we can make it hard for them (e.g. expecting them to sift through a\nfew months' worth of What's Cooking mails, to find out whether there has\nbeen any related work, and what is the branch name, if any, and where to\nfind that branch), and we can alternatively make it easy for them to help\nus make Git better.\n\nI would like us to choose the easier route for them. Because it would\nbenefit us.\n\nCiao,\nJohannes\n"}]}