{"thread":{"id":"45005","subject":"[ANNOUNCE] Git Merge Contributor Summit topic planning","startedAt":"2017-01-31T00:48:14Z","lastAt":"2017-02-01T21:35:18Z","messageCount":9,"participants":["Jeff King","Erik van Zijst","Junio C Hamano","Christian Couder","Stefan Beller"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"310581","messageId":"20170131004804.p5sule4rh2xrgtwe@sigill.intra.peff.net","threadId":"45005","inReplyTo":null,"subject":"[ANNOUNCE] Git Merge Contributor Summit topic planning","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-01-31T00:48:05Z","receivedAt":"2017-01-31T00:48:14Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"The Contributor Summit is only a few days away; I'd like to work out a\nfew bits of logistics ahead of time.\n\nWe're up to 26 attendees. The room layout will probably be three big\nround-tables, with a central projector. We should be able to have\neverybody pay attention to a single speaker, or break into 3 separate\nconversations.\n\nThe list of topics is totally open. If you're coming and have something\nyou'd like to present or discuss, then propose it here. If you're _not_\ncoming, you may still chime in with input on topics, but please don't\nsuggest a topic unless somebody who is there will agree to lead the\ndiscussion.\n\nWe'll write the final list on a whiteboard on Thursday morning, vote on\nwhat looks good, and then work our way down the list.  Topics don't\n_have_ to be proposed here ahead of time, but I'd encourage people to do\nso as it leaves time for others to consider them and possibly do any\nbackground thinking or research.\n\nThe rough schedule is:\n\n  0830 to 0930 - registration, breakfast, milling about and socializing;\n                 be aware that Git Merge Workshop attendees will be\n\t\t doing the same things in the same space, so show up\n\t\t with enough time to navigate a bit of a crowd.\n\n  0930 to 1215 - we retire to our Fortress of Solitude to talk about\n                 Very Important Git Things\n\n  1215 to 1330 - lunch\n\n  1330 to 1500 - Very Important Git Things, part deux. The end time\n\t\t isn't a hard deadline, so we can go as late as 1600 if\n\t\t the discussion keeps up.\n\nThere's no organized dinner planned. At our size, I think it's probably\nmost productive to let people form small groups for dinner if they want\nto. But if somebody is really interested in trying to do a big group\nreservation, they are welcome to try to organize it.\n\n-Peff\n"},{"id":"310582","messageId":"20170131005900.fo5qnbsb67rzgnjt@sigill.intra.peff.net","threadId":"45005","inReplyTo":"20170131004804.p5sule4rh2xrgtwe@sigill.intra.peff.net","subject":"Re: [ANNOUNCE] Git Merge Contributor Summit topic planning","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-01-31T00:59:01Z","receivedAt":"2017-01-31T01:05:51Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jan 31, 2017 at 01:48:05AM +0100, Jeff King wrote:\n\n> The list of topics is totally open. If you're coming and have something\n> you'd like to present or discuss, then propose it here. If you're _not_\n> coming, you may still chime in with input on topics, but please don't\n> suggest a topic unless somebody who is there will agree to lead the\n> discussion.\n\nHere are the two topics I plan on bringing:\n\n  - Git / Software Freedom Conservancy yearly report. I'll plan to give\n    a rundown of the past year's activities and financials, along with\n    some open questions that could benefit from community input.\n\n  - The git-scm.com website: who runs that thing, anyway? An overview\n    of the site, how it's managed, and what it needs.\n\nI plan to send out detailed emails on both topics to the list on\nWednesday, and then follow-up with a summary of any useful in-person\ndiscussion (since obviously not everybody will be at the summit).\n\nI'd encourage anybody with a topic to present to consider doing\nsomething similar.\n\n-Peff\n"},{"id":"310638","messageId":"1485941532-47993-1-git-send-email-erik.van.zijst@gmail.com","threadId":"45005","inReplyTo":"20170131004804.p5sule4rh2xrgtwe@sigill.intra.peff.net","subject":"Re: [ANNOUNCE] Git Merge Contributor Summit topic planning","fromName":"Erik van Zijst","fromEmail":"erik.van.zijst@gmail.com","sentAt":"2017-02-01T09:32:12Z","receivedAt":"2017-02-01T09:32:27Z","isPatch":false,"sender":{"key":"erik.van.zijst@gmail.com","avatar":null},"body":"On Tue, Jan 31, 2017 at 01:48:05AM +0100, Jeff King wrote:\n\n> The list of topics is totally open. If you're coming and have something\n> you'd like to present or discuss, then propose it here. If you're _not_\n> coming, you may still chime in with input on topics, but please don't\n> suggest a topic unless somebody who is there will agree to lead the\n> discussion.\n\nI would like to talk about the possibility of CDN-aided cloning\noperations as mentioned on this list earlier this week:\nhttp://public-inbox.org/git/CADoxLGPFgF7W4XJzt0X+xFJDoN6RmfFGx_96MO9GPSSOjDK0EQ@mail.gmail.com/\n\nAt Bitbucket we have recently rolled out so-called clonebundle support\nfor Mercurial repositories.\n\nFull clone operations are rather expensive on the server and are\nresponsible for a substantial part of our CPU and IO load. CDN-based\nclonebundles have allowed us to eliminate most of this load for\nMercurial repos and we've since built a clonebundle spike for Git.\n\nClients performing a full clone get redirected to a CDN where they seed\ntheir new local repo from a pre-built bundle file, and then pull/fetch\nany remaining changes. Mercurial has had native, built-in support for\nthis for a while now.\n\nI imagine other large code hosts could benefit from this as well and\nI'd love to gauge the group's interest for this. Could this make sense\nfor Git? Would it have a chance of landing?\n\nOur spike implements it as an optional capability during ref\nadvertisement. What are your thoughts on this?\n\nCheers,\nErik\n"},{"id":"310649","messageId":"20170201145300.4pn3faodhdb72jly@sigill.intra.peff.net","threadId":"45005","inReplyTo":"1485941532-47993-1-git-send-email-erik.van.zijst@gmail.com","subject":"Re: [ANNOUNCE] Git Merge Contributor Summit topic planning","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-02-01T14:53:00Z","receivedAt":"2017-02-01T14:53:20Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Feb 01, 2017 at 10:32:12AM +0100, Erik van Zijst wrote:\n\n> Clients performing a full clone get redirected to a CDN where they seed\n> their new local repo from a pre-built bundle file, and then pull/fetch\n> any remaining changes. Mercurial has had native, built-in support for\n> this for a while now.\n> \n> I imagine other large code hosts could benefit from this as well and\n> I'd love to gauge the group's interest for this. Could this make sense\n> for Git? Would it have a chance of landing?\n> \n> Our spike implements it as an optional capability during ref\n> advertisement. What are your thoughts on this?\n\nI think this is definitely an interesting topic to discuss tomorrow.\n\nHere are a few observations from my past thinking on the issue. I\nhaven't read the proposal from earlier this week yet, so some of them\nmay be obsolete.\n\nSeeding from a bundle CDN generally solves two problems: getting the\nbulk of the data from someplace with higher bandwidth (the CDN), and\ngetting the bulk of the data over a protocol that can be resumed (the\nbundle).\n\nBut we don't necessarily have to solve both problems simultaneously.\nAnd you might not want to. Storing a separate bundle on another server\nis complicated to configure, and doubles the amount of disk space you\nneed (just half of it is on the CDN). Using a bundle means you can't\nseed from a non-bundle source.\n\nSo for any solution, I'd want to consider how you can put together the\npieces. Can you seed from a non-bundle? Can you seed from yourself and\njust get resumability? If so, how hard is it to serve a pseudo-bundle\nbased on the packfiles you have on disk (i.e., getting resumability\nat least in the common cases without paying the disk cost). I.e., saving\nenough data that you could reconstruct the bundle byte-for-byte when you\nneed to.\n\nIf you _can_ do that latter part, and you take \"I only care about\nresumability\" to the simplest extreme, you'd probably end up with a\nprotocol more like:\n\n  Client: I need a packfile with this want/have\n  Server: OK, here it is; its opaque id is XYZ.\n  ... connection interrupted ...\n  Client: It's me again. I have up to byte N of pack XYZ\n  Server: OK, resuming\n          [or: I don't have XYZ anymore; start from scratch]\n\nThen generating XYZ and generating that bundle are basically the same\ntask.\n\nAll just food for thought. I look forward to digging into it more on the\nlist and in the in-person discussion.\n\n-Peff\n"},{"id":"310652","messageId":"xmqqh94dpyzs.fsf@gitster.mtv.corp.google.com","threadId":"45005","inReplyTo":"20170201145300.4pn3faodhdb72jly@sigill.intra.peff.net","subject":"Re: [ANNOUNCE] Git Merge Contributor Summit topic planning","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-02-01T18:06:15Z","receivedAt":"2017-02-01T18:06:23Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> If you _can_ do that latter part, and you take \"I only care about\n> resumability\" to the simplest extreme, you'd probably end up with a\n> protocol more like:\n>\n>   Client: I need a packfile with this want/have\n>   Server: OK, here it is; its opaque id is XYZ.\n>   ... connection interrupted ...\n>   Client: It's me again. I have up to byte N of pack XYZ\n>   Server: OK, resuming\n>           [or: I don't have XYZ anymore; start from scratch]\n>\n> Then generating XYZ and generating that bundle are basically the same\n> task.\n\nThe above allows a simple and naive implementation of generating a\npackstream and \"tee\"ing it to a spool file to be kept while sending\nto the first client that asks XYZ.\n\nThe story I heard from folks who run git servers at work for Android\nand other projects, however, is that they rarely see two requests\nwith want/have that result in an identical XYZ, unless \"have\" is an\nempty set (aka \"clone\").  In a busy repository, between two clone\nrequests relatively close together, somebody would be pushing, so\nyou'd need many XYZs in your spool even if you want to support only\nthe \"clone\" case.\n\nSo in the real life, I think that the exchange needs to be more\nlike this:\n\n    C: I need a packfile with this want/have\n    ... C/S negotiate what \"have\"s are common ...\n    S: Sorry, but our negitiation indicates that you are way too\n       behind.  I'll send you a packfile that brings you up to a\n       slightly older set of \"want\", so pretend that you asked for\n       these slightly older \"want\"s instead.  The opaque id of that\n       packfile is XYZ.  After getting XYZ, come back to me with\n       your original set of \"want\"s.  You would give me more recent\n       \"have\" in that request.  \n    ... connection interrupted ...\n    C: It's me again.  I have up to byte N of pack XYZ\n    S: OK, resuming (or: I do not have it anymore, start from scratch)\n    ... after 0 or more iterations C fully receives and digests XYZ ...\n\nand then the above will iterate until the server does not have to\nsay \"Sorry but you are way too behind\" and returns a packfile\nwithout having to tweak the \"want\".\n\nThat way, you can limit the number of XYZ you would need to keep to\na reasonable number.\n\nThe recent proposal by Jonathan Tan also allows the server side to\ntweak the final tips the client receives after the protocol exchange\nstarted.  I suspect the above two will become related.\n"},{"id":"310658","messageId":"CAP8UFD3NE65Uy1kWMCNrpC05Fs_WY=g7K2NmMcab6CGUy4c8LA@mail.gmail.com","threadId":"45005","inReplyTo":"20170131005900.fo5qnbsb67rzgnjt@sigill.intra.peff.net","subject":"Re: [ANNOUNCE] Git Merge Contributor Summit topic planning","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-02-01T19:51:59Z","receivedAt":"2017-02-01T19:52:06Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Tue, Jan 31, 2017 at 1:59 AM, Jeff King <peff@peff.net> wrote:\n> On Tue, Jan 31, 2017 at 01:48:05AM +0100, Jeff King wrote:\n>\n>> The list of topics is totally open. If you're coming and have something\n>> you'd like to present or discuss, then propose it here. If you're _not_\n>> coming, you may still chime in with input on topics, but please don't\n>> suggest a topic unless somebody who is there will agree to lead the\n>> discussion.\n>\n> Here are the two topics I plan on bringing:\n>\n>   - Git / Software Freedom Conservancy yearly report. I'll plan to give\n>     a rundown of the past year's activities and financials, along with\n>     some open questions that could benefit from community input.\n>\n>   - The git-scm.com website: who runs that thing, anyway? An overview\n>     of the site, how it's managed, and what it needs.\n>\n> I plan to send out detailed emails on both topics to the list on\n> Wednesday, and then follow-up with a summary of any useful in-person\n> discussion (since obviously not everybody will be at the summit).\n>\n> I'd encourage anybody with a topic to present to consider doing\n> something similar.\n\nGitLab people at the Summit (this includes me) would like to spend a\nfew minutes to introduce https://gitlab.com/gitlab-org/gitaly/ and\nanswer any questions.\n"},{"id":"310660","messageId":"CAGZ79kZyHyhaBSgP+jt+SRNx51t5XVQB7H0=RSS31wqAcQCptQ@mail.gmail.com","threadId":"45005","inReplyTo":"20170131004804.p5sule4rh2xrgtwe@sigill.intra.peff.net","subject":"Re: [ANNOUNCE] Git Merge Contributor Summit topic planning","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-02-01T20:37:29Z","receivedAt":"2017-02-01T20:37:35Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Jan 30, 2017 at 4:48 PM, Jeff King <peff@peff.net> wrote:\n> The Contributor Summit is only a few days away; I'd like to work out a\n> few bits of logistics ahead of time.\n>\n> We're up to 26 attendees. The room layout will probably be three big\n> round-tables, with a central projector. We should be able to have\n> everybody pay attention to a single speaker, or break into 3 separate\n> conversations.\n>\n> The list of topics is totally open. If you're coming and have something\n> you'd like to present or discuss, then propose it here. If you're _not_\n> coming, you may still chime in with input on topics, but please don't\n> suggest a topic unless somebody who is there will agree to lead the\n> discussion.\n\nsubmodules and X (How do submodules and worktrees interact,\nshould they?, Which functions need support for submodules, e.g. checkout,\nbranch,  grep, etc...? Are we interested in keeping a submodule its own\nlogical unit? Do we want to have dedicated plumbing commands for\nsubmodules?)\n\n... would be my line of talk,\n\nStefan\n"},{"id":"310662","messageId":"20170201212825.advj7f3ucnfbspbj@sigill.intra.peff.net","threadId":"45005","inReplyTo":"xmqqh94dpyzs.fsf@gitster.mtv.corp.google.com","subject":"Re: [ANNOUNCE] Git Merge Contributor Summit topic planning","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-02-01T21:28:26Z","receivedAt":"2017-02-01T21:28:36Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Feb 01, 2017 at 10:06:15AM -0800, Junio C Hamano wrote:\n\n> > If you _can_ do that latter part, and you take \"I only care about\n> > resumability\" to the simplest extreme, you'd probably end up with a\n> > protocol more like:\n> >\n> >   Client: I need a packfile with this want/have\n> >   Server: OK, here it is; its opaque id is XYZ.\n> >   ... connection interrupted ...\n> >   Client: It's me again. I have up to byte N of pack XYZ\n> >   Server: OK, resuming\n> >           [or: I don't have XYZ anymore; start from scratch]\n> >\n> > Then generating XYZ and generating that bundle are basically the same\n> > task.\n> \n> The above allows a simple and naive implementation of generating a\n> packstream and \"tee\"ing it to a spool file to be kept while sending\n> to the first client that asks XYZ.\n> \n> The story I heard from folks who run git servers at work for Android\n> and other projects, however, is that they rarely see two requests\n> with want/have that result in an identical XYZ, unless \"have\" is an\n> empty set (aka \"clone\").  In a busy repository, between two clone\n> requests relatively close together, somebody would be pushing, so\n> you'd need many XYZs in your spool even if you want to support only\n> the \"clone\" case.\n\nYeah, I agree a tag \"XYZ\" does not cover all cases, especially for\nfetches.\n\nWe do caching at GitHub based on the sha1(want+have+options) tag, and it\ndoes catch quite a lot of parallelism, but not all. It catches most\nclones, and many fetches that are done by \"thundering herds\" of similar\nclients.\n\nOne thing you could do with such a pure \"resume XYZ\" tag is to represent\nthe generated pack _without_ replicating the actual object bytes, but\ntake shortcuts by basing particular bits on the on-disk packfile. Just\nenough to serve a deterministic packfile for the same want/have bits.\n\nFor instance, if the server knew that XYZ meant\n\n  - send bytes m through n of packfile p, then...\n  \n  - send the object at position i of packfile p, as a delta against the\n    object at position j of packfile q\n\n  - ...and so on\n\nThen you could store very small \"instruction sheets\" for each XYZ that\nrely on the data in the packfiles. If those packfiles go away (e.g., due\nto a repack) that invalidates all of your current XYZ tags. That's OK as\nlong as this is an optimization, not a correctness requirement.\n\nI haven't actually built anything like this, though, so I don't have a\ncomplete language for the instruction sheets, nor numbers on how big\nthey would be for average cases.\n\n> So in the real life, I think that the exchange needs to be more\n> like this:\n> \n>     C: I need a packfile with this want/have\n>     ... C/S negotiate what \"have\"s are common ...\n>     S: Sorry, but our negitiation indicates that you are way too\n>        behind.  I'll send you a packfile that brings you up to a\n>        slightly older set of \"want\", so pretend that you asked for\n>        these slightly older \"want\"s instead.  The opaque id of that\n>        packfile is XYZ.  After getting XYZ, come back to me with\n>        your original set of \"want\"s.  You would give me more recent\n>        \"have\" in that request.  \n>     ... connection interrupted ...\n>     C: It's me again.  I have up to byte N of pack XYZ\n>     S: OK, resuming (or: I do not have it anymore, start from scratch)\n>     ... after 0 or more iterations C fully receives and digests XYZ ...\n> \n> and then the above will iterate until the server does not have to\n> say \"Sorry but you are way too behind\" and returns a packfile\n> without having to tweak the \"want\".\n\nYes, I think that is a reasonable variant. The client knows about\nseeding, but the XYZ conversation continues to happen inside the git\nprotocol. So it loses flexibility versus a true CDN redirection, but it\nwould \"just work\" when the server/client both understand the feature,\nwithout the server admin having to set up a separate bundle-over-http\ninfrastructure.\n\n-Peff\n"},{"id":"310663","messageId":"xmqqd1f1oar5.fsf@gitster.mtv.corp.google.com","threadId":"45005","inReplyTo":"20170201212825.advj7f3ucnfbspbj@sigill.intra.peff.net","subject":"Re: [ANNOUNCE] Git Merge Contributor Summit topic planning","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-02-01T21:35:10Z","receivedAt":"2017-02-01T21:35:18Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> For instance, if the server knew that XYZ meant\n>\n>   - send bytes m through n of packfile p, then...\n>   \n>   - send the object at position i of packfile p, as a delta against the\n>     object at position j of packfile q\n>\n>   - ...and so on\n>\n> Then you could store very small \"instruction sheets\" for each XYZ that\n> rely on the data in the packfiles. If those packfiles go away (e.g., due\n> to a repack) that invalidates all of your current XYZ tags. That's OK as\n> long as this is an optimization, not a correctness requirement.\n\nYes.  You can play optimization games.\n\n>> So in the real life, I think that the exchange needs to be more\n>> like this:\n>> \n>>     C: I need a packfile with this want/have\n>>     ... C/S negotiate what \"have\"s are common ...\n>>     S: Sorry, but our negitiation indicates that you are way too\n>>        behind.  I'll send you a packfile that brings you up to a\n>>        slightly older set of \"want\", so pretend that you asked for\n>>        these slightly older \"want\"s instead.  The opaque id of that\n>>        packfile is XYZ.  After getting XYZ, come back to me with\n>>        your original set of \"want\"s.  You would give me more recent\n>>        \"have\" in that request.  \n>>     ... connection interrupted ...\n>>     C: It's me again.  I have up to byte N of pack XYZ\n>>     S: OK, resuming (or: I do not have it anymore, start from scratch)\n>>     ... after 0 or more iterations C fully receives and digests XYZ ...\n>> \n>> and then the above will iterate until the server does not have to\n>> say \"Sorry but you are way too behind\" and returns a packfile\n>> without having to tweak the \"want\".\n>\n> Yes, I think that is a reasonable variant. The client knows about\n> seeding, but the XYZ conversation continues to happen inside the git\n> protocol. So it loses flexibility versus a true CDN redirection, but it\n> would \"just work\" when the server/client both understand the feature,\n> without the server admin having to set up a separate bundle-over-http\n> infrastructure.\n\nYou can also do a CDN offline as a natural extension.  When the\nserver says \"Sorry, you are way too behind.\", the above example\ntells \"I'll update you to a slightly stale version first\" to the\nclient.  An natural extension could say \"Go update yourself to a\nslightly stale version first by grabbing that bundle over there.\"\n\nBut I agree that doing everything in-line may be a logical and\nsimpler first step to get there.\n"}]}