{"thread":{"id":"50126","subject":"\"IMAP IDLE\"-like long-polling \"git fetch\"","startedAt":"2018-12-29T04:35:14Z","lastAt":"2019-05-02T09:42:57Z","messageCount":8,"participants":["Eric Wong","Konstantin Ryabitsev","Stefan Beller","Ævar Arnfjörð Bjarmason"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"365938","messageId":"20181229035621.cwjpknctq3rjnlhs@dcvr","threadId":"50126","inReplyTo":"20181229034342.11543-1-e@80x24.org","subject":"\"IMAP IDLE\"-like long-polling \"git fetch\"","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2018-12-29T03:56:21Z","receivedAt":"2018-12-29T04:35:14Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Hey all, I just added this to the TODO file for public-inbox[1] but\nobviously it's intended for git.git (meta@public-inbox cc-ed):\n\n> +* Contribute something like IMAP IDLE for \"git fetch\".\n> + Inboxes (and any git repos) can be kept up-to-date without\n> + relying on polling.\n\nI would've thought somebody had done this by now, but I guess\nit's dependent on a bunch of things (TLS layer nowadays, maybe\nHTTP/2), so git-daemon support alone wouldn't cut it...\n\nAnyways, until this is implemented, feel free to continue\nhammering a way on https://public-inbox.org/git/ with frequent\n\"git fetch\". I write C10K servers in my sleep -_-\n\n\n[1] https://public-inbox.org/meta/20181229034342.11543-1-e@80x24.org/\n"},{"id":"365939","messageId":"20181229043858.GA28509@pure.paranoia.local","threadId":"50126","inReplyTo":"20181229035621.cwjpknctq3rjnlhs@dcvr","subject":"Re: \"IMAP IDLE\"-like long-polling \"git fetch\"","fromName":"Konstantin Ryabitsev","fromEmail":"konstantin@linuxfoundation.org","sentAt":"2018-12-29T04:38:58Z","receivedAt":"2018-12-29T04:39:05Z","isPatch":false,"sender":{"key":"konstantin@linuxfoundation.org","avatar":"https://gravatar.com/avatar/7cb8827c6de56e1bd2dea16508c6708aa43feed3bf3813bcdacecdf96ceadd79?d=mp&s=160"},"body":"On Sat, Dec 29, 2018 at 03:56:21AM +0000, Eric Wong wrote:\n> Hey all, I just added this to the TODO file for public-inbox[1] but\n> obviously it's intended for git.git (meta@public-inbox cc-ed):\n> \n> > +* Contribute something like IMAP IDLE for \"git fetch\".\n> > + Inboxes (and any git repos) can be kept up-to-date without\n> > + relying on polling.\n> \n> I would've thought somebody had done this by now, but I guess\n> it's dependent on a bunch of things (TLS layer nowadays, maybe\n> HTTP/2), so git-daemon support alone wouldn't cut it...\n\nPolling is not all bad, especially for large repository collections. I'm\nnot sure you want to \"idle\" individual repositories when there's\nthousands of them. We ended up writing grokmirror for replicating\nrepo collections using manifest files.\n\n> Anyways, until this is implemented, feel free to continue\n> hammering a way on https://public-inbox.org/git/ with frequent\n> \"git fetch\". I write C10K servers in my sleep -_-\n\nThe archive is also mirrored at\nhttps://git.kernel.org/pub/scm/public-inbox/vger.kernel.org/git.git, and\nalso on kernel.googlesource.com.\n\n-K\n"},{"id":"365942","messageId":"20181229061352.GA5935@dcvr","threadId":"50126","inReplyTo":"20181229043858.GA28509@pure.paranoia.local","subject":"Re: \"IMAP IDLE\"-like long-polling \"git fetch\"","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2018-12-29T06:13:52Z","receivedAt":"2018-12-29T06:16:14Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Konstantin Ryabitsev wrote:\n> On Sat, Dec 29, 2018 at 03:56:21AM +0000, Eric Wong wrote:\n> > Hey all, I just added this to the TODO file for public-inbox[1] but\n> > obviously it's intended for git.git (meta@public-inbox cc-ed):\n> > \n> > > +* Contribute something like IMAP IDLE for \"git fetch\".\n> > > + Inboxes (and any git repos) can be kept up-to-date without\n> > > + relying on polling.\n> > \n> > I would've thought somebody had done this by now, but I guess\n> > it's dependent on a bunch of things (TLS layer nowadays, maybe\n> > HTTP/2), so git-daemon support alone wouldn't cut it...\n> \n> Polling is not all bad, especially for large repository collections. I'm\n> not sure you want to \"idle\" individual repositories when there's\n> thousands of them. We ended up writing grokmirror for replicating\n> repo collections using manifest files.\n\nI wasn't intending it for giant sites like korg, but for\nindividual hackers on their workstations tracking a handful of\nprojects they follow.\n\nThe cost for a hackers' machine would be the same as the current\nsituation where developers idle on IRC channels for the projects\nthey're involved in.\n\n> > Anyways, until this is implemented, feel free to continue\n> > hammering a way on https://public-inbox.org/git/ with frequent\n> > \"git fetch\". I write C10K servers in my sleep -_-\n> \n> The archive is also mirrored at\n> https://git.kernel.org/pub/scm/public-inbox/vger.kernel.org/git.git, and\n> also on kernel.googlesource.com.\n\nNow, I'm wondering if you can make a v2 public-inbox mirror of\ngit@vger and run it on lore. Converting public-inbox.org/git to\nv2 would break things for everybody fetching, right now :<\n"},{"id":"366454","messageId":"CAGZ79kb9Tbnxe1mSnxpqT_FO6Gdi6wxd-r2YarHXRF1sVRyxLA@mail.gmail.com","threadId":"50126","inReplyTo":"20181229043858.GA28509@pure.paranoia.local","subject":"Re: \"IMAP IDLE\"-like long-polling \"git fetch\"","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2019-01-09T22:27:25Z","receivedAt":"2019-01-09T22:27:39Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Dec 28, 2018 at 8:39 PM Konstantin Ryabitsev\n wrote:\n>\n> On Sat, Dec 29, 2018 at 03:56:21AM +0000, Eric Wong wrote:\n> > Hey all, I just added this to the TODO file for public-inbox[1] but\n> > obviously it's intended for git.git (meta@public-inbox cc-ed):\n> >\n> > > +* Contribute something like IMAP IDLE for \"git fetch\".\n> > > + Inboxes (and any git repos) can be kept up-to-date without\n> > > + relying on polling.\n> >\n> > I would've thought somebody had done this by now, but I guess\n> > it's dependent on a bunch of things (TLS layer nowadays, maybe\n> > HTTP/2), so git-daemon support alone wouldn't cut it...\n>\n> Polling is not all bad, especially for large repository collections.\n\nI disagree with that statement.\n\nIIRC, More than half the bandwidth of Googles git servers are used\nfor ls-remote calls (i.e. polling a lot of repos, most of them did *not*\nchange, by build bots which are really eager to try again after a minute).\n\nThat is why we use a superproject, with all other repositories as\na submodule for polling, as that would slash the ls-remote traffic\napproximately by the number of repositories.\n\nThere was an attempt in JGit to support this type of communication\nof long polling at\nhttps://git.eclipse.org/r/plugins/gitiles/jgit/jgit/+/2adc572628f9382ace5fbd791325dc64f7c968d3\nbut not a whole lot is left over in JGit as it was refactored at least\nonce again.\n\nIIRC the issues where in the lack of protocol definition that made it\nusable for a wider audience.\n"},{"id":"366458","messageId":"20190109224909.GC8258@pure.paranoia.local","threadId":"50126","inReplyTo":"CAGZ79kb9Tbnxe1mSnxpqT_FO6Gdi6wxd-r2YarHXRF1sVRyxLA@mail.gmail.com","subject":"Re: \"IMAP IDLE\"-like long-polling \"git fetch\"","fromName":"Konstantin Ryabitsev","fromEmail":"konstantin@linuxfoundation.org","sentAt":"2019-01-09T22:49:09Z","receivedAt":"2019-01-09T22:49:15Z","isPatch":false,"sender":{"key":"konstantin@linuxfoundation.org","avatar":"https://gravatar.com/avatar/7cb8827c6de56e1bd2dea16508c6708aa43feed3bf3813bcdacecdf96ceadd79?d=mp&s=160"},"body":"On Wed, Jan 09, 2019 at 02:27:25PM -0800, Stefan Beller wrote:\n> > > I would've thought somebody had done this by now, but I guess\n> > > it's dependent on a bunch of things (TLS layer nowadays, maybe\n> > > HTTP/2), so git-daemon support alone wouldn't cut it...\n> >\n> > Polling is not all bad, especially for large repository collections.\n> \n> I disagree with that statement.\n> \n> IIRC, More than half the bandwidth of Googles git servers are used\n> for ls-remote calls (i.e. polling a lot of repos, most of them did *not*\n> change, by build bots which are really eager to try again after a minute).\n\nOh, that's not the kind of polling I meant -- we monitor a single\nmanifest file containing the state of all repositories. It's a static\nfile served directly by any httpd daemon, and the only traffic is\nusually the \"not modified\" http header.\n\n-K\n"},{"id":"374805","messageId":"20190502085055.34kkll2deowat6il@dcvr","threadId":"50126","inReplyTo":"CAGZ79kb9Tbnxe1mSnxpqT_FO6Gdi6wxd-r2YarHXRF1sVRyxLA@mail.gmail.com","subject":"Re: \"IMAP IDLE\"-like long-polling \"git fetch\"","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2019-05-02T08:50:55Z","receivedAt":"2019-05-02T08:50:58Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Stefan Beller wrote:\n> IIRC, More than half the bandwidth of Googles git servers are used\n> for ls-remote calls (i.e. polling a lot of repos, most of them did *not*\n> change, by build bots which are really eager to try again after a minute).\n\nThinking back at that statement; I think polling can be\noptimized in git, at least.\n\nIIRC, your repos have lots of refs; right?\n(which is why it's a bandwidth problem)\n\nSince info/refs is a static file (hopefully updated by a\npost-update hook), the smart client can make an HTTP request\nto check If-Modified-Since: to avoid the big response.\n\nThe client would need to cache the mtime of the last requested\nrefs file; somewhere.\n\nIOW, do refs negotiation the \"dumb\" way; since it's no better\nthan the smart way, really. Keep doing object transfers the\nsmart way.\n\nDuring the initial clone, smart servers could probably\nhave a header informing clients that their info/refs\nis up-to-date and clients can do dumb refs negotiation.\n"},{"id":"374806","messageId":"87ftpxqkji.fsf@evledraar.gmail.com","threadId":"50126","inReplyTo":"20190502085055.34kkll2deowat6il@dcvr","subject":"Re: \"IMAP IDLE\"-like long-polling \"git fetch\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-05-02T09:21:05Z","receivedAt":"2019-05-02T09:21:10Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, May 02 2019, Eric Wong wrote:\n\n> Stefan Beller wrote:\n>> IIRC, More than half the bandwidth of Googles git servers are used\n>> for ls-remote calls (i.e. polling a lot of repos, most of them did *not*\n>> change, by build bots which are really eager to try again after a minute).\n>\n> Thinking back at that statement; I think polling can be\n> optimized in git, at least.\n>\n> IIRC, your repos have lots of refs; right?\n> (which is why it's a bandwidth problem)\n>\n> Since info/refs is a static file (hopefully updated by a\n> post-update hook), the smart client can make an HTTP request\n> to check If-Modified-Since: to avoid the big response.\n>\n> The client would need to cache the mtime of the last requested\n> refs file; somewhere.\n>\n> IOW, do refs negotiation the \"dumb\" way; since it's no better\n> than the smart way, really. Keep doing object transfers the\n> smart way.\n>\n> During the initial clone, smart servers could probably\n> have a header informing clients that their info/refs\n> is up-to-date and clients can do dumb refs negotiation.\n\nDoing this with If-Modified-Since sounds like an easier drop-in\nreplacement (just needs a client change), but I wonder if ETag isn't a\nbetter fit for this.\n\nI.e. we'd document some convention where the ETag is a hash of the refs\nthe client expects to be advertised in some format, it then sends that\nto the server.\n\nThat allows the same thing without anyone keeping more state than they\nkeep now in their local ref store\n\nOn the fancier side I think bloom filters are something that's been\ndiscussed (and I believe someone (Twitter?) had such an internal patch),\ni.e. the client sends a bloom filter of refs they have, and the server\nadvertises things they don't know about yet (and due to how bloom\nfilters work, some things they *do* know about already but tripped up\nthe bloom filter...).\n"},{"id":"374807","messageId":"20190502094255.kbpzffokvdch63qg@dcvr","threadId":"50126","inReplyTo":"87ftpxqkji.fsf@evledraar.gmail.com","subject":"Re: \"IMAP IDLE\"-like long-polling \"git fetch\"","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2019-05-02T09:42:55Z","receivedAt":"2019-05-02T09:42:57Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, May 02 2019, Eric Wong wrote:\n> \n> > Stefan Beller wrote:\n> >> IIRC, More than half the bandwidth of Googles git servers are used\n> >> for ls-remote calls (i.e. polling a lot of repos, most of them did *not*\n> >> change, by build bots which are really eager to try again after a minute).\n> >\n> > Thinking back at that statement; I think polling can be\n> > optimized in git, at least.\n> >\n> > IIRC, your repos have lots of refs; right?\n> > (which is why it's a bandwidth problem)\n> >\n> > Since info/refs is a static file (hopefully updated by a\n> > post-update hook), the smart client can make an HTTP request\n> > to check If-Modified-Since: to avoid the big response.\n> >\n> > The client would need to cache the mtime of the last requested\n> > refs file; somewhere.\n> >\n> > IOW, do refs negotiation the \"dumb\" way; since it's no better\n> > than the smart way, really. Keep doing object transfers the\n> > smart way.\n> >\n> > During the initial clone, smart servers could probably\n> > have a header informing clients that their info/refs\n> > is up-to-date and clients can do dumb refs negotiation.\n> \n> Doing this with If-Modified-Since sounds like an easier drop-in\n> replacement (just needs a client change), but I wonder if ETag isn't a\n> better fit for this.\n\nETags overall could work.\n\n> I.e. we'd document some convention where the ETag is a hash of the refs\n> the client expects to be advertised in some format, it then sends that\n> to the server.\n\nBut I was hoping to avoid the overhead of spawning git-http-backend\nentirely. And there's no consistent way to configure ETags on\ndifferent static servers.\n\n> That allows the same thing without anyone keeping more state than they\n> keep now in their local ref store\n\nI think caching the remote info/refs is useful anyways in case\nthe user changes their fetch refspec, and it could speed up\ninvocations of \"git ls-remote\".\n\n> On the fancier side I think bloom filters are something that's been\n> discussed (and I believe someone (Twitter?) had such an internal patch),\n> i.e. the client sends a bloom filter of refs they have, and the server\n> advertises things they don't know about yet (and due to how bloom\n> filters work, some things they *do* know about already but tripped up\n> the bloom filter...).\n\nI'm not smart enough to understand such fancy things :)\n"}]}