{"thread":{"id":"46222","subject":"Monitoring a repository for changes","startedAt":"2017-06-21T14:27:47Z","lastAt":"2017-06-21T22:37:02Z","messageCount":9,"participants":["Tim Hutt","Ævar Arnfjörð Bjarmason","Jeff King","Eric Wong","Stefan Beller","Jonathan Nieder"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"322810","messageId":"CAKuVd4Bdhreu6wrwDp1Bia56Db=TXYMQcitsznaRM+QcYVzJnQ@mail.gmail.com","threadId":"46222","inReplyTo":null,"subject":"Monitoring a repository for changes","fromName":"Tim Hutt","fromEmail":"tdhutt@gmail.com","sentAt":"2017-06-21T14:27:17Z","receivedAt":"2017-06-21T14:27:47Z","isPatch":false,"sender":{"key":"tdhutt@gmail.com","avatar":null},"body":"Hi,\n\nCurrently if you want to monitor a repository for changes there are\nthree options:\n\n* Polling - run a script to check for updates every 60 seconds.\n* Server side hooks\n* Web hooks (on Github, Bitbucket etc.)\n\nUnfortunately for many (most?) cases server-side hooks and web hooks\nare not suitable. They require you to both have admin access to the\nrepo and have a public server available to push updates to. That is a\nhuge faff when all I want to do is run some local code when a repo is\nupdated (e.g. play a sound).\n\nCurrently people resort to polling\n(https://stackoverflow.com/a/5199111/265521) which is just ugly. I\nwould like to propose that there should be a forth option that uses a\npersistent connection to monitor the repo. It would be used something\nlike this:\n\n    git watch https://github.com/git/git.git\n\nor\n\n    git watch git@github.com:git/git.git\n\nIt would then print simple messages to stdout. The complexity of what\nit prints is up for debate, - it could be something as simple as\n\"PUSH\\n\", or it could include more information, e.g. JSON-encoded\ninformation about the commits. I'd be happy with just \"PUSH\\n\" though.\n\nIn terms of implementation, the HTTP transport could use Server-Sent\nEvents, and the SSH transport can pretty much do whatever so that\nshould be easy.\n\nThoughts?\n\nTim\n"},{"id":"322813","messageId":"87efud8jrn.fsf@gmail.com","threadId":"46222","inReplyTo":"CAKuVd4Bdhreu6wrwDp1Bia56Db=TXYMQcitsznaRM+QcYVzJnQ@mail.gmail.com","subject":"Re: Monitoring a repository for changes","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-06-21T15:04:12Z","receivedAt":"2017-06-21T15:04:21Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jun 21 2017, Tim Hutt jotted:\n\n> Hi,\n>\n> Currently if you want to monitor a repository for changes there are\n> three options:\n>\n> * Polling - run a script to check for updates every 60 seconds.\n> * Server side hooks\n> * Web hooks (on Github, Bitbucket etc.)\n>\n> Unfortunately for many (most?) cases server-side hooks and web hooks\n> are not suitable. They require you to both have admin access to the\n> repo and have a public server available to push updates to. That is a\n> huge faff when all I want to do is run some local code when a repo is\n> updated (e.g. play a sound).\n>\n> Currently people resort to polling\n> (https://stackoverflow.com/a/5199111/265521) which is just ugly. I\n> would like to propose that there should be a forth option that uses a\n> persistent connection to monitor the repo. It would be used something\n> like this:\n>\n>     git watch https://github.com/git/git.git\n>\n> or\n>\n>     git watch git@github.com:git/git.git\n>\n> It would then print simple messages to stdout. The complexity of what\n> it prints is up for debate, - it could be something as simple as\n> \"PUSH\\n\", or it could include more information, e.g. JSON-encoded\n> information about the commits. I'd be happy with just \"PUSH\\n\" though.\n\nInsofar as this could be implemented in some standard way in Git it's\nlikely to have a large overlap with the \"protocol v2\" that keeps coming\nup here on-list. You might want to search for past threads discussing\nthat.\n\n> In terms of implementation, the HTTP transport could use Server-Sent\n> Events, and the SSH transport can pretty much do whatever so that\n> should be easy.\n\nIn case you didn't know, any of the non-trivially sized git hosting\nproviders (e.g. github, gitlab) provide you access over ssh, but you\ncan't just run any arbitrary command, it's a tiny set of whitelisted\ncommands. See the \"git-shell\" manual page (github doesn't use that exact\nsoftware, but something similar).\n\nBut overall, it would be nice to have some rationale for this approach\nother than that you think polling is ugly. There's a lot of advantages\nto polling for something you don't need near-instantly, e.g. imagine how\nmany active connections a site like GitHub would need to handle if\nsomething like this became widely used, that's in a lot of ways harder\nto scale and load balance than just having clients that poll something\nthat's trivially cached as static content.\n"},{"id":"322852","messageId":"20170621194425.vcu6qdmoddwpi3ht@sigill.intra.peff.net","threadId":"46222","inReplyTo":"87efud8jrn.fsf@gmail.com","subject":"Re: Monitoring a repository for changes","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-06-21T19:44:25Z","receivedAt":"2017-06-21T19:44:31Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jun 21, 2017 at 05:04:12PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> > In terms of implementation, the HTTP transport could use Server-Sent\n> > Events, and the SSH transport can pretty much do whatever so that\n> > should be easy.\n> \n> In case you didn't know, any of the non-trivially sized git hosting\n> providers (e.g. github, gitlab) provide you access over ssh, but you\n> can't just run any arbitrary command, it's a tiny set of whitelisted\n> commands. See the \"git-shell\" manual page (github doesn't use that exact\n> software, but something similar).\n\nThese days you don't even hit the actual fileservers with ssh at all.\nWe terminate all of the protocols (http, git://, and ssh) at a proxy\nlayer that kicks off git commands in the actual repositories using a\nseparate protocol. The ssh handshakes were a huge performance\nbottleneck, so by doing it that way we can scale out the front-end tier\nindependently of the repository storage (and of course it also provides\na convenient layer for mapping user visible repository names into\nsharded paths).\n\nNot to take away from your point. Just a little bit of trivia.\n\n> But overall, it would be nice to have some rationale for this approach\n> other than that you think polling is ugly. There's a lot of advantages\n> to polling for something you don't need near-instantly, e.g. imagine how\n> many active connections a site like GitHub would need to handle if\n> something like this became widely used, that's in a lot of ways harder\n> to scale and load balance than just having clients that poll something\n> that's trivially cached as static content.\n\nYeah. The naive way to implement this would be to have the client\nconnect and receive the ref advertisement. And then when it's a noop\n(nothing to fetch), instead of saying \"I want these objects\", say\n\"Please pause until one or more refs change\". But I don't think we'd\nwant to leave actual upload-pack processes sitting paused on the server.\nTheir memory usage is too high.\n\nFor this kind of \"long polling\" we have a separate front-end tier with a\ndaemon that keeps the per-client cost very low. We could possibly wedge\nthat into our proxy layer, but the system would be a lot simpler and\nmore flexible if this were done separately from the actual git protocol.\nE.g., if an HTTP endpoint were defined that paused and returned data\nonly when a particular repository's refs were updated.\n\nAnother option is to keep polling, but just make noop fetches a lot\ncheaper. The ref advertisement on some repositories can get into the\nmegabytes. I'd love to see protocol extensions for:\n\n  1. The client asking only for bits of the ref namespace they care\n     about. I have some preliminary patches for this, but I really need\n     to polish them.\n\n  2. Something ETag-ish where the client can say \"I already saw state X,\n     do you have updates?\" Even just handling \"no, no updates\" (like an\n     ETag) would be a big benefit. Bonus points if it can say \"since\n     state X, these are the changes; you are now at state Y\".\n\nThe sticking point on both is that the client needs to speak before the\nref advertisement begins, which is why we have to deal with the protocol\nv2 headache.\n\n-Peff\n"},{"id":"322854","messageId":"20170621195252.GA31582@starla","threadId":"46222","inReplyTo":"87efud8jrn.fsf@gmail.com","subject":"Re: Monitoring a repository for changes","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2017-06-21T19:52:52Z","receivedAt":"2017-06-21T19:52:57Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:\n> On Wed, Jun 21 2017, Tim Hutt jotted:\n> \n> > Hi,\n> >\n> > Currently if you want to monitor a repository for changes there are\n> > three options:\n> >\n> > * Polling - run a script to check for updates every 60 seconds.\n> > * Server side hooks\n> > * Web hooks (on Github, Bitbucket etc.)\n> >\n> > Unfortunately for many (most?) cases server-side hooks and web hooks\n> > are not suitable. They require you to both have admin access to the\n> > repo and have a public server available to push updates to. That is a\n> > huge faff when all I want to do is run some local code when a repo is\n> > updated (e.g. play a sound).\n\nYeah, it kinda sucks that way.\n\nCurrently, for one of my public-inbox mirrors which has ssh\naccess to the primary server on public-inbox.org, I have:\n\n\t#!/bin/sh\n\twhile true\n\tdo\n\t\t# GNU tail(1) uses inotify to avoid polling on Linux\n\t\tssh public-inbox.org tail -F /path/to/git-vger.git/info/refs | \\\n\t\t\t\twhile read sha1 ref\n\t\tdo\n\t\t\tfor GIT_DIR in git-vger.git\n\t\t\tdo\n\t\t\t\texport GIT_DIR\n\t\t\t\tgit fetch || continue\n\t\t\t\tgit update-server-info\n\t\t\t\tpublic-inbox-index # update Xapian index\n\t\t\tdone\n\t\tdone\n\tdone\n\nIt's not perfect as it requires multiple processes on the\nserver, but it's better than polling for my limited use.\n\n> > Currently people resort to polling\n> > (https://stackoverflow.com/a/5199111/265521) which is just ugly. I\n> > would like to propose that there should be a forth option that uses a\n> > persistent connection to monitor the repo. It would be used something\n> > like this:\n> >\n> >     git watch https://github.com/git/git.git\n> >\n> > or\n> >\n> >     git watch git@github.com:git/git.git\n> >\n> > It would then print simple messages to stdout. The complexity of what\n> > it prints is up for debate, - it could be something as simple as\n> > \"PUSH\\n\", or it could include more information, e.g. JSON-encoded\n> > information about the commits. I'd be happy with just \"PUSH\\n\" though.\n> \n> Insofar as this could be implemented in some standard way in Git it's\n> likely to have a large overlap with the \"protocol v2\" that keeps coming\n> up here on-list. You might want to search for past threads discussing\n> that.\n\nYeah, it hasn't been a priority for me, either...\n\n> > In terms of implementation, the HTTP transport could use Server-Sent\n> > Events, and the SSH transport can pretty much do whatever so that\n> > should be easy.\n> \n> In case you didn't know, any of the non-trivially sized git hosting\n> providers (e.g. github, gitlab) provide you access over ssh, but you\n> can't just run any arbitrary command, it's a tiny set of whitelisted\n> commands. See the \"git-shell\" manual page (github doesn't use that exact\n> software, but something similar).\n> \n> But overall, it would be nice to have some rationale for this approach\n> other than that you think polling is ugly. There's a lot of advantages\n> to polling for something you don't need near-instantly, e.g. imagine how\n> many active connections a site like GitHub would need to handle if\n> something like this became widely used, that's in a lot of ways harder\n> to scale and load balance than just having clients that poll something\n> that's trivially cached as static content.\n\nPolling becomes more expensive with TLS and high-latency\nconnections, and also increases power consumption if done\nfrequently for redundancy purposes.\n\nI've long wanted to do something better to allow others to keep\npublic-inbox mirrors up-to-date.  Having only 64-128 bytes of\noverhead per userspace per-connection should be totally doable\nbased on my experience working on cmogstored; at which point\nport exhaustion will become the limiting factor (or TLS overhead\nfor HTTPS).\n\nBut perhaps a cheaper option might be the traditional email/IRC\nnotification and having a client-side process watch for that\nbefore fetching.\n"},{"id":"322855","messageId":"CAGZ79kau0cxRw6_bFKw6-wc2J9fBPPjRfpRhSxD+BGi2usvNpg@mail.gmail.com","threadId":"46222","inReplyTo":"20170621194425.vcu6qdmoddwpi3ht@sigill.intra.peff.net","subject":"Re: Monitoring a repository for changes","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-06-21T19:55:05Z","receivedAt":"2017-06-21T19:55:16Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Jun 21, 2017 at 12:44 PM, Jeff King <peff@peff.net> wrote:\n>\n> Yeah. The naive way to implement this would be to have the client\n> connect and receive the ref advertisement. And then when it's a noop\n> (nothing to fetch), instead of saying \"I want these objects\", say\n> \"Please pause until one or more refs change\". But I don't think we'd\n> want to leave actual upload-pack processes sitting paused on the server.\n> Their memory usage is too high.\n\nhttps://git.eclipse.org/r/#/c/6587/\n\nJGit has had its experiments with some standing connection and then\nhaving some sort of Pub/Sub system. AFAICT it did not go anywhere\nbecause of the number of connections (even if you optimize for\nthe serverside, such that each connection is just the cost of a java\nthread and a file descriptor).\n\n>\n> The sticking point on both is that the client needs to speak before the\n> ref advertisement begins, which is why we have to deal with the protocol\n> v2 headache.\n\nI would not call it headache, but large project that is not to be tackled\nby one person alone. ;)\n"},{"id":"322867","messageId":"20170621211908.GF60603@aiede.mtv.corp.google.com","threadId":"46222","inReplyTo":"CAKuVd4Bdhreu6wrwDp1Bia56Db=TXYMQcitsznaRM+QcYVzJnQ@mail.gmail.com","subject":"Re: Monitoring a repository for changes","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-06-21T21:19:08Z","receivedAt":"2017-06-21T21:19:23Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nTim Hutt wrote:\n\n> Currently if you want to monitor a repository for changes there are\n> three options:\n>\n> * Polling - run a script to check for updates every 60 seconds.\n> * Server side hooks\n> * Web hooks (on Github, Bitbucket etc.)\n>\n> Unfortunately for many (most?) cases server-side hooks and web hooks\n> are not suitable. They require you to both have admin access to the\n> repo and have a public server available to push updates to. That is a\n> huge faff when all I want to do is run some local code when a repo is\n> updated (e.g. play a sound).\n\nOn the polling side, it is possible to improve things a little:\nhttps://www.kernel.org/mirroring-kernelorg-repositories.html\nhttps://github.com/mricon/grokmirror\n\nA hanging GET or websocket is more client-friendly but more expensive\nserver-side.  That doesn't rule out making it happen on some servers\nif someone does the work.  If I understand correctly then this\narchitecture tends to lead to centralization --- a small number of\nservices providing notifications pushed from multiple sources, as with\nhttps://developers.google.com/web/fundamentals/engage-and-retain/push-notifications/how-push-works\n\nIf someone wants to try adding something like grokmirror (which\ndescribes the state of multiple repositories, amortizing the\nper-request costs) to git, especially if it supports something\netag-like as Jeff King suggested, then I would be interested.\n\nThanks and hope that helps,\nJonathan\n"},{"id":"322885","messageId":"87a85180om.fsf@gmail.com","threadId":"46222","inReplyTo":"20170621195252.GA31582@starla","subject":"Re: Monitoring a repository for changes","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-06-21T21:56:25Z","receivedAt":"2017-06-21T21:56:32Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jun 21 2017, Eric Wong jotted:\n\n> Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:\n>> On Wed, Jun 21 2017, Tim Hutt jotted:\n>>\n>> > Hi,\n>> >\n>> > Currently if you want to monitor a repository for changes there are\n>> > three options:\n>> >\n>> > * Polling - run a script to check for updates every 60 seconds.\n>> > * Server side hooks\n>> > * Web hooks (on Github, Bitbucket etc.)\n>> >\n>> > Unfortunately for many (most?) cases server-side hooks and web hooks\n>> > are not suitable. They require you to both have admin access to the\n>> > repo and have a public server available to push updates to. That is a\n>> > huge faff when all I want to do is run some local code when a repo is\n>> > updated (e.g. play a sound).\n>\n> Yeah, it kinda sucks that way.\n>\n> Currently, for one of my public-inbox mirrors which has ssh\n> access to the primary server on public-inbox.org, I have:\n>\n> \t#!/bin/sh\n> \twhile true\n> \tdo\n> \t\t# GNU tail(1) uses inotify to avoid polling on Linux\n> \t\tssh public-inbox.org tail -F /path/to/git-vger.git/info/refs | \\\n> \t\t\t\twhile read sha1 ref\n> \t\tdo\n> \t\t\tfor GIT_DIR in git-vger.git\n> \t\t\tdo\n> \t\t\t\texport GIT_DIR\n> \t\t\t\tgit fetch || continue\n> \t\t\t\tgit update-server-info\n> \t\t\t\tpublic-inbox-index # update Xapian index\n> \t\t\tdone\n> \t\tdone\n> \tdone\n>\n> It's not perfect as it requires multiple processes on the\n> server, but it's better than polling for my limited use.\n>\n>> > Currently people resort to polling\n>> > (https://stackoverflow.com/a/5199111/265521) which is just ugly. I\n>> > would like to propose that there should be a forth option that uses a\n>> > persistent connection to monitor the repo. It would be used something\n>> > like this:\n>> >\n>> >     git watch https://github.com/git/git.git\n>> >\n>> > or\n>> >\n>> >     git watch git@github.com:git/git.git\n>> >\n>> > It would then print simple messages to stdout. The complexity of what\n>> > it prints is up for debate, - it could be something as simple as\n>> > \"PUSH\\n\", or it could include more information, e.g. JSON-encoded\n>> > information about the commits. I'd be happy with just \"PUSH\\n\" though.\n>>\n>> Insofar as this could be implemented in some standard way in Git it's\n>> likely to have a large overlap with the \"protocol v2\" that keeps coming\n>> up here on-list. You might want to search for past threads discussing\n>> that.\n>\n> Yeah, it hasn't been a priority for me, either...\n>\n>> > In terms of implementation, the HTTP transport could use Server-Sent\n>> > Events, and the SSH transport can pretty much do whatever so that\n>> > should be easy.\n>>\n>> In case you didn't know, any of the non-trivially sized git hosting\n>> providers (e.g. github, gitlab) provide you access over ssh, but you\n>> can't just run any arbitrary command, it's a tiny set of whitelisted\n>> commands. See the \"git-shell\" manual page (github doesn't use that exact\n>> software, but something similar).\n>>\n>> But overall, it would be nice to have some rationale for this approach\n>> other than that you think polling is ugly. There's a lot of advantages\n>> to polling for something you don't need near-instantly, e.g. imagine how\n>> many active connections a site like GitHub would need to handle if\n>> something like this became widely used, that's in a lot of ways harder\n>> to scale and load balance than just having clients that poll something\n>> that's trivially cached as static content.\n>\n> Polling becomes more expensive with TLS and high-latency\n> connections, and also increases power consumption if done\n> frequently for redundancy purposes.\n>\n> I've long wanted to do something better to allow others to keep\n> public-inbox mirrors up-to-date.  Having only 64-128 bytes of\n> overhead per userspace per-connection should be totally doable\n> based on my experience working on cmogstored; at which point\n> port exhaustion will become the limiting factor (or TLS overhead\n> for HTTPS).\n\nCome to think of it I should probably have asked you about this, but I\nhave a one-liner running that polls every 5 minutes, but will stop if I\nhaven't changed my git.git in a day:\n\n    while true; do if test $(find ~/g/git -type f -mmin -1440 | wc -l) -gt 0; then git pull; else echo too old; fi ; date ; sleep 300; done\n\n> But perhaps a cheaper option might be the traditional email/IRC\n> notification and having a client-side process watch for that\n> before fetching.\n\nIf there was a IRC channel with this info I could/would use that,\ngetting it via E-Mail would just get me into the same problem\npublic-inbox is currently solving for me, i.e. I might as well keep the\ngit ML up-to-date on that machine if I'm going to otherwise need to\nsubscribe to a \"hey there's a new message on the git ML\" list :)\n"},{"id":"322887","messageId":"20170621222013.GA24176@dcvr","threadId":"46222","inReplyTo":"87a85180om.fsf@gmail.com","subject":"Re: Monitoring a repository for changes","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2017-06-21T22:20:13Z","receivedAt":"2017-06-21T22:20:19Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:\n> On Wed, Jun 21 2017, Eric Wong jotted:\n> > I've long wanted to do something better to allow others to keep\n> > public-inbox mirrors up-to-date.  Having only 64-128 bytes of\n> > overhead per userspace per-connection should be totally doable\n> > based on my experience working on cmogstored; at which point\n> > port exhaustion will become the limiting factor (or TLS overhead\n> > for HTTPS).\n> \n> Come to think of it I should probably have asked you about this, but I\n> have a one-liner running that polls every 5 minutes, but will stop if I\n> haven't changed my git.git in a day:\n> \n>     while true; do if test $(find ~/g/git -type f -mmin -1440 | wc -l) -gt 0; then git pull; else echo too old; fi ; date ; sleep 300; done\n\nPolling https://public-inbox.org/git ?  no need to stop it,\nevery 5 seconds is fine if you're not worried about power\nconsumption on your end :)\n\n> > But perhaps a cheaper option might be the traditional email/IRC\n> > notification and having a client-side process watch for that\n> > before fetching.\n> \n> If there was a IRC channel with this info I could/would use that,\n> getting it via E-Mail would just get me into the same problem\n> public-inbox is currently solving for me, i.e. I might as well keep the\n> git ML up-to-date on that machine if I'm going to otherwise need to\n> subscribe to a \"hey there's a new message on the git ML\" list :)\n\nThe IRC server would have the same scalability problems faced by\nmaintaining persistent connections to git-daemon or HTTP\nservers, however.  And, yes, email does seem redundant, and\nmodern header sizes (with DKIM, etc) are gigantic; but\nconnection lifetime and concurrency is manageable to the server\neven if not instantaneous.\n\nI also considered having clients setup a listener of some sort,\n(possibly using UDP) but that would have all the problems with\ngit:// + firewalls.\n"},{"id":"322889","messageId":"20170621223657.GA1141@dcvr","threadId":"46222","inReplyTo":"20170621222013.GA24176@dcvr","subject":"Re: Monitoring a repository for changes","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2017-06-21T22:36:57Z","receivedAt":"2017-06-21T22:37:02Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Eric Wong <e@80x24.org> wrote:\n> And, yes, email does seem redundant, and\n> modern header sizes (with DKIM, etc) are gigantic; but\n> connection lifetime and concurrency is manageable to the server\n> even if not instantaneous.\n\nI should add that any email notification message should be\nsignificantly shorter than a normal message going to the list;\npossibly just a parsable subject line and empty body.\n"}]}