Re: Monitoring a repository for changes

From: Eric Wong <e@80x24.org>
Date: Jun 21, 2017, 22:20 UTC
Message-ID: <20170621222013.GA24176@dcvr>
In-Reply-To: <87a85180om.fsf@gmail.com>

Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:

Show 13 quoted lines

> On Wed, Jun 21 2017, Eric Wong jotted:
> > I've long wanted to do something better to allow others to keep
> > public-inbox mirrors up-to-date.  Having only 64-128 bytes of
> > overhead per userspace per-connection should be totally doable
> > based on my experience working on cmogstored; at which point
> > port exhaustion will become the limiting factor (or TLS overhead
> > for HTTPS).
> 
> Come to think of it I should probably have asked you about this, but I
> have a one-liner running that polls every 5 minutes, but will stop if I
> haven't changed my git.git in a day:
> 
>     while true; do if test $(find ~/g/git -type f -mmin -1440 | wc -l) -gt 0; then git pull; else echo too old; fi ; date ; sleep 300; done

Polling https://public-inbox.org/git ? no need to stop it, every 5 seconds is fine if you're not worried about power consumption on your end :)

Show 9 quoted lines

> > But perhaps a cheaper option might be the traditional email/IRC
> > notification and having a client-side process watch for that
> > before fetching.
> 
> If there was a IRC channel with this info I could/would use that,
> getting it via E-Mail would just get me into the same problem
> public-inbox is currently solving for me, i.e. I might as well keep the
> git ML up-to-date on that machine if I'm going to otherwise need to
> subscribe to a "hey there's a new message on the git ML" list :)

The IRC server would have the same scalability problems faced by maintaining persistent connections to git-daemon or HTTP servers, however. And, yes, email does seem redundant, and modern header sizes (with DKIM, etc) are gigantic; but connection lifetime and concurrency is manageable to the server even if not instantaneous.

I also considered having clients setup a listener of some sort, (possibly using UDP) but that would have all the problems with git:// + firewalls.