Thank you for public-inbox!

Hi Eric,

I would like to take five minutes to thank you for public-inbox. It is invaluable for me in the meantime. And I think I will never be able to thank you enough for it.

Just a couple of things where it is super useful to me:

- Recently, my mail provider started dropping mails left and right. They
  might even be addressed to me, and they never arrive in my inbox (and
  not even in the spam folder: they just never arrive). I spent some ten
  hours this past weekend to script identifying the mails in public-inbox
  that I never received, and to weed through them.

  It seems I missed some 4,000 mails. Thanks to you, I now saw those mails
  (although I had to delete most of them after reading only their subject,
  in the interest of time).

- Many a times when my automated builds identified a problem with the test
  suite, it was a lot quicker to use public inbox to identify the mail to
  respond to than to use my mail program.

- Sometimes, I find myself in want of replying to a past patch, but back
  in the day when it was sent I thought it would not be interesting to me,
  so I deleted it. With public-inbox, I can easily get it in raw format,
  i.e. put it back into my inbox so I can reply.

- I cannot tell you how many times I send a link to public-inbox to my
  colleagues rather than forwarding a mail, because the former will give
  them more context (and also semi-live updates in case somebody replied
  to said mail after I sent the link).

A couple of things in the future that public-inbox will make possible for me:

- You probably are aware of my GitGitGadget endeavor, a project similar in
  aim to SubmitGit, but a lot more integrated with the GitHub experience
  (and not requiring you to hand over your mail sending credentials to
  AWS).

  One particular feature I found myself really wanting in SubmitGit (but
  not possible due to its one-way design): I want my Pull Requests to be
  closed once the patches are integrated into git.git's `master` branch.

  While this feature is not available in GitGitGadget yet, I am well
  underway there. I already have a notes ref (`commit-to-mail`, available
  via https://github.git/gitgitgadget/git) that annotates commits in
  git.git with the Message-ID of the corresponding mail. By "I have", I
  mean: there is an automated task that uses public-inbox to keep that ref
  up to date.

  I also have an accompanying `mail-to-commit` notes ref that maps
  Message-IDs back to the corresponding commit in git.git. That notes ref
  "annotates" the (non-existing) blob you get when piping the Message-ID
  with a trailing newline to `git hash-object`.

  Again, this is information that would be absolutely unobtainable without
  public-inbox.

- Related, I want to annotate the GitHub Pull Requests handled by
  GitGitGadvget with the corresponding name of the branch in
  https://github.com/gitster/git.

  This requires that `mail-to-commit` I mentioned in the previous bullet
  point, and therefore would not be possible without public-inbox.

- A feature I plan on introducing into GitGitGadget is to attach comments
  to the GitHub Pull Request when anybody replies to the patch thread sent
  out by GitGitGadget.

  Also this feature would be impossible without public-inbox.

- Another really useful feature I plan on introducing is to attach
  comments to those PRs whenever a What's Cooking is talking about the
  corresponding branch.

  Once again, would be impossible without public-inbox.

So thank you, thank you, thank you, for public-inbox!

Ciao, Dscho

P.S.: FWIW I added a mirror of public-inbox to https://git-for-windows.visualstudio.com/git.public-inbox, so that my automated tasks, as well as my playing around, does not stress your server too much.

<snip>

You're very welcome, Johannes. And I'm hoping to have a few more goodies live this fall/winter for public-inbox :>

On Mon, Aug 27, 2018 at 04:25:13PM +0200, Johannes Schindelin wrote:

Let me echo that appreciation. I have always kept my own archive of the list, but it's so valuable to have stable URLs for communicating with other folks (and I find the general design of public-inbox _way_ more useful than gmane ever was).

I've thought about mirroring it to a public server as well, just for redundancy. But without the same domain, I'm not sure it would be all that useful as a community resource.

Eric, let me know if there's something there that would help (e.g., putting more servers in DNS round-robin).

-Peff

Jeff King <peff@peff.net> wrote:

I wouldn't get too attached to the domain, "public-inbox.org" is too long for my tastes anyways. "peff.net/git/$MESSAGE_ID" would actually be more user-friendly :>

A generic Message-ID redirection/finding service would be good, (maybe some DHT thing, but... has that taken off for git blobs, yet?)

Anyways I hope to teach public-inbox to auto-linkify Message-ID-looking strings "<XXXXXXXXXXX@XXXXXXXX>" into URLs for domain-portability, (but it's ambiguous with email addresses). But yeah, I don't like things being tied to domain names.

I've also been considering setting up a parallel instance on 80x24.org to use the more-scalable "v2" repository format developed for https://lore.kernel.org/lkml/

Speaking of lore, Konstantin confirmed he'll be getting more vger lists up: https://public-inbox.org/meta/20180827205811.GA1611@chatter

On 2018-08-29 12:02, Eric Wong wrote:

This would be very useful for people who use MUAs without Message-ID lookup capabilities, even without domain-portability.

On Wed, Aug 29 2018, Andrei Rybak wrote:

FWIW Many MUAs have some hidden way to do this, for example to find the E-Mail I'm replying to (your E-Mail) on GMail: https://mail.google.com/mail/u/0/#search/rfc822msgid%3A3eb1c5e8-3e89-0d2e-30b1-339f38c4c703%40gmail.com

I.e. rfc822msgid:<message-id>

Confusingly Google Groups accepts the same syntax, but will barf if you include the <>'s, but that's from memory, maybe I misrecall or they've fixed it.

On Wed, Aug 29, 2018 at 10:02:43AM +0000, Eric Wong wrote:

Yes, and I agree that the URL portability is one of the things I really love about public-inbox (after all, I do have my own archive and now I can follow people's public-inbox links into my very-fast local copy).

I guess I just wonder if I set up a mirror on another domain, would anybody actually _use_ it? I'd think most people would just go to public-inbox.org as the de facto URL.

That would be neat, but I think it actually makes references less useful in a lot of cases. URLs are universally understood, which means:

 - people who don't know about public-inbox can just follow the link
   (and in fact, that's how they learn how useful it is!)

 - even for people who do know about it, they are likely to read mails
   in their MUA. And most MUAs have some mechanism for easily following
   a URL, but won't know how to auto-linkify a message-id.

So I too dream of a world where I can say "give me more information on this identifier" and my tools search a peer to peer distributed hash table for it. But I don't think we live in that world yet.

At the very least, I think if we plan to reference without an http URL that we would use something like URI-ish, like <mid:ABC@XYZ>. That gives tools a better chance to say "OK, I know how to find message-ids" (though I still think that it's much less helpful out of the box compared to an http URL).

-Peff

Jeff King wrote:

If it's faster than public-inbox.org and you don't mind the traffic I would send, then I'll use it. :)

[...]

I agree: please don't stop using URLs.

Having the message-id in the URL is very useful for being able to migrate to another server.

In an ideal world, we'd be habitually using URLs that reference a redirector, in the same spirit as https://www.kernel.org/lore.html. That way, there is a very restricted syntax to use (e.g. just message-ids) that is stable and can be reconfigured to redirect to another service as appropriate. In principle it also allows tricks like redirecting based on geography or making the redirector go to the user's preferred archive interface based on a cookie.

Thanks, Jonathan

Jonathan Nieder <jrnieder@gmail.com> wrote:

Is performance a problem on public-inbox.org for you?

It should be pretty fast, but maybe there's corner-cases and I haven't had time to pay close attention, lately...

I've also been sorta considering downgrading to a $5/month VPS (from a $20/month VPS) to force myself to pay more attention to performance while saving myself a few bucks. But I wouldn't get to dogfood on SMP, anymore...

Eric Wong wrote:

It's pretty fast.  I'm just very, very picky about latency. ;-)

It's good to know you're interested in which corner cases are bad. The next time I have a noticeably slow page load, I'll contact meta@.

[...]

Sounds reasonable to me. If performance gets bad, that's just a reason for people to help out (either with patches or e.g. with donated VMs for hosting).

Speaking of the latter: what are your current resource requirements? E.g. which of the dimensions in [1] do you not fit into?

Thanks, Jonathan

[1] https://cloud.google.com/free/docs/always-free-usage-limits#compute_name

Jonathan Nieder <jrnieder@gmail.com> wrote:

Best way for good latency is to have a local mirror, but I guess Googlers still aren't allowed to run AGPL software?

Alright. It could also be a general datacenter/networking problem so https://status.linode.com/ (my VPS provider) is worth checking.

Not too much; but could always be better on the software side.

Dunno, I'm not seeing RAM, there.

Depending on traffic, it's around 200MB per-public-inbox-httpd worker (2 workers for 2 cores) when there's a traffic surge on from popular sites. Memory usage is the biggest disappointment and only happens when Varnish can't read fast enough. Everything in the PSGI code is is streamed if possible(*). My goal is to maintain <50MB per worker process, but it could be tough in Perl5. Anyways $20/month gets me 4GB RAM (so I have way more than I need).

CPU usage isn't even noticeable (only bursts) and I do other stuff on that server all the time.

HDDs wouldn't work well at all and I've noticed differences based on SSD quality with Xapian. Storage for Xapian+SQLite is 4-5x what's in git, so for this list, it's under 7G total (but more will be needed for Xapian reindexing/compact and git repacking).

(*) Individual messages for returning giant mboxes and threads are
    all lazily fleshed out from skeleton data structures as the
    client socket becomes writable (and quickly discarded after writing).
    Technically it's all compatible with any PSGI server, but all
    the streaming stuff is tailored to run on public-inbox-httpd.
    But there's also git-http-backend memory use which comes in
    bursts (bitmaps enabled, of course)

Jeff King <peff@peff.net> wrote:

Heh, one of the (unstated?) goals of public-inbox is to educate the users on how Message-IDs (and email in general) works. And to that end...

....More than dreaming, our goal should be to BUILD such a world :> After all, it was my intense dislike of centralization which drew me to DVCS and git in the first place.

That would be awesome if somelike like <mid:ABC@XYZ> could be a standard and adopted (likewise with <git:$object_id>).

I haven't checked, but are there existing/similar RFCs? Surely somebody has tried to get <git:$object_id> adopted by now, right?

On Thu, Aug 30, 2018 at 07:20:49AM +0000, Eric Wong wrote:

There's https://tools.ietf.org/html/rfc2392. They even call it "mid:".  :)

I don't know of any git URI scheme, though it's similar in spirit to magnet: links. Those are a bit verbose, though, because they're really a meta-format for a bunch of different content-addressable schemes.

-Peff