{"thread":{"id":"57024","subject":"Large delays in mailing list delivery?","startedAt":"2021-12-03T19:53:12Z","lastAt":"2022-02-07T21:34:28Z","messageCount":12,"participants":["Elijah Newren","Konstantin Ryabitsev","Ævar Arnfjörð Bjarmason","Jeff King","Thomas Guyot-Sionnest","Eric Wong"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"443021","messageId":"CABPp-BF_xsOpQ6GSaWs9u9JcnPQT_OXP-gCsAuxPtMj-X1tgOg@mail.gmail.com","threadId":"57024","inReplyTo":null,"subject":"Large delays in mailing list delivery?","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-12-03T19:52:58Z","receivedAt":"2021-12-03T19:53:12Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Are there some rather large delays in mailing list delivery these\ndays?  Anyone know who to contact to investigate?  [*]\n\nSome examples:\n\n* Stolee reviewed v2 of en/keep-cwd on Nov 29, three days after v3 was\nposted, and I mentioned v3 was current, he said that v3 had not yet\nappeared in his inbox[2] (though by the time of that response, half of\nit had appeared).\n\nI periodically look at lore.kernel.org/git because of the threaded\nview.  While I'm accustomed to a few hours delay between messages\narriving there and in my inbox, this week I noticed a few, among them\ntwo examples that I immediately remember:\n\n* https://lore.kernel.org/git/20211202030458.GA48278@newk/ from Dec 1\nstill hasn't arrived in my inbox.  I typed up a response, but waited\nfor it to show up so I could respond.  (I could use git-send-email to\nrespond, but that's a pain so I tend not to.)\n\n* v7 of ak/protect-any-current-branch has not arrived in my inbox yet,\ndespite being posted about 2 days ago.  I read through the series and\nthen was going to post a tiny comment or two yesterday, but only v6\nand older are in my email.\n\n\nThanks,\nElijah\n\n[1] And what are the odds that this issue will cause this email to be\ndelayed in delivery to everyone other than lore.kernel.org/git, and no\none sees it, until after the issue is resolved?  ;-)\n\n[2] https://lore.kernel.org/git/068b7faf-2ade-14a7-fee3-83fec26ae856@gmail.com/\n"},{"id":"443022","messageId":"20211203195856.7lfcy4gfvheb7lau@meerkat.local","threadId":"57024","inReplyTo":"CABPp-BF_xsOpQ6GSaWs9u9JcnPQT_OXP-gCsAuxPtMj-X1tgOg@mail.gmail.com","subject":"Re: Large delays in mailing list delivery?","fromName":"Konstantin Ryabitsev","fromEmail":"konstantin@linuxfoundation.org","sentAt":"2021-12-03T19:58:56Z","receivedAt":"2021-12-03T19:59:00Z","isPatch":false,"sender":{"key":"konstantin@linuxfoundation.org","avatar":"https://gravatar.com/avatar/7cb8827c6de56e1bd2dea16508c6708aa43feed3bf3813bcdacecdf96ceadd79?d=mp&s=160"},"body":"On Fri, Dec 03, 2021 at 11:52:58AM -0800, Elijah Newren wrote:\n> Are there some rather large delays in mailing list delivery these\n> days?  Anyone know who to contact to investigate?  [*]\n\nThe right person to contact is postmaster@vger.kernel.org, however I can\nactually answer your question (despite not actually being in charge of\nvger.kernel.org). Periodically, Gmail decides that there's just too much\nincoming mail for some accounts and will arbitrarily delay delivery by\nreturning a \"this account is receiving too much mail, please try later.\"\n\nI am willing to bet that this is what happened to you.\n\nBest regards,\n-K\n"},{"id":"443026","messageId":"211203.86sfv9qwdm.gmgdl@evledraar.gmail.com","threadId":"57024","inReplyTo":"CABPp-BF_xsOpQ6GSaWs9u9JcnPQT_OXP-gCsAuxPtMj-X1tgOg@mail.gmail.com","subject":"Re: Large delays in mailing list delivery?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-12-03T20:02:48Z","receivedAt":"2021-12-03T20:14:18Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, 3 Dec 2021 11:52:58 -0800 Elijah Newren wrote:\n\n> Are there some rather large delays in mailing list delivery these\n> days?  Anyone know who to contact to investigate?  [*]\n\nI've got E-Mail delays so large that I manually crafted In-Reply-To\netc. to reply to this :)\n\nI'm 99% sure it's some weird thing at GMail, and nothing to do with\nkernel.org.\n\nWhen I've experienced delays (sometimes of half a day or more) both\nhttps://public-inbox.org/git/ and https://lore.kernel.org/git/ have been\nupdated.\n\nKonstantin Ryabitsev notes (and would be in a position to know) that\nGMail throttles delivery (with an smtp 451 error?).\n\nBut the other day when Junio sent out a What's Cooking and I saw it on\nlore, but it wasn't in my mailbox for some hours. That time I looked a\nbi tinto it.\n\nI went into the gmail UI and searched for rfc822msgid:$id, nothing. It\nwas neither there on IMAP or in the UI.\n\nHowever when the E-Mail finally arrived I looked at the raw headers, and\nall the \"Received\" headers (including GMail's own internal routing) were\nwithin a couple of minutes of Junio having sent the mail (and in the\nmeantime it went through vger etc.).\n\nSo, I'm hazy on E-Mail infrastructure details these days (but worked no\nit in a past life), but that really seems to me like GMail in fact got\nthe E-Mail, but it was just sitting in some local queue of theirs before\nit got served to me.\n\nRight now I can't see the mail I'm replying to in my inbox[1], but I can\nreport the full headers once it arrives if that helps.\n\n1. A search for:\n   rfc822msgid:CABPp-BF_xsOpQ6GSaWs9u9JcnPQT_OXP-gCsAuxPtMj-X1tgOg@mail.gmail.com\n"},{"id":"443028","messageId":"20211203202427.o575sgrx4auqkmjp@meerkat.local","threadId":"57024","inReplyTo":"211203.86sfv9qwdm.gmgdl@evledraar.gmail.com","subject":"Re: Large delays in mailing list delivery?","fromName":"Konstantin Ryabitsev","fromEmail":"konstantin@linuxfoundation.org","sentAt":"2021-12-03T20:24:27Z","receivedAt":"2021-12-03T20:24:32Z","isPatch":false,"sender":{"key":"konstantin@linuxfoundation.org","avatar":"https://gravatar.com/avatar/7cb8827c6de56e1bd2dea16508c6708aa43feed3bf3813bcdacecdf96ceadd79?d=mp&s=160"},"body":"On Fri, Dec 03, 2021 at 09:02:48PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> When I've experienced delays (sometimes of half a day or more) both\n> https://public-inbox.org/git/ and https://lore.kernel.org/git/ have been\n> updated.\n\nBtw, you can source lore.kernel.org straight into your gmail inbox. :)\n\n    https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started\n    https://people.kernel.org/monsieuricon/lore-lei-part-2-now-with-imap\n\nOr, you can read it via nntp://nntp.lore.kernel.org/.\n\nBest regards,\n-K\n"},{"id":"443029","messageId":"211203.86k0glqvpp.gmgdl@evledraar.gmail.com","threadId":"57024","inReplyTo":"20211203202427.o575sgrx4auqkmjp@meerkat.local","subject":"Re: Large delays in mailing list delivery?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-12-03T20:26:08Z","receivedAt":"2021-12-03T20:28:38Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Dec 03 2021, Konstantin Ryabitsev wrote:\n\n> On Fri, Dec 03, 2021 at 09:02:48PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>> When I've experienced delays (sometimes of half a day or more) both\n>> https://public-inbox.org/git/ and https://lore.kernel.org/git/ have been\n>> updated.\n>\n> Btw, you can source lore.kernel.org straight into your gmail inbox. :)\n>\n>     https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started\n>     https://people.kernel.org/monsieuricon/lore-lei-part-2-now-with-imap\n>\n> Or, you can read it via nntp://nntp.lore.kernel.org/.\n\nNice. I was looking into doing that the other day but stopped at not\nfinding a way to do that with the public-inbox software itself.\n\nWell, \"the other day\" was probably ~6 months ago, but at the time I\ncould only find some TODO item for a Maildir export. Looks like that\nworks now, great!\n"},{"id":"443039","messageId":"YaqRr//Yu8cn+7s7@coredump.intra.peff.net","threadId":"57024","inReplyTo":"20211203202427.o575sgrx4auqkmjp@meerkat.local","subject":"Re: Large delays in mailing list delivery?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-12-03T21:52:47Z","receivedAt":"2021-12-03T21:52:49Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Dec 03, 2021 at 03:24:27PM -0500, Konstantin Ryabitsev wrote:\n\n> On Fri, Dec 03, 2021 at 09:02:48PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> > When I've experienced delays (sometimes of half a day or more) both\n> > https://public-inbox.org/git/ and https://lore.kernel.org/git/ have been\n> > updated.\n> \n> Btw, you can source lore.kernel.org straight into your gmail inbox. :)\n> \n>     https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started\n>     https://people.kernel.org/monsieuricon/lore-lei-part-2-now-with-imap\n> \n> Or, you can read it via nntp://nntp.lore.kernel.org/.\n\nI've been watching the lei stuff, and it's pretty cool. I was already\nindexing my local archive with notmuch, so right now I have an \"in\nbetween\" solution where I pull from lore and deliver into a local\nmailbox, like:\n\n-- >8 --\nROOT=$HOME/.cache/lists\n\ntest -n \"$QUIET\" && exec >/dev/null\ntest $# = 0 && set -- $(cd \"$ROOT\" && echo *)\n\nfor list in \"$@\"; do\n\tcd \"$ROOT/$list\" || exit 1\n\tgit fetch -v ${QUIET:+--quiet} || exit 1\n\tgit rev-list refs/lists/delivered..HEAD |\n\tgit diff-tree --format= --stdin --raw |\n\tawk '{print $4}' |\n\twhile read blob; do\n\t\ttest -n \"$QUIET\" || echo >&2 \"Delivering $blob...\"\n\t\tgit cat-file blob \"$blob\" |\n\t\tsafecat maildir/tmp maildir/new ||\n\t\texit 1\n\tdone || exit 1\n\tgit update-ref refs/lists/delivered HEAD || exit 1\ndone\n-- >8 --\n\nSome notes:\n\n  - ~/.cache/lists/git is a bare clone of https://lore.kernel.org/git/0;\n    I know this will run into problems if we eventually get enough\n    messages to start a new epoch, but that's still years away by the\n    current counting.\n\n  - maildir in the bare repo is a symlink to the actual maildir I\n    deliver to (~/mail/git)\n\n  - I use safecat here to deliver into the maildir, but notmuch-insert\n    would probably make more sense.\n\nI think this is less featureful than lei (especially some of the\nadvanced queries), but it was a drop-in replacement for my existing\nqueries and workflows. And it has low dependencies.\n\nOf course it doesn't help much if you're using gmail or something. :)\nI guess you could replace the safecat delivery with git-imap-send or\nsimilar.\n\nIt's polling, of course, but I assume that a noop fetch against the lore\nrepo is pretty cheap. I think Eric's public-inbox/lei code for doing\nupdates has an extra HTTP-endpoint check to avoid invoking even the noop\nGit (though it results in an extra HTTP request when there _is_\nsomething to fetch).\n\nAnd of course it's holding two copies of the messages (one in Git, and\nthen the delivered one). That's OK for my purposes, but I have noticed\nthat lei is generally much faster to answer queries, because maildirs\nhave awful cold-cache behavior because of all the inodes (and my\nmailserver is still on spinning disks).\n\nSo I don't really recommend anybody going down my same path if they\ncould just jump to using lei. But I thought the script above might help\nsomebody who wants to just replace one small bit of their\ninfrastructure/workflow without retraining fingers, etc.\n\n-Peff\n"},{"id":"443078","messageId":"2cd4a07b-6ffe-15f9-c8d4-fab322a79d2f@guyot-sionnest.net","threadId":"57024","inReplyTo":"20211203195856.7lfcy4gfvheb7lau@meerkat.local","subject":"Re: Large delays in mailing list delivery?","fromName":"Thomas Guyot-Sionnest","fromEmail":"thomas@guyot-sionnest.net","sentAt":"2021-12-04T16:51:43Z","receivedAt":"2021-12-04T17:00:02Z","isPatch":false,"sender":{"key":"thomas@guyot-sionnest.net","avatar":null},"body":"On 2021-12-03 14:58, Konstantin Ryabitsev wrote:\n> On Fri, Dec 03, 2021 at 11:52:58AM -0800, Elijah Newren wrote:\n>> Are there some rather large delays in mailing list delivery these\n>> days?  Anyone know who to contact to investigate?  [*]\n> The right person to contact is postmaster@vger.kernel.org, however I can\n> actually answer your question (despite not actually being in charge of\n> vger.kernel.org). Periodically, Gmail decides that there's just too much\n> incoming mail for some accounts and will arbitrarily delay delivery by\n> returning a \"this account is receiving too much mail, please try later.\"\n>\n> I am willing to bet that this is what happened to you.\n\nI have already contacted postmaster and they are aware, however you add\nvery good point. Somehow I knew it but it's the way you put it that made\nme realize: the emails are sent in bulk to gmail (to many or all\nrecipients for a single copy at once). The mail header even shows this,\nlisting the first recipient followed by \"+ 99 others\". If it's delayed\nfor all because of a single recipient that can only be because the 4xx\nerror is returned at the end of the DATA command (rather than on a\nspecific RCPT TO command), so no way for the sender to retry that single\nfaulty address alone later.\n\nI'm not sure if their software allows it but I suggested sending to a\nsingle recipient at a time for gmail, that would solve the issue.\n\nIn the meantime what I did is switch my subscription to another email\nusing a forwarder address on a domain I own - I still haven't received\nall the backlog but at least I'm getting new posts timely now.\n\nRegards,\n\n--\nThomas\n\n"},{"id":"443165","messageId":"211206.867dchr9nt.gmgdl@evledraar.gmail.com","threadId":"57024","inReplyTo":"20211203202427.o575sgrx4auqkmjp@meerkat.local","subject":"Re: Large delays in mailing list delivery?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-12-06T16:12:57Z","receivedAt":"2021-12-06T16:17:21Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Dec 03 2021, Konstantin Ryabitsev wrote:\n\n> On Fri, Dec 03, 2021 at 09:02:48PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>> When I've experienced delays (sometimes of half a day or more) both\n>> https://public-inbox.org/git/ and https://lore.kernel.org/git/ have been\n>> updated.\n>\n> Btw, you can source lore.kernel.org straight into your gmail inbox. :)\n>\n>     https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started\n>     https://people.kernel.org/monsieuricon/lore-lei-part-2-now-with-imap\n>\n> Or, you can read it via nntp://nntp.lore.kernel.org/.\n\n[CC'd meta@public-inbox.org, probably best to move this thread over\nthere sooner than later, but CC'ing git@ still in case this is\ninteresting to others]\n\nI poked a bit at setting this up but couldn't find from building\npublic-inbox.org & trying to page through the docs how I'd get from an\nexisting public-inbox.org/git/ checkout to a local Maildir.\n\nIf you could share some recipe or a pointer to the right docs for that\nthat would be much appreciated. Thanks!\n"},{"id":"443169","messageId":"20211206163626.GA4714@dcvr","threadId":"57024","inReplyTo":"211206.867dchr9nt.gmgdl@evledraar.gmail.com","subject":"Re: Large delays in mailing list delivery?","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2021-12-06T16:36:26Z","receivedAt":"2021-12-06T16:36:30Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:\n> On Fri, Dec 03 2021, Konstantin Ryabitsev wrote:\n> \n> > On Fri, Dec 03, 2021 at 09:02:48PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> >> When I've experienced delays (sometimes of half a day or more) both\n> >> https://public-inbox.org/git/ and https://lore.kernel.org/git/ have been\n> >> updated.\n> >\n> > Btw, you can source lore.kernel.org straight into your gmail inbox. :)\n> >\n> >     https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started\n> >     https://people.kernel.org/monsieuricon/lore-lei-part-2-now-with-imap\n> >\n> > Or, you can read it via nntp://nntp.lore.kernel.org/.\n> \n> [CC'd meta@public-inbox.org, probably best to move this thread over\n> there sooner than later, but CC'ing git@ still in case this is\n> interesting to others]\n> \n> I poked a bit at setting this up but couldn't find from building\n> public-inbox.org & trying to page through the docs how I'd get from an\n> existing public-inbox.org/git/ checkout to a local Maildir.\n\nExisting, public-inboxes can be set as \"externals\" and managed\nvia {add,forget,ls}-external sub-commands:\n\n\t# for locally-cloned inboxes:\n\tpublic-inbox-index /path/to/existing/inbox\n\tlei add-external /path/to/existing/inbox\n\n\t# relies on curl, memoizes data downloaded for each search:\n\tlei add-external https://yhbt.net/lore/git\n\nLocal externals will be included by every \"lei q\" invocation;\nHTTP(S) ones require \"lei q --remote\"\n\nIf you only want to use an external as a one-off without adding\nit, the -I/--include and -O/--only flags are useful:\n\n  lei q -O https://yhbt.net/lore/git -o /tmp/results SEARCH_TERMS\n\n> If you could share some recipe or a pointer to the right docs for that\n> that would be much appreciated. Thanks!\n\nlei-overview(7) manpage documents some things, at least:\nhttps://public-inbox.org/lei-overview.html  Patches welcome :>\n\nIMHO lei still kinda sucks, and I probably won't have time to\nwork on it for a bit :<\n"},{"id":"443228","messageId":"211207.86v901oxgc.gmgdl@evledraar.gmail.com","threadId":"57024","inReplyTo":"211203.86sfv9qwdm.gmgdl@evledraar.gmail.com","subject":"Re: Large delays in mailing list delivery?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-12-07T04:20:18Z","receivedAt":"2021-12-07T04:23:05Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Dec 03 2021, Ævar Arnfjörð Bjarmason wrote:\n\n> On Fri, 3 Dec 2021 11:52:58 -0800 Elijah Newren wrote:\n>\n>> Are there some rather large delays in mailing list delivery these\n>> days?  Anyone know who to contact to investigate?  [*]\n>\n> I've got E-Mail delays so large that I manually crafted In-Reply-To\n> etc. to reply to this :)\n>\n> I'm 99% sure it's some weird thing at GMail, and nothing to do with\n> kernel.org.\n>\n> When I've experienced delays (sometimes of half a day or more) both\n> https://public-inbox.org/git/ and https://lore.kernel.org/git/ have been\n> updated.\n>\n> Konstantin Ryabitsev notes (and would be in a position to know) that\n> GMail throttles delivery (with an smtp 451 error?).\n>\n> But the other day when Junio sent out a What's Cooking and I saw it on\n> lore, but it wasn't in my mailbox for some hours. That time I looked a\n> bi tinto it.\n>\n> I went into the gmail UI and searched for rfc822msgid:$id, nothing. It\n> was neither there on IMAP or in the UI.\n>\n> However when the E-Mail finally arrived I looked at the raw headers, and\n> all the \"Received\" headers (including GMail's own internal routing) were\n> within a couple of minutes of Junio having sent the mail (and in the\n> meantime it went through vger etc.).\n>\n> So, I'm hazy on E-Mail infrastructure details these days (but worked no\n> it in a past life), but that really seems to me like GMail in fact got\n> the E-Mail, but it was just sitting in some local queue of theirs before\n> it got served to me.\n>\n> Right now I can't see the mail I'm replying to in my inbox[1], but I can\n> report the full headers once it arrives if that helps.\n>\n> 1. A search for:\n>    rfc822msgid:CABPp-BF_xsOpQ6GSaWs9u9JcnPQT_OXP-gCsAuxPtMj-X1tgOg@mail.gmail.com\n\nFor what it's worth after it finally arrived the relevant part of the\nheaders is, which shows the issue Konstantin Ryabitsev\ndescribed. I.e. it was sitting for ~3 days between vger and GMail:\n    \n    [...]\n    Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18])\n            by mx.google.com with ESMTP id g10si21274138pfj.188.2021.12.06.17.50.50;\n            Mon, 06 Dec 2021 17:51:02 -0800 (PST)\n    Received-SPF: pass (google.com: domain of git-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;\n    Authentication-Results: mx.google.com;\n           dkim=pass header.i=@gmail.com header.s=20210112 header.b=V2iU5Wyg;\n           spf=pass (google.com: domain of git-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=git-owner@vger.kernel.org;\n           dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com\n    Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n            id S1343798AbhLCT4g (ORCPT <rfc822;bojan.kljakic.tech@gmail.com>\n            + 99 others); Fri, 3 Dec 2021 14:56:36 -0500\n    [...]\n\nI.e. not at all what I alluded to above, and at this point I'm not sure\nI ever really saw a mail like that (maybe I just misread the headers at\nthe time + confirmation bias, and I didn't go re-digging again now...).\n"},{"id":"447527","messageId":"220202.86leyt8uly.gmgdl@evledraar.gmail.com","threadId":"57024","inReplyTo":"20211206163626.GA4714@dcvr","subject":"Using public-inbox+lei+Emacs+mu+mu4e (was: Large delays in mailing list delivery?)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-02T09:34:23Z","receivedAt":"2022-02-02T09:51:10Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Dec 06 2021, Eric Wong wrote:\n\n> Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:\n>> On Fri, Dec 03 2021, Konstantin Ryabitsev wrote:\n>> \n>> > On Fri, Dec 03, 2021 at 09:02:48PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>> >> When I've experienced delays (sometimes of half a day or more) both\n>> >> https://public-inbox.org/git/ and https://lore.kernel.org/git/ have been\n>> >> updated.\n>> >\n>> > Btw, you can source lore.kernel.org straight into your gmail inbox. :)\n>> >\n>> >     https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started\n>> >     https://people.kernel.org/monsieuricon/lore-lei-part-2-now-with-imap\n>> >\n>> > Or, you can read it via nntp://nntp.lore.kernel.org/.\n>> \n>> [CC'd meta@public-inbox.org, probably best to move this thread over\n>> there sooner than later, but CC'ing git@ still in case this is\n>> interesting to others]\n>> \n>> I poked a bit at setting this up but couldn't find from building\n>> public-inbox.org & trying to page through the docs how I'd get from an\n>> existing public-inbox.org/git/ checkout to a local Maildir.\n>\n> Existing, public-inboxes can be set as \"externals\" and managed\n> via {add,forget,ls}-external sub-commands:\n>\n> \t# for locally-cloned inboxes:\n> \tpublic-inbox-index /path/to/existing/inbox\n> \tlei add-external /path/to/existing/inbox\n>\n> \t# relies on curl, memoizes data downloaded for each search:\n> \tlei add-external https://yhbt.net/lore/git\n>\n> Local externals will be included by every \"lei q\" invocation;\n> HTTP(S) ones require \"lei q --remote\"\n>\n> If you only want to use an external as a one-off without adding\n> it, the -I/--include and -O/--only flags are useful:\n>\n>   lei q -O https://yhbt.net/lore/git -o /tmp/results SEARCH_TERMS\n>\n>> If you could share some recipe or a pointer to the right docs for that\n>> that would be much appreciated. Thanks!\n>\n> lei-overview(7) manpage documents some things, at least:\n> https://public-inbox.org/lei-overview.html  Patches welcome :>\n>\n> IMHO lei still kinda sucks, and I probably won't have time to\n> work on it for a bit :<\n\nThanks. I finally got around to setting this up.\n\nThe above instructions didn't quite work for me, but here's what I did\n(in the form of a script lifted from a screen(1) config I've\ngot). Indented with un-indented comments. The \"stuff\" is screen's way of\n\"run this command\" (or well, input these characters):\n\nI had a ~/g/git-ml clone already, but this makes one:\n    \n    stuff \"cd ~/g/git-ml || git clone https://public-inbox.org/git ~/g/git-ml^M\"\n\nThe initial index:\n\n    ## This will create a .git/publici-inbox in ~/g/git-ml. Takes a while\n    ## the first time.\n    stuff \"time public-inbox-index -v \\$PWD^M\"\n\nI fiddled with this for a bit because it refused to work, turns out it\nwas missing the .git at the end, i.e. it expected a bare repo[1]:\n\n    ## When we add the lei external it *must* have the \".git\" part,\n    ## because it'll try to find the \"public-inbox\" folder at wherever we\n    ## point it.\n    stuff \"test -d ~/.config/lei || lei add-external ~/g/git-ml/.git^M\"\n\nA bit of a UX wart not to be able to specify no --limit, or maybe I'm\nmissing a way:\n\n    ## The one-off massive import of the Git ML. TODO: No way to specify\n    ## an infinite limit? Not --no-limit or --limit=0.\n    stuff \"test -d ~/Maildir/lei-q-git-ml || time lei q --limit=999999999 -v -o ~/Maildir/lei-q-git-ml l:git.vger.kernel.org^M\"\n\nThe initial indexing:\n\n    ## After the one-off import this will take forever *the first time*\n    ## (or around 20m), but subsequent invocations will be fast:\n    stuff \"time lei up ~/Maildir/lei-q-git-ml^M\"\n\nRuns an ad-hoc script to keep it up-to-date, which is quoted below:\n\n    ## Run it in a loop\n    stuff \"public-inbox-lei-pull-index^M\"\n\nThat script (which I whipped up just now. Is there a better/more\nstandard way? to keep a public-inbox+lei pair up-to-date with\nsleep/backoff etc?\n\t\n\t#!/bin/sh\n\tset -xe\n\t\n\trepo=~/g/git-ml\n\twhile true\n\tdo\n\t\toid=$(git -C $repo rev-parse HEAD)\n\t\tgit -C $repo pull\n\t\tnoid=$(git -C $repo rev-parse HEAD)\n\t\tif test \"$oid\" = \"$noid\"\n\t\tthen\n\t\t\techo Nothing to update\n\t\t\tsleep 60\n\t\t\tcontinue\n\t\tfi\n\t\t(\n\t\t\tcd $repo &&\n\t\t\tpublic-inbox-index -v \"$PWD\"\n\t\t)\n\t\tlei up ~/Maildir/lei-q-git-ml\n\t\tsleep 1\n\tdone\n\nI use Emacs+mu4e for my E-Mail. And since I index ~/Maildir having these\nfiles dropped in there will be added to its index. Then I just changed\nmy saved search to also look through that maildir (I guess the entire\nfirst condition could be dropped, but whatever):\n\n    \"(maildir:/personal-gmail/* OR maildir:/lei-q-git-ml/*) AND list:git.vger.kernel.org OR recip:git@vger.kernel.org OR recip:git-packagers@googlegroups.com\"\n\nBecause \"mu\" is generally good about de-duplicating stuff I've now got\nan inbox with mixed messages I can sync from GMail (including my \"Sent\"\nfolder), and I get up-to-the-minute ML traffic now (it's bee 10hrs-4day\ndelayed for 3-4 months at least).\n\nSo new messages are generally from the \"lei\" directory, but when I send\none it'll be dropped in the personal-gmail.\n\nI still need to check if it's doing the wrong thing with e.g. \"read\"\nflags if I read a mail synced via lei that later arrives in GMail. But I\nmostly don't use \"read\" statuses anyway...\n\n1.  Maybe this \"I only tested if it complied\" patch would make sense to catch that?\n\ndiff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm\nindex 2958d3f9..be49621f 100644\n--- a/lib/PublicInbox/LeiXSearch.pm\n+++ b/lib/PublicInbox/LeiXSearch.pm\n@@ -613,7 +613,7 @@ sub add_uri {\n \t}\n }\n \n-sub prepare_external {\n+sub _prepare_external {\n \tmy ($self, $loc, $boost) = @_; # n.b. already ordered by boost\n \tif (ref $loc) { # already a URI, or PublicInbox::Inbox-like object\n \t\treturn add_uri($self, $loc) if $loc->can('scheme');\n@@ -638,6 +638,14 @@ sub prepare_external {\n \tpush @{$self->{locals}}, $loc;\n }\n \n+sub prepare_external {\n+\tmy ($self, $loc, $boost) = @_;\n+\tmy $ret = _prepare_external($self, $loc, $boost);\n+\twarn \"W: we got nothing from $loc, did you mean $loc/.git?\"\n+\t\tif !$ret && -e \"$loc/.git\";\n+\treturn $ret;\n+}\n+\n sub _lcat_i { # LeiMailSync->each_src iterator callback\n \tmy ($oidbin, $id, $each_smsg) = @_;\n \t$each_smsg->({blob => unpack('H*', $oidbin), pct => 100});\n"},{"id":"447914","messageId":"20220207212724.GA9233@dcvr","threadId":"57024","inReplyTo":"220202.86leyt8uly.gmgdl@evledraar.gmail.com","subject":"Re: Using public-inbox+lei+Emacs+mu+mu4e (was: Large delays in mailing list delivery?)","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2022-02-07T21:27:24Z","receivedAt":"2022-02-07T21:34:28Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:\n> On Mon, Dec 06 2021, Eric Wong wrote:\n> > Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:\n> >> On Fri, Dec 03 2021, Konstantin Ryabitsev wrote:\n> >> \n> >> > On Fri, Dec 03, 2021 at 09:02:48PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> >> >> When I've experienced delays (sometimes of half a day or more) both\n> >> >> https://public-inbox.org/git/ and https://lore.kernel.org/git/ have been\n> >> >> updated.\n> >> >\n> >> > Btw, you can source lore.kernel.org straight into your gmail inbox. :)\n> >> >\n> >> >     https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started\n> >> >     https://people.kernel.org/monsieuricon/lore-lei-part-2-now-with-imap\n> >> >\n> >> > Or, you can read it via nntp://nntp.lore.kernel.org/.\n> >> \n> >> [CC'd meta@public-inbox.org, probably best to move this thread over\n> >> there sooner than later, but CC'ing git@ still in case this is\n> >> interesting to others]\n> >> \n> >> I poked a bit at setting this up but couldn't find from building\n> >> public-inbox.org & trying to page through the docs how I'd get from an\n> >> existing public-inbox.org/git/ checkout to a local Maildir.\n> >\n> > Existing, public-inboxes can be set as \"externals\" and managed\n> > via {add,forget,ls}-external sub-commands:\n> >\n> > \t# for locally-cloned inboxes:\n> > \tpublic-inbox-index /path/to/existing/inbox\n> > \tlei add-external /path/to/existing/inbox\n> >\n> > \t# relies on curl, memoizes data downloaded for each search:\n> > \tlei add-external https://yhbt.net/lore/git\n> >\n> > Local externals will be included by every \"lei q\" invocation;\n> > HTTP(S) ones require \"lei q --remote\"\n> >\n> > If you only want to use an external as a one-off without adding\n> > it, the -I/--include and -O/--only flags are useful:\n> >\n> >   lei q -O https://yhbt.net/lore/git -o /tmp/results SEARCH_TERMS\n> >\n> >> If you could share some recipe or a pointer to the right docs for that\n> >> that would be much appreciated. Thanks!\n> >\n> > lei-overview(7) manpage documents some things, at least:\n> > https://public-inbox.org/lei-overview.html  Patches welcome :>\n> >\n> > IMHO lei still kinda sucks, and I probably won't have time to\n> > work on it for a bit :<\n> \n> Thanks. I finally got around to setting this up.\n> \n> The above instructions didn't quite work for me, but here's what I did\n> (in the form of a script lifted from a screen(1) config I've\n> got). Indented with un-indented comments. The \"stuff\" is screen's way of\n> \"run this command\" (or well, input these characters):\n> \n> I had a ~/g/git-ml clone already, but this makes one:\n>     \n>     stuff \"cd ~/g/git-ml || git clone https://public-inbox.org/git ~/g/git-ml^M\"\n\nWait, lack of --mirror or --bare for git-clone on any\npublic-inbox is a big mistake in terms of inode+disk use\n\n> The initial index:\n> \n>     ## This will create a .git/publici-inbox in ~/g/git-ml. Takes a while\n>     ## the first time.\n>     stuff \"time public-inbox-index -v \\$PWD^M\"\n> \n> I fiddled with this for a bit because it refused to work, turns out it\n> was missing the .git at the end, i.e. it expected a bare repo[1]:\n> \n>     ## When we add the lei external it *must* have the \".git\" part,\n>     ## because it'll try to find the \"public-inbox\" folder at wherever we\n>     ## point it.\n>     stuff \"test -d ~/.config/lei || lei add-external ~/g/git-ml/.git^M\"\n\nYeah, public-inbox has never supported non-bare usage.\n\n> A bit of a UX wart not to be able to specify no --limit, or maybe I'm\n> missing a way:\n> \n>     ## The one-off massive import of the Git ML. TODO: No way to specify\n>     ## an infinite limit? Not --no-limit or --limit=0.\n>     stuff \"test -d ~/Maildir/lei-q-git-ml || time lei q --limit=999999999 -v -o ~/Maildir/lei-q-git-ml l:git.vger.kernel.org^M\"\n\nOops.  --no-limit or --limit=0 or --limit=-1 support will be\nadded at some point...\n\n> The initial indexing:\n> \n>     ## After the one-off import this will take forever *the first time*\n>     ## (or around 20m), but subsequent invocations will be fast:\n>     stuff \"time lei up ~/Maildir/lei-q-git-ml^M\"\n> \n> Runs an ad-hoc script to keep it up-to-date, which is quoted below:\n> \n>     ## Run it in a loop\n>     stuff \"public-inbox-lei-pull-index^M\"\n> \n> That script (which I whipped up just now. Is there a better/more\n> standard way? to keep a public-inbox+lei pair up-to-date with\n> sleep/backoff etc?\n\nNot yet, unfortunately.  I would like to have some sort of\nlong-polling support (similar to IDLE in IMAP) because I hate\nsleep/backoff.  But I don't think I'm physically nor mentally\ncapable of doing that or much of anything, anymore.\n\n> \t#!/bin/sh\n> \tset -xe\n> \t\n> \trepo=~/g/git-ml\n> \twhile true\n> \tdo\n> \t\toid=$(git -C $repo rev-parse HEAD)\n> \t\tgit -C $repo pull\n> \t\tnoid=$(git -C $repo rev-parse HEAD)\n> \t\tif test \"$oid\" = \"$noid\"\n> \t\tthen\n> \t\t\techo Nothing to update\n> \t\t\tsleep 60\n> \t\t\tcontinue\n> \t\tfi\n> \t\t(\n> \t\t\tcd $repo &&\n> \t\t\tpublic-inbox-index -v \"$PWD\"\n> \t\t)\n> \t\tlei up ~/Maildir/lei-q-git-ml\n> \t\tsleep 1\n> \tdone\n> \n> I use Emacs+mu4e for my E-Mail. And since I index ~/Maildir having these\n> files dropped in there will be added to its index. Then I just changed\n> my saved search to also look through that maildir (I guess the entire\n> first condition could be dropped, but whatever):\n> \n>     \"(maildir:/personal-gmail/* OR maildir:/lei-q-git-ml/*) AND list:git.vger.kernel.org OR recip:git@vger.kernel.org OR recip:git-packagers@googlegroups.com\"\n> \n> Because \"mu\" is generally good about de-duplicating stuff I've now got\n> an inbox with mixed messages I can sync from GMail (including my \"Sent\"\n> folder), and I get up-to-the-minute ML traffic now (it's bee 10hrs-4day\n> delayed for 3-4 months at least).\n> \n> So new messages are generally from the \"lei\" directory, but when I send\n> one it'll be dropped in the personal-gmail.\n> \n> I still need to check if it's doing the wrong thing with e.g. \"read\"\n> flags if I read a mail synced via lei that later arrives in GMail. But I\n> mostly don't use \"read\" statuses anyway...\n\nlei can export flags from its internal DBs to IMAP via \"lei export-kw\"\nsee lei-mail-sync-overview(7) for some details, but usability is still\nrough...\n\n> 1.  Maybe this \"I only tested if it complied\" patch would make sense to catch that?\n> \n> diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm\n> index 2958d3f9..be49621f 100644\n> --- a/lib/PublicInbox/LeiXSearch.pm\n> +++ b/lib/PublicInbox/LeiXSearch.pm\n> @@ -613,7 +613,7 @@ sub add_uri {\n>  \t}\n>  }\n>  \n> -sub prepare_external {\n> +sub _prepare_external {\n>  \tmy ($self, $loc, $boost) = @_; # n.b. already ordered by boost\n>  \tif (ref $loc) { # already a URI, or PublicInbox::Inbox-like object\n>  \t\treturn add_uri($self, $loc) if $loc->can('scheme');\n> @@ -638,6 +638,14 @@ sub prepare_external {\n>  \tpush @{$self->{locals}}, $loc;\n>  }\n>  \n> +sub prepare_external {\n> +\tmy ($self, $loc, $boost) = @_;\n> +\tmy $ret = _prepare_external($self, $loc, $boost);\n> +\twarn \"W: we got nothing from $loc, did you mean $loc/.git?\"\n> +\t\tif !$ret && -e \"$loc/.git\";\n> +\treturn $ret;\n> +}\n\nProbably add a note saying non-bare repos are a terrible idea,\nanyways, especially for v1 public-inboxes like public-inbox.org/git\n"}]}