{"thread":{"id":"45061","subject":"Cross-referencing the Git mailing list archive with their corresponding commits in `pu`","startedAt":"2017-02-06T15:34:39Z","lastAt":"2017-02-20T20:06:35Z","messageCount":11,"participants":["Johannes Schindelin","Junio C Hamano","Eric Wong","Jeff King","Lars Schneider"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"310914","messageId":"alpine.DEB.2.20.1702041206130.3496@virtualbox","threadId":"45061","inReplyTo":null,"subject":"Cross-referencing the Git mailing list archive with their corresponding commits in `pu`","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-02-06T15:34:28Z","receivedAt":"2017-02-06T15:34:39Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Josh,\n\nas discussed at the GitMerge, I am trying to come up with tooling that\nwill allow for substantially less tedious navigation between the local\nrepository, the mailing list, and what ends up in the `pu` branch.\n\nThat tooling would *still* not help lowering the barrier of entry for\ncontributing to Git by a lot, as it would *still* not address the problem\nthat mails sent from the most prevalent desktop mail client, as well as\nmails sent from the most prevalent web mail client, are simply and\nunceremoniously dropped. (This problem was acknowledged by quite a few\nnods even at the Contributors' Summit...) But still, we decided to start\n*somewhere* and this tooling is what we agreed on.\n\nIt is quite a bit harder going than I would like: as we have figured out,\nthe Subject: line is not a good way to link the commits with the original\nmails containing the patches, as commit messages are modified before being\npushed often enough to make this a fragile matching.\n\nSo I thought maybe the From: line (from the body, if available, otherwise\nfrom the header) in conjunction with the \"Date:\" header would work. But a\npreliminary study shows that there are 336 From: + Date: combinations in\nthe Git mailing list archive that are not unique. 71 of these are shared\nby three or more mails, even, and 9 are shared by more than 10 mails,\nrespectively. This is bad!\n\nUnsurprisingly, the top 10 of these cases were obviously caused by the\nbuiltin `git am` bug where it would not reset the author date properly.\nSurprisingly, though, there were a few cases from 2005, too.\n\nI had a quick look to find out what was the culprit (looking at the\n17-strong patch series \"Documentation fixes in response to my previous\nlisting\" by Nikolai Weibull, but I am at a loss there: the mail claims to\nbe sent by git-send-email and the patches appear to be generated by\ngit-format-patch as of v0.99.9l, neither of which had a Date:-related bug\nback in that time frame. My best guess is that the patches were mishandled\nby a tool similar to rebase -i (which entered Git only at v1.5.3).\n\nFor details, see:\nhttp://public-inbox.org/git/11340844841342-git-send-email-mailing-lists.git@rawuncut.elitemail.org/\n(this is also an example where public-inbox' thread detection went utterly\nwrong, including way too many mails in the \"thread\")\n\nThere was even a case of duplicated Date: headers in 2012. Now, this case\nis very curious, as there have been 7 mails with identical Date: header,\nbut it was not a 6-strong patch series. Instead, it was a 4-strong patch\nseries that needed three iterations before it was accepted, and the\nidentical Date: header appears only in v2's patches (*not* in its cover\nletter) and it *disappeared* in v3's 4/4, where it was set *back* by a\nweek (to the Date: it had in v1).\n\nFor details, see\nhttp://public-inbox.org/git/cover.1354693001.git.Sebastian.Leske@sleske.name/\nand\nhttp://public-inbox.org/git/cover.1354324110.git.Sebastian.Leske@sleske.name/\nand\nhttp://public-inbox.org/git/b115a546fa783b4121d118bb8fdb9270443f90fa.1353691892.git.Sebastian.Leske@sleske.name/\n\nThis last example also demonstrates a very curious test case for a\ndifferent difficulty in trying to reconstruct lost correspondences: the\npatch series was applied *twice*, independently of each other. First, on\nthe day v3 was submitted, it was applied on top of v1.8.1-rc0 (as commits\nee26a6e2b8..dd465ce66f), although it was not merged until v1.8.1-rc3. 22\ndays later, it was reapplied on top of maint so it could enter v1.8.0.3\n(back then, Git still had \"patchlevel\" versions): c2999adcd5..008c208c2c.\n\nAs you can see, there is a many-to-many relationship here, even if you do\nleave the *original* branch out of the picture entirely.\n\nWill keep you posted,\nDscho\n\nP.S.: I used public-inbox.org links instead of commit references to the\nGit repository containing the mailing list archive, because the format of\nsaid Git repository is so unfavorable that it was determined very quickly\nin a discussion between Patrick Reynolds (GitHub) and myself that it would\nput totally undue burden on GitHub to mirror it there (compare also Carlos\nNieto's talk at GitMerge titled \"Top Ten Worst Repositories to host on\nGitHub\").\n"},{"id":"310936","messageId":"xmqqwpd316f7.fsf@gitster.mtv.corp.google.com","threadId":"45061","inReplyTo":"alpine.DEB.2.20.1702041206130.3496@virtualbox","subject":"Re: Cross-referencing the Git mailing list archive with their corresponding commits in `pu`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-02-06T19:10:52Z","receivedAt":"2017-02-06T19:10:59Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> So I thought maybe the From: line (from the body, if available, otherwise\n> from the header) in conjunction with the \"Date:\" header would work.\n\nFYI, I use a post-applypatch hook to populate refs/notes/amlog notes\ntree when I queue a new patch; I am not sure how well the notes in\nit are preserved across rebases, but it could be a good starting\npoint. The notes tree is mirrored at git://github.com/git/gitster\nrepository.\n\nE.g.\n\n$ git show --notes=amlog --stat\ncommit 2488dcab22cee343fe35d9951160f0966a45fdb3\nAuthor: Patrick Steinhardt \nDate: Mon Feb 6 14:13:59 2017 +0100\n\n worktree: fix option descriptions for `prune`\n \n The `verbose` and `expire` options of the `git worktree prune`\n subcommand have wrong descriptions in that they pretend to relate to\n objects. But as the git-worktree(1) correctly states, these options have\n nothing to do with objects but only with worktrees. Fix the description\n accordingly.\n \n Signed-off-by: Patrick Steinhardt \n Signed-off-by: Junio C Hamano \n\nNotes (amlog):\n Message-Id: \n\n builtin/worktree.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n"},{"id":"310942","messageId":"20170206204820.GA7128@starla","threadId":"45061","inReplyTo":"alpine.DEB.2.20.1702041206130.3496@virtualbox","subject":"Re: Cross-referencing the Git mailing list archive with their corresponding commits in `pu`","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2017-02-06T20:48:20Z","receivedAt":"2017-02-06T20:48:26Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Johannes Schindelin wrote:\n> For details, see:\n> http://public-inbox.org/git/11340844841342-git-send-email-mailing-lists.git@rawuncut.elitemail.org/\n> (this is also an example where public-inbox' thread detection went utterly\n> wrong, including way too many mails in the \"thread\")\n\nThanks, it should be fixed in an hour or two when reindexing\nfinishes...\n\n\n\nbut it looks like reindexing is a little buggy in that it reuses\nthread IDs, too... (will fix)\n\nThe Tor .onion mirrors should be done, first, since they're on\nbetter hardware:\nhttp://hjrcffqmbrq6wope.onion/git/11340844841342-git-send-email-mailing-lists.git@rawuncut.elitemail.org/\nhttp://czquwvybam4bgbro.onion/git/11340844841342-git-send-email-mailing-lists.git@rawuncut.elitemail.org/\n\n> This last example also demonstrates a very curious test case for a\n> different difficulty in trying to reconstruct lost correspondences: the\n> patch series was applied *twice*, independently of each other. First, on\n> the day v3 was submitted, it was applied on top of v1.8.1-rc0 (as commits\n> ee26a6e2b8..dd465ce66f), although it was not merged until v1.8.1-rc3. 22\n> days later, it was reapplied on top of maint so it could enter v1.8.0.3\n> (back then, Git still had \"patchlevel\" versions): c2999adcd5..008c208c2c.\n> \n> As you can see, there is a many-to-many relationship here, even if you do\n> leave the *original* branch out of the picture entirely.\n\nFwiw, I've always seen the search ability of public-inbox as\nanalogous to rename detection in git; in that it can never be\nperfect, but can still be tweaked and improved after-the-fact\nand be used more flexibly.\n\nRight now, the thread searching public-inbox is loose in that it\nfavors overmatching based on Subject in addition to References.\nBut the actual threading algorithm (for display) is strict,\nrelying only on References. But yeah, there can be tweaks to\nimprove matching and introducing git (code) repository awareness\ninto the mail search...\n\n> Will keep you posted,\n\nLikewise :>\n\n> P.S.: I used public-inbox.org links instead of commit references to the\n> Git repository containing the mailing list archive, because the format of\n> said Git repository is so unfavorable that it was determined very quickly\n> in a discussion between Patrick Reynolds (GitHub) and myself that it would\n> put totally undue burden on GitHub to mirror it there (compare also Carlos\n> Nieto's talk at GitMerge titled \"Top Ten Worst Repositories to host on\n> GitHub\").\n\nAny suggestions on how the repository format can be improved?\n\nI haven't hit insurmountable performance problems, even on\nlow-end hardware; especially since I started storing blob ids in\nXapian itself, avoiding the expensive tree lookup via git.\n\nThe main problem seems to be tree size. Deepening (2/2/36 vs\n2/38) might be an option (I think Peff brought that up); but it\nmight be easier to switch to YYYYMM refs (working like\nlogrotate) and rely on Xapian to tie the entire thing together.\n\nSome change will definitely be needed for all LKML, but most\nprojects have less traffic than even git, and should be fine.\n\n\nBut, I am working to undermine centralized messaging systems\n(which GitHub and GitLab both are), so they would be wise to\nundermine public-inbox all the same ;>\n"},{"id":"310947","messageId":"20170206220754.5q2oddr5ej7c6qcg@sigill.intra.peff.net","threadId":"45061","inReplyTo":"20170206204820.GA7128@starla","subject":"Re: Cross-referencing the Git mailing list archive with their corresponding commits in `pu`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-02-06T22:07:55Z","receivedAt":"2017-02-06T22:08:04Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Feb 06, 2017 at 08:48:20PM +0000, Eric Wong wrote:\n\n> I haven't hit insurmountable performance problems, even on\n> low-end hardware; especially since I started storing blob ids in\n> Xapian itself, avoiding the expensive tree lookup via git.\n\nThe painful thing is traversing the object graph for clones and fetches.\nBitmaps help, but you still have to generate them.\n\n> The main problem seems to be tree size. Deepening (2/2/36 vs\n> 2/38) might be an option (I think Peff brought that up); but it\n> might be easier to switch to YYYYMM refs (working like\n> logrotate) and rely on Xapian to tie the entire thing together.\n\nYes, the hashing is definitely one issue. Some numbers here:\n\n http://public-inbox.org/git/20160805092805.w3nwv2l6jkbuwlzf@sigill.intra.peff.net/\n\nIf you have C commits on a tree with T entries, you have to do C*T hash\nlookups for a flat tree (for each commit, you have to see \"yup, already\nsaw that object\"). Sharding that across H entries at the top level drops\nthe tree cost from T to H + T/H (actually, it's a bit worse because we\nhave to read the secondary tree, too). Sharding again (at H') gets you\nH + H' + T/H/H'.\n\nLet's imagine you do one message per commit, so C=T. At 400K messages,\nthat's about 160 billion hash lookups flat. At H=256, it's about 700\nmillion. If you shard again with H'=256, it's 200 million. After that,\nthe additive terms start to dominate, and it's not worth going any\nfurther (and also, we're ignoring the extra-tree cost to each level).\n\nAt that point you're better off to start having fewer commits. I know\nthat the schema you use does put useful information into the commit\nmessage, but it's also redundant with what's in the messages themselves.\nAnd it sounds like you push most of that out to Xapian anyway.\n\nImagine your repo had one commit with 400K historical messages, and then\ngrouped the new messages so that on average we got about 10 messages per\ncommit (this doesn't seem unrealistic for something that commits every\nfew minutes; the messages tend to be bunched in time; I ran some\nnumbers against a 10-minute mark in the earlier message).\n\nThen after another 100K messages, we'd have C=10,001 and T=500K. With\ntwo levels of hashing at 256 each, that's ~5 million hash lookups to\nwalk the graph. And those numbers would be reasonable for a hosting site\nlike GitHub.\n\nI don't know what C is for the kernel repo, but I suspect with the right\ntuning it could be made into large-but-reasonable.\n\n-Peff\n"},{"id":"310956","messageId":"20170207001446.GB7128@starla","threadId":"45061","inReplyTo":"20170206220754.5q2oddr5ej7c6qcg@sigill.intra.peff.net","subject":"Re: Cross-referencing the Git mailing list archive with their corresponding commits in `pu`","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2017-02-07T00:14:46Z","receivedAt":"2017-02-07T00:14:59Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Jeff King wrote:\n> On Mon, Feb 06, 2017 at 08:48:20PM +0000, Eric Wong wrote:\n> \n> > I haven't hit insurmountable performance problems, even on\n> > low-end hardware; especially since I started storing blob ids in\n> > Xapian itself, avoiding the expensive tree lookup via git.\n> \n> The painful thing is traversing the object graph for clones and fetches.\n> Bitmaps help, but you still have to generate them.\n\nYep. \"public-inbox-init\" defaults to enabling bitmaps in the\nconfig for this reason.\n\n> > The main problem seems to be tree size. Deepening (2/2/36 vs\n> > 2/38) might be an option (I think Peff brought that up); but it\n> > might be easier to switch to YYYYMM refs (working like\n> > logrotate) and rely on Xapian to tie the entire thing together.\n> \n> Yes, the hashing is definitely one issue. Some numbers here:\n> \n> http://public-inbox.org/git/20160805092805.w3nwv2l6jkbuwlzf@sigill.intra.peff.net/\n> \n> If you have C commits on a tree with T entries, you have to do C*T hash\n> lookups for a flat tree (for each commit, you have to see \"yup, already\n> saw that object\"). Sharding that across H entries at the top level drops\n> the tree cost from T to H + T/H (actually, it's a bit worse because we\n> have to read the secondary tree, too). Sharding again (at H') gets you\n> H + H' + T/H/H'.\n> \n> Let's imagine you do one message per commit, so C=T. At 400K messages,\n> that's about 160 billion hash lookups flat. At H=256, it's about 700\n> million. If you shard again with H'=256, it's 200 million. After that,\n> the additive terms start to dominate, and it's not worth going any\n> further (and also, we're ignoring the extra-tree cost to each level).\n\nJust to make sure I'm following, here; the entire formulas are:\n\n\tC * H + H' + (T / H / H') # 2/2/36\n\tC * H + (T / H) # 2/38 (current)\n\nRight?\n\n> At that point you're better off to start having fewer commits. I know\n> that the schema you use does put useful information into the commit\n> message, but it's also redundant with what's in the messages themselves.\n> And it sounds like you push most of that out to Xapian anyway.\n\nYeah, there's no benefit to Xapian users for having any info in\nthe commit. However, keeping commit-per-message is still\nimportant to me to for better robustness from hardware and\nnetwork failures.\n\nBut yes, historical stuff could be squashed into a single commit\n(much like how linux.git started with v2.6.12-rc2 without\nhistory). Perhaps some folks will care about NNTP article\nnumbering being non-chronological...\n\n> Imagine your repo had one commit with 400K historical messages, and then\n> grouped the new messages so that on average we got about 10 messages per\n> commit (this doesn't seem unrealistic for something that commits every\n> few minutes; the messages tend to be bunched in time; I ran some\n> numbers against a 10-minute mark in the earlier message).\n> \n> Then after another 100K messages, we'd have C=10,001 and T=500K. With\n> two levels of hashing at 256 each, that's ~5 million hash lookups to\n> walk the graph. And those numbers would be reasonable for a hosting site\n> like GitHub.\n> \n> I don't know what C is for the kernel repo, but I suspect with the right\n> tuning it could be made into large-but-reasonable.\n\nLKML probably has an upper bound of 30K messages per month;\nso it could hit 100K in less than 4 months. Worst case might\nbe 360K messages a year\n\n\t360000 * (256 + 256 + ((360000 + old) / 256 / 256))\n\nThat's still at least 180 million hash lookups after a year or\nso of real-time updates; right? (But probably closer to 240\nmillion if there's 10 million old messages in there.\n\nInstead, I think I will add an option to support logrotate-style\nmonthly heads (YYYYMM); keeping 2/38 and C == T:\n\n\t30000 * (256 + (30000 / 256)) => 11 million\n\t30000 * (256 + 256 + (30000 / 256 / 256)) => 15 million\n\nThe monthly heads would each be discontiguous history-wise;\nso Xapian would become a requirement for users of this option\nfor Message-ID lookups, but histories would still be readable\nwith \"git log\"\n\nOne good side-effect of using monthly heads is --single-branch\nclones may be used if someone lacks the bandwidth or space to do\na full mirror. I'm not sure if the server-side (pack reuse,\nbitmaps) will benefit other aside from bandwidth reductions,\nthough.\n\n\nA (far-fetched) option I've considered would be to store entire\nmessages in the commit and have no trees or blobs at all. But\nthat would require a significant rework, and would also make\nXapian a hard requirement for even checking if a message is\ndeleted or not.\n"},{"id":"311177","messageId":"3F7E4C9E-593E-4FC5-B820-E6A0CDEB7476@gmail.com","threadId":"45061","inReplyTo":"xmqqwpd316f7.fsf@gitster.mtv.corp.google.com","subject":"Re: Cross-referencing the Git mailing list archive with their corresponding commits in `pu`","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-02-09T14:11:28Z","receivedAt":"2017-02-09T14:11:48Z","isPatch":false,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 06 Feb 2017, at 20:10, Junio C Hamano wrote:\n> \n> Johannes Schindelin writes:\n> \n>> So I thought maybe the From: line (from the body, if available, otherwise\n>> from the header) in conjunction with the \"Date:\" header would work.\n> \n> FYI, I use a post-applypatch hook to populate refs/notes/amlog notes\n> tree when I queue a new patch; I am not sure how well the notes in\n> it are preserved across rebases, but it could be a good starting\n> point. The notes tree is mirrored at git://github.com/git/gitster\n> repository.\n> \n> E.g.\n> \n> $ git show --notes=amlog --stat\n\nThat's super useful! Thanks for the pointer!\nWouldn't it make sense to push these notes to github.com/git/git ?\n\n- Lars\n"},{"id":"311224","messageId":"alpine.DEB.2.20.1702092242120.3496@virtualbox","threadId":"45061","inReplyTo":"3F7E4C9E-593E-4FC5-B820-E6A0CDEB7476@gmail.com","subject":"Re: Cross-referencing the Git mailing list archive with their corresponding commits in `pu`","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-02-09T21:53:34Z","receivedAt":"2017-02-09T22:12:24Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Lars,\n\nOn Thu, 9 Feb 2017, Lars Schneider wrote:\n\n> > On 06 Feb 2017, at 20:10, Junio C Hamano wrote:\n> > \n> > Johannes Schindelin writes:\n> > \n> >> So I thought maybe the From: line (from the body, if available,\n> >> otherwise from the header) in conjunction with the \"Date:\" header\n> >> would work.\n> > \n> > FYI, I use a post-applypatch hook to populate refs/notes/amlog notes\n> > tree when I queue a new patch; I am not sure how well the notes in it\n> > are preserved across rebases, but it could be a good starting point.\n> > The notes tree is mirrored at git://github.com/git/gitster repository.\n> > \n> > E.g.\n> > \n> > $ git show --notes=amlog --stat\n> \n> That's super useful! Thanks for the pointer!\n> Wouldn't it make sense to push these notes to github.com/git/git ?\n\nI am not quite sure about that. It is in a different namespace than what\nis usually cloned, and it currently adds 8MB to the download (there are\n\"amlog\" and \"commits\", the latter clearly being a sandbox).\n\nWhile I am thankful that there is at least some information available for\npatches integrated into `pu` since Nov 1 2016, the format is probably not\nstable (we are talking about free-form notes, after all), and it still\ndoes not help with catching the case where new patch series iterations (or\nin some case, new patch series, period) are missed.\n\nMake no mistake, it will be a huge undertaking to develop a tool that\nhelps with the management of patch series on top of the mailing list\ndriven patch review process. And even in the best case, it may be simply\ntoo hard for an automated tool to figure things out e.g. when Peff or\nJunio paste a tangentially related diff into a thread.\n\nIn the end, what I *really* would love to have is a system where you can\neasily query \"which reviewer comments on *any* of my patch series are new,\nor still unaddressed?\", and \"in what way was my patch modified relative to\nthe latest version I submitted?\". It may actually be impossible to create\nsuch a tool, as it cannot invent information/cross-references that it does\nnot have nor can deduce from available data.\n\nCiao,\nJohannes\n"},{"id":"311225","messageId":"xmqq8tpfnh47.fsf@gitster.mtv.corp.google.com","threadId":"45061","inReplyTo":"alpine.DEB.2.20.1702092242120.3496@virtualbox","subject":"Re: Cross-referencing the Git mailing list archive with their corresponding commits in `pu`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-02-09T22:18:00Z","receivedAt":"2017-02-09T22:18:41Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n>> > E.g.\n>> > \n>> > $ git show --notes=amlog --stat\n>> \n>> That's super useful! Thanks for the pointer!\n>> Wouldn't it make sense to push these notes to github.com/git/git ?\n>\n> I am not quite sure about that. It is in a different namespace than what\n> is usually cloned, and it currently adds 8MB to the download (there are\n> \"amlog\" and \"commits\", the latter clearly being a sandbox).\n\nI do not think the public mirrors of the primary repository should\nget amlog, either. It is more suited for those who are interested\nin broken-out topics, i.e. git://github.com/git/gitster.\n"},{"id":"312134","messageId":"alpine.DEB.2.20.1702171841450.3496@virtualbox","threadId":"45061","inReplyTo":"alpine.DEB.2.20.1702041206130.3496@virtualbox","subject":"Re: Cross-referencing the Git mailing list archive with their corresponding commits in `pu`","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-02-17T17:50:00Z","receivedAt":"2017-02-20T10:22:07Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Josh,\n\nOn Mon, 6 Feb 2017, Johannes Schindelin wrote:\n\n> as discussed at the GitMerge, I am trying to come up with tooling that\n> will allow for substantially less tedious navigation between the local\n> repository, the mailing list, and what ends up in the `pu` branch.\n\nI found a little bit more time last Friday to play with the\ncross-correlation between commits in `pu` and mails in\npublic-inbox/git.git and it is worse than I previously assumed.\n\nJust as a reminder: my plan was to start developing tools that will\nultimately help me as well as other contributors with the arcane mailing\nlist model of patch submission. And my first target was the seemingly\nsimple task of figuring out the mail corresponding to any given commit in\n`pu` (i.e. the mail that contained the patch, and whose mail thread is\nhence expected to have the entire patch review, and to which I would be\nexpected to respond if I find a problem with that commit).\n\nAnd since it is all-too-common that the oneline is adjusted before\napplying the patch, the Subject:/oneline pair is not a good candidate to\nfind matches.\n\nMy next best guess was that the author date would not be touched, so the\npair of Date: and authordate should make a good candidate.\n\nMy initial finding was that this is not without problems, as some mails\nwere sent with identical Date: lines (most likely due to bugs in the\ntools, e.g. the well-known and already fixed bug in git-am, and hence\ngit-rebase, where it would apply all patches using the first patch's\nauthor date), and worse: some of those mails contained actual patch series\nthat actually made it into Git's commit history.\n\nBut those are not the only problems.\n\nFor starters, I tried to cross-correlate *just* the commits that entered\n`pu` since one week ago (git rev-list --since=1.week.ago upstream/pu) with\nmails of the past month in the mailing list archive.\n\nOne obvious caveat is that RFC 2822 is ambiguous when it comes to the date\nformat. While it seems nice that you *can* write single-digit day numbers\nas single digit if you want, or with a leading zero, or with a leading\nspace, it makes it impossible to get away with exact matching. I did not\nreally want to complicate my research by parsing the dates and normalizing\nthem to epoch + timezone, also because I wanted results quick, so I simply\nnormalized the dates to have leading zeroes for single-digit day numbers,\nthat seems to work for the moment).\n\nThe first category of problematic commits come as no surprise: merges. We\ndo not even have a way to represent them as mails. I simply excluded them\nfrom the remainder of this study.\n\nThe second category should not be all that surprising, too: Junio often\nadjusts the release notes without sending those patches out for review.\nThose commits are:\n\n363588f (### match next, Junio C Hamano 2017-02-17)\n2076907 (Git 2.12-rc2, Junio C Hamano 2017-02-17)\n076c053 (Hopefully the final batch of mini-topics before the final,\n\tJunio C Hamano 2017-02-16)\nae86372 (Revert \"reset: add an example of how to split a commit into two\",\n\tJunio C Hamano 2017-02-16)\nd09b692 (A bit more for -rc2, Junio C Hamano 2017-02-15)\n\nThere is a third category, and this one *does* come as a surprise to me.\nIt appears that at least *some* patches' Date: lines are either ignored or\noverridden or changed on their way from the mailing list into Git's commit\nhistory. There was only one commit in that commit range:\n\n3c0cb0c (read_loose_refs(): read refs using resolve_ref_recursively(),\n\tMichael Haggerty 2017-02-09)\n\nThis one was committed with an author date \"Thu, 09 Feb 2017 21:53:52\n+0100\" but it appears that there was no mail sent to the Git mailing list\nwith that particular Date: header and the *actual* mail containing the\npatch was sent with a Date: header \"Fri, 10 Feb 2017 12:16:19 +0100\"\n(Message-ID:\nd8e906d969700acbca8dc717673d0a9cdc910f62.1486724698.git.mhagger@alum.mit.edu).\n\nIt is labor-intensive, but possible to find the correlation manually in\nthis case because the Subject: line has been left intact.\n\nHowever, this points to a serious problem with my approach: I try to\nre-create information that is actually not available (which Message-ID\ncorresponds to a given commit name). Since that information is not\navailable, it is quite possible that this information cannot be retrieved\naccurately (and Michael's commit demonstrates that this is not a merely\ntheoretic consideration). I do not know that I can fix this on my side.\n\n> P.S.: I used public-inbox.org links instead of commit references to the\n> Git repository containing the mailing list archive, because the format\n> of said Git repository is so unfavorable that it was determined very\n> quickly in a discussion between Patrick Reynolds (GitHub) and myself\n> that it would put totally undue burden on GitHub to mirror it there\n> (compare also Carlos Nieto's talk at GitMerge titled \"Top Ten Worst\n> Repositories to host on GitHub\").\n\nSince the main problem was the unfavorable commit history structure, I\n*think* that it may be possible to auto-process public-inbox.org/git.git\ninto a frequently-rewritten branch that squashes all commits from past\nyears into single, per-year commits (and the same for recent months, the\npast days, and a single commit accumulating the current day's commits) and\nthat that may solve the problematic structure. The blob names would remain\nidentical to what is on public-inbox, of course.\n\nCiao,\nJohannes\n\nP.S.: The *mini* scripts I used were\n\ncat generate-date-index.sh <<\\EOF\n#! /bin/sh\n\ncd public-inbox-git\n\nsince_commit=\"$1\"\ntest -n \"$since_commit\" ||\nsince_commit=$(git rev-list --since=1.month.ago master --reverse | head -n 1)\nfor sha1 in $(git diff --raw --no-abbrev $since_commit..master | cut -f 4 -d \\ )\ndo\n\tprintf '%s\\t%s\\n' \\\n\t\t\"$(git cat-file blob $sha1 |\n\t\tsed -n \\\n\t\t\t-e 's/^Date:[ \t]*\\([^,]*,\\) *\\([1-9] .*\\)/\\1 0\\2/p' \\\n\t\t\t-e 's/^Date:[ \t]*\\([^,]*,\\) *\\([0-9][0-9] .*\\)/\\1 \\2/p' \\\n\t\t\t-e '/^$/q')\" \\\n\t\t$sha1\ndone | less -S\nEOF\n\nto generate a file date-index.txt containing \"date\\tblob\" pairs where the blob\nrefers to the SHA-1 of the mail in public-inbox/git.git, and\n\ncat >match-pu.sh <<\\EOF\n#! /bin/sh\n\nfor commit in $(git rev-list --since=1.week.ago --no-merges upstream/pu)\ndo\n\tdate=\"$(git show -s --format=%aD $commit |\n\t\tsed 's/, \\([1-9]\\) /, 0\\1 /')\" # fix up Git's idea of RFC 2822\n\tmail_id=$(grep \"^$date\" date-index.txt | sed 's/.*\t//')\n\tcase \"$mail_id\" in\n\t'')\n\t\techo \"ERROR: no mail found for $commit (date $date)\" >&2\n\t\tgit show -s --pretty='tformat:%h (%s, %an %ad)' --date=short \\\n\t\t\t$commit >&2\n\t\t;;\n\t*' '*)\n\t\techo \"ERROR: multiple candidates found for $commit ($mail_id)\" >&2\n\t\t;;\n\t*)\n\t\techo \"$date $mail_id\"\n\t\t;;\n\tesac\ndone\nEOF\n\nto try to match the author dates with the ones in date-index.txt. The\nobvious next improvement is to list also Message-ID in date-index.txt.\n"},{"id":"312159","messageId":"xmqqefysk67v.fsf@gitster.mtv.corp.google.com","threadId":"45061","inReplyTo":"alpine.DEB.2.20.1702171841450.3496@virtualbox","subject":"Re: Cross-referencing the Git mailing list archive with their corresponding commits in `pu`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-02-20T19:33:24Z","receivedAt":"2017-02-20T19:33:31Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> There is a third category, and this one *does* come as a surprise to me.\n> It appears that at least *some* patches' Date: lines are either ignored or\n> overridden or changed on their way from the mailing list into Git's commit\n> history. There was only one commit in that commit range:\n>\n> 3c0cb0c (read_loose_refs(): read refs using resolve_ref_recursively(),\n> \tMichael Haggerty 2017-02-09)\n>\n> This one was committed with an author date \"Thu, 09 Feb 2017 21:53:52\n> +0100\" but it appears that there was no mail sent to the Git mailing list\n\nI think this is this one:\n\n \n\nRecent \"What's cooking\" lists the topic this one is part with this\ncomment:\n\n The tip one is newer than the one posted to the list but was sent\n privately by the author via his GitHub repository.\n\n"},{"id":"312161","messageId":"xmqq60k4k4oz.fsf@gitster.mtv.corp.google.com","threadId":"45061","inReplyTo":"xmqqefysk67v.fsf@gitster.mtv.corp.google.com","subject":"Re: Cross-referencing the Git mailing list archive with their corresponding commits in `pu`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-02-20T20:06:20Z","receivedAt":"2017-02-20T20:06:35Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Johannes Schindelin writes:\n>\n>> There is a third category, and this one *does* come as a surprise to me.\n>> It appears that at least *some* patches' Date: lines are either ignored or\n>> overridden or changed on their way from the mailing list into Git's commit\n>> history. There was only one commit in that commit range:\n>>\n>> 3c0cb0c (read_loose_refs(): read refs using resolve_ref_recursively(),\n>> \tMichael Haggerty 2017-02-09)\n>>\n>> This one was committed with an author date \"Thu, 09 Feb 2017 21:53:52\n>> +0100\" but it appears that there was no mail sent to the Git mailing list\n>\n> I think this is this one:\n>\n> \n>\n> Recent \"What's cooking\" lists the topic this one is part with this\n> comment:\n>\n> The tip one is newer than the one posted to the list but was sent\n> privately by the author via his GitHub repository.\n\nWe didn't have any pull from sub-maintainers during the period you\nchecked, but when we do, those could also fall into the category.\nEven though I see some l10n patches Cc'ed to the list, I won't be\nsurprised if not everything that is sent to Jiang Xin (i18n/l10n\ncoordinator) is, for example. It also is OK for sub-maintainers to\nhave their own commit to describe or otherwise improve their area\nand without sending a patch before doing so if they deem it\nappropriate [*1*].\n\nI actually think automation like yours would help another category:\nThere is a newer version of the series or an entirely new series on\nthe list, but the project's tree has not picked them up (yet).\n\nI from time to time sweep my inbox in an attempt to find and pick up\nleftover bits. Sometimes the authors remind me by pinging [*2*],\nwhich greatly helps. But another set of eyeballs that may be\nenhanced with a mechanised filter that catches \"messages without\ncorresponding commits\", which is the opposite of this \"third\"\ncategory, would be of great help, too [*3*].\n\n\n[Footnote]\n\n*1* ... like trivial fixes, for example, at their discretion. After\n all we entrusted their own area and we should give them the\n flexibility they can exercise with good taste ;-).\n\n*2* e.g. <2f67fc21-92f9-a03e-1b09-a237af6dbc46@alum.mit.edu>\n\n*3* ... even if a mechanised filter alone might strike too many\n false positives.\n"}]}