Re: [RFC/RFH] Fun things with git-notes, or: patch tracking backwards

Hi everyone

I'll start with the fun first, try this in a git.git:

  git fetch git://repo.or.cz/git/trast.git mailnotes &&
  GIT_NOTES_REF=FETCH_HEAD git log origin/pu

I played around with some python code over the weekend that automatically filters through git@vger.kernel.org history and scans for patches, threads, and "What's cooking" messages. So far it seems to be working ok.

The net effect is that we get a backwards patch tracker: instead of tracking the patches via some other means (say a web interface), the automatic annotations can reconstruct more information about the patches than what is eventually contained in the commits, and make it available via git-notes.

Right now it only applies the patches that it finds, and associates that with known commits to annotate them. I eventually want to scan for replies to patch text too, and insert them into the notes too. I also plan to publish topics for the patches somewhere (they're already applied locally) and track at least a rudimentary status, perhaps one of "unreplied", "replied" and "accepted". (Distinguishing "rejected" seems AI-complete.)

The whole process is a bunch of python scripts currently available at

  git://repo.or.cz/trackgit.git
  http://repo.or.cz/w/trackgit.git

It's all a bit after-the-fact right now, and can't cope with a few things yet, for example patch series that aren't in git.git appearing out of sequence to the mail reader.

Runtime is okay-ish so that I should be able to run it as a cronjob; note regeneration is almost negligible (<1min), importing a month's worth of mails takes on the order of 20min, and scanning history (to know about the base blobs) since v1.6.0 is around 3min.

So the RFC bit is, is this useful to anyone? What information would you like to see in it, and what could be left out?

And the RFH: I don't have a full mail archive, not even since I joined the list. There also doesn't seem to be a convenient download button on gmane. Does anyone have (or know of) an archive going at least back to v1.6.0 (not sure if any further back is interesting), which was released in August last year, that you could send to me?

Thanks in advance :-)

Hi Thomas,

On Mon, Feb 09, 2009 at 03:08:08PM +0100, Thomas Rast wrote:

you can download emails in mbox format from gmane by running

  wget http://download.gmane.org/gmane.comp.version-control.git/X/Y

which will download all emails starting at the email with "gmane id" X and ending at the email with "gmane id" Y-1.

So, if you want to download the whole archive, you could run

  wget http://download.gmane.org/gmane.comp.version-control.git/1/109090

(but of course that 109090 will be larger by the time you will reading this). However, there's a catch, as gmane's script execution time is limited to 30 sec, so you will not get the whole archive, but only the first couple of thousands emails (in my case and network I got around 6k emails in one run). Therefore you'll need some loop to download only a few thousand emails in each iteration.

Best, Gábor

SZEDER Gábor wrote:

Thanks, that will do nicely!

On Sunday 08 February 2009 20:25, Shawn O. Pearce wrote:

I came across the same problem. I also sent a tentative patchset to solve the problem about 12 hours ago:

http://thread.gmane.org/gmane.comp.version-control.git/109035

Thomas Rast wrote:

An update: I have fully automated the process, it now fetches mails from Gmane over HTTP which gives it the Gmane URLs for free. I'm rather happy with the latter feature, especially since Konsole has a feature to recognize and open links directly.

I have imported all commits, and mails since roughly July 2008 (starting with Gmane 89000). In this timeframe there were 1802 non-merge commits, and the mailnotes tree now holds 1122 annotations. I won't import mails any further back until the parsing-related code has become reasonably stable, but this at least covers the post-v1.6.0 commits.

Thomas Rast <trast@student.ethz.ch> writes:

Wonderful.

Thomas Rast <trast@student.ethz.ch> writes:

How do you match the mails to commits?

I am curious what the right balance for the matching algorithm should be, between being forgiving about amending of commit log message and the patch text to fix minor typos and obvious bugs, and being strict not to cause false matches to a message that contains the second iteration of the patch, when what was committed was the first iteration.

Junio C Hamano wrote:

Right now it's just the patch-id. Maybe filtering (author,subject) and then picking the one that is the most similar could work.

Thomas Rast <trast@student.ethz.ch> writes:

Yeah, I actually was thinking about matching the (date, author) tuple and nothing else, as it is unlikely you would have dups.

By the way, the note to f6b98e4 (git-web--browse: Fix check for /bin/start, 2009-02-08) is interesting.

Ramsay's 104332 was the same as what ended up in the commit, but his second patch that can be found by going to the thread from it is obviously a better alternative. In short, I screwed up, by not recalling the previous round. Sorry.

I find the "Extra-Notes:" tag a bit too loud, but I am probably a minority who thinks everything but the Message-ID can be dropped, so please don't take it as a feature request ;-)

Junio C Hamano wrote:

Thanks, good idea. I changed the code to parse the required data, and we're now up to 1502 annotations.

Unfortunately I noticed there's a bug in the mail input stage: Python's mailbox module assumes any '^From ' line starts a new mail, while gmane apparently uses a slightly different format based on the double newlines too (and they always have the same 'From news@gmane.org Tue Mar 04 03:33:20 2003' separator too), and doesn't quote '^From ' in the bodies. So any mail containing such body lines got chopped down in the middle, and any patches contained in them won't apply because of the missing headers.

A quick perl run shows that there are 26 mails affected among the 89000+ mails that I've (again) imported. The fix should be easy, but I'm already short on sleep.

I refactored the final formatting stage a bit to let it do several notes trees, and you can now take your pick:

  git://repo.or.cz/trackgit.git notes/full
  git://repo.or.cz/trackgit.git notes/terse

The latter only has 'Message-Id' and 'Archived-At'.

Thomas Rast wrote:

[...]

Q.E.D.!  The correct repository is

  git://repo.or.cz/git/trast.git notes/full
  git://repo.or.cz/git/trast.git notes/terse

Sorry for the noise.