{"thread":{"id":"17675","subject":"Re: [RFC/RFH] Fun things with git-notes, or: patch tracking backwards","startedAt":"2009-02-09T14:08:08Z","lastAt":"2009-02-11T22:58:20Z","messageCount":11,"participants":["SZEDER Gábor","Thomas Rast","Junio C Hamano","Giuseppe Bilotta"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"299090","messageId":"200902091508.11460.trast@student.ethz.ch","threadId":"17675","inReplyTo":null,"subject":"[RFC/RFH] Fun things with git-notes, or: patch tracking backwards","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2009-02-09T14:08:08Z","receivedAt":"2009-02-09T14:08:08Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Hi everyone\n\nI'll start with the fun first, try this in a git.git:\n\n git fetch git://repo.or.cz/git/trast.git mailnotes &&\n GIT_NOTES_REF=FETCH_HEAD git log origin/pu\n\nI played around with some python code over the weekend that\nautomatically filters through git@vger.kernel.org history and scans\nfor patches, threads, and \"What's cooking\" messages. So far it seems\nto be working ok.\n\nThe net effect is that we get a backwards patch tracker: instead of\ntracking the patches via some other means (say a web interface), the\nautomatic annotations can reconstruct more information about the\npatches than what is eventually contained in the commits, and make it\navailable via git-notes.\n\nRight now it only applies the patches that it finds, and associates\nthat with known commits to annotate them. I eventually want to scan\nfor replies to patch text too, and insert them into the notes too. I\nalso plan to publish topics for the patches somewhere (they're already\napplied locally) and track at least a rudimentary status, perhaps one\nof \"unreplied\", \"replied\" and \"accepted\". (Distinguishing \"rejected\"\nseems AI-complete.)\n\nThe whole process is a bunch of python scripts currently available at\n\n git://repo.or.cz/trackgit.git\n http://repo.or.cz/w/trackgit.git\n\nIt's all a bit after-the-fact right now, and can't cope with a few\nthings yet, for example patch series that aren't in git.git appearing\nout of sequence to the mail reader.\n\nRuntime is okay-ish so that I should be able to run it as a cronjob;\nnote regeneration is almost negligible (<1min), importing a month's\nworth of mails takes on the order of 20min, and scanning history (to\nknow about the base blobs) since v1.6.0 is around 3min.\n\nSo the RFC bit is, is this useful to anyone? What information would\nyou like to see in it, and what could be left out?\n\nAnd the RFH: I don't have a full mail archive, not even since I joined\nthe list. There also doesn't seem to be a convenient download button\non gmane. Does anyone have (or know of) an archive going at least\nback to v1.6.0 (not sure if any further back is interesting), which\nwas released in August last year, that you could send to me?\n\nThanks in advance :-)\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"103858","messageId":"20090209161907.GA953@neumann","threadId":"17675","inReplyTo":"200902091508.11460.trast@student.ethz.ch","subject":"Re: [RFC/RFH] Fun things with git-notes, or: patch tracking backwards","fromName":"SZEDER Gábor","fromEmail":"szeder@ira.uka.de","sentAt":"2009-02-09T16:19:07Z","receivedAt":"2009-02-09T16:19:07Z","isPatch":false,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"Hi Thomas,\n\nOn Mon, Feb 09, 2009 at 03:08:08PM +0100, Thomas Rast wrote:\n> And the RFH: I don't have a full mail archive, not even since I joined\n> the list. There also doesn't seem to be a convenient download button\n> on gmane.\n\nyou can download emails in mbox format from gmane by running\n\n wget http://download.gmane.org/gmane.comp.version-control.git/X/Y\n\nwhich will download all emails starting at the email with \"gmane id\" X\nand ending at the email with \"gmane id\" Y-1.\n\nSo, if you want to download the whole archive, you could run\n\n wget http://download.gmane.org/gmane.comp.version-control.git/1/109090\n\n(but of course that 109090 will be larger by the time you will reading\nthis). However, there's a catch, as gmane's script execution time is\nlimited to 30 sec, so you will not get the whole archive, but only the\nfirst couple of thousands emails (in my case and network I got around\n6k emails in one run). Therefore you'll need some loop to download\nonly a few thousand emails in each iteration.\n\n\nBest,\nGábor\n"},{"id":"103861","messageId":"200902091729.49487.trast@student.ethz.ch","threadId":"17675","inReplyTo":"20090209161907.GA953@neumann","subject":"Re: [RFC/RFH] Fun things with git-notes, or: patch tracking backwards","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2009-02-09T16:29:45Z","receivedAt":"2009-02-09T16:29:45Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"SZEDER Gábor wrote:\n> you can download emails in mbox format from gmane by running\n> \n> wget http://download.gmane.org/gmane.comp.version-control.git/X/Y\n> \n> which will download all emails starting at the email with \"gmane id\" X\n> and ending at the email with \"gmane id\" Y-1.\n\nThanks, that will do nicely!\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"299091","messageId":"1234201781-5537-1-git-send-email-giuseppe.bilotta@gmail.com","threadId":"17675","inReplyTo":"20090209161907.GA953@neumann","subject":"Re: tool and worktree","fromName":"Giuseppe Bilotta","fromEmail":"giuseppe.bilotta@gmail.com","sentAt":"2009-02-09T17:49:41Z","receivedAt":"2009-02-09T17:49:41Z","isPatch":false,"sender":{"key":"giuseppe.bilotta@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1464?v=4"},"body":"On Sunday 08 February 2009 20:25, Shawn O. Pearce wrote:\n> bill lam wrote:\n>> I track /etc using a config\n>> \n>> [core]\n>> \trepositoryformatversion = 0\n>> \tfilemode = true\n>> \tbare = false\n>> \tworktree = /etc\n>> \tlogAllRefUpdates = true\n>> \texcludesfile =\n>> \n>> But that can not be handled by tools,\n>> \n>> git gui : cannot use funny .git directory .\n> \n> If someone sends patches for git-gui, maybe. This use case of\n> different repository and worktree isn't very common for git-gui\n> so it doesn't support it.\n\nI came across the same problem. I also sent a tentative patchset to\nsolve the problem about 12 hours ago:\n\nhttp://thread.gmane.org/gmane.comp.version-control.git/109035\n\n-- \nGiuseppe \"Oblomov\" Bilotta\n"},{"id":"104096","messageId":"200902102342.29962.trast@student.ethz.ch","threadId":"17675","inReplyTo":"200902091508.11460.trast@student.ethz.ch","subject":"Re: [RFC/RFH] Fun things with git-notes, or: patch tracking backwards","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2009-02-10T22:42:27Z","receivedAt":"2009-02-10T22:42:27Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Thomas Rast wrote:\n> git fetch git://repo.or.cz/git/trast.git mailnotes &&\n> GIT_NOTES_REF=FETCH_HEAD git log origin/pu\n\nAn update: I have fully automated the process, it now fetches mails\nfrom Gmane over HTTP which gives it the Gmane URLs for free. I'm\nrather happy with the latter feature, especially since Konsole has a\nfeature to recognize and open links directly.\n\nI have imported all commits, and mails since roughly July 2008\n(starting with Gmane 89000). In this timeframe there were 1802\nnon-merge commits, and the mailnotes tree now holds 1122 annotations.\nI won't import mails any further back until the parsing-related code\nhas become reasonably stable, but this at least covers the post-v1.6.0\ncommits.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"104098","messageId":"7vbpt9lxax.fsf@gitster.siamese.dyndns.org","threadId":"17675","inReplyTo":"200902102342.29962.trast@student.ethz.ch","subject":"Re: [RFC/RFH] Fun things with git-notes, or: patch tracking backwards","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-02-10T22:52:22Z","receivedAt":"2009-02-10T22:52:22Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Rast writes:\n\n> Thomas Rast wrote:\n>> git fetch git://repo.or.cz/git/trast.git mailnotes &&\n>> GIT_NOTES_REF=FETCH_HEAD git log origin/pu\n>\n> An update: I have fully automated the process, it now fetches mails\n> from Gmane over HTTP which gives it the Gmane URLs for free. I'm\n> rather happy with the latter feature, especially since Konsole has a\n> feature to recognize and open links directly.\n>\n> I have imported all commits, and mails since roughly July 2008\n> (starting with Gmane 89000). In this timeframe there were 1802\n> non-merge commits, and the mailnotes tree now holds 1122 annotations.\n\nWonderful.\n"},{"id":"104101","messageId":"7v7i3xlwz4.fsf@gitster.siamese.dyndns.org","threadId":"17675","inReplyTo":"200902102342.29962.trast@student.ethz.ch","subject":"Re: [RFC/RFH] Fun things with git-notes, or: patch tracking backwards","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-02-10T22:59:27Z","receivedAt":"2009-02-10T22:59:27Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Rast writes:\n\n> I have imported all commits, and mails since roughly July 2008\n> (starting with Gmane 89000). In this timeframe there were 1802\n> non-merge commits, and the mailnotes tree now holds 1122 annotations.\n\nHow do you match the mails to commits?\n\nI am curious what the right balance for the matching algorithm should be,\nbetween being forgiving about amending of commit log message and the patch\ntext to fix minor typos and obvious bugs, and being strict not to cause\nfalse matches to a message that contains the second iteration of the\npatch, when what was committed was the first iteration.\n"},{"id":"104106","messageId":"200902110012.34717.trast@student.ethz.ch","threadId":"17675","inReplyTo":"7v7i3xlwz4.fsf@gitster.siamese.dyndns.org","subject":"Re: [RFC/RFH] Fun things with git-notes, or: patch tracking backwards","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2009-02-10T23:12:30Z","receivedAt":"2009-02-10T23:12:30Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Junio C Hamano wrote:\n> Thomas Rast writes:\n> \n> > I have imported all commits, and mails since roughly July 2008\n> > (starting with Gmane 89000). In this timeframe there were 1802\n> > non-merge commits, and the mailnotes tree now holds 1122 annotations.\n> \n> How do you match the mails to commits?\n> \n> I am curious what the right balance for the matching algorithm should be,\n> between being forgiving about amending of commit log message and the patch\n> text to fix minor typos and obvious bugs, and being strict not to cause\n> false matches to a message that contains the second iteration of the\n> patch, when what was committed was the first iteration.\n\nRight now it's just the patch-id. Maybe filtering (author,subject)\nand then picking the one that is the most similar could work.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"104110","messageId":"7vprhpkgzk.fsf@gitster.siamese.dyndns.org","threadId":"17675","inReplyTo":"200902110012.34717.trast@student.ethz.ch","subject":"Re: [RFC/RFH] Fun things with git-notes, or: patch tracking backwards","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-02-10T23:30:07Z","receivedAt":"2009-02-10T23:30:07Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Rast writes:\n\n> Junio C Hamano wrote:\n>> Thomas Rast writes:\n>> \n>> > I have imported all commits, and mails since roughly July 2008\n>> > (starting with Gmane 89000). In this timeframe there were 1802\n>> > non-merge commits, and the mailnotes tree now holds 1122 annotations.\n>> \n>> How do you match the mails to commits?\n>> \n>> I am curious what the right balance for the matching algorithm should be,\n>> between being forgiving about amending of commit log message and the patch\n>> text to fix minor typos and obvious bugs, and being strict not to cause\n>> false matches to a message that contains the second iteration of the\n>> patch, when what was committed was the first iteration.\n>\n> Right now it's just the patch-id. Maybe filtering (author,subject)\n> and then picking the one that is the most similar could work.\n\nYeah, I actually was thinking about matching the (date, author) tuple and\nnothing else, as it is unlikely you would have dups.\n\nBy the way, the note to f6b98e4 (git-web--browse: Fix check for\n/bin/start, 2009-02-08) is interesting.\n\nRamsay's 104332 was the same as what ended up in the commit, but his\nsecond patch that can be found by going to the thread from it is obviously\na better alternative. In short, I screwed up, by not recalling the\nprevious round. Sorry.\n\nI find the \"Extra-Notes:\" tag a bit too loud, but I am probably a minority\nwho thinks everything but the Message-ID can be dropped, so please don't\ntake it as a feature request ;-)\n"},{"id":"104309","messageId":"200902112352.40955.trast@student.ethz.ch","threadId":"17675","inReplyTo":"7vprhpkgzk.fsf@gitster.siamese.dyndns.org","subject":"Re: [RFC/RFH] Fun things with git-notes, or: patch tracking backwards","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2009-02-11T22:52:36Z","receivedAt":"2009-02-11T22:52:36Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Junio C Hamano wrote:\n> Thomas Rast writes:\n> > Right now it's just the patch-id. Maybe filtering (author,subject)\n> > and then picking the one that is the most similar could work.\n> \n> Yeah, I actually was thinking about matching the (date, author) tuple and\n> nothing else, as it is unlikely you would have dups.\n\nThanks, good idea. I changed the code to parse the required data, and\nwe're now up to 1502 annotations.\n\nUnfortunately I noticed there's a bug in the mail input stage:\nPython's mailbox module assumes any '^From ' line starts a new mail,\nwhile gmane apparently uses a slightly different format based on the\ndouble newlines too (and they always have the same 'From\nnews@gmane.org Tue Mar 04 03:33:20 2003' separator too), and doesn't\nquote '^From ' in the bodies. So any mail containing such body lines\ngot chopped down in the middle, and any patches contained in them\nwon't apply because of the missing headers.\n\nA quick perl run shows that there are 26 mails affected among the\n89000+ mails that I've (again) imported. The fix should be easy, but\nI'm already short on sleep.\n\n> I find the \"Extra-Notes:\" tag a bit too loud, but I am probably a minority\n> who thinks everything but the Message-ID can be dropped, so please don't\n> take it as a feature request ;-)\n\nI refactored the final formatting stage a bit to let it do several\nnotes trees, and you can now take your pick:\n\n git://repo.or.cz/trackgit.git notes/full\n git://repo.or.cz/trackgit.git notes/terse\n\nThe latter only has 'Message-Id' and 'Archived-At'.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"104310","messageId":"200902112358.23763.trast@student.ethz.ch","threadId":"17675","inReplyTo":"200902112352.40955.trast@student.ethz.ch","subject":"Re: [RFC/RFH] Fun things with git-notes, or: patch tracking backwards","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2009-02-11T22:58:20Z","receivedAt":"2009-02-11T22:58:20Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Thomas Rast wrote:\n> I'm already short on sleep.\n[...]\n> git://repo.or.cz/trackgit.git notes/full\n> git://repo.or.cz/trackgit.git notes/terse\n\nQ.E.D.! The correct repository is\n\n git://repo.or.cz/git/trast.git notes/full\n git://repo.or.cz/git/trast.git notes/terse\n\nSorry for the noise.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"}]}