threads / discuss / 48219

commit -> public-inbox link helper

Subject: commit -> public-inbox link helper

## tl;dr

7 messages between Apr 4, 2018 and Apr 20, 2018.

replies: 6people: 4as markdown or json

Johannes Schindelin· Apr 4, 2018, 16:35 UTC · lore
Hi team,

I found myself in dear need to quickly look up mails in the public-inbox mail archive corresponding to any given commit in git.git. Some time ago, I wrote a shell script to help me with that, and I found myself using it a couple of times, so I think it might be useful for others, too.

This script (I call it lookup-commit.sh) needs to be dropped into a *bare* clone of http://public-inbox.org/git, and called with its absolute or relative path from a git.git worktree, e.g.

	~/public-inbox-git.git/lookup-commit.sh \
		fea16b47b603e7e4fa7fca198bd49229c0e5da3d

This will take a while initially, or when the `master` branch of the public-inbox mirror was updated, as it will generate two files with plain-text mappings.

In its default mode, it will print the Message-ID of the mail (if found).

With --open, it opens the mail in a browser (macOS support is missing, mainly because I cannot test: just add an `open` alternative to `xdg-open`).

With --reply, it puts the mail into the `from-public-inbox` folder of a maildir-formatted ~/Mail/, so it is pretty specific to my setup here.

Note: the way mails are matched is by timestamp. In practice, this works
amazingly often (although not always, I reported findings short after
GitMerge 2017). My plan was to work on this when/as needed.

Note also: the script is very much in a 'works-for-me' state, and I could imagine that others might want to modify it to their needs. I would be delighted if somebody would take custody of it (as in: start a repository on GitHub, adding a README.md, adding a config setting to point to the location of the public-inbox mirror without having to copy the script, adding an option to install an alias to run the script, etc).

And now, without further ado, here it is, the script, in its current glory:

-- snipsnap -- #!/bin/sh

# This is a very simple helper to assist with finding the mail (if any) # corresponding to a given commit in git.git.

die () {
	echo "$*" >&2
	exit 1
}

mode=print while case "$1" in --open) mode=open;; --reply) mode=reply;; -*) die "Unknown option: $1";; *) break;; esac; do shift; done

test $# = 1 || die "Usage: $0 ( [--open] | [--reply] ) <commit>"

test reply != $mode || test -d "$HOME/Mail" || die "Need $HOME/Mail to reply"

commit="$1" tae="$(git show -s --format='%at %an <%ae>' "$1")" || die "Could not get Timestamp/Author/Email triplet from $1"

# We try to match the timestamp first; the author name and author email are # not as reliable: they might have been overridden via a "From:" line in the # mail's body timestamp=${tae%% *}

cd "$(dirname "$0")" || die "Could not cd to the public-inbox directory"

git rev-parse --quiet --verify \
	b60d038730d2c2bb8ab2b48c117db917ad529cf7 >/dev/null 2>&1 ||
die "Not a public-inbox directory: $(pwd)"

head="$(git rev-parse --verify master)" || die "Could not determine tip of master"

prevhead= test ! -f map.latest-rev || prevhead="$(cat map.latest-rev)"

if test $head != "$prevhead"
then
	range=${prevhead:+$prevhead..}$head
	echo "Inserting records for $range" >&2
	git log --format="%at %h %an <%ae>" $range >map.txt.add ||
	die "Could not enumerate $range"
	cat map.txt map.txt.add 2>/dev/null | sort -n >map.txt.new &&
	mv -f map.txt.new map.txt ||
	die "Could not insert new records"
	echo $head >map.latest-rev
fi
lines="$(grep "^$timestamp " <map.txt)"
if test 1 != $(echo "$lines" | wc -l)
then
	test -n "$lines" ||
	die "No records found for timestamp $timestamp"
	echo "Multiple records found:"
	for h in $(echo "$lines" | cut -d ' ' -f 2)
	do
		git show -s --format="%nOn %ad, %an <%ae> sent" $h
		git show $h |
		sed -n -e 's/^+Message-Id: <\(.*\)>/\1/ip' \
			-e 's/^+Subject: //ip'
	done
	exit 1
fi

# We found exactly one record: print the message ID h=${lines#$timestamp } h=${h%% *} messageid="$(git show $h | sed -n 's/^+Message-Id: <\(.*\)>/\1/ip')" || die "Could not determine Message-Id from $h"

case $mode in
print) echo $messageid;;
open)
	url=https://public-inbox.org/git/$messageid
	case "$(uname -s)" in
	Linux) xdg-open "$url";;
	MINGW*|MSYS*) start "$url";;
	*) die "Need to learn how to open URLs on $(uname -s)";;
	esac
	;;
reply)
	mkdir -p "$HOME/Mail/from-public-inbox/new" &&
	mkdir -p "$HOME/Mail/from-public-inbox/cur" &&
	mkdir -p "$HOME/Mail/from-public-inbox/tmp" ||
	die "Could not set up mail folder 'from-public-inbox'"
	path=$(git diff --name-only $h^!) &&
	mail="$(printf "%s_%09d.%s:2," $(date +%s.%N) $$ $(hostname -f))"
&&
	git show $h:$path >"$HOME/Mail/from-public-inbox/new/$mail" ||
	die "Could not write mail"
	;;
*)
	die "Unhandled mode: $mode"
	;;
esac
Jeff King· Apr 4, 2018, 18:36 UTC · re: Johannes Schindelin · lore

Re: commit -> public-inbox link helper

On Wed, Apr 04, 2018 at 06:35:59PM +0200, Johannes Schindelin wrote:
Show 17 quoted lines
> Hi team,
> 
> I found myself in dear need to quickly look up mails in the public-inbox
> mail archive corresponding to any given commit in git.git. Some time ago,
> I wrote a shell script to help me with that, and I found myself using it a
> couple of times, so I think it might be useful for others, too.
> 
> This script (I call it lookup-commit.sh) needs to be dropped into a *bare*
> clone of http://public-inbox.org/git, and called with its absolute or
> relative path from a git.git worktree, e.g.
> 
> 	~/public-inbox-git.git/lookup-commit.sh \
> 		fea16b47b603e7e4fa7fca198bd49229c0e5da3d
> 
> This will take a while initially, or when the `master` branch of the
> public-inbox mirror was updated, as it will generate two files with
> plain-text mappings.

Junio publishes a git-notes ref with the mapping you want. So you can do:

  git fetch git://github.com/gitster/git.git refs/notes/amlog:refs/notes/amlog
  mid=$(git notes --ref amlog show $commit | perl -lne '/<(.*)>/ and print $1')
  echo "https://public-inbox.org/git/$mid"

without having to download the gigantic list archive repo at all (though I do keep my own copy of the archive and index it with mairix, so I can use "mairix -t m:$mid" and then view the whole thing locally in mutt).

-Peff
Johannes Schindelin· Apr 4, 2018, 20:59 UTC · re: Jeff King · lore

Re: commit -> public-inbox link helper

Hi Peff,
On Wed, 4 Apr 2018, Jeff King wrote:
Show 28 quoted lines
> On Wed, Apr 04, 2018 at 06:35:59PM +0200, Johannes Schindelin wrote:
> 
> > I found myself in dear need to quickly look up mails in the public-inbox
> > mail archive corresponding to any given commit in git.git. Some time ago,
> > I wrote a shell script to help me with that, and I found myself using it a
> > couple of times, so I think it might be useful for others, too.
> > 
> > This script (I call it lookup-commit.sh) needs to be dropped into a *bare*
> > clone of http://public-inbox.org/git, and called with its absolute or
> > relative path from a git.git worktree, e.g.
> > 
> > 	~/public-inbox-git.git/lookup-commit.sh \
> > 		fea16b47b603e7e4fa7fca198bd49229c0e5da3d
> > 
> > This will take a while initially, or when the `master` branch of the
> > public-inbox mirror was updated, as it will generate two files with
> > plain-text mappings.
> 
> Junio publishes a git-notes ref with the mapping you want. So you can
> do:
> 
>   git fetch git://github.com/gitster/git.git refs/notes/amlog:refs/notes/amlog
>   mid=$(git notes --ref amlog show $commit | perl -lne '/<(.*)>/ and print $1')
>   echo "https://public-inbox.org/git/$mid"
> 
> without having to download the gigantic list archive repo at all (though
> I do keep my own copy of the archive and index it with mairix, so I can
> use "mairix -t m:$mid" and then view the whole thing locally in mutt).
Good to know! Thanks for the script.

And thanks also for the `--ref` trick: I had a look at the man page of git-notes, and it was not immediately obvious that it supports options before the sub-subcommand. The `--ref` description is buried pretty deep in there.

Thanks, Dscho

Mike Rappazzo· Apr 4, 2018, 19:58 UTC · re: Johannes Schindelin · lore

Re: commit -> public-inbox link helper

On Wed, Apr 4, 2018 at 12:35 PM, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

Show 135 quoted lines
> Hi team,
>
> I found myself in dear need to quickly look up mails in the public-inbox
> mail archive corresponding to any given commit in git.git. Some time ago,
> I wrote a shell script to help me with that, and I found myself using it a
> couple of times, so I think it might be useful for others, too.
>
> This script (I call it lookup-commit.sh) needs to be dropped into a *bare*
> clone of http://public-inbox.org/git, and called with its absolute or
> relative path from a git.git worktree, e.g.
>
>         ~/public-inbox-git.git/lookup-commit.sh \
>                 fea16b47b603e7e4fa7fca198bd49229c0e5da3d
>
> This will take a while initially, or when the `master` branch of the
> public-inbox mirror was updated, as it will generate two files with
> plain-text mappings.
>
> In its default mode, it will print the Message-ID of the mail (if found).
>
> With --open, it opens the mail in a browser (macOS support is missing,
> mainly because I cannot test: just add an `open` alternative to
> `xdg-open`).
>
> With --reply, it puts the mail into the `from-public-inbox` folder of a
> maildir-formatted ~/Mail/, so it is pretty specific to my setup here.
>
> Note: the way mails are matched is by timestamp. In practice, this works
> amazingly often (although not always, I reported findings short after
> GitMerge 2017). My plan was to work on this when/as needed.
>
> Note also: the script is very much in a 'works-for-me' state, and I could
> imagine that others might want to modify it to their needs. I would be
> delighted if somebody would take custody of it (as in: start a repository
> on GitHub, adding a README.md, adding a config setting to point to the
> location of the public-inbox mirror without having to copy the script,
> adding an option to install an alias to run the script, etc).
>
> And now, without further ado, here it is, the script, in its current glory:
>
> -- snipsnap --
> #!/bin/sh
>
> # This is a very simple helper to assist with finding the mail (if any)
> # corresponding to a given commit in git.git.
>
> die () {
>         echo "$*" >&2
>         exit 1
> }
>
> mode=print
> while case "$1" in
> --open) mode=open;;
> --reply) mode=reply;;
> -*) die "Unknown option: $1";;
> *) break;;
> esac; do shift; done
>
> test $# = 1 ||
> die "Usage: $0 ( [--open] | [--reply] ) <commit>"
>
> test reply != $mode ||
> test -d "$HOME/Mail" ||
> die "Need $HOME/Mail to reply"
>
> commit="$1"
> tae="$(git show -s --format='%at %an <%ae>' "$1")" ||
> die "Could not get Timestamp/Author/Email triplet from $1"
>
> # We try to match the timestamp first; the author name and author email are
> # not as reliable: they might have been overridden via a "From:" line in the
> # mail's body
> timestamp=${tae%% *}
>
> cd "$(dirname "$0")" ||
> die "Could not cd to the public-inbox directory"
>
> git rev-parse --quiet --verify \
>         b60d038730d2c2bb8ab2b48c117db917ad529cf7 >/dev/null 2>&1 ||
> die "Not a public-inbox directory: $(pwd)"
>
> head="$(git rev-parse --verify master)" ||
> die "Could not determine tip of master"
>
> prevhead=
> test ! -f map.latest-rev ||
> prevhead="$(cat map.latest-rev)"
>
> if test $head != "$prevhead"
> then
>         range=${prevhead:+$prevhead..}$head
>         echo "Inserting records for $range" >&2
>         git log --format="%at %h %an <%ae>" $range >map.txt.add ||
>         die "Could not enumerate $range"
>
>         cat map.txt map.txt.add 2>/dev/null | sort -n >map.txt.new &&
>         mv -f map.txt.new map.txt ||
>         die "Could not insert new records"
>
>         echo $head >map.latest-rev
> fi
>
> lines="$(grep "^$timestamp " <map.txt)"
> if test 1 != $(echo "$lines" | wc -l)
> then
>         test -n "$lines" ||
>         die "No records found for timestamp $timestamp"
>
>         echo "Multiple records found:"
>
>         for h in $(echo "$lines" | cut -d ' ' -f 2)
>         do
>                 git show -s --format="%nOn %ad, %an <%ae> sent" $h
>                 git show $h |
>                 sed -n -e 's/^+Message-Id: <\(.*\)>/\1/ip' \
>                         -e 's/^+Subject: //ip'
>         done
>
>         exit 1
> fi
>
> # We found exactly one record: print the message ID
> h=${lines#$timestamp }
> h=${h%% *}
> messageid="$(git show $h | sed -n 's/^+Message-Id: <\(.*\)>/\1/ip')" ||
> die "Could not determine Message-Id from $h"
>
> case $mode in
> print) echo $messageid;;
> open)
>         url=https://public-inbox.org/git/$messageid
>         case "$(uname -s)" in
>         Linux) xdg-open "$url";;
>         MINGW*|MSYS*) start "$url";;
         Darwin*) open "$url";;
Show 19 quoted lines
>         *) die "Need to learn how to open URLs on $(uname -s)";;
>         esac
>         ;;
> reply)
>         mkdir -p "$HOME/Mail/from-public-inbox/new" &&
>         mkdir -p "$HOME/Mail/from-public-inbox/cur" &&
>         mkdir -p "$HOME/Mail/from-public-inbox/tmp" ||
>         die "Could not set up mail folder 'from-public-inbox'"
>
>         path=$(git diff --name-only $h^!) &&
>         mail="$(printf "%s_%09d.%s:2," $(date +%s.%N) $$ $(hostname -f))"
> &&
>         git show $h:$path >"$HOME/Mail/from-public-inbox/new/$mail" ||
>         die "Could not write mail"
>         ;;
> *)
>         die "Unhandled mode: $mode"
>         ;;
> esac
Johannes Schindelin· Apr 4, 2018, 21:02 UTC · re: Mike Rappazzo · lore

Re: commit -> public-inbox link helper

Hi Mike,
as I said here:
On Wed, 4 Apr 2018, Mike Rappazzo wrote:
Show 17 quoted lines
> On Wed, Apr 4, 2018 at 12:35 PM, Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
> >
> > [...]
> >
> > With --open, it opens the mail in a browser (macOS support is missing,
> > mainly because I cannot test: just add an `open` alternative to
> > `xdg-open`).
> >
> > [...]
> > open)
> >         url=https://public-inbox.org/git/$messageid
> >         case "$(uname -s)" in
> >         Linux) xdg-open "$url";;
> >         MINGW*|MSYS*) start "$url";;
> 
>          Darwin*) open "$url";;

I am aware of this alternative, but as I do not currently develop on macOS apart from headless build agents, I did not add support for that.

Feel free to adopt the script, publish it in a GitHub repository, adapt it to use refs/notes/amlog instead of a public-inbox mirror, and then tell me where I can clone it ;-)

Ciao, Johannes

Eric Wong· Apr 20, 2018, 08:39 UTC · re: Johannes Schindelin · lore

Re: commit -> public-inbox link helper

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
Show 6 quoted lines
> Hi team,
> 
> I found myself in dear need to quickly look up mails in the public-inbox
> mail archive corresponding to any given commit in git.git. Some time ago,
> I wrote a shell script to help me with that, and I found myself using it a
> couple of times, so I think it might be useful for others, too.

Hello, I think you can dump all the info you need more quickly without cloning 1G of data by dumping NNTP OVER(view) information instead.

I've attached a short Perl script which dumps the tab-delimited file to stdout so you can process it with whatever. Columns relevant to you would probably be 2-5:

	NUM	Subject	From	Date	Message-ID

On public-inbox-nntpd, Dates are normalized to UTC in the OVER response right now, so you'd need to use TZ=UTC with --date=rfc-local

It works with both nntp://news.gmane.org/gmane.comp.version-control.git nntp://news.public-inbox.org/inbox.comp.version-control.git

> Note: the way mails are matched is by timestamp. In practice, this works
> amazingly often (although not always, I reported findings short after
> GitMerge 2017). My plan was to work on this when/as needed.

Thanks for that. I've added dt: (date-time) searching to public-inbox (d: (date-only) has been there forever):

        d:       date range as YYYYMMDD  e.g. d:19931002..20101002
                 Open-ended ranges such as d:19931002.. and d:..20101002
                 are also supported
        dt:      date-time range as YYYYMMDDhhmmss (e.g. dt:19931002011000..19931002011200)

To match an exact timestamp, both the begining and end range should be the same.

(It'd be nice if Xapian + Perl bindings could get date parsing as
 good as git's.  Too bad C++ / XS overwhelms my tiny brain and
 waiting for builds overwhelm my patience)
> git rev-parse --quiet --verify \
> 	b60d038730d2c2bb8ab2b48c117db917ad529cf7 >/dev/null 2>&1 ||
> die "Not a public-inbox directory: $(pwd)"

Eep. I'd don't think it's good to put such a hard dependency on a particular mirror I started. Somebody could start another one which wasn't sourced from gmane and the Received: headers would be different.

...And I'm pondering a conversion of what's running on https://public-inbox.org/git/ to the new v2 repository format:

	https://public-inbox.org/meta/20180419015813.GA20051@dcvr/
	https://public-inbox.org/meta/20180209205140.GA11047@dcvr/
	https://public-inbox.org/meta/20180215105509.GA22409@dcvr/

#!/usr/bin/perl -w use strict; use warnings; use IO::Socket::INET; my $usage = "$0 news://example.com/group.name [MIN] [MAX]\n"; my $url = shift or die $usage; my $umin = shift; my $umax = shift; my ($host, $port, $group) = ($url =~ m!://([^/]+)?(?::(\d+))?/(.+)!); $port ||= 119; defined $group or die "missing group in $url\n"; my %opts = ( Proto => 'tcp', PeerHost => $host, PeerPort => $port ); my $s = IO::Socket::INET->new(%opts) or die "connect to $host:$port: $!\n"; my $l = $s->getline; $l =~ /\A2\d\d / or die "bad greeting: $l\n";

$s->print("GROUP $group\n") or die "print $!"; $l = $s->getline; $l =~ /\A211 \d+ (\d+) (\d+) / or die "bad GROUP response: $l\n"; my ($min, $max) = ($1, $2); $min = $umin if $umin; $max = $umax if $umax;

$s->print("OVER $min-$max\n") or die "print $!"; $l = $s->getline; $l =~ /\A224 / or die "bad OVER response: $l\n";

while ($l = $s->getline) {
	last if $l eq ".\r\n";
	print $l;
}
Johannes Schindelin· Apr 20, 2018, 19:40 UTC · re: Eric Wong · lore

Re: commit -> public-inbox link helper

Hi Eric,
On Fri, 20 Apr 2018, Eric Wong wrote:
Show 11 quoted lines
> Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > 
> > I found myself in dear need to quickly look up mails in the
> > public-inbox mail archive corresponding to any given commit in
> > git.git. Some time ago, I wrote a shell script to help me with that,
> > and I found myself using it a couple of times, so I think it might be
> > useful for others, too.
> 
> Hello, I think you can dump all the info you need more quickly
> without cloning 1G of data by dumping NNTP OVER(view)
> information instead.
That might be true for the current state of affairs.

However, there *are* cases (I think I even linked to my original mail with my post-GitMerge 2017 analysis) where the triplet Date/Author/Email is not enough, where even some patch series have the identical triplet for every single patch.

Even if I did not hit those cases yet, and therefore did not implement that part, I needed to keep the door open for that. So I need a clone.

Ciao, Dscho

← back to recent threads