threads / discuss / 62218

Linking topic merges to mailing list threads

Subject: Linking topic merges to mailing list threads

## tl;dr

18 messages between Sep 30, 2024 and Oct 3, 2024.

replies: 17people: 8as markdown or json

Emily Shaffer· Sep 30, 2024, 16:21 UTC · lore
Hi all,

We've been wanting to gather metrics on Git's code review process - how long it takes from first contact on list to merge, how many iterations are needed, time between iterations, etc. One missing link is the actual merge time in `next` and `master` - a human can infer the link between the patch and the mailing list thread, but it's more challenging for a script to do it.

Would it be possible to modify the maintainer workflow to include a link to the cover letter as merged in the merge commit message (or the link to the latest iteration of the patch if it's a single-patch change)? What issues could come up with that workflow?

I guess one is that we could move from lore.kernel.org to something else, like we saw the migration from public-inbox.org some years ago. But the Message-ID was preserved between the two archives, so maybe it's enough to include the Message-ID in the merge commit? Another is, of course, the added burden on the maintainer. But maybe there is some script that is already used that we can modify to make the extra load negligible?

(Or, even better, if anybody else is already successfully measuring these kinds of metrics without such a reference, could you let me know how you're doing it? :) )

Thanks,
 - Emily
Konstantin Ryabitsev· Sep 30, 2024, 16:57 UTC · re: Emily Shaffer · lore

Re: Linking topic merges to mailing list threads

On Mon, Sep 30, 2024 at 09:21:11AM GMT, Emily Shaffer wrote:
Show 13 quoted lines
> Hi all,
> 
> We've been wanting to gather metrics on Git's code review process -
> how long it takes from first contact on list to merge, how many
> iterations are needed, time between iterations, etc. One missing link
> is the actual merge time in `next` and `master` - a human can infer
> the link between the patch and the mailing list thread, but it's more
> challenging for a script to do it.
> 
> Would it be possible to modify the maintainer workflow to include a
> link to the cover letter as merged in the merge commit message (or the
> link to the latest iteration of the patch if it's a single-patch
> change)? What issues could come up with that workflow?

One of the goals of b4 on the kernel side of things was to promote the use of cover letters as merge commit templates, but this requires buy-in from maintainers. It also doesn't really work for single-patch series.

For example, applying a series with "b4 shazam -M" will:
- fetch the series into FETCH_HEAD
- use the cover letter as the basis for the merge commit message
- insert the links to the source of the series
- open up the editor, allowing the maintainer to edit the merge commit message
Here's an example of such merge:
https://git.kernel.org/pub/scm/utils/b4/b4.git/commit/?id=b6b73918d94985bb2a017784fc14e013b36b38d0
> I guess one is that we could move from lore.kernel.org to something
> else, like we saw the migration from public-inbox.org some years ago.
> But the Message-ID was preserved between the two archives, so maybe
> it's enough to include the Message-ID in the merge commit?

This should be sufficient, yes, because you should still be able to find the origin thread even if lore.kernel.org is defunct at some point.

> Another is, of course, the added burden on the maintainer. But maybe there
> is some script that is already used that we can modify to make the extra
> load negligible?
There is. :)
> (Or, even better, if anybody else is already successfully measuring
> these kinds of metrics without such a reference, could you let me know
> how you're doing it? :) )

On the kernel side, any time the topic of metrics comes up, it gets immediately bogged down in "how much tracking is okay and how much is spying" kinds of discussions that have never resulted in anything, really.

-K
Junio C Hamano· Sep 30, 2024, 19:30 UTC · re: Emily Shaffer · lore

Re: Linking topic merges to mailing list threads

Emily Shaffer <nasamuffin@google.com> writes:
Show 8 quoted lines
> We've been wanting to gather metrics on Git's code review process -
> how long it takes from first contact on list to merge, how many
> iterations are needed, time between iterations, etc. One missing link
> is the actual merge time in `next` and `master` - a human can infer
> the link between the patch and the mailing list thread, but it's more
> challenging for a script to do it.
>
> Would it be possible to modify the maintainer workflow ...

I suspect that there is no need for any workflow change, as all the necessary information should be available from public sources.

The first-parent chain from 'next' (or 'master' for that matter) already record when they got merged. From there, C^1..C^2 are the commit objects that were merged. notes/amlog knows where they came from (i.e. their Message-Id). From lore/public-inbox you can find out how the iterations of topics went, as long as the topics are threaded properly (and if not, that would not be fixable with any maintainer workflow changes), just like how b4 can figure all of that out.

Ahh, nothing officially documents amlog and that is what you are missing. It would be very nice if somebody, preferrably somebody other than I, after trying the "maintainer workflow" by pretending to be a maintainer for a day or two with the new info revealed here, updates the Documentation/howto/maintain-git.txt file with the information below.

The script post-appplypatch found in the todo branch is made available as .git/hooks/post-applypatch so that "git am" knows to run it after creating a commit out of an e-mailed patch. It populates a mapping from commit object name to "Message-Id" of individual patch.

"git rebase" knows how to propagate this across rebases because I have

    [notes] rewriteref = refs/notes/amlog

in the .git/config (which means I have to use rebase not cherry-pick even when I am touching a single patch, as cherry-pick does not preserve notes by design).

Now I think you should have everything, together with what is already in Documentation/howto/maintain-git.txt, piece them together to illustrate the life of a patch series.

As I do not publish reflog for 'seen', you cannot do "when was the topic got picked up to 'seen'?", but as far as I am concerned, it is by design. Being in 'seen' does not mean anything other than I happened to have seen it, or saw that somebody indicate interest in it.

Thanks.
Taylor Blau· Sep 30, 2024, 20:45 UTC · re: Junio C Hamano · lore

[PATCH] Documentation: mention the amlog in howto/maintain-git.txt

Part of the maintainer's job is to keep up-to-date and publish the 'amlog' which stores a mapping between a patch's 'Message-Id' e-mail header and the commit generated by applying said patch.

But our Documentation/howto/maintain-git.txt does not mention the amlog, or the scripts which exist to help the maintainer keep the amlog up-to-date.

(This bit me during the first integration round I did as interim maintainer[1] involved a lot of manual clean-up. More recently it has come up as part of a research effort to better understand a patch's lifecycle on the list[2].)

Address this gap by briefly documenting the existence and purpose of the 'post-applypatch' hook in maintaining the amlog entries.

[1]: https://lore.kernel.org/git/Y19dnb2M+yObnftj@nand.local/ [2]: https://lore.kernel.org/git/CAJoAoZ=4ARuH3aHGe5yC_Xcnou_c396q_ZienYPY7YnEzZcyEg@mail.gmail.com/

Suggested-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 Documentation/howto/maintain-git.txt | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
diff --git a/Documentation/howto/maintain-git.txt b/Documentation/howto/maintain-git.txt
index da31332f11..fd1560327c 100644
--- a/Documentation/howto/maintain-git.txt
+++ b/Documentation/howto/maintain-git.txt
@@ -165,6 +165,22 @@ by doing the following:
    In practice, almost no patch directly goes to 'master' or
    'maint'.
 
+   The maintainer is expected to update refs/notes/amlog with a
+   mapping between the applied commit and the 'Message-Id'
+   corresponding to the e-mail which carried the patch.
+
+   This mapping is created with the aid of the "post-applypatch" hook
+   found in the 'todo' branch. That hook should be installed before
+   applying patches. It is also helpful to carry forward any relevant
+   amlog entries when rebasing, so the following config may be useful:
+
+      [notes]
+	rewriteref = refs/notes/amlog
+
+   Finally, take care that the amlog entries are pushed out during
+   integration cycles since external tools and contributors (in
+   addition to internal scripts) may rely on them.
+
  - Review the last issue of "What's cooking" message, review the
    topics ready for merging (topic->master and topic->maint).  Use
    "Meta/cook -w" script (where Meta/ contains a checkout of the

base-commit: 3857aae53f3633b7de63ad640737c657387ae0c6
-- 
2.46.2.633.gf09c3c1769.dirty
Kristoffer Haugsbakk· Sep 30, 2024, 21:06 UTC · re: Taylor Blau · lore

Re: [PATCH] Documentation: mention the amlog in howto/maintain-git.txt

On Mon, Sep 30, 2024, at 22:45, Taylor Blau wrote:
Show 46 quoted lines
> Part of the maintainer's job is to keep up-to-date and publish the
> 'amlog' which stores a mapping between a patch's 'Message-Id' e-mail
> header and the commit generated by applying said patch.
>
> But our Documentation/howto/maintain-git.txt does not mention the amlog,
> or the scripts which exist to help the maintainer keep the amlog
> up-to-date.
>
> (This bit me during the first integration round I did as interim
> maintainer[1] involved a lot of manual clean-up. More recently it has
> come up as part of a research effort to better understand a patch's
> lifecycle on the list[2].)
>
> Address this gap by briefly documenting the existence and purpose of the
> 'post-applypatch' hook in maintaining the amlog entries.
>
> [1]: https://lore.kernel.org/git/Y19dnb2M+yObnftj@nand.local/
> [2]:
> https://lore.kernel.org/git/CAJoAoZ=4ARuH3aHGe5yC_Xcnou_c396q_ZienYPY7YnEzZcyEg@mail.gmail.com/
>
> Suggested-by: Junio C Hamano <gitster@pobox.com>
> Signed-off-by: Taylor Blau <me@ttaylorr.com>
> ---
>  Documentation/howto/maintain-git.txt | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
>
> diff --git a/Documentation/howto/maintain-git.txt
> b/Documentation/howto/maintain-git.txt
> index da31332f11..fd1560327c 100644
> --- a/Documentation/howto/maintain-git.txt
> +++ b/Documentation/howto/maintain-git.txt
> @@ -165,6 +165,22 @@ by doing the following:
>     In practice, almost no patch directly goes to 'master' or
>     'maint'.
>
> +   The maintainer is expected to update refs/notes/amlog with a
> +   mapping between the applied commit and the 'Message-Id'
> +   corresponding to the e-mail which carried the patch.
> +
> +   This mapping is created with the aid of the "post-applypatch" hook
> +   found in the 'todo' branch. That hook should be installed before
> +   applying patches. It is also helpful to carry forward any relevant
> +   amlog entries when rebasing, so the following config may be useful:
> +
> +      [notes]
> +	rewriteref = refs/notes/amlog
Nit: `[notes]` is indented with spaces while the next line is indented
with a tab.  I guess it’s supposed to just be spaces in this context?
Show 12 quoted lines
> +
> +   Finally, take care that the amlog entries are pushed out during
> +   integration cycles since external tools and contributors (in
> +   addition to internal scripts) may rely on them.
> +
>   - Review the last issue of "What's cooking" message, review the
>     topics ready for merging (topic->master and topic->maint).  Use
>     "Meta/cook -w" script (where Meta/ contains a checkout of the
>
> base-commit: 3857aae53f3633b7de63ad640737c657387ae0c6
> --
> 2.46.2.633.gf09c3c1769.dirty

It might be worth explicitly mentioning the git-cherry-pick(1) footgun that Junio talked about in his email: you have to restrict yourself to git-rebase(1) and `git commit --amend`. Since git-cherry-pick(1) doesn’t care about (respect?) this configuration.

Right now it’s implied of course (“when rebasing”).
-- 
Kristoffer Haugsbakk
Taylor Blau· Oct 2, 2024, 15:04 UTC · re: Kristoffer Haugsbakk · lore

Re: [PATCH] Documentation: mention the amlog in howto/maintain-git.txt

On Mon, Sep 30, 2024 at 11:06:03PM +0200, Kristoffer Haugsbakk wrote:
Show 18 quoted lines
> > @@ -165,6 +165,22 @@ by doing the following:
> >     In practice, almost no patch directly goes to 'master' or
> >     'maint'.
> >
> > +   The maintainer is expected to update refs/notes/amlog with a
> > +   mapping between the applied commit and the 'Message-Id'
> > +   corresponding to the e-mail which carried the patch.
> > +
> > +   This mapping is created with the aid of the "post-applypatch" hook
> > +   found in the 'todo' branch. That hook should be installed before
> > +   applying patches. It is also helpful to carry forward any relevant
> > +   amlog entries when rebasing, so the following config may be useful:
> > +
> > +      [notes]
> > +	rewriteref = refs/notes/amlog
>
> Nit: `[notes]` is indented with spaces while the next line is indented
> with a tab.  I guess it’s supposed to just be spaces in this context?
Oops, good catch, thanks.
> It might be worth explicitly mentioning the git-cherry-pick(1) footgun
> that Junio talked about in his email: you have to restrict yourself to
> git-rebase(1) and `git commit --amend`.  Since git-cherry-pick(1)
> doesn’t care about (respect?) this configuration.

I think that's worth mentioning, and I added a small tidbit in the latest round mentioning it, thanks.

Thanks, Taylor

Junio C Hamano· Sep 30, 2024, 21:54 UTC · re: Taylor Blau · lore

Re: [PATCH] Documentation: mention the amlog in howto/maintain-git.txt

Taylor Blau <me@ttaylorr.com> writes:
Show 24 quoted lines
> Part of the maintainer's job is to keep up-to-date and publish the
> 'amlog' which stores a mapping between a patch's 'Message-Id' e-mail
> header and the commit generated by applying said patch.
>
> But our Documentation/howto/maintain-git.txt does not mention the amlog,
> or the scripts which exist to help the maintainer keep the amlog
> up-to-date.
>
> (This bit me during the first integration round I did as interim
> maintainer[1] involved a lot of manual clean-up. More recently it has
> come up as part of a research effort to better understand a patch's
> lifecycle on the list[2].)
>
> Address this gap by briefly documenting the existence and purpose of the
> 'post-applypatch' hook in maintaining the amlog entries.
>
> [1]: https://lore.kernel.org/git/Y19dnb2M+yObnftj@nand.local/
> [2]: https://lore.kernel.org/git/CAJoAoZ=4ARuH3aHGe5yC_Xcnou_c396q_ZienYPY7YnEzZcyEg@mail.gmail.com/
>
> Suggested-by: Junio C Hamano <gitster@pobox.com>
> Signed-off-by: Taylor Blau <me@ttaylorr.com>
> ---
>  Documentation/howto/maintain-git.txt | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
This addition to the procedure part of the documentation reads good.

We'd need a matching addition to "The Policy" part, to describe the motivation separately. The procedure exists only to realize what the policy gives, and we need something to back up the expectation "to update refs/notes/amlog" (i.e. because there is this policy).

Existig "policy" entries are only about how integration branches are maintained and used, but notes/amlog are solely about the individual patches, so we'd need an entirely new section there, I guess.

While at it, I notice that there is no mention on where these notes are published (the configuration you added to the text is solely about the local repository the maintainer uses).

I just added this change
 [remote "github2"]
         url = https://github.com/git/git
         fetch = +refs/heads/*:refs/remotes/github2/*
         pushurl = github.com:git/git.git
         push = refs/heads/maint:refs/heads/maint
         push = refs/heads/master:refs/heads/master
         push = refs/heads/next:refs/heads/next
         push = +refs/heads/seen:refs/heads/seen
+        push = +refs/notes/amlog

to github.com/git/git/ and other publishing repositories. My broken-out repository github.com/gitster/git/ have been pushed with the mirror mode, so there needs no change, but others like k.org repositories will start seeing this additional ref when I push out today's integration results.

The "policy" part of the change may read like the following.
Thanks.
 Documentation/howto/maintain-git.txt | 8 ++++++++
 1 file changed, 8 insertions(+)
diff --git c/Documentation/howto/maintain-git.txt w/Documentation/howto/maintain-git.txt
index da31332f11..9b72d435e6 100644
--- c/Documentation/howto/maintain-git.txt
+++ w/Documentation/howto/maintain-git.txt
@@ -35,6 +35,14 @@ The maintainer's Git time is spent on three activities.
 The Policy
 ----------
 
+Because most of the lines of code in Git are written by individual
+contributors, and contributions come in the form of e-mailed patches
+published on the mailing list, the project maintains a mapping from
+individual commits to the Message-Id of the e-mail that resulted in
+the commit, to help tracking the origin of the changes.  The notes
+in "refs/notes/amlog" are used for this purpose, and are published
+along with the broken-out branches to the maintainer's repository.
+
 The policy on Integration is informally mentioned in "A Note
 from the maintainer" message, which is periodically posted to
 the mailing list after each feature release is made:
Taylor Blau· Oct 2, 2024, 15:05 UTC · re: Junio C Hamano · lore

Re: [PATCH] Documentation: mention the amlog in howto/maintain-git.txt

On Mon, Sep 30, 2024 at 02:54:23PM -0700, Junio C Hamano wrote:
Show 26 quoted lines
> The "policy" part of the change may read like the following.
>
> Thanks.
>
>  Documentation/howto/maintain-git.txt | 8 ++++++++
>  1 file changed, 8 insertions(+)
>
> diff --git c/Documentation/howto/maintain-git.txt w/Documentation/howto/maintain-git.txt
> index da31332f11..9b72d435e6 100644
> --- c/Documentation/howto/maintain-git.txt
> +++ w/Documentation/howto/maintain-git.txt
> @@ -35,6 +35,14 @@ The maintainer's Git time is spent on three activities.
>  The Policy
>  ----------
>
> +Because most of the lines of code in Git are written by individual
> +contributors, and contributions come in the form of e-mailed patches
> +published on the mailing list, the project maintains a mapping from
> +individual commits to the Message-Id of the e-mail that resulted in
> +the commit, to help tracking the origin of the changes.  The notes
> +in "refs/notes/amlog" are used for this purpose, and are published
> +along with the broken-out branches to the maintainer's repository.
> +
>  The policy on Integration is informally mentioned in "A Note
>  from the maintainer" message, which is periodically posted to
>  the mailing list after each feature release is made:

Thanks, this looks good to me, and I added it in the latest version of this patch, with your Helped-by.

I moved this section to the end of the this section, not the beginning, since it seems more important to first discuss the mechanics of topic branches, next, seen, master, etc., before getting to the nuts and bolts of the amlog ;-).

Thanks, Taylor

Taylor Blau· Oct 2, 2024, 15:03 UTC · re: Taylor Blau · lore

[PATCH v2] Documentation: mention the amlog in howto/maintain-git.txt

Part of the maintainer's job is to keep up-to-date and publish the 'amlog' which stores a mapping between a patch's 'Message-Id' e-mail header and the commit generated by applying said patch.

But our Documentation/howto/maintain-git.txt does not mention the amlog, or the scripts which exist to help the maintainer keep the amlog up-to-date.

(This bit me during the first integration round I did as interim maintainer[1] involved a lot of manual clean-up. More recently it has come up as part of a research effort to better understand a patch's lifecycle on the list[2].)

Address this gap by briefly documenting the existence and purpose of the 'post-applypatch' hook in maintaining the amlog entries.

[1]: https://lore.kernel.org/git/Y19dnb2M+yObnftj@nand.local/ [2]: https://lore.kernel.org/git/CAJoAoZ=4ARuH3aHGe5yC_Xcnou_c396q_ZienYPY7YnEzZcyEg@mail.gmail.com/

Suggested-by: Junio C Hamano <gitster@pobox.com>
Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
Similar to v1, but with:
- an added change to "The Policy" section written by Junio
- a tab/space fix in the notes.rewriteRef example
- and a mention of the fact that the notes.rewriteRef configuration is
  not read by 'cherry-pick'.
 Documentation/howto/maintain-git.txt | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
diff --git a/Documentation/howto/maintain-git.txt b/Documentation/howto/maintain-git.txt
index da31332f11..76d6688d4c 100644
--- a/Documentation/howto/maintain-git.txt
+++ b/Documentation/howto/maintain-git.txt
@@ -122,6 +122,13 @@ Note that before v1.9.0 release, the version numbers used to be
 structured slightly differently.  vX.Y.Z were feature releases while
 vX.Y.Z.W were maintenance releases for vX.Y.Z.

+Because most of the lines of code in Git are written by individual
+contributors, and contributions come in the form of e-mailed patches
+published on the mailing list, the project maintains a mapping from
+individual commits to the Message-Id of the e-mail that resulted in
+the commit, to help tracking the origin of the changes. The notes
+in "refs/notes/amlog" are used for this purpose, and are published
+along with the broken-out branches to the maintainer's repository.

 A Typical Git Day
 -----------------
@@ -165,6 +172,24 @@ by doing the following:
    In practice, almost no patch directly goes to 'master' or
    'maint'.

+   The maintainer is expected to update refs/notes/amlog with a
+   mapping between the applied commit and the 'Message-Id'
+   corresponding to the e-mail which carried the patch.
+
+   This mapping is created with the aid of the "post-applypatch" hook
+   found in the 'todo' branch. That hook should be installed before
+   applying patches. It is also helpful to carry forward any relevant
+   amlog entries when rebasing, so the following config may be useful:
+
+      [notes]
+        rewriteRef = refs/notes/amlog
+
+   (note that this configuration is not read by 'cherry-pick').
+
+   Finally, take care that the amlog entries are pushed out during
+   integration cycles since external tools and contributors (in
+   addition to internal scripts) may rely on them.
+
  - Review the last issue of "What's cooking" message, review the
    topics ready for merging (topic->master and topic->maint).  Use
    "Meta/cook -w" script (where Meta/ contains a checkout of the

Range-diff against v1:
1:  a4b1da93e1 ! 1:  5cc8e2bcb8 Documentation: mention the amlog in howto/maintain-git.txt
    @@ Commit message
         [2]: https://lore.kernel.org/git/CAJoAoZ=4ARuH3aHGe5yC_Xcnou_c396q_ZienYPY7YnEzZcyEg@mail.gmail.com/

         Suggested-by: Junio C Hamano <gitster@pobox.com>
    +    Helped-by: Junio C Hamano <gitster@pobox.com>
         Signed-off-by: Taylor Blau <me@ttaylorr.com>

      ## Documentation/howto/maintain-git.txt ##
    +@@ Documentation/howto/maintain-git.txt: Note that before v1.9.0 release, the version numbers used to be
    + structured slightly differently.  vX.Y.Z were feature releases while
    + vX.Y.Z.W were maintenance releases for vX.Y.Z.
    +
    ++Because most of the lines of code in Git are written by individual
    ++contributors, and contributions come in the form of e-mailed patches
    ++published on the mailing list, the project maintains a mapping from
    ++individual commits to the Message-Id of the e-mail that resulted in
    ++the commit, to help tracking the origin of the changes. The notes
    ++in "refs/notes/amlog" are used for this purpose, and are published
    ++along with the broken-out branches to the maintainer's repository.
    +
    + A Typical Git Day
    + -----------------
     @@ Documentation/howto/maintain-git.txt: by doing the following:
         In practice, almost no patch directly goes to 'master' or
         'maint'.
    @@ Documentation/howto/maintain-git.txt: by doing the following:
     +   amlog entries when rebasing, so the following config may be useful:
     +
     +      [notes]
    -+	rewriteref = refs/notes/amlog
    ++        rewriteRef = refs/notes/amlog
    ++
    ++   (note that this configuration is not read by 'cherry-pick').
     +
     +   Finally, take care that the amlog entries are pushed out during
     +   integration cycles since external tools and contributors (in

base-commit: 3857aae53f3633b7de63ad640737c657387ae0c6
--
2.47.0.rc0.4.gd73fb26592.dirty
Junio C Hamano· Oct 2, 2024, 19:47 UTC · re: Taylor Blau · lore

Re: [PATCH v2] Documentation: mention the amlog in howto/maintain-git.txt

Taylor Blau <me@ttaylorr.com> writes:

Now the policy explains what is done (i.e. amlog gives a mapping) and for what purpose (i.e. we want to be able to go back to the origin), "... is expected to" in the actual procedure is redundant. In other words, the procedure section can focus on what is done without repeating why.

> +   The maintainer is expected to update refs/notes/amlog with a
> +   mapping between the applied commit and the 'Message-Id'
> +   corresponding to the e-mail which carried the patch.
I'd replace the above with something like:
      Applying the e-mailed patches using "git am" automatically
      records the mappings from Message-Id to resulting commit in
      the "amlog" note.  Periodically check that this is working
      with "git show -s --notes=amlog $commit".
> +   This mapping is created with the aid of the "post-applypatch" hook
"created" -> "maintained".
Show 8 quoted lines
> +   found in the 'todo' branch. That hook should be installed before
> +   applying patches. It is also helpful to carry forward any relevant
> +   amlog entries when rebasing, so the following config may be useful:
> +
> +      [notes]
> +        rewriteRef = refs/notes/amlog
> +
> +   (note that this configuration is not read by 'cherry-pick').
"(note ...)" ->
      Avoid "cherry-pick", as it does not propagate notes by design.
      Use either "git commit --amend" or "git rebase" to make
      corrections to an existing commit, even for a single-patch
      topic.
> +   Finally, take care that the amlog entries are pushed out during
> +   integration cycles since external tools and contributors (in
> +   addition to internal scripts) may rely on them.
The purpose of pushing amlog out does not need to be repeated here
in the procedure section.
	
      Make sure that push refspec for refs/notes/amlog is in the
      remote configuration for publishing repositories.  A few
      sample .git/config entries may look like this:
        [remote "github2"]
                url = https://github.com/git/git
                fetch = +refs/heads/*:refs/remotes/github2/*
                pushurl = github.com:git/git.git
                push = refs/heads/maint:refs/heads/maint
                push = refs/heads/master:refs/heads/master
                push = refs/heads/next:refs/heads/next
                push = +refs/heads/seen:refs/heads/seen
                push = +refs/notes/amlog
        [remote "github"]
                url = https://github.com/gitster/git
                pushurl = github.com:gitster/git.git
                mirror
Other than that, nicely done.
Thanks for filling the gap in the documentation.
Taylor Blau· Oct 3, 2024, 01:09 UTC · re: Junio C Hamano · lore

Re: [PATCH v2] Documentation: mention the amlog in howto/maintain-git.txt

On Wed, Oct 02, 2024 at 12:47:29PM -0700, Junio C Hamano wrote:
> Other than that, nicely done.
All very fair suggestions. A v3 is on its way...

Thanks, Taylor

Taylor Blau· Oct 3, 2024, 01:09 UTC · re: Taylor Blau · lore

[PATCH v3] Documentation: mention the amlog in howto/maintain-git.txt

Part of the maintainer's job is to keep up-to-date and publish the 'amlog' which stores a mapping between a patch's 'Message-Id' e-mail header and the commit generated by applying said patch.

But our Documentation/howto/maintain-git.txt does not mention the amlog, or the scripts which exist to help the maintainer keep the amlog up-to-date.

(This bit me during the first integration round I did as interim maintainer[1] involved a lot of manual clean-up. More recently it has come up as part of a research effort to better understand a patch's lifecycle on the list[2].)

Address this gap by briefly documenting the existence and purpose of the 'post-applypatch' hook in maintaining the amlog entries.

[1]: https://lore.kernel.org/git/Y19dnb2M+yObnftj@nand.local/ [2]: https://lore.kernel.org/git/CAJoAoZ=4ARuH3aHGe5yC_Xcnou_c396q_ZienYPY7YnEzZcyEg@mail.gmail.com/

Suggested-by: Junio C Hamano <gitster@pobox.com>
Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 Documentation/howto/maintain-git.txt | 44 ++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)
diff --git a/Documentation/howto/maintain-git.txt b/Documentation/howto/maintain-git.txt
index da31332f113..f52f32eda93 100644
--- a/Documentation/howto/maintain-git.txt
+++ b/Documentation/howto/maintain-git.txt
@@ -122,6 +122,13 @@ Note that before v1.9.0 release, the version numbers used to be
 structured slightly differently.  vX.Y.Z were feature releases while
 vX.Y.Z.W were maintenance releases for vX.Y.Z.
 
+Because most of the lines of code in Git are written by individual
+contributors, and contributions come in the form of e-mailed patches
+published on the mailing list, the project maintains a mapping from
+individual commits to the Message-Id of the e-mail that resulted in
+the commit, to help tracking the origin of the changes. The notes
+in "refs/notes/amlog" are used for this purpose, and are published
+along with the broken-out branches to the maintainer's repository.
 
 A Typical Git Day
 -----------------
@@ -165,6 +172,43 @@ by doing the following:
    In practice, almost no patch directly goes to 'master' or
    'maint'.
 
+   Applying the e-mailed patches using "git am" automatically records
+   the mappings from 'Message-Id' to the applied commit in the "amlog"
+   notes. Periodically check that this is working with "git show -s
+   --notes=amlog $commit".
+
+   This mapping is maintained with the aid of the "post-applypatch"
+   hook found in the 'todo' branch. That hook should be installed
+   before applying patches. It is also helpful to carry forward any
+   relevant amlog entries when rebasing, so the following config may
+   be useful:
+
+      [notes]
+        rewriteRef = refs/notes/amlog
+
+   Avoid "cherry-pick", as it does not propagate notes by design. Use
+   either "git commit --amend" or "git rebase" to make corrections to
+   an existing commit, even for a single-patch topic.
+
+   Make sure that a push refspec for 'refs/notes/amlog' is in the
+   remote configuration for publishing repositories. A few sample
+   configurations look like the following:
+
+      [remote "github"]
+        url = https://github.com/gitster/git
+        pushurl = github.com:gitster/git.git
+        mirror
+
+      [remote "github2"]
+        url = https://github.com/git/git
+        fetch = +refs/heads/*:refs/remotes/github2/*
+        pushurl = github.com:git/git.git
+        push = refs/heads/maint:refs/heads/maint
+        push = refs/heads/master:refs/heads/master
+        push = refs/notes/next:refs/notes/next
+        push = +refs/heads/seen:refs/heads/seen
+        push = +refs/notes/amlog
+
  - Review the last issue of "What's cooking" message, review the
    topics ready for merging (topic->master and topic->maint).  Use
    "Meta/cook -w" script (where Meta/ contains a checkout of the

Range-diff against v2:
1:  5cc8e2bcb88 ! 1:  88a13b9f2b6 Documentation: mention the amlog in howto/maintain-git.txt
    @@ Documentation/howto/maintain-git.txt: by doing the following:
         In practice, almost no patch directly goes to 'master' or
         'maint'.
      
    -+   The maintainer is expected to update refs/notes/amlog with a
    -+   mapping between the applied commit and the 'Message-Id'
    -+   corresponding to the e-mail which carried the patch.
    ++   Applying the e-mailed patches using "git am" automatically records
    ++   the mappings from 'Message-Id' to the applied commit in the "amlog"
    ++   notes. Periodically check that this is working with "git show -s
    ++   --notes=amlog $commit".
     +
    -+   This mapping is created with the aid of the "post-applypatch" hook
    -+   found in the 'todo' branch. That hook should be installed before
    -+   applying patches. It is also helpful to carry forward any relevant
    -+   amlog entries when rebasing, so the following config may be useful:
    ++   This mapping is maintained with the aid of the "post-applypatch"
    ++   hook found in the 'todo' branch. That hook should be installed
    ++   before applying patches. It is also helpful to carry forward any
    ++   relevant amlog entries when rebasing, so the following config may
    ++   be useful:
     +
     +      [notes]
     +        rewriteRef = refs/notes/amlog
     +
    -+   (note that this configuration is not read by 'cherry-pick').
    ++   Avoid "cherry-pick", as it does not propagate notes by design. Use
    ++   either "git commit --amend" or "git rebase" to make corrections to
    ++   an existing commit, even for a single-patch topic.
     +
    -+   Finally, take care that the amlog entries are pushed out during
    -+   integration cycles since external tools and contributors (in
    -+   addition to internal scripts) may rely on them.
    ++   Make sure that a push refspec for 'refs/notes/amlog' is in the
    ++   remote configuration for publishing repositories. A few sample
    ++   configurations look like the following:
    ++
    ++      [remote "github"]
    ++        url = https://github.com/gitster/git
    ++        pushurl = github.com:gitster/git.git
    ++        mirror
    ++
    ++      [remote "github2"]
    ++        url = https://github.com/git/git
    ++        fetch = +refs/heads/*:refs/remotes/github2/*
    ++        pushurl = github.com:git/git.git
    ++        push = refs/heads/maint:refs/heads/maint
    ++        push = refs/heads/master:refs/heads/master
    ++        push = refs/notes/next:refs/notes/next
    ++        push = +refs/heads/seen:refs/heads/seen
    ++        push = +refs/notes/amlog
     +
       - Review the last issue of "What's cooking" message, review the
         topics ready for merging (topic->master and topic->maint).  Use

base-commit: 3857aae53f3633b7de63ad640737c657387ae0c6
-- 
2.47.0.rc0.8.g9a975e77790
Junio C Hamano· Oct 3, 2024, 17:23 UTC · re: Taylor Blau · lore

Re: [PATCH v3] Documentation: mention the amlog in howto/maintain-git.txt

Taylor Blau <me@ttaylorr.com> writes:
> (This bit me during the first integration round I did as interim
> maintainer[1] involved a lot of manual clean-up. More recently it has
> come up as part of a research effort to better understand a patch's
> lifecycle on the list[2].)
Thanks.
Ramsay Jones· Oct 3, 2024, 18:32 UTC · re: Taylor Blau · lore

Re: [PATCH v3] Documentation: mention the amlog in howto/maintain-git.txt

On 03/10/2024 02:09, Taylor Blau wrote:
Show 82 quoted lines
> Part of the maintainer's job is to keep up-to-date and publish the
> 'amlog' which stores a mapping between a patch's 'Message-Id' e-mail
> header and the commit generated by applying said patch.
> 
> But our Documentation/howto/maintain-git.txt does not mention the amlog,
> or the scripts which exist to help the maintainer keep the amlog
> up-to-date.
> 
> (This bit me during the first integration round I did as interim
> maintainer[1] involved a lot of manual clean-up. More recently it has
> come up as part of a research effort to better understand a patch's
> lifecycle on the list[2].)
> 
> Address this gap by briefly documenting the existence and purpose of the
> 'post-applypatch' hook in maintaining the amlog entries.
> 
> [1]: https://lore.kernel.org/git/Y19dnb2M+yObnftj@nand.local/
> [2]: https://lore.kernel.org/git/CAJoAoZ=4ARuH3aHGe5yC_Xcnou_c396q_ZienYPY7YnEzZcyEg@mail.gmail.com/
> 
> Suggested-by: Junio C Hamano <gitster@pobox.com>
> Helped-by: Junio C Hamano <gitster@pobox.com>
> Signed-off-by: Taylor Blau <me@ttaylorr.com>
> ---
>  Documentation/howto/maintain-git.txt | 44 ++++++++++++++++++++++++++++
>  1 file changed, 44 insertions(+)
> 
> diff --git a/Documentation/howto/maintain-git.txt b/Documentation/howto/maintain-git.txt
> index da31332f113..f52f32eda93 100644
> --- a/Documentation/howto/maintain-git.txt
> +++ b/Documentation/howto/maintain-git.txt
> @@ -122,6 +122,13 @@ Note that before v1.9.0 release, the version numbers used to be
>  structured slightly differently.  vX.Y.Z were feature releases while
>  vX.Y.Z.W were maintenance releases for vX.Y.Z.
>  
> +Because most of the lines of code in Git are written by individual
> +contributors, and contributions come in the form of e-mailed patches
> +published on the mailing list, the project maintains a mapping from
> +individual commits to the Message-Id of the e-mail that resulted in
> +the commit, to help tracking the origin of the changes. The notes
> +in "refs/notes/amlog" are used for this purpose, and are published
> +along with the broken-out branches to the maintainer's repository.
>  
>  A Typical Git Day
>  -----------------
> @@ -165,6 +172,43 @@ by doing the following:
>     In practice, almost no patch directly goes to 'master' or
>     'maint'.
>  
> +   Applying the e-mailed patches using "git am" automatically records
> +   the mappings from 'Message-Id' to the applied commit in the "amlog"
> +   notes. Periodically check that this is working with "git show -s
> +   --notes=amlog $commit".
> +
> +   This mapping is maintained with the aid of the "post-applypatch"
> +   hook found in the 'todo' branch. That hook should be installed
> +   before applying patches. It is also helpful to carry forward any
> +   relevant amlog entries when rebasing, so the following config may
> +   be useful:
> +
> +      [notes]
> +        rewriteRef = refs/notes/amlog
> +
> +   Avoid "cherry-pick", as it does not propagate notes by design. Use
> +   either "git commit --amend" or "git rebase" to make corrections to
> +   an existing commit, even for a single-patch topic.
> +
> +   Make sure that a push refspec for 'refs/notes/amlog' is in the
> +   remote configuration for publishing repositories. A few sample
> +   configurations look like the following:
> +
> +      [remote "github"]
> +        url = https://github.com/gitster/git
> +        pushurl = github.com:gitster/git.git
> +        mirror
> +
> +      [remote "github2"]
> +        url = https://github.com/git/git
> +        fetch = +refs/heads/*:refs/remotes/github2/*
> +        pushurl = github.com:git/git.git
> +        push = refs/heads/maint:refs/heads/maint
> +        push = refs/heads/master:refs/heads/master
> +        push = refs/notes/next:refs/notes/next
Hmm, s/notes/heads/g perhaps?
Show 7 quoted lines
> +        push = +refs/heads/seen:refs/heads/seen
> +        push = +refs/notes/amlog
> +
>   - Review the last issue of "What's cooking" message, review the
>     topics ready for merging (topic->master and topic->maint).  Use
>     "Meta/cook -w" script (where Meta/ contains a checkout of the
> 

ATB, Ramsay Jones

Taylor Blau· Oct 3, 2024, 18:44 UTC · re: Ramsay Jones · lore

Re: [PATCH v3] Documentation: mention the amlog in howto/maintain-git.txt

On Thu, Oct 03, 2024 at 07:32:04PM +0100, Ramsay Jones wrote:
Show 9 quoted lines
> > +      [remote "github2"]
> > +        url = https://github.com/git/git
> > +        fetch = +refs/heads/*:refs/remotes/github2/*
> > +        pushurl = github.com:git/git.git
> > +        push = refs/heads/maint:refs/heads/maint
> > +        push = refs/heads/master:refs/heads/master
> > +        push = refs/notes/next:refs/notes/next
>
> Hmm, s/notes/heads/g perhaps?

Ugh, serves me right for trying to send this out late my time last night.

Perhaps Junio can tweak this when applying? Otherwise I can send out a new version.

Thanks, Taylor

Kristoffer Haugsbakk· Sep 30, 2024, 20:41 UTC · re: Emily Shaffer · lore

Re: Linking topic merges to mailing list threads

Like Junio explained refs/notes/amlog is a hidden gem for patch provenance.

Show 8 quoted lines
> Hi all,
>
> We've been wanting to gather metrics on Git's code review process -
> how long it takes from first contact on list to merge, how many
> iterations are needed, time between iterations, etc. One missing link
> is the actual merge time in `next` and `master` - a human can infer
> the link between the patch and the mailing list thread, but it's more
> challenging for a script to do it.

Is the starting point the email? I think you could fish out the Message-ID and do a grep inside the notes tree

    git grep --fixed-string --name-only \
        '00a9fe6b7d77c16c9fd6dfe746aacf9068a76942.1726206484.git.ps@pks.im' \
        refs/notes/amlog --
(the resulting hash will need to be cleaned: fanned directory layout[1])

Then try one commit at a time (because there might be unreachable commits from rewrite operations) using git-when-merged(1):[2]

    git when-merged --log 7cd8f1cc6e17af54fb78768c259a615b1ccc0205 next
    git when-merged --log 7cd8f1cc6e17af54fb78768c259a615b1ccc0205 master

† 1: e.g. 7c/d8/f1cc6e17af54fb78768c259a615b1ccc0205 🔗 2: https://github.com/mhagger/git-when-merged

-- 
Kristoffer Haugsbakk
Eric Wong· Oct 2, 2024, 22:50 UTC · re: Emily Shaffer · lore

Re: Linking topic merges to mailing list threads

Emily Shaffer <nasamuffin@google.com> wrote:
Show 8 quoted lines
> Hi all,
> 
> We've been wanting to gather metrics on Git's code review process -
> how long it takes from first contact on list to merge, how many
> iterations are needed, time between iterations, etc. One missing link
> is the actual merge time in `next` and `master` - a human can infer
> the link between the patch and the mailing list thread, but it's more
> challenging for a script to do it.

Searching by commit titles as a phrase against email subject (`s:') can probably make it easy w/o having to look up amlog or explicitly keep track of human-unmemorizable metadata such as Message-IDs.

Example: https://80x24.org/lore/pub/scm/git/git.git/365529e1ea19b44a7a253b780f3ae3a1cb2f081f/s/#merged

...In the "find merged patch emails" textarea. Yeah it's part of a 100% JS-free alternative to cgit, gitweb, etc...

(https://public-inbox.org/meta/20241002223902.4139389-4-e@80x24.org/
 implements the query generation)
Jeff King· Oct 2, 2024, 23:34 UTC · re: Eric Wong · lore

Re: Linking topic merges to mailing list threads

On Wed, Oct 02, 2024 at 10:50:57PM +0000, Eric Wong wrote:
Show 13 quoted lines
> Emily Shaffer <nasamuffin@google.com> wrote:
> > Hi all,
> > 
> > We've been wanting to gather metrics on Git's code review process -
> > how long it takes from first contact on list to merge, how many
> > iterations are needed, time between iterations, etc. One missing link
> > is the actual merge time in `next` and `master` - a human can infer
> > the link between the patch and the mailing list thread, but it's more
> > challenging for a script to do it.
> 
> Searching by commit titles as a phrase against email subject (`s:')
> can probably make it easy w/o having to look up amlog or explicitly
> keep track of human-unmemorizable metadata such as Message-IDs.

I do that a lot myself, but it sometimes get tripped up when people put patches inline, like:

  > blah blah blah
  Yes, good idea. Maybe like this:
  -- >8 --
  foo: frobnicate the bar
  Etc...

In that case you have to do an actual body search. I usually find these with phrase searches in the body text (I'm usually using notmuch, not public-inbox, but I think the same would be true). Of course it may also find more false positives, but usually they're from the same thread anyway.

Very occasionally somebody posts a patch snippet and Junio writes a commit message that never even hits the list, but that's pretty rare these days. :)

-Peff

← back to recent threads