Problem: git Notes not discoverable (+proposed solutions)

## Problem description

When a project has added git Notes for its commits, git by default doesn’t automatically fetch the Notes; so, the Notes aren’t automatically discoverable to contributors who are using “git log” to read the project commit logs — and especially not discoverable to new contributors, or “casual” users of the logs.

A user will see the Notes only if they _already_ know what git Notes are, and know that the project uses Notes, and the user knows how to get them.

But the reality is: most users do not even know what git Notes are, and don’t know how to get them if they exist. So most people end up never seeing them.

## Use-case/requirements

For the Ladybird browser project https://github.com/LadybirdBrowser/ladybird/, Notes for all 62,000+ commits in its repo commit history were recently added.

And same for the MDN Web Docs project https://developer.mozilla.org/en-US/ and https://github.com/mdn/content/ — Notes were very-recently added for all 23,000+ commits in its history. And that case is a relatively high-visibility project with a relatively high percentage of new, first-time contributors submitting patches each month (~33% on average: or ~73 first-time-contributor patches out of ~361 patches overall that get merged there every month).

(Disclosure: In both cases, I’m the person who added all the Notes…)

The Notes added to those projects provide a variety of relevant GitHub URLs for each commit — info that’s generally useful to everybody, and not only useful to say, the project maintainers or the core contributors.

Yet, the projects have no ready way to make the Notes automatically discoverable/gettable by non-maintainers/core-contributors — and especially, to be easily discoverable by new people showing up to the project.

I realize the projects can update their docs to tell people they can manually run “git fetch origin 'refs/notes/*:refs/notes/*'” to fetch Notes — or else do “git config --add remote.origin.fetch '+refs/notes/*:refs/notes/*'”.

But… people don’t always (or often…) read the docs.

## Patching git: Concrete proposed solutions

I’d be 100% happy to do the work of writing a patch to implement a solution (a git behavior change) for this — if I could get confirmation that the git maintainers would actually be open to reviewing such a patch.

As far as what the change would be: I realize this has been brought up before — but it seems the obvious solutions are to “just” change git so:

- Proposed solution #1: git auto-fetches all Notes when a repo is first cloned,
  and then auto re-fetches them again for every “git fetch” or“git pull”.

  I think that auto-fetching-of-Notes would ideally be the _default_ git
  behavior — but short of that, at least a new [notes] _option_ for enabling
  that behavior would help. That would seem somewhat more “approachable” to
  than “git config --add remote.origin.fetch '+refs/notes/*:refs/notes/*'”.

- Proposed solution #2: git checks if a clone lacks Notes vs remote, and emits:

  > Your clone is behind the origin remote by N notes. To fetch the notes
  > from the origin remote, run “git fetch origin 'refs/notes/*:refs/notes/*'”

Either way, I’d be very willing to put work myself into writing up a patch.

## Details

The Notes added to both the Ladybird and MDN projects provide a variety of relevant GitHub URLs for each commit — info generally useful to everybody, and not only useful to say, the project maintainers or core contributors.

For both cases, https://github.com/sideshowbarker/git-gloss/ was used for adding the Notes. That’s a specialized tool I wrote myself, for adding GitHub-related Notes which look like this:

  Author: https://github.com/Jon4t4n 🔰
  Commit: https://github.com/SerenityOS/serenity/commit/9812031a02
  Pull-request: https://github.com/SerenityOS/serenity/pull/20140
  Issue: https://github.com/SerenityOS/serenity/issues/19937
  Reviewed-by: https://github.com/AtkinsSJ ✅
  Reviewed-by: https://github.com/nico

That is: for each commit, the Notes include GitHub metadata URLs showing any GitHub pull-request associated with the commit, along with links to the GitHub profiles of the pull-request reviewers — as well any related GitHub issues — along with including a “canonical” URL for the commit, and a link to the GitHub profile of the commit author

🔰 – indicates this is author’s first commit to the repo ✅ – indicates a review approval

I suspect that over time, a lot of other GitHub-based projects may end up using that tool to add Notes for their project commits — maybe including some high-profile projects.

So, ideally, it’d be great to be able to avoid every project needing to separately document that “git fetch origin 'refs/notes/*:refs/notes/*'” or “git config --add remote.origin.fetch '+refs/notes/*:refs/notes/*'” are what people need to do in order to be able to get the Notes.

Instead, ideally, git by default would automatically fetch the Notes for them. Or, short of that, git would at least emit a message alerting users that Notes exist at the remote, and explaining to users how they can fetch those Notes.

I agree that git-notes is an under-utilized idea. There's a lot of potential to add context where it matters.

sideshowbarker <mike@w3.org> writes:

Best way to determine that in my experience is to just propose some kind of patch -- especially if the actual change is simple even if controversial.

This would certainly be the most turnkey approach -- but what could go wrong here? I can think of at least one potential danger: that your own notes would be wiped out on fetch if you don't remember to push them first. Laying out the risks involved with each approach would help the conversation by showing the effort you've put into the design.

It's my understanding that the git-notes feature is considered a little under-baked to 'turn on' more broadly like this. There are simply too many sharp edges:

- the 'push before fetch' footgun I mentioned above
- merge conflict resolution workflow for the notes themselves
- no 'set-and-forget' way to maintain multiple notes from multiple users

off the top of my head.

This is less controversial than turning it on by default, but IMO if it's not good enough to turn on by default, we shouldn't encourage its use so prevalently.

Very interesting -- my systems use case for git-notes is very similar, albeit for our in-house bug tracker. Clearly it's a good idea! ;-)

I'd love to see git-notes adopted more broadly, but I think there are some real problems to solve first. If others have their own thoughts/ gripes, I'd love to see such a list of deficiencies turned into a design and development plan. I'm no Git maintainer, but I would happy to provide what review I can.

On Mon, Sep 2, 2024 at 6:06 PM Sean Allred <allred.sean@gmail.com> wrote:

I also agree with this.

I had use cases involving notes in the past and this was the biggest dealbreaker. There is not standardized way to fetch notes in such a way that you can perform conflict resolution.

This sort of fits into a bigger problem in that any non-branch refs don't have the equivalent "refs/remotes/<blah>" area to fetch into for comparison, but changing refs/remotes is a big backwards compatibility issue. I considered in the past to add things like refs/remote-notes/, but that also ended up not going very far.

I would love to see some of these problems solved, but unfortunately have not had motivation or time to work on them, as we ended up not using notes. The problems are quite tricky to find suitable solutions and get folks to agree.

sideshowbarker <mike@w3.org> writes:

And even if they did, they wouldn't know how to use them, so not much is lost here.

Quite honestly, a project that uses notes in such a way that it is essential to understand/utilize the history should reexamine its use of notes and try to see if they can make its commits more useful without relying on notes, I would think.

I've seen from time to time people ask "I am thinking of doing this; will a patch be accepted? If so, I'll work on it." before showing any work, and my response always has been:

 (1) We don't know how useful and interesting your contribution would
     be for our audience, until we see it; and

 (2) If you truly believe in your work (find it useful, find writing
     it fun, etc.), that should be incentive enough for you to work
     on it, whether or not the result will land in my tree.  You
     should instead aim for something so brilliant that we would
     come to you begging for your permission to include it in our
     project.

A much more light-weight alternative would be to add an example to the tutorial to tweak the "remote.origin.fetch" refspec so that it will also fetch notes.

But stepping back a bit, none of the above (including your two) may practically be workable unless you limit the source of the notes to the upstream, or something. If you add notes yourself after you clone, and the upstream makes different changes to its notes, reconciling the diverged history of the notes tree would not be so pleasant. As a mechanism for the only publisher to publish auxiliary pieces of information to cloners, notes is a very useful mechanism, but for such a use case to be effective, the project participants must understand when they are supposed to use the notes read-only.

On Tue, Sep 03, 2024 at 02:34:02PM -0700, Junio C Hamano wrote:

The notes could be also used to annotate existing upstream history without altering it.

However, the problems with notes worflows make it impractical.

Thanks

Michal

Michal Suchánek <msuchanek@suse.de> writes:

Indeed. As a mechanism for the only publisher to publish auxiliary pieces of information to cloners, notes is a very useful mechanism, but for such a use case to be effective, the project participants must understand when they are supposed to use the notes read-only.

On Tue, Sep 3, 2024, at 23:34, Junio C Hamano wrote:

The given example is about them appearing in the Git log. If the notes are autofetched and there are notes in the default namespace (`commits`) then they will see them in the log. In turn they are using them (seeing them) without having to do anything themselves.

The two examples seem to be about adding Notes in bulk to established projects. Maybe this information would have been part of the commit message if they had the necessary foresight.

(Or maybe not: I wouldn’t add all relevant GitHub metadata to the commit messages)

The proposal seems very in line with downstream participants who don’t know how to use Git Notes. The downstream participants don’t have to do anything: the default note just appears in their logs because it is auto-fetched. Their only concern is whether the notes are informative or noisy. They don’t have to care what Notes are.

The participants don’t have to know how to use Notes (read-only): they are just there.

Then if they get curious and make notes themselves in the same namespace? That’s their own fault.

I don’t need a printer and my own pins to read a bulletin board.