{"thread":{"id":"61827","subject":"Problem: git Notes not discoverable (+proposed solutions)","startedAt":"2024-07-23T05:20:40Z","lastAt":"2024-09-04T20:03:20Z","messageCount":7,"participants":["sideshowbarker","Sean Allred","Junio C Hamano","Michal Suchánek","Kristoffer Haugsbakk","Jacob Keller"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"499124","messageId":"Zp89ntYaeFUumaTO@w3.org","threadId":"61827","inReplyTo":null,"subject":"Problem: git Notes not discoverable (+proposed solutions)","fromName":"sideshowbarker","fromEmail":"mike@w3.org","sentAt":"2024-07-23T05:20:30Z","receivedAt":"2024-07-23T05:20:40Z","isPatch":false,"sender":{"key":"mike@w3.org","avatar":null},"body":"## Problem description\n\nWhen a project has added git Notes for its commits, git by default doesn’t\nautomatically fetch the Notes; so, the Notes aren’t automatically discoverable\nto contributors who are using “git log” to read the project commit logs — and\nespecially not discoverable to new contributors, or “casual” users of the logs.\n\nA user will see the Notes only if they _already_ know what git Notes are, and\nknow that the project uses Notes, and the user knows how to get them.\n\nBut the reality is: most users do not even know what git Notes are, and don’t\nknow how to get them if they exist. So most people end up never seeing them.\n\n## Use-case/requirements\n\nFor the Ladybird browser project https://github.com/LadybirdBrowser/ladybird/,\nNotes for all 62,000+ commits in its repo commit history were recently added.\n\nAnd same for the MDN Web Docs project https://developer.mozilla.org/en-US/\nand https://github.com/mdn/content/ — Notes were very-recently added for all\n23,000+ commits in its history. And that case is a relatively high-visibility\nproject with a relatively high percentage of new, first-time contributors\nsubmitting patches each month (~33% on average: or ~73 first-time-contributor\npatches out of ~361 patches overall that get merged there every month).\n\n(Disclosure: In both cases, I’m the person who added all the Notes…)\n\nThe Notes added to those projects provide a variety of relevant GitHub URLs\nfor each commit — info that’s generally useful to everybody, and not only\nuseful to say, the project maintainers or the core contributors.\n\nYet, the projects have no ready way to make the Notes automatically\ndiscoverable/gettable by non-maintainers/core-contributors — and especially,\nto be easily discoverable by new people showing up to the project.\n\nI realize the projects can update their docs to tell people they can manually\nrun “git fetch origin 'refs/notes/*:refs/notes/*'” to fetch Notes — or else do\n“git config --add remote.origin.fetch '+refs/notes/*:refs/notes/*'”.\n\nBut… people don’t always (or often…) read the docs.\n\n## Patching git: Concrete proposed solutions\n\nI’d be 100% happy to do the work of writing a patch to implement a solution\n(a git behavior change) for this — if I could get confirmation that the git\nmaintainers would actually be open to reviewing such a patch.\n\nAs far as what the change would be: I realize this has been brought up\nbefore — but it seems the obvious solutions are to “just” change git so:\n\n- Proposed solution #1: git auto-fetches all Notes when a repo is first cloned,\n  and then auto re-fetches them again for every “git fetch” or“git pull”.\n\n  I think that auto-fetching-of-Notes would ideally be the _default_ git\n  behavior — but short of that, at least a new [notes] _option_ for enabling\n  that behavior would help. That would seem somewhat more “approachable” to\n  than “git config --add remote.origin.fetch '+refs/notes/*:refs/notes/*'”.\n\n- Proposed solution #2: git checks if a clone lacks Notes vs remote, and emits:\n\n  > Your clone is behind the origin remote by N notes. To fetch the notes\n  > from the origin remote, run “git fetch origin 'refs/notes/*:refs/notes/*'”\n\nEither way, I’d be very willing to put work myself into writing up a patch.\n\n## Details\n\nThe Notes added to both the Ladybird and MDN projects provide a variety of\nrelevant GitHub URLs for each commit — info generally useful to everybody,\nand not only useful to say, the project maintainers or core contributors.\n\nFor both cases, https://github.com/sideshowbarker/git-gloss/ was used for\nadding the Notes. That’s a specialized tool I wrote myself, for adding\nGitHub-related Notes which look like this:\n\n  Author: https://github.com/Jon4t4n 🔰\n  Commit: https://github.com/SerenityOS/serenity/commit/9812031a02\n  Pull-request: https://github.com/SerenityOS/serenity/pull/20140\n  Issue: https://github.com/SerenityOS/serenity/issues/19937\n  Reviewed-by: https://github.com/AtkinsSJ ✅\n  Reviewed-by: https://github.com/nico\n\nThat is: for each commit, the Notes include GitHub metadata URLs showing any\nGitHub pull-request associated with the commit, along with links to the GitHub\nprofiles of the pull-request reviewers — as well any related GitHub issues —\nalong with including a “canonical” URL for the commit, and a link to the\nGitHub profile of the commit author\n\n🔰 – indicates this is author’s first commit to the repo\n✅ – indicates a review approval\n\nI suspect that over time, a lot of other GitHub-based projects may end up\nusing that tool to add Notes for their project commits — maybe including some\nhigh-profile projects.\n\nSo, ideally, it’d be great to be able to avoid every project needing to\nseparately document that “git fetch origin 'refs/notes/*:refs/notes/*'” or\n“git config --add remote.origin.fetch '+refs/notes/*:refs/notes/*'” are what\npeople need to do in order to be able to get the Notes.\n\nInstead, ideally, git by default would automatically fetch the Notes for them.\nOr, short of that, git would at least emit a message alerting users that Notes\nexist at the remote, and explaining to users how they can fetch those Notes.\n\n-- \nhttps://sideshowbarker.github.io/w3c-faq/\n"},{"id":"501995","messageId":"m0wmjto8aq.fsf@epic96565.epic.com","threadId":"61827","inReplyTo":"Zp89ntYaeFUumaTO@w3.org","subject":"Re: Problem: git Notes not discoverable (+proposed solutions)","fromName":"Sean Allred","fromEmail":"allred.sean@gmail.com","sentAt":"2024-09-03T01:06:05Z","receivedAt":"2024-09-03T01:06:07Z","isPatch":false,"sender":{"key":"allred.sean@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2082195?v=4"},"body":"I agree that git-notes is an under-utilized idea. There's a lot of\npotential to add context where it matters.\n\nsideshowbarker <mike@w3.org> writes:\n> I’d be 100% happy to do the work of writing a patch to implement a solution\n> (a git behavior change) for this — if I could get confirmation that the git\n> maintainers would actually be open to reviewing such a patch.\n\nBest way to determine that in my experience is to just propose some kind\nof patch -- especially if the actual change is simple even if\ncontroversial.\n\n> As far as what the change would be: I realize this has been brought up\n> before — but it seems the obvious solutions are to “just” change git so:\n>\n> - Proposed solution #1: git auto-fetches all Notes when a repo is first cloned,\n>   and then auto re-fetches them again for every “git fetch” or“git pull”.\n>\n>   I think that auto-fetching-of-Notes would ideally be the _default_ git\n>   behavior — but short of that, at least a new [notes] _option_ for enabling\n>   that behavior would help. That would seem somewhat more “approachable” to\n>   than “git config --add remote.origin.fetch '+refs/notes/*:refs/notes/*'”.\n\nThis would certainly be the most turnkey approach -- but what could go\nwrong here? I can think of at least one potential danger: that your own\nnotes would be wiped out on fetch if you don't remember to push them\nfirst. Laying out the risks involved with each approach would help the\nconversation by showing the effort you've put into the design.\n\nIt's my understanding that the git-notes feature is considered a little\nunder-baked to 'turn on' more broadly like this. There are simply too\nmany sharp edges:\n\n- the 'push before fetch' footgun I mentioned above\n- merge conflict resolution workflow for the notes themselves\n- no 'set-and-forget' way to maintain multiple notes from multiple users\n\noff the top of my head.\n\n> - Proposed solution #2: git checks if a clone lacks Notes vs remote, and emits:\n>\n>   > Your clone is behind the origin remote by N notes. To fetch the notes\n>   > from the origin remote, run “git fetch origin 'refs/notes/*:refs/notes/*'”\n\nThis is less controversial than turning it on by default, but IMO if\nit's not good enough to turn on by default, we shouldn't encourage its\nuse so prevalently.\n\n> For both cases, https://github.com/sideshowbarker/git-gloss/ was used for\n> adding the Notes. That’s a specialized tool I wrote myself, for adding\n> GitHub-related Notes which look like this:\n>\n>   Author: https://github.com/Jon4t4n 🔰\n>   Commit: https://github.com/SerenityOS/serenity/commit/9812031a02\n>   Pull-request: https://github.com/SerenityOS/serenity/pull/20140\n>   Issue: https://github.com/SerenityOS/serenity/issues/19937\n>   Reviewed-by: https://github.com/AtkinsSJ ✅\n>   Reviewed-by: https://github.com/nico\n\nVery interesting -- my systems use case for git-notes is very similar,\nalbeit for our in-house bug tracker. Clearly it's a good idea! ;-)\n\nI'd love to see git-notes adopted more broadly, but I think there are\nsome real problems to solve first. If others have their own thoughts/\ngripes, I'd love to see such a list of deficiencies turned into a design\nand development plan. I'm no Git maintainer, but I would happy to\nprovide what review I can.\n\n-- \nSean Allred\n\nP.S. Sorry for the duplicate email, sideshowbarker; mu4e not-so-recently\nchanged its default keybindings and I have STILL not recovered from\nthis, apparently.\n"},{"id":"502070","messageId":"xmqq7cbsh16d.fsf@gitster.g","threadId":"61827","inReplyTo":"Zp89ntYaeFUumaTO@w3.org","subject":"Re: Problem: git Notes not discoverable (+proposed solutions)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-09-03T21:34:02Z","receivedAt":"2024-09-03T21:34:06Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"sideshowbarker <mike@w3.org> writes:\n\n> ## Problem description\n>\n> When a project has added git Notes for its commits, git by default doesn’t\n> automatically fetch the Notes; so, the Notes aren’t automatically discoverable\n> to contributors who are using “git log” to read the project commit logs — and\n> especially not discoverable to new contributors, or “casual” users of the logs.\n>\n> A user will see the Notes only if they _already_ know what git Notes are, and\n> know that the project uses Notes, and the user knows how to get them.\n>\n> But the reality is: most users do not even know what git Notes are, and don’t\n> know how to get them if they exist. So most people end up never seeing them.\n\nAnd even if they did, they wouldn't know how to use them, so not\nmuch is lost here.\n\nQuite honestly, a project that uses notes in such a way that it is\nessential to understand/utilize the history should reexamine its use\nof notes and try to see if they can make its commits more useful\nwithout relying on notes, I would think.\n\n> I’d be 100% happy to do the work of writing a patch to implement a solution\n> (a git behavior change) for this — if I could get confirmation that the git\n> maintainers would actually be open to reviewing such a patch.\n\nI've seen from time to time people ask \"I am thinking of doing this;\nwill a patch be accepted?  If so, I'll work on it.\" before showing\nany work, and my response always has been:\n\n (1) We don't know how useful and interesting your contribution would\n     be for our audience, until we see it; and\n\n (2) If you truly believe in your work (find it useful, find writing\n     it fun, etc.), that should be incentive enough for you to work\n     on it, whether or not the result will land in my tree.  You\n     should instead aim for something so brilliant that we would\n     come to you begging for your permission to include it in our\n     project.\n\n> As far as what the change would be: I realize this has been brought up\n> before — but it seems the obvious solutions are to “just” change git so:\n>\n> - Proposed solution #1: git auto-fetches all Notes when a repo is first cloned,\n>   and then auto re-fetches them again for every “git fetch” or“git pull”.\n>\n>   I think that auto-fetching-of-Notes would ideally be the _default_ git\n>   behavior — but short of that, at least a new [notes] _option_ for enabling\n>   that behavior would help. That would seem somewhat more “approachable” to\n>   than “git config --add remote.origin.fetch '+refs/notes/*:refs/notes/*'”.\n>\n> - Proposed solution #2: git checks if a clone lacks Notes vs remote, and emits:\n>\n>   > Your clone is behind the origin remote by N notes. To fetch the notes\n>   > from the origin remote, run “git fetch origin 'refs/notes/*:refs/notes/*'”\n>\n> Either way, I’d be very willing to put work myself into writing up a patch.\n\nA much more light-weight alternative would be to add an example to\nthe tutorial to tweak the \"remote.origin.fetch\" refspec so that it\nwill also fetch notes.\n\nBut stepping back a bit, none of the above (including your two) may\npractically be workable unless you limit the source of the notes to\nthe upstream, or something.  If you add notes yourself after you\nclone, and the upstream makes different changes to its notes,\nreconciling the diverged history of the notes tree would not be so\npleasant.  As a mechanism for the only publisher to publish\nauxiliary pieces of information to cloners, notes is a very useful\nmechanism, but for such a use case to be effective, the project\nparticipants must understand when they are supposed to use the notes\nread-only.\n"},{"id":"502094","messageId":"20240904071338.GW26466@kitsune.suse.cz","threadId":"61827","inReplyTo":"xmqq7cbsh16d.fsf@gitster.g","subject":"Re: Problem: git Notes not discoverable (+proposed solutions)","fromName":"Michal Suchánek","fromEmail":"msuchanek@suse.de","sentAt":"2024-09-04T07:13:38Z","receivedAt":"2024-09-04T07:13:41Z","isPatch":false,"sender":{"key":"msuchanek@suse.de","avatar":"https://avatars.githubusercontent.com/u/787652?v=4"},"body":"On Tue, Sep 03, 2024 at 02:34:02PM -0700, Junio C Hamano wrote:\n> sideshowbarker <mike@w3.org> writes:\n> \n> > ## Problem description\n> >\n> > When a project has added git Notes for its commits, git by default doesn’t\n> > automatically fetch the Notes; so, the Notes aren’t automatically discoverable\n> > to contributors who are using “git log” to read the project commit logs — and\n> > especially not discoverable to new contributors, or “casual” users of the logs.\n> >\n> > A user will see the Notes only if they _already_ know what git Notes are, and\n> > know that the project uses Notes, and the user knows how to get them.\n> >\n> > But the reality is: most users do not even know what git Notes are, and don’t\n> > know how to get them if they exist. So most people end up never seeing them.\n> \n> And even if they did, they wouldn't know how to use them, so not\n> much is lost here.\n> \n> Quite honestly, a project that uses notes in such a way that it is\n> essential to understand/utilize the history should reexamine its use\n> of notes and try to see if they can make its commits more useful\n> without relying on notes, I would think.\n\nThe notes could be also used to annotate existing upstream history\nwithout altering it.\n\nHowever, the problems with notes worflows make it impractical.\n\nThanks\n\nMichal\n"},{"id":"502097","messageId":"5cab62b7-d2ab-4395-83f8-0b34562d3568@app.fastmail.com","threadId":"61827","inReplyTo":"xmqq7cbsh16d.fsf@gitster.g","subject":"Re: Problem: git Notes not discoverable (+proposed solutions)","fromName":"Kristoffer Haugsbakk","fromEmail":"code@khaugsbakk.name","sentAt":"2024-09-04T07:24:30Z","receivedAt":"2024-09-04T07:24:52Z","isPatch":false,"sender":{"key":"code@khaugsbakk.name","avatar":"https://avatars.githubusercontent.com/u/2229597?v=4"},"body":"On Tue, Sep 3, 2024, at 23:34, Junio C Hamano wrote:\n> sideshowbarker <mike@w3.org> writes:\n>\n>> ## Problem description\n>>\n>> When a project has added git Notes for its commits, git by default doesn’t\n>> automatically fetch the Notes; so, the Notes aren’t automatically discoverable\n>> to contributors who are using “git log” to read the project commit logs — and\n>> especially not discoverable to new contributors, or “casual” users of the logs.\n>>\n>> A user will see the Notes only if they _already_ know what git Notes are, and\n>> know that the project uses Notes, and the user knows how to get them.\n>>\n>> But the reality is: most users do not even know what git Notes are, and don’t\n>> know how to get them if they exist. So most people end up never seeing them.\n>\n> And even if they did, they wouldn't know how to use them, so not\n> much is lost here.\n\nThe given example is about them appearing in the Git log. If the notes\nare autofetched and there are notes in the default namespace (`commits`)\nthen they will see them in the log. In turn they are using them (seeing\nthem) without having to do anything themselves.\n\n> Quite honestly, a project that uses notes in such a way that it is\n> essential to understand/utilize the history should reexamine its use\n> of notes and try to see if they can make its commits more useful\n> without relying on notes, I would think.\n\nThe two examples seem to be about adding Notes in bulk to established\nprojects. Maybe this information would have been part of the commit\nmessage if they had the necessary foresight.\n\n(Or maybe not: I wouldn’t add all relevant GitHub metadata to the commit\nmessages)\n\n>> As far as what the change would be: I realize this has been brought up\n>> before — but it seems the obvious solutions are to “just” change git so:\n>>\n>> - Proposed solution #1: git auto-fetches all Notes when a repo is first cloned,\n>>   and then auto re-fetches them again for every “git fetch” or“git pull”.\n>>\n>>   I think that auto-fetching-of-Notes would ideally be the _default_ git\n>>   behavior — but short of that, at least a new [notes] _option_ for enabling\n>>   that behavior would help. That would seem somewhat more “approachable” to\n>>   than “git config --add remote.origin.fetch '+refs/notes/*:refs/notes/*'”.\n>>\n>> - Proposed solution #2: git checks if a clone lacks Notes vs remote, and emits:\n>>\n>>   > Your clone is behind the origin remote by N notes. To fetch the notes\n>>   > from the origin remote, run “git fetch origin 'refs/notes/*:refs/notes/*'”\n>>\n>> Either way, I’d be very willing to put work myself into writing up a patch.\n>\n> A much more light-weight alternative would be to add an example to\n> the tutorial to tweak the \"remote.origin.fetch\" refspec so that it\n> will also fetch notes.\n>\n> But stepping back a bit, none of the above (including your two) may\n> practically be workable unless you limit the source of the notes to\n> the upstream, or something.  If you add notes yourself after you\n> clone, and the upstream makes different changes to its notes,\n> reconciling the diverged history of the notes tree would not be so\n> pleasant.  As a mechanism for the only publisher to publish\n> auxiliary pieces of information to cloners, notes is a very useful\n> mechanism, but for such a use case to be effective, the project\n> participants must understand when they are supposed to use the notes\n> read-only.\n\nThe proposal seems very in line with downstream participants who don’t\nknow how to use Git Notes. The downstream participants don’t have to do\nanything: the default note just appears in their logs because it is\nauto-fetched. Their only concern is whether the notes are informative or\nnoisy. They don’t have to care what Notes are.\n\nThe participants don’t have to know how to use Notes (read-only): they\nare just there.\n\nThen if they get curious and make notes themselves in the same\nnamespace? That’s their own fault.\n\nI don’t need a printer and my own pins to read a bulletin board.\n"},{"id":"502136","messageId":"xmqqbk13eafp.fsf@gitster.g","threadId":"61827","inReplyTo":"20240904071338.GW26466@kitsune.suse.cz","subject":"Re: Problem: git Notes not discoverable (+proposed solutions)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-09-04T14:54:34Z","receivedAt":"2024-09-04T14:54:36Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Michal Suchánek <msuchanek@suse.de> writes:\n\n> On Tue, Sep 03, 2024 at 02:34:02PM -0700, Junio C Hamano wrote:\n>\n> The notes could be also used to annotate existing upstream history\n> without altering it.\n>\n> However, the problems with notes worflows make it impractical.\n\nIndeed.  As a mechanism for the only publisher to publish auxiliary\npieces of information to cloners, notes is a very useful mechanism,\nbut for such a use case to be effective, the project participants\nmust understand when they are supposed to use the notes read-only.\n"},{"id":"502170","messageId":"CA+P7+xop8OY18nQaREFk6LeDdnn53oSGnigN-ddSHAU7mAMO8g@mail.gmail.com","threadId":"61827","inReplyTo":"m0wmjto8aq.fsf@epic96565.epic.com","subject":"Re: Problem: git Notes not discoverable (+proposed solutions)","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2024-09-04T20:03:09Z","receivedAt":"2024-09-04T20:03:20Z","isPatch":false,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Mon, Sep 2, 2024 at 6:06 PM Sean Allred <allred.sean@gmail.com> wrote:\n>\n> I agree that git-notes is an under-utilized idea. There's a lot of\n> potential to add context where it matters.\n>\n\nI also agree with this.\n\n> sideshowbarker <mike@w3.org> writes:\n> > I’d be 100% happy to do the work of writing a patch to implement a solution\n> > (a git behavior change) for this — if I could get confirmation that the git\n> > maintainers would actually be open to reviewing such a patch.\n>\n> Best way to determine that in my experience is to just propose some kind\n> of patch -- especially if the actual change is simple even if\n> controversial.\n>\n> > As far as what the change would be: I realize this has been brought up\n> > before — but it seems the obvious solutions are to “just” change git so:\n> >\n> > - Proposed solution #1: git auto-fetches all Notes when a repo is first cloned,\n> >   and then auto re-fetches them again for every “git fetch” or“git pull”.\n> >\n> >   I think that auto-fetching-of-Notes would ideally be the _default_ git\n> >   behavior — but short of that, at least a new [notes] _option_ for enabling\n> >   that behavior would help. That would seem somewhat more “approachable” to\n> >   than “git config --add remote.origin.fetch '+refs/notes/*:refs/notes/*'”.\n>\n> This would certainly be the most turnkey approach -- but what could go\n> wrong here? I can think of at least one potential danger: that your own\n> notes would be wiped out on fetch if you don't remember to push them\n> first. Laying out the risks involved with each approach would help the\n> conversation by showing the effort you've put into the design.\n>\n> It's my understanding that the git-notes feature is considered a little\n> under-baked to 'turn on' more broadly like this. There are simply too\n> many sharp edges:\n>\n> - the 'push before fetch' footgun I mentioned above\n> - merge conflict resolution workflow for the notes themselves\n\nI had use cases involving notes in the past and this was the biggest\ndealbreaker. There is not standardized way to fetch notes in such a\nway that you can perform conflict resolution.\n\nThis sort of fits into a bigger problem in that any non-branch refs\ndon't have the equivalent \"refs/remotes/<blah>\" area to fetch into for\ncomparison, but changing refs/remotes is a big backwards compatibility\nissue. I considered in the past to add things like refs/remote-notes/,\nbut that also ended up not going very far.\n\nI would love to see some of these problems solved, but unfortunately\nhave not had motivation or time to work on them, as we ended up not\nusing notes. The problems are quite tricky to find suitable solutions\nand get folks to agree.\n"}]}