Re: refs/notes/amlog problems, was Re: [PATCH v3 01/20] linear-assignment: a function to solve least-cost assignment problems

Jeff King <peff@peff.net> writes:

Yes, the reverse mapping in amlog was an experiment that did not work well in the end.

When I use "git am" to make a commit out of a message, a post-applypatch hook picks up the "Message-Id:" from the original message and adds a git note to the resulting commit. This is in line with how the notes are meant to be used. We have a commit object, and a piece of information that we want to associate with the commit object, which is not recorded as a part of the commit object. So we say "git notes add -m 'that piece of info' $commit" (the message-id happens to be that piece of info in this example).

And with notes.rewriteRef, "git commit --amend" etc. would copy the piece of info about the original commit to the rewritten commit.

	Side Note: there are a few workflow elements I do want to
	keep using but they currently *lose* the mapping info.  An
	obvious one is

	  $ git checkout -b to/pic master &&
	  ... review in MUA and then ...
	  $ git am -s mbox &&
	  ... review in tree, attempt to build, tweak, etc.
          $ git format-patch --stdout master..to/pic >P &&
          $ edit P &&
          $ git reset --hard master &&
          $ git am P

	which is far more versatile and efficient when doing certain
	transformations on the series than running "rebase -i" and
	reopening and editing the target files of the patches one by
	one in each step.  But because format-patch does not
	generate Message-Id header of the original one out of the
	commit, the post-applypatch hook run by "am" at the end of
	the steps would not have a chance to record that for the
	newly created commit.

	For this one, I think I can use "format-patch --notes=amlog"
	to produce the patch file and then teach post-applypatch
	script to pay attention to the Notes annotation without
	changing anything else to record the message id of the
	original.  Other workflow elements that lose the notes need
	to be identified and either a fix implemented or a
	workaround found for each of them.  For example, I suspect
	there is no workaround for "cherry-pick" and it would take a
	real fix.

A reverse mapping entry used to get created by post-applypatch to map the blob that represents the notes text added to the $commit to another text blob that contains the 40-hex of the commit object. This is the experiment that did not work well. As none of the later integrator's work e.g. "commit --amend", "rebase", "cherry-pick", etc. is about rewriting that blob, notes.rewriteRef mechanism would not kick in, and that is understandasble.

And these (incomplete) reverse mapping entries get in the way to maintain and correct the forward mapping. When a commit that got unreachable gets expired, I want "git notes prune" to remove notes on them, and I do not want to even think about what should happen to the entries in the notes tree that abuse the mechanism to map blobs that are otherwise *not* even reachable from the main history.

A much more important task is to make sure that the forward mapping that annotates invidual commits reachable from 'pu' and/or 'master' is maintained correctly by various tools. From a correctly maintained forward mapping, it should be straight forward to get a reverse mapping if needed.

Before this thread, I was planning to construct and publish the reverse mapping at the end of the day, but do so on a separate notes ref (see above---the hacky abuse gets in the way of maintaining and debugging the forward mapping, but a separate notes-ref that only contains hacks is less worrysome). But I have changed my mind and decided not to generate or publish one. It is sort of similar to the way the pack .idx is constructed only by the receiver [*1*].

I know there is no information loss (after all I was the only one who ran that experimental hack), but there is one objection that is still possible, even though I admit that is a weak argument.

If a plumbing "diff-{files,tree,index}" family had a sibling "diff-notes" to compare two notes-shaped trees while pretending that the object-name fan-out did not exist (i.e. instead, the trees being compared is without a subtree and full of 40-hex filenames), then it would be less cumbersome to incrementally update the reverse mapping by reading forward mapping with something like:

	git diff-notes --raw amlog@{1} amlog

to learn the commits whose notes have changed. But without such a plumbing, it is cumbersome to do so correctly. "git diff-tree -r" could serve as a rough substitute, until the note tree grows and get rebalanced by reorganizing the fan-out, and on the day it happens the reverse mapper needs to read and discard ghost changes that are only due to tree reorganizing [*2*].

[Footnotes]

*1* Even if the sender could give one when it creates a .pack, the
    receiver would not trust that it is matches the corresponding
    .pack before using it, and the cost to validate is similar to
    the cost to generate.

*2* That makes it less efficient on that day (which hopefully would
    happen once in a blue moon) but would not affect correctness.