Gated Merge?

I wish there were a way to let me annotate a commit with a statement like this:

 * When merging a branch that does not have me in it into a trunk
   that already has me, do this extra check on the result of the
   merge before allowing it to be recorded.

For example, suppose there is a topic that changes all instances of this

	__attribute__((format(printf, X, Y)))

into a new macro FORMAT_PRINTF(X, Y). With such a facility, I can annotate the commit at the tip of that series to define an extra check:

   ! git diff HEAD | grep '^+.*__attribute__((format(printf'

which would prevent me from merging a topic that introduces a new instance of the printf-format attribute that is spelled in an old-fashioned way.

In this particular example, it might be even better if the facility allowed me to declare something like this instead:

 * When merging a branch that does not have me in it into a trunk
   that already has me, do this extra processing on the result of
   the merge before recording it.

The difference between them is the "extra processing" part. For this example, it would be something that is built around:

    perl -p -e '
        s{__attribute__\(\(format\(\s*printf,\s*(\d+),\s*(\d+)\)\)\)}
         {FORMAT_PRINTF($1, $2)}gx
    '

to update the old-fashioned one to use the new macro, and the result would become an evil merge.

To realize this, there are low-hanging-fruit building blocks that are trivial:

 - Such an annotation can be made as a note attached to the commit
   in question;

 - Such a check can be done in pre-commit hook;

 - Such a pre-commit hook can iterate over this set of commits

    DIFF=$(git rev-list --left-right HEAD...MERGE_HEAD)
    
   collect these Merge Gate annotations from the commit appears on
   the left hand side (e.g. only exists on the trunk that a side
   branch is about to be merged in), and run them.

But the last one feels very costly, and I suspect that there must be a better way to do this. I do not think we want the "what to do" part (i.e. checking new lines for __attribute__ in the first example, or rewriting new lines that has __attribute_ in the second example) to be in git-core; that clearly is outside the scope of the plumbing. But the part that finds these "annotations", especially if we can come up with a fast implementation that may be hard to script, may be a good addition to the plumbing command set.

Thoughts?

On 12 February 2016 at 09:06, Junio C Hamano <gitster@pobox.com> wrote:

What is the benefit in doing this in notes vs having the tests in the working tree?

Pros:

 - merge-gates can be added after the commit, but will stick with the
commit if it moves around (as opposed to creating a second commit to
add the merge-gate to the working tree)

 - cross repository standards can be established that allow
merge-gates to be detected and run automatically (arguably could be
done with a standardised folder structure too, but that is more
disruptive)

 - can view the relevant merge-gates in git log against each commit
that they are protecting

Cons:

 - difficult to see the current complete set of merge-gates

 - difficult to make changes to a number of merge-gates at the same time

 - poorly defined behaviour when multiple merge-gates overlap in
functionality. Which gates execute first? What if I reorder the
commits?

My practical knowledge of notes is severely lacking so excuse me if I missed anything obvious.

Regards,

Andrew Ardill

(sorry for the double post Junio, gmail ate my plain text encoding at some point...)

On Thu, Feb 11, 2016 at 2:42 PM, Andrew Ardill <andrew.ardill@gmail.com> wrote:

The way I could imagine "merge gates" currently:

 * they are some kind of invariant to be checked, or an action
performed, an event
 * they can be introduced by a commit X.
 * they can be turned off by a commit Y
 * an offspring commit inherits all "merge gates" from its parents (in
order of the parents

So the event for one merge gate is in effect iff you merge one commit from X..Y

Now we want to know for a given commit C, which merge gates are there? To find all events we would need to walk the whole history of C and record all found events, which are not turned off.

So I'd think that could be accelerated with event bitmaps (which work similar to the reachability bitmaps, but having a different goal and record which events are alive for certain commits, such that you only need to walk a bounded set of commits to find out.

When first reading this email I thought this is a typical maintainers problem, so it should happen all the time in larger projects, such as linux, how do they handle system wide refactorings?

Andrew Ardill <andrew.ardill@gmail.com> writes:

Interesting. I have never thought of adding this information to the project history proper---I've viewed this as primarily an aid for keeping track of topics in-flight by an individual, i.e. something that the rest of the project do not want to even see.

I think that this will not be a problem in practice if we took your alternative approach to cap the topic with an extra commit that adds the Merge Gate test; because the workflow this targets treat a topic as a single unit, the extra commit will be rebased together with the real commits in the topic.

Yeah, I think that this is very similar to "something that the rest of the project do not want to even see" problem, if you use an in-tree approach.

Yes, we would need to have a quick way to enumerate commits with these notes in the rev-list output.

Hmm, I am not sure if/how that will be an issue.

True. With an in-tree approach, you can define the order with the filenames, for example, and the above will be clearer. Instead, you would need to worry about name clashes, though.

On Fri, Feb 12, 2016 at 09:44:13AM -0800, Junio C Hamano wrote:

After reading Andrew's message, I wondered if these are really any different than regular tests at all.

Let's say I implement feature X on a topic branch, and I add a regression test for it. Once that is merged, now we have that regression test forever[1], and any future topic branches that get merged from master must pass that test or be rejected.

If I update the interface for foo(), it is the same thing. We do not write a specific test for it, but we expect that the compiler will catch any callers of the old foo, because the new tree carries the updated definition that will not work with them (and we often structure our interface refactoring exactly to catch such things).

So let's go back to your FORMAT_PRINT example. The topic changes all of the format-attributes into FORMAT_PRINTF, but we never add a "test" that says "now there are no more bare format-attributes, and if there are, it is a regression".

But I don't think this needs to have anything to do with merges in particular, or rules like "when merging a branch that does not have me in it". It is about saying "from here on out, the tree state should match this property, and we can test it by running this script". And then running "make code-lint-tests" becomes part of the acceptance testing for a proposed topic merge, just like "make test" is already (and likewise, people forking _new_ branches from master after the topic is merged would make sure they do not fail the code-lint tests before even submitting the topic).

-Peff

[1] Of course it doesn't _have_ to be forever. We sometimes modify or
    back out tests as new situations come up, and we could do the same for
    these sorts of code-lint tests.

Jeff King <peff@peff.net> writes:

That certainly is true, but this strays more and more from my original motive of implementing an automated evil-merge scheme that is better than what I currently have.

We try to do our tree-wide refactoring in such a way that it would break the compilation (by changing function signature) when we can. Catching with "make test" would certainly generalize it, but the endgame result I was shooting for was to come up with a solution for each topic in-flight just once and keep replaying it until it is merged.

On Fri, Feb 12, 2016 at 11:06:04AM -0800, Junio C Hamano wrote:

Yeah, that makes sense. I do repeated merges myself, and it is a pain. I do not usually use your Reintegrate scripts, though when I have done so, they work pretty well. And I don't think you're going to come up with anything much better for the general case.

Let's split the problem into "detect a problem" from "apply the fix for a topic".

You do not want to stop detecting once the original topic is merged. The new rule is "we spell this as FORMAT_PRINTF" and you want to detect it in simultaneous topics, _and_ in future topics. So if it is worth checking in an auomated way, that should go into the tree.

So once we've detected a problem, how do we fix it?

For the specific case of FORMAT_PRINTF, you showed a possible "extra processing" snippet to fix the problem in an automated way. But if we realize that these gated attributes are really just another form of test, it should be obvious that we cannot do so automatically in the general case. You do not expect to fix a failed "make test" on a merge in an automated way; there are too many possible resolutions. The best we can do is detect the problem, create a patch, and then apply that patch as appropriate.

So I'm somewhat doubtful that it would be worth the effort to create infrastructure for "automate this fix" that is anything except "apply this patch and tell me if we now pass the test".

-Peff