Collaborative conflict resolution feature request

Hi Guys,

Sometimes in our private git instance in the company I work for we merge branches that have been forked for months and there can be several or more people involved in the conflict resolution.

At the moment we have two options:

- One person, a branch manager, solves them by ringing people, holding meetings, using best judgement, etc.
- Somebody solves the conflicts they are involved with, marks everything as resolved and pushes (leaving <<< ==== >>>> delimiters in for unsolved conflicts) for the next person to continue. This sort of works although you falsely mark everything as resolved, leaving merge tools useless and many broken, unbuildable commits around in the branch.

Note: rebase and squashing commits is banned in our org, basically anything that would rewrite history on a remote branch.

Is there any existing or upcoming feature in git that could help make conflict resolution a more distributed, collaborative kind of task?

I also opened this as an issue in github as I feel it could be solved by either tool potentiall:

https://github.com/isaacs/github/issues/1816

Regards,

Eric Curtin

Software Engineer Ovens Campus, Cork, Ireland

Dell EMC

Am 12.06.20 um 16:08 schrieb Curtin, Eric:

Third option: Do not merge the whole branch in one big do-it-all-at-once merge. Instead, pick strategic commits in the history of the branch such that, when you merge them one after the other, each has only conflicts in one particular area or topic, and so can be solved in a reasonable amount of time with reasonable resources.

-- Hannes

Hi,

On Fri, Jun 12, 2020 at 4:11 PM Curtin, Eric <Eric.Curtin@dell.com> wrote:

You might want to take a look at Michael Haggerty's 'git imerge':

https://github.com/mhagger/git-imerge

I also made the same suggestion on the issue.

Best, Christian.

Both great ideas! And have the same theory right? Merge until you come across the first conflicting commit in a branch to make the conflicts smaller right?

Thanks so much for your help! Any alternative ideas? I'm definitely going to try both techniques, although imerge seems like an automation of the first idea.

Anybody ever think of rewriting the imerge tool in C or whatever to get in merged into mainline git? Potentially I could do it as part of my masters thesis if Michael H and the git open source community agreed?

Regards,

Eric Curtin

Software Engineer Ovens Campus, Cork, Ireland

Dell EMC

From: Christian Couder <christian.couder@gmail.com>
Sent: Saturday 13 June 2020 13:08
To: Curtin, Eric <Eric.Curtin@dell.com>
Cc: git@vger.kernel.org <git@vger.kernel.org>; Geary, Niall <Niall.Geary@dell.com>; rowlands, scott <Scott.Rowlands@dell.com>; Michael Haggerty <mhagger@alum.mit.edu>
Subject: Re: Collaborative conflict resolution feature request 
 

[EXTERNAL EMAIL] 

Hi,

On Fri, Jun 12, 2020 at 4:11 PM Curtin, Eric <Eric.Curtin@dell.com> wrote:

You might want to take a look at Michael Haggerty's 'git imerge':

https://github.com/mhagger/git-imerge

I also made the same suggestion on the issue.

Best, Christian.

On 13/06/2020 13:38, Curtin, Eric wrote:

I've also responded to the issue on GitHub:

"Do you have an implicit XY-problem where your processes are reinforcing historical bad habits "we merge branches that have been forked for months"? It sounds like the process is saying "We enjoy technical debt" (of delaying the merge until it's really bad...).

Maybe have a parallel 'merge' branch that is used (say weekly) to do trial merges and will essentially record the conflict resolutions while they are fresh in folks memories. That branch is distinct from, either of the two main branches, but will act as a filter and a hand rail to highlight future difficulties."

Implicit in Git is the use of small patches, easy branching and frequent merges, available to the individual coder. Most older "change control systems" focus on *stopping* change. Git promotes change, because reproduction & testing is cheap (almost zero). Git provides *authentication* of code versions. The changes in the underlying materials (from hardware to software), i.e. to bits and bytes, ripped up the old rule book.

Also look at 'rerere'.

Philip Oakley <philipoakley@iee.email> writes:

An aside that probably would not directly help Eric, but I know the above workflow helps reasonably well. The 'pu' branch is rebuilt not on top of 'next', but is rebuilt with all topics (including those already in 'next') in flight directly on top of 'master', which serves as a way to anticipate conflicts that will require resolution in the future before the topics can enter 'next' branch. And these resolutions are ...

... remembered in the rerere database (and even trickier ones that rerere cannot handle are recorded in the merge-fix commits, but that is a separate story). When topics are ready to be merged to 'next' and to 'master', the correct resolutions are likely to be known and the result tested in 'pu' and 'next', respectively, for some time already.

Philip Oakley <philipoakley@iee.email> writes:

[...]

'rerere' is a superb feature, but isn't it local? If so, how could it help for collaboration? What's the idea? Is there a way to share 'rerere'?

Thanks, -- Sergey

On 15/06/2020 10:51, Sergey Organov wrote:

I saw this (rerere) is two parts. First was to ensure Eric was aware of it as a possible capability for use by the 'merge manager', and others, so that they didn't loose sight of their conflict resolutions for the time when the 'big merge window' came around. E.g. So a dev could do a local fix based on their rerere database and then send a patch to the merge manager indication their approach to the resolution.

Meanwhile, second, at the moment the rerere database is 'local', mainly, as I understand it because of the number of context lines a local user has chosen (hence not immediately portable).

I personally believe that it should be possible to some how exchange resolutions without that pre-optimisation of the context line choice. I had a look back at the old rerere script and that had small fingerprints of each resolution stored in the database (at least as I read it). However when I look at the modern rerere database it looks like it has full pre & post images, rather than just the conflicts, so I'm not yet sure what's really happening (i.e. I haven't dived into the c code).

A possibly more sensible approach (to exchanging resolutions) is simply to do the merge, without commit, then save that merge as if it's a single side commit (with the other merge parent listed in the commit message), and that commit can then be pushed/pulled etc and a variant of `rerere train` can be used to recreate the local database. In a sense the fake merge (why not a proper merge) is a variant of a stash where you aren't wanting to pollute the branch trees with this extra 'flotsam'.

Philip

Philip Oakley wrote:

There is a `contrib/rerere-train.sh` script in git's repository, that can recreate rerere resolutions from existing merge commits.

Extending the options Eric outlined, a collaborative conflict resolution workflow might thus be:

  * developers do test merges on temporary branches between their
    feature branch and the main development – or other feature
    branches if necessary (maybe create test merges on a regular
    basis to minimize the new conflicts)
  * these temporary branches get pushed, but not merged to other
    branches
  * the branch manager fetches these branches and uses
    `rerere-train.sh` to fill the local rerere database with
    conflict resolutions from the test merges
  * the temporary branches get deleted
  * the recorded resolutions get reused when needed (keep in mind
    rerere's gc config, see gc.rerereResolved and gc.rerereUnresolved)

The last discussion on rerere-train.sh on this list was here:

https://lore.kernel.org/git/BZAQIE4YND2I.Z7BFCW7BLH3K@penguin/

Hi Guys,

Yes I think you all understand the conundrum well. Conflict resolution by definition is a collaborative effort, but git doesn't support it as a, collaborative effort, only one user can resolve it in git. It will be hard to change my whole orgs thinking around avoiding conflicts or making them easier conflicts to solve. There will always be some conflicts.

I certainly want to play around with rere and see if it helps things. I suspect if I shared a technique like this with the 100 or so developers on the project this will be deemed too complex though.

A per file solution isn't great either as some files can be large. Per-conflict (between <<<< >>>> in a plain old text editor) is reasonable.

What would be most ideal is a:

git merge fix some conflicts, not others git push

so someone else can work on it kind of solution....

We don't do plaintext email patches, we do typically merge things via git cli/protocol or via GitHub Pull Request.

Regards,

Eric Curtin

Software Engineer Ovens Campus, Cork, Ireland

Dell EMC

"Curtin, Eric" <Eric.Curtin@dell.com> writes:

[As a side-note, I don't agree with "by definition" part of the statement, nor with "only one user" part, so what is left?]

What I'd like to stress though is that there is a pitfall here: is it feasible to try to support concurrent conflict resolution, or is it to be sequential (even if in multiple turns)? I incline to the latter.

Concurrent conflict resolution would lead to conflicts in conflict resolutions, that already sounds too complex to be useful for my taste, and we already are in recursion that must be stopped somewhere, so it's tempting to stop it one level up.

Admittedly, one can try to avoid conflicts in resolutions by splitting content to independent parts (where Git is not easily applicable by design, being the tool that manages entire content), but such a split doesn't sound realistic, as if it were possible, it'd be easier to apply it to avoid original conflicts in the first place. Then, a non-trivial conflict resolution often needs changes elsewhere anyway, making strict content split again problematic.

OTOH, anything sequential is intrinsically difficult for a distributed system like Git, so it probably should better be implemented elsewhere. This approach is simplified by the fact that fortunately Git doesn't care how exactly you come-up with the resulting merge commit[*], so one is free to use whatever external tool to achieve his goals.

-- Sergey

[*] Well, it actually does care in some corner cases, e.g., when it drops or tries to re-create merges on rebase, and that will only be eliminated once "evil merge" concept is finally buried, for better.

On Sat, Jun 13, 2020 at 2:38 PM Curtin, Eric <Eric.Curtin@dell.com> wrote:

(We usually reply inline instead of top-posting here.)

My opinion is that it would be nice to have something like git-imerge integrated into Git.

Christian Couder <christian.couder@gmail.com> writes:

I am not sure what you mean by "integrated into", but if you are talking about somehow reinventing it, I do not think it is a good idea at all. "imerge" works quite well already.

Or do you mean it would gain more exposure for being bundled together? I am not sure if that is a good idea, either.

Surely, it would be convenient for end users if a single download (or "apt-get install") gives everything useful, but that does not have to be done at the ultimate upstream level between me and Michael, and doing so at that level would mean the release schedule needs to be coordinated (one may need to wait unnecessarily for the other's pre-release freeze period, for example), among other loss of flexibility. Luckily, most end users would get their Git from packagers and they are good at doing the bundling (i.e. the "git-core" package may "suggest" the "git-imerge" package).

So...

Junio C Hamano <gitster@pobox.com> writes:

So my answer to your idea/opinion is that we shouldn't waste engineering effort to "have something like imerge integrated into git itself", but we should help distro packages to do the bundling of "git" itself and all the good things around it. One way of doing it may be by keeping an official curated list of "third-party things we find good" somewhere (it can be in-tree in my release tarballs, but it does not have to be---some page on git-scm.com could just be fine; as long as the quality of the list is maintained to our standards, where the packagers and end users see it does not really matter).

And such a list would also help those who prefer to build and install things by hand.

On 13/06/2020 20:34, Junio C Hamano wrote:

For the imerge tool, it may be worth having an extra sub-heading (Merge Tools?) within the "HOW TO RESOLVE CONFLICTS" section of the git-merge man page.

The merge.guitool configuration does list a lot of pre-configured tools (the list could be moved to the mergetool man page?)

Curating the list of tools maybe could be done in the same way the config entries are now being done, i.e. by area, so they can be included in the relevant man pages, with just a single source of 'nice tools'.

Philip

13.06.2020, 22:22, "Junio C Hamano" <gitster@pobox.com>:

No it doesn't. It has performance issues which makes it practically useless on the large scale repositories because of slow progress even on fastest machines. See for example

https://github.com/mhagger/git-imerge/issues/66 https://github.com/mhagger/git-imerge/issues/149

Also, maybe a bit irrelevant, but it was recently removed from Gentoo because it didn;t work with Python >= 3.7

Out of all the currently available options this solution helps. Thanks Junio! I played with incremental merge techniques this weekend. One problem with incrementally merging is that you start fixing conflicts that are later invalidated by subsequent commits. So it seems you end up doing more conflict resolution than necessary. Unless I'm misusing the technique.

I think in order to create collaborative distributed conflict resolution, you'd probably need a new type of commit a "partial-merge" commit, that is like a pseudo-commit that you can push and doesn't break builds. It would be a neat feature, at least for my team!

Regards,

Eric Curtin

Software Engineer

Ovens Campus, Cork, Ireland

Dell EMC

On 15/06/2020 10:28, Curtin, Eric wrote:

One thing I did note, from your prompting, is that merge doesn't take any `-- <files>` options which could allow a quick way of selecting just the few files that you, or the specific dev, is able to merge, so as to create a partial merge on your side branch, and leave behind the remaining difficult conflicts to be resolved later by the appropriate dev in a sequence of rolling partial merges.

It could be effectively a special strategy. IIUC the '--' separator is already supported by the underlying parser code, so may not be that hard? (perhaps a local contribution to the codebase;-). Just a thought.

Philip

Philip Oakley <philipoakley@iee.email> writes:

Assuming that there are paths A and B that would leave conflict in an attempted merge between commits X and Y, you somehow resolve the conflict in A and leave B unresolved, what would you pass to the other person and ask to resolve the conflicts in B? It cannot be a merge commit that records X and Y as its parents and a single tree object as the result, because the whole point of this is that you do not even know what to record for B.

I think the most important and useful part of this is to design the data format used for that task of passing from you to the other person. The way to specify which paths are yours etc. are much less interesting and trivial part of the story, I would think.

On Mon, Jun 15, 2020 at 10:11 AM Junio C Hamano <gitster@pobox.com> wrote:

I've thought about this (some) myself in the past. It seems to me that what is needed is the ability to pass the complete unmerged state on. That is, we need something like five commits, made in a way similar to the way a stash is made:

 * all resolved (stage 0) index entries (1 tree)
 * all index stage 1, 2, and 3 entries (3 trees)
 * and the current work-tree (1 tree)

This isn't quite complete as we also need the merge information; perhaps that goes into a sixth commit, or can be wedged into the commit message text of one (or more?) of the five trees.

The "resume unresolved merge" command then must unpack all of this into the appropriate places.

Chris

On Mon, Jun 15, 2020 at 10:32 AM Chris Torek <chris.torek@gmail.com> wrote:

A few further thoughts:

 * Given one or more saved merges and either a clean state or an
   ongoing merge, we need a tool to combine these.  There are a lot of
   corner cases here but in general, if merge X has file F in conflict and
   merge Y has file F resolved, we can take the resolution from Y.

 * Partial merges (in the work-tree copy of a file) that are not yet added
   may be the trickiest.  A simple heuristic would be to look for the
   conflict markers and see if one work-tree copy has a resolution
   where another work-tree copy has a conflict.  Or, though this is
   harder, use the ours/theirs copies in the saved index trees to find
   actual conflicted regions and compare this to the work-tree copy
   to find resolved regions.

 * There is also an obvious question about what to do when combining
   two different proposed resolutions where the stage-zero and/or
   work-tree copies of the files don't match.

None of these preclude the basic ability to save and restore—and of course transport, through fetch/push—the unmerged state, which I think is the required enabling technology. The ideas above are more for combining parallel merge efforts. If it's acceptable for dev A to merge his/her part and pass the result to dev B, who merges theirs, and so on, the above is not required.

Chris

Hi Junio, On 15/06/2020 17:57, Junio C Hamano wrote:

Are we confusing the file merge X.A and Y.A with X.B and Y.B?

The scenario envisaged is that dev.a has responsibility over the .A file merge, while dev.b will handle the merge for .B merge (e.g. different parts of the driver code).

So dev.a has resolved the .A conflicts, and the .B file(s) is still in the .ours state - no conflict (i.e. we have an 'ours' strategy for all files not in the A path(s)).

Yes the merge that dev.a has performed is passed onto dev.b to complete a second merge, apparently of the same commit, which is essentially similar to cherry picking the .B files from the .theirs commit [1].

if no merge is attempted for paths not in the A set then the other B set are never 'in conflict'. As I understand Eric's conundrum, it's that a plain merge throws too many conflicts at the one developer. This suggestion would be the way to reduce the number of conflicts just to the paths the one dev can handle. A lot will depend on how Eric's problem is partitioned and the depth of the overlaps.

A list of un-merged paths (i.e. not attempted) is one such format, surely?

Philip

[1] actually, it's more like a soft checkout or restore of those files from the merging branch. (exchange format becomes list of merged paths...).  Which  then leads easily to the available `git checkout` options (assuming the approach has validity for Eric's scenario).

Philip Oakley <philipoakley@iee.email> writes:

No.

Yes. The question is, that division of labor is agreed between humans dev.a and dev.b, but somehow need to be encoded in the data passed from dev.a to dev.b that says "I've done with As and am comfortable with the result; I didn't even look at Bs, but it's your turn to deal with them".

After seeing "git pull" leave conflicts, "git checkout HEAD -- Bs" (or checkout may be done out of MERGE_HEAD? I dunno) would be a way to get dev.a concentrate on As only, but the result cannot be tested sensibly anyway, and it cannot be committed as the result as taking "ours" or "theirs" for Bs was not dev.a's intention. How to express that "I didn't do anything to Bs" in such a way that can be distinguished from "I did look at Bs, and I believe taking all from my HEAD is the right resolution" was what I asked.

On Mon, 15 Jun 2020 at 11:30, Curtin, Eric <Eric.Curtin@dell.com> wrote:

I find that the solution in these cases is to first use interactive rebase to squash and reorganize the commits in the branches so you have a nice clean patch sequence. Once you have the branches cleaned up and squashed into a sequence of reasonable topic based chunks you then merge, sometimes it even means you dont get conflicts at all, git merge is pretty smart.

And after that you change your workflows so the rule is that whomever pushes first to the "trunk branch" wins, and the other guy has to do the conflict resolution. People will start merging earlier and more often so they can keep the conflicts to a minimum. :-) In other words I second what Philip Oakley said about bad workflows. Merge early, merge often, rollout early, rollout often, vote early, vote often. :-)

cheers, Yves

I understand what you guys are saying, I agree merge early, merge often does work best and it's what I've most often used. Our branching strategy is split by subsystem (and sometimes specific features). So contributors are not merging to the same long-term branches. So merge early, merge often, doesn't help with conflicts.

Why? We don't have so much automated tests, etc. And that's not an easy problem to solve although we are trying to improve in this area. For us to make worthwhile tests, we would have to emulate a lot of hardware from external vendors. Out test hardware is often shared among many. So branching ensures that if we hit problems, it most likely came from within our group, in the area of the code we are most knowledgeable about.

And when your branching strategy is like this, it ends up being the branch managers merging and having to fix conflicts, not the individual contributors. In our git repo you will see broken commits with these delimiters in (<<<< ==== >>>>) in order to "fake" this kind of collaborative conflict resolution feature as described in the initial email.

Regards,

Eric Curtin

Software Engineer Ovens Campus, Cork, Ireland

Dell EMC

I think concurrent doesn't make sense, only sequential.

Again, as said in the initial email, anything that rewrites history, recreates SHA's (such as rebase, squash, etc.) on a remote branch is not allowed in our repo. Of course with unpushed commits you can do some of these things as the remote end knows no different.

Regards,

Eric Curtin

Software Engineer Ovens Campus, Cork, Ireland

Dell EMC

On Thu, 18 Jun 2020 at 11:28, Curtin, Eric <Eric.Curtin@dell.com> wrote:

Ah I see, I missed that detail. We have a similar rule at work but only for the "trunk" branch (what most people call "master"), topic branches are allowed to change before the merge to trunk.

I guess there is no way to convince your policy makers that if commit A and B are different but have the same tree hash they refer to the same state on the disk? I have had audit conversations like that.

Anyway, sorry my reply wasn't helpful. Good luck.

cheers, Yves

Not at all, I do appreciate all the suggestions, I learned a lot from this thread in general. I think everybody in this thread has been very helpful.

This thread has gone a little cold. I did have a think about the sequential vs concurrent resolutions and even had a conversation with one of my colleagues about it.

Would it be reasonable if anyone could push a partial resolution but the book stops there (once a user hits a conflict of a conflict is must be solved locally)? I agree it doesn't make sense in most cases to support pushing recursive conflict resolutions (even though the other part of me says if the users wants to go down that path why stop them? You could even have a config setting to allow N levels of conflicts to be pushed, the default setting being exactly the way things are, none or 0!).

I know in my project we already "fake" this functionality like pointed out in the first email, it's just unclean the way we do it, leaves broken commits in the repo, you can no longer use difftools, etc.

Should I even consider this as a research idea for my thesis? Or another way of wording this is, if someone sent the code to the git maintainers Junio, etc. would it be merged into git?

Regards,

Eric Curtin

Software Engineer

Ovens Campus, Cork, Ireland

Dell EMC

On Fri, Jun 19, 2020 at 11:17 AM Curtin, Eric <Eric.Curtin@dell.com> wrote:

I think it could be an interesting feature to add to Git, as I agree that some people often have this kind of issues with conflicts.

What could perhaps work is to develop a new command, maybe called `git conflict` with subcommands for example to load, save and maybe push, fetch and resolve partial resolutions of conflicts. The conflicts could perhaps be stored as commits in the "refs/conflicts/" ref namespace.

Yes and actually the more I read about things like `git rerere`, most of the components are already there in git. What this concept kind of is, is a pushable git rerere cache. A workflow could be something like:

git merge/rebase # fails with conflicts, maybe a prompt that to tell the user that some conflict resolutions have already been pushed git rerere pop/apply # apply the already pushed conflicts # fix your conflict with whatever technique or tool you prefer git rerere create # commit your rerere (stash uses the term create) writing a message explaining why you resolved this conflict in a certain way git push # or maybe `git rerere push` so the next user can "git fetch/git merge/etc." and work on the conflict resolution

And maybe add some other git stash like commands like show/list. So other users can see who solved what conflict and why that decision was made. I'm using terms like pop/apply/create just to be consistent with stash also. The reason I make that a manual step and not an automatic step like git rerere tends to do, is that existing users of git won't be aware of these automated conflict resolutions and I'm sure many people won't like these being applied automatically by default.

Admittedly I don't use git stash for temporary local changes, I just use commits and branches, amend them, delete them etc. as I see fit. But I think the concept is very similar, these are not complete buildable commits.

But I think you're right Chris if we start from scratch and call it `git conflict` (although I was thinking of maybe `git resolve` or `git re`, since rerere already abbreviates resolution to just re) we don't have to worry about backwards compatibility with existing commands designed for a different purpose like rerere. I'm not too worried about the naming as long as the workflow is easy to use.

I might fly this by my lecturers as a research project. What do you think Chris and Junio? I'd be happy to send you patches of the work in progress as it's going along if you guys would be happy to see them?

Regards,

Eric Curtin

Software Engineer Ovens Campus, Cork, Ireland

Dell EMC

On Sat, Jun 13, 2020 at 2:08 PM Christian Couder <christian.couder@gmail.com> wrote:

Not sure if it could be useful in your case but some people also like a tool called git-mediate

See:
  - https://github.com/Peaker/git-mediate
  - https://medium.com/@yairchu/how-git-mediate-made-me-stop-fearing-merge-conflicts-and-start-treating-them-like-an-easy-game-of-a2c71b919984

"Curtin, Eric" <Eric.Curtin@dell.com> writes:

That'd be great.

What we sometimes do when such a case appears (rather rarely due to frequent merges, I admit) is to copy /entire/ git directory holding the merge in progress to the next person in charge. This is the only way that I'm aware of that keeps ability to do things like:

  $ f=foo.cc; git diff :1:./$f  :3:./$f

and

  $ f=foo.cc; git diff :1:./$f  :2:./$f

that helps a lot with complicated conflicts.

So, if such a solution ever appears, it'd apparently need a method of sharing or re-creating (parts of) the index? Doesn't sound as an easy problem.

-- Sergey