threads / discuss / 51169

Need help merging unrelated histories

Subject: Need help merging unrelated histories

## tl;dr

7 messages between May 24, 2019 and May 29, 2019.

replies: 6people: 4as markdown or json

Robert Dailey· May 24, 2019, 14:04 UTC · lore
Everything I'm going to describe is related to this repository:
https://github.com/powervr-graphics/Native_SDK

This repo has several distinct branches. None of them seem to be tied to each other. Instead of having a `master` where they branched off each of their releases (e.g. 3.1, 3.2, 4.0), it looks like they made a copy of previous branches with no ancestry and then continued coding on that.

What I'm trying to do is go back and rebase all of the X.X branches onto a new `master` branch. I started off with this:

$ git checkout --orphan master

Then I committed a `.gitattributes`. Next, I rebased the first (oldest) release branch:

$ git rebase -i --onto master origin/3.0 --root
Then I merged:
$ git checkout master && git merge --no-ff -
Next, I did 3.1:
$ git rebase -i --onto master origin/3.0 origin/3.1 -X theirs

Using interactive mode, Git is smart enough to detect duplicate commits and eliminates those, even though the 2 branches do not share a merge base. I continued doing this all the way up to rebasing 4.3, but that's when things got tough. I see a lot of 'UD', 'UA', 'AU', 'AA' merge conflicts. These are obviously due to the fact that the branches aren't connected. But I expected was that `-X theirs` would always favor what's on the branch being rebased. However, it seems this only affects modified conflicts, not adds or deletes.

I was trying to find a way to bulk-resolve these. I mean, if git sees a file added on the left AND the right, I want the right one (theirs; the one coming from the 4.3 branch). Even though the branches are unrelated in terms of their history, I want the net effect of the rebase to essentially reflect the files on 4.3 itself. If a file isn't present on HEAD, delete it. If a file exists on REBASE_HEAD but not on HEAD, then add it. If the same file exists on both, favor the one on REBASE_HEAD.

But I don't see a way of doing that. I tried `git checkout --theirs .` and `git checkout REBASE_HEAD -- .` but this doesn't work with all conflict types.

Can anyone provide some advice on how to properly restructure this repository to create some ancestry, as if all along a `master` existed and all release branches were based on this in a linear fashion?

Robert Dailey· May 24, 2019, 14:08 UTC · re: Robert Dailey · lore

Re: Need help merging unrelated histories

On Fri, May 24, 2019 at 9:04 AM Robert Dailey <rcdailey.lists@gmail.com> wrote:
Show 54 quoted lines
>
> Everything I'm going to describe is related to this repository:
>
> https://github.com/powervr-graphics/Native_SDK
>
> This repo has several distinct branches. None of them seem to be tied
> to each other. Instead of having a `master` where they branched off
> each of their releases (e.g. 3.1, 3.2, 4.0), it looks like they made a
> copy of previous branches with no ancestry and then continued coding
> on that.
>
> What I'm trying to do is go back and rebase all of the X.X branches
> onto a new `master` branch. I started off with this:
>
> $ git checkout --orphan master
>
> Then I committed a `.gitattributes`. Next, I rebased the first
> (oldest) release branch:
>
> $ git rebase -i --onto master origin/3.0 --root
>
> Then I merged:
>
> $ git checkout master && git merge --no-ff -
>
> Next, I did 3.1:
>
> $ git rebase -i --onto master origin/3.0 origin/3.1 -X theirs
>
> Using interactive mode, Git is smart enough to detect duplicate
> commits and eliminates those, even though the 2 branches do not share
> a merge base. I continued doing this all the way up to rebasing 4.3,
> but that's when things got tough. I see a lot of 'UD', 'UA', 'AU',
> 'AA' merge conflicts. These are obviously due to the fact that the
> branches aren't connected. But I expected was that `-X theirs` would
> always favor what's on the branch being rebased. However, it seems
> this only affects modified conflicts, not adds or deletes.
>
> I was trying to find a way to bulk-resolve these. I mean, if git sees
> a file added on the left AND the right, I want the right one (theirs;
> the one coming from the 4.3 branch). Even though the branches are
> unrelated in terms of their history, I want the net effect of the
> rebase to essentially reflect the files on 4.3 itself. If a file isn't
> present on HEAD, delete it. If a file exists on REBASE_HEAD but not on
> HEAD, then add it. If the same file exists on both, favor the one on
> REBASE_HEAD.
>
> But I don't see a way of doing that. I tried `git checkout --theirs .`
> and `git checkout REBASE_HEAD -- .` but this doesn't work with all
> conflict types.
>
> Can anyone provide some advice on how to properly restructure this
> repository to create some ancestry, as if all along a `master` existed
> and all release branches were based on this in a linear fashion?
By the way, everything I've done so far (up to 4.2) is located here:
https://github.com/rcdailey/Native_SDK

There is a `master` branch in my fork of that repo. I am not sure if I've done it correctly so far, or if how I've done this up to 4.2 explains why 4.3 is so hard.

Ævar Arnfjörð Bjarmason· May 24, 2019, 14:41 UTC · re: Robert Dailey · lore

Re: Need help merging unrelated histories

On Fri, May 24 2019, Robert Dailey wrote:
Show 53 quoted lines
> Everything I'm going to describe is related to this repository:
>
> https://github.com/powervr-graphics/Native_SDK
>
> This repo has several distinct branches. None of them seem to be tied
> to each other. Instead of having a `master` where they branched off
> each of their releases (e.g. 3.1, 3.2, 4.0), it looks like they made a
> copy of previous branches with no ancestry and then continued coding
> on that.
>
> What I'm trying to do is go back and rebase all of the X.X branches
> onto a new `master` branch. I started off with this:
>
> $ git checkout --orphan master
>
> Then I committed a `.gitattributes`. Next, I rebased the first
> (oldest) release branch:
>
> $ git rebase -i --onto master origin/3.0 --root
>
> Then I merged:
>
> $ git checkout master && git merge --no-ff -
>
> Next, I did 3.1:
>
> $ git rebase -i --onto master origin/3.0 origin/3.1 -X theirs
>
> Using interactive mode, Git is smart enough to detect duplicate
> commits and eliminates those, even though the 2 branches do not share
> a merge base. I continued doing this all the way up to rebasing 4.3,
> but that's when things got tough. I see a lot of 'UD', 'UA', 'AU',
> 'AA' merge conflicts. These are obviously due to the fact that the
> branches aren't connected. But I expected was that `-X theirs` would
> always favor what's on the branch being rebased. However, it seems
> this only affects modified conflicts, not adds or deletes.
>
> I was trying to find a way to bulk-resolve these. I mean, if git sees
> a file added on the left AND the right, I want the right one (theirs;
> the one coming from the 4.3 branch). Even though the branches are
> unrelated in terms of their history, I want the net effect of the
> rebase to essentially reflect the files on 4.3 itself. If a file isn't
> present on HEAD, delete it. If a file exists on REBASE_HEAD but not on
> HEAD, then add it. If the same file exists on both, favor the one on
> REBASE_HEAD.
>
> But I don't see a way of doing that. I tried `git checkout --theirs .`
> and `git checkout REBASE_HEAD -- .` but this doesn't work with all
> conflict types.
>
> Can anyone provide some advice on how to properly restructure this
> repository to create some ancestry, as if all along a `master` existed
> and all release branches were based on this in a linear fashion?

We don't have a merge strategy to do this, but should. I had a WIP patch for this here that I haven't picked up: https://public-inbox.org/git/87sh7sdtc1.fsf@evledraar.gmail.com/

But you can emulate the same thing with "read-tree" and "commit". I.e. manually craft a commit with plumbing that diverges from the "master" branch, then instead of stock "rebase" write some loop where you keep using "read-tree" to read what you want to stage into the index from the existing branched commits, then the equivalent of "git commit -c" etc. to commit it.

Andreas Schwab· May 24, 2019, 17:10 UTC · re: Robert Dailey · lore

Re: Need help merging unrelated histories

On Mai 24 2019, Robert Dailey <rcdailey.lists@gmail.com> wrote:
> Can anyone provide some advice on how to properly restructure this
> repository to create some ancestry, as if all along a `master` existed
> and all release branches were based on this in a linear fashion?

How about using git replace --graft, then git filter-branch to make it permanent?

Andreas.
-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."
Robert Dailey· May 28, 2019, 13:35 UTC · re: Andreas Schwab · lore

Re: Need help merging unrelated histories

On Fri, May 24, 2019 at 12:11 PM Andreas Schwab <schwab@linux-m68k.org> wrote:
Show 9 quoted lines
>
> On Mai 24 2019, Robert Dailey <rcdailey.lists@gmail.com> wrote:
>
> > Can anyone provide some advice on how to properly restructure this
> > repository to create some ancestry, as if all along a `master` existed
> > and all release branches were based on this in a linear fashion?
>
> How about using git replace --graft, then git filter-branch to make it
> permanent?

I unfortunately have little-to-no experience with low level plumbing commands. Could you provide an example of a series of commands to run? It will help me to figure out how to do it on my own for my specific use case. Thanks in advance.

Robert Dailey· May 28, 2019, 18:26 UTC · re: Robert Dailey · lore

Re: Need help merging unrelated histories

On Tue, May 28, 2019 at 8:35 AM Robert Dailey <rcdailey.lists@gmail.com> wrote:
Show 16 quoted lines
>
> On Fri, May 24, 2019 at 12:11 PM Andreas Schwab <schwab@linux-m68k.org> wrote:
> >
> > On Mai 24 2019, Robert Dailey <rcdailey.lists@gmail.com> wrote:
> >
> > > Can anyone provide some advice on how to properly restructure this
> > > repository to create some ancestry, as if all along a `master` existed
> > > and all release branches were based on this in a linear fashion?
> >
> > How about using git replace --graft, then git filter-branch to make it
> > permanent?
>
> I unfortunately have little-to-no experience with low level plumbing
> commands. Could you provide an example of a series of commands to run?
> It will help me to figure out how to do it on my own for my specific
> use case. Thanks in advance.

Toyed around with it a bit, and tried with the script below. There were no errors, but I noticed that it erased my `.gitattributes` file on master that I committed before performing the graft. Why did it remove the file? I assume I'm not doing this correctly. Please advise. Thank you.

```sh #!/usr/bin/env bash set -ex

if [[ ! -d Native_SDK.git ]]; then
    git clone --bare git@github.com:powervr-graphics/Native_SDK.git
else
    cd Native_SDK.git
    git fetch
    cd -
fi

rm -rf test_repo git clone Native_SDK.git test_repo cd test_repo

git checkout -f --orphan new_master echo '* text=auto' > .gitattributes git add .gitattributes git commit -m 'Add gitattributes file'

c=$(git log --oneline origin/3.4 | tail -1 | cut -d ' ' -f 1) git replace --graft $c new_master git filter-branch HEAD..origin/3.4 git merge --no-edit --no-ff origin/3.4 ```

Philip Oakley· May 29, 2019, 13:56 UTC · re: Robert Dailey · lore

Re: Need help merging unrelated histories

Hi Robert, A few comments to look at, which may help. On 28/05/2019 19:26, Robert Dailey wrote:

Show 36 quoted lines
> On Tue, May 28, 2019 at 8:35 AM Robert Dailey <rcdailey.lists@gmail.com> wrote:
>> On Fri, May 24, 2019 at 12:11 PM Andreas Schwab <schwab@linux-m68k.org> wrote:
>>> On Mai 24 2019, Robert Dailey <rcdailey.lists@gmail.com> wrote:
>>>
>>>> Can anyone provide some advice on how to properly restructure this
>>>> repository to create some ancestry, as if all along a `master` existed
>>>> and all release branches were based on this in a linear fashion?
>>> How about using git replace --graft, then git filter-branch to make it
>>> permanent?
>> I unfortunately have little-to-no experience with low level plumbing
>> commands. Could you provide an example of a series of commands to run?
>> It will help me to figure out how to do it on my own for my specific
>> use case. Thanks in advance.
> Toyed around with it a bit, and tried with the script below. There
> were no errors, but I noticed that it erased my `.gitattributes` file
> on master that I committed before performing the graft. Why did it
> remove the file? I assume I'm not doing this correctly. Please advise.
> Thank you.
>
> ```sh
> #!/usr/bin/env bash
> set -ex
>
> if [[ ! -d Native_SDK.git ]]; then
>      git clone --bare git@github.com:powervr-graphics/Native_SDK.git
> else
>      cd Native_SDK.git
>      git fetch
>      cd -
> fi
>
> rm -rf test_repo
> git clone Native_SDK.git test_repo
> cd test_repo
>
> git checkout -f --orphan new_master

Check here what is in the files system and index - see the caveat in the man page for the --orphan start point and potential use of `git rm -rf .`

> echo '* text=auto' > .gitattributes
Maybe also add some mega comment line to really see it..
> git add .gitattributes
> git commit -m 'Add gitattributes file'
Check what is actually committed, in case you have more/less than expected.
> c=$(git log --oneline origin/3.4 | tail -1 | cut -d ' ' -f 1)
Is this the one you expected (maybe gitk viewer )
> git replace --graft $c new_master

Typing "git replace" without arguments, also lists all replace refs. - check what the new one is. Investigate.

> git filter-branch HEAD..origin/3.4

Hmm is this the right range for re-write (not checked, just feels as if it's wrong). Maybe also need a local branch name for origin/3.4, just for comparing.

> git merge --no-edit --no-ff origin/3.4
> ```
same again

Hopefully the random thoughts may prompt something. Philip

← back to recent threads