{"thread":{"id":"52317","subject":"git-subtree split misbehaviour with a commit having empty ls-tree for the specified subdir","startedAt":"2019-11-22T20:42:34Z","lastAt":"2020-06-18T02:19:00Z","messageCount":10,"participants":["Ed Maste","Tom Clarkson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"386870","messageId":"CAPyFy2AsmaxU-BDf_teZJE5hiaVpTSZc8fftnuXPb_4-j7j5Fw@mail.gmail.com","threadId":"52317","inReplyTo":null,"subject":"git-subtree split misbehaviour with a commit having empty ls-tree for the specified subdir","fromName":"Ed Maste","fromEmail":"emaste@freebsd.org","sentAt":"2019-11-22T16:55:47Z","receivedAt":"2019-11-22T20:42:34Z","isPatch":false,"sender":{"key":"emaste@freebsd.org","avatar":"https://avatars.githubusercontent.com/u/1034582?v=4"},"body":"I encountered an issue while trying to use git subtree with the\nFreeBSD svn->git mirror: I found that when \"git subtree split\"\nencounters a commit with an empty \"git ls-tree\" for the subdirectory\nbeing split, it ends up recording the original parent as the new\nparent in the split history that's being created. This then leads to\nunrelated history appearing in the split subtree.\n\nBelow is a shell script that demonstrates the issue - this is not the\nprecise case that I encountered in the FreeBSD repo, but the behaviour\nis identical (and it doesn't take nearly 10 minutes to run). Running\nthe script and then \"git log\" of the commit printed by the final (git\nsubtree) command includes the unrelated history in dir2/.\n\nIt looks like this comes from the cache_set \"$rev\" \"$rev\" in\nprocess_split_commit() added in 39f5fff0d53. This is under the\nsuspicious-looking \"ugly. is there no better way to tell if this is a\nsubtree vs. a mainline commit? Does it matter\" comment. However, I\ndon't yet understand enough of git-subtree's operation to propose a\nfix.\n\n--repro.sh--\n#!/bin/sh\n\nrm -rf subrepo-issue\nmkdir -p subrepo-issue\ncd subrepo-issue\n\ngit init .\nmkdir -p dir1 dir2\ntouch dir1/file1 dir2/file2\ngit add dir1 dir2\ngit commit -m 'initial commit'\necho 'file2' > dir2/file2\ngit commit -m 'file2 modified' dir2/file2\ngit rm dir1/file1\ngit commit -m 'remove file1'\nmkdir -p dir1\ntouch dir1/file1\ngit add dir1\ngit commit -m 'restore file1'\necho 'file1' > dir1/file1\ngit commit -m 'file1 modified' dir1/file1\ngit subtree split --prefix=dir1/\n"},{"id":"388412","messageId":"D4C58338-10C6-4E5A-BF1F-F48EC2EBDAD5@icloud.com","threadId":"52317","inReplyTo":"CAPyFy2AsmaxU-BDf_teZJE5hiaVpTSZc8fftnuXPb_4-j7j5Fw@mail.gmail.com","subject":"Re: git-subtree split misbehaviour with a commit having empty ls-tree for the specified subdir","fromName":"Tom Clarkson","fromEmail":"tqclarkson@icloud.com","sentAt":"2019-12-18T00:17:28Z","receivedAt":"2019-12-18T00:17:35Z","isPatch":false,"sender":{"key":"tqclarkson@icloud.com","avatar":"https://gravatar.com/avatar/02ea2942fcaf5dc1140ccda995e0dee545ff210ca8adee02d50f35de7205e301?d=mp&s=160"},"body":"\n\n> On 23 Nov 2019, at 3:55 am, Ed Maste <emaste@freebsd.org> wrote:\n> \n> I encountered an issue while trying to use git subtree with the\n> FreeBSD svn->git mirror: I found that when \"git subtree split\"\n> encounters a commit with an empty \"git ls-tree\" for the subdirectory\n> being split, it ends up recording the original parent as the new\n> parent in the split history that's being created. This then leads to\n> unrelated history appearing in the split subtree.\n> \n> Below is a shell script that demonstrates the issue - this is not the\n> precise case that I encountered in the FreeBSD repo, but the behaviour\n> is identical (and it doesn't take nearly 10 minutes to run). Running\n> the script and then \"git log\" of the commit printed by the final (git\n> subtree) command includes the unrelated history in dir2/.\n> \n> It looks like this comes from the cache_set \"$rev\" \"$rev\" in\n> process_split_commit() added in 39f5fff0d53. This is under the\n> suspicious-looking \"ugly. is there no better way to tell if this is a\n> subtree vs. a mainline commit? Does it matter\" comment. However, I\n> don't yet understand enough of git-subtree's operation to propose a\n> fix.\n> \n> --repro.sh--\n> #!/bin/sh\n> \n> rm -rf subrepo-issue\n> mkdir -p subrepo-issue\n> cd subrepo-issue\n> \n> git init .\n> mkdir -p dir1 dir2\n> touch dir1/file1 dir2/file2\n> git add dir1 dir2\n> git commit -m 'initial commit'\n> echo 'file2' > dir2/file2\n> git commit -m 'file2 modified' dir2/file2\n> git rm dir1/file1\n> git commit -m 'remove file1'\n> mkdir -p dir1\n> touch dir1/file1\n> git add dir1\n> git commit -m 'restore file1'\n> echo 'file1' > dir1/file1\n> git commit -m 'file1 modified' dir1/file1\n> git subtree split --prefix=dir1/\n> \n\n\nThe algorithm I am looking at to replace the file based mainline detection is\n\n - If subtree root is unknown (as on the initial split), everything is mainline.\n\n - If subtree root is reachable and mainline root is not, it’s a subtree commit \n\n - Otherwise, treat as mainline. This will also pick up commits from other subtrees but they hopefully won’t contain the subtree folder. I don’t think there is an unambiguous way to distinguish a subtree merge from a regular merge - the message produced is pretty generic. It may be possible to check reachability of all known subtrees, but that adds a fair bit of complexity.\n\nThat leaves us with the question of how to record the empty mainline commits. The most correct result for your repro is probably four commits (add/delete everything/restore/modify), but I can see that falling over in a scenario where deleting a subtree is more like unlinking a library than editing that library to do nothing.\n\nIs it sufficiently correct for your scenario to treat ‘restore file1’ as the initial subtree commit?\n\n"},{"id":"388445","messageId":"CAPyFy2AKSVQJtSY0RNgJDJ5k1P=-gjNXVjDgPh+CdghhZtJXDw@mail.gmail.com","threadId":"52317","inReplyTo":"D4C58338-10C6-4E5A-BF1F-F48EC2EBDAD5@icloud.com","subject":"Re: git-subtree split misbehaviour with a commit having empty ls-tree for the specified subdir","fromName":"Ed Maste","fromEmail":"emaste@freebsd.org","sentAt":"2019-12-18T10:23:06Z","receivedAt":"2019-12-18T17:58:49Z","isPatch":false,"sender":{"key":"emaste@freebsd.org","avatar":"https://avatars.githubusercontent.com/u/1034582?v=4"},"body":"On Tue, 17 Dec 2019 at 19:17, Tom Clarkson <tqclarkson@icloud.com> wrote:\n>\n> The algorithm I am looking at to replace the file based mainline detection is\n>\n>  - If subtree root is unknown (as on the initial split), everything is mainline.\n>\n>  - If subtree root is reachable and mainline root is not, it’s a subtree commit\n>\n>  - Otherwise, treat as mainline. This will also pick up commits from other subtrees but they hopefully won’t contain the subtree folder. I don’t think there is an unambiguous way to distinguish a subtree merge from a regular merge - the message produced is pretty generic. It may be possible to check reachability of all known subtrees, but that adds a fair bit of complexity.\n>\n> That leaves us with the question of how to record the empty mainline commits. The most correct result for your repro is probably four commits (add/delete everything/restore/modify), but I can see that falling over in a scenario where deleting a subtree is more like unlinking a library than editing that library to do nothing.\n>\n> Is it sufficiently correct for your scenario to treat ‘restore file1’ as the initial subtree commit?\n\nMy reproduction scenario is really a demonstration of the real issue I\nencountered. Running the initial \"subtree split\" on the real repo\ntakes about 40 minutes so I wanted something trivial that shows the\nsame issue. In the demonstration case (i.e., actually removing and\nreadding the subtree) I think it's reasonable to start with the commit\nthat added it back.\n\nOverall I think your proposed algorithm is reasonable (even though I\nthink it won't address some of the cases in our repo). Will your\nalgorithm allow us to pass $dir to git rev-list, for the initial\nsplit?\n\nMy actual issue stems from the way svn2git converted some odd svn\nhistory, and is described in more detail on the freebsd-git mailing\nlist at https://lists.freebsd.org/pipermail/freebsd-git/2019-November/000218.html.\n\nPerhaps we can have some command-line options to provide metadata for\ncases that cannot be inferred? The cases in our repo come from svn2git\ncreating subtree merges to represent updates from vendor code. AFAIK\nthese should be basically identical to what subtree creates, except\nthat we don't have any of the metadata it adds.\n\nFor a concrete example (from the repo at\nhttps://github.com/freebsd/freebsd), 7f3a50b3b9f8 is a mainline commit\nthat added a new subtree, from 9ee787636908. I think that if I could\ninform subtree split that 9ee787636908 is the root it would work for\nme.\n"},{"id":"388491","messageId":"F0FBE3B6-0DF5-40A4-B1A3-18EF65D48FF3@icloud.com","threadId":"52317","inReplyTo":"CAPyFy2AKSVQJtSY0RNgJDJ5k1P=-gjNXVjDgPh+CdghhZtJXDw@mail.gmail.com","subject":"Re: git-subtree split misbehaviour with a commit having empty ls-tree for the specified subdir","fromName":"Tom Clarkson","fromEmail":"tqclarkson@icloud.com","sentAt":"2019-12-19T00:57:38Z","receivedAt":"2019-12-19T00:57:45Z","isPatch":false,"sender":{"key":"tqclarkson@icloud.com","avatar":"https://gravatar.com/avatar/02ea2942fcaf5dc1140ccda995e0dee545ff210ca8adee02d50f35de7205e301?d=mp&s=160"},"body":"\n\n> On 19 Dec 2019, at 4:58 am, Ed Maste <emaste@freebsd.org> wrote:\n> \n> On Tue, 17 Dec 2019 at 19:17, Tom Clarkson <tqclarkson@icloud.com> wrote:\n>> \n>> The algorithm I am looking at to replace the file based mainline detection is\n>> \n>> - If subtree root is unknown (as on the initial split), everything is mainline.\n>> \n>> - If subtree root is reachable and mainline root is not, it’s a subtree commit\n>> \n>> - Otherwise, treat as mainline. This will also pick up commits from other subtrees but they hopefully won’t contain the subtree folder. I don’t think there is an unambiguous way to distinguish a subtree merge from a regular merge - the message produced is pretty generic. It may be possible to check reachability of all known subtrees, but that adds a fair bit of complexity.\n>> \n>> That leaves us with the question of how to record the empty mainline commits. The most correct result for your repro is probably four commits (add/delete everything/restore/modify), but I can see that falling over in a scenario where deleting a subtree is more like unlinking a library than editing that library to do nothing.\n>> \n>> Is it sufficiently correct for your scenario to treat ‘restore file1’ as the initial subtree commit?\n> \n> My reproduction scenario is really a demonstration of the real issue I\n> encountered. Running the initial \"subtree split\" on the real repo\n> takes about 40 minutes so I wanted something trivial that shows the\n> same issue. In the demonstration case (i.e., actually removing and\n> readding the subtree) I think it's reasonable to start with the commit\n> that added it back.\n> \n> Overall I think your proposed algorithm is reasonable (even though I\n> think it won't address some of the cases in our repo). Will your\n> algorithm allow us to pass $dir to git rev-list, for the initial\n> split?\n\nIs this just for performance reasons? As I understand it that was left out because it would exclude relevant commits on an existing subtree, but it could make sense as an optimization for the first split of a large repo.\n\n> My actual issue stems from the way svn2git converted some odd svn\n> history, and is described in more detail on the freebsd-git mailing\n> list at https://lists.freebsd.org/pipermail/freebsd-git/2019-November/000218.html.\n> \n> Perhaps we can have some command-line options to provide metadata for\n> cases that cannot be inferred? The cases in our repo come from svn2git\n> creating subtree merges to represent updates from vendor code. AFAIK\n> these should be basically identical to what subtree creates, except\n> that we don't have any of the metadata it adds.\n\nThe existing --onto option comes pretty close - it marks everything in the rev-list of $onto as a subtree commit to be used as-is\n\nFor more flexibility, I think allowing more manipulation of the cache is the way to go - $cachedir is currently based on process id, but I don’t see any reason it can’t be based on prefix instead. So the process becomes something like\n\n # clear the cache - shouldn't usually be necessary, but it's a universal debugging step.\ngit subtree clear-cache --prefix=dir\n\n# ref and all its parents are before subtree add. Treat any children as inital commits.\ngit subtree ignore --prefix=dir ref\n\n# ref and all its parents are known subtree commits to be included without transformation.\ngit subtree existing --prefix=dir ref\n\n# Override an arbitrary mapping, either for performance or because that commit is problematic \ngit subtree map --prefix=dir mainline-ref subtree-ref\n\n# Run the existing algorithm, but skipping anything defined manually\ngit subtree split --prefix=dir\n\n\n> For a concrete example (from the repo at\n> https://github.com/freebsd/freebsd), 7f3a50b3b9f8 is a mainline commit\n> that added a new subtree, from 9ee787636908. I think that if I could\n> inform subtree split that 9ee787636908 is the root it would work for\n> me.\n\nAside from the metadata, that one is a bit different from a standard subtree add in that it copies three folders from the subtree repo rather than the root - so the contents of contrib/elftoolchain will never exactly match the actual elftoolchain repo, and 9ee787636908 is neither mainline nor subtree as subtree split understands it.\n\nIf you ignore 9ee787636908, the resulting subtree will be fairly clean, but won’t have much of a relationship to the external repo.\n\nIf you treat 9ee787636908 as an existing subtree, the second commit on your subtree will be based on 7f3a50b3b9f8, which deletes most of the contents of the subtree. You should still be able to merge in updates from the external repo, but if you try to push changes upstream the deletion will break things.\n\n\n"},{"id":"388614","messageId":"CAPyFy2Ar+OncJtgZZyAzxs0PkXy5rSU6ALS+MimK8x5TzWjLug@mail.gmail.com","threadId":"52317","inReplyTo":"F0FBE3B6-0DF5-40A4-B1A3-18EF65D48FF3@icloud.com","subject":"Re: git-subtree split misbehaviour with a commit having empty ls-tree for the specified subdir","fromName":"Ed Maste","fromEmail":"emaste@freebsd.org","sentAt":"2019-12-20T15:56:29Z","receivedAt":"2019-12-20T15:56:43Z","isPatch":false,"sender":{"key":"emaste@freebsd.org","avatar":"https://avatars.githubusercontent.com/u/1034582?v=4"},"body":"On Wed, 18 Dec 2019 at 19:57, Tom Clarkson <tqclarkson@icloud.com> wrote:\n>\n> > Overall I think your proposed algorithm is reasonable (even though I\n> > think it won't address some of the cases in our repo). Will your\n> > algorithm allow us to pass $dir to git rev-list, for the initial\n> > split?\n>\n> Is this just for performance reasons? As I understand it that was left out because it would exclude relevant commits on an existing subtree, but it could make sense as an optimization for the first split of a large repo.\n\nYes, it's for performance reasons on a first split that I'd like to\nsee it. On the FreeBSD repo the difference is some 40 minutes vs. a\nfew seconds.\n\n> So the process becomes something like\n>\n>  # clear the cache - shouldn't usually be necessary, but it's a universal debugging step.\n> git subtree clear-cache --prefix=dir\n>\n> # ref and all its parents are before subtree add. Treat any children as inital commits.\n> git subtree ignore --prefix=dir ref\n>\n> # ref and all its parents are known subtree commits to be included without transformation.\n> git subtree existing --prefix=dir ref\n>\n> # Override an arbitrary mapping, either for performance or because that commit is problematic\n> git subtree map --prefix=dir mainline-ref subtree-ref\n>\n> # Run the existing algorithm, but skipping anything defined manually\n> git subtree split --prefix=dir\n\nThis sounds about perfect.\n\n> > For a concrete example (from the repo at\n> > https://github.com/freebsd/freebsd), 7f3a50b3b9f8 is a mainline commit\n> > that added a new subtree, from 9ee787636908. I think that if I could\n> > inform subtree split that 9ee787636908 is the root it would work for\n> > me.\n>\n> Aside from the metadata, that one is a bit different from a standard subtree add in that it copies three folders from the subtree repo rather than the root - so the contents of contrib/elftoolchain will never exactly match the actual elftoolchain repo, and 9ee787636908 is neither mainline nor subtree as subtree split understands it.\n\nFair enough, and we have lots of examples of slightly strange history\nin svn that svn2git represents in interesting ways.\n\n> If you ignore 9ee787636908, the resulting subtree will be fairly clean, but won’t have much of a relationship to the external repo.\n>\n> If you treat 9ee787636908 as an existing subtree, the second commit on your subtree will be based on 7f3a50b3b9f8, which deletes most of the contents of the subtree. You should still be able to merge in updates from the external repo, but if you try to push changes upstream the deletion will break things.\n\nI think this is fine - our main goal here is to be able to update\ncontrib/ code within FreeBSD as we do today with svn, and we may well\nalways have some changes that are never intended to be pushed\nupstream.\n\nContinuing the example from our repo, there is more history in the\n\"subtree\" already, with 061ef1f9424f as the head. ca8624403626 is the\nmerge to mainline.\n"},{"id":"388796","messageId":"905A443A-7E2B-45C2-985F-46C3E295670A@icloud.com","threadId":"52317","inReplyTo":"CAPyFy2Ar+OncJtgZZyAzxs0PkXy5rSU6ALS+MimK8x5TzWjLug@mail.gmail.com","subject":"Re: git-subtree split misbehaviour with a commit having empty ls-tree for the specified subdir","fromName":"Tom Clarkson","fromEmail":"tqclarkson@icloud.com","sentAt":"2019-12-22T14:01:47Z","receivedAt":"2019-12-22T14:01:55Z","isPatch":false,"sender":{"key":"tqclarkson@icloud.com","avatar":"https://gravatar.com/avatar/02ea2942fcaf5dc1140ccda995e0dee545ff210ca8adee02d50f35de7205e301?d=mp&s=160"},"body":"\n> On 21 Dec 2019, at 2:56 am, Ed Maste <emaste@freebsd.org> wrote:\n> \n> On Wed, 18 Dec 2019 at 19:57, Tom Clarkson <tqclarkson@icloud.com> wrote:\n>> \n>>> Overall I think your proposed algorithm is reasonable (even though I\n>>> think it won't address some of the cases in our repo). Will your\n>>> algorithm allow us to pass $dir to git rev-list, for the initial\n>>> split?\n>> \n>> Is this just for performance reasons? As I understand it that was left out because it would exclude relevant commits on an existing subtree, but it could make sense as an optimization for the first split of a large repo.\n> \n> Yes, it's for performance reasons on a first split that I'd like to\n> see it. On the FreeBSD repo the difference is some 40 minutes vs. a\n> few seconds.\n\nI tried out the dir filter after getting the full revlist to produce a reasonable result.  It is a lot faster, but unfortunately it doesn’t produce the same output - If you use actual parent commits, you have to process the 50k irrelevant ones to keep a valid path. If you let rev-list find what it thinks is the most recent relevant change, the subtree merges resolve to nothing.\n\n>> So the process becomes something like\n>> \n>> # clear the cache - shouldn't usually be necessary, but it's a universal debugging step.\n>> git subtree clear-cache --prefix=dir\n>> \n>> # ref and all its parents are before subtree add. Treat any children as inital commits.\n>> git subtree ignore --prefix=dir ref\n>> \n>> # ref and all its parents are known subtree commits to be included without transformation.\n>> git subtree existing --prefix=dir ref\n>> \n>> # Override an arbitrary mapping, either for performance or because that commit is problematic\n>> git subtree map --prefix=dir mainline-ref subtree-ref\n>> \n>> # Run the existing algorithm, but skipping anything defined manually\n>> git subtree split --prefix=dir\n> \n> This sounds about perfect.\n> \n>>> For a concrete example (from the repo at\n>>> https://github.com/freebsd/freebsd), 7f3a50b3b9f8 is a mainline commit\n>>> that added a new subtree, from 9ee787636908. I think that if I could\n>>> inform subtree split that 9ee787636908 is the root it would work for\n>>> me.\n>> \n>> Aside from the metadata, that one is a bit different from a standard subtree add in that it copies three folders from the subtree repo rather than the root - so the contents of contrib/elftoolchain will never exactly match the actual elftoolchain repo, and 9ee787636908 is neither mainline nor subtree as subtree split understands it.\n> \n> Fair enough, and we have lots of examples of slightly strange history\n> in svn that svn2git represents in interesting ways.\n> \n>> If you ignore 9ee787636908, the resulting subtree will be fairly clean, but won’t have much of a relationship to the external repo.\n>> \n>> If you treat 9ee787636908 as an existing subtree, the second commit on your subtree will be based on 7f3a50b3b9f8, which deletes most of the contents of the subtree. You should still be able to merge in updates from the external repo, but if you try to push changes upstream the deletion will break things.\n> \n> I think this is fine - our main goal here is to be able to update\n> contrib/ code within FreeBSD as we do today with svn, and we may well\n> always have some changes that are never intended to be pushed\n> upstream.\n> \n> Continuing the example from our repo, there is more history in the\n> \"subtree\" already, with 061ef1f9424f as the head. ca8624403626 is the\n> merge to mainline.\n\n\nIf you want to try out my update, it’s at  https://github.com/gitgitgadget/git/pull/493. The commands I ended up with were\n\ngit subtree ignore --clear-cache --prefix=contrib/elftoolchain 4d43158\ngit subtree use --prefix=contrib/elftoolchain 9e78763\ngit subtree split --prefix=contrib/elftoolchain 53f2672ff78be42389cf41a8258f6e9ce36808fb\n\nOn my machine, ignore takes about 2 minutes to flag 200k commits as irrelevant. The split takes around 15 to go through the remaining 50k."},{"id":"390212","messageId":"CAPyFy2A+8mK3cBYYf4W3wg-qXR1S5wkX4kAOk2BXgG=hVefbYA@mail.gmail.com","threadId":"52317","inReplyTo":"905A443A-7E2B-45C2-985F-46C3E295670A@icloud.com","subject":"Re: git-subtree split misbehaviour with a commit having empty ls-tree for the specified subdir","fromName":"Ed Maste","fromEmail":"emaste@freebsd.org","sentAt":"2020-01-21T22:36:00Z","receivedAt":"2020-01-21T22:36:14Z","isPatch":false,"sender":{"key":"emaste@freebsd.org","avatar":"https://avatars.githubusercontent.com/u/1034582?v=4"},"body":"On Sun, 22 Dec 2019 at 09:01, Tom Clarkson <tqclarkson@icloud.com> wrote:\n>\n> If you want to try out my update, it’s at  https://github.com/gitgitgadget/git/pull/493. The commands I ended up with were\n>\n> git subtree ignore --clear-cache --prefix=contrib/elftoolchain 4d43158\n> git subtree use --prefix=contrib/elftoolchain 9e78763\n> git subtree split --prefix=contrib/elftoolchain 53f2672ff78be42389cf41a8258f6e9ce36808fb\n\nThanks Tom, I was finally able to get back to this, and confirm it\nworks as I'd expect / desire. I'll continue experimenting with the\nrest of the contrib/ software in FreeBSD; please let me know if\nthere's anything specific you'd like me to test with your patch set.\n"},{"id":"396483","messageId":"CAPyFy2CqAQWSxzVkfNqk5k=Tq_N82_62Z-rTawen9TtmdW9Ytg@mail.gmail.com","threadId":"52317","inReplyTo":"DB65AE2F-12DE-43B7-8B20-4E173794CAF2@icloud.com","subject":"Re: git-subtree split misbehaviour with a commit having empty ls-tree for the specified subdir","fromName":"Ed Maste","fromEmail":"emaste@freebsd.org","sentAt":"2020-04-28T18:08:26Z","receivedAt":"2020-04-28T18:08:41Z","isPatch":false,"sender":{"key":"emaste@freebsd.org","avatar":"https://avatars.githubusercontent.com/u/1034582?v=4"},"body":"On Sun, 22 Dec 2019 at 08:50, Tom Clarkson <tqclarkson@icloud.com> wrote:\n>\n> If you want to try out my update, it’s at  https://github.com/gitgitgadget/git/pull/493. The commands I ended up with were\n>\n> git subtree ignore --clear-cache --prefix=contrib/elftoolchain 4d43158\n> git subtree use --prefix=contrib/elftoolchain 9e78763\n> git subtree split --prefix=contrib/elftoolchain 53f2672ff78be42389cf41a8258f6e9ce36808fb\n>\n> On my machine, ignore takes about 2 minutes to flag 200k commits as irrelevant. The split takes around 15 to go through the remaining 50k.\n\nWhat's the next step with this patch set? Is there anything I can do to help?\n"},{"id":"399973","messageId":"CAPyFy2CMSGwPgGLh2Jbfvuf8oRBcvZ1LRv-m7AVvPybtpEybnw@mail.gmail.com","threadId":"52317","inReplyTo":"CAPyFy2Ar+OncJtgZZyAzxs0PkXy5rSU6ALS+MimK8x5TzWjLug@mail.gmail.com","subject":"Re: git-subtree split misbehaviour with a commit having empty ls-tree for the specified subdir","fromName":"Ed Maste","fromEmail":"emaste@freebsd.org","sentAt":"2020-06-17T14:46:40Z","receivedAt":"2020-06-17T14:46:56Z","isPatch":false,"sender":{"key":"emaste@freebsd.org","avatar":"https://avatars.githubusercontent.com/u/1034582?v=4"},"body":"On Fri, 20 Dec 2019 at 10:56, Ed Maste <emaste@freebsd.org> wrote:\n>\n> On Wed, 18 Dec 2019 at 19:57, Tom Clarkson <tqclarkson@icloud.com> wrote:\n> >\n> > > Overall I think your proposed algorithm is reasonable (even though I\n> > > think it won't address some of the cases in our repo). Will your\n> > > algorithm allow us to pass $dir to git rev-list, for the initial\n> > > split?\n> >\n> > Is this just for performance reasons? As I understand it that was left out because it would exclude relevant commits on an existing subtree, but it could make sense as an optimization for the first split of a large repo.\n>\n> Yes, it's for performance reasons on a first split that I'd like to\n> see it. On the FreeBSD repo the difference is some 40 minutes vs. a\n> few seconds.\n\nFollowing up on this old thread, I plan to revisit the optimization,\nimplementing something on top of your work in\nhttps://github.com/gitgitgadget/git/pull/493. I might look at adding a\n--initial flag to subtree split, having it essentially auto-detect a\nrevision to use as the value for --onto. For the common case of an\ninitial merge commit with two parents I think we can relatively easily\ndetermine which is the subtree parent. If that's not sufficiently\ngeneral (or broadly useful outside of our context) we could just\ncreate a helper script wrapping `subtree split` tailored to the\nFreeBSD cases. We have something like 100 projects we're looking to\nsplit, as part of our svn to git migration.\n"},{"id":"400023","messageId":"5CD94CF2-48D4-4EFD-9581-625E6C117F89@icloud.com","threadId":"52317","inReplyTo":"CAPyFy2CMSGwPgGLh2Jbfvuf8oRBcvZ1LRv-m7AVvPybtpEybnw@mail.gmail.com","subject":"Re: git-subtree split misbehaviour with a commit having empty ls-tree for the specified subdir","fromName":"Tom Clarkson","fromEmail":"tqclarkson@icloud.com","sentAt":"2020-06-18T01:13:02Z","receivedAt":"2020-06-18T02:19:00Z","isPatch":false,"sender":{"key":"tqclarkson@icloud.com","avatar":"https://gravatar.com/avatar/02ea2942fcaf5dc1140ccda995e0dee545ff210ca8adee02d50f35de7205e301?d=mp&s=160"},"body":"\n> On 18 Jun 2020, at 12:46 am, Ed Maste <emaste@freebsd.org> wrote:\n> \n> On Fri, 20 Dec 2019 at 10:56, Ed Maste <emaste@freebsd.org> wrote:\n>> \n>> On Wed, 18 Dec 2019 at 19:57, Tom Clarkson <tqclarkson@icloud.com> wrote:\n>>> \n>>>> Overall I think your proposed algorithm is reasonable (even though I\n>>>> think it won't address some of the cases in our repo). Will your\n>>>> algorithm allow us to pass $dir to git rev-list, for the initial\n>>>> split?\n>>> \n>>> Is this just for performance reasons? As I understand it that was left out because it would exclude relevant commits on an existing subtree, but it could make sense as an optimization for the first split of a large repo.\n>> \n>> Yes, it's for performance reasons on a first split that I'd like to\n>> see it. On the FreeBSD repo the difference is some 40 minutes vs. a\n>> few seconds.\n> \n> Following up on this old thread, I plan to revisit the optimization,\n> implementing something on top of your work in\n> https://github.com/gitgitgadget/git/pull/493. I might look at adding a\n> --initial flag to subtree split, having it essentially auto-detect a\n> revision to use as the value for --onto. For the common case of an\n> initial merge commit with two parents I think we can relatively easily\n> determine which is the subtree parent. If that's not sufficiently\n> general (or broadly useful outside of our context) we could just\n> create a helper script wrapping `subtree split` tailored to the\n> FreeBSD cases. We have something like 100 projects we're looking to\n> split, as part of our svn to git migration.\n\nThe new use command might be a better fit than onto in this case - it does the same thing as onto, except it also marks the commit as processed and therefore excludes them from the initial rev list.\n\nActually, on reading the code, I’m not sure onto does quite what the documentation suggests it does - by updating the cache it will shortcut processing of subtree commits that have already been merged into mainline, but has no mechanism for building onto an existing unrelated history.\n\nReliably differentiating subtree and mainline commits has always been tricky, but should be ok as part of an advanced flag/new command. Perhaps rev-list --merges <path> to find potential unmarked subtree merges, then take the one where the root tree matches the post merge subdir tree. No doubt it won’t catch everything, but I’d say that’s less of a risk than false positives.\n\nIn the context of a helper script, a new command or adding a --auto flag to use might be better than adding a flag to split - that way you could easily tell if the expected initial state was found rather than having to wait for the full process to produce something weird. \n\nThat would also let you mark the other side of the merge as ignored mainline history - a significant optimization when you’re excluding 200k commits, but risky to include more generally.\n\n"}]}