{"thread":{"id":"44269","subject":"git filter-branch --subdirectory-filter not working as expected, history of other folders is preserved","startedAt":"2016-10-10T13:43:02Z","lastAt":"2016-10-11T13:57:07Z","messageCount":5,"participants":["Seaders Oloinsigh","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"303722","messageId":"CAN40BKqPdPP2+K4FBzgDDfYiGkzk1gYcOeP==_t+pr5w0rj0EQ@mail.gmail.com","threadId":"44269","inReplyTo":null,"subject":"git filter-branch --subdirectory-filter not working as expected, history of other folders is preserved","fromName":"Seaders Oloinsigh","fromEmail":"seaders69@gmail.com","sentAt":"2016-10-10T13:42:36Z","receivedAt":"2016-10-10T13:43:02Z","isPatch":false,"sender":{"key":"seaders69@gmail.com","avatar":null},"body":"From\n\nhttps://git-scm.com/book/en/v2/Git-Tools-Rewriting-History\n\nNow your new project root is what was in the trunk subdirectory each time.\n> Git will also automatically remove commits that did not affect the\n> subdirectory.\n>\n\nWe have a git repository that looks like\n\nsdk/\nandroid/\nios/\nunity/\nwindows/\n\nWhich we'd like to split into 4 repositories, 1 for each platform.  To\nstart this process (for splitting android out), I ran,\n\ngit filter-branch -f --prune-empty --subdirectory-filter android -- --all\n\nWhich rewrote a ton of history and commits, and looked like it worked, but\non closer inspection had left a ton of history behind.\n\nIf I run\n\ngit log --all -- unity/\n\nIt returns a list of commits that happened in the unity/ subfolder of the\noriginal root.\n\ncommit c4ea2797...\n> Author: tom... <tom@...ve.com>\n> Date:   Thu Feb 25 14:20:59 2016 +0000\n>\n>     kick off build\n>\n\n> ...\n>\n\n\nWhich only contains an edit to a file with path \"unity/tom\" relative to the\nroot *before* the filter-branch, doesn't exist any more, and from my\nunderstanding of the docs, shouldn't have been taken across.\n\nIt's also not an isolated instance, if I run the same checks against\n\"ios/\", \"windows/\", any file that existed in a folder other than \"android\"\nto the old root, that history is also preserved.\n\nI've just about resorted to running multiple other, explicit filters to\nremove all references to those other folders, but it seems like this would\nbe redoing the job that I understood git filter-branch should have been\ndoing.\n\nAny help in this regard is greatly appreciated.\n\nseaders.\n"},{"id":"303727","messageId":"20161010153032.v2x773cbs4ifvzec@sigill.intra.peff.net","threadId":"44269","inReplyTo":"CAN40BKqPdPP2+K4FBzgDDfYiGkzk1gYcOeP==_t+pr5w0rj0EQ@mail.gmail.com","subject":"Re: git filter-branch --subdirectory-filter not working as expected, history of other folders is preserved","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-10-10T15:30:32Z","receivedAt":"2016-10-10T15:30:38Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Oct 10, 2016 at 02:42:36PM +0100, Seaders Oloinsigh wrote:\n\n> We have a git repository that looks like\n> \n> sdk/\n> android/\n> ios/\n> unity/\n> windows/\n> \n> Which we'd like to split into 4 repositories, 1 for each platform.  To\n> start this process (for splitting android out), I ran,\n> \n> git filter-branch -f --prune-empty --subdirectory-filter android -- --all\n\nOK, so that should rewrite each ref to have only the contents of the\n\"android\" directory at the top-level.\n\nNote that filter-branch saves a copy of the old refs in refs/original.\n\n> Which rewrote a ton of history and commits, and looked like it worked, but\n> on closer inspection had left a ton of history behind.\n> \n> If I run\n> \n> git log --all -- unity/\n> \n> It returns a list of commits that happened in the unity/ subfolder of the\n> original root.\n\nHere you asked for \"--all\", which includes refs/original. So you are\nseeing the original, unwritten commits (and none of your new ones, of\ncourse, because they do not have a unity/ directory!).\n\nTry:\n\n  git log --all --source -- unity\n\nto see which ref each commit is coming from.\n\nOr try:\n\n  git log --branches --tags -- unity\n\nto confirm that your branches and tags do not include that path.\n\nOr just:\n\n  git for-each-ref --format='delete %(refname)' refs/original |\n  git update-ref --stdin\n\nto get rid of the backup refs entirely.\n\n-Peff\n"},{"id":"303728","messageId":"CAN40BKoWroM4ZJHHdMvO8YvvCho0FnWzV9BG9E5xnCZbYOO2pg@mail.gmail.com","threadId":"44269","inReplyTo":"20161010153032.v2x773cbs4ifvzec@sigill.intra.peff.net","subject":"Re: git filter-branch --subdirectory-filter not working as expected, history of other folders is preserved","fromName":"Seaders Oloinsigh","fromEmail":"seaders69@gmail.com","sentAt":"2016-10-10T16:12:25Z","receivedAt":"2016-10-10T16:13:00Z","isPatch":false,"sender":{"key":"seaders69@gmail.com","avatar":null},"body":"Thanks for the reply, Jeff.\n\nClearing the backups of the branches, those starting with\n\"refs/original\" has gotten me closer, but I also needed to do that\nwith \"refs/tags\" as well, or change my filter-branch command to,\n\n  git filter-branch -f --prune-empty --tag-name-filter cat\n--subdirectory-filter android -- --all\n\nI still have remnants of that other history, though.\n\nDue to the structure of this repo, it looks like there are some\nbranches that never had anything to do with the android/ subdirectory,\nso they're not getting wiped out.  My branch is in a better state to\nhow I want it, but still, if I run your suggestion,\n\n  git log --all --source -- unity/\n\nI get output like\n\n> commit 4853c... refs/heads/unity-sdk-3_1_3\n> Author: serg... <serg...@...ve.com>\n> Date:   Thu Sep 11 16:30:01 2014 +0100\n>\n>    Started 3.1.3\n\nWhich is basically logs of branches which contain only edits within\nthe unity/ subdirectory of the original root.  There are other\nbranches like that for the other platforms / subdirectories of the\noriginal root, which if that is the case, I would consider\nfilter-branch with the subdirectory-filter isn't acting as expected,\nand doesn't get rid of all the history you want it to get rid of.\n\n\nOn Mon, Oct 10, 2016 at 4:30 PM, Jeff King <peff@peff.net> wrote:\n> On Mon, Oct 10, 2016 at 02:42:36PM +0100, Seaders Oloinsigh wrote:\n>\n>> We have a git repository that looks like\n>>\n>> sdk/\n>> android/\n>> ios/\n>> unity/\n>> windows/\n>>\n>> Which we'd like to split into 4 repositories, 1 for each platform.  To\n>> start this process (for splitting android out), I ran,\n>>\n>> git filter-branch -f --prune-empty --subdirectory-filter android -- --all\n>\n> OK, so that should rewrite each ref to have only the contents of the\n> \"android\" directory at the top-level.\n>\n> Note that filter-branch saves a copy of the old refs in refs/original.\n>\n>> Which rewrote a ton of history and commits, and looked like it worked, but\n>> on closer inspection had left a ton of history behind.\n>>\n>> If I run\n>>\n>> git log --all -- unity/\n>>\n>> It returns a list of commits that happened in the unity/ subfolder of the\n>> original root.\n>\n> Here you asked for \"--all\", which includes refs/original. So you are\n> seeing the original, unwritten commits (and none of your new ones, of\n> course, because they do not have a unity/ directory!).\n>\n> Try:\n>\n>   git log --all --source -- unity\n>\n> to see which ref each commit is coming from.\n>\n> Or try:\n>\n>   git log --branches --tags -- unity\n>\n> to confirm that your branches and tags do not include that path.\n>\n> Or just:\n>\n>   git for-each-ref --format='delete %(refname)' refs/original |\n>   git update-ref --stdin\n>\n> to get rid of the backup refs entirely.\n>\n> -Peff\n"},{"id":"303775","messageId":"20161010181907.bqekupn6npuimbir@sigill.intra.peff.net","threadId":"44269","inReplyTo":"CAN40BKoWroM4ZJHHdMvO8YvvCho0FnWzV9BG9E5xnCZbYOO2pg@mail.gmail.com","subject":"Re: git filter-branch --subdirectory-filter not working as expected, history of other folders is preserved","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-10-10T18:19:07Z","receivedAt":"2016-10-10T18:19:14Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Oct 10, 2016 at 05:12:25PM +0100, Seaders Oloinsigh wrote:\n\n> Due to the structure of this repo, it looks like there are some\n> branches that never had anything to do with the android/ subdirectory,\n> so they're not getting wiped out.  My branch is in a better state to\n> how I want it, but still, if I run your suggestion,\n> [...]\n\nHmm. Yeah, I think this is an artifact of the way that filter-branch\nworks with pathspec limiting. It keeps a mapping of commits that it has\nrewritten (including ones that were rewritten only because their\nancestors were), and realizes that a branch ref needs updated when the\ncommit it points to was rewritten.\n\nBut if we don't touch _any_ commits in the history reachable from a\nbranch (because they didn't even show up in our pathspec-limited\nrev-list), then it doesn't realize we touched the branch's history at\nall.\n\nI agree that the right outcome is for it to delete those branches\nentirely. I suspect the fix would be pretty tricky, though.\n\nIn the meantime, I think you can work around it by either:\n\n  1. Make a pass beforehand for refs that do not touch your desired\n     paths at all, like:\n\n       path=android ;# or whatever\n       git for-each-ref --format='%(refname)' |\n       while read ref; do\n         if test \"$(git rev-list --count \"$ref\" -- \"$path\")\" = 0; then\n\t   echo \"delete $ref\"\n\t fi\n       done |\n       git update-ref --stdin\n\n     and then filter what's left:\n\n       git filter-branch --subdirectory-filter $path -- --all\n\nor\n\n  2. Do the filter-branch, and because you know you specified --all and\n     that your filters would touch all histories, any ref which _wasn't_\n     touched can be deleted. That list is anything which didn't get a\n     backup entry in refs/original. So something like:\n\n       git for-each-ref --format='%(refname)' |\n       perl -lne 'print $1 if m{^refs/original/(.*)}' >backups\n\n       git for-each-ref --format='%(refname)' |\n       grep -v ^refs/original >refs\n\n       comm -23 refs backups |\n       sed \"s/^/delete /\" |\n       git update-ref --stdin\n\n-Peff\n"},{"id":"303871","messageId":"CAN40BKoHGHWrNyTg3h=VJdBtzpQbWXbc1F+2j+mrPgPC7uUmMQ@mail.gmail.com","threadId":"44269","inReplyTo":"20161010181907.bqekupn6npuimbir@sigill.intra.peff.net","subject":"Re: git filter-branch --subdirectory-filter not working as expected, history of other folders is preserved","fromName":"Seaders Oloinsigh","fromEmail":"seaders69@gmail.com","sentAt":"2016-10-11T13:56:39Z","receivedAt":"2016-10-11T13:57:07Z","isPatch":false,"sender":{"key":"seaders69@gmail.com","avatar":null},"body":"On Mon, Oct 10, 2016 at 7:19 PM, Jeff King <peff@peff.net> wrote:\n> On Mon, Oct 10, 2016 at 05:12:25PM +0100, Seaders Oloinsigh wrote:\n>\n>> Due to the structure of this repo, it looks like there are some\n>> branches that never had anything to do with the android/ subdirectory,\n>> so they're not getting wiped out.  My branch is in a better state to\n>> how I want it, but still, if I run your suggestion,\n>> [...]\n>\n> Hmm. Yeah, I think this is an artifact of the way that filter-branch\n> works with pathspec limiting. It keeps a mapping of commits that it has\n> rewritten (including ones that were rewritten only because their\n> ancestors were), and realizes that a branch ref needs updated when the\n> commit it points to was rewritten.\n>\n> But if we don't touch _any_ commits in the history reachable from a\n> branch (because they didn't even show up in our pathspec-limited\n> rev-list), then it doesn't realize we touched the branch's history at\n> all.\n>\n> I agree that the right outcome is for it to delete those branches\n> entirely. I suspect the fix would be pretty tricky, though.\n>\n> In the meantime, I think you can work around it by either:\n>\n>   1. Make a pass beforehand for refs that do not touch your desired\n>      paths at all, like:\n>\n>        path=android ;# or whatever\n>        git for-each-ref --format='%(refname)' |\n>        while read ref; do\n>          if test \"$(git rev-list --count \"$ref\" -- \"$path\")\" = 0; then\n>            echo \"delete $ref\"\n>          fi\n>        done |\n>        git update-ref --stdin\n>\n>      and then filter what's left:\n>\n>        git filter-branch --subdirectory-filter $path -- --all\n\nThis is the perfect solution for me.  Going through the delete\nbranches runthrough also quickened the filter-branch command, and I'm\nleft with a much more complete version of where I want to be.\n\nI would still contend that the filter-branch either doesn't work as\nexpected, or the docs need updating to provide extra steps like you've\ndone, because when dealing with a large repo like we have, running\nmultiple filter-branch commands, trying different combinations is\nquite a time sync, when you're left with the same incorrect solution\nagain and again.\n\n>\n> or\n>\n>   2. Do the filter-branch, and because you know you specified --all and\n>      that your filters would touch all histories, any ref which _wasn't_\n>      touched can be deleted. That list is anything which didn't get a\n>      backup entry in refs/original. So something like:\n>\n>        git for-each-ref --format='%(refname)' |\n>        perl -lne 'print $1 if m{^refs/original/(.*)}' >backups\n>\n>        git for-each-ref --format='%(refname)' |\n>        grep -v ^refs/original >refs\n>\n>        comm -23 refs backups |\n>        sed \"s/^/delete /\" |\n>        git update-ref --stdin\n>\n> -Peff\n"}]}