git filter-branch --subdirectory-filter not working as expected, history of other folders is preserved

From

https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History

Now your new project root is what was in the trunk subdirectory each time.

We have a git repository that looks like

sdk/ android/ ios/ unity/ windows/

Which we'd like to split into 4 repositories, 1 for each platform. To start this process (for splitting android out), I ran,

git filter-branch -f --prune-empty --subdirectory-filter android -- --all

Which rewrote a ton of history and commits, and looked like it worked, but on closer inspection had left a ton of history behind.

If I run

git log --all -- unity/

It returns a list of commits that happened in the unity/ subfolder of the original root.

commit c4ea2797...

Which only contains an edit to a file with path "unity/tom" relative to the root *before* the filter-branch, doesn't exist any more, and from my understanding of the docs, shouldn't have been taken across.

It's also not an isolated instance, if I run the same checks against "ios/", "windows/", any file that existed in a folder other than "android" to the old root, that history is also preserved.

I've just about resorted to running multiple other, explicit filters to remove all references to those other folders, but it seems like this would be redoing the job that I understood git filter-branch should have been doing.

Any help in this regard is greatly appreciated.

seaders.

On Mon, Oct 10, 2016 at 02:42:36PM +0100, Seaders Oloinsigh wrote:

OK, so that should rewrite each ref to have only the contents of the "android" directory at the top-level.

Note that filter-branch saves a copy of the old refs in refs/original.

Here you asked for "--all", which includes refs/original. So you are seeing the original, unwritten commits (and none of your new ones, of course, because they do not have a unity/ directory!).

Try:

  git log --all --source -- unity

to see which ref each commit is coming from.

Or try:

  git log --branches --tags -- unity

to confirm that your branches and tags do not include that path.

Or just:

  git for-each-ref --format='delete %(refname)' refs/original |
  git update-ref --stdin

to get rid of the backup refs entirely.

-Peff

Thanks for the reply, Jeff.

Clearing the backups of the branches, those starting with "refs/original" has gotten me closer, but I also needed to do that with "refs/tags" as well, or change my filter-branch command to,

  git filter-branch -f --prune-empty --tag-name-filter cat
--subdirectory-filter android -- --all

I still have remnants of that other history, though.

Due to the structure of this repo, it looks like there are some branches that never had anything to do with the android/ subdirectory, so they're not getting wiped out. My branch is in a better state to how I want it, but still, if I run your suggestion,

  git log --all --source -- unity/

I get output like

Which is basically logs of branches which contain only edits within the unity/ subdirectory of the original root. There are other branches like that for the other platforms / subdirectories of the original root, which if that is the case, I would consider filter-branch with the subdirectory-filter isn't acting as expected, and doesn't get rid of all the history you want it to get rid of.

On Mon, Oct 10, 2016 at 4:30 PM, Jeff King <peff@peff.net> wrote:

On Mon, Oct 10, 2016 at 05:12:25PM +0100, Seaders Oloinsigh wrote:

Hmm. Yeah, I think this is an artifact of the way that filter-branch works with pathspec limiting. It keeps a mapping of commits that it has rewritten (including ones that were rewritten only because their ancestors were), and realizes that a branch ref needs updated when the commit it points to was rewritten.

But if we don't touch _any_ commits in the history reachable from a branch (because they didn't even show up in our pathspec-limited rev-list), then it doesn't realize we touched the branch's history at all.

I agree that the right outcome is for it to delete those branches entirely. I suspect the fix would be pretty tricky, though.

In the meantime, I think you can work around it by either:

  1. Make a pass beforehand for refs that do not touch your desired
     paths at all, like:

       path=android ;# or whatever
       git for-each-ref --format='%(refname)' |
       while read ref; do
         if test "$(git rev-list --count "$ref" -- "$path")" = 0; then
	   echo "delete $ref"
	 fi
       done |
       git update-ref --stdin

     and then filter what's left:

       git filter-branch --subdirectory-filter $path -- --all

or

  2. Do the filter-branch, and because you know you specified --all and
     that your filters would touch all histories, any ref which _wasn't_
     touched can be deleted. That list is anything which didn't get a
     backup entry in refs/original. So something like:

       git for-each-ref --format='%(refname)' |
       perl -lne 'print $1 if m{^refs/original/(.*)}' >backups

       git for-each-ref --format='%(refname)' |
       grep -v ^refs/original >refs

       comm -23 refs backups |
       sed "s/^/delete /" |
       git update-ref --stdin

-Peff

On Mon, Oct 10, 2016 at 7:19 PM, Jeff King <peff@peff.net> wrote:

This is the perfect solution for me. Going through the delete branches runthrough also quickened the filter-branch command, and I'm left with a much more complete version of where I want to be.

I would still contend that the filter-branch either doesn't work as expected, or the docs need updating to provide extra steps like you've done, because when dealing with a large repo like we have, running multiple filter-branch commands, trying different combinations is quite a time sync, when you're left with the same incorrect solution again and again.