threads / discuss / 47989

How to use filter-branch with --state-branch?

Subject: How to use filter-branch with --state-branch?

## tl;dr

6 messages between Mar 6, 2018 and Mar 9, 2018.

replies: 5people: 3as markdown or json

Michele Locati· Mar 6, 2018, 15:17 UTC · lore

Recent versions of git filter-branch command introduced the --state-branch option. BTW I can't find any info about how this can be actually used.

We have this repository on github: https://github.com/concrete5/concrete5

When someone pushes to that repo, we clone it and execute `git filter-branch --subdirectory-filter concrete` to extract the concrete directory, and we push the result to https://github.com/concrete5/concrete5-core (including all the branches and tags)

The script at the moment is this one: https://github.com/concrete5/core_splitter/blob/70879e676b95160f7fc5d0ffc22b8f7420b0580b/bin/splitcore

I tried to use the --state-branch option on a local mirror, so that we could do an incremental filtering. Here's the script:

# Executed just one time
git clone --no-checkout --mirror \
   https://github.com/concrete5/concrete5.git work
cd work
git filter-branch \
   --subdirectory-filter concrete \
   --tag-name-filter cat \
   --prune-empty \
   --state-branch FILTERBRANCH_STATE \
   -- --all
# Executed every time the repo is updated
git remote update --prune
git filter-branch \
   --subdirectory-filter concrete \
   --tag-name-filter cat \
   --prune-empty \
   --state-branch FILTERBRANCH_STATE \
   -- --all

The first filter-branch call required 7168 steps, so did the second call... I also tried without the --prune option of remote update (I had to add --force to the second filter-branch), but nothing changed.

Ævar Arnfjörð Bjarmason· Mar 8, 2018, 09:25 UTC · re: Michele Locati · lore

Re: How to use filter-branch with --state-branch?

On Tue, Mar 06 2018, Michele Locati jotted:
Show 41 quoted lines
> Recent versions of git filter-branch command introduced the --state-branch
> option.
> BTW I can't find any info about how this can be actually used.
>
> We have this repository on github:
> https://github.com/concrete5/concrete5
>
> When someone pushes to that repo, we clone it and execute
> `git filter-branch --subdirectory-filter concrete`
> to extract the concrete directory, and we push the result to
> https://github.com/concrete5/concrete5-core
> (including all the branches and tags)
>
> The script at the moment is this one:
> https://github.com/concrete5/core_splitter/blob/70879e676b95160f7fc5d0ffc22b8f7420b0580b/bin/splitcore
>
> I tried to use the --state-branch option on a local mirror, so that we could
> do an incremental filtering. Here's the script:
>
> # Executed just one time
> git clone --no-checkout --mirror \
>    https://github.com/concrete5/concrete5.git work
> cd work
> git filter-branch \
>    --subdirectory-filter concrete \
>    --tag-name-filter cat \
>    --prune-empty \
>    --state-branch FILTERBRANCH_STATE \
>    -- --all
> # Executed every time the repo is updated
> git remote update --prune
> git filter-branch \
>    --subdirectory-filter concrete \
>    --tag-name-filter cat \
>    --prune-empty \
>    --state-branch FILTERBRANCH_STATE \
>    -- --all
>
> The first filter-branch call required 7168 steps, so did the second call...
> I also tried without the --prune option of remote update (I had to add
> --force to the second filter-branch), but nothing changed.

CC-ing the author of that feature. Usually I'd just look at how the tests for it work to answer your question, but I see this new feature made it in recently with no tests for it, which doesn't make me very happy :(

Ian Campbell· Mar 8, 2018, 09:40 UTC · re: Ævar Arnfjörð Bjarmason · lore

Re: How to use filter-branch with --state-branch?

On Thu, 2018-03-08 at 10:25 +0100, Ævar Arnfjörð Bjarmason wrote:
> > The first filter-branch call required 7168 steps, so did the second call...
> > I also tried without the --prune option of remote update (I had to add
> > --force to the second filter-branch), but nothing changed.
You can see an example of the usage in:
    https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git/

in the `scripts/` sub dir (flow is `cronjob` → `filter.sh` → `git filter-branch...`.

I think the big difference is rather than `--all` you need to give it the `previous..now` range since that is the update you wish to do (first time around you just give it `now`).

The devicetree-rebasing scripting arranges that by keeping the previous in a separate branch.

Ian.
Michele Locati· Mar 9, 2018, 13:04 UTC · re: Ian Campbell · lore

Re: How to use filter-branch with --state-branch?

2018-03-08 10:40 GMT+01:00 Ian Campbell <ijc@hellion.org.uk>:
Show 21 quoted lines
>
> On Thu, 2018-03-08 at 10:25 +0100, Ævar Arnfjörð Bjarmason wrote:
>
> > > The first filter-branch call required 7168 steps, so did the second call...
> > > I also tried without the --prune option of remote update (I had to add
> > > --force to the second filter-branch), but nothing changed.
>
> You can see an example of the usage in:
>     https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git/
>
> in the `scripts/` sub dir (flow is `cronjob` → `filter.sh` → `git
> filter-branch...`.
>
> I think the big difference is rather than `--all` you need to give it
> the `previous..now` range since that is the update you wish to do
> (first time around you just give it `now`).
>
> The devicetree-rebasing scripting arranges that by keeping the previous
> in a separate branch.
>
> Ian.
Thank you for your quick reply, Ian.
Just a couple of questions:
1. it seems to me it's not possible to process all the branches in one
go. Am I right?
2. Why do you have this line in filter.sh?
`rm -f .git/refs/original/refs/heads/${UPSTREAM_REWRITTEN}`

Thank you again, Michele

Ian Campbell· Mar 9, 2018, 13:23 UTC · re: Michele Locati · lore

Re: How to use filter-branch with --state-branch?

On Fri, 2018-03-09 at 14:04 +0100, Michele Locati wrote:
> Just a couple of questions:
> 
> 1. it seems to me it's not possible to process all the branches in one
> go. Am I right?

I'm not sure, I've never done such a thing, in fact I didn't know you could.

Really all this feature does is record the `.git/rewrite-map` (or whatever the correct name is) at the end of the rewrite and reinstate it again the next time, so it shouldn't really interact with many of the other options.

My method for storeing "last version processed" in a branch does conflict I suppose (since that branch would be rewritten) but that's an artefact of the surrounding scaffolding -- you could equally well keep the record in some file on the local system or in a non-branch-ish ref (I guess).

> 2. Why do you have this line in filter.sh?
> `rm -f .git/refs/original/refs/heads/${UPSTREAM_REWRITTEN}`

TBH I'm not really sure. I originally wrote this patch many years ago, it's just recently that I got around to upstreaming, so my memory is more fuzzy than might be expected.

I think perhaps I was trying to avoid this error:
    A previous backup already exists in $orig_namespace
    Force overwriting the backup with -f"

which comes if there is an existing backup (a safety feature in the non-incremental case).

Note quite sure why I didn't use `-f` as the message says, but I guess because it forces other things too which I didn't want to do?

Perhaps what I should have done is make that check conditional on the use of --state-branch.

I wonder if you could use the `original/refs/...` as the "last version processed"? Would be a lot less manual messing around than what I do!

Ian.
Michele Locati· Mar 9, 2018, 17:17 UTC · re: Ian Campbell · lore

Re: How to use filter-branch with --state-branch?

2018-03-09 14:23 GMT+01:00 Ian Campbell <ijc@hellion.org.uk>:
Show 45 quoted lines
> On Fri, 2018-03-09 at 14:04 +0100, Michele Locati wrote:
>> Just a couple of questions:
>>
>> 1. it seems to me it's not possible to process all the branches in one
>> go. Am I right?
>
> I'm not sure, I've never done such a thing, in fact I didn't know you
> could.
>
> Really all this feature does is record the `.git/rewrite-map` (or
> whatever the correct name is) at the end of the rewrite and reinstate
> it again the next time, so it shouldn't really interact with many of
> the other options.
>
> My method for storeing "last version processed" in a branch does
> conflict I suppose (since that branch would be rewritten) but that's an
> artefact of the surrounding scaffolding -- you could equally well keep
> the record in some file on the local system or in a non-branch-ish ref
> (I guess).
>
>> 2. Why do you have this line in filter.sh?
>> `rm -f .git/refs/original/refs/heads/${UPSTREAM_REWRITTEN}`
>
> TBH I'm not really sure. I originally wrote this patch many years ago,
> it's just recently that I got around to upstreaming, so my memory is
> more fuzzy than might be expected.
>
> I think perhaps I was trying to avoid this error:
>
>     A previous backup already exists in $orig_namespace
>     Force overwriting the backup with -f"
>
> which comes if there is an existing backup (a safety feature in the
> non-incremental case).
>
> Note quite sure why I didn't use `-f` as the message says, but I guess
> because it forces other things too which I didn't want to do?
>
> Perhaps what I should have done is make that check conditional on the
> use of --state-branch.
>
> I wonder if you could use the `original/refs/...` as the "last version
> processed"? Would be a lot less manual messing around than what I do!
>
> Ian.

I managed to get a general script that seems to work: see https://github.com/mlocati/incremental-git-filter-branch

Thanks again, Ian.

-- Michele

← back to recent threads