{"thread":{"id":"47989","subject":"How to use filter-branch with --state-branch?","startedAt":"2018-03-06T15:24:39Z","lastAt":"2018-03-09T17:17:07Z","messageCount":6,"participants":["Michele Locati","Ævar Arnfjörð Bjarmason","Ian Campbell"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"341131","messageId":"P56D58$52E406B139FE9D3C295CB5ABAB5C8893@locati.it","threadId":"47989","inReplyTo":null,"subject":"How to use filter-branch with --state-branch?","fromName":"Michele Locati","fromEmail":"michele@locati.it","sentAt":"2018-03-06T15:17:32Z","receivedAt":"2018-03-06T15:24:39Z","isPatch":false,"sender":{"key":"michele@locati.it","avatar":"https://avatars.githubusercontent.com/u/928116?v=4"},"body":"Recent versions of git filter-branch command introduced the --state-branch\noption.\nBTW I can't find any info about how this can be actually used.\n\nWe have this repository on github:\nhttps://github.com/concrete5/concrete5\n\nWhen someone pushes to that repo, we clone it and execute\n`git filter-branch --subdirectory-filter concrete`\nto extract the concrete directory, and we push the result to\nhttps://github.com/concrete5/concrete5-core\n(including all the branches and tags)\n\nThe script at the moment is this one:\nhttps://github.com/concrete5/core_splitter/blob/70879e676b95160f7fc5d0ffc22b8f7420b0580b/bin/splitcore\n\nI tried to use the --state-branch option on a local mirror, so that we could\ndo an incremental filtering. Here's the script:\n\n# Executed just one time\ngit clone --no-checkout --mirror \\\n   https://github.com/concrete5/concrete5.git work\ncd work\ngit filter-branch \\\n   --subdirectory-filter concrete \\\n   --tag-name-filter cat \\\n   --prune-empty \\\n   --state-branch FILTERBRANCH_STATE \\\n   -- --all\n# Executed every time the repo is updated\ngit remote update --prune\ngit filter-branch \\\n   --subdirectory-filter concrete \\\n   --tag-name-filter cat \\\n   --prune-empty \\\n   --state-branch FILTERBRANCH_STATE \\\n   -- --all\n\nThe first filter-branch call required 7168 steps, so did the second call...\nI also tried without the --prune option of remote update (I had to add\n--force to the second filter-branch), but nothing changed.\n"},{"id":"341259","messageId":"87zi3inckw.fsf@evledraar.gmail.com","threadId":"47989","inReplyTo":"P56D58$52E406B139FE9D3C295CB5ABAB5C8893@locati.it","subject":"Re: How to use filter-branch with --state-branch?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-03-08T09:25:35Z","receivedAt":"2018-03-08T09:25:48Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Mar 06 2018, Michele Locati jotted:\n\n> Recent versions of git filter-branch command introduced the --state-branch\n> option.\n> BTW I can't find any info about how this can be actually used.\n>\n> We have this repository on github:\n> https://github.com/concrete5/concrete5\n>\n> When someone pushes to that repo, we clone it and execute\n> `git filter-branch --subdirectory-filter concrete`\n> to extract the concrete directory, and we push the result to\n> https://github.com/concrete5/concrete5-core\n> (including all the branches and tags)\n>\n> The script at the moment is this one:\n> https://github.com/concrete5/core_splitter/blob/70879e676b95160f7fc5d0ffc22b8f7420b0580b/bin/splitcore\n>\n> I tried to use the --state-branch option on a local mirror, so that we could\n> do an incremental filtering. Here's the script:\n>\n> # Executed just one time\n> git clone --no-checkout --mirror \\\n>    https://github.com/concrete5/concrete5.git work\n> cd work\n> git filter-branch \\\n>    --subdirectory-filter concrete \\\n>    --tag-name-filter cat \\\n>    --prune-empty \\\n>    --state-branch FILTERBRANCH_STATE \\\n>    -- --all\n> # Executed every time the repo is updated\n> git remote update --prune\n> git filter-branch \\\n>    --subdirectory-filter concrete \\\n>    --tag-name-filter cat \\\n>    --prune-empty \\\n>    --state-branch FILTERBRANCH_STATE \\\n>    -- --all\n>\n> The first filter-branch call required 7168 steps, so did the second call...\n> I also tried without the --prune option of remote update (I had to add\n> --force to the second filter-branch), but nothing changed.\n\nCC-ing the author of that feature. Usually I'd just look at how the\ntests for it work to answer your question, but I see this new feature\nmade it in recently with no tests for it, which doesn't make me very\nhappy :(\n"},{"id":"341263","messageId":"1520502026.23648.29.camel@hellion.org.uk","threadId":"47989","inReplyTo":"87zi3inckw.fsf@evledraar.gmail.com","subject":"Re: How to use filter-branch with --state-branch?","fromName":"Ian Campbell","fromEmail":"ijc@hellion.org.uk","sentAt":"2018-03-08T09:40:26Z","receivedAt":"2018-03-08T10:17:45Z","isPatch":false,"sender":{"key":"ijc@hellion.org.uk","avatar":"https://avatars.githubusercontent.com/u/12985729?v=4"},"body":"On Thu, 2018-03-08 at 10:25 +0100, Ævar Arnfjörð Bjarmason wrote:\n\n> > The first filter-branch call required 7168 steps, so did the second call...\n> > I also tried without the --prune option of remote update (I had to add\n> > --force to the second filter-branch), but nothing changed.\n\nYou can see an example of the usage in:\n    https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git/\n\nin the `scripts/` sub dir (flow is `cronjob` → `filter.sh` → `git\nfilter-branch...`.\n\nI think the big difference is rather than `--all` you need to give it\nthe `previous..now` range since that is the update you wish to do\n(first time around you just give it `now`).\n\nThe devicetree-rebasing scripting arranges that by keeping the previous\nin a separate branch.\n\nIan.\n"},{"id":"341336","messageId":"CAGen01hYG40jKQSu7ZJpb8TXROC74NfVXVT82wNjYqx1yKYD=w@mail.gmail.com","threadId":"47989","inReplyTo":"1520502026.23648.29.camel@hellion.org.uk","subject":"Re: How to use filter-branch with --state-branch?","fromName":"Michele Locati","fromEmail":"michele@locati.it","sentAt":"2018-03-09T13:04:28Z","receivedAt":"2018-03-09T13:04:36Z","isPatch":false,"sender":{"key":"michele@locati.it","avatar":"https://avatars.githubusercontent.com/u/928116?v=4"},"body":"2018-03-08 10:40 GMT+01:00 Ian Campbell <ijc@hellion.org.uk>:\n>\n> On Thu, 2018-03-08 at 10:25 +0100, Ævar Arnfjörð Bjarmason wrote:\n>\n> > > The first filter-branch call required 7168 steps, so did the second call...\n> > > I also tried without the --prune option of remote update (I had to add\n> > > --force to the second filter-branch), but nothing changed.\n>\n> You can see an example of the usage in:\n>     https://git.kernel.org/pub/scm/linux/kernel/git/devicetree/devicetree-rebasing.git/\n>\n> in the `scripts/` sub dir (flow is `cronjob` → `filter.sh` → `git\n> filter-branch...`.\n>\n> I think the big difference is rather than `--all` you need to give it\n> the `previous..now` range since that is the update you wish to do\n> (first time around you just give it `now`).\n>\n> The devicetree-rebasing scripting arranges that by keeping the previous\n> in a separate branch.\n>\n> Ian.\n\nThank you for your quick reply, Ian.\n\nJust a couple of questions:\n\n1. it seems to me it's not possible to process all the branches in one\ngo. Am I right?\n\n2. Why do you have this line in filter.sh?\n`rm -f .git/refs/original/refs/heads/${UPSTREAM_REWRITTEN}`\n\nThank you again,\nMichele\n"},{"id":"341339","messageId":"1520601834.2474.27.camel@hellion.org.uk","threadId":"47989","inReplyTo":"CAGen01hYG40jKQSu7ZJpb8TXROC74NfVXVT82wNjYqx1yKYD=w@mail.gmail.com","subject":"Re: How to use filter-branch with --state-branch?","fromName":"Ian Campbell","fromEmail":"ijc@hellion.org.uk","sentAt":"2018-03-09T13:23:54Z","receivedAt":"2018-03-09T13:24:01Z","isPatch":false,"sender":{"key":"ijc@hellion.org.uk","avatar":"https://avatars.githubusercontent.com/u/12985729?v=4"},"body":"On Fri, 2018-03-09 at 14:04 +0100, Michele Locati wrote:\n> Just a couple of questions:\n> \n> 1. it seems to me it's not possible to process all the branches in one\n> go. Am I right?\n\nI'm not sure, I've never done such a thing, in fact I didn't know you\ncould.\n\nReally all this feature does is record the `.git/rewrite-map` (or\nwhatever the correct name is) at the end of the rewrite and reinstate\nit again the next time, so it shouldn't really interact with many of\nthe other options.\n\nMy method for storeing \"last version processed\" in a branch does\nconflict I suppose (since that branch would be rewritten) but that's an\nartefact of the surrounding scaffolding -- you could equally well keep\nthe record in some file on the local system or in a non-branch-ish ref\n(I guess).\n\n> 2. Why do you have this line in filter.sh?\n> `rm -f .git/refs/original/refs/heads/${UPSTREAM_REWRITTEN}`\n\nTBH I'm not really sure. I originally wrote this patch many years ago,\nit's just recently that I got around to upstreaming, so my memory is\nmore fuzzy than might be expected.\n\nI think perhaps I was trying to avoid this error:\n\n    A previous backup already exists in $orig_namespace\n    Force overwriting the backup with -f\"\n\nwhich comes if there is an existing backup (a safety feature in the\nnon-incremental case).\n\nNote quite sure why I didn't use `-f` as the message says, but I guess\nbecause it forces other things too which I didn't want to do?\n\nPerhaps what I should have done is make that check conditional on the\nuse of --state-branch.\n\nI wonder if you could use the `original/refs/...` as the \"last version\nprocessed\"? Would be a lot less manual messing around than what I do!\n\nIan.\n"},{"id":"341347","messageId":"CAGen01iRQgxibxmc88ogvgk2_gq-DNNdHMs1N+BfoM5rwWasqA@mail.gmail.com","threadId":"47989","inReplyTo":"1520601834.2474.27.camel@hellion.org.uk","subject":"Re: How to use filter-branch with --state-branch?","fromName":"Michele Locati","fromEmail":"michele@locati.it","sentAt":"2018-03-09T17:17:00Z","receivedAt":"2018-03-09T17:17:07Z","isPatch":false,"sender":{"key":"michele@locati.it","avatar":"https://avatars.githubusercontent.com/u/928116?v=4"},"body":"2018-03-09 14:23 GMT+01:00 Ian Campbell <ijc@hellion.org.uk>:\n> On Fri, 2018-03-09 at 14:04 +0100, Michele Locati wrote:\n>> Just a couple of questions:\n>>\n>> 1. it seems to me it's not possible to process all the branches in one\n>> go. Am I right?\n>\n> I'm not sure, I've never done such a thing, in fact I didn't know you\n> could.\n>\n> Really all this feature does is record the `.git/rewrite-map` (or\n> whatever the correct name is) at the end of the rewrite and reinstate\n> it again the next time, so it shouldn't really interact with many of\n> the other options.\n>\n> My method for storeing \"last version processed\" in a branch does\n> conflict I suppose (since that branch would be rewritten) but that's an\n> artefact of the surrounding scaffolding -- you could equally well keep\n> the record in some file on the local system or in a non-branch-ish ref\n> (I guess).\n>\n>> 2. Why do you have this line in filter.sh?\n>> `rm -f .git/refs/original/refs/heads/${UPSTREAM_REWRITTEN}`\n>\n> TBH I'm not really sure. I originally wrote this patch many years ago,\n> it's just recently that I got around to upstreaming, so my memory is\n> more fuzzy than might be expected.\n>\n> I think perhaps I was trying to avoid this error:\n>\n>     A previous backup already exists in $orig_namespace\n>     Force overwriting the backup with -f\"\n>\n> which comes if there is an existing backup (a safety feature in the\n> non-incremental case).\n>\n> Note quite sure why I didn't use `-f` as the message says, but I guess\n> because it forces other things too which I didn't want to do?\n>\n> Perhaps what I should have done is make that check conditional on the\n> use of --state-branch.\n>\n> I wonder if you could use the `original/refs/...` as the \"last version\n> processed\"? Would be a lot less manual messing around than what I do!\n>\n> Ian.\n\n\nI managed to get a general script that seems to work: see\nhttps://github.com/mlocati/incremental-git-filter-branch\n\nThanks again, Ian.\n\n--\nMichele\n"}]}