{"thread":{"id":"51710","subject":"RFC: Proposing git-filter-repo for inclusion in git.git","startedAt":"2019-08-22T18:26:22Z","lastAt":"2019-09-04T22:32:53Z","messageCount":73,"participants":["Elijah Newren","Junio C Hamano","Eric Wong","Derrick Stolee","Jeff King","Sergey Organov","Eric Sunshine","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"380965","messageId":"CABPp-BEr8LVM+yWTbi76hAq7Moe1hyp2xqxXfgVV4_teh_9skA@mail.gmail.com","threadId":"51710","inReplyTo":null,"subject":"RFC: Proposing git-filter-repo for inclusion in git.git","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-22T18:26:07Z","receivedAt":"2019-08-22T18:26:22Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi everyone,\n\ngit-filter-repo[1] is a tool for rewriting or restructuring git\nhistory. The recent Git Rev News article[2] on filter-repo introduced\nit and explained how it is much more versatile than git-filter-branch\nor BFG Repo Cleaner, and with speed comparable to BFG. (I probably\nalso should have gone into more detail on how it is also much safer\nthan filter-branch).\n\nI propose merging git-filter-repo.git into git.git. There was some\nprevious discussion at [3], but the tool was incomplete and still\nunder heavy development back then.\n\nQuestions, comments, or concerns with this proposal? Alternative\nproposals? If inclusion is acceptable, are there any other tasks that\nneed to be completed first?\n\n\nThanks,\nElijah\n\n[1] https://github.com/newren/git-filter-repo\n or\n https://github.com/newren/git-filter-repo/blob/master/Documentation/git-filter-repo.txt\n[2] https://git.github.io/rev_news/2019/08/21/edition-54/#an-introduction-to-git-filter-repo--written-by-elijah-newren\n[3] https://public-inbox.org/git/CABPp-BFC--s+D0ijRkFCRxP5Lxfi+__YF4EdxkpO5z+GoNW7Gg@mail.gmail.com/\n"},{"id":"380974","messageId":"xmqqlfvlne3k.fsf@gitster-ct.c.googlers.com","threadId":"51710","inReplyTo":"CABPp-BEr8LVM+yWTbi76hAq7Moe1hyp2xqxXfgVV4_teh_9skA@mail.gmail.com","subject":"Re: RFC: Proposing git-filter-repo for inclusion in git.git","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-08-22T20:23:59Z","receivedAt":"2019-08-22T20:24:07Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> Questions, comments, or concerns with this proposal? Alternative\n> proposals? If inclusion is acceptable, are there any other tasks that\n> need to be completed first?\n\nI do not want a discussion to begin with a Devil's Advocate\nresponse, but anyway...\n\nAre we planning to go to all batteries included approach? I have a\nfeeling that there are other tools (hello, \"git imerge\") that\nequally deserve attention by Git users; are we in the business of\nabsorbing them all? How big a project will our tree become, and how\nmuch more activity would have to be haneld by the readership of the\nGit mailing list?\n\nI'd rather see us shed non-core tools we already have (e.g. git-svn,\ncvs import/export) out of git.git and have them as independent\nprojects. But that may be just me.\n\nThe benefits I see in the \"leave as much as possible out\" approach\nare (not in particular order):\n\n - Distributed development and integration. The release schedule of\n Git-core does not have to be constrained by the readiness of\n non-core part.\n\n - Choice of development tools, language, etc. The core part will\n stay in C, but peripheral tools do not have to be constrained by\n our choice. Those who run their own project can choose what they\n want to use and how they want to structure their development\n process.\n\nThe primary downside I would want to avoid from absorbing non-core\nstuff is that such peripheral tools (git-cvsimport, I am looking at\nyou) can go stale when their development stalls, and nobody would be\nbold enough to suggest removing them.\n"},{"id":"380978","messageId":"CABPp-BFK65qL4GCs5bFuiPPYwMCDYrxMyYejacVS89d2GK4nDQ@mail.gmail.com","threadId":"51710","inReplyTo":"xmqqlfvlne3k.fsf@gitster-ct.c.googlers.com","subject":"Re: RFC: Proposing git-filter-repo for inclusion in git.git","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-22T21:12:42Z","receivedAt":"2019-08-22T21:12:56Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Aug 22, 2019 at 1:24 PM Junio C Hamano wrote:\n>\n> Elijah Newren writes:\n>\n> > Questions, comments, or concerns with this proposal? Alternative\n> > proposals? If inclusion is acceptable, are there any other tasks that\n> > need to be completed first?\n>\n> I do not want a discussion to begin with a Devil's Advocate\n> response, but anyway...\n>\n> Are we planning to go to all batteries included approach? I have a\n> feeling that there are other tools (hello, \"git imerge\") that\n> equally deserve attention by Git users; are we in the business of\n> absorbing them all? How big a project will our tree become, and how\n> much more activity would have to be haneld by the readership of the\n> Git mailing list?\n>\n> I'd rather see us shed non-core tools we already have (e.g. git-svn,\n> cvs import/export) out of git.git and have them as independent\n> projects. But that may be just me.\n\nOoh, if you're going to open this door, then a proposal I assumed\nwould be shot down but which I'd be just about as happy with is:\n\n * Remove git-filter-branch from git.git. Mention in the release\nnotes where people can go to get it.[1]\n\nfilter-branch is not merely a slow or difficult-to-use tool, it's one\nthat *fosters* mistakes by making it hard to get things right in\nseveral different ways. Granted, people exercise extra caution using\nfilter-branch because they know they need to, but there are so many\ngotchas that they're likely to accidentally mess something up. Those\nmess-ups are not always discovered immediately, and by then it's\nnearly cast into stone (rewriting being something you want to do very\nrarely).\n\nFor as long as git-filter-branch is part of git.git and other tools\nare outside, people will take that as a strong endorsement from us for\nfilter-branch and use it regardless of how much other education\nexists...and that causes problems.\n\n\nThoughts?\n\nElijah\n\n\n[1] We'd still have to decide where to put it. If no one else wants\nto do it, I could include it in git-filter-repo with the promise that\nit's there for backward compatibility for those that still need the\ntool, even if I recommend folks use filter-repo instead.\n"},{"id":"380980","messageId":"xmqqd0gwopej.fsf@gitster-ct.c.googlers.com","threadId":"51710","inReplyTo":"CABPp-BFK65qL4GCs5bFuiPPYwMCDYrxMyYejacVS89d2GK4nDQ@mail.gmail.com","subject":"Re: RFC: Proposing git-filter-repo for inclusion in git.git","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-08-22T21:34:28Z","receivedAt":"2019-08-22T21:34:36Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> Ooh, if you're going to open this door, then a proposal I assumed\n> would be shot down but which I'd be just about as happy with is:\n>\n> * Remove git-filter-branch from git.git. Mention in the release\n> notes where people can go to get it.[1]\n> ...\n\nYup, I think, especially given now filter-repo exists and is well\nknown by Git users, this is a good move in the mid to longer term.\n\n> For as long as git-filter-branch is part of git.git and other tools\n> are outside, people will take that as a strong endorsement from us for\n> filter-branch and use it regardless of how much other education\n> exists...and that causes problems.\n\nExactly.\n"},{"id":"380993","messageId":"20190823030059.47ftwfne6y436e5j@dcvr","threadId":"51710","inReplyTo":"CABPp-BFK65qL4GCs5bFuiPPYwMCDYrxMyYejacVS89d2GK4nDQ@mail.gmail.com","subject":"Re: RFC: Proposing git-filter-repo for inclusion in git.git","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2019-08-23T03:00:59Z","receivedAt":"2019-08-23T03:01:02Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Elijah Newren wrote:\n> * Remove git-filter-branch from git.git. Mention in the release\n> notes where people can go to get it.[1]\n> \n> filter-branch is not merely a slow or difficult-to-use tool, it's one\n> that *fosters* mistakes by making it hard to get things right in\n> several different ways. Granted, people exercise extra caution using\n> filter-branch because they know they need to, but there are so many\n> gotchas that they're likely to accidentally mess something up. Those\n> mess-ups are not always discovered immediately, and by then it's\n> nearly cast into stone (rewriting being something you want to do very\n> rarely).\n\nIs it possible to turn git-filter-branch into a fast, compatible,\nand (maybe) safe wrapper for git-filter-repo? That would \"fix\"\nfilter-branch and (if done carefully) not break existing uses.\n\nIt could also spew warnings to recommend safer switches.\n\nStability is a major reason I use git, the Linux kernel,\nand why I distrust+avoid desktop/GUI software. Removing\n\"unsafe\" features, even with good intentions, inevitably leads\nto frustrated users.\n"},{"id":"381009","messageId":"a784a61e-1320-be1e-9dfb-d533a01827ec@gmail.com","threadId":"51710","inReplyTo":"CABPp-BFK65qL4GCs5bFuiPPYwMCDYrxMyYejacVS89d2GK4nDQ@mail.gmail.com","subject":"Re: RFC: Proposing git-filter-repo for inclusion in git.git","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-08-23T12:02:02Z","receivedAt":"2019-08-23T12:02:05Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/22/2019 5:12 PM, Elijah Newren wrote:\n> On Thu, Aug 22, 2019 at 1:24 PM Junio C Hamano wrote:\n>>\n>> Elijah Newren writes:\n>>\n>>> Questions, comments, or concerns with this proposal? Alternative\n>>> proposals? If inclusion is acceptable, are there any other tasks that\n>>> need to be completed first?\n>>\n>> I do not want a discussion to begin with a Devil's Advocate\n>> response, but anyway...\n>>\n>> Are we planning to go to all batteries included approach? I have a\n>> feeling that there are other tools (hello, \"git imerge\") that\n>> equally deserve attention by Git users; are we in the business of\n>> absorbing them all? How big a project will our tree become, and how\n>> much more activity would have to be haneld by the readership of the\n>> Git mailing list?\n>>\n>> I'd rather see us shed non-core tools we already have (e.g. git-svn,\n>> cvs import/export) out of git.git and have them as independent\n>> projects. But that may be just me.\n\nYes please! Let's make the repo smaller.\n\n> Ooh, if you're going to open this door, then a proposal I assumed\n> would be shot down but which I'd be just about as happy with is:\n> \n> * Remove git-filter-branch from git.git. Mention in the release\n> notes where people can go to get it.[1]\n> \n[snip]\n>\n> [1] We'd still have to decide where to put it. If no one else wants\n> to do it, I could include it in git-filter-repo with the promise that\n> it's there for backward compatibility for those that still need the\n> tool, even if I recommend folks use filter-repo instead.\n\nMay I recommend an idea, which may be silly?\n\nWe could strip these \"extra\" tools out of git.git and place them in\ntheir own repos. The hope would be that they could build on their own\nand have their own test suites.\n\nThen, Git distributors could pick and choose the components they\nbundle with Git. Dscho would know more about this sort of thing as\nhe distributes MinGit, which strips these things out already.\n\nThe biggest question is: how do we make sure that as git.git moves\nforward that we don't break the ecosystem? Maybe we create a new,\nlarger repo that contains all of these subrepos? This would give\nthe community more experience dogfooding our own repo-splitting tools.\n\nPersonally, I like the idea of 'git subtree' over something like\n'git submodule'. Using 'git subtree' may mean that tools like\n'git-svn' that may be hard to split into a completely independent\nrepo could live primarily in the meta repo with a source dependence\non the included git.git subtree.\n\nThis \"meta-git.git\" repo could then be more flexible in adding\nnew tools like git-filter-repo or even git-lfs and friends. Again,\ndistributors could select a subset to include, but we would have\none place to run CI builds and make sure the tools are not\nobviously breaking as git.git updates.\n\nThanks,\n-Stolee\n"},{"id":"381022","messageId":"CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com","threadId":"51710","inReplyTo":"20190823030059.47ftwfne6y436e5j@dcvr","subject":"Re: RFC: Proposing git-filter-repo for inclusion in git.git","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-23T18:06:41Z","receivedAt":"2019-08-23T18:06:56Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Eric!\n\nOn Thu, Aug 22, 2019 at 8:01 PM Eric Wong wrote:\n>\n> Elijah Newren wrote:\n> > * Remove git-filter-branch from git.git. Mention in the release\n> > notes where people can go to get it.[1]\n> >\n> > filter-branch is not merely a slow or difficult-to-use tool, it's one\n> > that *fosters* mistakes by making it hard to get things right in\n> > several different ways. Granted, people exercise extra caution using\n> > filter-branch because they know they need to, but there are so many\n> > gotchas that they're likely to accidentally mess something up. Those\n> > mess-ups are not always discovered immediately, and by then it's\n> > nearly cast into stone (rewriting being something you want to do very\n> > rarely).\n>\n> Is it possible to turn git-filter-branch into a fast, compatible,\n> and (maybe) safe wrapper for git-filter-repo? That would \"fix\"\n> filter-branch and (if done carefully) not break existing uses.\n\nOoh, what an interesting question. I can probably ramble on a LOT\nlonger than you expected about this...\n\n== Short answer ==\n\nIt is certainly possible to reimplement git-filter-branch on top of\ngit-filter-repo, though slightly differently than how you appear to be\nsuggesting here. In doing so, you can provide the same user interface\nand have it be perfectly compatible. In fact, I've already created\nsuch a thing -- except that I took a few small liberties with\ncompatibility (and documented each), primarily to improve the speed --\nand I can use it instead of git-filter-branch to pass the git.git\ntestsuite. You can see it here:\nhttps://github.com/newren/git-filter-repo/blob/master/contrib/filter-repo-demos/filter-lamely\n\nHOWEVER, it is NOT possible at all to make such a thing be fast or\nsafe. Not even close. The performance and safety and are not\naccidents of the implementation, but are baked into the design from\ntop to bottom and cannot be fixed without breaking backward\ncompatibility in lots and lots of different ways. For the really\ncurious, I'll provide a possibly non-comprehensive list of why you\ncan't fix performance or safety if you require any compatibility at\nall (and maybe throw in one or two things that could be\nbackward-compatibly fixed in filter-branch, such as the commit\nencoding disaster):\n\n\n== Long answer ==\n\nPerformance:\n\n * In editing files, git-filter-branch by design checks out each and\nevery commit as it existed in the original repo. If your repo has\n10^5 files and 10^5 commits, but each commit only modifies 5 files,\nthen git-filter-branch will make you do 10^10 modifications, despite\nonly having (at most) 5*10^5 unique blobs.\n * If you try and cheat and try to make filter-branch only work on\nfiles modified in a commit, then two things happen (1) you run into\nproblems with deletions whenever the user is simply trying to rename\nfiles (because attempting to delete files that don't exist looks like\na no-op; it takes some chicanery and work to remap deletes across file\nrenames when the renames happen via arbitrary user-provided shell),\nand (2) even if you succeed at the map-deletes-for-renames chicanery\n(as I believe I did in my reimplementation), you still technically\nviolate backward compatibility because users are allowed to filter\nfiles in ways that depend upon topology of commits instead of\nfiltering solely based on file contents or names (though I have never\nseen any user ever do this).\n * Even if you don't need to edit files but only want to e.g. rename\nor remove some and thus can avoid checking out each file (i.e. you can\nuse --index-filter), you still are passing shell snippets for your\nfilters. This means that for every commit, you have to have a\nprepared git repo where users can run git commands. That's a lot of\nsetup. It also means you have to fork at least one process to run the\nuser-provided shell snippet, and odds are that the user's shell\nsnippet invokes lots of commands in some long pipeline, so you will\nhave lots and lots of forks. For every. single. commit. That's a\nmassive amount of overhead to rename a few files.\n * filter-branch is written in shell, which is kind of slow.\nNaturally, it makes sense to want to rewrite that in some other\nlanguage. However, filter-branch documentation states that several\nadditional shell functions are provided for users to call, e.g. 'map',\n'skip_commit', 'git_commit_non_empty_tree', If filter-branch itself\nisn't a shell script, then in order to make those shell functions\navailable to the users' shell snippets you have to prepend the shell\ndefinitions of these functions to every one of the users' shell\nsnippets and thus make these special shell functions be parsed with\neach and every commit.\n * filter-branch provides a --setup option which is a shell snippet\nthat can be sourced to make shell functions and variables available to\nall other filters. If filter-branch is a shell script, it can simply\neval this shell snippet once at the beginning. If you try to fix\nperformance by making filter-branch not be a shell script, then you\nhave to prepend the setup shell snippet to all other filters and parse\nit with every single commit.\n * git-filter-branch writes lots of files to $workdir/../map/ to keep\na mapping of commits, which it uses for the map() command it provides.\nOther files like $tempdir/backup-refs, $tempdir/raw-refs,\n$tempdir/heads, $tempdir/tree-state are all created internally too --\nand users could have accessed any of these. Users even had a pointer\nto follow in the form of Documentation that the 'map' command existed,\nwhich naturally uses the $workdir/../map/* files. So, even if you\ndon't have to edit files, for strict backward compatibility you need\nto still write a bunch of files to disk somewhere and keep them\nupdated for every commit. You can claim it was an implementation\ndetail that users should not have depended upon, but the truth is\nthey've had a decade where they could so. So, if you want full\ncompatibility, it has to be there. Besides, the regression tests\ndepend on at least one of these details, specifying an --index-filter\nthat reaches down and grabs backup-refs from $tempdir, and thus\nprovides resourceful users who do google searches an example that\nthere are files there for them to read and grab and use. (And if you\nwant to pass the existing regression tests, you have to at least put\nthe backup-refs file there even if it's irrelevant to your\nimplementation otherwise.)\n\nSafety:\n\nfilter-branch is riddled with gotchas resulting in various ways to\neasily corrupt repos or end up with a mess worse than what you started\nwith:\n\n* Someone can have a set of \"working and tested filters\" which they\ndocument or provide to a coworker, who then runs them on a different\nOS where the same commands are not working/tested (even the\ngit-filter-branch manpage is guilty here). BSD vs. GNU userland\ndifferences can really bite. If you're lucky, you get ugly error\nmessages spewed. But just as likely, the commands either don't do the\nfiltering requested, or silently corrupt making some unwanted change.\nThe unwanted change may only affect a few commits, so it's not\nnecessarily obvious either. (The fact that problems won't necessarily\nbe obvious means they are likely to go unnoticed until the rewritten\nhistory is in use for quite a while, at which point it's really hard\nto justify another flag-day for another rewrite.)\n * filenames with spaces (which are rare) are often mishandled by\nshell snippets since they cause problems for shell pipelines. Yes, I\nknow find -print0, xargs -0, ls-files -z, etc. Not everyone does.\nAnd even if they do, they may assume it's not relevant because someone\nelse renamed any such files in their repo back before the person doing\nthe filtering joined the project (or maybe they are just being lazy\nand not thinking about everything that could go wrong).\n * non-ascii filenames (which are rare) can be silently removed\ndespite being in a desired directory (the desire to select paths to\nkeep often use pipelines like 'git ls-files | grep -v ^WANTED_DIR/ |\nxargs git rm\". ls-files will only quote filenames if needed so folks\nmay not notice that one of the files didn't match the regex, again\nuntil it's much too late. Yes, someone who knows about core.quotePath\ncan avoid this (unless they have other special characters like \\t, \\n,\nor \"), and people who use ls-files -z can avoid this, but that doesn't\nmean they will).\n * Similarly, when moving files around, one can find that filenames\nwith non-ascii or special characters end up in a different directory,\none that includes a double quote character. (This is technically the\nsame issue as above with quoting, but perhaps an interesting different\nway that it can and has manifested as a problem.)\n * It's far too easy to accidentally mix up old and new history.\nIt's still possible with any tool, but filter-branch almost invites\nit. If we're lucky, the only downside is users getting frustrated\nthat they don't know how to shrink their repo and remove the old\nstuff. If we're unlucky, they merge old and new history and end up\nwith multiple \"copies\" of each commit, some of which have unwanted or\nsensitive files and others which don't. This comes about in multiple\ndifferent ways: the default to only doing a partial history rewrite\n('--all' is not the default and over 80% of the examples in the\nmanpage don't use it), the fact that there's no automatic post-run\ncleanup, the fact that --tag-name-filter (when used to rename tags)\ndoesn't remove the old tags but just adds new ones with the new name\n(the manpage documents this so it's presumably not a \"bug\" even though\nit feels like it), and the fact that little educational information is\nprovided to inform users of the ramifications of a rewrite and how to\navoid mixing old and new history (e.g. not only do other users need to\nunderstand that they need to rebase their changes for all their\nbranches on top of new history (or delete and reclone), but they also\nneed to manually delete all their tags before refetching, any\nreferences that were on any shared servers (e.g. the central repo\nfolks push to) that weren't part of the rewrite need to be deleted,\nand if the shared server has any locked-down refs such as\nrefs/changes/, refs/pull/, or refs/merge-requests/ then people need to\nexercise extra caution; since none of these are fool-proof, someone\nshould probably also add some server-side hooks to prevent folks from\naccidentally re-pushing old history, or make use of special facilities\n(such as gerrit's ban-commit command) to prevent it.)\n * annotated tags can be accidentally converted to lightweight tags.\nThe first way this happens is folks do a rewrite, realize they messed\nup, restore from the backups in refs/original/, and then redo their\nfilter-branch command. (The backup in refs/original/ is not a real\nbackup; it dereferences tags first.) Another way this happens is\ndespite passing --tags or --all on the command line, filter-branch\ndereferences the tag for them. The documentation does not make it all\nthat clear that in order to retain annotated tags as annotated, you\nmust use --tag-name-filter (and must not have restored from\nrefs/original/ in a previously botched rewrite).\n * Any commit messages that specify an encoding will become corrupted\nby the rewrite; filter-branch ignores the encoding, takes the original\nbytes, and feeds it to commit-tree without telling it the proper\nencoding. (This happens whether or not --msg-filter is used, though I\nsuspect --msg-filter provides additional ways to really mess things\nup).\n * commit messages (even if they are all UTF-8) by default become\ncorrupted due to not being updated -- any references to other commit\nhashes in commit messages will now refer to no-longer-extant commits.\n * no facilities for helping users find what unwanted crud they\nshould delete means they are much more likely to have incomplete or\npartial cleanups that sometimes result in confusion and people wasting\ntime trying to understand (e.g. folks tend to just look for big files\nto delete instead of big directories or extensions, and once they do\nso, then sometime later folks using the new repository who are going\nthrough history will notice a build artifact directory that has some\nfiles but not others, or a cache of dependencies (node_modules or\nsimilar) which couldn't have ever been functional since it's missing\nsome files)\n * if --prune-empty isn't specified, then the filtering process can\ncreate hoards of confusing empty commits\n * if --prune-empty is specified, then intentionally placed empty\ncommits from before the filtering operation are also pruned instead of\njust pruning commits that became empty due to filtering rules.\n * if --prune empty is specified, sometimes empty commits are missed\nand left around anyway (probably just a bug, but...).\n\nAlso, performance and safety combine:\n\n * Coming up with the correct shell snippet to do the filtering you\nwant is sometimes difficult unless you're just doing a trivial\nmodification such as deleting a couple files. People have often come\nto me for help, so I should be practiced and an expert, but even for\nfairly simple cases I still sometimes taken over 10 minutes and\nseveral iterations to get the right commands -- and that's assuming\nthey are working on a tiny repository. Unfortunately, people often\nlearn if the snippet is right or wrong by trying it out, but the\nrightness or wrongness can vary depending on special circumstances\n(spaces in filenames, non-ascii filenames, funny author names or\nemails, invalid timezones, presence of grafts or replace objects,\netc.), meaning they may have to wait a long time, hit an error, then\nrestart. The performance of filter-branch is so bad that this cycle\nis painful, reducing the time available to carefully re-check (to say\nnothing about what it does to the patience of the person doing the\nrewrite even if they do technically have more time available). This\nproblem is extra compounded because errors from broken filters may not\nbe shown for a long time and/or get lost in a sea of output. Even\nworse, broken filters often just result in silent incorrect rewrites.\n * To top it all off, even when users finally find working commands,\nthey naturally want to share them. But they may be unaware that their\nrepo didn't have some special cases that someone else's does. So,\nwhen someone else with a different repository runs the same commands,\nthey get hit by the problems above. Or, the user just runs commands\nthat really were vetted for special cases, but they run it on a\ndifferent OS where it doesn't work, as noted above.\n\n== End of long answer ==\n\n\nSummary of above: Anything compatible with git-filter-branch will be\nslower than molasses and extraordinarily unsafe.\n\n\n\n> It could also spew warnings to recommend safer switches.\n\nOoh, I can take a crack at that right now: \"For safety, don't use\n--tree-filter, --index-filter, --commit-filter, --tag-name-filter,\n--prune-empty, or (obviously) --force and _always_ use '--all' (and\nnothing else) for . For performance, don't use\n--tree-filter, --index-filter, --commit-filter, --tag-name-filter,\n--setup, --env-filter, or --msg-filter. Also, don't depend\nrefs/original/ stuff since that's unsafe (tag dereferencing), making\n--original useless. If you follow all these suggestions, -d is\nuseless too. Oh, and I forgot to include --parent-filter among the\nbad-for-performance cases because you should have been using\ngit-replace(1) instead of it for some time now, So that leaves us\nwith our subset that could theoretically be made safe and performant:\n--subdirectory-filter.\"\n\nTurns out, filter-repo does support this exact flag, so if you're\nwilling to restrict yourself to this subset, then filter-repo IS a\ndrop-in replacement. :-)\n\nAnd if you want not just recommendations of flags to avoid, bug flags\nyou can use, then, again, see filter-repo for flags you can use.\n\n\n> Stability is a major reason I use git, the Linux kernel,\n> and why I distrust+avoid desktop/GUI software. Removing\n> \"unsafe\" features, even with good intentions, inevitably leads\n> to frustrated users.\n\nI did not and would not suggest deleting git-filter-branch. I\nsuggested removing it from git.git and putting it elsewhere AND\ntelling people where that elsewhere is. That elsewhere might be\ngit-filter-repo, it could be a different repo, or it could even be my\nalternative faster (but still way too slow) re-implementation of\nfilter-branch.\n\nHope that helps,\nElijah\n"},{"id":"381025","messageId":"CABPp-BFxOA83uL=gsNir5zSgMsfBX6=9+YnWqp+Q4NiztiC6bw@mail.gmail.com","threadId":"51710","inReplyTo":"CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com","subject":"Re: RFC: Proposing git-filter-repo for inclusion in git.git","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-23T18:29:14Z","receivedAt":"2019-08-23T18:29:28Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Aug 23, 2019 at 11:06 AM Elijah Newren wrote:\n\n> Safety:\n\n\nOoh, and another one I remembered just after hitting 'send':\n * If the user provides a --tag-name-filter that maps multiple tags\nto the same name, no warning or error is provided; filter-branch\nsimply overwrites each tag in some undocumented pre-defined order\n(lexicographic) resulting in only one tag at the end. A regression\ntest will fail if you attempt to error out and warn the user, so if\nyou are trying to make a backward compatible reimplementation you have\nto add extra code to detect collisions and make sure that only the\nlexicographically last one is rewritten. (fast-import will naturally\nerror out if told to write the same tag more than once, so you have to\navoid triggering it.)\n\n\n> Summary of above: Anything compatible with git-filter-branch will be\n> slower than molasses and extraordinarily unsafe.\n I do not want a discussion to begin with a Devil's Advocate\n> response, but anyway...\n> \n> Are we planning to go to all batteries included approach? I have a\n> feeling that there are other tools (hello, \"git imerge\") that\n> equally deserve attention by Git users; are we in the business of\n> absorbing them all? How big a project will our tree become, and how\n> much more activity would have to be haneld by the readership of the\n> Git mailing list?\n> \n> I'd rather see us shed non-core tools we already have (e.g. git-svn,\n> cvs import/export) out of git.git and have them as independent\n> projects. But that may be just me.\n\nI like the general line of thinking here, but let me Devil's Advocate\nyour Devil's Advocate:\n\n - having separate repos and release schedules means that the\n dependency changes need to be coordinated. E.g., if a feature in\n git-filter-repo needs a new feature in git-core, then the feature\n needs to land in git-core first, then filter-repo needs to decide\n how to handle older versions. Whereas in the same repo, they can\n generally assume to move forward atomically.\n\n - some of the non-core stuff helps test coverage for the core parts of\n the system. E.g., what bugs might we find in fast-import that are\n only triggered by the filter-repo test suite? Similarly, the\n scripted tools in git.git often serve as canaries for\n backwards-incompatible changes to the plumbing.\n\n Something like the meta-git.git that Stolee proposed would help with\n that. But then we may end up dealing with other people's messes,\n which is one of the things we'd try to avoid with such a split.\n\n-Peff\n"},{"id":"381264","messageId":"20190826235226.15386-1-newren@gmail.com","threadId":"51710","inReplyTo":"xmqqd0gwopej.fsf@gitster-ct.c.googlers.com","subject":"[RFC PATCH 0/5] Remove git-filter-branch from git.git; host it elsewhere","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-26T23:52:21Z","receivedAt":"2019-08-26T23:52:35Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Following up on the suggestion to make git.git smaller and shed non-core\ntools, here's an RFC series to do so with git-filter-branch. This\nseries first removes dependencies on git-filter-branch (of which there\nwere very few), and then deletes git-filter-branch itself in the final\ncommit.\n\nI'm more than happy to consider alternate places for the filter-branch\nhistory (I had considered just merging it in with git-filter-repo), but\nfor now I just made it available here:\n https://github.com/newren/git-filter-branch\n\nThe rewrite above contains the history of the files deleted in Patch 5,\nplus a one-time copy of relevant build files (Makefiles, test-lib.sh,\netc. -- I didn't want the whole history of these), and then touchups to\nstreamline the build files and make them all work in this standalone\nrepo.\n\n\nSome highlevel notes on the patches:\n\n * Patches 1&2: are good cleanups & performance wins regardless of\n whether the rest of the series is taken\n \n * Patch 3: an attempt to improve i18n situation for external scripts,\n but discovered to not be necessary/useful for git-filter-branch\n specifically\n\n * Patch 4:\n * If we are good with deleting git-filter-branch now and just noting\n it in the release notes, then patch 4 could be simplified; there's\n no need to update git-filter-branch.txt in that case.\n * If, however, we want to do some external messaging for an\n additional release cycle or two before moving git-filter-branch\n out of git.git, this patch will help us until then to at least\n avoid recommending a tool which will likely mangle user's data in\n unexpected ways. But it'd be really helpful if folks could review\n and opine on the BFG stuff if so.\n\n * Patch 5: actually deletes git-filter-branch, its tests, and\n documentation.\n\n\nElijah Newren (5):\n t6006: simplify and optimize empty message test\n t3427: accelerate this test by using fast-export and fast-import\n git-sh-i18n: work with external scripts\n Recommend git-filter-repo instead of git-filter-branch in\n documentation\n Remove git-filter-branch, it is now external to git.git\n\n .gitignore | 1 -\n Documentation/git-fast-export.txt | 6 +-\n Documentation/git-filter-branch.txt | 481 --------------------\n Documentation/git-gc.txt | 17 +-\n Documentation/git-rebase.txt | 2 +-\n Documentation/git-replace.txt | 10 +-\n Documentation/git-svn.txt | 4 +-\n Documentation/githooks.txt | 7 +-\n Makefile | 1 -\n command-list.txt | 1 -\n contrib/svn-fe/svn-fe.txt | 4 +-\n git-filter-branch.sh | 662 ----------------------------\n git-sh-i18n.sh | 7 +-\n t/perf/p7000-filter-branch.sh | 24 -\n t/t3427-rebase-subtree.sh | 32 +-\n t/t6006-rev-list-format.sh | 5 +-\n t/t7003-filter-branch.sh | 505 ---------------------\n t/t7009-filter-branch-null-sha1.sh | 55 ---\n t/t9902-completion.sh | 12 +-\n 19 files changed, 63 insertions(+), 1773 deletions(-)\n delete mode 100644 Documentation/git-filter-branch.txt\n delete mode 100755 git-filter-branch.sh\n delete mode 100755 t/perf/p7000-filter-branch.sh\n delete mode 100755 t/t7003-filter-branch.sh\n delete mode 100755 t/t7009-filter-branch-null-sha1.sh\n\n-- \n2.23.0.5.g775ebaa2a0\n\n"},{"id":"381266","messageId":"20190826235226.15386-2-newren@gmail.com","threadId":"51710","inReplyTo":"20190826235226.15386-1-newren@gmail.com","subject":"[RFC PATCH 1/5] t6006: simplify and optimize empty message test","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-26T23:52:22Z","receivedAt":"2019-08-26T23:52:36Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Test t6006.71 (\"oneline with empty message\") was creating two commits\nwith simple commit messages, and then running filter-branch to rewrite\nthe commit messages to be empty. This test was written this way because\nthe --allow-empty-message option to git commit did not exist at the\ntime. Simplify this test and avoid the need to invoke filter-branch by\njust using --allow-empty-message when creating the commit.\n\nDespite only being one piece of the 71st test and there being 73 tests\noverall, this small change to just this one test speeds up the overall\nexecution time of t6006 (as measured by the best of 3 runs of `time\n./t6006-rev-list-format.sh`) by about 11% on Linux and by 13% on\nMac.\n\nSigned-off-by: Elijah Newren \n---\n t/t6006-rev-list-format.sh | 5 ++---\n 1 file changed, 2 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t6006-rev-list-format.sh b/t/t6006-rev-list-format.sh\nindex da113d975b..d30e41c9f7 100755\n--- a/t/t6006-rev-list-format.sh\n+++ b/t/t6006-rev-list-format.sh\n@@ -501,9 +501,8 @@ test_expect_success 'reflog identity' '\n '\n \n test_expect_success 'oneline with empty message' '\n-\tgit commit -m \"dummy\" --allow-empty &&\n-\tgit commit -m \"dummy\" --allow-empty &&\n-\tgit filter-branch --msg-filter \"sed -e s/dummy//\" HEAD^^.. &&\n+\tgit commit --allow-empty --allow-empty-message &&\n+\tgit commit --allow-empty --allow-empty-message &&\n \tgit rev-list --oneline HEAD >test.txt &&\n \ttest_line_count = 5 test.txt &&\n \tgit rev-list --oneline --graph HEAD >testg.txt &&\n-- \n2.23.0.5.g775ebaa2a0\n\n"},{"id":"381265","messageId":"20190826235226.15386-3-newren@gmail.com","threadId":"51710","inReplyTo":"20190826235226.15386-1-newren@gmail.com","subject":"[RFC PATCH 2/5] t3427: accelerate this test by using fast-export and fast-import","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-26T23:52:23Z","receivedAt":"2019-08-26T23:52:38Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"fast-export and fast-import can easily handle the simple rewrite that\nwas being done by filter-branch, and should be significantly faster on\nsystems with a slow fork. Timings from before and after on two laptops\nthat I have access to (measured via `time ./t3427-rebase-subtree.sh`,\ni.e. including everything in this test -- not just the filter-branch or\nfast-export/fast-import pair):\n\n Linux: 4.305s -> 3.684s (~17% speedup)\n Mac: 10.128s -> 7.038s (~30% speedup)\n\nSigned-off-by: Elijah Newren \n---\n t/t3427-rebase-subtree.sh | 32 ++++++++++++++++++++++++--------\n 1 file changed, 24 insertions(+), 8 deletions(-)\n\ndiff --git a/t/t3427-rebase-subtree.sh b/t/t3427-rebase-subtree.sh\nindex d8640522a0..d05fcce5dc 100755\n--- a/t/t3427-rebase-subtree.sh\n+++ b/t/t3427-rebase-subtree.sh\n@@ -42,7 +42,9 @@ test_expect_failure REBASE_P \\\n \t'Rebase -Xsubtree --preserve-merges --onto commit 4' '\n \treset_rebase &&\n \tgit checkout -b rebase-preserve-merges-4 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\tgit fast-export --no-data HEAD -- files_subtree/ \\\n+\t\t| sed -e \"s%\\([0-9a-f]\\{40\\} \\)files_subtree/%\\1%\" \\\n+\t\t| git fast-import --force --quiet &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --preserve-merges --onto files-master master &&\n \tverbose test \"$(commit_message HEAD~)\" = \"files_subtree/master4\"\n@@ -53,7 +55,9 @@ test_expect_failure REBASE_P \\\n \t'Rebase -Xsubtree --preserve-merges --onto commit 5' '\n \treset_rebase &&\n \tgit checkout -b rebase-preserve-merges-5 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\tgit fast-export --no-data HEAD -- files_subtree/ \\\n+\t\t| sed -e \"s%\\([0-9a-f]\\{40\\} \\)files_subtree/%\\1%\" \\\n+\t\t| git fast-import --force --quiet &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --preserve-merges --onto files-master master &&\n \tverbose test \"$(commit_message HEAD)\" = \"files_subtree/master5\"\n@@ -64,7 +68,9 @@ test_expect_failure REBASE_P \\\n \t'Rebase -Xsubtree --keep-empty --preserve-merges --onto commit 4' '\n \treset_rebase &&\n \tgit checkout -b rebase-keep-empty-4 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\tgit fast-export --no-data HEAD -- files_subtree/ \\\n+\t\t| sed -e \"s%\\([0-9a-f]\\{40\\} \\)files_subtree/%\\1%\" \\\n+\t\t| git fast-import --force --quiet &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master &&\n \tverbose test \"$(commit_message HEAD~2)\" = \"files_subtree/master4\"\n@@ -75,7 +81,9 @@ test_expect_failure REBASE_P \\\n \t'Rebase -Xsubtree --keep-empty --preserve-merges --onto commit 5' '\n \treset_rebase &&\n \tgit checkout -b rebase-keep-empty-5 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\tgit fast-export --no-data HEAD -- files_subtree/ \\\n+\t\t| sed -e \"s%\\([0-9a-f]\\{40\\} \\)files_subtree/%\\1%\" \\\n+\t\t| git fast-import --force --quiet &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master &&\n \tverbose test \"$(commit_message HEAD~)\" = \"files_subtree/master5\"\n@@ -86,7 +94,9 @@ test_expect_failure REBASE_P \\\n \t'Rebase -Xsubtree --keep-empty --preserve-merges --onto empty commit' '\n \treset_rebase &&\n \tgit checkout -b rebase-keep-empty-empty master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\tgit fast-export --no-data HEAD -- files_subtree/ \\\n+\t\t| sed -e \"s%\\([0-9a-f]\\{40\\} \\)files_subtree/%\\1%\" \\\n+\t\t| git fast-import --force --quiet &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master &&\n \tverbose test \"$(commit_message HEAD)\" = \"Empty commit\"\n@@ -96,7 +106,9 @@ test_expect_failure REBASE_P \\\n test_expect_failure 'Rebase -Xsubtree --onto commit 4' '\n \treset_rebase &&\n \tgit checkout -b rebase-onto-4 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\tgit fast-export --no-data HEAD -- files_subtree/ \\\n+\t\t| sed -e \"s%\\([0-9a-f]\\{40\\} \\)files_subtree/%\\1%\" \\\n+\t\t| git fast-import --force --quiet &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --onto files-master master &&\n \tverbose test \"$(commit_message HEAD~2)\" = \"files_subtree/master4\"\n@@ -106,7 +118,9 @@ test_expect_failure 'Rebase -Xsubtree --onto commit 4' '\n test_expect_failure 'Rebase -Xsubtree --onto commit 5' '\n \treset_rebase &&\n \tgit checkout -b rebase-onto-5 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\tgit fast-export --no-data HEAD -- files_subtree/ \\\n+\t\t| sed -e \"s%\\([0-9a-f]\\{40\\} \\)files_subtree/%\\1%\" \\\n+\t\t| git fast-import --force --quiet &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --onto files-master master &&\n \tverbose test \"$(commit_message HEAD~)\" = \"files_subtree/master5\"\n@@ -115,7 +129,9 @@ test_expect_failure 'Rebase -Xsubtree --onto commit 5' '\n test_expect_failure 'Rebase -Xsubtree --onto empty commit' '\n \treset_rebase &&\n \tgit checkout -b rebase-onto-empty master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\tgit fast-export --no-data HEAD -- files_subtree/ \\\n+\t\t| sed -e \"s%\\([0-9a-f]\\{40\\} \\)files_subtree/%\\1%\" \\\n+\t\t| git fast-import --force --quiet &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --onto files-master master &&\n \tverbose test \"$(commit_message HEAD)\" = \"Empty commit\"\n-- \n2.23.0.5.g775ebaa2a0\n\n"},{"id":"381269","messageId":"20190826235226.15386-4-newren@gmail.com","threadId":"51710","inReplyTo":"20190826235226.15386-1-newren@gmail.com","subject":"[RFC PATCH 3/5] git-sh-i18n: work with external scripts","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-26T23:52:24Z","receivedAt":"2019-08-26T23:52:39Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Scripts external to git could source $(git --exec-path)/git-sh-setup (as\nwe document in Documentation/git-sh-setup.txt). This will in turn\nsource git-sh-i18n, which will setup some handy internationalization\ninfrastructure. However, git-sh-i18n hardcodes the TEXTDOMAIN, meaning\nthat anyone using this infrastructure will only get translations that\nare shipped with git. Allow the external scripts to specify their own\ntranslation domain but otherwise use our infrastructure for accessing\ntranslations.\n\nMy original plan was to have git-filter-branch be the first testcase\nusing this feature, with a goal of minimizing the number of changes that\nneeded to be made to it when I moved it out of git.git. However, I\nrealized after creating this patch that no strings in git-filter-branch\nare translated. However, the generalization could be useful if we move\nother tools from git.git to an external location.\n\nSigned-off-by: Elijah Newren \n---\n git-sh-i18n.sh | 7 ++++++-\n 1 file changed, 6 insertions(+), 1 deletion(-)\n\ndiff --git a/git-sh-i18n.sh b/git-sh-i18n.sh\nindex 8eef60b43f..3d04d5d515 100644\n--- a/git-sh-i18n.sh\n+++ b/git-sh-i18n.sh\n@@ -5,7 +5,12 @@\n #\n \n # Export the TEXTDOMAIN* data that we need for Git\n-TEXTDOMAIN=git\n+if test -z \"$TEXTDOMAIN_OVERRIDE\"\n+then\n+\tTEXTDOMAIN=git\n+else\n+\tTEXTDOMAIN=\"$TEXTDOMAIN_OVERRIDE\"\n+fi\n export TEXTDOMAIN\n if test -z \"$GIT_TEXTDOMAINDIR\"\n then\n-- \n2.23.0.5.g775ebaa2a0\n\n"},{"id":"381268","messageId":"20190826235226.15386-5-newren@gmail.com","threadId":"51710","inReplyTo":"20190826235226.15386-1-newren@gmail.com","subject":"[RFC PATCH 4/5] Recommend git-filter-repo instead of git-filter-branch in documentation","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-26T23:52:25Z","receivedAt":"2019-08-26T23:52:41Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"filter-branch suffers from a huge number of pitfalls that can result in\nincorrectly rewritten history, and many of the problems can easily go\nundetected until the new repository is in use. This can result in\nproblems ranging from an even messier history than what led folks to\nfilter-branch in the first place, to data loss or corruption. These\nissues cannot be backward compatibly fixed, so add a warning to the\nfilter-branch manpage about this and recommand that another tool (such\nas filter-repo) be used instead.\n\nAlso, update other manpages that referenced filter-branch. Several of\nthese needed updates even if we could continue recommending\nfilter-branch, either due to implying that something was unique to\nfilter-branch when it applied more generally to all history rewriting\ntools (e.g. BFG, reposurgeon, fast-import, filter-repo), or because\nsomething about filter-branch was used as an example despite other more\ncommonly known examples now existing. Reword these sections to fix\nthese issues and to avoid recommending filter-branch.\n\nFinally, remove the section explaining BFG Repo Cleaner as an\nalternative to filter-branch. I feel somewhat bad about this,\nespecially since I feel like I learned so much from BFG that I put to\ngood use in filter-repo (which is much more than I can say for\nfilter-branch), but keeping that section presented a few problems:\n * In order to recommend that people quit using filter-branch, we need\n to provide them a recomendation for something else to use that\n can handle all the same types of rewrites. To my knowledge,\n filter-repo is the only such tool. So it needs to be mentioned.\n * I don't want to give conflicting recommendations to users\n * If we recommend two tools, we shouldn't expect users to learn both\n and pick which one to use; we should explain which problems one\n can solve that the other can't or when one is much faster than\n the other.\n * BFG and filter-repo have similar performance\n * All filtering types that BFG can do, filter-repo can also do. In\n fact, filter-repo comes with a reimplementation of BFG named\n bfg-ish which provides the same user-interface as BFG but with\n several bugfixes and new features that are hard to implement in\n BFG due to its technical underpinnings.\nWhile I could still mention both tools, it seems like I would need to\nprovide some kind of comparison and I would ultimately just say that\nfilter-repo can do everything BFG can, so ultimately it seems that it\nis just better to remove that section altogether.\n\nSigned-off-by: Elijah Newren \n---\n Documentation/git-fast-export.txt | 6 ++---\n Documentation/git-filter-branch.txt | 42 ++++++++---------------------\n Documentation/git-gc.txt | 17 ++++++------\n Documentation/git-rebase.txt | 2 +-\n Documentation/git-replace.txt | 10 +++----\n Documentation/git-svn.txt | 4 +--\n Documentation/githooks.txt | 7 ++---\n contrib/svn-fe/svn-fe.txt | 4 +--\n 8 files changed, 36 insertions(+), 56 deletions(-)\n\ndiff --git a/Documentation/git-fast-export.txt b/Documentation/git-fast-export.txt\nindex cc940eb9ad..784e934009 100644\n--- a/Documentation/git-fast-export.txt\n+++ b/Documentation/git-fast-export.txt\n@@ -17,9 +17,9 @@ This program dumps the given revisions in a form suitable to be piped\n into 'git fast-import'.\n \n You can use it as a human-readable bundle replacement (see\n-linkgit:git-bundle[1]), or as a kind of an interactive\n-'git filter-branch'.\n-\n+linkgit:git-bundle[1]), or as a format that can be edited before being\n+fed to 'git fast-import' in order to do history rewrites (an ability\n+relied on by tools like 'git filter-repo').\n \n OPTIONS\n -------\ndiff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt\nindex 6b53dd7e06..8c586eed55 100644\n--- a/Documentation/git-filter-branch.txt\n+++ b/Documentation/git-filter-branch.txt\n@@ -16,6 +16,17 @@ SYNOPSIS\n \t[--original ] [-d ] [-f | --force]\n \t[--state-branch ] [--] [...]\n \n+WARNING\n+-------\n+'git filter-branch' has a litany of gotchas that can and will cause\n+history to be rewritten incorrectly (in addition to abysmal\n+performance). These issues cannot be backward compatibly fixed and as\n+such, its use is not recommended. Please use an alternative history\n+filtering tool such as 'git filter-repo'. If you still need to use\n+'git filter-branch', please carefully read the \"Safety\" section of\n+https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/\n+and avoid as many of the pitfalls listed there as reasonably possible.\n+\n DESCRIPTION\n -----------\n Lets you rewrite Git revision history by rewriting the branches mentioned\n@@ -445,37 +456,6 @@ warned.\n (or if your git-gc is not new enough to support arguments to\n `--prune`, use `git repack -ad; git prune` instead).\n \n-NOTES\n------\n-\n-git-filter-branch allows you to make complex shell-scripted rewrites\n-of your Git history, but you probably don't need this flexibility if\n-you're simply _removing unwanted data_ like large files or passwords.\n-For those operations you may want to consider\n-http://rtyley.github.io/bfg-repo-cleaner/[The BFG Repo-Cleaner],\n-a JVM-based alternative to git-filter-branch, typically at least\n-10-50x faster for those use-cases, and with quite different\n-characteristics:\n-\n-* Any particular version of a file is cleaned exactly _once_. The BFG,\n- unlike git-filter-branch, does not give you the opportunity to\n- handle a file differently based on where or when it was committed\n- within your history. This constraint gives the core performance\n- benefit of The BFG, and is well-suited to the task of cleansing bad\n- data - you don't care _where_ the bad data is, you just want it\n- _gone_.\n-\n-* By default The BFG takes full advantage of multi-core machines,\n- cleansing commit file-trees in parallel. git-filter-branch cleans\n- commits sequentially (i.e. in a single-threaded manner), though it\n- _is_ possible to write filters that include their own parallelism,\n- in the scripts executed against each commit.\n-\n-* The http://rtyley.github.io/bfg-repo-cleaner/#examples[command options]\n- are much more restrictive than git-filter branch, and dedicated just\n- to the tasks of removing unwanted data- e.g:\n- `--strip-blobs-bigger-than 1M`.\n-\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 247f765604..0c114ad1ca 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -115,15 +115,14 @@ NOTES\n -----\n \n 'git gc' tries very hard not to delete objects that are referenced\n-anywhere in your repository. In\n-particular, it will keep not only objects referenced by your current set\n-of branches and tags, but also objects referenced by the index,\n-remote-tracking branches, refs saved by 'git filter-branch' in\n-refs/original/, reflogs (which may reference commits in branches\n-that were later amended or rewound), and anything else in the refs/* namespace.\n-If you are expecting some objects to be deleted and they aren't, check\n-all of those locations and decide whether it makes sense in your case to\n-remove those references.\n+anywhere in your repository. In particular, it will keep not only\n+objects referenced by your current set of branches and tags, but also\n+objects referenced by the index, remote-tracking branches, notes saved\n+by 'git notes' under refs/notes/, reflogs (which may reference commits\n+in branches that were later amended or rewound), and anything else in\n+the refs/* namespace. If you are expecting some objects to be deleted\n+and they aren't, check all of those locations and decide whether it\n+makes sense in your case to remove those references.\n \n On the other hand, when 'git gc' runs concurrently with another process,\n there is a risk of it deleting an object that the other process is using\ndiff --git a/Documentation/git-rebase.txt b/Documentation/git-rebase.txt\nindex 6156609cf7..2f201d85d4 100644\n--- a/Documentation/git-rebase.txt\n+++ b/Documentation/git-rebase.txt\n@@ -832,7 +832,7 @@ Hard case: The changes are not the same.::\n \tThis happens if the 'subsystem' rebase had conflicts, or used\n \t`--interactive` to omit, edit, squash, or fixup commits; or\n \tif the upstream used one of `commit --amend`, `reset`, or\n-\t`filter-branch`.\n+\ta full history rewriting command like `filter-repo`.\n \n \n The easy case\ndiff --git a/Documentation/git-replace.txt b/Documentation/git-replace.txt\nindex 246dc9943c..35595a2cd3 100644\n--- a/Documentation/git-replace.txt\n+++ b/Documentation/git-replace.txt\n@@ -123,10 +123,10 @@ The following format are available:\n CREATING REPLACEMENT OBJECTS\n ----------------------------\n \n-linkgit:git-filter-branch[1], linkgit:git-hash-object[1] and\n-linkgit:git-rebase[1], among other git commands, can be used to create\n-replacement objects from existing objects. The `--edit` option can\n-also be used with 'git replace' to create a replacement object by\n+linkgit:git-hash-object[1], linkgit:git-rebase[1], and\n+linkgit:git-filter-repo[1], among other git commands, can be used to\n+create replacement objects from existing objects. The `--edit` option\n+can also be used with 'git replace' to create a replacement object by\n editing an existing object.\n \n If you want to replace many blobs, trees or commits that are part of a\n@@ -148,8 +148,8 @@ pending objects.\n SEE ALSO\n --------\n linkgit:git-hash-object[1]\n-linkgit:git-filter-branch[1]\n linkgit:git-rebase[1]\n+linkgit:git-filter-repo[1]\n linkgit:git-tag[1]\n linkgit:git-branch[1]\n linkgit:git-commit[1]\ndiff --git a/Documentation/git-svn.txt b/Documentation/git-svn.txt\nindex 30711625fd..f2762dd5d4 100644\n--- a/Documentation/git-svn.txt\n+++ b/Documentation/git-svn.txt\n@@ -769,9 +769,9 @@ option for (hopefully) obvious reasons.\n +\n This option is NOT recommended as it makes it difficult to track down\n old references to SVN revision numbers in existing documentation, bug\n-reports and archives. If you plan to eventually migrate from SVN to Git\n+reports, and archives. If you plan to eventually migrate from SVN to Git\n and are certain about dropping SVN history, consider\n-linkgit:git-filter-branch[1] instead. filter-branch also allows\n+linkgit:git-filter-repo[1] instead. filter-repo also allows\n reformatting of metadata for ease-of-reading and rewriting authorship\n info for non-\"svn.authorsFile\" users.\n \ndiff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\nindex 82cd573776..997548f5ed 100644\n--- a/Documentation/githooks.txt\n+++ b/Documentation/githooks.txt\n@@ -425,9 +425,10 @@ post-rewrite\n \n This hook is invoked by commands that rewrite commits\n (linkgit:git-commit[1] when called with `--amend` and\n-linkgit:git-rebase[1]; currently `git filter-branch` does 'not' call\n-it!). Its first argument denotes the command it was invoked by:\n-currently one of `amend` or `rebase`. Further command-dependent\n+linkgit:git-rebase[1]; however, full-history (re)writing tools like\n+linkgit:git-fast-import[1] or linkgit:git-filter-repo[1] typically do\n+not call it!). Its first argument denotes the command it was invoked\n+by: currently one of `amend` or `rebase`. Further command-dependent\n arguments may be passed in the future.\n \n The hook receives a list of the rewritten commits on stdin, in the\ndiff --git a/contrib/svn-fe/svn-fe.txt b/contrib/svn-fe/svn-fe.txt\nindex a3425f4770..19333fc8df 100644\n--- a/contrib/svn-fe/svn-fe.txt\n+++ b/contrib/svn-fe/svn-fe.txt\n@@ -56,7 +56,7 @@ line. This line has the form `git-svn-id: URL@REVNO UUID`.\n \n The resulting repository will generally require further processing\n to put each project in its own repository and to separate the history\n-of each branch. The 'git filter-branch --subdirectory-filter' command\n+of each branch. The 'git filter-repo --subdirectory-filter' command\n may be useful for this purpose.\n \n BUGS\n@@ -67,5 +67,5 @@ The exit status does not reflect whether an error was detected.\n \n SEE ALSO\n --------\n-git-svn(1), svn2git(1), svk(1), git-filter-branch(1), git-fast-import(1),\n+git-svn(1), svn2git(1), svk(1), git-filter-repo(1), git-fast-import(1),\n https://svn.apache.org/repos/asf/subversion/trunk/notes/dump-load-format.txt\n-- \n2.23.0.5.g775ebaa2a0\n\n"},{"id":"381267","messageId":"20190826235226.15386-6-newren@gmail.com","threadId":"51710","inReplyTo":"20190826235226.15386-1-newren@gmail.com","subject":"[RFC PATCH 5/5] Remove git-filter-branch, it is now external to git.git","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-26T23:52:26Z","receivedAt":"2019-08-26T23:52:42Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Signed-off-by: Elijah Newren \n---\n .gitignore | 1 -\n Documentation/git-filter-branch.txt | 461 -------------------\n Makefile | 1 -\n command-list.txt | 1 -\n git-filter-branch.sh | 662 ----------------------------\n t/perf/p7000-filter-branch.sh | 24 -\n t/t7003-filter-branch.sh | 505 ---------------------\n t/t7009-filter-branch-null-sha1.sh | 55 ---\n t/t9902-completion.sh | 12 +-\n 9 files changed, 6 insertions(+), 1716 deletions(-)\n delete mode 100644 Documentation/git-filter-branch.txt\n delete mode 100755 git-filter-branch.sh\n delete mode 100755 t/perf/p7000-filter-branch.sh\n delete mode 100755 t/t7003-filter-branch.sh\n delete mode 100755 t/t7009-filter-branch-null-sha1.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex 521d8f4fb4..97f5d8afea 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -63,7 +63,6 @@\n /git-fast-import\n /git-fetch\n /git-fetch-pack\n-/git-filter-branch\n /git-fmt-merge-msg\n /git-for-each-ref\n /git-format-patch\ndiff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt\ndeleted file mode 100644\nindex 8c586eed55..0000000000\n--- a/Documentation/git-filter-branch.txt\n+++ /dev/null\n@@ -1,461 +0,0 @@\n-git-filter-branch(1)\n-====================\n-\n-NAME\n-----\n-git-filter-branch - Rewrite branches\n-\n-SYNOPSIS\n---------\n-[verse]\n-'git filter-branch' [--setup ] [--subdirectory-filter ]\n-\t[--env-filter ] [--tree-filter ]\n-\t[--index-filter ] [--parent-filter ]\n-\t[--msg-filter ] [--commit-filter ]\n-\t[--tag-name-filter ] [--prune-empty]\n-\t[--original ] [-d ] [-f | --force]\n-\t[--state-branch ] [--] [...]\n-\n-WARNING\n--------\n-'git filter-branch' has a litany of gotchas that can and will cause\n-history to be rewritten incorrectly (in addition to abysmal\n-performance). These issues cannot be backward compatibly fixed and as\n-such, its use is not recommended. Please use an alternative history\n-filtering tool such as 'git filter-repo'. If you still need to use\n-'git filter-branch', please carefully read the \"Safety\" section of\n-https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/\n-and avoid as many of the pitfalls listed there as reasonably possible.\n-\n-DESCRIPTION\n------------\n-Lets you rewrite Git revision history by rewriting the branches mentioned\n-in the , applying custom filters on each revision.\n-Those filters can modify each tree (e.g. removing a file or running\n-a perl rewrite on all files) or information about each commit.\n-Otherwise, all information (including original commit times or merge\n-information) will be preserved.\n-\n-The command will only rewrite the _positive_ refs mentioned in the\n-command line (e.g. if you pass 'a..b', only 'b' will be rewritten).\n-If you specify no filters, the commits will be recommitted without any\n-changes, which would normally have no effect. Nevertheless, this may be\n-useful in the future for compensating for some Git bugs or such,\n-therefore such a usage is permitted.\n-\n-*NOTE*: This command honors `.git/info/grafts` file and refs in\n-the `refs/replace/` namespace.\n-If you have any grafts or replacement refs defined, running this command\n-will make them permanent.\n-\n-*WARNING*! The rewritten history will have different object names for all\n-the objects and will not converge with the original branch. You will not\n-be able to easily push and distribute the rewritten branch on top of the\n-original branch. Please do not use this command if you do not know the\n-full implications, and avoid using it anyway, if a simple single commit\n-would suffice to fix your problem. (See the \"RECOVERING FROM UPSTREAM\n-REBASE\" section in linkgit:git-rebase[1] for further information about\n-rewriting published history.)\n-\n-Always verify that the rewritten version is correct: The original refs,\n-if different from the rewritten ones, will be stored in the namespace\n-'refs/original/'.\n-\n-Note that since this operation is very I/O expensive, it might\n-be a good idea to redirect the temporary directory off-disk with the\n-`-d` option, e.g. on tmpfs. Reportedly the speedup is very noticeable.\n-\n-\n-Filters\n-~~~~~~~\n-\n-The filters are applied in the order as listed below. The \n-argument is always evaluated in the shell context using the 'eval' command\n-(with the notable exception of the commit filter, for technical reasons).\n-Prior to that, the `$GIT_COMMIT` environment variable will be set to contain\n-the id of the commit being rewritten. Also, GIT_AUTHOR_NAME,\n-GIT_AUTHOR_EMAIL, GIT_AUTHOR_DATE, GIT_COMMITTER_NAME, GIT_COMMITTER_EMAIL,\n-and GIT_COMMITTER_DATE are taken from the current commit and exported to\n-the environment, in order to affect the author and committer identities of\n-the replacement commit created by linkgit:git-commit-tree[1] after the\n-filters have run.\n-\n-If any evaluation of returns a non-zero exit status, the whole\n-operation will be aborted.\n-\n-A 'map' function is available that takes an \"original sha1 id\" argument\n-and outputs a \"rewritten sha1 id\" if the commit has been already\n-rewritten, and \"original sha1 id\" otherwise; the 'map' function can\n-return several ids on separate lines if your commit filter emitted\n-multiple commits.\n-\n-\n-OPTIONS\n--------\n-\n---setup ::\n-\tThis is not a real filter executed for each commit but a one\n-\ttime setup just before the loop. Therefore no commit-specific\n-\tvariables are defined yet. Functions or variables defined here\n-\tcan be used or modified in the following filter steps except\n-\tthe commit filter, for technical reasons.\n-\n---subdirectory-filter ::\n-\tOnly look at the history which touches the given subdirectory.\n-\tThe result will contain that directory (and only that) as its\n-\tproject root. Implies <>.\n-\n---env-filter ::\n-\tThis filter may be used if you only need to modify the environment\n-\tin which the commit will be performed. Specifically, you might\n-\twant to rewrite the author/committer name/email/time environment\n-\tvariables (see linkgit:git-commit-tree[1] for details).\n-\n---tree-filter ::\n-\tThis is the filter for rewriting the tree and its contents.\n-\tThe argument is evaluated in shell with the working\n-\tdirectory set to the root of the checked out tree. The new tree\n-\tis then used as-is (new files are auto-added, disappeared files\n-\tare auto-removed - neither .gitignore files nor any other ignore\n-\trules *HAVE ANY EFFECT*!).\n-\n---index-filter ::\n-\tThis is the filter for rewriting the index. It is similar to the\n-\ttree filter but does not check out the tree, which makes it much\n-\tfaster. Frequently used with `git rm --cached\n-\t--ignore-unmatch ...`, see EXAMPLES below. For hairy\n-\tcases, see linkgit:git-update-index[1].\n-\n---parent-filter ::\n-\tThis is the filter for rewriting the commit's parent list.\n-\tIt will receive the parent string on stdin and shall output\n-\tthe new parent string on stdout. The parent string is in\n-\tthe format described in linkgit:git-commit-tree[1]: empty for\n-\tthe initial commit, \"-p parent\" for a normal commit and\n-\t\"-p parent1 -p parent2 -p parent3 ...\" for a merge commit.\n-\n---msg-filter ::\n-\tThis is the filter for rewriting the commit messages.\n-\tThe argument is evaluated in the shell with the original\n-\tcommit message on standard input; its standard output is\n-\tused as the new commit message.\n-\n---commit-filter ::\n-\tThis is the filter for performing the commit.\n-\tIf this filter is specified, it will be called instead of the\n-\t'git commit-tree' command, with arguments of the form\n-\t\" [(-p )...]\" and the log message on\n-\tstdin. The commit id is expected on stdout.\n-+\n-As a special extension, the commit filter may emit multiple\n-commit ids; in that case, the rewritten children of the original commit will\n-have all of them as parents.\n-+\n-You can use the 'map' convenience function in this filter, and other\n-convenience functions, too. For example, calling 'skip_commit \"$@\"'\n-will leave out the current commit (but not its changes! If you want\n-that, use 'git rebase' instead).\n-+\n-You can also use the `git_commit_non_empty_tree \"$@\"` instead of\n-`git commit-tree \"$@\"` if you don't wish to keep commits with a single parent\n-and that makes no change to the tree.\n-\n---tag-name-filter ::\n-\tThis is the filter for rewriting tag names. When passed,\n-\tit will be called for every tag ref that points to a rewritten\n-\tobject (or to a tag object which points to a rewritten object).\n-\tThe original tag name is passed via standard input, and the new\n-\ttag name is expected on standard output.\n-+\n-The original tags are not deleted, but can be overwritten;\n-use \"--tag-name-filter cat\" to simply update the tags. In this\n-case, be very careful and make sure you have the old tags\n-backed up in case the conversion has run afoul.\n-+\n-Nearly proper rewriting of tag objects is supported. If the tag has\n-a message attached, a new tag object will be created with the same message,\n-author, and timestamp. If the tag has a signature attached, the\n-signature will be stripped. It is by definition impossible to preserve\n-signatures. The reason this is \"nearly\" proper, is because ideally if\n-the tag did not change (points to the same object, has the same name, etc.)\n-it should retain any signature. That is not the case, signatures will always\n-be removed, buyer beware. There is also no support for changing the\n-author or timestamp (or the tag message for that matter). Tags which point\n-to other tags will be rewritten to point to the underlying commit.\n-\n---prune-empty::\n-\tSome filters will generate empty commits that leave the tree untouched.\n-\tThis option instructs git-filter-branch to remove such commits if they\n-\thave exactly one or zero non-pruned parents; merge commits will\n-\ttherefore remain intact. This option cannot be used together with\n-\t`--commit-filter`, though the same effect can be achieved by using the\n-\tprovided `git_commit_non_empty_tree` function in a commit filter.\n-\n---original ::\n-\tUse this option to set the namespace where the original commits\n-\twill be stored. The default value is 'refs/original'.\n-\n--d ::\n-\tUse this option to set the path to the temporary directory used for\n-\trewriting. When applying a tree filter, the command needs to\n-\ttemporarily check out the tree to some directory, which may consume\n-\tconsiderable space in case of large projects. By default it\n-\tdoes this in the `.git-rewrite/` directory but you can override\n-\tthat choice by this parameter.\n-\n--f::\n---force::\n-\t'git filter-branch' refuses to start with an existing temporary\n-\tdirectory or when there are already refs starting with\n-\t'refs/original/', unless forced.\n-\n---state-branch ::\n-\tThis option will cause the mapping from old to new objects to\n-\tbe loaded from named branch upon startup and saved as a new\n-\tcommit to that branch upon exit, enabling incremental of large\n-\ttrees. If '' does not exist it will be created.\n-\n-...::\n-\tArguments for 'git rev-list'. All positive refs included by\n-\tthese options are rewritten. You may also specify options\n-\tsuch as `--all`, but you must use `--` to separate them from\n-\tthe 'git filter-branch' options. Implies <>.\n-\n-\n-[[Remap_to_ancestor]]\n-Remap to ancestor\n-~~~~~~~~~~~~~~~~~\n-\n-By using linkgit:git-rev-list[1] arguments, e.g., path limiters, you can limit the\n-set of revisions which get rewritten. However, positive refs on the command\n-line are distinguished: we don't let them be excluded by such limiters. For\n-this purpose, they are instead rewritten to point at the nearest ancestor that\n-was not excluded.\n-\n-\n-EXIT STATUS\n------------\n-\n-On success, the exit status is `0`. If the filter can't find any commits to\n-rewrite, the exit status is `2`. On any other error, the exit status may be\n-any other non-zero value.\n-\n-\n-EXAMPLES\n---------\n-\n-Suppose you want to remove a file (containing confidential information\n-or copyright violation) from all commits:\n-\n--------------------------------------------------------\n-git filter-branch --tree-filter 'rm filename' HEAD\n--------------------------------------------------------\n-\n-However, if the file is absent from the tree of some commit,\n-a simple `rm filename` will fail for that tree and commit.\n-Thus you may instead want to use `rm -f filename` as the script.\n-\n-Using `--index-filter` with 'git rm' yields a significantly faster\n-version. Like with using `rm filename`, `git rm --cached filename`\n-will fail if the file is absent from the tree of a commit. If you\n-want to \"completely forget\" a file, it does not matter when it entered\n-history, so we also add `--ignore-unmatch`:\n-\n---------------------------------------------------------------------------\n-git filter-branch --index-filter 'git rm --cached --ignore-unmatch filename' HEAD\n---------------------------------------------------------------------------\n-\n-Now, you will get the rewritten history saved in HEAD.\n-\n-To rewrite the repository to look as if `foodir/` had been its project\n-root, and discard all other history:\n-\n--------------------------------------------------------\n-git filter-branch --subdirectory-filter foodir -- --all\n--------------------------------------------------------\n-\n-Thus you can, e.g., turn a library subdirectory into a repository of\n-its own. Note the `--` that separates 'filter-branch' options from\n-revision options, and the `--all` to rewrite all branches and tags.\n-\n-To set a commit (which typically is at the tip of another\n-history) to be the parent of the current initial commit, in\n-order to paste the other history behind the current history:\n-\n--------------------------------------------------------------------\n-git filter-branch --parent-filter 'sed \"s/^\\$/-p /\"' HEAD\n--------------------------------------------------------------------\n-\n-(if the parent string is empty - which happens when we are dealing with\n-the initial commit - add graftcommit as a parent). Note that this assumes\n-history with a single root (that is, no merge without common ancestors\n-happened). If this is not the case, use:\n-\n---------------------------------------------------------------------------\n-git filter-branch --parent-filter \\\n-\t'test $GIT_COMMIT = && echo \"-p \" || cat' HEAD\n---------------------------------------------------------------------------\n-\n-or even simpler:\n-\n------------------------------------------------\n-git replace --graft $commit-id $graft-id\n-git filter-branch $graft-id..HEAD\n------------------------------------------------\n-\n-To remove commits authored by \"Darl McBribe\" from the history:\n-\n-------------------------------------------------------------------------------\n-git filter-branch --commit-filter '\n-\tif [ \"$GIT_AUTHOR_NAME\" = \"Darl McBribe\" ];\n-\tthen\n-\t\tskip_commit \"$@\";\n-\telse\n-\t\tgit commit-tree \"$@\";\n-\tfi' HEAD\n-------------------------------------------------------------------------------\n-\n-The function 'skip_commit' is defined as follows:\n-\n---------------------------\n-skip_commit()\n-{\n-\tshift;\n-\twhile [ -n \"$1\" ];\n-\tdo\n-\t\tshift;\n-\t\tmap \"$1\";\n-\t\tshift;\n-\tdone;\n-}\n---------------------------\n-\n-The shift magic first throws away the tree id and then the -p\n-parameters. Note that this handles merges properly! In case Darl\n-committed a merge between P1 and P2, it will be propagated properly\n-and all children of the merge will become merge commits with P1,P2\n-as their parents instead of the merge commit.\n-\n-*NOTE* the changes introduced by the commits, and which are not reverted\n-by subsequent commits, will still be in the rewritten branch. If you want\n-to throw out _changes_ together with the commits, you should use the\n-interactive mode of 'git rebase'.\n-\n-You can rewrite the commit log messages using `--msg-filter`. For\n-example, 'git svn-id' strings in a repository created by 'git svn' can\n-be removed this way:\n-\n--------------------------------------------------------\n-git filter-branch --msg-filter '\n-\tsed -e \"/^git-svn-id:/d\"\n-'\n--------------------------------------------------------\n-\n-If you need to add 'Acked-by' lines to, say, the last 10 commits (none\n-of which is a merge), use this command:\n-\n---------------------------------------------------------\n-git filter-branch --msg-filter '\n-\tcat &&\n-\techo \"Acked-by: Bugs Bunny \"\n-' HEAD~10..HEAD\n---------------------------------------------------------\n-\n-The `--env-filter` option can be used to modify committer and/or author\n-identity. For example, if you found out that your commits have the wrong\n-identity due to a misconfigured user.email, you can make a correction,\n-before publishing the project, like this:\n-\n---------------------------------------------------------\n-git filter-branch --env-filter '\n-\tif test \"$GIT_AUTHOR_EMAIL\" = \"root@localhost\"\n-\tthen\n-\t\tGIT_AUTHOR_EMAIL=john@example.com\n-\tfi\n-\tif test \"$GIT_COMMITTER_EMAIL\" = \"root@localhost\"\n-\tthen\n-\t\tGIT_COMMITTER_EMAIL=john@example.com\n-\tfi\n-' -- --all\n---------------------------------------------------------\n-\n-To restrict rewriting to only part of the history, specify a revision\n-range in addition to the new branch name. The new branch name will\n-point to the top-most revision that a 'git rev-list' of this range\n-will print.\n-\n-Consider this history:\n-\n-------------------\n- D--E--F--G--H\n- / /\n-A--B-----C\n-------------------\n-\n-To rewrite only commits D,E,F,G,H, but leave A, B and C alone, use:\n-\n---------------------------------\n-git filter-branch ... C..H\n---------------------------------\n-\n-To rewrite commits E,F,G,H, use one of these:\n-\n-----------------------------------------\n-git filter-branch ... C..H --not D\n-git filter-branch ... D..H --not C\n-----------------------------------------\n-\n-To move the whole tree into a subdirectory, or remove it from there:\n-\n----------------------------------------------------------------\n-git filter-branch --index-filter \\\n-\t'git ls-files -s | sed \"s-\\t\\\"*-&newsubdir/-\" |\n-\t\tGIT_INDEX_FILE=$GIT_INDEX_FILE.new \\\n-\t\t\tgit update-index --index-info &&\n-\t mv \"$GIT_INDEX_FILE.new\" \"$GIT_INDEX_FILE\"' HEAD\n----------------------------------------------------------------\n-\n-\n-\n-CHECKLIST FOR SHRINKING A REPOSITORY\n-------------------------------------\n-\n-git-filter-branch can be used to get rid of a subset of files,\n-usually with some combination of `--index-filter` and\n-`--subdirectory-filter`. People expect the resulting repository to\n-be smaller than the original, but you need a few more steps to\n-actually make it smaller, because Git tries hard not to lose your\n-objects until you tell it to. First make sure that:\n-\n-* You really removed all variants of a filename, if a blob was moved\n- over its lifetime. `git log --name-only --follow --all -- filename`\n- can help you find renames.\n-\n-* You really filtered all refs: use `--tag-name-filter cat -- --all`\n- when calling git-filter-branch.\n-\n-Then there are two ways to get a smaller repository. A safer way is\n-to clone, that keeps your original intact.\n-\n-* Clone it with `git clone file:///path/to/repo`. The clone\n- will not have the removed objects. See linkgit:git-clone[1]. (Note\n- that cloning with a plain path just hardlinks everything!)\n-\n-If you really don't want to clone it, for whatever reasons, check the\n-following points instead (in this order). This is a very destructive\n-approach, so *make a backup* or go back to cloning it. You have been\n-warned.\n-\n-* Remove the original refs backed up by git-filter-branch: say `git\n- for-each-ref --format=\"%(refname)\" refs/original/ | xargs -n 1 git\n- update-ref -d`.\n-\n-* Expire all reflogs with `git reflog expire --expire=now --all`.\n-\n-* Garbage collect all unreferenced objects with `git gc --prune=now`\n- (or if your git-gc is not new enough to support arguments to\n- `--prune`, use `git repack -ad; git prune` instead).\n-\n-GIT\n----\n-Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex f9255344ae..20850def5d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -607,7 +607,6 @@ unexport CDPATH\n \n SCRIPT_SH += git-bisect.sh\n SCRIPT_SH += git-difftool--helper.sh\n-SCRIPT_SH += git-filter-branch.sh\n SCRIPT_SH += git-merge-octopus.sh\n SCRIPT_SH += git-merge-one-file.sh\n SCRIPT_SH += git-merge-resolve.sh\ndiff --git a/command-list.txt b/command-list.txt\nindex a9ac72bef4..1ba65d9516 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -90,7 +90,6 @@ git-fast-export ancillarymanipulators\n git-fast-import ancillarymanipulators\n git-fetch mainporcelain remote\n git-fetch-pack synchingrepositories\n-git-filter-branch ancillarymanipulators\n git-fmt-merge-msg purehelpers\n git-for-each-ref plumbinginterrogators\n git-format-patch mainporcelain\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\ndeleted file mode 100755\nindex 5c5afa2b98..0000000000\n--- a/git-filter-branch.sh\n+++ /dev/null\n@@ -1,662 +0,0 @@\n-#!/bin/sh\n-#\n-# Rewrite revision history\n-# Copyright (c) Petr Baudis, 2006\n-# Minimal changes to \"port\" it to core-git (c) Johannes Schindelin, 2007\n-#\n-# Lets you rewrite the revision history of the current branch, creating\n-# a new branch. You can specify a number of filters to modify the commits,\n-# files and trees.\n-\n-# The following functions will also be available in the commit filter:\n-\n-functions=$(cat << \\EOF\n-EMPTY_TREE=$(git hash-object -t tree /dev/null)\n-\n-warn () {\n-\techo \"$*\" >&2\n-}\n-\n-map()\n-{\n-\t# if it was not rewritten, take the original\n-\tif test -r \"$workdir/../map/$1\"\n-\tthen\n-\t\tcat \"$workdir/../map/$1\"\n-\telse\n-\t\techo \"$1\"\n-\tfi\n-}\n-\n-# if you run 'skip_commit \"$@\"' in a commit filter, it will print\n-# the (mapped) parents, effectively skipping the commit.\n-\n-skip_commit()\n-{\n-\tshift;\n-\twhile [ -n \"$1\" ];\n-\tdo\n-\t\tshift;\n-\t\tmap \"$1\";\n-\t\tshift;\n-\tdone;\n-}\n-\n-# if you run 'git_commit_non_empty_tree \"$@\"' in a commit filter,\n-# it will skip commits that leave the tree untouched, commit the other.\n-git_commit_non_empty_tree()\n-{\n-\tif test $# = 3 && test \"$1\" = $(git rev-parse \"$3^{tree}\"); then\n-\t\tmap \"$3\"\n-\telif test $# = 1 && test \"$1\" = $EMPTY_TREE; then\n-\t\t:\n-\telse\n-\t\tgit commit-tree \"$@\"\n-\tfi\n-}\n-# override die(): this version puts in an extra line break, so that\n-# the progress is still visible\n-\n-die()\n-{\n-\techo >&2\n-\techo \"$*\" >&2\n-\texit 1\n-}\n-EOF\n-)\n-\n-eval \"$functions\"\n-\n-finish_ident() {\n-\t# Ensure non-empty id name.\n-\techo \"case \\\"\\$GIT_$1_NAME\\\" in \\\"\\\") GIT_$1_NAME=\\\"\\${GIT_$1_EMAIL%%@*}\\\" && export GIT_$1_NAME;; esac\"\n-\t# And make sure everything is exported.\n-\techo \"export GIT_$1_NAME\"\n-\techo \"export GIT_$1_EMAIL\"\n-\techo \"export GIT_$1_DATE\"\n-}\n-\n-set_ident () {\n-\tparse_ident_from_commit author AUTHOR committer COMMITTER\n-\tfinish_ident AUTHOR\n-\tfinish_ident COMMITTER\n-}\n-\n-USAGE=\"[--setup ] [--subdirectory-filter ] [--env-filter ]\n-\t[--tree-filter ] [--index-filter ]\n-\t[--parent-filter ] [--msg-filter ]\n-\t[--commit-filter ] [--tag-name-filter ]\n-\t[--original ]\n-\t[-d ] [-f | --force] [--state-branch ]\n-\t[--] [...]\"\n-\n-OPTIONS_SPEC=\n-. git-sh-setup\n-\n-if [ \"$(is_bare_repository)\" = false ]; then\n-\trequire_clean_work_tree 'rewrite branches'\n-fi\n-\n-tempdir=.git-rewrite\n-filter_setup=\n-filter_env=\n-filter_tree=\n-filter_index=\n-filter_parent=\n-filter_msg=cat\n-filter_commit=\n-filter_tag_name=\n-filter_subdir=\n-state_branch=\n-orig_namespace=refs/original/\n-force=\n-prune_empty=\n-remap_to_ancestor=\n-while :\n-do\n-\tcase \"$1\" in\n-\t--)\n-\t\tshift\n-\t\tbreak\n-\t\t;;\n-\t--force|-f)\n-\t\tshift\n-\t\tforce=t\n-\t\tcontinue\n-\t\t;;\n-\t--remap-to-ancestor)\n-\t\t# deprecated ($remap_to_ancestor is set now automatically)\n-\t\tshift\n-\t\tremap_to_ancestor=t\n-\t\tcontinue\n-\t\t;;\n-\t--prune-empty)\n-\t\tshift\n-\t\tprune_empty=t\n-\t\tcontinue\n-\t\t;;\n-\t-*)\n-\t\t;;\n-\t*)\n-\t\tbreak;\n-\tesac\n-\n-\t# all switches take one argument\n-\tARG=\"$1\"\n-\tcase \"$#\" in 1) usage ;; esac\n-\tshift\n-\tOPTARG=\"$1\"\n-\tshift\n-\n-\tcase \"$ARG\" in\n-\t-d)\n-\t\ttempdir=\"$OPTARG\"\n-\t\t;;\n-\t--setup)\n-\t\tfilter_setup=\"$OPTARG\"\n-\t\t;;\n-\t--subdirectory-filter)\n-\t\tfilter_subdir=\"$OPTARG\"\n-\t\tremap_to_ancestor=t\n-\t\t;;\n-\t--env-filter)\n-\t\tfilter_env=\"$OPTARG\"\n-\t\t;;\n-\t--tree-filter)\n-\t\tfilter_tree=\"$OPTARG\"\n-\t\t;;\n-\t--index-filter)\n-\t\tfilter_index=\"$OPTARG\"\n-\t\t;;\n-\t--parent-filter)\n-\t\tfilter_parent=\"$OPTARG\"\n-\t\t;;\n-\t--msg-filter)\n-\t\tfilter_msg=\"$OPTARG\"\n-\t\t;;\n-\t--commit-filter)\n-\t\tfilter_commit=\"$functions; $OPTARG\"\n-\t\t;;\n-\t--tag-name-filter)\n-\t\tfilter_tag_name=\"$OPTARG\"\n-\t\t;;\n-\t--original)\n-\t\torig_namespace=$(expr \"$OPTARG/\" : '\\(.*[^/]\\)/*$')/\n-\t\t;;\n-\t--state-branch)\n-\t\tstate_branch=\"$OPTARG\"\n-\t\t;;\n-\t*)\n-\t\tusage\n-\t\t;;\n-\tesac\n-done\n-\n-case \"$prune_empty,$filter_commit\" in\n-,)\n-\tfilter_commit='git commit-tree \"$@\"';;\n-t,)\n-\tfilter_commit=\"$functions;\"' git_commit_non_empty_tree \"$@\"';;\n-,*)\n-\t;;\n-*)\n-\tdie \"Cannot set --prune-empty and --commit-filter at the same time\"\n-esac\n-\n-case \"$force\" in\n-t)\n-\trm -rf \"$tempdir\"\n-;;\n-'')\n-\ttest -d \"$tempdir\" &&\n-\t\tdie \"$tempdir already exists, please remove it\"\n-esac\n-orig_dir=$(pwd)\n-mkdir -p \"$tempdir/t\" &&\n-tempdir=\"$(cd \"$tempdir\"; pwd)\" &&\n-cd \"$tempdir/t\" &&\n-workdir=\"$(pwd)\" ||\n-die \"\"\n-\n-# Remove tempdir on exit\n-trap 'cd \"$orig_dir\"; rm -rf \"$tempdir\"' 0\n-\n-ORIG_GIT_DIR=\"$GIT_DIR\"\n-ORIG_GIT_WORK_TREE=\"$GIT_WORK_TREE\"\n-ORIG_GIT_INDEX_FILE=\"$GIT_INDEX_FILE\"\n-ORIG_GIT_AUTHOR_NAME=\"$GIT_AUTHOR_NAME\"\n-ORIG_GIT_AUTHOR_EMAIL=\"$GIT_AUTHOR_EMAIL\"\n-ORIG_GIT_AUTHOR_DATE=\"$GIT_AUTHOR_DATE\"\n-ORIG_GIT_COMMITTER_NAME=\"$GIT_COMMITTER_NAME\"\n-ORIG_GIT_COMMITTER_EMAIL=\"$GIT_COMMITTER_EMAIL\"\n-ORIG_GIT_COMMITTER_DATE=\"$GIT_COMMITTER_DATE\"\n-\n-GIT_WORK_TREE=.\n-export GIT_DIR GIT_WORK_TREE\n-\n-# Make sure refs/original is empty\n-git for-each-ref > \"$tempdir\"/backup-refs || exit\n-while read sha1 type name\n-do\n-\tcase \"$force,$name\" in\n-\t,$orig_namespace*)\n-\t\tdie \"Cannot create a new backup.\n-A previous backup already exists in $orig_namespace\n-Force overwriting the backup with -f\"\n-\t;;\n-\tt,$orig_namespace*)\n-\t\tgit update-ref -d \"$name\" $sha1\n-\t;;\n-\tesac\n-done < \"$tempdir\"/backup-refs\n-\n-# The refs should be updated if their heads were rewritten\n-git rev-parse --no-flags --revs-only --symbolic-full-name \\\n-\t--default HEAD \"$@\" > \"$tempdir\"/raw-refs || exit\n-while read ref\n-do\n-\tcase \"$ref\" in ^?*) continue ;; esac\n-\n-\tif git rev-parse --verify \"$ref\"^0 >/dev/null 2>&1\n-\tthen\n-\t\techo \"$ref\"\n-\telse\n-\t\twarn \"WARNING: not rewriting '$ref' (not a committish)\"\n-\tfi\n-done >\"$tempdir\"/heads <\"$tempdir\"/raw-refs\n-\n-test -s \"$tempdir\"/heads ||\n-\tdie \"You must specify a ref to rewrite.\"\n-\n-GIT_INDEX_FILE=\"$(pwd)/../index\"\n-export GIT_INDEX_FILE\n-\n-# map old->new commit ids for rewriting parents\n-mkdir ../map || die \"Could not create map/ directory\"\n-\n-if test -n \"$state_branch\"\n-then\n-\tstate_commit=$(git rev-parse --no-flags --revs-only \"$state_branch\")\n-\tif test -n \"$state_commit\"\n-\tthen\n-\t\techo \"Populating map from $state_branch ($state_commit)\" 1>&2\n-\t\tperl -e'open(MAP, \"-|\", \"git show $ARGV[0]:filter.map\") or die;\n-\t\t\twhile () {\n-\t\t\t\tm/(.*):(.*)/ or die;\n-\t\t\t\topen F, \">../map/$1\" or die;\n-\t\t\t\tprint F \"$2\" or die;\n-\t\t\t\tclose(F) or die;\n-\t\t\t}\n-\t\t\tclose(MAP) or die;' \"$state_commit\" \\\n-\t\t\t\t|| die \"Unable to load state from $state_branch:filter.map\"\n-\telse\n-\t\techo \"Branch $state_branch does not exist. Will create\" 1>&2\n-\tfi\n-fi\n-\n-# we need \"--\" only if there are no path arguments in $@\n-nonrevs=$(git rev-parse --no-revs \"$@\") || exit\n-if test -z \"$nonrevs\"\n-then\n-\tdashdash=--\n-else\n-\tdashdash=\n-\tremap_to_ancestor=t\n-fi\n-\n-git rev-parse --revs-only \"$@\" >../parse\n-\n-case \"$filter_subdir\" in\n-\"\")\n-\teval set -- \"$(git rev-parse --sq --no-revs \"$@\")\"\n-\t;;\n-*)\n-\teval set -- \"$(git rev-parse --sq --no-revs \"$@\" $dashdash \\\n-\t\t\"$filter_subdir\")\"\n-\t;;\n-esac\n-\n-git rev-list --reverse --topo-order --default HEAD \\\n-\t--parents --simplify-merges --stdin \"$@\" <../parse >../revs ||\n-\tdie \"Could not get the commits\"\n-commits=$(wc -l <../revs | tr -d \" \")\n-\n-test $commits -eq 0 && die_with_status 2 \"Found nothing to rewrite\"\n-\n-# Rewrite the commits\n-report_progress ()\n-{\n-\tif test -n \"$progress\" &&\n-\t\ttest $git_filter_branch__commit_count -gt $next_sample_at\n-\tthen\n-\t\tcount=$git_filter_branch__commit_count\n-\n-\t\tnow=$(date +%s)\n-\t\telapsed=$(($now - $start_timestamp))\n-\t\tremaining=$(( ($commits - $count) * $elapsed / $count ))\n-\t\tif test $elapsed -gt 0\n-\t\tthen\n-\t\t\tnext_sample_at=$(( ($elapsed + 1) * $count / $elapsed ))\n-\t\telse\n-\t\t\tnext_sample_at=$(($next_sample_at + 1))\n-\t\tfi\n-\t\tprogress=\" ($elapsed seconds passed, remaining $remaining predicted)\"\n-\tfi\n-\tprintf \"\\rRewrite $commit ($count/$commits)$progress \"\n-}\n-\n-git_filter_branch__commit_count=0\n-\n-progress= start_timestamp=\n-if date '+%s' 2>/dev/null | grep -q '^[0-9][0-9]*$'\n-then\n-\tnext_sample_at=0\n-\tprogress=\"dummy to ensure this is not empty\"\n-\tstart_timestamp=$(date '+%s')\n-fi\n-\n-if test -n \"$filter_index\" ||\n- test -n \"$filter_tree\" ||\n- test -n \"$filter_subdir\"\n-then\n-\tneed_index=t\n-else\n-\tneed_index=\n-fi\n-\n-eval \"$filter_setup\" < /dev/null ||\n-\tdie \"filter setup failed: $filter_setup\"\n-\n-while read commit parents; do\n-\tgit_filter_branch__commit_count=$(($git_filter_branch__commit_count+1))\n-\n-\treport_progress\n-\ttest -f \"$workdir\"/../map/$commit && continue\n-\n-\tcase \"$filter_subdir\" in\n-\t\"\")\n-\t\tif test -n \"$need_index\"\n-\t\tthen\n-\t\t\tGIT_ALLOW_NULL_SHA1=1 git read-tree -i -m $commit\n-\t\tfi\n-\t\t;;\n-\t*)\n-\t\t# The commit may not have the subdirectory at all\n-\t\terr=$(GIT_ALLOW_NULL_SHA1=1 \\\n-\t\t git read-tree -i -m $commit:\"$filter_subdir\" 2>&1) || {\n-\t\t\tif ! git rev-parse -q --verify $commit:\"$filter_subdir\"\n-\t\t\tthen\n-\t\t\t\trm -f \"$GIT_INDEX_FILE\"\n-\t\t\telse\n-\t\t\t\techo >&2 \"$err\"\n-\t\t\t\tfalse\n-\t\t\tfi\n-\t\t}\n-\tesac || die \"Could not initialize the index\"\n-\n-\tGIT_COMMIT=$commit\n-\texport GIT_COMMIT\n-\tgit cat-file commit \"$commit\" >../commit ||\n-\t\tdie \"Cannot read commit $commit\"\n-\n-\teval \"$(set_ident <../commit)\" ||\n-\t\tdie \"setting author/committer failed for commit $commit\"\n-\teval \"$filter_env\" < /dev/null ||\n-\t\tdie \"env filter failed: $filter_env\"\n-\n-\tif [ \"$filter_tree\" ]; then\n-\t\tgit checkout-index -f -u -a ||\n-\t\t\tdie \"Could not checkout the index\"\n-\t\t# files that $commit removed are now still in the working tree;\n-\t\t# remove them, else they would be added again\n-\t\tgit clean -d -q -f -x\n-\t\teval \"$filter_tree\" < /dev/null ||\n-\t\t\tdie \"tree filter failed: $filter_tree\"\n-\n-\t\t(\n-\t\t\tgit diff-index -r --name-only --ignore-submodules $commit -- &&\n-\t\t\tgit ls-files --others\n-\t\t) > \"$tempdir\"/tree-state || exit\n-\t\tgit update-index --add --replace --remove --stdin \\\n-\t\t\t< \"$tempdir\"/tree-state || exit\n-\tfi\n-\n-\teval \"$filter_index\" < /dev/null ||\n-\t\tdie \"index filter failed: $filter_index\"\n-\n-\tparentstr=\n-\tfor parent in $parents; do\n-\t\tfor reparent in $(map \"$parent\"); do\n-\t\t\tcase \"$parentstr \" in\n-\t\t\t*\" -p $reparent \"*)\n-\t\t\t\t;;\n-\t\t\t*)\n-\t\t\t\tparentstr=\"$parentstr -p $reparent\"\n-\t\t\t\t;;\n-\t\t\tesac\n-\t\tdone\n-\tdone\n-\tif [ \"$filter_parent\" ]; then\n-\t\tparentstr=\"$(echo \"$parentstr\" | eval \"$filter_parent\")\" ||\n-\t\t\t\tdie \"parent filter failed: $filter_parent\"\n-\tfi\n-\n-\t{\n-\t\twhile IFS='' read -r header_line && test -n \"$header_line\"\n-\t\tdo\n-\t\t\t# skip header lines...\n-\t\t\t:;\n-\t\tdone\n-\t\t# and output the actual commit message\n-\t\tcat\n-\t} <../commit |\n-\t\teval \"$filter_msg\" > ../message ||\n-\t\t\tdie \"msg filter failed: $filter_msg\"\n-\n-\tif test -n \"$need_index\"\n-\tthen\n-\t\ttree=$(git write-tree)\n-\telse\n-\t\ttree=$(git rev-parse \"$commit^{tree}\")\n-\tfi\n-\tworkdir=$workdir @SHELL_PATH@ -c \"$filter_commit\" \"git commit-tree\" \\\n-\t\t\"$tree\" $parentstr < ../message > ../map/$commit ||\n-\t\t\tdie \"could not write rewritten commit\"\n-done <../revs\n-\n-# If we are filtering for paths, as in the case of a subdirectory\n-# filter, it is possible that a specified head is not in the set of\n-# rewritten commits, because it was pruned by the revision walker.\n-# Ancestor remapping fixes this by mapping these heads to the unique\n-# nearest ancestor that survived the pruning.\n-\n-if test \"$remap_to_ancestor\" = t\n-then\n-\twhile read ref\n-\tdo\n-\t\tsha1=$(git rev-parse \"$ref\"^0)\n-\t\ttest -f \"$workdir\"/../map/$sha1 && continue\n-\t\tancestor=$(git rev-list --simplify-merges -1 \"$ref\" \"$@\")\n-\t\ttest \"$ancestor\" && echo $(map $ancestor) >> \"$workdir\"/../map/$sha1\n-\tdone < \"$tempdir\"/heads\n-fi\n-\n-# Finally update the refs\n-\n-_x40='[0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f]'\n-_x40=\"$_x40$_x40$_x40$_x40$_x40$_x40$_x40$_x40\"\n-echo\n-while read ref\n-do\n-\t# avoid rewriting a ref twice\n-\ttest -f \"$orig_namespace$ref\" && continue\n-\n-\tsha1=$(git rev-parse \"$ref\"^0)\n-\trewritten=$(map $sha1)\n-\n-\ttest $sha1 = \"$rewritten\" &&\n-\t\twarn \"WARNING: Ref '$ref' is unchanged\" &&\n-\t\tcontinue\n-\n-\tcase \"$rewritten\" in\n-\t'')\n-\t\techo \"Ref '$ref' was deleted\"\n-\t\tgit update-ref -m \"filter-branch: delete\" -d \"$ref\" $sha1 ||\n-\t\t\tdie \"Could not delete $ref\"\n-\t;;\n-\t$_x40)\n-\t\techo \"Ref '$ref' was rewritten\"\n-\t\tif ! git update-ref -m \"filter-branch: rewrite\" \\\n-\t\t\t\t\t\"$ref\" $rewritten $sha1 2>/dev/null; then\n-\t\t\tif test $(git cat-file -t \"$ref\") = tag; then\n-\t\t\t\tif test -z \"$filter_tag_name\"; then\n-\t\t\t\t\twarn \"WARNING: You said to rewrite tagged commits, but not the corresponding tag.\"\n-\t\t\t\t\twarn \"WARNING: Perhaps use '--tag-name-filter cat' to rewrite the tag.\"\n-\t\t\t\tfi\n-\t\t\telse\n-\t\t\t\tdie \"Could not rewrite $ref\"\n-\t\t\tfi\n-\t\tfi\n-\t;;\n-\t*)\n-\t\t# NEEDSWORK: possibly add -Werror, making this an error\n-\t\twarn \"WARNING: '$ref' was rewritten into multiple commits:\"\n-\t\twarn \"$rewritten\"\n-\t\twarn \"WARNING: Ref '$ref' points to the first one now.\"\n-\t\trewritten=$(echo \"$rewritten\" | head -n 1)\n-\t\tgit update-ref -m \"filter-branch: rewrite to first\" \\\n-\t\t\t\t\"$ref\" $rewritten $sha1 ||\n-\t\t\tdie \"Could not rewrite $ref\"\n-\t;;\n-\tesac\n-\tgit update-ref -m \"filter-branch: backup\" \"$orig_namespace$ref\" $sha1 ||\n-\t\t exit\n-done < \"$tempdir\"/heads\n-\n-# TODO: This should possibly go, with the semantics that all positive given\n-# refs are updated, and their original heads stored in refs/original/\n-# Filter tags\n-\n-if [ \"$filter_tag_name\" ]; then\n-\tgit for-each-ref --format='%(objectname) %(objecttype) %(refname)' refs/tags |\n-\twhile read sha1 type ref; do\n-\t\tref=\"${ref#refs/tags/}\"\n-\t\t# XXX: Rewrite tagged trees as well?\n-\t\tif [ \"$type\" != \"commit\" -a \"$type\" != \"tag\" ]; then\n-\t\t\tcontinue;\n-\t\tfi\n-\n-\t\tif [ \"$type\" = \"tag\" ]; then\n-\t\t\t# Dereference to a commit\n-\t\t\tsha1t=\"$sha1\"\n-\t\t\tsha1=\"$(git rev-parse -q \"$sha1\"^{commit})\" || continue\n-\t\tfi\n-\n-\t\t[ -f \"../map/$sha1\" ] || continue\n-\t\tnew_sha1=\"$(cat \"../map/$sha1\")\"\n-\t\tGIT_COMMIT=\"$sha1\"\n-\t\texport GIT_COMMIT\n-\t\tnew_ref=\"$(echo \"$ref\" | eval \"$filter_tag_name\")\" ||\n-\t\t\tdie \"tag name filter failed: $filter_tag_name\"\n-\n-\t\techo \"$ref -> $new_ref ($sha1 -> $new_sha1)\"\n-\n-\t\tif [ \"$type\" = \"tag\" ]; then\n-\t\t\tnew_sha1=$( ( printf 'object %s\\ntype commit\\ntag %s\\n' \\\n-\t\t\t\t\t\t\"$new_sha1\" \"$new_ref\"\n-\t\t\t\tgit cat-file tag \"$ref\" |\n-\t\t\t\tsed -n \\\n-\t\t\t\t -e '1,/^$/{\n-\t\t\t\t\t /^object /d\n-\t\t\t\t\t /^type /d\n-\t\t\t\t\t /^tag /d\n-\t\t\t\t\t}' \\\n-\t\t\t\t -e '/^-----BEGIN PGP SIGNATURE-----/q' \\\n-\t\t\t\t -e 'p' ) |\n-\t\t\t\tgit hash-object -t tag -w --stdin) ||\n-\t\t\t\tdie \"Could not create new tag object for $ref\"\n-\t\t\tif git cat-file tag \"$ref\" | \\\n-\t\t\t sane_grep '^-----BEGIN PGP SIGNATURE-----' >/dev/null 2>&1\n-\t\t\tthen\n-\t\t\t\twarn \"gpg signature stripped from tag object $sha1t\"\n-\t\t\tfi\n-\t\tfi\n-\n-\t\tgit update-ref \"refs/tags/$new_ref\" \"$new_sha1\" ||\n-\t\t\tdie \"Could not write tag $new_ref\"\n-\tdone\n-fi\n-\n-unset GIT_DIR GIT_WORK_TREE GIT_INDEX_FILE\n-unset GIT_AUTHOR_NAME GIT_AUTHOR_EMAIL GIT_AUTHOR_DATE\n-unset GIT_COMMITTER_NAME GIT_COMMITTER_EMAIL GIT_COMMITTER_DATE\n-test -z \"$ORIG_GIT_DIR\" || {\n-\tGIT_DIR=\"$ORIG_GIT_DIR\" && export GIT_DIR\n-}\n-test -z \"$ORIG_GIT_WORK_TREE\" || {\n-\tGIT_WORK_TREE=\"$ORIG_GIT_WORK_TREE\" &&\n-\texport GIT_WORK_TREE\n-}\n-test -z \"$ORIG_GIT_INDEX_FILE\" || {\n-\tGIT_INDEX_FILE=\"$ORIG_GIT_INDEX_FILE\" &&\n-\texport GIT_INDEX_FILE\n-}\n-test -z \"$ORIG_GIT_AUTHOR_NAME\" || {\n-\tGIT_AUTHOR_NAME=\"$ORIG_GIT_AUTHOR_NAME\" &&\n-\texport GIT_AUTHOR_NAME\n-}\n-test -z \"$ORIG_GIT_AUTHOR_EMAIL\" || {\n-\tGIT_AUTHOR_EMAIL=\"$ORIG_GIT_AUTHOR_EMAIL\" &&\n-\texport GIT_AUTHOR_EMAIL\n-}\n-test -z \"$ORIG_GIT_AUTHOR_DATE\" || {\n-\tGIT_AUTHOR_DATE=\"$ORIG_GIT_AUTHOR_DATE\" &&\n-\texport GIT_AUTHOR_DATE\n-}\n-test -z \"$ORIG_GIT_COMMITTER_NAME\" || {\n-\tGIT_COMMITTER_NAME=\"$ORIG_GIT_COMMITTER_NAME\" &&\n-\texport GIT_COMMITTER_NAME\n-}\n-test -z \"$ORIG_GIT_COMMITTER_EMAIL\" || {\n-\tGIT_COMMITTER_EMAIL=\"$ORIG_GIT_COMMITTER_EMAIL\" &&\n-\texport GIT_COMMITTER_EMAIL\n-}\n-test -z \"$ORIG_GIT_COMMITTER_DATE\" || {\n-\tGIT_COMMITTER_DATE=\"$ORIG_GIT_COMMITTER_DATE\" &&\n-\texport GIT_COMMITTER_DATE\n-}\n-\n-if test -n \"$state_branch\"\n-then\n-\techo \"Saving rewrite state to $state_branch\" 1>&2\n-\tstate_blob=$(\n-\t\tperl -e'opendir D, \"../map\" or die;\n-\t\t\topen H, \"|-\", \"git hash-object -w --stdin\" or die;\n-\t\t\tforeach (sort readdir(D)) {\n-\t\t\t\tnext if m/^\\.\\.?$/;\n-\t\t\t\topen F, \"<../map/$_\" or die;\n-\t\t\t\tchomp($f = );\n-\t\t\t\tprint H \"$_:$f\\n\" or die;\n-\t\t\t}\n-\t\t\tclose(H) or die;' || die \"Unable to save state\")\n-\tstate_tree=$(printf '100644 blob %s\\tfilter.map\\n' \"$state_blob\" | git mktree)\n-\tif test -n \"$state_commit\"\n-\tthen\n-\t\tstate_commit=$(echo \"Sync\" | git commit-tree \"$state_tree\" -p \"$state_commit\")\n-\telse\n-\t\tstate_commit=$(echo \"Sync\" | git commit-tree \"$state_tree\" )\n-\tfi\n-\tgit update-ref \"$state_branch\" \"$state_commit\"\n-fi\n-\n-cd \"$orig_dir\"\n-rm -rf \"$tempdir\"\n-\n-trap - 0\n-\n-if [ \"$(is_bare_repository)\" = false ]; then\n-\tgit read-tree -u -m HEAD || exit\n-fi\n-\n-exit 0\ndiff --git a/t/perf/p7000-filter-branch.sh b/t/perf/p7000-filter-branch.sh\ndeleted file mode 100755\nindex b029586ccb..0000000000\n--- a/t/perf/p7000-filter-branch.sh\n+++ /dev/null\n@@ -1,24 +0,0 @@\n-#!/bin/sh\n-\n-test_description='performance of filter-branch'\n-. ./perf-lib.sh\n-\n-test_perf_default_repo\n-test_checkout_worktree\n-\n-test_expect_success 'mark bases for tests' '\n-\tgit tag -f tip &&\n-\tgit tag -f base HEAD~100\n-'\n-\n-test_perf 'noop filter' '\n-\tgit checkout --detach tip &&\n-\tgit filter-branch -f base..HEAD\n-'\n-\n-test_perf 'noop prune-empty' '\n-\tgit checkout --detach tip &&\n-\tgit filter-branch -f --prune-empty base..HEAD\n-'\n-\n-test_done\ndiff --git a/t/t7003-filter-branch.sh b/t/t7003-filter-branch.sh\ndeleted file mode 100755\nindex e23de7d0b5..0000000000\n--- a/t/t7003-filter-branch.sh\n+++ /dev/null\n@@ -1,505 +0,0 @@\n-#!/bin/sh\n-\n-test_description='git filter-branch'\n-. ./test-lib.sh\n-. \"$TEST_DIRECTORY/lib-gpg.sh\"\n-\n-test_expect_success 'setup' '\n-\ttest_commit A &&\n-\tGIT_COMMITTER_DATE=\"@0 +0000\" GIT_AUTHOR_DATE=\"@0 +0000\" &&\n-\ttest_commit --notick B &&\n-\tgit checkout -b branch B &&\n-\ttest_commit D &&\n-\tmkdir dir &&\n-\ttest_commit dir/D &&\n-\ttest_commit E &&\n-\tgit checkout master &&\n-\ttest_commit C &&\n-\tgit checkout branch &&\n-\tgit merge C &&\n-\tgit tag F &&\n-\ttest_commit G &&\n-\ttest_commit H\n-'\n-# * (HEAD, branch) H\n-# * G\n-# * Merge commit 'C' into branch\n-# |\\\n-# | * (master) C\n-# * | E\n-# * | dir/D\n-# * | D\n-# |/\n-# * B\n-# * A\n-\n-\n-H=$(git rev-parse H)\n-\n-test_expect_success 'rewrite identically' '\n-\tgit filter-branch branch\n-'\n-test_expect_success 'result is really identical' '\n-\ttest $H = $(git rev-parse HEAD)\n-'\n-\n-test_expect_success 'rewrite bare repository identically' '\n-\t(git config core.bare true && cd .git &&\n-\t git filter-branch branch > filter-output 2>&1 &&\n-\t! fgrep fatal filter-output)\n-'\n-git config core.bare false\n-test_expect_success 'result is really identical' '\n-\ttest $H = $(git rev-parse HEAD)\n-'\n-\n-TRASHDIR=$(pwd)\n-test_expect_success 'correct GIT_DIR while using -d' '\n-\tmkdir drepo &&\n-\t( cd drepo &&\n-\tgit init &&\n-\ttest_commit drepo &&\n-\tgit filter-branch -d \"$TRASHDIR/dfoo\" \\\n-\t\t--index-filter \"cp \\\"$TRASHDIR\\\"/dfoo/backup-refs \\\"$TRASHDIR\\\"\" \\\n-\t) &&\n-\tgrep drepo \"$TRASHDIR/backup-refs\"\n-'\n-\n-test_expect_success 'tree-filter works with -d' '\n-\tgit init drepo-tree &&\n-\t(\n-\t\tcd drepo-tree &&\n-\t\ttest_commit one &&\n-\t\tgit filter-branch -d \"$TRASHDIR/dfoo\" \\\n-\t\t\t--tree-filter \"echo changed >one.t\" &&\n-\t\techo changed >expect &&\n-\t\tgit cat-file blob HEAD:one.t >actual &&\n-\t\ttest_cmp expect actual &&\n-\t\ttest_cmp one.t actual\n-\t)\n-'\n-\n-test_expect_success 'Fail if commit filter fails' '\n-\ttest_must_fail git filter-branch -f --commit-filter \"exit 1\" HEAD\n-'\n-\n-test_expect_success 'rewrite, renaming a specific file' '\n-\tgit filter-branch -f --tree-filter \"mv D.t doh || :\" HEAD\n-'\n-\n-test_expect_success 'test that the file was renamed' '\n-\ttest D = \"$(git show HEAD:doh --)\" &&\n-\t! test -f D.t &&\n-\ttest -f doh &&\n-\ttest D = \"$(cat doh)\"\n-'\n-\n-test_expect_success 'rewrite, renaming a specific directory' '\n-\tgit filter-branch -f --tree-filter \"mv dir diroh || :\" HEAD\n-'\n-\n-test_expect_success 'test that the directory was renamed' '\n-\ttest dir/D = \"$(git show HEAD:diroh/D.t --)\" &&\n-\t! test -d dir &&\n-\ttest -d diroh &&\n-\t! test -d diroh/dir &&\n-\ttest -f diroh/D.t &&\n-\ttest dir/D = \"$(cat diroh/D.t)\"\n-'\n-\n-V=$(git rev-parse HEAD)\n-\n-test_expect_success 'populate --state-branch' '\n-\tgit filter-branch --state-branch state -f --tree-filter \"touch file || :\" HEAD\n-'\n-\n-W=$(git rev-parse HEAD)\n-\n-test_expect_success 'using --state-branch to skip already rewritten commits' '\n-\ttest_when_finished git reset --hard $V &&\n-\tgit reset --hard $V &&\n-\tgit filter-branch --state-branch state -f --tree-filter \"touch file || :\" HEAD &&\n-\ttest_cmp_rev $W HEAD\n-'\n-\n-git tag oldD HEAD~4\n-test_expect_success 'rewrite one branch, keeping a side branch' '\n-\tgit branch modD oldD &&\n-\tgit filter-branch -f --tree-filter \"mv B.t boh || :\" D..modD\n-'\n-\n-test_expect_success 'common ancestor is still common (unchanged)' '\n-\ttest \"$(git merge-base modD D)\" = \"$(git rev-parse B)\"\n-'\n-\n-test_expect_success 'filter subdirectory only' '\n-\tmkdir subdir &&\n-\ttouch subdir/new &&\n-\tgit add subdir/new &&\n-\ttest_tick &&\n-\tgit commit -m \"subdir\" &&\n-\techo H > A.t &&\n-\ttest_tick &&\n-\tgit commit -m \"not subdir\" A.t &&\n-\techo A > subdir/new &&\n-\ttest_tick &&\n-\tgit commit -m \"again subdir\" subdir/new &&\n-\tgit rm A.t &&\n-\ttest_tick &&\n-\tgit commit -m \"again not subdir\" &&\n-\tgit branch sub &&\n-\tgit branch sub-earlier HEAD~2 &&\n-\tgit filter-branch -f --subdirectory-filter subdir \\\n-\t\trefs/heads/sub refs/heads/sub-earlier\n-'\n-\n-test_expect_success 'subdirectory filter result looks okay' '\n-\ttest 2 = $(git rev-list sub | wc -l) &&\n-\tgit show sub:new &&\n-\ttest_must_fail git show sub:subdir &&\n-\tgit show sub-earlier:new &&\n-\ttest_must_fail git show sub-earlier:subdir\n-'\n-\n-test_expect_success 'more setup' '\n-\tgit checkout master &&\n-\tmkdir subdir &&\n-\techo A > subdir/new &&\n-\tgit add subdir/new &&\n-\ttest_tick &&\n-\tgit commit -m \"subdir on master\" subdir/new &&\n-\tgit rm A.t &&\n-\ttest_tick &&\n-\tgit commit -m \"again subdir on master\" &&\n-\tgit merge branch\n-'\n-\n-test_expect_success 'use index-filter to move into a subdirectory' '\n-\tgit branch directorymoved &&\n-\tgit filter-branch -f --index-filter \\\n-\t\t \"git ls-files -s | sed \\\"s-\t-&newsubdir/-\\\" |\n-\t GIT_INDEX_FILE=\\$GIT_INDEX_FILE.new \\\n-\t\t\tgit update-index --index-info &&\n-\t\t mv \\\"\\$GIT_INDEX_FILE.new\\\" \\\"\\$GIT_INDEX_FILE\\\"\" directorymoved &&\n-\tgit diff --exit-code HEAD directorymoved:newsubdir\n-'\n-\n-test_expect_success 'stops when msg filter fails' '\n-\told=$(git rev-parse HEAD) &&\n-\ttest_must_fail git filter-branch -f --msg-filter false HEAD &&\n-\ttest $old = $(git rev-parse HEAD) &&\n-\trm -rf .git-rewrite\n-'\n-\n-test_expect_success 'author information is preserved' '\n-\t: > i &&\n-\tgit add i &&\n-\ttest_tick &&\n-\tGIT_AUTHOR_NAME=\"B V Uips\" git commit -m bvuips &&\n-\tgit branch preserved-author &&\n-\t(sane_unset GIT_AUTHOR_NAME &&\n-\t git filter-branch -f --msg-filter \"cat; \\\n-\t\t\ttest \\$GIT_COMMIT != $(git rev-parse master) || \\\n-\t\t\techo Hallo\" \\\n-\t\tpreserved-author) &&\n-\tgit rev-list --author=\"B V Uips\" preserved-author >actual &&\n-\ttest_line_count = 1 actual\n-'\n-\n-test_expect_success \"remove a certain author's commits\" '\n-\techo i > i &&\n-\ttest_tick &&\n-\tgit commit -m i i &&\n-\tgit branch removed-author &&\n-\tgit filter-branch -f --commit-filter \"\\\n-\t\tif [ \\\"\\$GIT_AUTHOR_NAME\\\" = \\\"B V Uips\\\" ];\\\n-\t\tthen\\\n-\t\t\tskip_commit \\\"\\$@\\\";\n-\t\telse\\\n-\t\t\tgit commit-tree \\\"\\$@\\\";\\\n-\t\tfi\" removed-author &&\n-\tcnt1=$(git rev-list master | wc -l) &&\n-\tcnt2=$(git rev-list removed-author | wc -l) &&\n-\ttest $cnt1 -eq $(($cnt2 + 1)) &&\n-\tgit rev-list --author=\"B V Uips\" removed-author >actual &&\n-\ttest_line_count = 0 actual\n-'\n-\n-test_expect_success 'barf on invalid name' '\n-\ttest_must_fail git filter-branch -f master xy-problem &&\n-\ttest_must_fail git filter-branch -f HEAD^\n-'\n-\n-test_expect_success '\"map\" works in commit filter' '\n-\tgit filter-branch -f --commit-filter \"\\\n-\t\tparent=\\$(git rev-parse \\$GIT_COMMIT^) &&\n-\t\tmapped=\\$(map \\$parent) &&\n-\t\tactual=\\$(echo \\\"\\$@\\\" | sed \\\"s/^.*-p //\\\") &&\n-\t\ttest \\$mapped = \\$actual &&\n-\t\tgit commit-tree \\\"\\$@\\\";\" master~2..master &&\n-\tgit rev-parse --verify master\n-'\n-\n-test_expect_success 'Name needing quotes' '\n-\n-\tgit checkout -b rerere A &&\n-\tmkdir foo &&\n-\tname=\"れれれ\" &&\n-\t>foo/$name &&\n-\tgit add foo &&\n-\tgit commit -m \"Adding a file\" &&\n-\tgit filter-branch --tree-filter \"rm -fr foo\" &&\n-\ttest_must_fail git ls-files --error-unmatch \"foo/$name\" &&\n-\ttest $(git rev-parse --verify rerere) != $(git rev-parse --verify A)\n-\n-'\n-\n-test_expect_success 'Subdirectory filter with disappearing trees' '\n-\tgit reset --hard &&\n-\tgit checkout master &&\n-\n-\tmkdir foo &&\n-\ttouch foo/bar &&\n-\tgit add foo &&\n-\ttest_tick &&\n-\tgit commit -m \"Adding foo\" &&\n-\n-\tgit rm -r foo &&\n-\ttest_tick &&\n-\tgit commit -m \"Removing foo\" &&\n-\n-\tmkdir foo &&\n-\ttouch foo/bar &&\n-\tgit add foo &&\n-\ttest_tick &&\n-\tgit commit -m \"Re-adding foo\" &&\n-\n-\tgit filter-branch -f --subdirectory-filter foo &&\n-\tgit rev-list master >actual &&\n-\ttest_line_count = 3 actual\n-'\n-\n-test_expect_success 'Tag name filtering retains tag message' '\n-\tgit tag -m atag T &&\n-\tgit cat-file tag T > expect &&\n-\tgit filter-branch -f --tag-name-filter cat &&\n-\tgit cat-file tag T > actual &&\n-\ttest_cmp expect actual\n-'\n-\n-faux_gpg_tag='object XXXXXX\n-type commit\n-tag S\n-tagger T A Gger 1206026339 -0500\n-\n-This is a faux gpg signed tag.\n------BEGIN PGP SIGNATURE-----\n-Version: FauxGPG v0.0.0 (FAUX/Linux)\n-\n-gdsfoewhxu/6l06f1kxyxhKdZkrcbaiOMtkJUA9ITAc1mlamh0ooasxkH1XwMbYQ\n-acmwXaWET20H0GeAGP+7vow=\n-=agpO\n------END PGP SIGNATURE-----\n-'\n-test_expect_success 'Tag name filtering strips gpg signature' '\n-\tsha1=$(git rev-parse HEAD) &&\n-\tsha1t=$(echo \"$faux_gpg_tag\" | sed -e s/XXXXXX/$sha1/ | git mktag) &&\n-\tgit update-ref \"refs/tags/S\" \"$sha1t\" &&\n-\techo \"$faux_gpg_tag\" | sed -e s/XXXXXX/$sha1/ | head -n 6 > expect &&\n-\tgit filter-branch -f --tag-name-filter cat &&\n-\tgit cat-file tag S > actual &&\n-\ttest_cmp expect actual\n-'\n-\n-test_expect_success GPG 'Filtering retains message of gpg signed commit' '\n-\tmkdir gpg &&\n-\ttouch gpg/foo &&\n-\tgit add gpg &&\n-\ttest_tick &&\n-\tgit commit -S -m \"Adding gpg\" &&\n-\n-\tgit log -1 --format=\"%s\" > expect &&\n-\tgit filter-branch -f --msg-filter \"cat\" &&\n-\tgit log -1 --format=\"%s\" > actual &&\n-\ttest_cmp expect actual\n-'\n-\n-test_expect_success 'Tag name filtering allows slashes in tag names' '\n-\tgit tag -m tag-with-slash X/1 &&\n-\tgit cat-file tag X/1 | sed -e s,X/1,X/2, > expect &&\n-\tgit filter-branch -f --tag-name-filter \"echo X/2\" &&\n-\tgit cat-file tag X/2 > actual &&\n-\ttest_cmp expect actual\n-'\n-test_expect_success 'setup --prune-empty comparisons' '\n-\tgit checkout --orphan master-no-a &&\n-\tgit rm -rf . &&\n-\tunset test_tick &&\n-\ttest_tick &&\n-\tGIT_COMMITTER_DATE=\"@0 +0000\" GIT_AUTHOR_DATE=\"@0 +0000\" &&\n-\ttest_commit --notick B B.t B Bx &&\n-\tgit checkout -b branch-no-a Bx &&\n-\ttest_commit D D.t D Dx &&\n-\tmkdir dir &&\n-\ttest_commit dir/D dir/D.t dir/D dir/Dx &&\n-\ttest_commit E E.t E Ex &&\n-\tgit checkout master-no-a &&\n-\ttest_commit C C.t C Cx &&\n-\tgit checkout branch-no-a &&\n-\tgit merge Cx -m \"Merge tag '\\''C'\\'' into branch\" &&\n-\tgit tag Fx &&\n-\ttest_commit G G.t G Gx &&\n-\ttest_commit H H.t H Hx &&\n-\tgit checkout branch\n-'\n-\n-test_expect_success 'Prune empty commits' '\n-\tgit rev-list HEAD > expect &&\n-\ttest_commit to_remove &&\n-\tgit filter-branch -f --index-filter \"git update-index --remove to_remove.t\" --prune-empty HEAD &&\n-\tgit rev-list HEAD > actual &&\n-\ttest_cmp expect actual\n-'\n-\n-test_expect_success 'prune empty collapsed merges' '\n-\ttest_config merge.ff false &&\n-\tgit rev-list HEAD >expect &&\n-\ttest_commit to_remove_2 &&\n-\tgit reset --hard HEAD^ &&\n-\ttest_merge non-ff to_remove_2 &&\n-\tgit filter-branch -f --index-filter \"git update-index --remove to_remove_2.t\" --prune-empty HEAD &&\n-\tgit rev-list HEAD >actual &&\n-\ttest_cmp expect actual\n-'\n-\n-test_expect_success 'prune empty works even without index/tree filters' '\n-\tgit rev-list HEAD >expect &&\n-\tgit commit --allow-empty -m empty &&\n-\tgit filter-branch -f --prune-empty HEAD &&\n-\tgit rev-list HEAD >actual &&\n-\ttest_cmp expect actual\n-'\n-\n-test_expect_success '--prune-empty is able to prune root commit' '\n-\tgit rev-list branch-no-a >expect &&\n-\tgit branch testing H &&\n-\tgit filter-branch -f --prune-empty --index-filter \"git update-index --remove A.t\" testing &&\n-\tgit rev-list testing >actual &&\n-\tgit branch -D testing &&\n-\ttest_cmp expect actual\n-'\n-\n-test_expect_success '--prune-empty is able to prune entire branch' '\n-\tgit branch prune-entire B &&\n-\tgit filter-branch -f --prune-empty --index-filter \"git update-index --remove A.t B.t\" prune-entire &&\n-\ttest_path_is_missing .git/refs/heads/prune-entire &&\n-\ttest_must_fail git reflog exists refs/heads/prune-entire\n-'\n-\n-test_expect_success '--remap-to-ancestor with filename filters' '\n-\tgit checkout master &&\n-\tgit reset --hard A &&\n-\ttest_commit add-foo foo 1 &&\n-\tgit branch moved-foo &&\n-\ttest_commit add-bar bar a &&\n-\tgit branch invariant &&\n-\torig_invariant=$(git rev-parse invariant) &&\n-\tgit branch moved-bar &&\n-\ttest_commit change-foo foo 2 &&\n-\tgit filter-branch -f --remap-to-ancestor \\\n-\t\tmoved-foo moved-bar A..master \\\n-\t\t-- -- foo &&\n-\ttest $(git rev-parse moved-foo) = $(git rev-parse moved-bar) &&\n-\ttest $(git rev-parse moved-foo) = $(git rev-parse master^) &&\n-\ttest $orig_invariant = $(git rev-parse invariant)\n-'\n-\n-test_expect_success 'automatic remapping to ancestor with filename filters' '\n-\tgit checkout master &&\n-\tgit reset --hard A &&\n-\ttest_commit add-foo2 foo 1 &&\n-\tgit branch moved-foo2 &&\n-\ttest_commit add-bar2 bar a &&\n-\tgit branch invariant2 &&\n-\torig_invariant=$(git rev-parse invariant2) &&\n-\tgit branch moved-bar2 &&\n-\ttest_commit change-foo2 foo 2 &&\n-\tgit filter-branch -f \\\n-\t\tmoved-foo2 moved-bar2 A..master \\\n-\t\t-- -- foo &&\n-\ttest $(git rev-parse moved-foo2) = $(git rev-parse moved-bar2) &&\n-\ttest $(git rev-parse moved-foo2) = $(git rev-parse master^) &&\n-\ttest $orig_invariant = $(git rev-parse invariant2)\n-'\n-\n-test_expect_success 'setup submodule' '\n-\trm -fr ?* .git &&\n-\tgit init &&\n-\ttest_commit file &&\n-\tmkdir submod &&\n-\tsubmodurl=\"$PWD/submod\" &&\n-\t( cd submod &&\n-\t git init &&\n-\t test_commit file-in-submod ) &&\n-\tgit submodule add \"$submodurl\" &&\n-\tgit commit -m \"added submodule\" &&\n-\ttest_commit add-file &&\n-\t( cd submod && test_commit add-in-submodule ) &&\n-\tgit add submod &&\n-\tgit commit -m \"changed submodule\" &&\n-\tgit branch original HEAD\n-'\n-\n-orig_head=$(git show-ref --hash --head HEAD)\n-\n-test_expect_success 'rewrite submodule with another content' '\n-\tgit filter-branch --tree-filter \"test -d submod && {\n-\t\t\t\t\t rm -rf submod &&\n-\t\t\t\t\t git rm -rf --quiet submod &&\n-\t\t\t\t\t mkdir submod &&\n-\t\t\t\t\t : > submod/file\n-\t\t\t\t\t } || :\" HEAD &&\n-\ttest $orig_head != $(git show-ref --hash --head HEAD)\n-'\n-\n-test_expect_success 'replace submodule revision' '\n-\tgit reset --hard original &&\n-\tgit filter-branch -f --tree-filter \\\n-\t \"if git ls-files --error-unmatch -- submod > /dev/null 2>&1\n-\t then git update-index --cacheinfo 160000 0123456789012345678901234567890123456789 submod\n-\t fi\" HEAD &&\n-\ttest $orig_head != $(git show-ref --hash --head HEAD)\n-'\n-\n-test_expect_success 'filter commit message without trailing newline' '\n-\tgit reset --hard original &&\n-\tcommit=$(printf \"no newline\" | git commit-tree HEAD^{tree}) &&\n-\tgit update-ref refs/heads/no-newline $commit &&\n-\tgit filter-branch -f refs/heads/no-newline &&\n-\techo $commit >expect &&\n-\tgit rev-parse refs/heads/no-newline >actual &&\n-\ttest_cmp expect actual\n-'\n-\n-test_expect_success 'tree-filter deals with object name vs pathname ambiguity' '\n-\ttest_when_finished \"git reset --hard original\" &&\n-\tambiguous=$(git rev-list -1 HEAD) &&\n-\tgit filter-branch --tree-filter \"mv file.t $ambiguous\" HEAD^.. &&\n-\tgit show HEAD:$ambiguous\n-'\n-\n-test_expect_success 'rewrite repository including refs that point at non-commit object' '\n-\ttest_when_finished \"git reset --hard original\" &&\n-\ttree=$(git rev-parse HEAD^{tree}) &&\n-\ttest_when_finished \"git replace -d $tree\" &&\n-\techo A >new &&\n-\tgit add new &&\n-\tnew_tree=$(git write-tree) &&\n-\tgit replace $tree $new_tree &&\n-\tgit tag -a -m \"tag to a tree\" treetag $new_tree &&\n-\tgit reset --hard HEAD &&\n-\tgit filter-branch -f -- --all >filter-output 2>&1 &&\n-\t! fgrep fatal filter-output\n-'\n-\n-test_done\ndiff --git a/t/t7009-filter-branch-null-sha1.sh b/t/t7009-filter-branch-null-sha1.sh\ndeleted file mode 100755\nindex 9ba9f24ad2..0000000000\n--- a/t/t7009-filter-branch-null-sha1.sh\n+++ /dev/null\n@@ -1,55 +0,0 @@\n-#!/bin/sh\n-\n-test_description='filter-branch removal of trees with null sha1'\n-. ./test-lib.sh\n-\n-test_expect_success 'setup: base commits' '\n-\ttest_commit one &&\n-\ttest_commit two &&\n-\ttest_commit three\n-'\n-\n-test_expect_success 'setup: a commit with a bogus null sha1 in the tree' '\n-\t{\n-\t\tgit ls-tree HEAD &&\n-\t\tprintf \"160000 commit $ZERO_OID\\\\tbroken\\\\n\"\n-\t} >broken-tree &&\n-\techo \"add broken entry\" >msg &&\n-\n-\ttree=$(git mktree expect &&\n-\tgit filter-branch \\\n-\t\t--force --prune-empty \\\n-\t\t--index-filter \"git rm --cached --ignore-unmatch broken\" &&\n-\tgit log -1 --format=%s >actual &&\n-\ttest_cmp expect actual\n-'\n-\n-test_done\ndiff --git a/t/t9902-completion.sh b/t/t9902-completion.sh\nindex 75512c3403..4e7f669c76 100755\n--- a/t/t9902-completion.sh\n+++ b/t/t9902-completion.sh\n@@ -28,10 +28,10 @@ complete ()\n #\n # (2) A test makes sure that common subcommands are included in the\n # completion for \"git \", and a plumbing is excluded. \"add\",\n-# \"filter-branch\" and \"ls-files\" are listed for this.\n+# \"rebase\" and \"ls-files\" are listed for this.\n \n-GIT_TESTING_ALL_COMMAND_LIST='add checkout check-attr filter-branch ls-files'\n-GIT_TESTING_PORCELAIN_COMMAND_LIST='add checkout filter-branch'\n+GIT_TESTING_ALL_COMMAND_LIST='add checkout check-attr rebase ls-files'\n+GIT_TESTING_PORCELAIN_COMMAND_LIST='add checkout rebase'\n \n . \"$GIT_BUILD_DIR/contrib/completion/git-completion.bash\"\n \n@@ -1392,12 +1392,12 @@ test_expect_success 'basic' '\n \t# built-in\n \tgrep -q \"^add \\$\" out &&\n \t# script\n-\tgrep -q \"^filter-branch \\$\" out &&\n+\tgrep -q \"^rebase \\$\" out &&\n \t# plumbing\n \t! grep -q \"^ls-files \\$\" out &&\n \n-\trun_completion \"git f\" &&\n-\t! grep -q -v \"^f\" out\n+\trun_completion \"git r\" &&\n+\t! grep -q -v \"^r\" out\n '\n \n test_expect_success 'double dash \"git\" itself' '\n-- \n2.23.0.5.g775ebaa2a0\n\n"},{"id":"381275","messageId":"39b07712-f6f0-44bd-c434-a871db4a4899@gmail.com","threadId":"51710","inReplyTo":"20190826235226.15386-2-newren@gmail.com","subject":"Re: [RFC PATCH 1/5] t6006: simplify and optimize empty message test","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-08-27T01:23:10Z","receivedAt":"2019-08-27T01:23:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/26/2019 7:52 PM, Elijah Newren wrote:\n> Test t6006.71 (\"oneline with empty message\") was creating two commits\n> with simple commit messages, and then running filter-branch to rewrite\n> the commit messages to be empty. This test was written this way because\n> the --allow-empty-message option to git commit did not exist at the\n> time. Simplify this test and avoid the need to invoke filter-branch by\n> just using --allow-empty-message when creating the commit.\n> \n> Despite only being one piece of the 71st test and there being 73 tests\n> overall, this small change to just this one test speeds up the overall\n> execution time of t6006 (as measured by the best of 3 runs of `time\n> ./t6006-rev-list-format.sh`) by about 11% on Linux and by 13% on\n> Mac.\n\nWow! A good cleanup to include, regardless of other concerns.\n\n> \n> Signed-off-by: Elijah Newren \n> ---\n> t/t6006-rev-list-format.sh | 5 ++---\n> 1 file changed, 2 insertions(+), 3 deletions(-)\n> \n> diff --git a/t/t6006-rev-list-format.sh b/t/t6006-rev-list-format.sh\n> index da113d975b..d30e41c9f7 100755\n> --- a/t/t6006-rev-list-format.sh\n> +++ b/t/t6006-rev-list-format.sh\n> @@ -501,9 +501,8 @@ test_expect_success 'reflog identity' '\n> '\n> \n> test_expect_success 'oneline with empty message' '\n> -\tgit commit -m \"dummy\" --allow-empty &&\n> -\tgit commit -m \"dummy\" --allow-empty &&\n> -\tgit filter-branch --msg-filter \"sed -e s/dummy//\" HEAD^^.. &&\n> +\tgit commit --allow-empty --allow-empty-message &&\n> +\tgit commit --allow-empty --allow-empty-message &&\n> \tgit rev-list --oneline HEAD >test.txt &&\n> \ttest_line_count = 5 test.txt &&\n> \tgit rev-list --oneline --graph HEAD >testg.txt &&\n\nLGTM.\n\n-Stolee\n"},{"id":"381276","messageId":"ad7446ac-34cc-b566-302b-053bf5dafc63@gmail.com","threadId":"51710","inReplyTo":"20190826235226.15386-3-newren@gmail.com","subject":"Re: [RFC PATCH 2/5] t3427: accelerate this test by using fast-export and fast-import","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-08-27T01:25:59Z","receivedAt":"2019-08-27T01:26:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/26/2019 7:52 PM, Elijah Newren wrote:\n> fast-export and fast-import can easily handle the simple rewrite that\n> was being done by filter-branch, and should be significantly faster on\n> systems with a slow fork. Timings from before and after on two laptops\n> that I have access to (measured via `time ./t3427-rebase-subtree.sh`,\n> i.e. including everything in this test -- not just the filter-branch or\n> fast-export/fast-import pair):\n> \n> Linux: 4.305s -> 3.684s (~17% speedup)\n> Mac: 10.128s -> 7.038s (~30% speedup)\n\nAgain, impressive speedup!\n\n> Signed-off-by: Elijah Newren \n> ---\n> t/t3427-rebase-subtree.sh | 32 ++++++++++++++++++++++++--------\n> 1 file changed, 24 insertions(+), 8 deletions(-)\n> \n> diff --git a/t/t3427-rebase-subtree.sh b/t/t3427-rebase-subtree.sh\n> index d8640522a0..d05fcce5dc 100755\n> --- a/t/t3427-rebase-subtree.sh\n> +++ b/t/t3427-rebase-subtree.sh\n> @@ -42,7 +42,9 @@ test_expect_failure REBASE_P \\\n> \t'Rebase -Xsubtree --preserve-merges --onto commit 4' '\n> \treset_rebase &&\n> \tgit checkout -b rebase-preserve-merges-4 master &&\n> -\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n> +\tgit fast-export --no-data HEAD -- files_subtree/ \\\n> +\t\t| sed -e \"s%\\([0-9a-f]\\{40\\} \\)files_subtree/%\\1%\" \\\n> +\t\t| git fast-import --force --quiet &&\n> \tgit commit -m \"Empty commit\" --allow-empty &&\n\nThere's a lot of repetition in these changes. Is this a good time\nto introduce a helper method? This trio of commands happens to be\ndifficult to parse, so I'd rather do it just once.\n\n> \tgit rebase -Xsubtree=files_subtree --preserve-merges --onto files-master master &&\n> \tverbose test \"$(commit_message HEAD~)\" = \"files_subtree/master4\"\n> @@ -53,7 +55,9 @@ test_expect_failure REBASE_P \\\n> \t'Rebase -Xsubtree --preserve-merges --onto commit 5' '\n> \treset_rebase &&\n> \tgit checkout -b rebase-preserve-merges-5 master &&\n> -\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n> +\tgit fast-export --no-data HEAD -- files_subtree/ \\\n> +\t\t| sed -e \"s%\\([0-9a-f]\\{40\\} \\)files_subtree/%\\1%\" \\\n> +\t\t| git fast-import --force --quiet &&\n> \tgit commit -m \"Empty commit\" --allow-empty &&\n> \tgit rebase -Xsubtree=files_subtree --preserve-merges --onto files-master master &&\n> \tverbose test \"$(commit_message HEAD)\" = \"files_subtree/master5\"\n> @@ -64,7 +68,9 @@ test_expect_failure REBASE_P \\\n> \t'Rebase -Xsubtree --keep-empty --preserve-merges --onto commit 4' '\n> \treset_rebase &&\n> \tgit checkout -b rebase-keep-empty-4 master &&\n> -\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n> +\tgit fast-export --no-data HEAD -- files_subtree/ \\\n> +\t\t| sed -e \"s%\\([0-9a-f]\\{40\\} \\)files_subtree/%\\1%\" \\\n> +\t\t| git fast-import --force --quiet &&\n> \tgit commit -m \"Empty commit\" --allow-empty &&\n> \tgit rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master &&\n> \tverbose test \"$(commit_message HEAD~2)\" = \"files_subtree/master4\"\n> @@ -75,7 +81,9 @@ test_expect_failure REBASE_P \\\n> \t'Rebase -Xsubtree --keep-empty --preserve-merges --onto commit 5' '\n> \treset_rebase &&\n> \tgit checkout -b rebase-keep-empty-5 master &&\n> -\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n> +\tgit fast-export --no-data HEAD -- files_subtree/ \\\n> +\t\t| sed -e \"s%\\([0-9a-f]\\{40\\} \\)files_subtree/%\\1%\" \\\n> +\t\t| git fast-import --force --quiet &&\n> \tgit commit -m \"Empty commit\" --allow-empty &&\n> \tgit rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master &&\n> \tverbose test \"$(commit_message HEAD~)\" = \"files_subtree/master5\"\n> @@ -86,7 +94,9 @@ test_expect_failure REBASE_P \\\n> \t'Rebase -Xsubtree --keep-empty --preserve-merges --onto empty commit' '\n> \treset_rebase &&\n> \tgit checkout -b rebase-keep-empty-empty master &&\n> -\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n> +\tgit fast-export --no-data HEAD -- files_subtree/ \\\n> +\t\t| sed -e \"s%\\([0-9a-f]\\{40\\} \\)files_subtree/%\\1%\" \\\n> +\t\t| git fast-import --force --quiet &&\n> \tgit commit -m \"Empty commit\" --allow-empty &&\n> \tgit rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master &&\n> \tverbose test \"$(commit_message HEAD)\" = \"Empty commit\"\n> @@ -96,7 +106,9 @@ test_expect_failure REBASE_P \\\n> test_expect_failure 'Rebase -Xsubtree --onto commit 4' '\n> \treset_rebase &&\n> \tgit checkout -b rebase-onto-4 master &&\n> -\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n> +\tgit fast-export --no-data HEAD -- files_subtree/ \\\n> +\t\t| sed -e \"s%\\([0-9a-f]\\{40\\} \\)files_subtree/%\\1%\" \\\n> +\t\t| git fast-import --force --quiet &&\n> \tgit commit -m \"Empty commit\" --allow-empty &&\n> \tgit rebase -Xsubtree=files_subtree --onto files-master master &&\n> \tverbose test \"$(commit_message HEAD~2)\" = \"files_subtree/master4\"\n> @@ -106,7 +118,9 @@ test_expect_failure 'Rebase -Xsubtree --onto commit 4' '\n> test_expect_failure 'Rebase -Xsubtree --onto commit 5' '\n> \treset_rebase &&\n> \tgit checkout -b rebase-onto-5 master &&\n> -\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n> +\tgit fast-export --no-data HEAD -- files_subtree/ \\\n> +\t\t| sed -e \"s%\\([0-9a-f]\\{40\\} \\)files_subtree/%\\1%\" \\\n> +\t\t| git fast-import --force --quiet &&\n> \tgit commit -m \"Empty commit\" --allow-empty &&\n> \tgit rebase -Xsubtree=files_subtree --onto files-master master &&\n> \tverbose test \"$(commit_message HEAD~)\" = \"files_subtree/master5\"\n> @@ -115,7 +129,9 @@ test_expect_failure 'Rebase -Xsubtree --onto commit 5' '\n> test_expect_failure 'Rebase -Xsubtree --onto empty commit' '\n> \treset_rebase &&\n> \tgit checkout -b rebase-onto-empty master &&\n> -\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n> +\tgit fast-export --no-data HEAD -- files_subtree/ \\\n> +\t\t| sed -e \"s%\\([0-9a-f]\\{40\\} \\)files_subtree/%\\1%\" \\\n> +\t\t| git fast-import --force --quiet &&\n> \tgit commit -m \"Empty commit\" --allow-empty &&\n> \tgit rebase -Xsubtree=files_subtree --onto files-master master &&\n> \tverbose test \"$(commit_message HEAD)\" = \"Empty commit\"\n> \n\n"},{"id":"381277","messageId":"88ee0658-66c2-902b-276a-25ffcd08b954@gmail.com","threadId":"51710","inReplyTo":"20190826235226.15386-4-newren@gmail.com","subject":"Re: [RFC PATCH 3/5] git-sh-i18n: work with external scripts","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-08-27T01:28:00Z","receivedAt":"2019-08-27T01:28:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/26/2019 7:52 PM, Elijah Newren wrote:\n> Scripts external to git could source $(git --exec-path)/git-sh-setup (as\n> we document in Documentation/git-sh-setup.txt). This will in turn\n> source git-sh-i18n, which will setup some handy internationalization\n> infrastructure. However, git-sh-i18n hardcodes the TEXTDOMAIN, meaning\n> that anyone using this infrastructure will only get translations that\n> are shipped with git. Allow the external scripts to specify their own\n> translation domain but otherwise use our infrastructure for accessing\n> translations.\n> \n> My original plan was to have git-filter-branch be the first testcase\n> using this feature, with a goal of minimizing the number of changes that\n> needed to be made to it when I moved it out of git.git. However, I\n> realized after creating this patch that no strings in git-filter-branch\n> are translated. However, the generalization could be useful if we move\n> other tools from git.git to an external location.\n\nHm. Maybe extract it from this series for the YAGNI principle?\n\nA noble goal, but in my opinion I'd prefer to see this be exercised\nby something immediately.\n\nThanks,\n-Stolee\n"},{"id":"381279","messageId":"e7df2ce3-f772-54b5-4e81-70510a897352@gmail.com","threadId":"51710","inReplyTo":"20190826235226.15386-5-newren@gmail.com","subject":"Re: [RFC PATCH 4/5] Recommend git-filter-repo instead of git-filter-branch in documentation","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-08-27T01:32:58Z","receivedAt":"2019-08-27T01:33:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/26/2019 7:52 PM, Elijah Newren wrote:\n> filter-branch suffers from a huge number of pitfalls that can result in\n> incorrectly rewritten history, and many of the problems can easily go\n> undetected until the new repository is in use. This can result in\n> problems ranging from an even messier history than what led folks to\n> filter-branch in the first place, to data loss or corruption. These\n> issues cannot be backward compatibly fixed, so add a warning to the\n> filter-branch manpage about this and recommand that another tool (such\n> as filter-repo) be used instead.\n> \n> Also, update other manpages that referenced filter-branch. Several of\n> these needed updates even if we could continue recommending\n> filter-branch, either due to implying that something was unique to\n> filter-branch when it applied more generally to all history rewriting\n> tools (e.g. BFG, reposurgeon, fast-import, filter-repo), or because\n> something about filter-branch was used as an example despite other more\n> commonly known examples now existing. Reword these sections to fix\n> these issues and to avoid recommending filter-branch.\n> \n> Finally, remove the section explaining BFG Repo Cleaner as an\n> alternative to filter-branch. I feel somewhat bad about this,\n> especially since I feel like I learned so much from BFG that I put to\n> good use in filter-repo (which is much more than I can say for\n> filter-branch), but keeping that section presented a few problems:\n> * In order to recommend that people quit using filter-branch, we need\n> to provide them a recomendation for something else to use that\n> can handle all the same types of rewrites. To my knowledge,\n> filter-repo is the only such tool. So it needs to be mentioned.\n> * I don't want to give conflicting recommendations to users\n> * If we recommend two tools, we shouldn't expect users to learn both\n> and pick which one to use; we should explain which problems one\n> can solve that the other can't or when one is much faster than\n> the other.\n> * BFG and filter-repo have similar performance\n> * All filtering types that BFG can do, filter-repo can also do. In\n> fact, filter-repo comes with a reimplementation of BFG named\n> bfg-ish which provides the same user-interface as BFG but with\n> several bugfixes and new features that are hard to implement in\n> BFG due to its technical underpinnings.\n> While I could still mention both tools, it seems like I would need to\n> provide some kind of comparison and I would ultimately just say that\n> filter-repo can do everything BFG can, so ultimately it seems that it\n> is just better to remove that section altogether.\n> \n> Signed-off-by: Elijah Newren \n> ---\n> Documentation/git-fast-export.txt | 6 ++---\n> Documentation/git-filter-branch.txt | 42 ++++++++---------------------\n> Documentation/git-gc.txt | 17 ++++++------\n> Documentation/git-rebase.txt | 2 +-\n> Documentation/git-replace.txt | 10 +++----\n> Documentation/git-svn.txt | 4 +--\n> Documentation/githooks.txt | 7 ++---\n> contrib/svn-fe/svn-fe.txt | 4 +--\n> 8 files changed, 36 insertions(+), 56 deletions(-)\n> \n> diff --git a/Documentation/git-fast-export.txt b/Documentation/git-fast-export.txt\n> index cc940eb9ad..784e934009 100644\n> --- a/Documentation/git-fast-export.txt\n> +++ b/Documentation/git-fast-export.txt\n> @@ -17,9 +17,9 @@ This program dumps the given revisions in a form suitable to be piped\n> into 'git fast-import'.\n> \n> You can use it as a human-readable bundle replacement (see\n> -linkgit:git-bundle[1]), or as a kind of an interactive\n> -'git filter-branch'.\n> -\n> +linkgit:git-bundle[1]), or as a format that can be edited before being\n> +fed to 'git fast-import' in order to do history rewrites (an ability\n> +relied on by tools like 'git filter-repo').\n> \n> OPTIONS\n> -------\n> diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt\n> index 6b53dd7e06..8c586eed55 100644\n> --- a/Documentation/git-filter-branch.txt\n> +++ b/Documentation/git-filter-branch.txt\n> @@ -16,6 +16,17 @@ SYNOPSIS\n> \t[--original ] [-d ] [-f | --force]\n> \t[--state-branch ] [--] [...]\n> \n> +WARNING\n> +-------\n> +'git filter-branch' has a litany of gotchas that can and will cause\n> +history to be rewritten incorrectly (in addition to abysmal\n> +performance). These issues cannot be backward compatibly fixed and as\n> +such, its use is not recommended. Please use an alternative history\n> +filtering tool such as 'git filter-repo'. If you still need to use\n> +'git filter-branch', please carefully read the \"Safety\" section of\n> +https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/\n\nIs it possible to present this URL as a hyperlink with a succinct\ndescription? Maybe 'carefully read [the \"Safety\" section of this\nmessage on the Git mailing list](url).' (I'm using Markdown notation\nhere as I don't know the equivalent for our docs.)\n\n> +and avoid as many of the pitfalls listed there as reasonably possible.\n> +\n> DESCRIPTION\n> -----------\n> Lets you rewrite Git revision history by rewriting the branches mentioned\n> @@ -445,37 +456,6 @@ warned.\n> (or if your git-gc is not new enough to support arguments to\n> `--prune`, use `git repack -ad; git prune` instead).\n> \n> -NOTES\n> ------\n> -\n> -git-filter-branch allows you to make complex shell-scripted rewrites\n> -of your Git history, but you probably don't need this flexibility if\n> -you're simply _removing unwanted data_ like large files or passwords.\n> -For those operations you may want to consider\n> -http://rtyley.github.io/bfg-repo-cleaner/[The BFG Repo-Cleaner],\n> -a JVM-based alternative to git-filter-branch, typically at least\n> -10-50x faster for those use-cases, and with quite different\n> -characteristics:\n> -\n> -* Any particular version of a file is cleaned exactly _once_. The BFG,\n> - unlike git-filter-branch, does not give you the opportunity to\n> - handle a file differently based on where or when it was committed\n> - within your history. This constraint gives the core performance\n> - benefit of The BFG, and is well-suited to the task of cleansing bad\n> - data - you don't care _where_ the bad data is, you just want it\n> - _gone_.\n> -\n> -* By default The BFG takes full advantage of multi-core machines,\n> - cleansing commit file-trees in parallel. git-filter-branch cleans\n> - commits sequentially (i.e. in a single-threaded manner), though it\n> - _is_ possible to write filters that include their own parallelism,\n> - in the scripts executed against each commit.\n> -\n> -* The http://rtyley.github.io/bfg-repo-cleaner/#examples[command options]\n> - are much more restrictive than git-filter branch, and dedicated just\n> - to the tasks of removing unwanted data- e.g:\n> - `--strip-blobs-bigger-than 1M`.\n> -\n> GIT\n> ---\n> Part of the linkgit:git[1] suite\n> diff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\n> index 247f765604..0c114ad1ca 100644\n> --- a/Documentation/git-gc.txt\n> +++ b/Documentation/git-gc.txt\n> @@ -115,15 +115,14 @@ NOTES\n> -----\n> \n> 'git gc' tries very hard not to delete objects that are referenced\n> -anywhere in your repository. In\n> -particular, it will keep not only objects referenced by your current set\n> -of branches and tags, but also objects referenced by the index,\n> -remote-tracking branches, refs saved by 'git filter-branch' in\n> -refs/original/, reflogs (which may reference commits in branches\n> -that were later amended or rewound), and anything else in the refs/* namespace.\n> -If you are expecting some objects to be deleted and they aren't, check\n> -all of those locations and decide whether it makes sense in your case to\n> -remove those references.\n> +anywhere in your repository. In particular, it will keep not only\n> +objects referenced by your current set of branches and tags, but also\n> +objects referenced by the index, remote-tracking branches, notes saved\n> +by 'git notes' under refs/notes/, reflogs (which may reference commits\n> +in branches that were later amended or rewound), and anything else in\n> +the refs/* namespace. If you are expecting some objects to be deleted\n> +and they aren't, check all of those locations and decide whether it\n> +makes sense in your case to remove those references.\n> \n> On the other hand, when 'git gc' runs concurrently with another process,\n> there is a risk of it deleting an object that the other process is using\n> diff --git a/Documentation/git-rebase.txt b/Documentation/git-rebase.txt\n> index 6156609cf7..2f201d85d4 100644\n> --- a/Documentation/git-rebase.txt\n> +++ b/Documentation/git-rebase.txt\n> @@ -832,7 +832,7 @@ Hard case: The changes are not the same.::\n> \tThis happens if the 'subsystem' rebase had conflicts, or used\n> \t`--interactive` to omit, edit, squash, or fixup commits; or\n> \tif the upstream used one of `commit --amend`, `reset`, or\n> -\t`filter-branch`.\n> +\ta full history rewriting command like `filter-repo`.\n> \n> \n> The easy case\n> diff --git a/Documentation/git-replace.txt b/Documentation/git-replace.txt\n> index 246dc9943c..35595a2cd3 100644\n> --- a/Documentation/git-replace.txt\n> +++ b/Documentation/git-replace.txt\n> @@ -123,10 +123,10 @@ The following format are available:\n> CREATING REPLACEMENT OBJECTS\n> ----------------------------\n> \n> -linkgit:git-filter-branch[1], linkgit:git-hash-object[1] and\n> -linkgit:git-rebase[1], among other git commands, can be used to create\n> -replacement objects from existing objects. The `--edit` option can\n> -also be used with 'git replace' to create a replacement object by\n> +linkgit:git-hash-object[1], linkgit:git-rebase[1], and\n> +linkgit:git-filter-repo[1], among other git commands, can be used to\n> +create replacement objects from existing objects. The `--edit` option\n> +can also be used with 'git replace' to create a replacement object by\n> editing an existing object.\n> \n> If you want to replace many blobs, trees or commits that are part of a\n> @@ -148,8 +148,8 @@ pending objects.\n> SEE ALSO\n> --------\n> linkgit:git-hash-object[1]\n> -linkgit:git-filter-branch[1]\n> linkgit:git-rebase[1]\n> +linkgit:git-filter-repo[1]\n> linkgit:git-tag[1]\n> linkgit:git-branch[1]\n> linkgit:git-commit[1]\n> diff --git a/Documentation/git-svn.txt b/Documentation/git-svn.txt\n> index 30711625fd..f2762dd5d4 100644\n> --- a/Documentation/git-svn.txt\n> +++ b/Documentation/git-svn.txt\n> @@ -769,9 +769,9 @@ option for (hopefully) obvious reasons.\n> +\n> This option is NOT recommended as it makes it difficult to track down\n> old references to SVN revision numbers in existing documentation, bug\n> -reports and archives. If you plan to eventually migrate from SVN to Git\n> +reports, and archives. If you plan to eventually migrate from SVN to Git\n> and are certain about dropping SVN history, consider\n> -linkgit:git-filter-branch[1] instead. filter-branch also allows\n> +linkgit:git-filter-repo[1] instead. filter-repo also allows\n> reformatting of metadata for ease-of-reading and rewriting authorship\n> info for non-\"svn.authorsFile\" users.\n> \n> diff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\n> index 82cd573776..997548f5ed 100644\n> --- a/Documentation/githooks.txt\n> +++ b/Documentation/githooks.txt\n> @@ -425,9 +425,10 @@ post-rewrite\n> \n> This hook is invoked by commands that rewrite commits\n> (linkgit:git-commit[1] when called with `--amend` and\n> -linkgit:git-rebase[1]; currently `git filter-branch` does 'not' call\n> -it!). Its first argument denotes the command it was invoked by:\n> -currently one of `amend` or `rebase`. Further command-dependent\n> +linkgit:git-rebase[1]; however, full-history (re)writing tools like\n> +linkgit:git-fast-import[1] or linkgit:git-filter-repo[1] typically do\n> +not call it!). Its first argument denotes the command it was invoked\n> +by: currently one of `amend` or `rebase`. Further command-dependent\n> arguments may be passed in the future.\n> \n> The hook receives a list of the rewritten commits on stdin, in the\n> diff --git a/contrib/svn-fe/svn-fe.txt b/contrib/svn-fe/svn-fe.txt\n> index a3425f4770..19333fc8df 100644\n> --- a/contrib/svn-fe/svn-fe.txt\n> +++ b/contrib/svn-fe/svn-fe.txt\n> @@ -56,7 +56,7 @@ line. This line has the form `git-svn-id: URL@REVNO UUID`.\n> \n> The resulting repository will generally require further processing\n> to put each project in its own repository and to separate the history\n> -of each branch. The 'git filter-branch --subdirectory-filter' command\n> +of each branch. The 'git filter-repo --subdirectory-filter' command\n> may be useful for this purpose.\n> \n> BUGS\n> @@ -67,5 +67,5 @@ The exit status does not reflect whether an error was detected.\n> \n> SEE ALSO\n> --------\n> -git-svn(1), svn2git(1), svk(1), git-filter-branch(1), git-fast-import(1),\n> +git-svn(1), svn2git(1), svk(1), git-filter-repo(1), git-fast-import(1),\n> https://svn.apache.org/repos/asf/subversion/trunk/notes/dump-load-format.txt\n> \n\n"},{"id":"381281","messageId":"922dbed9-3670-16b4-807c-ce68430ee591@gmail.com","threadId":"51710","inReplyTo":"20190826235226.15386-1-newren@gmail.com","subject":"Re: [RFC PATCH 0/5] Remove git-filter-branch from git.git; host it elsewhere","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-08-27T01:39:15Z","receivedAt":"2019-08-27T01:39:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/26/2019 7:52 PM, Elijah Newren wrote:\n> Following up on the suggestion to make git.git smaller and shed non-core\n> tools, here's an RFC series to do so with git-filter-branch. This\n> series first removes dependencies on git-filter-branch (of which there\n> were very few), and then deletes git-filter-branch itself in the final\n> commit.\n> \n> I'm more than happy to consider alternate places for the filter-branch\n> history (I had considered just merging it in with git-filter-repo), but\n> for now I just made it available here:\n> https://github.com/newren/git-filter-branch\n> \n> The rewrite above contains the history of the files deleted in Patch 5,\n> plus a one-time copy of relevant build files (Makefiles, test-lib.sh,\n> etc. -- I didn't want the whole history of these), and then touchups to\n> streamline the build files and make them all work in this standalone\n> repo.\n> \n> \n> Some highlevel notes on the patches:\n> \n> * Patches 1&2: are good cleanups & performance wins regardless of\n> whether the rest of the series is taken\n\nI agree! These are great. I just had a nit about extracting a helper\ninstead of copy-pasting the same three lines in multiple tests.\n\n> * Patch 3: an attempt to improve i18n situation for external scripts,\n> but discovered to not be necessary/useful for git-filter-branch\n> specifically\n\nI'm not sure this is super-important now, but could be saved for a\nlater date, when it is important.\n\n> * Patch 4:\n> * If we are good with deleting git-filter-branch now and just noting\n> it in the release notes, then patch 4 could be simplified; there's\n> no need to update git-filter-branch.txt in that case.\n> * If, however, we want to do some external messaging for an\n> additional release cycle or two before moving git-filter-branch\n> out of git.git, this patch will help us until then to at least\n> avoid recommending a tool which will likely mangle user's data in\n> unexpected ways. But it'd be really helpful if folks could review\n> and opine on the BFG stuff if so.\n\nI think this is a good step, and should be taken even if we never\nplan to take Patch 5.\n\n> * Patch 5: actually deletes git-filter-branch, its tests, and\n> documentation.\n\nThis is the one where others need to chime in with opinions. I\nthink this one can only be taken if we have a concrete plan about\nhow to support the tool _somehow_, even if it is \"go download the\nscript from this place; it may have broken since we last tested it.\"\n\nYes, we want to strongly recommend that people use newer, better\ntools. That's not always something users can accept. Having the\ntool live somewhere that is accessible can appease some users for\na while, and it can decay and die a slow death there.\n\nThanks,\n-Stolee\n"},{"id":"381318","messageId":"CABPp-BERNsoxdexwX1USuH4rcNVz6MOeao9mrLOrJbV8Azs9-w@mail.gmail.com","threadId":"51710","inReplyTo":"922dbed9-3670-16b4-807c-ce68430ee591@gmail.com","subject":"Re: [RFC PATCH 0/5] Remove git-filter-branch from git.git; host it elsewhere","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-27T06:17:55Z","receivedAt":"2019-08-27T06:18:09Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Aug 26, 2019 at 6:39 PM Derrick Stolee wrote:\n>\n> On 8/26/2019 7:52 PM, Elijah Newren wrote:\n> > Following up on the suggestion to make git.git smaller and shed non-core\n> > tools, here's an RFC series to do so with git-filter-branch. This\n> > series first removes dependencies on git-filter-branch (of which there\n> > were very few), and then deletes git-filter-branch itself in the final\n> > commit.\n> >\n> > I'm more than happy to consider alternate places for the filter-branch\n> > history (I had considered just merging it in with git-filter-repo), but\n> > for now I just made it available here:\n> > https://github.com/newren/git-filter-branch\n> >\n> > The rewrite above contains the history of the files deleted in Patch 5,\n> > plus a one-time copy of relevant build files (Makefiles, test-lib.sh,\n> > etc. -- I didn't want the whole history of these), and then touchups to\n> > streamline the build files and make them all work in this standalone\n> > repo.\n> >\n> >\n> > Some highlevel notes on the patches:\n> >\n> > * Patches 1&2: are good cleanups & performance wins regardless of\n> > whether the rest of the series is taken\n>\n> I agree! These are great. I just had a nit about extracting a helper\n> instead of copy-pasting the same three lines in multiple tests.\n>\n> > * Patch 3: an attempt to improve i18n situation for external scripts,\n> > but discovered to not be necessary/useful for git-filter-branch\n> > specifically\n>\n> I'm not sure this is super-important now, but could be saved for a\n> later date, when it is important.\n>\n> > * Patch 4:\n> > * If we are good with deleting git-filter-branch now and just noting\n> > it in the release notes, then patch 4 could be simplified; there's\n> > no need to update git-filter-branch.txt in that case.\n> > * If, however, we want to do some external messaging for an\n> > additional release cycle or two before moving git-filter-branch\n> > out of git.git, this patch will help us until then to at least\n> > avoid recommending a tool which will likely mangle user's data in\n> > unexpected ways. But it'd be really helpful if folks could review\n> > and opine on the BFG stuff if so.\n>\n> I think this is a good step, and should be taken even if we never\n> plan to take Patch 5.\n>\n> > * Patch 5: actually deletes git-filter-branch, its tests, and\n> > documentation.\n>\n> This is the one where others need to chime in with opinions. I\n> think this one can only be taken if we have a concrete plan about\n> how to support the tool _somehow_, even if it is \"go download the\n> script from this place; it may have broken since we last tested it.\"\n>\n> Yes, we want to strongly recommend that people use newer, better\n> tools. That's not always something users can accept. Having the\n> tool live somewhere that is accessible can appease some users for\n> a while, and it can decay and die a slow death there.\n\nPerhaps I should add some more words about the separate repo I\ncreated; even though it wasn't one of the five patches in this series\nit actually represented the lionshare of the work before I submitted\nthis. Anyway, it has a Makefile which supports the normal 'test',\n'doc', 'clean', 'install' (and variants), and 'dist' (and variants).\nThe 'test' target will run all the filter-branch tests taken from the\ngit.git testsuite (i.e. t7003 and t7009) without requiring a version\nof git built inside that separate repo, 'doc' will build both html and\nmanpages, etc. It doesn't look at any config.mak* files and has\nstripped out lots of stuff from the main repo, but it's relatively\nminimal and self-contained beyond an assumption that a normal copy of\ngit has been installed somehow already. Given how infrequently\nfilter-branch has needed fixes in the past, and the fact that it is\npretty good about relying on plumbing rather than porcelain, I suspect\nit might actually be a pretty light maintenance load to keep it\nrunning for a good long time.\n"},{"id":"381319","messageId":"CABPp-BEBVNERNBApmWTaR03qS2cHPwQ=yRe6aSesOFUsX-2qFA@mail.gmail.com","threadId":"51710","inReplyTo":"e7df2ce3-f772-54b5-4e81-70510a897352@gmail.com","subject":"Re: [RFC PATCH 4/5] Recommend git-filter-repo instead of git-filter-branch in documentation","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-27T06:23:46Z","receivedAt":"2019-08-27T06:23:59Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Aug 26, 2019 at 6:33 PM Derrick Stolee wrote:\n>\n> On 8/26/2019 7:52 PM, Elijah Newren wrote:\n\n> > +WARNING\n> > +-------\n> > +'git filter-branch' has a litany of gotchas that can and will cause\n> > +history to be rewritten incorrectly (in addition to abysmal\n> > +performance). These issues cannot be backward compatibly fixed and as\n> > +such, its use is not recommended. Please use an alternative history\n> > +filtering tool such as 'git filter-repo'. If you still need to use\n> > +'git filter-branch', please carefully read the \"Safety\" section of\n> > +https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/\n>\n> Is it possible to present this URL as a hyperlink with a succinct\n> description? Maybe 'carefully read [the \"Safety\" section of this\n> message on the Git mailing list](url).' (I'm using Markdown notation\n> here as I don't know the equivalent for our docs.)\n\nLooks like the syntax is\n URL[description]\ne.g.\n https://public-inbox.org/git/CABPp-BEDOH-etc-etc-etc@mail.gmail.com[the\n\"Safety\" section of yadda yadda]\n\nI'll fix that up.\n"},{"id":"381320","messageId":"20190827070324.7l2cpc3dlnithbm6@whir","threadId":"51710","inReplyTo":"20190826235226.15386-1-newren@gmail.com","subject":"Re: [RFC PATCH 0/5] Remove git-filter-branch from git.git; host it elsewhere","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2019-08-27T07:03:24Z","receivedAt":"2019-08-27T07:03:26Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Elijah Newren wrote:\n> Some highlevel notes on the patches:\n> \n> * Patches 1&2: are good cleanups & performance wins regardless of\n> whether the rest of the series is taken\n\nAgreed. Though weren't we moving away from pipes in tests\nbecause failures could go unnoticed? (I haven't been paying too\nmuch attention, though)\n\n> * Patch 5: actually deletes git-filter-branch, its tests, and\n> documentation.\n\nGiven how long we've had git-filter-branch, I suggest we keep it\naround but add a warning at runtime notifying users of it's\nimpending removal. Such warning should remain for >=10 years or\nwhatever an distro support cycle is, nowadays.\n\nAnd there should probably be a way to disable the warning via\ngit-config if it's too annoying.\n\nAFAIK, filter-branch is not causing support headaches for any\ngit developers today. With so many commands in git, it's\nunlikely newbies will ever get around to discover it :)\nSo I think think we should be in any rush to remove it.\n\nBut I agree that filter-branch isn't useful and certainly\nshouldn't be encouraged/promoted.\n\nYet there's probably still users which ARE happy with it, that\nwill never hit the edge cases and problems it poses; and will\nnever read release notes. And said users are probably getting\ngit from a slow-moving distro, so it'd be a disservice to them\nif they lost a tool they depend on without any warning.\n"},{"id":"381324","messageId":"87v9ujc827.fsf@osv.gnss.ru","threadId":"51710","inReplyTo":"20190827070324.7l2cpc3dlnithbm6@whir","subject":"Re: [RFC PATCH 0/5] Remove git-filter-branch from git.git; host it elsewhere","fromName":"Sergey Organov","fromEmail":"sorganov@gmail.com","sentAt":"2019-08-27T08:43:28Z","receivedAt":"2019-08-27T08:43:34Z","isPatch":true,"sender":{"key":"sorganov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8501568?v=4"},"body":"Eric Wong writes:\n\n\n[...]\n\n> AFAIK, filter-branch is not causing support headaches for any\n> git developers today. With so many commands in git, it's\n> unlikely newbies will ever get around to discover it :)\n> So I think think we should be in any rush to remove it.\n\nNah, discovering it is simple. Just Google for \"git change author\". That\neventually leads to a script that uses \"git filter-branch --env-filter\"\nto get the job done, and I'm afraid it is spread all over the world.\n\nSee, e.g.:\n\nhttps://help.github.com/en/articles/changing-author-info\n\n> But I agree that filter-branch isn't useful and certainly\n> shouldn't be encouraged/promoted.\n\nWell, is there more suitable way to change author for a (large) set of\ncommits then?\n\n> Yet there's probably still users which ARE happy with it, that\n> will never hit the edge cases and problems it poses; and will\n> never read release notes. And said users are probably getting\n> git from a slow-moving distro, so it'd be a disservice to them\n> if they lost a tool they depend on without any warning.\n\nPersonally, I'm far from happy with it, but I have no clue how to\nsubstitute it in the job above. Anybody?\n\n-- Sergey\n"},{"id":"381400","messageId":"CABPp-BGfoRX-ob986kd-vvBo3meg-MpJ8Jo85G_2GgARY=cxxg@mail.gmail.com","threadId":"51710","inReplyTo":"87v9ujc827.fsf@osv.gnss.ru","subject":"Re: [RFC PATCH 0/5] Remove git-filter-branch from git.git; host it elsewhere","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-27T22:18:19Z","receivedAt":"2019-08-27T22:18:33Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Aug 27, 2019 at 1:43 AM Sergey Organov wrote:\n>\n> Eric Wong writes:\n>\n>\n> [...]\n>\n> > AFAIK, filter-branch is not causing support headaches for any\n> > git developers today. With so many commands in git, it's\n> > unlikely newbies will ever get around to discover it :)\n> > So I think think we should be in any rush to remove it.\n>\n> Nah, discovering it is simple. Just Google for \"git change author\". That\n> eventually leads to a script that uses \"git filter-branch --env-filter\"\n> to get the job done, and I'm afraid it is spread all over the world.\n>\n> See, e.g.:\n>\n> https://help.github.com/en/articles/changing-author-info\n\nSide note: Is the goal to \"fix names and email addresses in this\nrepository\"? If so, this guide fails: it doesn't update tagger names\nor email addresses. Indeed, filter-branch doesn't provide a way to do\nthat. (Not to mention other problems like not updating references to\ncommit hashes in commit messages when it busy rewriting everything.)\n\n> > But I agree that filter-branch isn't useful and certainly\n> > shouldn't be encouraged/promoted.\n>\n> Well, is there more suitable way to change author for a (large) set of\n> commits then?\n\nI would say yes, use git filter-repo (note that this thread started\nwith me proposing filter-repo for inclusion in git.git -- and getting\nsuggestions that we should remove stuff instead of adding more stuff).\nI'm biased, but I think it's much better at this particular job as\nwell:\n\n\nYou can create a mailmap file and pass it to the --mailmap option to\ngit-filter-repo.\n\nOr, if you prefer (perhaps you don't like git's mailmap format as used\nby shortlog and now log, or perhaps you really want to be able to do\nregex replacement or something), you can use the --name-callback or\n--email-callback to work on those fields more directly.\n\nOr, if you prefer (e.g. you want to handle author vs. committer vs.\ntagger differently), you can use the --commit-callback and\n--tag-callback filters.\n\n\nAs an added bonus, filter-repo will also perform the rewrite far\nfaster than filter-branch (and rewrite commit hashes in commit\nmessages as alluded to above).\n\n> > Yet there's probably still users which ARE happy with it, that\n> > will never hit the edge cases and problems it poses; and will\n> > never read release notes. And said users are probably getting\n> > git from a slow-moving distro, so it'd be a disservice to them\n> > if they lost a tool they depend on without any warning.\n>\n> Personally, I'm far from happy with it, but I have no clue how to\n> substitute it in the job above. Anybody?\n\nThe start of this thread where I proposed git filter-repo for\ninclusion in git[1] had links to documentation and comparisons to\nother tools and such. You may find those links helpful; if not, let\nme know what needs to be fixed in the documentation.\n\nElijah\n\n[1] https://public-inbox.org/git/CABPp-BEr8LVM+yWTbi76hAq7Moe1hyp2xqxXfgVV4_teh_9skA@mail.gmail.com/\n"},{"id":"381410","messageId":"20190828002210.8862-1-newren@gmail.com","threadId":"51710","inReplyTo":"20190826235226.15386-1-newren@gmail.com","subject":"[PATCH v2 0/4] Warn about git-filter-branch usage and avoid it","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-28T00:22:06Z","receivedAt":"2019-08-28T00:22:19Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Here's a series that shifts the focus slightly to warning about\ngit-filter-branch usage and avoiding it ourselves. I have retained\npatch 4 but left it marked as RFC for further discussion. It appears\nthat folks generally seem to agree the first three patches are good\nto include now -- assuming my small fixes correctly address their\nrequests and suggestions.\n\nChanges since v1 (full range-diff below):\n * I might have had a little fun with a thesaurus (just trying to give\n reviewers something small to smile about...)\n * addressed feedback from Eric and Stolee, as detailed below\n * [Patch 2] factored out some common code\n * [Patch 3] fixed links in asciidoc documentation to make them more\n readable in both manpages and html format\n * [Patch 3] added a warning blurb to git-filter-branch itself\n\nIn particular, it'd be helpful if people could take a look at the changes\nto git-filter-branch.sh in patch 3 and comment on whether an environment\nvariable is fine or if we should make it a config setting or something.\n\nElijah Newren (4):\n t6006: simplify and optimize empty message test\n t3427: accelerate this test by using fast-export and fast-import\n Recommend git-filter-repo instead of git-filter-branch\n [RFC] Remove git-filter-branch, it is now external to git.git\n\n .gitignore | 1 -\n Documentation/git-fast-export.txt | 6 +-\n Documentation/git-filter-branch.txt | 481 --------------------\n Documentation/git-gc.txt | 17 +-\n Documentation/git-rebase.txt | 2 +-\n Documentation/git-replace.txt | 10 +-\n Documentation/git-svn.txt | 4 +-\n Documentation/githooks.txt | 7 +-\n Makefile | 1 -\n command-list.txt | 1 -\n contrib/svn-fe/svn-fe.txt | 4 +-\n git-filter-branch.sh | 662 ----------------------------\n t/perf/p7000-filter-branch.sh | 24 -\n t/t3427-rebase-subtree.sh | 22 +-\n t/t6006-rev-list-format.sh | 5 +-\n t/t7003-filter-branch.sh | 505 ---------------------\n t/t7009-filter-branch-null-sha1.sh | 55 ---\n t/t9902-completion.sh | 12 +-\n 18 files changed, 47 insertions(+), 1772 deletions(-)\n delete mode 100644 Documentation/git-filter-branch.txt\n delete mode 100755 git-filter-branch.sh\n delete mode 100755 t/perf/p7000-filter-branch.sh\n delete mode 100755 t/t7003-filter-branch.sh\n delete mode 100755 t/t7009-filter-branch-null-sha1.sh\n\nRange-diff:\n1: 7ddbeea2ca = 1: 7ddbeea2ca t6006: simplify and optimize empty message test\n2: 0172ca771e < -: ---------- t3427: accelerate this test by using fast-export and fast-import\n3: b814cc7b65 < -: ---------- git-sh-i18n: work with external scripts\n-: ---------- > 2: f18bd7a609 t3427: accelerate this test by using fast-export and fast-import\n4: dcec36d113 ! 3: 7008c16984 Recommend git-filter-repo instead of git-filter-branch in documentation\n @@ Metadata\n Author: Elijah Newren \n \n ## Commit message ##\n - Recommend git-filter-repo instead of git-filter-branch in documentation\n + Recommend git-filter-repo instead of git-filter-branch\n \n - filter-branch suffers from a huge number of pitfalls that can result in\n - incorrectly rewritten history, and many of the problems can easily go\n - undetected until the new repository is in use. This can result in\n - problems ranging from an even messier history than what led folks to\n - filter-branch in the first place, to data loss or corruption. These\n - issues cannot be backward compatibly fixed, so add a warning to the\n - filter-branch manpage about this and recommand that another tool (such\n - as filter-repo) be used instead.\n + filter-branch suffers from a deluge of disguised dangers that disfigure\n + history rewrites (i.e. deviate from the deliberate changes). Many of\n + these problems are unobtrusive and can easily go undiscovered until the\n + new repository is in use. This can result in problems ranging from an\n + even messier history than what led folks to filter-branch in the first\n + place, to data loss or corruption. These issues cannot be backward\n + compatibly fixed, so add a warning to both filter-branch and its manpage\n + recommending that another tool (such as filter-repo) be used instead.\n \n Also, update other manpages that referenced filter-branch. Several of\n these needed updates even if we could continue recommending\n @@ Documentation/git-filter-branch.txt: SYNOPSIS\n \n +WARNING\n +-------\n -+'git filter-branch' has a litany of gotchas that can and will cause\n -+history to be rewritten incorrectly (in addition to abysmal\n -+performance). These issues cannot be backward compatibly fixed and as\n -+such, its use is not recommended. Please use an alternative history\n -+filtering tool such as 'git filter-repo'. If you still need to use\n -+'git filter-branch', please carefully read the \"Safety\" section of\n -+https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/\n -+and avoid as many of the pitfalls listed there as reasonably possible.\n ++'git filter-branch' has a plethora of pitfalls that can produce non-obvious\n ++manglings of the intended history rewrite (and can leave you with little\n ++time to investigate such problems since it has such abysmal performance).\n ++These safety and performance issues cannot be backward compatibly fixed and\n ++as such, its use is not recommended. Please use an alternative history\n ++filtering tool such as https://github.com/newren/git-filter-repo/[git\n ++filter-repo]. If you still need to use 'git filter-branch', please\n ++carefully read the \"Safety\" section of the message on the Git mailing list\n ++https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/[detailing\n ++the land mines of filter-branch] and vigilantly avoid as many of the\n ++hazards listed there as reasonably possible.\n +\n DESCRIPTION\n -----------\n @@ contrib/svn-fe/svn-fe.txt: The exit status does not reflect whether an error was\n -git-svn(1), svn2git(1), svk(1), git-filter-branch(1), git-fast-import(1),\n +git-svn(1), svn2git(1), svk(1), git-filter-repo(1), git-fast-import(1),\n https://svn.apache.org/repos/asf/subversion/trunk/notes/dump-load-format.txt\n +\n + ## git-filter-branch.sh (mode change 100755 => 100644) ##\n +@@ git-filter-branch.sh: set_ident () {\n + \tfinish_ident COMMITTER\n + }\n + \n ++if [ -z \"$FILTER_BRANCH_SQUELCH_WARNING\" -a \\\n ++ -z \"$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS\" ]; then\n ++\tcat <] [--subdirectory-filter ] [--env-filter ]\n + \t[--tree-filter ] [--index-filter ]\n + \t[--parent-filter ] [--msg-filter ]\n5: 9dec8e06ee ! 4: ff3e04e558 Remove git-filter-branch, it is now external to git.git\n @@ Metadata\n ## Commit message ##\n Remove git-filter-branch, it is now external to git.git\n \n + git-filter-branch still exists, still has the same regression tests,\n + etc., but it is now being tracked in a separate repo that users will\n + need to download separately.\n +\n Signed-off-by: Elijah Newren \n \n ## .gitignore ##\n @@ Documentation/git-filter-branch.txt (deleted)\n -\n -WARNING\n --------\n --'git filter-branch' has a litany of gotchas that can and will cause\n --history to be rewritten incorrectly (in addition to abysmal\n --performance). These issues cannot be backward compatibly fixed and as\n --such, its use is not recommended. Please use an alternative history\n --filtering tool such as 'git filter-repo'. If you still need to use\n --'git filter-branch', please carefully read the \"Safety\" section of\n --https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/\n --and avoid as many of the pitfalls listed there as reasonably possible.\n +-'git filter-branch' has a plethora of pitfalls that can produce non-obvious\n +-manglings of the intended history rewrite (and can leave you with little\n +-time to investigate such problems since it has such abysmal performance).\n +-These safety and performance issues cannot be backward compatibly fixed and\n +-as such, its use is not recommended. Please use an alternative history\n +-filtering tool such as https://github.com/newren/git-filter-repo/[git\n +-filter-repo]. If you still need to use 'git filter-branch', please\n +-carefully read the \"Safety\" section of the message on the Git mailing list\n +-https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/[detailing\n +-the land mines of filter-branch] and vigilantly avoid as many of the\n +-hazards listed there as reasonably possible.\n -\n -DESCRIPTION\n ------------\n @@ git-filter-branch.sh (deleted)\n -\tfinish_ident COMMITTER\n -}\n -\n +-if [ -z \"$FILTER_BRANCH_SQUELCH_WARNING\" -a \\\n +- -z \"$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS\" ]; then\n +-\tcat <] [--subdirectory-filter ] [--env-filter ]\n -\t[--tree-filter ] [--index-filter ]\n -\t[--parent-filter ] [--msg-filter ]\n-- \n2.23.0.3.gcc10030edf.dirty\n\n"},{"id":"381411","messageId":"20190828002210.8862-2-newren@gmail.com","threadId":"51710","inReplyTo":"20190828002210.8862-1-newren@gmail.com","subject":"[PATCH v2 1/4] t6006: simplify and optimize empty message test","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-28T00:22:07Z","receivedAt":"2019-08-28T00:22:21Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Test t6006.71 (\"oneline with empty message\") was creating two commits\nwith simple commit messages, and then running filter-branch to rewrite\nthe commit messages to be empty. This test was written this way because\nthe --allow-empty-message option to git commit did not exist at the\ntime. Simplify this test and avoid the need to invoke filter-branch by\njust using --allow-empty-message when creating the commit.\n\nDespite only being one piece of the 71st test and there being 73 tests\noverall, this small change to just this one test speeds up the overall\nexecution time of t6006 (as measured by the best of 3 runs of `time\n./t6006-rev-list-format.sh`) by about 11% on Linux and by 13% on\nMac.\n\nSigned-off-by: Elijah Newren \n---\n t/t6006-rev-list-format.sh | 5 ++---\n 1 file changed, 2 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t6006-rev-list-format.sh b/t/t6006-rev-list-format.sh\nindex da113d975b..d30e41c9f7 100755\n--- a/t/t6006-rev-list-format.sh\n+++ b/t/t6006-rev-list-format.sh\n@@ -501,9 +501,8 @@ test_expect_success 'reflog identity' '\n '\n \n test_expect_success 'oneline with empty message' '\n-\tgit commit -m \"dummy\" --allow-empty &&\n-\tgit commit -m \"dummy\" --allow-empty &&\n-\tgit filter-branch --msg-filter \"sed -e s/dummy//\" HEAD^^.. &&\n+\tgit commit --allow-empty --allow-empty-message &&\n+\tgit commit --allow-empty --allow-empty-message &&\n \tgit rev-list --oneline HEAD >test.txt &&\n \ttest_line_count = 5 test.txt &&\n \tgit rev-list --oneline --graph HEAD >testg.txt &&\n-- \n2.23.0.3.gcc10030edf.dirty\n\n"},{"id":"381412","messageId":"20190828002210.8862-3-newren@gmail.com","threadId":"51710","inReplyTo":"20190828002210.8862-1-newren@gmail.com","subject":"[PATCH v2 2/4] t3427: accelerate this test by using fast-export and fast-import","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-28T00:22:08Z","receivedAt":"2019-08-28T00:22:22Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"fast-export and fast-import can easily handle the simple rewrite that\nwas being done by filter-branch, and should be significantly faster on\nsystems with a slow fork. Timings from before and after on two laptops\nthat I have access to (measured via `time ./t3427-rebase-subtree.sh`,\ni.e. including everything in this test -- not just the filter-branch or\nfast-export/fast-import pair):\n\n Linux: 4.305s -> 3.684s (~17% speedup)\n Mac: 10.128s -> 7.038s (~30% speedup)\n\nSigned-off-by: Elijah Newren \n---\n t/t3427-rebase-subtree.sh | 22 ++++++++++++++--------\n 1 file changed, 14 insertions(+), 8 deletions(-)\n\ndiff --git a/t/t3427-rebase-subtree.sh b/t/t3427-rebase-subtree.sh\nindex d8640522a0..943ae92226 100755\n--- a/t/t3427-rebase-subtree.sh\n+++ b/t/t3427-rebase-subtree.sh\n@@ -11,6 +11,12 @@ commit_message() {\n \tgit log --pretty=format:%s -1 \"$1\"\n }\n \n+extract_files_subtree() {\n+\tgit fast-export --no-data HEAD -- files_subtree/ \\\n+\t\t| sed -e \"s%\\([0-9a-f]\\{40\\} \\)files_subtree/%\\1%\" \\\n+\t\t| git fast-import --force --quiet\n+}\n+\n test_expect_success 'setup' '\n \ttest_commit README &&\n \tmkdir files &&\n@@ -42,7 +48,7 @@ test_expect_failure REBASE_P \\\n \t'Rebase -Xsubtree --preserve-merges --onto commit 4' '\n \treset_rebase &&\n \tgit checkout -b rebase-preserve-merges-4 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --preserve-merges --onto files-master master &&\n \tverbose test \"$(commit_message HEAD~)\" = \"files_subtree/master4\"\n@@ -53,7 +59,7 @@ test_expect_failure REBASE_P \\\n \t'Rebase -Xsubtree --preserve-merges --onto commit 5' '\n \treset_rebase &&\n \tgit checkout -b rebase-preserve-merges-5 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --preserve-merges --onto files-master master &&\n \tverbose test \"$(commit_message HEAD)\" = \"files_subtree/master5\"\n@@ -64,7 +70,7 @@ test_expect_failure REBASE_P \\\n \t'Rebase -Xsubtree --keep-empty --preserve-merges --onto commit 4' '\n \treset_rebase &&\n \tgit checkout -b rebase-keep-empty-4 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master &&\n \tverbose test \"$(commit_message HEAD~2)\" = \"files_subtree/master4\"\n@@ -75,7 +81,7 @@ test_expect_failure REBASE_P \\\n \t'Rebase -Xsubtree --keep-empty --preserve-merges --onto commit 5' '\n \treset_rebase &&\n \tgit checkout -b rebase-keep-empty-5 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master &&\n \tverbose test \"$(commit_message HEAD~)\" = \"files_subtree/master5\"\n@@ -86,7 +92,7 @@ test_expect_failure REBASE_P \\\n \t'Rebase -Xsubtree --keep-empty --preserve-merges --onto empty commit' '\n \treset_rebase &&\n \tgit checkout -b rebase-keep-empty-empty master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master &&\n \tverbose test \"$(commit_message HEAD)\" = \"Empty commit\"\n@@ -96,7 +102,7 @@ test_expect_failure REBASE_P \\\n test_expect_failure 'Rebase -Xsubtree --onto commit 4' '\n \treset_rebase &&\n \tgit checkout -b rebase-onto-4 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --onto files-master master &&\n \tverbose test \"$(commit_message HEAD~2)\" = \"files_subtree/master4\"\n@@ -106,7 +112,7 @@ test_expect_failure 'Rebase -Xsubtree --onto commit 4' '\n test_expect_failure 'Rebase -Xsubtree --onto commit 5' '\n \treset_rebase &&\n \tgit checkout -b rebase-onto-5 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --onto files-master master &&\n \tverbose test \"$(commit_message HEAD~)\" = \"files_subtree/master5\"\n@@ -115,7 +121,7 @@ test_expect_failure 'Rebase -Xsubtree --onto commit 5' '\n test_expect_failure 'Rebase -Xsubtree --onto empty commit' '\n \treset_rebase &&\n \tgit checkout -b rebase-onto-empty master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --onto files-master master &&\n \tverbose test \"$(commit_message HEAD)\" = \"Empty commit\"\n-- \n2.23.0.3.gcc10030edf.dirty\n\n"},{"id":"381413","messageId":"20190828002210.8862-4-newren@gmail.com","threadId":"51710","inReplyTo":"20190828002210.8862-1-newren@gmail.com","subject":"[PATCH v2 3/4] Recommend git-filter-repo instead of git-filter-branch","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-28T00:22:09Z","receivedAt":"2019-08-28T00:22:24Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"filter-branch suffers from a deluge of disguised dangers that disfigure\nhistory rewrites (i.e. deviate from the deliberate changes). Many of\nthese problems are unobtrusive and can easily go undiscovered until the\nnew repository is in use. This can result in problems ranging from an\neven messier history than what led folks to filter-branch in the first\nplace, to data loss or corruption. These issues cannot be backward\ncompatibly fixed, so add a warning to both filter-branch and its manpage\nrecommending that another tool (such as filter-repo) be used instead.\n\nAlso, update other manpages that referenced filter-branch. Several of\nthese needed updates even if we could continue recommending\nfilter-branch, either due to implying that something was unique to\nfilter-branch when it applied more generally to all history rewriting\ntools (e.g. BFG, reposurgeon, fast-import, filter-repo), or because\nsomething about filter-branch was used as an example despite other more\ncommonly known examples now existing. Reword these sections to fix\nthese issues and to avoid recommending filter-branch.\n\nFinally, remove the section explaining BFG Repo Cleaner as an\nalternative to filter-branch. I feel somewhat bad about this,\nespecially since I feel like I learned so much from BFG that I put to\ngood use in filter-repo (which is much more than I can say for\nfilter-branch), but keeping that section presented a few problems:\n * In order to recommend that people quit using filter-branch, we need\n to provide them a recomendation for something else to use that\n can handle all the same types of rewrites. To my knowledge,\n filter-repo is the only such tool. So it needs to be mentioned.\n * I don't want to give conflicting recommendations to users\n * If we recommend two tools, we shouldn't expect users to learn both\n and pick which one to use; we should explain which problems one\n can solve that the other can't or when one is much faster than\n the other.\n * BFG and filter-repo have similar performance\n * All filtering types that BFG can do, filter-repo can also do. In\n fact, filter-repo comes with a reimplementation of BFG named\n bfg-ish which provides the same user-interface as BFG but with\n several bugfixes and new features that are hard to implement in\n BFG due to its technical underpinnings.\nWhile I could still mention both tools, it seems like I would need to\nprovide some kind of comparison and I would ultimately just say that\nfilter-repo can do everything BFG can, so ultimately it seems that it\nis just better to remove that section altogether.\n\nSigned-off-by: Elijah Newren \n---\n Documentation/git-fast-export.txt | 6 ++--\n Documentation/git-filter-branch.txt | 45 +++++++++--------------------\n Documentation/git-gc.txt | 17 +++++------\n Documentation/git-rebase.txt | 2 +-\n Documentation/git-replace.txt | 10 +++----\n Documentation/git-svn.txt | 4 +--\n Documentation/githooks.txt | 7 +++--\n contrib/svn-fe/svn-fe.txt | 4 +--\n git-filter-branch.sh | 13 +++++++++\n 9 files changed, 52 insertions(+), 56 deletions(-)\n mode change 100755 => 100644 git-filter-branch.sh\n\ndiff --git a/Documentation/git-fast-export.txt b/Documentation/git-fast-export.txt\nindex cc940eb9ad..784e934009 100644\n--- a/Documentation/git-fast-export.txt\n+++ b/Documentation/git-fast-export.txt\n@@ -17,9 +17,9 @@ This program dumps the given revisions in a form suitable to be piped\n into 'git fast-import'.\n \n You can use it as a human-readable bundle replacement (see\n-linkgit:git-bundle[1]), or as a kind of an interactive\n-'git filter-branch'.\n-\n+linkgit:git-bundle[1]), or as a format that can be edited before being\n+fed to 'git fast-import' in order to do history rewrites (an ability\n+relied on by tools like 'git filter-repo').\n \n OPTIONS\n -------\ndiff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt\nindex 6b53dd7e06..e4047d472e 100644\n--- a/Documentation/git-filter-branch.txt\n+++ b/Documentation/git-filter-branch.txt\n@@ -16,6 +16,20 @@ SYNOPSIS\n \t[--original ] [-d ] [-f | --force]\n \t[--state-branch ] [--] [...]\n \n+WARNING\n+-------\n+'git filter-branch' has a plethora of pitfalls that can produce non-obvious\n+manglings of the intended history rewrite (and can leave you with little\n+time to investigate such problems since it has such abysmal performance).\n+These safety and performance issues cannot be backward compatibly fixed and\n+as such, its use is not recommended. Please use an alternative history\n+filtering tool such as https://github.com/newren/git-filter-repo/[git\n+filter-repo]. If you still need to use 'git filter-branch', please\n+carefully read the \"Safety\" section of the message on the Git mailing list\n+https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/[detailing\n+the land mines of filter-branch] and vigilantly avoid as many of the\n+hazards listed there as reasonably possible.\n+\n DESCRIPTION\n -----------\n Lets you rewrite Git revision history by rewriting the branches mentioned\n@@ -445,37 +459,6 @@ warned.\n (or if your git-gc is not new enough to support arguments to\n `--prune`, use `git repack -ad; git prune` instead).\n \n-NOTES\n------\n-\n-git-filter-branch allows you to make complex shell-scripted rewrites\n-of your Git history, but you probably don't need this flexibility if\n-you're simply _removing unwanted data_ like large files or passwords.\n-For those operations you may want to consider\n-http://rtyley.github.io/bfg-repo-cleaner/[The BFG Repo-Cleaner],\n-a JVM-based alternative to git-filter-branch, typically at least\n-10-50x faster for those use-cases, and with quite different\n-characteristics:\n-\n-* Any particular version of a file is cleaned exactly _once_. The BFG,\n- unlike git-filter-branch, does not give you the opportunity to\n- handle a file differently based on where or when it was committed\n- within your history. This constraint gives the core performance\n- benefit of The BFG, and is well-suited to the task of cleansing bad\n- data - you don't care _where_ the bad data is, you just want it\n- _gone_.\n-\n-* By default The BFG takes full advantage of multi-core machines,\n- cleansing commit file-trees in parallel. git-filter-branch cleans\n- commits sequentially (i.e. in a single-threaded manner), though it\n- _is_ possible to write filters that include their own parallelism,\n- in the scripts executed against each commit.\n-\n-* The http://rtyley.github.io/bfg-repo-cleaner/#examples[command options]\n- are much more restrictive than git-filter branch, and dedicated just\n- to the tasks of removing unwanted data- e.g:\n- `--strip-blobs-bigger-than 1M`.\n-\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 247f765604..0c114ad1ca 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -115,15 +115,14 @@ NOTES\n -----\n \n 'git gc' tries very hard not to delete objects that are referenced\n-anywhere in your repository. In\n-particular, it will keep not only objects referenced by your current set\n-of branches and tags, but also objects referenced by the index,\n-remote-tracking branches, refs saved by 'git filter-branch' in\n-refs/original/, reflogs (which may reference commits in branches\n-that were later amended or rewound), and anything else in the refs/* namespace.\n-If you are expecting some objects to be deleted and they aren't, check\n-all of those locations and decide whether it makes sense in your case to\n-remove those references.\n+anywhere in your repository. In particular, it will keep not only\n+objects referenced by your current set of branches and tags, but also\n+objects referenced by the index, remote-tracking branches, notes saved\n+by 'git notes' under refs/notes/, reflogs (which may reference commits\n+in branches that were later amended or rewound), and anything else in\n+the refs/* namespace. If you are expecting some objects to be deleted\n+and they aren't, check all of those locations and decide whether it\n+makes sense in your case to remove those references.\n \n On the other hand, when 'git gc' runs concurrently with another process,\n there is a risk of it deleting an object that the other process is using\ndiff --git a/Documentation/git-rebase.txt b/Documentation/git-rebase.txt\nindex 6156609cf7..2f201d85d4 100644\n--- a/Documentation/git-rebase.txt\n+++ b/Documentation/git-rebase.txt\n@@ -832,7 +832,7 @@ Hard case: The changes are not the same.::\n \tThis happens if the 'subsystem' rebase had conflicts, or used\n \t`--interactive` to omit, edit, squash, or fixup commits; or\n \tif the upstream used one of `commit --amend`, `reset`, or\n-\t`filter-branch`.\n+\ta full history rewriting command like `filter-repo`.\n \n \n The easy case\ndiff --git a/Documentation/git-replace.txt b/Documentation/git-replace.txt\nindex 246dc9943c..35595a2cd3 100644\n--- a/Documentation/git-replace.txt\n+++ b/Documentation/git-replace.txt\n@@ -123,10 +123,10 @@ The following format are available:\n CREATING REPLACEMENT OBJECTS\n ----------------------------\n \n-linkgit:git-filter-branch[1], linkgit:git-hash-object[1] and\n-linkgit:git-rebase[1], among other git commands, can be used to create\n-replacement objects from existing objects. The `--edit` option can\n-also be used with 'git replace' to create a replacement object by\n+linkgit:git-hash-object[1], linkgit:git-rebase[1], and\n+linkgit:git-filter-repo[1], among other git commands, can be used to\n+create replacement objects from existing objects. The `--edit` option\n+can also be used with 'git replace' to create a replacement object by\n editing an existing object.\n \n If you want to replace many blobs, trees or commits that are part of a\n@@ -148,8 +148,8 @@ pending objects.\n SEE ALSO\n --------\n linkgit:git-hash-object[1]\n-linkgit:git-filter-branch[1]\n linkgit:git-rebase[1]\n+linkgit:git-filter-repo[1]\n linkgit:git-tag[1]\n linkgit:git-branch[1]\n linkgit:git-commit[1]\ndiff --git a/Documentation/git-svn.txt b/Documentation/git-svn.txt\nindex 30711625fd..f2762dd5d4 100644\n--- a/Documentation/git-svn.txt\n+++ b/Documentation/git-svn.txt\n@@ -769,9 +769,9 @@ option for (hopefully) obvious reasons.\n +\n This option is NOT recommended as it makes it difficult to track down\n old references to SVN revision numbers in existing documentation, bug\n-reports and archives. If you plan to eventually migrate from SVN to Git\n+reports, and archives. If you plan to eventually migrate from SVN to Git\n and are certain about dropping SVN history, consider\n-linkgit:git-filter-branch[1] instead. filter-branch also allows\n+linkgit:git-filter-repo[1] instead. filter-repo also allows\n reformatting of metadata for ease-of-reading and rewriting authorship\n info for non-\"svn.authorsFile\" users.\n \ndiff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\nindex 82cd573776..997548f5ed 100644\n--- a/Documentation/githooks.txt\n+++ b/Documentation/githooks.txt\n@@ -425,9 +425,10 @@ post-rewrite\n \n This hook is invoked by commands that rewrite commits\n (linkgit:git-commit[1] when called with `--amend` and\n-linkgit:git-rebase[1]; currently `git filter-branch` does 'not' call\n-it!). Its first argument denotes the command it was invoked by:\n-currently one of `amend` or `rebase`. Further command-dependent\n+linkgit:git-rebase[1]; however, full-history (re)writing tools like\n+linkgit:git-fast-import[1] or linkgit:git-filter-repo[1] typically do\n+not call it!). Its first argument denotes the command it was invoked\n+by: currently one of `amend` or `rebase`. Further command-dependent\n arguments may be passed in the future.\n \n The hook receives a list of the rewritten commits on stdin, in the\ndiff --git a/contrib/svn-fe/svn-fe.txt b/contrib/svn-fe/svn-fe.txt\nindex a3425f4770..19333fc8df 100644\n--- a/contrib/svn-fe/svn-fe.txt\n+++ b/contrib/svn-fe/svn-fe.txt\n@@ -56,7 +56,7 @@ line. This line has the form `git-svn-id: URL@REVNO UUID`.\n \n The resulting repository will generally require further processing\n to put each project in its own repository and to separate the history\n-of each branch. The 'git filter-branch --subdirectory-filter' command\n+of each branch. The 'git filter-repo --subdirectory-filter' command\n may be useful for this purpose.\n \n BUGS\n@@ -67,5 +67,5 @@ The exit status does not reflect whether an error was detected.\n \n SEE ALSO\n --------\n-git-svn(1), svn2git(1), svk(1), git-filter-branch(1), git-fast-import(1),\n+git-svn(1), svn2git(1), svk(1), git-filter-repo(1), git-fast-import(1),\n https://svn.apache.org/repos/asf/subversion/trunk/notes/dump-load-format.txt\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\nold mode 100755\nnew mode 100644\nindex 5c5afa2b98..7b1865c1d5\n--- a/git-filter-branch.sh\n+++ b/git-filter-branch.sh\n@@ -83,6 +83,19 @@ set_ident () {\n \tfinish_ident COMMITTER\n }\n \n+if [ -z \"$FILTER_BRANCH_SQUELCH_WARNING\" -a \\\n+ -z \"$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS\" ]; then\n+\tcat <] [--subdirectory-filter ] [--env-filter ]\n \t[--tree-filter ] [--index-filter ]\n \t[--parent-filter ] [--msg-filter ]\n-- \n2.23.0.3.gcc10030edf.dirty\n\n"},{"id":"381414","messageId":"20190828002210.8862-5-newren@gmail.com","threadId":"51710","inReplyTo":"20190828002210.8862-1-newren@gmail.com","subject":"[RFC PATCH v2 4/4] Remove git-filter-branch, it is now external to git.git","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-28T00:22:10Z","receivedAt":"2019-08-28T00:22:26Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"git-filter-branch still exists, still has the same regression tests,\netc., but it is now being tracked in a separate repo that users will\nneed to download separately.\n\nSigned-off-by: Elijah Newren \n---\n .gitignore | 1 -\n Documentation/git-filter-branch.txt | 464 -------------------\n Makefile | 1 -\n command-list.txt | 1 -\n git-filter-branch.sh | 675 ----------------------------\n t/perf/p7000-filter-branch.sh | 24 -\n t/t7003-filter-branch.sh | 505 ---------------------\n t/t7009-filter-branch-null-sha1.sh | 55 ---\n t/t9902-completion.sh | 12 +-\n 9 files changed, 6 insertions(+), 1732 deletions(-)\n delete mode 100644 Documentation/git-filter-branch.txt\n delete mode 100644 git-filter-branch.sh\n delete mode 100755 t/perf/p7000-filter-branch.sh\n delete mode 100755 t/t7003-filter-branch.sh\n delete mode 100755 t/t7009-filter-branch-null-sha1.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex 521d8f4fb4..97f5d8afea 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -63,7 +63,6 @@\n /git-fast-import\n /git-fetch\n /git-fetch-pack\n-/git-filter-branch\n /git-fmt-merge-msg\n /git-for-each-ref\n /git-format-patch\ndiff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt\ndeleted file mode 100644\nindex e4047d472e..0000000000\n--- a/Documentation/git-filter-branch.txt\n+++ /dev/null\n@@ -1,464 +0,0 @@\n-git-filter-branch(1)\n-====================\n-\n-NAME\n-----\n-git-filter-branch - Rewrite branches\n-\n-SYNOPSIS\n---------\n-[verse]\n-'git filter-branch' [--setup ] [--subdirectory-filter ]\n-\t[--env-filter ] [--tree-filter ]\n-\t[--index-filter ] [--parent-filter ]\n-\t[--msg-filter ] [--commit-filter ]\n-\t[--tag-name-filter ] [--prune-empty]\n-\t[--original ] [-d ] [-f | --force]\n-\t[--state-branch ] [--] [...]\n-\n-WARNING\n--------\n-'git filter-branch' has a plethora of pitfalls that can produce non-obvious\n-manglings of the intended history rewrite (and can leave you with little\n-time to investigate such problems since it has such abysmal performance).\n-These safety and performance issues cannot be backward compatibly fixed and\n-as such, its use is not recommended. Please use an alternative history\n-filtering tool such as https://github.com/newren/git-filter-repo/[git\n-filter-repo]. If you still need to use 'git filter-branch', please\n-carefully read the \"Safety\" section of the message on the Git mailing list\n-https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/[detailing\n-the land mines of filter-branch] and vigilantly avoid as many of the\n-hazards listed there as reasonably possible.\n-\n-DESCRIPTION\n------------\n-Lets you rewrite Git revision history by rewriting the branches mentioned\n-in the , applying custom filters on each revision.\n-Those filters can modify each tree (e.g. removing a file or running\n-a perl rewrite on all files) or information about each commit.\n-Otherwise, all information (including original commit times or merge\n-information) will be preserved.\n-\n-The command will only rewrite the _positive_ refs mentioned in the\n-command line (e.g. if you pass 'a..b', only 'b' will be rewritten).\n-If you specify no filters, the commits will be recommitted without any\n-changes, which would normally have no effect. Nevertheless, this may be\n-useful in the future for compensating for some Git bugs or such,\n-therefore such a usage is permitted.\n-\n-*NOTE*: This command honors `.git/info/grafts` file and refs in\n-the `refs/replace/` namespace.\n-If you have any grafts or replacement refs defined, running this command\n-will make them permanent.\n-\n-*WARNING*! The rewritten history will have different object names for all\n-the objects and will not converge with the original branch. You will not\n-be able to easily push and distribute the rewritten branch on top of the\n-original branch. Please do not use this command if you do not know the\n-full implications, and avoid using it anyway, if a simple single commit\n-would suffice to fix your problem. (See the \"RECOVERING FROM UPSTREAM\n-REBASE\" section in linkgit:git-rebase[1] for further information about\n-rewriting published history.)\n-\n-Always verify that the rewritten version is correct: The original refs,\n-if different from the rewritten ones, will be stored in the namespace\n-'refs/original/'.\n-\n-Note that since this operation is very I/O expensive, it might\n-be a good idea to redirect the temporary directory off-disk with the\n-`-d` option, e.g. on tmpfs. Reportedly the speedup is very noticeable.\n-\n-\n-Filters\n-~~~~~~~\n-\n-The filters are applied in the order as listed below. The \n-argument is always evaluated in the shell context using the 'eval' command\n-(with the notable exception of the commit filter, for technical reasons).\n-Prior to that, the `$GIT_COMMIT` environment variable will be set to contain\n-the id of the commit being rewritten. Also, GIT_AUTHOR_NAME,\n-GIT_AUTHOR_EMAIL, GIT_AUTHOR_DATE, GIT_COMMITTER_NAME, GIT_COMMITTER_EMAIL,\n-and GIT_COMMITTER_DATE are taken from the current commit and exported to\n-the environment, in order to affect the author and committer identities of\n-the replacement commit created by linkgit:git-commit-tree[1] after the\n-filters have run.\n-\n-If any evaluation of returns a non-zero exit status, the whole\n-operation will be aborted.\n-\n-A 'map' function is available that takes an \"original sha1 id\" argument\n-and outputs a \"rewritten sha1 id\" if the commit has been already\n-rewritten, and \"original sha1 id\" otherwise; the 'map' function can\n-return several ids on separate lines if your commit filter emitted\n-multiple commits.\n-\n-\n-OPTIONS\n--------\n-\n---setup ::\n-\tThis is not a real filter executed for each commit but a one\n-\ttime setup just before the loop. Therefore no commit-specific\n-\tvariables are defined yet. Functions or variables defined here\n-\tcan be used or modified in the following filter steps except\n-\tthe commit filter, for technical reasons.\n-\n---subdirectory-filter ::\n-\tOnly look at the history which touches the given subdirectory.\n-\tThe result will contain that directory (and only that) as its\n-\tproject root. Implies <>.\n-\n---env-filter ::\n-\tThis filter may be used if you only need to modify the environment\n-\tin which the commit will be performed. Specifically, you might\n-\twant to rewrite the author/committer name/email/time environment\n-\tvariables (see linkgit:git-commit-tree[1] for details).\n-\n---tree-filter ::\n-\tThis is the filter for rewriting the tree and its contents.\n-\tThe argument is evaluated in shell with the working\n-\tdirectory set to the root of the checked out tree. The new tree\n-\tis then used as-is (new files are auto-added, disappeared files\n-\tare auto-removed - neither .gitignore files nor any other ignore\n-\trules *HAVE ANY EFFECT*!).\n-\n---index-filter ::\n-\tThis is the filter for rewriting the index. It is similar to the\n-\ttree filter but does not check out the tree, which makes it much\n-\tfaster. Frequently used with `git rm --cached\n-\t--ignore-unmatch ...`, see EXAMPLES below. For hairy\n-\tcases, see linkgit:git-update-index[1].\n-\n---parent-filter ::\n-\tThis is the filter for rewriting the commit's parent list.\n-\tIt will receive the parent string on stdin and shall output\n-\tthe new parent string on stdout. The parent string is in\n-\tthe format described in linkgit:git-commit-tree[1]: empty for\n-\tthe initial commit, \"-p parent\" for a normal commit and\n-\t\"-p parent1 -p parent2 -p parent3 ...\" for a merge commit.\n-\n---msg-filter ::\n-\tThis is the filter for rewriting the commit messages.\n-\tThe argument is evaluated in the shell with the original\n-\tcommit message on standard input; its standard output is\n-\tused as the new commit message.\n-\n---commit-filter ::\n-\tThis is the filter for performing the commit.\n-\tIf this filter is specified, it will be called instead of the\n-\t'git commit-tree' command, with arguments of the form\n-\t\" [(-p )...]\" and the log message on\n-\tstdin. The commit id is expected on stdout.\n-+\n-As a special extension, the commit filter may emit multiple\n-commit ids; in that case, the rewritten children of the original commit will\n-have all of them as parents.\n-+\n-You can use the 'map' convenience function in this filter, and other\n-convenience functions, too. For example, calling 'skip_commit \"$@\"'\n-will leave out the current commit (but not its changes! If you want\n-that, use 'git rebase' instead).\n-+\n-You can also use the `git_commit_non_empty_tree \"$@\"` instead of\n-`git commit-tree \"$@\"` if you don't wish to keep commits with a single parent\n-and that makes no change to the tree.\n-\n---tag-name-filter ::\n-\tThis is the filter for rewriting tag names. When passed,\n-\tit will be called for every tag ref that points to a rewritten\n-\tobject (or to a tag object which points to a rewritten object).\n-\tThe original tag name is passed via standard input, and the new\n-\ttag name is expected on standard output.\n-+\n-The original tags are not deleted, but can be overwritten;\n-use \"--tag-name-filter cat\" to simply update the tags. In this\n-case, be very careful and make sure you have the old tags\n-backed up in case the conversion has run afoul.\n-+\n-Nearly proper rewriting of tag objects is supported. If the tag has\n-a message attached, a new tag object will be created with the same message,\n-author, and timestamp. If the tag has a signature attached, the\n-signature will be stripped. It is by definition impossible to preserve\n-signatures. The reason this is \"nearly\" proper, is because ideally if\n-the tag did not change (points to the same object, has the same name, etc.)\n-it should retain any signature. That is not the case, signatures will always\n-be removed, buyer beware. There is also no support for changing the\n-author or timestamp (or the tag message for that matter). Tags which point\n-to other tags will be rewritten to point to the underlying commit.\n-\n---prune-empty::\n-\tSome filters will generate empty commits that leave the tree untouched.\n-\tThis option instructs git-filter-branch to remove such commits if they\n-\thave exactly one or zero non-pruned parents; merge commits will\n-\ttherefore remain intact. This option cannot be used together with\n-\t`--commit-filter`, though the same effect can be achieved by using the\n-\tprovided `git_commit_non_empty_tree` function in a commit filter.\n-\n---original ::\n-\tUse this option to set the namespace where the original commits\n-\twill be stored. The default value is 'refs/original'.\n-\n--d ::\n-\tUse this option to set the path to the temporary directory used for\n-\trewriting. When applying a tree filter, the command needs to\n-\ttemporarily check out the tree to some directory, which may consume\n-\tconsiderable space in case of large projects. By default it\n-\tdoes this in the `.git-rewrite/` directory but you can override\n-\tthat choice by this parameter.\n-\n--f::\n---force::\n-\t'git filter-branch' refuses to start with an existing temporary\n-\tdirectory or when there are already refs starting with\n-\t'refs/original/', unless forced.\n-\n---state-branch ::\n-\tThis option will cause the mapping from old to new objects to\n-\tbe loaded from named branch upon startup and saved as a new\n-\tcommit to that branch upon exit, enabling incremental of large\n-\ttrees. If '' does not exist it will be created.\n-\n-...::\n-\tArguments for 'git rev-list'. All positive refs included by\n-\tthese options are rewritten. You may also specify options\n-\tsuch as `--all`, but you must use `--` to separate them from\n-\tthe 'git filter-branch' options. Implies <>.\n-\n-\n-[[Remap_to_ancestor]]\n-Remap to ancestor\n-~~~~~~~~~~~~~~~~~\n-\n-By using linkgit:git-rev-list[1] arguments, e.g., path limiters, you can limit the\n-set of revisions which get rewritten. However, positive refs on the command\n-line are distinguished: we don't let them be excluded by such limiters. For\n-this purpose, they are instead rewritten to point at the nearest ancestor that\n-was not excluded.\n-\n-\n-EXIT STATUS\n------------\n-\n-On success, the exit status is `0`. If the filter can't find any commits to\n-rewrite, the exit status is `2`. On any other error, the exit status may be\n-any other non-zero value.\n-\n-\n-EXAMPLES\n---------\n-\n-Suppose you want to remove a file (containing confidential information\n-or copyright violation) from all commits:\n-\n--------------------------------------------------------\n-git filter-branch --tree-filter 'rm filename' HEAD\n--------------------------------------------------------\n-\n-However, if the file is absent from the tree of some commit,\n-a simple `rm filename` will fail for that tree and commit.\n-Thus you may instead want to use `rm -f filename` as the script.\n-\n-Using `--index-filter` with 'git rm' yields a significantly faster\n-version. Like with using `rm filename`, `git rm --cached filename`\n-will fail if the file is absent from the tree of a commit. If you\n-want to \"completely forget\" a file, it does not matter when it entered\n-history, so we also add `--ignore-unmatch`:\n-\n---------------------------------------------------------------------------\n-git filter-branch --index-filter 'git rm --cached --ignore-unmatch filename' HEAD\n---------------------------------------------------------------------------\n-\n-Now, you will get the rewritten history saved in HEAD.\n-\n-To rewrite the repository to look as if `foodir/` had been its project\n-root, and discard all other history:\n-\n--------------------------------------------------------\n-git filter-branch --subdirectory-filter foodir -- --all\n--------------------------------------------------------\n-\n-Thus you can, e.g., turn a library subdirectory into a repository of\n-its own. Note the `--` that separates 'filter-branch' options from\n-revision options, and the `--all` to rewrite all branches and tags.\n-\n-To set a commit (which typically is at the tip of another\n-history) to be the parent of the current initial commit, in\n-order to paste the other history behind the current history:\n-\n--------------------------------------------------------------------\n-git filter-branch --parent-filter 'sed \"s/^\\$/-p /\"' HEAD\n--------------------------------------------------------------------\n-\n-(if the parent string is empty - which happens when we are dealing with\n-the initial commit - add graftcommit as a parent). Note that this assumes\n-history with a single root (that is, no merge without common ancestors\n-happened). If this is not the case, use:\n-\n---------------------------------------------------------------------------\n-git filter-branch --parent-filter \\\n-\t'test $GIT_COMMIT = && echo \"-p \" || cat' HEAD\n---------------------------------------------------------------------------\n-\n-or even simpler:\n-\n------------------------------------------------\n-git replace --graft $commit-id $graft-id\n-git filter-branch $graft-id..HEAD\n------------------------------------------------\n-\n-To remove commits authored by \"Darl McBribe\" from the history:\n-\n-------------------------------------------------------------------------------\n-git filter-branch --commit-filter '\n-\tif [ \"$GIT_AUTHOR_NAME\" = \"Darl McBribe\" ];\n-\tthen\n-\t\tskip_commit \"$@\";\n-\telse\n-\t\tgit commit-tree \"$@\";\n-\tfi' HEAD\n-------------------------------------------------------------------------------\n-\n-The function 'skip_commit' is defined as follows:\n-\n---------------------------\n-skip_commit()\n-{\n-\tshift;\n-\twhile [ -n \"$1\" ];\n-\tdo\n-\t\tshift;\n-\t\tmap \"$1\";\n-\t\tshift;\n-\tdone;\n-}\n---------------------------\n-\n-The shift magic first throws away the tree id and then the -p\n-parameters. Note that this handles merges properly! In case Darl\n-committed a merge between P1 and P2, it will be propagated properly\n-and all children of the merge will become merge commits with P1,P2\n-as their parents instead of the merge commit.\n-\n-*NOTE* the changes introduced by the commits, and which are not reverted\n-by subsequent commits, will still be in the rewritten branch. If you want\n-to throw out _changes_ together with the commits, you should use the\n-interactive mode of 'git rebase'.\n-\n-You can rewrite the commit log messages using `--msg-filter`. For\n-example, 'git svn-id' strings in a repository created by 'git svn' can\n-be removed this way:\n-\n--------------------------------------------------------\n-git filter-branch --msg-filter '\n-\tsed -e \"/^git-svn-id:/d\"\n-'\n--------------------------------------------------------\n-\n-If you need to add 'Acked-by' lines to, say, the last 10 commits (none\n-of which is a merge), use this command:\n-\n---------------------------------------------------------\n-git filter-branch --msg-filter '\n-\tcat &&\n-\techo \"Acked-by: Bugs Bunny \"\n-' HEAD~10..HEAD\n---------------------------------------------------------\n-\n-The `--env-filter` option can be used to modify committer and/or author\n-identity. For example, if you found out that your commits have the wrong\n-identity due to a misconfigured user.email, you can make a correction,\n-before publishing the project, like this:\n-\n---------------------------------------------------------\n-git filter-branch --env-filter '\n-\tif test \"$GIT_AUTHOR_EMAIL\" = \"root@localhost\"\n-\tthen\n-\t\tGIT_AUTHOR_EMAIL=john@example.com\n-\tfi\n-\tif test \"$GIT_COMMITTER_EMAIL\" = \"root@localhost\"\n-\tthen\n-\t\tGIT_COMMITTER_EMAIL=john@example.com\n-\tfi\n-' -- --all\n---------------------------------------------------------\n-\n-To restrict rewriting to only part of the history, specify a revision\n-range in addition to the new branch name. The new branch name will\n-point to the top-most revision that a 'git rev-list' of this range\n-will print.\n-\n-Consider this history:\n-\n-------------------\n- D--E--F--G--H\n- / /\n-A--B-----C\n-------------------\n-\n-To rewrite only commits D,E,F,G,H, but leave A, B and C alone, use:\n-\n---------------------------------\n-git filter-branch ... C..H\n---------------------------------\n-\n-To rewrite commits E,F,G,H, use one of these:\n-\n-----------------------------------------\n-git filter-branch ... C..H --not D\n-git filter-branch ... D..H --not C\n-----------------------------------------\n-\n-To move the whole tree into a subdirectory, or remove it from there:\n-\n----------------------------------------------------------------\n-git filter-branch --index-filter \\\n-\t'git ls-files -s | sed \"s-\\t\\\"*-&newsubdir/-\" |\n-\t\tGIT_INDEX_FILE=$GIT_INDEX_FILE.new \\\n-\t\t\tgit update-index --index-info &&\n-\t mv \"$GIT_INDEX_FILE.new\" \"$GIT_INDEX_FILE\"' HEAD\n----------------------------------------------------------------\n-\n-\n-\n-CHECKLIST FOR SHRINKING A REPOSITORY\n-------------------------------------\n-\n-git-filter-branch can be used to get rid of a subset of files,\n-usually with some combination of `--index-filter` and\n-`--subdirectory-filter`. People expect the resulting repository to\n-be smaller than the original, but you need a few more steps to\n-actually make it smaller, because Git tries hard not to lose your\n-objects until you tell it to. First make sure that:\n-\n-* You really removed all variants of a filename, if a blob was moved\n- over its lifetime. `git log --name-only --follow --all -- filename`\n- can help you find renames.\n-\n-* You really filtered all refs: use `--tag-name-filter cat -- --all`\n- when calling git-filter-branch.\n-\n-Then there are two ways to get a smaller repository. A safer way is\n-to clone, that keeps your original intact.\n-\n-* Clone it with `git clone file:///path/to/repo`. The clone\n- will not have the removed objects. See linkgit:git-clone[1]. (Note\n- that cloning with a plain path just hardlinks everything!)\n-\n-If you really don't want to clone it, for whatever reasons, check the\n-following points instead (in this order). This is a very destructive\n-approach, so *make a backup* or go back to cloning it. You have been\n-warned.\n-\n-* Remove the original refs backed up by git-filter-branch: say `git\n- for-each-ref --format=\"%(refname)\" refs/original/ | xargs -n 1 git\n- update-ref -d`.\n-\n-* Expire all reflogs with `git reflog expire --expire=now --all`.\n-\n-* Garbage collect all unreferenced objects with `git gc --prune=now`\n- (or if your git-gc is not new enough to support arguments to\n- `--prune`, use `git repack -ad; git prune` instead).\n-\n-GIT\n----\n-Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex f9255344ae..20850def5d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -607,7 +607,6 @@ unexport CDPATH\n \n SCRIPT_SH += git-bisect.sh\n SCRIPT_SH += git-difftool--helper.sh\n-SCRIPT_SH += git-filter-branch.sh\n SCRIPT_SH += git-merge-octopus.sh\n SCRIPT_SH += git-merge-one-file.sh\n SCRIPT_SH += git-merge-resolve.sh\ndiff --git a/command-list.txt b/command-list.txt\nindex a9ac72bef4..1ba65d9516 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -90,7 +90,6 @@ git-fast-export ancillarymanipulators\n git-fast-import ancillarymanipulators\n git-fetch mainporcelain remote\n git-fetch-pack synchingrepositories\n-git-filter-branch ancillarymanipulators\n git-fmt-merge-msg purehelpers\n git-for-each-ref plumbinginterrogators\n git-format-patch mainporcelain\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\ndeleted file mode 100644\nindex 7b1865c1d5..0000000000\n--- a/git-filter-branch.sh\n+++ /dev/null\n@@ -1,675 +0,0 @@\n-#!/bin/sh\n-#\n-# Rewrite revision history\n-# Copyright (c) Petr Baudis, 2006\n-# Minimal changes to \"port\" it to core-git (c) Johannes Schindelin, 2007\n-#\n-# Lets you rewrite the revision history of the current branch, creating\n-# a new branch. You can specify a number of filters to modify the commits,\n-# files and trees.\n-\n-# The following functions will also be available in the commit filter:\n-\n-functions=$(cat << \\EOF\n-EMPTY_TREE=$(git hash-object -t tree /dev/null)\n-\n-warn () {\n-\techo \"$*\" >&2\n-}\n-\n-map()\n-{\n-\t# if it was not rewritten, take the original\n-\tif test -r \"$workdir/../map/$1\"\n-\tthen\n-\t\tcat \"$workdir/../map/$1\"\n-\telse\n-\t\techo \"$1\"\n-\tfi\n-}\n-\n-# if you run 'skip_commit \"$@\"' in a commit filter, it will print\n-# the (mapped) parents, effectively skipping the commit.\n-\n-skip_commit()\n-{\n-\tshift;\n-\twhile [ -n \"$1\" ];\n-\tdo\n-\t\tshift;\n-\t\tmap \"$1\";\n-\t\tshift;\n-\tdone;\n-}\n-\n-# if you run 'git_commit_non_empty_tree \"$@\"' in a commit filter,\n-# it will skip commits that leave the tree untouched, commit the other.\n-git_commit_non_empty_tree()\n-{\n-\tif test $# = 3 && test \"$1\" = $(git rev-parse \"$3^{tree}\"); then\n-\t\tmap \"$3\"\n-\telif test $# = 1 && test \"$1\" = $EMPTY_TREE; then\n-\t\t:\n-\telse\n-\t\tgit commit-tree \"$@\"\n-\tfi\n-}\n-# override die(): this version puts in an extra line break, so that\n-# the progress is still visible\n-\n-die()\n-{\n-\techo >&2\n-\techo \"$*\" >&2\n-\texit 1\n-}\n-EOF\n-)\n-\n-eval \"$functions\"\n-\n-finish_ident() {\n-\t# Ensure non-empty id name.\n-\techo \"case \\\"\\$GIT_$1_NAME\\\" in \\\"\\\") GIT_$1_NAME=\\\"\\${GIT_$1_EMAIL%%@*}\\\" && export GIT_$1_NAME;; esac\"\n-\t# And make sure everything is exported.\n-\techo \"export GIT_$1_NAME\"\n-\techo \"export GIT_$1_EMAIL\"\n-\techo \"export GIT_$1_DATE\"\n-}\n-\n-set_ident () {\n-\tparse_ident_from_commit author AUTHOR committer COMMITTER\n-\tfinish_ident AUTHOR\n-\tfinish_ident COMMITTER\n-}\n-\n-if [ -z \"$FILTER_BRANCH_SQUELCH_WARNING\" -a \\\n- -z \"$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS\" ]; then\n-\tcat <] [--subdirectory-filter ] [--env-filter ]\n-\t[--tree-filter ] [--index-filter ]\n-\t[--parent-filter ] [--msg-filter ]\n-\t[--commit-filter ] [--tag-name-filter ]\n-\t[--original ]\n-\t[-d ] [-f | --force] [--state-branch ]\n-\t[--] [...]\"\n-\n-OPTIONS_SPEC=\n-. git-sh-setup\n-\n-if [ \"$(is_bare_repository)\" = false ]; then\n-\trequire_clean_work_tree 'rewrite branches'\n-fi\n-\n-tempdir=.git-rewrite\n-filter_setup=\n-filter_env=\n-filter_tree=\n-filter_index=\n-filter_parent=\n-filter_msg=cat\n-filter_commit=\n-filter_tag_name=\n-filter_subdir=\n-state_branch=\n-orig_namespace=refs/original/\n-force=\n-prune_empty=\n-remap_to_ancestor=\n-while :\n-do\n-\tcase \"$1\" in\n-\t--)\n-\t\tshift\n-\t\tbreak\n-\t\t;;\n-\t--force|-f)\n-\t\tshift\n-\t\tforce=t\n-\t\tcontinue\n-\t\t;;\n-\t--remap-to-ancestor)\n-\t\t# deprecated ($remap_to_ancestor is set now automatically)\n-\t\tshift\n-\t\tremap_to_ancestor=t\n-\t\tcontinue\n-\t\t;;\n-\t--prune-empty)\n-\t\tshift\n-\t\tprune_empty=t\n-\t\tcontinue\n-\t\t;;\n-\t-*)\n-\t\t;;\n-\t*)\n-\t\tbreak;\n-\tesac\n-\n-\t# all switches take one argument\n-\tARG=\"$1\"\n-\tcase \"$#\" in 1) usage ;; esac\n-\tshift\n-\tOPTARG=\"$1\"\n-\tshift\n-\n-\tcase \"$ARG\" in\n-\t-d)\n-\t\ttempdir=\"$OPTARG\"\n-\t\t;;\n-\t--setup)\n-\t\tfilter_setup=\"$OPTARG\"\n-\t\t;;\n-\t--subdirectory-filter)\n-\t\tfilter_subdir=\"$OPTARG\"\n-\t\tremap_to_ancestor=t\n-\t\t;;\n-\t--env-filter)\n-\t\tfilter_env=\"$OPTARG\"\n-\t\t;;\n-\t--tree-filter)\n-\t\tfilter_tree=\"$OPTARG\"\n-\t\t;;\n-\t--index-filter)\n-\t\tfilter_index=\"$OPTARG\"\n-\t\t;;\n-\t--parent-filter)\n-\t\tfilter_parent=\"$OPTARG\"\n-\t\t;;\n-\t--msg-filter)\n-\t\tfilter_msg=\"$OPTARG\"\n-\t\t;;\n-\t--commit-filter)\n-\t\tfilter_commit=\"$functions; $OPTARG\"\n-\t\t;;\n-\t--tag-name-filter)\n-\t\tfilter_tag_name=\"$OPTARG\"\n-\t\t;;\n-\t--original)\n-\t\torig_namespace=$(expr \"$OPTARG/\" : '\\(.*[^/]\\)/*$')/\n-\t\t;;\n-\t--state-branch)\n-\t\tstate_branch=\"$OPTARG\"\n-\t\t;;\n-\t*)\n-\t\tusage\n-\t\t;;\n-\tesac\n-done\n-\n-case \"$prune_empty,$filter_commit\" in\n-,)\n-\tfilter_commit='git commit-tree \"$@\"';;\n-t,)\n-\tfilter_commit=\"$functions;\"' git_commit_non_empty_tree \"$@\"';;\n-,*)\n-\t;;\n-*)\n-\tdie \"Cannot set --prune-empty and --commit-filter at the same time\"\n-esac\n-\n-case \"$force\" in\n-t)\n-\trm -rf \"$tempdir\"\n-;;\n-'')\n-\ttest -d \"$tempdir\" &&\n-\t\tdie \"$tempdir already exists, please remove it\"\n-esac\n-orig_dir=$(pwd)\n-mkdir -p \"$tempdir/t\" &&\n-tempdir=\"$(cd \"$tempdir\"; pwd)\" &&\n-cd \"$tempdir/t\" &&\n-workdir=\"$(pwd)\" ||\n-die \"\"\n-\n-# Remove tempdir on exit\n-trap 'cd \"$orig_dir\"; rm -rf \"$tempdir\"' 0\n-\n-ORIG_GIT_DIR=\"$GIT_DIR\"\n-ORIG_GIT_WORK_TREE=\"$GIT_WORK_TREE\"\n-ORIG_GIT_INDEX_FILE=\"$GIT_INDEX_FILE\"\n-ORIG_GIT_AUTHOR_NAME=\"$GIT_AUTHOR_NAME\"\n-ORIG_GIT_AUTHOR_EMAIL=\"$GIT_AUTHOR_EMAIL\"\n-ORIG_GIT_AUTHOR_DATE=\"$GIT_AUTHOR_DATE\"\n-ORIG_GIT_COMMITTER_NAME=\"$GIT_COMMITTER_NAME\"\n-ORIG_GIT_COMMITTER_EMAIL=\"$GIT_COMMITTER_EMAIL\"\n-ORIG_GIT_COMMITTER_DATE=\"$GIT_COMMITTER_DATE\"\n-\n-GIT_WORK_TREE=.\n-export GIT_DIR GIT_WORK_TREE\n-\n-# Make sure refs/original is empty\n-git for-each-ref > \"$tempdir\"/backup-refs || exit\n-while read sha1 type name\n-do\n-\tcase \"$force,$name\" in\n-\t,$orig_namespace*)\n-\t\tdie \"Cannot create a new backup.\n-A previous backup already exists in $orig_namespace\n-Force overwriting the backup with -f\"\n-\t;;\n-\tt,$orig_namespace*)\n-\t\tgit update-ref -d \"$name\" $sha1\n-\t;;\n-\tesac\n-done < \"$tempdir\"/backup-refs\n-\n-# The refs should be updated if their heads were rewritten\n-git rev-parse --no-flags --revs-only --symbolic-full-name \\\n-\t--default HEAD \"$@\" > \"$tempdir\"/raw-refs || exit\n-while read ref\n-do\n-\tcase \"$ref\" in ^?*) continue ;; esac\n-\n-\tif git rev-parse --verify \"$ref\"^0 >/dev/null 2>&1\n-\tthen\n-\t\techo \"$ref\"\n-\telse\n-\t\twarn \"WARNING: not rewriting '$ref' (not a committish)\"\n-\tfi\n-done >\"$tempdir\"/heads <\"$tempdir\"/raw-refs\n-\n-test -s \"$tempdir\"/heads ||\n-\tdie \"You must specify a ref to rewrite.\"\n-\n-GIT_INDEX_FILE=\"$(pwd)/../index\"\n-export GIT_INDEX_FILE\n-\n-# map old->new commit ids for rewriting parents\n-mkdir ../map || die \"Could not create map/ directory\"\n-\n-if test -n \"$state_branch\"\n-then\n-\tstate_commit=$(git rev-parse --no-flags --revs-only \"$state_branch\")\n-\tif test -n \"$state_commit\"\n-\tthen\n-\t\techo \"Populating map from $state_branch ($state_commit)\" 1>&2\n-\t\tperl -e'open(MAP, \"-|\", \"git show $ARGV[0]:filter.map\") or die;\n-\t\t\twhile () {\n-\t\t\t\tm/(.*):(.*)/ or die;\n-\t\t\t\topen F, \">../map/$1\" or die;\n-\t\t\t\tprint F \"$2\" or die;\n-\t\t\t\tclose(F) or die;\n-\t\t\t}\n-\t\t\tclose(MAP) or die;' \"$state_commit\" \\\n-\t\t\t\t|| die \"Unable to load state from $state_branch:filter.map\"\n-\telse\n-\t\techo \"Branch $state_branch does not exist. Will create\" 1>&2\n-\tfi\n-fi\n-\n-# we need \"--\" only if there are no path arguments in $@\n-nonrevs=$(git rev-parse --no-revs \"$@\") || exit\n-if test -z \"$nonrevs\"\n-then\n-\tdashdash=--\n-else\n-\tdashdash=\n-\tremap_to_ancestor=t\n-fi\n-\n-git rev-parse --revs-only \"$@\" >../parse\n-\n-case \"$filter_subdir\" in\n-\"\")\n-\teval set -- \"$(git rev-parse --sq --no-revs \"$@\")\"\n-\t;;\n-*)\n-\teval set -- \"$(git rev-parse --sq --no-revs \"$@\" $dashdash \\\n-\t\t\"$filter_subdir\")\"\n-\t;;\n-esac\n-\n-git rev-list --reverse --topo-order --default HEAD \\\n-\t--parents --simplify-merges --stdin \"$@\" <../parse >../revs ||\n-\tdie \"Could not get the commits\"\n-commits=$(wc -l <../revs | tr -d \" \")\n-\n-test $commits -eq 0 && die_with_status 2 \"Found nothing to rewrite\"\n-\n-# Rewrite the commits\n-report_progress ()\n-{\n-\tif test -n \"$progress\" &&\n-\t\ttest $git_filter_branch__commit_count -gt $next_sample_at\n-\tthen\n-\t\tcount=$git_filter_branch__commit_count\n-\n-\t\tnow=$(date +%s)\n-\t\telapsed=$(($now - $start_timestamp))\n-\t\tremaining=$(( ($commits - $count) * $elapsed / $count ))\n-\t\tif test $elapsed -gt 0\n-\t\tthen\n-\t\t\tnext_sample_at=$(( ($elapsed + 1) * $count / $elapsed ))\n-\t\telse\n-\t\t\tnext_sample_at=$(($next_sample_at + 1))\n-\t\tfi\n-\t\tprogress=\" ($elapsed seconds passed, remaining $remaining predicted)\"\n-\tfi\n-\tprintf \"\\rRewrite $commit ($count/$commits)$progress \"\n-}\n-\n-git_filter_branch__commit_count=0\n-\n-progress= start_timestamp=\n-if date '+%s' 2>/dev/null | grep -q '^[0-9][0-9]*$'\n-then\n-\tnext_sample_at=0\n-\tprogress=\"dummy to ensure this is not empty\"\n-\tstart_timestamp=$(date '+%s')\n-fi\n-\n-if test -n \"$filter_index\" ||\n- test -n \"$filter_tree\" ||\n- test -n \"$filter_subdir\"\n-then\n-\tneed_index=t\n-else\n-\tneed_index=\n-fi\n-\n-eval \"$filter_setup\" < /dev/null ||\n-\tdie \"filter setup failed: $filter_setup\"\n-\n-while read commit parents; do\n-\tgit_filter_branch__commit_count=$(($git_filter_branch__commit_count+1))\n-\n-\treport_progress\n-\ttest -f \"$workdir\"/../map/$commit && continue\n-\n-\tcase \"$filter_subdir\" in\n-\t\"\")\n-\t\tif test -n \"$need_index\"\n-\t\tthen\n-\t\t\tGIT_ALLOW_NULL_SHA1=1 git read-tree -i -m $commit\n-\t\tfi\n-\t\t;;\n-\t*)\n-\t\t# The commit may not have the subdirectory at all\n-\t\terr=$(GIT_ALLOW_NULL_SHA1=1 \\\n-\t\t git read-tree -i -m $commit:\"$filter_subdir\" 2>&1) || {\n-\t\t\tif ! git rev-parse -q --verify $commit:\"$filter_subdir\"\n-\t\t\tthen\n-\t\t\t\trm -f \"$GIT_INDEX_FILE\"\n-\t\t\telse\n-\t\t\t\techo >&2 \"$err\"\n-\t\t\t\tfalse\n-\t\t\tfi\n-\t\t}\n-\tesac || die \"Could not initialize the index\"\n-\n-\tGIT_COMMIT=$commit\n-\texport GIT_COMMIT\n-\tgit cat-file commit \"$commit\" >../commit ||\n-\t\tdie \"Cannot read commit $commit\"\n-\n-\teval \"$(set_ident <../commit)\" ||\n-\t\tdie \"setting author/committer failed for commit $commit\"\n-\teval \"$filter_env\" < /dev/null ||\n-\t\tdie \"env filter failed: $filter_env\"\n-\n-\tif [ \"$filter_tree\" ]; then\n-\t\tgit checkout-index -f -u -a ||\n-\t\t\tdie \"Could not checkout the index\"\n-\t\t# files that $commit removed are now still in the working tree;\n-\t\t# remove them, else they would be added again\n-\t\tgit clean -d -q -f -x\n-\t\teval \"$filter_tree\" < /dev/null ||\n-\t\t\tdie \"tree filter failed: $filter_tree\"\n-\n-\t\t(\n-\t\t\tgit diff-index -r --name-only --ignore-submodules $commit -- &&\n-\t\t\tgit ls-files --others\n-\t\t) > \"$tempdir\"/tree-state || exit\n-\t\tgit update-index --add --replace --remove --stdin \\\n-\t\t\t< \"$tempdir\"/tree-state || exit\n-\tfi\n-\n-\teval \"$filter_index\" < /dev/null ||\n-\t\tdie \"index filter failed: $filter_index\"\n-\n-\tparentstr=\n-\tfor parent in $parents; do\n-\t\tfor reparent in $(map \"$parent\"); do\n-\t\t\tcase \"$parentstr \" in\n-\t\t\t*\" -p $reparent \"*)\n-\t\t\t\t;;\n-\t\t\t*)\n-\t\t\t\tparentstr=\"$parentstr -p $reparent\"\n-\t\t\t\t;;\n-\t\t\tesac\n-\t\tdone\n-\tdone\n-\tif [ \"$filter_parent\" ]; then\n-\t\tparentstr=\"$(echo \"$parentstr\" | eval \"$filter_parent\")\" ||\n-\t\t\t\tdie \"parent filter failed: $filter_parent\"\n-\tfi\n-\n-\t{\n-\t\twhile IFS='' read -r header_line && test -n \"$header_line\"\n-\t\tdo\n-\t\t\t# skip header lines...\n-\t\t\t:;\n-\t\tdone\n-\t\t# and output the actual commit message\n-\t\tcat\n-\t} <../commit |\n-\t\teval \"$filter_msg\" > ../message ||\n-\t\t\tdie \"msg filter failed: $filter_msg\"\n-\n-\tif test -n \"$need_index\"\n-\tthen\n-\t\ttree=$(git write-tree)\n-\telse\n-\t\ttree=$(git rev-parse \"$commit^{tree}\")\n-\tfi\n-\tworkdir=$workdir @SHELL_PATH@ -c \"$filter_commit\" \"git commit-tree\" \\\n-\t\t\"$tree\" $parentstr < ../message > ../map/$commit ||\n-\t\t\tdie \"could not write rewritten commit\"\n-done <../revs\n-\n-# If we are filtering for paths, as in the case of a subdirectory\n-# filter, it is possible that a specified head is not in the set of\n-# rewritten commits, because it was pruned by the revision walker.\n-# Ancestor remapping fixes this by mapping these heads to the unique\n-# nearest ancestor that survived the pruning.\n-\n-if test \"$remap_to_ancestor\" = t\n-then\n-\twhile read ref\n-\tdo\n-\t\tsha1=$(git rev-parse \"$ref\"^0)\n-\t\ttest -f \"$workdir\"/../map/$sha1 && continue\n-\t\tancestor=$(git rev-list --simplify-merges -1 \"$ref\" \"$@\")\n-\t\ttest \"$ancestor\" && echo $(map $ancestor) >> \"$workdir\"/../map/$sha1\n-\tdone < \"$tempdir\"/heads\n-fi\n-\n-# Finally update the refs\n-\n-_x40='[0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f]'\n-_x40=\"$_x40$_x40$_x40$_x40$_x40$_x40$_x40$_x40\"\n-echo\n-while read ref\n-do\n-\t# avoid rewriting a ref twice\n-\ttest -f \"$orig_namespace$ref\" && continue\n-\n-\tsha1=$(git rev-parse \"$ref\"^0)\n-\trewritten=$(map $sha1)\n-\n-\ttest $sha1 = \"$rewritten\" &&\n-\t\twarn \"WARNING: Ref '$ref' is unchanged\" &&\n-\t\tcontinue\n-\n-\tcase \"$rewritten\" in\n-\t'')\n-\t\techo \"Ref '$ref' was deleted\"\n-\t\tgit update-ref -m \"filter-branch: delete\" -d \"$ref\" $sha1 ||\n-\t\t\tdie \"Could not delete $ref\"\n-\t;;\n-\t$_x40)\n-\t\techo \"Ref '$ref' was rewritten\"\n-\t\tif ! git update-ref -m \"filter-branch: rewrite\" \\\n-\t\t\t\t\t\"$ref\" $rewritten $sha1 2>/dev/null; then\n-\t\t\tif test $(git cat-file -t \"$ref\") = tag; then\n-\t\t\t\tif test -z \"$filter_tag_name\"; then\n-\t\t\t\t\twarn \"WARNING: You said to rewrite tagged commits, but not the corresponding tag.\"\n-\t\t\t\t\twarn \"WARNING: Perhaps use '--tag-name-filter cat' to rewrite the tag.\"\n-\t\t\t\tfi\n-\t\t\telse\n-\t\t\t\tdie \"Could not rewrite $ref\"\n-\t\t\tfi\n-\t\tfi\n-\t;;\n-\t*)\n-\t\t# NEEDSWORK: possibly add -Werror, making this an error\n-\t\twarn \"WARNING: '$ref' was rewritten into multiple commits:\"\n-\t\twarn \"$rewritten\"\n-\t\twarn \"WARNING: Ref '$ref' points to the first one now.\"\n-\t\trewritten=$(echo \"$rewritten\" | head -n 1)\n-\t\tgit update-ref -m \"filter-branch: rewrite to first\" \\\n-\t\t\t\t\"$ref\" $rewritten $sha1 ||\n-\t\t\tdie \"Could not rewrite $ref\"\n-\t;;\n-\tesac\n-\tgit update-ref -m \"filter-branch: backup\" \"$orig_namespace$ref\" $sha1 ||\n-\t\t exit\n-done < \"$tempdir\"/heads\n-\n-# TODO: This should possibly go, with the semantics that all positive given\n-# refs are updated, and their original heads stored in refs/original/\n-# Filter tags\n-\n-if [ \"$filter_tag_name\" ]; then\n-\tgit for-each-ref --format='%(objectname) %(objecttype) %(refname)' refs/tags |\n-\twhile read sha1 type ref; do\n-\t\tref=\"${ref#refs/tags/}\"\n-\t\t# XXX: Rewrite tagged trees as well?\n-\t\tif [ \"$type\" != \"commit\" -a \"$type\" != \"tag\" ]; then\n-\t\t\tcontinue;\n-\t\tfi\n-\n-\t\tif [ \"$type\" = \"tag\" ]; then\n-\t\t\t# Dereference to a commit\n-\t\t\tsha1t=\"$sha1\"\n-\t\t\tsha1=\"$(git rev-parse -q \"$sha1\"^{commit})\" || continue\n-\t\tfi\n-\n-\t\t[ -f \"../map/$sha1\" ] || continue\n-\t\tnew_sha1=\"$(cat \"../map/$sha1\")\"\n-\t\tGIT_COMMIT=\"$sha1\"\n-\t\texport GIT_COMMIT\n-\t\tnew_ref=\"$(echo \"$ref\" | eval \"$filter_tag_name\")\" ||\n-\t\t\tdie \"tag name filter failed: $filter_tag_name\"\n-\n-\t\techo \"$ref -> $new_ref ($sha1 -> $new_sha1)\"\n-\n-\t\tif [ \"$type\" = \"tag\" ]; then\n-\t\t\tnew_sha1=$( ( printf 'object %s\\ntype commit\\ntag %s\\n' \\\n-\t\t\t\t\t\t\"$new_sha1\" \"$new_ref\"\n-\t\t\t\tgit cat-file tag \"$ref\" |\n-\t\t\t\tsed -n \\\n-\t\t\t\t -e '1,/^$/{\n-\t\t\t\t\t /^object /d\n-\t\t\t\t\t /^type /d\n-\t\t\t\t\t /^tag /d\n-\t\t\t\t\t}' \\\n-\t\t\t\t -e '/^-----BEGIN PGP SIGNATURE-----/q' \\\n-\t\t\t\t -e 'p' ) |\n-\t\t\t\tgit hash-object -t tag -w --stdin) ||\n-\t\t\t\tdie \"Could not create new tag object for $ref\"\n-\t\t\tif git cat-file tag \"$ref\" | \\\n-\t\t\t sane_grep '^-----BEGIN PGP SIGNATURE-----' >/dev/null 2>&1\n-\t\t\tthen\n-\t\t\t\twarn \"gpg signature stripped from tag object $sha1t\"\n-\t\t\tfi\n-\t\tfi\n-\n-\t\tgit update-ref \"refs/tags/$new_ref\" \"$new_sha1\" ||\n-\t\t\tdie \"Could not write tag $new_ref\"\n-\tdone\n-fi\n-\n-unset GIT_DIR GIT_WORK_TREE GIT_INDEX_FILE\n-unset GIT_AUTHOR_NAME GIT_AUTHOR_EMAIL GIT_AUTHOR_DATE\n-unset GIT_COMMITTER_NAME GIT_COMMITTER_EMAIL GIT_COMMITTER_DATE\n-test -z \"$ORIG_GIT_DIR\" || {\n-\tGIT_DIR=\"$ORIG_GIT_DIR\" && export GIT_DIR\n-}\n-test -z \"$ORIG_GIT_WORK_TREE\" || {\n-\tGIT_WORK_TREE=\"$ORIG_GIT_WORK_TREE\" &&\n-\texport GIT_WORK_TREE\n-}\n-test -z \"$ORIG_GIT_INDEX_FILE\" || {\n-\tGIT_INDEX_FILE=\"$ORIG_GIT_INDEX_FILE\" &&\n-\texport GIT_INDEX_FILE\n-}\n-test -z \"$ORIG_GIT_AUTHOR_NAME\" || {\n-\tGIT_AUTHOR_NAME=\"$ORIG_GIT_AUTHOR_NAME\" &&\n-\texport GIT_AUTHOR_NAME\n-}\n-test -z \"$ORIG_GIT_AUTHOR_EMAIL\" || {\n-\tGIT_AUTHOR_EMAIL=\"$ORIG_GIT_AUTHOR_EMAIL\" &&\n-\texport GIT_AUTHOR_EMAIL\n-}\n-test -z \"$ORIG_GIT_AUTHOR_DATE\" || {\n-\tGIT_AUTHOR_DATE=\"$ORIG_GIT_AUTHOR_DATE\" &&\n-\texport GIT_AUTHOR_DATE\n-}\n-test -z \"$ORIG_GIT_COMMITTER_NAME\" || {\n-\tGIT_COMMITTER_NAME=\"$ORIG_GIT_COMMITTER_NAME\" &&\n-\texport GIT_COMMITTER_NAME\n-}\n-test -z \"$ORIG_GIT_COMMITTER_EMAIL\" || {\n-\tGIT_COMMITTER_EMAIL=\"$ORIG_GIT_COMMITTER_EMAIL\" &&\n-\texport GIT_COMMITTER_EMAIL\n-}\n-test -z \"$ORIG_GIT_COMMITTER_DATE\" || {\n-\tGIT_COMMITTER_DATE=\"$ORIG_GIT_COMMITTER_DATE\" &&\n-\texport GIT_COMMITTER_DATE\n-}\n-\n-if test -n \"$state_branch\"\n-then\n-\techo \"Saving rewrite state to $state_branch\" 1>&2\n-\tstate_blob=$(\n-\t\tperl -e'opendir D, \"../map\" or die;\n-\t\t\topen H, \"|-\", \"git hash-object -w --stdin\" or die;\n-\t\t\tforeach (sort readdir(D)) {\n-\t\t\t\tnext if m/^\\.\\.?$/;\n-\t\t\t\topen F, \"<../map/$_\" or die;\n-\t\t\t\tchomp($f = );\n-\t\t\t\tprint H \"$_:$f\\n\" or die;\n-\t\t\t}\n-\t\t\tclose(H) or die;' || die \"Unable to save state\")\n-\tstate_tree=$(printf '100644 blob %s\\tfilter.map\\n' \"$state_blob\" | git mktree)\n-\tif test -n \"$state_commit\"\n-\tthen\n-\t\tstate_commit=$(echo \"Sync\" | git commit-tree \"$state_tree\" -p \"$state_commit\")\n-\telse\n-\t\tstate_commit=$(echo \"Sync\" | git commit-tree \"$state_tree\" )\n-\tfi\n-\tgit update-ref \"$state_branch\" \"$state_commit\"\n-fi\n-\n-cd \"$orig_dir\"\n-rm -rf \"$tempdir\"\n-\n-trap - 0\n-\n-if [ \"$(is_bare_repository)\" = false ]; then\n-\tgit read-tree -u -m HEAD || exit\n-fi\n-\n-exit 0\ndiff --git a/t/perf/p7000-filter-branch.sh b/t/perf/p7000-filter-branch.sh\ndeleted file mode 100755\nindex b029586ccb..0000000000\n--- a/t/perf/p7000-filter-branch.sh\n+++ /dev/null\n@@ -1,24 +0,0 @@\n-#!/bin/sh\n-\n-test_description='performance of filter-branch'\n-. ./perf-lib.sh\n-\n-test_perf_default_repo\n-test_checkout_worktree\n-\n-test_expect_success 'mark bases for tests' '\n-\tgit tag -f tip &&\n-\tgit tag -f base HEAD~100\n-'\n-\n-test_perf 'noop filter' '\n-\tgit checkout --detach tip &&\n-\tgit filter-branch -f base..HEAD\n-'\n-\n-test_perf 'noop prune-empty' '\n-\tgit checkout --detach tip &&\n-\tgit filter-branch -f --prune-empty base..HEAD\n-'\n-\n-test_done\ndiff --git a/t/t7003-filter-branch.sh b/t/t7003-filter-branch.sh\ndeleted file mode 100755\nindex e23de7d0b5..0000000000\n--- a/t/t7003-filter-branch.sh\n+++ /dev/null\n@@ -1,505 +0,0 @@\n-#!/bin/sh\n-\n-test_description='git filter-branch'\n-. ./test-lib.sh\n-. \"$TEST_DIRECTORY/lib-gpg.sh\"\n-\n-test_expect_success 'setup' '\n-\ttest_commit A &&\n-\tGIT_COMMITTER_DATE=\"@0 +0000\" GIT_AUTHOR_DATE=\"@0 +0000\" &&\n-\ttest_commit --notick B &&\n-\tgit checkout -b branch B &&\n-\ttest_commit D &&\n-\tmkdir dir &&\n-\ttest_commit dir/D &&\n-\ttest_commit E &&\n-\tgit checkout master &&\n-\ttest_commit C &&\n-\tgit checkout branch &&\n-\tgit merge C &&\n-\tgit tag F &&\n-\ttest_commit G &&\n-\ttest_commit H\n-'\n-# * (HEAD, branch) H\n-# * G\n-# * Merge commit 'C' into branch\n-# |\\\n-# | * (master) C\n-# * | E\n-# * | dir/D\n-# * | D\n-# |/\n-# * B\n-# * A\n-\n-\n-H=$(git rev-parse H)\n-\n-test_expect_success 'rewrite identically' '\n-\tgit filter-branch branch\n-'\n-test_expect_success 'result is really identical' '\n-\ttest $H = $(git rev-parse HEAD)\n-'\n-\n-test_expect_success 'rewrite bare repository identically' '\n-\t(git config core.bare true && cd .git &&\n-\t git filter-branch branch > filter-output 2>&1 &&\n-\t! fgrep fatal filter-output)\n-'\n-git config core.bare false\n-test_expect_success 'result is really identical' '\n-\ttest $H = $(git rev-parse HEAD)\n-'\n-\n-TRASHDIR=$(pwd)\n-test_expect_success 'correct GIT_DIR while using -d' '\n-\tmkdir drepo &&\n-\t( cd drepo &&\n-\tgit init &&\n-\ttest_commit drepo &&\n-\tgit filter-branch -d \"$TRASHDIR/dfoo\" \\\n-\t\t--index-filter \"cp \\\"$TRASHDIR\\\"/dfoo/backup-refs \\\"$TRASHDIR\\\"\" \\\n-\t) &&\n-\tgrep drepo \"$TRASHDIR/backup-refs\"\n-'\n-\n-test_expect_success 'tree-filter works with -d' '\n-\tgit init drepo-tree &&\n-\t(\n-\t\tcd drepo-tree &&\n-\t\ttest_commit one &&\n-\t\tgit filter-branch -d \"$TRASHDIR/dfoo\" \\\n-\t\t\t--tree-filter \"echo changed >one.t\" &&\n-\t\techo changed >expect &&\n-\t\tgit cat-file blob HEAD:one.t >actual &&\n-\t\ttest_cmp expect actual &&\n-\t\ttest_cmp one.t actual\n-\t)\n-'\n-\n-test_expect_success 'Fail if commit filter fails' '\n-\ttest_must_fail git filter-branch -f --commit-filter \"exit 1\" HEAD\n-'\n-\n-test_expect_success 'rewrite, renaming a specific file' '\n-\tgit filter-branch -f --tree-filter \"mv D.t doh || :\" HEAD\n-'\n-\n-test_expect_success 'test that the file was renamed' '\n-\ttest D = \"$(git show HEAD:doh --)\" &&\n-\t! test -f D.t &&\n-\ttest -f doh &&\n-\ttest D = \"$(cat doh)\"\n-'\n-\n-test_expect_success 'rewrite, renaming a specific directory' '\n-\tgit filter-branch -f --tree-filter \"mv dir diroh || :\" HEAD\n-'\n-\n-test_expect_success 'test that the directory was renamed' '\n-\ttest dir/D = \"$(git show HEAD:diroh/D.t --)\" &&\n-\t! test -d dir &&\n-\ttest -d diroh &&\n-\t! test -d diroh/dir &&\n-\ttest -f diroh/D.t &&\n-\ttest dir/D = \"$(cat diroh/D.t)\"\n-'\n-\n-V=$(git rev-parse HEAD)\n-\n-test_expect_success 'populate --state-branch' '\n-\tgit filter-branch --state-branch state -f --tree-filter \"touch file || :\" HEAD\n-'\n-\n-W=$(git rev-parse HEAD)\n-\n-test_expect_success 'using --state-branch to skip already rewritten commits' '\n-\ttest_when_finished git reset --hard $V &&\n-\tgit reset --hard $V &&\n-\tgit filter-branch --state-branch state -f --tree-filter \"touch file || :\" HEAD &&\n-\ttest_cmp_rev $W HEAD\n-'\n-\n-git tag oldD HEAD~4\n-test_expect_success 'rewrite one branch, keeping a side branch' '\n-\tgit branch modD oldD &&\n-\tgit filter-branch -f --tree-filter \"mv B.t boh || :\" D..modD\n-'\n-\n-test_expect_success 'common ancestor is still common (unchanged)' '\n-\ttest \"$(git merge-base modD D)\" = \"$(git rev-parse B)\"\n-'\n-\n-test_expect_success 'filter subdirectory only' '\n-\tmkdir subdir &&\n-\ttouch subdir/new &&\n-\tgit add subdir/new &&\n-\ttest_tick &&\n-\tgit commit -m \"subdir\" &&\n-\techo H > A.t &&\n-\ttest_tick &&\n-\tgit commit -m \"not subdir\" A.t &&\n-\techo A > subdir/new &&\n-\ttest_tick &&\n-\tgit commit -m \"again subdir\" subdir/new &&\n-\tgit rm A.t &&\n-\ttest_tick &&\n-\tgit commit -m \"again not subdir\" &&\n-\tgit branch sub &&\n-\tgit branch sub-earlier HEAD~2 &&\n-\tgit filter-branch -f --subdirectory-filter subdir \\\n-\t\trefs/heads/sub refs/heads/sub-earlier\n-'\n-\n-test_expect_success 'subdirectory filter result looks okay' '\n-\ttest 2 = $(git rev-list sub | wc -l) &&\n-\tgit show sub:new &&\n-\ttest_must_fail git show sub:subdir &&\n-\tgit show sub-earlier:new &&\n-\ttest_must_fail git show sub-earlier:subdir\n-'\n-\n-test_expect_success 'more setup' '\n-\tgit checkout master &&\n-\tmkdir subdir &&\n-\techo A > subdir/new &&\n-\tgit add subdir/new &&\n-\ttest_tick &&\n-\tgit commit -m \"subdir on master\" subdir/new &&\n-\tgit rm A.t &&\n-\ttest_tick &&\n-\tgit commit -m \"again subdir on master\" &&\n-\tgit merge branch\n-'\n-\n-test_expect_success 'use index-filter to move into a subdirectory' '\n-\tgit branch directorymoved &&\n-\tgit filter-branch -f --index-filter \\\n-\t\t \"git ls-files -s | sed \\\"s-\t-&newsubdir/-\\\" |\n-\t GIT_INDEX_FILE=\\$GIT_INDEX_FILE.new \\\n-\t\t\tgit update-index --index-info &&\n-\t\t mv \\\"\\$GIT_INDEX_FILE.new\\\" \\\"\\$GIT_INDEX_FILE\\\"\" directorymoved &&\n-\tgit diff --exit-code HEAD directorymoved:newsubdir\n-'\n-\n-test_expect_success 'stops when msg filter fails' '\n-\told=$(git rev-parse HEAD) &&\n-\ttest_must_fail git filter-branch -f --msg-filter false HEAD &&\n-\ttest $old = $(git rev-parse HEAD) &&\n-\trm -rf .git-rewrite\n-'\n-\n-test_expect_success 'author information is preserved' '\n-\t: > i &&\n-\tgit add i &&\n-\ttest_tick &&\n-\tGIT_AUTHOR_NAME=\"B V Uips\" git commit -m bvuips &&\n-\tgit branch preserved-author &&\n-\t(sane_unset GIT_AUTHOR_NAME &&\n-\t git filter-branch -f --msg-filter \"cat; \\\n-\t\t\ttest \\$GIT_COMMIT != $(git rev-parse master) || \\\n-\t\t\techo Hallo\" \\\n-\t\tpreserved-author) &&\n-\tgit rev-list --author=\"B V Uips\" preserved-author >actual &&\n-\ttest_line_count = 1 actual\n-'\n-\n-test_expect_success \"remove a certain author's commits\" '\n-\techo i > i &&\n-\ttest_tick &&\n-\tgit commit -m i i &&\n-\tgit branch removed-author &&\n-\tgit filter-branch -f --commit-filter \"\\\n-\t\tif [ \\\"\\$GIT_AUTHOR_NAME\\\" = \\\"B V Uips\\\" ];\\\n-\t\tthen\\\n-\t\t\tskip_commit \\\"\\$@\\\";\n-\t\telse\\\n-\t\t\tgit commit-tree \\\"\\$@\\\";\\\n-\t\tfi\" removed-author &&\n-\tcnt1=$(git rev-list master | wc -l) &&\n-\tcnt2=$(git rev-list removed-author | wc -l) &&\n-\ttest $cnt1 -eq $(($cnt2 + 1)) &&\n-\tgit rev-list --author=\"B V Uips\" removed-author >actual &&\n-\ttest_line_count = 0 actual\n-'\n-\n-test_expect_success 'barf on invalid name' '\n-\ttest_must_fail git filter-branch -f master xy-problem &&\n-\ttest_must_fail git filter-branch -f HEAD^\n-'\n-\n-test_expect_success '\"map\" works in commit filter' '\n-\tgit filter-branch -f --commit-filter \"\\\n-\t\tparent=\\$(git rev-parse \\$GIT_COMMIT^) &&\n-\t\tmapped=\\$(map \\$parent) &&\n-\t\tactual=\\$(echo \\\"\\$@\\\" | sed \\\"s/^.*-p //\\\") &&\n-\t\ttest \\$mapped = \\$actual &&\n-\t\tgit commit-tree \\\"\\$@\\\";\" master~2..master &&\n-\tgit rev-parse --verify master\n-'\n-\n-test_expect_success 'Name needing quotes' '\n-\n-\tgit checkout -b rerere A &&\n-\tmkdir foo &&\n-\tname=\"れれれ\" &&\n-\t>foo/$name &&\n-\tgit add foo &&\n-\tgit commit -m \"Adding a file\" &&\n-\tgit filter-branch --tree-filter \"rm -fr foo\" &&\n-\ttest_must_fail git ls-files --error-unmatch \"foo/$name\" &&\n-\ttest $(git rev-parse --verify rerere) != $(git rev-parse --verify A)\n-\n-'\n-\n-test_expect_success 'Subdirectory filter with disappearing trees' '\n-\tgit reset --hard &&\n-\tgit checkout master &&\n-\n-\tmkdir foo &&\n-\ttouch foo/bar &&\n-\tgit add foo &&\n-\ttest_tick &&\n-\tgit commit -m \"Adding foo\" &&\n-\n-\tgit rm -r foo &&\n-\ttest_tick &&\n-\tgit commit -m \"Removing foo\" &&\n-\n-\tmkdir foo &&\n-\ttouch foo/bar &&\n-\tgit add foo &&\n-\ttest_tick &&\n-\tgit commit -m \"Re-adding foo\" &&\n-\n-\tgit filter-branch -f --subdirectory-filter foo &&\n-\tgit rev-list master >actual &&\n-\ttest_line_count = 3 actual\n-'\n-\n-test_expect_success 'Tag name filtering retains tag message' '\n-\tgit tag -m atag T &&\n-\tgit cat-file tag T > expect &&\n-\tgit filter-branch -f --tag-name-filter cat &&\n-\tgit cat-file tag T > actual &&\n-\ttest_cmp expect actual\n-'\n-\n-faux_gpg_tag='object XXXXXX\n-type commit\n-tag S\n-tagger T A Gger 1206026339 -0500\n-\n-This is a faux gpg signed tag.\n------BEGIN PGP SIGNATURE-----\n-Version: FauxGPG v0.0.0 (FAUX/Linux)\n-\n-gdsfoewhxu/6l06f1kxyxhKdZkrcbaiOMtkJUA9ITAc1mlamh0ooasxkH1XwMbYQ\n-acmwXaWET20H0GeAGP+7vow=\n-=agpO\n------END PGP SIGNATURE-----\n-'\n-test_expect_success 'Tag name filtering strips gpg signature' '\n-\tsha1=$(git rev-parse HEAD) &&\n-\tsha1t=$(echo \"$faux_gpg_tag\" | sed -e s/XXXXXX/$sha1/ | git mktag) &&\n-\tgit update-ref \"refs/tags/S\" \"$sha1t\" &&\n-\techo \"$faux_gpg_tag\" | sed -e s/XXXXXX/$sha1/ | head -n 6 > expect &&\n-\tgit filter-branch -f --tag-name-filter cat &&\n-\tgit cat-file tag S > actual &&\n-\ttest_cmp expect actual\n-'\n-\n-test_expect_success GPG 'Filtering retains message of gpg signed commit' '\n-\tmkdir gpg &&\n-\ttouch gpg/foo &&\n-\tgit add gpg &&\n-\ttest_tick &&\n-\tgit commit -S -m \"Adding gpg\" &&\n-\n-\tgit log -1 --format=\"%s\" > expect &&\n-\tgit filter-branch -f --msg-filter \"cat\" &&\n-\tgit log -1 --format=\"%s\" > actual &&\n-\ttest_cmp expect actual\n-'\n-\n-test_expect_success 'Tag name filtering allows slashes in tag names' '\n-\tgit tag -m tag-with-slash X/1 &&\n-\tgit cat-file tag X/1 | sed -e s,X/1,X/2, > expect &&\n-\tgit filter-branch -f --tag-name-filter \"echo X/2\" &&\n-\tgit cat-file tag X/2 > actual &&\n-\ttest_cmp expect actual\n-'\n-test_expect_success 'setup --prune-empty comparisons' '\n-\tgit checkout --orphan master-no-a &&\n-\tgit rm -rf . &&\n-\tunset test_tick &&\n-\ttest_tick &&\n-\tGIT_COMMITTER_DATE=\"@0 +0000\" GIT_AUTHOR_DATE=\"@0 +0000\" &&\n-\ttest_commit --notick B B.t B Bx &&\n-\tgit checkout -b branch-no-a Bx &&\n-\ttest_commit D D.t D Dx &&\n-\tmkdir dir &&\n-\ttest_commit dir/D dir/D.t dir/D dir/Dx &&\n-\ttest_commit E E.t E Ex &&\n-\tgit checkout master-no-a &&\n-\ttest_commit C C.t C Cx &&\n-\tgit checkout branch-no-a &&\n-\tgit merge Cx -m \"Merge tag '\\''C'\\'' into branch\" &&\n-\tgit tag Fx &&\n-\ttest_commit G G.t G Gx &&\n-\ttest_commit H H.t H Hx &&\n-\tgit checkout branch\n-'\n-\n-test_expect_success 'Prune empty commits' '\n-\tgit rev-list HEAD > expect &&\n-\ttest_commit to_remove &&\n-\tgit filter-branch -f --index-filter \"git update-index --remove to_remove.t\" --prune-empty HEAD &&\n-\tgit rev-list HEAD > actual &&\n-\ttest_cmp expect actual\n-'\n-\n-test_expect_success 'prune empty collapsed merges' '\n-\ttest_config merge.ff false &&\n-\tgit rev-list HEAD >expect &&\n-\ttest_commit to_remove_2 &&\n-\tgit reset --hard HEAD^ &&\n-\ttest_merge non-ff to_remove_2 &&\n-\tgit filter-branch -f --index-filter \"git update-index --remove to_remove_2.t\" --prune-empty HEAD &&\n-\tgit rev-list HEAD >actual &&\n-\ttest_cmp expect actual\n-'\n-\n-test_expect_success 'prune empty works even without index/tree filters' '\n-\tgit rev-list HEAD >expect &&\n-\tgit commit --allow-empty -m empty &&\n-\tgit filter-branch -f --prune-empty HEAD &&\n-\tgit rev-list HEAD >actual &&\n-\ttest_cmp expect actual\n-'\n-\n-test_expect_success '--prune-empty is able to prune root commit' '\n-\tgit rev-list branch-no-a >expect &&\n-\tgit branch testing H &&\n-\tgit filter-branch -f --prune-empty --index-filter \"git update-index --remove A.t\" testing &&\n-\tgit rev-list testing >actual &&\n-\tgit branch -D testing &&\n-\ttest_cmp expect actual\n-'\n-\n-test_expect_success '--prune-empty is able to prune entire branch' '\n-\tgit branch prune-entire B &&\n-\tgit filter-branch -f --prune-empty --index-filter \"git update-index --remove A.t B.t\" prune-entire &&\n-\ttest_path_is_missing .git/refs/heads/prune-entire &&\n-\ttest_must_fail git reflog exists refs/heads/prune-entire\n-'\n-\n-test_expect_success '--remap-to-ancestor with filename filters' '\n-\tgit checkout master &&\n-\tgit reset --hard A &&\n-\ttest_commit add-foo foo 1 &&\n-\tgit branch moved-foo &&\n-\ttest_commit add-bar bar a &&\n-\tgit branch invariant &&\n-\torig_invariant=$(git rev-parse invariant) &&\n-\tgit branch moved-bar &&\n-\ttest_commit change-foo foo 2 &&\n-\tgit filter-branch -f --remap-to-ancestor \\\n-\t\tmoved-foo moved-bar A..master \\\n-\t\t-- -- foo &&\n-\ttest $(git rev-parse moved-foo) = $(git rev-parse moved-bar) &&\n-\ttest $(git rev-parse moved-foo) = $(git rev-parse master^) &&\n-\ttest $orig_invariant = $(git rev-parse invariant)\n-'\n-\n-test_expect_success 'automatic remapping to ancestor with filename filters' '\n-\tgit checkout master &&\n-\tgit reset --hard A &&\n-\ttest_commit add-foo2 foo 1 &&\n-\tgit branch moved-foo2 &&\n-\ttest_commit add-bar2 bar a &&\n-\tgit branch invariant2 &&\n-\torig_invariant=$(git rev-parse invariant2) &&\n-\tgit branch moved-bar2 &&\n-\ttest_commit change-foo2 foo 2 &&\n-\tgit filter-branch -f \\\n-\t\tmoved-foo2 moved-bar2 A..master \\\n-\t\t-- -- foo &&\n-\ttest $(git rev-parse moved-foo2) = $(git rev-parse moved-bar2) &&\n-\ttest $(git rev-parse moved-foo2) = $(git rev-parse master^) &&\n-\ttest $orig_invariant = $(git rev-parse invariant2)\n-'\n-\n-test_expect_success 'setup submodule' '\n-\trm -fr ?* .git &&\n-\tgit init &&\n-\ttest_commit file &&\n-\tmkdir submod &&\n-\tsubmodurl=\"$PWD/submod\" &&\n-\t( cd submod &&\n-\t git init &&\n-\t test_commit file-in-submod ) &&\n-\tgit submodule add \"$submodurl\" &&\n-\tgit commit -m \"added submodule\" &&\n-\ttest_commit add-file &&\n-\t( cd submod && test_commit add-in-submodule ) &&\n-\tgit add submod &&\n-\tgit commit -m \"changed submodule\" &&\n-\tgit branch original HEAD\n-'\n-\n-orig_head=$(git show-ref --hash --head HEAD)\n-\n-test_expect_success 'rewrite submodule with another content' '\n-\tgit filter-branch --tree-filter \"test -d submod && {\n-\t\t\t\t\t rm -rf submod &&\n-\t\t\t\t\t git rm -rf --quiet submod &&\n-\t\t\t\t\t mkdir submod &&\n-\t\t\t\t\t : > submod/file\n-\t\t\t\t\t } || :\" HEAD &&\n-\ttest $orig_head != $(git show-ref --hash --head HEAD)\n-'\n-\n-test_expect_success 'replace submodule revision' '\n-\tgit reset --hard original &&\n-\tgit filter-branch -f --tree-filter \\\n-\t \"if git ls-files --error-unmatch -- submod > /dev/null 2>&1\n-\t then git update-index --cacheinfo 160000 0123456789012345678901234567890123456789 submod\n-\t fi\" HEAD &&\n-\ttest $orig_head != $(git show-ref --hash --head HEAD)\n-'\n-\n-test_expect_success 'filter commit message without trailing newline' '\n-\tgit reset --hard original &&\n-\tcommit=$(printf \"no newline\" | git commit-tree HEAD^{tree}) &&\n-\tgit update-ref refs/heads/no-newline $commit &&\n-\tgit filter-branch -f refs/heads/no-newline &&\n-\techo $commit >expect &&\n-\tgit rev-parse refs/heads/no-newline >actual &&\n-\ttest_cmp expect actual\n-'\n-\n-test_expect_success 'tree-filter deals with object name vs pathname ambiguity' '\n-\ttest_when_finished \"git reset --hard original\" &&\n-\tambiguous=$(git rev-list -1 HEAD) &&\n-\tgit filter-branch --tree-filter \"mv file.t $ambiguous\" HEAD^.. &&\n-\tgit show HEAD:$ambiguous\n-'\n-\n-test_expect_success 'rewrite repository including refs that point at non-commit object' '\n-\ttest_when_finished \"git reset --hard original\" &&\n-\ttree=$(git rev-parse HEAD^{tree}) &&\n-\ttest_when_finished \"git replace -d $tree\" &&\n-\techo A >new &&\n-\tgit add new &&\n-\tnew_tree=$(git write-tree) &&\n-\tgit replace $tree $new_tree &&\n-\tgit tag -a -m \"tag to a tree\" treetag $new_tree &&\n-\tgit reset --hard HEAD &&\n-\tgit filter-branch -f -- --all >filter-output 2>&1 &&\n-\t! fgrep fatal filter-output\n-'\n-\n-test_done\ndiff --git a/t/t7009-filter-branch-null-sha1.sh b/t/t7009-filter-branch-null-sha1.sh\ndeleted file mode 100755\nindex 9ba9f24ad2..0000000000\n--- a/t/t7009-filter-branch-null-sha1.sh\n+++ /dev/null\n@@ -1,55 +0,0 @@\n-#!/bin/sh\n-\n-test_description='filter-branch removal of trees with null sha1'\n-. ./test-lib.sh\n-\n-test_expect_success 'setup: base commits' '\n-\ttest_commit one &&\n-\ttest_commit two &&\n-\ttest_commit three\n-'\n-\n-test_expect_success 'setup: a commit with a bogus null sha1 in the tree' '\n-\t{\n-\t\tgit ls-tree HEAD &&\n-\t\tprintf \"160000 commit $ZERO_OID\\\\tbroken\\\\n\"\n-\t} >broken-tree &&\n-\techo \"add broken entry\" >msg &&\n-\n-\ttree=$(git mktree expect &&\n-\tgit filter-branch \\\n-\t\t--force --prune-empty \\\n-\t\t--index-filter \"git rm --cached --ignore-unmatch broken\" &&\n-\tgit log -1 --format=%s >actual &&\n-\ttest_cmp expect actual\n-'\n-\n-test_done\ndiff --git a/t/t9902-completion.sh b/t/t9902-completion.sh\nindex 75512c3403..4e7f669c76 100755\n--- a/t/t9902-completion.sh\n+++ b/t/t9902-completion.sh\n@@ -28,10 +28,10 @@ complete ()\n #\n # (2) A test makes sure that common subcommands are included in the\n # completion for \"git \", and a plumbing is excluded. \"add\",\n-# \"filter-branch\" and \"ls-files\" are listed for this.\n+# \"rebase\" and \"ls-files\" are listed for this.\n \n-GIT_TESTING_ALL_COMMAND_LIST='add checkout check-attr filter-branch ls-files'\n-GIT_TESTING_PORCELAIN_COMMAND_LIST='add checkout filter-branch'\n+GIT_TESTING_ALL_COMMAND_LIST='add checkout check-attr rebase ls-files'\n+GIT_TESTING_PORCELAIN_COMMAND_LIST='add checkout rebase'\n \n . \"$GIT_BUILD_DIR/contrib/completion/git-completion.bash\"\n \n@@ -1392,12 +1392,12 @@ test_expect_success 'basic' '\n \t# built-in\n \tgrep -q \"^add \\$\" out &&\n \t# script\n-\tgrep -q \"^filter-branch \\$\" out &&\n+\tgrep -q \"^rebase \\$\" out &&\n \t# plumbing\n \t! grep -q \"^ls-files \\$\" out &&\n \n-\trun_completion \"git f\" &&\n-\t! grep -q -v \"^f\" out\n+\trun_completion \"git r\" &&\n+\t! grep -q -v \"^r\" out\n '\n \n test_expect_success 'double dash \"git\" itself' '\n-- \n2.23.0.3.gcc10030edf.dirty\n\n"},{"id":"381418","messageId":"CAPig+cQco8vtj498G_3tbsF_PFxXidUPgPZtOdNYqySGuz5c_A@mail.gmail.com","threadId":"51710","inReplyTo":"20190828002210.8862-3-newren@gmail.com","subject":"Re: [PATCH v2 2/4] t3427: accelerate this test by using fast-export and fast-import","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2019-08-28T06:00:38Z","receivedAt":"2019-08-28T06:00:52Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Tue, Aug 27, 2019 at 8:22 PM Elijah Newren wrote:\n> fast-export and fast-import can easily handle the simple rewrite that\n> was being done by filter-branch, and should be significantly faster on\n> systems with a slow fork. Timings from before and after on two laptops\n> that I have access to (measured via `time ./t3427-rebase-subtree.sh`,\n> i.e. including everything in this test -- not just the filter-branch or\n> fast-export/fast-import pair):\n> [...]\n> Signed-off-by: Elijah Newren \n> ---\n> diff --git a/t/t3427-rebase-subtree.sh b/t/t3427-rebase-subtree.sh\n> @@ -11,6 +11,12 @@ commit_message() {\n> +extract_files_subtree() {\n\nStyle nit: add space before opening '('\n\n(However, commit_message() function just above this doesn't follow\nthat style, so...)\n\n> + git fast-export --no-data HEAD -- files_subtree/ \\\n> + | sed -e \"s%\\([0-9a-f]\\{40\\} \\)files_subtree/%\\1%\" \\\n> + | git fast-import --force --quiet\n\nThis would be a bit less noisy if you ended each line with the pipe\noperator, allowing you to drop the backslashes:\n\n git fast-export --no-data HEAD -- files_subtree/ |\n sed -e \"s%\\([0-9a-f]\\{40\\} \\)files_subtree/%\\1%\" |\n git fast-import --force --quiet\n\n> +}\n\nNot sure any of this is worth a re-roll.\n"},{"id":"381419","messageId":"CAPig+cSydZnie3CSwPgfSaNcGxr1jj94YBXAL47vEXrtxaQXUw@mail.gmail.com","threadId":"51710","inReplyTo":"20190828002210.8862-4-newren@gmail.com","subject":"Re: [PATCH v2 3/4] Recommend git-filter-repo instead of git-filter-branch","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2019-08-28T06:17:47Z","receivedAt":"2019-08-28T06:18:02Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Tue, Aug 27, 2019 at 8:22 PM Elijah Newren wrote:\n> filter-branch suffers from a deluge of disguised dangers that disfigure\n> history rewrites (i.e. deviate from the deliberate changes). [...]\n>\n> Signed-off-by: Elijah Newren \n> ---\n> diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt\n> @@ -16,6 +16,20 @@ SYNOPSIS\n> +WARNING\n> +-------\n> +'git filter-branch' has a plethora of pitfalls that can produce non-obvious\n> +manglings of the intended history rewrite (and can leave you with little\n> +time to investigate such problems since it has such abysmal performance).\n> +These safety and performance issues cannot be backward compatibly fixed and\n> +as such, its use is not recommended. Please use an alternative history\n> +filtering tool such as https://github.com/newren/git-filter-repo/[git\n> +filter-repo]. If you still need to use 'git filter-branch', please\n> +carefully read the \"Safety\" section of the message on the Git mailing list\n> +https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/[detailing\n> +the land mines of filter-branch] and vigilantly avoid as many of the\n> +hazards listed there as reasonably possible.\n\nIs there a good reason to not simply copy the \"Safety\" section from\nthat email directly into this document so that readers don't have to\ngo chasing down the link (especially those who are reading\ndocumentation offline)?\n\n> diff --git a/Documentation/git-rebase.txt b/Documentation/git-rebase.txt\n> @@ -832,7 +832,7 @@ Hard case: The changes are not the same.::\n> This happens if the 'subsystem' rebase had conflicts, or used\n> `--interactive` to omit, edit, squash, or fixup commits; or\n> if the upstream used one of `commit --amend`, `reset`, or\n> - `filter-branch`.\n> + a full history rewriting command like `filter-repo`.\n\nDo we want a clickable link to `filter-repo` here?\n\n> diff --git a/Documentation/git-replace.txt b/Documentation/git-replace.txt\n> @@ -123,10 +123,10 @@ The following format are available:\n> +linkgit:git-hash-object[1], linkgit:git-rebase[1], and\n> +linkgit:git-filter-repo[1], among other git commands, can be used to\n> [...]\n> @@ -148,8 +148,8 @@ pending objects.\n> linkgit:git-hash-object[1]\n> linkgit:git-rebase[1]\n> +linkgit:git-filter-repo[1]\n\nAre these 'linkgit:' references to `filter-repo` going to be\nmeaningful if that tool is not incorporated into the Git project\nproper? Perhaps use a generic clickable link instead.\n\nSame comment applies to other 'linkgit:' invocations in the remainder\nof the patch.\n\n> diff --git a/git-filter-branch.sh b/git-filter-branch.sh\n> old mode 100755\n> new mode 100644\n\nWhy lose the executable bit?\n\n> @@ -83,6 +83,19 @@ set_ident () {\n> +if [ -z \"$FILTER_BRANCH_SQUELCH_WARNING\" -a \\\n> + -z \"$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS\" ]; then\n\nIf this script didn't already have a mix of styles, I'd say something\nabout modern style being:\n\n if test -z \"$FILTER_BRANCH_SQUELCH_WARNING\" &&\n test -z \"$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS\"\n then\n ...\n fi\n\n> + cat < +WARNING: git-filter-branch has a glut of gotchas generating mangled history\n> + rewrites. Please use an alternative filtering tool such as 'git\n> + filter-repo' (https://github.com/newren/git-filter-repo/) instead.\n> + See the filter-branch manual page for more details; to squelch\n> + this warning and pause, set FILTER_BRANCH_SQUELCH_WARNING=1.\n\nThe \"and pause\" threw me. There's more than a bit of ambiguity\nsurrounding it. Perhaps drop it?\n"},{"id":"381424","messageId":"87blw93c5n.fsf@osv.gnss.ru","threadId":"51710","inReplyTo":"CABPp-BGfoRX-ob986kd-vvBo3meg-MpJ8Jo85G_2GgARY=cxxg@mail.gmail.com","subject":"Re: [RFC PATCH 0/5] Remove git-filter-branch from git.git; host it elsewhere","fromName":"Sergey Organov","fromEmail":"sorganov@gmail.com","sentAt":"2019-08-28T08:52:04Z","receivedAt":"2019-08-28T08:52:11Z","isPatch":true,"sender":{"key":"sorganov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8501568?v=4"},"body":"Elijah Newren writes:\n\n> On Tue, Aug 27, 2019 at 1:43 AM Sergey Organov wrote:\n>>\n>> Eric Wong writes:\n>>\n>>\n>> [...]\n>>\n>> > AFAIK, filter-branch is not causing support headaches for any\n>> > git developers today. With so many commands in git, it's\n>> > unlikely newbies will ever get around to discover it :)\n>> > So I think think we should be in any rush to remove it.\n>>\n>> Nah, discovering it is simple. Just Google for \"git change author\". That\n>> eventually leads to a script that uses \"git filter-branch --env-filter\"\n>> to get the job done, and I'm afraid it is spread all over the world.\n>>\n>> See, e.g.:\n>>\n>> https://help.github.com/en/articles/changing-author-info\n>\n> Side note: Is the goal to \"fix names and email addresses in this\n> repository\"? If so, this guide fails: it doesn't update tagger names\n> or email addresses. Indeed, filter-branch doesn't provide a way to do\n> that. (Not to mention other problems like not updating references to\n> commit hashes in commit messages when it busy rewriting everything.)\n\nNo. Maybe the original goal was like that, by I, personally, use\nmodified version of this to change my \"Author\" credentials from\n\"internal\" to \"public\" in branches that I'm going to send upstream, so\nthe actual aim is to change e-mail of particular Author from a@b to c@d\nin all the commits in a (feature) branch.\n\n>\n>> > But I agree that filter-branch isn't useful and certainly\n>> > shouldn't be encouraged/promoted.\n>>\n>> Well, is there more suitable way to change author for a (large) set of\n>> commits then?\n>\n> I would say yes, use git filter-repo (note that this thread started\n> with me proposing filter-repo for inclusion in git.git -- and getting\n> suggestions that we should remove stuff instead of adding more stuff).\n> I'm biased, but I think it's much better at this particular job as\n> well:\n\nWell, I don't want to change the entire repo, and I don't immediately\nsee how to do it with git filter-repo. Is it at all possible?\n\n> You can create a mailmap file and pass it to the --mailmap option to\n> git-filter-repo.\n>\n> Or, if you prefer (perhaps you don't like git's mailmap format as used\n> by shortlog and now log, or perhaps you really want to be able to do\n> regex replacement or something), you can use the --name-callback or\n> --email-callback to work on those fields more directly.\n>\n> Or, if you prefer (e.g. you want to handle author vs. committer vs.\n> tagger differently), you can use the --commit-callback and\n> --tag-callback filters.\n>\n> As an added bonus, filter-repo will also perform the rewrite far\n> faster than filter-branch (and rewrite commit hashes in commit\n> messages as alluded to above).\n\nThese things are nice to have indeed, but it always changes the entire\nrepo, right? If so, it's not a suitable substitute for git-filter-branch\nfor particular job at hand.\n\nActually, I'd rather expect some support for this in \"git rebase\", being\ngit history editing/reshaping tool, but it looks like it only has it in\nthe form that is very difficult to automate.\n\n>\n>> > Yet there's probably still users which ARE happy with it, that\n>> > will never hit the edge cases and problems it poses; and will\n>> > never read release notes. And said users are probably getting\n>> > git from a slow-moving distro, so it'd be a disservice to them\n>> > if they lost a tool they depend on without any warning.\n>>\n>> Personally, I'm far from happy with it, but I have no clue how to\n>> substitute it in the job above. Anybody?\n>\n> The start of this thread where I proposed git filter-repo for\n> inclusion in git[1] had links to documentation and comparisons to\n> other tools and such. You may find those links helpful; if not, let\n> me know what needs to be fixed in the documentation.\n\nThank you for the references, I find it a very nice tool to have!\n\nPity it's not an entire substitute for git filter-branch.\n\n-- Sergey\n"},{"id":"381433","messageId":"nycvar.QRO.7.76.6.1908281307250.46@tvgsbejvaqbjf.bet","threadId":"51710","inReplyTo":"CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com","subject":"Re: RFC: Proposing git-filter-repo for inclusion in git.git","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-08-28T11:09:21Z","receivedAt":"2019-08-28T11:10:03Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Elijah,\n\nOn Fri, 23 Aug 2019, Elijah Newren wrote:\n\n> On Thu, Aug 22, 2019 at 8:01 PM Eric Wong wrote:\n> >\n> > Elijah Newren wrote:\n> > > * Remove git-filter-branch from git.git. Mention in the release\n> > > notes where people can go to get it.[1]\n> > >\n> > > filter-branch is not merely a slow or difficult-to-use tool, it's one\n> > > that *fosters* mistakes by making it hard to get things right in\n> > > several different ways. Granted, people exercise extra caution using\n> > > filter-branch because they know they need to, but there are so many\n> > > gotchas that they're likely to accidentally mess something up. Those\n> > > mess-ups are not always discovered immediately, and by then it's\n> > > nearly cast into stone (rewriting being something you want to do very\n> > > rarely).\n> >\n> > Is it possible to turn git-filter-branch into a fast, compatible,\n> > and (maybe) safe wrapper for git-filter-repo? That would \"fix\"\n> > filter-branch and (if done carefully) not break existing uses.\n>\n> Ooh, what an interesting question. I can probably ramble on a LOT\n> longer than you expected about this...\n>\n> [...]\n\nFWIW if anybody cares about my opinion: I would be totally fine with\nintegrating git-filter-repo into git.git, have it there for a major\nversion or two, then patch `git filter-branch` to spew out a deprecation\nwarning, and then remove that latter command a major version (or two)\nlater.\n\nCiao,\nDscho\n"},{"id":"381447","messageId":"xmqqsgpl1g8i.fsf@gitster-ct.c.googlers.com","threadId":"51710","inReplyTo":"nycvar.QRO.7.76.6.1908281307250.46@tvgsbejvaqbjf.bet","subject":"Re: RFC: Proposing git-filter-repo for inclusion in git.git","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-08-28T15:06:53Z","receivedAt":"2019-08-28T15:07:02Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> FWIW if anybody cares about my opinion: I would be totally fine with\n> integrating git-filter-repo into git.git, have it there for a major\n> version or two, then patch `git filter-branch` to spew out a deprecation\n> warning, and then remove that latter command a major version (or two)\n> later.\n\nYup, that's just the usual deprecate then delete sequence. The\ncompatibility wrapper brought up in the discussion earlier would be\na big plus ;-)\n"},{"id":"381462","messageId":"CABPp-BEYRmhrb4Tx3bGzkx8y53T_0BYhLE5J0cEmxj18WtZs9A@mail.gmail.com","threadId":"51710","inReplyTo":"87blw93c5n.fsf@osv.gnss.ru","subject":"Re: [RFC PATCH 0/5] Remove git-filter-branch from git.git; host it elsewhere","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-28T17:16:18Z","receivedAt":"2019-08-28T17:16:32Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Sergey,\n\nOn Wed, Aug 28, 2019 at 1:52 AM Sergey Organov wrote:\n>\n> Elijah Newren writes:\n>\n> > On Tue, Aug 27, 2019 at 1:43 AM Sergey Organov wrote:\n> >>\n> >> Eric Wong writes:\n> >>\n> >>\n> >> [...]\n> >>\n> >> > AFAIK, filter-branch is not causing support headaches for any\n> >> > git developers today. With so many commands in git, it's\n> >> > unlikely newbies will ever get around to discover it :)\n> >> > So I think think we should be in any rush to remove it.\n> >>\n> >> Nah, discovering it is simple. Just Google for \"git change author\". That\n> >> eventually leads to a script that uses \"git filter-branch --env-filter\"\n> >> to get the job done, and I'm afraid it is spread all over the world.\n> >>\n> >> See, e.g.:\n> >>\n> >> https://help.github.com/en/articles/changing-author-info\n> >\n> > Side note: Is the goal to \"fix names and email addresses in this\n> > repository\"? If so, this guide fails: it doesn't update tagger names\n> > or email addresses. Indeed, filter-branch doesn't provide a way to do\n> > that. (Not to mention other problems like not updating references to\n> > commit hashes in commit messages when it busy rewriting everything.)\n>\n> No. Maybe the original goal was like that, by I, personally, use\n> modified version of this to change my \"Author\" credentials from\n> \"internal\" to \"public\" in branches that I'm going to send upstream, so\n> the actual aim is to change e-mail of particular Author from a@b to c@d\n> in all the commits in a (feature) branch.\n\nThere's an interesting usecase I hadn't heard of or thought of before.\nQuick question to see if I'm understanding correctly: \"all commits in\na branch\" or \"all commits *unique* to a branch\"?\n\n(Perhaps the only commits with the author you want to change are among\nthe commits that are unique to that branch and so the distinction\ndoesn't matter, but it wasn't clear from the description.)\n\n> >> > But I agree that filter-branch isn't useful and certainly\n> >> > shouldn't be encouraged/promoted.\n> >>\n> >> Well, is there more suitable way to change author for a (large) set of\n> >> commits then?\n> >\n> > I would say yes, use git filter-repo (note that this thread started\n> > with me proposing filter-repo for inclusion in git.git -- and getting\n> > suggestions that we should remove stuff instead of adding more stuff).\n> > I'm biased, but I think it's much better at this particular job as\n> > well:\n>\n> Well, I don't want to change the entire repo, and I don't immediately\n> see how to do it with git filter-repo. Is it at all possible?\n\nYes, it is possible. filter-repo has a hidden --refs argument\ndefaulting to --all; you could instead set it to e.g.\norigin/master..master.\n\n--refs is the only hidden option in filter-repo. I know it may look\nfunny that I spent a bunch of effort to create the\n--reference-excluded-parents option to fast-export explicitly so that\nit would be possible to do partial history rewrites like this, and\nthen to hide and avoid documenting this option (though I did hint that\nit existed in the documentation if you search for \"Partial-repo\nfiltering\"), but there was a few reasons for this:\n\n * mixing old and new history for most rewrites that\nfilter-branch/filter-repo/bfg/etc are used for can really mess things\nup and make it hard to recover from. I don't like trying to clean up\nrepos with accidental duplicate copies of most commits in the repo,\nand I suspect others like it even less. So, anything that makes it\neasier to make such mistakes needs to have a really good rationale in\norder for me to expose it.\n * The only usecases I knew of for partial repo filtering prior to\nyour email were (1) side-stepping insanely slow execution time of poor\nfiltering tools like filter-branch, and (2) performing operations\nbetter suited to git-rebase anyway (e.g. the --signoff option to\nrebase did not exist once upon a time and so folks could have used\nfilter-branch to fake it, but using rebase is the better way to make\nthis change). And, even after your email, I'm not sure that has\nchanged though, as noted below.\n\n> > You can create a mailmap file and pass it to the --mailmap option to\n> > git-filter-repo.\n> >\n> > Or, if you prefer (perhaps you don't like git's mailmap format as used\n> > by shortlog and now log, or perhaps you really want to be able to do\n> > regex replacement or something), you can use the --name-callback or\n> > --email-callback to work on those fields more directly.\n> >\n> > Or, if you prefer (e.g. you want to handle author vs. committer vs.\n> > tagger differently), you can use the --commit-callback and\n> > --tag-callback filters.\n> >\n> > As an added bonus, filter-repo will also perform the rewrite far\n> > faster than filter-branch (and rewrite commit hashes in commit\n> > messages as alluded to above).\n>\n> These things are nice to have indeed, but it always changes the entire\n> repo, right? If so, it's not a suitable substitute for git-filter-branch\n> for particular job at hand.\n\nIt *defaults* to changing the entire repo. You may also be interested\nto note that two of the demos in contrib/filter-repo-demos[1]\nexplicitly make use of partial history filtering -- one of the two\nbeing the filter-branch reimplementation (I created a script that\nreimplemented filter-branch on top of filter-repo and made it accept\nthe exact same flags as filter-branch. That script passes all the\nfilter-branch regression tests from git.git and it's much faster than\nfilter-branch, though it's still glacially slow compared to\nfilter-repo, and has all the safety problems that filter-branch does).\n\n[1] https://github.com/newren/git-filter-repo/tree/master/contrib/filter-repo-demos\n\n> Actually, I'd rather expect some support for this in \"git rebase\", being\n> git history editing/reshaping tool, but it looks like it only has it in\n> the form that is very difficult to automate.\n\nI agree that git rebase would be the better choice here; I typically\nfeel it's the better choice for rewrites of recent history. I think\nit provides just what you need:\n\n git rebase --exec=\"git commit --amend --reset-author -C HEAD\" $UPSTREAM\n\n(Assuming, of course, that you've either set the right environment\nvariables or set user.name and user.email to the new values you want\nso that commit's --reset-author flag can reset to the *new* author.)\n\n> >> > Yet there's probably still users which ARE happy with it, that\n> >> > will never hit the edge cases and problems it poses; and will\n> >> > never read release notes. And said users are probably getting\n> >> > git from a slow-moving distro, so it'd be a disservice to them\n> >> > if they lost a tool they depend on without any warning.\n> >>\n> >> Personally, I'm far from happy with it, but I have no clue how to\n> >> substitute it in the job above. Anybody?\n> >\n> > The start of this thread where I proposed git filter-repo for\n> > inclusion in git[1] had links to documentation and comparisons to\n> > other tools and such. You may find those links helpful; if not, let\n> > me know what needs to be fixed in the documentation.\n>\n> Thank you for the references, I find it a very nice tool to have!\n>\n> Pity it's not an entire substitute for git filter-branch.\n\nAu contraire, I believe it is. :-)\n\nThanks for the interesting usecase. It sounds like we both think this\none happens to be better solved by rebase, and the command snippet I\nprovided above should show you to use rebase to solve it. However, if\nyou come up with any others where partial repo filtering makes sense,\nI'm always willing to reconsider my decision to make the --refs\nargument hidden; it may just mean adding more warnings, but it might\nalso involve changing other defaults (e.g. the automatic\nrepacking/pruning). I'd need concrete usecases to know for sure how\nI'd want to handle it.\n\nHope that helps,\nElijah\n"},{"id":"381464","messageId":"87muftqfit.fsf@osv.gnss.ru","threadId":"51710","inReplyTo":"CABPp-BEYRmhrb4Tx3bGzkx8y53T_0BYhLE5J0cEmxj18WtZs9A@mail.gmail.com","subject":"Re: [RFC PATCH 0/5] Remove git-filter-branch from git.git; host it elsewhere","fromName":"Sergey Organov","fromEmail":"sorganov@gmail.com","sentAt":"2019-08-28T19:03:06Z","receivedAt":"2019-08-28T19:03:12Z","isPatch":true,"sender":{"key":"sorganov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8501568?v=4"},"body":"Hi Elijah,\n\nElijah Newren writes:\n\n> Hi Sergey,\n>\n> On Wed, Aug 28, 2019 at 1:52 AM Sergey Organov wrote:\n>>\n>> Elijah Newren writes:\n>>\n>> > On Tue, Aug 27, 2019 at 1:43 AM Sergey Organov wrote:\n>> >>\n>> >> Eric Wong writes:\n>> >>\n>> >>\n>> >> [...]\n\n[...]\n\n>> >\n>> > Side note: Is the goal to \"fix names and email addresses in this\n>> > repository\"? If so, this guide fails: it doesn't update tagger names\n>> > or email addresses. Indeed, filter-branch doesn't provide a way to do\n>> > that. (Not to mention other problems like not updating references to\n>> > commit hashes in commit messages when it busy rewriting everything.)\n>>\n>> No. Maybe the original goal was like that, by I, personally, use\n>> modified version of this to change my \"Author\" credentials from\n>> \"internal\" to \"public\" in branches that I'm going to send upstream, so\n>> the actual aim is to change e-mail of particular Author from a@b to c@d\n>> in all the commits in a (feature) branch.\n>\n> There's an interesting usecase I hadn't heard of or thought of before.\n> Quick question to see if I'm understanding correctly: \"all commits in\n> a branch\" or \"all commits *unique* to a branch\"?\n>\n> (Perhaps the only commits with the author you want to change are among\n> the commits that are unique to that branch and so the distinction\n> doesn't matter, but it wasn't clear from the description.)\n\nYes, this is exactly the case for me, as I'm changing entirely linear\ntopic branch that is going to become patch series to send out. No\ncomplications.\n\n>\n>> >> > But I agree that filter-branch isn't useful and certainly\n>> >> > shouldn't be encouraged/promoted.\n>> >>\n>> >> Well, is there more suitable way to change author for a (large) set of\n>> >> commits then?\n>> >\n>> > I would say yes, use git filter-repo (note that this thread started\n>> > with me proposing filter-repo for inclusion in git.git -- and getting\n>> > suggestions that we should remove stuff instead of adding more stuff).\n>> > I'm biased, but I think it's much better at this particular job as\n>> > well:\n>>\n>> Well, I don't want to change the entire repo, and I don't immediately\n>> see how to do it with git filter-repo. Is it at all possible?\n>\n> Yes, it is possible. filter-repo has a hidden --refs argument\n> defaulting to --all; you could instead set it to e.g.\n> origin/master..master.\n\nCool!\n\n>\n> --refs is the only hidden option in filter-repo. I know it may look\n> funny that I spent a bunch of effort to create the\n> --reference-excluded-parents option to fast-export explicitly so that\n> it would be possible to do partial history rewrites like this, and\n> then to hide and avoid documenting this option (though I did hint that\n> it existed in the documentation if you search for \"Partial-repo\n> filtering\"), but there was a few reasons for this:\n>\n> * mixing old and new history for most rewrites that\n> filter-branch/filter-repo/bfg/etc are used for can really mess things\n> up and make it hard to recover from. I don't like trying to clean up\n> repos with accidental duplicate copies of most commits in the repo,\n> and I suspect others like it even less. So, anything that makes it\n> easier to make such mistakes needs to have a really good rationale in\n> order for me to expose it.\n> * The only usecases I knew of for partial repo filtering prior to\n> your email were (1) side-stepping insanely slow execution time of poor\n> filtering tools like filter-branch, and (2) performing operations\n> better suited to git-rebase anyway (e.g. the --signoff option to\n> rebase did not exist once upon a time and so folks could have used\n> filter-branch to fake it, but using rebase is the better way to make\n> this change). And, even after your email, I'm not sure that has\n> changed though, as noted below.\n\nYeah, I share your worries.\n\n[...]\n\n>> Actually, I'd rather expect some support for this in \"git rebase\", being\n>> git history editing/reshaping tool, but it looks like it only has it in\n>> the form that is very difficult to automate.\n>\n> I agree that git rebase would be the better choice here; I typically\n> feel it's the better choice for rewrites of recent history. I think\n> it provides just what you need:\n>\n> git rebase --exec=\"git commit --amend --reset-author -C HEAD\" $UPSTREAM\n>\n> (Assuming, of course, that you've either set the right environment\n> variables or set user.name and user.email to the new values you want\n> so that commit's --reset-author flag can reset to the *new* author.)\n\nThis should do the trick for me most of times, thanks a lot for the clue!\n\nHowever, the script that I'm using doesn't change _all_ the authors, it\nonly changes those that match particular specific author specified in\nthe script. I didn't yet actually need this feature, but I can well\nimagine it's probable that I will have commits by other author(s) in the\nbranch and I won't want to attribute their job to myself.\n\nHmm... That said, using the generic \"--exec\" to \"git rebase\" I could\nprobably come-up with a script that will check the Author of the latest\ncommit and will choose to either rewrite it or not. Nothing terribly\ncomplex.\n\n>\n>> >> > Yet there's probably still users which ARE happy with it, that\n>> >> > will never hit the edge cases and problems it poses; and will\n>> >> > never read release notes. And said users are probably getting\n>> >> > git from a slow-moving distro, so it'd be a disservice to them\n>> >> > if they lost a tool they depend on without any warning.\n>> >>\n>> >> Personally, I'm far from happy with it, but I have no clue how to\n>> >> substitute it in the job above. Anybody?\n>> >\n>> > The start of this thread where I proposed git filter-repo for\n>> > inclusion in git[1] had links to documentation and comparisons to\n>> > other tools and such. You may find those links helpful; if not, let\n>> > me know what needs to be fixed in the documentation.\n>>\n>> Thank you for the references, I find it a very nice tool to have!\n>>\n>> Pity it's not an entire substitute for git filter-branch.\n>\n> Au contraire, I believe it is. :-)\n\nI take your word for it :-)\n\n>\n> Thanks for the interesting usecase. It sounds like we both think this\n> one happens to be better solved by rebase, and the command snippet I\n> provided above should show you to use rebase to solve it. However, if\n> you come up with any others where partial repo filtering makes sense,\n> I'm always willing to reconsider my decision to make the --refs\n> argument hidden; it may just mean adding more warnings, but it might\n> also involve changing other defaults (e.g. the automatic\n> repacking/pruning). I'd need concrete usecases to know for sure how\n> I'd want to handle it.\n\nOK, thanks a lot! Doesn't seem to be necessary for now due to the rebase\ntrick you've suggested.\n\n>\n> Hope that helps,\n\nSure it does!\n\n-- Sergey\n"},{"id":"381472","messageId":"CABPp-BHzHy9D7e7VbN6689-mu2AqqN-Y+inhAFjSHFi+HFSxrg@mail.gmail.com","threadId":"51710","inReplyTo":"CAPig+cSydZnie3CSwPgfSaNcGxr1jj94YBXAL47vEXrtxaQXUw@mail.gmail.com","subject":"Re: [PATCH v2 3/4] Recommend git-filter-repo instead of git-filter-branch","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-28T21:48:01Z","receivedAt":"2019-08-28T21:48:16Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Aug 27, 2019 at 11:17 PM Eric Sunshine wrote:\n>\n> On Tue, Aug 27, 2019 at 8:22 PM Elijah Newren wrote:\n> > filter-branch suffers from a deluge of disguised dangers that disfigure\n> > history rewrites (i.e. deviate from the deliberate changes). [...]\n> >\n> > Signed-off-by: Elijah Newren \n> > ---\n> > diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt\n> > @@ -16,6 +16,20 @@ SYNOPSIS\n> > +WARNING\n> > +-------\n> > +'git filter-branch' has a plethora of pitfalls that can produce non-obvious\n> > +manglings of the intended history rewrite (and can leave you with little\n> > +time to investigate such problems since it has such abysmal performance).\n> > +These safety and performance issues cannot be backward compatibly fixed and\n> > +as such, its use is not recommended. Please use an alternative history\n> > +filtering tool such as https://github.com/newren/git-filter-repo/[git\n> > +filter-repo]. If you still need to use 'git filter-branch', please\n> > +carefully read the \"Safety\" section of the message on the Git mailing list\n> > +https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/[detailing\n> > +the land mines of filter-branch] and vigilantly avoid as many of the\n> > +hazards listed there as reasonably possible.\n>\n> Is there a good reason to not simply copy the \"Safety\" section from\n> that email directly into this document so that readers don't have to\n> go chasing down the link (especially those who are reading\n> documentation offline)?\n\nMakes sense, I can include it. However, saying e.g. \"the\ngit-filter-branch manpage is missing...\" or \"the git-filter-branch\nmanpage actually documents as expected\" feels\nreally weird to include on the git-filter-branch manpage. I'll try to\ntouch it up.\n\n> > diff --git a/Documentation/git-rebase.txt b/Documentation/git-rebase.txt\n> > @@ -832,7 +832,7 @@ Hard case: The changes are not the same.::\n> > This happens if the 'subsystem' rebase had conflicts, or used\n> > `--interactive` to omit, edit, squash, or fixup commits; or\n> > if the upstream used one of `commit --amend`, `reset`, or\n> > - `filter-branch`.\n> > + a full history rewriting command like `filter-repo`.\n>\n> Do we want a clickable link to `filter-repo` here?\n\nI guess it can't hurt.\n\n> > diff --git a/Documentation/git-replace.txt b/Documentation/git-replace.txt\n> > @@ -123,10 +123,10 @@ The following format are available:\n> > +linkgit:git-hash-object[1], linkgit:git-rebase[1], and\n> > +linkgit:git-filter-repo[1], among other git commands, can be used to\n> > [...]\n> > @@ -148,8 +148,8 @@ pending objects.\n> > linkgit:git-hash-object[1]\n> > linkgit:git-rebase[1]\n> > +linkgit:git-filter-repo[1]\n>\n> Are these 'linkgit:' references to `filter-repo` going to be\n> meaningful if that tool is not incorporated into the Git project\n> proper? Perhaps use a generic clickable link instead.\n>\n> Same comment applies to other 'linkgit:' invocations in the remainder\n> of the patch.\n\nI'm fixing them up.\n\n> > diff --git a/git-filter-branch.sh b/git-filter-branch.sh\n> > old mode 100755\n> > new mode 100644\n>\n> Why lose the executable bit?\n\nWhoops. Did some rebasing and fixups, then continued editing my\nbuffer of the file after one of the rebases, realized the file was\ndeleted (because of the final patch in the series), moved the file out\nof the way and rebased again and copied the file back into place, and\nforgot to check the filemode.\n\n> > @@ -83,6 +83,19 @@ set_ident () {\n> > +if [ -z \"$FILTER_BRANCH_SQUELCH_WARNING\" -a \\\n> > + -z \"$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS\" ]; then\n>\n> If this script didn't already have a mix of styles, I'd say something\n> about modern style being:\n>\n> if test -z \"$FILTER_BRANCH_SQUELCH_WARNING\" &&\n> test -z \"$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS\"\n> then\n> ...\n> fi\n>\n> > + cat < > +WARNING: git-filter-branch has a glut of gotchas generating mangled history\n> > + rewrites. Please use an alternative filtering tool such as 'git\n> > + filter-repo' (https://github.com/newren/git-filter-repo/) instead.\n> > + See the filter-branch manual page for more details; to squelch\n> > + this warning and pause, set FILTER_BRANCH_SQUELCH_WARNING=1.\n>\n> The \"and pause\" threw me. There's more than a bit of ambiguity\n> surrounding it. Perhaps drop it?\n\nSure, will do.\n"},{"id":"381484","messageId":"20190829000657.26415-1-newren@gmail.com","threadId":"51710","inReplyTo":"20190828002210.8862-1-newren@gmail.com","subject":"[PATCH v3 0/4] Warn about git-filter-branch usage and avoid it","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-29T00:06:53Z","receivedAt":"2019-08-29T00:07:06Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Here's a series warns about git-filter-branch usage and avoids it\nourselves.\n\nChanges since v2 (full range-diff below):\n * [Patch 2] testcase syntax cleanups\n * [Patch 3] fixed \"linkgit:\" references to filter-repo to be url\n links (or footnotes)\n * [Patch 3] fixed the mode on filter-branch.sh (oops) and dropped\n the ambiguous \"and pause\". Linkified \"filter-repo\" in a place\n where there was no link.\n * [Patch 3] As suggested by Eric (Sunshine), just make the manpage\n and directly include the safety and performance sections of the\n referenced email (the performance section was referenced by the\n safety section). Being included directly in the manpage should\n help with folks reading the documentation offline. Anyway, the\n text is really long, so it took a while to format it nicely,\n recheck for typos, reword based on the fact that it'll be in the\n manpage (because it's weird to have the manpage refer to itself),\n etc.\n * [Patch 4] Dropped almost all the original patch 4; only including\n the bits about t9902-completion.sh. Removed the RFC label, since\n that one piece should be good for including now.\n\nElijah Newren (4):\n t6006: simplify and optimize empty message test\n t3427: accelerate this test by using fast-export and fast-import\n Recommend git-filter-repo instead of git-filter-branch\n t9902: use a non-deprecated command for testing\n\n Documentation/git-fast-export.txt | 6 +-\n Documentation/git-filter-branch.txt | 302 +++++++++++++++++++++++++---\n Documentation/git-gc.txt | 17 +-\n Documentation/git-rebase.txt | 3 +-\n Documentation/git-replace.txt | 10 +-\n Documentation/git-svn.txt | 10 +-\n Documentation/githooks.txt | 10 +-\n contrib/svn-fe/svn-fe.txt | 4 +-\n git-filter-branch.sh | 13 ++\n t/t3427-rebase-subtree.sh | 24 ++-\n t/t6006-rev-list-format.sh | 5 +-\n t/t9902-completion.sh | 12 +-\n 12 files changed, 339 insertions(+), 77 deletions(-)\n\nRange-diff:\n1: 7ddbeea2ca = 1: 7ddbeea2ca t6006: simplify and optimize empty message test\n2: f18bd7a609 ! 2: e1e63189c1 t3427: accelerate this test by using fast-export and fast-import\n @@ Commit message\n Signed-off-by: Elijah Newren \n \n ## t/t3427-rebase-subtree.sh ##\n -@@ t/t3427-rebase-subtree.sh: commit_message() {\n +@@ t/t3427-rebase-subtree.sh: This test runs git rebase and tests the subtree strategy.\n + . ./test-lib.sh\n + . \"$TEST_DIRECTORY\"/lib-rebase.sh\n + \n +-commit_message() {\n ++commit_message () {\n \tgit log --pretty=format:%s -1 \"$1\"\n }\n \n -+extract_files_subtree() {\n -+\tgit fast-export --no-data HEAD -- files_subtree/ \\\n -+\t\t| sed -e \"s%\\([0-9a-f]\\{40\\} \\)files_subtree/%\\1%\" \\\n -+\t\t| git fast-import --force --quiet\n ++extract_files_subtree () {\n ++\tgit fast-export --no-data HEAD -- files_subtree/ |\n ++\t\tsed -e \"s%\\([0-9a-f]\\{40\\} \\)files_subtree/%\\1%\" |\n ++\t\tgit fast-import --force --quiet\n +}\n +\n test_expect_success 'setup' '\n3: 7008c16984 ! 3: 59c7446927 Recommend git-filter-repo instead of git-filter-branch\n @@ Documentation/git-filter-branch.txt: SYNOPSIS\n +as such, its use is not recommended. Please use an alternative history\n +filtering tool such as https://github.com/newren/git-filter-repo/[git\n +filter-repo]. If you still need to use 'git filter-branch', please\n -+carefully read the \"Safety\" section of the message on the Git mailing list\n ++carefully read <> (and <>) to learn about the land\n ++mines of filter-branch, and then vigilantly avoid as many of the hazards\n ++listed there as reasonably possible.\n ++\n +https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/[detailing\n -+the land mines of filter-branch] and vigilantly avoid as many of the\n -+hazards listed there as reasonably possible.\n ++the land mines of filter-branch]\n +\n DESCRIPTION\n -----------\n @@ Documentation/git-filter-branch.txt: warned.\n - are much more restrictive than git-filter branch, and dedicated just\n - to the tasks of removing unwanted data- e.g:\n - `--strip-blobs-bigger-than 1M`.\n --\n ++[[PERFORMANCE]]\n ++PERFORMANCE\n ++-----------\n ++\n ++The performance of filter-branch is glacially slow; its design makes it\n ++impossible for a backward-compatible implementation to ever be fast:\n ++\n ++* In editing files, git-filter-branch by design checks out each and\n ++every commit as it existed in the original repo. If your repo has 10\\^5\n ++files and 10\\^5 commits, but each commit only modifies 5 files, then\n ++git-filter-branch will make you do 10\\^10 modifications, despite only\n ++having (at most) 5*10^5 unique blobs.\n ++\n ++* If you try and cheat and try to make filter-branch only work on\n ++files modified in a commit, then two things happen\n ++\n ++ . you run into problems with deletions whenever the user is simply\n ++ trying to rename files (because attempting to delete files that\n ++ don't exist looks like a no-op; it takes some chicanery to remap\n ++ deletes across file renames when the renames happen via arbitrary\n ++ user-provided shell)\n ++\n ++ . even if you succeed at the map-deletes-for-renames chicanery, you\n ++ still technically violate backward compatibility because users are\n ++ allowed to filter files in ways that depend upon topology of commits\n ++ instead of filtering solely based on file contents or names (though\n ++ I have never seen any user ever do this).\n ++\n ++* Even if you don't need to edit files but only want to e.g. rename or\n ++remove some and thus can avoid checking out each file (i.e. you can use\n ++--index-filter), you still are passing shell snippets for your filters.\n ++This means that for every commit, you have to have a prepared git repo\n ++where users can run git commands. That's a lot of setup. It also means\n ++you have to fork at least one process to run the user-provided shell\n ++snippet, and odds are that the user's shell snippet invokes lots of\n ++commands in some long pipeline, so you will have lots and lots of forks.\n ++For every. single. commit. That's a massive amount of overhead to\n ++rename a few files.\n ++\n ++* filter-branch is written in shell, which is kind of slow. Naturally,\n ++it makes sense to want to rewrite that in some other language. However,\n ++filter-branch documentation states that several additional shell\n ++functions are provided for users to call, e.g. 'map', 'skip_commit',\n ++'git_commit_non_empty_tree'. If filter-branch itself isn't a shell\n ++script, then in order to make those shell functions available to the\n ++users' shell snippets you have to prepend the shell definitions of these\n ++functions to every one of the users' shell snippets and thus make these\n ++special shell functions be parsed with each and every commit.\n ++\n ++* filter-branch provides a --setup option which is a shell snippet that\n ++can be sourced to make shell functions and variables available to all\n ++other filters. If filter-branch is a shell script, it can simply eval\n ++this shell snippet once at the beginning. If you try to fix performance\n ++by making filter-branch not be a shell script, then you have to prepend\n ++the setup shell snippet to all other filters and parse it with every\n ++single commit.\n ++\n ++* filter-branch writes lots of files to $workdir/../map/ to keep a\n ++mapping of commits, which it uses pruning commits and remapping to\n ++ancestors and the map() command more generally. Other files like\n ++$tempdir/backup-refs, $tempdir/raw-refs, $tempdir/heads,\n ++$tempdir/tree-state are all created internally too. It is possible\n ++(though strongly discouraged) that users could have accessed any of\n ++these directly. Users even had a pointer to follow in the form of\n ++Documentation that the 'map' command existed, which naturally uses the\n ++$workdir/../map/* files. So, even if you don't have to edit files, for\n ++strict backward compatibility you need to still write a bunch of files\n ++to disk somewhere and keep them updated for every commit. You can claim\n ++it was an implementation detail that users should not have depended\n ++upon, but the truth is they've had a decade where they could so. So, if\n ++you want full compatibility, it has to be there. Besides, the\n ++regression tests depend on at least one of these details, specifying an\n ++--index-filter that reaches down and grabs backup-refs from $tempdir,\n ++and thus provides resourceful users who do google searches an example\n ++that there are files there for them to read and grab and use. (And if\n ++you want to pass the existing regression tests, you have to at least put\n ++the backup-refs file there even if it's irrelevant to your\n ++implementation otherwise.)\n ++\n ++All of that said, performance of filter-branch could be improved by\n ++reimplementing it in a non-shell language and taking a couple small\n ++liberties with backward compatibility (such as having it only run\n ++filters on files changed within each commit). filter-repo provides a\n ++demo script named\n ++https://github.com/newren/git-filter-repo/blob/master/contrib/filter-repo-demos/filter-lamely[filter-lamely]\n ++which does exactly that and which passes all the git-filter-branch\n ++regression tests. It's much faster than git-filter-branch, though it\n ++suffers from all the same safety issues as git-filter-branch, and is\n ++still glacially slow compared to\n ++https://github.com/newren/git-filter-repo/[git filter-repo].\n ++\n ++[[SAFETY]]\n ++SAFETY\n ++------\n ++\n ++filter-branch is riddled with gotchas resulting in various ways to\n ++easily corrupt repos or end up with a mess worse than what you started\n ++with:\n ++\n ++* Someone can have a set of \"working and tested filters\" which they\n ++document or provide to a coworker, who then runs them on a different OS\n ++where the same commands are not working/tested (some examples in the\n ++git-filter-branch manpage are also affected by this). BSD vs. GNU\n ++userland differences can really bite. If you're lucky, you get ugly\n ++error messages spewed. But just as likely, the commands either don't do\n ++the filtering requested, or silently corrupt making some unwanted\n ++change. The unwanted change may only affect a few commits, so it's not\n ++necessarily obvious either. (The fact that problems won't necessarily\n ++be obvious means they are likely to go unnoticed until the rewritten\n ++history is in use for quite a while, at which point it's really hard to\n ++justify another flag-day for another rewrite.)\n ++\n ++* Filenames with spaces (which are rare) are often mishandled by shell\n ++snippets since they cause problems for shell pipelines. Not everyone is\n ++familiar with find -print0, xargs -0, ls-files -z, etc. Even people who\n ++are familiar with these may assume such needs are not relevant because\n ++someone else renamed any such files in their repo back before the person\n ++doing the filtering joined the project. And, often, even those familiar\n ++with handling arguments with spaces my not do so just because they\n ++aren't in the mindset of thinking about everything that could possibly\n ++go wrong.\n ++\n ++* Non-ascii filenames (which are rare) can be silently removed despite\n ++being in a desired directory. The desire to select paths to keep often\n ++use pipelines like `git ls-files | grep -v ^WANTED_DIR/ | xargs git rm`.\n ++ls-files will only quote filenames if needed so folks may not notice\n ++that one of the files didn't match the regex, again until it's much too\n ++late. Yes, someone who knows about core.quotePath can avoid this\n ++(unless they have other special characters like \\t, \\n, or \"), and\n ++people who use ls-files -z with something other than grep can avoid\n ++this, but that doesn't mean they will.\n ++\n ++* Similarly, when moving files around, one can find that filenames with\n ++non-ascii or special characters end up in a different directory, one\n ++that includes a double quote character. (This is technically the same\n ++issue as above with quoting, but perhaps an interesting different way\n ++that it can and has manifested as a problem.)\n ++\n ++* It's far too easy to accidentally mix up old and new history. It's\n ++still possible with any tool, but filter-branch almost invites it. If\n ++we're lucky, the only downside is users getting frustrated that they\n ++don't know how to shrink their repo and remove the old stuff. If we're\n ++unlucky, they merge old and new history and end up with multiple\n ++\"copies\" of each commit, some of which have unwanted or sensitive files\n ++and others which don't. This comes about in multiple different ways:\n ++\n ++ ** the default to only doing a partial history rewrite ('--all' is not\n ++ the default and over 80% of the examples in the manpage don't use\n ++ it)\n ++\n ++ ** the fact that there's no automatic post-run cleanup\n ++\n ++ ** the fact that --tag-name-filter (when used to rename tags) doesn't\n ++ remove the old tags but just adds new ones with the new name (this\n ++ manpage has documented this for a long time so it's presumably not\n ++ a \"bug\" even though it feels like it)\n ++\n ++ ** the fact that little educational information is provided to inform\n ++ users of the ramifications of a rewrite and how to avoid mixing old\n ++ and new history. For example, this man page discusses how users\n ++ need to understand that they need to rebase their changes for all\n ++ their branches on top of new history (or delete and reclone), but\n ++ that's only one of multiple concerns to consider. See the\n ++ \"DISCUSSION\" section of the git filter-repo manual page for more\n ++ details.\n ++\n ++* Annotated tags can be accidentally converted to lightweight tags, due\n ++to either of two issues:\n ++\n ++ . Someone can do a history rewrite, realize they messed up, restore\n ++ from the backups in refs/original/, and then redo their\n ++ filter-branch command. (The backup in refs/original/ is not a real\n ++ backup; it dereferences tags first.)\n ++\n ++ . Running filter-branch with either --tags or --all in your . In order to retain annotated tags as annotated, you must\n ++ use --tag-name-filter (and must not have restored from\n ++ refs/original/ in a previously botched rewrite).\n ++\n ++* Any commit messages that specify an encoding will become corrupted\n ++by the rewrite; filter-branch ignores the encoding, takes the original\n ++bytes, and feeds it to commit-tree without telling it the proper\n ++encoding. (This happens whether or not --msg-filter is used, though I\n ++suspect --msg-filter provides additional ways to really mess things\n ++up).\n ++\n ++* Commit messages (even if they are all UTF-8) by default become\n ++corrupted due to not being updated -- any references to other commit\n ++hashes in commit messages will now refer to no-longer-extant commits.\n ++\n ++* There are no facilities for helping users find what unwanted crud they\n ++should delete, which means they are much more likely to have incomplete\n ++or partial cleanups that sometimes result in confusion and people\n ++wasting time trying to understand. (For example, folks tend to just\n ++look for big files to delete instead of big directories or extensions,\n ++and once they do so, then sometime later folks using the new repository\n ++who are going through history will notice a build artifact directory\n ++that has some files but not others, or a cache of dependencies\n ++(node_modules or similar) which couldn't have ever been functional since\n ++it's missing some files.)\n ++\n ++* If --prune-empty isn't specified, then the filtering process can\n ++create hoards of confusing empty commits\n ++\n ++* If --prune-empty is specified, then intentionally placed empty\n ++commits from before the filtering operation are also pruned instead of\n ++just pruning commits that became empty due to filtering rules.\n ++\n ++* If --prune empty is specified, sometimes empty commits are missed\n ++and left around anyway (a somewhat rare bug, but it happens...)\n ++\n ++* A minor issue, but users who have a goal to update all names and\n ++emails in a repository may be led to --env-filter which will only update\n ++authors and committers, missing taggers.\n ++\n ++* If the user provides a --tag-name-filter that maps multiple tags to\n ++the same name, no warning or error is provided; filter-branch simply\n ++overwrites each tag in some undocumented pre-defined order resulting in\n ++only one tag at the end. If you try to \"fix\" this bug in filter-branch\n ++and make it error out and warn the user instead, one of the\n ++filter-branch regression tests will fail. (So, if you are trying to\n ++make a backward compatible reimplementation you have to add extra code\n ++to detect collisions and make sure that only the lexicographically last\n ++one is rewritten to avoid fast-import from seeing both since fast-import\n ++will naturally do the sane thing and error out if told to write the same\n ++tag more than once.)\n ++\n ++Also, the poor performance of filter-branch often leads to safety issues:\n ++\n ++* Coming up with the correct shell snippet to do the filtering you want\n ++is sometimes difficult unless you're just doing a trivial modification\n ++such as deleting a couple files. People have often come to me for help,\n ++so I should be practiced and an expert, but even for fairly simple cases\n ++I still sometimes take over 10 minutes and several iterations to get\n ++the right commands -- and that's assuming they are working on a tiny\n ++repository. Unfortunately, people often learn if the snippet is right\n ++or wrong by trying it out, but the rightness or wrongness can vary\n ++depending on special circumstances (spaces in filenames, non-ascii\n ++filenames, funny author names or emails, invalid timezones, presence of\n ++grafts or replace objects, etc.), meaning they may have to wait a long\n ++time, hit an error, then restart. The performance of filter-branch is\n ++so bad that this cycle is painful, reducing the time available to\n ++carefully re-check (to say nothing about what it does to the patience of\n ++the person doing the rewrite even if they do technically have more time\n ++available). This problem is extra compounded because errors from broken\n ++filters may not be shown for a long time and/or get lost in a sea of\n ++output. Even worse, broken filters often just result in silent\n ++incorrect rewrites.\n ++\n ++* To top it all off, even when users finally find working commands, they\n ++naturally want to share them. But they may be unaware that their repo\n ++didn't have some special cases that someone else's does. So, when\n ++someone else with a different repository runs the same commands, they\n ++get hit by the problems above. Or, the user just runs commands that\n ++really were vetted for special cases, but they run it on a different OS\n ++where it doesn't work, as noted above.\n + \n GIT\n ---\n - Part of the linkgit:git[1] suite\n \n ## Documentation/git-gc.txt ##\n @@ Documentation/git-gc.txt: NOTES\n @@ Documentation/git-rebase.txt: Hard case: The changes are not the same.::\n \t`--interactive` to omit, edit, squash, or fixup commits; or\n \tif the upstream used one of `commit --amend`, `reset`, or\n -\t`filter-branch`.\n -+\ta full history rewriting command like `filter-repo`.\n ++\ta full history rewriting command like\n ++\thttps://github.com/newren/git-filter-repo[`filter-repo`].\n \n \n The easy case\n @@ Documentation/git-replace.txt: The following format are available:\n -replacement objects from existing objects. The `--edit` option can\n -also be used with 'git replace' to create a replacement object by\n +linkgit:git-hash-object[1], linkgit:git-rebase[1], and\n -+linkgit:git-filter-repo[1], among other git commands, can be used to\n ++https://github.com/newren/git-filter-repo[git-filter-repo], among other git commands, can be used to\n +create replacement objects from existing objects. The `--edit` option\n +can also be used with 'git replace' to create a replacement object by\n editing an existing object.\n @@ Documentation/git-replace.txt: pending objects.\n linkgit:git-hash-object[1]\n -linkgit:git-filter-branch[1]\n linkgit:git-rebase[1]\n -+linkgit:git-filter-repo[1]\n linkgit:git-tag[1]\n linkgit:git-branch[1]\n linkgit:git-commit[1]\n + linkgit:git-var[1]\n + linkgit:git[1]\n ++https://github.com/newren/git-filter-repo[git-filter-repo]\n + \n + GIT\n + ---\n \n ## Documentation/git-svn.txt ##\n @@ Documentation/git-svn.txt: option for (hopefully) obvious reasons.\n @@ Documentation/git-svn.txt: option for (hopefully) obvious reasons.\n This option is NOT recommended as it makes it difficult to track down\n old references to SVN revision numbers in existing documentation, bug\n -reports and archives. If you plan to eventually migrate from SVN to Git\n -+reports, and archives. If you plan to eventually migrate from SVN to Git\n - and are certain about dropping SVN history, consider\n +-and are certain about dropping SVN history, consider\n -linkgit:git-filter-branch[1] instead. filter-branch also allows\n -+linkgit:git-filter-repo[1] instead. filter-repo also allows\n - reformatting of metadata for ease-of-reading and rewriting authorship\n - info for non-\"svn.authorsFile\" users.\n +-reformatting of metadata for ease-of-reading and rewriting authorship\n +-info for non-\"svn.authorsFile\" users.\n ++reports, and archives. If you plan to eventually migrate from SVN to\n ++Git and are certain about dropping SVN history, consider\n ++https://github.com/newren/git-filter-repo[git-filter-repo] instead.\n ++filter-repo also allows reformatting of metadata for ease-of-reading\n ++and rewriting authorship info for non-\"svn.authorsFile\" users.\n \n + svn.useSvmProps::\n + svn-remote..useSvmProps::\n \n ## Documentation/githooks.txt ##\n @@ Documentation/githooks.txt: post-rewrite\n @@ Documentation/githooks.txt: post-rewrite\n -linkgit:git-rebase[1]; currently `git filter-branch` does 'not' call\n -it!). Its first argument denotes the command it was invoked by:\n -currently one of `amend` or `rebase`. Further command-dependent\n +-arguments may be passed in the future.\n +linkgit:git-rebase[1]; however, full-history (re)writing tools like\n -+linkgit:git-fast-import[1] or linkgit:git-filter-repo[1] typically do\n -+not call it!). Its first argument denotes the command it was invoked\n -+by: currently one of `amend` or `rebase`. Further command-dependent\n - arguments may be passed in the future.\n ++linkgit:git-fast-import[1] or\n ++https://github.com/newren/git-filter-repo[git-filter-repo] typically\n ++do not call it!). Its first argument denotes the command it was\n ++invoked by: currently one of `amend` or `rebase`. Further\n ++command-dependent arguments may be passed in the future.\n \n The hook receives a list of the rewritten commits on stdin, in the\n + format\n \n ## contrib/svn-fe/svn-fe.txt ##\n @@ contrib/svn-fe/svn-fe.txt: line. This line has the form `git-svn-id: URL@REVNO UUID`.\n @@ contrib/svn-fe/svn-fe.txt: The exit status does not reflect whether an error was\n +git-svn(1), svn2git(1), svk(1), git-filter-repo(1), git-fast-import(1),\n https://svn.apache.org/repos/asf/subversion/trunk/notes/dump-load-format.txt\n \n - ## git-filter-branch.sh (mode change 100755 => 100644) ##\n + ## git-filter-branch.sh ##\n @@ git-filter-branch.sh: set_ident () {\n \tfinish_ident COMMITTER\n }\n @@ git-filter-branch.sh: set_ident () {\n + rewrites. Please use an alternative filtering tool such as 'git\n + filter-repo' (https://github.com/newren/git-filter-repo/) instead.\n + See the filter-branch manual page for more details; to squelch\n -+ this warning and pause, set FILTER_BRANCH_SQUELCH_WARNING=1.\n ++ this warning, set FILTER_BRANCH_SQUELCH_WARNING=1.\n +\n +EOF\n +\tsleep 5\n4: ff3e04e558 < -: ---------- Remove git-filter-branch, it is now external to git.git\n-: ---------- > 4: 1dbca82408 t9902: use a non-deprecated command for testing\n-- \n2.23.0.3.g59c7446927.dirty\n\n"},{"id":"381485","messageId":"20190829000657.26415-2-newren@gmail.com","threadId":"51710","inReplyTo":"20190829000657.26415-1-newren@gmail.com","subject":"[PATCH v3 1/4] t6006: simplify and optimize empty message test","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-29T00:06:54Z","receivedAt":"2019-08-29T00:07:06Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Test t6006.71 (\"oneline with empty message\") was creating two commits\nwith simple commit messages, and then running filter-branch to rewrite\nthe commit messages to be empty. This test was written this way because\nthe --allow-empty-message option to git commit did not exist at the\ntime. Simplify this test and avoid the need to invoke filter-branch by\njust using --allow-empty-message when creating the commit.\n\nDespite only being one piece of the 71st test and there being 73 tests\noverall, this small change to just this one test speeds up the overall\nexecution time of t6006 (as measured by the best of 3 runs of `time\n./t6006-rev-list-format.sh`) by about 11% on Linux and by 13% on\nMac.\n\nSigned-off-by: Elijah Newren \n---\n t/t6006-rev-list-format.sh | 5 ++---\n 1 file changed, 2 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t6006-rev-list-format.sh b/t/t6006-rev-list-format.sh\nindex da113d975b..d30e41c9f7 100755\n--- a/t/t6006-rev-list-format.sh\n+++ b/t/t6006-rev-list-format.sh\n@@ -501,9 +501,8 @@ test_expect_success 'reflog identity' '\n '\n \n test_expect_success 'oneline with empty message' '\n-\tgit commit -m \"dummy\" --allow-empty &&\n-\tgit commit -m \"dummy\" --allow-empty &&\n-\tgit filter-branch --msg-filter \"sed -e s/dummy//\" HEAD^^.. &&\n+\tgit commit --allow-empty --allow-empty-message &&\n+\tgit commit --allow-empty --allow-empty-message &&\n \tgit rev-list --oneline HEAD >test.txt &&\n \ttest_line_count = 5 test.txt &&\n \tgit rev-list --oneline --graph HEAD >testg.txt &&\n-- \n2.23.0.3.g59c7446927.dirty\n\n"},{"id":"381486","messageId":"20190829000657.26415-3-newren@gmail.com","threadId":"51710","inReplyTo":"20190829000657.26415-1-newren@gmail.com","subject":"[PATCH v3 2/4] t3427: accelerate this test by using fast-export and fast-import","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-29T00:06:55Z","receivedAt":"2019-08-29T00:07:10Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"fast-export and fast-import can easily handle the simple rewrite that\nwas being done by filter-branch, and should be significantly faster on\nsystems with a slow fork. Timings from before and after on two laptops\nthat I have access to (measured via `time ./t3427-rebase-subtree.sh`,\ni.e. including everything in this test -- not just the filter-branch or\nfast-export/fast-import pair):\n\n Linux: 4.305s -> 3.684s (~17% speedup)\n Mac: 10.128s -> 7.038s (~30% speedup)\n\nSigned-off-by: Elijah Newren \n---\n t/t3427-rebase-subtree.sh | 24 +++++++++++++++---------\n 1 file changed, 15 insertions(+), 9 deletions(-)\n\ndiff --git a/t/t3427-rebase-subtree.sh b/t/t3427-rebase-subtree.sh\nindex d8640522a0..c1f6102921 100755\n--- a/t/t3427-rebase-subtree.sh\n+++ b/t/t3427-rebase-subtree.sh\n@@ -7,10 +7,16 @@ This test runs git rebase and tests the subtree strategy.\n . ./test-lib.sh\n . \"$TEST_DIRECTORY\"/lib-rebase.sh\n \n-commit_message() {\n+commit_message () {\n \tgit log --pretty=format:%s -1 \"$1\"\n }\n \n+extract_files_subtree () {\n+\tgit fast-export --no-data HEAD -- files_subtree/ |\n+\t\tsed -e \"s%\\([0-9a-f]\\{40\\} \\)files_subtree/%\\1%\" |\n+\t\tgit fast-import --force --quiet\n+}\n+\n test_expect_success 'setup' '\n \ttest_commit README &&\n \tmkdir files &&\n@@ -42,7 +48,7 @@ test_expect_failure REBASE_P \\\n \t'Rebase -Xsubtree --preserve-merges --onto commit 4' '\n \treset_rebase &&\n \tgit checkout -b rebase-preserve-merges-4 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --preserve-merges --onto files-master master &&\n \tverbose test \"$(commit_message HEAD~)\" = \"files_subtree/master4\"\n@@ -53,7 +59,7 @@ test_expect_failure REBASE_P \\\n \t'Rebase -Xsubtree --preserve-merges --onto commit 5' '\n \treset_rebase &&\n \tgit checkout -b rebase-preserve-merges-5 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --preserve-merges --onto files-master master &&\n \tverbose test \"$(commit_message HEAD)\" = \"files_subtree/master5\"\n@@ -64,7 +70,7 @@ test_expect_failure REBASE_P \\\n \t'Rebase -Xsubtree --keep-empty --preserve-merges --onto commit 4' '\n \treset_rebase &&\n \tgit checkout -b rebase-keep-empty-4 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master &&\n \tverbose test \"$(commit_message HEAD~2)\" = \"files_subtree/master4\"\n@@ -75,7 +81,7 @@ test_expect_failure REBASE_P \\\n \t'Rebase -Xsubtree --keep-empty --preserve-merges --onto commit 5' '\n \treset_rebase &&\n \tgit checkout -b rebase-keep-empty-5 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master &&\n \tverbose test \"$(commit_message HEAD~)\" = \"files_subtree/master5\"\n@@ -86,7 +92,7 @@ test_expect_failure REBASE_P \\\n \t'Rebase -Xsubtree --keep-empty --preserve-merges --onto empty commit' '\n \treset_rebase &&\n \tgit checkout -b rebase-keep-empty-empty master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master &&\n \tverbose test \"$(commit_message HEAD)\" = \"Empty commit\"\n@@ -96,7 +102,7 @@ test_expect_failure REBASE_P \\\n test_expect_failure 'Rebase -Xsubtree --onto commit 4' '\n \treset_rebase &&\n \tgit checkout -b rebase-onto-4 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --onto files-master master &&\n \tverbose test \"$(commit_message HEAD~2)\" = \"files_subtree/master4\"\n@@ -106,7 +112,7 @@ test_expect_failure 'Rebase -Xsubtree --onto commit 4' '\n test_expect_failure 'Rebase -Xsubtree --onto commit 5' '\n \treset_rebase &&\n \tgit checkout -b rebase-onto-5 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --onto files-master master &&\n \tverbose test \"$(commit_message HEAD~)\" = \"files_subtree/master5\"\n@@ -115,7 +121,7 @@ test_expect_failure 'Rebase -Xsubtree --onto commit 5' '\n test_expect_failure 'Rebase -Xsubtree --onto empty commit' '\n \treset_rebase &&\n \tgit checkout -b rebase-onto-empty master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --onto files-master master &&\n \tverbose test \"$(commit_message HEAD)\" = \"Empty commit\"\n-- \n2.23.0.3.g59c7446927.dirty\n\n"},{"id":"381487","messageId":"20190829000657.26415-5-newren@gmail.com","threadId":"51710","inReplyTo":"20190829000657.26415-1-newren@gmail.com","subject":"[PATCH v3 4/4] t9902: use a non-deprecated command for testing","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-29T00:06:57Z","receivedAt":"2019-08-29T00:07:11Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"t9902 had a list of three random porcelain commands as a sanity check,\none of which was filter-branch. Since we are recommending people not\nuse filter-branch, let's update this test to use rebase instead of\nfilter-branch.\n\nSigned-off-by: Elijah Newren \n---\n t/t9902-completion.sh | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/t/t9902-completion.sh b/t/t9902-completion.sh\nindex 75512c3403..4e7f669c76 100755\n--- a/t/t9902-completion.sh\n+++ b/t/t9902-completion.sh\n@@ -28,10 +28,10 @@ complete ()\n #\n # (2) A test makes sure that common subcommands are included in the\n # completion for \"git \", and a plumbing is excluded. \"add\",\n-# \"filter-branch\" and \"ls-files\" are listed for this.\n+# \"rebase\" and \"ls-files\" are listed for this.\n \n-GIT_TESTING_ALL_COMMAND_LIST='add checkout check-attr filter-branch ls-files'\n-GIT_TESTING_PORCELAIN_COMMAND_LIST='add checkout filter-branch'\n+GIT_TESTING_ALL_COMMAND_LIST='add checkout check-attr rebase ls-files'\n+GIT_TESTING_PORCELAIN_COMMAND_LIST='add checkout rebase'\n \n . \"$GIT_BUILD_DIR/contrib/completion/git-completion.bash\"\n \n@@ -1392,12 +1392,12 @@ test_expect_success 'basic' '\n \t# built-in\n \tgrep -q \"^add \\$\" out &&\n \t# script\n-\tgrep -q \"^filter-branch \\$\" out &&\n+\tgrep -q \"^rebase \\$\" out &&\n \t# plumbing\n \t! grep -q \"^ls-files \\$\" out &&\n \n-\trun_completion \"git f\" &&\n-\t! grep -q -v \"^f\" out\n+\trun_completion \"git r\" &&\n+\t! grep -q -v \"^r\" out\n '\n \n test_expect_success 'double dash \"git\" itself' '\n-- \n2.23.0.3.g59c7446927.dirty\n\n"},{"id":"381488","messageId":"20190829000657.26415-4-newren@gmail.com","threadId":"51710","inReplyTo":"20190829000657.26415-1-newren@gmail.com","subject":"[PATCH v3 3/4] Recommend git-filter-repo instead of git-filter-branch","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-29T00:06:56Z","receivedAt":"2019-08-29T00:07:12Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"filter-branch suffers from a deluge of disguised dangers that disfigure\nhistory rewrites (i.e. deviate from the deliberate changes). Many of\nthese problems are unobtrusive and can easily go undiscovered until the\nnew repository is in use. This can result in problems ranging from an\neven messier history than what led folks to filter-branch in the first\nplace, to data loss or corruption. These issues cannot be backward\ncompatibly fixed, so add a warning to both filter-branch and its manpage\nrecommending that another tool (such as filter-repo) be used instead.\n\nAlso, update other manpages that referenced filter-branch. Several of\nthese needed updates even if we could continue recommending\nfilter-branch, either due to implying that something was unique to\nfilter-branch when it applied more generally to all history rewriting\ntools (e.g. BFG, reposurgeon, fast-import, filter-repo), or because\nsomething about filter-branch was used as an example despite other more\ncommonly known examples now existing. Reword these sections to fix\nthese issues and to avoid recommending filter-branch.\n\nFinally, remove the section explaining BFG Repo Cleaner as an\nalternative to filter-branch. I feel somewhat bad about this,\nespecially since I feel like I learned so much from BFG that I put to\ngood use in filter-repo (which is much more than I can say for\nfilter-branch), but keeping that section presented a few problems:\n * In order to recommend that people quit using filter-branch, we need\n to provide them a recomendation for something else to use that\n can handle all the same types of rewrites. To my knowledge,\n filter-repo is the only such tool. So it needs to be mentioned.\n * I don't want to give conflicting recommendations to users\n * If we recommend two tools, we shouldn't expect users to learn both\n and pick which one to use; we should explain which problems one\n can solve that the other can't or when one is much faster than\n the other.\n * BFG and filter-repo have similar performance\n * All filtering types that BFG can do, filter-repo can also do. In\n fact, filter-repo comes with a reimplementation of BFG named\n bfg-ish which provides the same user-interface as BFG but with\n several bugfixes and new features that are hard to implement in\n BFG due to its technical underpinnings.\nWhile I could still mention both tools, it seems like I would need to\nprovide some kind of comparison and I would ultimately just say that\nfilter-repo can do everything BFG can, so ultimately it seems that it\nis just better to remove that section altogether.\n\nSigned-off-by: Elijah Newren \n---\n Documentation/git-fast-export.txt | 6 +-\n Documentation/git-filter-branch.txt | 302 +++++++++++++++++++++++++---\n Documentation/git-gc.txt | 17 +-\n Documentation/git-rebase.txt | 3 +-\n Documentation/git-replace.txt | 10 +-\n Documentation/git-svn.txt | 10 +-\n Documentation/githooks.txt | 10 +-\n contrib/svn-fe/svn-fe.txt | 4 +-\n git-filter-branch.sh | 13 ++\n 9 files changed, 316 insertions(+), 59 deletions(-)\n\ndiff --git a/Documentation/git-fast-export.txt b/Documentation/git-fast-export.txt\nindex cc940eb9ad..784e934009 100644\n--- a/Documentation/git-fast-export.txt\n+++ b/Documentation/git-fast-export.txt\n@@ -17,9 +17,9 @@ This program dumps the given revisions in a form suitable to be piped\n into 'git fast-import'.\n \n You can use it as a human-readable bundle replacement (see\n-linkgit:git-bundle[1]), or as a kind of an interactive\n-'git filter-branch'.\n-\n+linkgit:git-bundle[1]), or as a format that can be edited before being\n+fed to 'git fast-import' in order to do history rewrites (an ability\n+relied on by tools like 'git filter-repo').\n \n OPTIONS\n -------\ndiff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt\nindex 6b53dd7e06..c3f874b692 100644\n--- a/Documentation/git-filter-branch.txt\n+++ b/Documentation/git-filter-branch.txt\n@@ -16,6 +16,22 @@ SYNOPSIS\n \t[--original ] [-d ] [-f | --force]\n \t[--state-branch ] [--] [...]\n \n+WARNING\n+-------\n+'git filter-branch' has a plethora of pitfalls that can produce non-obvious\n+manglings of the intended history rewrite (and can leave you with little\n+time to investigate such problems since it has such abysmal performance).\n+These safety and performance issues cannot be backward compatibly fixed and\n+as such, its use is not recommended. Please use an alternative history\n+filtering tool such as https://github.com/newren/git-filter-repo/[git\n+filter-repo]. If you still need to use 'git filter-branch', please\n+carefully read <> (and <>) to learn about the land\n+mines of filter-branch, and then vigilantly avoid as many of the hazards\n+listed there as reasonably possible.\n+\n+https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/[detailing\n+the land mines of filter-branch]\n+\n DESCRIPTION\n -----------\n Lets you rewrite Git revision history by rewriting the branches mentioned\n@@ -445,36 +461,262 @@ warned.\n (or if your git-gc is not new enough to support arguments to\n `--prune`, use `git repack -ad; git prune` instead).\n \n-NOTES\n------\n-\n-git-filter-branch allows you to make complex shell-scripted rewrites\n-of your Git history, but you probably don't need this flexibility if\n-you're simply _removing unwanted data_ like large files or passwords.\n-For those operations you may want to consider\n-http://rtyley.github.io/bfg-repo-cleaner/[The BFG Repo-Cleaner],\n-a JVM-based alternative to git-filter-branch, typically at least\n-10-50x faster for those use-cases, and with quite different\n-characteristics:\n-\n-* Any particular version of a file is cleaned exactly _once_. The BFG,\n- unlike git-filter-branch, does not give you the opportunity to\n- handle a file differently based on where or when it was committed\n- within your history. This constraint gives the core performance\n- benefit of The BFG, and is well-suited to the task of cleansing bad\n- data - you don't care _where_ the bad data is, you just want it\n- _gone_.\n-\n-* By default The BFG takes full advantage of multi-core machines,\n- cleansing commit file-trees in parallel. git-filter-branch cleans\n- commits sequentially (i.e. in a single-threaded manner), though it\n- _is_ possible to write filters that include their own parallelism,\n- in the scripts executed against each commit.\n-\n-* The http://rtyley.github.io/bfg-repo-cleaner/#examples[command options]\n- are much more restrictive than git-filter branch, and dedicated just\n- to the tasks of removing unwanted data- e.g:\n- `--strip-blobs-bigger-than 1M`.\n+[[PERFORMANCE]]\n+PERFORMANCE\n+-----------\n+\n+The performance of filter-branch is glacially slow; its design makes it\n+impossible for a backward-compatible implementation to ever be fast:\n+\n+* In editing files, git-filter-branch by design checks out each and\n+every commit as it existed in the original repo. If your repo has 10\\^5\n+files and 10\\^5 commits, but each commit only modifies 5 files, then\n+git-filter-branch will make you do 10\\^10 modifications, despite only\n+having (at most) 5*10^5 unique blobs.\n+\n+* If you try and cheat and try to make filter-branch only work on\n+files modified in a commit, then two things happen\n+\n+ . you run into problems with deletions whenever the user is simply\n+ trying to rename files (because attempting to delete files that\n+ don't exist looks like a no-op; it takes some chicanery to remap\n+ deletes across file renames when the renames happen via arbitrary\n+ user-provided shell)\n+\n+ . even if you succeed at the map-deletes-for-renames chicanery, you\n+ still technically violate backward compatibility because users are\n+ allowed to filter files in ways that depend upon topology of commits\n+ instead of filtering solely based on file contents or names (though\n+ I have never seen any user ever do this).\n+\n+* Even if you don't need to edit files but only want to e.g. rename or\n+remove some and thus can avoid checking out each file (i.e. you can use\n+--index-filter), you still are passing shell snippets for your filters.\n+This means that for every commit, you have to have a prepared git repo\n+where users can run git commands. That's a lot of setup. It also means\n+you have to fork at least one process to run the user-provided shell\n+snippet, and odds are that the user's shell snippet invokes lots of\n+commands in some long pipeline, so you will have lots and lots of forks.\n+For every. single. commit. That's a massive amount of overhead to\n+rename a few files.\n+\n+* filter-branch is written in shell, which is kind of slow. Naturally,\n+it makes sense to want to rewrite that in some other language. However,\n+filter-branch documentation states that several additional shell\n+functions are provided for users to call, e.g. 'map', 'skip_commit',\n+'git_commit_non_empty_tree'. If filter-branch itself isn't a shell\n+script, then in order to make those shell functions available to the\n+users' shell snippets you have to prepend the shell definitions of these\n+functions to every one of the users' shell snippets and thus make these\n+special shell functions be parsed with each and every commit.\n+\n+* filter-branch provides a --setup option which is a shell snippet that\n+can be sourced to make shell functions and variables available to all\n+other filters. If filter-branch is a shell script, it can simply eval\n+this shell snippet once at the beginning. If you try to fix performance\n+by making filter-branch not be a shell script, then you have to prepend\n+the setup shell snippet to all other filters and parse it with every\n+single commit.\n+\n+* filter-branch writes lots of files to $workdir/../map/ to keep a\n+mapping of commits, which it uses pruning commits and remapping to\n+ancestors and the map() command more generally. Other files like\n+$tempdir/backup-refs, $tempdir/raw-refs, $tempdir/heads,\n+$tempdir/tree-state are all created internally too. It is possible\n+(though strongly discouraged) that users could have accessed any of\n+these directly. Users even had a pointer to follow in the form of\n+Documentation that the 'map' command existed, which naturally uses the\n+$workdir/../map/* files. So, even if you don't have to edit files, for\n+strict backward compatibility you need to still write a bunch of files\n+to disk somewhere and keep them updated for every commit. You can claim\n+it was an implementation detail that users should not have depended\n+upon, but the truth is they've had a decade where they could so. So, if\n+you want full compatibility, it has to be there. Besides, the\n+regression tests depend on at least one of these details, specifying an\n+--index-filter that reaches down and grabs backup-refs from $tempdir,\n+and thus provides resourceful users who do google searches an example\n+that there are files there for them to read and grab and use. (And if\n+you want to pass the existing regression tests, you have to at least put\n+the backup-refs file there even if it's irrelevant to your\n+implementation otherwise.)\n+\n+All of that said, performance of filter-branch could be improved by\n+reimplementing it in a non-shell language and taking a couple small\n+liberties with backward compatibility (such as having it only run\n+filters on files changed within each commit). filter-repo provides a\n+demo script named\n+https://github.com/newren/git-filter-repo/blob/master/contrib/filter-repo-demos/filter-lamely[filter-lamely]\n+which does exactly that and which passes all the git-filter-branch\n+regression tests. It's much faster than git-filter-branch, though it\n+suffers from all the same safety issues as git-filter-branch, and is\n+still glacially slow compared to\n+https://github.com/newren/git-filter-repo/[git filter-repo].\n+\n+[[SAFETY]]\n+SAFETY\n+------\n+\n+filter-branch is riddled with gotchas resulting in various ways to\n+easily corrupt repos or end up with a mess worse than what you started\n+with:\n+\n+* Someone can have a set of \"working and tested filters\" which they\n+document or provide to a coworker, who then runs them on a different OS\n+where the same commands are not working/tested (some examples in the\n+git-filter-branch manpage are also affected by this). BSD vs. GNU\n+userland differences can really bite. If you're lucky, you get ugly\n+error messages spewed. But just as likely, the commands either don't do\n+the filtering requested, or silently corrupt making some unwanted\n+change. The unwanted change may only affect a few commits, so it's not\n+necessarily obvious either. (The fact that problems won't necessarily\n+be obvious means they are likely to go unnoticed until the rewritten\n+history is in use for quite a while, at which point it's really hard to\n+justify another flag-day for another rewrite.)\n+\n+* Filenames with spaces (which are rare) are often mishandled by shell\n+snippets since they cause problems for shell pipelines. Not everyone is\n+familiar with find -print0, xargs -0, ls-files -z, etc. Even people who\n+are familiar with these may assume such needs are not relevant because\n+someone else renamed any such files in their repo back before the person\n+doing the filtering joined the project. And, often, even those familiar\n+with handling arguments with spaces my not do so just because they\n+aren't in the mindset of thinking about everything that could possibly\n+go wrong.\n+\n+* Non-ascii filenames (which are rare) can be silently removed despite\n+being in a desired directory. The desire to select paths to keep often\n+use pipelines like `git ls-files | grep -v ^WANTED_DIR/ | xargs git rm`.\n+ls-files will only quote filenames if needed so folks may not notice\n+that one of the files didn't match the regex, again until it's much too\n+late. Yes, someone who knows about core.quotePath can avoid this\n+(unless they have other special characters like \\t, \\n, or \"), and\n+people who use ls-files -z with something other than grep can avoid\n+this, but that doesn't mean they will.\n+\n+* Similarly, when moving files around, one can find that filenames with\n+non-ascii or special characters end up in a different directory, one\n+that includes a double quote character. (This is technically the same\n+issue as above with quoting, but perhaps an interesting different way\n+that it can and has manifested as a problem.)\n+\n+* It's far too easy to accidentally mix up old and new history. It's\n+still possible with any tool, but filter-branch almost invites it. If\n+we're lucky, the only downside is users getting frustrated that they\n+don't know how to shrink their repo and remove the old stuff. If we're\n+unlucky, they merge old and new history and end up with multiple\n+\"copies\" of each commit, some of which have unwanted or sensitive files\n+and others which don't. This comes about in multiple different ways:\n+\n+ ** the default to only doing a partial history rewrite ('--all' is not\n+ the default and over 80% of the examples in the manpage don't use\n+ it)\n+\n+ ** the fact that there's no automatic post-run cleanup\n+\n+ ** the fact that --tag-name-filter (when used to rename tags) doesn't\n+ remove the old tags but just adds new ones with the new name (this\n+ manpage has documented this for a long time so it's presumably not\n+ a \"bug\" even though it feels like it)\n+\n+ ** the fact that little educational information is provided to inform\n+ users of the ramifications of a rewrite and how to avoid mixing old\n+ and new history. For example, this man page discusses how users\n+ need to understand that they need to rebase their changes for all\n+ their branches on top of new history (or delete and reclone), but\n+ that's only one of multiple concerns to consider. See the\n+ \"DISCUSSION\" section of the git filter-repo manual page for more\n+ details.\n+\n+* Annotated tags can be accidentally converted to lightweight tags, due\n+to either of two issues:\n+\n+ . Someone can do a history rewrite, realize they messed up, restore\n+ from the backups in refs/original/, and then redo their\n+ filter-branch command. (The backup in refs/original/ is not a real\n+ backup; it dereferences tags first.)\n+\n+ . Running filter-branch with either --tags or --all in your . In order to retain annotated tags as annotated, you must\n+ use --tag-name-filter (and must not have restored from\n+ refs/original/ in a previously botched rewrite).\n+\n+* Any commit messages that specify an encoding will become corrupted\n+by the rewrite; filter-branch ignores the encoding, takes the original\n+bytes, and feeds it to commit-tree without telling it the proper\n+encoding. (This happens whether or not --msg-filter is used, though I\n+suspect --msg-filter provides additional ways to really mess things\n+up).\n+\n+* Commit messages (even if they are all UTF-8) by default become\n+corrupted due to not being updated -- any references to other commit\n+hashes in commit messages will now refer to no-longer-extant commits.\n+\n+* There are no facilities for helping users find what unwanted crud they\n+should delete, which means they are much more likely to have incomplete\n+or partial cleanups that sometimes result in confusion and people\n+wasting time trying to understand. (For example, folks tend to just\n+look for big files to delete instead of big directories or extensions,\n+and once they do so, then sometime later folks using the new repository\n+who are going through history will notice a build artifact directory\n+that has some files but not others, or a cache of dependencies\n+(node_modules or similar) which couldn't have ever been functional since\n+it's missing some files.)\n+\n+* If --prune-empty isn't specified, then the filtering process can\n+create hoards of confusing empty commits\n+\n+* If --prune-empty is specified, then intentionally placed empty\n+commits from before the filtering operation are also pruned instead of\n+just pruning commits that became empty due to filtering rules.\n+\n+* If --prune empty is specified, sometimes empty commits are missed\n+and left around anyway (a somewhat rare bug, but it happens...)\n+\n+* A minor issue, but users who have a goal to update all names and\n+emails in a repository may be led to --env-filter which will only update\n+authors and committers, missing taggers.\n+\n+* If the user provides a --tag-name-filter that maps multiple tags to\n+the same name, no warning or error is provided; filter-branch simply\n+overwrites each tag in some undocumented pre-defined order resulting in\n+only one tag at the end. If you try to \"fix\" this bug in filter-branch\n+and make it error out and warn the user instead, one of the\n+filter-branch regression tests will fail. (So, if you are trying to\n+make a backward compatible reimplementation you have to add extra code\n+to detect collisions and make sure that only the lexicographically last\n+one is rewritten to avoid fast-import from seeing both since fast-import\n+will naturally do the sane thing and error out if told to write the same\n+tag more than once.)\n+\n+Also, the poor performance of filter-branch often leads to safety issues:\n+\n+* Coming up with the correct shell snippet to do the filtering you want\n+is sometimes difficult unless you're just doing a trivial modification\n+such as deleting a couple files. People have often come to me for help,\n+so I should be practiced and an expert, but even for fairly simple cases\n+I still sometimes take over 10 minutes and several iterations to get\n+the right commands -- and that's assuming they are working on a tiny\n+repository. Unfortunately, people often learn if the snippet is right\n+or wrong by trying it out, but the rightness or wrongness can vary\n+depending on special circumstances (spaces in filenames, non-ascii\n+filenames, funny author names or emails, invalid timezones, presence of\n+grafts or replace objects, etc.), meaning they may have to wait a long\n+time, hit an error, then restart. The performance of filter-branch is\n+so bad that this cycle is painful, reducing the time available to\n+carefully re-check (to say nothing about what it does to the patience of\n+the person doing the rewrite even if they do technically have more time\n+available). This problem is extra compounded because errors from broken\n+filters may not be shown for a long time and/or get lost in a sea of\n+output. Even worse, broken filters often just result in silent\n+incorrect rewrites.\n+\n+* To top it all off, even when users finally find working commands, they\n+naturally want to share them. But they may be unaware that their repo\n+didn't have some special cases that someone else's does. So, when\n+someone else with a different repository runs the same commands, they\n+get hit by the problems above. Or, the user just runs commands that\n+really were vetted for special cases, but they run it on a different OS\n+where it doesn't work, as noted above.\n \n GIT\n ---\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 247f765604..0c114ad1ca 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -115,15 +115,14 @@ NOTES\n -----\n \n 'git gc' tries very hard not to delete objects that are referenced\n-anywhere in your repository. In\n-particular, it will keep not only objects referenced by your current set\n-of branches and tags, but also objects referenced by the index,\n-remote-tracking branches, refs saved by 'git filter-branch' in\n-refs/original/, reflogs (which may reference commits in branches\n-that were later amended or rewound), and anything else in the refs/* namespace.\n-If you are expecting some objects to be deleted and they aren't, check\n-all of those locations and decide whether it makes sense in your case to\n-remove those references.\n+anywhere in your repository. In particular, it will keep not only\n+objects referenced by your current set of branches and tags, but also\n+objects referenced by the index, remote-tracking branches, notes saved\n+by 'git notes' under refs/notes/, reflogs (which may reference commits\n+in branches that were later amended or rewound), and anything else in\n+the refs/* namespace. If you are expecting some objects to be deleted\n+and they aren't, check all of those locations and decide whether it\n+makes sense in your case to remove those references.\n \n On the other hand, when 'git gc' runs concurrently with another process,\n there is a risk of it deleting an object that the other process is using\ndiff --git a/Documentation/git-rebase.txt b/Documentation/git-rebase.txt\nindex 6156609cf7..a8cfc0ad82 100644\n--- a/Documentation/git-rebase.txt\n+++ b/Documentation/git-rebase.txt\n@@ -832,7 +832,8 @@ Hard case: The changes are not the same.::\n \tThis happens if the 'subsystem' rebase had conflicts, or used\n \t`--interactive` to omit, edit, squash, or fixup commits; or\n \tif the upstream used one of `commit --amend`, `reset`, or\n-\t`filter-branch`.\n+\ta full history rewriting command like\n+\thttps://github.com/newren/git-filter-repo[`filter-repo`].\n \n \n The easy case\ndiff --git a/Documentation/git-replace.txt b/Documentation/git-replace.txt\nindex 246dc9943c..f271d758c3 100644\n--- a/Documentation/git-replace.txt\n+++ b/Documentation/git-replace.txt\n@@ -123,10 +123,10 @@ The following format are available:\n CREATING REPLACEMENT OBJECTS\n ----------------------------\n \n-linkgit:git-filter-branch[1], linkgit:git-hash-object[1] and\n-linkgit:git-rebase[1], among other git commands, can be used to create\n-replacement objects from existing objects. The `--edit` option can\n-also be used with 'git replace' to create a replacement object by\n+linkgit:git-hash-object[1], linkgit:git-rebase[1], and\n+https://github.com/newren/git-filter-repo[git-filter-repo], among other git commands, can be used to\n+create replacement objects from existing objects. The `--edit` option\n+can also be used with 'git replace' to create a replacement object by\n editing an existing object.\n \n If you want to replace many blobs, trees or commits that are part of a\n@@ -148,13 +148,13 @@ pending objects.\n SEE ALSO\n --------\n linkgit:git-hash-object[1]\n-linkgit:git-filter-branch[1]\n linkgit:git-rebase[1]\n linkgit:git-tag[1]\n linkgit:git-branch[1]\n linkgit:git-commit[1]\n linkgit:git-var[1]\n linkgit:git[1]\n+https://github.com/newren/git-filter-repo[git-filter-repo]\n \n GIT\n ---\ndiff --git a/Documentation/git-svn.txt b/Documentation/git-svn.txt\nindex 30711625fd..53774f5b64 100644\n--- a/Documentation/git-svn.txt\n+++ b/Documentation/git-svn.txt\n@@ -769,11 +769,11 @@ option for (hopefully) obvious reasons.\n +\n This option is NOT recommended as it makes it difficult to track down\n old references to SVN revision numbers in existing documentation, bug\n-reports and archives. If you plan to eventually migrate from SVN to Git\n-and are certain about dropping SVN history, consider\n-linkgit:git-filter-branch[1] instead. filter-branch also allows\n-reformatting of metadata for ease-of-reading and rewriting authorship\n-info for non-\"svn.authorsFile\" users.\n+reports, and archives. If you plan to eventually migrate from SVN to\n+Git and are certain about dropping SVN history, consider\n+https://github.com/newren/git-filter-repo[git-filter-repo] instead.\n+filter-repo also allows reformatting of metadata for ease-of-reading\n+and rewriting authorship info for non-\"svn.authorsFile\" users.\n \n svn.useSvmProps::\n svn-remote..useSvmProps::\ndiff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\nindex 82cd573776..5a789c91df 100644\n--- a/Documentation/githooks.txt\n+++ b/Documentation/githooks.txt\n@@ -425,10 +425,12 @@ post-rewrite\n \n This hook is invoked by commands that rewrite commits\n (linkgit:git-commit[1] when called with `--amend` and\n-linkgit:git-rebase[1]; currently `git filter-branch` does 'not' call\n-it!). Its first argument denotes the command it was invoked by:\n-currently one of `amend` or `rebase`. Further command-dependent\n-arguments may be passed in the future.\n+linkgit:git-rebase[1]; however, full-history (re)writing tools like\n+linkgit:git-fast-import[1] or\n+https://github.com/newren/git-filter-repo[git-filter-repo] typically\n+do not call it!). Its first argument denotes the command it was\n+invoked by: currently one of `amend` or `rebase`. Further\n+command-dependent arguments may be passed in the future.\n \n The hook receives a list of the rewritten commits on stdin, in the\n format\ndiff --git a/contrib/svn-fe/svn-fe.txt b/contrib/svn-fe/svn-fe.txt\nindex a3425f4770..19333fc8df 100644\n--- a/contrib/svn-fe/svn-fe.txt\n+++ b/contrib/svn-fe/svn-fe.txt\n@@ -56,7 +56,7 @@ line. This line has the form `git-svn-id: URL@REVNO UUID`.\n \n The resulting repository will generally require further processing\n to put each project in its own repository and to separate the history\n-of each branch. The 'git filter-branch --subdirectory-filter' command\n+of each branch. The 'git filter-repo --subdirectory-filter' command\n may be useful for this purpose.\n \n BUGS\n@@ -67,5 +67,5 @@ The exit status does not reflect whether an error was detected.\n \n SEE ALSO\n --------\n-git-svn(1), svn2git(1), svk(1), git-filter-branch(1), git-fast-import(1),\n+git-svn(1), svn2git(1), svk(1), git-filter-repo(1), git-fast-import(1),\n https://svn.apache.org/repos/asf/subversion/trunk/notes/dump-load-format.txt\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\nindex 5c5afa2b98..f805965d87 100755\n--- a/git-filter-branch.sh\n+++ b/git-filter-branch.sh\n@@ -83,6 +83,19 @@ set_ident () {\n \tfinish_ident COMMITTER\n }\n \n+if [ -z \"$FILTER_BRANCH_SQUELCH_WARNING\" -a \\\n+ -z \"$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS\" ]; then\n+\tcat <] [--subdirectory-filter ] [--env-filter ]\n \t[--tree-filter ] [--index-filter ]\n \t[--parent-filter ] [--msg-filter ]\n-- \n2.23.0.3.g59c7446927.dirty\n\n"},{"id":"381529","messageId":"CAPig+cQ-E8J9WfH96GKt5j0gb0s0XL5jC3AKyvJK36rpAvucmw@mail.gmail.com","threadId":"51710","inReplyTo":"20190829000657.26415-4-newren@gmail.com","subject":"Re: [PATCH v3 3/4] Recommend git-filter-repo instead of git-filter-branch","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2019-08-29T18:10:55Z","receivedAt":"2019-08-29T18:11:11Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Wed, Aug 28, 2019 at 8:07 PM Elijah Newren wrote:\n> filter-branch suffers from a deluge of disguised dangers that disfigure\n> history rewrites (i.e. deviate from the deliberate changes). [...]\n> Signed-off-by: Elijah Newren \n> ---\n> diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt\n> @@ -16,6 +16,22 @@ SYNOPSIS\n> +WARNING\n> +-------\n> +'git filter-branch' has a plethora of pitfalls that can produce non-obvious\n> +manglings of the intended history rewrite (and can leave you with little\n> +time to investigate such problems since it has such abysmal performance).\n> +These safety and performance issues cannot be backward compatibly fixed and\n> +as such, its use is not recommended. Please use an alternative history\n> +filtering tool such as https://github.com/newren/git-filter-repo/[git\n> +filter-repo]. If you still need to use 'git filter-branch', please\n> +carefully read <> (and <>) to learn about the land\n> +mines of filter-branch, and then vigilantly avoid as many of the hazards\n> +listed there as reasonably possible.\n> +\n> +https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/[detailing\n> +the land mines of filter-branch]\n\nThis stray link looks like leftover gunk from the previous revision.\n\n> +PERFORMANCE\n> +-----------\n> +\n> +The performance of filter-branch is glacially slow; its design makes it\n\nThe rest of this document spells it git-filter-branch or 'git\nfilter-branch', not plain filter-branch.\n\n> +* In editing files, git-filter-branch by design checks out each and\n> +every commit as it existed in the original repo. If your repo has 10\\^5\n> +files and 10\\^5 commits, but each commit only modifies 5 files, then\n> +git-filter-branch will make you do 10\\^10 modifications, despite only\n> +having (at most) 5*10^5 unique blobs.\n> +\n> +* If you try and cheat and try to make filter-branch only work on\n> +files modified in a commit, then two things happen\n\ns/filter-branch/git-&/\n\n> +\n> + . you run into problems with deletions whenever the user is simply\n> + trying to rename files (because attempting to delete files that\n> + don't exist looks like a no-op; it takes some chicanery to remap\n> + deletes across file renames when the renames happen via arbitrary\n> + user-provided shell)\n> +\n> + . even if you succeed at the map-deletes-for-renames chicanery, you\n> + still technically violate backward compatibility because users are\n> + allowed to filter files in ways that depend upon topology of commits\n> + instead of filtering solely based on file contents or names (though\n> + I have never seen any user ever do this).\n\nMaybe avoid first-person:\n\n ... contents or names (though this has not been observed in\n the wild).\n\n> +* filter-branch is written in shell, which is kind of slow. Naturally,\n> +it makes sense to want to rewrite that in some other language. However,\n> +filter-branch documentation states that several additional shell\n> +functions are provided for users to call, e.g. 'map', 'skip_commit',\n> +'git_commit_non_empty_tree'. If filter-branch itself isn't a shell\n> +script, then in order to make those shell functions available to the\n> +users' shell snippets you have to prepend the shell definitions of these\n> +functions to every one of the users' shell snippets and thus make these\n> +special shell functions be parsed with each and every commit.\n> +\n> +* filter-branch provides a --setup option which is a shell snippet that\n> +can be sourced to make shell functions and variables available to all\n> +other filters. If filter-branch is a shell script, it can simply eval\n> +this shell snippet once at the beginning. If you try to fix performance\n> +by making filter-branch not be a shell script, then you have to prepend\n> +the setup shell snippet to all other filters and parse it with every\n> +single commit.\n\nEven though they made sense in the context of the original email\nmessage, these two bullet points may not belong in the man page since\nsomeone reading the man page is doing so to learn about\ngit-filter-branch usage, not because he or she is thinking about\nre-implementing it. It might make sense, however, to collapse these\npoints to some general statement about shell being slow and process\nstartup being costly.\n\nAlso, these bullet points and others below need a s/filter-branch/git-&/.\n\n> +* filter-branch writes lots of files to $workdir/../map/ to keep a\n\nShould that path have three dots \"...\" instead of two \"..\"?\n\n> +mapping of commits, which it uses pruning commits and remapping to\n> +ancestors and the map() command more generally. Other files like\n> +$tempdir/backup-refs, $tempdir/raw-refs, $tempdir/heads,\n> +$tempdir/tree-state are all created internally too. It is possible\n> +(though strongly discouraged) that users could have accessed any of\n> +these directly. Users even had a pointer to follow in the form of\n> +Documentation that the 'map' command existed, which naturally uses the\n> +$workdir/../map/* files. So, even if you don't have to edit files, for\n> +strict backward compatibility you need to still write a bunch of files\n> +to disk somewhere and keep them updated for every commit. You can claim\n> +it was an implementation detail that users should not have depended\n> +upon, but the truth is they've had a decade where they could so. So, if\n> +you want full compatibility, it has to be there. Besides, the\n> +regression tests depend on at least one of these details, specifying an\n> +--index-filter that reaches down and grabs backup-refs from $tempdir,\n> +and thus provides resourceful users who do google searches an example\n> +that there are files there for them to read and grab and use. (And if\n> +you want to pass the existing regression tests, you have to at least put\n> +the backup-refs file there even if it's irrelevant to your\n> +implementation otherwise.)\n\nAs with the earlier comment, this bullet point is aimed at someone\nthinking about re-implementing the command; it sounds out of place in\nthe \"Performance\" section of the man page. However, it does make sense\nto mention all the files git-filter-branch creates since that can have\nan impact on performance. So, perhaps this section can be collapsed so\nit just talks about that.\n\n> +All of that said, performance of filter-branch could be improved by\n> +reimplementing it in a non-shell language and taking a couple small\n> +liberties with backward compatibility (such as having it only run\n> +filters on files changed within each commit). filter-repo provides a\n> +demo script named\n> +https://github.com/newren/git-filter-repo/blob/master/contrib/filter-repo-demos/filter-lamely[filter-lamely]\n> +which does exactly that and which passes all the git-filter-branch\n> +regression tests. It's much faster than git-filter-branch, though it\n> +suffers from all the same safety issues as git-filter-branch, and is\n> +still glacially slow compared to\n> +https://github.com/newren/git-filter-repo/[git filter-repo].\n\nThis paragraph could be collapsed to say merely that, for those with\nexisting tooling relying upon git-filter-branch, filter-repo's\n\"filter-lamely\" provides a drop-in replacement with somewhat improved\nperformance and a few caveats.\n\nTaking the above comments into consideration, here is a possible\nrewrite of the final three bullet points and the closing paragraph:\n\n * filter-branch is written in shell, which is kind of slow, and it\n potentially can run many other commands which can slow down its\n operation significantly, especially on platforms for which\n process startup is costly.\n\n * filter-branch writes lots of files to $workdir/.../map/ to keep\n a mapping of commits, which it uses for pruning commits and\n remapping to ancestors and for the map() command more generally.\n Other files like $tempdir/backup-refs, $tempdir/raw-refs,\n $tempdir/heads, $tempdir/tree-state are created internally too.\n Such file creation can be costly in general, but especially on\n platforms with slow filesystems.\n\n The tool https://github.com/newren/git-filter-repo/[git\n filter-repo] is an alternative to git-filter-branch which does not\n suffer from these performance problems or the safety problems\n (mentioned below). For those with existing tooling which relies\n upon git-filter-branch, 'git repo-filter' also provides\n https://github.com/newren/git-filter-repo/blob/master/contrib/filter-repo-demos/filter-lamely[filter-lamely],\n a drop-in git-filter-branch replacement (with a few caveats).\n\n> +SAFETY\n> +------\n> +\n> +* Non-ascii filenames (which are rare) can be silently removed despite\n\nPerhaps drop \"(which are rare)\" to make this sound more formal and\nless like an email message.\n\nComment below also are intended to make the prose sound a bit more formal.\n\n> +being in a desired directory. The desire to select paths to keep often\n> +use pipelines like `git ls-files | grep -v ^WANTED_DIR/ | xargs git rm`.\n> +ls-files will only quote filenames if needed so folks may not notice\n\ns/ls-files/git-&/\n\n> +that one of the files didn't match the regex, again until it's much too\n> +late. Yes, someone who knows about core.quotePath can avoid this\n> +(unless they have other special characters like \\t, \\n, or \"), and\n> +people who use ls-files -z with something other than grep can avoid\n> +this, but that doesn't mean they will.\n> +\n> +* It's far too easy to accidentally mix up old and new history. It's\n> +still possible with any tool, but filter-branch almost invites it. If\n> +we're lucky, the only downside is users getting frustrated that they\n\ns/we're//\n\n> +don't know how to shrink their repo and remove the old stuff. If we're\n\ns/we're//\n\n> +unlucky, they merge old and new history and end up with multiple\n> +\"copies\" of each commit, some of which have unwanted or sensitive files\n> +and others which don't. This comes about in multiple different ways:\n> +\n> + ** the default to only doing a partial history rewrite ('--all' is not\n> + the default and over 80% of the examples in the manpage don't use\n> + it)\n\nMaybe just shorten this to:\n\n ('--all is not the default, and few examples show it)\n\n> + ** the fact that there's no automatic post-run cleanup\n> +\n> + ** the fact that --tag-name-filter (when used to rename tags) doesn't\n> + remove the old tags but just adds new ones with the new name (this\n> + manpage has documented this for a long time so it's presumably not\n> + a \"bug\" even though it feels like it)\n\nPerhaps drop the final parenthetical comment.\n\n> + ** the fact that little educational information is provided to inform\n> + users of the ramifications of a rewrite and how to avoid mixing old\n> + and new history. For example, this man page discusses how users\n> + need to understand that they need to rebase their changes for all\n> + their branches on top of new history (or delete and reclone), but\n> + that's only one of multiple concerns to consider. See the\n> + \"DISCUSSION\" section of the git filter-repo manual page for more\n> + details.\n> +\n> +* Annotated tags can be accidentally converted to lightweight tags, due\n> +to either of two issues:\n> +\n> + . Someone can do a history rewrite, realize they messed up, restore\n> + from the backups in refs/original/, and then redo their\n> + filter-branch command. (The backup in refs/original/ is not a real\n> + backup; it dereferences tags first.)\n> +\n> + . Running filter-branch with either --tags or --all in your + options>. In order to retain annotated tags as annotated, you must\n> + use --tag-name-filter (and must not have restored from\n> + refs/original/ in a previously botched rewrite).\n\nShould these bullet points use \"**\" rather than \".\"?\n\n> +* Any commit messages that specify an encoding will become corrupted\n> +by the rewrite; filter-branch ignores the encoding, takes the original\n> +bytes, and feeds it to commit-tree without telling it the proper\n> +encoding. (This happens whether or not --msg-filter is used, though I\n> +suspect --msg-filter provides additional ways to really mess things\n> +up).\n\nPerhaps shorten simply to:\n\n (This happens whether or not --msg-filter is used.)\n\n> +* If the user provides a --tag-name-filter that maps multiple tags to\n> +the same name, no warning or error is provided; filter-branch simply\n> +overwrites each tag in some undocumented pre-defined order resulting in\n> +only one tag at the end. If you try to \"fix\" this bug in filter-branch\n> +and make it error out and warn the user instead, one of the\n> +filter-branch regression tests will fail. (So, if you are trying to\n> +make a backward compatible reimplementation you have to add extra code\n> +to detect collisions and make sure that only the lexicographically last\n> +one is rewritten to avoid fast-import from seeing both since fast-import\n> +will naturally do the sane thing and error out if told to write the same\n> +tag more than once.)\n\nMaybe drop everything from \"If you try to 'fix'...\" to the end of paragraph.\n\n> +Also, the poor performance of filter-branch often leads to safety issues:\n> +\n> +* Coming up with the correct shell snippet to do the filtering you want\n> +is sometimes difficult unless you're just doing a trivial modification\n> +such as deleting a couple files. People have often come to me for help,\n> +so I should be practiced and an expert, but even for fairly simple cases\n> +I still sometimes take over 10 minutes and several iterations to get\n> +the right commands -- and that's assuming they are working on a tiny\n> +repository. Unfortunately, people often learn if the snippet is right\n> +or wrong by trying it out, but the rightness or wrongness can vary\n> +depending on special circumstances (spaces in filenames, non-ascii\n> +filenames, funny author names or emails, invalid timezones, presence of\n> +grafts or replace objects, etc.), meaning they may have to wait a long\n> +time, hit an error, then restart. The performance of filter-branch is\n> +so bad that this cycle is painful, reducing the time available to\n> +carefully re-check (to say nothing about what it does to the patience of\n> +the person doing the rewrite even if they do technically have more time\n> +available). This problem is extra compounded because errors from broken\n> +filters may not be shown for a long time and/or get lost in a sea of\n> +output. Even worse, broken filters often just result in silent\n> +incorrect rewrites.\n\nDrop the \"People have often come to me...\" sentence from this paragraph.\n"},{"id":"381556","messageId":"CABPp-BEP94Z6e+SFyFnPPgnhQUn7dJqcmHCbpOiJ4-NDVm-ozA@mail.gmail.com","threadId":"51710","inReplyTo":"CAPig+cQ-E8J9WfH96GKt5j0gb0s0XL5jC3AKyvJK36rpAvucmw@mail.gmail.com","subject":"Re: [PATCH v3 3/4] Recommend git-filter-repo instead of git-filter-branch","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-30T00:04:06Z","receivedAt":"2019-08-30T00:04:22Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Eric,\n\nThanks for the careful and thoughtful review.\n\nOn Thu, Aug 29, 2019 at 11:11 AM Eric Sunshine wrote:\n>\n> On Wed, Aug 28, 2019 at 8:07 PM Elijah Newren wrote:\n> > filter-branch suffers from a deluge of disguised dangers that disfigure\n> > history rewrites (i.e. deviate from the deliberate changes). [...]\n> > Signed-off-by: Elijah Newren \n> > ---\n> > diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt\n> > @@ -16,6 +16,22 @@ SYNOPSIS\n> > +WARNING\n> > +-------\n> > +'git filter-branch' has a plethora of pitfalls that can produce non-obvious\n> > +manglings of the intended history rewrite (and can leave you with little\n> > +time to investigate such problems since it has such abysmal performance).\n> > +These safety and performance issues cannot be backward compatibly fixed and\n> > +as such, its use is not recommended. Please use an alternative history\n> > +filtering tool such as https://github.com/newren/git-filter-repo/[git\n> > +filter-repo]. If you still need to use 'git filter-branch', please\n> > +carefully read <> (and <>) to learn about the land\n> > +mines of filter-branch, and then vigilantly avoid as many of the hazards\n> > +listed there as reasonably possible.\n> > +\n> > +https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/[detailing\n> > +the land mines of filter-branch]\n>\n> This stray link looks like leftover gunk from the previous revision.\n\nUgh, indeed.\n\n>\n> > +PERFORMANCE\n> > +-----------\n> > +\n> > +The performance of filter-branch is glacially slow; its design makes it\n>\n> The rest of this document spells it git-filter-branch or 'git\n> filter-branch', not plain filter-branch.\n>\n> > +* In editing files, git-filter-branch by design checks out each and\n> > +every commit as it existed in the original repo. If your repo has 10\\^5\n> > +files and 10\\^5 commits, but each commit only modifies 5 files, then\n> > +git-filter-branch will make you do 10\\^10 modifications, despite only\n> > +having (at most) 5*10^5 unique blobs.\n> > +\n> > +* If you try and cheat and try to make filter-branch only work on\n> > +files modified in a commit, then two things happen\n>\n> s/filter-branch/git-&/\n\nI can fix these up.\n\n>\n> > +\n> > + . you run into problems with deletions whenever the user is simply\n> > + trying to rename files (because attempting to delete files that\n> > + don't exist looks like a no-op; it takes some chicanery to remap\n> > + deletes across file renames when the renames happen via arbitrary\n> > + user-provided shell)\n> > +\n> > + . even if you succeed at the map-deletes-for-renames chicanery, you\n> > + still technically violate backward compatibility because users are\n> > + allowed to filter files in ways that depend upon topology of commits\n> > + instead of filtering solely based on file contents or names (though\n> > + I have never seen any user ever do this).\n>\n> Maybe avoid first-person:\n>\n> ... contents or names (though this has not been observed in\n> the wild).\n\nThanks for providing alternative wording.\n\n> > +* filter-branch is written in shell, which is kind of slow. Naturally,\n> > +it makes sense to want to rewrite that in some other language. However,\n> > +filter-branch documentation states that several additional shell\n> > +functions are provided for users to call, e.g. 'map', 'skip_commit',\n> > +'git_commit_non_empty_tree'. If filter-branch itself isn't a shell\n> > +script, then in order to make those shell functions available to the\n> > +users' shell snippets you have to prepend the shell definitions of these\n> > +functions to every one of the users' shell snippets and thus make these\n> > +special shell functions be parsed with each and every commit.\n> > +\n> > +* filter-branch provides a --setup option which is a shell snippet that\n> > +can be sourced to make shell functions and variables available to all\n> > +other filters. If filter-branch is a shell script, it can simply eval\n> > +this shell snippet once at the beginning. If you try to fix performance\n> > +by making filter-branch not be a shell script, then you have to prepend\n> > +the setup shell snippet to all other filters and parse it with every\n> > +single commit.\n>\n> Even though they made sense in the context of the original email\n> message, these two bullet points may not belong in the man page since\n> someone reading the man page is doing so to learn about\n> git-filter-branch usage, not because he or she is thinking about\n> re-implementing it. It might make sense, however, to collapse these\n> points to some general statement about shell being slow and process\n> startup being costly.\n\nHmm. I see where you're coming from, but the performance section\nisn't really user actionable stuff anyway; it's just a warning. And I\nhave repeatedly seen over the years the question brought up on the\nlist of \"Can we make filter-branch fast by making it a builtin?\" (Or\n\"Can't _you_ make filter-branch fast by rewriting it in C?\")\n\nI could try to reword it so that there's some general statement about\nshell being slow and process startup being costly, and then add these\ntwo items as sub-bullets to try to stave off that obvious but\nmisguided question from coming up. Or maybe I just add a reference to\nthe original email?\n\n> Also, these bullet points and others below need a s/filter-branch/git-&/.\n\nThanks, will fix.\n\n> > +* filter-branch writes lots of files to $workdir/../map/ to keep a\n>\n> Should that path have three dots \"...\" instead of two \"..\"?\n\nNo, it's a literal parent directory reference. Users have access to\n$workdir; it's where their commands run. There is no name for the\nparent of that directory, other than by appending '/..' to wherever\nthey are. Maybe if I had spelled it as $(pwd)/../map/ it would be\nbetter?\n\nOr maybe I don't need to name the files at all; does it really matter\nto the user?\n\n> > +mapping of commits, which it uses pruning commits and remapping to\n> > +ancestors and the map() command more generally. Other files like\n> > +$tempdir/backup-refs, $tempdir/raw-refs, $tempdir/heads,\n> > +$tempdir/tree-state are all created internally too. It is possible\n> > +(though strongly discouraged) that users could have accessed any of\n> > +these directly. Users even had a pointer to follow in the form of\n> > +Documentation that the 'map' command existed, which naturally uses the\n> > +$workdir/../map/* files. So, even if you don't have to edit files, for\n> > +strict backward compatibility you need to still write a bunch of files\n> > +to disk somewhere and keep them updated for every commit. You can claim\n> > +it was an implementation detail that users should not have depended\n> > +upon, but the truth is they've had a decade where they could so. So, if\n> > +you want full compatibility, it has to be there. Besides, the\n> > +regression tests depend on at least one of these details, specifying an\n> > +--index-filter that reaches down and grabs backup-refs from $tempdir,\n> > +and thus provides resourceful users who do google searches an example\n> > +that there are files there for them to read and grab and use. (And if\n> > +you want to pass the existing regression tests, you have to at least put\n> > +the backup-refs file there even if it's irrelevant to your\n> > +implementation otherwise.)\n>\n> As with the earlier comment, this bullet point is aimed at someone\n> thinking about re-implementing the command; it sounds out of place in\n> the \"Performance\" section of the man page. However, it does make sense\n> to mention all the files git-filter-branch creates since that can have\n> an impact on performance. So, perhaps this section can be collapsed so\n> it just talks about that.\n\nI think there's both a how-performance-affects-user component and a\ncomponent addressing the common incorrect question/statement/thought\nthat filter-branch performance could just be fixed by making it a\nbuiltin. But splitting this may make sense. And maybe the portions\naddressing making-it-a-builtin-wouldn't-fix-it could be a short\nsentence with a link to the original email for more details.\n\n> > +All of that said, performance of filter-branch could be improved by\n> > +reimplementing it in a non-shell language and taking a couple small\n> > +liberties with backward compatibility (such as having it only run\n> > +filters on files changed within each commit). filter-repo provides a\n> > +demo script named\n> > +https://github.com/newren/git-filter-repo/blob/master/contrib/filter-repo-demos/filter-lamely[filter-lamely]\n> > +which does exactly that and which passes all the git-filter-branch\n> > +regression tests. It's much faster than git-filter-branch, though it\n> > +suffers from all the same safety issues as git-filter-branch, and is\n> > +still glacially slow compared to\n> > +https://github.com/newren/git-filter-repo/[git filter-repo].\n>\n> This paragraph could be collapsed to say merely that, for those with\n> existing tooling relying upon git-filter-branch, filter-repo's\n> \"filter-lamely\" provides a drop-in replacement with somewhat improved\n> performance and a few caveats.\n\nSounds good.\n\n> Taking the above comments into consideration, here is a possible\n> rewrite of the final three bullet points and the closing paragraph:\n\nOh, sweet, thanks for providing this. I really like the simplicity of\nyour suggested wording in general; it will be really helpful in\nrewording. I do have a nitpick with each one, though...\n\n> * filter-branch is written in shell, which is kind of slow, and it\n> potentially can run many other commands which can slow down its\n> operation significantly, especially on platforms for which\n> process startup is costly.\n\nEven if it's not the emphasis you intended, I'm worried this makes it\nsound as if filter-branch performance is only bad on Windows or Mac.\nCompared to invoking a function (even in a bytecode interpreted\nlanguage), creating and running another process is slow on any\nplatform.\n\n> * filter-branch writes lots of files to $workdir/.../map/ to keep\n> a mapping of commits, which it uses for pruning commits and\n> remapping to ancestors and for the map() command more generally.\n> Other files like $tempdir/backup-refs, $tempdir/raw-refs,\n> $tempdir/heads, $tempdir/tree-state are created internally too.\n> Such file creation can be costly in general, but especially on\n> platforms with slow filesystems.\n\nAgain, it may not have been your intended emphasis, but I think this\nmay be read as singling out slow filesystems, and make people think\nthe performance problems from this bullet point only affects some\nOSes. Filesystems are part of the problem. Disks being slow is part\nof the problem. But it's not all of it. I guess part of what really\ngets me with these is that they represent forced synchronization (e.g.\nthe kernel has to flush the data upon close() to make sure any other\nprocesses can see the file contents and all further filtering is\nblocked waiting for this to finish). By way of comparison, in\nfilter-repo I have to both write data to fast-import and read back\ninformation from fast-import (in order to find out the new commit\nnames, for example). When I did the straightforward thing of writing\na commit, writing a 'get-mark' directive, and then reading the answer,\nit ruined performance. So I had to be a bit smarter and defer reading\nback the resulting sha1. There's no room for anything similarly\nclever in filter-branch; writing these files out is a synchronization\npoint that is needed before the user's filter can be eval'ed.\n\n> The tool https://github.com/newren/git-filter-repo/[git\n> filter-repo] is an alternative to git-filter-branch which does not\n> suffer from these performance problems or the safety problems\n> (mentioned below). For those with existing tooling which relies\n> upon git-filter-branch, 'git repo-filter' also provides\n> https://github.com/newren/git-filter-repo/blob/master/contrib/filter-repo-demos/filter-lamely[filter-lamely],\n> a drop-in git-filter-branch replacement (with a few caveats).\n\nThis suggests filter-lamely doesn't suffer from performance or safety\nproblems, which is very misleading. filter-lamely doesn't improve the\nsafety story at all and only ameliorates the performance problems\nsomewhat.\n\n> > +SAFETY\n> > +------\n> > +\n> > +* Non-ascii filenames (which are rare) can be silently removed despite\n>\n> Perhaps drop \"(which are rare)\" to make this sound more formal and\n> less like an email message.\n\nMakes sense; and I'm guessing I should also drop it from the bullet\npoint above this one.\n\nI'll stop commenting on the individual comments since there's not much\nto say with most of them other than they look like obviously good\nsuggestions...\n\n> Comment below also are intended to make the prose sound a bit more formal.\n>\n> > +being in a desired directory. The desire to select paths to keep often\n> > +use pipelines like `git ls-files | grep -v ^WANTED_DIR/ | xargs git rm`.\n> > +ls-files will only quote filenames if needed so folks may not notice\n>\n> s/ls-files/git-&/\n>\n> > +that one of the files didn't match the regex, again until it's much too\n> > +late. Yes, someone who knows about core.quotePath can avoid this\n> > +(unless they have other special characters like \\t, \\n, or \"), and\n> > +people who use ls-files -z with something other than grep can avoid\n> > +this, but that doesn't mean they will.\n> > +\n> > +* It's far too easy to accidentally mix up old and new history. It's\n> > +still possible with any tool, but filter-branch almost invites it. If\n> > +we're lucky, the only downside is users getting frustrated that they\n>\n> s/we're//\n>\n> > +don't know how to shrink their repo and remove the old stuff. If we're\n>\n> s/we're//\n>\n> > +unlucky, they merge old and new history and end up with multiple\n> > +\"copies\" of each commit, some of which have unwanted or sensitive files\n> > +and others which don't. This comes about in multiple different ways:\n> > +\n> > + ** the default to only doing a partial history rewrite ('--all' is not\n> > + the default and over 80% of the examples in the manpage don't use\n> > + it)\n>\n> Maybe just shorten this to:\n>\n> ('--all is not the default, and few examples show it)\n\nI know I said I'd not comment unless I disagreed, but I just wanted to\nsay thanks so much for providing concrete suggestions in so many\nplaces. It's *very* helpful.\n\n> > + ** the fact that there's no automatic post-run cleanup\n> > +\n> > + ** the fact that --tag-name-filter (when used to rename tags) doesn't\n> > + remove the old tags but just adds new ones with the new name (this\n> > + manpage has documented this for a long time so it's presumably not\n> > + a \"bug\" even though it feels like it)\n>\n> Perhaps drop the final parenthetical comment.\n>\n> > + ** the fact that little educational information is provided to inform\n> > + users of the ramifications of a rewrite and how to avoid mixing old\n> > + and new history. For example, this man page discusses how users\n> > + need to understand that they need to rebase their changes for all\n> > + their branches on top of new history (or delete and reclone), but\n> > + that's only one of multiple concerns to consider. See the\n> > + \"DISCUSSION\" section of the git filter-repo manual page for more\n> > + details.\n> > +\n> > +* Annotated tags can be accidentally converted to lightweight tags, due\n> > +to either of two issues:\n> > +\n> > + . Someone can do a history rewrite, realize they messed up, restore\n> > + from the backups in refs/original/, and then redo their\n> > + filter-branch command. (The backup in refs/original/ is not a real\n> > + backup; it dereferences tags first.)\n> > +\n> > + . Running filter-branch with either --tags or --all in your > + options>. In order to retain annotated tags as annotated, you must\n> > + use --tag-name-filter (and must not have restored from\n> > + refs/original/ in a previously botched rewrite).\n>\n> Should these bullet points use \"**\" rather than \".\"?\n\nI guess it could but the \"either of two issues\" above it made me think\nof numbering them. I also had a couple sub-bullets in the performance\nsection that were numbered. But I guess it is slightly weird coming\nso close after another section that used un-numbered subbullets. I\nguess I'll just make them all un-numbered.\n\n> > +* Any commit messages that specify an encoding will become corrupted\n> > +by the rewrite; filter-branch ignores the encoding, takes the original\n> > +bytes, and feeds it to commit-tree without telling it the proper\n> > +encoding. (This happens whether or not --msg-filter is used, though I\n> > +suspect --msg-filter provides additional ways to really mess things\n> > +up).\n>\n> Perhaps shorten simply to:\n>\n> (This happens whether or not --msg-filter is used.)\n>\n> > +* If the user provides a --tag-name-filter that maps multiple tags to\n> > +the same name, no warning or error is provided; filter-branch simply\n> > +overwrites each tag in some undocumented pre-defined order resulting in\n> > +only one tag at the end. If you try to \"fix\" this bug in filter-branch\n> > +and make it error out and warn the user instead, one of the\n> > +filter-branch regression tests will fail. (So, if you are trying to\n> > +make a backward compatible reimplementation you have to add extra code\n> > +to detect collisions and make sure that only the lexicographically last\n> > +one is rewritten to avoid fast-import from seeing both since fast-import\n> > +will naturally do the sane thing and error out if told to write the same\n> > +tag more than once.)\n>\n> Maybe drop everything from \"If you try to 'fix'...\" to the end of paragraph.\n\nOr just replace that long section you highlight with a parenthetical\ncomment, \"(a git-filter-branch regression test requires this.)\"\n\n> > +Also, the poor performance of filter-branch often leads to safety issues:\n> > +\n> > +* Coming up with the correct shell snippet to do the filtering you want\n> > +is sometimes difficult unless you're just doing a trivial modification\n> > +such as deleting a couple files. People have often come to me for help,\n> > +so I should be practiced and an expert, but even for fairly simple cases\n> > +I still sometimes take over 10 minutes and several iterations to get\n> > +the right commands -- and that's assuming they are working on a tiny\n> > +repository. Unfortunately, people often learn if the snippet is right\n> > +or wrong by trying it out, but the rightness or wrongness can vary\n> > +depending on special circumstances (spaces in filenames, non-ascii\n> > +filenames, funny author names or emails, invalid timezones, presence of\n> > +grafts or replace objects, etc.), meaning they may have to wait a long\n> > +time, hit an error, then restart. The performance of filter-branch is\n> > +so bad that this cycle is painful, reducing the time available to\n> > +carefully re-check (to say nothing about what it does to the patience of\n> > +the person doing the rewrite even if they do technically have more time\n> > +available). This problem is extra compounded because errors from broken\n> > +filters may not be shown for a long time and/or get lost in a sea of\n> > +output. Even worse, broken filters often just result in silent\n> > +incorrect rewrites.\n>\n> Drop the \"People have often come to me...\" sentence from this paragraph.\n\n\nThanks again for the careful reading and many suggestions!\n"},{"id":"381562","messageId":"20190830055707.7785-1-newren@gmail.com","threadId":"51710","inReplyTo":"20190829000657.26415-1-newren@gmail.com","subject":"[PATCH v4 0/4] Warn about git-filter-branch usage and avoid it","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-30T05:57:03Z","receivedAt":"2019-08-30T05:57:20Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Here's a series warns about git-filter-branch usage and avoids it\nourselves.\n\nChanges since v3\n * Incorporated Eric's detailed feedback on the git-filter-branch\n manpage, some notes:\n * s/filter-branch/git-&/ (and similar for ls-files)\n * Multiple sections removed (and existing sections had a\n number of sentences removed)\n * I ended up not linking to the original html, but just added\n a small \"Side Note\" in a sub-bullet to address how fixing the\n\twritten-in-shell attribute of git-filter-branch would do less\n\tthan proponents expect.\n * ...and lots of other miscellaneous wording fixes and cleanups\n * The full range-diff is below, but it's kinda hard to read due to\n line wrapping and such.\n\nElijah Newren (4):\n t6006: simplify and optimize empty message test\n t3427: accelerate this test by using fast-export and fast-import\n Recommend git-filter-repo instead of git-filter-branch\n t9902: use a non-deprecated command for testing\n\n Documentation/git-fast-export.txt | 6 +-\n Documentation/git-filter-branch.txt | 272 +++++++++++++++++++++++++---\n Documentation/git-gc.txt | 17 +-\n Documentation/git-rebase.txt | 3 +-\n Documentation/git-replace.txt | 10 +-\n Documentation/git-svn.txt | 10 +-\n Documentation/githooks.txt | 10 +-\n contrib/svn-fe/svn-fe.txt | 4 +-\n git-filter-branch.sh | 13 ++\n t/t3427-rebase-subtree.sh | 24 ++-\n t/t6006-rev-list-format.sh | 5 +-\n t/t9902-completion.sh | 12 +-\n 12 files changed, 309 insertions(+), 77 deletions(-)\n\nRange-diff:\n1: 7ddbeea2ca = 1: 7ddbeea2ca t6006: simplify and optimize empty message test\n2: e1e63189c1 = 2: e1e63189c1 t3427: accelerate this test by using fast-export and fast-import\n3: 59c7446927 ! 3: ed6505584f Recommend git-filter-repo instead of git-filter-branch\n @@ Documentation/git-filter-branch.txt: SYNOPSIS\n +carefully read <> (and <>) to learn about the land\n +mines of filter-branch, and then vigilantly avoid as many of the hazards\n +listed there as reasonably possible.\n -+\n -+https://public-inbox.org/git/CABPp-BEDOH-row-hxY4u_cP30ptqOpcCvPibwyZ2wBu142qUbA@mail.gmail.com/[detailing\n -+the land mines of filter-branch]\n +\n DESCRIPTION\n -----------\n @@ Documentation/git-filter-branch.txt: warned.\n +PERFORMANCE\n +-----------\n +\n -+The performance of filter-branch is glacially slow; its design makes it\n ++The performance of git-filter-branch is glacially slow; its design makes it\n +impossible for a backward-compatible implementation to ever be fast:\n +\n +* In editing files, git-filter-branch by design checks out each and\n @@ Documentation/git-filter-branch.txt: warned.\n +git-filter-branch will make you do 10\\^10 modifications, despite only\n +having (at most) 5*10^5 unique blobs.\n +\n -+* If you try and cheat and try to make filter-branch only work on\n ++* If you try and cheat and try to make git-filter-branch only work on\n +files modified in a commit, then two things happen\n +\n -+ . you run into problems with deletions whenever the user is simply\n -+ trying to rename files (because attempting to delete files that\n -+ don't exist looks like a no-op; it takes some chicanery to remap\n -+ deletes across file renames when the renames happen via arbitrary\n -+ user-provided shell)\n ++ ** you run into problems with deletions whenever the user is simply\n ++ trying to rename files (because attempting to delete files that\n ++ don't exist looks like a no-op; it takes some chicanery to remap\n ++ deletes across file renames when the renames happen via arbitrary\n ++ user-provided shell)\n +\n -+ . even if you succeed at the map-deletes-for-renames chicanery, you\n -+ still technically violate backward compatibility because users are\n -+ allowed to filter files in ways that depend upon topology of commits\n -+ instead of filtering solely based on file contents or names (though\n -+ I have never seen any user ever do this).\n ++ ** even if you succeed at the map-deletes-for-renames chicanery, you\n ++ still technically violate backward compatibility because users are\n ++ allowed to filter files in ways that depend upon topology of\n ++ commits instead of filtering solely based on file contents or names\n ++ (though this has not been observed in the wild).\n +\n +* Even if you don't need to edit files but only want to e.g. rename or\n +remove some and thus can avoid checking out each file (i.e. you can use\n +--index-filter), you still are passing shell snippets for your filters.\n +This means that for every commit, you have to have a prepared git repo\n -+where users can run git commands. That's a lot of setup. It also means\n -+you have to fork at least one process to run the user-provided shell\n -+snippet, and odds are that the user's shell snippet invokes lots of\n -+commands in some long pipeline, so you will have lots and lots of forks.\n -+For every. single. commit. That's a massive amount of overhead to\n -+rename a few files.\n -+\n -+* filter-branch is written in shell, which is kind of slow. Naturally,\n -+it makes sense to want to rewrite that in some other language. However,\n -+filter-branch documentation states that several additional shell\n -+functions are provided for users to call, e.g. 'map', 'skip_commit',\n -+'git_commit_non_empty_tree'. If filter-branch itself isn't a shell\n -+script, then in order to make those shell functions available to the\n -+users' shell snippets you have to prepend the shell definitions of these\n -+functions to every one of the users' shell snippets and thus make these\n -+special shell functions be parsed with each and every commit.\n -+\n -+* filter-branch provides a --setup option which is a shell snippet that\n -+can be sourced to make shell functions and variables available to all\n -+other filters. If filter-branch is a shell script, it can simply eval\n -+this shell snippet once at the beginning. If you try to fix performance\n -+by making filter-branch not be a shell script, then you have to prepend\n -+the setup shell snippet to all other filters and parse it with every\n -+single commit.\n -+\n -+* filter-branch writes lots of files to $workdir/../map/ to keep a\n -+mapping of commits, which it uses pruning commits and remapping to\n -+ancestors and the map() command more generally. Other files like\n -+$tempdir/backup-refs, $tempdir/raw-refs, $tempdir/heads,\n -+$tempdir/tree-state are all created internally too. It is possible\n -+(though strongly discouraged) that users could have accessed any of\n -+these directly. Users even had a pointer to follow in the form of\n -+Documentation that the 'map' command existed, which naturally uses the\n -+$workdir/../map/* files. So, even if you don't have to edit files, for\n -+strict backward compatibility you need to still write a bunch of files\n -+to disk somewhere and keep them updated for every commit. You can claim\n -+it was an implementation detail that users should not have depended\n -+upon, but the truth is they've had a decade where they could so. So, if\n -+you want full compatibility, it has to be there. Besides, the\n -+regression tests depend on at least one of these details, specifying an\n -+--index-filter that reaches down and grabs backup-refs from $tempdir,\n -+and thus provides resourceful users who do google searches an example\n -+that there are files there for them to read and grab and use. (And if\n -+you want to pass the existing regression tests, you have to at least put\n -+the backup-refs file there even if it's irrelevant to your\n -+implementation otherwise.)\n -+\n -+All of that said, performance of filter-branch could be improved by\n -+reimplementing it in a non-shell language and taking a couple small\n -+liberties with backward compatibility (such as having it only run\n -+filters on files changed within each commit). filter-repo provides a\n -+demo script named\n -+https://github.com/newren/git-filter-repo/blob/master/contrib/filter-repo-demos/filter-lamely[filter-lamely]\n -+which does exactly that and which passes all the git-filter-branch\n -+regression tests. It's much faster than git-filter-branch, though it\n -+suffers from all the same safety issues as git-filter-branch, and is\n -+still glacially slow compared to\n -+https://github.com/newren/git-filter-repo/[git filter-repo].\n ++where those filters can be run. That's a significant setup.\n ++\n ++* Further, several additional files are created or updated per commit by\n ++git-filter-branch. Some of these are for supporting the convenience\n ++functions provided by git-filter-branch (such as map()), while others\n ++are for keeping track of internal state (but could have also been\n ++accessed by user filters; one of git-filter-branch's regression tests\n ++does so). This essentially amounts to using the filesystem as an IPC\n ++mechanism between git-filter-branch and the user-provided filters.\n ++Disks tend to be a slow IPC mechanism, and writing these files also\n ++effectively represents a forced synchronization point between separate\n ++processes that we hit with every commit.\n ++\n ++* The user-provided shell commands will likely involve a pipeline of\n ++commands, resulting in the creation of many processes per commit.\n ++Creating and running another process takes a widely varying amount of\n ++time between operating systems, but on any platform it is very slow\n ++relative to invoking a function.\n ++\n ++* git-filter-branch itself is written in shell, which is kind of slow.\n ++This is the one performance issue that could be backward-compatibly\n ++fixed, but compared to the above problems that are intrinsic to the\n ++design of git-filter-branch, the language of the tool itself is a\n ++relatively minor issue.\n ++\n ++ ** Side note: Unfortunately, people tend to fixate on the\n ++ written-in-shell aspect and periodically ask if git-filter-branch\n ++ could be rewritten in another language to fix the performance\n ++ issues. Not only does that ignore the bigger intrinsic problems\n ++ with the design, it'd help less than you'd expect: if\n ++ git-filter-branch itself were not shell, then the convenience\n ++ functions (map(), skip_commit(), etc) and the `--setup` argument\n ++ could no longer be executed once at the beginning of the program\n ++ but would instead need to be prepended to every user filter (and\n ++ thus re-executed with every commit).\n ++\n ++The https://github.com/newren/git-filter-repo/[git filter-repo] tool is\n ++an alternative to git-filter-branch which does not suffer from these\n ++performance problems or the safety problems (mentioned below). For those\n ++with existing tooling which relies upon git-filter-branch, 'git\n ++repo-filter' also provides\n ++https://github.com/newren/git-filter-repo/blob/master/contrib/filter-repo-demos/filter-lamely[filter-lamely],\n ++a drop-in git-filter-branch replacement (with a few caveats). While\n ++filter-lamely suffers from all the same safety issues as\n ++git-filter-branch, it at least ameloriates the performance issues a\n ++little.\n +\n +[[SAFETY]]\n +SAFETY\n +------\n +\n -+filter-branch is riddled with gotchas resulting in various ways to\n ++git-filter-branch is riddled with gotchas resulting in various ways to\n +easily corrupt repos or end up with a mess worse than what you started\n +with:\n +\n @@ Documentation/git-filter-branch.txt: warned.\n +history is in use for quite a while, at which point it's really hard to\n +justify another flag-day for another rewrite.)\n +\n -+* Filenames with spaces (which are rare) are often mishandled by shell\n -+snippets since they cause problems for shell pipelines. Not everyone is\n -+familiar with find -print0, xargs -0, ls-files -z, etc. Even people who\n -+are familiar with these may assume such needs are not relevant because\n ++* Filenames with spaces are often mishandled by shell snippets since\n ++they cause problems for shell pipelines. Not everyone is familiar with\n ++find -print0, xargs -0, git-ls-files -z, etc. Even people who are\n ++familiar with these may assume such needs are not relevant because\n +someone else renamed any such files in their repo back before the person\n +doing the filtering joined the project. And, often, even those familiar\n +with handling arguments with spaces my not do so just because they\n +aren't in the mindset of thinking about everything that could possibly\n +go wrong.\n +\n -+* Non-ascii filenames (which are rare) can be silently removed despite\n -+being in a desired directory. The desire to select paths to keep often\n -+use pipelines like `git ls-files | grep -v ^WANTED_DIR/ | xargs git rm`.\n -+ls-files will only quote filenames if needed so folks may not notice\n -+that one of the files didn't match the regex, again until it's much too\n -+late. Yes, someone who knows about core.quotePath can avoid this\n -+(unless they have other special characters like \\t, \\n, or \"), and\n -+people who use ls-files -z with something other than grep can avoid\n -+this, but that doesn't mean they will.\n ++* Non-ascii filenames can be silently removed despite being in a desired\n ++directory. The desire to select paths to keep often use pipelines like\n ++`git ls-files | grep -v ^WANTED_DIR/ | xargs git rm`. ls-files will\n ++only quote filenames if needed so folks may not notice that one of the\n ++files didn't match the regex, again until it's much too late. Yes,\n ++someone who knows about core.quotePath can avoid this (unless they have\n ++other special characters like \\t, \\n, or \"), and people who use ls-files\n ++-z with something other than grep can avoid this, but that doesn't mean\n ++they will.\n +\n +* Similarly, when moving files around, one can find that filenames with\n +non-ascii or special characters end up in a different directory, one\n @@ Documentation/git-filter-branch.txt: warned.\n +that it can and has manifested as a problem.)\n +\n +* It's far too easy to accidentally mix up old and new history. It's\n -+still possible with any tool, but filter-branch almost invites it. If\n -+we're lucky, the only downside is users getting frustrated that they\n -+don't know how to shrink their repo and remove the old stuff. If we're\n -+unlucky, they merge old and new history and end up with multiple\n -+\"copies\" of each commit, some of which have unwanted or sensitive files\n -+and others which don't. This comes about in multiple different ways:\n ++still possible with any tool, but git-filter-branch almost invites it.\n ++If lucky, the only downside is users getting frustrated that they don't\n ++know how to shrink their repo and remove the old stuff. If unlucky,\n ++they merge old and new history and end up with multiple \"copies\" of each\n ++commit, some of which have unwanted or sensitive files and others which\n ++don't. This comes about in multiple different ways:\n +\n + ** the default to only doing a partial history rewrite ('--all' is not\n -+ the default and over 80% of the examples in the manpage don't use\n -+ it)\n ++ the default and few examples show it)\n +\n + ** the fact that there's no automatic post-run cleanup\n +\n + ** the fact that --tag-name-filter (when used to rename tags) doesn't\n -+ remove the old tags but just adds new ones with the new name (this\n -+ manpage has documented this for a long time so it's presumably not\n -+ a \"bug\" even though it feels like it)\n ++ remove the old tags but just adds new ones with the new name\n +\n + ** the fact that little educational information is provided to inform\n + users of the ramifications of a rewrite and how to avoid mixing old\n @@ Documentation/git-filter-branch.txt: warned.\n +* Annotated tags can be accidentally converted to lightweight tags, due\n +to either of two issues:\n +\n -+ . Someone can do a history rewrite, realize they messed up, restore\n -+ from the backups in refs/original/, and then redo their\n -+ filter-branch command. (The backup in refs/original/ is not a real\n -+ backup; it dereferences tags first.)\n ++ ** Someone can do a history rewrite, realize they messed up, restore\n ++ from the backups in refs/original/, and then redo their\n ++ git-filter-branch command. (The backup in refs/original/ is not a\n ++ real backup; it dereferences tags first.)\n +\n -+ . Running filter-branch with either --tags or --all in your . In order to retain annotated tags as annotated, you must\n -+ use --tag-name-filter (and must not have restored from\n -+ refs/original/ in a previously botched rewrite).\n ++ ** Running git-filter-branch with either --tags or --all in your\n ++ . In order to retain annotated tags as\n ++ annotated, you must use --tag-name-filter (and must not have\n ++ restored from refs/original/ in a previously botched rewrite).\n +\n +* Any commit messages that specify an encoding will become corrupted\n -+by the rewrite; filter-branch ignores the encoding, takes the original\n ++by the rewrite; git-filter-branch ignores the encoding, takes the original\n +bytes, and feeds it to commit-tree without telling it the proper\n -+encoding. (This happens whether or not --msg-filter is used, though I\n -+suspect --msg-filter provides additional ways to really mess things\n -+up).\n ++encoding. (This happens whether or not --msg-filter is used.)\n +\n +* Commit messages (even if they are all UTF-8) by default become\n +corrupted due to not being updated -- any references to other commit\n @@ Documentation/git-filter-branch.txt: warned.\n +authors and committers, missing taggers.\n +\n +* If the user provides a --tag-name-filter that maps multiple tags to\n -+the same name, no warning or error is provided; filter-branch simply\n ++the same name, no warning or error is provided; git-filter-branch simply\n +overwrites each tag in some undocumented pre-defined order resulting in\n -+only one tag at the end. If you try to \"fix\" this bug in filter-branch\n -+and make it error out and warn the user instead, one of the\n -+filter-branch regression tests will fail. (So, if you are trying to\n -+make a backward compatible reimplementation you have to add extra code\n -+to detect collisions and make sure that only the lexicographically last\n -+one is rewritten to avoid fast-import from seeing both since fast-import\n -+will naturally do the sane thing and error out if told to write the same\n -+tag more than once.)\n ++only one tag at the end. (A git-filter-branch regression test requires\n ++this.)\n +\n -+Also, the poor performance of filter-branch often leads to safety issues:\n ++Also, the poor performance of git-filter-branch often leads to safety issues:\n +\n +* Coming up with the correct shell snippet to do the filtering you want\n +is sometimes difficult unless you're just doing a trivial modification\n -+such as deleting a couple files. People have often come to me for help,\n -+so I should be practiced and an expert, but even for fairly simple cases\n -+I still sometimes take over 10 minutes and several iterations to get\n -+the right commands -- and that's assuming they are working on a tiny\n -+repository. Unfortunately, people often learn if the snippet is right\n -+or wrong by trying it out, but the rightness or wrongness can vary\n -+depending on special circumstances (spaces in filenames, non-ascii\n -+filenames, funny author names or emails, invalid timezones, presence of\n -+grafts or replace objects, etc.), meaning they may have to wait a long\n -+time, hit an error, then restart. The performance of filter-branch is\n -+so bad that this cycle is painful, reducing the time available to\n -+carefully re-check (to say nothing about what it does to the patience of\n -+the person doing the rewrite even if they do technically have more time\n -+available). This problem is extra compounded because errors from broken\n -+filters may not be shown for a long time and/or get lost in a sea of\n -+output. Even worse, broken filters often just result in silent\n -+incorrect rewrites.\n ++such as deleting a couple files. Unfortunately, people often learn if\n ++the snippet is right or wrong by trying it out, but the rightness or\n ++wrongness can vary depending on special circumstances (spaces in\n ++filenames, non-ascii filenames, funny author names or emails, invalid\n ++timezones, presence of grafts or replace objects, etc.), meaning they\n ++may have to wait a long time, hit an error, then restart. The\n ++performance of git-filter-branch is so bad that this cycle is painful,\n ++reducing the time available to carefully re-check (to say nothing about\n ++what it does to the patience of the person doing the rewrite even if\n ++they do technically have more time available). This problem is extra\n ++compounded because errors from broken filters may not be shown for a\n ++long time and/or get lost in a sea of output. Even worse, broken\n ++filters often just result in silent incorrect rewrites.\n +\n +* To top it all off, even when users finally find working commands, they\n +naturally want to share them. But they may be unaware that their repo\n4: 1dbca82408 = 4: ca8e124cb3 t9902: use a non-deprecated command for testing\n5: 762d63d6a5 < -: ---------- Remove git-filter-branch, it is now external to git.git\n-- \n2.23.0.38.g892688c90e\n\n"},{"id":"381563","messageId":"20190830055707.7785-2-newren@gmail.com","threadId":"51710","inReplyTo":"20190830055707.7785-1-newren@gmail.com","subject":"[PATCH v4 1/4] t6006: simplify and optimize empty message test","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-30T05:57:04Z","receivedAt":"2019-08-30T05:57:20Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Test t6006.71 (\"oneline with empty message\") was creating two commits\nwith simple commit messages, and then running filter-branch to rewrite\nthe commit messages to be empty. This test was written this way because\nthe --allow-empty-message option to git commit did not exist at the\ntime. Simplify this test and avoid the need to invoke filter-branch by\njust using --allow-empty-message when creating the commit.\n\nDespite only being one piece of the 71st test and there being 73 tests\noverall, this small change to just this one test speeds up the overall\nexecution time of t6006 (as measured by the best of 3 runs of `time\n./t6006-rev-list-format.sh`) by about 11% on Linux and by 13% on\nMac.\n\nSigned-off-by: Elijah Newren \n---\n t/t6006-rev-list-format.sh | 5 ++---\n 1 file changed, 2 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t6006-rev-list-format.sh b/t/t6006-rev-list-format.sh\nindex da113d975b..d30e41c9f7 100755\n--- a/t/t6006-rev-list-format.sh\n+++ b/t/t6006-rev-list-format.sh\n@@ -501,9 +501,8 @@ test_expect_success 'reflog identity' '\n '\n \n test_expect_success 'oneline with empty message' '\n-\tgit commit -m \"dummy\" --allow-empty &&\n-\tgit commit -m \"dummy\" --allow-empty &&\n-\tgit filter-branch --msg-filter \"sed -e s/dummy//\" HEAD^^.. &&\n+\tgit commit --allow-empty --allow-empty-message &&\n+\tgit commit --allow-empty --allow-empty-message &&\n \tgit rev-list --oneline HEAD >test.txt &&\n \ttest_line_count = 5 test.txt &&\n \tgit rev-list --oneline --graph HEAD >testg.txt &&\n-- \n2.23.0.38.g892688c90e\n\n"},{"id":"381564","messageId":"20190830055707.7785-3-newren@gmail.com","threadId":"51710","inReplyTo":"20190830055707.7785-1-newren@gmail.com","subject":"[PATCH v4 2/4] t3427: accelerate this test by using fast-export and fast-import","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-30T05:57:05Z","receivedAt":"2019-08-30T05:57:21Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"fast-export and fast-import can easily handle the simple rewrite that\nwas being done by filter-branch, and should be significantly faster on\nsystems with a slow fork. Timings from before and after on two laptops\nthat I have access to (measured via `time ./t3427-rebase-subtree.sh`,\ni.e. including everything in this test -- not just the filter-branch or\nfast-export/fast-import pair):\n\n Linux: 4.305s -> 3.684s (~17% speedup)\n Mac: 10.128s -> 7.038s (~30% speedup)\n\nSigned-off-by: Elijah Newren \n---\n t/t3427-rebase-subtree.sh | 24 +++++++++++++++---------\n 1 file changed, 15 insertions(+), 9 deletions(-)\n\ndiff --git a/t/t3427-rebase-subtree.sh b/t/t3427-rebase-subtree.sh\nindex d8640522a0..c1f6102921 100755\n--- a/t/t3427-rebase-subtree.sh\n+++ b/t/t3427-rebase-subtree.sh\n@@ -7,10 +7,16 @@ This test runs git rebase and tests the subtree strategy.\n . ./test-lib.sh\n . \"$TEST_DIRECTORY\"/lib-rebase.sh\n \n-commit_message() {\n+commit_message () {\n \tgit log --pretty=format:%s -1 \"$1\"\n }\n \n+extract_files_subtree () {\n+\tgit fast-export --no-data HEAD -- files_subtree/ |\n+\t\tsed -e \"s%\\([0-9a-f]\\{40\\} \\)files_subtree/%\\1%\" |\n+\t\tgit fast-import --force --quiet\n+}\n+\n test_expect_success 'setup' '\n \ttest_commit README &&\n \tmkdir files &&\n@@ -42,7 +48,7 @@ test_expect_failure REBASE_P \\\n \t'Rebase -Xsubtree --preserve-merges --onto commit 4' '\n \treset_rebase &&\n \tgit checkout -b rebase-preserve-merges-4 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --preserve-merges --onto files-master master &&\n \tverbose test \"$(commit_message HEAD~)\" = \"files_subtree/master4\"\n@@ -53,7 +59,7 @@ test_expect_failure REBASE_P \\\n \t'Rebase -Xsubtree --preserve-merges --onto commit 5' '\n \treset_rebase &&\n \tgit checkout -b rebase-preserve-merges-5 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --preserve-merges --onto files-master master &&\n \tverbose test \"$(commit_message HEAD)\" = \"files_subtree/master5\"\n@@ -64,7 +70,7 @@ test_expect_failure REBASE_P \\\n \t'Rebase -Xsubtree --keep-empty --preserve-merges --onto commit 4' '\n \treset_rebase &&\n \tgit checkout -b rebase-keep-empty-4 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master &&\n \tverbose test \"$(commit_message HEAD~2)\" = \"files_subtree/master4\"\n@@ -75,7 +81,7 @@ test_expect_failure REBASE_P \\\n \t'Rebase -Xsubtree --keep-empty --preserve-merges --onto commit 5' '\n \treset_rebase &&\n \tgit checkout -b rebase-keep-empty-5 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master &&\n \tverbose test \"$(commit_message HEAD~)\" = \"files_subtree/master5\"\n@@ -86,7 +92,7 @@ test_expect_failure REBASE_P \\\n \t'Rebase -Xsubtree --keep-empty --preserve-merges --onto empty commit' '\n \treset_rebase &&\n \tgit checkout -b rebase-keep-empty-empty master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master &&\n \tverbose test \"$(commit_message HEAD)\" = \"Empty commit\"\n@@ -96,7 +102,7 @@ test_expect_failure REBASE_P \\\n test_expect_failure 'Rebase -Xsubtree --onto commit 4' '\n \treset_rebase &&\n \tgit checkout -b rebase-onto-4 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --onto files-master master &&\n \tverbose test \"$(commit_message HEAD~2)\" = \"files_subtree/master4\"\n@@ -106,7 +112,7 @@ test_expect_failure 'Rebase -Xsubtree --onto commit 4' '\n test_expect_failure 'Rebase -Xsubtree --onto commit 5' '\n \treset_rebase &&\n \tgit checkout -b rebase-onto-5 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --onto files-master master &&\n \tverbose test \"$(commit_message HEAD~)\" = \"files_subtree/master5\"\n@@ -115,7 +121,7 @@ test_expect_failure 'Rebase -Xsubtree --onto commit 5' '\n test_expect_failure 'Rebase -Xsubtree --onto empty commit' '\n \treset_rebase &&\n \tgit checkout -b rebase-onto-empty master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --onto files-master master &&\n \tverbose test \"$(commit_message HEAD)\" = \"Empty commit\"\n-- \n2.23.0.38.g892688c90e\n\n"},{"id":"381566","messageId":"20190830055707.7785-5-newren@gmail.com","threadId":"51710","inReplyTo":"20190830055707.7785-1-newren@gmail.com","subject":"[PATCH v4 4/4] t9902: use a non-deprecated command for testing","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-30T05:57:07Z","receivedAt":"2019-08-30T05:57:22Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"t9902 had a list of three random porcelain commands as a sanity check,\none of which was filter-branch. Since we are recommending people not\nuse filter-branch, let's update this test to use rebase instead of\nfilter-branch.\n\nSigned-off-by: Elijah Newren \n---\n t/t9902-completion.sh | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/t/t9902-completion.sh b/t/t9902-completion.sh\nindex 75512c3403..4e7f669c76 100755\n--- a/t/t9902-completion.sh\n+++ b/t/t9902-completion.sh\n@@ -28,10 +28,10 @@ complete ()\n #\n # (2) A test makes sure that common subcommands are included in the\n # completion for \"git \", and a plumbing is excluded. \"add\",\n-# \"filter-branch\" and \"ls-files\" are listed for this.\n+# \"rebase\" and \"ls-files\" are listed for this.\n \n-GIT_TESTING_ALL_COMMAND_LIST='add checkout check-attr filter-branch ls-files'\n-GIT_TESTING_PORCELAIN_COMMAND_LIST='add checkout filter-branch'\n+GIT_TESTING_ALL_COMMAND_LIST='add checkout check-attr rebase ls-files'\n+GIT_TESTING_PORCELAIN_COMMAND_LIST='add checkout rebase'\n \n . \"$GIT_BUILD_DIR/contrib/completion/git-completion.bash\"\n \n@@ -1392,12 +1392,12 @@ test_expect_success 'basic' '\n \t# built-in\n \tgrep -q \"^add \\$\" out &&\n \t# script\n-\tgrep -q \"^filter-branch \\$\" out &&\n+\tgrep -q \"^rebase \\$\" out &&\n \t# plumbing\n \t! grep -q \"^ls-files \\$\" out &&\n \n-\trun_completion \"git f\" &&\n-\t! grep -q -v \"^f\" out\n+\trun_completion \"git r\" &&\n+\t! grep -q -v \"^r\" out\n '\n \n test_expect_success 'double dash \"git\" itself' '\n-- \n2.23.0.38.g892688c90e\n\n"},{"id":"381565","messageId":"20190830055707.7785-4-newren@gmail.com","threadId":"51710","inReplyTo":"20190830055707.7785-1-newren@gmail.com","subject":"[PATCH v4 3/4] Recommend git-filter-repo instead of git-filter-branch","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-30T05:57:06Z","receivedAt":"2019-08-30T05:57:23Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"filter-branch suffers from a deluge of disguised dangers that disfigure\nhistory rewrites (i.e. deviate from the deliberate changes). Many of\nthese problems are unobtrusive and can easily go undiscovered until the\nnew repository is in use. This can result in problems ranging from an\neven messier history than what led folks to filter-branch in the first\nplace, to data loss or corruption. These issues cannot be backward\ncompatibly fixed, so add a warning to both filter-branch and its manpage\nrecommending that another tool (such as filter-repo) be used instead.\n\nAlso, update other manpages that referenced filter-branch. Several of\nthese needed updates even if we could continue recommending\nfilter-branch, either due to implying that something was unique to\nfilter-branch when it applied more generally to all history rewriting\ntools (e.g. BFG, reposurgeon, fast-import, filter-repo), or because\nsomething about filter-branch was used as an example despite other more\ncommonly known examples now existing. Reword these sections to fix\nthese issues and to avoid recommending filter-branch.\n\nFinally, remove the section explaining BFG Repo Cleaner as an\nalternative to filter-branch. I feel somewhat bad about this,\nespecially since I feel like I learned so much from BFG that I put to\ngood use in filter-repo (which is much more than I can say for\nfilter-branch), but keeping that section presented a few problems:\n * In order to recommend that people quit using filter-branch, we need\n to provide them a recomendation for something else to use that\n can handle all the same types of rewrites. To my knowledge,\n filter-repo is the only such tool. So it needs to be mentioned.\n * I don't want to give conflicting recommendations to users\n * If we recommend two tools, we shouldn't expect users to learn both\n and pick which one to use; we should explain which problems one\n can solve that the other can't or when one is much faster than\n the other.\n * BFG and filter-repo have similar performance\n * All filtering types that BFG can do, filter-repo can also do. In\n fact, filter-repo comes with a reimplementation of BFG named\n bfg-ish which provides the same user-interface as BFG but with\n several bugfixes and new features that are hard to implement in\n BFG due to its technical underpinnings.\nWhile I could still mention both tools, it seems like I would need to\nprovide some kind of comparison and I would ultimately just say that\nfilter-repo can do everything BFG can, so ultimately it seems that it\nis just better to remove that section altogether.\n\nSigned-off-by: Elijah Newren \n---\n Documentation/git-fast-export.txt | 6 +-\n Documentation/git-filter-branch.txt | 272 +++++++++++++++++++++++++---\n Documentation/git-gc.txt | 17 +-\n Documentation/git-rebase.txt | 3 +-\n Documentation/git-replace.txt | 10 +-\n Documentation/git-svn.txt | 10 +-\n Documentation/githooks.txt | 10 +-\n contrib/svn-fe/svn-fe.txt | 4 +-\n git-filter-branch.sh | 13 ++\n 9 files changed, 286 insertions(+), 59 deletions(-)\n\ndiff --git a/Documentation/git-fast-export.txt b/Documentation/git-fast-export.txt\nindex cc940eb9ad..784e934009 100644\n--- a/Documentation/git-fast-export.txt\n+++ b/Documentation/git-fast-export.txt\n@@ -17,9 +17,9 @@ This program dumps the given revisions in a form suitable to be piped\n into 'git fast-import'.\n \n You can use it as a human-readable bundle replacement (see\n-linkgit:git-bundle[1]), or as a kind of an interactive\n-'git filter-branch'.\n-\n+linkgit:git-bundle[1]), or as a format that can be edited before being\n+fed to 'git fast-import' in order to do history rewrites (an ability\n+relied on by tools like 'git filter-repo').\n \n OPTIONS\n -------\ndiff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt\nindex 6b53dd7e06..c199f2ee20 100644\n--- a/Documentation/git-filter-branch.txt\n+++ b/Documentation/git-filter-branch.txt\n@@ -16,6 +16,19 @@ SYNOPSIS\n \t[--original ] [-d ] [-f | --force]\n \t[--state-branch ] [--] [...]\n \n+WARNING\n+-------\n+'git filter-branch' has a plethora of pitfalls that can produce non-obvious\n+manglings of the intended history rewrite (and can leave you with little\n+time to investigate such problems since it has such abysmal performance).\n+These safety and performance issues cannot be backward compatibly fixed and\n+as such, its use is not recommended. Please use an alternative history\n+filtering tool such as https://github.com/newren/git-filter-repo/[git\n+filter-repo]. If you still need to use 'git filter-branch', please\n+carefully read <> (and <>) to learn about the land\n+mines of filter-branch, and then vigilantly avoid as many of the hazards\n+listed there as reasonably possible.\n+\n DESCRIPTION\n -----------\n Lets you rewrite Git revision history by rewriting the branches mentioned\n@@ -445,36 +458,235 @@ warned.\n (or if your git-gc is not new enough to support arguments to\n `--prune`, use `git repack -ad; git prune` instead).\n \n-NOTES\n------\n-\n-git-filter-branch allows you to make complex shell-scripted rewrites\n-of your Git history, but you probably don't need this flexibility if\n-you're simply _removing unwanted data_ like large files or passwords.\n-For those operations you may want to consider\n-http://rtyley.github.io/bfg-repo-cleaner/[The BFG Repo-Cleaner],\n-a JVM-based alternative to git-filter-branch, typically at least\n-10-50x faster for those use-cases, and with quite different\n-characteristics:\n-\n-* Any particular version of a file is cleaned exactly _once_. The BFG,\n- unlike git-filter-branch, does not give you the opportunity to\n- handle a file differently based on where or when it was committed\n- within your history. This constraint gives the core performance\n- benefit of The BFG, and is well-suited to the task of cleansing bad\n- data - you don't care _where_ the bad data is, you just want it\n- _gone_.\n-\n-* By default The BFG takes full advantage of multi-core machines,\n- cleansing commit file-trees in parallel. git-filter-branch cleans\n- commits sequentially (i.e. in a single-threaded manner), though it\n- _is_ possible to write filters that include their own parallelism,\n- in the scripts executed against each commit.\n-\n-* The http://rtyley.github.io/bfg-repo-cleaner/#examples[command options]\n- are much more restrictive than git-filter branch, and dedicated just\n- to the tasks of removing unwanted data- e.g:\n- `--strip-blobs-bigger-than 1M`.\n+[[PERFORMANCE]]\n+PERFORMANCE\n+-----------\n+\n+The performance of git-filter-branch is glacially slow; its design makes it\n+impossible for a backward-compatible implementation to ever be fast:\n+\n+* In editing files, git-filter-branch by design checks out each and\n+every commit as it existed in the original repo. If your repo has 10\\^5\n+files and 10\\^5 commits, but each commit only modifies 5 files, then\n+git-filter-branch will make you do 10\\^10 modifications, despite only\n+having (at most) 5*10^5 unique blobs.\n+\n+* If you try and cheat and try to make git-filter-branch only work on\n+files modified in a commit, then two things happen\n+\n+ ** you run into problems with deletions whenever the user is simply\n+ trying to rename files (because attempting to delete files that\n+ don't exist looks like a no-op; it takes some chicanery to remap\n+ deletes across file renames when the renames happen via arbitrary\n+ user-provided shell)\n+\n+ ** even if you succeed at the map-deletes-for-renames chicanery, you\n+ still technically violate backward compatibility because users are\n+ allowed to filter files in ways that depend upon topology of\n+ commits instead of filtering solely based on file contents or names\n+ (though this has not been observed in the wild).\n+\n+* Even if you don't need to edit files but only want to e.g. rename or\n+remove some and thus can avoid checking out each file (i.e. you can use\n+--index-filter), you still are passing shell snippets for your filters.\n+This means that for every commit, you have to have a prepared git repo\n+where those filters can be run. That's a significant setup.\n+\n+* Further, several additional files are created or updated per commit by\n+git-filter-branch. Some of these are for supporting the convenience\n+functions provided by git-filter-branch (such as map()), while others\n+are for keeping track of internal state (but could have also been\n+accessed by user filters; one of git-filter-branch's regression tests\n+does so). This essentially amounts to using the filesystem as an IPC\n+mechanism between git-filter-branch and the user-provided filters.\n+Disks tend to be a slow IPC mechanism, and writing these files also\n+effectively represents a forced synchronization point between separate\n+processes that we hit with every commit.\n+\n+* The user-provided shell commands will likely involve a pipeline of\n+commands, resulting in the creation of many processes per commit.\n+Creating and running another process takes a widely varying amount of\n+time between operating systems, but on any platform it is very slow\n+relative to invoking a function.\n+\n+* git-filter-branch itself is written in shell, which is kind of slow.\n+This is the one performance issue that could be backward-compatibly\n+fixed, but compared to the above problems that are intrinsic to the\n+design of git-filter-branch, the language of the tool itself is a\n+relatively minor issue.\n+\n+ ** Side note: Unfortunately, people tend to fixate on the\n+ written-in-shell aspect and periodically ask if git-filter-branch\n+ could be rewritten in another language to fix the performance\n+ issues. Not only does that ignore the bigger intrinsic problems\n+ with the design, it'd help less than you'd expect: if\n+ git-filter-branch itself were not shell, then the convenience\n+ functions (map(), skip_commit(), etc) and the `--setup` argument\n+ could no longer be executed once at the beginning of the program\n+ but would instead need to be prepended to every user filter (and\n+ thus re-executed with every commit).\n+\n+The https://github.com/newren/git-filter-repo/[git filter-repo] tool is\n+an alternative to git-filter-branch which does not suffer from these\n+performance problems or the safety problems (mentioned below). For those\n+with existing tooling which relies upon git-filter-branch, 'git\n+repo-filter' also provides\n+https://github.com/newren/git-filter-repo/blob/master/contrib/filter-repo-demos/filter-lamely[filter-lamely],\n+a drop-in git-filter-branch replacement (with a few caveats). While\n+filter-lamely suffers from all the same safety issues as\n+git-filter-branch, it at least ameloriates the performance issues a\n+little.\n+\n+[[SAFETY]]\n+SAFETY\n+------\n+\n+git-filter-branch is riddled with gotchas resulting in various ways to\n+easily corrupt repos or end up with a mess worse than what you started\n+with:\n+\n+* Someone can have a set of \"working and tested filters\" which they\n+document or provide to a coworker, who then runs them on a different OS\n+where the same commands are not working/tested (some examples in the\n+git-filter-branch manpage are also affected by this). BSD vs. GNU\n+userland differences can really bite. If you're lucky, you get ugly\n+error messages spewed. But just as likely, the commands either don't do\n+the filtering requested, or silently corrupt making some unwanted\n+change. The unwanted change may only affect a few commits, so it's not\n+necessarily obvious either. (The fact that problems won't necessarily\n+be obvious means they are likely to go unnoticed until the rewritten\n+history is in use for quite a while, at which point it's really hard to\n+justify another flag-day for another rewrite.)\n+\n+* Filenames with spaces are often mishandled by shell snippets since\n+they cause problems for shell pipelines. Not everyone is familiar with\n+find -print0, xargs -0, git-ls-files -z, etc. Even people who are\n+familiar with these may assume such needs are not relevant because\n+someone else renamed any such files in their repo back before the person\n+doing the filtering joined the project. And, often, even those familiar\n+with handling arguments with spaces my not do so just because they\n+aren't in the mindset of thinking about everything that could possibly\n+go wrong.\n+\n+* Non-ascii filenames can be silently removed despite being in a desired\n+directory. The desire to select paths to keep often use pipelines like\n+`git ls-files | grep -v ^WANTED_DIR/ | xargs git rm`. ls-files will\n+only quote filenames if needed so folks may not notice that one of the\n+files didn't match the regex, again until it's much too late. Yes,\n+someone who knows about core.quotePath can avoid this (unless they have\n+other special characters like \\t, \\n, or \"), and people who use ls-files\n+-z with something other than grep can avoid this, but that doesn't mean\n+they will.\n+\n+* Similarly, when moving files around, one can find that filenames with\n+non-ascii or special characters end up in a different directory, one\n+that includes a double quote character. (This is technically the same\n+issue as above with quoting, but perhaps an interesting different way\n+that it can and has manifested as a problem.)\n+\n+* It's far too easy to accidentally mix up old and new history. It's\n+still possible with any tool, but git-filter-branch almost invites it.\n+If lucky, the only downside is users getting frustrated that they don't\n+know how to shrink their repo and remove the old stuff. If unlucky,\n+they merge old and new history and end up with multiple \"copies\" of each\n+commit, some of which have unwanted or sensitive files and others which\n+don't. This comes about in multiple different ways:\n+\n+ ** the default to only doing a partial history rewrite ('--all' is not\n+ the default and few examples show it)\n+\n+ ** the fact that there's no automatic post-run cleanup\n+\n+ ** the fact that --tag-name-filter (when used to rename tags) doesn't\n+ remove the old tags but just adds new ones with the new name\n+\n+ ** the fact that little educational information is provided to inform\n+ users of the ramifications of a rewrite and how to avoid mixing old\n+ and new history. For example, this man page discusses how users\n+ need to understand that they need to rebase their changes for all\n+ their branches on top of new history (or delete and reclone), but\n+ that's only one of multiple concerns to consider. See the\n+ \"DISCUSSION\" section of the git filter-repo manual page for more\n+ details.\n+\n+* Annotated tags can be accidentally converted to lightweight tags, due\n+to either of two issues:\n+\n+ ** Someone can do a history rewrite, realize they messed up, restore\n+ from the backups in refs/original/, and then redo their\n+ git-filter-branch command. (The backup in refs/original/ is not a\n+ real backup; it dereferences tags first.)\n+\n+ ** Running git-filter-branch with either --tags or --all in your\n+ . In order to retain annotated tags as\n+ annotated, you must use --tag-name-filter (and must not have\n+ restored from refs/original/ in a previously botched rewrite).\n+\n+* Any commit messages that specify an encoding will become corrupted\n+by the rewrite; git-filter-branch ignores the encoding, takes the original\n+bytes, and feeds it to commit-tree without telling it the proper\n+encoding. (This happens whether or not --msg-filter is used.)\n+\n+* Commit messages (even if they are all UTF-8) by default become\n+corrupted due to not being updated -- any references to other commit\n+hashes in commit messages will now refer to no-longer-extant commits.\n+\n+* There are no facilities for helping users find what unwanted crud they\n+should delete, which means they are much more likely to have incomplete\n+or partial cleanups that sometimes result in confusion and people\n+wasting time trying to understand. (For example, folks tend to just\n+look for big files to delete instead of big directories or extensions,\n+and once they do so, then sometime later folks using the new repository\n+who are going through history will notice a build artifact directory\n+that has some files but not others, or a cache of dependencies\n+(node_modules or similar) which couldn't have ever been functional since\n+it's missing some files.)\n+\n+* If --prune-empty isn't specified, then the filtering process can\n+create hoards of confusing empty commits\n+\n+* If --prune-empty is specified, then intentionally placed empty\n+commits from before the filtering operation are also pruned instead of\n+just pruning commits that became empty due to filtering rules.\n+\n+* If --prune empty is specified, sometimes empty commits are missed\n+and left around anyway (a somewhat rare bug, but it happens...)\n+\n+* A minor issue, but users who have a goal to update all names and\n+emails in a repository may be led to --env-filter which will only update\n+authors and committers, missing taggers.\n+\n+* If the user provides a --tag-name-filter that maps multiple tags to\n+the same name, no warning or error is provided; git-filter-branch simply\n+overwrites each tag in some undocumented pre-defined order resulting in\n+only one tag at the end. (A git-filter-branch regression test requires\n+this.)\n+\n+Also, the poor performance of git-filter-branch often leads to safety issues:\n+\n+* Coming up with the correct shell snippet to do the filtering you want\n+is sometimes difficult unless you're just doing a trivial modification\n+such as deleting a couple files. Unfortunately, people often learn if\n+the snippet is right or wrong by trying it out, but the rightness or\n+wrongness can vary depending on special circumstances (spaces in\n+filenames, non-ascii filenames, funny author names or emails, invalid\n+timezones, presence of grafts or replace objects, etc.), meaning they\n+may have to wait a long time, hit an error, then restart. The\n+performance of git-filter-branch is so bad that this cycle is painful,\n+reducing the time available to carefully re-check (to say nothing about\n+what it does to the patience of the person doing the rewrite even if\n+they do technically have more time available). This problem is extra\n+compounded because errors from broken filters may not be shown for a\n+long time and/or get lost in a sea of output. Even worse, broken\n+filters often just result in silent incorrect rewrites.\n+\n+* To top it all off, even when users finally find working commands, they\n+naturally want to share them. But they may be unaware that their repo\n+didn't have some special cases that someone else's does. So, when\n+someone else with a different repository runs the same commands, they\n+get hit by the problems above. Or, the user just runs commands that\n+really were vetted for special cases, but they run it on a different OS\n+where it doesn't work, as noted above.\n \n GIT\n ---\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 247f765604..0c114ad1ca 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -115,15 +115,14 @@ NOTES\n -----\n \n 'git gc' tries very hard not to delete objects that are referenced\n-anywhere in your repository. In\n-particular, it will keep not only objects referenced by your current set\n-of branches and tags, but also objects referenced by the index,\n-remote-tracking branches, refs saved by 'git filter-branch' in\n-refs/original/, reflogs (which may reference commits in branches\n-that were later amended or rewound), and anything else in the refs/* namespace.\n-If you are expecting some objects to be deleted and they aren't, check\n-all of those locations and decide whether it makes sense in your case to\n-remove those references.\n+anywhere in your repository. In particular, it will keep not only\n+objects referenced by your current set of branches and tags, but also\n+objects referenced by the index, remote-tracking branches, notes saved\n+by 'git notes' under refs/notes/, reflogs (which may reference commits\n+in branches that were later amended or rewound), and anything else in\n+the refs/* namespace. If you are expecting some objects to be deleted\n+and they aren't, check all of those locations and decide whether it\n+makes sense in your case to remove those references.\n \n On the other hand, when 'git gc' runs concurrently with another process,\n there is a risk of it deleting an object that the other process is using\ndiff --git a/Documentation/git-rebase.txt b/Documentation/git-rebase.txt\nindex 6156609cf7..a8cfc0ad82 100644\n--- a/Documentation/git-rebase.txt\n+++ b/Documentation/git-rebase.txt\n@@ -832,7 +832,8 @@ Hard case: The changes are not the same.::\n \tThis happens if the 'subsystem' rebase had conflicts, or used\n \t`--interactive` to omit, edit, squash, or fixup commits; or\n \tif the upstream used one of `commit --amend`, `reset`, or\n-\t`filter-branch`.\n+\ta full history rewriting command like\n+\thttps://github.com/newren/git-filter-repo[`filter-repo`].\n \n \n The easy case\ndiff --git a/Documentation/git-replace.txt b/Documentation/git-replace.txt\nindex 246dc9943c..f271d758c3 100644\n--- a/Documentation/git-replace.txt\n+++ b/Documentation/git-replace.txt\n@@ -123,10 +123,10 @@ The following format are available:\n CREATING REPLACEMENT OBJECTS\n ----------------------------\n \n-linkgit:git-filter-branch[1], linkgit:git-hash-object[1] and\n-linkgit:git-rebase[1], among other git commands, can be used to create\n-replacement objects from existing objects. The `--edit` option can\n-also be used with 'git replace' to create a replacement object by\n+linkgit:git-hash-object[1], linkgit:git-rebase[1], and\n+https://github.com/newren/git-filter-repo[git-filter-repo], among other git commands, can be used to\n+create replacement objects from existing objects. The `--edit` option\n+can also be used with 'git replace' to create a replacement object by\n editing an existing object.\n \n If you want to replace many blobs, trees or commits that are part of a\n@@ -148,13 +148,13 @@ pending objects.\n SEE ALSO\n --------\n linkgit:git-hash-object[1]\n-linkgit:git-filter-branch[1]\n linkgit:git-rebase[1]\n linkgit:git-tag[1]\n linkgit:git-branch[1]\n linkgit:git-commit[1]\n linkgit:git-var[1]\n linkgit:git[1]\n+https://github.com/newren/git-filter-repo[git-filter-repo]\n \n GIT\n ---\ndiff --git a/Documentation/git-svn.txt b/Documentation/git-svn.txt\nindex 30711625fd..53774f5b64 100644\n--- a/Documentation/git-svn.txt\n+++ b/Documentation/git-svn.txt\n@@ -769,11 +769,11 @@ option for (hopefully) obvious reasons.\n +\n This option is NOT recommended as it makes it difficult to track down\n old references to SVN revision numbers in existing documentation, bug\n-reports and archives. If you plan to eventually migrate from SVN to Git\n-and are certain about dropping SVN history, consider\n-linkgit:git-filter-branch[1] instead. filter-branch also allows\n-reformatting of metadata for ease-of-reading and rewriting authorship\n-info for non-\"svn.authorsFile\" users.\n+reports, and archives. If you plan to eventually migrate from SVN to\n+Git and are certain about dropping SVN history, consider\n+https://github.com/newren/git-filter-repo[git-filter-repo] instead.\n+filter-repo also allows reformatting of metadata for ease-of-reading\n+and rewriting authorship info for non-\"svn.authorsFile\" users.\n \n svn.useSvmProps::\n svn-remote..useSvmProps::\ndiff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\nindex 82cd573776..5a789c91df 100644\n--- a/Documentation/githooks.txt\n+++ b/Documentation/githooks.txt\n@@ -425,10 +425,12 @@ post-rewrite\n \n This hook is invoked by commands that rewrite commits\n (linkgit:git-commit[1] when called with `--amend` and\n-linkgit:git-rebase[1]; currently `git filter-branch` does 'not' call\n-it!). Its first argument denotes the command it was invoked by:\n-currently one of `amend` or `rebase`. Further command-dependent\n-arguments may be passed in the future.\n+linkgit:git-rebase[1]; however, full-history (re)writing tools like\n+linkgit:git-fast-import[1] or\n+https://github.com/newren/git-filter-repo[git-filter-repo] typically\n+do not call it!). Its first argument denotes the command it was\n+invoked by: currently one of `amend` or `rebase`. Further\n+command-dependent arguments may be passed in the future.\n \n The hook receives a list of the rewritten commits on stdin, in the\n format\ndiff --git a/contrib/svn-fe/svn-fe.txt b/contrib/svn-fe/svn-fe.txt\nindex a3425f4770..19333fc8df 100644\n--- a/contrib/svn-fe/svn-fe.txt\n+++ b/contrib/svn-fe/svn-fe.txt\n@@ -56,7 +56,7 @@ line. This line has the form `git-svn-id: URL@REVNO UUID`.\n \n The resulting repository will generally require further processing\n to put each project in its own repository and to separate the history\n-of each branch. The 'git filter-branch --subdirectory-filter' command\n+of each branch. The 'git filter-repo --subdirectory-filter' command\n may be useful for this purpose.\n \n BUGS\n@@ -67,5 +67,5 @@ The exit status does not reflect whether an error was detected.\n \n SEE ALSO\n --------\n-git-svn(1), svn2git(1), svk(1), git-filter-branch(1), git-fast-import(1),\n+git-svn(1), svn2git(1), svk(1), git-filter-repo(1), git-fast-import(1),\n https://svn.apache.org/repos/asf/subversion/trunk/notes/dump-load-format.txt\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\nindex 5c5afa2b98..f805965d87 100755\n--- a/git-filter-branch.sh\n+++ b/git-filter-branch.sh\n@@ -83,6 +83,19 @@ set_ident () {\n \tfinish_ident COMMITTER\n }\n \n+if [ -z \"$FILTER_BRANCH_SQUELCH_WARNING\" -a \\\n+ -z \"$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS\" ]; then\n+\tcat <] [--subdirectory-filter ] [--env-filter ]\n \t[--tree-filter ] [--index-filter ]\n \t[--parent-filter ] [--msg-filter ]\n-- \n2.23.0.38.g892688c90e\n\n"},{"id":"381616","messageId":"nycvar.QRO.7.76.6.1908302221210.46@tvgsbejvaqbjf.bet","threadId":"51710","inReplyTo":"CABPp-BEYRmhrb4Tx3bGzkx8y53T_0BYhLE5J0cEmxj18WtZs9A@mail.gmail.com","subject":"Re: [RFC PATCH 0/5] Remove git-filter-branch from git.git; host it elsewhere","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-08-30T20:40:39Z","receivedAt":"2019-08-30T20:40:58Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Elijah,\n\n\nOn Wed, 28 Aug 2019, Elijah Newren wrote:\n\n> Hi Sergey,\n>\n> On Wed, Aug 28, 2019 at 1:52 AM Sergey Organov wrote:\n> >\n> > Elijah Newren writes:\n> >\n> > > On Tue, Aug 27, 2019 at 1:43 AM Sergey Organov wrote:\n> > >>\n> > >> Eric Wong writes:\n> > >>\n> > >>\n> > >> [...]\n> > >>\n> > >> > AFAIK, filter-branch is not causing support headaches for any\n> > >> > git developers today. With so many commands in git, it's\n> > >> > unlikely newbies will ever get around to discover it :)\n> > >> > So I think think we should be in any rush to remove it.\n> > >>\n> > >> Nah, discovering it is simple. Just Google for \"git change author\". That\n> > >> eventually leads to a script that uses \"git filter-branch --env-filter\"\n> > >> to get the job done, and I'm afraid it is spread all over the world.\n> > >>\n> > >> See, e.g.:\n> > >>\n> > >> https://help.github.com/en/articles/changing-author-info\n> > >\n> > > Side note: Is the goal to \"fix names and email addresses in this\n> > > repository\"? If so, this guide fails: it doesn't update tagger names\n> > > or email addresses. Indeed, filter-branch doesn't provide a way to do\n> > > that. (Not to mention other problems like not updating references to\n> > > commit hashes in commit messages when it busy rewriting everything.)\n> >\n> > No. Maybe the original goal was like that, by I, personally, use\n> > modified version of this to change my \"Author\" credentials from\n> > \"internal\" to \"public\" in branches that I'm going to send upstream, so\n> > the actual aim is to change e-mail of particular Author from a@b to c@d\n> > in all the commits in a (feature) branch.\n>\n> There's an interesting usecase I hadn't heard of or thought of before.\n\nI'll throw in another use case that's kinda related: extracting the\nhistory of one file (or subdirectory).\n\nIn my most recent instance of this, I wanted to publish the script I\nused to use for submitting patch series to the Git mailing list,\nmaintaining tags for iterations and generating cover letters from branch\ndescriptions and interdiffs (this script eventually became GitGitGadget,\nhttps://github.com/gitgitgadget/gitgitgadget/commits?after=6fb0ede48f86e729292ee1542729bc0f5a30cfa6+0\ndemonstrates this).\n\nTo do that, I ran a `git filter-branch` in the repository where I track\nall the scripts I deem unsuitable for public consumption, to remove all\nfiles but `mail-patch-series.sh`, then pushed it to\nhttps://github.com/dscho/mail-patch-series\n\nPlease note that most crucially, I wanted to rewrite a newly-created\nbranch, and only that branch.\n\nCould I have done the same using `git fast-export`, filtering the output\nwith a Perl script, then passing it to `git fast-import`? Sure, I was\nreally tempted to do that. In the end, it took less of _my_ time to just\nlet `git filter-branch` do its work with a not-too-complicated index\nfilter.\n\nIn another instance, a long, long time ago, I needed to restart a\nrepository which had included way too many files for its own good, then\nrename the old repository and start with a fresh `master` that contained\nbut a single commit whose tree was identical to the previous `master`'s\ntip commit. I simply grafted that commit, ran `git filter-branch` and\nhad precisely what I needed.\n\nI would be _delighted_ if these kinds of use case (rewriting a branch,\nor even just a commit range) became more of a first-class citizen with\n`git filter-repo`.\n\nThanks,\nDscho\n"},{"id":"381617","messageId":"CABPp-BHMXAQGPaBYyg2dtVeN5h8fW8G4YdhddCeAjY5r74BAzw@mail.gmail.com","threadId":"51710","inReplyTo":"nycvar.QRO.7.76.6.1908302221210.46@tvgsbejvaqbjf.bet","subject":"Re: [RFC PATCH 0/5] Remove git-filter-branch from git.git; host it elsewhere","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-30T23:22:10Z","receivedAt":"2019-08-30T23:22:43Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Dscho,\n\nOn Fri, Aug 30, 2019 at 1:40 PM Johannes Schindelin\n wrote:\n>\n> Hi Elijah,\n>\n>\n> On Wed, 28 Aug 2019, Elijah Newren wrote:\n>\n> > Hi Sergey,\n> >\n> > On Wed, Aug 28, 2019 at 1:52 AM Sergey Organov wrote:\n> > >\n> > > Elijah Newren writes:\n> > >\n> > > > On Tue, Aug 27, 2019 at 1:43 AM Sergey Organov wrote:\n> > > >>\n> > > >> Eric Wong writes:\n> > > >>\n> > > >>\n> > > >> [...]\n> > > >>\n> > > >> > AFAIK, filter-branch is not causing support headaches for any\n> > > >> > git developers today. With so many commands in git, it's\n> > > >> > unlikely newbies will ever get around to discover it :)\n> > > >> > So I think think we should be in any rush to remove it.\n> > > >>\n> > > >> Nah, discovering it is simple. Just Google for \"git change author\". That\n> > > >> eventually leads to a script that uses \"git filter-branch --env-filter\"\n> > > >> to get the job done, and I'm afraid it is spread all over the world.\n> > > >>\n> > > >> See, e.g.:\n> > > >>\n> > > >> https://help.github.com/en/articles/changing-author-info\n> > > >\n> > > > Side note: Is the goal to \"fix names and email addresses in this\n> > > > repository\"? If so, this guide fails: it doesn't update tagger names\n> > > > or email addresses. Indeed, filter-branch doesn't provide a way to do\n> > > > that. (Not to mention other problems like not updating references to\n> > > > commit hashes in commit messages when it busy rewriting everything.)\n> > >\n> > > No. Maybe the original goal was like that, by I, personally, use\n> > > modified version of this to change my \"Author\" credentials from\n> > > \"internal\" to \"public\" in branches that I'm going to send upstream, so\n> > > the actual aim is to change e-mail of particular Author from a@b to c@d\n> > > in all the commits in a (feature) branch.\n> >\n> > There's an interesting usecase I hadn't heard of or thought of before.\n>\n> I'll throw in another use case that's kinda related: extracting the\n> history of one file (or subdirectory).\n\nThanks for sending these along! I do have some comments, and a bunch\nof questions...\n\n> In my most recent instance of this, I wanted to publish the script I\n> used to use for submitting patch series to the Git mailing list,\n> maintaining tags for iterations and generating cover letters from branch\n> descriptions and interdiffs (this script eventually became GitGitGadget,\n> https://github.com/gitgitgadget/gitgitgadget/commits?after=6fb0ede48f86e729292ee1542729bc0f5a30cfa6+0\n> demonstrates this).\n>\n> To do that, I ran a `git filter-branch` in the repository where I track\n> all the scripts I deem unsuitable for public consumption, to remove all\n> files but `mail-patch-series.sh`, then pushed it to\n> https://github.com/dscho/mail-patch-series\n>\n> Please note that most crucially, I wanted to rewrite a newly-created\n> branch, and only that branch.\n>\n> Could I have done the same using `git fast-export`, filtering the output\n> with a Perl script, then passing it to `git fast-import`? Sure, I was\n> really tempted to do that. In the end, it took less of _my_ time to just\n> let `git filter-branch` do its work with a not-too-complicated index\n> filter.\n\nWhy a perl script? Shouldn't\n git fast-export [--no-data] HEAD -- $PATH | git fast-import --force --quiet\ndo the trick? And it's probably simpler and shorter than the index\nfilter you used.\n\nThat said, yeah it'd be nice to get automatic rewriting of commit\nhashes in commit messages and other niceties from filter-repo (e.g.\nfuture automatic reattaching of notes to the rewritten commits). Some\nquestions:\n\n * What's the backup strategy in case you specify the wrong filters\n(e.g. you have a typo in the pathnames)? filter-repo encourages folks\nto make a clone and then filter the fresh clone, because if anything\ngoes awry, you can just delete and restart. (I am heavily opposed to\nthe refs/original/ backup mechanism used by filter-branch, for\nmultiple reasons.) Is your safety stance just \"If I mess up it's my\nown fault; do the rewrite?\" Or are you okay with cloning before\nfiltering?\n * If you're okay with cloning before filtering...then is there an\nissue with rewriting all branches, and just pushing the one you need?\n(Is there an issue with \"this branch is small, the others are huge,\nand filter-branch is slow -- so rewriting one branch saves me lots of\ntime\"? Or are there other issues at play too?)\n * What if the user has auxiliary information for the branch in other\nrefs? For example, git-notes pointing at any of the commits, or tags\nin the history of the branch that might be relevant, or perhaps even\nreplace refs in combination with GIT_NO_REPLACE_OBJECTS=1? Is this an\n\"I don't care, toss that stuff and just rewrite just this branch?\"\n * filter-repo by default creates new replace references so that you\ncan refer to new commit IDs using old (unabbreviated) commit IDs.\nWould that be considered helpful for this usecase? unhelpful?\nirrelevant, since you'll just push the branch you want somewhere and\nnuke the temporary clone?\n\n\nI'm not by any means ruling out the possibility of documenting --refs\nand adjusting the defaults when it is used so the user can just run\nsomething like\n git filter-repo --path $PATH --refs $MYBRANCH\nbut I feel like I need to understand answers to questions like the\nabove ones so that I can know how to phrase warnings and adjust\ndefaults and update the documentation.\n\n> In another instance, a long, long time ago, I needed to restart a\n> repository which had included way too many files for its own good, then\n> rename the old repository and start with a fresh `master` that contained\n> but a single commit whose tree was identical to the previous `master`'s\n> tip commit. I simply grafted that commit, ran `git filter-branch` and\n> had precisely what I needed.\n\nfilter-repo supports grafts and replace objects, the same as\nfilter-branch. (Although, technically, I didn't have to do a thing to\nsupport it; fast-export does the special handling of rewriting based\non grafts and replace objects.) So, I'd say this is fully supported.\n\nSide question: the git-replace documents suggest that the graft file\nis deprecated. Are there any timeframes or plans for phasing out\nbeyond the git-replace manpage existing? Should I avoid documenting\nthe graft file support in filter-repo? Should I include examples\nusing not just git-replace but also using the graft file?\n\n> I would be _delighted_ if these kinds of use case (rewriting a branch,\n> or even just a commit range) became more of a first-class citizen with\n> `git filter-repo`.\n\nI've got all the pieces for supporting a single branch or a commit\nrange (e.g. 'git filter-repo --path foo --refs ^master~4 ^stable~23\nmybranch'), but the defaults (error out unless in a bare repo, move\nrefs/remotes/origin/* to refs/heads/*, disconnect origin remote,\nexpire reflogs & repack & prune, create new replace references so\nfolks can access new commits using old commit IDs) may be somewhat\nfriction-filled for this usecase. Those defaults other than the new\nreplace refs happen to all be turned off with the combination of\n--force and --target, so, assuming turning them off is what you need,\nyou could cheat and just specify 'git filter-repo --force --target .\n--refs $MYBRANCH' today and perhaps get what you want, but that's a\nreally non-intuitive command line that is way too ugly to recommend.\nAnd I don't want to tie myself to '--target .' being the magic sauce\nin the future either.\n"},{"id":"381643","messageId":"nycvar.QRO.7.76.6.1909021111440.46@tvgsbejvaqbjf.bet","threadId":"51710","inReplyTo":"CABPp-BHMXAQGPaBYyg2dtVeN5h8fW8G4YdhddCeAjY5r74BAzw@mail.gmail.com","subject":"Re: [RFC PATCH 0/5] Remove git-filter-branch from git.git; host it elsewhere","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-09-02T09:29:52Z","receivedAt":"2019-09-02T09:30:34Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Elijah,\n\nOn Fri, 30 Aug 2019, Elijah Newren wrote:\n\n> On Fri, Aug 30, 2019 at 1:40 PM Johannes Schindelin\n> wrote:\n>\n> > [...]\n> > In my most recent instance of this, I wanted to publish the script I\n> > used to use for submitting patch series to the Git mailing list,\n> > maintaining tags for iterations and generating cover letters from branch\n> > descriptions and interdiffs (this script eventually became GitGitGadget,\n> > https://github.com/gitgitgadget/gitgitgadget/commits?after=6fb0ede48f86e729292ee1542729bc0f5a30cfa6+0\n> > demonstrates this).\n> >\n> > To do that, I ran a `git filter-branch` in the repository where I track\n> > all the scripts I deem unsuitable for public consumption, to remove all\n> > files but `mail-patch-series.sh`, then pushed it to\n> > https://github.com/dscho/mail-patch-series\n> >\n> > Please note that most crucially, I wanted to rewrite a newly-created\n> > branch, and only that branch.\n> >\n> > Could I have done the same using `git fast-export`, filtering the output\n> > with a Perl script, then passing it to `git fast-import`? Sure, I was\n> > really tempted to do that. In the end, it took less of _my_ time to just\n> > let `git filter-branch` do its work with a not-too-complicated index\n> > filter.\n>\n> Why a perl script? Shouldn't\n> git fast-export [--no-data] HEAD -- $PATH | git fast-import --force --quiet\n> do the trick? And it's probably simpler and shorter than the index\n> filter you used.\n\nDoes that not keep the full `$PATH`? I wanted the resulting branch to\nhave the file in the top-level directory.\n\n> That said, yeah it'd be nice to get automatic rewriting of commit\n> hashes in commit messages and other niceties from filter-repo (e.g.\n> future automatic reattaching of notes to the rewritten commits). Some\n> questions:\n>\n> * What's the backup strategy in case you specify the wrong filters\n> (e.g. you have a typo in the pathnames)? filter-repo encourages folks\n> to make a clone and then filter the fresh clone, because if anything\n> goes awry, you can just delete and restart. (I am heavily opposed to\n> the refs/original/ backup mechanism used by filter-branch, for\n> multiple reasons.) Is your safety stance just \"If I mess up it's my\n> own fault; do the rewrite?\" Or are you okay with cloning before\n> filtering?\n\nPlease note that the `refs/original/` refs should not have been written\nat all anymore, not after reflogs were introduced.\n\nIncidentally, that is my answer to your question: the reflog is my\nbackup.\n\n> * If you're okay with cloning before filtering...then is there an\n> issue with rewriting all branches, and just pushing the one you need?\n> (Is there an issue with \"this branch is small, the others are huge,\n> and filter-branch is slow -- so rewriting one branch saves me lots of\n> time\"? Or are there other issues at play too?)\n\nI am not okay with cloning before filtering.\n\nFirst of all, it is wasteful.\n\nSecond of all, in my case it would have been *particularly* wasteful\nbecause the repository in question also has quite a few quite large\nblobs (hysterical raisins, don't ask).\n\n> * What if the user has auxiliary information for the branch in other\n> refs? For example, git-notes pointing at any of the commits, or tags\n> in the history of the branch that might be relevant, or perhaps even\n> replace refs in combination with GIT_NO_REPLACE_OBJECTS=1? Is this an\n> \"I don't care, toss that stuff and just rewrite just this branch?\"\n\nIn my case: there are no notes. The only time when I make heavy use of\nnotes is in GitGitGadget. I don't use that feature otherwise.\n\n> * filter-repo by default creates new replace references so that you\n> can refer to new commit IDs using old (unabbreviated) commit IDs.\n> Would that be considered helpful for this usecase? unhelpful?\n> irrelevant, since you'll just push the branch you want somewhere and\n> nuke the temporary clone?\n\nI definitely did not need that mapping in all of my `git filter-branch`\nuse cases.\n\nOf course, I can see how it can come in handy in other circumstances,\njust not in the ones I experienced so far.\n\n> I'm not by any means ruling out the possibility of documenting --refs\n> and adjusting the defaults when it is used so the user can just run\n> something like\n> git filter-repo --path $PATH --refs $MYBRANCH\n> but I feel like I need to understand answers to questions like the\n> above ones so that I can know how to phrase warnings and adjust\n> defaults and update the documentation.\n\nIn all the scenarios where I used `git filter-branch` (some dozen per\nyear, so not all *that* many), I needed to rewrite one particular\nbranch, typically a freshly-created one. I never, ever ever needed to\nrewrite all the refs in the repository. Not once ;-)\n\n> > In another instance, a long, long time ago, I needed to restart a\n> > repository which had included way too many files for its own good, then\n> > rename the old repository and start with a fresh `master` that contained\n> > but a single commit whose tree was identical to the previous `master`'s\n> > tip commit. I simply grafted that commit, ran `git filter-branch` and\n> > had precisely what I needed.\n>\n> filter-repo supports grafts and replace objects, the same as\n> filter-branch. (Although, technically, I didn't have to do a thing to\n> support it; fast-export does the special handling of rewriting based\n> on grafts and replace objects.) So, I'd say this is fully supported.\n>\n> Side question: the git-replace documents suggest that the graft file\n> is deprecated. Are there any timeframes or plans for phasing out\n> beyond the git-replace manpage existing? Should I avoid documenting\n> the graft file support in filter-repo? Should I include examples\n> using not just git-replace but also using the graft file?\n\nI had meant to prepare a patch series to remove `grafts` support that\nJunio could carry in `pu` until the time he considers it appropriate to\nmerge to `master`, but it seems that this task fell under the rag.\n\nThe deprecation itself has been introduced in tags/v2.18.0-rc0~54^2~4,\ni.e. it is official as of Git v2.18.0, which was released in mid-June\nlast year.\n\nMy personal gut feeling is that we should let it simmer for another year\nbefore removing support for the `grafts` file (and we may want to update\nthe label \"grafted\" when `git log` shows a shallow commit before we\nremove that support for `grafts`).\n\nSo I'll not work on that patch for now.\n\n> > I would be _delighted_ if these kinds of use case (rewriting a branch,\n> > or even just a commit range) became more of a first-class citizen with\n> > `git filter-repo`.\n>\n> I've got all the pieces for supporting a single branch or a commit\n> range (e.g. 'git filter-repo --path foo --refs ^master~4 ^stable~23\n> mybranch'), but the defaults (error out unless in a bare repo, move\n> refs/remotes/origin/* to refs/heads/*, disconnect origin remote,\n> expire reflogs & repack & prune, create new replace references so\n> folks can access new commits using old commit IDs) may be somewhat\n> friction-filled for this usecase. Those defaults other than the new\n> replace refs happen to all be turned off with the combination of\n> --force and --target, so, assuming turning them off is what you need,\n> you could cheat and just specify 'git filter-repo --force --target .\n> --refs $MYBRANCH' today and perhaps get what you want, but that's a\n> really non-intuitive command line that is way too ugly to recommend.\n> And I don't want to tie myself to '--target .' being the magic sauce\n> in the future either.\n\nI agree. I would love for my use cases to become more of first-class\ncitizens. Maybe `--branch ` could serve as the knob?\n\nWhat I also found really helpful in `git filter-branch` is that it was\npossible to pass one-liner shell scripts directly to the command, giving\na lot of freedom about the transformations. I understand that Python\nmakes it hard to write spaghetti-code one-liners, so you cannot really\npass the snippet in via the command-line, but I hope there is a way to\nscript things in `git filter-repo`?\n\nCiao,\nDscho\n"},{"id":"381651","messageId":"nycvar.QRO.7.76.6.1909021641490.46@tvgsbejvaqbjf.bet","threadId":"51710","inReplyTo":"20190830055707.7785-3-newren@gmail.com","subject":"Re: [PATCH v4 2/4] t3427: accelerate this test by using fast-export and fast-import","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-09-02T14:45:37Z","receivedAt":"2019-09-02T14:45:57Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Elijah,\n\nOn Thu, 29 Aug 2019, Elijah Newren wrote:\n\n> fast-export and fast-import can easily handle the simple rewrite that\n> was being done by filter-branch, and should be significantly faster on\n> systems with a slow fork. Timings from before and after on two laptops\n> that I have access to (measured via `time ./t3427-rebase-subtree.sh`,\n> i.e. including everything in this test -- not just the filter-branch or\n> fast-export/fast-import pair):\n>\n> Linux: 4.305s -> 3.684s (~17% speedup)\n> Mac: 10.128s -> 7.038s (~30% speedup)\n\nThis patch seems to accelerate t3427 on my Windows laptop, too, from\n~1m37s to ~1m17s, i.e. ~20%.\n\nCiao,\nDscho\n"},{"id":"381652","messageId":"nycvar.QRO.7.76.6.1909021645460.46@tvgsbejvaqbjf.bet","threadId":"51710","inReplyTo":"20190830055707.7785-2-newren@gmail.com","subject":"Re: [PATCH v4 1/4] t6006: simplify and optimize empty message test","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-09-02T14:47:14Z","receivedAt":"2019-09-02T14:47:26Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Elijah,\n\nOn Thu, 29 Aug 2019, Elijah Newren wrote:\n\n> Despite only being one piece of the 71st test and there being 73 tests\n> overall, this small change to just this one test speeds up the overall\n> execution time of t6006 (as measured by the best of 3 runs of `time\n> ./t6006-rev-list-format.sh`) by about 11% on Linux and by 13% on\n> Mac.\n\nA similar effect can be observed on my Windows laptop: from 25s to 21s,\ni.e. ~15%.\n\nThanks,\nDscho\n"},{"id":"381736","messageId":"CABPp-BF5Y415ALi288CWnuJSG4nrdbjBvh5pHJ1752V-qQFPKw@mail.gmail.com","threadId":"51710","inReplyTo":"nycvar.QRO.7.76.6.1909021111440.46@tvgsbejvaqbjf.bet","subject":"Re: [RFC PATCH 0/5] Remove git-filter-branch from git.git; host it elsewhere","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-09-03T17:37:08Z","receivedAt":"2019-09-03T17:37:23Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Dscho,\n\nOn Mon, Sep 2, 2019 at 2:30 AM Johannes Schindelin\n wrote:\n>\n> Hi Elijah,\n>\n> On Fri, 30 Aug 2019, Elijah Newren wrote:\n>\n> > On Fri, Aug 30, 2019 at 1:40 PM Johannes Schindelin\n> > wrote:\n> >\n> > > [...]\n> > > In my most recent instance of this, I wanted to publish the script I\n> > > used to use for submitting patch series to the Git mailing list,\n> > > maintaining tags for iterations and generating cover letters from branch\n> > > descriptions and interdiffs (this script eventually became GitGitGadget,\n> > > https://github.com/gitgitgadget/gitgitgadget/commits?after=6fb0ede48f86e729292ee1542729bc0f5a30cfa6+0\n> > > demonstrates this).\n> > >\n> > > To do that, I ran a `git filter-branch` in the repository where I track\n> > > all the scripts I deem unsuitable for public consumption, to remove all\n> > > files but `mail-patch-series.sh`, then pushed it to\n> > > https://github.com/dscho/mail-patch-series\n> > >\n> > > Please note that most crucially, I wanted to rewrite a newly-created\n> > > branch, and only that branch.\n> > >\n> > > Could I have done the same using `git fast-export`, filtering the output\n> > > with a Perl script, then passing it to `git fast-import`? Sure, I was\n> > > really tempted to do that. In the end, it took less of _my_ time to just\n> > > let `git filter-branch` do its work with a not-too-complicated index\n> > > filter.\n> >\n> > Why a perl script? Shouldn't\n> > git fast-export [--no-data] HEAD -- $PATH | git fast-import --force --quiet\n> > do the trick? And it's probably simpler and shorter than the index\n> > filter you used.\n>\n> Does that not keep the full `$PATH`? I wanted the resulting branch to\n> have the file in the top-level directory.\n\nAh, gotcha; I read your original description to suggest that the\nscript was already at the toplevel.\n\n> > That said, yeah it'd be nice to get automatic rewriting of commit\n> > hashes in commit messages and other niceties from filter-repo (e.g.\n> > future automatic reattaching of notes to the rewritten commits). Some\n> > questions:\n> >\n> > * What's the backup strategy in case you specify the wrong filters\n> > (e.g. you have a typo in the pathnames)? filter-repo encourages folks\n> > to make a clone and then filter the fresh clone, because if anything\n> > goes awry, you can just delete and restart. (I am heavily opposed to\n> > the refs/original/ backup mechanism used by filter-branch, for\n> > multiple reasons.) Is your safety stance just \"If I mess up it's my\n> > own fault; do the rewrite?\" Or are you okay with cloning before\n> > filtering?\n>\n> Please note that the `refs/original/` refs should not have been written\n> at all anymore, not after reflogs were introduced.\n>\n> Incidentally, that is my answer to your question: the reflog is my\n> backup.\n\nThe reflog is great, but while it works in your special case please note that:\n\n1. Anyone filtering a subset of refs more in number than one may have\nsome difficulty restoring correctly (they have to look in several\nreflogs, and can't script restoring from all older reflog versions).\n2. Few folks have core.logAllRefUpdates set to 'always', meaning\nthey'll lack a backup for some refs if the reflog is relied upon.\n3. If the filter specifies only keeping a list of files that happen to\nnot exist within one of the branches (perhaps a filename was typo'ed)\nand if pruning empty commits, then the branch can be deleted, and git\ndoesn't have a mechanism for deleting a branch without deleting its\nreflog as far as I know.\n\nPoint 1 is kind of minor, but points 2 and 3 are showstoppers in\nregards to me recommending the reflogs as a reliable recovery\nmechanism after general filtering operations, and this is true for\neither filter-branch or filter-repo. (That said, I can definitely\nallow people to choose their risks and just provide some\nhere-be-dragons warnings.)\n\n> > * If you're okay with cloning before filtering...then is there an\n> > issue with rewriting all branches, and just pushing the one you need?\n> > (Is there an issue with \"this branch is small, the others are huge,\n> > and filter-branch is slow -- so rewriting one branch saves me lots of\n> > time\"? Or are there other issues at play too?)\n>\n> I am not okay with cloning before filtering.\n>\n> First of all, it is wasteful.\n>\n> Second of all, in my case it would have been *particularly* wasteful\n> because the repository in question also has quite a few quite large\n> blobs (hysterical raisins, don't ask).\n>\n> > * What if the user has auxiliary information for the branch in other\n> > refs? For example, git-notes pointing at any of the commits, or tags\n> > in the history of the branch that might be relevant, or perhaps even\n> > replace refs in combination with GIT_NO_REPLACE_OBJECTS=1? Is this an\n> > \"I don't care, toss that stuff and just rewrite just this branch?\"\n>\n> In my case: there are no notes. The only time when I make heavy use of\n> notes is in GitGitGadget. I don't use that feature otherwise.\n>\n> > * filter-repo by default creates new replace references so that you\n> > can refer to new commit IDs using old (unabbreviated) commit IDs.\n> > Would that be considered helpful for this usecase? unhelpful?\n> > irrelevant, since you'll just push the branch you want somewhere and\n> > nuke the temporary clone?\n>\n> I definitely did not need that mapping in all of my `git filter-branch`\n> use cases.\n>\n> Of course, I can see how it can come in handy in other circumstances,\n> just not in the ones I experienced so far.\n>\n> > I'm not by any means ruling out the possibility of documenting --refs\n> > and adjusting the defaults when it is used so the user can just run\n> > something like\n> > git filter-repo --path $PATH --refs $MYBRANCH\n> > but I feel like I need to understand answers to questions like the\n> > above ones so that I can know how to phrase warnings and adjust\n> > defaults and update the documentation.\n>\n> In all the scenarios where I used `git filter-branch` (some dozen per\n> year, so not all *that* many), I needed to rewrite one particular\n> branch, typically a freshly-created one. I never, ever ever needed to\n> rewrite all the refs in the repository. Not once ;-)\n\nThanks for answering all these and providing the extra context. Very helpful.\n\n> > > In another instance, a long, long time ago, I needed to restart a\n> > > repository which had included way too many files for its own good, then\n> > > rename the old repository and start with a fresh `master` that contained\n> > > but a single commit whose tree was identical to the previous `master`'s\n> > > tip commit. I simply grafted that commit, ran `git filter-branch` and\n> > > had precisely what I needed.\n> >\n> > filter-repo supports grafts and replace objects, the same as\n> > filter-branch. (Although, technically, I didn't have to do a thing to\n> > support it; fast-export does the special handling of rewriting based\n> > on grafts and replace objects.) So, I'd say this is fully supported.\n> >\n> > Side question: the git-replace documents suggest that the graft file\n> > is deprecated. Are there any timeframes or plans for phasing out\n> > beyond the git-replace manpage existing? Should I avoid documenting\n> > the graft file support in filter-repo? Should I include examples\n> > using not just git-replace but also using the graft file?\n>\n> I had meant to prepare a patch series to remove `grafts` support that\n> Junio could carry in `pu` until the time he considers it appropriate to\n> merge to `master`, but it seems that this task fell under the rag.\n>\n> The deprecation itself has been introduced in tags/v2.18.0-rc0~54^2~4,\n> i.e. it is official as of Git v2.18.0, which was released in mid-June\n> last year.\n>\n> My personal gut feeling is that we should let it simmer for another year\n> before removing support for the `grafts` file (and we may want to update\n> the label \"grafted\" when `git log` shows a shallow commit before we\n> remove that support for `grafts`).\n>\n> So I'll not work on that patch for now.\n\nThanks for the extra history.\n\n> > > I would be _delighted_ if these kinds of use case (rewriting a branch,\n> > > or even just a commit range) became more of a first-class citizen with\n> > > `git filter-repo`.\n> >\n> > I've got all the pieces for supporting a single branch or a commit\n> > range (e.g. 'git filter-repo --path foo --refs ^master~4 ^stable~23\n> > mybranch'), but the defaults (error out unless in a bare repo, move\n> > refs/remotes/origin/* to refs/heads/*, disconnect origin remote,\n> > expire reflogs & repack & prune, create new replace references so\n> > folks can access new commits using old commit IDs) may be somewhat\n> > friction-filled for this usecase. Those defaults other than the new\n> > replace refs happen to all be turned off with the combination of\n> > --force and --target, so, assuming turning them off is what you need,\n> > you could cheat and just specify 'git filter-repo --force --target .\n> > --refs $MYBRANCH' today and perhaps get what you want, but that's a\n> > really non-intuitive command line that is way too ugly to recommend.\n> > And I don't want to tie myself to '--target .' being the magic sauce\n> > in the future either.\n>\n> I agree. I would love for my use cases to become more of first-class\n> citizens. Maybe `--branch ` could serve as the knob?\n\nI think I can put something together to make your usecases better. I\ndislike --branch, though, because:\n * It suggests it doesn't work for tags or other things outside of refs/heads/*\n * It suggests that revision ranges are unwelcome.\nSo, I'd prefer a more generic --refs which can potentially take\nmultiple arguments, e.g. any of\n --refs mybranch\n --refs mytag\n --refs HEAD~5..mybranch\n --refs ^origin/master ^origin/other-feature mybranch1 mybranch2\n\n> What I also found really helpful in `git filter-branch` is that it was\n> possible to pass one-liner shell scripts directly to the command, giving\n> a lot of freedom about the transformations. I understand that Python\n> makes it hard to write spaghetti-code one-liners, so you cannot really\n> pass the snippet in via the command-line, but I hope there is a way to\n> script things in `git filter-repo`?\n\nI agree, having the ability to use a programming language snippet or\neven full script for special cases is really nice. You can totally do\nthat filter-repo, at three different levels:\n\n1. Light-control for easier cases: You can use the command line\narguments --filename-callback, --message-callback, --name-callback,\n--email-callback, or --refname-callback and provide python snippets\n(usually one-liners) that return a new value. Most of these are for\nediting fields shared across multiple object types (e.g. the name\ncallback will be used to edit author && committer && tagger names).\nHowever, the filename callback allows both editing the filename and\nalso filtering based upon filename (you can return the original\nfilename, OR a new name, OR you can return None to state that you want\nfiles with that name filtered out of commits).\n\n2. Moderate-control: You can use the command line arguments\n--blob-callback, --commit-callback, --tag-callback, or\n--reset-callback and provide python snippets (possibly one-liners but\nmore likely to be complex enough that you want newlines) that modify\nthese fast-import-stream objects. These provide more control but tend\nto be slightly more work. (For example, if you want to rename\nbranches, you need to worry about three callbacks: commit && tag &&\nreset; by comparison, you'd only need to use the refname callback from\nlighter control. Also, if you're worried about filtering based on\nfilenames, you'll need to dig the filenames (and modes and change\ntypes) out of the list commit.file_changes).\n\n3. All-in: You can write a python script that imports filter-repo as a\npython module and (among other things) set up your own\nfunctions/classes as callbacks. You have to do a bit more setup to\nspecify the options you are running with, list how many export and\nimport processes you want to run with, name all your callbacks, etc.,\nbut it allows you to do anything from just providing a slightly more\ninvolved callback up to and including creating your own filtering tool\nwith a totally different user interface while still leveraging\nfilter-repo's capability. There are multiple examples provided along\nthat range too (including bfg-ish and\nfilter-lamely/filter-branch-ish.)\n\nFor more details about all of these, see:\nhttps://github.com/newren/git-filter-repo/blob/a6a6a1b0/README.md#callbacks\nhttps://github.com/newren/git-filter-repo/blob/a6a6a1b0/README.md#using-filter-repo-as-a-library\n"},{"id":"381752","messageId":"20190903185524.13467-1-newren@gmail.com","threadId":"51710","inReplyTo":"20190828002210.8862-1-newren@gmail.com","subject":"[PATCH v5 0/4] Warn about git-filter-branch usage and avoid it","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-09-03T18:55:20Z","receivedAt":"2019-09-03T18:55:31Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"It's been about 5 days with no further feedback, other than some timings\nfrom Dscho for Windows showing that my fixes help there too. So, I did\none last re-read, made a couple small wording tweaks, and am resending as\nready for inclusion.\n\nChanges since v4:\n * Included the windows timings from Dscho in the commit messages for\n the first two perf patches\n * A few slight wording tweaks to the manpage\n\nElijah Newren (4):\n t6006: simplify and optimize empty message test\n t3427: accelerate this test by using fast-export and fast-import\n Recommend git-filter-repo instead of git-filter-branch\n t9902: use a non-deprecated command for testing\n\n Documentation/git-fast-export.txt | 6 +-\n Documentation/git-filter-branch.txt | 273 +++++++++++++++++++++++++---\n Documentation/git-gc.txt | 17 +-\n Documentation/git-rebase.txt | 3 +-\n Documentation/git-replace.txt | 10 +-\n Documentation/git-svn.txt | 10 +-\n Documentation/githooks.txt | 10 +-\n contrib/svn-fe/svn-fe.txt | 4 +-\n git-filter-branch.sh | 13 ++\n t/t3427-rebase-subtree.sh | 24 ++-\n t/t6006-rev-list-format.sh | 5 +-\n t/t9902-completion.sh | 12 +-\n 12 files changed, 310 insertions(+), 77 deletions(-)\n\nRange-diff:\n1: 7ddbeea2ca ! 1: ccea0e5846 t6006: simplify and optimize empty message test\n @@ Commit message\n Despite only being one piece of the 71st test and there being 73 tests\n overall, this small change to just this one test speeds up the overall\n execution time of t6006 (as measured by the best of 3 runs of `time\n - ./t6006-rev-list-format.sh`) by about 11% on Linux and by 13% on\n - Mac.\n + ./t6006-rev-list-format.sh`) by about 11% on Linux, 13% on Mac, and\n + about 15% on Windows.\n \n Signed-off-by: Elijah Newren \n \n2: e1e63189c1 ! 2: 6d73135006 t3427: accelerate this test by using fast-export and fast-import\n @@ Commit message\n \n fast-export and fast-import can easily handle the simple rewrite that\n was being done by filter-branch, and should be significantly faster on\n - systems with a slow fork. Timings from before and after on two laptops\n - that I have access to (measured via `time ./t3427-rebase-subtree.sh`,\n - i.e. including everything in this test -- not just the filter-branch or\n - fast-export/fast-import pair):\n + systems with a slow fork. Timings from before and after on a few\n + laptops that I or others measured on (measured via `time\n + ./t3427-rebase-subtree.sh`, i.e. including everything in this test --\n + not just the filter-branch or fast-export/fast-import pair):\n \n - Linux: 4.305s -> 3.684s (~17% speedup)\n - Mac: 10.128s -> 7.038s (~30% speedup)\n + Linux: 4.305s -> 3.684s (~17% speedup)\n + Mac: 10.128s -> 7.038s (~30% speedup)\n + Windows: 1m 37s -> 1m 17s (~26% speedup)\n \n Signed-off-by: Elijah Newren \n \n3: ed6505584f ! 3: 2f225c8697 Recommend git-filter-repo instead of git-filter-branch\n @@ Documentation/git-filter-branch.txt: warned.\n +document or provide to a coworker, who then runs them on a different OS\n +where the same commands are not working/tested (some examples in the\n +git-filter-branch manpage are also affected by this). BSD vs. GNU\n -+userland differences can really bite. If you're lucky, you get ugly\n -+error messages spewed. But just as likely, the commands either don't do\n -+the filtering requested, or silently corrupt making some unwanted\n -+change. The unwanted change may only affect a few commits, so it's not\n -+necessarily obvious either. (The fact that problems won't necessarily\n -+be obvious means they are likely to go unnoticed until the rewritten\n -+history is in use for quite a while, at which point it's really hard to\n -+justify another flag-day for another rewrite.)\n ++userland differences can really bite. If lucky, error messages are\n ++spewed. But just as likely, the commands either don't do the filtering\n ++requested, or silently corrupt by making some unwanted change. The\n ++unwanted change may only affect a few commits, so it's not necessarily\n ++obvious either. (The fact that problems won't necessarily be obvious\n ++means they are likely to go unnoticed until the rewritten history is in\n ++use for quite a while, at which point it's really hard to justify\n ++another flag-day for another rewrite.)\n +\n +* Filenames with spaces are often mishandled by shell snippets since\n +they cause problems for shell pipelines. Not everyone is familiar with\n +find -print0, xargs -0, git-ls-files -z, etc. Even people who are\n -+familiar with these may assume such needs are not relevant because\n ++familiar with these may assume such flags are not relevant because\n +someone else renamed any such files in their repo back before the person\n -+doing the filtering joined the project. And, often, even those familiar\n -+with handling arguments with spaces my not do so just because they\n ++doing the filtering joined the project. And often, even those familiar\n ++with handling arguments with spaces may not do so just because they\n +aren't in the mindset of thinking about everything that could possibly\n +go wrong.\n +\n +* Non-ascii filenames can be silently removed despite being in a desired\n -+directory. The desire to select paths to keep often use pipelines like\n ++directory. Keeping only wanted paths is often done using pipelines like\n +`git ls-files | grep -v ^WANTED_DIR/ | xargs git rm`. ls-files will\n -+only quote filenames if needed so folks may not notice that one of the\n -+files didn't match the regex, again until it's much too late. Yes,\n -+someone who knows about core.quotePath can avoid this (unless they have\n -+other special characters like \\t, \\n, or \"), and people who use ls-files\n -+-z with something other than grep can avoid this, but that doesn't mean\n -+they will.\n ++only quote filenames if needed, so folks may not notice that one of the\n ++files didn't match the regex (at least not until it's much too late).\n ++Yes, someone who knows about core.quotePath can avoid this (unless they\n ++have other special characters like \\t, \\n, or \"), and people who use\n ++ls-files -z with something other than grep can avoid this, but that\n ++doesn't mean they will.\n +\n +* Similarly, when moving files around, one can find that filenames with\n +non-ascii or special characters end up in a different directory, one\n @@ Documentation/git-filter-branch.txt: warned.\n +the same name, no warning or error is provided; git-filter-branch simply\n +overwrites each tag in some undocumented pre-defined order resulting in\n +only one tag at the end. (A git-filter-branch regression test requires\n -+this.)\n ++this surprising behavior.)\n +\n -+Also, the poor performance of git-filter-branch often leads to safety issues:\n ++Also, the poor performance of git-filter-branch often leads to safety\n ++issues:\n +\n +* Coming up with the correct shell snippet to do the filtering you want\n +is sometimes difficult unless you're just doing a trivial modification\n4: ca8e124cb3 = 4: 048eba375b t9902: use a non-deprecated command for testing\n-- \n2.23.0.39.gf92d9de5c3\n\n"},{"id":"381753","messageId":"20190903185524.13467-3-newren@gmail.com","threadId":"51710","inReplyTo":"20190903185524.13467-1-newren@gmail.com","subject":"[PATCH v5 2/4] t3427: accelerate this test by using fast-export and fast-import","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-09-03T18:55:22Z","receivedAt":"2019-09-03T18:55:35Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"fast-export and fast-import can easily handle the simple rewrite that\nwas being done by filter-branch, and should be significantly faster on\nsystems with a slow fork. Timings from before and after on a few\nlaptops that I or others measured on (measured via `time\n./t3427-rebase-subtree.sh`, i.e. including everything in this test --\nnot just the filter-branch or fast-export/fast-import pair):\n\n Linux: 4.305s -> 3.684s (~17% speedup)\n Mac: 10.128s -> 7.038s (~30% speedup)\n Windows: 1m 37s -> 1m 17s (~26% speedup)\n\nSigned-off-by: Elijah Newren \n---\n t/t3427-rebase-subtree.sh | 24 +++++++++++++++---------\n 1 file changed, 15 insertions(+), 9 deletions(-)\n\ndiff --git a/t/t3427-rebase-subtree.sh b/t/t3427-rebase-subtree.sh\nindex d8640522a0..c1f6102921 100755\n--- a/t/t3427-rebase-subtree.sh\n+++ b/t/t3427-rebase-subtree.sh\n@@ -7,10 +7,16 @@ This test runs git rebase and tests the subtree strategy.\n . ./test-lib.sh\n . \"$TEST_DIRECTORY\"/lib-rebase.sh\n \n-commit_message() {\n+commit_message () {\n \tgit log --pretty=format:%s -1 \"$1\"\n }\n \n+extract_files_subtree () {\n+\tgit fast-export --no-data HEAD -- files_subtree/ |\n+\t\tsed -e \"s%\\([0-9a-f]\\{40\\} \\)files_subtree/%\\1%\" |\n+\t\tgit fast-import --force --quiet\n+}\n+\n test_expect_success 'setup' '\n \ttest_commit README &&\n \tmkdir files &&\n@@ -42,7 +48,7 @@ test_expect_failure REBASE_P \\\n \t'Rebase -Xsubtree --preserve-merges --onto commit 4' '\n \treset_rebase &&\n \tgit checkout -b rebase-preserve-merges-4 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --preserve-merges --onto files-master master &&\n \tverbose test \"$(commit_message HEAD~)\" = \"files_subtree/master4\"\n@@ -53,7 +59,7 @@ test_expect_failure REBASE_P \\\n \t'Rebase -Xsubtree --preserve-merges --onto commit 5' '\n \treset_rebase &&\n \tgit checkout -b rebase-preserve-merges-5 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --preserve-merges --onto files-master master &&\n \tverbose test \"$(commit_message HEAD)\" = \"files_subtree/master5\"\n@@ -64,7 +70,7 @@ test_expect_failure REBASE_P \\\n \t'Rebase -Xsubtree --keep-empty --preserve-merges --onto commit 4' '\n \treset_rebase &&\n \tgit checkout -b rebase-keep-empty-4 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master &&\n \tverbose test \"$(commit_message HEAD~2)\" = \"files_subtree/master4\"\n@@ -75,7 +81,7 @@ test_expect_failure REBASE_P \\\n \t'Rebase -Xsubtree --keep-empty --preserve-merges --onto commit 5' '\n \treset_rebase &&\n \tgit checkout -b rebase-keep-empty-5 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master &&\n \tverbose test \"$(commit_message HEAD~)\" = \"files_subtree/master5\"\n@@ -86,7 +92,7 @@ test_expect_failure REBASE_P \\\n \t'Rebase -Xsubtree --keep-empty --preserve-merges --onto empty commit' '\n \treset_rebase &&\n \tgit checkout -b rebase-keep-empty-empty master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master &&\n \tverbose test \"$(commit_message HEAD)\" = \"Empty commit\"\n@@ -96,7 +102,7 @@ test_expect_failure REBASE_P \\\n test_expect_failure 'Rebase -Xsubtree --onto commit 4' '\n \treset_rebase &&\n \tgit checkout -b rebase-onto-4 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --onto files-master master &&\n \tverbose test \"$(commit_message HEAD~2)\" = \"files_subtree/master4\"\n@@ -106,7 +112,7 @@ test_expect_failure 'Rebase -Xsubtree --onto commit 4' '\n test_expect_failure 'Rebase -Xsubtree --onto commit 5' '\n \treset_rebase &&\n \tgit checkout -b rebase-onto-5 master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --onto files-master master &&\n \tverbose test \"$(commit_message HEAD~)\" = \"files_subtree/master5\"\n@@ -115,7 +121,7 @@ test_expect_failure 'Rebase -Xsubtree --onto commit 5' '\n test_expect_failure 'Rebase -Xsubtree --onto empty commit' '\n \treset_rebase &&\n \tgit checkout -b rebase-onto-empty master &&\n-\tgit filter-branch --prune-empty -f --subdirectory-filter files_subtree &&\n+\textract_files_subtree &&\n \tgit commit -m \"Empty commit\" --allow-empty &&\n \tgit rebase -Xsubtree=files_subtree --onto files-master master &&\n \tverbose test \"$(commit_message HEAD)\" = \"Empty commit\"\n-- \n2.23.0.39.gf92d9de5c3\n\n"},{"id":"381754","messageId":"20190903185524.13467-2-newren@gmail.com","threadId":"51710","inReplyTo":"20190903185524.13467-1-newren@gmail.com","subject":"[PATCH v5 1/4] t6006: simplify and optimize empty message test","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-09-03T18:55:21Z","receivedAt":"2019-09-03T18:55:35Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Test t6006.71 (\"oneline with empty message\") was creating two commits\nwith simple commit messages, and then running filter-branch to rewrite\nthe commit messages to be empty. This test was written this way because\nthe --allow-empty-message option to git commit did not exist at the\ntime. Simplify this test and avoid the need to invoke filter-branch by\njust using --allow-empty-message when creating the commit.\n\nDespite only being one piece of the 71st test and there being 73 tests\noverall, this small change to just this one test speeds up the overall\nexecution time of t6006 (as measured by the best of 3 runs of `time\n./t6006-rev-list-format.sh`) by about 11% on Linux, 13% on Mac, and\nabout 15% on Windows.\n\nSigned-off-by: Elijah Newren \n---\n t/t6006-rev-list-format.sh | 5 ++---\n 1 file changed, 2 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t6006-rev-list-format.sh b/t/t6006-rev-list-format.sh\nindex da113d975b..d30e41c9f7 100755\n--- a/t/t6006-rev-list-format.sh\n+++ b/t/t6006-rev-list-format.sh\n@@ -501,9 +501,8 @@ test_expect_success 'reflog identity' '\n '\n \n test_expect_success 'oneline with empty message' '\n-\tgit commit -m \"dummy\" --allow-empty &&\n-\tgit commit -m \"dummy\" --allow-empty &&\n-\tgit filter-branch --msg-filter \"sed -e s/dummy//\" HEAD^^.. &&\n+\tgit commit --allow-empty --allow-empty-message &&\n+\tgit commit --allow-empty --allow-empty-message &&\n \tgit rev-list --oneline HEAD >test.txt &&\n \ttest_line_count = 5 test.txt &&\n \tgit rev-list --oneline --graph HEAD >testg.txt &&\n-- \n2.23.0.39.gf92d9de5c3\n\n"},{"id":"381756","messageId":"20190903185524.13467-4-newren@gmail.com","threadId":"51710","inReplyTo":"20190903185524.13467-1-newren@gmail.com","subject":"[PATCH v5 3/4] Recommend git-filter-repo instead of git-filter-branch","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-09-03T18:55:23Z","receivedAt":"2019-09-03T18:55:36Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"filter-branch suffers from a deluge of disguised dangers that disfigure\nhistory rewrites (i.e. deviate from the deliberate changes). Many of\nthese problems are unobtrusive and can easily go undiscovered until the\nnew repository is in use. This can result in problems ranging from an\neven messier history than what led folks to filter-branch in the first\nplace, to data loss or corruption. These issues cannot be backward\ncompatibly fixed, so add a warning to both filter-branch and its manpage\nrecommending that another tool (such as filter-repo) be used instead.\n\nAlso, update other manpages that referenced filter-branch. Several of\nthese needed updates even if we could continue recommending\nfilter-branch, either due to implying that something was unique to\nfilter-branch when it applied more generally to all history rewriting\ntools (e.g. BFG, reposurgeon, fast-import, filter-repo), or because\nsomething about filter-branch was used as an example despite other more\ncommonly known examples now existing. Reword these sections to fix\nthese issues and to avoid recommending filter-branch.\n\nFinally, remove the section explaining BFG Repo Cleaner as an\nalternative to filter-branch. I feel somewhat bad about this,\nespecially since I feel like I learned so much from BFG that I put to\ngood use in filter-repo (which is much more than I can say for\nfilter-branch), but keeping that section presented a few problems:\n * In order to recommend that people quit using filter-branch, we need\n to provide them a recomendation for something else to use that\n can handle all the same types of rewrites. To my knowledge,\n filter-repo is the only such tool. So it needs to be mentioned.\n * I don't want to give conflicting recommendations to users\n * If we recommend two tools, we shouldn't expect users to learn both\n and pick which one to use; we should explain which problems one\n can solve that the other can't or when one is much faster than\n the other.\n * BFG and filter-repo have similar performance\n * All filtering types that BFG can do, filter-repo can also do. In\n fact, filter-repo comes with a reimplementation of BFG named\n bfg-ish which provides the same user-interface as BFG but with\n several bugfixes and new features that are hard to implement in\n BFG due to its technical underpinnings.\nWhile I could still mention both tools, it seems like I would need to\nprovide some kind of comparison and I would ultimately just say that\nfilter-repo can do everything BFG can, so ultimately it seems that it\nis just better to remove that section altogether.\n\nSigned-off-by: Elijah Newren \n---\n Documentation/git-fast-export.txt | 6 +-\n Documentation/git-filter-branch.txt | 273 +++++++++++++++++++++++++---\n Documentation/git-gc.txt | 17 +-\n Documentation/git-rebase.txt | 3 +-\n Documentation/git-replace.txt | 10 +-\n Documentation/git-svn.txt | 10 +-\n Documentation/githooks.txt | 10 +-\n contrib/svn-fe/svn-fe.txt | 4 +-\n git-filter-branch.sh | 13 ++\n 9 files changed, 287 insertions(+), 59 deletions(-)\n\ndiff --git a/Documentation/git-fast-export.txt b/Documentation/git-fast-export.txt\nindex cc940eb9ad..784e934009 100644\n--- a/Documentation/git-fast-export.txt\n+++ b/Documentation/git-fast-export.txt\n@@ -17,9 +17,9 @@ This program dumps the given revisions in a form suitable to be piped\n into 'git fast-import'.\n \n You can use it as a human-readable bundle replacement (see\n-linkgit:git-bundle[1]), or as a kind of an interactive\n-'git filter-branch'.\n-\n+linkgit:git-bundle[1]), or as a format that can be edited before being\n+fed to 'git fast-import' in order to do history rewrites (an ability\n+relied on by tools like 'git filter-repo').\n \n OPTIONS\n -------\ndiff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt\nindex 6b53dd7e06..5876598852 100644\n--- a/Documentation/git-filter-branch.txt\n+++ b/Documentation/git-filter-branch.txt\n@@ -16,6 +16,19 @@ SYNOPSIS\n \t[--original ] [-d ] [-f | --force]\n \t[--state-branch ] [--] [...]\n \n+WARNING\n+-------\n+'git filter-branch' has a plethora of pitfalls that can produce non-obvious\n+manglings of the intended history rewrite (and can leave you with little\n+time to investigate such problems since it has such abysmal performance).\n+These safety and performance issues cannot be backward compatibly fixed and\n+as such, its use is not recommended. Please use an alternative history\n+filtering tool such as https://github.com/newren/git-filter-repo/[git\n+filter-repo]. If you still need to use 'git filter-branch', please\n+carefully read <> (and <>) to learn about the land\n+mines of filter-branch, and then vigilantly avoid as many of the hazards\n+listed there as reasonably possible.\n+\n DESCRIPTION\n -----------\n Lets you rewrite Git revision history by rewriting the branches mentioned\n@@ -445,36 +458,236 @@ warned.\n (or if your git-gc is not new enough to support arguments to\n `--prune`, use `git repack -ad; git prune` instead).\n \n-NOTES\n------\n-\n-git-filter-branch allows you to make complex shell-scripted rewrites\n-of your Git history, but you probably don't need this flexibility if\n-you're simply _removing unwanted data_ like large files or passwords.\n-For those operations you may want to consider\n-http://rtyley.github.io/bfg-repo-cleaner/[The BFG Repo-Cleaner],\n-a JVM-based alternative to git-filter-branch, typically at least\n-10-50x faster for those use-cases, and with quite different\n-characteristics:\n-\n-* Any particular version of a file is cleaned exactly _once_. The BFG,\n- unlike git-filter-branch, does not give you the opportunity to\n- handle a file differently based on where or when it was committed\n- within your history. This constraint gives the core performance\n- benefit of The BFG, and is well-suited to the task of cleansing bad\n- data - you don't care _where_ the bad data is, you just want it\n- _gone_.\n-\n-* By default The BFG takes full advantage of multi-core machines,\n- cleansing commit file-trees in parallel. git-filter-branch cleans\n- commits sequentially (i.e. in a single-threaded manner), though it\n- _is_ possible to write filters that include their own parallelism,\n- in the scripts executed against each commit.\n-\n-* The http://rtyley.github.io/bfg-repo-cleaner/#examples[command options]\n- are much more restrictive than git-filter branch, and dedicated just\n- to the tasks of removing unwanted data- e.g:\n- `--strip-blobs-bigger-than 1M`.\n+[[PERFORMANCE]]\n+PERFORMANCE\n+-----------\n+\n+The performance of git-filter-branch is glacially slow; its design makes it\n+impossible for a backward-compatible implementation to ever be fast:\n+\n+* In editing files, git-filter-branch by design checks out each and\n+every commit as it existed in the original repo. If your repo has 10\\^5\n+files and 10\\^5 commits, but each commit only modifies 5 files, then\n+git-filter-branch will make you do 10\\^10 modifications, despite only\n+having (at most) 5*10^5 unique blobs.\n+\n+* If you try and cheat and try to make git-filter-branch only work on\n+files modified in a commit, then two things happen\n+\n+ ** you run into problems with deletions whenever the user is simply\n+ trying to rename files (because attempting to delete files that\n+ don't exist looks like a no-op; it takes some chicanery to remap\n+ deletes across file renames when the renames happen via arbitrary\n+ user-provided shell)\n+\n+ ** even if you succeed at the map-deletes-for-renames chicanery, you\n+ still technically violate backward compatibility because users are\n+ allowed to filter files in ways that depend upon topology of\n+ commits instead of filtering solely based on file contents or names\n+ (though this has not been observed in the wild).\n+\n+* Even if you don't need to edit files but only want to e.g. rename or\n+remove some and thus can avoid checking out each file (i.e. you can use\n+--index-filter), you still are passing shell snippets for your filters.\n+This means that for every commit, you have to have a prepared git repo\n+where those filters can be run. That's a significant setup.\n+\n+* Further, several additional files are created or updated per commit by\n+git-filter-branch. Some of these are for supporting the convenience\n+functions provided by git-filter-branch (such as map()), while others\n+are for keeping track of internal state (but could have also been\n+accessed by user filters; one of git-filter-branch's regression tests\n+does so). This essentially amounts to using the filesystem as an IPC\n+mechanism between git-filter-branch and the user-provided filters.\n+Disks tend to be a slow IPC mechanism, and writing these files also\n+effectively represents a forced synchronization point between separate\n+processes that we hit with every commit.\n+\n+* The user-provided shell commands will likely involve a pipeline of\n+commands, resulting in the creation of many processes per commit.\n+Creating and running another process takes a widely varying amount of\n+time between operating systems, but on any platform it is very slow\n+relative to invoking a function.\n+\n+* git-filter-branch itself is written in shell, which is kind of slow.\n+This is the one performance issue that could be backward-compatibly\n+fixed, but compared to the above problems that are intrinsic to the\n+design of git-filter-branch, the language of the tool itself is a\n+relatively minor issue.\n+\n+ ** Side note: Unfortunately, people tend to fixate on the\n+ written-in-shell aspect and periodically ask if git-filter-branch\n+ could be rewritten in another language to fix the performance\n+ issues. Not only does that ignore the bigger intrinsic problems\n+ with the design, it'd help less than you'd expect: if\n+ git-filter-branch itself were not shell, then the convenience\n+ functions (map(), skip_commit(), etc) and the `--setup` argument\n+ could no longer be executed once at the beginning of the program\n+ but would instead need to be prepended to every user filter (and\n+ thus re-executed with every commit).\n+\n+The https://github.com/newren/git-filter-repo/[git filter-repo] tool is\n+an alternative to git-filter-branch which does not suffer from these\n+performance problems or the safety problems (mentioned below). For those\n+with existing tooling which relies upon git-filter-branch, 'git\n+repo-filter' also provides\n+https://github.com/newren/git-filter-repo/blob/master/contrib/filter-repo-demos/filter-lamely[filter-lamely],\n+a drop-in git-filter-branch replacement (with a few caveats). While\n+filter-lamely suffers from all the same safety issues as\n+git-filter-branch, it at least ameloriates the performance issues a\n+little.\n+\n+[[SAFETY]]\n+SAFETY\n+------\n+\n+git-filter-branch is riddled with gotchas resulting in various ways to\n+easily corrupt repos or end up with a mess worse than what you started\n+with:\n+\n+* Someone can have a set of \"working and tested filters\" which they\n+document or provide to a coworker, who then runs them on a different OS\n+where the same commands are not working/tested (some examples in the\n+git-filter-branch manpage are also affected by this). BSD vs. GNU\n+userland differences can really bite. If lucky, error messages are\n+spewed. But just as likely, the commands either don't do the filtering\n+requested, or silently corrupt by making some unwanted change. The\n+unwanted change may only affect a few commits, so it's not necessarily\n+obvious either. (The fact that problems won't necessarily be obvious\n+means they are likely to go unnoticed until the rewritten history is in\n+use for quite a while, at which point it's really hard to justify\n+another flag-day for another rewrite.)\n+\n+* Filenames with spaces are often mishandled by shell snippets since\n+they cause problems for shell pipelines. Not everyone is familiar with\n+find -print0, xargs -0, git-ls-files -z, etc. Even people who are\n+familiar with these may assume such flags are not relevant because\n+someone else renamed any such files in their repo back before the person\n+doing the filtering joined the project. And often, even those familiar\n+with handling arguments with spaces may not do so just because they\n+aren't in the mindset of thinking about everything that could possibly\n+go wrong.\n+\n+* Non-ascii filenames can be silently removed despite being in a desired\n+directory. Keeping only wanted paths is often done using pipelines like\n+`git ls-files | grep -v ^WANTED_DIR/ | xargs git rm`. ls-files will\n+only quote filenames if needed, so folks may not notice that one of the\n+files didn't match the regex (at least not until it's much too late).\n+Yes, someone who knows about core.quotePath can avoid this (unless they\n+have other special characters like \\t, \\n, or \"), and people who use\n+ls-files -z with something other than grep can avoid this, but that\n+doesn't mean they will.\n+\n+* Similarly, when moving files around, one can find that filenames with\n+non-ascii or special characters end up in a different directory, one\n+that includes a double quote character. (This is technically the same\n+issue as above with quoting, but perhaps an interesting different way\n+that it can and has manifested as a problem.)\n+\n+* It's far too easy to accidentally mix up old and new history. It's\n+still possible with any tool, but git-filter-branch almost invites it.\n+If lucky, the only downside is users getting frustrated that they don't\n+know how to shrink their repo and remove the old stuff. If unlucky,\n+they merge old and new history and end up with multiple \"copies\" of each\n+commit, some of which have unwanted or sensitive files and others which\n+don't. This comes about in multiple different ways:\n+\n+ ** the default to only doing a partial history rewrite ('--all' is not\n+ the default and few examples show it)\n+\n+ ** the fact that there's no automatic post-run cleanup\n+\n+ ** the fact that --tag-name-filter (when used to rename tags) doesn't\n+ remove the old tags but just adds new ones with the new name\n+\n+ ** the fact that little educational information is provided to inform\n+ users of the ramifications of a rewrite and how to avoid mixing old\n+ and new history. For example, this man page discusses how users\n+ need to understand that they need to rebase their changes for all\n+ their branches on top of new history (or delete and reclone), but\n+ that's only one of multiple concerns to consider. See the\n+ \"DISCUSSION\" section of the git filter-repo manual page for more\n+ details.\n+\n+* Annotated tags can be accidentally converted to lightweight tags, due\n+to either of two issues:\n+\n+ ** Someone can do a history rewrite, realize they messed up, restore\n+ from the backups in refs/original/, and then redo their\n+ git-filter-branch command. (The backup in refs/original/ is not a\n+ real backup; it dereferences tags first.)\n+\n+ ** Running git-filter-branch with either --tags or --all in your\n+ . In order to retain annotated tags as\n+ annotated, you must use --tag-name-filter (and must not have\n+ restored from refs/original/ in a previously botched rewrite).\n+\n+* Any commit messages that specify an encoding will become corrupted\n+by the rewrite; git-filter-branch ignores the encoding, takes the original\n+bytes, and feeds it to commit-tree without telling it the proper\n+encoding. (This happens whether or not --msg-filter is used.)\n+\n+* Commit messages (even if they are all UTF-8) by default become\n+corrupted due to not being updated -- any references to other commit\n+hashes in commit messages will now refer to no-longer-extant commits.\n+\n+* There are no facilities for helping users find what unwanted crud they\n+should delete, which means they are much more likely to have incomplete\n+or partial cleanups that sometimes result in confusion and people\n+wasting time trying to understand. (For example, folks tend to just\n+look for big files to delete instead of big directories or extensions,\n+and once they do so, then sometime later folks using the new repository\n+who are going through history will notice a build artifact directory\n+that has some files but not others, or a cache of dependencies\n+(node_modules or similar) which couldn't have ever been functional since\n+it's missing some files.)\n+\n+* If --prune-empty isn't specified, then the filtering process can\n+create hoards of confusing empty commits\n+\n+* If --prune-empty is specified, then intentionally placed empty\n+commits from before the filtering operation are also pruned instead of\n+just pruning commits that became empty due to filtering rules.\n+\n+* If --prune empty is specified, sometimes empty commits are missed\n+and left around anyway (a somewhat rare bug, but it happens...)\n+\n+* A minor issue, but users who have a goal to update all names and\n+emails in a repository may be led to --env-filter which will only update\n+authors and committers, missing taggers.\n+\n+* If the user provides a --tag-name-filter that maps multiple tags to\n+the same name, no warning or error is provided; git-filter-branch simply\n+overwrites each tag in some undocumented pre-defined order resulting in\n+only one tag at the end. (A git-filter-branch regression test requires\n+this surprising behavior.)\n+\n+Also, the poor performance of git-filter-branch often leads to safety\n+issues:\n+\n+* Coming up with the correct shell snippet to do the filtering you want\n+is sometimes difficult unless you're just doing a trivial modification\n+such as deleting a couple files. Unfortunately, people often learn if\n+the snippet is right or wrong by trying it out, but the rightness or\n+wrongness can vary depending on special circumstances (spaces in\n+filenames, non-ascii filenames, funny author names or emails, invalid\n+timezones, presence of grafts or replace objects, etc.), meaning they\n+may have to wait a long time, hit an error, then restart. The\n+performance of git-filter-branch is so bad that this cycle is painful,\n+reducing the time available to carefully re-check (to say nothing about\n+what it does to the patience of the person doing the rewrite even if\n+they do technically have more time available). This problem is extra\n+compounded because errors from broken filters may not be shown for a\n+long time and/or get lost in a sea of output. Even worse, broken\n+filters often just result in silent incorrect rewrites.\n+\n+* To top it all off, even when users finally find working commands, they\n+naturally want to share them. But they may be unaware that their repo\n+didn't have some special cases that someone else's does. So, when\n+someone else with a different repository runs the same commands, they\n+get hit by the problems above. Or, the user just runs commands that\n+really were vetted for special cases, but they run it on a different OS\n+where it doesn't work, as noted above.\n \n GIT\n ---\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 247f765604..0c114ad1ca 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -115,15 +115,14 @@ NOTES\n -----\n \n 'git gc' tries very hard not to delete objects that are referenced\n-anywhere in your repository. In\n-particular, it will keep not only objects referenced by your current set\n-of branches and tags, but also objects referenced by the index,\n-remote-tracking branches, refs saved by 'git filter-branch' in\n-refs/original/, reflogs (which may reference commits in branches\n-that were later amended or rewound), and anything else in the refs/* namespace.\n-If you are expecting some objects to be deleted and they aren't, check\n-all of those locations and decide whether it makes sense in your case to\n-remove those references.\n+anywhere in your repository. In particular, it will keep not only\n+objects referenced by your current set of branches and tags, but also\n+objects referenced by the index, remote-tracking branches, notes saved\n+by 'git notes' under refs/notes/, reflogs (which may reference commits\n+in branches that were later amended or rewound), and anything else in\n+the refs/* namespace. If you are expecting some objects to be deleted\n+and they aren't, check all of those locations and decide whether it\n+makes sense in your case to remove those references.\n \n On the other hand, when 'git gc' runs concurrently with another process,\n there is a risk of it deleting an object that the other process is using\ndiff --git a/Documentation/git-rebase.txt b/Documentation/git-rebase.txt\nindex 6156609cf7..a8cfc0ad82 100644\n--- a/Documentation/git-rebase.txt\n+++ b/Documentation/git-rebase.txt\n@@ -832,7 +832,8 @@ Hard case: The changes are not the same.::\n \tThis happens if the 'subsystem' rebase had conflicts, or used\n \t`--interactive` to omit, edit, squash, or fixup commits; or\n \tif the upstream used one of `commit --amend`, `reset`, or\n-\t`filter-branch`.\n+\ta full history rewriting command like\n+\thttps://github.com/newren/git-filter-repo[`filter-repo`].\n \n \n The easy case\ndiff --git a/Documentation/git-replace.txt b/Documentation/git-replace.txt\nindex 246dc9943c..f271d758c3 100644\n--- a/Documentation/git-replace.txt\n+++ b/Documentation/git-replace.txt\n@@ -123,10 +123,10 @@ The following format are available:\n CREATING REPLACEMENT OBJECTS\n ----------------------------\n \n-linkgit:git-filter-branch[1], linkgit:git-hash-object[1] and\n-linkgit:git-rebase[1], among other git commands, can be used to create\n-replacement objects from existing objects. The `--edit` option can\n-also be used with 'git replace' to create a replacement object by\n+linkgit:git-hash-object[1], linkgit:git-rebase[1], and\n+https://github.com/newren/git-filter-repo[git-filter-repo], among other git commands, can be used to\n+create replacement objects from existing objects. The `--edit` option\n+can also be used with 'git replace' to create a replacement object by\n editing an existing object.\n \n If you want to replace many blobs, trees or commits that are part of a\n@@ -148,13 +148,13 @@ pending objects.\n SEE ALSO\n --------\n linkgit:git-hash-object[1]\n-linkgit:git-filter-branch[1]\n linkgit:git-rebase[1]\n linkgit:git-tag[1]\n linkgit:git-branch[1]\n linkgit:git-commit[1]\n linkgit:git-var[1]\n linkgit:git[1]\n+https://github.com/newren/git-filter-repo[git-filter-repo]\n \n GIT\n ---\ndiff --git a/Documentation/git-svn.txt b/Documentation/git-svn.txt\nindex 30711625fd..53774f5b64 100644\n--- a/Documentation/git-svn.txt\n+++ b/Documentation/git-svn.txt\n@@ -769,11 +769,11 @@ option for (hopefully) obvious reasons.\n +\n This option is NOT recommended as it makes it difficult to track down\n old references to SVN revision numbers in existing documentation, bug\n-reports and archives. If you plan to eventually migrate from SVN to Git\n-and are certain about dropping SVN history, consider\n-linkgit:git-filter-branch[1] instead. filter-branch also allows\n-reformatting of metadata for ease-of-reading and rewriting authorship\n-info for non-\"svn.authorsFile\" users.\n+reports, and archives. If you plan to eventually migrate from SVN to\n+Git and are certain about dropping SVN history, consider\n+https://github.com/newren/git-filter-repo[git-filter-repo] instead.\n+filter-repo also allows reformatting of metadata for ease-of-reading\n+and rewriting authorship info for non-\"svn.authorsFile\" users.\n \n svn.useSvmProps::\n svn-remote..useSvmProps::\ndiff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\nindex 82cd573776..5a789c91df 100644\n--- a/Documentation/githooks.txt\n+++ b/Documentation/githooks.txt\n@@ -425,10 +425,12 @@ post-rewrite\n \n This hook is invoked by commands that rewrite commits\n (linkgit:git-commit[1] when called with `--amend` and\n-linkgit:git-rebase[1]; currently `git filter-branch` does 'not' call\n-it!). Its first argument denotes the command it was invoked by:\n-currently one of `amend` or `rebase`. Further command-dependent\n-arguments may be passed in the future.\n+linkgit:git-rebase[1]; however, full-history (re)writing tools like\n+linkgit:git-fast-import[1] or\n+https://github.com/newren/git-filter-repo[git-filter-repo] typically\n+do not call it!). Its first argument denotes the command it was\n+invoked by: currently one of `amend` or `rebase`. Further\n+command-dependent arguments may be passed in the future.\n \n The hook receives a list of the rewritten commits on stdin, in the\n format\ndiff --git a/contrib/svn-fe/svn-fe.txt b/contrib/svn-fe/svn-fe.txt\nindex a3425f4770..19333fc8df 100644\n--- a/contrib/svn-fe/svn-fe.txt\n+++ b/contrib/svn-fe/svn-fe.txt\n@@ -56,7 +56,7 @@ line. This line has the form `git-svn-id: URL@REVNO UUID`.\n \n The resulting repository will generally require further processing\n to put each project in its own repository and to separate the history\n-of each branch. The 'git filter-branch --subdirectory-filter' command\n+of each branch. The 'git filter-repo --subdirectory-filter' command\n may be useful for this purpose.\n \n BUGS\n@@ -67,5 +67,5 @@ The exit status does not reflect whether an error was detected.\n \n SEE ALSO\n --------\n-git-svn(1), svn2git(1), svk(1), git-filter-branch(1), git-fast-import(1),\n+git-svn(1), svn2git(1), svk(1), git-filter-repo(1), git-fast-import(1),\n https://svn.apache.org/repos/asf/subversion/trunk/notes/dump-load-format.txt\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\nindex 5c5afa2b98..f805965d87 100755\n--- a/git-filter-branch.sh\n+++ b/git-filter-branch.sh\n@@ -83,6 +83,19 @@ set_ident () {\n \tfinish_ident COMMITTER\n }\n \n+if [ -z \"$FILTER_BRANCH_SQUELCH_WARNING\" -a \\\n+ -z \"$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS\" ]; then\n+\tcat <] [--subdirectory-filter ] [--env-filter ]\n \t[--tree-filter ] [--index-filter ]\n \t[--parent-filter ] [--msg-filter ]\n-- \n2.23.0.39.gf92d9de5c3\n\n"},{"id":"381755","messageId":"20190903185524.13467-5-newren@gmail.com","threadId":"51710","inReplyTo":"20190903185524.13467-1-newren@gmail.com","subject":"[PATCH v5 4/4] t9902: use a non-deprecated command for testing","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-09-03T18:55:24Z","receivedAt":"2019-09-03T18:55:38Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"t9902 had a list of three random porcelain commands as a sanity check,\none of which was filter-branch. Since we are recommending people not\nuse filter-branch, let's update this test to use rebase instead of\nfilter-branch.\n\nSigned-off-by: Elijah Newren \n---\n t/t9902-completion.sh | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/t/t9902-completion.sh b/t/t9902-completion.sh\nindex 75512c3403..4e7f669c76 100755\n--- a/t/t9902-completion.sh\n+++ b/t/t9902-completion.sh\n@@ -28,10 +28,10 @@ complete ()\n #\n # (2) A test makes sure that common subcommands are included in the\n # completion for \"git \", and a plumbing is excluded. \"add\",\n-# \"filter-branch\" and \"ls-files\" are listed for this.\n+# \"rebase\" and \"ls-files\" are listed for this.\n \n-GIT_TESTING_ALL_COMMAND_LIST='add checkout check-attr filter-branch ls-files'\n-GIT_TESTING_PORCELAIN_COMMAND_LIST='add checkout filter-branch'\n+GIT_TESTING_ALL_COMMAND_LIST='add checkout check-attr rebase ls-files'\n+GIT_TESTING_PORCELAIN_COMMAND_LIST='add checkout rebase'\n \n . \"$GIT_BUILD_DIR/contrib/completion/git-completion.bash\"\n \n@@ -1392,12 +1392,12 @@ test_expect_success 'basic' '\n \t# built-in\n \tgrep -q \"^add \\$\" out &&\n \t# script\n-\tgrep -q \"^filter-branch \\$\" out &&\n+\tgrep -q \"^rebase \\$\" out &&\n \t# plumbing\n \t! grep -q \"^ls-files \\$\" out &&\n \n-\trun_completion \"git f\" &&\n-\t! grep -q -v \"^f\" out\n+\trun_completion \"git r\" &&\n+\t! grep -q -v \"^r\" out\n '\n \n test_expect_success 'double dash \"git\" itself' '\n-- \n2.23.0.39.gf92d9de5c3\n\n"},{"id":"381767","messageId":"xmqq1rwxt7eu.fsf@gitster-ct.c.googlers.com","threadId":"51710","inReplyTo":"20190903185524.13467-2-newren@gmail.com","subject":"Re: [PATCH v5 1/4] t6006: simplify and optimize empty message test","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-09-03T21:08:09Z","receivedAt":"2019-09-03T21:08:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> Test t6006.71 (\"oneline with empty message\") was creating two commits\n> with simple commit messages, and then running filter-branch to rewrite\n> the commit messages to be empty. This test was written this way because\n> the --allow-empty-message option to git commit did not exist at the\n> time. Simplify this test and avoid the need to invoke filter-branch by\n> just using --allow-empty-message when creating the commit.\n\nThe result of filter-branch seems to have one empty line as the body\n(i.e. \"echo X; git cat-file commit A; echo Y\" will show two blank\nlines between the committer line and Y), while \"--allow-empty-message\"\ndoes not leave any body (i.e. the same will give you only one blank\nline there).\n\nWas this test verifying the right thing in the first place, I have\nto wonder.\n\nIOW,\n\n\tgit commit --allow-empty --cleanup=verbatim -m \"$LF\" &&\n\nwould be more faithful conversion of the original (and hopefully\njust as performant).\n\n> Despite only being one piece of the 71st test and there being 73 tests\n> overall, this small change to just this one test speeds up the overall\n> execution time of t6006 (as measured by the best of 3 runs of `time\n> ./t6006-rev-list-format.sh`) by about 11% on Linux, 13% on Mac, and\n> about 15% on Windows.\n\nQuite an improvement ;-)\n\n>\n> Signed-off-by: Elijah Newren \n> ---\n> t/t6006-rev-list-format.sh | 5 ++---\n> 1 file changed, 2 insertions(+), 3 deletions(-)\n>\n> diff --git a/t/t6006-rev-list-format.sh b/t/t6006-rev-list-format.sh\n> index da113d975b..d30e41c9f7 100755\n> --- a/t/t6006-rev-list-format.sh\n> +++ b/t/t6006-rev-list-format.sh\n> @@ -501,9 +501,8 @@ test_expect_success 'reflog identity' '\n> '\n> \n> test_expect_success 'oneline with empty message' '\n> -\tgit commit -m \"dummy\" --allow-empty &&\n> -\tgit commit -m \"dummy\" --allow-empty &&\n> -\tgit filter-branch --msg-filter \"sed -e s/dummy//\" HEAD^^.. &&\n> +\tgit commit --allow-empty --allow-empty-message &&\n> +\tgit commit --allow-empty --allow-empty-message &&\n> \tgit rev-list --oneline HEAD >test.txt &&\n> \ttest_line_count = 5 test.txt &&\n> \tgit rev-list --oneline --graph HEAD >testg.txt &&\n"},{"id":"381769","messageId":"xmqqsgpdrs01.fsf@gitster-ct.c.googlers.com","threadId":"51710","inReplyTo":"20190903185524.13467-3-newren@gmail.com","subject":"Re: [PATCH v5 2/4] t3427: accelerate this test by using fast-export and fast-import","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-09-03T21:26:22Z","receivedAt":"2019-09-03T21:26:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> +extract_files_subtree () {\n> +\tgit fast-export --no-data HEAD -- files_subtree/ |\n> +\t\tsed -e \"s%\\([0-9a-f]\\{40\\} \\)files_subtree/%\\1%\" |\n> +\t\tgit fast-import --force --quiet\n> +}\n\nClever, if a bit filthy ;-). We expect to see something like\n\n\tM 100644 dead...beef files_subtree/bar\n\tM 100755 c0f.....fee files_subtree/foo\n\nin the --no-data output, and the assumption here is that 40-hex\nfollowed by \" files_subtree/\" would never appear anywhere in the\nstream other than these tree dump, so the sed script can rewrite\nthe above to\n\n\tM 100644 dead...beef bar\n\tM 100755 c0f.....fee foo\n\nby getting rid of the leading directory name (plus the slash at the\nend).\n\nThanks.\n"},{"id":"381771","messageId":"xmqqo901rrcs.fsf@gitster-ct.c.googlers.com","threadId":"51710","inReplyTo":"20190903185524.13467-4-newren@gmail.com","subject":"Re: [PATCH v5 3/4] Recommend git-filter-repo instead of git-filter-branch","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-09-03T21:40:19Z","receivedAt":"2019-09-03T21:40:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> diff --git a/git-filter-branch.sh b/git-filter-branch.sh\n> index 5c5afa2b98..f805965d87 100755\n> --- a/git-filter-branch.sh\n> +++ b/git-filter-branch.sh\n> @@ -83,6 +83,19 @@ set_ident () {\n> \tfinish_ident COMMITTER\n> }\n> \n> +if [ -z \"$FILTER_BRANCH_SQUELCH_WARNING\" -a \\\n> + -z \"$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS\" ]; then\n\nThis is probably the only place where [] instead of \"test\" is used\nin our shell scripts.\n\nif test -z \"$FILTER_BRANCH_SQUELCH_WARNING$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS\"\nthen\n ...\n\n> +\tcat < +WARNING: git-filter-branch has a glut of gotchas generating mangled history\n> + rewrites. Please use an alternative filtering tool such as 'git\n> + filter-repo' (https://github.com/newren/git-filter-repo/) instead.\n> + See the filter-branch manual page for more details; to squelch\n> + this warning, set FILTER_BRANCH_SQUELCH_WARNING=1.\n> +\n> +EOF\n> +\tsleep 5\n> +fi\n\nThis should say it is \"sleeping while showing the message and can\nsafely be killed before starting to do any harm\"; alternatively it\nshould lose the \"sleep\". The user would have fear against typing ^C\nto get out of a bulk history rewrite command, and the message itself\nis making the fear worse. If your goal is to discourage its use,\nthen it would be a good idea to make it clear when it is safe to\nkill it before going and studying the alternative. Otherwise, the\nsleep does not help that much---the main complaint is that filter\nbranch is too slow, so the user has plenty of time to read the\nmessage anyway, right? ;-)\n\n> USAGE=\"[--setup ] [--subdirectory-filter ] [--env-filter ]\n> \t[--tree-filter ] [--index-filter ]\n> \t[--parent-filter ] [--msg-filter ]\n"},{"id":"381773","messageId":"CABPp-BHx7=kZJ5rgz5q8xRSbVtbE-UEHMyZxjvEq1tRXW7mCzQ@mail.gmail.com","threadId":"51710","inReplyTo":"xmqq1rwxt7eu.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v5 1/4] t6006: simplify and optimize empty message test","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-09-03T21:58:12Z","receivedAt":"2019-09-03T21:58:26Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Sep 3, 2019 at 2:08 PM Junio C Hamano wrote:\n>\n> Elijah Newren writes:\n>\n> > Test t6006.71 (\"oneline with empty message\") was creating two commits\n> > with simple commit messages, and then running filter-branch to rewrite\n> > the commit messages to be empty. This test was written this way because\n> > the --allow-empty-message option to git commit did not exist at the\n> > time. Simplify this test and avoid the need to invoke filter-branch by\n> > just using --allow-empty-message when creating the commit.\n>\n> The result of filter-branch seems to have one empty line as the body\n> (i.e. \"echo X; git cat-file commit A; echo Y\" will show two blank\n> lines between the committer line and Y), while \"--allow-empty-message\"\n> does not leave any body (i.e. the same will give you only one blank\n> line there).\n\nAh, good catch. I checked out the commit before 1fb5fdd25f0\n(\"rev-list: fix --pretty=oneline with empty message\", 2010-03-21), to\ntry and see the error before that testcase was introduced. I tried it\non a repo with both an actual empty commit message, and one with a\ncommit message consisting solely of a newline. Both styles exhibited\nthe bug that the testcase was introduced to guard against.\n\n> Was this test verifying the right thing in the first place, I have\n> to wonder.\n>\n> IOW,\n>\n> git commit --allow-empty --cleanup=verbatim -m \"$LF\" &&\n>\n> would be more faithful conversion of the original (and hopefully\n> just as performant).\n\nYeah, it'd be a more faithful conversion of the original, though the\noriginal didn't match the testcase description nor the commit message\n(it claimed it was testing with an empty message). Also, in terms of\nfuture proofing, any code changes are more likely to omit a needed\ntrailing LF if the commit message doesn't have one than if it does, so\nI think it's a more robust test with this change.\n\nI can update the commit message to explain this, or, if you prefer, I\ncould duplicate the testcase and tweak the second as you suggest so we\ntest both with and without the LF. What's your preference?\n\n> > Despite only being one piece of the 71st test and there being 73 tests\n> > overall, this small change to just this one test speeds up the overall\n> > execution time of t6006 (as measured by the best of 3 runs of `time\n> > ./t6006-rev-list-format.sh`) by about 11% on Linux, 13% on Mac, and\n> > about 15% on Windows.\n>\n> Quite an improvement ;-)\n>\n> >\n> > Signed-off-by: Elijah Newren \n> > ---\n> > t/t6006-rev-list-format.sh | 5 ++---\n> > 1 file changed, 2 insertions(+), 3 deletions(-)\n> >\n> > diff --git a/t/t6006-rev-list-format.sh b/t/t6006-rev-list-format.sh\n> > index da113d975b..d30e41c9f7 100755\n> > --- a/t/t6006-rev-list-format.sh\n> > +++ b/t/t6006-rev-list-format.sh\n> > @@ -501,9 +501,8 @@ test_expect_success 'reflog identity' '\n> > '\n> >\n> > test_expect_success 'oneline with empty message' '\n> > - git commit -m \"dummy\" --allow-empty &&\n> > - git commit -m \"dummy\" --allow-empty &&\n> > - git filter-branch --msg-filter \"sed -e s/dummy//\" HEAD^^.. &&\n> > + git commit --allow-empty --allow-empty-message &&\n> > + git commit --allow-empty --allow-empty-message &&\n> > git rev-list --oneline HEAD >test.txt &&\n> > test_line_count = 5 test.txt &&\n> > git rev-list --oneline --graph HEAD >testg.txt &&\n"},{"id":"381775","messageId":"xmqqk1aprpa9.fsf@gitster-ct.c.googlers.com","threadId":"51710","inReplyTo":"CABPp-BHx7=kZJ5rgz5q8xRSbVtbE-UEHMyZxjvEq1tRXW7mCzQ@mail.gmail.com","subject":"Re: [PATCH v5 1/4] t6006: simplify and optimize empty message test","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-09-03T22:25:02Z","receivedAt":"2019-09-03T22:25:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> Ah, good catch. I checked out the commit before 1fb5fdd25f0\n> (\"rev-list: fix --pretty=oneline with empty message\", 2010-03-21), to\n> try and see the error before that testcase was introduced. I tried it\n> on a repo with both an actual empty commit message, and one with a\n> commit message consisting solely of a newline. Both styles exhibited\n> the bug that the testcase was introduced to guard against.\n\nThat's a good thing to know to decide what is a reasonable\nthing to do here.\n\nAs we are creating two commits, perhaps adding one with and another\nwithout the extra blank line may give us more diversity, and\nexplaining why we are adding two slightly different one\n(i.e. because the original bug was there for both shapes of commits)\nwould help us not wasting the time we already spent discussing this\nchange ;-)\n\nOf course, we can alternatively just keep the patch as-is and update\nthe explanation as to why we are testing with commits different from\nthe original when we are supposed to be making this change for\nperformance reasons (i.e. the symptom manifests either way, so why\nnot using the form that is easier to create?).\n\nThanks for working on this ;-)\n"},{"id":"381777","messageId":"xmqqftldroae.fsf@gitster-ct.c.googlers.com","threadId":"51710","inReplyTo":"xmqqsgpdrs01.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v5 2/4] t3427: accelerate this test by using fast-export and fast-import","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-09-03T22:46:33Z","receivedAt":"2019-09-03T22:46:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Elijah Newren writes:\n>\n>> +extract_files_subtree () {\n>> +\tgit fast-export --no-data HEAD -- files_subtree/ |\n>> +\t\tsed -e \"s%\\([0-9a-f]\\{40\\} \\)files_subtree/%\\1%\" |\n>> +\t\tgit fast-import --force --quiet\n>> +}\n\nThis change has obvious interactions with Dscho's d51b771d (\"t3427:\nmove the `filter-branch` invocation into the `setup` case\",\n2019-07-31) that is still in flight, but in a good way. There only\nneeds a single callsite for the above helper function after that\nstep.\n\nI think I'll discard this step from the \"move us closer to deprecate\nfilter-branch\" topic, and ask you and Dscho to work together to have\nit or its moral equivalent included as part of js/rebase-r-strategy\ntopic.\n\nThanks.\n"},{"id":"381825","messageId":"CABPp-BEEb1gZbChjbNQt_anUL65Le3Zf=seM9m3yEN1sEBf=BA@mail.gmail.com","threadId":"51710","inReplyTo":"xmqqo901rrcs.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v5 3/4] Recommend git-filter-repo instead of git-filter-branch","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-09-04T20:30:23Z","receivedAt":"2019-09-04T20:30:37Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Sep 3, 2019 at 2:40 PM Junio C Hamano wrote:\n>\n> Elijah Newren writes:\n>\n> > diff --git a/git-filter-branch.sh b/git-filter-branch.sh\n> > index 5c5afa2b98..f805965d87 100755\n> > --- a/git-filter-branch.sh\n> > +++ b/git-filter-branch.sh\n> > @@ -83,6 +83,19 @@ set_ident () {\n> > finish_ident COMMITTER\n> > }\n> >\n> > +if [ -z \"$FILTER_BRANCH_SQUELCH_WARNING\" -a \\\n> > + -z \"$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS\" ]; then\n>\n> This is probably the only place where [] instead of \"test\" is used\n> in our shell scripts.\n>\n> if test -z \"$FILTER_BRANCH_SQUELCH_WARNING$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS\"\n> then\n> ...\n\nYeah, git-filter-branch.sh has approximately twice as many uses of []\nthan \"test\", so it seemed in line with its coding style. I can switch\nit over.\n\n> > + cat < > +WARNING: git-filter-branch has a glut of gotchas generating mangled history\n> > + rewrites. Please use an alternative filtering tool such as 'git\n> > + filter-repo' (https://github.com/newren/git-filter-repo/) instead.\n> > + See the filter-branch manual page for more details; to squelch\n> > + this warning, set FILTER_BRANCH_SQUELCH_WARNING=1.\n> > +\n> > +EOF\n> > + sleep 5\n> > +fi\n>\n> This should say it is \"sleeping while showing the message and can\n> safely be killed before starting to do any harm\"; alternatively it\n> should lose the \"sleep\". The user would have fear against typing ^C\n> to get out of a bulk history rewrite command, and the message itself\n> is making the fear worse. If your goal is to discourage its use,\n> then it would be a good idea to make it clear when it is safe to\n> kill it before going and studying the alternative. Otherwise, the\n> sleep does not help that much---the main complaint is that filter\n> branch is too slow, so the user has plenty of time to read the\n> message anyway, right? ;-)\n\nMakes sense; will fix.\n"},{"id":"381827","messageId":"CABPp-BFi9puCJ8H08ZD_VSM2pygt3GoL52QNuaVLkN_130=n0w@mail.gmail.com","threadId":"51710","inReplyTo":"xmqqftldroae.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v5 2/4] t3427: accelerate this test by using fast-export and fast-import","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-09-04T20:32:47Z","receivedAt":"2019-09-04T20:33:00Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Sep 3, 2019 at 3:46 PM Junio C Hamano wrote:\n>\n> Junio C Hamano writes:\n>\n> > Elijah Newren writes:\n> >\n> >> +extract_files_subtree () {\n> >> + git fast-export --no-data HEAD -- files_subtree/ |\n> >> + sed -e \"s%\\([0-9a-f]\\{40\\} \\)files_subtree/%\\1%\" |\n> >> + git fast-import --force --quiet\n> >> +}\n>\n> This change has obvious interactions with Dscho's d51b771d (\"t3427:\n> move the `filter-branch` invocation into the `setup` case\",\n> 2019-07-31) that is still in flight, but in a good way. There only\n> needs a single callsite for the above helper function after that\n> step.\n>\n> I think I'll discard this step from the \"move us closer to deprecate\n> filter-branch\" topic, and ask you and Dscho to work together to have\n> it or its moral equivalent included as part of js/rebase-r-strategy\n> topic.\n\nSounds good. I'll resubmit it separately as a patch on top of his topic.\n"},{"id":"381842","messageId":"20190904223239.571-1-newren@gmail.com","threadId":"51710","inReplyTo":"20190903185524.13467-1-newren@gmail.com","subject":"[PATCH v6 0/3] Warn about git-filter-branch usage and avoid it","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-09-04T22:32:36Z","receivedAt":"2019-09-04T22:32:49Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Changes since v5 (full range-diff below):\n * Dropped patch 3 (which was rebased on top of js/rebase-r-strategy and\n submitted separately)[1]\n * Updated t6006 to include both an empty commit message and a commit\n message with just a line feed\n * Made the two small tweaks Junio suggested to git-filter-branch.sh\n\n[1] https://public-inbox.org/git/20190904214048.29331-1-newren@gmail.com/\n\nElijah Newren (3):\n t6006: simplify, fix, and optimize empty message test\n Recommend git-filter-repo instead of git-filter-branch\n t9902: use a non-deprecated command for testing\n\n Documentation/git-fast-export.txt | 6 +-\n Documentation/git-filter-branch.txt | 273 +++++++++++++++++++++++++---\n Documentation/git-gc.txt | 17 +-\n Documentation/git-rebase.txt | 3 +-\n Documentation/git-replace.txt | 10 +-\n Documentation/git-svn.txt | 10 +-\n Documentation/githooks.txt | 10 +-\n contrib/svn-fe/svn-fe.txt | 4 +-\n git-filter-branch.sh | 14 ++\n t/t6006-rev-list-format.sh | 5 +-\n t/t9902-completion.sh | 12 +-\n 11 files changed, 296 insertions(+), 68 deletions(-)\n\nRange-diff:\n1: ccea0e5846 ! 1: d5370568a4 t6006: simplify and optimize empty message test\n @@ Metadata\n Author: Elijah Newren \n \n ## Commit message ##\n - t6006: simplify and optimize empty message test\n + t6006: simplify, fix, and optimize empty message test\n \n Test t6006.71 (\"oneline with empty message\") was creating two commits\n with simple commit messages, and then running filter-branch to rewrite\n - the commit messages to be empty. This test was written this way because\n - the --allow-empty-message option to git commit did not exist at the\n - time. Simplify this test and avoid the need to invoke filter-branch by\n - just using --allow-empty-message when creating the commit.\n + the commit messages to be \"empty\". This test was introduced in commit\n + 1fb5fdd25f0 (\"rev-list: fix --pretty=oneline with empty message\",\n + 2010-03-21) and written this way because the --allow-empty-message\n + option to git commit did not exist at the time.\n +\n + However, the filter-branch invocation used differed slightly from\n + --allow-empty-message in that it would have a commit message consisting\n + solely of a single newline, and as such was not testing what the\n + original commit intended to test. Since both a truly empty commit\n + message and a commit message with a single linefeed could trigger the\n + original bug, modify the test slightly to include an example of each.\n \n Despite only being one piece of the 71st test and there being 73 tests\n overall, this small change to just this one test speeds up the overall\n @@ t/t6006-rev-list-format.sh: test_expect_success 'reflog identity' '\n -\tgit commit -m \"dummy\" --allow-empty &&\n -\tgit commit -m \"dummy\" --allow-empty &&\n -\tgit filter-branch --msg-filter \"sed -e s/dummy//\" HEAD^^.. &&\n -+\tgit commit --allow-empty --allow-empty-message &&\n ++\tgit commit --allow-empty --cleanup=verbatim -m \"$LF\" &&\n +\tgit commit --allow-empty --allow-empty-message &&\n \tgit rev-list --oneline HEAD >test.txt &&\n \ttest_line_count = 5 test.txt &&\n2: 6d73135006 < -: ---------- t3427: accelerate this test by using fast-export and fast-import\n3: 2f225c8697 ! 2: 8635410b88 Recommend git-filter-repo instead of git-filter-branch\n @@ git-filter-branch.sh: set_ident () {\n \tfinish_ident COMMITTER\n }\n \n -+if [ -z \"$FILTER_BRANCH_SQUELCH_WARNING\" -a \\\n -+ -z \"$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS\" ]; then\n ++if test -z \"$FILTER_BRANCH_SQUELCH_WARNING$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS\"\n ++then\n +\tcat <] [--subdirectory-filter ] [--env-filter ]\n4: 048eba375b = 3: 19edb94ec2 t9902: use a non-deprecated command for testing\n-- \n2.23.0.3.g19edb94ec2\n\n"},{"id":"381843","messageId":"20190904223239.571-2-newren@gmail.com","threadId":"51710","inReplyTo":"20190904223239.571-1-newren@gmail.com","subject":"[PATCH v6 1/3] t6006: simplify, fix, and optimize empty message test","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-09-04T22:32:37Z","receivedAt":"2019-09-04T22:32:50Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Test t6006.71 (\"oneline with empty message\") was creating two commits\nwith simple commit messages, and then running filter-branch to rewrite\nthe commit messages to be \"empty\". This test was introduced in commit\n1fb5fdd25f0 (\"rev-list: fix --pretty=oneline with empty message\",\n2010-03-21) and written this way because the --allow-empty-message\noption to git commit did not exist at the time.\n\nHowever, the filter-branch invocation used differed slightly from\n--allow-empty-message in that it would have a commit message consisting\nsolely of a single newline, and as such was not testing what the\noriginal commit intended to test. Since both a truly empty commit\nmessage and a commit message with a single linefeed could trigger the\noriginal bug, modify the test slightly to include an example of each.\n\nDespite only being one piece of the 71st test and there being 73 tests\noverall, this small change to just this one test speeds up the overall\nexecution time of t6006 (as measured by the best of 3 runs of `time\n./t6006-rev-list-format.sh`) by about 11% on Linux, 13% on Mac, and\nabout 15% on Windows.\n\nSigned-off-by: Elijah Newren \n---\n t/t6006-rev-list-format.sh | 5 ++---\n 1 file changed, 2 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t6006-rev-list-format.sh b/t/t6006-rev-list-format.sh\nindex da113d975b..cfb74d0e03 100755\n--- a/t/t6006-rev-list-format.sh\n+++ b/t/t6006-rev-list-format.sh\n@@ -501,9 +501,8 @@ test_expect_success 'reflog identity' '\n '\n \n test_expect_success 'oneline with empty message' '\n-\tgit commit -m \"dummy\" --allow-empty &&\n-\tgit commit -m \"dummy\" --allow-empty &&\n-\tgit filter-branch --msg-filter \"sed -e s/dummy//\" HEAD^^.. &&\n+\tgit commit --allow-empty --cleanup=verbatim -m \"$LF\" &&\n+\tgit commit --allow-empty --allow-empty-message &&\n \tgit rev-list --oneline HEAD >test.txt &&\n \ttest_line_count = 5 test.txt &&\n \tgit rev-list --oneline --graph HEAD >testg.txt &&\n-- \n2.23.0.3.g19edb94ec2\n\n"},{"id":"381844","messageId":"20190904223239.571-3-newren@gmail.com","threadId":"51710","inReplyTo":"20190904223239.571-1-newren@gmail.com","subject":"[PATCH v6 2/3] Recommend git-filter-repo instead of git-filter-branch","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-09-04T22:32:38Z","receivedAt":"2019-09-04T22:32:52Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"filter-branch suffers from a deluge of disguised dangers that disfigure\nhistory rewrites (i.e. deviate from the deliberate changes). Many of\nthese problems are unobtrusive and can easily go undiscovered until the\nnew repository is in use. This can result in problems ranging from an\neven messier history than what led folks to filter-branch in the first\nplace, to data loss or corruption. These issues cannot be backward\ncompatibly fixed, so add a warning to both filter-branch and its manpage\nrecommending that another tool (such as filter-repo) be used instead.\n\nAlso, update other manpages that referenced filter-branch. Several of\nthese needed updates even if we could continue recommending\nfilter-branch, either due to implying that something was unique to\nfilter-branch when it applied more generally to all history rewriting\ntools (e.g. BFG, reposurgeon, fast-import, filter-repo), or because\nsomething about filter-branch was used as an example despite other more\ncommonly known examples now existing. Reword these sections to fix\nthese issues and to avoid recommending filter-branch.\n\nFinally, remove the section explaining BFG Repo Cleaner as an\nalternative to filter-branch. I feel somewhat bad about this,\nespecially since I feel like I learned so much from BFG that I put to\ngood use in filter-repo (which is much more than I can say for\nfilter-branch), but keeping that section presented a few problems:\n * In order to recommend that people quit using filter-branch, we need\n to provide them a recomendation for something else to use that\n can handle all the same types of rewrites. To my knowledge,\n filter-repo is the only such tool. So it needs to be mentioned.\n * I don't want to give conflicting recommendations to users\n * If we recommend two tools, we shouldn't expect users to learn both\n and pick which one to use; we should explain which problems one\n can solve that the other can't or when one is much faster than\n the other.\n * BFG and filter-repo have similar performance\n * All filtering types that BFG can do, filter-repo can also do. In\n fact, filter-repo comes with a reimplementation of BFG named\n bfg-ish which provides the same user-interface as BFG but with\n several bugfixes and new features that are hard to implement in\n BFG due to its technical underpinnings.\nWhile I could still mention both tools, it seems like I would need to\nprovide some kind of comparison and I would ultimately just say that\nfilter-repo can do everything BFG can, so ultimately it seems that it\nis just better to remove that section altogether.\n\nSigned-off-by: Elijah Newren \n---\n Documentation/git-fast-export.txt | 6 +-\n Documentation/git-filter-branch.txt | 273 +++++++++++++++++++++++++---\n Documentation/git-gc.txt | 17 +-\n Documentation/git-rebase.txt | 3 +-\n Documentation/git-replace.txt | 10 +-\n Documentation/git-svn.txt | 10 +-\n Documentation/githooks.txt | 10 +-\n contrib/svn-fe/svn-fe.txt | 4 +-\n git-filter-branch.sh | 14 ++\n 9 files changed, 288 insertions(+), 59 deletions(-)\n\ndiff --git a/Documentation/git-fast-export.txt b/Documentation/git-fast-export.txt\nindex cc940eb9ad..784e934009 100644\n--- a/Documentation/git-fast-export.txt\n+++ b/Documentation/git-fast-export.txt\n@@ -17,9 +17,9 @@ This program dumps the given revisions in a form suitable to be piped\n into 'git fast-import'.\n \n You can use it as a human-readable bundle replacement (see\n-linkgit:git-bundle[1]), or as a kind of an interactive\n-'git filter-branch'.\n-\n+linkgit:git-bundle[1]), or as a format that can be edited before being\n+fed to 'git fast-import' in order to do history rewrites (an ability\n+relied on by tools like 'git filter-repo').\n \n OPTIONS\n -------\ndiff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt\nindex 6b53dd7e06..5876598852 100644\n--- a/Documentation/git-filter-branch.txt\n+++ b/Documentation/git-filter-branch.txt\n@@ -16,6 +16,19 @@ SYNOPSIS\n \t[--original ] [-d ] [-f | --force]\n \t[--state-branch ] [--] [...]\n \n+WARNING\n+-------\n+'git filter-branch' has a plethora of pitfalls that can produce non-obvious\n+manglings of the intended history rewrite (and can leave you with little\n+time to investigate such problems since it has such abysmal performance).\n+These safety and performance issues cannot be backward compatibly fixed and\n+as such, its use is not recommended. Please use an alternative history\n+filtering tool such as https://github.com/newren/git-filter-repo/[git\n+filter-repo]. If you still need to use 'git filter-branch', please\n+carefully read <> (and <>) to learn about the land\n+mines of filter-branch, and then vigilantly avoid as many of the hazards\n+listed there as reasonably possible.\n+\n DESCRIPTION\n -----------\n Lets you rewrite Git revision history by rewriting the branches mentioned\n@@ -445,36 +458,236 @@ warned.\n (or if your git-gc is not new enough to support arguments to\n `--prune`, use `git repack -ad; git prune` instead).\n \n-NOTES\n------\n-\n-git-filter-branch allows you to make complex shell-scripted rewrites\n-of your Git history, but you probably don't need this flexibility if\n-you're simply _removing unwanted data_ like large files or passwords.\n-For those operations you may want to consider\n-http://rtyley.github.io/bfg-repo-cleaner/[The BFG Repo-Cleaner],\n-a JVM-based alternative to git-filter-branch, typically at least\n-10-50x faster for those use-cases, and with quite different\n-characteristics:\n-\n-* Any particular version of a file is cleaned exactly _once_. The BFG,\n- unlike git-filter-branch, does not give you the opportunity to\n- handle a file differently based on where or when it was committed\n- within your history. This constraint gives the core performance\n- benefit of The BFG, and is well-suited to the task of cleansing bad\n- data - you don't care _where_ the bad data is, you just want it\n- _gone_.\n-\n-* By default The BFG takes full advantage of multi-core machines,\n- cleansing commit file-trees in parallel. git-filter-branch cleans\n- commits sequentially (i.e. in a single-threaded manner), though it\n- _is_ possible to write filters that include their own parallelism,\n- in the scripts executed against each commit.\n-\n-* The http://rtyley.github.io/bfg-repo-cleaner/#examples[command options]\n- are much more restrictive than git-filter branch, and dedicated just\n- to the tasks of removing unwanted data- e.g:\n- `--strip-blobs-bigger-than 1M`.\n+[[PERFORMANCE]]\n+PERFORMANCE\n+-----------\n+\n+The performance of git-filter-branch is glacially slow; its design makes it\n+impossible for a backward-compatible implementation to ever be fast:\n+\n+* In editing files, git-filter-branch by design checks out each and\n+every commit as it existed in the original repo. If your repo has 10\\^5\n+files and 10\\^5 commits, but each commit only modifies 5 files, then\n+git-filter-branch will make you do 10\\^10 modifications, despite only\n+having (at most) 5*10^5 unique blobs.\n+\n+* If you try and cheat and try to make git-filter-branch only work on\n+files modified in a commit, then two things happen\n+\n+ ** you run into problems with deletions whenever the user is simply\n+ trying to rename files (because attempting to delete files that\n+ don't exist looks like a no-op; it takes some chicanery to remap\n+ deletes across file renames when the renames happen via arbitrary\n+ user-provided shell)\n+\n+ ** even if you succeed at the map-deletes-for-renames chicanery, you\n+ still technically violate backward compatibility because users are\n+ allowed to filter files in ways that depend upon topology of\n+ commits instead of filtering solely based on file contents or names\n+ (though this has not been observed in the wild).\n+\n+* Even if you don't need to edit files but only want to e.g. rename or\n+remove some and thus can avoid checking out each file (i.e. you can use\n+--index-filter), you still are passing shell snippets for your filters.\n+This means that for every commit, you have to have a prepared git repo\n+where those filters can be run. That's a significant setup.\n+\n+* Further, several additional files are created or updated per commit by\n+git-filter-branch. Some of these are for supporting the convenience\n+functions provided by git-filter-branch (such as map()), while others\n+are for keeping track of internal state (but could have also been\n+accessed by user filters; one of git-filter-branch's regression tests\n+does so). This essentially amounts to using the filesystem as an IPC\n+mechanism between git-filter-branch and the user-provided filters.\n+Disks tend to be a slow IPC mechanism, and writing these files also\n+effectively represents a forced synchronization point between separate\n+processes that we hit with every commit.\n+\n+* The user-provided shell commands will likely involve a pipeline of\n+commands, resulting in the creation of many processes per commit.\n+Creating and running another process takes a widely varying amount of\n+time between operating systems, but on any platform it is very slow\n+relative to invoking a function.\n+\n+* git-filter-branch itself is written in shell, which is kind of slow.\n+This is the one performance issue that could be backward-compatibly\n+fixed, but compared to the above problems that are intrinsic to the\n+design of git-filter-branch, the language of the tool itself is a\n+relatively minor issue.\n+\n+ ** Side note: Unfortunately, people tend to fixate on the\n+ written-in-shell aspect and periodically ask if git-filter-branch\n+ could be rewritten in another language to fix the performance\n+ issues. Not only does that ignore the bigger intrinsic problems\n+ with the design, it'd help less than you'd expect: if\n+ git-filter-branch itself were not shell, then the convenience\n+ functions (map(), skip_commit(), etc) and the `--setup` argument\n+ could no longer be executed once at the beginning of the program\n+ but would instead need to be prepended to every user filter (and\n+ thus re-executed with every commit).\n+\n+The https://github.com/newren/git-filter-repo/[git filter-repo] tool is\n+an alternative to git-filter-branch which does not suffer from these\n+performance problems or the safety problems (mentioned below). For those\n+with existing tooling which relies upon git-filter-branch, 'git\n+repo-filter' also provides\n+https://github.com/newren/git-filter-repo/blob/master/contrib/filter-repo-demos/filter-lamely[filter-lamely],\n+a drop-in git-filter-branch replacement (with a few caveats). While\n+filter-lamely suffers from all the same safety issues as\n+git-filter-branch, it at least ameloriates the performance issues a\n+little.\n+\n+[[SAFETY]]\n+SAFETY\n+------\n+\n+git-filter-branch is riddled with gotchas resulting in various ways to\n+easily corrupt repos or end up with a mess worse than what you started\n+with:\n+\n+* Someone can have a set of \"working and tested filters\" which they\n+document or provide to a coworker, who then runs them on a different OS\n+where the same commands are not working/tested (some examples in the\n+git-filter-branch manpage are also affected by this). BSD vs. GNU\n+userland differences can really bite. If lucky, error messages are\n+spewed. But just as likely, the commands either don't do the filtering\n+requested, or silently corrupt by making some unwanted change. The\n+unwanted change may only affect a few commits, so it's not necessarily\n+obvious either. (The fact that problems won't necessarily be obvious\n+means they are likely to go unnoticed until the rewritten history is in\n+use for quite a while, at which point it's really hard to justify\n+another flag-day for another rewrite.)\n+\n+* Filenames with spaces are often mishandled by shell snippets since\n+they cause problems for shell pipelines. Not everyone is familiar with\n+find -print0, xargs -0, git-ls-files -z, etc. Even people who are\n+familiar with these may assume such flags are not relevant because\n+someone else renamed any such files in their repo back before the person\n+doing the filtering joined the project. And often, even those familiar\n+with handling arguments with spaces may not do so just because they\n+aren't in the mindset of thinking about everything that could possibly\n+go wrong.\n+\n+* Non-ascii filenames can be silently removed despite being in a desired\n+directory. Keeping only wanted paths is often done using pipelines like\n+`git ls-files | grep -v ^WANTED_DIR/ | xargs git rm`. ls-files will\n+only quote filenames if needed, so folks may not notice that one of the\n+files didn't match the regex (at least not until it's much too late).\n+Yes, someone who knows about core.quotePath can avoid this (unless they\n+have other special characters like \\t, \\n, or \"), and people who use\n+ls-files -z with something other than grep can avoid this, but that\n+doesn't mean they will.\n+\n+* Similarly, when moving files around, one can find that filenames with\n+non-ascii or special characters end up in a different directory, one\n+that includes a double quote character. (This is technically the same\n+issue as above with quoting, but perhaps an interesting different way\n+that it can and has manifested as a problem.)\n+\n+* It's far too easy to accidentally mix up old and new history. It's\n+still possible with any tool, but git-filter-branch almost invites it.\n+If lucky, the only downside is users getting frustrated that they don't\n+know how to shrink their repo and remove the old stuff. If unlucky,\n+they merge old and new history and end up with multiple \"copies\" of each\n+commit, some of which have unwanted or sensitive files and others which\n+don't. This comes about in multiple different ways:\n+\n+ ** the default to only doing a partial history rewrite ('--all' is not\n+ the default and few examples show it)\n+\n+ ** the fact that there's no automatic post-run cleanup\n+\n+ ** the fact that --tag-name-filter (when used to rename tags) doesn't\n+ remove the old tags but just adds new ones with the new name\n+\n+ ** the fact that little educational information is provided to inform\n+ users of the ramifications of a rewrite and how to avoid mixing old\n+ and new history. For example, this man page discusses how users\n+ need to understand that they need to rebase their changes for all\n+ their branches on top of new history (or delete and reclone), but\n+ that's only one of multiple concerns to consider. See the\n+ \"DISCUSSION\" section of the git filter-repo manual page for more\n+ details.\n+\n+* Annotated tags can be accidentally converted to lightweight tags, due\n+to either of two issues:\n+\n+ ** Someone can do a history rewrite, realize they messed up, restore\n+ from the backups in refs/original/, and then redo their\n+ git-filter-branch command. (The backup in refs/original/ is not a\n+ real backup; it dereferences tags first.)\n+\n+ ** Running git-filter-branch with either --tags or --all in your\n+ . In order to retain annotated tags as\n+ annotated, you must use --tag-name-filter (and must not have\n+ restored from refs/original/ in a previously botched rewrite).\n+\n+* Any commit messages that specify an encoding will become corrupted\n+by the rewrite; git-filter-branch ignores the encoding, takes the original\n+bytes, and feeds it to commit-tree without telling it the proper\n+encoding. (This happens whether or not --msg-filter is used.)\n+\n+* Commit messages (even if they are all UTF-8) by default become\n+corrupted due to not being updated -- any references to other commit\n+hashes in commit messages will now refer to no-longer-extant commits.\n+\n+* There are no facilities for helping users find what unwanted crud they\n+should delete, which means they are much more likely to have incomplete\n+or partial cleanups that sometimes result in confusion and people\n+wasting time trying to understand. (For example, folks tend to just\n+look for big files to delete instead of big directories or extensions,\n+and once they do so, then sometime later folks using the new repository\n+who are going through history will notice a build artifact directory\n+that has some files but not others, or a cache of dependencies\n+(node_modules or similar) which couldn't have ever been functional since\n+it's missing some files.)\n+\n+* If --prune-empty isn't specified, then the filtering process can\n+create hoards of confusing empty commits\n+\n+* If --prune-empty is specified, then intentionally placed empty\n+commits from before the filtering operation are also pruned instead of\n+just pruning commits that became empty due to filtering rules.\n+\n+* If --prune empty is specified, sometimes empty commits are missed\n+and left around anyway (a somewhat rare bug, but it happens...)\n+\n+* A minor issue, but users who have a goal to update all names and\n+emails in a repository may be led to --env-filter which will only update\n+authors and committers, missing taggers.\n+\n+* If the user provides a --tag-name-filter that maps multiple tags to\n+the same name, no warning or error is provided; git-filter-branch simply\n+overwrites each tag in some undocumented pre-defined order resulting in\n+only one tag at the end. (A git-filter-branch regression test requires\n+this surprising behavior.)\n+\n+Also, the poor performance of git-filter-branch often leads to safety\n+issues:\n+\n+* Coming up with the correct shell snippet to do the filtering you want\n+is sometimes difficult unless you're just doing a trivial modification\n+such as deleting a couple files. Unfortunately, people often learn if\n+the snippet is right or wrong by trying it out, but the rightness or\n+wrongness can vary depending on special circumstances (spaces in\n+filenames, non-ascii filenames, funny author names or emails, invalid\n+timezones, presence of grafts or replace objects, etc.), meaning they\n+may have to wait a long time, hit an error, then restart. The\n+performance of git-filter-branch is so bad that this cycle is painful,\n+reducing the time available to carefully re-check (to say nothing about\n+what it does to the patience of the person doing the rewrite even if\n+they do technically have more time available). This problem is extra\n+compounded because errors from broken filters may not be shown for a\n+long time and/or get lost in a sea of output. Even worse, broken\n+filters often just result in silent incorrect rewrites.\n+\n+* To top it all off, even when users finally find working commands, they\n+naturally want to share them. But they may be unaware that their repo\n+didn't have some special cases that someone else's does. So, when\n+someone else with a different repository runs the same commands, they\n+get hit by the problems above. Or, the user just runs commands that\n+really were vetted for special cases, but they run it on a different OS\n+where it doesn't work, as noted above.\n \n GIT\n ---\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 247f765604..0c114ad1ca 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -115,15 +115,14 @@ NOTES\n -----\n \n 'git gc' tries very hard not to delete objects that are referenced\n-anywhere in your repository. In\n-particular, it will keep not only objects referenced by your current set\n-of branches and tags, but also objects referenced by the index,\n-remote-tracking branches, refs saved by 'git filter-branch' in\n-refs/original/, reflogs (which may reference commits in branches\n-that were later amended or rewound), and anything else in the refs/* namespace.\n-If you are expecting some objects to be deleted and they aren't, check\n-all of those locations and decide whether it makes sense in your case to\n-remove those references.\n+anywhere in your repository. In particular, it will keep not only\n+objects referenced by your current set of branches and tags, but also\n+objects referenced by the index, remote-tracking branches, notes saved\n+by 'git notes' under refs/notes/, reflogs (which may reference commits\n+in branches that were later amended or rewound), and anything else in\n+the refs/* namespace. If you are expecting some objects to be deleted\n+and they aren't, check all of those locations and decide whether it\n+makes sense in your case to remove those references.\n \n On the other hand, when 'git gc' runs concurrently with another process,\n there is a risk of it deleting an object that the other process is using\ndiff --git a/Documentation/git-rebase.txt b/Documentation/git-rebase.txt\nindex 6156609cf7..a8cfc0ad82 100644\n--- a/Documentation/git-rebase.txt\n+++ b/Documentation/git-rebase.txt\n@@ -832,7 +832,8 @@ Hard case: The changes are not the same.::\n \tThis happens if the 'subsystem' rebase had conflicts, or used\n \t`--interactive` to omit, edit, squash, or fixup commits; or\n \tif the upstream used one of `commit --amend`, `reset`, or\n-\t`filter-branch`.\n+\ta full history rewriting command like\n+\thttps://github.com/newren/git-filter-repo[`filter-repo`].\n \n \n The easy case\ndiff --git a/Documentation/git-replace.txt b/Documentation/git-replace.txt\nindex 246dc9943c..f271d758c3 100644\n--- a/Documentation/git-replace.txt\n+++ b/Documentation/git-replace.txt\n@@ -123,10 +123,10 @@ The following format are available:\n CREATING REPLACEMENT OBJECTS\n ----------------------------\n \n-linkgit:git-filter-branch[1], linkgit:git-hash-object[1] and\n-linkgit:git-rebase[1], among other git commands, can be used to create\n-replacement objects from existing objects. The `--edit` option can\n-also be used with 'git replace' to create a replacement object by\n+linkgit:git-hash-object[1], linkgit:git-rebase[1], and\n+https://github.com/newren/git-filter-repo[git-filter-repo], among other git commands, can be used to\n+create replacement objects from existing objects. The `--edit` option\n+can also be used with 'git replace' to create a replacement object by\n editing an existing object.\n \n If you want to replace many blobs, trees or commits that are part of a\n@@ -148,13 +148,13 @@ pending objects.\n SEE ALSO\n --------\n linkgit:git-hash-object[1]\n-linkgit:git-filter-branch[1]\n linkgit:git-rebase[1]\n linkgit:git-tag[1]\n linkgit:git-branch[1]\n linkgit:git-commit[1]\n linkgit:git-var[1]\n linkgit:git[1]\n+https://github.com/newren/git-filter-repo[git-filter-repo]\n \n GIT\n ---\ndiff --git a/Documentation/git-svn.txt b/Documentation/git-svn.txt\nindex 30711625fd..53774f5b64 100644\n--- a/Documentation/git-svn.txt\n+++ b/Documentation/git-svn.txt\n@@ -769,11 +769,11 @@ option for (hopefully) obvious reasons.\n +\n This option is NOT recommended as it makes it difficult to track down\n old references to SVN revision numbers in existing documentation, bug\n-reports and archives. If you plan to eventually migrate from SVN to Git\n-and are certain about dropping SVN history, consider\n-linkgit:git-filter-branch[1] instead. filter-branch also allows\n-reformatting of metadata for ease-of-reading and rewriting authorship\n-info for non-\"svn.authorsFile\" users.\n+reports, and archives. If you plan to eventually migrate from SVN to\n+Git and are certain about dropping SVN history, consider\n+https://github.com/newren/git-filter-repo[git-filter-repo] instead.\n+filter-repo also allows reformatting of metadata for ease-of-reading\n+and rewriting authorship info for non-\"svn.authorsFile\" users.\n \n svn.useSvmProps::\n svn-remote..useSvmProps::\ndiff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\nindex 82cd573776..5a789c91df 100644\n--- a/Documentation/githooks.txt\n+++ b/Documentation/githooks.txt\n@@ -425,10 +425,12 @@ post-rewrite\n \n This hook is invoked by commands that rewrite commits\n (linkgit:git-commit[1] when called with `--amend` and\n-linkgit:git-rebase[1]; currently `git filter-branch` does 'not' call\n-it!). Its first argument denotes the command it was invoked by:\n-currently one of `amend` or `rebase`. Further command-dependent\n-arguments may be passed in the future.\n+linkgit:git-rebase[1]; however, full-history (re)writing tools like\n+linkgit:git-fast-import[1] or\n+https://github.com/newren/git-filter-repo[git-filter-repo] typically\n+do not call it!). Its first argument denotes the command it was\n+invoked by: currently one of `amend` or `rebase`. Further\n+command-dependent arguments may be passed in the future.\n \n The hook receives a list of the rewritten commits on stdin, in the\n format\ndiff --git a/contrib/svn-fe/svn-fe.txt b/contrib/svn-fe/svn-fe.txt\nindex a3425f4770..19333fc8df 100644\n--- a/contrib/svn-fe/svn-fe.txt\n+++ b/contrib/svn-fe/svn-fe.txt\n@@ -56,7 +56,7 @@ line. This line has the form `git-svn-id: URL@REVNO UUID`.\n \n The resulting repository will generally require further processing\n to put each project in its own repository and to separate the history\n-of each branch. The 'git filter-branch --subdirectory-filter' command\n+of each branch. The 'git filter-repo --subdirectory-filter' command\n may be useful for this purpose.\n \n BUGS\n@@ -67,5 +67,5 @@ The exit status does not reflect whether an error was detected.\n \n SEE ALSO\n --------\n-git-svn(1), svn2git(1), svk(1), git-filter-branch(1), git-fast-import(1),\n+git-svn(1), svn2git(1), svk(1), git-filter-repo(1), git-fast-import(1),\n https://svn.apache.org/repos/asf/subversion/trunk/notes/dump-load-format.txt\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\nindex 5c5afa2b98..fea7964617 100755\n--- a/git-filter-branch.sh\n+++ b/git-filter-branch.sh\n@@ -83,6 +83,20 @@ set_ident () {\n \tfinish_ident COMMITTER\n }\n \n+if test -z \"$FILTER_BRANCH_SQUELCH_WARNING$GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS\"\n+then\n+\tcat <] [--subdirectory-filter ] [--env-filter ]\n \t[--tree-filter ] [--index-filter ]\n \t[--parent-filter ] [--msg-filter ]\n-- \n2.23.0.3.g19edb94ec2\n\n"},{"id":"381845","messageId":"20190904223239.571-4-newren@gmail.com","threadId":"51710","inReplyTo":"20190904223239.571-1-newren@gmail.com","subject":"[PATCH v6 3/3] t9902: use a non-deprecated command for testing","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-09-04T22:32:39Z","receivedAt":"2019-09-04T22:32:53Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"t9902 had a list of three random porcelain commands as a sanity check,\none of which was filter-branch. Since we are recommending people not\nuse filter-branch, let's update this test to use rebase instead of\nfilter-branch.\n\nSigned-off-by: Elijah Newren \n---\n t/t9902-completion.sh | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/t/t9902-completion.sh b/t/t9902-completion.sh\nindex 75512c3403..4e7f669c76 100755\n--- a/t/t9902-completion.sh\n+++ b/t/t9902-completion.sh\n@@ -28,10 +28,10 @@ complete ()\n #\n # (2) A test makes sure that common subcommands are included in the\n # completion for \"git \", and a plumbing is excluded. \"add\",\n-# \"filter-branch\" and \"ls-files\" are listed for this.\n+# \"rebase\" and \"ls-files\" are listed for this.\n \n-GIT_TESTING_ALL_COMMAND_LIST='add checkout check-attr filter-branch ls-files'\n-GIT_TESTING_PORCELAIN_COMMAND_LIST='add checkout filter-branch'\n+GIT_TESTING_ALL_COMMAND_LIST='add checkout check-attr rebase ls-files'\n+GIT_TESTING_PORCELAIN_COMMAND_LIST='add checkout rebase'\n \n . \"$GIT_BUILD_DIR/contrib/completion/git-completion.bash\"\n \n@@ -1392,12 +1392,12 @@ test_expect_success 'basic' '\n \t# built-in\n \tgrep -q \"^add \\$\" out &&\n \t# script\n-\tgrep -q \"^filter-branch \\$\" out &&\n+\tgrep -q \"^rebase \\$\" out &&\n \t# plumbing\n \t! grep -q \"^ls-files \\$\" out &&\n \n-\trun_completion \"git f\" &&\n-\t! grep -q -v \"^f\" out\n+\trun_completion \"git r\" &&\n+\t! grep -q -v \"^r\" out\n '\n \n test_expect_success 'double dash \"git\" itself' '\n-- \n2.23.0.3.g19edb94ec2\n\n"}]}