{"thread":{"id":"18776","subject":"[ANNOUNCE] git_fast_filter","startedAt":"2009-04-08T03:35:10Z","lastAt":"2009-04-08T16:55:20Z","messageCount":3,"participants":["Elijah Newren","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"110777","messageId":"51419b2c0904072035u1182b507o836a67ac308d32b9@mail.gmail.com","threadId":"18776","inReplyTo":null,"subject":"[ANNOUNCE] git_fast_filter","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2009-04-08T03:35:10Z","receivedAt":"2009-04-08T03:35:10Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Just thought I'd make this available, in case there's others with\nniche needs that find it useful...\n\n\ngit_fast_filter assists with quickly rewriting the history of a repository\nby making it easy to write scripts whose purpose is to serve as safe\nfilters between fast-export and fast-import.  git_fast_filter comes with\nexample programs, a basic test-suite, and a double your money back\nsatisfaction guarantee.  (I love free software.)  You can get it from\n\n  git://gitorious.org/git_fast_filter/mainline.git\n\nIn more detail...\n\n=== Purpose ===\n\ngit_fast_filter is designed to make it easy to filter or rewrite the\nhistory of a repository.  As such, it fills the same role as\ngit-filter-branch, and was written primarily to overcome the sometimes\nsevere speed shortcomings of git-filter-branch.  In particular, using\ngit_fast_filter can avoid thousands or millions of new process forks, and\ncan allow you to rewrite the same file only one time instead of 50,000\ntimes.  However, while using git_fast_filter is fairly simple and quick, it\nis hard to beat writing a simple git-filter-branch one-liner for efficiency\nof human time.  Also, the two tools use very different methods of rewriting\nhistory and do not have exactly overlapping feature sets, so the best tool\nfor a particular job is going to be very problem dependent.\n\nAs human time is often more important than computer time, especially for\none-shot rewrites, git-filter-branch will probably continue to be the more\ncommon tool.  However, git_fast_filter is useful in cases where computer\ntime of a rewrite matters (particularly larger repositories and more\ninvolved rewrites that need to be run and tested many times on large data\nsets).  Also git_fast_filter has a couple features that may come in handy\nin special cases (assisting with generating fast-export output from\nscratch, interleaving commits from seperate repositories, and bidirectional\ncollaboration between filtered and unfiltered repositories).\n\n=== Idea ===\n\nThe way git_fast_filter works is by providing a simple python library,\ngit_fast_filter.py.  This library can be used in simple python scripts to\ncreate a filter for the output of git-fast-export.  Thus, the typical\ncalling convention is of the form:\n\n    git fast-export | filter_script.py | git fast-import\n\n=== Example ===\n\nAn example script that renames the 'master' branch to 'other is shown\nbelow (this is similar to the example in the git-fast-export manpage, but\nis safe against the string 'refs/heads/master' appearing in some file or\ncommit message in the repository):\n\n  #!/usr/bin/python\n\n  from git_fast_filter import Commit, FastExportFilter\n\n  def my_commit_callback(commit):\n    if commit.branch == \"refs/heads/master\":\n      commit.branch = \"refs/heads/other\"\n\n  filter = FastExportFilter(commit_callback = my_commit_callback)\n  filter.run()\n\nThe user can then run this script by:\n  $ mkdir target && cd target && git init\n  $ (cd /PATH/LEADING/TO/source && git fast-export --all) \\\n       | /PATH/TO/filter_script.py | git fast-import\n\n(Note: The user can have the script take care of the git init, the cd's,\nand the invocations of git fast-export and git fast-import by just passing\ndirectory names to FastExportFilter.run; however, writing out the details\nexplicitly as in the above example makes it clearer what is going on.)\n\n\nElijah\n"},{"id":"110812","messageId":"alpine.DEB.1.00.0904081144580.9157@intel-tinevez-2-302","threadId":"18776","inReplyTo":"51419b2c0904072035u1182b507o836a67ac308d32b9@mail.gmail.com","subject":"Re: [ANNOUNCE] git_fast_filter","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-04-08T09:45:41Z","receivedAt":"2009-04-08T09:45:41Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 7 Apr 2009, Elijah Newren wrote:\n\n> Just thought I'd make this available, in case there's others with niche \n> needs that find it useful...\n\nHave you seen\n\n\thttp://thread.gmane.org/gmane.comp.version-control.git/52323\n\nI was rather disappointed that skimo left the patch series in a rather \nhalf-useful state.\n\nCiao,\nDscho\n"},{"id":"110845","messageId":"51419b2c0904080955u71d18093rd3e8b6a29cf9bbfb@mail.gmail.com","threadId":"18776","inReplyTo":"alpine.DEB.1.00.0904081144580.9157@intel-tinevez-2-302","subject":"Re: [ANNOUNCE] git_fast_filter","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2009-04-08T16:55:20Z","receivedAt":"2009-04-08T16:55:20Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi,\n\nOn Wed, Apr 8, 2009 at 3:45 AM, Johannes Schindelin\n<Johannes.Schindelin@gmx.de> wrote:\n> On Tue, 7 Apr 2009, Elijah Newren wrote:\n>\n>> Just thought I'd make this available, in case there's others with niche\n>> needs that find it useful...\n>\n> Have you seen\n>\n>        http://thread.gmane.org/gmane.comp.version-control.git/52323\n>\n> I was rather disappointed that skimo left the patch series in a rather\n> half-useful state.\n\nNo, I had not.  Looks useful, though it appears to be missing a number\nof options from git-filter-branch, such as --subdirectory-filter,\n--tree-filter, and --prune-empty.  I'm guessing that's related to your\n\"half-useful\" comment?\n\nI was particularly interested in --tree-filter for my rewrite, since I\nneeded it (to remove all cvs keywords and a few other touch-ups) and\nit was the slowest of the filters I'd be using.  Problem is, in a\nrepository with 40,000 commits and 70,000 files in the latest commits,\n--tree-filter is unacceptably slow.  On average, a commit would have\nabout 35,000 files (assuming approximately linear growth over the\ncommit history), meaning that I'd have to modify 40,000 x 35,000 files\n= 1,400,000,000 files[1].  However, on average, less than a dozen\nfiles changed per commit, so there are less than 40,000 x 12 = 480,000\nunique files that actually need to be rewritten.  git-fast-export\nprovides (and git-fast-import expects) just those half million files,\nand rewriting half a million files instead of 1.4 billion files is the\ndifference between a 45 minute rewrite and a 3 month one.\n\nI didn't see a way to easily avoid the 1.4 billion file rewrite using\ngit-filter-branch (or git-rewrite-commits had I known about it), and\nwriting something to parse and modify git-fast-export output seemed\nlike the easiest solution.  Perhaps I could have written some fancy\nindex-filter script that recorded original and modified file sha1sums\nsomewhere and used that to only check out certain files and rewrite\nthem, but such an idea hadn't occurred to me (and I'm not sure it\nwould have been the better route even if it had).  Maybe there's\nsomething I missed that would have made this easy, though?\n\n\nElijah\n\n[1] For simplicity, I'm ignoring the 'binary' files that should not\nhave any cvs-keyword unmunging performed on them.  However, it does\npresent an issue, particularly with extra process forks, since you\nneed to determine which files are safe to modify.  I used libmagic\n(the library behind the unix 'file' command) to avoid the need to run\n'file' repeatedly.\n"}]}