{"thread":{"id":"32828","subject":"A fast alternative to git-filter-branch - The BFG Repo-Cleaner","startedAt":"2013-02-05T00:04:20Z","lastAt":"2013-02-05T00:04:20Z","messageCount":1,"participants":["Roberto Tyley"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"208663","messageId":"CAFY1edb6osN+Qe33K9e6imaMG=3_ZUJx7Q1R++RHfY6h+zGXYQ@mail.gmail.com","threadId":"32828","inReplyTo":null,"subject":"A fast alternative to git-filter-branch - The BFG Repo-Cleaner","fromName":"Roberto Tyley","fromEmail":"roberto.tyley@gmail.com","sentAt":"2013-02-05T00:04:20Z","receivedAt":"2013-02-05T00:04:20Z","isPatch":false,"sender":{"key":"roberto.tyley@gmail.com","avatar":"https://avatars.githubusercontent.com/u/52038?v=4"},"body":"I recently released The BFG Repo-Cleaner, a new tool for cleansing bad\ndata out of Git repository histories. The BFG is typically at least\n10-50x faster than git-filter-branch at these tasks:\n\n* Removing Crazy Big Files from repo history\n* Removing Passwords, Credentials & other Private data\n\nhttp://rtyley.github.com/bfg-repo-cleaner/\n\nAs an example, these are timings for deleting an arbitrary file from\nthe large GCC repository (148495 commits):\n\nThe BFG : 3m29s\n$ bfg -D README-fixinc\n\ngit filter-branch : 472m31s\n$ git filter-branch --index-filter 'git rm --cached --ignore-unmatch\ngcc/README-fixinc' --prune-empty --tag-name-filter cat -- --all\n\n(roughly a 135x speed increase, reducing the task of processing a\nlarge codebase from an overnight job to the work of a few minutes....\nall timings done in a 4GB tmpfs ramdisk)\n\n\nThe BFG has some simple but very powerful command-line options, which\nperform at similar speed:\n\nremove all blobs bigger than 1 megabyte :\n$ bfg --strip-blobs-bigger-than 1M  my-repo.git\n\nreplace all passwords (listed in a file 'passwords.txt') with ***REMOVED*** :\n$ bfg --replace-banned-strings passwords.txt  my-repo.git\n\n\nThe main source of the BFG's performance advantage comes from\npreventing repeated examination of the same tree objects. The approach\nof git-filter-branch performs filtering for each commit, against the\ncomplete file-hierarchy of each commit, one after the other, even\nthough commit trees are largely very similar. For the use-cases of The\nBFG that's unnecessary- we don't care where, and in which commit, a\n'bad' file exists - we just want it dealt with. Consequently the BFG\nprocesses the Git object db on a memoised tree-by-tree basis,\nprocessing each and every file & folder exactly once - the final\nprocessing of the commit hierarchy is very quick. This _does_ mean\nthat it's not possible to delete files based on their absolute path\nwithin the repo, but they can deleted based on their filename,\nblob-id, or contents. This, and multi-core processing by default,\ngives the dramatic speed-up while still providing the same results.\nThere's more performance data here:\nhttps://docs.google.com/spreadsheet/ccc?key=0AsR1d5Zpes8HdER3VGU1a3dOcmVHMmtzT2dsS2xNenc\n\nI'd welcome feedback, and if anyone has cause to filter a repository's\nhistory in future, I'd appreciate you giving the BFG a try and letting\nme know how you found it.\n\nthanks,\nRoberto Tyley\nsoftware dev @ The Guardian\n\nhttp://rtyley.github.com/bfg-repo-cleaner/\n"}]}