{"thread":{"id":"35537","subject":"[PATCH] docs: add filter-branch note about The BFG","startedAt":"2013-12-17T10:53:19Z","lastAt":"2013-12-18T14:25:16Z","messageCount":7,"participants":["Roberto Tyley","Jonathan Nieder","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"232093","messageId":"1387277599-69719-1-git-send-email-roberto.tyley@gmail.com","threadId":"35537","inReplyTo":null,"subject":"[PATCH] docs: add filter-branch note about The BFG","fromName":"Roberto Tyley","fromEmail":"roberto.tyley@gmail.com","sentAt":"2013-12-17T10:53:19Z","receivedAt":"2013-12-17T10:53:19Z","isPatch":true,"sender":{"key":"roberto.tyley@gmail.com","avatar":"https://avatars.githubusercontent.com/u/52038?v=4"},"body":"The BFG is a tool specifically designed for the task of removing\nunwanted data from Git repository history - a common use-case for which\ngit-filter-branch has been the traditional workhorse.\n\nIt's beneficial to let users know that filter-branch has an alternative\nhere:\n\n* speed : The BFG is 10-50x faster\n http://rtyley.github.io/bfg-repo-cleaner/#speed\n* complexity of configuration : filter-branch is a very flexible tool,\n but demands very careful usage in order to get the desired results\n http://rtyley.github.io/bfg-repo-cleaner/#examples\n\nObviously, filter-branch has it's advantages too - it permits very\ncomplex rewrites, and doesn't require a JVM - but for the common\nuse-case of deleting unwanted data, it's helpful to users to be aware\nthat an alternative exists.\n\nThe BFG was released under the GPL in February 2013, and has since seen\nwidespread production use (The Guardian, RedHat, Google, UK Government\nDigital Service), been tested against large repos (~300K commits, ~5GB\npackfiles) and received significant positive feedback from users:\n\nhttp://rtyley.github.io/bfg-repo-cleaner/#feedback\n\nSigned-off-by: Roberto Tyley \n---\n Documentation/git-filter-branch.txt | 14 +++++++++++++-\n 1 file changed, 13 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt\nindex e4c8e82..918e965 100644\n--- a/Documentation/git-filter-branch.txt\n+++ b/Documentation/git-filter-branch.txt\n@@ -18,6 +18,12 @@ SYNOPSIS\n \n DESCRIPTION\n -----------\n+\n+NOTE: For simply removing unwanted data from repository history, you may\n+want to use link:http://rtyley.github.io/bfg-repo-cleaner/[The BFG Repo-Cleaner]\n+instead - it's generally faster and simpler for eliminating large files\n+or private data.\n+\n Lets you rewrite Git revision history by rewriting the branches mentioned\n in the , applying custom filters on each revision.\n Those filters can modify each tree (e.g. removing a file or running\n@@ -393,7 +399,7 @@ git filter-branch --index-filter \\\n Checklist for Shrinking a Repository\n ------------------------------------\n \n-git-filter-branch is often used to get rid of a subset of files,\n+git-filter-branch can be used to get rid of a subset of files,\n usually with some combination of `--index-filter` and\n `--subdirectory-filter`. People expect the resulting repository to\n be smaller than the original, but you need a few more steps to\n@@ -429,6 +435,12 @@ warned.\n (or if your git-gc is not new enough to support arguments to\n `--prune`, use `git repack -ad; git prune` instead).\n \n+SEE ALSO\n+--------\n+link:http://rtyley.github.io/bfg-repo-cleaner/[The BFG Repo-Cleaner]\n+- a tool specifically designed for removing unwanted data from Git\n+repository history.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\n-- \n1.8.3.4 (Apple Git-47)\n"},{"id":"232111","messageId":"xmqqk3f3mjl5.fsf@gitster.dls.corp.google.com","threadId":"35537","inReplyTo":"1387277599-69719-1-git-send-email-roberto.tyley@gmail.com","subject":"Re: [PATCH] docs: add filter-branch note about The BFG","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-12-17T18:13:42Z","receivedAt":"2013-12-17T18:13:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Roberto Tyley writes:\n\n> The BFG is a tool specifically designed for the task of removing\n> unwanted data from Git repository history - a common use-case for which\n> git-filter-branch has been the traditional workhorse.\n>\n> It's beneficial to let users know that filter-branch has an alternative\n> here:\n>\n> * speed : The BFG is 10-50x faster\n> http://rtyley.github.io/bfg-repo-cleaner/#speed\n> * complexity of configuration : filter-branch is a very flexible tool,\n> but demands very careful usage in order to get the desired results\n> http://rtyley.github.io/bfg-repo-cleaner/#examples\n>\n> Obviously, filter-branch has it's advantages too - it permits very\n> complex rewrites, and doesn't require a JVM - but for the common\n> use-case of deleting unwanted data, it's helpful to users to be aware\n> that an alternative exists.\n>\n> The BFG was released under the GPL in February 2013, and has since seen\n> widespread production use (The Guardian, RedHat, Google, UK Government\n> Digital Service), been tested against large repos (~300K commits, ~5GB\n> packfiles) and received significant positive feedback from users:\n>\n> http://rtyley.github.io/bfg-repo-cleaner/#feedback\n>\n> Signed-off-by: Roberto Tyley \n> ---\n> Documentation/git-filter-branch.txt | 14 +++++++++++++-\n> 1 file changed, 13 insertions(+), 1 deletion(-)\n>\n> diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt\n> index e4c8e82..918e965 100644\n> --- a/Documentation/git-filter-branch.txt\n> +++ b/Documentation/git-filter-branch.txt\n> @@ -18,6 +18,12 @@ SYNOPSIS\n> \n> DESCRIPTION\n> -----------\n> +\n> +NOTE: For simply removing unwanted data from repository history, you may\n> +want to use link:http://rtyley.github.io/bfg-repo-cleaner/[The BFG Repo-Cleaner]\n> +instead - it's generally faster and simpler for eliminating large files\n> +or private data.\n> +\n\nMy understanding is that the primary speed up of BFG comes from the\ndesign decision it made to fitler each blob only once, unlike\nfilter-branch that allows you to (and forces you to) decide how the\nsame blob is filtered depending on the places it appears in space\n(i.e. the path in the project's directory hierarchy) and time\n(i.e. the commit it appears in). For \"removing unwanted data\", I\nthink nobody needs the flexibility to filter differently depending\non the context, an it is a good idea to refer those with such need\nto BFG.\n\nHaving said that, \"You may want to use ...\" without giving the\nreason why we recommend the other tool leaves the reader wondering\nwhat the pros and cons are, and why git-filter-branch exists if BFG\nis the first thing its document recommends even before it describes\nwhat git-filter-branch is and does. \"You may want to check ...\"\nmight be slightly better, but probably by not that much improvement.\n\nRewriting \"it's generally faster ...\" part to give a bit more info\nto allow readers decide the pros and cons themselves may be needed.\n\n> Lets you rewrite Git revision history by rewriting the branches mentioned\n> in the , applying custom filters on each revision.\n> Those filters can modify each tree (e.g. removing a file or running\n> @@ -393,7 +399,7 @@ git filter-branch --index-filter \\\n> Checklist for Shrinking a Repository\n> ------------------------------------\n> \n> -git-filter-branch is often used to get rid of a subset of files,\n> +git-filter-branch can be used to get rid of a subset of files,\n> usually with some combination of `--index-filter` and\n> `--subdirectory-filter`. People expect the resulting repository to\n> be smaller than the original, but you need a few more steps to\n> @@ -429,6 +435,12 @@ warned.\n> (or if your git-gc is not new enough to support arguments to\n> `--prune`, use `git repack -ad; git prune` instead).\n> \n> +SEE ALSO\n> +--------\n> +link:http://rtyley.github.io/bfg-repo-cleaner/[The BFG Repo-Cleaner]\n> +- a tool specifically designed for removing unwanted data from Git\n> +repository history.\n> +\n> GIT\n> ---\n> Part of the linkgit:git[1] suite\n"},{"id":"232109","messageId":"20131217184003.GJ2311@google.com","threadId":"35537","inReplyTo":"1387277599-69719-1-git-send-email-roberto.tyley@gmail.com","subject":"Re: [PATCH] docs: add filter-branch note about The BFG","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2013-12-17T18:40:03Z","receivedAt":"2013-12-17T18:40:03Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nRoberto Tyley wrote:\n\n> The BFG is a tool specifically designed for the task of removing\n> unwanted data from Git repository history - a common use-case for which\n> git-filter-branch has been the traditional workhorse.\n>\n> It's beneficial to let users know that filter-branch has an alternative\n> here:\n\nThat sounds like a good suggestion for the SEE ALSO section, or an\nexplanation of when each tool should be used would be a good thing to\nput in NOTES, but...\n\n[...]\n> --- a/Documentation/git-filter-branch.txt\n> +++ b/Documentation/git-filter-branch.txt\n> @@ -18,6 +18,12 @@ SYNOPSIS\n> \n> DESCRIPTION\n> -----------\n> +\n> +NOTE: For simply removing unwanted data from repository history, you may\n> +want to use link:http://rtyley.github.io/bfg-repo-cleaner/[The BFG Repo-Cleaner]\n> +instead - it's generally faster and simpler for eliminating large files\n> +or private data.\n\n... this shouting NOTE at the top of the description is way over the\ntop.\n\nSo as is, this patch looks like a net negative.\n\nThanks and hope that helps,\nJonathan\n"},{"id":"232133","messageId":"CAFY1edaEZzDUuG9kopbAp9h2Frc2aLRKkjKMUnpSonML2xZN=A@mail.gmail.com","threadId":"35537","inReplyTo":"xmqqk3f3mjl5.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH] docs: add filter-branch note about The BFG","fromName":"Roberto Tyley","fromEmail":"roberto.tyley@gmail.com","sentAt":"2013-12-18T01:04:26Z","receivedAt":"2013-12-18T01:04:26Z","isPatch":true,"sender":{"key":"roberto.tyley@gmail.com","avatar":"https://avatars.githubusercontent.com/u/52038?v=4"},"body":"On 17 December 2013 18:13, Junio C Hamano wrote:\n>\n> Having said that, \"You may want to use ...\" without giving the\n> reason why we recommend the other tool leaves the reader wondering\n> what the pros and cons are, and why git-filter-branch exists if BFG\n> is the first thing its document recommends even before it describes\n> what git-filter-branch is and does. \"You may want to check ...\"\n> might be slightly better, but probably by not that much improvement.\n>\n> Rewriting \"it's generally faster ...\" part to give a bit more info\n> to allow readers decide the pros and cons themselves may be needed.\n\nThanks for that feedback, it makes sense. Here's an alternative\nversion which gives more information on the pros and cons of each\ntool, and why you might want to use either - as Jonathan suggested,\nthis would be for the NOTES section at the bottom of the file, where\nit's less intrusive:\n\nNotes\n-----\n\ngit-filter-branch allows you to make complex shell-scripted rewrites\nof your Git history, but you may not need this flexibility if you're\nsimply _removing unwanted data_ like large files or passwords. For\nthose operations you may want to consider\nlink:http://rtyley.github.io/bfg-repo-cleaner/[The BFG Repo-Cleaner],\na JVM-based alternative to git-filter-branch, typically at least\n10-50x faster for those use-cases, and with quite different\nproperties:\n\n* The BFG takes advantage of multi-core machines, cleaning commit\nfile-trees in parallel, which git-filter-branch currently does not do.\n* Any particular version of a file is cleaned exactly _once_. The BFG,\nunlike git-filter-branch, does not give you the opportunity to handle\na file differently based on where or when it was committed within your\nhistory.\n* The link:http://rtyley.github.io/bfg-repo-cleaner/#examples[command-set]\nis much more restrictive than git-filter branch, and dedicated just to\nthe tasks of removing unwanted data - e.g. `--strip-blobs-bigger-than\n1M`.\n\n\nI can re-submit this as a patch if it's acceptable?\n"},{"id":"232138","messageId":"xmqqr49ak8fi.fsf@gitster.dls.corp.google.com","threadId":"35537","inReplyTo":"CAFY1edaEZzDUuG9kopbAp9h2Frc2aLRKkjKMUnpSonML2xZN=A@mail.gmail.com","subject":"Re: [PATCH] docs: add filter-branch note about The BFG","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-12-18T05:57:37Z","receivedAt":"2013-12-18T05:57:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Roberto Tyley writes:\n\n> * The BFG takes advantage of multi-core machines, cleaning commit\n> file-trees in parallel, which git-filter-branch currently does not do.\n> * Any particular version of a file is cleaned exactly _once_. The BFG,\n> unlike git-filter-branch, does not give you the opportunity to handle\n> a file differently based on where or when it was committed within your\n> history.\n> * The link:http://rtyley.github.io/bfg-repo-cleaner/#examples[command-set]\n> is much more restrictive than git-filter branch, and dedicated just to\n> the tasks of removing unwanted data - e.g. `--strip-blobs-bigger-than\n> 1M`.\n\nI do not know offhand if the above formats well with AsciiDoc. You\nmay have to do it like this:\n\n* The first line of the bulletted paragraph is\n followed by the second and subsequent lines indented\n to align with the first one.\n\nThe first bullet point may be somewhat misleading, though. Nothing\nstops your script you use in filter-branch from processing blobs\nbelonging to a single tree in parallel---the user just needs to do a\nbit more work to do so.\n\nI think the second point is the most characteristic in BFG (and that\nis what allows easy parallelization of the filtering). Also, it\ncannot be stressed enough that the \"removing unwanted contents\" use\ncase can take advantage of the \"bad contents in a blob is bad, no\nmatter where in the tree and when in the history the blob appears\".\nThat is what makes BFG particularly shine for the use case. Its\ndesign very much aligns the objective the use case wants to achieve.\n"},{"id":"232150","messageId":"1387376716-72615-1-git-send-email-roberto.tyley@gmail.com","threadId":"35537","inReplyTo":"xmqqr49ak8fi.fsf@gitster.dls.corp.google.com","subject":"[PATCH v2] Tweaked notes on gfb<->bfg differences","fromName":"Roberto Tyley","fromEmail":"roberto.tyley@gmail.com","sentAt":"2013-12-18T14:25:15Z","receivedAt":"2013-12-18T14:25:15Z","isPatch":true,"sender":{"key":"roberto.tyley@gmail.com","avatar":"https://avatars.githubusercontent.com/u/52038?v=4"},"body":"On 18 December 2013 05:57, Junio C Hamano wrote:\n> The first bullet point may be somewhat misleading, though. Nothing\n> stops your script you use in filter-branch from processing blobs\n> belonging to a single tree in parallel---the user just needs to do a\n> bit more work to do so.\n\nThanks, I've moved this entry down and clarified the capabilities -\nI think there's quite a big difference to the user (of a multi-core\nmachine) as to whether parallelism happens by default, or whether they\nhave to work out how to introduce it into the operation they're trying\nto perform.\n\n> I think the second point is the most characteristic in BFG (and that\n> is what allows easy parallelization of the filtering). Also, it\n> cannot be stressed enough that the \"removing unwanted contents\" use\n> case can take advantage of the \"bad contents in a blob is bad, no\n> matter where in the tree and when in the history the blob appears\".\n> That is what makes BFG particularly shine for the use case. Its\n> design very much aligns the objective the use case wants to achieve.\n\nI've moved this up to the first bullet-point and added text emphasising\nthe significance the constraint plays in making the BFG work quickly\nwhile satisfying the common use-case.\n\n\nRoberto Tyley (1):\n docs: add filter-branch notes on The BFG\n\n Documentation/git-filter-branch.txt | 33 ++++++++++++++++++++++++++++++++-\n 1 file changed, 32 insertions(+), 1 deletion(-)\n\n-- \n1.8.3.4 (Apple Git-47)\n"},{"id":"232151","messageId":"1387376716-72615-2-git-send-email-roberto.tyley@gmail.com","threadId":"35537","inReplyTo":"1387376716-72615-1-git-send-email-roberto.tyley@gmail.com","subject":"[PATCH v2] docs: add filter-branch notes on The BFG","fromName":"Roberto Tyley","fromEmail":"roberto.tyley@gmail.com","sentAt":"2013-12-18T14:25:16Z","receivedAt":"2013-12-18T14:25:16Z","isPatch":true,"sender":{"key":"roberto.tyley@gmail.com","avatar":"https://avatars.githubusercontent.com/u/52038?v=4"},"body":"The BFG is a tool specifically designed for the task of removing\nunwanted data from Git repository history - a common use-case for which\ngit-filter-branch has been the traditional workhorse.\n\nIt's beneficial to let users know that filter-branch has an alternative\nhere:\n\n* speed : The BFG is 10-50x faster\n http://rtyley.github.io/bfg-repo-cleaner/#speed\n* complexity of configuration : filter-branch is a very flexible tool,\n but demands very careful usage in order to get the desired results\n http://rtyley.github.io/bfg-repo-cleaner/#examples\n\nObviously, filter-branch has it's advantages too - it permits very\ncomplex rewrites, and doesn't require a JVM - but for the common\nuse-case of deleting unwanted data, it's helpful to users to be aware\nthat an alternative exists.\n\nThe BFG was released under the GPL in February 2013, and has since seen\nwidespread production use (The Guardian, RedHat, Google, UK Government\nDigital Service), been tested against large repos (~300K commits, ~5GB\npackfiles) and received significant positive feedback from users:\n\nhttp://rtyley.github.io/bfg-repo-cleaner/#feedback\n\nSigned-off-by: Roberto Tyley \n---\n Documentation/git-filter-branch.txt | 33 ++++++++++++++++++++++++++++++++-\n 1 file changed, 32 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt\nindex e4c8e82..2eba627 100644\n--- a/Documentation/git-filter-branch.txt\n+++ b/Documentation/git-filter-branch.txt\n@@ -393,7 +393,7 @@ git filter-branch --index-filter \\\n Checklist for Shrinking a Repository\n ------------------------------------\n \n-git-filter-branch is often used to get rid of a subset of files,\n+git-filter-branch can be used to get rid of a subset of files,\n usually with some combination of `--index-filter` and\n `--subdirectory-filter`. People expect the resulting repository to\n be smaller than the original, but you need a few more steps to\n@@ -429,6 +429,37 @@ warned.\n (or if your git-gc is not new enough to support arguments to\n `--prune`, use `git repack -ad; git prune` instead).\n \n+Notes\n+-----\n+\n+git-filter-branch allows you to make complex shell-scripted rewrites\n+of your Git history, but you probably don't need this flexibility if\n+you're simply _removing unwanted data_ like large files or passwords.\n+For those operations you may want to consider\n+link:http://rtyley.github.io/bfg-repo-cleaner/[The BFG Repo-Cleaner],\n+a JVM-based alternative to git-filter-branch, typically at least\n+10-50x faster for those use-cases, and with quite different\n+characteristics:\n+\n+* Any particular version of a file is cleaned exactly _once_. The BFG,\n+ unlike git-filter-branch, does not give you the opportunity to\n+ handle a file differently based on where or when it was committed\n+ within your history. This constraint gives the core performance\n+ benefit of The BFG, and is well-suited to the task of cleansing bad\n+ data - you don't care _where_ the bad data is, you just want it\n+ _gone_.\n+\n+* By default The BFG takes full advantage of multi-core machines,\n+ cleansing commit file-trees in parallel. git-filter-branch cleans\n+ commits sequentially (ie in a single-threaded manner), though it\n+ _is_ possible to write filters that include their own parallellism,\n+ in the scripts executed against each commit.\n+\n+* The link:http://rtyley.github.io/bfg-repo-cleaner/#examples[command options]\n+ are much more restrictive than git-filter branch, and dedicated just\n+ to the tasks of removing unwanted data- e.g:\n+ `--strip-blobs-bigger-than 1M`.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\n-- \n1.8.3.4 (Apple Git-47)\n"}]}