{"thread":{"id":"38153","subject":"filter-branch performance","startedAt":"2014-12-09T18:52:33Z","lastAt":"2014-12-10T23:44:58Z","messageCount":7,"participants":["Henning Moll","Jeff King","Roberto Tyley","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"253428","messageId":"548744F1.9000902@gmx.de","threadId":"38153","inReplyTo":null,"subject":"filter-branch performance","fromName":"Henning Moll","fromEmail":"newsscott@gmx.de","sentAt":"2014-12-09T18:52:33Z","receivedAt":"2014-12-09T18:52:33Z","isPatch":false,"sender":{"key":"newsscott@gmx.de","avatar":null},"body":"Hi,\n\ni am runningthis command\n\ngit filter-branch --env-filter 'export \nGIT_COMMITTER_EMAIL=\"$GIT_AUTHOR_EMAIL\" \nGIT_COMMITTER_NAME=\"$GIT_AUTHOR_NAME\" \nGIT_COMMITTER_DATE=\"$GIT_AUTHOR_DATE\"' --prune-empty --tag-name-filter \ncat -- --all\n\n\nin a repository which i copied to /dev/shm before. According to \"top\", \nthe git process only consumes about 5 percent of the CPU. The load is \nbetween 0.70 and 1.00.\n\nI assume that there is a lot of process forking going on. Could that be \nthe cause?\n\nAny ideas how to further improve?\n\nBest regards\nHenning\n"},{"id":"253430","messageId":"20141209185933.GC31158@peff.net","threadId":"38153","inReplyTo":"548744F1.9000902@gmx.de","subject":"Re: filter-branch performance","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-12-09T18:59:34Z","receivedAt":"2014-12-09T18:59:34Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Dec 09, 2014 at 07:52:33PM +0100, Henning Moll wrote:\n\n> i am runningthis command\n> \n> git filter-branch --env-filter 'export\n> GIT_COMMITTER_EMAIL=\"$GIT_AUTHOR_EMAIL\"\n> GIT_COMMITTER_NAME=\"$GIT_AUTHOR_NAME\" GIT_COMMITTER_DATE=\"$GIT_AUTHOR_DATE\"'\n> --prune-empty --tag-name-filter cat -- --all\n> \n> in a repository which i copied to /dev/shm before. According to \"top\", the\n> git process only consumes about 5 percent of the CPU. The load is between\n> 0.70 and 1.00.\n> \n> I assume that there is a lot of process forking going on. Could that be the\n> cause?\n\nYes. filter-branch is a shell scripts, and it is probably running\nmultiple git commands per commit it is filtering.\n\n> Any ideas how to further improve?\n\nIn your case you are not touching the tree contents at all. Last time I\nlooked into this, I believe that filter-branch always loaded the index\nfor each commit, even if no --index-filter is being used. So teaching\nfilter-branch to optimize this case would be one strategy.\n\nAnother is to try using \"git fast-export | git fast-import\", and munging\nthe data stream in between. That's may be more work, depending how fancy\nyou want to get with accurate parsing (look into fast-export's\n--no-data, which omits blob data; that should make things faster and\nmake hacky context-less parsing less likely to cause problems).\n\n-Peff\n"},{"id":"253500","messageId":"CAFY1edYYC9TZmLE6b3=QAoTB1zQHi_Y97rHL-5wk5Pbpa_oj_w@mail.gmail.com","threadId":"38153","inReplyTo":"20141209185933.GC31158@peff.net","subject":"Re: filter-branch performance","fromName":"Roberto Tyley","fromEmail":"roberto.tyley@gmail.com","sentAt":"2014-12-10T14:18:24Z","receivedAt":"2014-12-10T14:18:24Z","isPatch":false,"sender":{"key":"roberto.tyley@gmail.com","avatar":"https://avatars.githubusercontent.com/u/52038?v=4"},"body":"On 9 December 2014 at 18:59, Jeff King wrote:\n> On Tue, Dec 09, 2014 at 07:52:33PM +0100, Henning Moll wrote:\n>> I assume that there is a lot of process forking going on. Could that be the\n>> cause?\n>\n> Yes. filter-branch is a shell scripts, and it is probably running\n> multiple git commands per commit it is filtering.\n>\n>> Any ideas how to further improve?\n\nDepending on how much time you can sink into improving the performance\n(versus just allowing the process to run to completion), you could\nalso look into a non-forking solution, as well as not bothering to\nload the commit trees. To me non-forking means putting everything into\nthe JVM by using JGit, like the BFG does, though libgit2 might also be\nan option.\n\nChanging the BFG's code to do the transformation in your script is\nabsolutely trivial - define a commit-node cleaner like this:\n\nobject SetCommitterToAuthor extends CommitNodeCleaner {\n override def fixer(kit: CommitNodeCleaner.Kit) = c =>\nc.copy(committer = c.author) // PersonIdent class holds name, email &\ntime\n}\n\n...trivial if you don't mind compiling Scala with SBT that is, and I'm\nsure some people do! A DSL for non-Scala people to define their own\nBFG scripts would be good, I must get on that some day.\n\nThe BFG is generally faster than filter-branch for 3 reasons:\n\n1. No forking - everything stays in the JVM process\n2. Embarrassingly parallel algorithm makes good use of multi-core machines\n3. Memoization means no Git object (file or folder) is cleaned more than once\n\nIn the case of your problem, only the first factor will be noticeably\nhelpful. Unfortunately commits do need to be cleaned sequentially, as\ntheir hashes depend on the hashes of their parents, and filter-branch\ndoesn't clean /commits/ more than once, the way it does with files or\nfolders - so the last 2 reasons in the list won't be significant.\n\nFor your specific use case tho', the fact that BFG doesn't load the\nfile tree at all unless it needs to clean it will also help.\n\nI decided to knock up an egregious hack in the BFG to see what\nperformance would be like. I ran it against a fairly large repo\n(https://github.com/bfg-repo-cleaner-demos/intellij-community-original),\n100k commits, stored in /dev/shm, and used the SetCommitterToAuthor\ncode above. The BFG run completed in 31.7 seconds, you can see the\nresulting repo here:\n\nhttps://github.com/rtyley/intellij-community-set-committer-to-author\n\nI started running the same test some time ago using filter-branch,\nunfortunately that test has not completed yet - the BFG appears to be\nsubstantially faster.\n\nBefore:\n$ git cat-file -p b02bf46c4e93c2e8570910cdd68eb6f4ce21ff81\ntree 7a412e49ecdbd966d7efe5fe746ff3ea3b6067d1\nparent 8794219e3e84aed3cc8af926ffd74beafa51fb6b\nauthor peter 1370854045 +0200\ncommitter peter 1370854098 +0200\n\nAfter:\n$ git cat-file -p 3adb7b2a5c87320a5a028b6a59a7132c75a6e91c\ntree 7a412e49ecdbd966d7efe5fe746ff3ea3b6067d1\nparent 5efcdb551789b0d0bb541de9325f09521c5fbcb6\nauthor peter 1370854045 +0200\ncommitter peter 1370854045 +0200 <- time fixed\n\nThe relevant code is in:\nhttps://github.com/rtyley/bfg-repo-cleaner/compare/set-committer-to-author\n"},{"id":"253501","messageId":"20141210143706.GA2996@peff.net","threadId":"38153","inReplyTo":"CAFY1edYYC9TZmLE6b3=QAoTB1zQHi_Y97rHL-5wk5Pbpa_oj_w@mail.gmail.com","subject":"Re: filter-branch performance","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-12-10T14:37:07Z","receivedAt":"2014-12-10T14:37:07Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Dec 10, 2014 at 02:18:24PM +0000, Roberto Tyley wrote:\n\n> Depending on how much time you can sink into improving the performance\n> (versus just allowing the process to run to completion), you could\n> also look into a non-forking solution, as well as not bothering to\n> load the commit trees. To me non-forking means putting everything into\n> the JVM by using JGit, like the BFG does, though libgit2 might also be\n> an option.\n> \n> Changing the BFG's code to do the transformation in your script is\n> absolutely trivial - define a commit-node cleaner like this:\n> \n> object SetCommitterToAuthor extends CommitNodeCleaner {\n> override def fixer(kit: CommitNodeCleaner.Kit) = c =>\n> c.copy(committer = c.author) // PersonIdent class holds name, email &\n> time\n> }\n\nThanks. I _almost_ mentioned BFG in the original email, but I didn't\nthink it could do arbitrary fixes like this. Can you monkey-patch in\narbitrary code, or do you have to rebuild all of BFG to include the\nsnippet above?\n\n> ...trivial if you don't mind compiling Scala with SBT that is, and I'm\n> sure some people do! A DSL for non-Scala people to define their own\n> BFG scripts would be good, I must get on that some day.\n\nThat would be cool. Even if the DSL was just Java, if you could do\nsomething like:\n\n vi fix.java\n javac fix.java\n bfg --filter=fix.class\n\nthat would be very useful (and I am probably showing my lack of Java chops\nby getting the compilation command or filenames wrong :) ).\n\n> I started running the same test some time ago using filter-branch,\n> unfortunately that test has not completed yet - the BFG appears to be\n> substantially faster.\n\nNo fair if you didn't run filter-branch on a PC and BFG on a Raspberry\nPi. You have to give us a fighting chance. :)\n\n-Peff\n"},{"id":"253503","messageId":"CAFY1edZqt-VY7afBjhtzQV6QW0zS8Cbd09XJbNZ9bj+T6GWOGg@mail.gmail.com","threadId":"38153","inReplyTo":"20141210143706.GA2996@peff.net","subject":"Re: filter-branch performance","fromName":"Roberto Tyley","fromEmail":"roberto.tyley@gmail.com","sentAt":"2014-12-10T15:25:44Z","receivedAt":"2014-12-10T15:25:44Z","isPatch":false,"sender":{"key":"roberto.tyley@gmail.com","avatar":"https://avatars.githubusercontent.com/u/52038?v=4"},"body":"On 10 December 2014 at 14:37, Jeff King wrote:\n> On Wed, Dec 10, 2014 at 02:18:24PM +0000, Roberto Tyley wrote:\n>> object SetCommitterToAuthor extends CommitNodeCleaner {\n>> override def fixer(kit: CommitNodeCleaner.Kit) = c =>\n>> c.copy(committer = c.author) // PersonIdent class holds name, email &\n>> time\n>> }\n>\n> Thanks. I _almost_ mentioned BFG in the original email, but I didn't\n> think it could do arbitrary fixes like this. Can you monkey-patch in\n> arbitrary code, or do you have to rebuild all of BFG to include the\n> snippet above?\n\nWell, I publish a bfg-library jar to Maven Central, so you don't need\nto rebuild that:\n\nhttp://search.maven.org/#search%7Cga%7C1%7Ca%3A%22bfg-library_2.11%22\n\n...in principle you can write a Java/Groovy/whatever project that\ncalls that jar (your entry point would be\ncom.madgag.git.bfg.cleaner.RepoRewriter) - tho' to be honest, I can't\nswear to how /friendly/ the API would be to call from non-Scala-land\nthough, as I haven't tried it.\n\nIncidentally, if people want to try compiling this monkey-patched BFG\nat home, this is how you'd do it:\n\n* Install SBT - http://www.scala-sbt.org/download.html (or 'brew\ninstall sbt' for Mac OS X)\n* git clone https://github.com/rtyley/bfg-repo-cleaner.git --branch\nset-committer-to-author\n* cd bfg-repo-cleaner\n* sbt \"bfg/run --no-blob-protection\"\n\nThere will be a lot of automated downloading of dependencies, and\ncompilation will be slow the first time around, but at least there\naren't that many steps. I do realise that being Scala/JVM based makes\nworking on the BFG a bit of a specialist activity at the moment!\n\n>> A DSL for non-Scala people to define their own\n>> BFG scripts would be good, I must get on that some day.\n>\n> That would be cool. Even if the DSL was just Java, if you could do\n> something like:\n>\n> vi fix.java\n> javac fix.java\n> bfg --filter=fix.class\n>\n> that would be very useful (and I am probably showing my lack of Java chops\n> by getting the compilation command or filenames wrong :) ).\n\nYour syntax is right :) I'll give it some thought.\n\n\n>> I started running the same test some time ago using filter-branch,\n>> unfortunately that test has not completed yet - the BFG appears to be\n>> substantially faster.\n>\n> No fair if you didn't run filter-branch on a PC and BFG on a Raspberry\n> Pi. You have to give us a fighting chance. :)\n\nI guess I made that rod for my own back :) http://youtu.be/Ir4IHzPhJuI\nfor those who haven't seen it.\n"},{"id":"253508","messageId":"xmqqfvcnjxry.fsf@gitster.dls.corp.google.com","threadId":"38153","inReplyTo":"CAFY1edYYC9TZmLE6b3=QAoTB1zQHi_Y97rHL-5wk5Pbpa_oj_w@mail.gmail.com","subject":"Re: filter-branch performance","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-12-10T16:05:21Z","receivedAt":"2014-12-10T16:05:21Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Roberto Tyley writes:\n\n> The BFG is generally faster than filter-branch for 3 reasons:\n>\n> 1. No forking - everything stays in the JVM process\n> 2. Embarrassingly parallel algorithm makes good use of multi-core machines\n> 3. Memoization means no Git object (file or folder) is cleaned more than once\n>\n> In the case of your problem, only the first factor will be noticeably\n> helpful. Unfortunately commits do need to be cleaned sequentially, as\n> their hashes depend on the hashes of their parents, and filter-branch\n> doesn't clean /commits/ more than once, the way it does with files or\n> folders - so the last 2 reasons in the list won't be significant.\n\nJust this part. If your history is bushy, you should be able to\nrewrite histories of merged branches in parallel up to the point\nthey are merged---rewriting of the merge commit of course has to\nwait until all the branches have been rewritten, though.\n"},{"id":"253547","messageId":"CAFY1eda-utVReuQnotSUDPV4-=hiMupbNdLZrYnEiaDryXQboQ@mail.gmail.com","threadId":"38153","inReplyTo":"xmqqfvcnjxry.fsf@gitster.dls.corp.google.com","subject":"Re: filter-branch performance","fromName":"Roberto Tyley","fromEmail":"roberto.tyley@gmail.com","sentAt":"2014-12-10T23:44:58Z","receivedAt":"2014-12-10T23:44:58Z","isPatch":false,"sender":{"key":"roberto.tyley@gmail.com","avatar":"https://avatars.githubusercontent.com/u/52038?v=4"},"body":"On 10 December 2014 at 16:05, Junio C Hamano wrote:\n> Roberto Tyley writes:\n>\n>> The BFG is generally faster than filter-branch for 3 reasons:\n>>\n>> 1. No forking - everything stays in the JVM process\n>> 2. Embarrassingly parallel algorithm makes good use of multi-core machines\n>> 3. Memoization means no Git object (file or folder) is cleaned more than once\n>>\n>> In the case of your problem, only the first factor will be noticeably\n>> helpful. Unfortunately commits do need to be cleaned sequentially, as\n>> their hashes depend on the hashes of their parents, and filter-branch\n>> doesn't clean /commits/ more than once, the way it does with files or\n>> folders - so the last 2 reasons in the list won't be significant.\n>\n> Just this part. If your history is bushy, you should be able to\n> rewrite histories of merged branches in parallel up to the point\n> they are merged---rewriting of the merge commit of course has to\n> wait until all the branches have been rewritten, though.\n\nThat's true, and the bfg does take advantage of that parallelism, so\nas well as point 1, point 2 will provide some benefit if history is\nbushy enough :)\n"}]}