{"thread":{"id":"50037","subject":"Git blame performance on files with a lot of history","startedAt":"2018-12-14T18:29:22Z","lastAt":"2018-12-17T20:59:36Z","messageCount":7,"participants":["Clement Moyroud","Bryan Turner","Derrick Stolee","Ævar Arnfjörð Bjarmason"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"365361","messageId":"CABXAcUzoNJ6s3=2xZfWYQUZ_AUefwP=5UVUgMnafKHHtufzbSA@mail.gmail.com","threadId":"50037","inReplyTo":null,"subject":"Git blame performance on files with a lot of history","fromName":"Clement Moyroud","fromEmail":"clement.moyroud@gmail.com","sentAt":"2018-12-14T18:29:17Z","receivedAt":"2018-12-14T18:29:22Z","isPatch":false,"sender":{"key":"clement.moyroud@gmail.com","avatar":null},"body":"Hello,\n\nMy group at work is migrating a CVS repo to Git. The biggest issue we\nface so far is the performance of git blame, especially compared to\nCVS on the same file. One file especially causes us trouble: it's a\n30k lines file with 25 years of history in 3k+ commits. The complete\nrepo has 200k+ commits over that same period of time.\n\nCurrently, 'cvs annotate' takes 2.7 seconds, while 'git blame'\n(without -M nor -C) takes 145s.\n\nI tried using the commit-graph with the Bloom filter, per\nhttps://public-inbox.org/git/61559c5b-546e-d61b-d2e1-68de692f5972@gmail.com/.\nNo dice:\n > time GIT_TEST_BLOOM_FILTERS=1\n/wv/cmoyroud/calibre-src/git-bloom-filters/git-bloom-bin/bin/git\ncommit-graph write --reachable\n Annotating commits in commit graph: 573705, done.\n Computing commit graph generation numbers: 100% (286441/286441), done.\n Computing commit diff Bloom filters: 100% (286441/286441), done.\n GIT_TEST_BLOOM_FILTERS=1 commit-graph write --reachable 386.80s\nuser 31.78s system 78% cpu 8:53.87 total\n > time GIT_TEST_BLOOM_FILTERS=1 GIT_TRACE_BLOOM_FILTER=2\nGIT_USE_POC_BLOOM_FILTER=y /path/to/git blame master --\nimportant/file.C > /tmp/foo.compiler.bloom\n Blaming lines: 100% (33179/33179), done.\n GIT_TEST_BLOOM_FILTERS=1 GIT_TRACE_BLOOM_FILTER=2\nGIT_USE_POC_BLOOM_FILTER=y 145.11s user 0.97s system 99% cpu 2:26.22\ntotal\n > time /path/to/git blame master -- important/file.C >\n/tmp/foo.compiler.nobloom\n Blaming lines: 100% (33179/33179), done.\n GIT_TEST_BLOOM_FILTERS=1 GIT_TEST_BLOOM_FILTERS=1\nGIT_USE_POC_BLOOM_FILTER=y 141.69s user 0.77s system 99% cpu 2:22.56\ntotal\n\nI used Derrick Stolee's tree at\nhttps://github.com/derrickstolee/git/tree/bloom/stolee\n\nLooking at the blame code, it does not seem to be able to use the\ncommit graph, so I tried the same rev-list command from the e-mail,\nusing my own file:\n > GIT_TRACE_BLOOM_FILTER=2 GIT_USE_POC_BLOOM_FILTER=y\n/path/to/git rev-list --count --full-history HEAD -- important/file.C\n 3576\n\nNo trace information there either. Running 'strings' on the binary\nreports the env. variable names, so I'm not totally crazy. Let me know\nif I tried the right thing :)\n\nLooks like blame performance is gonna be the biggest issue for us, so\nI'm really interested in seeing improvements there. Let me know if\nthere's anything else I can try.\n\nCheers,\n\nClément\n"},{"id":"365369","messageId":"CAGyf7-EZaiVBnoBfRJXAK7KdaXzPaDFVocUrB8LQNU_Hi16u7g@mail.gmail.com","threadId":"50037","inReplyTo":"CABXAcUzoNJ6s3=2xZfWYQUZ_AUefwP=5UVUgMnafKHHtufzbSA@mail.gmail.com","subject":"Re: Git blame performance on files with a lot of history","fromName":"Bryan Turner","fromEmail":"bturner@atlassian.com","sentAt":"2018-12-14T19:10:47Z","receivedAt":"2018-12-14T19:11:00Z","isPatch":false,"sender":{"key":"bturner@atlassian.com","avatar":"https://gravatar.com/avatar/16bcf3167981c1ef7c804e502642366d888a35b0d0b0a4ca01fdc442aa1acb1e?d=mp&s=160"},"body":"On Fri, Dec 14, 2018 at 10:29 AM Clement Moyroud\n wrote:\n>\n> Hello,\n>\n> My group at work is migrating a CVS repo to Git. The biggest issue we\n> face so far is the performance of git blame, especially compared to\n> CVS on the same file. One file especially causes us trouble: it's a\n> 30k lines file with 25 years of history in 3k+ commits. The complete\n> repo has 200k+ commits over that same period of time.\n\nAfter you converted the repository from CVS to Git, did you run a manual repack?\n\nThe process of converting a repository from another SCM often results\nin poor delta chain selections which result in a repository that's\nunnecessarily large on disk, and/or performs quite slowly.\n\nSomething like `git repack -Adf --depth=50 --window=200` discards the\nexisting delta chains and chooses new ones, and may result in\nsignificantly improved performance. A smaller depth, like --depth=20,\nmight result in even more performance improvement, but may also make\nthe repository larger on disk; you'll need to find the balance that\nworks for you.\n\nMight be something worth testing, if you haven't?\n\nBryan\n"},{"id":"365380","messageId":"3f3e7b11-19ef-cc2f-3bd4-e03d9ba8dc91@gmail.com","threadId":"50037","inReplyTo":"CABXAcUzoNJ6s3=2xZfWYQUZ_AUefwP=5UVUgMnafKHHtufzbSA@mail.gmail.com","subject":"Re: Git blame performance on files with a lot of history","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-12-14T21:31:24Z","receivedAt":"2018-12-14T21:31:28Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/14/2018 1:29 PM, Clement Moyroud wrote:\n> My group at work is migrating a CVS repo to Git. The biggest issue we\n> face so far is the performance of git blame, especially compared to\n> CVS on the same file. One file especially causes us trouble: it's a\n> 30k lines file with 25 years of history in 3k+ commits. The complete\n> repo has 200k+ commits over that same period of time.\n\nI think the 30k lines is the bigger issue than the 200k+ commits. I'm \nnot terribly familiar with the blame code, though.\n\n> Currently, 'cvs annotate' takes 2.7 seconds, while 'git blame'\n> (without -M nor -C) takes 145s.\n>\n> I tried using the commit-graph with the Bloom filter, per\n> https://public-inbox.org/git/61559c5b-546e-d61b-d2e1-68de692f5972@gmail.com/.\n\nThanks for the interest in this prototype feature. Sorry that it doesn't \nappear to help you in this case. It should definitely be a follow-up \nwhen that feature gets polished to production-quality.\n> Looking at the blame code, it does not seem to be able to use the\n> commit graph, so I tried the same rev-list command from the e-mail,\n> using my own file:\n> > GIT_TRACE_BLOOM_FILTER=2 GIT_USE_POC_BLOOM_FILTER=y\n> /path/to/git rev-list --count --full-history HEAD -- important/file.C\n> 3576\n>\nPlease double-check that you have the 'core.commitGraph' config setting \nenabled, or you will not read the commit-graph at run-time:\n\n     git config core.commitGraph true\n\nI see that the commit introducing GIT_TRACE_BLOOM_FILTER [1] does \nnothing if the commit-graph is not loaded.\n\nThanks,\n-Stolee\n\n[1] \nhttps://github.com/derrickstolee/git/commit/adc469894b755512c9d02f099700ead2a7a78377\n"},{"id":"365387","messageId":"87ftuz208b.fsf@evledraar.gmail.com","threadId":"50037","inReplyTo":"CABXAcUzoNJ6s3=2xZfWYQUZ_AUefwP=5UVUgMnafKHHtufzbSA@mail.gmail.com","subject":"Re: Git blame performance on files with a lot of history","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-12-14T22:48:52Z","receivedAt":"2018-12-14T22:48:58Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Dec 14 2018, Clement Moyroud wrote:\n\n> My group at work is migrating a CVS repo to Git. The biggest issue we\n> face so far is the performance of git blame, especially compared to\n> CVS on the same file. One file especially causes us trouble: it's a\n> 30k lines file with 25 years of history in 3k+ commits. The complete\n> repo has 200k+ commits over that same period of time.\n\nThere's a real-world repo with a shape & size very similar to this that\nhas good performance, gcc.git: https://github.com/gcc-mirror/gcc\n\n $ wc -l ChangeLog\n 20240 ChangeLog\n $ git log --oneline -- ChangeLog | wc -l\n 2676\n $ git log --oneline | wc -l\n 165309\n $ time git blame ChangeLog >/dev/null\n\n real 0m1.977s\n user 0m1.909s\n sys 0m0.069s\n\nIts history began in 1997, and the changes to the ChangeLog file by its\nnature is fairly evenly spread through that period.\n\nSo check out that repo to see if you have similar or worse\nperformance. Does your work repo show the same problem with a history\nproduced with 'git fast-export --anonymize', and if so is that something\nyou'd be OK with sharing?\n"},{"id":"365495","messageId":"CABXAcUwCrJGTQHwbM0Pg59DudZ6vCtZYJT5W=YJVnGGE3Jijew@mail.gmail.com","threadId":"50037","inReplyTo":"87ftuz208b.fsf@evledraar.gmail.com","subject":"Re: Git blame performance on files with a lot of history","fromName":"Clement Moyroud","fromEmail":"clement.moyroud@gmail.com","sentAt":"2018-12-17T20:30:56Z","receivedAt":"2018-12-17T20:31:01Z","isPatch":false,"sender":{"key":"clement.moyroud@gmail.com","avatar":null},"body":"On Fri, Dec 14, 2018 at 2:48 PM Ævar Arnfjörð Bjarmason\n wrote:\n>\n>\n> On Fri, Dec 14 2018, Clement Moyroud wrote:\n>\n> > My group at work is migrating a CVS repo to Git. The biggest issue we\n> > face so far is the performance of git blame, especially compared to\n> > CVS on the same file. One file especially causes us trouble: it's a\n> > 30k lines file with 25 years of history in 3k+ commits. The complete\n> > repo has 200k+ commits over that same period of time.\n>\n> There's a real-world repo with a shape & size very similar to this that\n> has good performance, gcc.git: https://github.com/gcc-mirror/gcc\n>\n> $ wc -l ChangeLog\n> 20240 ChangeLog\n> $ git log --oneline -- ChangeLog | wc -l\n> 2676\n> $ git log --oneline | wc -l\n> 165309\n> $ time git blame ChangeLog >/dev/null\n>\n> real 0m1.977s\n> user 0m1.909s\n> sys 0m0.069s\n>\n> Its history began in 1997, and the changes to the ChangeLog file by its\n> nature is fairly evenly spread through that period.\n>\n> So check out that repo to see if you have similar or worse\n> performance. Does your work repo show the same problem with a history\n> produced with 'git fast-export --anonymize', and if so is that something\n> you'd be OK with sharing?\n\nHi Ævar,\n\nI see around 3s here on the GCC repo, but I'm on a VM and the repo is\ncloned on an NFS disk, so I'd say it matches :) It's around 45x faster\nthan my repo, on the same NFS share and VM. So there's definitely\nsomething to improve here on my end (see my reply to Bryan re: repack\nin a separate e-mail).\n\nThe anonymized export won't work in that case: all file contents are\nreplaced with 'anonymous blob ', so there's no per-line history for\nblame to follow. Let me see if I can post-process a non-anonymized\nversion to keep the relevant data available.\n\nCheers,\n\nClément\n"},{"id":"365496","messageId":"CABXAcUyiJOcZwjSyvCKJC0vKDRbSP21sY_Fa7-WfmQHsczYbzQ@mail.gmail.com","threadId":"50037","inReplyTo":"CAGyf7-EZaiVBnoBfRJXAK7KdaXzPaDFVocUrB8LQNU_Hi16u7g@mail.gmail.com","subject":"Re: Git blame performance on files with a lot of history","fromName":"Clement Moyroud","fromEmail":"clement.moyroud@gmail.com","sentAt":"2018-12-17T20:43:31Z","receivedAt":"2018-12-17T20:43:35Z","isPatch":false,"sender":{"key":"clement.moyroud@gmail.com","avatar":null},"body":"did hOn Fri, Dec 14, 2018 at 11:10 AM Bryan Turner\n wrote:\n>\n> After you converted the repository from CVS to Git, did you run a manual repack?\n>\n> The process of converting a repository from another SCM often results\n> in poor delta chain selections which result in a repository that's\n> unnecessarily large on disk, and/or performs quite slowly.\n>\n\nYep I did a repack, using 'git repack -A -d --pack-kept-objects'. On\nNFS it'd be even\nworse, because of all the small objects Git would have to go through.\n\n> Something like `git repack -Adf --depth=50 --window=200` discards the\n> existing delta chains and chooses new ones, and may result in\n> significantly improved performance. A smaller depth, like --depth=20,\n> might result in even more performance improvement, but may also make\n> the repository larger on disk; you'll need to find the balance that\n> works for you.\n>\n\nI re-ran with 'git repack -Adf --depth=20 --window=200' and that did\nhelp quite a bit:\n > time git blame master -- important/file.C > /tmp/foo\n Blaming lines: 100% (33179/33179), done.\n git blame master -- important/file.C > /tmp/foo 50.70s user 0.55s\nsystem 99% cpu 51.298 total\n\nThat's roughly a 3x improvement, great. I just need another 10x one\nand we'll be in business :) Based on\nexperiments with the Bloom filter, maybe that'll help enough to get\nour users on board.\n\nCheers,\n\nClément\n"},{"id":"365497","messageId":"CABXAcUxNAr0H+EOmzv_TsW026Y-yDET01PeU3PtgcOmuFE5rjQ@mail.gmail.com","threadId":"50037","inReplyTo":"3f3e7b11-19ef-cc2f-3bd4-e03d9ba8dc91@gmail.com","subject":"Re: Git blame performance on files with a lot of history","fromName":"Clement Moyroud","fromEmail":"clement.moyroud@gmail.com","sentAt":"2018-12-17T20:59:33Z","receivedAt":"2018-12-17T20:59:36Z","isPatch":false,"sender":{"key":"clement.moyroud@gmail.com","avatar":null},"body":"On Fri, Dec 14, 2018 at 1:31 PM Derrick Stolee wrote:\n>\n> Please double-check that you have the 'core.commitGraph' config setting\n> enabled, or you will not read the commit-graph at run-time:\n>\n> git config core.commitGraph true\n>\n\nYeah, this is what happens when trying too many things at once :( I\nhad removed it to get\nwith/without scores, and forgot to re-enable it before trying my last\nset of experiments.\nHere are the results with it enabled:\n> time GIT_TRACE_BLOOM_FILTER=2 GIT_USE_POC_BLOOM_FILTER=y /path/to/git rev-list --count --full-history HEAD -- important/file.C\n10:32:06.665057 revision.c:483 bloom filter total queries:\n286363 definitely not: 234605 maybe: 51758 false positives: 48212 fp\nratio: 0.168360\nGIT_TRACE_BLOOM_FILTER=2 GIT_USE_POC_BLOOM_FILTER=y rev-list --count\nHEAD - 2.62s user 0.14s system 97% cpu 2.830 total\n> time /path/to/git rev-list --count --full-history HEAD -- ic/lv/src/iclv/drc_compiler.C\n3576\n/path/to/git rev-list 8.86s user 0.15s system 99% cpu 9.031 total\n\nSo I'm getting a 3x benefit, not bad! This is on the re-repacked repo,\nwhich is why I ran again\nwith and without the Bloom filter.\n\nLet's see what this does for blame:\n> time GIT_TRACE_BLOOM_FILTER=2 GIT_USE_POC_BLOOM_FILTER=y /path/to/git blame master -- important/file.C > /tmp/foo\nBlaming lines: 100% (33179/33179), done.\n12:50:42.703522 revision.c:483 bloom filter total queries: 0\ndefinitely not: 0 maybe: 0 false positives: 0 fp ratio: -nan\nGIT_TRACE_BLOOM_FILTER=2 GIT_USE_POC_BLOOM_FILTER=y blame master --\n> 132.59s user 2.15s system 99% cpu 2:14.95 total\n\nSeems like it's not implemented for blame operations. I'll be happy to\ntest any implementation.\n\nTake care,\n\nClément\n"}]}