{"thread":{"id":"45561","subject":"Terrible bad performance for it blame --date=iso -C -C master -- ","startedAt":"2017-03-31T07:40:22Z","lastAt":"2017-05-03T12:29:51Z","messageCount":11,"participants":["Ulrich Windl","Junio C Hamano","Samuel Lijin","SZEDER Gábor","Jakub Narębski"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"315874","messageId":"58DE0598020000A1000256CE@gwsmtp1.uni-regensburg.de","threadId":"45561","inReplyTo":"58DE0598020000A1000256CE@gwsmtp1.uni-regensburg.de","subject":"Terrible bad performance for it blame --date=iso -C -C master -- ","fromName":"Ulrich Windl","fromEmail":"ulrich.windl@rz.uni-regensburg.de","sentAt":"2017-03-31T07:30:32Z","receivedAt":"2017-03-31T07:40:22Z","isPatch":false,"sender":{"key":"ulrich.windl@rz.uni-regensburg.de","avatar":null},"body":"\nHi!\n\nI was running \"vc-annotate\" in Emacs for a file from a large repository (>40000 files, a big percentage being binary, about 10 commits). For the first file the result was presented rather soon, but for a second file the command did not finish even after about 10 minutes!\n\nThe file in question is a rather short text file (124 kB), and according to git log it has one commit.\n\nWhile being bored, I did an strace of the command to find out that a huge number of files is inspected.\n\nI'm using git of openSUSE LEap 42.2 (git-2.10.2-3.1.x86_64).\n\nHere's the stat of the process:\n\n% cat /proc/6307/stat\n6307 (git) R 6189 6307 6307 0 -1 4194304 30514692 0 1860 0 142035 6333 0 0 20 0 1 0 14519800 5850595328 391884 18446744073709551615 4194304 6038988 140726752300720 140726752293728 5549816 0 0 67108864 0 0 0 0 17 2 0 0 28211 0 0 8138160 8173100 9625600 140726752306005 140726752306072 140726752306072 140726752309227 0\n\nRegards,\nUlrich\n\n"},{"id":"315919","messageId":"xmqq60ip1m0f.fsf@gitster.mtv.corp.google.com","threadId":"45561","inReplyTo":"58DE0598020000A1000256CE@gwsmtp1.uni-regensburg.de","subject":"Re: Terrible bad performance for it blame --date=iso -C -C master -- ","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-03-31T15:52:48Z","receivedAt":"2017-03-31T15:52:55Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Ulrich Windl\" writes:\n\n> I was running \"vc-annotate\" in Emacs for a file from a large\n> repository (>40000 files, a big percentage being binary, about 10\n> commits). For the first file the result was presented rather soon, but\n> for a second file the command did not finish even after about 10\n> minutes!\n>\n> The file in question is a rather short text file (124 kB), and\n> according to git log it has one commit.\n>\n> While being bored, I did an strace of the command to find out that a\n> huge number of files is inspected.\n\nWith -C -C the user (vc-annotate?) is asking to inspect huge number\nof files, to find if the contents of the file (except for the part\nthat came from its own previous version) came from other existing\nfiles. So this is very much expected.\n\nIt might not be a bad idea to teach \"blame\" not to pay attention to\nany path that is marked as \"-diff\" (e.g. binary files) when trying\nto see if remaining contents appeared by borrowing from them. We do\nnot have that heuristics (yet).\n"},{"id":"315923","messageId":"CAJZjrdXyRqs=oscynWghB13OiR8qjP10O2EV-wiP-acb7wqY-A@mail.gmail.com","threadId":"45561","inReplyTo":"xmqq60ip1m0f.fsf@gitster.mtv.corp.google.com","subject":"Re: Terrible bad performance for it blame --date=iso -C -C master -- ","fromName":"Samuel Lijin","fromEmail":"sxlijin@gmail.com","sentAt":"2017-03-31T16:23:02Z","receivedAt":"2017-03-31T16:23:49Z","isPatch":false,"sender":{"key":"sxlijin@gmail.com","avatar":"https://gravatar.com/avatar/01777bf1eae64e2b4dca97dcac182a6abbcf6fd8cb4d5b8fa33edf9f8cc21746?d=mp&s=160"},"body":"Hi Ulrich,\n\nIs there any chance you could share the repo where this is coming from?\n\nThis is actually something a colleague and I are looking into seeing\nif we can crunch out some performance gains since -C -C isn't\nthreaded.\n\nSam\n\nOn Fri, Mar 31, 2017 at 10:52 AM, Junio C Hamano wrote:\n> \"Ulrich Windl\" writes:\n>\n>> I was running \"vc-annotate\" in Emacs for a file from a large\n>> repository (>40000 files, a big percentage being binary, about 10\n>> commits). For the first file the result was presented rather soon, but\n>> for a second file the command did not finish even after about 10\n>> minutes!\n>>\n>> The file in question is a rather short text file (124 kB), and\n>> according to git log it has one commit.\n>>\n>> While being bored, I did an strace of the command to find out that a\n>> huge number of files is inspected.\n>\n> With -C -C the user (vc-annotate?) is asking to inspect huge number\n> of files, to find if the contents of the file (except for the part\n> that came from its own previous version) came from other existing\n> files. So this is very much expected.\n>\n> It might not be a bad idea to teach \"blame\" not to pay attention to\n> any path that is marked as \"-diff\" (e.g. binary files) when trying\n> to see if remaining contents appeared by borrowing from them. We do\n> not have that heuristics (yet).\n"},{"id":"316085","messageId":"58E1F239020000A100025732@gwsmtp1.uni-regensburg.de","threadId":"45561","inReplyTo":"xmqq60ip1m0f.fsf@gitster.mtv.corp.google.com","subject":"Antw: Re: Terrible bad performance for it blame --date=iso -C -C master -- ","fromName":"Ulrich Windl","fromEmail":"ulrich.windl@rz.uni-regensburg.de","sentAt":"2017-04-03T06:56:57Z","receivedAt":"2017-04-03T06:57:06Z","isPatch":false,"sender":{"key":"ulrich.windl@rz.uni-regensburg.de","avatar":null},"body":"Junio,\n\nthanks for explaining! So if there are at least two commits, blame is fast, but with only one commit blame tries hard to find another commit that might have contributed to the one file?\nI verified that without those \"-C\" options the result is very quick.\nIn the other case (for the user bored of waiting seeking for some entertainment ;-)) a \"-v (verbose) option could be useful.\nOr at the very least: If git is expecting that some operation will take (or already did take) a lot of time, give some message explaining why it is taking a lot of time, and maybe how to avoid that.\n\nRegards,\nUlrich\n\n\n>>> Junio C Hamano schrieb am 31.03.2017 um 17:52 in Nachricht\n:\n> \"Ulrich Windl\" writes:\n> \n>> I was running \"vc-annotate\" in Emacs for a file from a large\n>> repository (>40000 files, a big percentage being binary, about 10\n>> commits). For the first file the result was presented rather soon, but\n>> for a second file the command did not finish even after about 10\n>> minutes!\n>>\n>> The file in question is a rather short text file (124 kB), and\n>> according to git log it has one commit.\n>>\n>> While being bored, I did an strace of the command to find out that a\n>> huge number of files is inspected.\n> \n> With -C -C the user (vc-annotate?) is asking to inspect huge number\n> of files, to find if the contents of the file (except for the part\n> that came from its own previous version) came from other existing\n> files. So this is very much expected.\n> \n> It might not be a bad idea to teach \"blame\" not to pay attention to\n> any path that is marked as \"-diff\" (e.g. binary files) when trying\n> to see if remaining contents appeared by borrowing from them. We do\n> not have that heuristics (yet).\n\n\n\n\n"},{"id":"316087","messageId":"20170403105641.16912-1-szeder.dev@gmail.com","threadId":"45561","inReplyTo":"58E1F239020000A100025732@gwsmtp1.uni-regensburg.de","subject":"Re: Terrible bad performance for it blame --date=iso -C","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2017-04-03T10:56:41Z","receivedAt":"2017-04-03T10:56:57Z","isPatch":false,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"> In the other case (for the user bored of waiting seeking for some\n> entertainment ;-)) a \"-v (verbose) option could be useful. Or at the\n> very least: If git is expecting that some operation will take (or\n> already did take) a lot of time, give some message explaining why it\n> is taking a lot of time, and maybe how to avoid that.\n\nIt already does so by default since v2.8.0, see aba37f495 (blame: add\nsupport for --[no-]progress option, 2015-12-12).\n\n $ time git blame sha1_file.c |wc -l\n 4026\n \n real 0m1.744s\n user 0m1.672s\n sys 0m0.068s\n $ time git blame -C -C sha1_file.c |wc -l\n Blaming lines: 100% (4026/4026), done.\n 4026\n \n real 0m3.832s\n user 0m3.716s\n sys 0m0.112s\n\nHowever, after a short peek at that commit, it only displays progress\nby default when stderr is a terminal, which might not be the case when\ninvoked from emacs.\n\n"},{"id":"316089","messageId":"0ccc5cab-26b7-4b02-b964-452b61e92579@gmail.com","threadId":"45561","inReplyTo":"20170403105641.16912-1-szeder.dev@gmail.com","subject":"Re: Terrible bad performance for it blame --date=iso -C","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2017-04-03T15:16:01Z","receivedAt":"2017-04-03T15:16:29Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 03.04.2017 o 12:56, SZEDER Gábor pisze:\n> Ulrich Windl wrote:\n\n>> In the other case (for the user bored of waiting seeking for some\n>> entertainment ;-)) a \"-v (verbose) option could be useful. Or at the\n>> very least: If git is expecting that some operation will take (or\n>> already did take) a lot of time, give some message explaining why it\n>> is taking a lot of time, and maybe how to avoid that.\n> \n> It already does so by default since v2.8.0, see aba37f495 (blame: add\n> support for --[no-]progress option, 2015-12-12).\n> \n> $ time git blame sha1_file.c |wc -l\n> 4026\n> \n> real 0m1.744s\n> user 0m1.672s\n> sys 0m0.068s\n> $ time git blame -C -C sha1_file.c |wc -l\n> Blaming lines: 100% (4026/4026), done.\n> 4026\n> \n> real 0m3.832s\n> user 0m3.716s\n> sys 0m0.112s\n> \n> However, after a short peek at that commit, it only displays progress\n> by default when stderr is a terminal, which might not be the case when\n> invoked from emacs.\n\nEmacs (magit?) should use `git blame --porcelain`, and do its own\nprogress report, just like 'git gui blame' and incremental blame mode\nof gitweb.\n\nActually... there already is git-blamed - Minor mode for incremental\nblame for Git, and mo-git-blame - An interactive, iterative 'git blame'\nmode for Emacs, both available on ELPA (Emacs Lisp Package Archive).\n\nHTH\n-- \nJakub Narębski\n\n"},{"id":"316133","messageId":"58E33BC4020000A100025797@gwsmtp1.uni-regensburg.de","threadId":"45561","inReplyTo":"0ccc5cab-26b7-4b02-b964-452b61e92579@gmail.com","subject":"Antw: Re: Terrible bad performance for it blame --date=iso -C","fromName":"Ulrich Windl","fromEmail":"ulrich.windl@rz.uni-regensburg.de","sentAt":"2017-04-04T06:23:00Z","receivedAt":"2017-04-04T06:23:10Z","isPatch":false,"sender":{"key":"ulrich.windl@rz.uni-regensburg.de","avatar":null},"body":">>> Jakub Narebski schrieb am 03.04.2017 um 17:16 in\nNachricht\n<0ccc5cab-26b7-4b02-b964-452b61e92579@gmail.com>:\n> W dniu 03.04.2017 o 12:56, SZEDER Gábor pisze:\n>> Ulrich Windl wrote:\n> \n>>> In the other case (for the user bored of waiting seeking for some\n>>> entertainment ;-)) a \"-v (verbose) option could be useful. Or at the\n>>> very least: If git is expecting that some operation will take (or\n>>> already did take) a lot of time, give some message explaining why it\n>>> is taking a lot of time, and maybe how to avoid that.\n>> \n>> It already does so by default since v2.8.0, see aba37f495 (blame: add\n>> support for --[no-]progress option, 2015-12-12).\n>> \n>> $ time git blame sha1_file.c |wc -l\n>> 4026\n>> \n>> real 0m1.744s\n>> user 0m1.672s\n>> sys 0m0.068s\n>> $ time git blame -C -C sha1_file.c |wc -l\n>> Blaming lines: 100% (4026/4026), done.\n>> 4026\n>> \n>> real 0m3.832s\n>> user 0m3.716s\n>> sys 0m0.112s\n>> \n>> However, after a short peek at that commit, it only displays progress\n>> by default when stderr is a terminal, which might not be the case when\n>> invoked from emacs.\n> \n> Emacs (magit?) should use `git blame --porcelain`, and do its own\n> progress report, just like 'git gui blame' and incremental blame mode\n> of gitweb.\n\nI was thinking similar: The pain vc-annotate obviously should work without\nthose \"-C\" options, and with prefix argument (C-u in Emacs) it could\nstart looking for copied stuff. HMO...\n\nWorse than no progress reporting is the inability to kill the process (if you\nrun out of patience) with C-g (that stops most commands in Emacs).\n\n> \n> Actually... there already is git-blamed - Minor mode for incremental\n> blame for Git, and mo-git-blame - An interactive, iterative 'git blame'\n> mode for Emacs, both available on ELPA (Emacs Lisp Package Archive).\n\nI confess taht is still use RCS from time to time, and I prefer the\nhigher-level Emacs commands ;-)\n\nRegards,\nUlrich\n\n"},{"id":"318382","messageId":"CAJZjrdUt+=oLt0cDcs2+bJMCFR4h8UUSFmZkx6-1menz4haPig@mail.gmail.com","threadId":"45561","inReplyTo":"xmqq60ip1m0f.fsf@gitster.mtv.corp.google.com","subject":"Re: Terrible bad performance for it blame --date=iso -C -C master -- ","fromName":"Samuel Lijin","fromEmail":"sxlijin@gmail.com","sentAt":"2017-05-01T17:58:42Z","receivedAt":"2017-05-01T17:59:29Z","isPatch":false,"sender":{"key":"sxlijin@gmail.com","avatar":"https://gravatar.com/avatar/01777bf1eae64e2b4dca97dcac182a6abbcf6fd8cb4d5b8fa33edf9f8cc21746?d=mp&s=160"},"body":"On Fri, Mar 31, 2017 at 10:52 AM, Junio C Hamano wrote:\n> \"Ulrich Windl\" writes:\n>\n>> I was running \"vc-annotate\" in Emacs for a file from a large\n>> repository (>40000 files, a big percentage being binary, about 10\n>> commits). For the first file the result was presented rather soon, but\n>> for a second file the command did not finish even after about 10\n>> minutes!\n>>\n>> The file in question is a rather short text file (124 kB), and\n>> according to git log it has one commit.\n>>\n>> While being bored, I did an strace of the command to find out that a\n>> huge number of files is inspected.\n>\n> With -C -C the user (vc-annotate?) is asking to inspect huge number\n> of files, to find if the contents of the file (except for the part\n> that came from its own previous version) came from other existing\n> files. So this is very much expected.\n>\n> It might not be a bad idea to teach \"blame\" not to pay attention to\n> any path that is marked as \"-diff\" (e.g. binary files) when trying\n> to see if remaining contents appeared by borrowing from them. We do\n> not have that heuristics (yet).\n\nCould you elaborate on this? Do you mean to tell diffcore-rename to\nignore diff_filespec objects if they're binary?\n"},{"id":"318436","messageId":"xmqqfugof4zb.fsf@gitster.mtv.corp.google.com","threadId":"45561","inReplyTo":"CAJZjrdUt+=oLt0cDcs2+bJMCFR4h8UUSFmZkx6-1menz4haPig@mail.gmail.com","subject":"Re: Terrible bad performance for it blame --date=iso -C -C master -- ","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-05-02T00:59:52Z","receivedAt":"2017-05-02T01:00:01Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Samuel Lijin writes:\n\n> On Fri, Mar 31, 2017 at 10:52 AM, Junio C Hamano wrote:\n>\n>> It might not be a bad idea to teach \"blame\" not to pay attention to\n>> any path that is marked as \"-diff\" (e.g. binary files) when trying\n>> to see if remaining contents appeared by borrowing from them. We do\n>> not have that heuristics (yet).\n>\n> Could you elaborate on this? Do you mean to tell diffcore-rename to\n> ignore diff_filespec objects if they're binary?\n\nNo and yes ;-). I do not think it is a good idea to unconditionally\nignore binary in diffcore-rename.\n\nBut when we know that the rename detection is called from inside\nblame.c, where by definition we would be digging line-oriented\ncontents, there is no point checking if the contents we are looking\nfor came from an existing binary file.\n"},{"id":"318609","messageId":"CAJZjrdVZ7gTZvm=CcG7pUPWtXjsMsHgyMRiE8xoXm=bZ4j6FEQ@mail.gmail.com","threadId":"45561","inReplyTo":"xmqqfugof4zb.fsf@gitster.mtv.corp.google.com","subject":"Re: Terrible bad performance for it blame --date=iso -C -C master -- ","fromName":"Samuel Lijin","fromEmail":"sxlijin@gmail.com","sentAt":"2017-05-03T07:12:06Z","receivedAt":"2017-05-03T07:12:52Z","isPatch":false,"sender":{"key":"sxlijin@gmail.com","avatar":"https://gravatar.com/avatar/01777bf1eae64e2b4dca97dcac182a6abbcf6fd8cb4d5b8fa33edf9f8cc21746?d=mp&s=160"},"body":"On Mon, May 1, 2017 at 7:59 PM, Junio C Hamano wrote:\n> Samuel Lijin writes:\n>\n>> On Fri, Mar 31, 2017 at 10:52 AM, Junio C Hamano wrote:\n>>\n>>> It might not be a bad idea to teach \"blame\" not to pay attention to\n>>> any path that is marked as \"-diff\" (e.g. binary files) when trying\n>>> to see if remaining contents appeared by borrowing from them. We do\n>>> not have that heuristics (yet).\n>>\n>> Could you elaborate on this? Do you mean to tell diffcore-rename to\n>> ignore diff_filespec objects if they're binary?\n>\n> No and yes ;-). I do not think it is a good idea to unconditionally\n> ignore binary in diffcore-rename.\n>\n> But when we know that the rename detection is called from inside\n> blame.c, where by definition we would be digging line-oriented\n> contents, there is no point checking if the contents we are looking\n> for came from an existing binary file.\n\nA followup thought: perhaps it would make sense for blame.c rename\ndetection to only consider files with the same extension?\n"},{"id":"318647","messageId":"5909CD36020000A100025CC7@gwsmtp1.uni-regensburg.de","threadId":"45561","inReplyTo":"CAJZjrdVZ7gTZvm=CcG7pUPWtXjsMsHgyMRiE8xoXm=bZ4j6FEQ@mail.gmail.com","subject":"Antw: Re: Terrible bad performance for it blame --date=iso -C -C master -- ","fromName":"Ulrich Windl","fromEmail":"ulrich.windl@rz.uni-regensburg.de","sentAt":"2017-05-03T12:29:42Z","receivedAt":"2017-05-03T12:29:51Z","isPatch":false,"sender":{"key":"ulrich.windl@rz.uni-regensburg.de","avatar":null},"body":">>> Samuel Lijin schrieb am 03.05.2017 um 09:12 in Nachricht\n:\n> On Mon, May 1, 2017 at 7:59 PM, Junio C Hamano wrote:\n>> Samuel Lijin writes:\n>>\n>>> On Fri, Mar 31, 2017 at 10:52 AM, Junio C Hamano wrote:\n>>>\n>>>> It might not be a bad idea to teach \"blame\" not to pay attention to\n>>>> any path that is marked as \"-diff\" (e.g. binary files) when trying\n>>>> to see if remaining contents appeared by borrowing from them. We do\n>>>> not have that heuristics (yet).\n>>>\n>>> Could you elaborate on this? Do you mean to tell diffcore-rename to\n>>> ignore diff_filespec objects if they're binary?\n>>\n>> No and yes ;-). I do not think it is a good idea to unconditionally\n>> ignore binary in diffcore-rename.\n>>\n>> But when we know that the rename detection is called from inside\n>> blame.c, where by definition we would be digging line-oriented\n>> contents, there is no point checking if the contents we are looking\n>> for came from an existing binary file.\n> \n> A followup thought: perhaps it would make sense for blame.c rename\n> detection to only consider files with the same extension?\n\nAs traditionally file type and file name extension had little to do with each other in UNIX, I thing that would not be a good solution (to assume the opposite). I also know that Git does not care too much about hierarchies also, but my idea was so find candidates \"nearby\", i.e. look in the subdirectories of the first level and in the parent directory; then, if needed, widen the radius of the search...\n\nRegards,\nUlrich\n\n\n\n\n\n"}]}