{"thread":{"id":"11420","subject":"[RFH] git-log vs git-rev-list performance","startedAt":"2007-12-29T12:18:52Z","lastAt":"2007-12-29T20:05:11Z","messageCount":3,"participants":["Marco Costalba","Linus Torvalds"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"64172","messageId":"e5bfff550712290418h22d7f7edqda519e7f4dcd25b8@mail.gmail.com","threadId":"11420","inReplyTo":null,"subject":"[RFH] git-log vs git-rev-list performance","fromName":"Marco Costalba","fromEmail":"mcostalba@gmail.com","sentAt":"2007-12-29T12:18:52Z","receivedAt":"2007-12-29T12:18:52Z","isPatch":false,"sender":{"key":"mcostalba@gmail.com","avatar":null},"body":"Hi all,\n\n   perhaps this will turn out to be a bit academic, anyway...\n\nI'm almost ready to release qgit-2.1, a lot of improvements went in\nbut still I cannot beat the speed of stable qgit-1.5 series.\n\nI really profiled this puppy and I think qgit-2.1 is _internally_ much\nfaster then qgit-1.5, but at the end of the day qgit-1.5 is about 13%\nfaster then the new 2.1 at loading repositories.\n\n\nThe loading speed is the sum of two factors:\n\n- speed of underlying git-rev-list / git-log command\n\n- qgit overhead in parsing and storing the git plumbing outpt\n\n\nI think qgit-2.0 code is much more efficient then the old one because\nthe overhead went down from 17% of qgit-1.5 to current 6% of qgit-2.1,\nit means that opening and loading a repository with qgit-2.1 is only\n6% slower then running the underlying git command from the command\nline.\n\n\nSo the problem seems to be in the underlying command that for qgit-1.5\nis git-rev-list while for new qgit-2.1 is git-log\n\n\nTo have some numbers I have tested on the Linux repository with the\nactual git commands used by the two versions of qgit:\n\n[marco@localhost linux-2.6]$ git --version\ngit version 1.5.4-rc1.GIT\n\n\n[marco@localhost linux-2.6]$ time git log --topo-order --no-color\n--parents --boundary -z --log-size\n--pretty=format:\"%m%HX%PX%n%an<%ae>%n%at%n%s%n%b\" HEAD > /dev/null\n3.60user 0.09system 0:03.70elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+27156minor)pagefaults 0swaps\n\n\n[marco@localhost linux-2.6]$ time git rev-list --topo-order --no-color\n--parents --boundary -z --header HEAD > /dev/null\n2.89user 0.08system 0:02.98elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+27156minor)pagefaults 0swaps\n\n\n[ BTW, yes I have a new laptop! ;-) ]\n\n\nNote that the output is smaller for git-log because --pretty=format\nasks for less info then the --header option used by git-rev-list. To\nbe precise, output is  57.113.153 bytes for git-rev-list against\n41.755.328 bytes for git-log.\n\n\n\nSo the bottom line is that git-log is 24% slower then git-rev-list\nalthough size of its output is 36% smaller!\n\n\nCould someone be so kind to explain me why these differences? I'm not\nso confident with git-log /git-rev-list internals.\n\n\nThanks\nMarco\n"},{"id":"64183","messageId":"alpine.LFD.0.9999.0712291038560.2778@woody.linux-foundation.org","threadId":"11420","inReplyTo":"e5bfff550712290418h22d7f7edqda519e7f4dcd25b8@mail.gmail.com","subject":"Re: [RFH] git-log vs git-rev-list performance","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-12-29T18:51:39Z","receivedAt":"2007-12-29T18:51:39Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sat, 29 Dec 2007, Marco Costalba wrote:\n>\n> [marco@localhost linux-2.6]$ time git log --topo-order --no-color\n> --parents --boundary -z --log-size\n> --pretty=format:\"%m%HX%PX%n%an<%ae>%n%at%n%s%n%b\" HEAD > /dev/null\n\nDon't compare \"--pretty=format\" to the pre-formatted versions.\n\nUse \"--pretty=raw\" for \"git log\" if you want to approximate \"git \nrev-list --header\". \n\nOr alternatively use the same \"--pretty=format:\" for git-rev-list.\n\nIf you start using anything else, you only have yourself to blame. OF \nCOURSE it's more expensive to pretty-format the messages.\n\nI get\n\n\t[torvalds@woody linux]$ time git rev-list \\\n\t\t--pretty=format:\"%m%HX%PX%n%an<%ae>%n%at%n%s%n%b\" \\\n\t\t--topo-order --parents --boundary --header \\\n\t\t--log-size HEAD\t> /dev/null\n\n\treal    0m1.596s\n\tuser    0m1.556s\n\tsys     0m0.040s\n\n\t[torvalds@woody linux]$ time git log \\\n\t\t--pretty=format:\"%m%HX%PX%n%an<%ae>%n%at%n%s%n%b\" \\\n\t\t--topo-order --parents --boundary \\\n\t\t--log-size HEAD > /dev/null\n\n\treal    0m1.597s\n\tuser    0m1.548s\n\tsys     0m0.048s\n\nso I'd say that with the same output, the timings are pretty much the \nsame (except \"git log\" is more capable - \"--log-size\" does nothing for \n\"git rev-list\", for example).\n\nBut if you ask for different formats, they'll have different performance, \neven if you then use the same command (ie \"git log\" will be slower than \n\"git log\" depending on the command line arguments!)\n\n\t\tLinus\n"},{"id":"64188","messageId":"e5bfff550712291205m56521977tfe380969e06e93d4@mail.gmail.com","threadId":"11420","inReplyTo":"alpine.LFD.0.9999.0712291038560.2778@woody.linux-foundation.org","subject":"Re: [RFH] git-log vs git-rev-list performance","fromName":"Marco Costalba","fromEmail":"mcostalba@gmail.com","sentAt":"2007-12-29T20:05:11Z","receivedAt":"2007-12-29T20:05:11Z","isPatch":false,"sender":{"key":"mcostalba@gmail.com","avatar":null},"body":"On Dec 29, 2007 7:51 PM, Linus Torvalds <torvalds@linux-foundation.org> wrote:\n>\n>\n> On Sat, 29 Dec 2007, Marco Costalba wrote:\n> >\n> > [marco@localhost linux-2.6]$ time git log --topo-order --no-color\n> > --parents --boundary -z --log-size\n> > --pretty=format:\"%m%HX%PX%n%an<%ae>%n%at%n%s%n%b\" HEAD > /dev/null\n>\n> Don't compare \"--pretty=format\" to the pre-formatted versions.\n>\n> Use \"--pretty=raw\" for \"git log\" if you want to approximate \"git\n> rev-list --header\".\n>\n\nI have switched to --pretty=format instead of preformatted one to save\nRAM, becuase needed memory is about 35% less with a custom format, the\npreformatted ones give me additional info that is not shown on qgit so\nit's just a waste.\n\nAs example a full Linux tree loaded with qgit takes less then 80MB,\nwith gitk as comparison we are above 400MB although of course the\noptimized format is not the whole reason for this difference.\n\nWhat I have seen looking expecially at the pretty.c sources with a\nprofiler is that the custom format is continuosly reparsed _for each\nrevision_ also if it never changes during the whole git-log run. This\ncould explain why the custom format although cheaper in terms of\nquantity of outputted data is slower then a preformatted one.\n\nA caching of the parsed custom --pretty=format at the beginning of\ngit-log could help...\n\nMarco\n"}]}