{"thread":{"id":"6766","subject":"how to speed up \"git log\"?","startedAt":"2007-02-11T11:52:28Z","lastAt":"2007-02-18T06:33:51Z","messageCount":23,"participants":["Bruno Haible","Johannes Schindelin","Shawn O. Pearce","Robin Rosenberg","Junio C Hamano","Linus Torvalds"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"34160","messageId":"200702111252.28393.bruno@clisp.org","threadId":"6766","inReplyTo":null,"subject":"how to speed up \"git log\"?","fromName":"Bruno Haible","fromEmail":"bruno@clisp.org","sentAt":"2007-02-11T11:52:28Z","receivedAt":"2007-02-11T11:52:28Z","isPatch":false,"sender":{"key":"bruno@clisp.org","avatar":null},"body":"Hi,\n\nAre there some known tricks to speed up the operation of \"git log\"?\n\nOn a file in a local copy of the coreutils git repository,\n\"git log tr.c > output\" takes\n - 33 seconds of CPU time (33 user, 0 system) on a Linux/x86 500MHz system,\n - 24 seconds of CPU time (12 user, 12 system) on a MacOS X PowerPC 1.1 GHz\n system.\nThe result shows only 147 commits and a total of 40 KB textual output.\n\n1) Why so much user CPU time?\n2) Why so much system CPU time, but only on MacOS X?\n\nBruno\n"},{"id":"34174","messageId":"Pine.LNX.4.63.0702111745170.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6766","inReplyTo":"200702111252.28393.bruno@clisp.org","subject":"Re: how to speed up \"git log\"?","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-02-11T16:49:03Z","receivedAt":"2007-02-11T16:49:03Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sun, 11 Feb 2007, Bruno Haible wrote:\n\n> Are there some known tricks to speed up the operation of \"git log\"?\n> \n> On a file in a local copy of the coreutils git repository,\n> \"git log tr.c > output\" takes\n> - 33 seconds of CPU time (33 user, 0 system) on a Linux/x86 500MHz system,\n> - 24 seconds of CPU time (12 user, 12 system) on a MacOS X PowerPC 1.1 GHz\n> system.\n> The result shows only 147 commits and a total of 40 KB textual output.\n\nYes, because there were only 147 commits which changed the file. But git \nlooked at all commits to find that.\n\nBasically, we don't do file versions. File versions do not make sense, \nsince they strip away the context. See also\n\nhttp://news.gmane.org/group/gmane.comp.version-control.git/thread=37838\n\nfor a real flamewar revolving around that very subject.\n\n> 1) Why so much user CPU time?\n\nSee above.\n\nPlus, you are probably not really interested in _all_ revisions changing \nthat file, are you? Usually the output of git-log -- even with pathname \nfiltering -- starts almost instantaneous, and is piped to your pager. So, \nyour numbers are misleading.\n\n> 2) Why so much system CPU time, but only on MacOS X?\n\nProbably the mmap() problem. Does it go away when you use git 1.5.0-rc4?\n\nHth,\nDscho\n"},{"id":"34203","messageId":"20070211230035.GD31488@spearce.org","threadId":"6766","inReplyTo":"Pine.LNX.4.63.0702111745170.22628@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: how to speed up \"git log\"?","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-02-11T23:00:35Z","receivedAt":"2007-02-11T23:00:35Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Johannes Schindelin wrote:\n> > 1) Why so much user CPU time?\n> \n> See above.\n\nSome of the ideas Nico and I have kicked around for a pack v4 (post\n1.5.0, obviously) would speed up revision traversal by bypassing\nsome of the costly decompression overheads.\n \n> > 2) Why so much system CPU time, but only on MacOS X?\n> \n> Probably the mmap() problem. Does it go away when you use git 1.5.0-rc4?\n\nWhat does 1.5.0-rc4 do here that didn't happen before? Are you\nreferring to the mmap sliding window? Because NO_MMAP might be\nfaster on MacOS X then using mmap (thanks to its slower mmap)... but\nI can't say I have performance tested it either way.\n\n-- \nShawn.\n"},{"id":"34205","messageId":"Pine.LNX.4.63.0702120003450.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6766","inReplyTo":"20070211230035.GD31488@spearce.org","subject":"Re: how to speed up \"git log\"?","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-02-11T23:08:31Z","receivedAt":"2007-02-11T23:08:31Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sun, 11 Feb 2007, Shawn O. Pearce wrote:\n\n> Johannes Schindelin wrote:\n> > > 1) Why so much user CPU time?\n> > \n> > See above.\n> \n> Some of the ideas Nico and I have kicked around for a pack v4 (post\n> 1.5.0, obviously) would speed up revision traversal by bypassing\n> some of the costly decompression overheads.\n\nMaybe. But my point (which you did not quote) was this: git log _starts_ \nvery fast, and the information you are most likely after is shown right \naway. So I don't think it makes sense investing much time to enhance \nperformance for a full log.\n\n> > > 2) Why so much system CPU time, but only on MacOS X?\n> > \n> > Probably the mmap() problem. Does it go away when you use git 1.5.0-rc4?\n> \n> What does 1.5.0-rc4 do here that didn't happen before? Are you\n> referring to the mmap sliding window?\n\nNo. I was referring to v1.5.0-rc0~62, but was too lazy to look that up.\n\nCiao,\nDscho\n"},{"id":"34213","messageId":"200702120041.27419.bruno@clisp.org","threadId":"6766","inReplyTo":"Pine.LNX.4.63.0702111745170.22628@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: how to speed up \"git log\"?","fromName":"Bruno Haible","fromEmail":"bruno@clisp.org","sentAt":"2007-02-11T23:41:27Z","receivedAt":"2007-02-11T23:41:27Z","isPatch":false,"sender":{"key":"bruno@clisp.org","avatar":null},"body":"Hello Johannes,\n\nThanks for the helpful answer.\n\n> Yes, because there were only 147 commits which changed the file. But git \n> looked at all commits to find that.\n\nOuch.\n\n> Basically, we don't do file versions. File versions do not make sense, \n> since they strip away the context.\n\nIs there some other concept or command that git offers? I'm in the situation\nwhere I know that 'tr' in coreutils version 5.2.1 had a certain bug and\nversion 6.4 does not have the bug, and I want to review all commits that\nare relevant to this. I know that the only changes in tr.c are relevant\nfor this, and I'm interested in a display of the minimum amount of relevant\ncommit messages. If \"git log\" is not the right command for this question,\nwhich command is it?\n\n> > 2) Why so much system CPU time, but only on MacOS X?\n> \n> Probably the mmap() problem. Does it go away when you use git 1.5.0-rc4?\n\nNo, it became even worse: git-1.5.0-rc4 is twice as slow as git-1.4.4 for\nthis command:\n git-1.4.4: 25 seconds real time, 24 seconds of CPU time (12 user, 12 system)\n git-1.5.0: 50 seconds real time, 39 seconds of CPU time (20 user, 19 system)\n\nBruno\n"},{"id":"34215","messageId":"20070211234649.GG31488@spearce.org","threadId":"6766","inReplyTo":"200702120041.27419.bruno@clisp.org","subject":"Re: how to speed up \"git log\"?","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-02-11T23:46:49Z","receivedAt":"2007-02-11T23:46:49Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Bruno Haible wrote:\n> Is there some other concept or command that git offers? I'm in the situation\n> where I know that 'tr' in coreutils version 5.2.1 had a certain bug and\n> version 6.4 does not have the bug, and I want to review all commits that\n> are relevant to this. I know that the only changes in tr.c are relevant\n> for this, and I'm interested in a display of the minimum amount of relevant\n> commit messages. If \"git log\" is not the right command for this question,\n> which command is it?\n\nTwo options come to mind:\n\n `git log v5.2.1..v6.4 -- tr.c`\n `git bisect`\n\nThe former has a few different flavors, e.g. you can run the\nsame arguments to `gitk` to view the changes in a graphical form.\nThe latter will help you do a binary search through the commits\nwhich affected tr.c between the known good and known bad revisions,\nallowing you to test the possible candidates for the defect.\n \n> > > 2) Why so much system CPU time, but only on MacOS X?\n> > \n> > Probably the mmap() problem. Does it go away when you use git 1.5.0-rc4?\n> \n> No, it became even worse: git-1.5.0-rc4 is twice as slow as git-1.4.4 for\n> this command:\n> git-1.4.4: 25 seconds real time, 24 seconds of CPU time (12 user, 12 system)\n> git-1.5.0: 50 seconds real time, 39 seconds of CPU time (20 user, 19 system)\n\nThat's not so good... This is `git log -- tr.c >/dev/null` ?\n\n-- \nShawn.\n"},{"id":"34214","messageId":"200702120052.23468.bruno@clisp.org","threadId":"6766","inReplyTo":"20070211152840.GA2781@steel.home","subject":"Re: how to speed up \"git log\"?","fromName":"Bruno Haible","fromEmail":"bruno@clisp.org","sentAt":"2007-02-11T23:52:23Z","receivedAt":"2007-02-11T23:52:23Z","isPatch":false,"sender":{"key":"bruno@clisp.org","avatar":null},"body":"Alex Riesen wrote:\n> - do not use \"tr.c\", unless you really need it: git has to read more\n> of a commit in this case. Just \"git log\" takes only 0.9 sec on the\n> machine above.\n\n\"git log\" is indeed faster, but is useless for the given task, since it doesn't\nshow which of the 4 megabytes of commit messages apply to tr.c.\n\n> > On a file in a local copy of the coreutils git repository,\n> > \"git log tr.c > output\" takes\n> \n> Why do you need _all_ commits, btw?\n\nI want to quickly find the cause of a behaviour change between tr.c of\ncoreutils 5.2.1 and the one of coreutils 6.4. It's a period of 1.5 years,\nbut limited to a single file. Can't git produce this quickly?\n\n> > 2) Why so much system CPU time, but only on MacOS X?\n> \n> MacOS X is famous for its bad perfomance when doing serious work.\n> The mmap(2) of it, in particular.\n\nBut at least, a MacOS X machine is still interactively usable when it uses\n6 times more swap than the machine's RAM size. Whereas a Linux 2.4 machine\nis interactively unusable already with 1.5 to 2 times more swap than the\nmachine has RAM.\n\nBruno\n"},{"id":"34219","messageId":"Pine.LNX.4.63.0702120051240.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6766","inReplyTo":"200702120041.27419.bruno@clisp.org","subject":"Re: how to speed up \"git log\"?","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-02-11T23:56:36Z","receivedAt":"2007-02-11T23:56:36Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 12 Feb 2007, Bruno Haible wrote:\n\n> > Yes, because there were only 147 commits which changed the file. But git \n> > looked at all commits to find that.\n> \n> Ouch.\n\nNot so ouch:\n\n> > Basically, we don't do file versions. File versions do not make sense, \n> > since they strip away the context.\n\nYou could have it faster, but you'd break a very useful concept by doing \nso.\n\n> Is there some other concept or command that git offers? I'm in the \n> situation where I know that 'tr' in coreutils version 5.2.1 had a \n> certain bug and version 6.4 does not have the bug, and I want to review \n> all commits that are relevant to this.\n\nSo, only look at those:\n\n\tgit log v5.2.1..v6.4 tr.c\n\n(provided you have the tags for the releases). You can start reviewing \nright away, since the output will start very fast (much faster than it \ntakes to complete the log!).\n\nIf you want to get the patches to tr.c with the logs, just add \"-p\":\n\n\tgit log -p v5.2.1..v6.4 tr.c\n\n> > > 2) Why so much system CPU time, but only on MacOS X?\n> > \n> > Probably the mmap() problem. Does it go away when you use git \n> > 1.5.0-rc4?\n> \n> No, it became even worse: git-1.5.0-rc4 is twice as slow as git-1.4.4 for\n> this command:\n> git-1.4.4: 25 seconds real time, 24 seconds of CPU time (12 user, 12 system)\n> git-1.5.0: 50 seconds real time, 39 seconds of CPU time (20 user, 19 system)\n\nHmmm. I don't have MacOSX any more, so I cannot investigate. You might \nfind this the perfect opening into working on git ;-)\n\nHth,\nDscho\n"},{"id":"34221","messageId":"200702120059.17676.robin.rosenberg.lists@dewire.com","threadId":"6766","inReplyTo":"200702120041.27419.bruno@clisp.org","subject":"Re: how to speed up \"git log\"?","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2007-02-11T23:59:17Z","receivedAt":"2007-02-11T23:59:17Z","isPatch":false,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"måndag 12 februari 2007 00:41 skrev Bruno Haible:\n> Hello Johannes,\n> \n> Thanks for the helpful answer.\n> \n> > Yes, because there were only 147 commits which changed the file. But git \n> > looked at all commits to find that.\n> \n> Ouch.\n> \n> > Basically, we don't do file versions. File versions do not make sense, \n> > since they strip away the context.\n> \n> Is there some other concept or command that git offers? I'm in the situation\n> where I know that 'tr' in coreutils version 5.2.1 had a certain bug and\n> version 6.4 does not have the bug, and I want to review all commits that\n> are relevant to this. I know that the only changes in tr.c are relevant\n> for this, and I'm interested in a display of the minimum amount of relevant\n> commit messages. If \"git log\" is not the right command for this question,\n> which command is it?\n\nSince you know that you are not interested in the whole history, you can limit your scan.\n\ngit log COREUTILS-5_2_1..COREUTILS-6_4 src/tr.c\n\n> > > 2) Why so much system CPU time, but only on MacOS X?\n> > \n> > Probably the mmap() problem. Does it go away when you use git 1.5.0-rc4?\n> \n> No, it became even worse: git-1.5.0-rc4 is twice as slow as git-1.4.4 for\n> this command:\n> git-1.4.4: 25 seconds real time, 24 seconds of CPU time (12 user, 12 system)\n> git-1.5.0: 50 seconds real time, 39 seconds of CPU time (20 user, 19 system)\n\nCould the UTF-8 stuff have anything to do with this?\n\n-- robin\n"},{"id":"34230","messageId":"200702120302.00576.bruno@clisp.org","threadId":"6766","inReplyTo":"200702120059.17676.robin.rosenberg.lists@dewire.com","subject":"Re: how to speed up \"git log\"?","fromName":"Bruno Haible","fromEmail":"bruno@clisp.org","sentAt":"2007-02-12T02:02:00Z","receivedAt":"2007-02-12T02:02:00Z","isPatch":false,"sender":{"key":"bruno@clisp.org","avatar":null},"body":"Thanks for the responses.\n\nRobin Rosenberg wrote:\n> Since you know that you are not interested in the whole history, you can limit your scan.\n> \n> git log COREUTILS-5_2_1..COREUTILS-6_4 src/tr.c\n\nThanks, that indeed does the trick: it reduces the time from 33 sec to 11 sec.\n\nTo reduce the time even more, and to allow more flexibility among the\nsearch criteria (e.g. \"I need the commits from date X to date Y, on this\nfile set, from anyone except me\"), I would need to connect git to a database.\ngit cannot store all kinds of indices and reverse mappings to allow all\nkinds of queries; that's really a classical database application area.\n\n> > No, it became even worse: git-1.5.0-rc4 is twice as slow as git-1.4.4 for\n> > this command:\n> > git-1.4.4: 25 seconds real time, 24 seconds of CPU time (12 user, 12 system)\n> > git-1.5.0: 50 seconds real time, 39 seconds of CPU time (20 user, 19 system)\n> \n> Could the UTF-8 stuff have anything to do with this?\n\nActually, no. Brown paper bag on me for doing benches in different\nconditions. The timing difference is an effect of the buffer cache / page\ncache:\n\n - After the second repetition of the command (i.e. when all files are cached\n in RAM), the timings are\n 25 seconds real time, 24 seconds of CPU time (13 user, 11 system)\n both in git-1.4.4 and -1.5.0-rc4.\n\n - After unmounting and remounting the disk containing the repository (i.e.\n when none of the files are cached in RAM), the timings are\n 49 seconds real time, 38 seconds of CPU time (20 user, 18 system)\n\nSorry for the false alarm.\n\nBruno\n"},{"id":"34237","messageId":"7vmz3kaugq.fsf@assigned-by-dhcp.cox.net","threadId":"6766","inReplyTo":"200702120059.17676.robin.rosenberg.lists@dewire.com","subject":"Re: how to speed up \"git log\"?","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-02-12T04:08:37Z","receivedAt":"2007-02-12T04:08:37Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Robin Rosenberg writes:\n\n>> No, it became even worse: git-1.5.0-rc4 is twice as slow as git-1.4.4 for\n>> this command:\n>> git-1.4.4: 25 seconds real time, 24 seconds of CPU time (12 user, 12 system)\n>> git-1.5.0: 50 seconds real time, 39 seconds of CPU time (20 user, 19 system)\n>\n> Could the UTF-8 stuff have anything to do with this?\n\nI doubt it -- sliding mmap() in the current git, while is a good\nchange overall for handling really huge repos, would most likely\nperform poorer than the fixed mmap() in 1.4.4 series on\nplatforms with slow mmap(), most notably on MacOS X.\n\nIt _might_ be possible that turning some sliding mmap() calls\ninto pread() makes it perform better on MacOS X.\n\nI wonder what happens it git is compiled with NO_MMAP there...\n"},{"id":"34239","messageId":"Pine.LNX.4.64.0702112009440.8424@woody.linux-foundation.org","threadId":"6766","inReplyTo":"200702120041.27419.bruno@clisp.org","subject":"Re: how to speed up \"git log\"?","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-02-12T04:20:43Z","receivedAt":"2007-02-12T04:20:43Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Mon, 12 Feb 2007, Bruno Haible wrote:\n\n> Hello Johannes,\n> \n> Thanks for the helpful answer.\n> \n> > Yes, because there were only 147 commits which changed the file. But git \n> > looked at all commits to find that.\n> \n> Ouch.\n\nThis should become a FAQ.\n\nGit simply DOES NOT HAVE per-file history. And having it is actually a \nBUG in other systems.\n\nNot having per-file history is what allows git to do\n\n\tgit log directory-or-file-set\n\nratehr than being able to track just one file. You can't do it sanely \nwith per-file history (because to tie the per-file histories back \ntogether in a logical sequence, you need the global history to sort it \nagain!)\n\nSo:\n\n - git is \"slow\" on single-file things, because such things DON'T EVEN \n EXIST in git!\n\n When you do \"git log \", itreally always ends up being a \n full git log. \n\n - but this is fundamentally what allows you to track multiple directories \n well. It's what makes things like \"gitk drivers/scsi/\" actually work, \n where you really can see the history for a random *collection* of \n files. Nobody else can do it, afaik, and git just considers a single \n filename to be a case of the \"random collection of files\".\n\nThe example I gave to corecode was to do\n\n\tgitk builtin-rev-list.c\n\tgitk builtin-rev-parse.c\n\tgitk builtin-rev-parse.c builtin-rev-list.c\n\nadn realize that doing the history for two files together is NOT AT ALL \nEQUIVALENT to doing the history for those files individually and stitching \nit together.\n\n(The reason the above is a great example is that both of the files alone \nhave a very simple linear history, but when you look at the *combined* \nhistory you actually see concurrent development, and merges: you see \nmerge commits that simply don't \"exist\" when only looking at the history \nof one of them separately).\n\n> Is there some other concept or command that git offers? I'm in the situation\n> where I know that 'tr' in coreutils version 5.2.1 had a certain bug and\n> version 6.4 does not have the bug, and I want to review all commits that\n> are relevant to this. I know that the only changes in tr.c are relevant\n> for this, and I'm interested in a display of the minimum amount of relevant\n> commit messages. If \"git log\" is not the right command for this question,\n> which command is it?\n\nDo\n\n\tgit log v5.2.1..v6.4 -- tr.c\n\n(or whatever your tag-names for releases are) where you can limit the log \ngeneration cost by giving the beginning commit. But yeah, it *will* look \nat the whole history in between, so if there is a long long history \nbetween v5.2.1 and v6.4, you'll still end up using reasonable amounts of \nCPU.\n\n> > Probably the mmap() problem. Does it go away when you use git 1.5.0-rc4?\n> \n> No, it became even worse: git-1.5.0-rc4 is twice as slow as git-1.4.4 for\n> this command:\n> git-1.4.4: 25 seconds real time, 24 seconds of CPU time (12 user, 12 system)\n> git-1.5.0: 50 seconds real time, 39 seconds of CPU time (20 user, 19 system)\n\nThat's an interesting fact in itself. Do you have the repo available \nsomewhere?\n\nYes, some of the operations can be improved upon by not wasting quite so \nmuch time uncompressing stuff, so we could at least help this a bit. But \nthat's a long-term thing. The slowdown is bad, and that probably has some \nsimple explanation.\n\n\t\tLinus\n"},{"id":"34253","messageId":"20070212060641.GC699@spearce.org","threadId":"6766","inReplyTo":"7vmz3kaugq.fsf@assigned-by-dhcp.cox.net","subject":"Re: how to speed up \"git log\"?","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-02-12T06:06:41Z","receivedAt":"2007-02-12T06:06:41Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano wrote:\n> I doubt it -- sliding mmap() in the current git, while is a good\n> change overall for handling really huge repos, would most likely\n> perform poorer than the fixed mmap() in 1.4.4 series on\n> platforms with slow mmap(), most notably on MacOS X.\n> \n> It _might_ be possible that turning some sliding mmap() calls\n> into pread() makes it perform better on MacOS X.\n> \n> I wonder what happens it git is compiled with NO_MMAP there...\n\nSo I ran three trials, v1.5.0-rc4-26-gcc46a74 with and without\nNO_MMAP against v1.4.4.4 on a freshly repacked git.git.\n\nv150-mmap:\n 3.33 real 3.12 user 0.05 sys\n 3.32 real 3.12 user 0.05 sys\n 3.34 real 3.12 user 0.05 sys\n\nv150-nommap:\n 3.46 real 3.13 user 0.16 sys\n 3.43 real 3.13 user 0.16 sys\n 3.46 real 3.13 user 0.16 sys\n\nv1444-mmap:\n 3.30 real 3.09 user 0.05 sys\n 3.30 real 3.09 user 0.05 sys\n 3.25 real 3.09 user 0.04 sys\n\nCFLAGS=\"-O2\"; the above timings are three representative samples\nout of 10 runs each, all hot cache.\n\nClearly the sliding mmap window isn't hurting us in this case by\nvery much, and NO_MMAP really isn't helping matters at all.\n\n-- \nShawn.\n"},{"id":"34255","messageId":"7vmz3jaorx.fsf@assigned-by-dhcp.cox.net","threadId":"6766","inReplyTo":"20070212060641.GC699@spearce.org","subject":"Re: how to speed up \"git log\"?","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-02-12T06:11:30Z","receivedAt":"2007-02-12T06:11:30Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n> Junio C Hamano wrote:\n>> I doubt it -- sliding mmap() in the current git, while is a good\n>> change overall for handling really huge repos, would most likely\n>> perform poorer than the fixed mmap() in 1.4.4 series on\n>> platforms with slow mmap(), most notably on MacOS X.\n>> \n>> It _might_ be possible that turning some sliding mmap() calls\n>> into pread() makes it perform better on MacOS X.\n>> \n>> I wonder what happens it git is compiled with NO_MMAP there...\n>\n> So I ran three trials, v1.5.0-rc4-26-gcc46a74 with and without\n> NO_MMAP against v1.4.4.4 on a freshly repacked git.git.\n\nI do not think freshly repacked git.git is a good test case for\na real-world workload where this really matters. Doesn't your\ndefault pack window large enough to cover it with a single\nwindow, or perhaps two at most?\n"},{"id":"34256","messageId":"20070212062224.GD699@spearce.org","threadId":"6766","inReplyTo":"7vmz3jaorx.fsf@assigned-by-dhcp.cox.net","subject":"Re: how to speed up \"git log\"?","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-02-12T06:22:24Z","receivedAt":"2007-02-12T06:22:24Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano wrote:\n> \"Shawn O. Pearce\" writes:\n> > So I ran three trials, v1.5.0-rc4-26-gcc46a74 with and without\n> > NO_MMAP against v1.4.4.4 on a freshly repacked git.git.\n> \n> I do not think freshly repacked git.git is a good test case for\n> a real-world workload where this really matters. Doesn't your\n> default pack window large enough to cover it with a single\n> window, or perhaps two at most?\n\nIts one window, maybe two, as git.git is ~12 MiB and the window\nsize is 1 MiB (NO_MMAP) or 32 MiB (with mmap).\n\nOn linux.git:\n\nv150-mmap:\n 2.23 real 1.99 user 0.10 sys\n 2.19 real 1.98 user 0.10 sys\n 2.19 real 1.98 user 0.10 sys\n\nv150-nommap:\n 2.63 real 1.99 user 0.50 sys\n 2.67 real 1.98 user 0.51 sys\n 2.63 real 1.99 user 0.51 sys\n\nv1444:\n 2.15 real 1.94 user 0.09 sys\n 2.19 real 1.95 user 0.10 sys\n 2.16 real 1.94 user 0.10 sys\n\nAgain, we aren't too far away from v1.4.4.4, but the NO_MMAP clearly\nis hurting us, even on Mac OS X.\n\n-- \nShawn.\n"},{"id":"34258","messageId":"20070212062813.GE699@spearce.org","threadId":"6766","inReplyTo":"20070212062224.GD699@spearce.org","subject":"Re: how to speed up \"git log\"?","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-02-12T06:28:13Z","receivedAt":"2007-02-12T06:28:13Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"\"Shawn O. Pearce\" wrote:\n> Junio C Hamano wrote:\n> > \"Shawn O. Pearce\" writes:\n> > > So I ran three trials, v1.5.0-rc4-26-gcc46a74 with and without\n> > > NO_MMAP against v1.4.4.4 on a freshly repacked git.git.\n\nI probably should have mentioned, my run (in all cases) was:\n\n\tgit rev-list HEAD -- Makefile 2>/dev/null\n\ncheap, a file that exists pretty much everywhere, and that triggers\nthe path limiter in the revision walking code.\n\nBTW, I discovered by accident tonight that this works:\n\n\tcp git-rev-list ../git-1444\n\t../git-1444 rev-list\n\nwhich is so not something I would have expected. :-) I honestly\nexpected the wrapper to puke and say it doesn't know what command\n1444 is.\n\n-- \nShawn.\n"},{"id":"34272","messageId":"Pine.LNX.4.63.0702121215190.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6766","inReplyTo":"200702120302.00576.bruno@clisp.org","subject":"Re: how to speed up \"git log\"?","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-02-12T11:19:24Z","receivedAt":"2007-02-12T11:19:24Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 12 Feb 2007, Bruno Haible wrote:\n\n> Robin Rosenberg wrote:\n> > Since you know that you are not interested in the whole history, you can limit your scan.\n> > \n> > git log COREUTILS-5_2_1..COREUTILS-6_4 src/tr.c\n> \n> Thanks, that indeed does the trick: it reduces the time from 33 sec to 11 sec.\n> \n> To reduce the time even more, and to allow more flexibility among the \n> search criteria (e.g. \"I need the commits from date X to date Y, on this \n> file set, from anyone except me\"), I would need to connect git to a \n> database. git cannot store all kinds of indices and reverse mappings to \n> allow all kinds of queries; that's really a classical database \n> application area.\n\n[in the following paragraph, \"index\" means the index on a classical \ndatabase table]\n\nAnd -- as everywhere else with classical databases -- you have to ask if \nit is worth it. Given the fact that a one-time use of such an index is \n_worse_ than doing it without index at all (building and writing the \nindex is _at least_ as expensive as searching once without an index), I'd \nrather doubt it.\n\nHowever, if you do similar kinds of searches quite often, it makes tons of \nsense to connect to a database. We already use sqlite in cvsserver, so I'd \ntry that.\n\nCiao,\nDscho\n"},{"id":"34274","messageId":"200702121227.15757.bruno@clisp.org","threadId":"6766","inReplyTo":"Pine.LNX.4.64.0702112009440.8424@woody.linux-foundation.org","subject":"Re: how to speed up \"git log\"?","fromName":"Bruno Haible","fromEmail":"bruno@clisp.org","sentAt":"2007-02-12T11:27:15Z","receivedAt":"2007-02-12T11:27:15Z","isPatch":false,"sender":{"key":"bruno@clisp.org","avatar":null},"body":"Linus,\n\n> > git-1.4.4: 25 seconds real time, 24 seconds of CPU time (12 user, 12 system)\n> > git-1.5.0: 50 seconds real time, 39 seconds of CPU time (20 user, 19 system)\n> \n> That's an interesting fact in itself.\n\nSorry, these measurements happened to be done in different conditions:\nrepo fully cached in RAM vs. repo not yet in buffer cache / page cache.\n\nWhen measured under the same conditions, no speed difference is visible\nbetween git-1.4.4 and git-1.5.0-rc4.\n\nBruno\n"},{"id":"34890","messageId":"200702172019.20536.bruno@clisp.org","threadId":"6766","inReplyTo":"20070211152840.GA2781@steel.home","subject":"Re: how to speed up \"git log\"?","fromName":"Bruno Haible","fromEmail":"bruno@clisp.org","sentAt":"2007-02-17T19:19:20Z","receivedAt":"2007-02-17T19:19:20Z","isPatch":false,"sender":{"key":"bruno@clisp.org","avatar":null},"body":"Alex Riesen wrote:\n> MacOS X is famous for its bad perfomance when doing serious work.\n> The mmap(2) of it, in particular.\n\nYou can't blame MacOS X mmap(2) for git's slow execution of \"git log\".\nHere are is execution times of \"git log tr.c > output\"\n\n - with git-1.5.0-rc4 built with -DNO_MMAP\n\n real 0m26.032s\n user 0m13.580s\n sys 0m11.730s\n\n - with git-1.5.0-rc4 built with the default settings:\n\n real 0m25.469s\n user 0m13.530s\n sys 0m11.490s\n\nYou can see that using mmap() provides a speedup of about 2% on MacOS X,\nwhich is similar to the 4% than Shawn measured on Linux.\n\nBruno\n"},{"id":"34902","messageId":"Pine.LNX.4.63.0702180019040.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6766","inReplyTo":"200702172019.20536.bruno@clisp.org","subject":"Re: how to speed up \"git log\"?","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-02-17T23:20:49Z","receivedAt":"2007-02-17T23:20:49Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sat, 17 Feb 2007, Bruno Haible wrote:\n\n> Alex Riesen wrote:\n> > MacOS X is famous for its bad perfomance when doing serious work.\n> > The mmap(2) of it, in particular.\n> \n> You can't blame MacOS X mmap(2) for git's slow execution of \"git log\".\n\nNo, but you can blame the person calling git log and waiting until it \nfinishes. See the list archives for reasons why.\n\nIf this comes up one more time, I'm very tempted to write a scathing \nremark in the FAQ.\n\nCiao,\nDscho\n"},{"id":"34911","messageId":"200702180109.26412.bruno@clisp.org","threadId":"6766","inReplyTo":"Pine.LNX.4.63.0702180019040.22628@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: piped to a pager (was: how to speed up \"git log\"?)","fromName":"Bruno Haible","fromEmail":"bruno@clisp.org","sentAt":"2007-02-18T00:09:26Z","receivedAt":"2007-02-18T00:09:26Z","isPatch":false,"sender":{"key":"bruno@clisp.org","avatar":null},"body":"Johannes Schindelin wrote:\n> you can blame the person calling git log and waiting until it \n> finishes. See the list archives for reasons why.\n... and earlier:\n> Usually the output of git-log -- even with pathname \n> filtering -- starts almost instantaneous, and is piped to your pager.\n\nThe pager ('less') in a console is not a good solution for everone:\n - People used to GUI editors (kate, nedit, ...) miss a scroll bar for\n navigation. You can't use kate or nedit as a pager.\n - PAGER=\"vi -\" also reads all input before it displays anything.\n - PAGER=\"xless\" likewise.\n - In Emacs shell-mode, with PAGER=\"\", you see the output as it is produced,\n but it's disturbing to work in a buffer which is growing, where the scrollbar\n continues to change its position.\n\nIt's OK for many people, but not for everyone.\n\nBruno\n"},{"id":"34916","messageId":"Pine.LNX.4.63.0702180109400.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6766","inReplyTo":"200702180109.26412.bruno@clisp.org","subject":"Re: piped to a pager (was: how to speed up \"git log\"?)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-02-18T00:10:00Z","receivedAt":"2007-02-18T00:10:00Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sun, 18 Feb 2007, Bruno Haible wrote:\n\n> Johannes Schindelin wrote:\n> > you can blame the person calling git log and waiting until it \n> > finishes. See the list archives for reasons why.\n> ... and earlier:\n> > Usually the output of git-log -- even with pathname \n> > filtering -- starts almost instantaneous, and is piped to your pager.\n> \n> The pager ('less') in a console is not a good solution for everone:\n> - People used to GUI editors (kate, nedit, ...) miss a scroll bar for\n> navigation. You can't use kate or nedit as a pager.\n> - PAGER=\"vi -\" also reads all input before it displays anything.\n> - PAGER=\"xless\" likewise.\n> - In Emacs shell-mode, with PAGER=\"\", you see the output as it is produced,\n> but it's disturbing to work in a buffer which is growing, where the scrollbar\n> continues to change its position.\n> \n> It's OK for many people, but not for everyone.\n\nSo why don't you go scratch that itch, and write a decent GUI pager?\n\nCiao,\nDscho\n"},{"id":"34938","messageId":"20070218063351.GB31350@spearce.org","threadId":"6766","inReplyTo":"200702172019.20536.bruno@clisp.org","subject":"Re: how to speed up \"git log\"?","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-02-18T06:33:51Z","receivedAt":"2007-02-18T06:33:51Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Bruno Haible wrote:\n> Alex Riesen wrote:\n> > MacOS X is famous for its bad perfomance when doing serious work.\n> > The mmap(2) of it, in particular.\n> \n> You can see that using mmap() provides a speedup of about 2% on MacOS X,\n> which is similar to the 4% than Shawn measured on Linux.\n\nUh, I was testing on Mac OS X (G4 PowerBook).\n\n-- \nShawn.\n"}]}