{"thread":{"id":"1220","subject":"Is cogito really this inefficient","startedAt":"2005-07-13T12:50:52Z","lastAt":"2005-07-19T23:54:52Z","messageCount":13,"participants":["Russell King","Matthias Urlichs","Linus Torvalds","Catalin Marinas","Junio C Hamano","Petr Baudis"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"6076","messageId":"20050713135052.C6791@flint.arm.linux.org.uk","threadId":"1220","inReplyTo":null,"subject":"Is cogito really this inefficient","fromName":"Russell King","fromEmail":"rmk@arm.linux.org.uk","sentAt":"2005-07-13T12:50:52Z","receivedAt":"2005-07-13T12:50:52Z","isPatch":false,"sender":{"key":"rmk@arm.linux.org.uk","avatar":null},"body":"This says it all.  1min 22secs to generate a patch from a locally\nmodified but uncommitted file.\n\ncp, edit, diff would be several orders of magnitude faster.  What's\ngoing on?\n\n$ /usr/bin/time cg-diff drivers/serial/8250.c > o\nCommand exited with non-zero status 1\n14.40user 17.47system 1:22.96elapsed 38%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (180major+14692minor)pagefaults 0swaps\n\ndiff --git a/drivers/serial/8250.c b/drivers/serial/8250.c\n--- a/drivers/serial/8250.c\n+++ b/drivers/serial/8250.c\n@@ -2333,6 +2333,7 @@ static int __devinit serial8250_probe(st\n \t\t\tdev_err(dev, \"unable to register port at index %d \"\n \t\t\t\t\"(IO%lx MEM%lx IRQ%d): %d\\n\", i,\n \t\t\t\tp->iobase, p->mapbase, p->irq, ret);\n+\t\t\tprintk(KERN_ERR \"uartclk was %d\\n\", port.uartclk);\n \t\t}\n \t}\n \treturn 0;\n\n-- \nRussell King\n"},{"id":"6078","messageId":"pan.2005.07.13.16.51.28.520681@smurf.noris.de","threadId":"1220","inReplyTo":"20050713135052.C6791@flint.arm.linux.org.uk","subject":"Re: Is cogito really this inefficient","fromName":"Matthias Urlichs","fromEmail":"smurf@smurf.noris.de","sentAt":"2005-07-13T16:51:30Z","receivedAt":"2005-07-13T16:51:30Z","isPatch":false,"sender":{"key":"matthias@urlichs.de","avatar":"https://gravatar.com/avatar/2708905af227313eba6f2b2ae0f7d0259b5ac5d71baef58fe5a13c699ce0bbf0?d=mp&s=160"},"body":"Hi, Russell King wrote:\n\n> This says it all.  1min 22secs to generate a patch from a locally\n> modified but uncommitted file.\n\nI only get that when the index is out-of-date WRT the file modification\ndates, so cg-diff has to examine every file.\n\nThe good news is that the index is being updated as it finds that the\nfiles are in sync, so expect this to be significantly faster the next time\naround.\n\n-- \nMatthias Urlichs   |   {M:U} IT Design @ m-u-it.de   |  smurf@smurf.noris.de\nDisclaimer: The quote was selected randomly. Really. | http://smurf.noris.de\n - -\nPraise the sea; on shore remain.\n\t\t-- John Florio\n"},{"id":"6087","messageId":"Pine.LNX.4.58.0507131325170.17536@g5.osdl.org","threadId":"1220","inReplyTo":"20050713135052.C6791@flint.arm.linux.org.uk","subject":"Re: Is cogito really this inefficient","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-07-13T20:28:18Z","receivedAt":"2005-07-13T20:28:18Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 13 Jul 2005, Russell King wrote:\n>\n> This says it all.  1min 22secs to generate a patch from a locally\n> modified but uncommitted file.\n\nNo, there's something else going on.\n\nMost likely that something forced a total index file re-validation, and\nthe time you see is every single checked out file having its SHA1\nre-computed.\n\nWas this a recently cloned tree, or what was the last operation you did on\nthat tree before that command? Something must have invalidated the index.\n\n\t\t\tLinus\n"},{"id":"6121","messageId":"20050714083700.A26322@flint.arm.linux.org.uk","threadId":"1220","inReplyTo":"Pine.LNX.4.58.0507131325170.17536@g5.osdl.org","subject":"Re: Is cogito really this inefficient","fromName":"Russell King","fromEmail":"rmk@arm.linux.org.uk","sentAt":"2005-07-14T07:37:00Z","receivedAt":"2005-07-14T07:37:00Z","isPatch":false,"sender":{"key":"rmk@arm.linux.org.uk","avatar":null},"body":"On Wed, Jul 13, 2005 at 01:28:18PM -0700, Linus Torvalds wrote:\n> On Wed, 13 Jul 2005, Russell King wrote:\n> > This says it all.  1min 22secs to generate a patch from a locally\n> > modified but uncommitted file.\n> \n> No, there's something else going on.\n> \n> Most likely that something forced a total index file re-validation, and\n> the time you see is every single checked out file having its SHA1\n> re-computed.\n> \n> Was this a recently cloned tree, or what was the last operation you did on\n> that tree before that command? Something must have invalidated the index.\n\ncg-update origin\nand then I edited drivers/serial/8250.c\n\nAs discovered using:\n\n\tsh -x /usr/bin/cg-diff drivers/serial/8250.c\n\nit appears that cg-diff does a\n\n\tgit-update-cache --refresh >/dev/null\n\neach time it's run, which is taking the bulk of the time.  Also note\nthat curiously, it exits with status 1.\n\n-- \nRussell King\n"},{"id":"6122","messageId":"20050714083831.B26322@flint.arm.linux.org.uk","threadId":"1220","inReplyTo":"pan.2005.07.13.16.51.28.520681@smurf.noris.de","subject":"Re: Is cogito really this inefficient","fromName":"Russell King","fromEmail":"rmk@arm.linux.org.uk","sentAt":"2005-07-14T07:38:31Z","receivedAt":"2005-07-14T07:38:31Z","isPatch":false,"sender":{"key":"rmk@arm.linux.org.uk","avatar":null},"body":"On Wed, Jul 13, 2005 at 06:51:30PM +0200, Matthias Urlichs wrote:\n> Hi, Russell King wrote:\n> \n> > This says it all.  1min 22secs to generate a patch from a locally\n> > modified but uncommitted file.\n> \n> I only get that when the index is out-of-date WRT the file modification\n> dates, so cg-diff has to examine every file.\n> \n> The good news is that the index is being updated as it finds that the\n> files are in sync, so expect this to be significantly faster the next time\n> around.\n\nIt isn't.  First time it was 1min11, second time _immediately_ after\nit was 1min22.  See my reply to Linus.\n\nOddly, show-diff seemed to be a lot more efficient in previous git\nrevisions.\n\n-- \nRussell King\n"},{"id":"6128","messageId":"tnxu0ixoiuo.fsf@arm.com","threadId":"1220","inReplyTo":"20050714083700.A26322@flint.arm.linux.org.uk","subject":"Re: Is cogito really this inefficient","fromName":"Catalin Marinas","fromEmail":"catalin.marinas@gmail.com","sentAt":"2005-07-14T09:08:31Z","receivedAt":"2005-07-14T09:08:31Z","isPatch":false,"sender":{"key":"catalin.marinas@gmail.com","avatar":null},"body":"Russell King <rmk@arm.linux.org.uk> wrote:\n> it appears that cg-diff does a\n>\n> \tgit-update-cache --refresh >/dev/null\n>\n> each time it's run, which is taking the bulk of the time.  Also note\n> that curiously, it exits with status 1.\n\nDoes git-ls-files --unmerged show any files?\n\n-- \nCatalin\n"},{"id":"6129","messageId":"20050714105938.A31383@flint.arm.linux.org.uk","threadId":"1220","inReplyTo":"tnxu0ixoiuo.fsf@arm.com","subject":"Re: Is cogito really this inefficient","fromName":"Russell King","fromEmail":"rmk@arm.linux.org.uk","sentAt":"2005-07-14T09:59:38Z","receivedAt":"2005-07-14T09:59:38Z","isPatch":false,"sender":{"key":"rmk@arm.linux.org.uk","avatar":null},"body":"On Thu, Jul 14, 2005 at 10:08:31AM +0100, Catalin Marinas wrote:\n> Russell King <rmk@arm.linux.org.uk> wrote:\n> > it appears that cg-diff does a\n> >\n> > \tgit-update-cache --refresh >/dev/null\n> >\n> > each time it's run, which is taking the bulk of the time.  Also note\n> > that curiously, it exits with status 1.\n> \n> Does git-ls-files --unmerged show any files?\n\nNo, and it returns fairly quickly:\n\n$ /usr/bin/time git-ls-files --unmerged\n0.29user 0.03system 0:00.43elapsed 73%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+655minor)pagefaults 0swaps\n\nActually, I should've left the sh -x /usr/bin/cg-diff drivers/serial/8250.c\nrunning a little longer.  It's not the git-update-cache command which\nis taking the time, it's git-diff-cache.\n\nRunning the diff several times, both with and without changes to\ndrivers/serial/8250.c, it seems that sometimes it's faster.  I guess\nit has to do with dentry invalidation...\n\nHowever, the point is - I've only asked for _one_ file.  Why do we need\nto look at _every_ file in the tree?\n\nI could understand this behaviour if I'd asked for a diff across the\nwhole tree, but I didn't.\n\nInternally, the sha1 of the unmodified drivers/serial/8250.c should be\nknown, so should be trivial to unpack that and generate a diff.  Given\nthe cache, this should be something which should be lightning fast\nwhen the requested fileset to diff is already known.\n\n-- \nRussell King\n"},{"id":"6133","messageId":"Pine.LNX.4.58.0507140813150.19183@g5.osdl.org","threadId":"1220","inReplyTo":"20050714083700.A26322@flint.arm.linux.org.uk","subject":"Re: Is cogito really this inefficient","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-07-14T15:26:07Z","receivedAt":"2005-07-14T15:26:07Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 14 Jul 2005, Russell King wrote:\n> \n> cg-update origin\n> and then I edited drivers/serial/8250.c\n\nHmm.. \n\n> it appears that cg-diff does a\n> \n> \tgit-update-cache --refresh >/dev/null\n> \n> each time it's run, which is taking the bulk of the time.  Also note\n> that curiously, it exits with status 1.\n\nThat part is normal - a update-cache is fast (it takes me 0.08 sec for the\nkernel) if the cache is already mostly up-to-date, and the non-zero exit\nstatus just means that some file was different (ie it's telling the caller\nthat there are edits in your tree - drivers/serial/8250.c).\n\nThe update-cache is slow only if the index isn't up-to-date, which can\nhappen either if somebody plays games with the index, or if somebody\ntouches all the files in the tree.\n\nIt's quite possible that some path in cg-update ends up not updating the \nindex properly. For example, I notice that the \"fast-forward\" uses \n\"git-checkout-cache -f -a\", which can do so (lack of \"-u\" fila), but then \nit does do a \"git-update-cache --refresh\" later, so that doesn't seem to \nbe it either.\n\nIf you do a \"git-diff-files\" every once in a while, it will _scream_ at\nyou whenever you have files that aren't up-to-date in the cache. That's\nnormal in small doses, of course (eg your edit of drivers/serial/8250.c\nwould make that one not up-to-date), but if you get a _lot_ of files\nlisted, that's usually a sign that something screwed up your index. \n\n\t\tLinus\n"},{"id":"6134","messageId":"Pine.LNX.4.58.0507140832490.19183@g5.osdl.org","threadId":"1220","inReplyTo":"20050714105938.A31383@flint.arm.linux.org.uk","subject":"Re: Is cogito really this inefficient","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-07-14T15:51:44Z","receivedAt":"2005-07-14T15:51:44Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 14 Jul 2005, Russell King wrote:\n> \n> Actually, I should've left the sh -x /usr/bin/cg-diff drivers/serial/8250.c\n> running a little longer.  It's not the git-update-cache command which\n> is taking the time, it's git-diff-cache.\n\nOk. git-diff-cache actually ends up reading your HEAD tree, and that, in\nturn, is 1000+ tree objects. So it can take a while for the whole tree,\nespecially in the nonpacked and uncached case.\n\ngit-diff-tree (comparing two trees) is smart enough to limit itself to \njust the sub-trees that have been named, and would have compared the two \ntrees by looking up just eight objects (three subdirectories from each \ntree, and then the file itself from both trees). \n\nBut git-diff-cache isn't - because it's comparing the tree against the\nindex file, and the index is inevitably the whole tree.\n\nAnd I now think I know what makes it slow. Not only are you basically\nopening 1100 files (the tree objects - there's really that many\nsubdirectories in the kernel. Scary), but because you have alternate\nobject directories, and almost all of the objects are in the alternate\n(not your primary), you'll basically always end up _first_ looking in the\nprimary, failing, and then looking in the alternate.\n\nTogether with the hashing, you'll be looking all over the place, in other\nwords ;)\n\nWhich means that you'll be needing a fair amount of memory to keep all of\nthose negative dentries etc cached (and the directory tree too).\n\nThis is something the pack-files will just help enormously with, but it\nwas only recently that we turned git around to check the pack-files\n_first_, and the object directories second, so you probably won't see it\n(not to mention that you probably don't have big pack-files at all ;)\n\nI'll look into making diff-cache be more efficient. I normally don't use\nit myself, so I didn't bother (I use git-diff-files, which is way more\nefficient, but doesn't show the difference against the _tree_, it shows\nthe difference against the index. Since cogito tries to hide the index\nfrom you, cogito can't very well use that).\n\n\t\t\tLinus\n"},{"id":"6152","messageId":"Pine.LNX.4.58.0507141725280.19183@g5.osdl.org","threadId":"1220","inReplyTo":"Pine.LNX.4.58.0507140832490.19183@g5.osdl.org","subject":"Re: Is cogito really this inefficient","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-07-15T00:29:09Z","receivedAt":"2005-07-15T00:29:09Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 14 Jul 2005, Linus Torvalds wrote:\n>\n> I'll look into making diff-cache be more efficient. I normally don't use\n> it myself, so I didn't bother (I use git-diff-files, which is way more\n> efficient, but doesn't show the difference against the _tree_, it shows\n> the difference against the index. Since cogito tries to hide the index\n> from you, cogito can't very well use that).\n\nOk, done.\n\nI made git-diff-cache _and_ git-diff-files limit the pathnames early, so\nthat they don't even bother expanding the tree objects that are\nirrelevant, and don't bother even validating index objects that don't\nmatch the pathnames given.\n\nJunio - I think this makes gitcore-pathspec pretty pointless, but I didn't\nactually remove it. I guess \"git-diff-helper\" still uses it.\n\n\t\tLinus\n"},{"id":"6167","messageId":"7vll48250r.fsf@assigned-by-dhcp.cox.net","threadId":"1220","inReplyTo":"Pine.LNX.4.58.0507141725280.19183@g5.osdl.org","subject":"Re: Is cogito really this inefficient","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-07-15T02:10:28Z","receivedAt":"2005-07-15T02:10:28Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds <torvalds@osdl.org> writes:\n\n> On Thu, 14 Jul 2005, Linus Torvalds wrote:\n>>\n>> I'll look into making diff-cache be more efficient. I normally don't use\n>> it myself, so I didn't bother (I use git-diff-files, which is way more\n>> efficient, but doesn't show the difference against the _tree_, it shows\n>> the difference against the index. Since cogito tries to hide the index\n>> from you, cogito can't very well use that).\n>\n> Ok, done.\n\nWonderful.\n\n> Junio - I think this makes gitcore-pathspec pretty pointless, but I didn't\n> actually remove it. I guess \"git-diff-helper\" still uses it.\n\nAnd probably it shouldn't; diff-helper should be raw-to-patch\nconverter, nothing more.\n\nUsually I'd volunteer to clean up the remaining mess (which was\noriginally my mess anyway) myself, but since I'd already asked\nsmurf to help cleaning up the diff option parsing, and recently\nI've suddenly got quite busy in the day job, so ...\n"},{"id":"6174","messageId":"20050715104826.H25428@flint.arm.linux.org.uk","threadId":"1220","inReplyTo":"Pine.LNX.4.58.0507141725280.19183@g5.osdl.org","subject":"Re: Is cogito really this inefficient","fromName":"Russell King","fromEmail":"rmk@arm.linux.org.uk","sentAt":"2005-07-15T09:48:26Z","receivedAt":"2005-07-15T09:48:26Z","isPatch":false,"sender":{"key":"rmk@arm.linux.org.uk","avatar":null},"body":"On Thu, Jul 14, 2005 at 05:29:09PM -0700, Linus Torvalds wrote:\n> On Thu, 14 Jul 2005, Linus Torvalds wrote:\n> > I'll look into making diff-cache be more efficient. I normally don't use\n> > it myself, so I didn't bother (I use git-diff-files, which is way more\n> > efficient, but doesn't show the difference against the _tree_, it shows\n> > the difference against the index. Since cogito tries to hide the index\n> > from you, cogito can't very well use that).\n> \n> Ok, done.\n\nThanks Linus.  I'll look forward to trying this out.\n\n-- \nRussell King\n"},{"id":"6267","messageId":"20050719235452.GE2255@pasky.ji.cz","threadId":"1220","inReplyTo":"Pine.LNX.4.58.0507140813150.19183@g5.osdl.org","subject":"Re: Is cogito really this inefficient","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2005-07-19T23:54:52Z","receivedAt":"2005-07-19T23:54:52Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Thu, Jul 14, 2005 at 05:26:07PM CEST, I got a letter\nwhere Linus Torvalds <torvalds@osdl.org> told me that...\n> It's quite possible that some path in cg-update ends up not updating the \n> index properly. For example, I notice that the \"fast-forward\" uses \n> \"git-checkout-cache -f -a\", which can do so (lack of \"-u\" fila), but then \n> it does do a \"git-update-cache --refresh\" later, so that doesn't seem to \n> be it either.\n\nJust a side note for casual readers, Cogito could use a cleanup here -\nfrom large part it ignores things like git-checkout-cache -u simply\nbecause there was no such option at the time that part of Cogito was\nwritten. I myself am not even too familiar about those gazillions of\nfunny new options, and as long as it works, I prefer not to touch that\ncode, but if someone is bored and wants to get familiar with core git\nusage as well as Cogito internals...\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nIf you want the holes in your knowledge showing up try teaching\nsomeone.  -- Alan Cox\n"}]}