{"thread":{"id":"20205","subject":"Performance issue of 'git branch'","startedAt":"2009-07-22T23:59:14Z","lastAt":"2009-08-07T04:21:33Z","messageCount":73,"participants":["Carlos R. Mafra","Linus Torvalds","SZEDER Gábor","Junio C Hamano","Jakub Narebski","Anders Kaseorg","Tony Finch","Johannes Schindelin","david@lang.hm","Theodore Tso","Shawn O. Pearce","Daniel Barkalow","Timo Hirvonen","Reece Dunn","Mike Hommey","demerphq","Avi Kivity","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"118522","messageId":"20090722235914.GA13150@Pilar.aei.mpg.de","threadId":"20205","inReplyTo":null,"subject":"Performance issue of 'git branch'","fromName":"Carlos R. Mafra","fromEmail":"crmafra2@gmail.com","sentAt":"2009-07-22T23:59:14Z","receivedAt":"2009-07-22T23:59:14Z","isPatch":false,"sender":{"key":"crmafra2@gmail.com","avatar":null},"body":"Hi,\n\nWhen I run 'git branch' in the linux-2.6 repo I think it takes\ntoo long to finish (with cold cache):\n\n[mafra@Pilar:linux-2.6]$ time git branch\n 27-stable\n 28-stable\n 29-stable\n 30-stable\n dev-private\n* master\n option\n sparse\n stern\n0.00user 0.05system 0:05.73elapsed 1%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (209major+1380minor)pagefaults 0swaps\n\nThis is with git 1.6.4.rc1.10.g2a67 and the kernel is 2.6.31-rc3+. The\nmachine is a 64bit Vaio laptop which is 1+ year old (so it is not \"slow\").\n\nRepeating the command a second time takes basically zero seconds, but\nthis is more or less what I would expect in the first time too.\n\nI use git to track linux-2.6 for 2 years now, and I remember that\n'git branch' is slow for quite some time, so it is not a regression\nor something. It is just now that I took the courage to report this\nsmall issue.\n\nI did a 'strace' and this is where it spent most of the time:\n\n1248301060.654911 open(\".git/refs/heads/sparse\", O_RDONLY) = 6\n1248301060.654985 read(6, \"60afdf6a4065a170ad829b4d79a86ec0\"..., 255) = 41\n1248301060.655056 read(6, \"\", 214) = 0\n1248301060.655116 close(6) = 0\n1248301060.680754 lstat(\".git/refs/heads/stern\", 0x7fff80bfa8d0) = -1 ENOENT (No such file or directory)\n1248301064.018491 fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0\n1248301064.018641 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f409ffa7000\n1248301064.018722 write(1, \" 27-stable\\33[m\\n\", 15) = 15\n\nI don't know why .git/refs/heads/stern does not exist and why it takes\nso long with it. That branch is functional ('git checkout stern' succeeds),\nas well as all the others. But strangely .git/refs/heads/ contains only\n\n[mafra@Pilar:linux-2.6]$ ls .git/refs/heads/\ndev-private master sparse\n\nwhich, apart from \"master\", are the last branches that I created.\n\nI occasionally run 'git gc --aggressive --prune\" to optimize the repo,\nbut other than that I don't do anything fancy, just 'pull' almost\nevery day and 'bisect' (which is becoming a rare event now :-)\n\nSo I would like to ask what should I do to recover the missing files\nin .git/refs/heads/ (which apparently is the cause for my issue) and\nhow I can avoid losing them in the first place.\n\nAlso, is there a way to \"fix\" the 4-secs pause in that lstat() in\ncase the files in .git/refs/heads/ get lost again?\n\nThanks in advance,\nCarlos\n"},{"id":"118523","messageId":"alpine.LFD.2.01.0907221714300.3352@localhost.localdomain","threadId":"20205","inReplyTo":"20090722235914.GA13150@Pilar.aei.mpg.de","subject":"Re: Performance issue of 'git branch'","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-23T00:21:48Z","receivedAt":"2009-07-23T00:21:48Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 23 Jul 2009, Carlos R. Mafra wrote:\n> \n> When I run 'git branch' in the linux-2.6 repo I think it takes\n> too long to finish (with cold cache):\n> \n> [mafra@Pilar:linux-2.6]$ time git branch\n> 27-stable\n> 28-stable\n> 29-stable\n> 30-stable\n> dev-private\n> * master\n> option\n> sparse\n> stern\n> 0.00user 0.05system 0:05.73elapsed 1%CPU (0avgtext+0avgdata 0maxresident)k\n> 0inputs+0outputs (209major+1380minor)pagefaults 0swaps\n> \n> This is with git 1.6.4.rc1.10.g2a67 and the kernel is 2.6.31-rc3+. The\n> machine is a 64bit Vaio laptop which is 1+ year old (so it is not \"slow\").\n\nWhen have you last repacked the repository?\n\nWhat you're descibing is basically IO overhead, and if you don't have \npacked references, it's going to read a lot of small files.\n\n> I use git to track linux-2.6 for 2 years now, and I remember that\n> 'git branch' is slow for quite some time, so it is not a regression\n> or something. It is just now that I took the courage to report this\n> small issue.\n> \n> I did a 'strace' and this is where it spent most of the time:\n> \n> 1248301060.654911 open(\".git/refs/heads/sparse\", O_RDONLY) = 6\n> 1248301060.654985 read(6, \"60afdf6a4065a170ad829b4d79a86ec0\"..., 255) = 41\n> 1248301060.655056 read(6, \"\", 214) = 0\n> 1248301060.655116 close(6) = 0\n> 1248301060.680754 lstat(\".git/refs/heads/stern\", 0x7fff80bfa8d0) = -1 ENOENT (No such file or directory)\n> 1248301064.018491 fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0\n> 1248301064.018641 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f409ffa7000\n> 1248301064.018722 write(1, \" 27-stable\\33[m\\n\", 15) = 15\n> \n> I don't know why .git/refs/heads/stern does not exist and why it takes\n> so long with it. That branch is functional ('git checkout stern' succeeds),\n> as well as all the others. But strangely .git/refs/heads/ contains only\n> \n> [mafra@Pilar:linux-2.6]$ ls .git/refs/heads/\n> dev-private master sparse\n> \n> which, apart from \"master\", are the last branches that I created.\n\nOk, this actually means that you _have_ repacked the repo, and the rest of \nthe branches are all nicely packed in .git/packed-refs.\n\nBut that four _second_ lstat() is really disgusting.\n\nLet me guess: if you do a \"ls -ld .git/refs/heads\" you get a very big \ndirectory, despite it only having three entries in it. And your filesystem \ndoesn't have name hashing enabled, so searching for a non-existent file \ninvolves looking through _all_ of the empty slots.\n\nTry this:\n\n\tgit pack-refs --all\n\n\trmdir .git/refs/heads\n\trmdir .git/refs/tags\n\n\tmkdir .git/refs/heads\n\tmkdir .git/refs/tags\n\nand see if it magically speeds up.\n\n\t\t\tLinus\n"},{"id":"118524","messageId":"20090723002323.GA23021@saturnine","threadId":"20205","inReplyTo":"20090722235914.GA13150@Pilar.aei.mpg.de","subject":"Re: Performance issue of 'git branch'","fromName":"SZEDER Gábor","fromEmail":"szeder@ira.uka.de","sentAt":"2009-07-23T00:23:23Z","receivedAt":"2009-07-23T00:23:23Z","isPatch":false,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"Hi,\n\n\nOn Thu, Jul 23, 2009 at 01:59:14AM +0200, Carlos R. Mafra wrote:\n\n> I don't know why .git/refs/heads/stern does not exist and why it takes\n> so long with it. That branch is functional ('git checkout stern' succeeds),\n> as well as all the others. But strangely .git/refs/heads/ contains only\n> \n> [mafra@Pilar:linux-2.6]$ ls .git/refs/heads/\n> dev-private master sparse\n> \n> which, apart from \"master\", are the last branches that I created.\n> \n> I occasionally run 'git gc --aggressive --prune\" to optimize the repo,\n> but other than that I don't do anything fancy, just 'pull' almost\n> every day and 'bisect' (which is becoming a rare event now :-)\n> \n> So I would like to ask what should I do to recover the missing files\n> in .git/refs/heads/ (which apparently is the cause for my issue) and\n> how I can avoid losing them in the first place.\n\nhave a look at .git/packed-refs and 'git pack-refs'.\n\n\nBest,\nGábor\n"},{"id":"118526","messageId":"alpine.LFD.2.01.0907221742010.3352@localhost.localdomain","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907221714300.3352@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-23T00:51:38Z","receivedAt":"2009-07-23T00:51:38Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 22 Jul 2009, Linus Torvalds wrote:\n> \n> Try this:\n> \n> \tgit pack-refs --all\n> \n> \trmdir .git/refs/heads\n> \trmdir .git/refs/tags\n> \n> \tmkdir .git/refs/heads\n> \tmkdir .git/refs/tags\n> \n> and see if it magically speeds up.\n\nIn fact, you could also just try\n\n\tmv .git/refs .git/temp-refs &&\n\tcp -a .git/temp-refs .git/refs &&\n\trm -rf .git/temp-refs\n\nwhich will re-create other subdirectories too (like .git/refs/remotes \netc).\n\nOf course, depending on your particular filesystem, a better fix might be \nto enable filename hashing, which gets rid of the whole \"look through all \nthe old empty stale directory entries to see if there's a filename there\" \nissue. That won't fix 'readdir()' performance, but it should fix your \ninsane 4-second lstat() thing.\n\nIf you have ext3, you'd do something like\n\n\ttune2fs -O dir_index /dev/\n\nbut as mentioned, even with directory indexing it can actually make sense \nto recreate directories that at some point _used_ to be large, but got \nshrunk down to something much smaller. It's a generic directory problem \n(not just ext3, not just unix, it's a common issue across filesystems. \nIt's not _universal_ - some smarter filesystems really do shrink their \ndirectories - but it's certainly not unusual).\n\n\t\tLinus\n"},{"id":"118527","messageId":"alpine.LFD.2.01.0907221753290.3352@localhost.localdomain","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907221742010.3352@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-23T00:55:51Z","receivedAt":"2009-07-23T00:55:51Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 22 Jul 2009, Linus Torvalds wrote:\n> \n> If you have ext3, you'd do something like\n> \n> \ttune2fs -O dir_index /dev/\n\nOne last email note on this subject. Really. Promise.\n\nIf you do that \"tune2fs -O dir_index\" thing, it will only take effect for \n_newly_ created directories. So you'll still need to do that whole \n\"mv+cp+rm\" dance, just to make sure that the refs directories are all new.\n\nI think you can also force all directories to be indexed by using fsck, \nbut I forget the details. I'm sure man-pages will have it. Or google.\n\n\t\tLinus\n"},{"id":"118528","messageId":"20090723012207.GA9368@Pilar.aei.mpg.de","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907221714300.3352@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Carlos R. Mafra","fromEmail":"crmafra2@gmail.com","sentAt":"2009-07-23T01:22:07Z","receivedAt":"2009-07-23T01:22:07Z","isPatch":false,"sender":{"key":"crmafra2@gmail.com","avatar":null},"body":"On Wed 22.Jul'09 at 17:21:48 -0700, Linus Torvalds wrote:\n> \n> When have you last repacked the repository?\n\nLast week or so, with 'git repack -d -a'\n\n\n> > [mafra@Pilar:linux-2.6]$ ls .git/refs/heads/\n> > dev-private master sparse\n> > \n> > which, apart from \"master\", are the last branches that I created.\n> \n> Ok, this actually means that you _have_ repacked the repo, and the rest of \n> the branches are all nicely packed in .git/packed-refs.\n\nYes, now I saw the other branches inside packed-refs.\n\n> But that four _second_ lstat() is really disgusting.\n> \n> Let me guess: if you do a \"ls -ld .git/refs/heads\" you get a very big \n> directory, despite it only having three entries in it. \n\n[mafra@Pilar:linux-2.6]$ ls -ld .git/refs/heads\ndrwxr-xr-x 2 mafra mafra 4096 2009-07-22 23:01 .git/refs/heads/\n\n> And your filesystem \n> doesn't have name hashing enabled, so searching for a non-existent file \n> involves looking through _all_ of the empty slots.\n\nI use ext3 without changing any defaults that I know of (I simply compile\nand boot the kernel of the day), and I have no idea if name hashing\nis enabled here.\n\n> Try this:\n> \n> \tgit pack-refs --all\n> \n> \trmdir .git/refs/heads\n> \trmdir .git/refs/tags\n> \n> \tmkdir .git/refs/heads\n> \tmkdir .git/refs/tags\n> \n> and see if it magically speeds up.\n\nIt didn't change things, unfortunately.\n\nAfter 'echo 3 > /proc/sys/vm/drop_caches' it still takes too long,\n\n1248310449.693085 munmap(0x7f50bcd11000, 164) = 0\n1248310449.693187 lstat(\".git/refs/heads/sparse\", 0x7fff618c0960) = -1 ENOENT (No such file or directory)\n1248310449.719112 lstat(\".git/refs/heads/stern\", 0x7fff618c0960) = -1 ENOENT (No such file or directory)\n1248310453.014041 fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 3), ...}) = 0\n1248310453.014183 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f50bcd11000\n\nPerhaps I should delete the \"stern\" branch, but I would like to learn why\nit is slowing things, because it also happened before (in fact it is always\nlike this, afaicr)\n\nDo you have another theory? (now .git/refs/heads is empty)\n\nThanks,\nCarlos\n"},{"id":"118530","messageId":"20090723020238.GA8948@Pilar.aei.mpg.de","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907221753290.3352@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Carlos R. Mafra","fromEmail":"crmafra2@gmail.com","sentAt":"2009-07-23T02:02:38Z","receivedAt":"2009-07-23T02:02:38Z","isPatch":false,"sender":{"key":"crmafra2@gmail.com","avatar":null},"body":"On Wed 22.Jul'09 at 17:55:51 -0700, Linus Torvalds wrote:\n> On Wed, 22 Jul 2009, Linus Torvalds wrote:\n> > \n> > If you have ext3, you'd do something like\n> > \n> > \ttune2fs -O dir_index /dev/\n> \n> One last email note on this subject. Really. Promise.\n> \n> If you do that \"tune2fs -O dir_index\" thing, it will only take effect for \n> _newly_ created directories. So you'll still need to do that whole \n> \"mv+cp+rm\" dance, just to make sure that the refs directories are all new.\n\nOk, now I also did the \"dir_index\" thing followed by the mv+cp+rm instructions.\nIt doesn't change the 3.5 secs delay in that single line,\n\n1248313742.355195 lstat(\".git/refs/heads/sparse\", 0x7fff0c663ab0) = -1 ENOENT (No such file or directory)\n1248313742.381178 lstat(\".git/refs/heads/stern\", 0x7fff0c663ab0) = -1 ENOENT (No such file or directory)\n1248313745.804637 fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0\n\nJust to double check,\n\n[root@Pilar linux-2.6]# tune2fs -l /dev/sda5 |grep dir_index\nFilesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file\n\n(and I did the mv+cp+rm after setting \"dir_index\")\n\nIs there another way to check what is going on with that anomalous lstat()?\n[ perhaps I will try 'perf' after I read how to use it ]\n\nThanks,\nCarlos\n"},{"id":"118531","messageId":"alpine.LFD.2.01.0907221850000.3352@localhost.localdomain","threadId":"20205","inReplyTo":"20090723012207.GA9368@Pilar.aei.mpg.de","subject":"Re: Performance issue of 'git branch'","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-23T02:20:00Z","receivedAt":"2009-07-23T02:20:00Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 23 Jul 2009, Carlos R. Mafra wrote:\n> > Let me guess: if you do a \"ls -ld .git/refs/heads\" you get a very big \n> > directory, despite it only having three entries in it. \n> \n> [mafra@Pilar:linux-2.6]$ ls -ld .git/refs/heads\n> drwxr-xr-x 2 mafra mafra 4096 2009-07-22 23:01 .git/refs/heads/\n\nHmm. That's just a single block. \n\nThen I really don't see why the lstat takes so long.\n\n> After 'echo 3 > /proc/sys/vm/drop_caches' it still takes too long,\n> \n> 1248310449.693085 munmap(0x7f50bcd11000, 164) = 0\n> 1248310449.693187 lstat(\".git/refs/heads/sparse\", 0x7fff618c0960) = -1 ENOENT (No such file or directory)\n> 1248310449.719112 lstat(\".git/refs/heads/stern\", 0x7fff618c0960) = -1 ENOENT (No such file or directory)\n> 1248310453.014041 fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 3), ...}) = 0\n> 1248310453.014183 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f50bcd11000\n\nUse 'strace -T', which shows how long the actual system calls take, rather \nthan '-tt' which just shows when they started.\n\nMaybe the four seconds is something else than the lstat - page faults on \nthe pack-file in between the lstat and the fstat, for example.\n\n> Perhaps I should delete the \"stern\" branch, but I would like to learn why\n> it is slowing things, because it also happened before (in fact it is always\n> like this, afaicr)\n\nAbsolutely. Don't delete it until we figure out what takes so long there.\n\n> Do you have another theory? (now .git/refs/heads is empty)\n\nClearly it's IO, but if that 'lstat()' was just a red herring, then I \nsuspect it's IO on the pack-file. If so, I'd further guess that your VAIO \nhas some pitiful 4200rpm harddisk that is slow as hell and has horrible \nseek latencies, and the CPU is way overpowered compared to the cruddy \ndisk.\n\nIt probably does the object lookup. You can see some debug output if you \ndo\n\n\tGIT_DEBUG_LOOKUP=1 git branch\n\nand that will show you the patterns. It won't be very pretty, especially \nif you have several pack-files, but maybe we can figure out what's up.\n\nHmm. I wonder.. I suspect 'git branch' looks up _all_ refs, and then \nafterwards it filters them. So even though it only prints out a few \nbranches, maybe it will look at all the tags etc of the whole repository.\n\nOoh yes. That would do it. It's going to peel and look up every single ref \nit finds, so it's going to look up _hundreds_ of objects (all the tags, \nall the commits they point to, etc etc). Even if it then only shows a \ncouple of branches.\n\nJunio, any ideas?\n\n\t\tLinus\n"},{"id":"118532","messageId":"alpine.LFD.2.01.0907221921570.3352@localhost.localdomain","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907221850000.3352@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-23T02:23:39Z","receivedAt":"2009-07-23T02:23:39Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 22 Jul 2009, Linus Torvalds wrote:\n> \n> Ooh yes. That would do it. It's going to peel and look up every single ref \n> it finds, so it's going to look up _hundreds_ of objects (all the tags, \n> all the commits they point to, etc etc). Even if it then only shows a \n> couple of branches.\n> \n> Junio, any ideas?\n\nI had one of my own.\n\nDoes this fix it?\n\nIt uses the \"raw\" version of 'for_each_ref()' (which doesn't verify that \nthe ref is valid), and then does the \"type verification\" before it starts \ndoing any gentle commit lookup.\n\nThat should hopefully mean that it no longer does tons of object lookups \non refs that it's not actually interested in. \n\n\t\tLinus\n\n---\n builtin-branch.c | 10 +++++-----\n 1 files changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin-branch.c b/builtin-branch.c\nindex 5687d60..54a89ff 100644\n--- a/builtin-branch.c\n+++ b/builtin-branch.c\n@@ -240,6 +240,10 @@ static int append_ref(const char *refname, const unsigned char *sha1, int flags,\n \tif (ARRAY_SIZE(ref_kind) <= i)\n \t\treturn 0;\n \n+\t/* Don't add types the caller doesn't want */\n+\tif ((kind & ref_list->kinds) == 0)\n+\t\treturn 0;\n+\n \tcommit = lookup_commit_reference_gently(sha1, 1);\n \tif (!commit)\n \t\treturn error(\"branch '%s' does not point at a commit\", refname);\n@@ -248,10 +252,6 @@ static int append_ref(const char *refname, const unsigned char *sha1, int flags,\n \tif (!is_descendant_of(commit, ref_list->with_commit))\n \t\treturn 0;\n \n-\t/* Don't add types the caller doesn't want */\n-\tif ((kind & ref_list->kinds) == 0)\n-\t\treturn 0;\n-\n \tif (merge_filter != NO_FILTER)\n \t\tadd_pending_object(&ref_list->revs,\n \t\t\t\t (struct object *)commit, refname);\n@@ -426,7 +426,7 @@ static void print_ref_list(int kinds, int detached, int verbose, int abbrev, str\n \tref_list.with_commit = with_commit;\n \tif (merge_filter != NO_FILTER)\n \t\tinit_revisions(&ref_list.revs, NULL);\n-\tfor_each_ref(append_ref, &ref_list);\n+\tfor_each_rawref(append_ref, &ref_list);\n \tif (merge_filter != NO_FILTER) {\n \t\tstruct commit *filter;\n \t\tfilter = lookup_commit_reference_gently(merge_filter_ref, 0);\n"},{"id":"118533","messageId":"20090723022533.GB8948@Pilar.aei.mpg.de","threadId":"20205","inReplyTo":"20090723002323.GA23021@saturnine","subject":"Re: Performance issue of 'git branch'","fromName":"Carlos R. Mafra","fromEmail":"crmafra2@gmail.com","sentAt":"2009-07-23T02:25:33Z","receivedAt":"2009-07-23T02:25:33Z","isPatch":false,"sender":{"key":"crmafra2@gmail.com","avatar":null},"body":"Hi,\n\nOn Wed 22.Jul'09 at 19:23:23 -0500, SZEDER Gábor wrote:\n> > So I would like to ask what should I do to recover the missing files\n> > in .git/refs/heads/ (which apparently is the cause for my issue) and\n> > how I can avoid losing them in the first place.\n> \n> have a look at .git/packed-refs and 'git pack-refs'.\n\nYes, now I learned that the files were not really missing\nas in \"there is something wrong\".\n\nI will also start to use 'git pack-refs --prune' from time to time\nnow, in adition to 'git gc --prune' and 'git repack -d -a'.\n\nBut the takes-too-long 'git branch' issue is apparently caused\nby something else.\n\nThanks Gábor,\nCarlos\n"},{"id":"118534","messageId":"alpine.LFD.2.01.0907221925100.3352@localhost.localdomain","threadId":"20205","inReplyTo":"20090723020238.GA8948@Pilar.aei.mpg.de","subject":"Re: Performance issue of 'git branch'","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-23T02:28:40Z","receivedAt":"2009-07-23T02:28:40Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 23 Jul 2009, Carlos R. Mafra wrote:\n> \n> Is there another way to check what is going on with that anomalous lstat()?\n\nI really don't think it's the lstat any more. Your directories look small \nand simple, and clearly the indexing made no difference.\n\nSee earlier email about using \"strace -T\" instead of \"-tt\". Also, I sent \nyou a patch to try out just a minute ago, I think that may be it.\n\n> [ perhaps I will try 'perf' after I read how to use it ]\n\nI really like 'perf' (it does what oprofile did for me, but without the \nheadaches), but it doesn't help with IO profiling.\n\nI've actually often wanted to have a 'strace' that shows page faults as \nspecial system calls, but it's sadly nontrivial ;(\n\n\t\t\tLinus\n"},{"id":"118535","messageId":"alpine.LFD.2.01.0907221959330.21520@localhost.localdomain","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907221921570.3352@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-23T03:08:57Z","receivedAt":"2009-07-23T03:08:57Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 22 Jul 2009, Linus Torvalds wrote:\n> \n> It uses the \"raw\" version of 'for_each_ref()' (which doesn't verify that \n> the ref is valid), and then does the \"type verification\" before it starts \n> doing any gentle commit lookup.\n> \n> That should hopefully mean that it no longer does tons of object lookups \n> on refs that it's not actually interested in. \n\nHmm. On my kernel repo, doing\n\n\tGIT_DEBUG_LOOKUP=1 git branch | wc -l\n\nI get\n - before: 2121\n - after: 39\n\n(where two of the lines are the actual 'git branch' output). So yeah, this \nshould make a big difference. It now looks up just two objects (one of \nthem duplicated because it checks \"HEAD\" - but the duplicate lookup won't \nresult in any extra IO, so it's only two _uncached_ accesses).\n\nThe GIT_DEBUG_LOOKUP debug output probably does match the number of \ncold-cache IO's fairly well for something like this (at least to a first \napproximation), so I really hope my patch will fix your problem.\n\n\t\t\tLinus\n"},{"id":"118537","messageId":"20090723031843.GA9152@Pilar.aei.mpg.de","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907221921570.3352@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Carlos R. Mafra","fromEmail":"crmafra2@gmail.com","sentAt":"2009-07-23T03:18:44Z","receivedAt":"2009-07-23T03:18:44Z","isPatch":false,"sender":{"key":"crmafra2@gmail.com","avatar":null},"body":"First of all:\n * yes, my VAIO has a slow 4200 rpm disc :-(\n * strace -T indeed showed that lstat() was not guilty\n * GIT_DEBUG_LOOKUP=1 git branch produced ugly 2200+ lines\n\nNow to the patch,\n\nOn Wed 22.Jul'09 at 19:23:39 -0700, Linus Torvalds wrote:\n> > Ooh yes. That would do it. It's going to peel and look up every single ref \n> > it finds, so it's going to look up _hundreds_ of objects (all the tags, \n> > all the commits they point to, etc etc). Even if it then only shows a \n> > couple of branches.\n> > \n> > Junio, any ideas?\n> \n> I had one of my own.\n> \n> Does this fix it?\n\nYes!\n\n[mafra@Pilar:linux-2.6]$ time git branch\n 27-stable\n 28-stable\n 29-stable\n 30-stable\n dev-private\n* master\n option\n sparse\n stern\n0.00user 0.01system 0:01.50elapsed 1%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (42major+757minor)pagefaults 0swaps\n\n01.50 is not that good, but it doesn't \"feel\" terrible as 4 seconds.\n[ It is incredible how 4 secs feels really bad while 2 is acceptable... ]\n\nSo thank you very much, Linus! A 50% improvement here!\n\nAnd I am happy to have finally reported it, after quietly suffering for so long \nthinking that \"git is as fast as possible, so it is probably my fault\".\n\nPS: Out of curiosity, how many femtoseconds does it take in your \nstate-of-the-art machine? :-)\n"},{"id":"118538","messageId":"alpine.LFD.2.01.0907222009340.21520@localhost.localdomain","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907221959330.21520@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-23T03:21:01Z","receivedAt":"2009-07-23T03:21:01Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 22 Jul 2009, Linus Torvalds wrote:\n> \n> The GIT_DEBUG_LOOKUP debug output probably does match the number of \n> cold-cache IO's fairly well for something like this (at least to a first \n> approximation), so I really hope my patch will fix your problem.\n\nSide note: the object lookup binary search we do is simple and reasonably \nefficient, but it is _not_ very cache-friendly (where \"cache-friendly\" \nalso in this case means IO caches).\n\nThere are more cache-friendly ways of searching, although the really \nclever ones would require us to switch the format of the pack-file index \naround. Which would be a fairly big pain (in addition to making the lookup \na lot more complex).\n\nThe _simpler_ cache-friendly alternative is likely to try the \"guess \nlocation by assuming the SHA1's are evenly spread out\" thing doesn't jump \nback-and-forth like a binary search does.\n\nWe tried it a few years ago, but didn't do cold-cache numbers. And \nrepositories were smaller too.\n\nWith something like the kernel repo, with 1.2+ million objects, a binary \nsearch needs about 21 comparisons for each object we look up. The index \nhas a first-level fan-out of 256, so that takes away 8 of them, but we're \nstill talking about 13 comparisons. With bad locality except for the very \nlast ones.\n\nAssuming a 4kB page-size, and about 170 index entries per page (~7 binary \nsearch levels), that's 6 pages we have to page-fault in for each search. \nAnd we probably won't start seeing lots of cache reuse until we hit \nhundreds or thousands of objects searched for.\n\nWith soemthing like \"three iterations of newton-raphson + linear search\", \nwe might end up with more index entries looked at, but we'd quite possibly \nget much better locality.\n\nI suspect the old newton-raphson patches we had (Discussions and patches \nback in April 2007 on this list) could be resurrected pretty easily.\n\n\t\tLinus\n"},{"id":"118539","messageId":"20090723032715.GA7630@Pilar.aei.mpg.de","threadId":"20205","inReplyTo":"20090723031843.GA9152@Pilar.aei.mpg.de","subject":"Re: Performance issue of 'git branch'","fromName":"Carlos R. Mafra","fromEmail":"crmafra2@gmail.com","sentAt":"2009-07-23T03:27:16Z","receivedAt":"2009-07-23T03:27:16Z","isPatch":false,"sender":{"key":"crmafra2@gmail.com","avatar":null},"body":"On Thu 23.Jul'09 at 5:18:44 +0200, Carlos R. Mafra wrote:\n\n> * GIT_DEBUG_LOOKUP=1 git branch produced ugly 2200+ lines\n\nWith your patch applied it went down to 132 lines.\n"},{"id":"118540","messageId":"20090723034005.GA11463@Pilar.aei.mpg.de","threadId":"20205","inReplyTo":"20090723031843.GA9152@Pilar.aei.mpg.de","subject":"Re: Performance issue of 'git branch'","fromName":"Carlos R. Mafra","fromEmail":"crmafra2@gmail.com","sentAt":"2009-07-23T03:40:05Z","receivedAt":"2009-07-23T03:40:05Z","isPatch":false,"sender":{"key":"crmafra2@gmail.com","avatar":null},"body":"On Thu 23.Jul'09 at 5:18:44 +0200, Carlos R. Mafra wrote:\n\n> 0.00user 0.01system 0:01.50elapsed 1%CPU (0avgtext+0avgdata 0maxresident)k\n> 0inputs+0outputs (42major+757minor)pagefaults 0swaps\n> \n> 01.50 is not that good, but it doesn't \"feel\" terrible as 4 seconds.\n> [ It is incredible how 4 secs feels really bad while 2 is acceptable... ]\n\nI need to sleep, as the number 4 seconds got stuck in my head. In my original\nreport it was much worse\n\n0.00user 0.05system 0:05.73elapsed\n\nSo now it was a 75% improvement!\n"},{"id":"118541","messageId":"alpine.LFD.2.01.0907222041341.21520@localhost.localdomain","threadId":"20205","inReplyTo":"20090723031843.GA9152@Pilar.aei.mpg.de","subject":"Re: Performance issue of 'git branch'","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-23T03:47:10Z","receivedAt":"2009-07-23T03:47:10Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 23 Jul 2009, Carlos R. Mafra wrote:\n> \n> PS: Out of curiosity, how many femtoseconds does it take in your \n> state-of-the-art machine? :-)\n\nCold cache? 0.15s before the patch. 0.03s after.\n\nSo we're not talking femto-seconds, but I've got Intel SSD's that do \nrandom reads in well under a millisecond. Your pitiful 4200rpm drive \nprobably takes 20ms for each seek. You don't really need that many IO's \nfor it to take a second or two. Or four.\n\nThe kernel will do IO in bigger chunks than a single page, and there is \n_some_ locality to it all, so you won't see IO for each lookup. But with \n2000+ lines of GIT_DEBUG_LOOKUP, you probably do end up having a \nnoticeable fraction of them being IO-causing, and another fraction causing \nseeks.\n\nBut I'll see if I can dig up my non-binary-search patch and see if I can \nmake it go faster. My machine is fast, but not so fast that I can't \nmeasure it ;)\n\n\t\tLinus\n"},{"id":"118542","messageId":"alpine.LFD.2.01.0907222050500.21520@localhost.localdomain","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907222041341.21520@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-23T04:10:49Z","receivedAt":"2009-07-23T04:10:49Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 22 Jul 2009, Linus Torvalds wrote:\n> \n> But I'll see if I can dig up my non-binary-search patch and see if I can \n> make it go faster. My machine is fast, but not so fast that I can't \n> measure it ;)\n\nOh. We actually merged a fixed version of it. I'd completely forgotten.\n \nEnabled with 'GIT_USE_LOOKUP'. But it seems to give worse performance, \ndespite giving me fewer searches: I get 2121 probes with binary searching, \nbut only 1325 with the newton-raphson method (for the non-fixed 'git \nbranch' case).\n\nUsing GIT_USE_LOOKUP actually results in fewer pagefaults (1391 vs 1473), \nbut it's still slower. Interesting. Carlos, try it on your machine (just \ndo\n\n\texport GIT_USE_LOOKUP=1\n\ttime git branch\n\nto try it, and 'unset GIT_USE_LOOKUP' to disable it.\n\n(And note that the \"=1\" part isn't important - the only thing that matters \nis whether the environment variable is set or not - setting it to '0' will \n_not_ disable it, you need to 'unset' it).\n\nWith my fix to 'git branch', it doesn't matter. I get the same \nperformance, and same number of page faults (676) regardless. So my patch \nmakes the GIT_USE_LOOKUP=1 thing irrelevant.\n\n\t\tLinus\n"},{"id":"118543","messageId":"7vtz146mgr.fsf@alter.siamese.dyndns.org","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907221921570.3352@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-07-23T04:40:36Z","receivedAt":"2009-07-23T04:40:36Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds writes:\n\n> On Wed, 22 Jul 2009, Linus Torvalds wrote:\n>> \n>> Ooh yes. That would do it. It's going to peel and look up every single ref \n>> it finds, so it's going to look up _hundreds_ of objects (all the tags, \n>> all the commits they point to, etc etc). Even if it then only shows a \n>> couple of branches.\n>> \n>> Junio, any ideas?\n>\n> I had one of my own.\n\nIt seems that I missed all the fun while going out to dinner.\n\n> It uses the \"raw\" version of 'for_each_ref()' (which doesn't verify that \n> the ref is valid), and then does the \"type verification\" before it starts \n> doing any gentle commit lookup.\n\nHmm, we now have to remember what this patch did, if we ever wanted to\nintroduce negative refs later (see ef06b91 do_for_each_ref: perform the\nsame sanity check for leftovers., 2006-11-18). Not exactly nice to spread\nthe codepaths that need to be updated. Is the cold cache performance of\n\"git branch\" to list your local branches that important?\n"},{"id":"118548","messageId":"7vtz1456de.fsf@alter.siamese.dyndns.org","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907222050500.21520@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-07-23T05:13:33Z","receivedAt":"2009-07-23T05:13:33Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds writes:\n\n> On Wed, 22 Jul 2009, Linus Torvalds wrote:\n>> \n>> But I'll see if I can dig up my non-binary-search patch and see if I can \n>> make it go faster. My machine is fast, but not so fast that I can't \n>> measure it ;)\n>\n> Oh. We actually merged a fixed version of it. I'd completely forgotten.\n\nAs the commit message of 628522e (sha1-lookup: more memory efficient\nsearch in sorted list of SHA-1, 2007-12-29) shows, it didn't get any great\nperformance improvements, even though it did make the probing quite a lot\nless memory intensive.\n\nPerhaps you can spot obvious inefficiency in the code that I failed to\nsee, just like you recently did for \"show --cc\" codepath?\n"},{"id":"118551","messageId":"20090723051717.GA20063@Pilar.aei.mpg.de","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907222050500.21520@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Carlos R. Mafra","fromEmail":"crmafra2@gmail.com","sentAt":"2009-07-23T05:17:18Z","receivedAt":"2009-07-23T05:17:18Z","isPatch":false,"sender":{"key":"crmafra2@gmail.com","avatar":null},"body":"On Wed 22.Jul'09 at 21:10:49 -0700, Linus Torvalds wrote:\n> Enabled with 'GIT_USE_LOOKUP'. But it seems to give worse performance, \n> despite giving me fewer searches: I get 2121 probes with binary searching, \n> but only 1325 with the newton-raphson method (for the non-fixed 'git \n> branch' case).\n> \n> Using GIT_USE_LOOKUP actually results in fewer pagefaults (1391 vs 1473), \n> but it's still slower. Interesting. Carlos, try it on your machine (just \n> do\n> \n> \texport GIT_USE_LOOKUP=1\n> \ttime git branch\n> \n> to try it, and 'unset GIT_USE_LOOKUP' to disable it.\n\n\nGIT_USE_LOOKUP=1 makes is a bit slower overall. \n\nWithout your patch, I get fewer pagefaults (1254 vs 1404) when\nit is set, but it takes ~0.5s longer (it varies a bit).\n\n> With my fix to 'git branch', it doesn't matter. I get the same \n> performance, and same number of page faults (676) regardless. So my patch \n> makes the GIT_USE_LOOKUP=1 thing irrelevant.\n\nWith your patch and GIT_USE_LOOKUP=1 I get 751 pagefaults, versus 775\nif GIT_USE_LOOKUP is unset, but it is faster when unset.\n\nSo your patch without GIT_USE_LOOKUP=1 is the fastest option.\n"},{"id":"118553","messageId":"alpine.LFD.2.01.0907222204310.21520@localhost.localdomain","threadId":"20205","inReplyTo":"7vtz146mgr.fsf@alter.siamese.dyndns.org","subject":"Re: Performance issue of 'git branch'","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-23T05:36:09Z","receivedAt":"2009-07-23T05:36:09Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 22 Jul 2009, Junio C Hamano wrote:\n> \n> Hmm, we now have to remember what this patch did, if we ever wanted to\n> introduce negative refs later (see ef06b91 do_for_each_ref: perform the\n> same sanity check for leftovers., 2006-11-18). Not exactly nice to spread\n> the codepaths that need to be updated. Is the cold cache performance of\n> \"git branch\" to list your local branches that important?\n\nHmm. I do think that 7.5s is _way_ too long to wait for something as \nsimple as \"what branches do I have?\".\n\nAnd yes, it's also an operation that I'd expect to be quite possibly the \nfirst one you do when moving to a new repo, so cold-cache is realistic.\n\nAnd the 'rawref' thing is exactly the same as the 'ref' version, except it \ndoesn't do the null_sha1 check and the 'has_sha1-file()' check.\n\nAnd since git branch will do something _better_ than the 'has_sha1_file()' \ncheck (by virtue of actually looking up the commit), I don't think that \npart is an issue. So the only issue is the is_null_sha1() thing.\n\nAnd quite frankly, while the null-sha1 check may make sense, the way the \nflag is named right now (DO_FOR_EACH_INCLUDE_BROKEN), I think we might be \nbetter off re-thinking things later if we ever end up caring. That \n'is_null_sha1()' check should possibly be under a separate flag.\n\nThat said, while I think my patch was the simplest and most \nthe problem could certainly have been fixed differently.\n\nFor example, instead of using 'for_each_ref()' and then splitting them by \nkind with that \"detect kind\" loop, it could instead have done two loops, \nie\n\n\tif (kinds & REF_LOCAL_BRANCH)\n\t\tfor_each_ref_in(\"refs/heads/\", append_local, &ref_list);\n\tif (kinds & REF_REMOTE_BRANCH)\n\t\tfor_each_ref_in(\"refs/remotes/\", append_remote, &ref_list);\n\nand avoided the other refs we aren't interested in _that_ way instead.\n\nBut it would be a bigger and involved patch. It gets really messy too (I \ntried), because when you use 'for_each_ref_in()' it removes the prefix as \nit goes along, but then the code in builtin-branch.c wants the prefix \nafter all.\n\n\t\tLinus\n"},{"id":"118555","messageId":"7veis83q0e.fsf@alter.siamese.dyndns.org","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907222204310.21520@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-07-23T05:52:17Z","receivedAt":"2009-07-23T05:52:17Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds writes:\n\n> On Wed, 22 Jul 2009, Junio C Hamano wrote:\n>> \n>> Hmm, we now have to remember what this patch did, if we ever wanted to\n>> introduce negative refs later (see ef06b91 do_for_each_ref: perform the\n>> same sanity check for leftovers., 2006-11-18). Not exactly nice to spread\n>> the codepaths that need to be updated.\n> ...\n> And since git branch will do something _better_ than the 'has_sha1_file()' \n> check (by virtue of actually looking up the commit), I don't think that \n> part is an issue. So the only issue is the is_null_sha1() thing.\n\nExactly.\n\nThat is_null_sha1() thing was a remnant of your idea to represent deleted\nref that has a packed counterpart by storing 0{40} in a loose ref, so that\nwe can implement deletion efficiently.\n\nSince we currently implement deletion by repacking packed refs if the ref\nhas a packed (possibly stale) one, we do not use such a \"negative ref\",\nand skipping 0{40} done by the normal (i.e. non-raw) for_each_ref() family\nis not necessary.\n\nI was inclined to say that, because I never saw anybody complained that\ndeleting refs was too slow, we declare that we would forever stick to the\ncurrent implementation of ref deletion, and remove the is_null_sha1()\ncheck from the do_one_ref() function, even for include-broken case.\n\nBut after thinking about it again, I'd say \"if null, then skip\" should be\noutside the DO_FOR_EACH_INCLUDE_BROKEN anyway, because the null check is\nnot about brokenness of the ref, but is about a possible future expansion\nto represent deleted ref with such a \"negative ref\" entry.\n\nIf we remove is_null_sha1() from do_one_ref(), or if we move it out of the\n\"include broken\" thing, my \"Not exactly nice\" comment can be rescinded, as\ndoing the former (i.e. removal of is_null_sha1() check) is a promise that\nwe will never have to worry about negative refs, and doing the latter will\nstill protect callers of do_for_each_rawref() from negative refs if we\never introduce them in some future.\n"},{"id":"118556","messageId":"7vy6qg2aus.fsf@alter.siamese.dyndns.org","threadId":"20205","inReplyTo":"7veis83q0e.fsf@alter.siamese.dyndns.org","subject":"Re: Performance issue of 'git branch'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-07-23T06:04:59Z","receivedAt":"2009-07-23T06:04:59Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Exactly.\n>\n> That is_null_sha1() thing was a remnant of your idea to represent deleted\n> ref that has a packed counterpart by storing 0{40} in a loose ref, so that\n> we can implement deletion efficiently.\n>\n> Since we currently implement deletion by repacking packed refs if the ref\n> has a packed (possibly stale) one, we do not use such a \"negative ref\",\n> and skipping 0{40} done by the normal (i.e. non-raw) for_each_ref() family\n> is not necessary.\n>\n> I was inclined to say that, because I never saw anybody complained that\n> deleting refs was too slow, we declare that we would forever stick to the\n> current implementation of ref deletion, and remove the is_null_sha1()\n> check from the do_one_ref() function, even for include-broken case.\n>\n> But after thinking about it again, I'd say \"if null, then skip\" should be\n> outside the DO_FOR_EACH_INCLUDE_BROKEN anyway, because the null check is\n> not about brokenness of the ref, but is about a possible future expansion\n> to represent deleted ref with such a \"negative ref\" entry.\n>\n> If we remove is_null_sha1() from do_one_ref(), or if we move it out of the\n> \"include broken\" thing, my \"Not exactly nice\" comment can be rescinded, as\n> doing the former (i.e. removal of is_null_sha1() check) is a promise that\n> we will never have to worry about negative refs, and doing the latter will\n> still protect callers of do_for_each_rawref() from negative refs if we\n> ever introduce them in some future.\n\nThat is, a patch like this (this should go to 'maint'), and my worries\nwill go away.\n\n-- >8 --\nSubject: do_one_ref(): null_sha1 check is not about broken ref\n\nf8948e2 (remote prune: warn dangling symrefs, 2009-02-08) introduced a\nmore dangerous variant of for_each_ref() family that skips the check for\ndangling refs, but it also made another unrelated check optional by\nmistake.\n\nThe check to see if a ref points at 0{40} is not about brokenness, but is\nabout a possible future plan to represent a deleted ref by writing 40 \"0\"\nin a loose ref when there is a stale version of the same ref already in\n.git/packed-refs, so that we can implement deletion of a ref without\nhaving to rewrite the packed refs file excluding the ref being deleted.\nThis check has to be outside of the conditional.\n\nSigned-off-by: Junio C Hamano \n---\n refs.c | 5 +++--\n 1 files changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex bb0762e..3da3c8c 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -531,9 +531,10 @@ static int do_one_ref(const char *base, each_ref_fn fn, int trim,\n {\n \tif (strncmp(base, entry->name, trim))\n \t\treturn 0;\n+\t/* Is this a \"negative ref\" that represents a deleted ref? */\n+\tif (is_null_sha1(entry->sha1))\n+\t\treturn 0;\n \tif (!(flags & DO_FOR_EACH_INCLUDE_BROKEN)) {\n-\t\tif (is_null_sha1(entry->sha1))\n-\t\t\treturn 0;\n \t\tif (!has_sha1_file(entry->sha1)) {\n \t\t\terror(\"%s does not point to a valid object!\", entry->name);\n \t\t\treturn 0;\n"},{"id":"118579","messageId":"m3prbr371l.fsf@localhost.localdomain","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907221925100.3352@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-07-23T12:42:03Z","receivedAt":"2009-07-23T12:42:03Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Linus Torvalds writes:\n\n> On Thu, 23 Jul 2009, Carlos R. Mafra wrote:\n> > \n> > Is there another way to check what is going on with that anomalous lstat()?\n> \n> I really don't think it's the lstat any more. Your directories look small \n> and simple, and clearly the indexing made no difference.\n> \n> See earlier email about using \"strace -T\" instead of \"-tt\". Also, I sent \n> you a patch to try out just a minute ago, I think that may be it.\n> \n> > [ perhaps I will try 'perf' after I read how to use it ]\n> \n> I really like 'perf' (it does what oprofile did for me, but without the \n> headaches), but it doesn't help with IO profiling.\n> \n> I've actually often wanted to have a 'strace' that shows page faults as \n> special system calls, but it's sadly nontrivial ;(\n\nBTW. Would SystemTap help there? Among contributed scripts there is\niotimes, so perhaps it would be possible to have iotrace...\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"118581","messageId":"20090723144559.GA20167@Pilar.aei.mpg.de","threadId":"20205","inReplyTo":"m3prbr371l.fsf@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Carlos R. Mafra","fromEmail":"crmafra2@gmail.com","sentAt":"2009-07-23T14:45:59Z","receivedAt":"2009-07-23T14:45:59Z","isPatch":false,"sender":{"key":"crmafra2@gmail.com","avatar":null},"body":"On Thu 23.Jul'09 at 5:42:03 -0700, Jakub Narebski wrote:\n> Linus Torvalds writes:\n> \n> > On Thu, 23 Jul 2009, Carlos R. Mafra wrote:\n> > > \n> > > Is there another way to check what is going on with that anomalous lstat()?\n> > \n> > I really don't think it's the lstat any more. Your directories look small \n> > and simple, and clearly the indexing made no difference.\n> > \n> > See earlier email about using \"strace -T\" instead of \"-tt\". Also, I sent \n> > you a patch to try out just a minute ago, I think that may be it.\n> > \n> > > [ perhaps I will try 'perf' after I read how to use it ]\n> > \n> > I really like 'perf' (it does what oprofile did for me, but without the \n> > headaches), but it doesn't help with IO profiling.\n> > \n> > I've actually often wanted to have a 'strace' that shows page faults as \n> > special system calls, but it's sadly nontrivial ;(\n> \n> BTW. Would SystemTap help there? Among contributed scripts there is\n> iotimes, so perhaps it would be possible to have iotrace...\n\n\nI played a bit with 'blktrace' and 'btrace' and had two terminals\nopen side by side, one with 'strace git branch' and the other with\n'blktrace'.\n\nIt was pretty obvious that exactly at the point where 'git branch'\nwas stalling (without Linus' patch) -- which I thought had to do\nwith lstat() -- there was a flurry of activity going on in 'btrace' \noutput.\n\nIt would be nice if 'btrace' could be somehow unified with 'strace',\nif that makes any sense.\n\nHere are some numbers from my tests with blktrace (blkparse and btrace):\n\n[root@Pilar mafra]# grep git blkparse-patch.txt |wc -l\n811\n[root@Pilar mafra]# grep git blkparse-nopatch.txt |wc -l\n3479\n\nwhere those lines with 'git' are something like\n\n8,5 0 677 1.787350654 18591 I R 204488479 + 40 [git]\n8,0 0 678 1.787370489 18591 A R 204488783 + 96 <- (8,5) 137529800\n8,5 0 679 1.787371886 18591 Q R 204488783 + 96 [git]\n8,5 0 680 1.787375378 18591 G R 204488783 + 96 [git]\n8,5 0 681 1.787377613 18591 I R 204488783 + 96 [git]\n\nAnd the summary lines also indicate that the non-patched git makes\nthe disc work much harder:\n\n*************** Without Linus' patch ******************************************\n\nTotal (8,5):\n Reads Queued: 764, 20,008KiB Writes Queued: 0, 0KiB\n Read Dispatches: 764, 20,008KiB Write Dispatches: 0, 0KiB\n Reads Requeued: 0 Writes Requeued: 0\n Reads Completed: 764, 20,008KiB Writes Completed: 0, 0KiB\n Read Merges: 0, 0KiB Write Merges: 0, 0KiB\n IO unplugs: 299 Timer unplugs: 2\n\nThroughput (R/W): 4,003KiB/s / 0KiB/s\nEvents (8,5): 5,266 entries\nSkips: 0 forward (0 - 0.0%)\n\n************** With Linus' patch **********************************************\n\nTotal (sda5):\n Reads Queued: 171, 3,128KiB\t Writes Queued: 6, 24KiB\n Read Dispatches: 171, 3,128KiB\t Write Dispatches: 2, 24KiB\n Reads Requeued: 0\t\t Writes Requeued: 0\n Reads Completed: 171, 3,128KiB\t Writes Completed: 2, 24KiB\n Read Merges: 0, 0KiB\t Write Merges: 4, 16KiB\n IO unplugs: 80 \t Timer unplugs: 0\n\nThroughput (R/W): 1,632KiB/s / 12KiB/s\nEvents (sda5): 1,226 entries\nSkips: 0 forward (0 - 0.0%)\n"},{"id":"118585","messageId":"20090723160740.GA5736@Pilar.aei.mpg.de","threadId":"20205","inReplyTo":"7vtz146mgr.fsf@alter.siamese.dyndns.org","subject":"Re: Performance issue of 'git branch'","fromName":"Carlos R. Mafra","fromEmail":"crmafra2@gmail.com","sentAt":"2009-07-23T16:07:40Z","receivedAt":"2009-07-23T16:07:40Z","isPatch":false,"sender":{"key":"crmafra2@gmail.com","avatar":null},"body":"On Wed 22.Jul'09 at 21:40:36 -0700, Junio C Hamano wrote:\n> Is the cold cache performance of \"git branch\" to list your \n> local branches that important?\n\nI simply felt like something not optimal was going on, and in\nsome sense I still feel it even with Linus' patch applied...\n\nDon't get me wrong, I am super happy that Linus fixed it\nso quickly and I am grateful for that, but I am surely missing\nsome git internal reason why 'git branch' is not instantaneous\nas I _naively_ expected.\n\nHaving learned about .git/packed-refs last night, today I tried\nthis (with cold cache),\n\n[mafra@Pilar:linux-2.6]$ time awk '{print $2}' .git/packed-refs |grep heads| awk -F \"/\" '{print $3}'\n0.00user 0.00system 0:00.12elapsed 0%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (3major+311minor)pagefaults 0swaps\n27-stable\n28-stable\n29-stable\n30-stable\ndev-private\nmaster\noption\nsparse\nstern\n\nand notice how that makes my pitiful harddisc look like Linus' SSD! And the\nresult is the same. \n\n[ If some branches are not inside .git/packed-refs but are listed in .git/refs/heads \n(like some of them were last night), it would require some modification to the\nscript, but it would still be faster ]\n\nHowever, I know that I am missing something here and I would be happy to \nlearn what.\n\nThanks in advance,\nCarlos\n"},{"id":"118587","messageId":"alpine.LFD.2.01.0907230913230.21520@localhost.localdomain","threadId":"20205","inReplyTo":"20090723160740.GA5736@Pilar.aei.mpg.de","subject":"Re: Performance issue of 'git branch'","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-23T16:19:21Z","receivedAt":"2009-07-23T16:19:21Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 23 Jul 2009, Carlos R. Mafra wrote:\n> \n> Having learned about .git/packed-refs last night, today I tried\n> this (with cold cache),\n> \n> [mafra@Pilar:linux-2.6]$ time awk '{print $2}' .git/packed-refs |grep heads| awk -F \"/\" '{print $3}'\n> 0.00user 0.00system 0:00.12elapsed 0%CPU (0avgtext+0avgdata 0maxresident)k\n> 0inputs+0outputs (3major+311minor)pagefaults 0swaps\n> 27-stable\n> 28-stable\n> 29-stable\n> 30-stable\n> dev-private\n> master\n> option\n> sparse\n> stern\n> \n> and notice how that makes my pitiful harddisc look like Linus' SSD! And the\n> result is the same. \n\nThe result is the same, yes, but it doesn't do error checking.\n\nWhat \"git branch\" does over and beyond just looking at the heads is to \nalso look at the commits those heads point to. And the reason it sucks for \nyou is that the commits are pretty spread out (particularly in the index \nfile, but also in the pack-file) on disk. So each \"verify this head\" will \nlikely involve at least one seek, and possibly four or five. \n\nAnd on your disk, five seeks is a tenth of a second. You can run hdparm, \nand it will probably say that you get 30MB/s off that laptop drive - but \nwhen doing small random reads you'll probably get performance in the order \nof a few tens of kilobytes, not megabytes. (With read-ahead and \nread-around it's probably going to be mostly ~64kB IO's and you'll \nprobably get hundreds of kB per second, but you're going to care about \njust a few kB total of those).\n\nSo we _could_ make 'git branch' not actually read and verify the commits. \nIt doesn't strictly _need_ to, unless you use 'git branch -v' or \nsomething. That would speed it up further, but the verification is nice, \nand as long as performance isn't _horrible_ I think we're better off doing \nit.\n\nAfter all, you'll see the problem only once.\n\n\t\t\tLinus\n"},{"id":"118589","messageId":"alpine.LFD.2.01.0907230922530.21520@localhost.localdomain","threadId":"20205","inReplyTo":"m3prbr371l.fsf@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-23T16:25:06Z","receivedAt":"2009-07-23T16:25:06Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 23 Jul 2009, Jakub Narebski wrote:\n> \n> BTW. Would SystemTap help there? Among contributed scripts there is\n> iotimes, so perhaps it would be possible to have iotrace...\n\nThe problem I've had with all iotracers is that it's easy enough to get an \nIO trace, but it's basically almost impossible to integrate it with what \nactually _caused_ the IO.\n\nUsing 'strace -T' shows very clearly what operations are taking a long \ntime. It's very useful for seeing what you should not do for good \nperformance - including IO - and where it comes from. It's just that page \nfaults are invisible to it.\n\n\t\tLinus\n"},{"id":"118591","messageId":"alpine.DEB.1.10.0907231244020.10001@vinegar-pot.mit.edu","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907221921570.3352@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Anders Kaseorg","fromEmail":"andersk@mit.edu","sentAt":"2009-07-23T16:48:20Z","receivedAt":"2009-07-23T16:48:20Z","isPatch":false,"sender":{"key":"andersk@mit.edu","avatar":"https://avatars.githubusercontent.com/u/26471?v=4"},"body":"On Wed, 22 Jul 2009, Linus Torvalds wrote:\n> It uses the \"raw\" version of 'for_each_ref()' (which doesn't verify that \n> the ref is valid), and then does the \"type verification\" before it starts \n> doing any gentle commit lookup.\n\nI submitted essentially the same patch in May:\n http://article.gmane.org/gmane.comp.version-control.git/120097\nwith the additional optimization that we don’t need to lookup commits at \nall unless we’re using -v, --merged, --no-merged, or --contains. In my \ntests, it makes `git branch` 5 times faster on an uncached linux-2.6 \nrepository.\n\nAnders\n"},{"id":"118592","messageId":"20090723165335.GA15598@Pilar.aei.mpg.de","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907230913230.21520@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Carlos R. Mafra","fromEmail":"crmafra2@gmail.com","sentAt":"2009-07-23T16:53:35Z","receivedAt":"2009-07-23T16:53:35Z","isPatch":false,"sender":{"key":"crmafra2@gmail.com","avatar":null},"body":"On Thu 23.Jul'09 at 9:19:21 -0700, Linus Torvalds wrote:\n> > \n> > and notice how that makes my pitiful harddisc look like Linus' SSD! And the\n> > result is the same. \n> \n> The result is the same, yes, but it doesn't do error checking.\n\nOh, I see.\n\n> So we _could_ make 'git branch' not actually read and verify the commits. \n> It doesn't strictly _need_ to, unless you use 'git branch -v' or \n> something. That would speed it up further, but the verification is nice, \n> and as long as performance isn't _horrible_ I think we're better off doing \n> it.\n\nRight, but I would definitely like having some option like --dont-check to \n'git branch', and I think I would use it as default (unless experience\ntells that errors happen often).\n\n> After all, you'll see the problem only once.\n\nTrue, but paradoxically that is also the reason why I notice it and\nmakes it feel bad.\n\nEverytime I did the first 'git branch' those 5 seconds really hurt, because\nI wondered why it couldn't be done in 0s like subsequent commands.\n\nBut sure, this was definitely not a pressing issue and your patch made\nit even less. I am happy that it takes 1s now, and I really appreciated\nyour patch! \n\nThanks,\nCarlos\n"},{"id":"118593","messageId":"alpine.LFD.2.01.0907231018300.21520@localhost.localdomain","threadId":"20205","inReplyTo":"7vy6qg2aus.fsf@alter.siamese.dyndns.org","subject":"Re: Performance issue of 'git branch'","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-23T17:19:29Z","receivedAt":"2009-07-23T17:19:29Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 22 Jul 2009, Junio C Hamano wrote:\n>\n> Subject: do_one_ref(): null_sha1 check is not about broken ref\n\nAck. If we want to make it conditional at some point, we'd want to use a \ndifferent flag. \n\nI do wonder if we should simply remove the code entirely?\n\n\t\tLinus\n"},{"id":"118594","messageId":"alpine.LSU.2.00.0907231846190.30197@hermes-2.csi.cam.ac.uk","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907222009340.21520@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Tony Finch","fromEmail":"dot@dotat.at","sentAt":"2009-07-23T17:47:48Z","receivedAt":"2009-07-23T17:47:48Z","isPatch":false,"sender":{"key":"dot@dotat.at","avatar":"https://avatars.githubusercontent.com/u/68429?v=4"},"body":"On Wed, 22 Jul 2009, Linus Torvalds wrote:\n>\n> I suspect the old newton-raphson patches we had (Discussions and patches\n> back in April 2007 on this list) could be resurrected pretty easily.\n\nThat sounds interesting, but I can't find the thread you are referring to.\nDo you have a URL or a subject I can feed to Google?\n\nTony.\n-- \nf.anthony.n.finch http://dotat.at/\nGERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.\nMODERATE OR GOOD.\n"},{"id":"118596","messageId":"alpine.LFD.2.01.0907231153010.21520@localhost.localdomain","threadId":"20205","inReplyTo":"alpine.LSU.2.00.0907231846190.30197@hermes-2.csi.cam.ac.uk","subject":"Re: Performance issue of 'git branch'","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-23T18:57:42Z","receivedAt":"2009-07-23T18:57:42Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 23 Jul 2009, Tony Finch wrote:\n\n> On Wed, 22 Jul 2009, Linus Torvalds wrote:\n> >\n> > I suspect the old newton-raphson patches we had (Discussions and patches\n> > back in April 2007 on this list) could be resurrected pretty easily.\n> \n> That sounds interesting, but I can't find the thread you are referring to.\n> Do you have a URL or a subject I can feed to Google?\n\nSome googling found this:\n\n\thttp://marc.info/?l=git&m=117537594112450&w=2\n\nbut what got merged (half a year later) was a much fancier thing by Junio. \nSee sha1-lookup.c.\n\nThat original \"single iteration of newton-raphson\" patch was buggy, but \nit's perhaps interesting as a concept patch.\n\n\t\tLinus\n"},{"id":"118597","messageId":"20090723190320.GA5556@Pilar.aei.mpg.de","threadId":"20205","inReplyTo":"alpine.DEB.1.10.0907231244020.10001@vinegar-pot.mit.edu","subject":"Re: Performance issue of 'git branch'","fromName":"Carlos R. Mafra","fromEmail":"crmafra2@gmail.com","sentAt":"2009-07-23T19:03:20Z","receivedAt":"2009-07-23T19:03:20Z","isPatch":false,"sender":{"key":"crmafra2@gmail.com","avatar":null},"body":"On Thu 23.Jul'09 at 12:48:20 -0400, Anders Kaseorg wrote:\n> \n> I submitted essentially the same patch in May:\n> http://article.gmane.org/gmane.comp.version-control.git/120097\n> with the additional optimization that we don't need to lookup commits at\n> all unless we're using -v, --merged, --no-merged, or --contains. In my \n> tests, it makes `git branch` 5 times faster on an uncached linux-2.6 \n> repository.\n\nI also tested your patch even if you said that it was \"essentially the same\". \n\nBut after repeating the tests 6 times for both your and Linus' patch\n(taking care to let the system rest a bit after clearing the cache), your\npatch is faster,\n\n0.62 +/- 0.24 (Anders)\n1.35 +/- 0.23 (Linus)\n\nAnd this is the raw data for your patch,\n\n0.00user 0.01system 0:00.54elapsed 2%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (7major+727minor)pagefaults 0swaps\n\n0.00user 0.00system 0:00.18elapsed 5%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (1major+733minor)pagefaults 0swaps\n\n0.00user 0.00system 0:00.66elapsed 1%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (9major+723minor)pagefaults 0swaps\n\n0.00user 0.01system 0:00.74elapsed 2%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (14major+720minor)pagefaults 0swaps\n\n0.00user 0.00system 0:00.80elapsed 0%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (16major+718minor)pagefaults 0swaps\n\n0.00user 0.00system 0:00.83elapsed 0%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (16major+718minor)pagefaults 0swaps\n\n\nand for Linus'\n\n0.00user 0.01system 0:01.56elapsed 1%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (43major+755minor)pagefaults 0swaps\n\n0.00user 0.01system 0:01.09elapsed 1%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (24major+775minor)pagefaults 0swaps\n\n0.00user 0.01system 0:01.33elapsed 1%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (32major+767minor)pagefaults 0swaps\n\n0.00user 0.00system 0:01.53elapsed 0%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (39major+760minor)pagefaults 0swaps\n\n0.00user 0.01system 0:01.06elapsed 2%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (24major+775minor)pagefaults 0swaps\n\n0.00user 0.00system 0:01.54elapsed 0%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (39major+760minor)pagefaults 0swaps\n"},{"id":"118598","messageId":"alpine.LFD.2.01.0907231158280.21520@localhost.localdomain","threadId":"20205","inReplyTo":"20090723165335.GA15598@Pilar.aei.mpg.de","subject":"Re: Performance issue of 'git branch'","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-23T19:05:24Z","receivedAt":"2009-07-23T19:05:24Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 23 Jul 2009, Carlos R. Mafra wrote:\n>\n> Everytime I did the first 'git branch' those 5 seconds really hurt, because\n> I wondered why it couldn't be done in 0s like subsequent commands.\n> \n> But sure, this was definitely not a pressing issue and your patch made\n> it even less. I am happy that it takes 1s now, and I really appreciated\n> your patch! \n\nYou could try something like this (on _top_ of the previous patch). \n\nNot very exhaustively tested, but it's pretty simple.\n\nIt will still do _some_ object lookups. In particular, it will do the HEAD \nlookup in 'print_ref_list()', even if it's not strictly necessary. But it \nshould cut down the noise further.\n\n\t\tLinus\n\n---\n builtin-branch.c | 24 ++++++++++++++----------\n 1 files changed, 14 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin-branch.c b/builtin-branch.c\nindex 54a89ff..82c2cf0 100644\n--- a/builtin-branch.c\n+++ b/builtin-branch.c\n@@ -191,7 +191,7 @@ struct ref_item {\n \n struct ref_list {\n \tstruct rev_info revs;\n-\tint index, alloc, maxwidth;\n+\tint index, alloc, maxwidth, verbose;\n \tstruct ref_item *list;\n \tstruct commit_list *with_commit;\n \tint kinds;\n@@ -244,17 +244,20 @@ static int append_ref(const char *refname, const unsigned char *sha1, int flags,\n \tif ((kind & ref_list->kinds) == 0)\n \t\treturn 0;\n \n-\tcommit = lookup_commit_reference_gently(sha1, 1);\n-\tif (!commit)\n-\t\treturn error(\"branch '%s' does not point at a commit\", refname);\n+\tcommit = NULL;\n+\tif (ref_list->verbose || ref_list->with_commit || merge_filter != NO_FILTER) {\n+\t\tcommit = lookup_commit_reference_gently(sha1, 1);\n+\t\tif (!commit)\n+\t\t\treturn error(\"branch '%s' does not point at a commit\", refname);\n \n-\t/* Filter with with_commit if specified */\n-\tif (!is_descendant_of(commit, ref_list->with_commit))\n-\t\treturn 0;\n+\t\t/* Filter with with_commit if specified */\n+\t\tif (!is_descendant_of(commit, ref_list->with_commit))\n+\t\t\treturn 0;\n \n-\tif (merge_filter != NO_FILTER)\n-\t\tadd_pending_object(&ref_list->revs,\n-\t\t\t\t (struct object *)commit, refname);\n+\t\tif (merge_filter != NO_FILTER)\n+\t\t\tadd_pending_object(&ref_list->revs,\n+\t\t\t\t\t (struct object *)commit, refname);\n+\t}\n \n \t/* Resize buffer */\n \tif (ref_list->index >= ref_list->alloc) {\n@@ -423,6 +426,7 @@ static void print_ref_list(int kinds, int detached, int verbose, int abbrev, str\n \n \tmemset(&ref_list, 0, sizeof(ref_list));\n \tref_list.kinds = kinds;\n+\tref_list.verbose = verbose;\n \tref_list.with_commit = with_commit;\n \tif (merge_filter != NO_FILTER)\n \t\tinit_revisions(&ref_list.revs, NULL);\n"},{"id":"118599","messageId":"alpine.LFD.2.01.0907231212180.21520@localhost.localdomain","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907231158280.21520@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-23T19:13:41Z","receivedAt":"2009-07-23T19:13:41Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 23 Jul 2009, Linus Torvalds wrote:\n> \n> You could try something like this (on _top_ of the previous patch). \n> \n> Not very exhaustively tested, but it's pretty simple.\n> \n> It will still do _some_ object lookups. In particular, it will do the HEAD \n> lookup in 'print_ref_list()', even if it's not strictly necessary. But it \n> should cut down the noise further.\n\nAnd this (on top of them all) will basically avoid even that one.\n\nIn fact, I think this is a cleanup. I think I'll resubmit the whole series \nwith proper commit messages etc.\n\n\t\tLinus\n\n---\n builtin-branch.c | 38 +++++++++++++++++++++++---------------\n 1 files changed, 23 insertions(+), 15 deletions(-)\n\ndiff --git a/builtin-branch.c b/builtin-branch.c\nindex 82c2cf0..1a03d5f 100644\n--- a/builtin-branch.c\n+++ b/builtin-branch.c\n@@ -191,7 +191,7 @@ struct ref_item {\n \n struct ref_list {\n \tstruct rev_info revs;\n-\tint index, alloc, maxwidth, verbose;\n+\tint index, alloc, maxwidth, verbose, abbrev;\n \tstruct ref_item *list;\n \tstruct commit_list *with_commit;\n \tint kinds;\n@@ -418,15 +418,34 @@ static int calc_maxwidth(struct ref_list *refs)\n \treturn w;\n }\n \n+\n+static void show_detached(struct ref_list *ref_list)\n+{\n+\tstruct commit *head_commit = lookup_commit_reference_gently(head_sha1, 1);\n+\n+\tif (head_commit && is_descendant_of(head_commit, ref_list->with_commit)) {\n+\t\tstruct ref_item item;\n+\t\titem.name = xstrdup(\"(no branch)\");\n+\t\titem.len = strlen(item.name);\n+\t\titem.kind = REF_LOCAL_BRANCH;\n+\t\titem.dest = NULL;\n+\t\titem.commit = head_commit;\n+\t\tif (item.len > ref_list->maxwidth)\n+\t\t\tref_list->maxwidth = item.len;\n+\t\tprint_ref_item(&item, ref_list->maxwidth, ref_list->verbose, ref_list->abbrev, 1, \"\");\n+\t\tfree(item.name);\n+\t}\n+}\n+\n static void print_ref_list(int kinds, int detached, int verbose, int abbrev, struct commit_list *with_commit)\n {\n \tint i;\n \tstruct ref_list ref_list;\n-\tstruct commit *head_commit = lookup_commit_reference_gently(head_sha1, 1);\n \n \tmemset(&ref_list, 0, sizeof(ref_list));\n \tref_list.kinds = kinds;\n \tref_list.verbose = verbose;\n+\tref_list.abbrev = abbrev;\n \tref_list.with_commit = with_commit;\n \tif (merge_filter != NO_FILTER)\n \t\tinit_revisions(&ref_list.revs, NULL);\n@@ -446,19 +465,8 @@ static void print_ref_list(int kinds, int detached, int verbose, int abbrev, str\n \tqsort(ref_list.list, ref_list.index, sizeof(struct ref_item), ref_cmp);\n \n \tdetached = (detached && (kinds & REF_LOCAL_BRANCH));\n-\tif (detached && head_commit &&\n-\t is_descendant_of(head_commit, with_commit)) {\n-\t\tstruct ref_item item;\n-\t\titem.name = xstrdup(\"(no branch)\");\n-\t\titem.len = strlen(item.name);\n-\t\titem.kind = REF_LOCAL_BRANCH;\n-\t\titem.dest = NULL;\n-\t\titem.commit = head_commit;\n-\t\tif (item.len > ref_list.maxwidth)\n-\t\t\tref_list.maxwidth = item.len;\n-\t\tprint_ref_item(&item, ref_list.maxwidth, verbose, abbrev, 1, \"\");\n-\t\tfree(item.name);\n-\t}\n+\tif (detached)\n+\t\tshow_detached(&ref_list);\n \n \tfor (i = 0; i < ref_list.index; i++) {\n \t\tint current = !detached &&\n"},{"id":"118603","messageId":"20090723195548.GA28494@Pilar.aei.mpg.de","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907231212180.21520@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Carlos R. Mafra","fromEmail":"crmafra2@gmail.com","sentAt":"2009-07-23T19:55:48Z","receivedAt":"2009-07-23T19:55:48Z","isPatch":false,"sender":{"key":"crmafra2@gmail.com","avatar":null},"body":"On Thu 23.Jul'09 at 12:13:41 -0700, Linus Torvalds wrote:\n> > It will still do _some_ object lookups. In particular, it will do the HEAD \n> > lookup in 'print_ref_list()', even if it's not strictly necessary. But it \n> > should cut down the noise further.\n> \n> And this (on top of them all) will basically avoid even that one.\n\nOk, I applied (both) on top of the first one.\n\nAfter 7 tests I got these, \n\ntime:\n\n 0.61 +/- 0.08\n\nGIT_DEBUG_LOOKUP=1 git branch |wc -l\n \n 9\n \nwhich are in fact only the branches list.\n\nCompared to yesterday, that is a huge improvement (0.6s vs 5.7s)\nand (9 vs 2200+). At least for me 0.6s is \"instantaneous\", so\nthe issue is really gone.\n\nThanks a lot to everyone!\n"},{"id":"118605","messageId":"alpine.LSU.2.00.0907232310220.22113@hermes-2.csi.cam.ac.uk","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907231153010.21520@localhost.localdomain","subject":"Newton-Raphson, was Re: Performance issue of 'git branch'","fromName":"Tony Finch","fromEmail":"dot@dotat.at","sentAt":"2009-07-23T22:48:43Z","receivedAt":"2009-07-23T22:48:43Z","isPatch":false,"sender":{"key":"dot@dotat.at","avatar":"https://avatars.githubusercontent.com/u/68429?v=4"},"body":"On Thu, 23 Jul 2009, Linus Torvalds wrote:\n>\n> Some googling found this:\n> \thttp://marc.info/?l=git&m=117537594112450&w=2\n> but what got merged (half a year later) was a much fancier thing by Junio.\n> See sha1-lookup.c.\n\nThanks. Edésio Costa e Silva also gave me a useful pointer.\n\n> That original \"single iteration of newton-raphson\" patch was buggy, but\n> it's perhaps interesting as a concept patch.\n\nI think Newton-Raphson is a brilliant but misleading idea. (As Junio said,\n\"egg of Columbus\" - it certainly blew my mind!) However, Newton's method\nworks with smooth curves, but a pack index is a straight line plus\nstochastic deviations. If you try to apply Newton's method then the more\nyou zoom in the more the random variations will send you away from the\nplace you want to be. So I think your first N-R patch was closer to being\nright than its successors.\n\nWhat you should do is ONE linear interpolation on the entire index. (i.e.\nIf you have N objects in the pack and you want to find one with SHA-1 id\nS, take the top four bytes of S and multiply by N/2^32.) Note that if you\ndo a level-1 256-way fan-out lookup first then the random variations will\nmake you LESS likely to land near the right place.\n\nAfter doing the first-order linear interpolation, it's probably sensible\nto do a page-wise linear search (in case you don't land directly on\nthe page containing the target SHA-1) then a binary search within the\nfinal page for efficiency with a hot cache.\n\nThis should give you O(1) seeks in the index per object lookup.\n\nTony.\n-- \nf.anthony.n.finch http://dotat.at/\nGERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.\nMODERATE OR GOOD."},{"id":"118606","messageId":"alpine.DEB.1.00.0907240114410.8306@pacific.mpi-cbg.de","threadId":"20205","inReplyTo":"alpine.LSU.2.00.0907232310220.22113@hermes-2.csi.cam.ac.uk","subject":"Re: Newton-Raphson, was Re: Performance issue of 'git branch'","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-07-23T23:24:21Z","receivedAt":"2009-07-23T23:24:21Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 23 Jul 2009, Tony Finch wrote:\n\n> I think Newton-Raphson is a brilliant but misleading idea. (As Junio \n> said, \"egg of Columbus\" - it certainly blew my mind!) However, Newton's \n> method works with smooth curves, but a pack index is a straight line \n> plus stochastic deviations. If you try to apply Newton's method then the \n> more you zoom in the more the random variations will send you away from \n> the place you want to be.\n\nNo.\n\nThink about it, absent any further information than \"it is a hash, i.e. \ndistributed pretty equally in _any_ byte\", even subsets of a sorted list \nwill me more or less linear. And assuming that they are linear is _still_ \nyour best bet.\n\nAssuming that subsets of said sorted list will _still_ minimize the \naverage number of steps to take until you find the correct entry.\n\nUnless you have more information about the nature of the hashes, of \ncourse.\n\n> This should give you O(1) seeks in the index per object lookup.\n\nThere is no way to achieve that, best thing you can hope for is _expected_ \nO(1) (e.g. with a hashmap, with exponential worst case).\n\nCiao,\nDscho\n"},{"id":"118607","messageId":"alpine.LSU.2.00.0907240031470.12180@hermes-2.csi.cam.ac.uk","threadId":"20205","inReplyTo":"alpine.DEB.1.00.0907240114410.8306@pacific.mpi-cbg.de","subject":"Re: Newton-Raphson, was Re: Performance issue of 'git branch'","fromName":"Tony Finch","fromEmail":"dot@dotat.at","sentAt":"2009-07-23T23:50:47Z","receivedAt":"2009-07-23T23:50:47Z","isPatch":false,"sender":{"key":"dot@dotat.at","avatar":"https://avatars.githubusercontent.com/u/68429?v=4"},"body":"On Fri, 24 Jul 2009, Johannes Schindelin wrote:\n>\n> Think about it, absent any further information than \"it is a hash, i.e.\n> distributed pretty equally in _any_ byte\", even subsets of a sorted list\n> will me more or less linear. And assuming that they are linear is _still_\n> your best bet.\n\nThe even distribution of the lower-order bytes is irrelevant. We're\nlooking at the top 20-ish bits for a pack with a million-ish objects. The\nmore you zoom in the less linear a sorted list of hashes will be, so\nassuming linearity at all scales is wrong. It's a bit like fractal\nmountains.\n\n> There is no way to achieve [O(1) seeks], best thing you can hope for is\n> _expected_ O(1) (e.g. with a hashmap, with exponential worst case).\n\nOf course it's expected. However the worst case is nowhere near\nexponential: it's linear because the second-order search is a linear\npagewise scan. But I think in practice, the larger the pack the more that\nthe randomization of the hash function will smooth out performance\noddities. (Sorry, I don't know enough statistics to be able to say what\nthe expected error of the linear interpolation is, though I expect it's a\nfairly simple formula.) For small packs the number of seeks is 1 anyway.\n\nTony.\n-- \nf.anthony.n.finch http://dotat.at/\nGERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.\nMODERATE OR GOOD.\n"},{"id":"118608","messageId":"alpine.DEB.1.00.0907240241430.8306@pacific.mpi-cbg.de","threadId":"20205","inReplyTo":"alpine.LSU.2.00.0907240031470.12180@hermes-2.csi.cam.ac.uk","subject":"Re: Newton-Raphson, was Re: Performance issue of 'git branch'","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-07-24T00:43:59Z","receivedAt":"2009-07-24T00:43:59Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Fri, 24 Jul 2009, Tony Finch wrote:\n\n> On Fri, 24 Jul 2009, Johannes Schindelin wrote:\n> >\n> > Think about it, absent any further information than \"it is a hash, i.e.\n> > distributed pretty equally in _any_ byte\", even subsets of a sorted list\n> > will me more or less linear. And assuming that they are linear is _still_\n> > your best bet.\n> \n> The even distribution of the lower-order bytes is irrelevant.\n\nI was not talking about lower-order bytes. All bytes are pretty much \nevenly distributed. That's why SHA-1 is a good hash.\n\n> We're looking at the top 20-ish bits for a pack with a million-ish \n> objects. The more you zoom in the less linear a sorted list of hashes \n> will be, so assuming linearity at all scales is wrong. It's a bit like \n> fractal mountains.\n\nIf you really find irregularities like that, then SHA-1 is really a lousy \nhash. Irregularities like this are typically exploitable.\n\nIf you know of such an irregularity, you might want to write a paper that \nSHA-1 is broken and get famous.\n\n> > There is no way to achieve [O(1) seeks], best thing you can hope for \n> > is _expected_ O(1) (e.g. with a hashmap, with exponential worst case).\n> \n> Of course it's expected. However the worst case is nowhere near\n> exponential: it's linear because the second-order search is a linear\n> pagewise scan. But I think in practice, the larger the pack the more that\n> the randomization of the hash function will smooth out performance\n> oddities. (Sorry, I don't know enough statistics to be able to say what\n> the expected error of the linear interpolation is, though I expect it's a\n> fairly simple formula.) For small packs the number of seeks is 1 anyway.\n\nI will believe it when I see it.\n\nCiao,\nDscho\n"},{"id":"118653","messageId":"alpine.LFD.2.01.0907241327410.3960@localhost.localdomain","threadId":"20205","inReplyTo":"20090723195548.GA28494@Pilar.aei.mpg.de","subject":"Re: Performance issue of 'git branch'","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-24T20:36:13Z","receivedAt":"2009-07-24T20:36:13Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 23 Jul 2009, Carlos R. Mafra wrote:\n>\n> After 7 tests I got these, \n> \n> time:\n> \n> 0.61 +/- 0.08\n\nBtw, I think 0.61s is still too much. Can you send me the output of \n'strace -Ttt' on your machine?\n\nIt's entirely possible that it's all the actual binary (and shared \nlibrary) loading, of course. You do have a slow harddisk. But it takes \n0.035s for me, and I'm wondering if there is something else than just CPU \nspeed and IO speed accounting for the 20x performance difference.\n\n(That said, maybe 20x is right - my SSD latency almost certainly is 20x \nbetter).\n\n\t\t\tLinus\n"},{"id":"118654","messageId":"alpine.LFD.2.01.0907241346450.3960@localhost.localdomain","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907241327410.3960@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-24T20:47:22Z","receivedAt":"2009-07-24T20:47:22Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Fri, 24 Jul 2009, Linus Torvalds wrote:\n> \n> Btw, I think 0.61s is still too much. Can you send me the output of \n> 'strace -Ttt' on your machine?\n\nNever mind. I'm seeing even worse behavior on a laptop I just dug up \n(another 4200 rpm harddisk).\n\nI'll dig some more.\n\n\t\tLinus\n"},{"id":"118656","messageId":"alpine.LFD.2.01.0907241349390.3960@localhost.localdomain","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907241346450.3960@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-24T21:21:20Z","receivedAt":"2009-07-24T21:21:20Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Fri, 24 Jul 2009, Linus Torvalds wrote:\n> \n> Never mind. I'm seeing even worse behavior on a laptop I just dug up \n> (another 4200 rpm harddisk).\n> \n> I'll dig some more.\n\nYeah, it seems to be the loading overhead. I'm seeing a 'time git branch' \ntake 1.2s in the cold-cache case, in a directory that isn't even a git \ndirectory.\n\nAnd 80% of it comes before we even get to 'main()'. Shared library \nloading, SELinux crud etc. A lot of it seems to be 'libfreebl3' and \n'libselinux', which is some crazy sh*t.\n\nIt seems to be all from 'curl' support.\n\nThat seems _really_ sad. Lookie here:\n\n [torvalds@nehalem git]$ ldd git\n\tlinux-vdso.so.1 => (0x00007fff61da7000)\n\tlibcurl.so.4 => /usr/lib64/libcurl.so.4 (0x00007f2f1a498000)\n\tlibz.so.1 => /lib64/libz.so.1 (0x0000003cdb800000)\n\tlibcrypto.so.8 => /usr/lib64/libcrypto.so.8 (0x0000003ba7a00000)\n\tlibpthread.so.0 => /lib64/libpthread.so.0 (0x0000003cdb400000)\n\tlibc.so.6 => /lib64/libc.so.6 (0x0000003cda800000)\n\tlibidn.so.11 => /lib64/libidn.so.11 (0x0000003ceaa00000)\n\tlibssh2.so.1 => /usr/lib64/libssh2.so.1 (0x0000003ba8e00000)\n\tlibldap-2.4.so.2 => /usr/lib64/libldap-2.4.so.2 (0x00007f2f1a250000)\n\tlibrt.so.1 => /lib64/librt.so.1 (0x0000003cdbc00000)\n\tlibgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x0000003ce6e00000)\n\tlibkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x0000003ce7e00000)\n\tlibk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x0000003ce7200000)\n\tlibcom_err.so.2 => /lib64/libcom_err.so.2 (0x0000003ce6a00000)\n\tlibssl3.so => /lib64/libssl3.so (0x0000003490200000)\n\tlibsmime3.so => /lib64/libsmime3.so (0x000000348fe00000)\n\tlibnss3.so => /lib64/libnss3.so (0x000000348f600000)\n\tlibplds4.so => /lib64/libplds4.so (0x0000003cbc800000)\n\tlibplc4.so => /lib64/libplc4.so (0x0000003cbdc00000)\n\tlibnspr4.so => /lib64/libnspr4.so (0x0000003cbd800000)\n\tlibdl.so.2 => /lib64/libdl.so.2 (0x0000003cdb000000)\n\t/lib64/ld-linux-x86-64.so.2 (0x0000003cda400000)\n\tlibssl.so.8 => /usr/lib64/libssl.so.8 (0x0000003ba7e00000)\n\tliblber-2.4.so.2 => /usr/lib64/liblber-2.4.so.2 (0x0000003ceee00000)\n\tlibresolv.so.2 => /lib64/libresolv.so.2 (0x0000003ce5600000)\n\tlibsasl2.so.2 => /usr/lib64/libsasl2.so.2 (0x00007f2f1a030000)\n\tlibkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x0000003ce7a00000)\n\tlibkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x0000003ce7600000)\n\tlibnssutil3.so => /lib64/libnssutil3.so (0x000000348fa00000)\n\tlibcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f2f19df8000)\n\tlibselinux.so.1 => /lib64/libselinux.so.1 (0x0000003cdc400000)\n\tlibfreebl3.so => /lib64/libfreebl3.so (0x00007f2f19b99000)\n [torvalds@nehalem git]$ make -j16 NO_CURL=1\n [torvalds@nehalem git]$ ldd git\n\tlinux-vdso.so.1 => (0x00007fff2f960000)\n\tlibz.so.1 => /lib64/libz.so.1 (0x0000003cdb800000)\n\tlibcrypto.so.8 => /usr/lib64/libcrypto.so.8 (0x0000003ba7a00000)\n\tlibpthread.so.0 => /lib64/libpthread.so.0 (0x0000003cdb400000)\n\tlibc.so.6 => /lib64/libc.so.6 (0x0000003cda800000)\n\tlibdl.so.2 => /lib64/libdl.so.2 (0x0000003cdb000000)\n\t/lib64/ld-linux-x86-64.so.2 (0x0000003cda400000)\n\nWhat a huge difference!\n\nAnd the NO_CURL version really does load a lot faster in cold-cache. We're \nnot talking small differences:\n\n - compiled with NO_CURL, five runs of \"echo 3 > /proc/sys/vm/drop_caches\" \n followed by \"time git branch\":\n\n\treal\t0m0.654s\n\treal\t0m0.562s\n\treal\t0m0.519s\n\treal\t0m0.534s\n\treal\t0m0.734s\n\n Total number of system calls: 194\n\n - compiled with curl, same thing:\n\n\treal\t0m1.503s\n\treal\t0m1.455s\n\treal\t0m1.267s\n\treal\t0m1.819s\n\treal\t0m0.985s\n\n Total number of system calls: 407!\n\nie we're talking a _huge_ hit in startup times for that curl support. \nThat's really really sad - especially considering how all the curl support \nis for very random occasional stuff. I never use it myself, for example, \nsince I don't use http at all. And even for people who do, they only need \nit for non-local operations.\n\nI wonder if there is some way to only load the crazy curl stuff when we \nactually want open a http: connection.\n\n\t\t\tLinus\n"},{"id":"118665","messageId":"alpine.LFD.2.01.0907241505400.3960@localhost.localdomain","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907241349390.3960@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-24T22:13:07Z","receivedAt":"2009-07-24T22:13:07Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\nOn Fri, 24 Jul 2009, Linus Torvalds wrote:\n> \n> ie we're talking a _huge_ hit in startup times for that curl support. \n> That's really really sad - especially considering how all the curl support \n> is for very random occasional stuff. I never use it myself, for example, \n> since I don't use http at all. And even for people who do, they only need \n> it for non-local operations.\n> \n> I wonder if there is some way to only load the crazy curl stuff when we \n> actually want open a http: connection.\n\nHere's the simple step#1: make 'git-http-fetch' be an external program \nrather than a built-in.\n\nSadly, I have no idea hot to turn the transport.c code into an external \nwalker sanely (turn the ref/object walkers into an exec of an external \nprogram). So we still end up linking with curl. But maybe somebody \n(Daniel? Dscho?) who knows the transport code could try to make it an \nexternal process?\n\nThe performance angle of http fetching is non-existent, we really should \ntry very hard to make the curl-dependent parts be in a binary of their \nown.\n\n\t\tLinus\n\n---\n>From 3cfc50d497266dc73a414ed1460b36b712ad10de Mon Sep 17 00:00:00 2001\nFrom: Linus Torvalds \nDate: Fri, 24 Jul 2009 14:54:55 -0700\nSubject: [PATCH] git-http-fetch: not a builtin\n\nWe should really try to avoid having a dependency on the curl libraries\nfor the core 'git' executable. It adds huge overheads, for no advantage.\n\nThis splits up git-http-fetch so that it isn't built-in. We still do\nend up linking with curl for the git binary due to the transport.c http\nwalker, but that's at least partially an independent issue.\n\nSigned-off-by: Linus Torvalds \n---\n Makefile | 8 +++++++-\n git.c | 3 ---\n builtin-http-fetch.c => http-fetch.c | 5 ++++-\n 3 files changed, 11 insertions(+), 5 deletions(-)\n rename builtin-http-fetch.c => http-fetch.c (95%)\n\ndiff --git a/Makefile b/Makefile\nindex bde27ed..8cbd863 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -978,9 +978,12 @@ else\n \telse\n \t\tCURL_LIBCURL = -lcurl\n \tendif\n-\tBUILTIN_OBJS += builtin-http-fetch.o\n+\tPROGRAMS += git-http-fetch$X\n+\n+\t# FIXME! Sadly 'transport.c' still needs these for the builtin case\n \tEXTLIBS += $(CURL_LIBCURL)\n \tLIB_OBJS += http.o http-walker.o\n+\n \tcurl_check := $(shell (echo 070908; curl-config --vernum) | sort -r | sed -ne 2p)\n \tifeq \"$(curl_check)\" \"070908\"\n \t\tifndef NO_EXPAT\n@@ -1485,6 +1488,9 @@ git-imap-send$X: imap-send.o $(GITLIBS)\n \n http.o http-walker.o http-push.o transport.o: http.h\n \n+git-http-fetch$X: revision.o http.o http-push.o $(GITLIBS)\n+\t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \\\n+\t\t$(LIBS) $(CURL_LIBCURL) $(EXPAT_LIBEXPAT)\n git-http-push$X: revision.o http.o http-push.o $(GITLIBS)\n \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \\\n \t\t$(LIBS) $(CURL_LIBCURL) $(EXPAT_LIBEXPAT)\ndiff --git a/git.c b/git.c\nindex 807d875..c1e8f05 100644\n--- a/git.c\n+++ b/git.c\n@@ -309,9 +309,6 @@ static void handle_internal_command(int argc, const char **argv)\n \t\t{ \"get-tar-commit-id\", cmd_get_tar_commit_id },\n \t\t{ \"grep\", cmd_grep, RUN_SETUP | USE_PAGER },\n \t\t{ \"help\", cmd_help },\n-#ifndef NO_CURL\n-\t\t{ \"http-fetch\", cmd_http_fetch, RUN_SETUP },\n-#endif\n \t\t{ \"init\", cmd_init_db },\n \t\t{ \"init-db\", cmd_init_db },\n \t\t{ \"log\", cmd_log, RUN_SETUP | USE_PAGER },\ndiff --git a/builtin-http-fetch.c b/http-fetch.c\nsimilarity index 95%\nrename from builtin-http-fetch.c\nrename to http-fetch.c\nindex f3e63d7..e8f44ba 100644\n--- a/builtin-http-fetch.c\n+++ b/http-fetch.c\n@@ -1,8 +1,9 @@\n #include \"cache.h\"\n #include \"walker.h\"\n \n-int cmd_http_fetch(int argc, const char **argv, const char *prefix)\n+int main(int argc, const char **argv)\n {\n+\tconst char *prefix;\n \tstruct walker *walker;\n \tint commits_on_stdin = 0;\n \tint commits;\n@@ -18,6 +19,8 @@ int cmd_http_fetch(int argc, const char **argv, const char *prefix)\n \tint get_verbosely = 0;\n \tint get_recover = 0;\n \n+\tprefix = setup_git_directory();\n+\n \tgit_config(git_default_config, NULL);\n \n \twhile (arg < argc && argv[arg][0] == '-') {\n-- \n1.6.4.rc1.5.gb84f\n"},{"id":"118666","messageId":"alpine.DEB.1.10.0907241518120.28013@asgard.lang.hm","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907241505400.3960@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"","fromEmail":"david@lang.hm","sentAt":"2009-07-24T22:18:44Z","receivedAt":"2009-07-24T22:18:44Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Fri, 24 Jul 2009, Linus Torvalds wrote:\n\n> On Fri, 24 Jul 2009, Linus Torvalds wrote:\n>>\n>> ie we're talking a _huge_ hit in startup times for that curl support.\n>> That's really really sad - especially considering how all the curl support\n>> is for very random occasional stuff. I never use it myself, for example,\n>> since I don't use http at all. And even for people who do, they only need\n>> it for non-local operations.\n>>\n>> I wonder if there is some way to only load the crazy curl stuff when we\n>> actually want open a http: connection.\n>\n> Here's the simple step#1: make 'git-http-fetch' be an external program\n> rather than a built-in.\n>\n> Sadly, I have no idea hot to turn the transport.c code into an external\n> walker sanely (turn the ref/object walkers into an exec of an external\n> program). So we still end up linking with curl. But maybe somebody\n> (Daniel? Dscho?) who knows the transport code could try to make it an\n> external process?\n>\n> The performance angle of http fetching is non-existent, we really should\n> try very hard to make the curl-dependent parts be in a binary of their\n> own.\n\nwhat does the performance look like if you just do a static compile \ninstead?\n\nDavid Lang\n\n> \t\tLinus\n>\n> ---\n>> From 3cfc50d497266dc73a414ed1460b36b712ad10de Mon Sep 17 00:00:00 2001\n> From: Linus Torvalds \n> Date: Fri, 24 Jul 2009 14:54:55 -0700\n> Subject: [PATCH] git-http-fetch: not a builtin\n>\n> We should really try to avoid having a dependency on the curl libraries\n> for the core 'git' executable. It adds huge overheads, for no advantage.\n>\n> This splits up git-http-fetch so that it isn't built-in. We still do\n> end up linking with curl for the git binary due to the transport.c http\n> walker, but that's at least partially an independent issue.\n>\n> Signed-off-by: Linus Torvalds \n> ---\n> Makefile | 8 +++++++-\n> git.c | 3 ---\n> builtin-http-fetch.c => http-fetch.c | 5 ++++-\n> 3 files changed, 11 insertions(+), 5 deletions(-)\n> rename builtin-http-fetch.c => http-fetch.c (95%)\n>\n> diff --git a/Makefile b/Makefile\n> index bde27ed..8cbd863 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -978,9 +978,12 @@ else\n> \telse\n> \t\tCURL_LIBCURL = -lcurl\n> \tendif\n> -\tBUILTIN_OBJS += builtin-http-fetch.o\n> +\tPROGRAMS += git-http-fetch$X\n> +\n> +\t# FIXME! Sadly 'transport.c' still needs these for the builtin case\n> \tEXTLIBS += $(CURL_LIBCURL)\n> \tLIB_OBJS += http.o http-walker.o\n> +\n> \tcurl_check := $(shell (echo 070908; curl-config --vernum) | sort -r | sed -ne 2p)\n> \tifeq \"$(curl_check)\" \"070908\"\n> \t\tifndef NO_EXPAT\n> @@ -1485,6 +1488,9 @@ git-imap-send$X: imap-send.o $(GITLIBS)\n>\n> http.o http-walker.o http-push.o transport.o: http.h\n>\n> +git-http-fetch$X: revision.o http.o http-push.o $(GITLIBS)\n> +\t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \\\n> +\t\t$(LIBS) $(CURL_LIBCURL) $(EXPAT_LIBEXPAT)\n> git-http-push$X: revision.o http.o http-push.o $(GITLIBS)\n> \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \\\n> \t\t$(LIBS) $(CURL_LIBCURL) $(EXPAT_LIBEXPAT)\n> diff --git a/git.c b/git.c\n> index 807d875..c1e8f05 100644\n> --- a/git.c\n> +++ b/git.c\n> @@ -309,9 +309,6 @@ static void handle_internal_command(int argc, const char **argv)\n> \t\t{ \"get-tar-commit-id\", cmd_get_tar_commit_id },\n> \t\t{ \"grep\", cmd_grep, RUN_SETUP | USE_PAGER },\n> \t\t{ \"help\", cmd_help },\n> -#ifndef NO_CURL\n> -\t\t{ \"http-fetch\", cmd_http_fetch, RUN_SETUP },\n> -#endif\n> \t\t{ \"init\", cmd_init_db },\n> \t\t{ \"init-db\", cmd_init_db },\n> \t\t{ \"log\", cmd_log, RUN_SETUP | USE_PAGER },\n> diff --git a/builtin-http-fetch.c b/http-fetch.c\n> similarity index 95%\n> rename from builtin-http-fetch.c\n> rename to http-fetch.c\n> index f3e63d7..e8f44ba 100644\n> --- a/builtin-http-fetch.c\n> +++ b/http-fetch.c\n> @@ -1,8 +1,9 @@\n> #include \"cache.h\"\n> #include \"walker.h\"\n>\n> -int cmd_http_fetch(int argc, const char **argv, const char *prefix)\n> +int main(int argc, const char **argv)\n> {\n> +\tconst char *prefix;\n> \tstruct walker *walker;\n> \tint commits_on_stdin = 0;\n> \tint commits;\n> @@ -18,6 +19,8 @@ int cmd_http_fetch(int argc, const char **argv, const char *prefix)\n> \tint get_verbosely = 0;\n> \tint get_recover = 0;\n>\n> +\tprefix = setup_git_directory();\n> +\n> \tgit_config(git_default_config, NULL);\n>\n> \twhile (arg < argc && argv[arg][0] == '-') {\n>\n"},{"id":"118670","messageId":"alpine.LFD.2.01.0907241529420.3960@localhost.localdomain","threadId":"20205","inReplyTo":"alpine.DEB.1.10.0907241518120.28013@asgard.lang.hm","subject":"Re: Performance issue of 'git branch'","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-24T22:42:24Z","receivedAt":"2009-07-24T22:42:24Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Fri, 24 Jul 2009, david@lang.hm wrote:\n> \n> what does the performance look like if you just do a static compile instead?\n\nI don't even know - I don't have a static version of curl. I could install \none, of course, but since I don't think that's the solution anyway, I'm \nnot going to bother.\n\nThe real solution really is to not have curl support in the main binary.\n\nOne option might be to make _all_ the transport code be outside of the \ncore binary, or course. That's a fairly simple but somewhat sad solution \n(ie make all of push/pull/fetch/clone/ls-remote/etc be external binaries)\n\n\t\tLinus\n"},{"id":"118676","messageId":"alpine.DEB.1.10.0907241545340.28013@asgard.lang.hm","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907241529420.3960@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"","fromEmail":"david@lang.hm","sentAt":"2009-07-24T22:46:29Z","receivedAt":"2009-07-24T22:46:29Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Fri, 24 Jul 2009, Linus Torvalds wrote:\n\n> On Fri, 24 Jul 2009, david@lang.hm wrote:\n>>\n>> what does the performance look like if you just do a static compile instead?\n>\n> I don't even know - I don't have a static version of curl. I could install\n> one, of course, but since I don't think that's the solution anyway, I'm\n> not going to bother.\n\nI wasn't thinking a static version of curl, I was thinking a static \nversion of the git binaries. see how fast things could be if no startup \nlinking was nessasary.\n\nDavid Lang\n\n> The real solution really is to not have curl support in the main binary.\n>\n> One option might be to make _all_ the transport code be outside of the\n> core binary, or course. That's a fairly simple but somewhat sad solution\n> (ie make all of push/pull/fetch/clone/ls-remote/etc be external binaries)\n>\n> \t\tLinus\n>\n"},{"id":"118681","messageId":"20090724225415.GC6832@mit.edu","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907241349390.3960@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Theodore Tso","fromEmail":"tytso@mit.edu","sentAt":"2009-07-24T22:54:15Z","receivedAt":"2009-07-24T22:54:15Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Fri, Jul 24, 2009 at 02:21:20PM -0700, Linus Torvalds wrote:\n> \n> I wonder if there is some way to only load the crazy curl stuff when we \n> actually want open a http: connection.\n\nWell, we could use dlopen(), but I'm not sure that qualifies as a\n_sane_ solution --- especially given that there are approximately 15\ninterfaces used by git, that we'd have to resolve using dlsym().\n\n\t \t \t \t - Ted\n"},{"id":"118683","messageId":"20090724225917.GA11191@spearce.org","threadId":"20205","inReplyTo":"20090724225415.GC6832@mit.edu","subject":"Re: Performance issue of 'git branch'","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-07-24T22:59:17Z","receivedAt":"2009-07-24T22:59:17Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Theodore Tso wrote:\n> On Fri, Jul 24, 2009 at 02:21:20PM -0700, Linus Torvalds wrote:\n> > \n> > I wonder if there is some way to only load the crazy curl stuff when we \n> > actually want open a http: connection.\n> \n> Well, we could use dlopen(), but I'm not sure that qualifies as a\n> _sane_ solution --- especially given that there are approximately 15\n> interfaces used by git, that we'd have to resolve using dlsym().\n\nYea, that's not sane.\n\nProbably the better approach is to have git fetch and git push be a\ndifferent binary from main git, so we only pay the libcurl loading\noverheads when we hit transport.\n\n-- \nShawn.\n"},{"id":"118686","messageId":"7vr5w564qf.fsf@alter.siamese.dyndns.org","threadId":"20205","inReplyTo":"20090724225917.GA11191@spearce.org","subject":"Re: Performance issue of 'git branch'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-07-24T23:28:08Z","receivedAt":"2009-07-24T23:28:08Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n> Theodore Tso wrote:\n>> On Fri, Jul 24, 2009 at 02:21:20PM -0700, Linus Torvalds wrote:\n>> > \n>> > I wonder if there is some way to only load the crazy curl stuff when we \n>> > actually want open a http: connection.\n>> \n>> Well, we could use dlopen(), but I'm not sure that qualifies as a\n>> _sane_ solution --- especially given that there are approximately 15\n>> interfaces used by git, that we'd have to resolve using dlsym().\n>\n> Yea, that's not sane.\n>\n> Probably the better approach is to have git fetch and git push be a\n> different binary from main git, so we only pay the libcurl loading\n> overheads when we hit transport.\n\nEven though that still will hurt people who do not use http, I think it\nwould be a right approach (in the sense that it should not be too painful\nand with a reasonable gain for local-only operations).\n"},{"id":"118691","messageId":"20090724234648.GA4616@Pilar.aei.mpg.de","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907241349390.3960@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Carlos R. Mafra","fromEmail":"crmafra2@gmail.com","sentAt":"2009-07-24T23:46:49Z","receivedAt":"2009-07-24T23:46:49Z","isPatch":false,"sender":{"key":"crmafra2@gmail.com","avatar":null},"body":"Sorry for the delay and missing the \"strace -ttT\" request,\nbut today was a \"Physics\" day and took me longer to \nnotice your email.\n\nOn Fri 24.Jul'09 at 14:21:20 -0700, Linus Torvalds wrote:\n> \n> What a huge difference!\n> \n> And the NO_CURL version really does load a lot faster in cold-cache. We're \n> not talking small differences:\n\nWith NO_CURL=1 the strace log contained 242 lines (vs 404), but\nthe time difference was not as great as you got. But it was\nbetter:\n\n0.55 +- 0.06 (for 8 runs)\n\nSo I repeated the tests with curl enabled and this time\nI got:\n\n0.77 +- 0.03 (for 6 runs)\n\n(yesterday I got 0.61 +- 0.08, so there is lot of noise)\n\nSo it is better, but not by the same factor as you saw.\nBut I may have an explanation for this.\n\nAfter I clear the cache I wait a few seconds to stabilize,\nand I do the 'time git branch' test when I see that\nthere is no activity in the disk by looking at\nthe 'btrace' output in another xterm. \n\nI noticed that after dropping the cache and before\nI do the test there is lot of activity of something\ncalled 'preload', with lines which look like these:\n\n8,0 0 42881 495.067655112 17777 Q R 51244367 + 552 [preload]\n8,0 0 42882 495.067659931 17777 G R 51244367 + 552 [preload]\n8,0 0 42883 495.067664401 17777 I R 51244367 + 552 [preload]\n\nI hadn't noticed this before and now I checked that,\n\n\"preload is an adaptive readahead daemon that prefetches files mapped by\napplications from the disk to reduce application startup time.\"\n\nSo I guess that my tests here for your NO_CURL=1 idea is inconclusive,\nas I am not sure what preload is prefetching.\n"},{"id":"118697","messageId":"20090725004122.GA28477@Pilar.aei.mpg.de","threadId":"20205","inReplyTo":"20090724234648.GA4616@Pilar.aei.mpg.de","subject":"Re: Performance issue of 'git branch'","fromName":"Carlos R. Mafra","fromEmail":"crmafra2@gmail.com","sentAt":"2009-07-25T00:41:22Z","receivedAt":"2009-07-25T00:41:22Z","isPatch":false,"sender":{"key":"crmafra2@gmail.com","avatar":null},"body":"On Sat 25.Jul'09 at 1:46:48 +0200, Carlos R. Mafra wrote:\n> \n> So I guess that my tests here for your NO_CURL=1 idea is inconclusive,\n> as I am not sure what preload is prefetching.\n\nOk, so I killed /usr/sbin/preload and did the tests again. The \nresults were much more stable, with average 0.40 vs 0.79\n(NO_CURL=1 being faster). The pagefaults were pretty stable too,\n(40major+654minor vs 12major+401minor). \n\nI will use NO_CURL=1 from now on!\n"},{"id":"118699","messageId":"alpine.LFD.2.01.0907241934260.3960@localhost.localdomain","threadId":"20205","inReplyTo":"alpine.DEB.1.10.0907241545340.28013@asgard.lang.hm","subject":"Re: Performance issue of 'git branch'","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-25T02:39:27Z","receivedAt":"2009-07-25T02:39:27Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Fri, 24 Jul 2009, david@lang.hm wrote:\n\n> On Fri, 24 Jul 2009, Linus Torvalds wrote:\n> \n> > On Fri, 24 Jul 2009, david@lang.hm wrote:\n> > > \n> > > what does the performance look like if you just do a static compile\n> > > instead?\n> > \n> > I don't even know - I don't have a static version of curl. I could install\n> > one, of course, but since I don't think that's the solution anyway, I'm\n> > not going to bother.\n> \n> I wasn't thinking a static version of curl, I was thinking a static version of\n> the git binaries. see how fast things could be if no startup linking was\n> nessasary.\n\nWell, that's what I meant. If I add '-static' to the link flags, I get\n\n\t/usr/bin/ld: cannot find -lcurl\n\tcollect2: ld returned 1 exit status\n\nbecause I simply don't have a static library version of curl (and if I do \nNO_CURL, I fail the link due to not having a static version of zlib).\n\nThat's what I meant by \"I could install a static version of curl\" - I \ncould install the debug libraries, but it just isn't a normal thing to do \non any modern distribution. The right thing to do really would be to not \nhave -lcurl for the main git binary at all.\n\nPreferably done by having http walking handled by an external process (the \nway we already do rsync), but it's probably easier to just make all the \nclone/fetch/ls-remote things be a separate binary.\n\nOf course, I'd personally solve the problem with NO_CURL=1, but that's \nprobably not acceptable in general.\n\n\t\t\tLinus\n"},{"id":"118700","messageId":"alpine.LNX.2.00.0907242242310.2147@iabervon.org","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907241934260.3960@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2009-07-25T02:53:11Z","receivedAt":"2009-07-25T02:53:11Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Fri, 24 Jul 2009, Linus Torvalds wrote:\n\n> On Fri, 24 Jul 2009, david@lang.hm wrote:\n> \n> > On Fri, 24 Jul 2009, Linus Torvalds wrote:\n> > \n> > > On Fri, 24 Jul 2009, david@lang.hm wrote:\n> > > > \n> > > > what does the performance look like if you just do a static compile\n> > > > instead?\n> > > \n> > > I don't even know - I don't have a static version of curl. I could install\n> > > one, of course, but since I don't think that's the solution anyway, I'm\n> > > not going to bother.\n> > \n> > I wasn't thinking a static version of curl, I was thinking a static version of\n> > the git binaries. see how fast things could be if no startup linking was\n> > nessasary.\n> \n> Well, that's what I meant. If I add '-static' to the link flags, I get\n> \n> \t/usr/bin/ld: cannot find -lcurl\n> \tcollect2: ld returned 1 exit status\n> \n> because I simply don't have a static library version of curl (and if I do \n> NO_CURL, I fail the link due to not having a static version of zlib).\n> \n> That's what I meant by \"I could install a static version of curl\" - I \n> could install the debug libraries, but it just isn't a normal thing to do \n> on any modern distribution. The right thing to do really would be to not \n> have -lcurl for the main git binary at all.\n> \n> Preferably done by having http walking handled by an external process (the \n> way we already do rsync), but it's probably easier to just make all the \n> clone/fetch/ls-remote things be a separate binary.\n\nI think it's actually easy enough to have a separate binary to handle the \nhttp walking, particularly since I've got code lying around to handle \nimporting from a foreign VCS with a separate binary that I can just remove \nsome of the features from.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"118741","messageId":"alpine.LFD.2.01.0907251046140.3960@localhost.localdomain","threadId":"20205","inReplyTo":"20090725004122.GA28477@Pilar.aei.mpg.de","subject":"Re: Performance issue of 'git branch'","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-25T18:04:29Z","receivedAt":"2009-07-25T18:04:29Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sat, 25 Jul 2009, Carlos R. Mafra wrote:\n> \n> Ok, so I killed /usr/sbin/preload and did the tests again. The \n> results were much more stable, with average 0.40 vs 0.79\n> (NO_CURL=1 being faster). The pagefaults were pretty stable too,\n> (40major+654minor vs 12major+401minor). \n> \n> I will use NO_CURL=1 from now on!\n\nI actually find it interesting that this whole NO_CURL issue is actually a \nlot more noticeable for me in the hot-cache case than all the other 'git \nbranch' issues were.\n\nI went back to a version a few days ago (before all the optimizations), \nand on my machine with a hot cache I get (for my kernel repo - I don't \nuse branches there, but I have an old 'akpm' branch for taking a emailed \npatch series from Andrew):\n\n\t[torvalds@nehalem linux]$ time ~/git/git branch\n\t akpm\n\t* master\n\n\treal\t0m0.005s\n\tuser\t0m0.004s\n\tsys\t0m0.000s\n\nso it's five milliseconds. Big deal, fast enough, right?\n\nOk, so fast-forward to today, with the optimizations to builtin-branch.c:\n\n\t[torvalds@nehalem linux]$ time ~/git/git branch\n\t akpm\n\t* master\n\n\treal\t0m0.004s\n\tuser\t0m0.000s\n\tsys\t0m0.004s\n\nWoot! I shaved a millisecond off it by avoiding all those page faults and \nobject lookups. Good, but hey, all that unnecessary lookup was just a 25% \ncost.\n\nSo let's build it with NO_CURL:\n\n\t[torvalds@nehalem linux]$ time ~/git/git branch\n\t akpm\n\t* master\n\n\treal\t0m0.002s\n\tuser\t0m0.000s\n\tsys\t0m0.000s\n\nHeh. The whole NO_CURL=1 thing is actually a _bigger_ optimization than \nanything else I did to git-branch. Cost of curl: 100%.\n\nThe difference in number of system calls and page faults is really quite \nstaggering. System calls: 397->184, page faults: 619->293. Just from not \ndoing that curl loading. No wonder performance actually doubles.\n\nNow, I admit that 5ms vs 2ms probably doesn't really matter much, but \ndang, performance was a primary goal in git, so I'm a bit upset at how bad \ncurl screwed us. Plus those things do add up when scripting things, and \nthose 300+ page faults are basically true for _all_ git programs.\n\nSo it's not just 'git branch': doing 'git show' shows the exact same \nthing: 6ms -> 4ms, 448->235 system calls, and 1549->1176 page faults.\n\nSo curl really must die. It may not matter for the expensive operations, \nbut a lot of scripting is about running all those \"cheap\" things that just \nadd up over time.\n\n\t\t\tLinus\n"},{"id":"118765","messageId":"20090725215739.d074e947.tihirvon@gmail.com","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907251046140.3960@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Timo Hirvonen","fromEmail":"tihirvon@gmail.com","sentAt":"2009-07-25T18:57:39Z","receivedAt":"2009-07-25T18:57:39Z","isPatch":false,"sender":{"key":"tihirvon@gmail.com","avatar":null},"body":"Linus Torvalds wrote:\n\n> So curl really must die. It may not matter for the expensive operations, \n> but a lot of scripting is about running all those \"cheap\" things that just \n> add up over time.\n\nSELinux is the problem, not curl.\n\nOn my Arch Linux machine:\n\n $ ldd bin/git\n\tlinux-vdso.so.1 => (0x00007fff42306000)\n\tlibcurl.so.4 => /usr/lib/libcurl.so.4 (0x00007f8714532000)\n\tlibz.so.1 => /usr/lib/libz.so.1 (0x00007f871431d000)\n\tlibcrypto.so.0.9.8 => /usr/lib/libcrypto.so.0.9.8 (0x00007f8713f8f000)\n\tlibpthread.so.0 => /lib/libpthread.so.0 (0x00007f8713d74000)\n\tlibc.so.6 => /lib/libc.so.6 (0x00007f8713a21000)\n\tlibrt.so.1 => /lib/librt.so.1 (0x00007f8713819000)\n\tlibssl.so.0.9.8 => /usr/lib/libssl.so.0.9.8 (0x00007f87135ca000)\n\tlibdl.so.2 => /lib/libdl.so.2 (0x00007f87133c6000)\n\t/lib/ld-linux-x86-64.so.2 (0x00007f8714778000)\n\nYour:\n\n [torvalds@nehalem git]$ ldd git\n\tlinux-vdso.so.1 => (0x00007fff61da7000)\n\tlibcurl.so.4 => /usr/lib64/libcurl.so.4 (0x00007f2f1a498000)\n\tlibz.so.1 => /lib64/libz.so.1 (0x0000003cdb800000)\n\tlibcrypto.so.8 => /usr/lib64/libcrypto.so.8 (0x0000003ba7a00000)\n\tlibpthread.so.0 => /lib64/libpthread.so.0 (0x0000003cdb400000)\n\tlibc.so.6 => /lib64/libc.so.6 (0x0000003cda800000)\n\tlibidn.so.11 => /lib64/libidn.so.11 (0x0000003ceaa00000)\n\tlibssh2.so.1 => /usr/lib64/libssh2.so.1 (0x0000003ba8e00000)\n\tlibldap-2.4.so.2 => /usr/lib64/libldap-2.4.so.2 (0x00007f2f1a250000)\n\tlibrt.so.1 => /lib64/librt.so.1 (0x0000003cdbc00000)\n\tlibgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x0000003ce6e00000)\n\tlibkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x0000003ce7e00000)\n\tlibk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x0000003ce7200000)\n\tlibcom_err.so.2 => /lib64/libcom_err.so.2 (0x0000003ce6a00000)\n\tlibssl3.so => /lib64/libssl3.so (0x0000003490200000)\n\tlibsmime3.so => /lib64/libsmime3.so (0x000000348fe00000)\n\tlibnss3.so => /lib64/libnss3.so (0x000000348f600000)\n\tlibplds4.so => /lib64/libplds4.so (0x0000003cbc800000)\n\tlibplc4.so => /lib64/libplc4.so (0x0000003cbdc00000)\n\tlibnspr4.so => /lib64/libnspr4.so (0x0000003cbd800000)\n\tlibdl.so.2 => /lib64/libdl.so.2 (0x0000003cdb000000)\n\t/lib64/ld-linux-x86-64.so.2 (0x0000003cda400000)\n\tlibssl.so.8 => /usr/lib64/libssl.so.8 (0x0000003ba7e00000)\n\tliblber-2.4.so.2 => /usr/lib64/liblber-2.4.so.2 (0x0000003ceee00000)\n\tlibresolv.so.2 => /lib64/libresolv.so.2 (0x0000003ce5600000)\n\tlibsasl2.so.2 => /usr/lib64/libsasl2.so.2 (0x00007f2f1a030000)\n\tlibkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x0000003ce7a00000)\n\tlibkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x0000003ce7600000)\n\tlibnssutil3.so => /lib64/libnssutil3.so (0x000000348fa00000)\n\tlibcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f2f19df8000)\n\tlibselinux.so.1 => /lib64/libselinux.so.1 (0x0000003cdc400000)\n\tlibfreebl3.so => /lib64/libfreebl3.so (0x00007f2f19b99000)\n"},{"id":"118766","messageId":"3f4fd2640907251206n2c7f29a1r14204302d4b27cd4@mail.gmail.com","threadId":"20205","inReplyTo":"20090725215739.d074e947.tihirvon@gmail.com","subject":"Re: Performance issue of 'git branch'","fromName":"Reece Dunn","fromEmail":"msclrhd@googlemail.com","sentAt":"2009-07-25T19:06:41Z","receivedAt":"2009-07-25T19:06:41Z","isPatch":false,"sender":{"key":"msclrhd@googlemail.com","avatar":null},"body":"2009/7/25 Timo Hirvonen :\n> Linus Torvalds wrote:\n>\n>> So curl really must die. It may not matter for the expensive operations,\n>> but a lot of scripting is about running all those \"cheap\" things that just\n>> add up over time.\n>\n> SELinux is the problem, not curl.\n>\n> On my Arch Linux machine:\n>\n>   $ ldd bin/git\n>        linux-vdso.so.1 =>  (0x00007fff42306000)\n>        libcurl.so.4 => /usr/lib/libcurl.so.4 (0x00007f8714532000)\n>        libz.so.1 => /usr/lib/libz.so.1 (0x00007f871431d000)\n>        libcrypto.so.0.9.8 => /usr/lib/libcrypto.so.0.9.8 (0x00007f8713f8f000)\n>        libpthread.so.0 => /lib/libpthread.so.0 (0x00007f8713d74000)\n>        libc.so.6 => /lib/libc.so.6 (0x00007f8713a21000)\n>        librt.so.1 => /lib/librt.so.1 (0x00007f8713819000)\n>        libssl.so.0.9.8 => /usr/lib/libssl.so.0.9.8 (0x00007f87135ca000)\n>        libdl.so.2 => /lib/libdl.so.2 (0x00007f87133c6000)\n>        /lib/ld-linux-x86-64.so.2 (0x00007f8714778000)\n\nIt will depend on the dependencies of curl that are applied. BLFS\n(http://www.linuxfromscratch.org/blfs/view/stable/basicnet/curl.html)\nlist the following dependencies:\n\n pkg-config-0.22\n OpenSSL-0.9.8g or GnuTLS-1.6.3\n OpenLDAP-2.3.39\n libidn-0.6.14\n MIT Kerberos V5-1.6 or Heimdal-1.1\n krb4\n SPNEGO\n c-ares\n\nand the dependencies of those packages and so forth.\n\nOn Ubuntu 9.04, I get:\n\n$ ldd /usr/bin/git\n\tlinux-gate.so.1 => (0xb80ae000)\n\tlibcurl-gnutls.so.4 => /usr/lib/libcurl-gnutls.so.4 (0xb805b000)\n\tlibz.so.1 => /lib/libz.so.1 (0xb8045000)\n\tlibpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb802b000)\n\tlibc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7ec8000)\n\tlibidn.so.11 => /usr/lib/libidn.so.11 (0xb7e95000)\n\tliblber-2.4.so.2 => /usr/lib/liblber-2.4.so.2 (0xb7e87000)\n\tlibldap_r-2.4.so.2 => /usr/lib/libldap_r-2.4.so.2 (0xb7e43000)\n\tlibrt.so.1 => /lib/tls/i686/cmov/librt.so.1 (0xb7e39000)\n\tlibgssapi_krb5.so.2 => /usr/lib/libgssapi_krb5.so.2 (0xb7e0e000)\n\tlibgnutls.so.26 => /usr/lib/libgnutls.so.26 (0xb7d71000)\n\tlibtasn1.so.3 => /usr/lib/libtasn1.so.3 (0xb7d5f000)\n\tlibgcrypt.so.11 => /lib/libgcrypt.so.11 (0xb7cf6000)\n\t/lib/ld-linux.so.2 (0xb80af000)\n\tlibresolv.so.2 => /lib/tls/i686/cmov/libresolv.so.2 (0xb7ce0000)\n\tlibsasl2.so.2 => /usr/lib/libsasl2.so.2 (0xb7cc7000)\n\tlibdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7cc3000)\n\tlibkrb5.so.3 => /usr/lib/libkrb5.so.3 (0xb7c31000)\n\tlibk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0xb7c0d000)\n\tlibcom_err.so.2 => /lib/libcom_err.so.2 (0xb7c09000)\n\tlibkrb5support.so.0 => /usr/lib/libkrb5support.so.0 (0xb7bff000)\n\tlibkeyutils.so.1 => /lib/libkeyutils.so.1 (0xb7bfb000)\n\tlibgpg-error.so.0 => /lib/libgpg-error.so.0 (0xb7bf7000)\n\n- Reece\n"},{"id":"118776","messageId":"20090725203130.GB1640@glandium.org","threadId":"20205","inReplyTo":"20090725215739.d074e947.tihirvon@gmail.com","subject":"Re: Performance issue of 'git branch'","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2009-07-25T20:31:30Z","receivedAt":"2009-07-25T20:31:30Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Sat, Jul 25, 2009 at 09:57:39PM +0300, Timo Hirvonen wrote:\n> Linus Torvalds wrote:\n> \n> > So curl really must die. It may not matter for the expensive operations, \n> > but a lot of scripting is about running all those \"cheap\" things that just \n> > add up over time.\n> \n> SELinux is the problem, not curl.\n\nI think it's NSS, the problem, not SELinux. Linus's libcurl is built\nagainst NSS, which is the default on Fedora.\n\nMike\n"},{"id":"118781","messageId":"alpine.LFD.2.01.0907251353410.3960@localhost.localdomain","threadId":"20205","inReplyTo":"20090725203130.GB1640@glandium.org","subject":"Re: Performance issue of 'git branch'","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-25T21:02:19Z","receivedAt":"2009-07-25T21:02:19Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sat, 25 Jul 2009, Mike Hommey wrote:\n\n> On Sat, Jul 25, 2009 at 09:57:39PM +0300, Timo Hirvonen wrote:\n> > Linus Torvalds wrote:\n> > \n> > > So curl really must die. It may not matter for the expensive operations, \n> > > but a lot of scripting is about running all those \"cheap\" things that just \n> > > add up over time.\n> > \n> > SELinux is the problem, not curl.\n> \n> I think it's NSS, the problem, not SELinux. Linus's libcurl is built\n> against NSS, which is the default on Fedora.\n\nWell, it kind of doesn't matter. The fact is, libcurl is a bloated \nmonster, and adds zero to 99% of what git people do.\n\nThe fact that apparently sometimes it's less bloated than other times \ndoesn't really change anything fundamental, does it?\n\n\t\t\tLinus\n"},{"id":"118782","messageId":"20090725210407.GA3556@Pilar.aei.mpg.de","threadId":"20205","inReplyTo":"20090725215739.d074e947.tihirvon@gmail.com","subject":"Re: Performance issue of 'git branch'","fromName":"Carlos R. Mafra","fromEmail":"crmafra2@gmail.com","sentAt":"2009-07-25T21:04:07Z","receivedAt":"2009-07-25T21:04:07Z","isPatch":false,"sender":{"key":"crmafra2@gmail.com","avatar":null},"body":"On Sat 25.Jul'09 at 21:57:39 +0300, Timo Hirvonen wrote:\n> Linus Torvalds wrote:\n> \n> > So curl really must die. It may not matter for the expensive operations, \n> > but a lot of scripting is about running all those \"cheap\" things that just \n> > add up over time.\n> \n> SELinux is the problem, not curl.\n\nI don't have SELinux, and without curl it takes ~50% less time (on\ntop of Linus' previous optimizations!).\n\nThe time to open() all the libs really sums up to a considerable \nfraction (when the total time is low, not when compared to the \nhuge 6 secs of before)\n\nWithout curl:\n[mafra@Pilar:linux-2.6]$ grep open strace-nocurl.log |grep lib \\\n> | awk -F \"<\" '{print $2}' | sed s/\\>// | awk '{s += $1} END {print s}'\n0.070104\n\nWith curl:\n[mafra@Pilar:linux-2.6]$ grep open strace-curl.log |grep lib \\\n> | awk -F \"<\" '{print $2}' | sed s/\\>// | awk '{s += $1} END {print s}'\n0.249764\n\nPS: It is interesting that in my laptop the time required\nto open libcurl alone is 20x the total time of 'git branch' for Linus'\nin his supercomputer:\nopen(\"/usr/lib64/libcurl.so.4\", O_RDONLY) = 3 <0.066239>\n"},{"id":"118784","messageId":"alpine.LFD.2.01.0907251406390.3960@localhost.localdomain","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907251353410.3960@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-25T21:13:13Z","receivedAt":"2009-07-25T21:13:13Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sat, 25 Jul 2009, Linus Torvalds wrote:\n>\n> The fact that apparently sometimes it's less bloated than other times \n> doesn't really change anything fundamental, does it?\n\nBtw, does anybody know how/why libdl seems to get linked in too?\n\nWe're not doing -ldl, and I'm not seeing any need for it, but it's \ndefinitely there on fedora, at least.\n\nIt seems to come from libcrypto. I can get rid of it with NO_OPENSSL, and \nthat cuts down on the number of system calls in my startup by 16 (getting \nrid of both libcrypto and libdl). I wonder if there is some way to get the \noptimized openssl sha1 routines _without_ that silly ldl thing.\n\n\t\tLinus\n"},{"id":"118805","messageId":"alpine.DEB.1.00.0907260123090.8306@pacific.mpi-cbg.de","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907251406390.3960@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-07-25T23:23:58Z","receivedAt":"2009-07-25T23:23:58Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sat, 25 Jul 2009, Linus Torvalds wrote:\n\n> On Sat, 25 Jul 2009, Linus Torvalds wrote:\n> >\n> > The fact that apparently sometimes it's less bloated than other times \n> > doesn't really change anything fundamental, does it?\n> \n> Btw, does anybody know how/why libdl seems to get linked in too?\n> \n> We're not doing -ldl, and I'm not seeing any need for it, but it's \n> definitely there on fedora, at least.\n> \n> It seems to come from libcrypto. I can get rid of it with NO_OPENSSL, and \n> that cuts down on the number of system calls in my startup by 16 (getting \n> rid of both libcrypto and libdl). I wonder if there is some way to get the \n> optimized openssl sha1 routines _without_ that silly ldl thing.\n\nOpenSSL allows for so-called engines implementing certain algorithms. \nThese engines are dynamic libraries, loaded via dlopen().\n\nCiao,\nDscho\n"},{"id":"118812","messageId":"alpine.LFD.2.01.0907252147130.3960@localhost.localdomain","threadId":"20205","inReplyTo":"alpine.DEB.1.00.0907260123090.8306@pacific.mpi-cbg.de","subject":"Re: Performance issue of 'git branch'","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-26T04:49:41Z","receivedAt":"2009-07-26T04:49:41Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sun, 26 Jul 2009, Johannes Schindelin wrote:\n> > \n> > It seems to come from libcrypto. I can get rid of it with NO_OPENSSL, and \n> > that cuts down on the number of system calls in my startup by 16 (getting \n> > rid of both libcrypto and libdl). I wonder if there is some way to get the \n> > optimized openssl sha1 routines _without_ that silly ldl thing.\n> \n> OpenSSL allows for so-called engines implementing certain algorithms. \n> These engines are dynamic libraries, loaded via dlopen().\n\nAh. Ok, that explains it.\n\nIt's a bit sad, since the _only_ thing we load all of libcrypto for is the \n(fairly trivial) SHA1 code. \n\nBut at the same time, last time I benchmarked the different SHA1 \nlibraries, the openssl one was the fastest. I think it has tuned assembly \nlanguage for most architectures. Our regular mozilla-based C code is \nperfectly fine, but it doesn't hold a candle to assembler tuning.\n\nOh well. \n\n\t\tLinus\n"},{"id":"118816","messageId":"20090726075455.GA22525@glandium.org","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907251353410.3960@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2009-07-26T07:54:55Z","receivedAt":"2009-07-26T07:54:55Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Sat, Jul 25, 2009 at 02:02:19PM -0700, Linus Torvalds wrote:\n> \n> \n> On Sat, 25 Jul 2009, Mike Hommey wrote:\n> \n> > On Sat, Jul 25, 2009 at 09:57:39PM +0300, Timo Hirvonen wrote:\n> > > Linus Torvalds wrote:\n> > > \n> > > > So curl really must die. It may not matter for the expensive operations, \n> > > > but a lot of scripting is about running all those \"cheap\" things that just \n> > > > add up over time.\n> > > \n> > > SELinux is the problem, not curl.\n> > \n> > I think it's NSS, the problem, not SELinux. Linus's libcurl is built\n> > against NSS, which is the default on Fedora.\n> \n> Well, it kind of doesn't matter. The fact is, libcurl is a bloated \n> monster, and adds zero to 99% of what git people do.\n\nEspecially consideting the http transport fails to be useful in various\nscenarios.\n\nMike\n"},{"id":"118827","messageId":"alpine.DEB.1.00.0907261214220.8306@pacific.mpi-cbg.de","threadId":"20205","inReplyTo":"20090726075455.GA22525@glandium.org","subject":"Re: Performance issue of 'git branch'","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-07-26T10:16:37Z","receivedAt":"2009-07-26T10:16:37Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sun, 26 Jul 2009, Mike Hommey wrote:\n\n> On Sat, Jul 25, 2009 at 02:02:19PM -0700, Linus Torvalds wrote:\n> > \n> > \n> > On Sat, 25 Jul 2009, Mike Hommey wrote:\n> > \n> > > On Sat, Jul 25, 2009 at 09:57:39PM +0300, Timo Hirvonen wrote:\n> > > > Linus Torvalds wrote:\n> > > > \n> > > > > So curl really must die. It may not matter for the expensive operations, \n> > > > > but a lot of scripting is about running all those \"cheap\" things that just \n> > > > > add up over time.\n> > > > \n> > > > SELinux is the problem, not curl.\n> > > \n> > > I think it's NSS, the problem, not SELinux. Linus's libcurl is built\n> > > against NSS, which is the default on Fedora.\n> > \n> > Well, it kind of doesn't matter. The fact is, libcurl is a bloated \n> > monster, and adds zero to 99% of what git people do.\n> \n> Especially consideting the http transport fails to be useful in various\n> scenarios.\n\nI beg your pardon? Maybe \"s/useful/desirable/\"?\n\nIn many scenarios, http transport is the _last resort_ against overzealous \nadministrators. The fact that you might be lucky enough not to need that \nresort is a blessing, and does not give you the right to ridicule those \nwho are unfortunate enough not to share your good luck.\n\nCiao,\nDscho\n"},{"id":"118828","messageId":"9b18b3110907260323h575a0b06rcf058c8b3370b742@mail.gmail.com","threadId":"20205","inReplyTo":"alpine.DEB.1.00.0907261214220.8306@pacific.mpi-cbg.de","subject":"Re: Performance issue of 'git branch'","fromName":"demerphq","fromEmail":"demerphq@gmail.com","sentAt":"2009-07-26T10:23:33Z","receivedAt":"2009-07-26T10:23:33Z","isPatch":false,"sender":{"key":"demerphq@gmail.com","avatar":null},"body":"2009/7/26 Johannes Schindelin :\n> Hi,\n>\n> On Sun, 26 Jul 2009, Mike Hommey wrote:\n>\n>> On Sat, Jul 25, 2009 at 02:02:19PM -0700, Linus Torvalds wrote:\n>> >\n>> >\n>> > On Sat, 25 Jul 2009, Mike Hommey wrote:\n>> >\n>> > > On Sat, Jul 25, 2009 at 09:57:39PM +0300, Timo Hirvonen wrote:\n>> > > > Linus Torvalds wrote:\n>> > > >\n>> > > > > So curl really must die. It may not matter for the expensive operations,\n>> > > > > but a lot of scripting is about running all those \"cheap\" things that just\n>> > > > > add up over time.\n>> > > >\n>> > > > SELinux is the problem, not curl.\n>> > >\n>> > > I think it's NSS, the problem, not SELinux. Linus's libcurl is built\n>> > > against NSS, which is the default on Fedora.\n>> >\n>> > Well, it kind of doesn't matter. The fact is, libcurl is a bloated\n>> > monster, and adds zero to 99% of what git people do.\n>>\n>> Especially consideting the http transport fails to be useful in various\n>> scenarios.\n>\n> I beg your pardon?  Maybe \"s/useful/desirable/\"?\n>\n> In many scenarios, http transport is the _last resort_ against overzealous\n> administrators.  The fact that you might be lucky enough not to need that\n> resort is a blessing, and does not give you the right to ridicule those\n> who are unfortunate enough not to share your good luck.\n\nI think he meant that it is buggy and does not work correctly in\nvarious scenarios.\n\nEg: Last I checked it couldn't handle repos where the main branch\nwasn''t called master, and I've seen other messages that make me think\nit doesn't work correctly on edge cases.\n\ncheers,\nYves\n\n\n\n-- \nperl -Mre=debug -e \"/just|another|perl|hacker/\"\n"},{"id":"118830","messageId":"9b18b3110907260327q5a6d370g670f79792dfa93e0@mail.gmail.com","threadId":"20205","inReplyTo":"9b18b3110907260323h575a0b06rcf058c8b3370b742@mail.gmail.com","subject":"Re: Performance issue of 'git branch'","fromName":"demerphq","fromEmail":"demerphq@gmail.com","sentAt":"2009-07-26T10:27:35Z","receivedAt":"2009-07-26T10:27:35Z","isPatch":false,"sender":{"key":"demerphq@gmail.com","avatar":null},"body":"2009/7/26 demerphq :\n> 2009/7/26 Johannes Schindelin :\n>> Hi,\n>>\n>> On Sun, 26 Jul 2009, Mike Hommey wrote:\n>>\n>>> On Sat, Jul 25, 2009 at 02:02:19PM -0700, Linus Torvalds wrote:\n>>> >\n>>> >\n>>> > On Sat, 25 Jul 2009, Mike Hommey wrote:\n>>> >\n>>> > > On Sat, Jul 25, 2009 at 09:57:39PM +0300, Timo Hirvonen wrote:\n>>> > > > Linus Torvalds wrote:\n>>> > > >\n>>> > > > > So curl really must die. It may not matter for the expensive operations,\n>>> > > > > but a lot of scripting is about running all those \"cheap\" things that just\n>>> > > > > add up over time.\n>>> > > >\n>>> > > > SELinux is the problem, not curl.\n>>> > >\n>>> > > I think it's NSS, the problem, not SELinux. Linus's libcurl is built\n>>> > > against NSS, which is the default on Fedora.\n>>> >\n>>> > Well, it kind of doesn't matter. The fact is, libcurl is a bloated\n>>> > monster, and adds zero to 99% of what git people do.\n>>>\n>>> Especially consideting the http transport fails to be useful in various\n>>> scenarios.\n>>\n>> I beg your pardon?  Maybe \"s/useful/desirable/\"?\n>>\n>> In many scenarios, http transport is the _last resort_ against overzealous\n>> administrators.  The fact that you might be lucky enough not to need that\n>> resort is a blessing, and does not give you the right to ridicule those\n>> who are unfortunate enough not to share your good luck.\n>\n> I think he meant that it is buggy and does not work correctly in\n> various scenarios.\n>\n> Eg: Last I checked it couldn't handle repos where the main branch\n> wasn''t called master, and I've seen other messages that make me think\n> it doesn't work correctly on edge cases.\n\nEr, I meant that to go to Johannes directly, not to spam the list or\nthe cc's with my hazy recollection, and I should have added: \"but\nperhaps im confusing http and rsync\".\n\nSorry for the noise.\n\nYves\n\n\n\n-- \nperl -Mre=debug -e \"/just|another|perl|hacker/\"\n"},{"id":"118837","messageId":"20090726162927.GC17272@mit.edu","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907252147130.3960@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Theodore Tso","fromEmail":"tytso@mit.edu","sentAt":"2009-07-26T16:29:28Z","receivedAt":"2009-07-26T16:29:28Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Sat, Jul 25, 2009 at 09:49:41PM -0700, Linus Torvalds wrote:\n> \n> But at the same time, last time I benchmarked the different SHA1 \n> libraries, the openssl one was the fastest. I think it has tuned assembly \n> language for most architectures. Our regular mozilla-based C code is \n> perfectly fine, but it doesn't hold a candle to assembler tuning.\n\nSo maybe git should import the SHA1 code into its own source base?\nIt's not like the SHA1 code changes often, or is likely to have\nsecurity issues (at least, not buffer overruns; if SHA1 gets thorouhly\nbroken we might have to change algorithms, but that's a different\nkettle of fish :-).\n\n\t\t\t\t\t- Ted\n"},{"id":"118840","messageId":"4A6C8D3B.101@redhat.com","threadId":"20205","inReplyTo":"20090724225917.GA11191@spearce.org","subject":"Re: Performance issue of 'git branch'","fromName":"Avi Kivity","fromEmail":"avi@redhat.com","sentAt":"2009-07-26T17:07:07Z","receivedAt":"2009-07-26T17:07:07Z","isPatch":false,"sender":{"key":"avi@redhat.com","avatar":null},"body":"On 07/25/2009 01:59 AM, Shawn O. Pearce wrote:\n> Theodore Tso wrote:\n> \n>> On Fri, Jul 24, 2009 at 02:21:20PM -0700, Linus Torvalds wrote:\n>> \n>>> I wonder if there is some way to only load the crazy curl stuff when we\n>>> actually want open a http: connection.\n>>> \n>> Well, we could use dlopen(), but I'm not sure that qualifies as a\n>> _sane_ solution --- especially given that there are approximately 15\n>> interfaces used by git, that we'd have to resolve using dlsym().\n>> \n>\n> Yea, that's not sane.\n>\n> Probably the better approach is to have git fetch and git push be a\n> different binary from main git, so we only pay the libcurl loading\n> overheads when we hit transport.\n> \n\nOr make the transports shared libraries, and use dlopen() to open the \ntransport and dlsym() to resolve the struct transport object exported by \nthe library.\n\n-- \nerror compiling committee.c: too many arguments to function\n"},{"id":"118841","messageId":"alpine.DEB.1.00.0907261915580.8306@pacific.mpi-cbg.de","threadId":"20205","inReplyTo":"4A6C8D3B.101@redhat.com","subject":"Re: Performance issue of 'git branch'","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-07-26T17:16:38Z","receivedAt":"2009-07-26T17:16:38Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sun, 26 Jul 2009, Avi Kivity wrote:\n\n> On 07/25/2009 01:59 AM, Shawn O. Pearce wrote:\n> > Theodore Tso wrote:\n> > \n> > > On Fri, Jul 24, 2009 at 02:21:20PM -0700, Linus Torvalds wrote:\n> > > \n> > > > I wonder if there is some way to only load the crazy curl stuff when we\n> > > > actually want open a http: connection.\n> > > > \n> > > Well, we could use dlopen(), but I'm not sure that qualifies as a\n> > > _sane_ solution --- especially given that there are approximately 15\n> > > interfaces used by git, that we'd have to resolve using dlsym().\n> > > \n> >\n> > Yea, that's not sane.\n> >\n> > Probably the better approach is to have git fetch and git push be a\n> > different binary from main git, so we only pay the libcurl loading\n> > overheads when we hit transport.\n> > \n> \n> Or make the transports shared libraries, and use dlopen() to open the\n> transport and dlsym() to resolve the struct transport object exported by the\n> library.\n\n... and introduce all kinds of braindamage to the Makefile so we can \nproperly compile .dll files on Windows?\n\nUmm, thanks, but no.\n\nCiao,\nDscho\n"},{"id":"119881","messageId":"20090807042132.GA14751@sigill.intra.peff.net","threadId":"20205","inReplyTo":"alpine.LFD.2.01.0907241505400.3960@localhost.localdomain","subject":"Re: Performance issue of 'git branch'","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-08-07T04:21:33Z","receivedAt":"2009-08-07T04:21:33Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jul 24, 2009 at 03:13:07PM -0700, Linus Torvalds wrote:\n\n> Subject: [PATCH] git-http-fetch: not a builtin\n> \n> We should really try to avoid having a dependency on the curl libraries\n> for the core 'git' executable. It adds huge overheads, for no advantage.\n> \n> This splits up git-http-fetch so that it isn't built-in. We still do\n> end up linking with curl for the git binary due to the transport.c http\n> walker, but that's at least partially an independent issue.\n>\n> [...]\n>\n> +git-http-fetch$X: revision.o http.o http-push.o $(GITLIBS)\n> +\t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \\\n> +\t\t$(LIBS) $(CURL_LIBCURL) $(EXPAT_LIBEXPAT)\n\nErr, this seems to horribly break git-http-fetch (see if you can spot\nthe logic error in dependencies). Patch is below.\n\nNobody noticed, I expect, because nothing in git _uses_ http-fetch\nanymore, now that git-clone is no longer a shell script. I only noticed\nbecause it tried to build http-push on one of my NO_EXPAT machines.\n\nIt might be an interesting exercise to dust off the old shell scripts\nonce in a while and see if they still pass their original tests while\nrunning on top of a more modern git. It would test that we haven't\nbroken the plumbing interfaces.\n\n-- >8 --\nSubject: [PATCH] Makefile: build http-fetch against http-fetch.o\n\nAs opposed to http-push.o. We can also drop EXPAT_LIBEXPAT,\nsince fetch does not need it.\n\nThis appears to be a bad cut-and-paste in commit 1088261f.\n\nSigned-off-by: Jeff King \n---\n Makefile | 4 ++--\n 1 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 97d904b..d6362d3 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1502,9 +1502,9 @@ http.o http-walker.o http-push.o: http.h\n \n http.o http-walker.o: $(LIB_H)\n \n-git-http-fetch$X: revision.o http.o http-push.o $(GITLIBS)\n+git-http-fetch$X: revision.o http.o http-fetch.o http-walker.o $(GITLIBS)\n \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \\\n-\t\t$(LIBS) $(CURL_LIBCURL) $(EXPAT_LIBEXPAT)\n+\t\t$(LIBS) $(CURL_LIBCURL)\n git-http-push$X: revision.o http.o http-push.o $(GITLIBS)\n \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \\\n \t\t$(LIBS) $(CURL_LIBCURL) $(EXPAT_LIBEXPAT)\n-- \n1.6.4.117.g6056d.dirty\n"}]}