{"thread":{"id":"16178","subject":"Slow \"git rev-list origin/master --not --all\" or \"git fetch\" slow when downloading nothing","startedAt":"2008-11-05T09:38:31Z","lastAt":"2008-11-05T22:08:06Z","messageCount":5,"participants":["Santi Béjar","Junio C Hamano","Linus Torvalds"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"94965","messageId":"adf1fd3d0811050138j7b8bbed1nd94a999f55e38d61@mail.gmail.com","threadId":"16178","inReplyTo":null,"subject":"Slow \"git rev-list origin/master --not --all\" or \"git fetch\" slow when downloading nothing","fromName":"Santi Béjar","fromEmail":"santi@agolina.net","sentAt":"2008-11-05T09:38:31Z","receivedAt":"2008-11-05T09:38:31Z","isPatch":false,"sender":{"key":"santi@agolina.net","avatar":null},"body":"Hi *,\n\n In cold cache \"git rev-list origin/master --not --all\" is slow\nreading many files:\n\ncold cache:\n$ /usr/bin/time git rev-list origin/master --not --all\n0.03user 0.02system 0:04.57elapsed 1%CPU (0avgtext+0avgdata 0maxresident)k\n77848inputs+0outputs (410major+1798minor)pagefaults 0swaps\n\nhot cache:\n$ /usr/bin/time git rev-list origin/master --not --all\n0.01user 0.00system 0:00.06elapsed 31%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+2207minor)pagefaults 0swaps\n\nI think that, in this particular case (when the arguments are the tips\nof some of the branches), this should not read that many files.\n\nMoreover, this is used in \"git fetch\" (git rev-list --quiet --objects\n --not --all) to detect if all the objects are\nreachable from the local repository. When nothing has changed in the\nremote repository (so refs//* has all the remote refs) the\n\"git fetch\" could be almost instantaneous (even in coldcache), but\ncurrently it is not because of the above.\n\nThanks,\nSanti\n"},{"id":"94976","messageId":"7vprlakri4.fsf@gitster.siamese.dyndns.org","threadId":"16178","inReplyTo":"adf1fd3d0811050138j7b8bbed1nd94a999f55e38d61@mail.gmail.com","subject":"Re* Slow \"git rev-list origin/master --not --all\" or \"git fetch\" slow when downloading nothing","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-11-05T17:32:51Z","receivedAt":"2008-11-05T17:32:51Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Santi Béjar\" writes:\n\n> In cold cache \"git rev-list origin/master --not --all\" is slow\n> reading many files:\n>\n> cold cache:\n> $ /usr/bin/time git rev-list origin/master --not --all\n> 0.03user 0.02system 0:04.57elapsed 1%CPU (0avgtext+0avgdata 0maxresident)k\n> 77848inputs+0outputs (410major+1798minor)pagefaults 0swaps\n>\n> hot cache:\n> $ /usr/bin/time git rev-list origin/master --not --all\n> 0.01user 0.00system 0:00.06elapsed 31%CPU (0avgtext+0avgdata 0maxresident)k\n> 0inputs+0outputs (0major+2207minor)pagefaults 0swaps\n>\n> I think that, in this particular case (when the arguments are the tips\n> of some of the branches), this should not read that many files.\n\nWhat kind of \"many files\" are you making git read? Do you have too many\nunpacked refs? Too many loose objects? \n\n> ... When nothing has changed in the remote repository (so\n> refs//* has all the remote refs) the \"git fetch\" could be almost\n> instantaneous (even in coldcache),...\n\nYou at least need to read:\n\n - what \"--all\" refs point at; to find this out, you need to read all\n unpacked refs files, and one packed-refs file;\n\n - commit objects that these refs point at; to cull refs that do not point\n at committish and dereference tag objects that point at commit, you\n need to read these objects (either loose objects or in packs);\n\n - commit objects on the ancestry graph starting from the commit pointed\n at by origin/master and the commits from \"--all\" refs, until your\n traversal from origin/master hit one of the ancestors of \"--all\" refs.\n\nI noticed that when you \"git pack-refs\" (or \"git gc\"), we do not remove\nthe leading directories of loose refs that become empty because of\npruning. This can cause many opendir() when you used to have too many\nhierachy of refs even after packing them.\n\n dir.c | 13 +++++++++----\n pack-refs.c | 5 +++++\n 2 files changed, 14 insertions(+), 4 deletions(-)\n\ndiff --git c/dir.c w/dir.c\nindex 0131983..7241631 100644\n--- c/dir.c\n+++ w/dir.c\n@@ -779,7 +779,12 @@ int is_inside_dir(const char *dir)\n \treturn get_relative_cwd(buffer, sizeof(buffer), dir) != NULL;\n }\n \n-int remove_dir_recursively(struct strbuf *path, int only_empty)\n+/*\n+ * option:\n+ * 1: remove empty directory;\n+ * 2: remove empty subdirectories, but not the directory itself\n+ */\n+int remove_dir_recursively(struct strbuf *path, int option)\n {\n \tDIR *dir = opendir(path->buf);\n \tstruct dirent *e;\n@@ -803,9 +808,9 @@ int remove_dir_recursively(struct strbuf *path, int only_empty)\n \t\tif (lstat(path->buf, &st))\n \t\t\t; /* fall thru */\n \t\telse if (S_ISDIR(st.st_mode)) {\n-\t\t\tif (!remove_dir_recursively(path, only_empty))\n+\t\t\tif (!remove_dir_recursively(path, !!option))\n \t\t\t\tcontinue; /* happy */\n-\t\t} else if (!only_empty && !unlink(path->buf))\n+\t\t} else if (!option && !unlink(path->buf))\n \t\t\tcontinue; /* happy, too */\n \n \t\t/* path too long, stat fails, or non-directory still exists */\n@@ -815,7 +820,7 @@ int remove_dir_recursively(struct strbuf *path, int only_empty)\n \tclosedir(dir);\n \n \tstrbuf_setlen(path, original_len);\n-\tif (!ret)\n+\tif (!ret && option != 2)\n \t\tret = rmdir(path->buf);\n \treturn ret;\n }\ndiff --git c/pack-refs.c w/pack-refs.c\nindex 2c76fb1..30fbae8 100644\n--- c/pack-refs.c\n+++ w/pack-refs.c\n@@ -2,6 +2,7 @@\n #include \"refs.h\"\n #include \"tag.h\"\n #include \"pack-refs.h\"\n+#include \"dir.h\"\n \n struct ref_to_prune {\n \tstruct ref_to_prune *next;\n@@ -73,10 +74,14 @@ static void prune_ref(struct ref_to_prune *r)\n \n static void prune_refs(struct ref_to_prune *r)\n {\n+\tstruct strbuf refs = STRBUF_INIT;\n+\n+\tstrbuf_addstr(&refs, git_path(\"refs\"));\n \twhile (r) {\n \t\tprune_ref(r);\n \t\tr = r->next;\n \t}\n+\tremove_dir_recursively(&refs, 2);\n }\n \n static struct lock_file packed;\n"},{"id":"94978","messageId":"alpine.LFD.2.00.0811050935270.3419@nehalem.linux-foundation.org","threadId":"16178","inReplyTo":"adf1fd3d0811050138j7b8bbed1nd94a999f55e38d61@mail.gmail.com","subject":"Re: Slow \"git rev-list origin/master --not --all\" or \"git fetch\" slow when downloading nothing","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2008-11-05T17:37:35Z","receivedAt":"2008-11-05T17:37:35Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 5 Nov 2008, Santi Béjar wrote:\n> \n> In cold cache \"git rev-list origin/master --not --all\" is slow\n> reading many files:\n\nHmm. It sounds like you possibly don't have packed refs. \n\nHave you done \"git gc\" on that thing lately? What does \"strace\" say?\n\n\t\tLinus\n"},{"id":"94999","messageId":"adf1fd3d0811051337i5fd501e1hdd3a6452930581a@mail.gmail.com","threadId":"16178","inReplyTo":"alpine.LFD.2.00.0811050935270.3419@nehalem.linux-foundation.org","subject":"Re: Slow \"git rev-list origin/master --not --all\" or \"git fetch\" slow when downloading nothing","fromName":"Santi Béjar","fromEmail":"santi@agolina.net","sentAt":"2008-11-05T21:37:30Z","receivedAt":"2008-11-05T21:37:30Z","isPatch":false,"sender":{"key":"santi@agolina.net","avatar":null},"body":"On Wed, Nov 5, 2008 at 6:37 PM, Linus Torvalds\n wrote:\n>\n>\n> On Wed, 5 Nov 2008, Santi Béjar wrote:\n>>\n>> In cold cache \"git rev-list origin/master --not --all\" is slow\n>> reading many files:\n\nSorry, s/files/data/\n\n>\n> Hmm. It sounds like you possibly don't have packed refs.\n\nThey are packed up to v2.6.27.\n\n>\n> Have you done \"git gc\" on that thing lately? What does \"strace\" say?\n>\n> Linus\n>\n\nIt is a recently cloned linux-2.6 repo, with 10 or so packs and 70\nloose objects. It spends a good fraction in:\n\nbrk(0x8318000) = 0x8318000\n\nand\n\nmmap2(NULL, 33554432, PROT_READ, MAP_PRIVATE, 4, 0x2000) = 0xb2227000\n\n(strace attatched).\n\nI should have given more info:\n\nIt is an old computer (Pentium 4 2.5 GHz)\nand the repo is on an external USB drive.\n\nOn Wed, Nov 5, 2008 at 6:32 PM, Junio C Hamano wrote:\n> \"Santi Béjar\" writes:\n>\n>> In cold cache \"git rev-list origin/master --not --all\" is slow\n>> reading many files:\n>>\n>> cold cache:\n>> $ /usr/bin/time git rev-list origin/master --not --all\n>> 0.03user 0.02system 0:04.57elapsed 1%CPU (0avgtext+0avgdata 0maxresident)k\n>> 77848inputs+0outputs (410major+1798minor)pagefaults 0swaps\n>>\n>> hot cache:\n>> $ /usr/bin/time git rev-list origin/master --not --all\n>> 0.01user 0.00system 0:00.06elapsed 31%CPU (0avgtext+0avgdata 0maxresident)k\n>> 0inputs+0outputs (0major+2207minor)pagefaults 0swaps\n>>\n>> I think that, in this particular case (when the arguments are the tips\n>> of some of the branches), this should not read that many files.\n>\n> What kind of \"many files\" are you making git read? Do you have too many\n> unpacked refs? Too many loose objects?\n>\n\nSee above.\n\n>> ... When nothing has changed in the remote repository (so\n>> refs//* has all the remote refs) the \"git fetch\" could be almost\n>> instantaneous (even in coldcache),...\n>\n> You at least need to read:\n>\n> - what \"--all\" refs point at; to find this out, you need to read all\n> unpacked refs files, and one packed-refs file;\n>\n> - commit objects that these refs point at; to cull refs that do not point\n> at committish and dereference tag objects that point at commit, you\n> need to read these objects (either loose objects or in packs);\n>\n> - commit objects on the ancestry graph starting from the commit pointed\n> at by origin/master and the commits from \"--all\" refs, until your\n> traversal from origin/master hit one of the ancestors of \"--all\" refs.\n\nYes, in the general case it is, but in this case we can bypass the\nchecking of the --all refs after checking if all the given refs are\nequal to some of the --all refs.\n\nFor me the main thing is the slow \"git fetch\" when downloading\nnothing. I think it was/is faster without the quickfetch path (at\nleast in the coldcache case).\nMaybe we can just check if the new fetched refs are already the tips\nof the corresponding remote branch.\n\nSanti\n"},{"id":"95003","messageId":"alpine.LFD.2.00.0811051344050.3419@nehalem.linux-foundation.org","threadId":"16178","inReplyTo":"adf1fd3d0811051337i5fd501e1hdd3a6452930581a@mail.gmail.com","subject":"Re: Slow \"git rev-list origin/master --not --all\" or \"git fetch\" slow when downloading nothing","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2008-11-05T22:08:06Z","receivedAt":"2008-11-05T22:08:06Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 5 Nov 2008, Santi Béjar wrote:\n> >\n> > Hmm. It sounds like you possibly don't have packed refs.\n> \n> They are packed up to v2.6.27.\n\nYeah, your strace isn't at all horrible. You don't open very many files at \nall, and you don't have any big directories.\n\nThe biggest cost when things are cold-cache is probably the seeking from \njust opening all those index files. \n\n> It is an old computer (Pentium 4 2.5 GHz) and the repo is on an external \n> USB drive.\n\nThere should be basically no CPU spent on that load, so your computer is \nfine. But I think the issue is the dog-slow IO on the USB drive, \nespecially since there are multiple pack-files and thus index files.\n\nYour strace would be more interesting with \"-Ttt\", but much of the cost is \nlikely in the page faulting of the mmap'ed data, and none of that would \nshow up in the trace, except indirectly (ie just looking at the times \nbetween the system calls).\n\n> Yes, in the general case it is, but in this case we can bypass the\n> checking of the --all refs after checking if all the given refs are\n> equal to some of the --all refs.\n\nI don't think we'll actually walk anything, because all commits will end \nup being negative.\n\nBut we'll look up the objects for even the negative commits, yes. So we're \ndoing several \"unnecessary\" object lookups, and in that sense we could \nmake this much faster by not even bothering to look them up.\n\nBut we do that to validate that the refs are _valid_, so in that sense the \nobject lookup is not \"unnecessary\" at all. Oh, and we need to peel them to \nsee if they are tag objects, in order to mark the _commit_ uninteresting \nif the object itself was uninteresting.\n\nSo in practice we do end up having to pretty much parse them all. \n\nWe could do some crazy special case for the empty set, but it would be \nbetter to see if you can improve performance with a slow disk some other, \nless hacky, way. If you use a USB stick to move between machines, maybe \nyou can make sure that it's fully packed (ie a single index file) before \nmoving it to the USB stick? That would likely help quite a bit.\n\n\t\tLinus\n"}]}