{"thread":{"id":"35399","subject":"slow git-cherry-pick.","startedAt":"2013-11-24T10:45:32Z","lastAt":"2013-12-04T12:46:43Z","messageCount":9,"participants":["Paweł Sikora","Duy Nguyen","Junio C Hamano","Thomas Rast"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"231012","messageId":"2142926.gg3W3MsbJZ@localhost.localdomain","threadId":"35399","inReplyTo":null,"subject":"slow git-cherry-pick.","fromName":"Paweł Sikora","fromEmail":"pawel.sikora@agmk.net","sentAt":"2013-11-24T10:45:32Z","receivedAt":"2013-11-24T10:45:32Z","isPatch":false,"sender":{"key":"pawel.sikora@agmk.net","avatar":null},"body":"Hi,\n\ni've recently reinstalled a fresh system (fc20-beta) on my workstation\nand observing a big slowdown on git cherry-pick operation (git-1.8.4.2-1).\nthe previous centos installation with an old git version works faster\n(few seconds per cherry pick). now the same operation takes >1 min.\n\nmy git repo isn't very big[1] but it's checked out on the linear lvm\nwhere random i/o generally hurts and strace shows that current git version\nperforms 2x{lstat}+1x{open,read,close} [2] on whole checkout before\ncherry-pick. there is also a .gitattributes searching on all levels\nwhich doing another tons of i/o. looks like git-status on action but\nwhy on whole repo while cherry-pick touches limited set of files?\n\nis it a bug or feature?\n\nBR,\nPaweł.\n\nplease CC me on reply.\n\n\n[1]\n$ du -sh .git/objects/\n4.2G .git/objects/\n$ find sources -type f|wc -l\n9536\n$ find buildenv -type f|wc -l\n14637\n\n[2]\nlstat(\"buildenv/boost-1.51.0/include/boost/bimap.hpp\", {st_mode=S_IFREG|0664, \nst_size=387, ...}) = 0\nlstat(\"buildenv/boost-1.51.0/include/boost/bimap.hpp\", {st_mode=S_IFREG|0664, \nst_size=387, ...}) = 0\nopen(\"buildenv/boost-1.51.0/include/boost/bimap.hpp\", O_RDONLY) = 5\nread(5, \"// Boost.Bimap\\n//\\n// Copyright (\"..., 387) = 387\nclose(5)\n\n-- \ngpg key fingerprint = 60B4 9886 AD53 EB3E 88BB 1EB5 C52E D01B 683B 9411\n"},{"id":"231013","messageId":"CACsJy8CAAov1nSGPx79U+md4xROsCydidHPcipOb_sdFwNdSkA@mail.gmail.com","threadId":"35399","inReplyTo":"2142926.gg3W3MsbJZ@localhost.localdomain","subject":"Re: slow git-cherry-pick.","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-24T12:47:10Z","receivedAt":"2013-11-24T12:47:10Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Nov 24, 2013 at 5:45 PM, Paweł Sikora wrote:\n> i've recently reinstalled a fresh system (fc20-beta) on my workstation\n> and observing a big slowdown on git cherry-pick operation (git-1.8.4.2-1).\n> the previous centos installation with an old git version works faster\n> (few seconds per cherry pick). now the same operation takes >1 min.\n\nWhat is the git version the reinstallation? Do you cherry-pick on one\ncommit or a commit range?\n-- \nDuy\n"},{"id":"231014","messageId":"CACsJy8BNVwVWh7847Vvtz0mRAY3OX4DkYZ_D_9EX7n6MfTjfTA@mail.gmail.com","threadId":"35399","inReplyTo":"CACsJy8CAAov1nSGPx79U+md4xROsCydidHPcipOb_sdFwNdSkA@mail.gmail.com","subject":"Re: slow git-cherry-pick.","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-24T12:48:13Z","receivedAt":"2013-11-24T12:48:13Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Nov 24, 2013 at 7:47 PM, Duy Nguyen wrote:\n> On Sun, Nov 24, 2013 at 5:45 PM, Paweł Sikora wrote:\n>> i've recently reinstalled a fresh system (fc20-beta) on my workstation\n>> and observing a big slowdown on git cherry-pick operation (git-1.8.4.2-1).\n>> the previous centos installation with an old git version works faster\n>> (few seconds per cherry pick). now the same operation takes >1 min.\n>\n> What is the git version the reinstallation? Do you cherry-pick on one\n\nI accidentally the word \"before\" in the first question..\n\n> commit or a commit range?\n-- \nDuy\n"},{"id":"231022","messageId":"32998962.pucYdvRloz@localhost.localdomain","threadId":"35399","inReplyTo":"CACsJy8CAAov1nSGPx79U+md4xROsCydidHPcipOb_sdFwNdSkA@mail.gmail.com","subject":"Re: slow git-cherry-pick.","fromName":"Paweł Sikora","fromEmail":"pawel.sikora@agmk.net","sentAt":"2013-11-24T19:17:25Z","receivedAt":"2013-11-24T19:17:25Z","isPatch":false,"sender":{"key":"pawel.sikora@agmk.net","avatar":null},"body":"On Sunday 24 of November 2013 19:47:10 Duy Nguyen wrote:\n> On Sun, Nov 24, 2013 at 5:45 PM, Paweł Sikora wrote:\n> > i've recently reinstalled a fresh system (fc20-beta) on my workstation\n> > and observing a big slowdown on git cherry-pick operation (git-1.8.4.2-1).\n> > the previous centos installation with an old git version works faster\n> > (few seconds per cherry pick). now the same operation takes >1 min.\n> \n> What is the git version before the reinstallation?\n\ngit-1.7.11.3-1.el5.rf.\n\ni've checked this version on another machine with centos-5.$latest\nand it does similar amout of stat/read operation quickly (~6s).\nthis \"fast\" centos-5 machine has /home on raid-0 (2x500GB) while\nmy \"slow (>1min)\" workstation has /home on linear lvm (250G+1T).\n\nso, i suppose that my \"slow\" working copy crosses disks boundary\nor spread over 1TB drive and the random git i/o impacts performance.\n\nthe question still remains - does the git need to scan whole checkout\nduring picking well defined set of files?\n\n> Do you cherry-pick on one commit or a commit range?\n\nsingle commit.\n\n-- \ngpg key fingerprint = 60B4 9886 AD53 EB3E 88BB 1EB5 C52E D01B 683B 9411\n"},{"id":"231076","messageId":"xmqqd2loz92n.fsf@gitster.dls.corp.google.com","threadId":"35399","inReplyTo":"32998962.pucYdvRloz@localhost.localdomain","subject":"Re: slow git-cherry-pick.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-11-25T17:26:40Z","receivedAt":"2013-11-25T17:26:40Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Paweł Sikora writes:\n\n> On Sunday 24 of November 2013 19:47:10 Duy Nguyen wrote:\n>> On Sun, Nov 24, 2013 at 5:45 PM, Paweł Sikora wrote:\n>> > i've recently reinstalled a fresh system (fc20-beta) on my workstation\n>> > and observing a big slowdown on git cherry-pick operation (git-1.8.4.2-1).\n>> > the previous centos installation with an old git version works faster\n>> > (few seconds per cherry pick). now the same operation takes >1 min.\n>> \n>> What is the git version before the reinstallation?\n>\n> git-1.7.11.3-1.el5.rf.\n>\n> i've checked this version on another machine with centos-5.$latest\n> and it does similar amout of stat/read operation quickly (~6s).\n> this \"fast\" centos-5 machine has /home on raid-0 (2x500GB) while\n> my \"slow (>1min)\" workstation has /home on linear lvm (250G+1T).\n>\n> so, i suppose that my \"slow\" working copy crosses disks boundary\n> or spread over 1TB drive and the random git i/o impacts performance.\n>\n> the question still remains - does the git need to scan whole checkout\n> during picking well defined set of files?\n\nWe do update-index to see what local changes you have upfront in\norder to avoid stomping on them (and we do not know upfront what\npaths the cherry-picked commit would change, given that there may be\nrenames involved), so the answer is unfortunately yes, we would need\nto do lstat(2) the whole thing.\n\nDoing that lstat(2) more lazily and do away with the update-index\nmight be possible, but I suspect that may be quite a lot of work.\n"},{"id":"231458","messageId":"2679995.THr9ekVpkO@localhost.localdomain","threadId":"35399","inReplyTo":"xmqqd2loz92n.fsf@gitster.dls.corp.google.com","subject":"Re: slow git-cherry-pick.","fromName":"Paweł Sikora","fromEmail":"pawel.sikora@agmk.net","sentAt":"2013-12-03T19:31:26Z","receivedAt":"2013-12-03T19:31:26Z","isPatch":false,"sender":{"key":"pawel.sikora@agmk.net","avatar":null},"body":"On Monday 25 of November 2013 09:26:40 Junio C Hamano wrote:\n> Paweł Sikora writes:\n> > On Sunday 24 of November 2013 19:47:10 Duy Nguyen wrote:\n> >> On Sun, Nov 24, 2013 at 5:45 PM, Paweł Sikora \nwrote:\n> >> > i've recently reinstalled a fresh system (fc20-beta) on my workstation\n> >> > and observing a big slowdown on git cherry-pick operation\n> >> > (git-1.8.4.2-1).\n> >> > the previous centos installation with an old git version works faster\n> >> > (few seconds per cherry pick). now the same operation takes >1 min.\n> >> \n> >> What is the git version before the reinstallation?\n> > \n> > git-1.7.11.3-1.el5.rf.\n> > \n> > i've checked this version on another machine with centos-5.$latest\n> > and it does similar amout of stat/read operation quickly (~6s).\n> > this \"fast\" centos-5 machine has /home on raid-0 (2x500GB) while\n> > my \"slow (>1min)\" workstation has /home on linear lvm (250G+1T).\n> > \n> > so, i suppose that my \"slow\" working copy crosses disks boundary\n> > or spread over 1TB drive and the random git i/o impacts performance.\n> > \n> > the question still remains - does the git need to scan whole checkout\n> > during picking well defined set of files?\n> \n> We do update-index to see what local changes you have upfront in\n> order to avoid stomping on them (and we do not know upfront what\n> paths the cherry-picked commit would change, given that there may be\n> renames involved), so the answer is unfortunately yes, we would need\n> to do lstat(2) the whole thing.\n\nthis is quite weird for me (user). git pull also fetches objects from\nserver and aborts user's action if working copy contains uncommitted\nmodifications on files that will be modified by incoming objects.\n\nfrom the other side, the cherry-pick needs to stat() the whole working\ncopy to achieve similar precondition. looks like suboptimal implementation.\n\n> Doing that lstat(2) more lazily and do away with the update-index\n> might be possible, but I suspect that may be quite a lot of work.\n\nmaybe you can use the existing implementation used by 'pull' ?\n\n\n-- \ngpg key fingerprint = 60B4 9886 AD53 EB3E 88BB 1EB5 C52E D01B 683B 9411\n"},{"id":"231464","messageId":"87fvq9n154.fsf@linux-1gf2.Speedport_W723_V_Typ_A_1_00_098","threadId":"35399","inReplyTo":"2142926.gg3W3MsbJZ@localhost.localdomain","subject":"Re: slow git-cherry-pick.","fromName":"Thomas Rast","fromEmail":"tr@thomasrast.ch","sentAt":"2013-12-03T20:13:43Z","receivedAt":"2013-12-03T20:13:43Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Paweł Sikora writes:\n>\n\nUmm, there's a gem here that the thread missed so far:\n\n> my git repo isn't very big[1] but it's checked out on the linear lvm\n> where random i/o generally hurts and strace shows that current git version\n> performs 2x{lstat}+1x{open,read,close} [2] on whole checkout before\n ^^^^^^^^^\n\nThere's no reason why it should do the lstat() *twice* for every file.\nBut Paweł is right; the code path roughly goes like this:\n\nint cmd_cherry_pick(int argc, const char **argv, const char *prefix)\n{\n[...]\n\tres = sequencer_pick_revisions(&opts);\n\nint sequencer_pick_revisions(struct replay_opts *opts)\n{\n[...]\n\tread_and_refresh_cache(opts);\n[...]\n\treturn pick_commits(todo_list, opts);\n}\n\nstatic int pick_commits(struct commit_list *todo_list, struct replay_opts *opts)\n{\n[...]\n\tread_and_refresh_cache(opts);\n\n\nI'm too tired to dig further, but AFAICT it's just a rather obvious case\nof duplication of effort.\n\n-- \nThomas Rast\ntr@thomasrast.ch\n"},{"id":"231481","messageId":"CACsJy8Be2USmGA--FLT3LERTde327Ue65CCjoLHi5SzNzUX1dw@mail.gmail.com","threadId":"35399","inReplyTo":"87fvq9n154.fsf@linux-1gf2.Speedport_W723_V_Typ_A_1_00_098","subject":"Re: slow git-cherry-pick.","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-04T01:07:23Z","receivedAt":"2013-12-04T01:07:23Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Dec 4, 2013 at 3:13 AM, Thomas Rast wrote:\n> Paweł Sikora writes:\n>>\n>\n> Umm, there's a gem here that the thread missed so far:\n>\n>> my git repo isn't very big[1] but it's checked out on the linear lvm\n>> where random i/o generally hurts and strace shows that current git version\n>> performs 2x{lstat}+1x{open,read,close} [2] on whole checkout before\n> ^^^^^^^^^\n>\n> There's no reason why it should do the lstat() *twice* for every file.\n> But Paweł is right; the code path roughly goes like this:\n>\n> int cmd_cherry_pick(int argc, const char **argv, const char *prefix)\n> {\n> [...]\n> res = sequencer_pick_revisions(&opts);\n>\n> int sequencer_pick_revisions(struct replay_opts *opts)\n> {\n> [...]\n> read_and_refresh_cache(opts);\n> [...]\n> return pick_commits(todo_list, opts);\n> }\n>\n> static int pick_commits(struct commit_list *todo_list, struct replay_opts *opts)\n> {\n> [...]\n> read_and_refresh_cache(opts);\n>\n>\n> I'm too tired to dig further, but AFAICT it's just a rather obvious case\n> of duplication of effort.\n\nThat's something to optimize, but it's single commit picking,\nsequencer_pick_revisions() should call single_pick() instead of\npick_commits().\n\nThe read+close on the whole checkout looks like there's problem with\nrefresh operation and git decides to read up and verify sha-1 by\ncontent. Pawel, if you run \"strace git update-index --refresh\" twice,\ndoes it still show 1 stat + 1 read for every entry on the second try?\n-- \nDuy\n"},{"id":"231502","messageId":"1827709.rD7YulLKCL@localhost.localdomain","threadId":"35399","inReplyTo":"CACsJy8Be2USmGA--FLT3LERTde327Ue65CCjoLHi5SzNzUX1dw@mail.gmail.com","subject":"Re: slow git-cherry-pick.","fromName":"Paweł Sikora","fromEmail":"pawel.sikora@agmk.net","sentAt":"2013-12-04T12:46:43Z","receivedAt":"2013-12-04T12:46:43Z","isPatch":false,"sender":{"key":"pawel.sikora@agmk.net","avatar":null},"body":"On Wednesday 04 of December 2013 08:07:23 Duy Nguyen wrote:\n> On Wed, Dec 4, 2013 at 3:13 AM, Thomas Rast wrote:\n> > Paweł Sikora writes:\n> > \n> > Umm, there's a gem here that the thread missed so far:\n> >> my git repo isn't very big[1] but it's checked out on the linear lvm\n> >> where random i/o generally hurts and strace shows that current git\n> >> version\n> >> performs 2x{lstat}+1x{open,read,close} [2] on whole checkout before\n> >> \n> > ^^^^^^^^^\n> > \n> > There's no reason why it should do the lstat() *twice* for every file.\n> > But Paweł is right; the code path roughly goes like this:\n> > \n> > int cmd_cherry_pick(int argc, const char **argv, const char *prefix)\n> > {\n> > [...]\n> > \n> > res = sequencer_pick_revisions(&opts);\n> > \n> > int sequencer_pick_revisions(struct replay_opts *opts)\n> > {\n> > [...]\n> > \n> > read_and_refresh_cache(opts);\n> > \n> > [...]\n> > \n> > return pick_commits(todo_list, opts);\n> > \n> > }\n> > \n> > static int pick_commits(struct commit_list *todo_list, struct replay_opts\n> > *opts) {\n> > [...]\n> > \n> > read_and_refresh_cache(opts);\n> > \n> > I'm too tired to dig further, but AFAICT it's just a rather obvious case\n> > of duplication of effort.\n> \n> That's something to optimize, but it's single commit picking,\n> sequencer_pick_revisions() should call single_pick() instead of\n> pick_commits().\n> \n> The read+close on the whole checkout looks like there's problem with\n> refresh operation and git decides to read up and verify sha-1 by\n> content. Pawel, if you run \"strace git update-index --refresh\" twice,\n> does it still show 1 stat + 1 read for every entry on the second try?\n\nthe 'git update-index --refresh' runs quickly and strace shows only lstat()\non every file. i see no massive open/read actions in this case.\n\n$ strace -o strace-try1.log git update-index --refresh\nhmdb: needs update\n$ strace -o strace-try2.log git update-index --refresh\nhmdb: needs update\n\n$ grep -c lstat strace-try1.log \n33793\n$ grep -c lstat strace-try2.log\n33793\n\n-- \ngpg key fingerprint = 60B4 9886 AD53 EB3E 88BB 1EB5 C52E D01B 683B 9411\n"}]}