{"thread":{"id":"30481","subject":"Seeing high CPU usage during git-upload-pack","startedAt":"2012-05-09T07:01:37Z","lastAt":"2012-05-10T20:41:11Z","messageCount":4,"participants":["David Ebbo","Nguyen Thai Ngoc Duy"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"191180","messageId":"CAPeUw3EsFCxki6obAAeVBb0g4MKNWBteL3VueG5djbajkFiv3A@mail.gmail.com","threadId":"30481","inReplyTo":null,"subject":"Seeing high CPU usage during git-upload-pack","fromName":"David Ebbo","fromEmail":"david.ebbo@gmail.com","sentAt":"2012-05-09T07:01:37Z","receivedAt":"2012-05-09T07:01:37Z","isPatch":false,"sender":{"key":"david.ebbo@gmail.com","avatar":"https://gravatar.com/avatar/afaf05bed461bda07768f1ab79afcbb8b56055774757f1b8968c67e97cbd61b6?d=mp&s=160"},"body":"We’re using git-upload-pack in a simple git server. When the repo is\non a slow share, we’re noticing that git-upload-pack is hugging the\nCPU during the whole pack-objects operation.\n\nMore details:\n\n- git-upload-pack gets launched (with params git-upload-pack\n--stateless-rpc //path/to/repo)\n- it itself launches a git process (with params pack-objects --revs\n--all --stdout --progress --delta-base-offset). This git process is\nthe one that does all the disk I/O, but it's not the one using up the\nCPU time.\n- git-upload-pack appears to be waiting for the git process, and is at\nhigh CPU during that time, suggesting that it's doing some kind of\nbusy wait. To further test that, I stopped the git process in the\ndebugger, preventing it from making progress. At that point,\ngit-upload-pack pegs the CPU forever.\n\nWould someone familiar with the sources be able to comment on this\nbusy wait behavior, and on whether it could conceivably be done in a\nway that doesn’t use up some much CPU?\n\nThanks,\nDavid\n"},{"id":"191190","messageId":"CACsJy8DMm0ZNiQYQ1CvOfDJu5ogP42Y06792iA_xyzeHDGLJaw@mail.gmail.com","threadId":"30481","inReplyTo":"CAPeUw3EsFCxki6obAAeVBb0g4MKNWBteL3VueG5djbajkFiv3A@mail.gmail.com","subject":"Re: Seeing high CPU usage during git-upload-pack","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-05-09T10:37:38Z","receivedAt":"2012-05-09T10:37:38Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, May 9, 2012 at 2:01 PM, David Ebbo wrote:\n> We’re using git-upload-pack in a simple git server. When the repo is\n> on a slow share, we’re noticing that git-upload-pack is hugging the\n> CPU during the whole pack-objects operation.\n\nWhat OS is it run on? What git version?\n\n> More details:\n>\n> - git-upload-pack gets launched (with params git-upload-pack\n> --stateless-rpc //path/to/repo)\n> - it itself launches a git process (with params pack-objects --revs\n> --all --stdout --progress --delta-base-offset). This git process is\n> the one that does all the disk I/O, but it's not the one using up the\n> CPU time.\n> - git-upload-pack appears to be waiting for the git process, and is at\n> high CPU during that time, suggesting that it's doing some kind of\n> busy wait. To further test that, I stopped the git process in the\n> debugger, preventing it from making progress. At that point,\n> git-upload-pack pegs the CPU forever.\n\nHow about stopping upload-pack and see where it stops? There's a main\nloop in upload-pack.c, create_pack_file() that is only active when\ndata comes. I looked but failed to see how it becomes a busywait loop.\nMaybe you can put some debugging there.\n\nSetting env var GIT_TRACE_PACKET and GIT_DEBUG_SEND_PACK before\nrunning upload-pack might also help.\n\n> Would someone familiar with the sources be able to comment on this\n> busy wait behavior, and on whether it could conceivably be done in a\n> way that doesn’t use up some much CPU?\n-- \nDuy\n"},{"id":"191199","messageId":"CACsJy8CbaM6UNCT3ZP0=VZE9GhH2eJpgKe-dnGU5ubn_4gS=Pw@mail.gmail.com","threadId":"30481","inReplyTo":"CAPeUw3HCkKKBP6RnZzR0TYOO0hHtJ4_ma-Je4a-xKhPpNPJhHw@mail.gmail.com","subject":"Re: Seeing high CPU usage during git-upload-pack","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-05-09T15:11:07Z","receivedAt":"2012-05-09T15:11:07Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Hi,\n\nI don't see anything private in the mail, so I CC git@vger again. It's\na good thing to keep git@vger in the loop because there will be more\n(experienced) people to help you out.\n\nOn Wed, May 9, 2012 at 9:37 PM, David Ebbo wrote:\n> Thanks Duy for your looking at this.\n>\n> On Wed, May 9, 2012 at 3:37 AM, Nguyen Thai Ngoc Duy wrote:\n>>\n>> What OS is it run on? What git version?\n>>\n>\n> This is using 1.7.9 on Windows (msysgit).\n\nAh... msysgit reimplements poll() function, in compat/win32/poll.c. If\nit goes rouge, you could get an active loop. But this is just my\nsuspicion. I don't know anything about msysgit's poll implementation\nso I can be completely wrong here.\n\n>> How about stopping upload-pack and see where it stops? There's a main\n>> loop in upload-pack.c, create_pack_file() that is only active when\n>> data comes. I looked but failed to see how it becomes a busywait loop.\n>> Maybe you can put some debugging there.\n>>\n>> Setting env var GIT_TRACE_PACKET and GIT_DEBUG_SEND_PACK before\n>> running upload-pack might also help.\n>>\n>\n> I'll try those env vars, and will debug as needed (I haven't done that\n> yet, so I'll need to get set up )\n>\n> Thanks for the pointers!\n-- \nDuy\n"},{"id":"191354","messageId":"CAPeUw3HyCz921upWA3YBtJXTOKTcdP5tDxUv3drR7=aovJ07_g@mail.gmail.com","threadId":"30481","inReplyTo":"CACsJy8CbaM6UNCT3ZP0=VZE9GhH2eJpgKe-dnGU5ubn_4gS=Pw@mail.gmail.com","subject":"Re: Seeing high CPU usage during git-upload-pack","fromName":"David Ebbo","fromEmail":"david.ebbo@gmail.com","sentAt":"2012-05-10T20:41:11Z","receivedAt":"2012-05-10T20:41:11Z","isPatch":false,"sender":{"key":"david.ebbo@gmail.com","avatar":"https://gravatar.com/avatar/afaf05bed461bda07768f1ab79afcbb8b56055774757f1b8968c67e97cbd61b6?d=mp&s=160"},"body":"On Wed, May 9, 2012 at 8:11 AM, Nguyen Thai Ngoc Duy wrote:\n>\n> Ah... msysgit reimplements poll() function, in compat/win32/poll.c. If\n> it goes rouge, you could get an active loop. But this is just my\n> suspicion. I don't know anything about msysgit's poll implementation\n> so I can be completely wrong here.\n>\n\nSorry for the delayed replay, and thanks for pointing out that this is\nmsysgit specific and not general to all platform. I guess the next\nstep is to discuss it on their forum (msysgit@googlegroups.com).\n\nthanks!\nDavid\n"}]}