{"thread":{"id":"11164","subject":"Re: Git and GCC. Why not with fork, exec and pipes like in linux?","startedAt":"2007-12-06T19:25:28Z","lastAt":"2007-12-06T20:37:32Z","messageCount":2,"participants":["J.C. Pizarro"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"62189","messageId":"998d0e4a0712061125h3d44139ctb7f5600bc8467292@mail.gmail.com","threadId":"11164","inReplyTo":null,"subject":"Re: Git and GCC. Why not with fork, exec and pipes like in linux?","fromName":"J.C. Pizarro","fromEmail":"jcpiza@gmail.com","sentAt":"2007-12-06T19:25:28Z","receivedAt":"2007-12-06T19:25:28Z","isPatch":false,"sender":{"key":"jcpiza@gmail.com","avatar":null},"body":"On 2007/12/06, \"Jon Smirl\" wrote:\n> On 12/6/07, Linus Torvalds wrote:\n> > On Thu, 6 Dec 2007, Jeff King wrote:\n> > >\n> > > What is really disappointing is that we saved only about 20% of the\n> > > time. I didn't sit around watching the stages, but my guess is that we\n> > > spent a long time in the single threaded \"writing objects\" stage with a\n> > > thrashing delta cache.\n> >\n> > I don't think you spent all that much time writing the objects. That part\n> > isn't very intensive, it's mostly about the IO.\n> >\n> > I suspect you may simply be dominated by memory-throughput issues. The\n> > delta matching doesn't cache all that well, and using two or more cores\n> > isn't going to help all that much if they are largely waiting for memory\n> > (and quite possibly also perhaps fighting each other for a shared cache?\n> > Is this a Core 2 with the shared L2?)\n>\n> When I lasted looked at the code, the problem was in evenly dividing\n> the work. I was using a four core machine and most of the time one\n> core would end up with 3-5x the work of the lightest loaded core.\n> Setting pack.threads up to 20 fixed the problem. With a high number of\n> threads I was able to get a 4hr pack to finished in something like\n> 1:15.\n>\n> A scheme where each core could work a minute without communicating to\n> the other cores would be best. It would also be more efficient if the\n> cores could avoid having sync points between them.\n>\n> --\n> Jon Smirl\n> jonsmirl@gmail.com\n\nFor multicores CPUs, don't divide the work in threads.\nTo divide the work in processes!\n\nTips, tricks and hacks: to use fork, exec, pipes and another IPC mechanisms like\nmutexes, shared memory's IPC, file locks, pipes, semaphores, RPCs, sockets, etc.\nto access concurrently and parallely to the filelocked database.\n\nFor Intel Quad Core e.g., x4 cores, it need a parent process and 4\nchild processes\nlinked to the parent with pipes.\n\nThe parent process can be\n* no-threaded using select/epoll/libevent\n* threaded using Pth (GNU Portable Threads), NPTL (from RedHat) or whatever.\n\n J.C.Pizarro\n"},{"id":"62197","messageId":"998d0e4a0712061237j6ed43aaav5934e4fe63398233@mail.gmail.com","threadId":"11164","inReplyTo":"998d0e4a0712061125h3d44139ctb7f5600bc8467292@mail.gmail.com","subject":"Re: Git and GCC. Why not with fork, exec and pipes like in linux?","fromName":"J.C. Pizarro","fromEmail":"jcpiza@gmail.com","sentAt":"2007-12-06T20:37:32Z","receivedAt":"2007-12-06T20:37:32Z","isPatch":false,"sender":{"key":"jcpiza@gmail.com","avatar":null},"body":"On 2007/12/6, J.C. Pizarro , i wrote:\n> For multicores CPUs, don't divide the work in threads.\n> To divide the work in processes!\n>\n> Tips, tricks and hacks: to use fork, exec, pipes and another IPC mechanisms like\n> mutexes, shared memory's IPC, file locks, pipes, semaphores, RPCs, sockets, etc.\n> to access concurrently and parallely to the filelocked database.\n\nI'm sorry, we don't need exec. We need fork, pipes and another IPC mechanisms\nbecause it so shares easy the C code for parallelism.\n\nThanks to Linus because GIT is implemented in C language to interact with\nsystem calls of the kernel written in C.\n\n> For Intel Quad Core e.g., x4 cores, it need a parent process and 4\n> child processes linked to the parent with pipes.\n\nFor peak performance (e.g 99.9% usage), the minimum number of child\nprocesses should be more than 4, normally between e.g. 6 and 10 processes\ndepending on the statistics of idle's stalls of the cores.\n\n> The parent process can be\n> * no-threaded using select/epoll/libevent\n> * threaded using Pth (GNU Portable Threads), NPTL (from RedHat) or whatever.\n\nNote: there is a little design's problem with slowdown of I/O bandwith when\nthe parent is multithreaded and the children MUST to be multithreaded that\nwe can't avoid them to be non-multithreaded for maximum I/O bandwith.\n\nThe \"finding of the smallest spanning forest with deltas\" consumes a lot of\nCPU, so if it scales well in a CPU x4 cores then it can to reduce 4\nhours to 1 hour.\n\n J.C.Pizarro :)\n"}]}