Re: Git and GCC. Why not with fork, exec and pipes like in linux?

From: J.C. Pizarro <jcpiza@gmail.com>
Date: Dec 6, 2007, 20:37 UTC
Message-ID: <998d0e4a0712061237j6ed43aaav5934e4fe63398233@mail.gmail.com>
In-Reply-To: <998d0e4a0712061125h3d44139ctb7f5600bc8467292@mail.gmail.com>

On 2007/12/6, J.C. Pizarro <jcpiza@gmail.com>, i wrote:

Show 6 quoted lines

> For multicores CPUs, don't divide the work in threads.
> To divide the work in processes!
>
> Tips, tricks and hacks: to use fork, exec, pipes and another IPC mechanisms like
> mutexes, shared memory's IPC, file locks, pipes, semaphores, RPCs, sockets, etc.
> to access concurrently and parallely to the filelocked database.

I'm sorry, we don't need exec. We need fork, pipes and another IPC mechanisms because it so shares easy the C code for parallelism.

Thanks to Linus because GIT is implemented in C language to interact with system calls of the kernel written in C.

> For Intel Quad Core e.g., x4 cores, it need a parent process and 4
> child processes linked to the parent with pipes.

For peak performance (e.g 99.9% usage), the minimum number of child processes should be more than 4, normally between e.g. 6 and 10 processes depending on the statistics of idle's stalls of the cores.

> The parent process can be
> * no-threaded using select/epoll/libevent
> * threaded using Pth (GNU Portable Threads), NPTL (from RedHat) or whatever.

Note: there is a little design's problem with slowdown of I/O bandwith when
the parent is multithreaded and the children MUST to be multithreaded that
we can't avoid them to be non-multithreaded for maximum I/O bandwith.

The "finding of the smallest spanning forest with deltas" consumes a lot of CPU, so if it scales well in a CPU x4 cores then it can to reduce 4 hours to 1 hour.

   J.C.Pizarro :)