Re: Efficient way to import snapshots?

Linus Torvalds <torvalds@linux-foundation.org> writes:

I tried a git-add . on a TeX live tree (lots of itsy files). About 80% of the processor time was wait.

I think that SHA1 is costly enough that the processor(s) could get saturated when enough is done in parallel.

Potential for multithreading is here enough to make any Summer of Code student weep with joy:

a) one thread for opendir/readdir at every directory level
b) one thread for stating the files in readdir order (more likely to
   correspond to the disk layout than sorted order)
c) one thread on each directory level doing a mergesort on a different
   link field (some merge passes could be parallelized even, but let's
   not get overexcited)
d) asynch I/O requesting the data for all files to be submitted to
   SHA1
e) one thread (but no more threads than there are CPUs) per
   independent file/tree for doing SHA1
f) asynch I/O reading the index sequentially
g) one thread doing a merge pass of already sorted stuff (this can
   start once the top level directory has been read and sorted
   completely, possibly having to stop until a complete subdirectory
   comes in).
h) asynch I/O writing out the results of the merge sequentially

In fact, git-ls-files|git-add --stdin is already exploiting a bit of parallelism (and will probably profit from CFS by making much more use of the buffering capacity of the pipe). It is counterintuitive that hand-built chains work more efficiently than explicit git commands.