RE: [PATCH v3] Add git-grep threads param

From: Victor Leschuk <vleschuk@accesssoftek.com>
Date: Oct 27, 2015, 09:14 UTC
Message-ID: <6AE1604EE3EC5F4296C096518C6B77EE5D0FDAB9FE@mail.accesssoftek.com>
In-Reply-To: <CA+55aFwrU25x25XrRODgS1oRXqN60rmYPiXLgfs3mqRco4Oi9A@mail.gmail.com>

Hello Linus,

>> According to several tests on systems with different number of CPU cores
>> the hard-coded number of 8 threads is not optimal for all systems:

> Did you also compare cold-cache filesystem performance?

>  One of the reasons for doing threaded grep is for CPU scaling. But another is for IO scaling. If your git tree is over NFS, doing grep eight threads at a time if likely going to make things much faster even if you are on a single CPU.

Yes, I have performed tests on cold-cache FS and it looks like number of threads affects performance. Here are the results for grepping linux kernel repo on a 4-core machine (similar test was conducted on 8-core machine):

Threads: 4 Time: 39.13
Threads: 8 Time: 34.39
Threads: 16 Time: 31.46
Threads: 32 Time: 27.40

Here is test scenario:

#!/bin/bash
TIMEFORMAT=%R
GIT=/home/del/git-dev/bin/git
TESTS=10
for n in 4 8 16 32; do
    echo -n "Threads: $n Time: "
    for i in $(seq 1 $TESTS); do
        echo 3 > /proc/sys/vm/drop_caches
        time $GIT grep --threads $n -e '#define' --and \( -e MAX_PATH -e PATH_MAX \)  >/dev/null
    done 2>&1 | awk -v ntests=${TESTS} '{sum+=$1} END{printf "%.2f\n", sum/ntests}'
done

Note: With hot-cache grepping with 4 threads gives fastest results on both 4-core and 8-core machines.

Thus I think it can be useful for users to be able to tune the threads number according to their needs.