From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date: Tue, 29 Mar 2022 11:45:17 GMT
Subject: Re: [PATCH v4 00/13] core.fsyncmethod: add 'batch' mode for faster fsyncing of multiple objects
Message-ID: <220329.868rst2cei.gmgdl@evledraar.gmail.com>
In-Reply-To: <pull.1134.v4.git.1648514552.gitgitgadget@gmail.com>


On Tue, Mar 29 2022, Neeraj K. Singh via GitGitGadget wrote:

> V4 changes:
>
>  * Make ODB transactions nestable.
>  * Add an ODB transaction around writing out the cached tree.
>  * Change update-index to use a more straightforward way of managing ODB
>    transactions.
>  * Fix missing 'local's in lib-unique-files
>  * Add a per-iteration setup mechanism to test_perf.
>  * Fix camelCasing in warning message.

Despite my
https://lore.kernel.org/git/220329.86czi52ekn.gmgdl@evledraar.gmail.com/
I eventually gave up on trying to extract meaningful numbers from
t/perf, I can never quite find out if they're because of its
shellscripts shenanigans or actual code.

(And also; I realize I didn't follow-up on
https://lore.kernel.org/git/CANQDOdcFN5GgOPZ3hqCsjHDTiRfRpqoAKxjF1n9D6S8oD9--_A@mail.gmail.com/,
sorry):

But I came up with this (uses my thin
https://gitlab.com/avar/git-hyperfine/ wrapper, and you should be able
to apt get hyperfine):
	
	#!/bin/sh
	set -xe
	
	if ! test -d /tmp/scalar.git
	then
		git clone --bare https://github.com/Microsoft/scalar.git /tmp/scalar.git
		mv /tmp/scalar.git/objects/pack/*.pack /tmp/scalar.git/my.pack
	fi
	git hyperfine \
	        --warmup 1 -r 3 \
		-L rev neeraj-v4,avar-RFC \
		-s 'make CFLAGS=-O3 && rm -rf repo && git init repo && cp -R t repo/ && git ls-files -- t >repo/.git/to-add.txt' \
		-p 'rm -rf repo/.git/objects/* repo/.git/index' \
		$@'./git -c core.fsync=loose-object -c core.fsyncMethod=batch -C repo update-index --add --stdin <repo/.git/to-add.txt'
	
	git hyperfine \
	        --warmup 1 -r 3 \
		-L rev neeraj-v4,avar-RFC \
		-s 'make CFLAGS=-O3 && rm -rf repo && git init repo && cp -R t repo/' \
		-p 'rm -rf repo/.git/objects/* repo/.git/index' \
		$@'./git -c core.fsync=loose-object -c core.fsyncMethod=batch -C repo add .'
	
	git hyperfine \
	        --warmup 1 -r 3 \
		-L rev neeraj-v4,avar-RFC \
	        -s 'make CFLAGS=-O3' \
	        -p 'git init --bare dest.git' \
	        -c 'rm -rf dest.git' \
	        $@'./git -C dest.git -c core.fsyncMethod=batch unpack-objects </tmp/scalar.git/my.pack'

Those tags are your v4 here & the v2 of the RFC I sent at
https://lore.kernel.org/git/RFC-cover-v2-0.7-00000000000-20220323T140753Z-avarab@gmail.com/

Which shows my RFC v2 is ~20% faster with:

    $ PFX='strace' ~/g/git.meta/benchmark.sh "strace "

    1.22 ± 0.02 times faster than 'strace ./git -c core.fsync=loose-object -c core.fsyncMethod=batch -C repo update-index --add --stdin <repo/.git/to-add.txt' in 'neeraj-v4'
    1.22 ± 0.01 times faster than 'strace ./git -c core.fsync=loose-object -c core.fsyncMethod=batch -C repo add .' in 'neeraj-v4'
    1.00 ± 0.01 times faster than 'strace ./git -C dest.git -c core.fsyncMethod=batch unpack-objects </tmp/scalar.git/my.pack' in 'neeraj-v4'

But only for add/update-index, is the unpack-objects not using the
tmp-objdir? (presumably yes).

As noted before I've found "strace" to be a handy way to "simulate"
slower FS ops on a ramdisk (I get about the same numbers sometimes on
the actual non-SSD disk, but due to load on the system (that I'm not in
full control of[1]) I can't get hyperfine to be happy with the
non-fuzzyness:

    1.06 ± 0.02 times faster than './git -c core.fsync=loose-object -c core.fsyncMethod=batch -C repo update-index --add --stdin <repo/.git/to-add.txt' in 'neeraj-v4'
    1.06 ± 0.03 times faster than './git -c core.fsync=loose-object -c core.fsyncMethod=batch -C repo add .' in 'neeraj-v4'
    1.01 ± 0.01 times faster than './git -C dest.git -c core.fsyncMethod=batch unpack-objects </tmp/scalar.git/my.pack' in 'neeraj-v4'

FWIW these are my actual non-fuzzy-with-strace numbers on the
not-ramdisk, as you can see the intervals overlap, but for the first two
the "min" time is never close to the RFC v2:
	
	$ XDG_RUNTIME_DIR=/tmp/ghf ~/g/git.meta/benchmark.sh
	+ test -d /tmp/scalar.git
	+ git hyperfine --warmup 1 -r 3 -L rev neeraj-v4,avar-RFC -s make CFLAGS=-O3 && rm -rf repo && git init repo && cp -R t repo/ && git ls-files -- t >repo/.git/to-add.txt -p rm -rf repo/.git/objects/* repo/.git/index ./git -c core.fsync=loose-object -c core.fsyncMethod=batch -C repo update-index --add --stdin <repo/.git/to-add.txt
	Benchmark 1: ./git -c core.fsync=loose-object -c core.fsyncMethod=batch -C repo update-index --add --stdin <repo/.git/to-add.txt' in 'neeraj-v4
	  Time (mean ± σ):      1.043 s ±  0.143 s    [User: 0.184 s, System: 0.193 s]
	  Range (min … max):    0.943 s …  1.207 s    3 runs
	
	Benchmark 2: ./git -c core.fsync=loose-object -c core.fsyncMethod=batch -C repo update-index --add --stdin <repo/.git/to-add.txt' in 'avar-RFC
	  Time (mean ± σ):     877.6 ms ± 183.4 ms    [User: 197.9 ms, System: 149.4 ms]
	  Range (min … max):   697.8 ms … 1064.4 ms    3 runs
	
	Summary
	  './git -c core.fsync=loose-object -c core.fsyncMethod=batch -C repo update-index --add --stdin <repo/.git/to-add.txt' in 'avar-RFC' ran
	    1.19 ± 0.30 times faster than './git -c core.fsync=loose-object -c core.fsyncMethod=batch -C repo update-index --add --stdin <repo/.git/to-add.txt' in 'neeraj-v4'
	+ git hyperfine --warmup 1 -r 3 -L rev neeraj-v4,avar-RFC -s make CFLAGS=-O3 && rm -rf repo && git init repo && cp -R t repo/ -p rm -rf repo/.git/objects/* repo/.git/index ./git -c core.fsync=loose-object -c core.fsyncMethod=batch -C repo add .
	Benchmark 1: ./git -c core.fsync=loose-object -c core.fsyncMethod=batch -C repo add .' in 'neeraj-v4
	  Time (mean ± σ):      1.019 s ±  0.057 s    [User: 0.213 s, System: 0.194 s]
	  Range (min … max):    0.963 s …  1.076 s    3 runs
	
	Benchmark 2: ./git -c core.fsync=loose-object -c core.fsyncMethod=batch -C repo add .' in 'avar-RFC
	  Time (mean ± σ):     918.6 ms ±  34.4 ms    [User: 207.8 ms, System: 164.1 ms]
	  Range (min … max):   880.6 ms … 947.5 ms    3 runs
	
	Summary
	  './git -c core.fsync=loose-object -c core.fsyncMethod=batch -C repo add .' in 'avar-RFC' ran
	    1.11 ± 0.07 times faster than './git -c core.fsync=loose-object -c core.fsyncMethod=batch -C repo add .' in 'neeraj-v4'
	+ git hyperfine --warmup 1 -r 3 -L rev neeraj-v4,avar-RFC -s make CFLAGS=-O3 -p git init --bare dest.git -c rm -rf dest.git ./git -C dest.git -c core.fsyncMethod=batch unpack-objects </tmp/scalar.git/my.pack
	Benchmark 1: ./git -C dest.git -c core.fsyncMethod=batch unpack-objects </tmp/scalar.git/my.pack' in 'neeraj-v4
	  Time (mean ± σ):      1.362 s ±  0.285 s    [User: 1.021 s, System: 0.186 s]
	  Range (min … max):    1.192 s …  1.691 s    3 runs
	
	  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
	
	Benchmark 2: ./git -C dest.git -c core.fsyncMethod=batch unpack-objects </tmp/scalar.git/my.pack' in 'avar-RFC
	 ⠏ Performing warmup runs         ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ ⠙ Performing warmup runs         ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  Time (mean ± σ):      1.188 s ±  0.009 s    [User: 1.025 s, System: 0.161 s]
	  Range (min … max):    1.180 s …  1.199 s    3 runs
	 
	Summary
	  './git -C dest.git -c core.fsyncMethod=batch unpack-objects </tmp/scalar.git/my.pack' in 'avar-RFC' ran
	    1.15 ± 0.24 times faster than './git -C dest.git -c core.fsyncMethod=batch unpack-objects </tmp/scalar.git/my.pack' in 'neeraj-v4'

1. I do my git hacking on a bare metal box I rent with some friends, and
   one of them is running one those persistent video game daemons
   written in Java. So I think all my non-RAM I/O numbers are
   continually fuzzed by what players are doing in Minecraft or whatever
   that thing is...