[RFC PATCH v2 4/7] update-index: have the index fsync() flush the loose objects

As with unpack-objects in a preceding commit have update-index.c make use of the HASH_N_OBJECTS{,_{FIRST,LAST}} flags. We now have a "batch" mode again for "update-index".

Adding the t/* directory from git.git on a Linux ramdisk is a bit faster than with the tmp-objdir indirection:

	$ git hyperfine -L rev ns/batched-fsync,HEAD -s 'make CFLAGS=-O3 && rm -rf repo && git init repo && cp -R t repo/ && git ls-files -- t >repo/.git/to-add.txt' -p 'rm -rf repo/.git/objects/* repo/.git/index' './git -c core.fsync=loose-object -c core.fsyncMethod=batch -C repo update-index --add --stdin <repo/.git/to-add.txt' --warmup 1 -r 10Benchmark 1: ./git -c core.fsync=loose-object -c core.fsyncMethod=batch -C repo update-index --add --stdin <repo/.git/to-add.txt' in 'ns/batched-fsync
	  Time (mean ± σ):     281.1 ms ±   2.6 ms    [User: 186.2 ms, System: 92.3 ms]
	  Range (min … max):   278.3 ms … 287.0 ms    10 runs

	Benchmark 2: ./git -c core.fsync=loose-object -c core.fsyncMethod=batch -C repo update-index --add --stdin <repo/.git/to-add.txt' in 'HEAD
	  Time (mean ± σ):     265.9 ms ±   2.6 ms    [User: 181.7 ms, System: 82.1 ms]
	  Range (min … max):   262.0 ms … 270.3 ms    10 runs

	Summary
	  './git -c core.fsync=loose-object -c core.fsyncMethod=batch -C repo update-index --add --stdin <repo/.git/to-add.txt' in 'HEAD' ran
	    1.06 ± 0.01 times faster than './git -c core.fsync=loose-object -c core.fsyncMethod=batch -C repo update-index --add --stdin <repo/.git/to-add.txt' in 'ns/batched-fsync'

And as before running that with "strace --summary-only" slows things down a bit (probably mimicking slower I/O a bit). I then get:

	Summary
	  'strace --summary-only ./git -c core.fsync=loose-object -c core.fsyncMethod=batch -C repo update-index --add --stdin <repo/.git/to-add.txt' in 'HEAD' ran
	    1.19 ± 0.03 times faster than 'strace --summary-only ./git -c core.fsync=loose-object -c core.fsyncMethod=batch -C repo update-index --add --stdin <repo/.git/to-add.txt' in 'ns/batched-fsync'

This one has a twist though, instead of fsync()-ing on the last object we write we'll not do that, and instead defer the fsync() until we write the index itself. This is outlined in [1] (as "METHOD THREE").

Because of this under FSYNC_METHOD_BATCH we'll do the N objects (possibly only one, because we're lazy) as HASH_N_OBJECTS, and we'll even now support doing this via N arguments on the command-line.

Then we won't fsync() any of it, but we will rename it in-place (which, if we were still using the tmp-objdir, would leave it "staged" in the tmp-objdir).

We'll then have the fsync() for the index update "flush" that out, and thus avoid two fsync() calls when one will do.

Running this with the "git hyperfine" command mentioned in a preceding commit with "strace --summary-only" shows that we do 1 fsync() now instead of 2, and have one more sync_file_range(), as expected.

We also go from ~51k syscalls to ~39k, with ~2x the number of link() and unlink() in ns/batched-fsync, and of course one fsync() instead of two()>

The flow of this code isn't quite set up for re-plugging the tmp-objdir back in. In particular we no longer pass HASH_N_OBJECTS_FIRST (but doing so would be trivial)< and there's no HASH_N_OBJECTS_LAST.

So this and other callers would need some light transaction-y API, or to otherwise pass down a "yes, I'd like to flush it" down to finalize_hashfile(), but doing so will be trivial.

And since we've started structuring it this way it'll become easy to do any arbitrary number of things down the line that would "bulk fsync" before the final fsync(). Now we write some objects and fsync() on the index, but between those two could do any number of other things where we'd defer the fsync().

This sort of thing might be especially interesting for "git repack" when it writes e.g. a *.bitmap, *.rev, *.pack and *.idx. In that case we could skip the fsync() on all of those, and only do it on the *.idx before we renamed it in-place. I *think* nothing cares about a *.pack without an *.idx, but even then we could fsync *.idx, rename *.pack, rename *.idx and still safely do only one fsync(). See "git show --first-parent" on 62874602032 (Merge branch 'tb/pack-finalize-ordering' into maint, 2021-10-12) for a good overview of the code involved in that.

1. https://lore.kernel.org/git/220323.86sfr9ndpr.gmgdl@evledraar.gmail.com/

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 builtin/update-index.c |  7 ++++---
 cache.h                |  1 +
 read-cache.c           | 29 ++++++++++++++++++++++++++++-
 3 files changed, 33 insertions(+), 4 deletions(-)