Re: [PATCH] refs: sync loose refs to disk before committing them

From: Jeff King <peff@peff.net>
Date: Nov 5, 2021, 07:17 UTC
Message-ID: <YYTaiIlEKxHRVdCy@coredump.intra.peff.net>
In-Reply-To: <YYTYJpyrxtyR8yYZ@coredump.intra.peff.net>

On Fri, Nov 05, 2021 at 03:07:18AM -0400, Jeff King wrote:

Show 16 quoted lines

>   2. It's not clear what the performance implications will be,
>      especially on a busy server doing a lot of ref updates, or on a
>      filesystem where fsync() ends up syncing everything, not just the
>      one file (my impression is ext3 is such a system, but not ext4).
>      Whereas another solution may be journaling data and metadata writes
>      in order without worrying about the durability of writing them to
>      disk.
> 
>      I suspect for small updates (say, a push of one or two refs), this
>      will have little impact. We'd generally fsync the incoming packfile
>      and its idx anyway, so we're adding may one or two fsyncs on top of
>      that. But if you're pushing 100 refs, that will be 100 sequential
>      fsyncs, which may add up to quite a bit of latency. It would be
>      nice if we could batch these by somehow (e.g., by opening up all of
>      the lockfiles, writing and fsyncing them, and then renaming one by
>      one).

So here's a quick experiment that shows a worst case: a small push that updates a bunch of refs. After building Git with and without your patch, I set up a small repo like:

  git init
  git commit --allow-empty -m foo
  for i in $(seq 100); do
    git update-ref refs/heads/$i HEAD
  done

To give a clean slate between runs, I stuck this in a script called "setup":

  #!/bin/sh
  rm -rf dst.git
  git init --bare dst.git
  sync

And then ran:

  $ hyperfine -L v orig,fsync -p ./setup '/tmp/{v}/bin/git push dst.git refs/heads/*'
  Benchmark 1: /tmp/orig/bin/git push dst.git refs/heads/*
    Time (mean ± σ):       9.9 ms ±   0.2 ms    [User: 6.3 ms, System: 4.7 ms]
    Range (min … max):     9.5 ms …  10.5 ms    111 runs
   
  Benchmark 2: /tmp/fsync/bin/git push dst.git refs/heads/*
    Time (mean ± σ):     401.0 ms ±   7.7 ms    [User: 9.4 ms, System: 15.2 ms]
    Range (min … max):   389.4 ms … 412.4 ms    10 runs
   
  Summary
    '/tmp/orig/bin/git push dst.git refs/heads/*' ran
     40.68 ± 1.16 times faster than '/tmp/fsync/bin/git push dst.git refs/heads/*'

So it really does produce a noticeable impact (this is on a system with a decent SSD and no other disk load, so I'd expect it to be about average for modern hardware).

Now this test isn't entirely fair. 100 refs is a larger than average number to be pushing, and the effect is out-sized because there's virtually no time spent dealing with the objects themselves, nor is there any network latency. But 400ms feels like a non-trivial amount of time just in absolute numbers.

The numbers scale pretty linearly, as you'd expect. Pushing 10 refs takes ~40ms, 100 takes ~400ms, and 1000 takes ~4s. The non-fsyncing version gets slower, too (there's more work to do), but much more slowly (6ms, 10ms, and 50ms respectively).

So this will definitely hurt at edge / pathological cases.

-Peff