Re: [PATCH v3 1/2] pack-bitmap.c: remove unnecessary "open_pack_index()" calls

From: Jeff King <peff@peff.net>
Date: Nov 14, 2022, 22:03 UTC
Message-ID: <Y3K7TEpB8EzczjTb@coredump.intra.peff.net>
In-Reply-To: <aaeb021538cdfeb83dc6004fe7b3ac35a23aef49.1668063122.git.dyroneteng@gmail.com>

On Thu, Nov 10, 2022 at 03:10:11PM +0800, Teng Long wrote:

Show 10 quoted lines

> It's likely that we'll end up relying on objects in that pack later
> in the process (in which case we're doing the work of opening the
> pack index optimistically), but not guaranteed.
> 
> For instance, consider a repository with a large number of small
> packs, and one large pack with a bitmap. If we see that bitmap pack
> last in our loop which calls open_pack_bitmap_1(), the current code
> will have opened *all* pack index files in the repository. If the
> request can be served out of the bitmapped pack alone, then the time
> spent opening these idx files was wasted.S

By the way, I wondered if it was possible to measure a slowdown in this case. It is, but you have to try pretty hard. Something like this:

  # one bitmapped pack
  git repack -adb

  # and then a bunch of other packs
  git rev-list HEAD |
  head -10000 |
  while read commit; do
    echo $commit | git pack-objects .git/objects/pack/pack
  done

  # make the bitmapped one newest, since otherwise our non-bitmap lookup
  # of the initial traversal commit causes us to open all the other
  # packs first!
  bitmap=$(echo .git/objects/pack/pack-*.bitmap)
  touch ${bitmap%.bitmap}.*

  hyperfine -L v old,new './git.{v} rev-list --count --use-bitmap-index HEAD'

where "new" and "old" are builds with and without this patch. I get:

  Benchmark 1: ./git.old rev-list --count --use-bitmap-index HEAD
    Time (mean ± σ):     117.9 ms ±   1.8 ms    [User: 26.9 ms, System: 90.0 ms]
    Range (min … max):   113.4 ms … 120.3 ms    25 runs
  
  Benchmark 2: ./git.new rev-list --count --use-bitmap-index HEAD
    Time (mean ± σ):      71.8 ms ±   2.6 ms    [User: 21.2 ms, System: 50.5 ms]
    Range (min … max):    67.0 ms …  75.1 ms    41 runs
  
  Summary
    './git.new rev-list --count --use-bitmap-index HEAD' ran
      1.64 ± 0.06 times faster than './git.old rev-list --count --use-bitmap-index HEAD'

which implies to me two things:

  - this probably isn't helping anybody much in the real world, as
    evidenced by the contortions I had to go through to set up the
    situation (and which would be made much better by repacking, which
    would also speed up non-bitmap operations).

  - it's worth doing anyway. Even if it only shaves off microseconds,
    the existing call is just pointless.

-Peff