[PATCH 0/2] Fix background maintenance regression in Git 2.45.0

Here is an issue I noticed while exploring issues with my local copy of a large monorepo. I was intending to show some engineers how nice the objects were maintained by background maintenance, but saw hundreds of small pack-files that were up to two months old. This time matched when I upgraded to the microsoft/git fork that included the 2.45.0 release of Git.

The issue is that 'git multi-pack-index repack' was taught to call 'git pack-objects' with the new '--stdin-packs' option. However, this changes the object selection algorithm. Instead of using the objects referenced by the multi-pack-index, it compares pack-files using a list of "included" and "excluded" pack-files. This loses some granularity of how the multi-pack-index chooses among duplicate objects.

The end result is that some objects that would normally have been included in the new pack-file are no longer included. The copy that the multi-pack-index references is in the pack-file that was intended to be repacked, so that pack-file cannot be expired in the next 'git multi-pack-index expire' step and is included again in the batch of objects to repack.

In the context of the change that is reverted by this series, it seems the motivation of the change was two-fold:

 1. some I/O benefits to using pack names over object names, and
 2. the ability to use an object walk to improve delta compression.

In my local prototyping, I've found that we could improve 'git pack-objects' to use an object walk when given a set of objects over stdin without needing to use pack-file names. I do not believe the '--stdin-packs' option should be used for the 'git multi-pack-index repack' mechanism, or at least should be done with great care and only in specific cases where some assumptions can be made around duplicate objects and closure under reachability.

However, the prototype I've built to get these benefits is non-trivial due to working to guarantee that partial clones do not accidentally download missing blobs. That will follow in a separate series that can be reviewed at a slower pace.

Thanks, -Stolee

Derrick Stolee (2):
  t5319: add failing test case for repack/expire
  midx-write: revert use of --stdin-packs

 midx-write.c                | 18 ++++++------
 t/t5319-multi-pack-index.sh | 55 +++++++++++++++++++++++++++++++++++++
 2 files changed, 64 insertions(+), 9 deletions(-)

base-commit: bea9ecd24b0c3bf06cab4a851694fe09e7e51408
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1764%2Fderrickstolee%2Fincremental-repack-fix-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1764/derrickstolee/incremental-repack-fix-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1764