[PATCH 0/4] fetch: add --must-have and remote.*.mustHave

Fetch negotiation aims to find enough information from haves and wants such that the server can be reasonably confident that it will send all necessary objects and not too many "extra" objects that the client already has. However, this can break down if there are too many references, since Git truncates the list of haves based on a few factors (a 256 count limit or the server sending an ACK at the right time).

We already have the --negotiation-tip feature to focus the set of references that are used in negotiation, but I feel like this is designed backwards. I'd rather that we have a way to say "this is an important set of refs, but feel free to add more refs if needed" than "only use these refs for negotiation".

Here's an example that demonstrates the problem. In an internal monorepo, developers work off of the 'main' branch so there are thousands of user branches that each add a few commits different from the 'main' branch. However, there is also a long-lived 'release' branch. This branch has a first-parent history that is parallel to 'main' and each of those commits is a merge whose second parent is a commit from 'main' that had a successful CI run. There are additional changes in the 'release' branch merge commits that add some changelog data, so there is a nontrivial set of novel blob content in that branch and not just a different set of commits.

The problem we had was that our georeplication system was regularly fetching from the origin and trying to get all data from all reachable branches. When the 'release' branch updated, the client would run out of haves before advertising its copy of the 'release' branch, but it would still list the new 'release' tip as a want. The server would then think that the client had never fetched that branch before and would send all of the changelog data from the whole history of the repo. (This led to a lot of downstream problems; we mitigated by setting a refspec that stopped fetching the 'release' branch, but this is not ideal.)

What I'd like is a mechanism to say "always advertise the client's version of 'main' and 'release' but also opportunistically include some user branches".

Based on my understanding, the '--negotiation-tip' option is close but not quite what I want. I could have the client only advertise 'release' and 'main' and never advertise any user branches. But then we'd download all content from each user branch every time it updates. Perhaps this would happen even with opportunistic inclusion of more haves, but I'd like to explore this area more.

There's also an issue that the '--negotiation-tip' feature doesn't seem to have a config key that enables it without CLI arguments. This is something that we could consider independently.

This patch series adds a new '--must-have' argument in the same places that '--negotiation-tip' exists. This adds a set of references that will always be included in the negotiation as haves, but then the rest of the negotiation can proceed as normal. It also adds a new 'remote.<name>.mustHave' config option to enable this behavior without CLI arguments.

The series is organized into four changes:

 1. The first change updates a test that started failing after I added some
    tests before it. This fix should be valuable on its own, but I need it
    for my tests to pass.
 2. The second change adds the --must-have option for fetch and tests it
    relative to --negotiation-tip.
 3. The third change adds the config option to enable --must-have by
    default. Any use in the CLI completely overrides the config option.
 4. The fourth change adds capabilities for the config to update negotiation
    during git push.

During development, I had briefly considered only using config values, but that required some strange changes to care about the remote name in the transport layer. This was most different in the 'git push' integration. When I discovered the '--negotiation-tip' feature during the process, that gave me a clear pattern to follow with the addition of a config on top.

I've CC'd some folks who have been working on or near the '--negotiation-tip' feature for feedback as to how these things fit together in the bigger picture.

Big picture questions to think about:

 * Is this a valuable addition to the fetch negotiation?
 * Is the interaction between --must-have and --negotiation-tip correct?
 * Is the "must have" name sensical to users? I expect that this only
   matters to experts, but I'm open to better names that could be more
   self-documenting.
 * Should we add a similar config key for --negotiation-tip?

Thanks, -Stolee

Derrick Stolee (4):
  t5516: fix test order flakiness
  fetch: add --must-have option for negotiation
  remote: add mustHave config as default for --must-have
  send-pack: pass --must-have for push negotiation

 Documentation/config/remote.adoc |  23 ++++++
 Documentation/fetch-options.adoc |  22 ++++++
 builtin/fetch.c                  |  14 +++-
 fetch-pack.c                     |  92 +++++++++++++++++++++--
 fetch-pack.h                     |  10 ++-
 remote.c                         |   6 ++
 remote.h                         |   1 +
 send-pack.c                      |  12 ++-
 send-pack.h                      |   1 +
 t/t5510-fetch.sh                 | 121 +++++++++++++++++++++++++++++++
 t/t5516-fetch-push.sh            |  17 ++++-
 transport.c                      |   5 +-
 transport.h                      |   6 ++
 13 files changed, 319 insertions(+), 11 deletions(-)

base-commit: 6e8d538aab8fe4dd07ba9fb87b5c7edcfa5706ad
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2085%2Fderrickstolee%2Fmust-have-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2085/derrickstolee/must-have-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/2085