[PATCH v2 7/8] fetch-pack: ensure fetch sends a promisor-remote reply
- From
Christian Couder <christian.couder@gmail.com>
- Date
- Nov 20, 2026, 16:48 UTC
- Message-ID
- <20261120164812.707946-17-christian.couder@gmail.com>
- In-Reply-To
- <20261120164812.707946-10-christian.couder@gmail.com>
In protocol v2, when the client accepts promisor remotes, the `promisor-remote=...` reply is only sent with the `ls-refs` and `bundle-uri` protocol commands, by get_remote_refs() and get_remote_bundle_uri() in "connect.c".
The request for the `fetch` protocol command is built by send_fetch_request() in "fetch-pack.c" which doesn't send that reply. do_fetch_pack_v2() calls promisor_remote_reply(), but only to mark the accepted remotes, as e15a6b2f8b (promisor-remote: change promisor_remote_reply()'s signature, 2026-02-16) assumed that fetch-pack would not need to send a reply.
With file:// and ssh:// URLs on a Git server, one upload-pack process handles both `ls-refs` and `fetch`. The acceptance recorded during `ls-refs` by promisor_remote_receive() in "serve.c" is still there at `fetch` time, so the clone succeeds. But it fails when each request is handled on its own, for example with HTTP where different upload-pack processes are used.
It also fails when only object IDs are fetched, as no `ls-refs` command is then sent. In these cases, the server doesn't know that the client accepted a promisor remote, so it doesn't allow promisor objects to be missing from the pack it sends, and it fails with "could not fetch <oid> from promisor remote".
Let's fix that by building the list of capabilities for the `fetch` requests once in do_fetch_pack_v2(), using add_server_options() and add_promisor_remote_reply(), and by passing that list to send_fetch_request(). This way the `promisor-remote=...` reply is always properly sent with the `fetch` requests, and the accepted remotes are still marked at the same point as before.
The `fetch` requests sent by negotiate_using_fetch() don't ask for a pack, so they don't need that reply.
Let's also add tests in t5710 to check that the reply is sent with the `fetch` request, and that fetching only an object ID works. For HTTP, let's add a new t5711 test script, copying the setup from t5710.
Reported-by: Matt Cote <mcote@nvidia.com> Signed-off-by: Christian Couder <christian.couder@gmail.com> --- fetch-pack.c | 18 +-- t/meson.build | 1 + t/t5710-promisor-remote-capability.sh | 40 +++++++ t/t5711-promisor-remote-capability-http.sh | 126 +++++++++++++++++++++ 4 files changed, 176 insertions(+), 9 deletions(-) create mode 100755 t/t5711-promisor-remote-capability-http.sh
diff --git a/fetch-pack.c b/fetch-pack.c index 176d625936..3d6d3e33a2 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -1380,16 +1380,15 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out, const struct ref *wants, struct oidset *common, int *haves_to_send, int *in_vain, int sideband_all, int seen_ack, - struct oidset *negotiation_include_oids) + struct oidset *negotiation_include_oids, + const struct strvec *capabilities) { int haves_added; int done_sent = 0; struct strbuf req_buf = STRBUF_INIT; - struct strvec capabilities = STRVEC_INIT; - add_server_options(&capabilities, args->server_options); write_command_and_capabilities(&req_buf, "fetch", - &capabilities, NULL); + capabilities, NULL); if (args->use_thin_pack) packet_buf_write(&req_buf, "thin-pack"); @@ -1454,7 +1453,6 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out, if (write_in_full(fd_out, req_buf.buf, req_buf.len) < 0) die_errno(_("unable to write request to remote")); - strvec_clear(&capabilities); strbuf_release(&req_buf); return done_sent; } @@ -1698,12 +1696,12 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, struct string_list packfile_uris = STRING_LIST_INIT_DUP; int i; struct strvec index_pack_args = STRVEC_INIT; - const char *promisor_remote_config; + struct strvec capabilities = STRVEC_INIT; fsck_options_init(&fsck_options, the_repository, FSCK_OPTIONS_MISSING_GITMODULES); - if (server_feature_v2("promisor-remote", &promisor_remote_config)) - promisor_remote_reply(promisor_remote_config, NULL); + add_server_options(&capabilities, args->server_options); + add_promisor_remote_reply(&capabilities); if (args->filter_options.choice == LOFC_AUTO) { struct strbuf errbuf = STRBUF_INIT; @@ -1781,7 +1779,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, &haves_to_send, &in_vain, reader.use_sideband, seen_ack, - &negotiation_include_oids)) { + &negotiation_include_oids, + &capabilities)) { trace2_region_leave_printf("negotiation_v2", "round", the_repository, "%d", negotiation_round); @@ -1924,6 +1923,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, string_list_clear(&packfile_uris, 0); strvec_clear(&index_pack_args); + strvec_clear(&capabilities); if (fsck_finish(&fsck_options)) die("fsck failed"); diff --git a/t/meson.build b/t/meson.build index 1cf1d95f70..84644703f1 100644 --- a/t/meson.build +++ b/t/meson.build @@ -766,6 +766,7 @@ integration_tests = [ 't5704-protocol-violations.sh', 't5705-session-id-in-capabilities.sh', 't5710-promisor-remote-capability.sh', + 't5711-promisor-remote-capability-http.sh', 't5730-protocol-v2-bundle-uri-file.sh', 't5731-protocol-v2-bundle-uri-git.sh', 't5732-protocol-v2-bundle-uri-http.sh', diff --git a/t/t5710-promisor-remote-capability.sh b/t/t5710-promisor-remote-capability.sh index 463e9e00b0..e5d6f9d1c6 100755 --- a/t/t5710-promisor-remote-capability.sh +++ b/t/t5710-promisor-remote-capability.sh @@ -999,6 +999,46 @@ test_expect_success "clone and fetch with --filter=auto" ' check_missing_objects server 1 "$oid" ' +test_expect_success "clone sends the promisor-remote reply with fetch" ' + git -C server config promisor.advertise true && + test_when_finished "rm -rf client trace" && + + GIT_TRACE_PACKET="$(pwd)/trace" git clone \ + -c remote.lop.promisor=true \ + -c remote.lop.url="$TRASH_DIRECTORY_URL/lop" \ + -c promisor.acceptfromserver=All \ + --no-local --filter="blob:limit=5k" server client && + + # Check that the reply is part of the fetch request + sed -n "/clone> command=fetch/,/clone> 0001/p" trace >request && + test_grep "clone> promisor-remote=lop" request && + + # Check that the largest object is still missing on the server + check_missing_objects server 1 "$oid" +' + +test_expect_success "fetch by object ID sends the promisor-remote reply" ' + git -C server config promisor.advertise true && + test_when_finished "rm -rf client trace" && + + git init client && + git -C client remote add origin "$TRASH_DIRECTORY_URL/server" && + git -C client config remote.lop.promisor true && + git -C client config remote.lop.url "$TRASH_DIRECTORY_URL/lop" && + git -C client config promisor.acceptfromserver All && + + # Fetching only an object ID skips the "ls-refs" command, so the + # reply can only be sent with "fetch" + head=$(git -C server rev-parse HEAD) && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch --no-tags \ + --filter="blob:limit=5k" origin "$head" && + test_grep ! "fetch> command=ls-refs" trace && + test_grep "fetch> promisor-remote=lop" trace && + + # Check that the largest object is still missing on the server + check_missing_objects server 1 "$oid" +' + test_expect_success "clone with promisor.advertise set to 'true' but don't delete the client" ' git -C server config promisor.advertise true && diff --git a/t/t5711-promisor-remote-capability-http.sh b/t/t5711-promisor-remote-capability-http.sh new file mode 100755 index 0000000000..c08cca3e51 --- /dev/null +++ b/t/t5711-promisor-remote-capability-http.sh @@ -0,0 +1,126 @@ +#!/bin/sh + +test_description='handling of promisor remote advertisement over HTTP' + +. ./test-lib.sh + +if ! test_have_prereq PERL_TEST_HELPERS +then + skip_all='skipping promisor remote capabilities tests; Perl not available' + test_done +fi + +. "$TEST_DIRECTORY"/lib-httpd.sh +start_httpd + +GIT_TEST_MULTI_PACK_INDEX=0 +GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0 + +# The setup and helpers below are copied from t5710, except that the +# server repository is in the HTTP document root. + +SERVER="$HTTPD_DOCUMENT_ROOT_PATH/server.git" + +check_missing_objects () { + git -C "$1" rev-list --objects --all --missing=print > all.txt && + perl -ne 'print if s/^[?]//' all.txt >missing.txt && + test_line_count = "$2" missing.txt && + test "$3" = "$(cat missing.txt)" +} + +initialize_server () { + # Repack everything first + git -C "$SERVER" -c repack.writebitmaps=false repack -a -d && + + # Repack without the largest object and create a promisor pack on server + git -C "$SERVER" -c repack.writebitmaps=false repack -a -d \ + --filter=blob:limit=5k --filter-to="$(pwd)/pack" && + promisor_file=$(ls "$SERVER"/objects/pack/*.pack | sed "s/\.pack/.promisor/") && + >"$promisor_file" && + + # Check objects missing on the server + check_missing_objects "$SERVER" 1 "$1" +} + +copy_to_lop () { + oid_path="$(test_oid_to_path $1)" && + path="$SERVER/objects/$oid_path" && + path2="lop/objects/$oid_path" && + mkdir -p $(dirname "$path2") && + cp "$path" "$path2" +} + +# On Windows, `pwd` returns a path like 'D:/foo/bar'. Prepend '/' to turn +# it into '/D:/foo/bar', which is what git expects in file:// URLs on Windows. +# On Unix, the path already starts with '/', so this is a no-op. +pwd_path=$(pwd) +case "$pwd_path" in +[a-zA-Z]:*) pwd_path="/$pwd_path" ;; +esac + +TRASH_DIRECTORY_URL="file://$pwd_path" + +test_expect_success 'setup: create "template" repository' ' + git init template && + test_commit -C template 1 && + test_commit -C template 2 && + test_commit -C template 3 && + test-tool genrandom foo 10k >template/foo && + git -C template add foo && + git -C template commit -m foo +' + +# A bare repo will act as a server repo with unpacked objects. +test_expect_success 'setup: create bare "server" repository' ' + git clone --bare --no-local template "$SERVER" && + mv "$SERVER"/objects/pack/pack-* . && + packfile=$(ls pack-*.pack) && + git -C "$SERVER" unpack-objects --strict <"$packfile" +' + +test_expect_success "setup for testing promisor remote advertisement" ' + # Create another bare repo called "lop" (for Large Object Promisor) + git init --bare lop && + + # Copy the largest object from server to lop + obj="HEAD:foo" && + oid="$(git -C "$SERVER" rev-parse $obj)" && + copy_to_lop "$oid" && + + initialize_server "$oid" && + + # Configure lop as promisor remote for server + git -C "$SERVER" remote add lop "$TRASH_DIRECTORY_URL/lop" && + git -C "$SERVER" config remote.lop.promisor true && + + git -C lop config uploadpack.allowFilter true && + git -C lop config uploadpack.allowAnySHA1InWant true && + git -C "$SERVER" config uploadpack.allowFilter true && + git -C "$SERVER" config uploadpack.allowAnySHA1InWant true && + git -C "$SERVER" config promisor.advertise true +' + +# Over HTTP, each request is handled by a separate upload-pack process, +# so the server knows that the client accepted "lop" only if the reply +# is sent with the "fetch" request itself. +test_expect_success "clone over HTTP with an accepted promisor remote" ' + test_when_finished "rm -rf client trace" && + + GIT_TRACE_PACKET="$(pwd)/trace" git -c protocol.version=2 clone \ + -c remote.lop.promisor=true \ + -c remote.lop.url="$TRASH_DIRECTORY_URL/lop" \ + -c promisor.acceptfromserver=All \ + --filter="blob:limit=5k" "$HTTPD_URL/smart/server.git" client && + + # Check that the reply is part of the fetch request + sed -n "/clone> command=fetch/,/clone> 0001/p" trace >request && + test_grep "clone> promisor-remote=lop" request && + + # Check that the largest object was lazily fetched from lop + test_cmp template/foo client/foo && + + # Check that the largest object is still missing on the server + check_missing_objects "$SERVER" 1 "$oid" +' + +test_done
-- 2.56.0