From: Harald Nordgren via GitGitGadget Date: Fri, 25 Sep 2026 10:49:58 GMT Subject: [PATCH v3 0/4] fetch: avoid fetching every branch of a new remote in a shallow repo Message-ID: In-Reply-To: Avoid fetching every branch of a new remote in a shallow repo. Changes in v3: * Replace the special ":"/"+:" fetch refspec with remote..refmap, reusing git's existing --refmap mechanism instead of inventing new refspec syntax. * Split the change into 4 commits. Changes in v2: * Replaced the opt-in fetch.shallow config entirely with a new special fetch refspec (+:) that git remote add now defaults new remotes to in a shallow repository. The new refspec fetches whichever branches any local branch tracks at that remote, plus the remote's default branch. Harald Nordgren (4): fetch: add remote..refmap fetch: infer branches to fetch from a refmap-only remote remote: add "git remote add --limited-fetch" remote: default to --limited-fetch in a shallow repository Documentation/config/remote.adoc | 7 ++ Documentation/fetch-options.adoc | 7 ++ Documentation/git-remote.adoc | 14 ++- builtin/fetch.c | 142 +++++++++++++++++++++++++------ builtin/remote.c | 31 +++++-- remote.c | 41 ++++++++- remote.h | 9 ++ t/meson.build | 1 + t/t5505-remote.sh | 75 ++++++++++++++++ t/t5510-fetch.sh | 17 ++++ t/t5585-fetch-refmap.sh | 124 +++++++++++++++++++++++++++ 11 files changed, 433 insertions(+), 35 deletions(-) create mode 100755 t/t5585-fetch-refmap.sh base-commit: 0f8e75abebff0877cae681a3d5ff31ac47f54220 Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-2412%2FHaraldNordgren%2Ffetch-shallow-narrow-refspec-v3 Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-2412/HaraldNordgren/fetch-shallow-narrow-refspec-v3 Pull-Request: https://github.com/git/git/pull/2412 Range-diff vs v2: -: ---------- > 1: b04c00b974 fetch: add remote..refmap 1: f04f5ce219 ! 2: 4ec508a223 fetch: avoid fetching every branch of a new remote in a shallow repo @@ Metadata Author: Harald Nordgren ## Commit message ## - fetch: avoid fetching every branch of a new remote in a shallow repo + fetch: infer branches to fetch from a refmap-only remote - git remote add sets a new remote up to fetch every branch by default. - In an already shallow repository, that turns the next plain fetch or - pull into a slow or hanging one, even when only one or two branches - are ever used. + Configuring remote..refmap without remote..fetch used to + make a refspec-less "git fetch " fail with "--refmap option is + only meaningful with command-line refspec(s)", since a refmap only + says where to put fetched refs, not what to fetch. - Add a special fetch refspec, "+:", that fetches only the branches our - local branches are built on, plus the remote's default branch so a - brand new remote is usable right away, without needing to first set - anything up to track it. git remote add now uses it instead of the - usual wildcard refspec whenever the repository is already shallow. + Make that case infer what to fetch: the local branches whose + @{upstream} is already on that remote, plus the remote's default + branch, which is always included so it is available even before + anything is set up to track it. This lets a remote be configured to + fetch only the branches actually in use, without listing them by + hand in remote..fetch, and without needing to touch the + command line every time. Signed-off-by: Harald Nordgren - ## Documentation/git-remote.adoc ## -@@ Documentation/git-remote.adoc: Add a remote named __ for the repository at - __. The command `git fetch ` can then be used to create and - update remote-tracking branches `/`. - + -+If the repository is already a shallow repository (see linkgit:git-clone[1] -+`--depth`) and neither `-t` nor `--mirror` is given, the remote is set up -+with the special tracking refspec (see the __ entry in -+linkgit:git-fetch[1]) instead of the default glob refspec, so that -+`git fetch ` does not need to negotiate history for every branch -+the remote has. -++ - With `-f` option, `git fetch ` is run immediately after - the remote information is set up. - + + ## Documentation/config/remote.adoc ## +@@ Documentation/config/remote.adoc: remote..fetch:: + + remote..refmap:: + The default value of the `--refmap` option for linkgit:git-fetch[1]. +- Only takes effect when the fetch names what to fetch explicitly, +- either on the command line or via `remote..fetch`. See the +- `--refmap` entry in linkgit:git-fetch[1]. ++ If `remote..fetch` is not set either, a refspec-less fetch ++ infers what to fetch from local branches built on this remote, ++ instead of fetching every branch it has. See the `--refmap` entry ++ in linkgit:git-fetch[1]. + + remote..push:: + The default set of "refspec" for linkgit:git-push[1]. See - ## Documentation/pull-fetch-param.adoc ## -@@ Documentation/pull-fetch-param.adoc: Negative refspecs can themselves be pattern refspecs. However, they may only - contain a __ and do not specify a __. Fully spelled out hex object - names are also not supported. - + -+The special refspec `:` (or `+:` to allow non-fast-forward updates) -+directs Git to fetch whichever branches at the remote our local branches -+have as their `@{upstream}`, plus the remote's default branch, which is -+always included so that it is available even before anything is set up -+to track it. Unlike the same syntax when pushing, the local branch name -+plays no role here beyond identifying which remote branch to fetch: the -+result is written to `refs/remotes//` using the remote's -+own branch name, and several local branches built on the same remote -+branch only cause it to be fetched once. -++ - `tag ` means the same as `refs/tags/:refs/tags/`; - it requests fetching everything up to the given tag. + ## Documentation/fetch-options.adoc ## +@@ Documentation/fetch-options.adoc: endif::git-pull[] + command-line arguments. See section on "Configured Remote-tracking + Branches" for details. + +-`remote..refmap` provides the default value for this option, the +-same way `remote..fetch` provides the default refspecs to fetch. ++When a refmap is active (from `--refmap` or `remote..refmap`) but ++there is nothing to fetch, neither on the command line nor from ++`remote..fetch`, Git infers what to fetch from the local branches ++whose `@{upstream}` is on that remote, plus the remote's default branch, ++which is always included so that it is available even before anything ++is set up to track it. + + `-t`:: + `--tags`:: ## builtin/fetch.c ## @@ builtin/fetch.c: static void filter_prefetch_refspec(struct refspec *rs) - } - } - -+static void fetch_tracked_branches(struct remote *remote, -+ const struct ref *remote_refs, -+ struct ref ***tail, -+ char **default_branch_out, -+ int force) -+{ -+ struct string_list tracked = STRING_LIST_INIT_DUP; -+ struct string_list_item *item; -+ struct ref *head; -+ char *default_branch = NULL; -+ -+ branches_tracking_remote(remote, &tracked); -+ -+ head = get_remote_ref(remote_refs, "HEAD"); -+ if (head && head->symref && *head->symref) { -+ default_branch = xstrdup(head->symref); -+ string_list_insert(&tracked, default_branch); -+ } -+ free_one_ref(head); -+ -+ for_each_string_list_item(item, &tracked) { -+ struct refspec_item ref = { .force = force }; -+ const char *branch_name = item->string; -+ int is_default_branch = default_branch && -+ !strcmp(item->string, default_branch); -+ -+ skip_prefix(branch_name, "refs/heads/", &branch_name); -+ ref.src = xstrdup(is_default_branch ? "HEAD" : item->string); -+ ref.dst = xstrfmt("refs/remotes/%s/%s", remote->name, branch_name); -+ get_fetch_map(remote_refs, &ref, tail, 1); -+ free(ref.src); -+ free(ref.dst); -+ } -+ -+ if (default_branch && default_branch_out) { -+ const char *branch_name = default_branch; -+ skip_prefix(branch_name, "refs/heads/", &branch_name); -+ *default_branch_out = xstrdup(branch_name); -+ } -+ -+ free(default_branch); -+ string_list_clear(&tracked, 0); -+} -+ static struct ref *get_ref_map(struct remote *remote, const struct ref *remote_refs, struct refspec *rs, @@ builtin/fetch.c: static void filter_prefetch_refspec(struct refspec *rs) int i; struct ref *rm; @@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote, - /* Note: has_merge implies non-NULL branch->remote_name */ - (has_merge && !strcmp(branch->remote_name, remote->name)))) { - for (i = 0; i < remote->fetch.nr; i++) { -+ if (remote->fetch.items[i].tracking) { -+ fetch_tracked_branches(remote, remote_refs, &tail, -+ bootstrap_head_branch, -+ remote->fetch.items[i].force); -+ continue; -+ } - get_fetch_map(remote_refs, &remote->fetch.items[i], &tail, 0); - if (remote->fetch.items[i].dst && - remote->fetch.items[i].dst[0]) + struct ref **tail = &ref_map; + struct refspec *effective_refmap = + refmap.nr ? &refmap : remote ? &remote->refmap : NULL; ++ struct refspec inferred_rs; ++ int inferred_branches = 0; + + /* opportunistically-updated references: */ + struct ref *orefs = NULL, **oref_tail = &orefs; +@@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote, + struct hashmap existing_refs; + int existing_refs_populated = 0; + ++ refspec_init_fetch(&inferred_rs, the_hash_algo); ++ + filter_prefetch_refspec(rs); + if (remote) + filter_prefetch_refspec(&remote->fetch); + ++ if (!rs->nr && remote && !remote->fetch.nr && ++ effective_refmap && effective_refmap->nr) { ++ struct string_list tracked = STRING_LIST_INIT_DUP; ++ struct string_list_item *item; ++ struct ref *head; ++ char *default_branch = NULL; ++ char *default_branch_dst = NULL; ++ ++ branches_tracking_remote(remote, &tracked); ++ for_each_string_list_item(item, &tracked) ++ refspec_append(&inferred_rs, item->string); ++ ++ /* Always fetch the default branch too, as "HEAD". */ ++ head = get_remote_ref(remote_refs, "HEAD"); ++ if (head && head->symref && *head->symref) ++ default_branch = xstrdup(head->symref); ++ free_one_ref(head); ++ ++ for (i = 0; default_branch && !default_branch_dst && ++ !string_list_has_string(&tracked, default_branch) && ++ i < effective_refmap->nr; i++) { ++ struct refspec_item *map = &effective_refmap->items[i]; ++ ++ if (map->pattern) ++ match_refname_with_pattern(map->src, default_branch, ++ map->dst, &default_branch_dst); ++ else if (!strcmp(map->src, default_branch)) ++ default_branch_dst = xstrdup(map->dst); ++ } ++ ++ if (default_branch_dst) { ++ struct refspec_item head_item = { .force = 1 }; ++ ++ head_item.src = xstrdup("HEAD"); ++ head_item.dst = default_branch_dst; ++ get_fetch_map(remote_refs, &head_item, &tail, 1); ++ free(head_item.src); ++ } ++ ++ if (default_branch && bootstrap_head_branch) { ++ const char *branch_name = default_branch; ++ skip_prefix(branch_name, "refs/heads/", &branch_name); ++ *bootstrap_head_branch = xstrdup(branch_name); ++ } ++ ++ free(default_branch); ++ free(default_branch_dst); ++ string_list_clear(&tracked, 0); ++ ++ rs = &inferred_rs; ++ inferred_branches = 1; ++ } ++ + if (rs->nr) { + struct refspec *fetch_refspec; + + for (i = 0; i < rs->nr; i++) { +- get_fetch_map(remote_refs, &rs->items[i], &tail, 0); ++ get_fetch_map(remote_refs, &rs->items[i], &tail, ++ inferred_branches); + if (rs->items[i].dst && rs->items[i].dst[0]) + *autotags = 1; + } +@@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote, + + for (i = 0; i < fetch_refspec->nr; i++) + get_fetch_map(ref_map, &fetch_refspec->items[i], &oref_tail, 1); ++ } else if (inferred_branches) { ++ /* Already fully handled above. */ + } else if (effective_refmap && effective_refmap->nr) { + die("--refmap option is only meaningful with command-line refspec(s)"); + } else { +@@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote, + if (existing_refs_populated) + hashmap_clear_and_free(&existing_refs, struct refname_hash_entry, ent); + ++ refspec_clear(&inferred_rs); + return ref_map; + } + @@ builtin/fetch.c: static void warn_set_head(const char *remote, const char *head_name, } @@ builtin/fetch.c: static int do_fetch(struct transport *transport, refspec_ref_prefixes(rs, &transport_ls_refs_options.ref_prefixes); } else { struct branch *branch = branch_get(NULL); +- +- if (transport->remote->fetch.nr) { + int tracks_this_remote = branch && branch_has_merge_config(branch) && + !strcmp(branch->remote_name, transport->remote->name); -+ int tracking_refspec = transport->remote->fetch.nr == 1 && -+ transport->remote->fetch.items[0].tracking; - -- if (transport->remote->fetch.nr) { -+ if (tracking_refspec) { ++ struct refspec *effective_refmap = refmap.nr ? &refmap : ++ &transport->remote->refmap; ++ int inferred_branches = !transport->remote->fetch.nr && ++ effective_refmap->nr; ++ ++ if (inferred_branches) { + struct string_list tracked = STRING_LIST_INIT_DUP; + struct string_list_item *item; + @@ builtin/fetch.c: static int do_fetch(struct transport *transport, } - if (branch && branch_has_merge_config(branch) && - !strcmp(branch->remote_name, transport->remote->name)) { -+ if (transport->remote->fetch.nr && ++ if ((transport->remote->fetch.nr || inferred_branches) && + follow_remote_head != FOLLOW_REMOTE_NEVER) + do_set_head = 1; + if (tracks_this_remote) { @@ builtin/fetch.c: static int do_fetch(struct transport *transport, * When using batched updates, we want to commit the non-rejected * updates and also handle the rejections. - ## builtin/remote.c ## -@@ - #include "rebase.h" - #include "refs.h" - #include "refspec.h" -+#include "shallow.h" - #include "odb.h" - #include "strvec.h" - #include "commit-reach.h" -@@ builtin/remote.c: static void add_branch(const char *key, const char *branchname, - const char *remotename, int mirror, struct strbuf *tmp) - { - strbuf_reset(tmp); -- strbuf_addch(tmp, '+'); -- if (mirror) -- strbuf_addf(tmp, "refs/%s:refs/%s", -- branchname, branchname); -- else -- strbuf_addf(tmp, "refs/heads/%s:refs/remotes/%s/%s", -- branchname, remotename, branchname); -+ if (!strcmp(branchname, ":")) { -+ strbuf_addstr(tmp, "+:"); -+ } else { -+ strbuf_addch(tmp, '+'); -+ if (mirror) -+ strbuf_addf(tmp, "refs/%s:refs/%s", -+ branchname, branchname); -+ else -+ strbuf_addf(tmp, "refs/heads/%s:refs/remotes/%s/%s", -+ branchname, remotename, branchname); -+ } - repo_config_set_multivar(the_repository, key, tmp->buf, "^$", 0); - } - -@@ builtin/remote.c: static int add(int argc, const char **argv, const char *prefix, - strbuf_reset(&buf); - strbuf_addf(&buf, "remote.%s.fetch", name); - if (track.nr == 0) -- string_list_append(&track, "*"); -- for (size_t i = 0; i < track.nr; i++) { -+ string_list_append(&track, -+ mirror == MIRROR_NONE && -+ is_repository_shallow(the_repository) ? -+ ":" : "*"); -+ for (size_t i = 0; i < track.nr; i++) - add_branch(buf.buf, track.items[i].string, - name, mirror, &buf2); -- } - } - - if (mirror & MIRROR_PUSH) { - - ## refspec.c ## -@@ refspec.c: static int parse_refspec(struct refspec_item *item, const char *refspec, - return 1; - } - -+ if (fetch && rhs == lhs && rhs[1] == '\0') { -+ item->tracking = 1; -+ return 1; -+ } -+ - if (rhs) { - size_t rlen = strlen(++rhs); - is_glob = (1 <= rlen && strchr(rhs, '*')); -@@ refspec.c: void refspec_item_clear(struct refspec_item *item) - item->force = 0; - item->pattern = 0; - item->matching = 0; -+ item->tracking = 0; - item->exact_sha1 = 0; - } - -@@ refspec.c: static int refspec_find_negative_match(struct refspec *rs, struct refspec_item * - struct refspec_item *refspec = &rs->items[i]; - char *expn_name; - -- if (refspec->negative) -+ if (refspec->negative || refspec->tracking) - continue; - - /* Note the reversal of src and dst */ - - ## refspec.h ## -@@ refspec.h: struct refspec_item { - unsigned matching : 1; - unsigned exact_sha1 : 1; - unsigned negative : 1; -+ unsigned tracking : 1; - - char *src; - char *dst; - ## remote.c ## -@@ remote.c: void ref_push_report_free(struct ref_push_report *report) - - int remote_find_tracking(struct remote *remote, struct refspec_item *refspec) - { -+ if (remote->fetch.nr == 1 && remote->fetch.items[0].tracking) { -+ struct strbuf prefix = STRBUF_INIT; -+ const char *branch_name; -+ int ret = -1; -+ -+ strbuf_addf(&prefix, "refs/remotes/%s/", remote->name); -+ if (!refspec->src) { -+ if (refspec->dst && -+ skip_prefix(refspec->dst, prefix.buf, &branch_name)) { -+ refspec->src = xstrfmt("refs/heads/%s", branch_name); -+ refspec->force = 1; -+ ret = 0; -+ } -+ } else if (skip_prefix(refspec->src, "refs/heads/", &branch_name)) { -+ free(refspec->dst); -+ refspec->dst = xstrfmt("%s%s", prefix.buf, branch_name); -+ refspec->force = 1; -+ ret = 0; -+ } -+ strbuf_release(&prefix); -+ return ret; -+ } -+ - return refspec_find_match(&remote->fetch, refspec); - } - @@ remote.c: int branch_merge_matches(struct branch *branch, return refname_match(branch->merge[i]->src, refname); } @@ remote.c: int branch_merge_matches(struct branch *branch, __attribute__((format (printf,2,3))) static char *error_buf(struct strbuf *err, const char *fmt, ...) { -@@ remote.c: int get_fetch_map(const struct ref *remote_refs, - { - struct ref *ref_map, **rmp; - -- if (refspec->negative) -+ if (refspec->negative || refspec->tracking) - return 0; - - if (refspec->pattern) { ## remote.h ## @@ remote.h: int branch_has_merge_config(struct branch *branch); @@ t/meson.build: integration_tests = [ 't5582-fetch-negative-refspec.sh', 't5583-push-branches.sh', 't5584-http-429-retry.sh', -+ 't5585-fetch-tracking-refspec.sh', ++ 't5585-fetch-refmap.sh', 't5600-clone-fail-cleanup.sh', 't5601-clone.sh', 't5602-clone-remote-exec.sh', - ## t/t5505-remote.sh ## -@@ t/t5505-remote.sh: test_expect_success 'filters are listed by git remote -v only' ' - test_grep ! "\[blob:none\]" out - ' - -+test_expect_success 'add remote -t keeps an explicit refspec in a shallow repository' ' -+ test_when_finished "rm -rf shallow-add" && -+ git clone --no-local --depth=1 --branch main --single-branch \ -+ one shallow-add && -+ ( -+ cd shallow-add && -+ git remote add -t main upstream ../two && -+ test_cmp_config "+refs/heads/main:refs/remotes/upstream/main" \ -+ remote.upstream.fetch -+ ) -+' -+ -+test_expect_success 'add remote keeps the wildcard refspec in a full repository' ' -+ test_when_finished "rm -rf full-add" && -+ git clone --no-local one full-add && -+ ( -+ cd full-add && -+ git remote add upstream ../two && -+ test_cmp_config "+refs/heads/*:refs/remotes/upstream/*" \ -+ remote.upstream.fetch -+ ) -+' -+ -+test_expect_success 'a remote added in a shallow repository defaults to the tracking refspec' ' -+ test_when_finished "rm -rf shallow-add" && -+ git clone --no-local --depth=1 --branch main --single-branch \ -+ one shallow-add && -+ ( -+ cd shallow-add && -+ git remote add upstream ../two && -+ test_cmp_config "+:" remote.upstream.fetch && -+ git fetch upstream && -+ git for-each-ref --format="%(refname)" refs/remotes/upstream >actual && -+ cat >expect <<-\EOF && -+ refs/remotes/upstream/HEAD -+ refs/remotes/upstream/main -+ EOF -+ test_cmp expect actual && -+ git branch --set-upstream-to=upstream && -+ test_cmp_config upstream branch.main.remote && -+ test_cmp_config refs/heads/main branch.main.merge -+ ) -+' -+ - test_expect_success 'check remote-tracking' ' - ( - cd test && - - ## t/t5585-fetch-tracking-refspec.sh (new) ## + ## t/t5585-fetch-refmap.sh (new) ## @@ +#!/bin/sh + -+test_description='"git fetch" with the tracking refspec ":" (or "+:") ++test_description='"git fetch" with a remote..refmap but no remote..fetch + -+The special refspec fetches whichever branches at the remote our local -+branches have as their @{upstream}, plus the default branch of that -+remote, which is always included so that it is available even before -+anything is set up to track it. git remote add uses it by default for -+a new remote in an already shallow repository, so that adding a -+second remote there does not turn the next plain fetch or pull into -+one that negotiates history for every branch the remote has. ++When a remote has a refmap configured but no fetch refspec, a ++refspec-less fetch infers what to fetch from the local branches whose ++@{upstream} is on that remote, plus the default branch of that remote, ++which is always included so that it is available even before anything ++is set up to track it. +' + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main @@ t/t5585-fetch-tracking-refspec.sh (new) + ( + cd client && + git remote add upstream .. && -+ test_cmp_config "+:" remote.upstream.fetch ++ test_might_fail git config unset remote.upstream.fetch && ++ git config remote.upstream.refmap \ ++ "+refs/heads/*:refs/remotes/upstream/*" + ) +' + @@ t/t5585-fetch-tracking-refspec.sh (new) + ) +' + -+test_expect_success 'git remote show does not choke on the tracking refspec' ' ++test_expect_success 'git remote show does not choke on a refmap-only remote' ' + ( + cd client && + git remote show upstream -: ---------- > 3: 604e584956 remote: add "git remote add --limited-fetch" -: ---------- > 4: 31462e9445 remote: default to --limited-fetch in a shallow repository -- gitgitgadget