From: Harald Nordgren via GitGitGadget Date: Tue, 27 Oct 2026 12:24:39 GMT Subject: [PATCH v12 0/5] fetch: avoid fetching every branch of a new remote in a shallow repo Message-ID: In-Reply-To: Avoid fetching every branch of a new remote in a shallow repo. Changes in v12: * Renamed remote..refmap to remote..fetchRefmap throughout code, docs, and tests, so it reads unambiguously as fetch-only. * Added the missing code path in get_ref_map() so a branch fetched by name via remote..fetch is also stored through remote..fetchRefmap, with a new test (the "cabal" example: fetch = master/next/seen plus a refmap storing them all under one namespace). * Reworded the fetchRefmap commit message and doc entry to drop a redundant sentence, fix a confusing paragraph, and spell out that remote..fetch, when configured, still decides what's fetched while the refmap only remaps it. * Added a prefetch coverage test to t5586-fetch-refmap.sh. Changes in v11: * remote_find_tracking() now falls back to remote..refmap when remote..fetch doesn't match. Negative refspecs in fetch still exclude refs, and both cases have tests * The helpers write straight into a strvec, take the output parameter last and the remote as const * The followRemoteHEAD lookup is done once, not duplicated in two branches * Clearer commit messages and docs: a working example plus a reference to 49ca2fba39, the goal stated first, an incorrect "used to die" claim dropped, and an explanation of why a refmap turns on the inference * use_limited_fetch is a bool Changes in v10: * Resolve merge conflict after master integrated ch/fetch-followremotehead-lazy-validation. Changes in v9: * Moved the do_fetch() if/else-if/else split out of the preliminary helper-extraction commit and into the refmap-inference commit that needs it. Preliminary commit is now a simpler refactor. Changes in v8: * Split into a preliminary collect_upstream_from_remote() helper commit plus a simplified refmap-inference commit. * Fixed a regression that could drop the current branch's @{upstream} when remote..fetch was narrowed. * Threaded struct repository * through instead of hardcoding the_repository. * Minor naming and comment cleanups. Changes in v7: * Fixed fatal: --refmap option is only meaningful with command-line refspec(s) firing just from configuring remote..refmap alone, with nothing else to fetch. * When both remote..fetch and remote..refmap are configured, a refspec-less fetch follows .fetch as usual, .refmap only remaps something already being fetched by name. Added tests. Changes in v6: * Remove leftover reference to deleted default-branch logic in commit message. Changes in v5: * Renumber test file from t5585 to t5586. Changes in v4: * Removed the automatic default-branch fetch. A fresh remote fetches nothing until you track a branch explicitly. * Fixed fetch report showing "new ref HEAD" instead of "new branch " * Reworded remote..refmap docs and commit message. Changes in v3: * Replace the special ":"/"+:" fetch refspec with remote..refmap, reusing git's existing --refmap mechanism instead of inventing new refspec syntax. * Split the change into 4 commits. Changes in v2: * Replaced the opt-in fetch.shallow config entirely with a new special fetch refspec (+:) that git remote add now defaults new remotes to in a shallow repository. The new refspec fetches whichever branches any local branch tracks at that remote, plus the remote's default branch. Harald Nordgren (5): fetch: add remote..fetchRefmap fetch: extract collect_upstream_from_remote() helper fetch: infer branches to fetch from a refmap-only remote remote: add "git remote add --limited-fetch" remote: default to --limited-fetch in a shallow repository Documentation/config/remote.adoc | 9 +++ Documentation/fetch-options.adoc | 10 +++ Documentation/git-remote.adoc | 14 +++- builtin/fetch.c | 100 +++++++++++++++++------ builtin/remote.c | 31 +++++-- remote.c | 60 +++++++++++++- remote.h | 20 +++++ t/meson.build | 1 + t/t5505-remote.sh | 76 ++++++++++++++++++ t/t5510-fetch.sh | 130 ++++++++++++++++++++++++++++++ t/t5586-fetch-refmap.sh | 133 +++++++++++++++++++++++++++++++ 11 files changed, 551 insertions(+), 33 deletions(-) create mode 100755 t/t5586-fetch-refmap.sh base-commit: 37b3475fb530c14109211faca33716a173db61ed Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-2412%2FHaraldNordgren%2Ffetch-shallow-narrow-refspec-v12 Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-2412/HaraldNordgren/fetch-shallow-narrow-refspec-v12 Pull-Request: https://github.com/git/git/pull/2412 Range-diff vs v11: 1: d66b3c0bb1 ! 1: a1f5e81fdc fetch: add remote..refmap @@ Metadata Author: Harald Nordgren ## Commit message ## - fetch: add remote..refmap + fetch: add remote..fetchRefmap - Add a per-remote config variable, remote..refmap, that provides - the default value for --refmap the same way remote..fetch - already provides the default refspecs to fetch. Like --refmap itself, - it only maps refs that are actually being fetched, so it has nothing - to do when there is nothing explicit to fetch, on the command line or - via remote..fetch. + Add a per-remote config variable, remote..fetchRefmap, that + provides the default value for --refmap the same way + remote..fetch already provides the default refspecs to fetch. - A refspec-less fetch with a refmap configured but nothing else to say - what to fetch falls back to the same defaults as a remote with no - refspec or refmap at all. Only an explicit --refmap given on the - command line with no command-line refspec to go with it is still - rejected, since that combination is a plain mistake to type, unlike a - remote..refmap configured on its own. A remote..fetch - that is also configured is used as before. + Unlike passing --refmap on the command line, which requires an + explicit refspec to go with it, configuring remote..fetchRefmap + on its own does not change what a refspec-less fetch does: it falls + back to the same defaults as a remote with no fetchRefmap at all. + + When remote..fetch is also configured, it still decides what to + fetch, and the refmap maps what it fetches. This lets a remote name + only a few branches and store them all through one refmap: + + [remote "cabal"] + fetch = master + fetch = next + fetch = seen + fetchRefmap = +refs/heads/*:refs/remotes/cabal/* + + Looking up the remote-tracking branch for a remote branch, as + @{upstream} and "git branch --track" do, falls back to the refmap when + remote..fetch has no match, so that these work for such + branches too. A branch excluded by a negative refspec in + remote..fetch stays excluded. Signed-off-by: Harald Nordgren @@ Documentation/config/remote.adoc: remote..fetch:: The default set of "refspec" for linkgit:git-fetch[1]. See linkgit:git-fetch[1]. -+remote..refmap:: -+ The default value of the `--refmap` option for linkgit:git-fetch[1]. -+ Used to map remote refs being fetched to remote-tracking refs to -+ store. See the `--refmap` entry in linkgit:git-fetch[1]. ++remote..fetchRefmap:: ++ The default value of the `--refmap` option, used to map refs being ++ fetched to remote-tracking refs to store, the same way ++ `remote..fetch` does for an explicit refspec. ++ `remote..fetch`, if also configured, still decides what is ++ fetched. See the `--refmap` entry in linkgit:git-fetch[1]. + remote..push:: The default set of "refspec" for linkgit:git-push[1]. See @@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote, struct ref **tail = &ref_map; + /* + * The --refmap command line option, if given, takes precedence -+ * over remote..refmap. ++ * over remote..fetchRefmap. + */ + struct refspec *effective_refmap = + refmap.nr ? &refmap : remote ? &remote->refmap : NULL; /* opportunistically-updated references: */ struct ref *orefs = NULL, **oref_tail = &orefs; +@@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote, + filter_prefetch_refspec(rs); + if (remote) + filter_prefetch_refspec(&remote->fetch); ++ if (effective_refmap) ++ filter_prefetch_refspec(effective_refmap); + + if (rs->nr) { + struct refspec *fetch_refspec; @@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote, * by ref_remove_duplicates() in favor of one of these * opportunistic entries with FETCH_HEAD_IGNORE. @@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote, else fetch_refspec = &remote->fetch; +@@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote, + if (has_merge && + !strcmp(branch->remote_name, remote->name)) + add_merge_config(&ref_map, remote_refs, branch, &tail); ++ ++ for (i = 0; i < remote->refmap.nr; i++) ++ get_fetch_map(ref_map, &remote->refmap.items[i], ++ &oref_tail, 1); + } else if (!prefetch) { + ref_map = get_remote_ref(remote_refs, "HEAD"); + if (!ref_map) ## remote.c ## @@ remote.c: static struct remote *make_remote(struct remote_state *remote_state, @@ remote.c: static int handle_config(const char *key, const char *value, return -1; refspec_append(&remote->fetch, v); free(v); -+ } else if (!strcmp(subkey, "refmap")) { ++ } else if (!strcmp(subkey, "fetchrefmap")) { + char *v; + if (git_config_string(&v, key, value)) + return -1; @@ t/t5510-fetch.sh: test_expect_success 'explicit --refmap option overrides remote ) ' -+test_expect_success 'remote.*.refmap acts like --refmap on the command line' ' -+ test_when_finished "git -C three config --unset remote.origin.refmap" && ++test_expect_success 'remote.*.fetchRefmap acts like --refmap on the command line' ' ++ test_when_finished "git -C three config --unset remote.origin.fetchRefmap" && + git branch -f side && -+ git -C three config remote.origin.refmap \ ++ git -C three config remote.origin.fetchRefmap \ + "refs/heads/*:refs/remotes/other/*" && + ( + cd three && @@ t/t5510-fetch.sh: test_expect_success 'explicit --refmap option overrides remote + test_cmp expect actual +} + -+test_expect_success 'remote..refmap without tracking (baseline)' ' ++test_expect_success 'remote..fetchRefmap without tracking (baseline)' ' + test_when_finished "rm -fr fetch-refmap-baseline fetch-refmap-upstream" && + git init -b main fetch-refmap-upstream && + test_commit -C fetch-refmap-upstream base && @@ t/t5510-fetch.sh: test_expect_success 'explicit --refmap option overrides remote + + # Without fetch refspec, but with fetch refmap. + git config --unset-all remote.origin.fetch && -+ git config remote.origin.refmap "+refs/heads/*:refs/remotes/origin/*" && ++ git config remote.origin.fetchRefmap "+refs/heads/*:refs/remotes/origin/*" && + + # Nothing tracked, nothing fetched, no error. + git fetch origin && @@ t/t5510-fetch.sh: test_expect_success 'explicit --refmap option overrides remote + + # With both refmap and fetch configured, remote..fetch + # wins: a refspec-less fetch follows it as usual, and -+ # remote..refmap plays no part in deciding what to -+ # fetch, only in remapping something that is already being -+ # fetched by name. ++ # remote..fetchRefmap plays no part in deciding what to ++ # fetch, only in mapping what is fetched. + git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*" && + git fetch origin && + check_fetched_refs <<-\EOF @@ t/t5510-fetch.sh: test_expect_success 'explicit --refmap option overrides remote + ) +' + -+test_expect_success 'remote..refmap tracks a branch outside remote..fetch' ' ++test_expect_success 'remote..fetchRefmap stores branches named in remote..fetch' ' ++ test_when_finished "rm -fr fetch-refmap-upstream downstream" && ++ git init -b master fetch-refmap-upstream && ++ test_commit -C fetch-refmap-upstream base && ++ git -C fetch-refmap-upstream branch next && ++ git -C fetch-refmap-upstream branch seen && ++ git -C fetch-refmap-upstream branch other && ++ git init downstream && ++ ( ++ cd downstream && ++ git remote add cabal ../fetch-refmap-upstream && ++ git config --unset-all remote.cabal.fetch && ++ git config --add remote.cabal.fetch master && ++ git config --add remote.cabal.fetch next && ++ git config --add remote.cabal.fetch seen && ++ git config remote.cabal.fetchRefmap "+refs/heads/*:refs/remotes/cabal/*" && ++ git fetch cabal && ++ check_fetched_refs <<-\EOF && ++ refs/remotes/cabal/HEAD ++ refs/remotes/cabal/master ++ refs/remotes/cabal/next ++ refs/remotes/cabal/seen ++ EOF ++ git branch --track topic cabal/next && ++ echo refs/remotes/cabal/next >expect && ++ git rev-parse --symbolic-full-name topic@{upstream} >actual && ++ test_cmp expect actual ++ ) ++' ++ ++test_expect_success 'remote..fetchRefmap tracks a branch outside remote..fetch' ' + test_when_finished "rm -fr fetch-refmap-upstream downstream" && + git init -b main fetch-refmap-upstream && + test_commit -C fetch-refmap-upstream base && @@ t/t5510-fetch.sh: test_expect_success 'explicit --refmap option overrides remote + ( + cd downstream && + git remote add -t main origin ../fetch-refmap-upstream && -+ git config remote.origin.refmap "+refs/heads/*:refs/remotes/origin/*" && ++ git config remote.origin.fetchRefmap "+refs/heads/*:refs/remotes/origin/*" && + git fetch origin other && + git branch --track topic origin/other && + test_cmp_config origin branch.topic.remote && @@ t/t5510-fetch.sh: test_expect_success 'explicit --refmap option overrides remote + ) +' + -+test_expect_success 'remote..refmap does not track a branch excluded by remote..fetch' ' ++test_expect_success 'remote..fetchRefmap does not track a branch excluded by remote..fetch' ' + test_when_finished "rm -fr fetch-refmap-upstream downstream" && + git init -b main fetch-refmap-upstream && + test_commit -C fetch-refmap-upstream base && @@ t/t5510-fetch.sh: test_expect_success 'explicit --refmap option overrides remote + cd downstream && + git remote add origin ../fetch-refmap-upstream && + git config --add remote.origin.fetch "^refs/heads/secret" && -+ git config remote.origin.refmap "+refs/heads/*:refs/remotes/origin/*" && ++ git config remote.origin.fetchRefmap "+refs/heads/*:refs/remotes/origin/*" && + git fetch origin secret && + git rev-parse --verify refs/remotes/origin/secret && + test_must_fail git branch --track topic origin/secret 2>err && 2: f685051033 = 2: e8af5bc76b fetch: extract collect_upstream_from_remote() helper 3: 5c76b3b88d ! 3: aad6e5f66f fetch: infer branches to fetch from a refmap-only remote @@ Metadata ## Commit message ## fetch: infer branches to fetch from a refmap-only remote - We want remote..refmap to work on its own, without also setting - remote..fetch, so a remote can be limited to the branches - already in use locally, without listing them by hand or touching the - command line every time. + We want remote..fetchRefmap to work on its own, without also + setting remote..fetch, so a remote can be limited to the + branches already in use locally, without listing them by hand or + touching the command line every time. Infer what to fetch from the local branches whose @{upstream} is already on that remote. The refmap itself still only controls where @@ Commit message Signed-off-by: Harald Nordgren ## Documentation/config/remote.adoc ## -@@ Documentation/config/remote.adoc: remote..fetch:: - remote..refmap:: - The default value of the `--refmap` option for linkgit:git-fetch[1]. - Used to map remote refs being fetched to remote-tracking refs to -- store. See the `--refmap` entry in linkgit:git-fetch[1]. -+ store. If `remote..fetch` is not set either, a refspec-less -+ fetch infers what to fetch from local branches built on this -+ remote, instead of needing `remote..fetch` listed by hand. +@@ Documentation/config/remote.adoc: remote..fetchRefmap:: + fetched to remote-tracking refs to store, the same way + `remote..fetch` does for an explicit refspec. + `remote..fetch`, if also configured, still decides what is +- fetched. See the `--refmap` entry in linkgit:git-fetch[1]. ++ fetched. If it is not set either, a refspec-less fetch instead ++ infers what to fetch from local branches built on this remote. + See the `--refmap` entry in linkgit:git-fetch[1]. remote..push:: @@ Documentation/fetch-options.adoc: endif::git-pull[] + -`remote..refmap` provides the default value for this option, the -same way `remote..fetch` provides the default refspecs to fetch. -+When a refmap is active (from `--refmap` or `remote..refmap`) but ++When a refmap is active (from `--refmap` or `remote..fetchRefmap`) but +nothing to fetch is specified on the command line, nor is there a +`remote..fetch`, infer what to fetch from local branches built on +this remote: remote-tracking branches from this remote that are also @@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote, filter_prefetch_refspec(rs); if (remote) filter_prefetch_refspec(&remote->fetch); + if (effective_refmap) + filter_prefetch_refspec(effective_refmap); + if (!rs->nr && remote && !remote->fetch.nr && + effective_refmap && effective_refmap->nr) { @@ builtin/fetch.c: static int do_fetch(struct transport *transport, + } else { + /* + * The --refmap command line option, if given, takes -+ * precedence over remote..refmap. ++ * precedence over remote..fetchRefmap. + */ + struct refspec *effective_refmap = refmap.nr ? &refmap : + &transport->remote->refmap; @@ t/t5586-fetch-refmap.sh (new) @@ +#!/bin/sh + -+test_description='"git fetch" with a remote..refmap but no remote..fetch ++test_description='"git fetch" with a remote..fetchRefmap but no remote..fetch + +When a remote has a refmap configured but no fetch refspec, a +refspec-less fetch infers what to fetch from the local branches whose @@ t/t5586-fetch-refmap.sh (new) + cd client && + git remote add upstream .. && + test_might_fail git config unset remote.upstream.fetch && -+ git config remote.upstream.refmap \ ++ git config remote.upstream.fetchRefmap \ + "+refs/heads/*:refs/remotes/upstream/*" + ) +' @@ t/t5586-fetch-refmap.sh (new) + ) +' + ++test_expect_success 'prefetch stores tracked branches under refs/prefetch/' ' ++ test_commit main-3 && ++ ( ++ cd client && ++ git rev-parse refs/remotes/upstream/main >expect && ++ git fetch --prefetch upstream && ++ git rev-parse refs/remotes/upstream/main >actual && ++ test_cmp expect actual && ++ git -C .. rev-parse main >expect && ++ git rev-parse refs/prefetch/remotes/upstream/main >actual && ++ test_cmp expect actual ++ ) ++' ++ +test_expect_success 'git remote show does not choke on a refmap-only remote' ' + ( + cd client && 4: 417f14d708 ! 4: 97650da870 remote: add "git remote add --limited-fetch" @@ Commit message branches nobody asked for. Give "git remote add" a --limited-fetch option that sets up - remote..refmap instead of remote..fetch, so that + remote..fetchRefmap instead of remote..fetch, so that "git fetch " only fetches the branches already tracked, as described in the previous commit. It is rejected together with -t/--track or --mirror, since those already say explicitly what to @@ Documentation/git-remote.adoc: the `refs/remotes//` namespace, a refspec t multiple branches without grabbing all branches. + +With `--limited-fetch` option, instead of a `remote..fetch` refspec -+that tracks all branches, `remote..refmap` is set up so that a ++that tracks all branches, `remote..fetchRefmap` is set up so that a +refspec-less `git fetch ` only fetches branches our local branches +are built on. See the `--refmap` entry in linkgit:git-fetch[1] for +details. @@ builtin/remote.c: static int add(int argc, const char **argv, const char *prefix - add_branch(buf.buf, track.items[i].string, - name, mirror, &buf2); + if (use_limited_fetch) { -+ strbuf_addf(&buf, "remote.%s.refmap", name); ++ strbuf_addf(&buf, "remote.%s.fetchRefmap", name); + strbuf_reset(&buf2); + strbuf_addf(&buf2, "+refs/heads/*:refs/remotes/%s/*", name); + repo_config_set(the_repository, buf.buf, buf2.buf); @@ t/t5505-remote.sh: test_expect_success 'filters are listed by git remote -v only + cd full-add && + git remote add --limited-fetch upstream ../two && + test_cmp_config "+refs/heads/*:refs/remotes/upstream/*" \ -+ remote.upstream.refmap && ++ remote.upstream.fetchRefmap && + test_must_fail git config get remote.upstream.fetch + ) +' 5: de0daa8623 ! 5: a802c9f82f remote: default to --limited-fetch in a shallow repository @@ Documentation/git-remote.adoc: Add a remote named __ for the repository at the remote information is set up. + @@ Documentation/git-remote.adoc: With `--limited-fetch` option, instead of a `remote..fetch` refspec - that tracks all branches, `remote..refmap` is set up so that a + that tracks all branches, `remote..fetchRefmap` is set up so that a refspec-less `git fetch ` only fetches branches our local branches are built on. See the `--refmap` entry in linkgit:git-fetch[1] for -details. @@ t/t5505-remote.sh: test_expect_success 'filters are listed by git remote -v only + cd shallow-add && + git remote add upstream ../two && + test_cmp_config "+refs/heads/*:refs/remotes/upstream/*" \ -+ remote.upstream.refmap && ++ remote.upstream.fetchRefmap && + test_must_fail git config get remote.upstream.fetch && + git fetch upstream main && + git branch --set-upstream-to=upstream/main && @@ t/t5505-remote.sh: test_expect_success 'filters are listed by git remote -v only + git remote add --no-limited-fetch upstream ../two && + test_cmp_config "+refs/heads/*:refs/remotes/upstream/*" \ + remote.upstream.fetch && -+ test_must_fail git config get remote.upstream.refmap ++ test_must_fail git config get remote.upstream.fetchRefmap + ) +' + -- gitgitgadget