Avoid fetching every branch of a new remote in a shallow repo.
Changes in v4:
* Removed the automatic default-branch fetch. A fresh remote fetches
nothing until you track a branch explicitly.
* Fixed fetch report showing "new ref HEAD" instead of "new branch "
* Reworded remote..refmap docs and commit message.
Changes in v3:
* Replace the special ":"/"+:" fetch refspec with remote.<name>.refmap,
reusing git's existing --refmap mechanism instead of inventing new
refspec syntax.
* Split the change into 4 commits.
Changes in v2:
* Replaced the opt-in fetch.shallow config entirely with a new special
fetch refspec (+:) that git remote add now defaults new remotes to in a
shallow repository. The new refspec fetches whichever branches any local
branch tracks at that remote, plus the remote's default branch.
Harald Nordgren (4):
fetch: add remote.<name>.refmap
fetch: infer branches to fetch from a refmap-only remote
remote: add "git remote add --limited-fetch"
remote: default to --limited-fetch in a shallow repository
Documentation/config/remote.adoc | 8 +++
Documentation/fetch-options.adoc | 5 ++
Documentation/git-remote.adoc | 14 +++-
builtin/fetch.c | 60 +++++++++++++---
builtin/remote.c | 31 ++++++--
remote.c | 41 ++++++++++-
remote.h | 9 +++
t/meson.build | 1 +
t/t5505-remote.sh | 76 ++++++++++++++++++++
t/t5510-fetch.sh | 17 +++++
t/t5585-fetch-refmap.sh | 119 +++++++++++++++++++++++++++++++
11 files changed, 363 insertions(+), 18 deletions(-)
create mode 100755 t/t5585-fetch-refmap.sh
base-commit: a018953688f1b10bddf91bff8747068f5f4746a4
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-2412%2FHaraldNordgren%2Ffetch-shallow-narrow-refspec-v4
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-2412/HaraldNordgren/fetch-shallow-narrow-refspec-v4
Pull-Request: https://github.com/git/git/pull/2412
Range-diff vs v3:
1: b04c00b974 ! 1: d48a7004e4 fetch: add remote.<name>.refmap
@@ Commit message
Add a per-remote config variable, remote.<name>.refmap, that provides
the default value for --refmap the same way remote.<name>.fetch
- already provides the default refspecs to fetch. It only takes effect
- when there is something explicit to fetch, on the command line or via
- remote.<name>.fetch, matching how --refmap itself already behaves.
+ already provides the default refspecs to fetch. Like --refmap itself,
+ it only maps refs that are actually being fetched, so it has nothing
+ to do when there is nothing explicit to fetch, on the command line or
+ via remote.<name>.fetch.
Signed-off-by: Harald Nordgren <haraldnordgren@gmail.com>
@@ Documentation/config/remote.adoc: remote.<name>.fetch::
+remote.<name>.refmap::
+ The default value of the `--refmap` option for linkgit:git-fetch[1].
-+ Only takes effect when the fetch names what to fetch explicitly,
-+ either on the command line or via `remote.<name>.fetch`. See the
-+ `--refmap` entry in linkgit:git-fetch[1].
++ Used to map remote refs being fetched to remote-tracking refs to
++ store. See the `--refmap` entry in linkgit:git-fetch[1].
+
remote.<name>.push::
The default set of "refspec" for linkgit:git-push[1]. See
2: 4ec508a223 ! 2: 45b26e2bb2 fetch: infer branches to fetch from a refmap-only remote
@@ Commit message
says where to put fetched refs, not what to fetch.
Make that case infer what to fetch: the local branches whose
- @{upstream} is already on that remote, plus the remote's default
- branch, which is always included so it is available even before
- anything is set up to track it. This lets a remote be configured to
- fetch only the branches actually in use, without listing them by
- hand in remote.<name>.fetch, and without needing to touch the
- command line every time.
+ @{upstream} is already on that remote. This lets a remote be
+ configured to fetch only the branches actually in use, without
+ listing them by hand in remote.<name>.fetch, and without needing to
+ touch the command line every time.
Signed-off-by: Harald Nordgren <haraldnordgren@gmail.com>
## Documentation/config/remote.adoc ##
@@ Documentation/config/remote.adoc: remote.<name>.fetch::
-
remote.<name>.refmap::
The default value of the `--refmap` option for linkgit:git-fetch[1].
-- Only takes effect when the fetch names what to fetch explicitly,
-- either on the command line or via `remote.<name>.fetch`. See the
-- `--refmap` entry in linkgit:git-fetch[1].
-+ If `remote.<name>.fetch` is not set either, a refspec-less fetch
-+ infers what to fetch from local branches built on this remote,
-+ instead of fetching every branch it has. See the `--refmap` entry
-+ in linkgit:git-fetch[1].
+ Used to map remote refs being fetched to remote-tracking refs to
+- store. See the `--refmap` entry in linkgit:git-fetch[1].
++ store. If `remote.<name>.fetch` is not set either, a refspec-less
++ fetch infers what to fetch from local branches built on this
++ remote, instead of fetching every branch it has. See the
++ `--refmap` entry in linkgit:git-fetch[1].
remote.<name>.push::
The default set of "refspec" for linkgit:git-push[1]. See
@@ Documentation/fetch-options.adoc: endif::git-pull[]
+When a refmap is active (from `--refmap` or `remote.<name>.refmap`) but
+there is nothing to fetch, neither on the command line nor from
+`remote.<name>.fetch`, Git infers what to fetch from the local branches
-+whose `@{upstream}` is on that remote, plus the remote's default branch,
-+which is always included so that it is available even before anything
-+is set up to track it.
++whose `@{upstream}` is on that remote.
`-t`::
`--tags`::
## builtin/fetch.c ##
-@@ builtin/fetch.c: static void filter_prefetch_refspec(struct refspec *rs)
- static struct ref *get_ref_map(struct remote *remote,
- const struct ref *remote_refs,
- struct refspec *rs,
-- int tags, int *autotags)
-+ int tags, int *autotags,
-+ char **bootstrap_head_branch)
- {
- int i;
- struct ref *rm;
@@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote,
struct ref **tail = &ref_map;
struct refspec *effective_refmap =
@@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote,
+ effective_refmap && effective_refmap->nr) {
+ struct string_list tracked = STRING_LIST_INIT_DUP;
+ struct string_list_item *item;
-+ struct ref *head;
-+ char *default_branch = NULL;
-+ char *default_branch_dst = NULL;
+
+ branches_tracking_remote(remote, &tracked);
+ for_each_string_list_item(item, &tracked)
+ refspec_append(&inferred_rs, item->string);
-+
-+ /* Always fetch the default branch too, as "HEAD". */
-+ head = get_remote_ref(remote_refs, "HEAD");
-+ if (head && head->symref && *head->symref)
-+ default_branch = xstrdup(head->symref);
-+ free_one_ref(head);
-+
-+ for (i = 0; default_branch && !default_branch_dst &&
-+ !string_list_has_string(&tracked, default_branch) &&
-+ i < effective_refmap->nr; i++) {
-+ struct refspec_item *map = &effective_refmap->items[i];
-+
-+ if (map->pattern)
-+ match_refname_with_pattern(map->src, default_branch,
-+ map->dst, &default_branch_dst);
-+ else if (!strcmp(map->src, default_branch))
-+ default_branch_dst = xstrdup(map->dst);
-+ }
-+
-+ if (default_branch_dst) {
-+ struct refspec_item head_item = { .force = 1 };
-+
-+ head_item.src = xstrdup("HEAD");
-+ head_item.dst = default_branch_dst;
-+ get_fetch_map(remote_refs, &head_item, &tail, 1);
-+ free(head_item.src);
-+ }
-+
-+ if (default_branch && bootstrap_head_branch) {
-+ const char *branch_name = default_branch;
-+ skip_prefix(branch_name, "refs/heads/", &branch_name);
-+ *bootstrap_head_branch = xstrdup(branch_name);
-+ }
-+
-+ free(default_branch);
-+ free(default_branch_dst);
+ string_list_clear(&tracked, 0);
+
+ rs = &inferred_rs;
@@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote,
return ref_map;
}
-@@ builtin/fetch.c: static void warn_set_head(const char *remote, const char *head_name,
- }
-
- static int set_head(const struct ref *remote_refs, struct remote *remote,
-- int follow_remote_head)
-+ int follow_remote_head, const char *known_head_branch)
- {
- int result = 0, create_only, baremirror, was_detached;
- struct strbuf b_head = STRBUF_INIT, b_remote_head = STRBUF_INIT,
- b_local_head = STRBUF_INIT;
- const char *no_warn_branch = remote->no_warn_branch;
- char *head_name = NULL;
-- struct ref *ref, *matches;
-+ struct ref *ref, *matches = NULL;
- struct ref *fetch_map = NULL, **fetch_map_tail = &fetch_map;
- struct refspec_item refspec = {
- .force = 0,
-@@ builtin/fetch.c: static int set_head(const struct ref *remote_refs, struct remote *remote,
- struct string_list heads = STRING_LIST_INIT_DUP;
- struct ref_store *refs = get_main_ref_store(the_repository);
-
-- get_fetch_map(remote_refs, &refspec, &fetch_map_tail, 0);
-- matches = guess_remote_head(find_ref_by_name(remote_refs, "HEAD"),
-- fetch_map, REMOTE_GUESS_HEAD_ALL);
-- for (ref = matches; ref; ref = ref->next) {
-- string_list_append(&heads, strip_refshead(ref->name));
-- }
-+ if (known_head_branch) {
-+ head_name = xstrdup(known_head_branch);
-+ } else {
-+ get_fetch_map(remote_refs, &refspec, &fetch_map_tail, 0);
-+ matches = guess_remote_head(find_ref_by_name(remote_refs, "HEAD"),
-+ fetch_map, REMOTE_GUESS_HEAD_ALL);
-+ for (ref = matches; ref; ref = ref->next) {
-+ string_list_append(&heads, strip_refshead(ref->name));
-+ }
-
-- if (!heads.nr)
-- result = 1;
-- else if (heads.nr > 1)
-- result = 1;
-- else
-- head_name = xstrdup(heads.items[0].string);
-+ if (!heads.nr)
-+ result = 1;
-+ else if (heads.nr > 1)
-+ result = 1;
-+ else
-+ head_name = xstrdup(heads.items[0].string);
-+ }
-
- if (!head_name)
- goto cleanup;
-@@ builtin/fetch.c: static int do_fetch(struct transport *transport,
- struct strmap rejected_refs = STRMAP_INIT;
- int summary_width = 0;
- int follow_remote_head;
-+ char *bootstrap_head_branch = NULL;
-
- if (tags == TAGS_DEFAULT) {
- if (transport->remote->fetch_tags == 2)
@@ builtin/fetch.c: static int do_fetch(struct transport *transport,
refspec_ref_prefixes(rs, &transport_ls_refs_options.ref_prefixes);
} else {
@@ builtin/fetch.c: static int do_fetch(struct transport *transport,
+ strvec_push(&transport_ls_refs_options.ref_prefixes,
+ item->string);
+ string_list_clear(&tracked, 0);
-+ strvec_push(&transport_ls_refs_options.ref_prefixes, "HEAD");
+ } else if (transport->remote->fetch.nr) {
refspec_ref_prefixes(&transport->remote->fetch,
&transport_ls_refs_options.ref_prefixes);
@@ builtin/fetch.c: static int do_fetch(struct transport *transport,
for (i = 0; i < branch->merge_nr; i++) {
strvec_push(&transport_ls_refs_options.ref_prefixes,
@@ builtin/fetch.c: static int do_fetch(struct transport *transport,
- transport_ls_refs_options_release(&transport_ls_refs_options);
ref_map = get_ref_map(transport->remote, remote_refs, rs,
-- tags, &autotags);
-+ tags, &autotags, &bootstrap_head_branch);
+ tags, &autotags);
+
if (!update_head_ok)
check_not_current_branch(ref_map);
-@@ builtin/fetch.c: static int do_fetch(struct transport *transport,
- * Way too many cases where this can go wrong so let's just
- * ignore errors and fail silently for now.
- */
-- set_head(remote_refs, transport->remote, follow_remote_head);
-+ set_head(remote_refs, transport->remote, follow_remote_head,
-+ bootstrap_head_branch);
- }
-
- cleanup:
-+ free(bootstrap_head_branch);
- /*
- * When using batched updates, we want to commit the non-rejected
- * updates and also handle the rejections.
## remote.c ##
@@ remote.c: int branch_merge_matches(struct branch *branch,
@@ t/t5585-fetch-refmap.sh (new)
+
+When a remote has a refmap configured but no fetch refspec, a
+refspec-less fetch infers what to fetch from the local branches whose
-+@{upstream} is on that remote, plus the default branch of that remote,
-+which is always included so that it is available even before anything
-+is set up to track it.
++@{upstream} is on that remote.
+'
+
+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
@@ t/t5585-fetch-refmap.sh (new)
+ )
+'
+
-+test_expect_success 'fetching the new remote does not need every branch it has' '
++test_expect_success 'a bare fetch needs nothing until a branch is tracked' '
+ (
+ cd client &&
+ git fetch upstream &&
+ git for-each-ref --format="%(refname)" refs/remotes/upstream >actual &&
-+ cat >expect <<-\EOF &&
-+ refs/remotes/upstream/HEAD
-+ refs/remotes/upstream/main
-+ EOF
-+ test_cmp expect actual
++ test_must_be_empty actual
+ )
+'
+
-+test_expect_success 'set-upstream-to now resolves right after that first fetch' '
++test_expect_success 'an explicit one-time fetch lets a branch be tracked' '
+ (
+ cd client &&
-+ git branch --set-upstream-to=upstream &&
++ git fetch upstream main &&
++ git branch --set-upstream-to=upstream/main &&
+ test_cmp_config upstream branch.main.remote &&
+ test_cmp_config refs/heads/main branch.main.merge
+ )
3: 604e584956 ! 3: e2f072c254 remote: add "git remote add --limited-fetch"
@@ Documentation/git-remote.adoc: the `refs/remotes/<name>/` namespace, a refspec t
+With `--limited-fetch` option, instead of a `remote.<name>.fetch` refspec
+that tracks all branches, `remote.<name>.refmap` is set up so that a
+refspec-less `git fetch <name>` only fetches branches our local branches
-+are built on, plus the remote's default branch. See the `--refmap` entry
-+in linkgit:git-fetch[1] for details.
++are built on. See the `--refmap` entry in linkgit:git-fetch[1] for
++details.
++
With `-m <master>` option, a symbolic-ref `refs/remotes/<name>/HEAD` is set
up to point at remote's _<master>_ branch. See also the set-head command.
4: 31462e9445 ! 4: 43b9711a2c remote: default to --limited-fetch in a shallow repository
@@ Documentation/git-remote.adoc: Add a remote named _<name>_ for the repository at
@@ Documentation/git-remote.adoc: With `--limited-fetch` option, instead of a `remote.<name>.fetch` refspec
that tracks all branches, `remote.<name>.refmap` is set up so that a
refspec-less `git fetch <name>` only fetches branches our local branches
- are built on, plus the remote's default branch. See the `--refmap` entry
--in linkgit:git-fetch[1] for details.
-+in linkgit:git-fetch[1] for details. `--no-limited-fetch` explicitly
-+disables this, overriding the shallow-repository default described above.
+ are built on. See the `--refmap` entry in linkgit:git-fetch[1] for
+-details.
++details. `--no-limited-fetch` explicitly disables this, overriding the
++shallow-repository default described above.
+
With `-m <master>` option, a symbolic-ref `refs/remotes/<name>/HEAD` is set
up to point at remote's _<master>_ branch. See also the set-head command.
@@ t/t5505-remote.sh: test_expect_success 'filters are listed by git remote -v only
+ test_cmp_config "+refs/heads/*:refs/remotes/upstream/*" \
+ remote.upstream.refmap &&
+ test_must_fail git config get remote.upstream.fetch &&
++ git fetch upstream main &&
++ git branch --set-upstream-to=upstream/main &&
++ test_cmp_config upstream branch.main.remote &&
++ test_cmp_config refs/heads/main branch.main.merge &&
+ git fetch upstream &&
+ git for-each-ref --format="%(refname)" refs/remotes/upstream >actual &&
+ cat >expect <<-\EOF &&
+ refs/remotes/upstream/HEAD
+ refs/remotes/upstream/main
+ EOF
-+ test_cmp expect actual &&
-+ git branch --set-upstream-to=upstream &&
-+ test_cmp_config upstream branch.main.remote &&
-+ test_cmp_config refs/heads/main branch.main.merge
++ test_cmp expect actual
+ )
+'
+Harald Nordgren via GitGitGadget <gitgitgadget@gmail.com>