Re: [PATCH v2] fetch: avoid fetching every branch of a new remote in a shallow repo

"Harald Nordgren via GitGitGadget" <gitgitgadget@gmail.com> writes:

That's way too much for a single patch. It needs to be split into digestible chunks, but I offhand do not know how many pieces are appropriate, so let's think aloud together and try to refine the design while we do so.

The outline of our design so far should give something like this in our configuration file.

	[remote "second"]
		fetch = :+

	[branch "topic1"]
		remote = second
		merge = refs/heads/main

	[branch "topic2"]
		remote = second
		merge = refs/heads/next

In the "fetch only what we build on" mode, we would collect local branches $X where branch.$X.remote == second and then collect branch.$X.merge for these local branches. In this case, we would decide to fetch 'main' and 'next' branches in the end.

But notice that this does not give us sufficient information. There is no explicit clue that tells that the remote-tracking branches for this remote 'second' should be stored under refs/remotes/second/ hierarchy. A normal remote that is defined like so:

	[remote "origin"]
		fetch = +refs/heads/*:refs/remotes/origin/*

does not have such a problem, as it makes it crystal clear that their branches go under refs/remotes/origin/ hierarchy.

So using "fetch = :+" is *not* a good idea, as I said. Let's scrap that syntax.

One thing we could do is probably to introduce

	[remote "second"]
		refmap = +refs/heads/*:refs/remotes/second/*

instead to give this clue (see "git fetch --help" for what a refmap is; it looks similar to refspec but only defines how their refs are mapped to our namespace without specifying what to be fetched, which is exactly what we need here). We do not use remote.second.fetch at all.

It would be an easy first step to teach that an explicit

	$ git fetch second main next

with such a remote.second.refmap should behave the same way as

	$ git fetch --refmap='+refs/heads/*:refs/remotes/second/*' second \
		main next

in a repository without the refmote.second.refmap configuration. As "git fetch --refmap=... second main next" should already work, it would be only the matter of supplementing the command line argument with configured default.

Then teach "git fetch" to further treat

	$ git fetch --refmap='+refs/heads/*:refs/remotes/second/*' second

i.e., fetch with refmap but without specifying what exactly to fetch, as a request to fetch their branches we build on (and nothing else), using the refmap, in other words, the lack of "what to fetch" in the above command line signals "git fetch" to rewrite the above to

	$ git fetch --refmap='+refs/heads/*:refs/remotes/second/*' second \
		main next

internally. Since we have the previous remote.X.refmap step already, it means that with remote.second.refmap configured properly, the user can only say

	$ git fetch second

and it would do the right thing in our scenario.

Another and final step would be to teach "git remote add" to add

        [remote "second"]
                refmap = +refs/heads/*:refs/remotes/second/*

when you want to fetch only what you build on. I am not sure what should trigger the decision. Your initial message said something about shallow and sparse and an earlier review refuted one of them (I do not recall which offhand, but probably sparse). It probably is a good idea to start with an explicit command line option to "git remote add --limited-fetch" in a single commit.

And then add heuristics (like "in a shallow clone, this mode is turned on by default, but an explicit '--no-limited-fetch' can countermand it") in another commit.

So far, we identified four distinct commits, each bite sized.

 - remote.X.refmap configuration acts as if --refmap=... command
   line argument was passed.

 - passing refmap without saying what to fetch enumerates what their
   branches we build on, and pretend as if the user listed these
   branches on the command line to fetch.

 - "git remote add --limited-fetch" creates remote.X.refmap instead
   of remote.X.fetch as necessary.

 - "git remote add" without explicit "--[no-]limited-fetch" uses
   heuristics to enable it.

Or something like that, perhaps?