threads / discuss / 20755

Question regarding git fetch

Subject: Question regarding git fetch

## tl;dr

17 messages between Aug 27, 2009 and Aug 28, 2009.

replies: 16people: 6as markdown or json

Tom Lambda· Aug 27, 2009, 15:30 UTC · lore

I noticed that git-fetch seems smarter when it is run without a <refspec> argument than when one specifies a branch name. I use a simple setup where a remote central repository is defined when it is cloned the first time (clone -o central ...). This leads to these default parameters:

remote.central.fetch=+refs/heads/*:refs/remotes/central/* branch.master.remote=central branch.master.merge=refs/heads/master

When I use "git fetch central" each branch in central's refs/heads/ is automatically fetched to my refs/remotes/central/ as expected.

What was a little bit surprising to me is that running "git fetch central master" does not update refs/remotes/central/master but simply updates FETCH_HEAD.

I read the manual and I know that updating FETCH_HEAD is the expected default behavior for git-fetch. However, I really like the fact that git fetch (without <refspec>) knows that ANY branch in refs/heads/ corresponds to refs/remotes/central/. Is there a way to change the configuration to have "git fetch central branch" always updating refs/remotes/central/branch whatever the specified branch.

I would prefer not to have to specify the <dst> each time: git fetch central branch:remotes/central/branch

Thank you, Tom

-- 
View this message in context: http://n2.nabble.com/Question-regarding-git-fetch-tp3527289p3527289.html
Sent from the git mailing list archive at Nabble.com.
Avery Pennarun· Aug 27, 2009, 15:36 UTC · re: Tom Lambda · lore

Re: Question regarding git fetch

On Thu, Aug 27, 2009 at 3:30 PM, Tom Lambda<tom.lambda@gmail.com> wrote:
> What was a little bit surprising to me is that running "git fetch central
> master" does not update refs/remotes/central/master but simply updates
> FETCH_HEAD.

I've often wanted this myself, especially when doing things like "git pull origin master". However, I know the current behaviour is also useful sometimes, and changing it would introduce an unexpected side effect. Git currently promises that your refs/remotes/* branches will never be updated unless you explicitly request it, even if you're fetching, merging, and pulling other stuff. This means you can write scripts to do complicated things without triggering unexpected user-visible side effects.

So basically, I agree that it would often be much more user-friendly to do what you're asking. But it would be less scripting-friendly. I don't think anyone has thought of an answer that better balances the two.

Have fun,
Avery
Eric Raible· Aug 27, 2009, 16:21 UTC · re: Avery Pennarun · lore

Re: Question regarding git fetch

Avery Pennarun <apenwarr <at> gmail.com> writes:
> Git currently promises that your refs/remotes/* branches will
> never be updated unless you explicitly request it, even if you're
> fetching, merging, and pulling other stuff.

So your claim is that "git fetch central" is somehow more explicit than "git fetch central master"?

I understand that "git fetch central" will use remote.central.fetch (which _is_ explicit), but the command itself is certainly _less_ explicit about specifying the remote branch.

Either way, AFAICT it seems purely historical that "git fetch central master" doesn't update remotes/central.

- Eric
Avery Pennarun· Aug 27, 2009, 16:28 UTC · re: Eric Raible · lore

Re: Question regarding git fetch

On Thu, Aug 27, 2009 at 4:21 PM, Eric Raible<raible@gmail.com> wrote:
Show 8 quoted lines
> Avery Pennarun <apenwarr <at> gmail.com> writes:
>
>> Git currently promises that your refs/remotes/* branches will
>> never be updated unless you explicitly request it, even if you're
>> fetching, merging, and pulling other stuff.
>
> So your claim is that "git fetch central" is somehow more
> explicit than "git fetch central master"?

It is, because there's nothing else you could possibly do with it. It fetches too many things, so you don't know their names (other than the ones it puts into refs/remotes/*).

So it's not exactly that it's more explicit - in fact it's rather confusing and non-explicit that the three-parameter version does something almost entirely different from the four-parameter one - but the three-parameter version is at least completely unambiguous.

> Either way, AFAICT it seems purely historical that
> "git fetch central master" doesn't update remotes/central.
This seems true to me.
Have fun,
Avery
Björn Steinbrink· Aug 27, 2009, 16:46 UTC · re: Avery Pennarun · lore

Re: Question regarding git fetch

On 2009.08.27 15:36:53 +0000, Avery Pennarun wrote:
Show 18 quoted lines
> On Thu, Aug 27, 2009 at 3:30 PM, Tom Lambda<tom.lambda@gmail.com> wrote:
> > What was a little bit surprising to me is that running "git fetch central
> > master" does not update refs/remotes/central/master but simply updates
> > FETCH_HEAD.
> 
> I've often wanted this myself, especially when doing things like "git
> pull origin master".  However, I know the current behaviour is also
> useful sometimes, and changing it would introduce an unexpected side
> effect.  Git currently promises that your refs/remotes/* branches will
> never be updated unless you explicitly request it, even if you're
> fetching, merging, and pulling other stuff.  This means you can write
> scripts to do complicated things without triggering unexpected
> user-visible side effects.
> 
> So basically, I agree that it would often be much more user-friendly
> to do what you're asking.  But it would be less scripting-friendly.  I
> don't think anyone has thought of an answer that better balances the
> two.

It would also be pretty hard to implement that. Given the default fetch refspec, it would "simply" be a matter of mapping the given ref to the refspec, so e.g. "foo" becomes "refs/heads/foo:refs/remotes/origin/foo". But even just using "git remote add -t master foo git://..." breaks that, as the fetch refspec in the config will no longer be a glob, and thus no such mapping is possible.

Björn
Avery Pennarun· Aug 27, 2009, 17:22 UTC · re: Björn Steinbrink · lore

Re: Question regarding git fetch

2009/8/27 Björn Steinbrink <B.Steinbrink@gmx.de>:
Show 6 quoted lines
> It would also be pretty hard to implement that. Given the default fetch
> refspec, it would "simply" be a matter of mapping the given ref to the
> refspec, so e.g. "foo" becomes "refs/heads/foo:refs/remotes/origin/foo".
> But even just using "git remote add -t master foo git://..." breaks
> that, as the fetch refspec in the config will no longer be a glob, and
> thus no such mapping is possible.

Hmm, I don't really see why that introduces a problem. If you use -t to override explicitly which refs you want to save, then it's not a problem if git doesn't save other refs, right?

I'd be more concerned about the inconsistency between
   git fetch git://whatever master
and
   git fetch origin master

There's no really good way for the first one to know it needs to update any branches, even though 'origin' might be an alias for git://whatever. So users will still be confused.

Thinking of that also reminds me of another surprise.  If you do:
   git fetch git://whatever

...it seems to do nothing at all, as far as I can see. Which makes sense, I guess, since I wouldn't really expect it to be meaningful. But it seems to connect up to the remote server anyway just in case.

I suppose I have no useful suggestions here, other than an interpretation of why users find the current behaviour confusing.

Have fun,
Avery
Jeff King· Aug 27, 2009, 20:48 UTC · re: Avery Pennarun · lore

Re: Question regarding git fetch

On Thu, Aug 27, 2009 at 05:22:55PM +0000, Avery Pennarun wrote:
Show 11 quoted lines
> 2009/8/27 Björn Steinbrink <B.Steinbrink@gmx.de>:
> > It would also be pretty hard to implement that. Given the default fetch
> > refspec, it would "simply" be a matter of mapping the given ref to the
> > refspec, so e.g. "foo" becomes "refs/heads/foo:refs/remotes/origin/foo".
> > But even just using "git remote add -t master foo git://..." breaks
> > that, as the fetch refspec in the config will no longer be a glob, and
> > thus no such mapping is possible.
> 
> Hmm, I don't really see why that introduces a problem.  If you use -t
> to override explicitly which refs you want to save, then it's not a
> problem if git doesn't save other refs, right?

I think you can handle both cases just by matching the fetched refs to the LHS of the refspec.

So if you fetch "refs/heads/foo", and if you have a refspec of:
  refs/heads/*:refs/remotes/origin/*

then you see that the LHS glob matches, and the RHS expands to refs/remotes/origin/foo. And if you have a more restrictive refspec, that would work, too:

  refs/heads/foo:refs/remotes/origin/foo
would still match, but
  refs/heads/bar:refs/remotes/origin/bar

does not match on the LHS, so you write nothing. It would even handle multiple refspecs properly.

And this matching is not really any different than what the fetch code does when applying the refspec to what the remote offers. So I don't think it should be any significant new code; it's just a matter of activating that matching and updating the local tracking refs based on what we actually fetched, instead of what the remote advertised.

Show 9 quoted lines
> I'd be more concerned about the inconsistency between
> 
>    git fetch git://whatever master
> and
>    git fetch origin master
> 
> There's no really good way for the first one to know it needs to
> update any branches, even though 'origin' might be an alias for
> git://whatever.  So users will still be confused.

Well, you can always reverse-lookup each remote to see if the URL matches. Of course you would never know that "http://whatever" is an alias for "git://whatever". Personally, I don't imagine that users really expect git to reverse-map remotes in that way (after all, why would they input git://whatever long-hand if they knew that it was a remote).

Show 7 quoted lines
> Thinking of that also reminds me of another surprise.  If you do:
> 
>    git fetch git://whatever
> 
> ...it seems to do nothing at all, as far as I can see.  Which makes
> sense, I guess, since I wouldn't really expect it to be meaningful.
> But it seems to connect up to the remote server anyway just in case.

Shouldn't it fetch HEAD from the remote and store it in FETCH_HEAD? And then tell you that it did that?

I get:
  $ mkdir foo && cd foo && git init
  Initialized empty Git repository in /home/peff/foo/.git/
  $ git fetch ~/compile/git
  remote: Counting objects: 85011, done.
  remote: Compressing objects: 100% (23942/23942), done.
  remote: Total 85011 (delta 61389), reused 83076 (delta 59655)
  Receiving objects: 100% (85011/85011), 19.03 MiB | 11385 KiB/s, done.
  Resolving deltas: 100% (61389/61389), done.
  From /home/peff/compile/git
   * branch            HEAD       -> FETCH_HEAD
  $ git fetch git://git.kernel.org/pub/scm/git/git.git
  From git://git.kernel.org/pub/scm/git/git
   * branch            HEAD       -> FETCH_HEAD
-Peff
Jeff King· Aug 27, 2009, 21:34 UTC · re: Jeff King · lore

Re: Question regarding git fetch

On Thu, Aug 27, 2009 at 04:48:35PM -0400, Jeff King wrote:
Show 5 quoted lines
> And this matching is not really any different than what the fetch code
> does when applying the refspec to what the remote offers. So I don't
> think it should be any significant new code; it's just a matter of
> activating that matching and updating the local tracking refs based on
> what we actually fetched, instead of what the remote advertised.

Sure enough, here is a really simple proof of concept. This is the first time I have really looked into the fetch code, so I hope I'm not totally breaking something else. :)

Basically it works like this. Both "git fetch remote" and "git fetch remote refspec" work by making a "ref map" via get_ref_map. This is a list of refs, with some refs having a "peer ref" that points to the local counterpart. Those without a peer are just stored in FETCH_HEAD. The function works by starting with a list of possible remote refs, and then applying the refspecs to it.

Calling fetch without explicit refspecs means we will create the map using the configured refspecs. Calling with commandline refspecs means we will use those. So what this patch does is to first apply the commandline refspecs to narrow the list, and then apply the configured refspecs on top of that to make a new list, and then concatenate the lists.

So you will end up with "two" refs to fetch, which just happen to match the same source. One will go to FETCH_HEAD, and one will go to the tracking ref. I.e.:

  $ git remote add origin ~/compile/git
  $ git fetch origin next
  From /home/peff/compile/git
   * branch            next       -> FETCH_HEAD
   * [new branch]      next       -> origin/next

And it should work the same if you supply a more interesting refspec, too:

  $ git fetch -v origin next:foo
  From /home/peff/compile/git
   * [new branch]      next       -> foo
   = [up to date]      next       -> origin/next
or even a wildcard:
  $ git fetch -v origin refs/heads/*:refs/foo/*
  From /home/peff/compile/git
   * [new branch]      master     -> refs/foo/master
   * [new branch]      next       -> refs/foo/next
   * [new branch]      master     -> origin/master
   = [up to date]      next       -> origin/next

I haven't thought long enough to be convinced there aren't any bad side effects, though. I guess if you had non-traditional configured refspecs, you might be surprised to see them applied in this case. Something like:

  # we usually fetch the remote's master straight into our production
  # branch for deployment
  $ git config remote.origin.fetch refs/heads/master:refs/heads/production
  # but today let's demo it first
  $ git fetch origin master:demo

I don't know if people are actually using refspecs in that way. But we do need to consider that this is a potentially destructive change.

Anyway, here is the patch.
---
diff --git a/builtin-fetch.c b/builtin-fetch.c
index 817dd6b..d9e44c7 100644
--- a/builtin-fetch.c
+++ b/builtin-fetch.c
@@ -119,6 +119,9 @@ static struct ref *get_ref_map(struct transport *transport,
 	const struct ref *remote_refs = transport_get_remote_refs(transport);
 
 	if (ref_count || tags == TAGS_SET) {
+		struct ref *tracking_refs = NULL;
+		struct ref **tracking_tail = &tracking_refs;
+
 		for (i = 0; i < ref_count; i++) {
 			get_fetch_map(remote_refs, &refs[i], &tail, 0);
 			if (refs[i].dst && refs[i].dst[0])
@@ -129,6 +132,12 @@ static struct ref *get_ref_map(struct transport *transport,
 			rm->merge = 1;
 		if (tags == TAGS_SET)
 			get_fetch_map(remote_refs, tag_refspec, &tail, 0);
+
+		for (i = 0; i < transport->remote->fetch_refspec_nr; i++)
+			get_fetch_map(ref_map, &transport->remote->fetch[i],
+					&tracking_tail, 0);
+		*tail = tracking_refs;
+		tail = tracking_tail;
 	} else {
 		/* Use the defaults */
 		struct remote *remote = transport->remote;
Junio C Hamano· Aug 27, 2009, 21:44 UTC · re: Jeff King · lore

Re: Question regarding git fetch

Jeff King <peff@peff.net> writes:
Show 6 quoted lines
>   # we usually fetch the remote's master straight into our production
>   # branch for deployment
>   $ git config remote.origin.fetch refs/heads/master:refs/heads/production
>
>   # but today let's demo it first
>   $ git fetch origin master:demo

I think this is a good example that any change results from this discussion should apply _only_ to cases where command line refspecs lack colon (i.e. used to mean "do not store this anywhere but in FETCH_HEAD").

Jeff King· Aug 27, 2009, 21:50 UTC · re: Junio C Hamano · lore

Re: Question regarding git fetch

On Thu, Aug 27, 2009 at 02:44:54PM -0700, Junio C Hamano wrote:
Show 12 quoted lines
> Jeff King <peff@peff.net> writes:
> 
> >   # we usually fetch the remote's master straight into our production
> >   # branch for deployment
> >   $ git config remote.origin.fetch refs/heads/master:refs/heads/production
> >
> >   # but today let's demo it first
> >   $ git fetch origin master:demo
> 
> I think this is a good example that any change results from this
> discussion should apply _only_ to cases where command line refspecs lack
> colon (i.e. used to mean "do not store this anywhere but in FETCH_HEAD").

I don't think the colon is the issue. Consider the same situation, but I say:

  # but today let's demo it first
  $ git fetch origin master
  $ git checkout -b demo FETCH_HEAD

I'm still screwed. The issue is that you consider your configured refspec destinations to be precious, and not merely a cache for what's happening on the remote side.

-Peff
Jeff King· Aug 27, 2009, 21:53 UTC · re: Jeff King · lore

Re: Question regarding git fetch

On Thu, Aug 27, 2009 at 05:50:07PM -0400, Jeff King wrote:
Show 14 quoted lines
> > I think this is a good example that any change results from this
> > discussion should apply _only_ to cases where command line refspecs lack
> > colon (i.e. used to mean "do not store this anywhere but in FETCH_HEAD").
> 
> I don't think the colon is the issue. Consider the same situation, but I
> say:
> 
>   # but today let's demo it first
>   $ git fetch origin master
>   $ git checkout -b demo FETCH_HEAD
> 
> I'm still screwed. The issue is that you consider your configured
> refspec destinations to be precious, and not merely a cache for what's
> happening on the remote side.

Which, btw, led me to consider whether there are heuristics for deciding when a fetch refspec means one thing and not the other. I don't think there are reliable ones (probably the default configured refs/remotes/$remotename/* would not yield false positives, but I think limiting to that would yield false negatives). So maybe this is something that should be configurable, disabled by default for now, and maybe enabled by default in the future (v1.7.0?).

-Peff
Avery Pennarun· Aug 27, 2009, 22:12 UTC · re: Jeff King · lore

Re: Question regarding git fetch

On Thu, Aug 27, 2009 at 9:50 PM, Jeff King<peff@peff.net> wrote:
Show 10 quoted lines
> I don't think the colon is the issue. Consider the same situation, but I
> say:
>
>  # but today let's demo it first
>  $ git fetch origin master
>  $ git checkout -b demo FETCH_HEAD
>
> I'm still screwed. The issue is that you consider your configured
> refspec destinations to be precious, and not merely a cache for what's
> happening on the remote side.
Is the "precious remote ref" concept perhaps an imaginary one?

After all, if I *really* care about the prior state of the remote, I can just make it a remote branch. And if (as often happens) I just want to know what's new in that ref since last time I merged, it's simply

   git log master..origin/master

This works even if master has extra commits vs. origin/master, since the double-dot invokes git-merge-base.

I think this might be a much more common than the case where people actually want to see "what's changed since last time I checked what's changed." At least, the latter question has never been very interesting to me, or if it is, it's easy for me to tell by eye.

Avery
Jeff King· Aug 27, 2009, 22:16 UTC · re: Avery Pennarun · lore

Re: Question regarding git fetch

On Thu, Aug 27, 2009 at 10:12:50PM +0000, Avery Pennarun wrote:
Show 5 quoted lines
> > I'm still screwed. The issue is that you consider your configured
> > refspec destinations to be precious, and not merely a cache for what's
> > happening on the remote side.
> 
> Is the "precious remote ref" concept perhaps an imaginary one?

Maybe. I certainly don't use it. But I am trying to consider corner cases where somebody who _isn't_ me is going to get screwed by a change we make.

> After all, if I *really* care about the prior state of the remote, I
> can just make it a remote branch.  And if (as often happens) I just
Do you mean "local branch" here?
Show 7 quoted lines
> want to know what's new in that ref since last time I merged, it's
> simply
> 
>    git log master..origin/master
> 
> This works even if master has extra commits vs. origin/master, since
> the double-dot invokes git-merge-base.

Well, ".." doesn't use git-merge-base. But yes, I actually do this, except I do:

  gitk master...origin/master
-Peff
Avery Pennarun· Aug 27, 2009, 22:24 UTC · re: Jeff King · lore

Re: Question regarding git fetch

On Thu, Aug 27, 2009 at 10:16 PM, Jeff King<peff@peff.net> wrote:
Show 5 quoted lines
> On Thu, Aug 27, 2009 at 10:12:50PM +0000, Avery Pennarun wrote:
>> After all, if I *really* care about the prior state of the remote, I
>> can just make it a remote branch.  And if (as often happens) I just
>
> Do you mean "local branch" here?
Yes.
Show 12 quoted lines
>> want to know what's new in that ref since last time I merged, it's
>> simply
>>
>>    git log master..origin/master
>>
>> This works even if master has extra commits vs. origin/master, since
>> the double-dot invokes git-merge-base.
>
> Well, ".." doesn't use git-merge-base. But yes, I actually do this,
> except I do:
>
>  gitk master...origin/master

Sorry, not my day for accuracy, it seems. I use both, and I should have remembered that the triple-dot is the one that worries about the merge-base.

Have fun,
Avery
Junio C Hamano· Aug 27, 2009, 21:20 UTC · re: Avery Pennarun · lore

Re: Question regarding git fetch

Avery Pennarun <apenwarr@gmail.com> writes:
Show 13 quoted lines
> On Thu, Aug 27, 2009 at 3:30 PM, Tom Lambda<tom.lambda@gmail.com> wrote:
>> What was a little bit surprising to me is that running "git fetch central
>> master" does not update refs/remotes/central/master but simply updates
>> FETCH_HEAD.
>
> I've often wanted this myself, especially when doing things like "git
> pull origin master".  However, I know the current behaviour is also
> useful sometimes, and changing it would introduce an unexpected side
> effect.  Git currently promises that your refs/remotes/* branches will
> never be updated unless you explicitly request it, even if you're
> fetching, merging, and pulling other stuff.  This means you can write
> scripts to do complicated things without triggering unexpected
> user-visible side effects.

I think it is reasonable, in 1.7.0, to change "git fetch" with command line refspacs that lack colon (i.e. saying "fetch this ref" without saying "and store it here") so that it updates remote tracking refs if and only if an appropriate remote.$remote.fetch is configured to do so. E.g. when I fetch from Eric for git-svn updates with

	$ git pull git-svn master
because I do have
	[remote "git-svn"]
                url = git://yhbt.net/git-svn
                fetch = +refs/heads/*:refs/remotes/git-svn/*
defined, it is Ok to update refs/remotes/git-svn/master (but not others).
On the other hand, if my refspecs for "git svn" _were_ like this:
	[remote "git-svn"]
		url = git://yhbt.net/git-svn
                fetch = +refs/heads/master:refs/remotes/git-svn/master
then I would _not_ want this:
	$ git fetch git-svn dev
to create a new tracking branch refs/remotes/git-svn/dev.

It used to be that the only way to check the progress of other people were to do this:

	$ git fetch git-svn master
	$ git log git-svn/master..FETCH_HEAD

But these days, even if we changed the "git fetch" semantics, we can still rely on reflogs to do the equivalent with:

	$ git fetch git-svn
	$ git log git-svn/master@{1}..git-svn/master

In other words, I think the "feature" that an explicit "fetch but do not store this time" request to prevent "git fetch" from updating the tracking branches outlived its usefulness.

Jeff King· Aug 27, 2009, 21:39 UTC · re: Junio C Hamano · lore

Re: Question regarding git fetch

On Thu, Aug 27, 2009 at 02:20:46PM -0700, Junio C Hamano wrote:
Show 13 quoted lines
> I think it is reasonable, in 1.7.0, to change "git fetch" with command
> line refspacs that lack colon (i.e. saying "fetch this ref" without saying
> "and store it here") so that it updates remote tracking refs if and only
> if an appropriate remote.$remote.fetch is configured to do so.  E.g. when
> I fetch from Eric for git-svn updates with
> 
> 	$ git pull git-svn master
> 
> because I do have
> 
> 	[remote "git-svn"]
>                 url = git://yhbt.net/git-svn
>                 fetch = +refs/heads/*:refs/remotes/git-svn/*
Does the colon really matter? That is, should
  $ git fetch git-svn master
store a tracking ref, but
  $ git fetch git-svn master:foo

not? In both cases, git has learned during the course of an operation what the new value of the tracking ref could be, so why not store it?

The obstacle I see is that somebody may be using configured refspecs as something other than a tracking ref. That is, they care about _when_ those refs on the RHS of the refspec are updated, and are not treating them just as a cache of what the remote side has.

Show 11 quoted lines
> On the other hand, if my refspecs for "git svn" _were_ like this:
> 
> 	[remote "git-svn"]
> 		url = git://yhbt.net/git-svn
>                 fetch = +refs/heads/master:refs/remotes/git-svn/master
> 
> then I would _not_ want this:
> 
> 	$ git fetch git-svn dev
> 
> to create a new tracking branch refs/remotes/git-svn/dev.

Of course not. It's not in your configured refspec, so you have requested no such tracking ref. I think we would have to limit ourselves to how the tracking refs are defined in the refspecs. Otherwise how would we know to put the tracking ref in refs/remotes/git-svn and not some other place?

So I think it is a simple matter of applying the configured refspecs with information we happen to have gotten during the course of a somewhat-related operation.

-Peff
Tom Lambda· Aug 28, 2009, 13:24 UTC · re: Tom Lambda · lore

Re: Question regarding git fetch

Thank you all for your answers and the interesting following discussion.
>From a user perspective (I do not know how git works internally), the

approach proposed by Juno looks very intuitive to me. Now I understand there are other cases and users to take into account, which make the implementation complex. I hope this will be part of git 1.7.0.

--Tom
-- 
View this message in context: http://n2.nabble.com/Question-regarding-git-fetch-tp3527289p3534609.html
Sent from the git mailing list archive at Nabble.com.

← back to recent threads