Random thoughts on "upstream"

I do agree that separating

 - where to push to, and
 - where to fetch for integrating with

is a necessary step for supporting triangular workflow better.

The "upstream" is that on top of which you build your work. You clone from there to bootstrap yourself, you add your work (which may include integrating the work of your contributors, if you are a mid-tier maintainer/integrator aka a lieutenant) on top of it, and arrange the result to reach the "upstream" in some way.

For the simplest (and still widely used) workflow that employs a central shared repository, the way to make the result to reach the "upstream" is by directly pushing into it yourself. In that sense, the word "upstream" and the traditional behaviour of "git push" that pushes back to the 'origin' (or branch.$name.remote) to update your "upstream" (or branch.$name.merge at 'origin') both make perfect sense.

Also, if you are rebasing, @{u} refers to that place you integrate with, i.e. your "upstream", in the central shared repository workflow.

But in a triangular workflow, the way to make the result reach the "upstream" is *not* by pushing there yourself. For developers at the leaf level, it is to push to their own repository (often on GitHub), which is different from where they (initially) clone from in order to bootstrap themselves, and (subsequently) pull from in order to keep them up-to-date. And then they request the published work to be pulled by the "upstream".

Note that even in a triangular workflow, @{u} should still refer to the place you integrate with, i.e. your "upstream", not to the place you push to publish the result of your work.

There is an interesting twist. For a maintainer/integrator (either a mid-tier one who has "upstream" somebody else controls, or the top-level one whose conceptual "upstream" is himself), "git pull" does not necessarily always go to the "upstream". You also respond to pull requests made to you, by those who consider you as their "upstream". As the antonym of "upstream", "downstream" would be a good word to refer to them (and that is why calling the place you push to "downstream" is not a good idea). The integrator responds to such a pull request by running "git pull" against his "downstream".

Because an integrator will pull from more than one "downstream" by definition (otherwise he is merely a relay or a gatekeeper), a pull request typically carries the full URL and the branch name to be pulled from, and there is not much need for an integrator to have a short-hand mechanism to allow "git pull" (without other arguments on the command line) to go to a "downstream".

But for an integrator, the place they consider their "upstream" is not the place they push to publish. A mid-tier integrator aka lieutenant has exactly the same issue as leaf developers do on the "push" side.

In a triangular workflow, "git push" that goes to 'origin' does not make sense, and remote.pushdefault (and branch.$name.pushremote) solves a half of the problem by allowing them to name the repository they publish their work as the destination of a push.

The other half of the issue is that there is no good way to say "The result of my work on this branch should go to that branch in the repository I publish to; they are not named the same".

The value "upstream" for push.default does not make sense in the context of a triangular workflow, as you may well base your work on 'master' from the upstream, which is a branch with a very generic purpose (e.g. "to advance the state of the overall project"), but your work may have a more specific purpose (e.g. "to improve frotz feature") that deserves to be on a branch named after that specific purpose in the repository you publish your work on. That is, after working on your 'frotz' branch this way:

    git checkout -t -b frotz origin/master
    work work work, commit commit commit

you do not want to push the result as 'master'; you want to call it 'frotz' in the published repository.

The value "current" for push.default dictates that the work you did on your branch 'frotz' must be published as the same name 'frotz', which may be good enough.

For those who do not find it good enough, a possible improvement is to introduce another mechanism that tweaks the remote.publish.push refspec. I am thinking aloud here, but imagine that you are working on your 'frotz' branch with this configuration:

    [remote "origin"]
    	url = ... this is where your upstream is ...
        fetch = +refs/heads/*:refs/remotes/origin/*
    [remote "publish"]
        url = ... this is where you push to ...
        ;; push = +refs/heads/*:refs/heads/topics/*
    [remote]
        pushdefault = publish
    [push]
        default = single

    ; the remainder is per-branch
    [branch "frotz"]
        remote = origin
        merge = refs/heads/master

and you eventually want to push the result to topics/frotz branch there.

In a triangular workflow, when you need to depend on some feature that is only available in the updated upstream, you would rebase your work on your upstream, e.g.

    git pull --rebase

which would go to branch.frotz.remote (= origin) and rebase your work on top of their master. Before such a rebase, you could

    git fetch
    git log ..@{u}

to see what new stuff you could exploit if you did rebase now.

Up to this point, there was no need to change anything; the current Git does the above perfectly fine. But then you try to push:

    git push

Thanks to the recent introduction of remote.pushdefault, this push will no longer go to 'origin' but goes to 'publish'. But not necessarily to the desired branch.

With 'upstream', it may try to go to 'master' (as branch.frotz.merge is set there), and with 'current', it may go to 'frotz'.

Uncommenting

    [remote "publish"]
        push = +refs/heads/*:refs/heads/topics/*

will let "git push publish" (or "git push") push 'frotz' to update 'topics/frotz' branch over there, which is what we want in this example, but it will also push other branches at the same time, which is not what we want.

Now I have a curious value "single" in the above configuration that is not understood by current Git. It is to trigger a new rule to modify the way remote.publish.push refspec is used:

    When on branch 'frotz', where pushremote_get() would return
    'publish', find where refs/heads/frotz would be pushed with the
    refspec for that remote (i.e. refs/heads/*:refs/heads/topics/*
    maps refs/heads/frotz to refs/heads/topics/frotz) and push only
    that, i.e. update refs/heads/topics/frotz over there with the
    tip of 'frotz' we are pushing.

That may be a solution for those who do not find 'current' good enough.

On Thu, May 16, 2013 at 12:55 PM, Junio C Hamano <gitster@pobox.com> wrote:

If the user has branch.autosetupmerge=always, that's not correct; even doing:

% git checkout -b frotz origin/master

Would setup the upstream. And I believe for v2.0 we do want branch.autosetupmerge=always, but we might not want to always override the push location.

What happens if I want to push to 'refs/heads/topics/frotz-for-juno'?

Felipe Contreras <felipe.contreras@gmail.com> writes:

You would weigh pros-and-cons of supporting such a "single branch only" special case, and add a branch level override, and if the benefit outweighs the cost of complexity, design and implement it.

The push.default setting is to make sure we have a simple mechanism to cover more common cases, and my suspicion is what 'current' gives us is already there without the need for 'single'.

On Thu, May 16, 2013 at 6:17 PM, Junio C Hamano <gitster@pobox.com> wrote:

I already implemented it. I already sent the patch.

Junio C Hamano <gitster@pobox.com> writes:

Actually, I suspect that you shouldn't even need to do that pros-and-cons analysis, because the 'single' thing should cover as a natural extension of the existing infrastructure. You should only need to have something like this:

	[remote "there"]
        	url = ... were you push ...
		push = refs/heads/frotz:refs/heads/topics/frotz-for-juno
                push = refs/heads/*:refs/heads/topics/*

Without the 'single', your 'frotz' will be pushed to update heads/topics/frotz-for-juno, not heads/topics/frotz, because the exact refspec match will prevent it from matched twice by the wildcarded one. The imagined 'single' mode would just limit the push to the current branch, so it would end up pushing to the branch you want to update, without sending an extra copy to the same name.

On Thu, May 16, 2013 at 8:31 PM, Junio C Hamano <gitster@pobox.com> wrote:

And would 'git branch --set-downstream-to github/frotz-for-juno' do that? If not it's basically useless for 99% of the users who never fiddle with push refspecs.

On Thu, May 16, 2013 at 8:59 PM, Felipe Contreras <felipe.contreras@gmail.com> wrote:

And 'git branch -vv' would not show that either. Nor can the user do '@{downstream}'.

Junio C Hamano wrote:

Which is the exact argument I presented on the other thread. However, Felipe has a point: we shouldn't cripple @{f} (I think "fork" is a good name for it) in the name of generality. Think of what remote.<name>.push and branch.<name>.push are actually going to be used for:

remote.<name>.push are most probably going to be used with wildcards (like in the Gerrit case): I don't mind if @{f} is not able to tell me where my branch will go (and it's probably even a pseudo-ref, in which case the answer is really meaningless). The limitation doesn't affect remote.<name>.fetch, because @{u} only operates between remote branches and local branches, not at the ref-mapping level.

branch.<name>.push cannot be used with wildcards, and @{f} is pretty straightforward to determine in this case. When the information is easily available for end-user consumption, why would we _not_ want to expose it by stuffing branch.<name>.push into remote.<name>.push configuration?

So, I think the way forward is:

1. Introduce @{f[ork]} to show $(branch.<name>.pushremote or
remote.pushdefault)/$(HEAD).  We can update 'git status' to show this
information as well: the ahead/ behind with @{f} is no less important.

2. (optional) Make remote.<name>.fetch non-mandatory.  Currently, a
lot of git-core breaks unless this is present, and this is an annoying
wart.

3. Introduce remote.<name>.push.  Make it clear that it is intended
for the Gerrit-user.  @{f} cannot be determined with this [*1*].

4. Introduce branch.<name>.push for a branch-specific override to
remote.<name>.push.  @{f} can be determined when this is in effect.

[Footnotes]

*1* Do we want to do some magic wildcard-expansion to get/set @{f} anyway? In which case, branch.<name>.push is really not necessary. Which approach is less ugly?

Ramkumar Ramachandra <artagnon@gmail.com> writes:

Please clarify the semantics of @{f}. Does it conceptually refer to where the current branch is going to be pushed to (i.e. a pair of (<remote>, <ref>))? Will we have a remote tracking branch for it to record what we pushed there the last time? I am guessing that your answers to both of these questions are "Yes", and frotz@{f} would resolve to refs/remotes/there/topics/frotz-for-juno in the sample set-up in the message you are responding to.

	Side note: I do not think "fork" rings bell to the end
	users.  Who is forking from what?  I am guessing you are
	trying to make a short form of "the branch in my public
	pepository I push this branch to, and other people would
	consider it my fork of the upstream project", but it is hard
	to do the reverse, i.e. a new person who is presented a word
	'fork' to guess that you wanted to refer to the above by
	that word.

I am not saying branch.<name>.push should not exist. Having a finest-grained special case would be an escape hatch you can use when you want to do something unusual.

But it has exactly the same issue as branch.<name>.pushremote; adding it without having the single "all of my pushes go to here, not to 'origin'" would have meant that for N branches you have to set the same thing N times. We fixed it with remote.pushdefault before the series graduated. If you only add branch.<name>.push, then people have to configure it N times, for N branches they want to push out.

Reusing the existing push refspecs was just a suggestion to solve that issue, and I am not married to that particular design. You or Felipe may be able to come up with a better alternative to achieve the same goal and that is perfectly fine. I just wanted to make sure that we do not force the user to repeatedly set the same thing over and over in the common case.

By the way, about "crippling".

I do not think of a reason why you cannot implement that @{f} with the 'single' matching (or its better version you may come up with). If "git push" can figure out where it would push to, you certainly should be able to borrow that same logic to see what tracking branch you are locally using to track the last push result for the current branch in response to @{f} request, no?

Junio C Hamano wrote:

Yes.

GitHub is an overwhelmingly popular service, and many end-users are used to clicking on the "Fork" button on various projects to contribute. Here, "fork" is short for "my fork". Do you have a better name in mind?

Oh, I'm completely against just adding branch.<name>.push as I've pointed out on the other thread. Even in the part you clipped out, I clearly stated remote.<name>.push above a branch-specific thing in the priorities.

Ofcourse.

Ofcourse, I'm not saying it's not possible.

1. Getting @{f} requires extra computation, and that might be ugly/
undesirable/ surprising considering how @{u} doesn't require it.

2. Setting @{f} with branch --set-fork-to won't operate on the
branch.<name> section, and this might be surprising.

3. If remote.<name>.push is only going to be used by the Gerrit
people, @{f} is not going to work anyway.

These issues aren't deal breakers, but are certainly worth mentioning.
 Frankly, I'm not overtly fond of the branch.<name>.push idea, and am
tilting towards this now.  What do you think?

Ramkumar Ramachandra <artagnon@gmail.com> writes:

In my original "Random thoughts", I deliberately avoided introducing a new name for the publishing point, because I did not think of a good name myself ("publish" for "what I publish this as?" was what I had in mind, and while it is better than "downstream" which this thing clearly isn't, I do not think it is a great name).

As I said number of times, I am not good at naming, even though I can tell when a proposed name (like "publish") is not a good one X-<.

Matching it with the way to set this per-branch might be a good alternative. If branch.<name>.push is it, then it would mean "@{push}".

This is minor, but I am not sure "doesn't require it" part should always stay true in the first place. If you are always forking your 'master' from the upstream 'master' and your 'next' from the upstream 'next', there is no real reason why the user must record an obvious value in branch.master.merge and branch.next.merge variables.

Hmm, why?

As I said, I do _not_ mean branch.<name>.push must _not_ exist.

Setting it explicitly with --fork-to or whatever option is perfectly fine; the logic to resolve @{f} notation would honor it naturally if you reused the same logic as "git push" decides where to push, no?

I do not understand what you mean by this.

I think I'd better rephrase what I wanted to say in the message you are responding to.

 * "branch.<name>.push" would be a way to set per-branch push
   destination. There may be other ways to do the same but I do not
   think it is bad and it probably will fall into bikeshedding
   category to attempt to come up with something different for the
   same purpose of setting per-branch destination.

 * There should be a mechanism to allow people _not_ configure it
   per-branch if they do not want to push to randomly custimized
   destinations per-branch (i.e. want to always push $branch to
   refs/heads/$branch, or refs/remotes/satellite/$branch).

So that is what I think.

Having said that I am not sure where your "not overly fond of" comes from, as I do not see a problem with branch.<name>.push. The only problem I may have with the approach would arise _only_ if we make it the sole way to allow people push to different names, forcing people who want to push N branches to configure it N times. Other than that, I think it is an acceptable solution to give per-branch control.

Junio C Hamano wrote:

It doesn't strike me as being elegant at all.

[branch "hot-branch"]
    remote = ram
    push = refs/heads/for-junio/hot-branch

1.  I git branch -M for-each-ref and push.  Screwed.

2.  Let's say you give me the thumbs up to write hot-branch to origin.
 Excitedly, I change the remote to origin.  Now I already have push
semantics for origin (you obviously want me to write to rr/, so I can
cooperate with others):

[remote "origin"]
    push = refs/heads/*:refs/heads/rr/*

Unfortunately, it doesn't kick in unless I change branch.<name>.push.

I guess what I'm saying is: refspec semantics are inherent properties of the remote, not of the local branch. Since there is no intermediate refs/remotes/* sitting between the remote and local branches, this is _not_ like branch.<name>.merge at all. That is why I'm tiling towards remote.<name>.push: we're not giving up any functionality anyway; there's nothing we can't do without branch.<name>.push.

Ramkumar Ramachandra wrote:

[remote "ram"]
    push = refs/heads/link:refs/heads/for-junio/link

is saying: if the branch name matches "link", push it to for-junio/link.

[branch "link"]
    push = refs/heads/for-junio/link

is saying: push _this_ branch to for-junio/link, irrespective of what it is called.

An example illustrating this clearly:

# on branch link-v2 # work ... $ git push # work ... # ok, I'm ready to replace link $ git branch -M link $ git push # where should the push go?

Also: putting branch-specific configuration in remote.<name>.push
would mean that it can potentially accumulate a lot of cruft from
deleted branches.  It's the same kind problem we face with .gitignore,
no?

On Sat, May 18, 2013 at 3:07 PM, Ramkumar Ramachandra <artagnon@gmail.com> wrote:

Exactly where you told it to go.

Not many people would use 'remote.<name>.push' and in very specific circumstances. It's not a problem.

On Sat, May 18, 2013 at 1:27 PM, Ramkumar Ramachandra <artagnon@gmail.com> wrote:

s/remote/pushremote/

Can't parse that.

If course it doesn't that's what you told git to do. Of course, you could simply do:

% git branch --set-downstream-to origin/rr/for-junio/hot-branch

Or if you are going to push multiple branches and set a 'remote.<name>.push':

% git branch --unset-downstream

Yes there is, I thought you already acknowledged that:

% git branch --set-downstream-to github/fc/master % git rebase -i @{downstream}

Ramkumar Ramachandra <artagnon@gmail.com> writes:

-ECANTSEEWHATYOUMEAN.  Your proofreader is screwed ;-)

I thought branch.$name.push is "What ref do I push this out as?"

At this point you have two remotes, me as your upstream, and your publishing point, and remote.pushdefault set to the latter since originally you cannot push to 'origin':

    [push]
        default = simple ;# adopt 2.0 default early
    [remote]
        pushdefault = ram

    [remote "ram"}
	url = github.com:ram/something.git
        fetch = +refs/heads/*:refs/remotes/ram/*
        push = +refs/heads/*:refs/heads/*

    [remote "origin"]
        url = git://git.kernel.org/pub/scm/git/git.git
        fetch = +refs/heads/*:refs/remotes/origin/*

    # assuming you forked from my master
    [branch "hot-branch"]
        remote = origin
        merge = refs/heads/master
   +    pushremote = origin
   +    push = refs/heads/rr/hot-branch

The above pseudo-diff would be the changes you would be making to your configuration when you were asked to push your 'hot-branch' to 'origin' as 'rr/hot-branch', I think.

If somebody implements the "push.default = single" (see the original message you are responding to), then the change might be smaller.

    [push]
   -    default = simple ;# adopt 2.0 default early
   +    default = single

    [remote]
        pushdefault = ram

    [remote "ram"}
	url = github.com:ram/something.git
        fetch = +refs/heads/*:refs/remotes/ram/*
        push = +refs/heads/*:refs/heads/*

    [remote "origin"]
        url = git://git.kernel.org/pub/scm/git/git.git
        fetch = +refs/heads/*:refs/remotes/origin/*
   +    push = +refs/heads/*:refs/heads/rr/*

    # assuming you forked from my master
    [branch "hot-branch"]
        remote = origin
        merge = refs/heads/master
   +    pushremote = origin

Somewhat true, if you write this with branch.hot-branch.push without push.default=single implicitly introduces

	refs/heads/hot-branch:refs/heads/rr/hot-branch

refspec to the remote named by branch.hot-branch.pushremote.

Junio C Hamano wrote:

I think this is a bad hack covering up an underlying problem: it is ugly, confusing, and inextensible in my opinion. I think of push.default as merely a default remote.<name>.push value for all remotes, that can be overridden on a per-remote basis.

I suspect that the issue you're trying to address is:

[remote "ram"]
    push = refs/heads/*:refs/heads/rr/*

not dictating which refs to push when I say 'git push' (it'll push all the refs under refs/heads/*, not respecting push.default=current in my scheme). That is a valid problem, and it is a problem with our refspec: I can say "HEAD" (which is what push.default=current is) to mean current branch, but I can't dictate the rhs of the refspec then. In other words, I cannot have the refspec "HEAD:refs/heads/rr/%1", and _this_ is the problem.

What do you think?

On Sun, May 19, 2013 at 6:44 AM, Ramkumar Ramachandra <artagnon@gmail.com> wrote:

You can't represent push.default = single either.

Felipe Contreras wrote:

Right. And I propose that we extend the refspec to be able to represent it, instead of having "single" sticking out like a sore thumb (and possibly introducing more sore thumbs like this in the future).

On Sun, May 19, 2013 at 6:54 AM, Ramkumar Ramachandra <artagnon@gmail.com> wrote:

Yeah, go ahead and have fun making refspecs Turing complete. Why? Because you decided to call something a "sore thumb". Calling something a "sore thumb" doesn't make it so.

We, the sane people, will people will keep using simple configuration options. Hopefully some day there won't be much need for many of the current configurations, including push.default.

Ramkumar Ramachandra <artagnon@gmail.com> writes:

That is not what I was addressing. You outlined your scenario as "you were not, but you are now, allowed to push an approved ref into 'origin'". And you do so under a different name. That is why I set that rr/ renaming push refspec for a remote ORIGIN not RAM.

And that was done with extensivility your example implied in mind: you may later be allowed to push other branches as well to origin. That is why the refspec definition for 'origin' does not hardcode the name of the branch you are permitted to push there at this moment. The fact that hot-branch goes to origin is encapsulated in the branch.hot-branch.pushremote. The rule, under which the name of any branch that goes to the origin is renamed, is encapsulated in remote.origin.push refspec (the introduction of the new mode "push.default = single" is necessary to make this work).

When making that change, our fictitious ram did not have to touch "remote.ram.push" *at all*. Independent of what the owner of "origin" and ram agreed in that example, ram would keep doing exactly the same thing to his own publishing point so that people who are working off of his work would get updates from the place known to contain his work from before.

So with "git push ram", it will push everything to "ram" under the same name *without* rr/ renaming, but that was *by design*, not something I wanted to or I needded to work around. And you would also push to "origin" by doing "git push" while on hot-topic branch.

Junio C Hamano wrote:

My problem with this entire scheme _is_ the magic push.default = single. Currently, push.default only kicks in when no remote.<name>.push refspec is specified (in other words, it is a default value of remote.<name>.push for all remotes), and I don't think we should change this. If the user wants to configure the push refspec (either for rewriting, or for determining what to push), there is exactly one thing to change: remote.<name>.push. So I can have:

[remote "theodore"]
    push = master
    push = +pu

This means that I will always push master without force and pu with force, irrespective of the branch I'm on.

[remote "origin"]
    push = refs/heads/*:refs/heads/rr/*

This means that I will always push all branches without force with rewriting, irrespective of the branch I'm on.

[remote "ram"]
    push = HEAD:refs/heads/rr/%1

This means that I will always push the current branch without force, with rewriting.

[remote "felipe"]
     # empty

This means that remote."felipe".push falls back to the refspec specified by push.default.

Isn't branch.<name>.push is completely unnecessary? Does this make sense to you? Isn't it more straightforward and general (how do I get a push.default = single on a per-remote basis) than your solution?

On Thu, May 23, 2013 at 5:42 AM, Ramkumar Ramachandra <artagnon@gmail.com> wrote:

No, it's not; 'git push --set-downstream' is not going to configure that for the user. Nor will the user be able to see the downstream with 'git branch -vv', nor will the user be able to see the downstream with 'git rev-parse branch@{downstream}'.

You keep ignoring those use-cases I already mentioned multiple times because they don't fit your idealistic model.

Ramkumar Ramachandra <artagnon@gmail.com> writes:

Exactly. And some people are unhappy about it, because they prefer to work per-branch basis, as opposed to having to perfect everything into a publishable state first and then finally push everything out in one go.

I am not saying default=single would be _the_ single right way to solve it, but "when you have default=single, remote.$name.push is used only to describe the mapping, not forcing you to push everything out at once" is one possible solution to that.

Junio C Hamano wrote:

The big advantage it has, in my opinion, is ease of implementation: you'd essentially have to grab the remote.<name>.push refspec, rewrite it to replace refs/heads/*: with $HEAD: and feed that refspec to the transport layer. Compare that to inventing a new refspec syntax, which is a mammoth task. We can always extend the refspec later, if we want that. I wonder if it makes sense to bake the functionality into current though: will we be breaking anything?

(My opinion has changed after wrestling with the horrible transport layer)

Ramkumar Ramachandra wrote:

Um, sorry.  It involves two independent steps:

1. add_refspec() $HEAD:$HEAD@{push} to feed to the transport layer.
(@{push} is already in progress; see list)

2. Forcing the transport layer to ignore remote->push and use the fed
refspec instead.  (I'm not sure how to do this yet)