Re: [RFD] making separate-remote layout easier to use

Now separate-remote layout is the default for newly cloned repositories, I think it is a good time to make further effort to make things easier to use. Here are some of the ideas off the top of my head.

* Do not change what branch the sole local branch is named.

In the current implementation, we find out what branch the remote HEAD points at, and match the name of the sole local branch to it. HEAD of most central distribution point repositories that use git with the default configuration have HEAD pointing at refs/heads/master and a cloned repository would continue to use refs/heads/master as the sole local branch when it starts as before, but some projects may use alternate wordings such as 'main', and a new repository would have refs/heads/main branch when cloned from such a repository instead. I think this behaviour is a reasonable default, so let's keep this.

Distribution point repositories are of interest here, because that is what most people clone from, and this message is about making the default easier to use by those casual cloners.

* Change the default contents of $GIT_DIR/remotes/origin

Currently we list all the remote branches that exist when the clone is made, making sure that the branch pointed at by HEAD at the remote is listed first. The intent is to track every branch at the remote but merge in the "primary" branch to us.

Two issues that have been raised about this is:

 - the intent to track every branch is good, but the current
   implementation means we would not notice new branches at the
   remote.

 - merging the "primary" branch is good only when the user is on
   the corresponding "primary" branch.  It is usually a wrong
   thing to do when on another branch.

The first issue can be solved, with the help of recent "glob pattern refspec in fetch" from Andy. I am thinking about making the default contents of $GIT_DIR/remotes/origin to be:

	URL: <url of the cloned repository>
	Pull: +refs/heads/<primary>:refs/remotes/<origin>/<primary>
	Pull: +refs/heads/*:refs/remotes/<origin>/*

to address this issue.

	Side note: <primary> is what HEAD pointed at at the
        remote when the clone was made, and <origin> is usually
        'origin' but "git clone -o $origin" can override it.

Forcing with '+' is debatable, but with separate-remote layout, remotes/*/ hierarchy is to track what the remote has, and you cannot do much else other than noticing and warning when the remote end does funny things to its refs anyway, so I think having '+' might be a better default.

The right thing to do to address the second issue is less clear. If the "upstream" has two more-or-less equally prominent branches, say 'main' and 'test', it may make sense to use corresponding two branches on the local side and merge 'main' from the remote when on local 'main' and merge 'test' when on local 'test'. Even when dealing with a specific topic branch, that would hold true for most of the time. A topic branch refs/heads/bug#2073 to work on the bug 2073 can be published at the central distribution point. The proposed updates to $GIT_DIR/remotes/origin file would track it with refs/remotes/origin/bug#2073 and interested people can create a local branch refs/heads/bug#2073 from it and work on it, which makes it easy to polish a topic branch in a collaborative way.

I am not sure if 'merge in corresponding branch' is the only valid workflow, however. I am reluctant to make the system automatically do so if the solution makes other workflows more painful to follow. Automatically merging remotes/origin/$foo when on $foo branch is not good enough, in other words (also, there may be a hierarchy under remotes/ other than origin). It might make sense to introduce "Merge: " in remotes/ file and if they are present use "Pull: " only to decide what are fetched and use "Merge: " to decide what is merged (if we were doing the system from scratch, the former would have been named "Fetch: " but it is too late now).

Junio C Hamano wrote:

[...]

The question is: do we continue to use remotes/ file, or do we
save remotes info in the config file: remote.<name>.url,
remote.<name>.fetch, remote.<name>.push and branch.<name>.merge
(in our case '[remote "origin"]' section)?
 

I hope that it also works with the remote.origin config file section instead of $GIT_DIR/remotes/origin

Perhaps, perhaps not. It would be nice to have configuration option that would tell that history of given branch is being changed, and the ability to ask about it remotely, so git-clone would be able to add this + _when needed_ automatically.

But it's a fact that with separate remote the need to use fast-forward check is lessened, and it might be more important to not confuse first time user with having to modify $GIT_DIR/remotes/origin or remote.origin config section to fetch from the repository he/she cloned from.

If you add "Merge: " in remotes/, then please add it also in remote section in config file. Config file has now branch.<branchname>.merge (and it would be nice if clone would set ou this for local branches corresponding to remote branches), but it is not the same.

Jakub Narebski <jnareb@gmail.com> writes:

It is not "the question"; it is irrelevant because $GIT_DIR/remotes/origin and [remote "origin"] are pretty much interchangeable, and will hopefully continue to be.

        [remote "origin"]
                url = master.kernel.org:/pub/scm/git/git.git/
                fetch = +refs/heads/master:refs/remotes/origin/master
                fetch = +refs/heads/*:refs/remotes/origin/*

would be the way to express the equilvalent of the example in the previous message.

Junio C Hamano wrote:

Well, the question is if git-clone would generate appropriate config entries in addition to or instead of $GIT_DIR/remotes/origin file...

By the way, what do you think about extended refspec syntax, namely in addition to <remote name>:<local name> it would allow to say <remote name>:<local name>:<local branch to merge into>?

Jakub Narebski <jnareb@gmail.com> writes:

What I meant by "irrelevant" is that it is purely syntax. I would rather nail down the desired semantics first, and then worry about the syntax. We have done remotes/origin file by default and allowed the same thing specified in configuration file for people who prefer configuration file, and I do not see an immediate need to change that.

Only if/when the desired semantics turns out to be too complex to be expressed in the remotes/ file scheme and at the same time if configuration file is easier to express the same information, we should start talking about favoring configuration file over remotes/ file, and perhaps generating the config entries by default without generating remotes/.

In either case, I do not think generating _both_ by default is a bad idea. They will become out of sync and the user either need to delete one or the other to avoid confusion.

I do not think it is reasonable to assume that one remote branch is to be always merged into one fixed local branch, so that is too limiting as an extension. In the configuration file, I think we already allow the default remote to be fetched from and remote branches to be merged per branch statically. A reasonable thing to do is to see how satisfied users are with that facility, what they find missing there. Worrying about the syntax should wait until we know what kind of things need to be expressed; that can be done both by extending the existing configuration and/or introducing a new entries in remotes/ files but the point is that is mere implementation detail.

Junio C Hamano <junkio@cox.net> writes:

Eh, excuse me.  "generating both is a BAD idea" is what I meant...

Junio C Hamano <junkio@cox.net> wrote:

I'm all for:

 * changing the default made git-clone to be [remote "<origin>"]
 * continuing to support parsing of $GIT_DIR/remotes/*

We moved away from the $GIT_DIR/branches directory to $GIT_DIR/remotes, yet we still support $GIT_DIR/branches in remote handling code. I see no reason why we cannot start to use remote.<name>.url by default while continuing to support the older branches and remotes formats.

For one thing the newer remote.<name>.fetch seems to make more sense to new users than Pull: lines do. For another it appears to be supported since v1.4.0, which was released June 10th. Most users cloning a repository with >1.4.4.1 will probably only use 1.4.0 or later on that same repository, so there is probably low risk of breakage due to the remote not being recognized by a pre-1.4.0 Git.

Shawn Pearce <spearce@spearce.org> writes:

IIRC, git never used "branches", so there was no "moving away". It was merely to interpret what cg-init left in the repository. Since the very early days remotes/ has been _the_ format git used, and I do not see any strong reason to change it now. The continued support for branches is an irrelevant topic and is not a justification to favor configuration over remotes/ nor the other way around.

That is quite subjective.

Again, "low risk of breakage" does not justify switching.

As I said in the other message, I think configuration vs remotes/ is futile subject and premature topic to talk about before nailing down what semantics we would want. Either format can have the expressive power, once we know what we want to express.

Jakub Narebski <jnareb@gmail.com> wrote:

Yes, it does.  Fortunately Git is relatively consistent.  :-)
 

Yes. From an out-of-the-box-make-Git-appear-to-be-easier-to-use point of view tossing + into the Pull:/remote.<name>.fetch setup might seem like the right thing to do. It lets the end-user blindly follow the upstream repository they cloned from. Almost.

The problem becomes when the user makes a topic branch off say Junio's `pu` branch and later after doing a fetch and looking at the log of `remotes/origin/pu` decide to pull the updated `pu` into their topic branch, so they can continue testing. But now they are in merge hell as Junio has completely rebased `pu` and what was there isn't, and what's there now ain't compatible... and the new user curses at Git.

Needing to put + in front of a refspec (or needing to fetch it with --force) is the user agreeing that something _evil_ is going on with the upstream and that they acknowledge this may cause problems for them locally.

I would prefer to see the upstream be able to publish a short description of each branch, where the repository owner can describe the policy of that branch, as well as have a machine readable setting on each branch indicating if that branch will be rewound from time to time, or never rewound.

git-clone should skip rewinding branches by default, unless the user
adds an option (e.g. --include-rewinding-branches).  This way new
users to git.git don't get the `pu` branch unless they really mean
to get it, at which point they have hopefully also read the upstream's
description of the `pu` branch and its rewinding policy, and can
at least start to grasp what is going to happen if they start to
work with the branch.
 

I'm against adding anything to the remotes/ file format.

We already have branch.<name>.merge to indicate what the default source for a git-pull on the branch named <name> should be. git-branch probably should fill that entry in when a branch is created from a remotes ref.

Shawn Pearce <spearce@spearce.org> writes:

I like this approach very much.

Junio C Hamano <junkio@cox.net> wrote:

I think the idea of glob'ing off the remote during clone (and using the glob for future fetches) is a good idea. So I think what I'm suggesting above is to make glob'ing by default not copy the rewinding branches, unless the glob spec itself starts with '+' (which is what say --include-rewriting-branches would do).

So now we're at a point of:

 * how do we get this branch information from the remote?
 * how does the remote store this branch information?

I'm leaning towards the repo config for the latter, with say:

	[branch "pu"]
		rewinds = true
		description = This branch contains some proposed updates to git.\n\
	It rewinds often, as different updates are proposed or merged into 'next'.

but looking at that think its rather horrible looking.  :-)

However for the former it may be useful if the client can download the repo config from the remote repository, such as to fetch the branch description data.

Shawn Pearce <spearce@spearce.org> writes:

For the former, I was hoping that we could do a new action against remote repository that is "get remote information". Just like we have "ls-remote" as a protocol neutral front-end to get what git native protocol would give "peek-remote", git native protocol would have get-repository-info action to transfer this information, and http and dumb transports do a file download from a known file, just like they download files prepared with update-server-info. Most likely, that get-repository-info is just a glorified single file transfer and nothing more.

As an implementation detail, I think it is one possibility to store this information in the config of the repository being cloned from. We should be able to use repo-config to parse that file in config-like syntax to extract what we would want, so we do not have to come up with a new syntax nor a new parser, which is a big plus.

	Side note: repo-config, especially the updating side of
	it, is one of the programs I find the most brittle in
	the whole system, and it always makes me hesitate when
	somebody proposes to use config file in any important
	way read-write for this reason.  But this "reading out
	per-branch information from the file" is read-only so I
	would not worry too much about it.

Strictly speaking, however, the config file is a wrong place to store it. For one thing it has core.sharedrepository and receive.denynonfastforwards that are true configuration to control the behaviour of git _at_ _the_ _repository_ the configuration is at. The new "branch property" are primarily to help the other end, and the "filtering rewinding ones" we want at the clone/fetch side wants that information only and not interested in the true configuration information at that repository.

So my preference is:

 * store it in a fixed filename under $GIT_DIR/$something, in a
   format that is similar to the true config file.

 * Dumb transports would just do a file transfer, and we add a
   new command (like "upload-pack", "upload-archive") for git
   native transport to read that file.  We might want to have a
   protocol neutral wrapper (just like "ls-remote").

 * Cloners and fetchers would do the file transfer and use
   repo-config to read from that transferred file to find out
   which are rewinding branches.

Junio C Hamano <junkio@cox.net> wrote:

Yes, that is completely sane.  :)
 

Also quite sane. Nobody wants to write new code if we don't really have to. :)

Now you tell me. git-gui uses repo-config for all of its option data. It writes back to the config file to save user configured options, including window layout at quit. It also uses both the user global and the local repository config files. :)

We can safely update the index, symrefs and refs. Why can't we also safely update the config file? The biggest problem is its also a hand-modified file complete with support for comments, so users expect the file to stay intact during edits. Aside from that little issue its really a fairly simple format...

Yes, of course. I never suggested copy the entire configuration as is. What we really want to do is take the branch data from the remote and rewrite it for every branch we fetched into a ref, e.g.:

  remote:
    [branch "master"]
        rewinds = false
        description = blah blah

  local:
    [branch "remotes/origin/master"]
        rewinds = false
        description = blah blah

while skipping other configuration sections and options that have no meaning for the local repository. core.sharedrepository is just one of *many* configuration options that the local repository should not inherit. remote.<name>.url is another... etc.

At which point you might as well put it into $GIT_DIR/config.
People are going to use git-repo-config to edit that file, even if
you say its brittle and maybe shouldn't be used for heavy editing.
Its just damn handy.
 
Besides of which, lets not forget that something like:

  [branch "master"]
      rewinds = false
  [branch "pu"]
      rewinds = true

is not only data for the client to examine. It can be useful in say git-receive-pack as a much more fine-grained alternative to receive.denynonfastforwards. If the server's policy is to not rewind a branch then receive-pack shouldn't let a remote user rewind it. At which point its useful to have that branch data in $GIT_DIR/config. :-)

Shawn Pearce <spearce@spearce.org> writes:

I never misunderstood your plan as such. I was more worried about a case where you clone from a not-so-public repository and the config file has some information of sensitive nature. Currently we do not have any such variable, so I am probably being a bit paranoid here, but using the config for the purpose of what we are discussing right now closes the door to store sensitive information and declare that the config file at the remote is "none of the cloner's business".

However,...

This is a good point. If we want to ever have a 'not to be leaked' configuration per repository, we can have a separate, private, configuration file to store that, so I probably should not worry too much about this at this point. I'm Ok with having this in the usual config -- the convenience outweighs the purity.

To summarize:

 * A repository can have branch."foo".rewinds configuration to
   mark the "foo" branch to be subject to rewinding.  This
   configuration variable defaults to false.

 * Receive.denynonfastforwards is currently a boolean but a new
   value, "per-branch", is also allowed.  When it is set to
   "per-branch", non fast-forward update is allowed only when
   branch."foo".rewinds is true for the branch.

 * A cloner creates "Pull: refs/heads/*:refs/remotes/origin/*"
   (or config equivalent) upon cloning.  When it clones, it
   downloads the config from the remote in order to see which
   are marked as "rewinds".  It makes tracking branches for only
   the ones that are not marked as "rewinds" by default.  We
   might have an option to do '+' variant glob, and clone all
   branches.

 * Subsequent fetch, when the glob is not '+' variant, would
   download the config from the remote in order to see which are
   marked as "rewinds", and ignores the branches that are marked
   as such.

I am a bit unhappy that subsequent fetches have to re-read the remote config every time. I can sort-of-see we can cram the "this is expected to be rewound" information as part of peek-remote exchange to avoid the overhead, but I do not think it is easily doable for dumb transports without breaking the backward compatibility.

To avoid this, we need to remember what remote branches we have seen but decided not to track, perhaps because they were marked as "rewinds". Subsequent fetches first learn what are on the remote side, and only when it sees a branch that we do not track and we have not seen, it needs to re-read the remote config to see if that is to be tracked (in which case we would create a new tracking branch) or ignored (in which case we would remember that we will ignore this branch in the future fetches).

Junio C Hamano wrote:

Perhaps below the "Pull: refs/heads/*:refs/remotes/origin/*" (or config
equivalent) we should list (in some format) branches which are not to be
fetched becaus they don't fast forward. One option, namely
  Pull:-refs/heads/pu:refs/remotes/origin/pu
(or config equivalent)comes to mind. It has the advantage that when we
decide to fetch this branch, it would be as easy as changing '-' to '+'.
Or we can use '!' instead of '-' if you think it is more readable (probably
is).

On Sunday 26 November 2006 04:58, Junio C Hamano wrote:

I am not so sure about this. IMHO, any user expects to see "pu" branch cloned too after cloning git.git (at least me, and a newbie probably too).

I agree about the remote branch description saying that "pu" branch will be rewinded. git-clone should set "+" before such a branch.

As said, the real problem begins when the user tries to branch off her own local branch from "pu". At this point, "git branch" or "git checkout -b ..." should warn the user that he has to expect troubles when branching off from such a branch, and only allow it with a "--force" option.

Josef Weidendorfer <Josef.Weidendorfer@gmx.de> writes:

I think you are right that we can keep our sanity without omitting rewinding ones from the tracked set of branches. In fact, remotes/origin/* hierarchy are not to be checked out directly anyway, even after we do the "bare SHA-1 stored in $GIT_DIR/HEAD" update -- that should give you a read-only checkout that you cannot commit on top of. As long as we warn users who try to merge from these rewinding branches, there is no reason not to track them.

That is true if you make "git branch my-pu remotes/origin/pu" to somehow set up the default merge source of "my-pu" branch to be remotes/origin/pu to encourage merging from it.

I am still not convinced it is a valid assumption that a branch often want to merge from the branch it was forked off of, and even less so that "git branch" and "git checkout -b" are the places to do that configuration.

But for the sake of discussion let's pretend for now that it were a good idea. If we know remotes/origin/pu is expected to be rewound, the logic that configures "my-pu" to merge from the fork origin should be able to notice that situation, and refrain from doing the configuration, to prevent the user from issuing "git pull" without saying "from where", which can be done with your merge.stopsansdefault option in your other message.

On Sat, 2006-11-25 at 21:58, Junio C Hamano wrote:

I see how it is! :-) When Shawn says it, it is to be liked. But when I allude to it a _year_ ago, I'm ignored.

    From: Jon Loeliger <jdl@freescale.com>
    To: git@vger.kernel.org
    Subject: Re: Trying to Update All Heads of a Repository
    Date: Fri, 4 Nov 2005 07:49:40 -0700 
    Message-ID:  <E1EY2su-0006LW-IN@jdl.com>

    So, from the git-pull man page:

        For "git push", the local ref that matches <src> is
        used to fast forward the remote ref that matches
        <dst>. If the optional plus + is used, the remote
        ref is updated even if it does not result in a fast
        forward update.

    Ah-ha!  Wait.  But here's the conceptual missing piece:
    When might I _know_ I have the situation where a
    fast-forward update might not happen?  And as a remote
    puller, that would be "never" -- unless I know something
    about the nature of the remote end.  That is, like you
    said, "pu" is subject to wild fluctuation and non-linear
    behavior.  But any random puller can't know that a
    priori.  So far, that is out-of-band information about a
    branch that needs to be "available".

Yep, out-of-band information that needs to be available indeed...

Ah well. :-)

jdl

Jon Loeliger <jdl@freescale.com> writes:

Think of it not as a difference between you and Shawn, but as a difference in me back then and me now, who finally saw the light in your revelation after gaining a year's worth of experience ;-).

Seriously, I did not ignore it, and I think some of your other comments in the message contributed to improved documentation. When your message was posted, the issue Shawn raised in his message was not so prominent, because people did not even get that "Pull: refs/heads/pu:refs/heads/pu" line in their remotes/ file by default back then. It was back in 0.99.9 era and a few days after your message, a patch was posted to put more than just the 'master' mapping in the remotes/ file clone creates. It became an issue after that change but we lacked the resources nor sense of urgency to do the out-of-band stuff, as we were busy working on other things.

Shawn Pearce wrote:

As I said, branch.<name>.merge is about something else: it just means that if we are on <name> branch "git pull" will merge branch.<name>.merge branch into it.

I think the "Merge: " or remote.<repo>.merge is about changing current implicit rule: first branch is to be merged with current branch (if not specified otherwise) when pull-ing, into explicit rule: branch marked as "Merge: " is to be merged with current branch (unless specified otherwise).

Correct me if I'm wrong, Junio.

On Saturday 25 November 2006 22:53, Junio C Hamano wrote:

I vote for a config option

 core.pull.stopWithoutDefaultMerge

which would error out on a

 git pull

when no branch.<current branch>.merge is set with a message like

 "No default remote branch set to merge into current branch <branch>.
  Set branch.<branch>.remote/merge to make this work"

It is arguable if this option should change the behavior of "git pull <repo>". Probably not: I would assume that the user explicitly wants to merge the primary branch of <repo>.

Additionally, set branch.<primary>.remote/merge on git-clone, together with core.pull.stopWithoutDefaultMerge=1.

On Sat, 2006-11-25 at 15:53, Junio C Hamano wrote:

Heh.  It was too late back here too!

    From: 	Jon Loeliger <jdl@freescale.com>
    To: 	Git List <git@vger.kernel.org>
    Subject: 	Re: GIT 0.99.7d, and end of week status.
    Date: 	Mon, 26 Sep 2005 15:17:32 -0500
    Message-Id:  <1127765852.5735.36.camel@cashmere.sps.mot.com>

jdl