proper remote ref namespaces

Hello,

Recently there was some discussion about git-notes and how we do not fetch notes from remotes by default. The big problem with doing so is because refs/remotes/* hierarchy is only setup for branches (heads), so we don't have any clean location to put them.

Around the time of git 1.8.0, Johan Herland made a proposal for remotes to put all their refs in refs/remtoes/*, by moving heads into refs/remotes/<remote>heads/* [1]

In addition, his proposal was to include remote tags into refs/remotes/<remote>/tags and also refs/remotes/<remote>/replace and notes similarly.

During this discussion there was many people who liked the idea, and others who rejected it. The main rejection reason was two fold:

(a) tags are "global" per project, so their namespace should be treated global as it is now.

The proposal's counter to this is that tags aren't guaranteed to be global, because today two remotes you fetch might have tags that are the same name with different pointers. This is currently hidden, and git silently picks the tag it fetched first.

(b) script compatibility, as changing the ref layout such that new git can't work with old repository would be bad

the counter to this, is that we make git smart enough to recognize old remote format, and continue to work with it. Scripts which depend on this layout will break, but that may not be such a huge concern.

Personally, I think this proposal at least for heads, notes, replace, and other remote refs we'd like to pull is very useful. I don't rightly know the answer for tags. The linked discussion below covers several pages of back and forth between a few people about which method is best.

I like the idea of simplifying tags and branches and notes and others to all fetch the same way. local stuff is in refs/heads or refs/notes and remote stuff is (by default) in refs/remotes/<remote>/tags etc

But it does bring up some discussion as today we "auto follow" tags into refs/tags, and it can get weird for tags since now "ambiguous" tags must mean if there are tags of same name which point to different refs, and we'd need to teach a bunch of logic to the ref lookup code.

I am looking at ways to help git-notes be easier to use, so that we by default fetch notes, and enable easier merge, since we'd have default locations to merge from and to. This would make the sharing of notes a lot easier, which is one of their primary sticking points.. you can't really share them without *everyone* working to do it the same way you do. By making a default policy, sharing becomes natural, and users can easily add *public* notes to commits for things such as bug ids and other things which are not discovered until after the commit is created.

In addition, the easy ability to share replaces might also be helpful, though IMHO not as valuable as git-notes.

I think that the only logical refs layout is "refs/remotes/<remote>/(heads|tags|notes|replace)"

and adding "refs/remote-notes" and "refs/remote-replace" is not really a clean solution.

Given that the 1.8.0 proposal mostly died, does anyone have any thoughts now?

The proposal suggested by Johan makes sense to me, and I believe we can code up logic to make it easy for new git to keep logic of the old layout.

Personally, I think the best solution is to only store that layout for a given clone, using a config option that defaults to false, where new-git sets it to true for all clones. Then, provide a command to renew remotes-layout that does this if the user wishes. Thus, clones for the old style will be handled, and new clones would have the new layout. (ie: no mixing layouts in a single repository).

I'm really not sure if this is the best solution, but seems like the cleanest solution.

Regards, Jake

[1] http://thread.gmane.org/gmane.comp.version-control.git/165799/focus=165885

On 15-08-12 02:43 AM, Jacob Keller wrote:

Thanks for resurrecting this discussion. I feel this is a fundamental inconsistency in git's core design. Not a fatal flaw by any means, but something that keeps git from reaching its full potential.

The tags would be disambiguated by their remote name, e.g. "origin/tags/v5.6" vs. "hacker/tags/v5.6". The change would actually simplify tag auto-following.

Not a lot. Existing DWIMery already handles ambiguous branches, by preferring a local branch name over any remote ones. The only teaching that's really needed is to resolve shorthand like "origin/v5.6" to "refs/remotes/origin/tags/v5.6" (i.e. to look for anything matching "refs/remotes/origin/*/v5.6") but that doesn't seem very difficult.

There is the question of how the user can even know if there's ambiguity. Aside from paying attention to "git fetch" output, I think we could extend "git tag" (and "git branch") in the case where the user specifies an existing tag (or branch). Right now both commands fail because the name already exists. All we need to do is extend that error message a bit, e.g.:

	> git tag v5.6
	fatal: tag 'next' already exists as
	v5.6 -> a3a0e5d67d554a685abd897bc3ce4ffa4e74c812
	origin/v5.6 remoteX/v5.6 -> 504346bbf9b02387b51f232f4db9c1860789b135
	hacker/v5.6 -> fb4aa3533f81700789b3fb119e527410678e8d8d

Here we see that our local v5.6, and hacker's v5.6, are unique, but origin and remoteX both have the same v5.6.

Well, same-ish. I think those SHA IDs should be the things the tags point at, not the tag objects' IDs. This keeps things consistent between lightweight and annotated/signed tags. It does mean though that origin/v5.6 and remoteX/v5.6 might be different tag objects that happen to point at the same thing. I'm not sure the distinction is all that germane, and it's something that the user could disambiguate with "git tag -v" or "git show".

I think that proposal is good. Perhaps something like config.remoterefslayout, where 0 means the existing layout and 1 is the new layout (with larger numbers reserved for future changes).

Implementing the new layout should, I think, just involve changing the default fetch refspecs when adding a remote.

Then it's just a matter of making the test suite pass.  :)

		M.

Marc Branchaud <marcnarc@xiplink.com> writes:

You need to remember that there are five useful things you can do to mutable things.

 - Creation can be covered by teaching "clone" to put a new-style
   refspec but the same change needs to also go to "remote add".

 - Reading is done by the DWIMery in ref_rev_parse_rules[] (your
   point above).

 - Updating is automatic, as "fetch" does not have any funny
   built-in intuit and blindly follows configured fetch refspec.

 - Deletion by "branch -d -r" and "remote remove" needs to be
   careful about designing how the case where both old and new
   hierarchies exist should be handled (my gut feeling is "delete
   both", but there may be funny corner cases).

 - Enumeration by "branch -l -r" probably shares the same issue as
   deletion.

Jacob Keller <jacob.keller@gmail.com> writes:

I wouldn't call this topic "proper" namespaces, though. What we have is widely used and it shouldn't be broken. Call it "enhanced", perhaps.

Some design boundaries:

 - Moving the remote-tracking branch hierarchy from refs/remotes/$O/*
   to refs/remotes/$O/heads/* would not fly, because it will break
   existing repositories.  Do not even waste time on pursuing
   refs/remotes/$O/{heads,tags,notes...}/*

 - Extending the current refs/remotes/$O/* (for branches) and doing
   refs/remote-tags/$O/* (for tags) may work, would not break
   existing repositories, and could to be extended further to cover
   refs/remote-notes/$O and friends.  It however may not look pretty
   (weak objection) and more importantly, it would make it harder to
   "cull" things that came from a single remote.

Just thinking aloud, perhaps we can introduce a brand new top level hierarchy refs/remote/$O/{heads,tags,notes,...}, and give backward compatibility by making a moral equivalent of a symbolic link from refs/remote/$O/heads to refs/remotes/$O/. The true remote-tracknig refs continue to live in refs/remotes/$O/* and old tools that do not know the new layout would not be hurt. New tools that want to ignore and look only at refs/remote/$O/* can do so through the moral equivalent of a symbolic link. "remote remove" needs to be taught about this single exception (i.e. "the symbolic link"), but the places it needs to touch is limited only to two places and will not grow.

If somebody got confused, notice that in the above description, I said refs/remotes/ and refs/remote/. The former must stay. The name of the latter is open to bikeshedding. Some may prefer a name that is more distinct (refs/tracking/ or something, perhaps?). I happen to prefer a name that is similar, but this preference is very weak and I can persuaded to go either way.