threads / discuss / 40072

Re: enhanced remote ref namespaces

Subject: Re: enhanced remote ref namespaces

## tl;dr

5 messages between Aug 12, 2015 and Aug 12, 2015.

replies: 4people: 3as markdown or json

Jacob Keller· Aug 12, 2015, 18:34 UTC · lore
On Wed, Aug 12, 2015 at 9:10 AM, Junio C Hamano <gitster@pobox.com> wrote:
Show 11 quoted lines
> Jacob Keller <jacob.keller@gmail.com> writes:
>
>> Recently there was some discussion about git-notes and how we do not
>> fetch notes from remotes by default. The big problem with doing so is
>> because refs/remotes/* hierarchy is only setup for branches (heads),
>> so we don't have any clean location to put them.
>
> I wouldn't call this topic "proper" namespaces, though.  What we
> have is widely used and it shouldn't be broken.  Call it "enhanced",
> perhaps.
>
Ok.
Show 7 quoted lines
> Some design boundaries:
>
>  - Moving the remote-tracking branch hierarchy from refs/remotes/$O/*
>    to refs/remotes/$O/heads/* would not fly, because it will break
>    existing repositories.  Do not even waste time on pursuing
>    refs/remotes/$O/{heads,tags,notes...}/*
>

even if we maintained new git's abililty to work with this, ie: only external-to-git scripts would break and only for new clones? Maybe we don't want to go this route, but it seems like the way that the original proposal was headed.

Show 19 quoted lines
>  - Extending the current refs/remotes/$O/* (for branches) and doing
>    refs/remote-tags/$O/* (for tags) may work, would not break
>    existing repositories, and could to be extended further to cover
>    refs/remote-notes/$O and friends.  It however may not look pretty
>    (weak objection) and more importantly, it would make it harder to
>    "cull" things that came from a single remote.
>
> Just thinking aloud, perhaps we can introduce a brand new top level
> hierarchy refs/remote/$O/{heads,tags,notes,...}, and give backward
> compatibility by making a moral equivalent of a symbolic link from
> refs/remote/$O/heads to refs/remotes/$O/.  The true remote-tracknig
> refs continue to live in refs/remotes/$O/* and old tools that do not
> know the new layout would not be hurt.  New tools that want to
> ignore and look only at refs/remote/$O/* can do so through the moral
> equivalent of a symbolic link.  "remote remove" needs to be taught
> about this single exception (i.e. "the symbolic link"), but the
> places it needs to touch is limited only to two places and will not
> grow.
>
I think this proposal makes the sense..  I'd go with something like:
refs/tracking/<origin>/(heads|tags|notes|replace|etc)/*
with a symlink from or into
refs/tracking/<origin>/heads -> refs/remotes/<origin>

I prefer tracking vs remote because "remote" and "remotes" are too similar for my taste. tracking I think gets the idea across pretty straight forward. Using a symlink personally I'd symlink the refs/tracking into refs/remotes rather than make refs/remotes the real storage.

Show 6 quoted lines
> If somebody got confused, notice that in the above description, I
> said refs/remotes/ and refs/remote/.  The former must stay.  The
> name of the latter is open to bikeshedding.  Some may prefer a name
> that is more distinct (refs/tracking/ or something, perhaps?).  I
> happen to prefer a name that is similar, but this preference is very
> weak and I can persuaded to go either way.

I don't like it being so similar that we now have typos between remotes and remote.. ie: remotes/<origin> works for heads, but "remotes/<origin>/tags" does not... that sounds like it would get confusing.

Symlinking the old location seems reasonable to me, as it would leave all the same data in the locations expected by the old tools, while keeping all actual storage in the new location.

In this way, we would no longer need configuration settings. It honestly doesn't matter to me which direction we symlink either.

As for the other complex issue is what to do about "refs/tracking/<origin>/tags

The big discussion on the original thread is about how tags would work. I'm personally ok with *ignoring* tags and leaving it the way it is for now, and just doing this as a solid place to stick notes/replace/etc.

Or, we could go the route of continuing to stick tags into "refs/tags" at the same time as also sticking them into refs/tracking/<origin>/tags

Or.. we could go the full route of fixing up lookup of tags such that we put tags in refs/tracking/<origin>/tags and we have it lookup tags there via something like:

1) local tags preferred
2) any remote tag as long as all remote tags point to the same commit
object (how we select which to use is not very relevant here... we
could actually go with as long as the tag object is the same so two
remotes with annotated tags pointing to the same object but different
tag id would be ambiguous as well)
3) warn the user must disambiguate the tag via the remote name.

We could also teach fetch to warn about remote tags which are ambiguous (ie: two remotes with same named tag objects pointing to different things)

How goes this sound? I think it makes sense... I don't know how to do all this without breaking some backwards compatibility though... I think we can maintain expectations for the general user but I feel that any change here will break *someones* scripts.

Regards, Jake

Junio C Hamano· Aug 12, 2015, 18:54 UTC · re: Jacob Keller · lore
Jacob Keller <jacob.keller@gmail.com> writes:
Show 19 quoted lines
>> Just thinking aloud, perhaps we can introduce a brand new top level
>> hierarchy refs/remote/$O/{heads,tags,notes,...}, and give backward
>> compatibility by making a moral equivalent of a symbolic link from
>> refs/remote/$O/heads to refs/remotes/$O/.  The true remote-tracknig
>> refs continue to live in refs/remotes/$O/* and old tools that do not
>> know the new layout would not be hurt.  New tools that want to
>> ignore and look only at refs/remote/$O/* can do so through the moral
>> equivalent of a symbolic link.  "remote remove" needs to be taught
>> about this single exception (i.e. "the symbolic link"), but the
>> places it needs to touch is limited only to two places and will not
>> grow.
>
> I think this proposal makes the sense..  I'd go with something like:
>
> refs/tracking/<origin>/(heads|tags|notes|replace|etc)/*
>
> with a symlink from or into
>
> refs/tracking/<origin>/heads -> refs/remotes/<origin>
I actually do not think we even need the "symlink" thing.

We can just say refs/remotes/$O has been and forever will be where the remote-tracking branches will live. With or without the symlink for backward compatibility, the updated Git will need to be aware of the two places to deal with older and newer repositories anyway.

"everything is now under refs/tracking/$O, not spread over many unbounded number of places like refs/remotes-$foo/$O" may appear to be cleaner but two is already not too bad and will greatly reduce the transition pain.

Jacob Keller· Aug 12, 2015, 19:02 UTC · re: Junio C Hamano · lore
On Wed, Aug 12, 2015 at 11:54 AM, Junio C Hamano <gitster@pobox.com> wrote:
Show 33 quoted lines
> Jacob Keller <jacob.keller@gmail.com> writes:
>
>>> Just thinking aloud, perhaps we can introduce a brand new top level
>>> hierarchy refs/remote/$O/{heads,tags,notes,...}, and give backward
>>> compatibility by making a moral equivalent of a symbolic link from
>>> refs/remote/$O/heads to refs/remotes/$O/.  The true remote-tracknig
>>> refs continue to live in refs/remotes/$O/* and old tools that do not
>>> know the new layout would not be hurt.  New tools that want to
>>> ignore and look only at refs/remote/$O/* can do so through the moral
>>> equivalent of a symbolic link.  "remote remove" needs to be taught
>>> about this single exception (i.e. "the symbolic link"), but the
>>> places it needs to touch is limited only to two places and will not
>>> grow.
>>
>> I think this proposal makes the sense..  I'd go with something like:
>>
>> refs/tracking/<origin>/(heads|tags|notes|replace|etc)/*
>>
>> with a symlink from or into
>>
>> refs/tracking/<origin>/heads -> refs/remotes/<origin>
>
> I actually do not think we even need the "symlink" thing.
>
> We can just say refs/remotes/$O has been and forever will be where
> the remote-tracking branches will live.  With or without the symlink
> for backward compatibility, the updated Git will need to be aware of
> the two places to deal with older and newer repositories anyway.
>
> "everything is now under refs/tracking/$O, not spread over many
> unbounded number of places like refs/remotes-$foo/$O" may appear to
> be cleaner but two is already not too bad and will greatly reduce
> the transition pain.

Ok, so.... just have branches be in refs/remotes/$O and all new things be in refs/tracking/$O/category

that sounds reasonable enough to me.

That still hasn't really resolved the question of how to deal with tags, but it does solve the question of how to deal with replace and notes refs.

Regards, Jake

Junio C Hamano· Aug 12, 2015, 19:49 UTC · re: Jacob Keller · lore
Jacob Keller <jacob.keller@gmail.com> writes:
> That still hasn't really resolved the question of how to deal with
> tags, but it does solve the question of how to deal with replace and
> notes refs.
I do not think it would be a good change to add a
    [remote "foo"]
        fetch = refs/tags/*:refs/tracking/foo/tags/*

as people do expect to see tags from upstream to appear in their local tag namespace when cloning from the central meeting point of their project (e.g. Linus's kernel repository).

I'm willing to believe those who argued that they have both private tags and public tags in their repository, but I think that would be something better supported by adding "git tag --local foo" that creates a local tag in e.g. refs/local-tags/foo, and by teaching DWIMmery to also look in there when the user says "foo".

Alternatively, they can use their local branch namespace, which is already private to them.

Johan Herland· Aug 12, 2015, 22:52 UTC · re: Jacob Keller · lore
On Wed, Aug 12, 2015 at 8:34 PM, Jacob Keller <jacob.keller@gmail.com> wrote:
Show 12 quoted lines
> On Wed, Aug 12, 2015 at 9:10 AM, Junio C Hamano <gitster@pobox.com> wrote:
>> Some design boundaries:
>>
>>  - Moving the remote-tracking branch hierarchy from refs/remotes/$O/*
>>    to refs/remotes/$O/heads/* would not fly, because it will break
>>    existing repositories.  Do not even waste time on pursuing
>>    refs/remotes/$O/{heads,tags,notes...}/*
>
> even if we maintained new git's abililty to work with this, ie: only
> external-to-git scripts would break and only for new clones? Maybe we
> don't want to go this route, but it seems like the way that the
> original proposal was headed.

I don't think it's worth trying to go that route. Even though it's theoretically possible to solve it, the complexity added by e.g. multiple git versions operating on the same repository (maybe even simultaneously?) suggests that it's much simpler to just go with a new clean hierarchy that simply Works for new Git versions, and the older versions don't have to deal with at all. IMHO, maybe even going as far as incrementing core.repositoryFormatVersion, so that older Git versions will refuse to work with new-style repos...

[...]
Show 11 quoted lines
>> If somebody got confused, notice that in the above description, I
>> said refs/remotes/ and refs/remote/.  The former must stay.  The
>> name of the latter is open to bikeshedding.  Some may prefer a name
>> that is more distinct (refs/tracking/ or something, perhaps?).  I
>> happen to prefer a name that is similar, but this preference is very
>> weak and I can persuaded to go either way.
>
> I don't like it being so similar that we now have typos between
> remotes and remote.. ie: remotes/<origin> works for heads, but
> "remotes/<origin>/tags" does not... that sounds like it would get
> confusing.

I like refs/tracking. At some point I was planning to use refs/peers, but that's merely another color for the bikeshed...

Show 13 quoted lines
> Symlinking the old location seems reasonable to me, as it would leave
> all the same data in the locations expected by the old tools, while
> keeping all actual storage in the new location.
>
> In this way, we would no longer need configuration settings. It
> honestly doesn't matter to me which direction we symlink either.
>
> As for the other complex issue is what to do about "refs/tracking/<origin>/tags
>
> The big discussion on the original thread is about how tags would
> work. I'm personally ok with *ignoring* tags and leaving it the way it
> is for now, and just doing this as a solid place to stick
> notes/replace/etc.

That is probably the best plan for now: Solve most of the problem, and punt on the controversial parts.

> Or, we could go the route of continuing to stick tags into "refs/tags"
> at the same time as also sticking them into
> refs/tracking/<origin>/tags

I don't like the copying approach; makes it harder to deduce which tags are (truly) local, and which came from a remote.

Show 13 quoted lines
> Or.. we could go the full route of fixing up lookup of tags such that
> we put tags in refs/tracking/<origin>/tags and we have it lookup tags
> there via something like:
>
> 1) local tags preferred
>
> 2) any remote tag as long as all remote tags point to the same commit
> object (how we select which to use is not very relevant here... we
> could actually go with as long as the tag object is the same so two
> remotes with annotated tags pointing to the same object but different
> tag id would be ambiguous as well)
>
> 3) warn the user must disambiguate the tag via the remote name.

As was probably obvious from the old threads, this is where I'd like to go. Eventually.

> We could also teach fetch to warn about remote tags which are
> ambiguous (ie: two remotes with same named tag objects pointing to
> different things)

Agreed. This would actually be a good improvement Right Now, without changing any ref layout at all: If get fetch would otherwise copy a remote tag into refs/tags/*, but doesn't because there is already a _different_ tag in its place, then warn loudly to the user, so that user has a chance to discover the tag difference.

> How goes this sound? I think it makes sense... I don't know how to do
> all this without breaking some backwards compatibility though... I
> think we can maintain expectations for the general user but I feel
> that any change here will break *someones* scripts.

As I said above: Punt on tags for now, and you might be able to not break anyone's scripts (and if you do, it's probably a poorly written script). Provided that you leave a symlink to/from refs/remotes/$O in place, you're AFAICS only adding functionality, not (visibly) changing existing behavior.

BTW, thanks for resurrecting this topic!
...Johan
-- 
Johan Herland, <johan@herland.net>
www.herland.net

← back to recent threads