threads / rfc / 18083

[RFC] Refspec patterns with * in the middle

Subject: [RFC] Refspec patterns with * in the middle

## tl;dr

12 messages between Mar 1, 2009 and Mar 3, 2009.

replies: 11people: 4as markdown or json

Daniel Barkalow· Mar 1, 2009, 23:42 UTC · lore

I've got an annoying repository where all of the branches upstream[*] have names, for a project "my-proj" like:

some/constant/stuff/$VERSION/junk/my-proj
I'd like to be able to use refspecs like:
 fetch = some/constant/stuff/*/junk/my-proj:refs/remotes/origin/*

I've written an implementation (which mainly involved having only one place do the matching and replacement for pattern refspecs, and then making that one place a little more clever), so that's not hard. But we currently prohibit refspecs like this, and I think we may want to prohibit some patterns of this general form, in order to keep typos from doing surprising things.

My use case is actually, more precisely:
some/constant/stuff/$PROJ-$NUMBER/junk/my-proj

Where $NUMBER is the version number, and $PROJ is usually, but not quite always "my-proj"; the exception being that it might be effectively a superproject. So I'd like to have:

 fetch = some/constant/stuff/my-proj-*/junk/my-proj:refs/remotes/origin/*
But I can live with remote branches like "my-proj-2.4" instead of "2.4".

I think it would make sense, and limit typo damage, to say that the * can only expand to something with a '/' in it if the star has a slash or the end of the string on each side.

What are other people's thoughts? Should I have a config option for how flexible matching is permitted? Any particular constraints I should have in general?

[*] (This is actually a Perforce upstream, but I want to keep the mapping from Perforce into git simple and not grow another pattern-matching format, so I'm just copying Perforce paths as ref names, and then I want the match them with the usual code)

	-Daniel
*This .sig left intentionally blank*
Jay Soffian· Mar 2, 2009, 12:54 UTC · re: Daniel Barkalow · lore

Re: [RFC] Refspec patterns with * in the middle

On Sun, Mar 1, 2009 at 6:42 PM, Daniel Barkalow <barkalow@iabervon.org> wrote:
Show 31 quoted lines
> I've got an annoying repository where all of the branches upstream[*] have
> names, for a project "my-proj" like:
>
> some/constant/stuff/$VERSION/junk/my-proj
>
> I'd like to be able to use refspecs like:
>
>  fetch = some/constant/stuff/*/junk/my-proj:refs/remotes/origin/*
>
> I've written an implementation (which mainly involved having only one
> place do the matching and replacement for pattern refspecs, and then
> making that one place a little more clever), so that's not hard. But we
> currently prohibit refspecs like this, and I think we may want to prohibit
> some patterns of this general form, in order to keep typos from doing
> surprising things.
>
> My use case is actually, more precisely:
>
> some/constant/stuff/$PROJ-$NUMBER/junk/my-proj
>
> Where $NUMBER is the version number, and $PROJ is usually, but not quite
> always "my-proj"; the exception being that it might be effectively a
> superproject. So I'd like to have:
>
>  fetch = some/constant/stuff/my-proj-*/junk/my-proj:refs/remotes/origin/*
>
> But I can live with remote branches like "my-proj-2.4" instead of "2.4".
>
> I think it would make sense, and limit typo damage, to say that the * can
> only expand to something with a '/' in it if the star has a slash or the
> end of the string on each side.

That seems more confusing that just saying: '*' matches everything but the path separator ('/'), and whatever it matches on the LHS of the ':' is what it expands to on the RHS. I'm not sure how a typo would damage anything, but this could always be enabled with core.refspec.glob_anywhere or some such.

I think regex support is too much:
  fetch = some/constant/stuff/(my-proj-[^/]*)/junk/my-proj:refs/remotes/origin/\1

(Which in a git config, I think may need a double-backslash, but I forget what the config parser does.)

j.
Junio C Hamano· Mar 2, 2009, 17:08 UTC · re: Daniel Barkalow · lore

Re: [RFC] Refspec patterns with * in the middle

Daniel Barkalow <barkalow@iabervon.org> writes:
Show 9 quoted lines
> My use case is actually, more precisely:
>
> some/constant/stuff/$PROJ-$NUMBER/junk/my-proj
>
> Where $NUMBER is the version number, and $PROJ is usually, but not quite 
> always "my-proj"; the exception being that it might be effectively a 
> superproject. So I'd like to have:
>
>  fetch = some/constant/stuff/my-proj-*/junk/my-proj:refs/remotes/origin/*

... and expect "some/constant/stuff/my-proj-2.5/junk/my-proj" to be mapped to "refs/remotes/origin/2.5"? I think it does not look too bad.

Show 5 quoted lines
> But I can live with remote branches like "my-proj-2.4" instead of "2.4".
>
> I think it would make sense, and limit typo damage, to say that the * can 
> only expand to something with a '/' in it if the star has a slash or the 
> end of the string on each side.

I do not understand what you mean by "* can only expand to something with a '/' in it if ..." part. None of the examples in your message have a case where the asterisk matches across directory boundaries, and I thought you would simply say "* does not match /" and be done with that.

What scenario do you have in mind that wants to match a slash with an asterisk?

Jay Soffian· Mar 2, 2009, 18:01 UTC · re: Junio C Hamano · lore

Re: [RFC] Refspec patterns with * in the middle

On Mon, Mar 2, 2009 at 12:08 PM, Junio C Hamano <gitster@pobox.com> wrote:
Show 28 quoted lines
> Daniel Barkalow <barkalow@iabervon.org> writes:
>
>> My use case is actually, more precisely:
>>
>> some/constant/stuff/$PROJ-$NUMBER/junk/my-proj
>>
>> Where $NUMBER is the version number, and $PROJ is usually, but not quite
>> always "my-proj"; the exception being that it might be effectively a
>> superproject. So I'd like to have:
>>
>>  fetch = some/constant/stuff/my-proj-*/junk/my-proj:refs/remotes/origin/*
>
> ... and expect "some/constant/stuff/my-proj-2.5/junk/my-proj" to be mapped
> to "refs/remotes/origin/2.5"?  I think it does not look too bad.
>
>> But I can live with remote branches like "my-proj-2.4" instead of "2.4".
>>
>> I think it would make sense, and limit typo damage, to say that the * can
>> only expand to something with a '/' in it if the star has a slash or the
>> end of the string on each side.
>
> I do not understand what you mean by "* can only expand to something with
> a '/' in it if ..." part.  None of the examples in your message have a
> case where the asterisk matches across directory boundaries, and I thought
> you would simply say "* does not match /" and be done with that.
>
> What scenario do you have in mind that wants to match a slash with an
> asterisk?
I think he means the following are valid:
 - foo/bar/*/baz
 - foo/bar/baz/*
But the following is not:
 - foo/bar*/baz

IOW, '*' can only appear as a non-terminating symbol if it is bounded by '/' on each side.

j.
Daniel Barkalow· Mar 2, 2009, 18:25 UTC · re: Jay Soffian · lore

Re: [RFC] Refspec patterns with * in the middle

On Mon, 2 Mar 2009, Jay Soffian wrote:
Show 41 quoted lines
> On Mon, Mar 2, 2009 at 12:08 PM, Junio C Hamano <gitster@pobox.com> wrote:
> > Daniel Barkalow <barkalow@iabervon.org> writes:
> >
> >> My use case is actually, more precisely:
> >>
> >> some/constant/stuff/$PROJ-$NUMBER/junk/my-proj
> >>
> >> Where $NUMBER is the version number, and $PROJ is usually, but not quite
> >> always "my-proj"; the exception being that it might be effectively a
> >> superproject. So I'd like to have:
> >>
> >>  fetch = some/constant/stuff/my-proj-*/junk/my-proj:refs/remotes/origin/*
> >
> > ... and expect "some/constant/stuff/my-proj-2.5/junk/my-proj" to be mapped
> > to "refs/remotes/origin/2.5"?  I think it does not look too bad.
> >
> >> But I can live with remote branches like "my-proj-2.4" instead of "2.4".
> >>
> >> I think it would make sense, and limit typo damage, to say that the * can
> >> only expand to something with a '/' in it if the star has a slash or the
> >> end of the string on each side.
> >
> > I do not understand what you mean by "* can only expand to something with
> > a '/' in it if ..." part.  None of the examples in your message have a
> > case where the asterisk matches across directory boundaries, and I thought
> > you would simply say "* does not match /" and be done with that.
> >
> > What scenario do you have in mind that wants to match a slash with an
> > asterisk?
> 
> I think he means the following are valid:
> 
>  - foo/bar/*/baz
>  - foo/bar/baz/*
> 
> But the following is not:
> 
>  - foo/bar*/baz
> 
> IOW, '*' can only appear as a non-terminating symbol if it is bounded
> by '/' on each side.

You have my criterion right, but I want that to be valid, but only match things like "foo/bar-something/baz", not "foo/bar-a/b/baz".

	-Daniel
*This .sig left intentionally blank*
Jay Soffian· Mar 2, 2009, 22:07 UTC · re: Daniel Barkalow · lore

Re: [RFC] Refspec patterns with * in the middle

On Mon, Mar 2, 2009 at 1:25 PM, Daniel Barkalow <barkalow@iabervon.org> wrote:
Show 9 quoted lines
>> But the following is not:
>>
>>  - foo/bar*/baz
>>
>> IOW, '*' can only appear as a non-terminating symbol if it is bounded
>> by '/' on each side.
>
> You have my criterion right, but I want that to be valid, but only match
> things like "foo/bar-something/baz", not "foo/bar-a/b/baz".

Ah, that makes sense. Perhaps use "**" to mean matching across path components which is what rsync does:

 o  a '*' matches any non-empty path component (it stops at slashes).
 o  use '**' to match anything, including slashes.
?
j.
Junio C Hamano· Mar 2, 2009, 22:39 UTC · re: Jay Soffian · lore

Re: [RFC] Refspec patterns with * in the middle

Jay Soffian <jaysoffian@gmail.com> writes:
Show 18 quoted lines
> On Mon, Mar 2, 2009 at 1:25 PM, Daniel Barkalow <barkalow@iabervon.org> wrote:
>>> But the following is not:
>>>
>>>  - foo/bar*/baz
>>>
>>> IOW, '*' can only appear as a non-terminating symbol if it is bounded
>>> by '/' on each side.
>>
>> You have my criterion right, but I want that to be valid, but only match
>> things like "foo/bar-something/baz", not "foo/bar-a/b/baz".
>
> Ah, that makes sense. Perhaps use "**" to mean matching across path
> components which is what rsync does:
>
>  o  a '*' matches any non-empty path component (it stops at slashes).
>  o  use '**' to match anything, including slashes.
>
> ?
I personally do not think that makes much sense (and I find ** ugly, too).

We traditionally supported '*' only at the end, and it always has meant "match through the end, including slashes".

Requiring 'match including slashes' to be spelled as '**' only when it is not at the end is unnecessarily confusing.

Is there a valid use case when * wants to match across directory boundaries when it is not at the end? I offhand do not think of a sane one.

So, it might make it easiest to understand if we say * usually does not match slash, except when it is used at the end immediately after a slash, in which case it means "match through the end".

Jay Soffian· Mar 2, 2009, 22:58 UTC · re: Junio C Hamano · lore

Re: [RFC] Refspec patterns with * in the middle

On Mon, Mar 2, 2009 at 5:39 PM, Junio C Hamano <gitster@pobox.com> wrote:
> I personally do not think that makes much sense (and I find ** ugly, too).
>
> We traditionally supported '*' only at the end, and it always has meant
> "match through the end, including slashes".
Oh, good point, I had not thought that through very well.
Show 10 quoted lines
> Requiring 'match including slashes' to be spelled as '**' only when it is
> not at the end is unnecessarily confusing.
>
> Is there a valid use case when * wants to match across directory
> boundaries when it is not at the end?  I offhand do not think of a sane
> one.
>
> So, it might make it easiest to understand if we say * usually does not
> match slash, except when it is used at the end immediately after a slash,
> in which case it means "match through the end".
That seems simple enough.
j.
Daniel Barkalow· Mar 2, 2009, 23:00 UTC · re: Junio C Hamano · lore

Re: [RFC] Refspec patterns with * in the middle

On Mon, 2 Mar 2009, Junio C Hamano wrote:
Show 32 quoted lines
> Jay Soffian <jaysoffian@gmail.com> writes:
> 
> > On Mon, Mar 2, 2009 at 1:25 PM, Daniel Barkalow <barkalow@iabervon.org> wrote:
> >>> But the following is not:
> >>>
> >>>  - foo/bar*/baz
> >>>
> >>> IOW, '*' can only appear as a non-terminating symbol if it is bounded
> >>> by '/' on each side.
> >>
> >> You have my criterion right, but I want that to be valid, but only match
> >> things like "foo/bar-something/baz", not "foo/bar-a/b/baz".
> >
> > Ah, that makes sense. Perhaps use "**" to mean matching across path
> > components which is what rsync does:
> >
> >  o  a '*' matches any non-empty path component (it stops at slashes).
> >  o  use '**' to match anything, including slashes.
> >
> > ?
> 
> I personally do not think that makes much sense (and I find ** ugly, too).
> 
> We traditionally supported '*' only at the end, and it always has meant
> "match through the end, including slashes".
> 
> Requiring 'match including slashes' to be spelled as '**' only when it is
> not at the end is unnecessarily confusing.
> 
> Is there a valid use case when * wants to match across directory
> boundaries when it is not at the end?  I offhand do not think of a sane
> one.
Maybe:
/refs/imported/$GROUP/$USER/project -> $GROUP/$USER
or
/refs/imported/sandbox/$USER/$TOPIC/project -> $USER/$TOPIC

The issue, in my case, is importing from a system where branches contain projects instead of projects containing branches (and everything is a single namespace). So I want to match an insane (for us) LHS with a sane RHS to get stuff into reasonable shape. I don't really care about any patterns where the branch identifier is multiple components, but I wouldn't be surprised if somebody did.

Oh, and it looks like "?" is reserved and currently unused, so we could have * match one or more full path components, and ? match partial path components.

	-Daniel
*This .sig left intentionally blank*
Junio C Hamano· Mar 2, 2009, 23:30 UTC · re: Daniel Barkalow · lore

Re: [RFC] Refspec patterns with * in the middle

Daniel Barkalow <barkalow@iabervon.org> writes:
Show 6 quoted lines
> The issue, in my case, is importing from a system where branches contain 
> projects instead of projects containing branches (and everything is a 
> single namespace). So I want to match an insane (for us) LHS with a sane 
> RHS to get stuff into reasonable shape. I don't really care about any 
> patterns where the branch identifier is multiple components, but I 
> wouldn't be surprised if somebody did.

Isn't this just getting more and more insane? Is this really worth supporting, I have to wonder...

> Oh, and it looks like "?" is reserved and currently unused, so we could 
> have * match one or more full path components, and ? match partial path 
> components.

Well, "?" is not allowed exactly because it often is used to match a single character by things like for-each-ref.

David Kastrup· Mar 3, 2009, 16:25 UTC · re: Junio C Hamano · lore

Re: [RFC] Refspec patterns with * in the middle

Junio C Hamano <gitster@pobox.com> writes:
Show 18 quoted lines
> Daniel Barkalow <barkalow@iabervon.org> writes:
>
>> The issue, in my case, is importing from a system where branches contain 
>> projects instead of projects containing branches (and everything is a 
>> single namespace). So I want to match an insane (for us) LHS with a sane 
>> RHS to get stuff into reasonable shape. I don't really care about any 
>> patterns where the branch identifier is multiple components, but I 
>> wouldn't be surprised if somebody did.
>
> Isn't this just getting more and more insane?  Is this really worth
> supporting, I have to wonder...
>
>> Oh, and it looks like "?" is reserved and currently unused, so we could 
>> have * match one or more full path components, and ? match partial path 
>> components.
>
> Well, "?" is not allowed exactly because it often is used to match a
> single character by things like for-each-ref.

Let me present a use case where the current matching rules have changed working well at some point of time.

We have a product called dsp which sits in Subversion under the top server repository as

dsp/{trunk,branches,tags}
Then we have projects that have a structure including
projekte/some-project/{trunk,branches,tags}/dsp

which usually is an external link to dsp/trunk. However, sometimes a project needs its own local variations of dsp, in which case projekte/some-project/trunk/dsp becomes a properly populated copy of dsp/trunk.

Now in git, I want to track those non-externals as branches of the dsp checkout: that way I get cherry-picking and merging and stuff like that. To have this work out, I need to have the dsp git project pulling stuff from any projekte/some-project/trunk/dsp as a branch called some-project.

I had a git-svn configuration at one point of time that did that, but the matching rules have changed since then.

So in this case, the existing Subversion structure (which makes quite good sense) makes wildcards in non-terminal positions desirable.

-- 
David Kastrup
Daniel Barkalow· Mar 2, 2009, 18:22 UTC · re: Junio C Hamano · lore

Re: [RFC] Refspec patterns with * in the middle

On Mon, 2 Mar 2009, Junio C Hamano wrote:
Show 14 quoted lines
> Daniel Barkalow <barkalow@iabervon.org> writes:
> 
> > My use case is actually, more precisely:
> >
> > some/constant/stuff/$PROJ-$NUMBER/junk/my-proj
> >
> > Where $NUMBER is the version number, and $PROJ is usually, but not quite 
> > always "my-proj"; the exception being that it might be effectively a 
> > superproject. So I'd like to have:
> >
> >  fetch = some/constant/stuff/my-proj-*/junk/my-proj:refs/remotes/origin/*
> 
> ... and expect "some/constant/stuff/my-proj-2.5/junk/my-proj" to be mapped
> to "refs/remotes/origin/2.5"?  I think it does not look too bad.
Yeah.
Show 13 quoted lines
> > But I can live with remote branches like "my-proj-2.4" instead of "2.4".
> >
> > I think it would make sense, and limit typo damage, to say that the * can 
> > only expand to something with a '/' in it if the star has a slash or the 
> > end of the string on each side.
> 
> I do not understand what you mean by "* can only expand to something with
> a '/' in it if ..." part.  None of the examples in your message have a
> case where the asterisk matches across directory boundaries, and I thought
> you would simply say "* does not match /" and be done with that.
> 
> What scenario do you have in mind that wants to match a slash with an
> asterisk?

Backwards compatibility? Currently, refs/heads/*:refs/remotes/origin/* will match refs/heads/js/notes with refs/remotes/origin/js/notes, because it simply copies everything after the fixed prefix.

I was assuming that that particular matching is an existing requirement (and that you used it regularly to share topic-maintenance work between machines).

So my idea was that a path component of "*" would be able to match multiple levels, while a path component of "something-*" or "*-something" can only match a single path component that starts or ends with the right thing. Looking at it now, I think it might be more sane to use "%" for the second case, and say: "*" must be used as a full path component, and matches one or more path components; "%" may be used as part of a path component, and doesn't match the path separator; you can only use one such character, and both sides of the refspec have to agree on which.

I think the problem cases were typos like:
refs/heads*:refs/remotes/origin/* (used to cause problems)
and
refs/heads/*:refs/remotes/origin* (gives you the originmaster remote?)
	-Daniel
*This .sig left intentionally blank*

← back to recent threads