threads / discuss / 28207

Buggy handling of non-canonical ref names

Subject: Buggy handling of non-canonical ref names

## tl;dr

10 messages between Aug 24, 2011 and Aug 25, 2011.

replies: 9people: 3as markdown or json

Michael Haggerty· Aug 24, 2011, 15:49 UTC · lore

git has inconsistencies (I would say bugs) in how it handles reference names that are in non-canonical format (e.g., "/foo/bar", "foo///bar", etc). Some commands work with reference names that are in non-canonical form, whereas others do not. Sometimes the behavior depends on whether the reference is loose or packed.

For example "git branch" and "git update-ref" seem to swallow them OK:
    $ git update-ref /foo/bar HEAD
    $ cat .git/foo/bar
    ef6cf90ba11dd6205f8b974692d795ea0b1c0bdd
    $ git branch /bar/baz
    $ git for-each-ref | grep baz
    ef6cf90ba11dd6205f8b974692d795ea0b1c0bdd commit refs/heads/bar/baz
But other commands do not:
    $ git rev-parse /foo/bar --
    /foo/bar
    fatal: ambiguous argument '/foo/bar': unknown revision or path not
in the working tree.
    Use '--' to separate paths from revisions
    $ git rev-parse foo///bar --
    foo///bar
    fatal: ambiguous argument 'foo///bar': unknown revision or path not
in the working tree.
    Use '--' to separate paths from revisions
    $ git rev-parse foo/bar --
    ef6cf90ba11dd6205f8b974692d795ea0b1c0bdd
    --

It seems like most of the canonicalization of reference names is via path canonicalization using the likes of git_snpath() and git_path(), with the exception of "git check-ref-format --print", which does the work using its own code. But packed references are not transformed into paths, so they don't handle canonicalization at all. For example, in the following case, there is a difference between how packed and unpacked references are handled:

    $ git update-ref refs/heads/foo/bar HEAD
    $ git update-ref -d refs/heads/foo///bar HEAD
    [Reference was really deleted]
    $ git update-ref refs/heads/foo/bar HEAD
    $ git pack-refs --all
    $ git update-ref -d refs/heads/foo///bar HEAD
    error: unable to resolve reference refs/heads/foo///bar: No such
file or directory
    $ git for-each-ref | grep foo
    ffae1b1dc75896bd89ff3cbe7037f40474e57e2a commit refs/heads/foo/bar
    [Reference was not deleted]

What is the policy about reference names and their canonicalization? In what situations should one be able to use non-canonical refnames, and in what situations should it be forbidden?

Given that refnames are closely associated with filesystem paths, it is important that every code path either canonicalize or reject non-canonical refnames (i.e., failure to do so could be a security problem). In the absence of clear rules, it will be very difficult to ensure that this is being done consistently.

I also noticed that check_ref_format() accepts ref names that start with a "/", but that the leading slash is *not* stripped off as part of the canonicalization:

    $ git check-ref-format /foo/bar ; echo $?
    0
    $ git check-ref-format --print /foo/bar
    /foo/bar

However, creating a reference with such a name is equivalent to creating a reference without the leading slash:

    $ git update-ref /foo/bar HEAD
    $ cat .git/foo/bar
    ef6cf90ba11dd6205f8b974692d795ea0b1c0bdd
    $ git branch /bar/baz
    $ git for-each-ref | grep baz
    ef6cf90ba11dd6205f8b974692d795ea0b1c0bdd commit refs/heads/bar/baz
Therefore, it seems to belong in the equivalence class of "foo/bar".

The test suite for "git check-ref-format" says nothing about how leading slashes should be handled.

Either leading slashes should not be allowed in ref names at all or they should be canonicalized away.

Michael
-- 
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/
Junio C Hamano· Aug 24, 2011, 18:40 UTC · re: Michael Haggerty · lore

Re: Buggy handling of non-canonical ref names

Michael Haggerty <mhagger@alum.mit.edu> writes:
> What is the policy about reference names and their canonicalization?

The overall policy has been that we care about well-formed input, and everything else is "undefined", even though as you found out some of them try to work sensibly.

Show 5 quoted lines
>     $ git check-ref-format /foo/bar ; echo $?
>     0
>
>     $ git check-ref-format --print /foo/bar
>     /foo/bar
I think these are bogus. Patches welcome.
Show 9 quoted lines
> However, creating a reference with such a name is equivalent to creating
> a reference without the leading slash:
>
>     $ git update-ref /foo/bar HEAD
>     $ cat .git/foo/bar
>     ef6cf90ba11dd6205f8b974692d795ea0b1c0bdd
>     $ git branch /bar/baz
>     $ git for-each-ref | grep baz
>     ef6cf90ba11dd6205f8b974692d795ea0b1c0bdd commit refs/heads/bar/baz
These are just examples of "undefined being nice to the user as a bonus".
Carlos Martín Nieto· Aug 24, 2011, 21:32 UTC · re: Junio C Hamano · lore

Re: Buggy handling of non-canonical ref names

On Wed, 2011-08-24 at 11:40 -0700, Junio C Hamano wrote:
Show 16 quoted lines
> Michael Haggerty <mhagger@alum.mit.edu> writes:
> 
> > What is the policy about reference names and their canonicalization?
> 
> The overall policy has been that we care about well-formed input, and
> everything else is "undefined", even though as you found out some of them
> try to work sensibly.
> 
> >     $ git check-ref-format /foo/bar ; echo $?
> >     0
> >
> >     $ git check-ref-format --print /foo/bar
> >     /foo/bar
> 
> I think these are bogus. Patches welcome.
> 

The rules in the manpage don't forbit it, as we assume that $GIT_DIR/ is going to be put in front. This makes /foo/bar mean the same as foo/bar (both become would cause git to look at $GIT_DIR/foo/bar), but I agree that it can be quite confusing.

Show 11 quoted lines
> > However, creating a reference with such a name is equivalent to creating
> > a reference without the leading slash:
> >
> >     $ git update-ref /foo/bar HEAD
> >     $ cat .git/foo/bar
> >     ef6cf90ba11dd6205f8b974692d795ea0b1c0bdd
> >     $ git branch /bar/baz
> >     $ git for-each-ref | grep baz
> >     ef6cf90ba11dd6205f8b974692d795ea0b1c0bdd commit refs/heads/bar/baz
> 
> These are just examples of "undefined being nice to the user as a bonus".

Or not, the user might have expected `git branch /bar/baz` to create a reference in $GIT_DIR/bar/baz with the SHA of the current branch (yes, it's probably unlikely, but this is what one might think when giving references as absolute paths).

Be as it may, allowing /foo/bar is not a good thing, how about this patch? It doesn't modify the manpage, as I wasn't sure whether this should become an explicit rule.

-- >8 --
Subject: [PATCH] Don't allow reference names to start with '/'

Being able to name references using absolute filenames is confusing at best and might give people wrong ideas.

Signed-off-by: Carlos Martín Nieto <cmn@elego.de>
---
 refs.c |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)
diff --git a/refs.c b/refs.c
index 6f313a9..ab549e4 100644
--- a/refs.c
+++ b/refs.c
@@ -879,6 +879,10 @@ int check_ref_format(const char *ref)
 	int ret = CHECK_REF_FORMAT_OK;
 	const char *cp = ref;
 
+	/* we don't want refs expressed as absolute paths */
+	if (*cp == '/')
+		return CHECK_REF_FORMAT_ERROR;
+
 	level = 0;
 	while (1) {
 		while ((ch = *cp++) == '/')
-- 
1.7.5.2.354.g349bf
Junio C Hamano· Aug 24, 2011, 22:27 UTC · re: Junio C Hamano · lore

Re: Buggy handling of non-canonical ref names

Junio C Hamano <gitster@pobox.com> writes:
Show 15 quoted lines
> Michael Haggerty <mhagger@alum.mit.edu> writes:
>
>> What is the policy about reference names and their canonicalization?
>
> The overall policy has been that we care about well-formed input, and
> everything else is "undefined", even though as you found out some of them
> try to work sensibly.
>
>>     $ git check-ref-format /foo/bar ; echo $?
>>     0
>>
>>     $ git check-ref-format --print /foo/bar
>>     /foo/bar
>
> I think these are bogus. Patches welcome.

I actually think the former is correct and the latter should strip the leading slash. Essentially what "check-ref-format" (and the underlying check_ref_format() function) validates is if the given user string can be used under $GIT_DIR/refs/ to name a ref, and $GIT_DIR/refs//foo/bar is (because we "tolerate duplicated slashes" cf. comment in the function) the same as $GIT_DIR/refs/foo/bar and is allowed.

I do not know what the original motivation behind --print was, but it seems to want to show it canonicalized, so it should strip the leading slash as well.

I think what is missing is a unified way to canonicalize the refnames (which led to the inconsistencies you observed), and I strongly suspect that check_ref_format() should learn to return the canonicalized format (if asked by the caller) and the caller should use the canonicalized version after feeding end-user input to it.

Then the plumbing "check-ref-format --print" can use it just like any other caller that should be (or already are) using check_ref_format() to validate the end-user input.

Yes, such a change will update the overall policy I stated earlier and narrow the scope of "undefined" down a bit, by uniformly codifying that leading and duplicate slashes are removed to be nice to the user.

Michael Haggerty· Aug 25, 2011, 07:54 UTC · re: Junio C Hamano · lore

Re: Buggy handling of non-canonical ref names

On 08/25/2011 12:27 AM, Junio C Hamano wrote:
Show 22 quoted lines
> Junio C Hamano <gitster@pobox.com> writes:
>> Michael Haggerty <mhagger@alum.mit.edu> writes:
>>> What is the policy about reference names and their canonicalization?
>>
>> The overall policy has been that we care about well-formed input, and
>> everything else is "undefined", even though as you found out some of them
>> try to work sensibly.
>>
>>>     $ git check-ref-format /foo/bar ; echo $?
>>>     0
>>>
>>>     $ git check-ref-format --print /foo/bar
>>>     /foo/bar
>>
>> I think these are bogus. Patches welcome.
> 
> I actually think the former is correct and the latter should strip the
> leading slash. Essentially what "check-ref-format" (and the underlying
> check_ref_format() function) validates is if the given user string can be
> used under $GIT_DIR/refs/ to name a ref, and $GIT_DIR/refs//foo/bar is
> (because we "tolerate duplicated slashes" cf. comment in the function) the
> same as $GIT_DIR/refs/foo/bar and is allowed.

I can live with either way, but I should point out that such an extra slash can be problematic when used naively in conjunction with Python's standard glue-together-pathname function, os.path.join() [1]:

    $ python
    >>> import os
    >>> os.path.join('.git', '/foo/bar')
    '/foo/bar'
    >>>
Maybe there are other examples of libraries with these semantics.
Show 9 quoted lines
> I think what is missing is a unified way to canonicalize the refnames
> (which led to the inconsistencies you observed), and I strongly suspect
> that check_ref_format() should learn to return the canonicalized format
> (if asked by the caller) and the caller should use the canonicalized
> version after feeding end-user input to it.
> 
> Then the plumbing "check-ref-format --print" can use it just like any
> other caller that should be (or already are) using check_ref_format()
> to validate the end-user input.

Indeed, regardless of the policy about leading slashes, this is a good plan. I will try to find time to work on it.

> Yes, such a change will update the overall policy I stated earlier and
> narrow the scope of "undefined" down a bit, by uniformly codifying that
> leading and duplicate slashes are removed to be nice to the user.
Michael
[1] http://docs.python.org/library/os.path.html#os.path.join
-- 
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/
Michael Haggerty· Aug 25, 2011, 08:08 UTC · re: Michael Haggerty · lore

[PATCH] Do not allow refnames to start with a slash

Previously, refnames with leading slashes were handled inconsistently. So forbid them altogether.

Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
---
This patch chooses the alternative of forbidding leading slashes,
because I feel that it is the safer option.  It is like Carlos's patch
but with documentation changes, test-suite additions, and an
corresponding change to git.py.  If anybody knows of other
implementations of the ref format checking code, please let me know.

This patch applies to either master or maint. Since it restricts the DWIM behavior in some cases, master is probably the better choice.

 Documentation/git-check-ref-format.txt |    7 ++++---
 git_remote_helpers/git/git.py          |    3 ++-
 refs.c                                 |    3 +++
 t/t1402-check-ref-format.sh            |    4 ++++
 4 files changed, 13 insertions(+), 4 deletions(-)
diff --git a/Documentation/git-check-ref-format.txt b/Documentation/git-check-ref-format.txt
index c9fdf84..238b185 100644
--- a/Documentation/git-check-ref-format.txt
+++ b/Documentation/git-check-ref-format.txt
@@ -34,15 +34,16 @@ git imposes the following rules on how references are named:
   category like `heads/`, `tags/` etc. but the actual names are not
   restricted.
 
-. They cannot have two consecutive dots `..` anywhere.
+. They must not start or end with slash `/`.
+
+. They cannot have two consecutive dots `..` anywhere or end with a
+  dot `.`.
 
 . They cannot have ASCII control characters (i.e. bytes whose
   values are lower than \040, or \177 `DEL`), space, tilde `~`,
   caret `{caret}`, colon `:`, question-mark `?`, asterisk `*`,
   or open bracket `[` anywhere.
 
-. They cannot end with a slash `/` nor a dot `.`.
-
 . They cannot end with the sequence `.lock`.
 
 . They cannot contain a sequence `@{`.
diff --git a/git_remote_helpers/git/git.py b/git_remote_helpers/git/git.py
index a383e6c..6df53aa 100644
--- a/git_remote_helpers/git/git.py
+++ b/git_remote_helpers/git/git.py
@@ -55,7 +55,8 @@ def valid_git_ref (ref_name):
     # command.  The rules were derived from the git check-ref-format(1)
     # manual page.  This code should be replaced by a call to
     # check_ref_format() in the git library, when such is available.
-    if ref_name.endswith('/') or \
+    if ref_name.startswith('/') or \
+       ref_name.endswith('/') or \
        ref_name.startswith('.') or \
        ref_name.count('/.') or \
        ref_name.count('..') or \
diff --git a/refs.c b/refs.c
index 6f313a9..84c5af3 100644
--- a/refs.c
+++ b/refs.c
@@ -880,6 +880,9 @@ int check_ref_format(const char *ref)
 	const char *cp = ref;
 
 	level = 0;
+	if (*cp == '/')
+		/* no leading slashes */
+		return CHECK_REF_FORMAT_ERROR;
 	while (1) {
 		while ((ch = *cp++) == '/')
 			; /* tolerate duplicated slashes */
diff --git a/t/t1402-check-ref-format.sh b/t/t1402-check-ref-format.sh
index 1b0f82f..b05ca26 100755
--- a/t/t1402-check-ref-format.sh
+++ b/t/t1402-check-ref-format.sh
@@ -18,6 +18,10 @@ invalid_ref 'foo'
 valid_ref 'foo/bar/baz'
 valid_ref 'refs///heads/foo'
 invalid_ref 'heads/foo/'
+invalid_ref '/foo'
+invalid_ref 'foo/'
+invalid_ref '/foo/bar'
+invalid_ref 'foo/bar/'
 invalid_ref './foo'
 invalid_ref '.refs/foo'
 invalid_ref 'heads/foo..bar'
-- 
1.7.6.8.gd2879
Junio C Hamano· Aug 25, 2011, 18:17 UTC · re: Michael Haggerty · lore

Re: [PATCH] Do not allow refnames to start with a slash

Michael Haggerty <mhagger@alum.mit.edu> writes:
Show 8 quoted lines
> diff --git a/t/t1402-check-ref-format.sh b/t/t1402-check-ref-format.sh
> index 1b0f82f..b05ca26 100755
> --- a/t/t1402-check-ref-format.sh
> +++ b/t/t1402-check-ref-format.sh
> @@ -18,6 +18,10 @@ invalid_ref 'foo'
>  valid_ref 'foo/bar/baz'
>  valid_ref 'refs///heads/foo'
>  invalid_ref 'heads/foo/'
> +invalid_ref '/foo'
> +invalid_ref '/foo/bar'

As refs///heads/foo is defined to be valid_ref, I am moderately against this change, which can break existing scripts.

> +invalid_ref 'foo/'
> +invalid_ref 'foo/bar/'

We already check trailing slash in existing checks, so I do not see a point in these two additional lines.

>  invalid_ref './foo'
>  invalid_ref '.refs/foo'
>  invalid_ref 'heads/foo..bar'

I was about to write "other than that the patch looks good", but then realized there is nothing remaining after rejecting the new "now slash cannot come at the beginning".

Michael Haggerty· Aug 25, 2011, 19:19 UTC · re: Junio C Hamano · lore

[PATCH] check-ref-format --print: Normalize refnames that start with slashes

And add tests that such refnames are accepted and normalized correctly.

Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
---

I didn't convince Junio that it is preferable to reject refnames that start with slashes, so go the other way instead.

This patch should be applied to maint, as it fixes a bug. The normalization should probably be moved to refs.[ch] as mentioned on the mailing list, but that can be done separately, on master.

 builtin/check-ref-format.c  |    6 ++++--
 t/t1402-check-ref-format.sh |    7 +++++++
 2 files changed, 11 insertions(+), 2 deletions(-)
diff --git a/builtin/check-ref-format.c b/builtin/check-ref-format.c
index ae3f281..7118021 100644
--- a/builtin/check-ref-format.c
+++ b/builtin/check-ref-format.c
@@ -12,8 +12,8 @@ static const char builtin_check_ref_format_usage[] =
 "   or: git check-ref-format --branch <branchname-shorthand>";
 
 /*
- * Replace each run of adjacent slashes in src with a single slash,
- * and write the result to dst.
+ * Remove leading slashes and replace each run of adjacent slashes in
+ * src with a single slash, and write the result to dst.
  *
  * This function is similar to normalize_path_copy(), but stripped down
  * to meet check_ref_format's simpler needs.
@@ -23,6 +23,8 @@ static void collapse_slashes(char *dst, const char *src)
 	char ch;
 	char prev = '\0';
 
+	while (*src == '/')
+		src++;
 	while ((ch = *src++) != '\0') {
 		if (prev == '/' && ch == prev)
 			continue;
diff --git a/t/t1402-check-ref-format.sh b/t/t1402-check-ref-format.sh
index 1b0f82f..e6fafb2 100755
--- a/t/t1402-check-ref-format.sh
+++ b/t/t1402-check-ref-format.sh
@@ -18,6 +18,9 @@ invalid_ref 'foo'
 valid_ref 'foo/bar/baz'
 valid_ref 'refs///heads/foo'
 invalid_ref 'heads/foo/'
+valid_ref '/heads/foo'
+valid_ref '///heads/foo'
+invalid_ref '/foo'
 invalid_ref './foo'
 invalid_ref '.refs/foo'
 invalid_ref 'heads/foo..bar'
@@ -28,6 +31,7 @@ valid_ref 'heads/foo@bar'
 invalid_ref 'heads/v@{ation'
 invalid_ref 'heads/foo\bar'
 
+
 test_expect_success "check-ref-format --branch @{-1}" '
 	T=$(git write-tree) &&
 	sha1=$(echo A | git commit-tree $T) &&
@@ -70,7 +74,10 @@ invalid_ref_normalized() {
 
 valid_ref_normalized 'heads/foo' 'heads/foo'
 valid_ref_normalized 'refs///heads/foo' 'refs/heads/foo'
+valid_ref_normalized '/heads/foo' 'heads/foo'
+valid_ref_normalized '///heads/foo' 'heads/foo'
 invalid_ref_normalized 'foo'
+invalid_ref_normalized '/foo'
 invalid_ref_normalized 'heads/foo/../bar'
 invalid_ref_normalized 'heads/./foo'
 invalid_ref_normalized 'heads\foo'
-- 
1.7.6.8.gd2879
Junio C Hamano· Aug 25, 2011, 20:42 UTC · re: Michael Haggerty · lore

Re: [PATCH] check-ref-format --print: Normalize refnames that start with slashes

Michael Haggerty <mhagger@alum.mit.edu> writes:
> And add tests that such refnames are accepted and normalized
> correctly.
Thanks. I think this is a bit simpler, though...
 builtin/check-ref-format.c  |    4 +---
 t/t1402-check-ref-format.sh |    1 -
 2 files changed, 1 insertions(+), 4 deletions(-)
diff --git b/builtin/check-ref-format.c a/builtin/check-ref-format.c
index 7118021..0723cf2 100644
--- b/builtin/check-ref-format.c
+++ a/builtin/check-ref-format.c
@@ -21,10 +21,8 @@ static const char builtin_check_ref_format_usage[] =
 static void collapse_slashes(char *dst, const char *src)
 {
 	char ch;
-	char prev = '\0';
+	char prev = '/';
 
-	while (*src == '/')
-		src++;
 	while ((ch = *src++) != '\0') {
 		if (prev == '/' && ch == prev)
 			continue;
diff --git b/t/t1402-check-ref-format.sh a/t/t1402-check-ref-format.sh
index e6fafb2..7563043 100755
--- b/t/t1402-check-ref-format.sh
+++ a/t/t1402-check-ref-format.sh
@@ -31,7 +31,6 @@ valid_ref 'heads/foo@bar'
 invalid_ref 'heads/v@{ation'
 invalid_ref 'heads/foo\bar'
 
-
 test_expect_success "check-ref-format --branch @{-1}" '
 	T=$(git write-tree) &&
 	sha1=$(echo A | git commit-tree $T) &&
Michael Haggerty· Aug 25, 2011, 21:57 UTC · re: Junio C Hamano · lore

Re: [PATCH] check-ref-format --print: Normalize refnames that start with slashes

On 08/25/2011 10:42 PM, Junio C Hamano wrote:
Show 6 quoted lines
> Michael Haggerty <mhagger@alum.mit.edu> writes:
> 
>> And add tests that such refnames are accepted and normalized
>> correctly.
> 
> Thanks. I think this is a bit simpler, though [...]
Indeed.
Michael
-- 
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/

← back to recent threads