threads / discuss / 24481

URL decoding changed semantics of + in URLs

Subject: URL decoding changed semantics of + in URLs

## tl;dr

18 messages between Jul 23, 2010 and Aug 6, 2010.

replies: 17people: 7as markdown or json

Thomas Rast· Jul 23, 2010, 13:18 UTC · lore
Hi Jeff

As pointed out by Jasper St. Pierre on #git, it is no longer possible to clone

  git://git.gnome.org/gtk+

because your 9d2e942 (decode file:// and ssh:// URLs, 2010-05-23) decodes + characters in URLs to spaces in the http style. It was later fixed by ce83eda (url.c: "<scheme>://" part at the beginning should not be URL decoded, 2010-06-23) but the later part of the url still decodes + as space.

The tests that go along with the commit make it clear that it was an intended change. But the interesting thing is, I cannot find any reference in any RFC that + must have this meaning. In particular,

  http://www.ietf.org/rfc/rfc2396.txt

doesn't say much about + and the only escaping defined is the usual %xx style. So is there a standard that mandates this, or was it just a well-meaning but unnecessary backwards incompatible change?

-- 
Thomas Rast
trast@{inf,student}.ethz.ch
Thomas Rast· Jul 23, 2010, 13:21 UTC · re: Thomas Rast · lore

Re: URL decoding changed semantics of + in URLs

Thomas Rast wrote:
Show 6 quoted lines
> Hi Jeff
> 
> As pointed out by Jasper St. Pierre on #git, it is no longer possible
> to clone
> 
>   git://git.gnome.org/gtk+
Well, that reads wrong now that I'm seeing it again.

I of course meant to say that the above URL no longer works and must be spelled

    git://git.gnome.org/gtk%2B
instead.
-- 
Thomas Rast
trast@{inf,student}.ethz.ch
Ævar Arnfjörð Bjarmason· Jul 23, 2010, 14:10 UTC · re: Thomas Rast · lore

Re: URL decoding changed semantics of + in URLs

On Fri, Jul 23, 2010 at 13:18, Thomas Rast <trast@student.ethz.ch> wrote:
> doesn't say much about + and the only escaping defined is the usual
> %xx style.  So is there a standard that mandates this, or was it just
> a well-meaning but unnecessary backwards incompatible change?
+ and %20 are as far as I know only interchangable in *query strings*,
so having to clone 'git://git.gnome.org/gtk%2B' where you could
previously clone 'git://git.gnome.org/gtk+' is a bug. Git shouldn't be
changing that + to a %20.

I haven't followed why we need to escape + to %20 at all, even in the query string. E.g. curl(1) doesn't do that before sending requests to Apache, which can handle either one. The + v.s. %20 duality is always handled at the server AFAIK.

Jasper St. Pierre· Jul 23, 2010, 14:25 UTC · re: Ævar Arnfjörð Bjarmason · lore

Re: URL decoding changed semantics of + in URLs

Yep. http://www.ietf.org/rfc/rfc2396.txt defines '+' as a reserved character, but doesn't give a purpose for it. www-form-encoded replaces space with '+' but in a URL it can mean anything it wants.

On Fri, Jul 23, 2010 at 10:10 AM, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:

Show 16 quoted lines
> On Fri, Jul 23, 2010 at 13:18, Thomas Rast <trast@student.ethz.ch> wrote:
>
>> doesn't say much about + and the only escaping defined is the usual
>> %xx style.  So is there a standard that mandates this, or was it just
>> a well-meaning but unnecessary backwards incompatible change?
>
> + and %20 are as far as I know only interchangable in *query strings*,
> so having to clone 'git://git.gnome.org/gtk%2B' where you could
> previously clone 'git://git.gnome.org/gtk+' is a bug. Git shouldn't be
> changing that + to a %20.
>
> I haven't followed why we need to escape + to %20 at all, even in the
> query string. E.g. curl(1) doesn't do that before sending requests to
> Apache, which can handle either one. The + v.s. %20 duality is always
> handled at the server AFAIK.
>
Thomas Rast· Jul 23, 2010, 21:23 UTC · re: Ævar Arnfjörð Bjarmason · lore

[PATCH] Do not unquote + into ' ' in URLs

Since 9d2e942 (decode file:// and ssh:// URLs, 2010-05-23) the URL logic unquotes escaped URLs. For the %2B type of escape, this is conformant with RFC 2396. However, it also unquotes + into a space character, which is only appropriate for the query strings in HTTP. This notably broke fetching from the gtk+ repository.

Remove the corresponding bit of code.
Reported-by: Jasper St. Pierre <jstpierre@mecheye.net>
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
---
Jasper St. Pierre wrote:
> Yep. http://www.ietf.org/rfc/rfc2396.txt defines '+' as a reserved character,
> but doesn't give a purpose for it. www-form-encoded replaces space with '+'
> but in a URL it can mean anything it wants.
So let's do this then, instead?

Based on the discussion, I would consider this a bugfix that should go in 1.7.2.1.

 t/t5601-clone.sh |   10 ++++++++--
 url.c            |    5 +----
 2 files changed, 9 insertions(+), 6 deletions(-)
diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh
index 8abb71a..4431dfd 100755
--- a/t/t5601-clone.sh
+++ b/t/t5601-clone.sh
@@ -178,8 +178,14 @@ test_expect_success 'clone respects global branch.autosetuprebase' '
 
 test_expect_success 'respect url-encoding of file://' '
 	git init x+y &&
-	test_must_fail git clone "file://$PWD/x+y" xy-url &&
-	git clone "file://$PWD/x%2By" xy-url
+	git clone "file://$PWD/x+y" xy-url-1 &&
+	git clone "file://$PWD/x%2By" xy-url-2
+'
+
+test_expect_success 'do not query-string-decode + in URLs' '
+	rm -rf x+y &&
+	git init "x y" &&
+	test_must_fail git clone "file://$PWD/x+y" xy-no-plus
 '
 
 test_expect_success 'do not respect url-encoding of non-url path' '
diff --git a/url.c b/url.c
index 2306236..fa4b8d4 100644
--- a/url.c
+++ b/url.c
@@ -90,10 +90,7 @@ static char *url_decode_internal(const char **query, const char *stop_at, struct
 			}
 		}
 
-		if (c == '+')
-			strbuf_addch(out, ' ');
-		else
-			strbuf_addch(out, c);
+		strbuf_addch(out, c);
 		q++;
 	} while (1);
 	*query = q;
-- 
1.7.2.rc3.335.g26d7d
Ævar Arnfjörð Bjarmason· Jul 23, 2010, 22:20 UTC · re: Thomas Rast · lore

Re: [PATCH] Do not unquote + into ' ' in URLs

On Fri, Jul 23, 2010 at 21:23, Thomas Rast <trast@student.ethz.ch> wrote:
Show 67 quoted lines
> Since 9d2e942 (decode file:// and ssh:// URLs, 2010-05-23) the URL
> logic unquotes escaped URLs.  For the %2B type of escape, this is
> conformant with RFC 2396.  However, it also unquotes + into a space
> character, which is only appropriate for the query strings in HTTP.
> This notably broke fetching from the gtk+ repository.
>
> Remove the corresponding bit of code.
>
> Reported-by: Jasper St. Pierre <jstpierre@mecheye.net>
> Signed-off-by: Thomas Rast <trast@student.ethz.ch>
> ---
>
> Jasper St. Pierre wrote:
>> Yep. http://www.ietf.org/rfc/rfc2396.txt defines '+' as a reserved character,
>> but doesn't give a purpose for it. www-form-encoded replaces space with '+'
>> but in a URL it can mean anything it wants.
>
> So let's do this then, instead?
>
> Based on the discussion, I would consider this a bugfix that should go
> in 1.7.2.1.
>
>
>  t/t5601-clone.sh |   10 ++++++++--
>  url.c            |    5 +----
>  2 files changed, 9 insertions(+), 6 deletions(-)
>
> diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh
> index 8abb71a..4431dfd 100755
> --- a/t/t5601-clone.sh
> +++ b/t/t5601-clone.sh
> @@ -178,8 +178,14 @@ test_expect_success 'clone respects global branch.autosetuprebase' '
>
>  test_expect_success 'respect url-encoding of file://' '
>        git init x+y &&
> -       test_must_fail git clone "file://$PWD/x+y" xy-url &&
> -       git clone "file://$PWD/x%2By" xy-url
> +       git clone "file://$PWD/x+y" xy-url-1 &&
> +       git clone "file://$PWD/x%2By" xy-url-2
> +'
> +
> +test_expect_success 'do not query-string-decode + in URLs' '
> +       rm -rf x+y &&
> +       git init "x y" &&
> +       test_must_fail git clone "file://$PWD/x+y" xy-no-plus
>  '
>
>  test_expect_success 'do not respect url-encoding of non-url path' '
> diff --git a/url.c b/url.c
> index 2306236..fa4b8d4 100644
> --- a/url.c
> +++ b/url.c
> @@ -90,10 +90,7 @@ static char *url_decode_internal(const char **query, const char *stop_at, struct
>                        }
>                }
>
> -               if (c == '+')
> -                       strbuf_addch(out, ' ');
> -               else
> -                       strbuf_addch(out, c);
> +               strbuf_addch(out, c);
>                q++;
>        } while (1);
>        *query = q;
> --
> 1.7.2.rc3.335.g26d7d
>
This looks good, Ack.

But as icing, it'd be nice to extend these tests to create files / clone repositories with the rest of the reserved characters: http://en.wikipedia.org/wiki/Percent-encoding#Percent-encoding_reserved_characters

Of course that would have to be wrapped in something that skips the tests if those paths can't be created. E.g. "/" is a no-no on Unix, and some of the others will probably cause troubles on other systems.

Junio C Hamano· Jul 23, 2010, 22:26 UTC · re: Thomas Rast · lore

Re: [PATCH] Do not unquote + into ' ' in URLs

Thomas Rast <trast@student.ethz.ch> writes:
Show 5 quoted lines
> Since 9d2e942 (decode file:// and ssh:// URLs, 2010-05-23) the URL
> logic unquotes escaped URLs.  For the %2B type of escape, this is
> conformant with RFC 2396.  However, it also unquotes + into a space
> character, which is only appropriate for the query strings in HTTP.
> This notably broke fetching from the gtk+ repository.
Wait a minute.
> Based on the discussion, I would consider this a bugfix that should go
> in 1.7.2.1.

Some form of this may need to be applied to help the client side, but what will happen to

  http-backend.c::get_info_refs()
   -> http-backend.c::get_parameter()
     -> http-backend.c::get_parameters()
       -> url.c::url_decode_parameter_value()
         -> url.c::url_decode_internal()
codepath, which is the server-side handing of query strings?
Thomas Rast· Jul 23, 2010, 23:04 UTC · re: Junio C Hamano · lore

Re: [PATCH] Do not unquote + into ' ' in URLs

Junio C Hamano wrote:
Show 23 quoted lines
> Thomas Rast <trast@student.ethz.ch> writes:
> 
> > Since 9d2e942 (decode file:// and ssh:// URLs, 2010-05-23) the URL
> > logic unquotes escaped URLs.  For the %2B type of escape, this is
> > conformant with RFC 2396.  However, it also unquotes + into a space
> > character, which is only appropriate for the query strings in HTTP.
> > This notably broke fetching from the gtk+ repository.
> 
> Wait a minute.
> 
> > Based on the discussion, I would consider this a bugfix that should go
> > in 1.7.2.1.
> 
> Some form of this may need to be applied to help the client side, but what
> will happen to
> 
>   http-backend.c::get_info_refs()
>    -> http-backend.c::get_parameter()
>      -> http-backend.c::get_parameters()
>        -> url.c::url_decode_parameter_value()
>          -> url.c::url_decode_internal()
> 
> codepath, which is the server-side handing of query strings?

You're right, I forgot about those. I imagine it would be one of two cases:

* It should never have dequoted + before the ? query-delimiter, so we
  need to fix that too;
* It should never have dequoted + before the ? query-delimiter, but
  it's too late to change that now.
Shawn, can you help with this?
-- 
Thomas Rast
trast@{inf,student}.ethz.ch
Thomas Rast· Jul 24, 2010, 14:49 UTC · re: Thomas Rast · lore

[PATCH v2] Do not unquote + into ' ' in URLs

Since 9d2e942 (decode file:// and ssh:// URLs, 2010-05-23) the URL logic unquotes escaped URLs. For the %2B type of escape, this is conformant with RFC 2396. However, it also unquotes + into a space character, which is only appropriate for the query strings in HTTP. This notably broke fetching from the gtk+ repository.

We cannot just remove the corresponding code since the same url_decode_internal() is also used by the HTTP backend to decode query parameters. Introduce a new argument that controls whether the + decoding happens, and use it only in the (client-side) url_decode().

Reported-by: Jasper St. Pierre <jstpierre@mecheye.net>
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
---
I wrote:
Show 10 quoted lines
> Junio C Hamano wrote:
> > 
> >   http-backend.c::get_info_refs()
> >    -> http-backend.c::get_parameter()
> >      -> http-backend.c::get_parameters()
> >        -> url.c::url_decode_parameter_value()
> >          -> url.c::url_decode_internal()
> 
> You're right, I forgot about those.  I imagine it would be one of two
> cases:
[...]
> Shawn, can you help with this?
The third case, of course, is:
* It only uses these functions for parameter decoding, which of course
  was correct to begin with.

So after hopefully drinking enough coffee, I made this one. The catch is that I'm not entirely clear whether *not* decoding the + client-side anywhere in the URL is correct for http:// URLs? If the client decodes and re-encodes the URL, then the + would be turned into a %2B on the re-encoding. Then again maybe UI-facing URLs should never have a query part at all?

 t/t5601-clone.sh |   10 ++++++++--
 url.c            |   11 ++++++-----
 2 files changed, 14 insertions(+), 7 deletions(-)
diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh
index 8abb71a..4431dfd 100755
--- a/t/t5601-clone.sh
+++ b/t/t5601-clone.sh
@@ -178,8 +178,14 @@ test_expect_success 'clone respects global branch.autosetuprebase' '
 
 test_expect_success 'respect url-encoding of file://' '
 	git init x+y &&
-	test_must_fail git clone "file://$PWD/x+y" xy-url &&
-	git clone "file://$PWD/x%2By" xy-url
+	git clone "file://$PWD/x+y" xy-url-1 &&
+	git clone "file://$PWD/x%2By" xy-url-2
+'
+
+test_expect_success 'do not query-string-decode + in URLs' '
+	rm -rf x+y &&
+	git init "x y" &&
+	test_must_fail git clone "file://$PWD/x+y" xy-no-plus
 '
 
 test_expect_success 'do not respect url-encoding of non-url path' '
diff --git a/url.c b/url.c
index 2306236..cd8f74f 100644
--- a/url.c
+++ b/url.c
@@ -67,7 +67,8 @@ static int url_decode_char(const char *q)
 	return val;
 }
 
-static char *url_decode_internal(const char **query, const char *stop_at, struct strbuf *out)
+static char *url_decode_internal(const char **query, const char *stop_at,
+				 struct strbuf *out, int decode_plus)
 {
 	const char *q = *query;
 
@@ -90,7 +91,7 @@ static char *url_decode_internal(const char **query, const char *stop_at, struct
 			}
 		}
 
-		if (c == '+')
+		if (decode_plus && c == '+')
 			strbuf_addch(out, ' ');
 		else
 			strbuf_addch(out, c);
@@ -110,17 +111,17 @@ char *url_decode(const char *url)
 		strbuf_add(&out, url, colon - url);
 		url = colon;
 	}
-	return url_decode_internal(&url, NULL, &out);
+	return url_decode_internal(&url, NULL, &out, 0);
 }
 
 char *url_decode_parameter_name(const char **query)
 {
 	struct strbuf out = STRBUF_INIT;
-	return url_decode_internal(query, "&=", &out);
+	return url_decode_internal(query, "&=", &out, 1);
 }
 
 char *url_decode_parameter_value(const char **query)
 {
 	struct strbuf out = STRBUF_INIT;
-	return url_decode_internal(query, "&", &out);
+	return url_decode_internal(query, "&", &out, 1);
 }
-- 
1.7.2.278.g76edd.dirty
Jasper St. Pierre· Jul 31, 2010, 21:18 UTC · re: Thomas Rast · lore

Re: [PATCH v2] Do not unquote + into ' ' in URLs

Hmm.. could we get a follow-up on this patch? It's annoying because it's both backwards and forwards incompatible: the %2b workaround doesn't work in older versions.

On Sat, Jul 24, 2010 at 10:49 AM, Thomas Rast <trast@student.ethz.ch> wrote:
Show 115 quoted lines
> Since 9d2e942 (decode file:// and ssh:// URLs, 2010-05-23) the URL
> logic unquotes escaped URLs.  For the %2B type of escape, this is
> conformant with RFC 2396.  However, it also unquotes + into a space
> character, which is only appropriate for the query strings in HTTP.
> This notably broke fetching from the gtk+ repository.
>
> We cannot just remove the corresponding code since the same
> url_decode_internal() is also used by the HTTP backend to decode query
> parameters.  Introduce a new argument that controls whether the +
> decoding happens, and use it only in the (client-side) url_decode().
>
> Reported-by: Jasper St. Pierre <jstpierre@mecheye.net>
> Signed-off-by: Thomas Rast <trast@student.ethz.ch>
> ---
>
> I wrote:
>> Junio C Hamano wrote:
>> >
>> >   http-backend.c::get_info_refs()
>> >    -> http-backend.c::get_parameter()
>> >      -> http-backend.c::get_parameters()
>> >        -> url.c::url_decode_parameter_value()
>> >          -> url.c::url_decode_internal()
>>
>> You're right, I forgot about those.  I imagine it would be one of two
>> cases:
> [...]
>> Shawn, can you help with this?
>
> The third case, of course, is:
>
> * It only uses these functions for parameter decoding, which of course
>  was correct to begin with.
>
> So after hopefully drinking enough coffee, I made this one.  The catch
> is that I'm not entirely clear whether *not* decoding the +
> client-side anywhere in the URL is correct for http:// URLs?  If the
> client decodes and re-encodes the URL, then the + would be turned into
> a %2B on the re-encoding.  Then again maybe UI-facing URLs should
> never have a query part at all?
>
>
>  t/t5601-clone.sh |   10 ++++++++--
>  url.c            |   11 ++++++-----
>  2 files changed, 14 insertions(+), 7 deletions(-)
>
> diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh
> index 8abb71a..4431dfd 100755
> --- a/t/t5601-clone.sh
> +++ b/t/t5601-clone.sh
> @@ -178,8 +178,14 @@ test_expect_success 'clone respects global branch.autosetuprebase' '
>
>  test_expect_success 'respect url-encoding of file://' '
>        git init x+y &&
> -       test_must_fail git clone "file://$PWD/x+y" xy-url &&
> -       git clone "file://$PWD/x%2By" xy-url
> +       git clone "file://$PWD/x+y" xy-url-1 &&
> +       git clone "file://$PWD/x%2By" xy-url-2
> +'
> +
> +test_expect_success 'do not query-string-decode + in URLs' '
> +       rm -rf x+y &&
> +       git init "x y" &&
> +       test_must_fail git clone "file://$PWD/x+y" xy-no-plus
>  '
>
>  test_expect_success 'do not respect url-encoding of non-url path' '
> diff --git a/url.c b/url.c
> index 2306236..cd8f74f 100644
> --- a/url.c
> +++ b/url.c
> @@ -67,7 +67,8 @@ static int url_decode_char(const char *q)
>        return val;
>  }
>
> -static char *url_decode_internal(const char **query, const char *stop_at, struct strbuf *out)
> +static char *url_decode_internal(const char **query, const char *stop_at,
> +                                struct strbuf *out, int decode_plus)
>  {
>        const char *q = *query;
>
> @@ -90,7 +91,7 @@ static char *url_decode_internal(const char **query, const char *stop_at, struct
>                        }
>                }
>
> -               if (c == '+')
> +               if (decode_plus && c == '+')
>                        strbuf_addch(out, ' ');
>                else
>                        strbuf_addch(out, c);
> @@ -110,17 +111,17 @@ char *url_decode(const char *url)
>                strbuf_add(&out, url, colon - url);
>                url = colon;
>        }
> -       return url_decode_internal(&url, NULL, &out);
> +       return url_decode_internal(&url, NULL, &out, 0);
>  }
>
>  char *url_decode_parameter_name(const char **query)
>  {
>        struct strbuf out = STRBUF_INIT;
> -       return url_decode_internal(query, "&=", &out);
> +       return url_decode_internal(query, "&=", &out, 1);
>  }
>
>  char *url_decode_parameter_value(const char **query)
>  {
>        struct strbuf out = STRBUF_INIT;
> -       return url_decode_internal(query, "&", &out);
> +       return url_decode_internal(query, "&", &out, 1);
>  }
> --
> 1.7.2.278.g76edd.dirty
>
>
Thomas Rast· Jul 31, 2010, 21:33 UTC · re: Jasper St. Pierre · lore

Re: [PATCH v2] Do not unquote + into ' ' in URLs

Jasper St. Pierre wrote:
> Hmm.. could we get a follow-up on this patch? It's annoying because it's both
> backwards and forwards incompatible: the %2b workaround doesn't work in
> older versions.

I was hoping for an Ack from someone who knows the whole HTTP business, because I don't feel confident enough to say it won't break anything there.

To be precise, if the client ever attempts to decode URLs with query strings (for whatever reasons) then it would break. Probably there are no such URLs, but I don't know for sure.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch
Ralf Ebert· Aug 6, 2010, 10:46 UTC · re: Thomas Rast · lore

Re: [PATCH v2] Do not unquote + into ' ' in URLs

Hi,
> To be precise, if the client ever attempts to decode URLs with query
> strings (for whatever reasons) then it would break.  Probably there
> are no such URLs, but I don't know for sure.

url_decode seems to be called only from connect.c to support percent-encoding for git/ssh-URLs, so at least as of now, there should be no problem here. Since applying the patch, git clone 'git://git.gnome.org/gtk+' works ok again for me.

Greetings,
Ralf
Jeff King· Jul 26, 2010, 15:40 UTC · re: Thomas Rast · lore

Re: URL decoding changed semantics of + in URLs

On Fri, Jul 23, 2010 at 03:18:30PM +0200, Thomas Rast wrote:
Show 20 quoted lines
> As pointed out by Jasper St. Pierre on #git, it is no longer possible
> to clone
> 
>   git://git.gnome.org/gtk+
> 
> because your 9d2e942 (decode file:// and ssh:// URLs, 2010-05-23)
> decodes + characters in URLs to spaces in the http style.  It was
> later fixed by ce83eda (url.c: "<scheme>://" part at the beginning
> should not be URL decoded, 2010-06-23) but the later part of the url
> still decodes + as space.
> 
> The tests that go along with the commit make it clear that it was an
> intended change.  But the interesting thing is, I cannot find any
> reference in any RFC that + must have this meaning.  In particular,
> 
>   http://www.ietf.org/rfc/rfc2396.txt
> 
> doesn't say much about + and the only escaping defined is the usual
> %xx style.  So is there a standard that mandates this, or was it just
> a well-meaning but unnecessary backwards incompatible change?

Sorry for the slow reply. I am in the middle of moving and just got a usable machine running.

I think it is well-meaning but unnecessary. That is, I think %-decoding is probably still a good idea, and I just dragged '+'-decoding along out of cluelessness. I thought I cross-checked with what curl was doing to http URLs, but now I can't even get it to do anything but pass the path component literally to the webserver. So I'm really not sure what I tested before.

As Jasper noted, "+" is a reserved character, which means "gtk+" probably _should_ be escaped. But clearly it doesn't happen in practice, and I am more interested in not breaking current use than in nitpicking with a standard.

So I agree with the spirit of your patches, and reading over your v2, it looks fine to me, but I didn't do any extensive testing.

-Peff
Ævar Arnfjörð Bjarmason· Jul 26, 2010, 17:57 UTC · re: Jeff King · lore

Re: URL decoding changed semantics of + in URLs

On Mon, Jul 26, 2010 at 15:40, Jeff King <peff@peff.net> wrote:
> As Jasper noted, "+" is a reserved character, which means "gtk+"
> probably _should_ be escaped. But clearly it doesn't happen in practice,
> and I am more interested in not breaking current use than in nitpicking
> with a standard.

Reserved characters only have to be escaped in certain contexts, from RFC 2396:

   Many URI include components consisting of or delimited by, certain
   special characters.  These characters are called "reserved", since
   their usage within the URI component is limited to their reserved
   purpose.  If the data for a URI component would conflict with the
   reserved purpose, then the conflicting data must be escaped before
   forming the URI.
      reserved    = ";" | "/" | "?" | ":" | "@" | "&" | "=" | "+" |
                    "$" | ","

E.g. @ is special in the hostname part (http://user@example.org), but writing http://example.org/Git@Big%20companies:%20A%20Study is just fine.

Which is why curl passes it along literally, it *can* escape them, and real webservers like Apache handle reserved characters equivalently (in their unreserved contexts) whether they're escaped or not, but the git-daemon at git.gnome.org evidently doesn't implement RFC 2396 carefully enough.

So we shouldn't escape + for backwards compatibility and because it's not necessary, but we should probably also fix git-daemon to accept both forms if that hasn't been done already.

Jasper St. Pierre· Jul 26, 2010, 18:22 UTC · re: Ævar Arnfjörð Bjarmason · lore

Re: URL decoding changed semantics of + in URLs

Where is '+' used in the URL context? I don't see that it's a replacement for '+' aside from formencoded, which is from CGI, not HTTP or the URI spec.

I also can't access something called "test 2.txt" from Apache with "http://localhost/test+2.txt", so I don't think it's unescaping the '+'.

I don't think we should do anything about the '+' case, except where used in formencoded parameters (aka the "query string"), where it is used.

On Mon, Jul 26, 2010 at 1:57 PM, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:

Show 34 quoted lines
> On Mon, Jul 26, 2010 at 15:40, Jeff King <peff@peff.net> wrote:
>
>> As Jasper noted, "+" is a reserved character, which means "gtk+"
>> probably _should_ be escaped. But clearly it doesn't happen in practice,
>> and I am more interested in not breaking current use than in nitpicking
>> with a standard.
>
> Reserved characters only have to be escaped in certain contexts, from
> RFC 2396:
>
>   Many URI include components consisting of or delimited by, certain
>   special characters.  These characters are called "reserved", since
>   their usage within the URI component is limited to their reserved
>   purpose.  If the data for a URI component would conflict with the
>   reserved purpose, then the conflicting data must be escaped before
>   forming the URI.
>
>      reserved    = ";" | "/" | "?" | ":" | "@" | "&" | "=" | "+" |
>                    "$" | ","
>
> E.g. @ is special in the hostname part (http://user@example.org), but
> writing http://example.org/Git@Big%20companies:%20A%20Study is just
> fine.
>
> Which is why curl passes it along literally, it *can* escape them, and
> real webservers like Apache handle reserved characters equivalently
> (in their unreserved contexts) whether they're escaped or not, but the
> git-daemon at git.gnome.org evidently doesn't implement RFC 2396
> carefully enough.
>
> So we shouldn't escape + for backwards compatibility and because it's
> not necessary, but we should probably also fix git-daemon to accept
> both forms if that hasn't been done already.
>
Matthieu Moy· Jul 26, 2010, 18:30 UTC · re: Jasper St. Pierre · lore

Re: URL decoding changed semantics of + in URLs

"Jasper St. Pierre" <jstpierre@mecheye.net> writes:
> I don't think we should do anything about the '+' case, except where used in
> formencoded parameters (aka the "query string"), where it is used.
This is what the latest version of the patch does, AIUI, yes.
-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/
Ævar Arnfjörð Bjarmason· Jul 26, 2010, 18:35 UTC · re: Jasper St. Pierre · lore

Re: URL decoding changed semantics of + in URLs

On Mon, Jul 26, 2010 at 18:22, Jasper St. Pierre <jstpierre@mecheye.net> wrote:
> Where is '+' used in the URL context? I don't see that it's a
> replacement for '+'
> aside from formencoded, which is from CGI, not HTTP or the URI spec.

In my example? Nowhere, but I used : and @, which are also reserved characters.

> I also can't access something called "test 2.txt" from Apache with
> "http://localhost/test+2.txt", so I don't think it's unescaping the '+'.

Yes, that's not supposed to work. But you should be able to access "test+2.txt" using /test+2.txt and /test%2B.txt and /%74%65%73%74%2B2%2E%74%78%74 for that matter.

git-daemon only seems to handle the first form. Which is probably a bug, maybe it doesn't *have to* URI unescape its arguments, but it's probably a good idea anyway. E.g. some systems that handle URIs will convert : to %3A automatically when passing them through. That would break a git URL.

> I don't think we should do anything about the '+' case, except where used in
> formencoded parameters (aka the "query string"), where it is used.

I don't really have an opinion on what we should do. It's not a problem for me, I'm just noting how it could break, and that maybe we should try harder and support URI escaping where we handle URLs.

Jasper St. Pierre· Jul 26, 2010, 18:44 UTC · re: Ævar Arnfjörð Bjarmason · lore

Re: URL decoding changed semantics of + in URLs

Hm. I've always wondered about the percent double-encoding problem, where the "%" in "%2B" double-encodes to "%252B", which might require the mandatory server implementation then.

I do think that because "git://git.gnome.org/git+" was working, something on the server was going right.

On Mon, Jul 26, 2010 at 2:35 PM, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:

Show 28 quoted lines
> On Mon, Jul 26, 2010 at 18:22, Jasper St. Pierre <jstpierre@mecheye.net> wrote:
>> Where is '+' used in the URL context? I don't see that it's a
>> replacement for '+'
>> aside from formencoded, which is from CGI, not HTTP or the URI spec.
>
> In my example? Nowhere, but I used : and @, which are also reserved
> characters.
>
>> I also can't access something called "test 2.txt" from Apache with
>> "http://localhost/test+2.txt", so I don't think it's unescaping the '+'.
>
> Yes, that's not supposed to work. But you should be able to access
> "test+2.txt" using /test+2.txt and /test%2B.txt and
> /%74%65%73%74%2B2%2E%74%78%74 for that matter.
>
> git-daemon only seems to handle the first form. Which is probably a
> bug, maybe it doesn't *have to* URI unescape its arguments, but it's
> probably a good idea anyway. E.g. some systems that handle URIs will
> convert : to %3A automatically when passing them through. That would
> break a git URL.
>
>> I don't think we should do anything about the '+' case, except where used in
>> formencoded parameters (aka the "query string"), where it is used.
>
> I don't really have an opinion on what we should do. It's not a
> problem for me, I'm just noting how it could break, and that maybe we
> should try harder and support URI escaping where we handle URLs.
>

← back to recent threads