threads / patch / 44316

patchsubmodule--helper: normalize funny urls

Subject: [PATCH] submodule--helper: normalize funny urls

## tl;dr

8 messages between Oct 17, 2016 and Oct 18, 2016. Diffs are folded; open one to read it.

replies: 7people: 3as markdown or json

Stefan Beller· Oct 17, 2016, 22:16 UTC · lore
Currently a URL for the superproject ending in

(A) .../path/to/dir (B) .../path/to/dir/ (C) .../path/to/dir/. (D) .../path/to/dir/./. (E) .../path/to/dir/.///.//.

is treated the same in (A) and (B), but (C, D, E) are different.

We never produce the URLs in (C,D,E) ourselves, they come to use, because the user used it as the URL for cloning a superproject. Normalize these paths.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 By being strict in Git, I think we also fix the Git for Windows painpoints.
 
 This goes on top of origin/sb/submodule-ignore-trailing-slash.
 
 Thanks,
 Stefan
 builtin/submodule--helper.c | 49 ++++++++++++++++++++++++++++++++++-----------
 t/t0060-path-utils.sh       | 11 ++++++----
 2 files changed, 44 insertions(+), 16 deletions(-)
Show changes to 2 files +44 −16

builtin/submodule--helper.c, t/t0060-path-utils.sh

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 260f46f..ca90763 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -76,6 +76,30 @@ static int chop_last_dir(char **remoteurl, int is_relative)
 	return 0;
 }
 
+static void strip_url_ending(char *url, size_t *_len)
+{
+	int check_url_stripping = 1;
+	size_t len = _len ? *_len : strlen(url);
+
+	while (check_url_stripping) {
+		check_url_stripping = 0;
+		if (is_dir_sep(url[len-2]) && url[len-1] == '.') {
+			url[len-2] = '\0';
+			len -= 2;
+			check_url_stripping = 1;
+		}
+
+		if (is_dir_sep(url[len-1])) {
+			url[len-1] = '\0';
+			len --;
+			check_url_stripping = 1;
+		}
+	}
+
+	if (_len)
+		*_len = len;
+}
+
 /*
  * The `url` argument is the URL that navigates to the submodule origin
  * repo. When relative, this URL is relative to the superproject origin
@@ -93,14 +117,16 @@ static int chop_last_dir(char **remoteurl, int is_relative)
  * the superproject working tree otherwise.
  *
  * NEEDSWORK: This works incorrectly on the domain and protocol part.
- * remote_url      url              outcome          expectation
- * http://a.com/b  ../c             http://a.com/c   as is
- * http://a.com/b/ ../c             http://a.com/c   same as previous line, but
- *                                                   ignore trailing slash in url
- * http://a.com/b  ../../c          http://c         error out
- * http://a.com/b  ../../../c       http:/c          error out
- * http://a.com/b  ../../../../c    http:c           error out
- * http://a.com/b  ../../../../../c    .:c           error out
+ * remote_url       url              outcome          expectation
+ * http://a.com/b   ../c             http://a.com/c   as is
+ * http://a.com/b/  ../c             http://a.com/c   same as previous line, but
+ *                                                    ignore trailing '/' in url
+ * http://a.com/b/. ../c             http://a.com/c   same as previous line, but
+ *                                                    ignore trailing '/.' in url
+ * http://a.com/b   ../../c          http://c         error out
+ * http://a.com/b   ../../../c       http:/c          error out
+ * http://a.com/b   ../../../../c    http:c           error out
+ * http://a.com/b   ../../../../../c    .:c           error out
  * NEEDSWORK: Given how chop_last_dir() works, this function is broken
  * when a local part has a colon in its path component, too.
  */
@@ -115,8 +141,7 @@ static char *relative_url(const char *remote_url,
 	struct strbuf sb = STRBUF_INIT;
 	size_t len = strlen(remoteurl);
 
-	if (is_dir_sep(remoteurl[len-1]))
-		remoteurl[len-1] = '\0';
+	strip_url_ending(remoteurl, &len);
 
 	if (!url_is_local_not_ssh(remoteurl) || is_absolute_path(remoteurl))
 		is_relative = 0;
@@ -149,10 +174,10 @@ static char *relative_url(const char *remote_url,
 	}
 	strbuf_reset(&sb);
 	strbuf_addf(&sb, "%s%s%s", remoteurl, colonsep ? ":" : "/", url);
-	if (ends_with(url, "/"))
-		strbuf_setlen(&sb, sb.len - 1);
 	free(remoteurl);
 
+	strip_url_ending(sb.buf, &sb.len);
+
 	if (starts_with_dot_slash(sb.buf))
 		out = xstrdup(sb.buf + 2);
 	else
diff --git a/t/t0060-path-utils.sh b/t/t0060-path-utils.sh
index 25b48e5..e154e5f 100755
--- a/t/t0060-path-utils.sh
+++ b/t/t0060-path-utils.sh
@@ -329,14 +329,17 @@ test_submodule_relative_url "(null)" "./foo" "../submodule" "submodule"
 test_submodule_relative_url "(null)" "//somewhere else/repo" "../subrepo" "//somewhere else/subrepo"
 test_submodule_relative_url "(null)" "$PWD/subsuper_update_r" "../subsubsuper_update_r" "$(pwd)/subsubsuper_update_r"
 test_submodule_relative_url "(null)" "$PWD/super_update_r2" "../subsuper_update_r" "$(pwd)/subsuper_update_r"
-test_submodule_relative_url "(null)" "$PWD/." "../." "$(pwd)/."
-test_submodule_relative_url "(null)" "$PWD" "./." "$(pwd)/."
+test_submodule_relative_url "(null)" "$PWD/sub/." "../." "$(pwd)"
+test_submodule_relative_url "(null)" "$PWD/sub/./." "../." "$(pwd)"
+test_submodule_relative_url "(null)" "$PWD/sub/.////././/./." "../." "$(pwd)"
+test_submodule_relative_url "(null)" "$PWD" "./." "$(pwd)"
 test_submodule_relative_url "(null)" "$PWD/addtest" "../repo" "$(pwd)/repo"
 test_submodule_relative_url "(null)" "$PWD" "./å äö" "$(pwd)/å äö"
-test_submodule_relative_url "(null)" "$PWD/." "../submodule" "$(pwd)/submodule"
+test_submodule_relative_url "(null)" "$PWD/sub" "../submodule" "$(pwd)/submodule"
+test_submodule_relative_url "(null)" "$PWD/sub/." "../submodule" "$(pwd)/submodule"
 test_submodule_relative_url "(null)" "$PWD/submodule" "../submodule" "$(pwd)/submodule"
 test_submodule_relative_url "(null)" "$PWD/home2/../remote" "../bundle1" "$(pwd)/home2/../bundle1"
-test_submodule_relative_url "(null)" "$PWD/submodule_update_repo" "./." "$(pwd)/submodule_update_repo/."
+test_submodule_relative_url "(null)" "$PWD/submodule_update_repo" "./." "$(pwd)/submodule_update_repo"
 test_submodule_relative_url "(null)" "file:///tmp/repo" "../subrepo" "file:///tmp/subrepo"
 test_submodule_relative_url "(null)" "foo/bar" "../submodule" "foo/submodule"
 test_submodule_relative_url "(null)" "foo" "../submodule" "submodule"
-- 
2.10.1.480.g573bd76
Junio C Hamano· Oct 17, 2016, 22:49 UTC · re: Stefan Beller · lore

Re: [PATCH] submodule--helper: normalize funny urls

Stefan Beller <sbeller@google.com> writes:
Show 8 quoted lines
> +static void strip_url_ending(char *url, size_t *_len)
> +{
> +	int check_url_stripping = 1;
> +	size_t len = _len ? *_len : strlen(url);
> +
> +	while (check_url_stripping) {
> +		check_url_stripping = 0;
> +		if (is_dir_sep(url[len-2]) && url[len-1] == '.') {
This is "strip /. at the end" it seems.
Does anything in the loop control guarantees 2 <= len at this point?
Show 6 quoted lines
> +			url[len-2] = '\0';
> +			len -= 2;
> +			check_url_stripping = 1;
> +		}
> +
> +		if (is_dir_sep(url[len-1])) {
This is "strip / at the end" it seems.
Does anything in the loop control guarantees 1 <= len at this point?
Show 5 quoted lines
> +			url[len-1] = '\0';
> +			len--;
> +			check_url_stripping = 1;
> +		}
> +	}
Stefan Beller· Oct 17, 2016, 22:50 UTC · re: Junio C Hamano · lore

Re: [PATCH] submodule--helper: normalize funny urls

On Mon, Oct 17, 2016 at 3:49 PM, Junio C Hamano <gitster@pobox.com> wrote:
Show 14 quoted lines
> Stefan Beller <sbeller@google.com> writes:
>
>> +static void strip_url_ending(char *url, size_t *_len)
>> +{
>> +     int check_url_stripping = 1;
>> +     size_t len = _len ? *_len : strlen(url);
>> +
>> +     while (check_url_stripping) {
>> +             check_url_stripping = 0;
>> +             if (is_dir_sep(url[len-2]) && url[len-1] == '.') {
>
> This is "strip /. at the end" it seems.
>
> Does anything in the loop control guarantees 2 <= len at this point?

Oh, thanks for pointing that out. I thought about that and missed to add it. I'll reroll with the length check once we hear back from Windows folks, that this is a viable strategy for them, too.

Thanks, Stefan

Show 17 quoted lines
>
>> +                     url[len-2] = '\0';
>> +                     len -= 2;
>> +                     check_url_stripping = 1;
>> +             }
>> +
>> +             if (is_dir_sep(url[len-1])) {
>
> This is "strip / at the end" it seems.
>
> Does anything in the loop control guarantees 1 <= len at this point?
>
>> +                     url[len-1] = '\0';
>> +                     len--;
>> +                     check_url_stripping = 1;
>> +             }
>> +     }
Johannes Schindelin· Oct 18, 2016, 11:23 UTC · re: Stefan Beller · lore

Re: [PATCH] submodule--helper: normalize funny urls

Hi Stefan,
On Mon, 17 Oct 2016, Stefan Beller wrote:
Show 19 quoted lines
> On Mon, Oct 17, 2016 at 3:49 PM, Junio C Hamano <gitster@pobox.com> wrote:
> > Stefan Beller <sbeller@google.com> writes:
> >
> >> +static void strip_url_ending(char *url, size_t *_len)
> >> +{
> >> +     int check_url_stripping = 1;
> >> +     size_t len = _len ? *_len : strlen(url);
> >> +
> >> +     while (check_url_stripping) {
> >> +             check_url_stripping = 0;
> >> +             if (is_dir_sep(url[len-2]) && url[len-1] == '.') {
> >
> > This is "strip /. at the end" it seems.
> >
> > Does anything in the loop control guarantees 2 <= len at this point?
> 
> Oh, thanks for pointing that out. I thought about that and missed to add it.
> I'll reroll with the length check once we hear back from Windows folks,
> that this is a viable strategy for them, too.
It is a viable strategy for me, too.

Thanks, Dscho

Junio C Hamano· Oct 18, 2016, 17:15 UTC · re: Junio C Hamano · lore

Re: [PATCH] submodule--helper: normalize funny urls

Junio C Hamano <gitster@pobox.com> writes:
Show 31 quoted lines
> Stefan Beller <sbeller@google.com> writes:
>
>> +static void strip_url_ending(char *url, size_t *_len)
>> +{
>> +	int check_url_stripping = 1;
>> +	size_t len = _len ? *_len : strlen(url);
>> +
>> +	while (check_url_stripping) {
>> +		check_url_stripping = 0;
>> +		if (is_dir_sep(url[len-2]) && url[len-1] == '.') {
>
> This is "strip /. at the end" it seems.
>
> Does anything in the loop control guarantees 2 <= len at this point?
>
>> +			url[len-2] = '\0';
>> +			len -= 2;
>> +			check_url_stripping = 1;
>> +		}
>> +
>> +		if (is_dir_sep(url[len-1])) {
>
> This is "strip / at the end" it seems.
>
> Does anything in the loop control guarantees 1 <= len at this point?
>
>> +			url[len-1] = '\0';
>> +			len--;
>> +			check_url_stripping = 1;
>> +		}
>> +	}
I also somehow find the "check-url-stripping" variable ugly.
	while (URL still has something that could be stripped) {
		if (ends with "/.") {
			strip "/.";
			continue;
		}
		if (ends with "/") {
			strip "/";
			continue;
		}
		break;
	}
perhaps?
Stefan Beller· Oct 18, 2016, 17:17 UTC · re: Junio C Hamano · lore

Re: [PATCH] submodule--helper: normalize funny urls

On Tue, Oct 18, 2016 at 10:15 AM, Junio C Hamano <gitster@pobox.com> wrote:
>
> I also somehow find the "check-url-stripping" variable ugly.
>
>         while (URL still has something that could be stripped) {
for(;;) {

here ? (this code would not need a variable, and for wins over while: $ git grep "while (1)" |wc -l 107 $ git grep "for (;;)" |wc -l 128 )

Show 12 quoted lines
>                 if (ends with "/.") {
>                         strip "/.";
>                         continue;
>                 }
>                 if (ends with "/") {
>                         strip "/";
>                         continue;
>                 }
>                 break;
>         }
>
> perhaps?
Junio C Hamano· Oct 18, 2016, 17:51 UTC · re: Stefan Beller · lore

Re: [PATCH] submodule--helper: normalize funny urls

Stefan Beller <sbeller@google.com> writes:
Show 9 quoted lines
> for(;;) {
>
> here ? (this code would not need a variable, and
> for wins over while:
> $ git grep "while (1)" |wc -l
> 107
> $ git grep "for (;;)" |wc -l
> 128
> )

I dunno; the numbers tells me there is no strong preference by wide margin either way.

I am not sure if the end shape does not really need an exit condition. If there is a need for one, "for (; condition;)" would look strange. If there isn't, "for (;;)" is actually my personal preference over "while (1)".

Junio C Hamano· Oct 18, 2016, 15:24 UTC · re: Stefan Beller · lore

Re: [PATCH] submodule--helper: normalize funny urls

Stefan Beller <sbeller@google.com> writes:
Show 9 quoted lines
> Currently a URL for the superproject ending in
>
> (A)    .../path/to/dir
> (B)    .../path/to/dir/
> (C)    .../path/to/dir/.
> (D)    .../path/to/dir/./.
> (E)    .../path/to/dir/.///.//.
>
> is treated the same in (A) and (B), but (C, D, E) are different.

You may know what you meant to say with "treated", but the readers do not know "treated" in what situation you are talking about. We need to tell the readers that the bug being fixed is about resolving a relative URL "../<something>" off of the URL of the superproject to compute the remote URL for a submodule repository.

> We never produce the URLs in (C,D,E) ourselves, they come to use, because
> the user used it as the URL for cloning a superproject.
> Normalize these paths.

As you know the externally-visible impact of this change (which I asked you, but didn't see an on-list answer to, by the way), please describe what this means to the end user in the log message. It is normally done in an earlier part of the log message to describe the problem being solved and its background.

If I understand the issue correctly, it may go like this:
	The remote URL for the submodule can be specified relative
	to the URL of the superproject in .gitmodules.  A top-level
	git://site.xz/toplevel.git can specify in its .gitmodules
		[submodule "sub"]
			url = ../submodule.git
			path = sub
	to say that git://site.xz/submodule.git is where the
	submodule bound at its "sub/" is found.
	However, when the toplevel is cloned like this:
		git clone git://site.xz/toplevel.git/. top
	i.e. the toplevel specifies its URL with trailing "/.", the
	code set the URL to git://site.xz/toplevel.git/submodule.git
	for the submodule, which is nonsense.  This was because the
	logic failed to treat trailing "/." any differently from
	trailing "/<anything-without-slash>" when resolving a
	relative URL "../<something>" off of it.  Stripping "/." at
	the end does *not* take you one level up, even though
	stripping "/<anything-without-slash>" does!
	
And then describe the solution/bugfix, listing A-C (or A-E) and
telling that these will be treated the same way.

← back to recent threads