threads / patch / 14826

patch, 5 partsFix 'url.*.insteadOf' for submodule URLs

Subject: [PATCH 0/5] Fix 'url.*.insteadOf' for submodule URLs

## tl;dr

12 messages between Aug 3, 2008 and Aug 4, 2008. Diffs are folded; open one to read it.

replies: 11people: 3as markdown or json

Johan Herland· Aug 3, 2008, 22:57 UTC · lore

As suggested in a thread some time ago, one could redefine the URL used to fetch submodules by adding a 'url.*.insteadOf' rule prior to the first invocation of 'git submodule update'.

However, this does not work with current Git, because the super-repo config (which is home to the 'url.*.insteadOf' rule) is not consulted by the 'git clone' that is invoked by 'git submodule update'.

These patches fix this issue by making 'git submodule' explicitly rewrite the submodule URL according to the super-repo config, prior to calling 'git clone'.

In order for the 'git submodule' shell script to properly rewrite URLs, it must gain access to the URL rewriting functionality in remote.c. To expose the URL rewriting functionality to 'git submodule' (and others, if needed), a new option ('rewrite-url') has been added to 'git config'.

The patch series is based on master. Whether or not this should be considered for v1.6.0 is to be decided by Junio. (My personal opinion is that although we're late in the release cycle, the patches are fairly straightforward, and should not pose a great risk of regressions.)

Johan Herland (5):
  Add testcase for 'git submodule' with url.*.insteadOf set in the
    super-repo
  Teach 'git config' to rewrite URLs according to current
    url.*.insteadOf rules
  Add selftest for new option '--rewrite-url' to 'git config'
  Add documentation on the new --rewrite-url option to 'git config'
  Teach 'git submodule' to rewrite submodule URLs according to
    super-repo's rules
 Documentation/git-config.txt |   10 ++++++++++
 builtin-config.c             |   23 ++++++++++++++++++++++-
 git-submodule.sh             |    6 ++++++
 t/t1300-repo-config.sh       |   14 ++++++++++++++
 t/t7400-submodule-basic.sh   |   11 +++++++++++
 5 files changed, 63 insertions(+), 1 deletions(-)
Have fun!
...Johan
Johan Herland· Aug 3, 2008, 23:00 UTC · re: Johan Herland · lore

[PATCH 1/5] Add testcase for 'git submodule' with url.*.insteadOf set in the super-repo

Currently, setting url.*.insteadOf in the super-repo in order to rewrite submodule URLs, don't work. When cloning/fetching the submodule, the super-repo config is never consulted, and thus the url.*.insteadOf rule is never seen. This adds a testcase that confirms the current behaviour.

Signed-off-by: Johan Herland <johan@herland.net>
---
 t/t7400-submodule-basic.sh |   11 +++++++++++
 1 files changed, 11 insertions(+), 0 deletions(-)
Show changes to t/t7400-submodule-basic.sh +11 −0
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index cbc0c34..bafc46c 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -209,4 +209,15 @@ test_expect_success 'update --init' '
 
 '
 
+test_expect_failure 'update --init with url.*.insteadOf' '
+
+	rm -rf init &&
+	git config -f .gitmodules submodule.example.url "http://example.com/init2" 
&&
+	git config --remove-section submodule.example
+	git config "url.$(pwd)/.insteadOf" "http://example.com/" &&
+	git submodule update --init init &&
+	test -d init/.git
+
+'
+
 test_done
-- 
1.6.0.rc1.34.g0fe8c
Johan Herland· Aug 3, 2008, 23:04 UTC · re: Johan Herland · lore

[PATCH 1/5 v2] Add testcase for 'git submodule' with url.*.insteadOf set in the super-repo

Currently, setting url.*.insteadOf in the super-repo in order to rewrite submodule URLs, don't work. When cloning/fetching the submodule, the super-repo config is never consulted, and thus the url.*.insteadOf rule is never seen. This adds a testcase that confirms the current behaviour.

Signed-off-by: Johan Herland <johan@herland.net>
---
 t/t7400-submodule-basic.sh |   11 +++++++++++
 1 files changed, 11 insertions(+), 0 deletions(-)
The previous patch was whitespace-damaged. Sorry. Trying again.
...Johan
Show changes to t/t7400-submodule-basic.sh +11 −0
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index cbc0c34..bafc46c 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -209,4 +209,15 @@ test_expect_success 'update --init' '
 
 '
 
+test_expect_failure 'update --init with url.*.insteadOf' '
+
+	rm -rf init &&
+	git config -f .gitmodules submodule.example.url "http://example.com/init2" &&
+	git config --remove-section submodule.example
+	git config "url.$(pwd)/.insteadOf" "http://example.com/" &&
+	git submodule update --init init &&
+	test -d init/.git
+
+'
+
 test_done
-- 
1.6.0.rc1.34.g0fe8c
Johan Herland· Aug 3, 2008, 23:01 UTC · re: Johan Herland · lore

[PATCH 2/5] Teach 'git config' to rewrite URLs according to current url.*.insteadOf rules

This patch adds the --rewrite-url option to 'git config'. The option takes exactly one argument; a URL that is to be rewritten according to the longest matching url.*.insteadOf rule in the current config. The resulting URL is printed on stdout, and 0 is returned.

The rationale for this patch is to enable access to Git's URL rewriting functionality from shell scripts.

The URL rewriting functionality is implemented by piggybacking on the existing URL rewriting code in remote.c.

Signed-off-by: Johan Herland <johan@herland.net>
---
 builtin-config.c |   23 ++++++++++++++++++++++-
 1 files changed, 22 insertions(+), 1 deletions(-)
Show changes to builtin-config.c +22 −1
diff --git a/builtin-config.c b/builtin-config.c
index 91fdc49..977e6cb 100644
--- a/builtin-config.c
+++ b/builtin-config.c
@@ -1,9 +1,10 @@
 #include "builtin.h"
 #include "cache.h"
 #include "color.h"
+#include "remote.h"
 
 static const char git_config_set_usage[] =
-"git config [ --global | --system | [ -f | --file ] config-file ] [ --bool | --int | --bool-or-int ] [ -z | --null ] 
[--get | --get-all | --get-regexp | --replace-all | --add | --unset | --unset-all] name [value [value_regex]] | --rename-section old_name new_name | --remove-section name | --list | --get-color 
var [default] | --get-colorbool name [stdout-is-tty]";
+"git config [ --global | --system | [ -f | --file ] config-file ] [ --bool | --int | --bool-or-int ] [ -z | --null ] 
[--get | --get-all | --get-regexp | --replace-all | --add | --unset | --unset-all] name [value [value_regex]] | --rename-section old_name new_name | --remove-section name | --list | --get-color 
var [default] | --get-colorbool name [stdout-is-tty] | --rewrite-url url";
 
 static char *key;
 static regex_t *key_regexp;
@@ -281,6 +282,24 @@ static int get_colorbool(int argc, const char **argv)
 	}
 }
 
+static int rewrite_url(int argc, const char **argv)
+{
+	/*
+	 * git config --rewrite_url <url>
+	 *
+	 * returns <url> after rewriting it according to url.*.insteadOf rules.
+	 */
+
+	if (argc > 1)
+		usage(git_config_set_usage);
+
+	struct remote *remote = remote_get(argv[0]);
+	if (remote->url_nr != 1)
+		die("Expected exactly one URL from remote_get()!");
+	printf("%s\n", remote->url[0]);
+	return 0;
+}
+
 int cmd_config(int argc, const char **argv, const char *prefix)
 {
 	int nongit;
@@ -362,6 +381,8 @@ int cmd_config(int argc, const char **argv, const char *prefix)
 			return get_color(argc-2, argv+2);
 		} else if (!strcmp(argv[1], "--get-colorbool")) {
 			return get_colorbool(argc-2, argv+2);
+		} else if (!strcmp(argv[1], "--rewrite-url")) {
+			return rewrite_url(argc-2, argv+2);
 		} else
 			break;
 		argc--;
-- 
1.6.0.rc1.34.g0fe8c
Johan Herland· Aug 3, 2008, 23:02 UTC · re: Johan Herland · lore

[PATCH 3/5] Add selftest for new option '--rewrite-url' to 'git config'

Simple check that URL rewriting functionality works as expected.
Signed-off-by: Johan Herland <johan@herland.net>
---
 t/t1300-repo-config.sh |   14 ++++++++++++++
 1 files changed, 14 insertions(+), 0 deletions(-)
Show changes to t/t1300-repo-config.sh +14 −0
diff --git a/t/t1300-repo-config.sh b/t/t1300-repo-config.sh
index 64567fb..dcd04c2 100755
--- a/t/t1300-repo-config.sh
+++ b/t/t1300-repo-config.sh
@@ -741,4 +741,18 @@ test_expect_success 'symlinked configuration' '
 
 '
 
+test_expect_success '--rewrite-url' '
+
+	git config url.ssh://example.com/foo/.insteadOf http://example.com/ &&
+	git config url.../.insteadOf git://example.com/ &&
+	a=$(git config --rewrite-url http://example.com/baz/xyzzy.git) &&
+	test "zssh://example.com/foo/baz/xyzzy.git" = "z$a" &&
+	b=$(git config --rewrite-url git://example.com/baz/xyzzy.git) &&
+	test "z../baz/xyzzy.git" = "z$b" &&
+	c=$(git config --rewrite-url ssh://example.com/foo/baz/xyzzy.git) &&
+	test "zssh://example.com/foo/baz/xyzzy.git" = "z$c" &&
+	d=$(git config --rewrite-url rsync://example.com/baz/xyzzy.git) &&
+	test "zrsync://example.com/baz/xyzzy.git" = "z$d"
+'
+
 test_done
-- 
1.6.0.rc1.34.g0fe8c
Johan Herland· Aug 3, 2008, 23:02 UTC · re: Johan Herland · lore

[PATCH 4/5] Add documentation on the new --rewrite-url option to 'git config'

Signed-off-by: Johan Herland <johan@herland.net>
---
 Documentation/git-config.txt |   10 ++++++++++
 1 files changed, 10 insertions(+), 0 deletions(-)
Show changes to Documentation/git-config.txt +10 −0
diff --git a/Documentation/git-config.txt b/Documentation/git-config.txt
index 28e1861..154c80b 100644
--- a/Documentation/git-config.txt
+++ b/Documentation/git-config.txt
@@ -22,6 +22,7 @@ SYNOPSIS
 'git config' [<file-option>] [-z|--null] -l | --list
 'git config' [<file-option>] --get-color name [default]
 'git config' [<file-option>] --get-colorbool name [stdout-is-tty]
+'git config' [<file-option>] --rewrite-url url
 
 DESCRIPTION
 -----------
@@ -157,6 +158,15 @@ See also <<FILES>>.
 	output.  The optional `default` parameter is used instead, if
 	there is no color configured for `name`.
 
+--rewrite-url url::
+
+	Rewrite `url` according to the longest matching URL rewriting rule
+	(see documentation on `url.<base>.insteadOf` for more information
+	on URL rewriting rules), and output the resulting URL to the
+	standard output. If there is no matching URL rewriting rule, the
+	original `url` is printed on the standard output. In either case,
+	the exit code is 0.
+
 [[FILES]]
 FILES
 -----
-- 
1.6.0.rc1.34.g0fe8c
Johan Herland· Aug 3, 2008, 23:02 UTC · re: Johan Herland · lore

[PATCH 5/5] Teach 'git submodule' to rewrite submodule URLs according to super-repo's rules

When a 'url.*.insteadOf' rule in the superrepo matches a submodule URL, we should rewrite the submodule URL accordingly, so that we don't request the submodule from location that is inaccessible (or unwanted by the user for some other reason).

Signed-off-by: Johan Herland <johan@herland.net>
---
 git-submodule.sh           |    6 ++++++
 t/t7400-submodule-basic.sh |    2 +-
 2 files changed, 7 insertions(+), 1 deletions(-)
Show changes to 2 files +7 −1

git-submodule.sh, t/t7400-submodule-basic.sh

diff --git a/git-submodule.sh b/git-submodule.sh
index b40f876..ea2aa3b 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -96,6 +96,12 @@ module_clone()
 	test -e "$path" &&
 	die "A file already exist at path '$path'"
 
+	# The user may have added url.*.insteadOf rules intending to rewrite
+	# submodule URLs. We must explicitly do this rewrite before calling
+	# 'git clone', since 'git clone' will not consult the super-repo
+	# config and thus never see any url.*.insteadOf rules placed therein.
+	url=$(git config --rewrite-url "$url")
+
 	git-clone -n "$url" "$path" ||
 	die "Clone of '$url' into submodule path '$path' failed"
 }
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index bafc46c..9c56de2 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -209,7 +209,7 @@ test_expect_success 'update --init' '
 
 '
 
-test_expect_failure 'update --init with url.*.insteadOf' '
+test_expect_success 'update --init with url.*.insteadOf' '
 
 	rm -rf init &&
 	git config -f .gitmodules submodule.example.url "http://example.com/init2" &&
-- 
1.6.0.rc1.34.g0fe8c
Johannes Schindelin· Aug 3, 2008, 23:27 UTC · re: Johan Herland · lore

Re: [PATCH 0/5] Fix 'url.*.insteadOf' for submodule URLs

Hi,
On Mon, 4 Aug 2008, Johan Herland wrote:
> As suggested in a thread some time ago, one could redefine the URL used 
> to fetch submodules by adding a 'url.*.insteadOf' rule prior to the 
> first invocation of 'git submodule update'.

If I suggested it, but forgot the "--global" flag to "git config", I apologize.

Ciao, Dscho

Johan Herland· Aug 3, 2008, 23:47 UTC · re: Johannes Schindelin · lore

Re: [PATCH 0/5] Fix 'url.*.insteadOf' for submodule URLs

On Monday 04 August 2008, Johannes Schindelin wrote:
Show 7 quoted lines
> On Mon, 4 Aug 2008, Johan Herland wrote:
> > As suggested in a thread some time ago, one could redefine the URL used
> > to fetch submodules by adding a 'url.*.insteadOf' rule prior to the
> > first invocation of 'git submodule update'.
>
> If I suggested it, but forgot the "--global" flag to "git config", I
> apologize.

Does this mean that you don't agree with the rationale for this patch? I.e. that submodule URLs should not be rewritten according to the rules in the super-repo (but instead require such rules to be set in the user's global config)?

There are (at least) two reasons for why I think this should work without having to use '--global':

1. Consistency: Other git commands in the supermodule does _not_ require the 
URL rewriting rule to reside in the global config. Why should 'git 
submodule' be different.
2. I believe there are valid use cases for adding URL rewriting rules to the 
repo config instead of the global config. You may want to check out Fred's 
version of project X (including submodules), without making your other 
clones of project X start cloning/fetching from Fred.

Puzzled, ...Johan

-- 
Johan Herland, <johan@herland.net>
www.herland.net
Junio C Hamano· Aug 4, 2008, 02:06 UTC · re: Johan Herland · lore

Re: [PATCH 0/5] Fix 'url.*.insteadOf' for submodule URLs

Johan Herland <johan@herland.net> writes:
> 1. Consistency: Other git commands in the supermodule does _not_ require the 
> URL rewriting rule to reside in the global config. Why should 'git 
> submodule' be different.

When it comes to "submodules", I do not think such consistency argument makes much sense. "git submodule" command crosses module boundary, normal commands don't. They are naturally different and they should be.

Your kind of consistency means breaking the separation between module boundary, doesn't it?

Having said that...
> 2. I believe there are valid use cases for adding URL rewriting rules to the 
> repo config instead of the global config. You may want to check out Fred's 
> version of project X (including submodules), without making your other 
> clones of project X start cloning/fetching from Fred.

I think you are referring to the example given in an earlier thread to peek what your neighbor did between you two, without affecting other people.

Personally I think it is partly showing the shortcoming of the current "git submodule" that minimally supports the workflow to follow what the canonical repository does, and partly showing that it is an abuse of that interface to rewrite config file to temporarily switch to peek somewhere else in such a workflow.

Let's step back and think what we would do if there is no submodule involved. That is, you usually follow origin, but you temporarily want to peek at what Fred did. How would you do this?

	$ git fetch $fred $branch_fred_wants_you_to_review
        $ git checkout FETCH_HEAD ;# this detaches HEAD.

And you take a look around. Perhaps you like the change and decide to merge that to your branch. Perhaps you create your own branch on top of that state, build a few fix-up commits, and give the result back to Fred.

Shouldn't peeking what Fred did in the whole submodule hierarchy be essentially the same thing? That is,

	$ git submodule for-each-submodule sh -c '
                git fetch "$fred/$1" $branch_fred_wants_you_to_review &&
                git checkout FETCH_HEAD
	' -

where "for-each-submodule" would iterate over the submodules in the current superproject that you are interested in (that is, you actually have corresponding repositories there), and runs any given command with the path to the submodule in that directory.

Hmm?
Johan Herland· Aug 4, 2008, 07:52 UTC · re: Junio C Hamano · lore

Re: [PATCH 0/5] Fix 'url.*.insteadOf' for submodule URLs

On Monday 04 August 2008, Junio C Hamano wrote:
Show 11 quoted lines
> Johan Herland <johan@herland.net> writes:
> > 1. Consistency: Other git commands in the supermodule does _not_
> > require the URL rewriting rule to reside in the global config. Why
> > should 'git submodule' be different.
>
> When it comes to "submodules", I do not think such consistency argument
> makes much sense.  "git submodule" command crosses module boundary,
> normal commands don't.  They are naturally different and they should be.
>
> Your kind of consistency means breaking the separation between module
> boundary, doesn't it?

To a certain degree, yes, but (1) only for this specific config rule (url.*.insteadOf), and (2) only at the specific point in time when the submodule is cloned. This means that after the submodule is cloned, it is indeed not affected by the super-repo's config. E.g. you can change the origin URL of the submodule back to the original (un-rewritten) URL in the submodule's config, and the super-repo will not try to rewrite it at again.

I believe that the module boundary should not be crossed in general. But this patch only "crosses" that boundary before it really exists (i.e. before the submodule has been cloned into the super-repo). The event of introducing a submodule into a super-repo, is AFAICS the _only_ event where I would consider to cross the module boundary (and then only for a good reason).

Show 10 quoted lines
> Having said that...
>
> > 2. I believe there are valid use cases for adding URL rewriting rules
> > to the repo config instead of the global config. You may want to check
> > out Fred's version of project X (including submodules), without making
> > your other clones of project X start cloning/fetching from Fred.
>
> I think you are referring to the example given in an earlier thread to
> peek what your neighbor did between you two, without affecting other
> people.
Correct.
Show 5 quoted lines
> Personally I think it is partly showing the shortcoming of the current
> "git submodule" that minimally supports the workflow to follow what the
> canonical repository does, and partly showing that it is an abuse of that
> interface to rewrite config file to temporarily switch to peek somewhere
> else in such a workflow.

Indeed. However, the development of submodules usability seems so slow that even though this use case is an ugly workaround, it's one I thought we'd have to live with for some time... [1]

Show 25 quoted lines
> Let's step back and think what we would do if there is no submodule
> involved.  That is, you usually follow origin, but you temporarily want
> to peek at what Fred did.  How would you do this?
>
> 	$ git fetch $fred $branch_fred_wants_you_to_review
> 	$ git checkout FETCH_HEAD ;# this detaches HEAD.
>
> And you take a look around.  Perhaps you like the change and decide to
> merge that to your branch.  Perhaps you create your own branch on top of
> that state, build a few fix-up commits, and give the result back to Fred.
>
> Shouldn't peeking what Fred did in the whole submodule hierarchy be
> essentially the same thing?  That is,
>
> 	$ git submodule for-each-submodule sh -c '
> 		git fetch "$fred/$1" $branch_fred_wants_you_to_review &&
> 		git checkout FETCH_HEAD
> 	' -
>
> where "for-each-submodule" would iterate over the submodules in the
> current superproject that you are interested in (that is, you actually
> have corresponding repositories there), and runs any given command with
> the path to the submodule in that directory.
>
> Hmm?

Yes, having such functionality in 'git submodule' would be wonderful. However, implementations of such functionality have been slow in coming, and apparently hard to implement without being workflow-agnostic (if I remember correctly).

In light of improvements to 'git submodule', feel free to disregard my patch series (although I still find 'git config --rewrite-url' useful for dry-testing my 'url.*.insteadOf' rules...).

Thanks for providing constructive feedback.
Have fun!
...Johan

[1]: Maybe 'git submodule' would improve more quickly if we ate our own dogfood, i.e. if we included submodules (e.g. gitk and git-gui) in git.git?

-- 
Johan Herland, <johan@herland.net>
www.herland.net
Junio C Hamano· Aug 3, 2008, 23:57 UTC · re: Johannes Schindelin · lore

Re: [PATCH 0/5] Fix 'url.*.insteadOf' for submodule URLs

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
Show 8 quoted lines
> On Mon, 4 Aug 2008, Johan Herland wrote:
>
>> As suggested in a thread some time ago, one could redefine the URL used 
>> to fetch submodules by adding a 'url.*.insteadOf' rule prior to the 
>> first invocation of 'git submodule update'.
>
> If I suggested it, but forgot the "--global" flag to "git config", I 
> apologize.
That's a nice way to say that the patchset is unnecessary ;-).

← back to recent threads