threads / patch / 12188

patch, 2 partsAdd support for host aliases in config files

Subject: [PATCH 2/2] Add support for host aliases in config files

## tl;dr

11 messages between Feb 19, 2008 and Feb 20, 2008. Diffs are folded; open one to read it.

replies: 10people: 5as markdown or json

Daniel Barkalow· Feb 19, 2008, 04:28 UTC · lore

This allows users with different preferences for access methods to the same remote repositories to rewrite each other's URLs by pattern matching across a large set of similiarly set up repositories to each get the desired access.

For example, if you don't have a kernel.org account, you might want settings like:

[host "kernel.org"]
      base = git://git.kernel.org/pub/
      rewritebase = master.kernel.org:/pub
Then, if you give git a URL like:
  master.kernel.org:/pub/scm/linux/kernel/git/linville/wireless-2.6.git
it will act like you gave it:
  git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-2.6.git

and you can cut-and-paste pull requests in email without fixing them by hand, for example.

Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
---
I believe this takes care of all the comments I've gotten on this patch.

I looked at how the documentation appears in (for example) git-fetch.1 with the change to urls.txt, and it looks informative to me and not unbalanced. Of course, in order to be perfectly accurate, git clone needs to be converted to using the same code as everything else, because it won't yet actually support these alises, unfortunately for my dreams of being able to set up a default host and then have "git clone :module" work.

 Documentation/config.txt |   23 ++++++++++
 Documentation/urls.txt   |   25 +++++++++++
 remote.c                 |  102 ++++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 147 insertions(+), 3 deletions(-)
Show changes to 3 files +147 −3

Documentation/config.txt, Documentation/urls.txt, remote.c

diff --git a/Documentation/config.txt b/Documentation/config.txt
index f2f6a77..7a9cba4 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -646,6 +646,29 @@ help.format::
 	Values 'man', 'info', 'web' and 'html' are supported. 'man' is
 	the default. 'web' and 'html' are the same.
 
+host.<name>.*::
+	These options provide a way to rewrite URLs when there is a
+	pattern of URLs with a common prefix which should be replaced
+	with a different prefix. For every 'rewritebase' prefix given,
+	any URL git receives that starts with that prefix will be
+	rewritten to have the 'base' prefix instead. In cases where
+	some site serves a large number of repositories, and serves
+	them with multiple access methods, and some users need to use
+	different access methods, this feature allows people to
+	specify any of the equivalent URLs and have git automatically
+	rewrite the URL to the best alternative for the particular
+	user, even for a never-before-seen repository on the site. The
+	effect of multiple matching rewritebase prefixes is undefined.
+
+host.<name>.base::
+	The base URL which should be used for this particular
+	host.
+
+host.<name>.rewritebase::
+	Additional base URLs which refer to this host. If a URL
+	matches this, any access to it will use the URL formed with
+	the corresponding base URL instead of the given URL.
+
 http.proxy::
 	Override the HTTP proxy, normally configured using the 'http_proxy'
 	environment variable (see linkgit:curl[1]).  This can be overridden
diff --git a/Documentation/urls.txt b/Documentation/urls.txt
index 81ac17f..11c05db 100644
--- a/Documentation/urls.txt
+++ b/Documentation/urls.txt
@@ -44,3 +44,28 @@ endif::git-clone[]
 ifdef::git-clone[]
 They are equivalent, except the former implies --local option.
 endif::git-clone[]
+
+
+If there are a large number of similarly-named remote repositories and
+you want to use a different format for them (such that the URLs you
+use will be rewritten into URLs that work), you can create a
+configuration section of the form:
+
+------------
+	[host "<host>"]
+		base = <actual url base>
+		rewritebase = <other url base>
+------------
+
+If you have a section:
+
+------------
+	[host "xz"]
+		base = git://git.host.xz/
+		rewritebase = host.xz:/path/to/
+		rewritebase = work:
+------------
+
+a URL like "work:repo.git" or like "host.xz:/path/to/repo.git" will be
+rewritten in any context that takes a URL to be
+"git://git.host.xz/repo.git".
diff --git a/remote.c b/remote.c
index 457d8a4..fc83e2d 100644
--- a/remote.c
+++ b/remote.c
@@ -2,6 +2,14 @@
 #include "remote.h"
 #include "refs.h"
 
+struct host {
+	const char *name;
+	const char *base;
+	const char **alias;
+	int alias_nr;
+	int alias_alloc;
+};
+
 static struct remote **remotes;
 static int remotes_alloc;
 static int remotes_nr;
@@ -13,9 +21,33 @@ static int branches_nr;
 static struct branch *current_branch;
 static const char *default_remote_name;
 
+static struct host **hosts;
+static int hosts_alloc;
+static int hosts_nr;
+
 #define BUF_SIZE (2048)
 static char buffer[BUF_SIZE];
 
+static const char *alias_url(const char *url)
+{
+	int i, j;
+	for (i = 0; i < hosts_nr; i++) {
+		if (!hosts[i])
+			continue;
+		for (j = 0; j < hosts[i]->alias_nr; j++) {
+			if (!prefixcmp(url, hosts[i]->alias[j])) {
+				char *ret = malloc(strlen(hosts[i]->base) -
+						   strlen(hosts[i]->alias[j]) +
+						   strlen(url) + 1);
+				strcpy(ret, hosts[i]->base);
+				strcat(ret, url + strlen(hosts[i]->alias[j]));
+				return ret;
+			}
+		}
+	}
+	return url;
+}
+
 static void add_push_refspec(struct remote *remote, const char *ref)
 {
 	ALLOC_GROW(remote->push_refspec,
@@ -38,6 +70,11 @@ static void add_url(struct remote *remote, const char *url)
 	remote->url[remote->url_nr++] = url;
 }
 
+static void add_url_alias(struct remote *remote, const char *url)
+{
+	add_url(remote, alias_url(url));
+}
+
 static struct remote *make_remote(const char *name, int len)
 {
 	struct remote *ret;
@@ -95,6 +132,35 @@ static struct branch *make_branch(const char *name, int len)
 	return ret;
 }
 
+static struct host *make_host(const char *name, int len)
+{
+	struct host *ret;
+	int i;
+
+	for (i = 0; i < hosts_nr; i++) {
+		if (len ? (!strncmp(name, hosts[i]->name, len) &&
+			   !hosts[i]->name[len]) :
+		    !strcmp(name, hosts[i]->name))
+			return hosts[i];
+	}
+
+	ALLOC_GROW(hosts, hosts_nr + 1, hosts_alloc);
+	ret = xcalloc(1, sizeof(struct host));
+	hosts[hosts_nr++] = ret;
+	if (len)
+		ret->name = xstrndup(name, len);
+	else
+		ret->name = xstrdup(name);
+
+	return ret;
+}
+
+static void add_alias(struct host *host, const char *name)
+{
+	ALLOC_GROW(host->alias, host->alias_nr + 1, host->alias_alloc);
+	host->alias[host->alias_nr++] = name;
+}
+
 static void read_remotes_file(struct remote *remote)
 {
 	FILE *f = fopen(git_path("remotes/%s", remote->name), "r");
@@ -128,7 +194,7 @@ static void read_remotes_file(struct remote *remote)
 
 		switch (value_list) {
 		case 0:
-			add_url(remote, xstrdup(s));
+			add_url_alias(remote, xstrdup(s));
 			break;
 		case 1:
 			add_push_refspec(remote, xstrdup(s));
@@ -180,7 +246,7 @@ static void read_branches_file(struct remote *remote)
 	} else {
 		branch = "refs/heads/master";
 	}
-	add_url(remote, p);
+	add_url_alias(remote, p);
 	add_fetch_refspec(remote, branch);
 	remote->fetch_tags = 1; /* always auto-follow */
 }
@@ -210,6 +276,23 @@ static int handle_config(const char *key, const char *value)
 		}
 		return 0;
 	}
+	if (!prefixcmp(key, "host.")) {
+		struct host *host;
+		name = key + 5;
+		subkey = strrchr(name, '.');
+		if (!subkey)
+			return 0;
+		host = make_host(name, subkey - name);
+		if (!strcmp(subkey, ".base")) {
+			if (!value)
+				return config_error_nonbool(key);
+			host->base = xstrdup(value);
+		} else if (!strcmp(subkey, ".rewritebase")) {
+			if (!value)
+				return config_error_nonbool(key);
+			add_alias(host, xstrdup(value));
+		}
+	}
 	if (prefixcmp(key,  "remote."))
 		return 0;
 	name = key + 7;
@@ -261,6 +344,18 @@ static int handle_config(const char *key, const char *value)
 	return 0;
 }
 
+static void alias_all_urls(void)
+{
+	int i, j;
+	for (i = 0; i < remotes_nr; i++) {
+		if (!remotes[i])
+			continue;
+		for (j = 0; j < remotes[i]->url_nr; j++) {
+			remotes[i]->url[j] = alias_url(remotes[i]->url[j]);
+		}
+	}
+}
+
 static void read_config(void)
 {
 	unsigned char sha1[20];
@@ -277,6 +372,7 @@ static void read_config(void)
 			make_branch(head_ref + strlen("refs/heads/"), 0);
 	}
 	git_config(handle_config);
+	alias_all_urls();
 }
 
 struct refspec *parse_ref_spec(int nr_refspec, const char **refspec)
@@ -342,7 +438,7 @@ struct remote *remote_get(const char *name)
 			read_branches_file(ret);
 	}
 	if (!ret->url)
-		add_url(ret, name);
+		add_url_alias(ret, name);
 	if (!ret->url)
 		return NULL;
 	ret->fetch = parse_ref_spec(ret->fetch_refspec_nr, ret->fetch_refspec);
-- 
1.5.4.1.191.gfbf10
Jon Loeliger· Feb 19, 2008, 15:11 UTC · re: Daniel Barkalow · lore

Re: [PATCH 2/2] Add support for host aliases in config files

Daniel Barkalow wrote:
Show 25 quoted lines
> This allows users with different preferences for access methods to the
> same remote repositories to rewrite each other's URLs by pattern
> matching across a large set of similiarly set up repositories to each
> get the desired access.
> 
> For example, if you don't have a kernel.org account, you might want
> settings like:
> 
> [host "kernel.org"]
>       base = git://git.kernel.org/pub/
>       rewritebase = master.kernel.org:/pub
> 
> Then, if you give git a URL like:
> 
>   master.kernel.org:/pub/scm/linux/kernel/git/linville/wireless-2.6.git
> 
> it will act like you gave it:
> 
>   git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-2.6.git
> 
> and you can cut-and-paste pull requests in email without fixing them
> by hand, for example.
> 
> Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
> ---

So, I may be dense, but I'm having a hard time distinguishing the names "base" and "rewritebase" just from a User Interface perspective. It's not clear to me which of those is the A and which is the B part in s/A/B/. "Rewritebase" might be the "from" basis, or it might be the "to" basis. Can we come up with more descriptive property names here?

Is the rewrite always just prefix substitution/replacement? What if there was some generic path rewrite needed? (KISS? :-))

Also, is "host" really the right config key? Maybe "rewritehost"? Or is this thought to be the basis of other (perhaps not rewrite related) properties of the same host as well?

Maybe it's just me.

Thanks, jdl

Johannes Schindelin· Feb 19, 2008, 15:49 UTC · re: Jon Loeliger · lore

Re: [PATCH 2/2] Add support for host aliases in config files

Hi,
On Tue, 19 Feb 2008, Jon Loeliger wrote:
Show 31 quoted lines
> Daniel Barkalow wrote:
>
> > For example, if you don't have a kernel.org account, you might want 
> > settings like:
> > 
> > [host "kernel.org"]
> >       base = git://git.kernel.org/pub/
> >       rewritebase = master.kernel.org:/pub
> > 
> > Then, if you give git a URL like:
> > 
> >   master.kernel.org:/pub/scm/linux/kernel/git/linville/wireless-2.6.git
> > 
> > it will act like you gave it:
> > 
> >   git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-2.6.git
> > 
> > and you can cut-and-paste pull requests in email without fixing them 
> > by hand, for example.
> > 
> > Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
> > ---
> 
> So, I may be dense, but I'm having a hard time distinguishing the names 
> "base" and "rewritebase" just from a User Interface perspective.  It's 
> not clear to me which of those is the A and which is the B part in 
> s/A/B/.  "Rewritebase" might be the "from" basis, or it might be the 
> "to" basis.  Can we come up with more descriptive property names here?
> 
> Is the rewrite always just prefix substitution/replacement?
> What if there was some generic path rewrite needed? (KISS? :-))
I just tried this:
$ git config rewrite."master.kernel.org:/pub".url \
	git://git.kernel.org/pub/
and it worked.  IOW, the config will look like this:
[rewrite "master.kernel.org:/pub"]
        url = git://git.kernel.org/pub/
So maybe this is easier to grasp?

Ciao, Dscho

Daniel Barkalow· Feb 19, 2008, 16:52 UTC · re: Johannes Schindelin · lore

Re: [PATCH 2/2] Add support for host aliases in config files

On Tue, 19 Feb 2008, Johannes Schindelin wrote:
Show 47 quoted lines
> Hi,
> 
> On Tue, 19 Feb 2008, Jon Loeliger wrote:
> 
> > Daniel Barkalow wrote:
> >
> > > For example, if you don't have a kernel.org account, you might want 
> > > settings like:
> > > 
> > > [host "kernel.org"]
> > >       base = git://git.kernel.org/pub/
> > >       rewritebase = master.kernel.org:/pub
> > > 
> > > Then, if you give git a URL like:
> > > 
> > >   master.kernel.org:/pub/scm/linux/kernel/git/linville/wireless-2.6.git
> > > 
> > > it will act like you gave it:
> > > 
> > >   git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-2.6.git
> > > 
> > > and you can cut-and-paste pull requests in email without fixing them 
> > > by hand, for example.
> > > 
> > > Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
> > > ---
> > 
> > So, I may be dense, but I'm having a hard time distinguishing the names 
> > "base" and "rewritebase" just from a User Interface perspective.  It's 
> > not clear to me which of those is the A and which is the B part in 
> > s/A/B/.  "Rewritebase" might be the "from" basis, or it might be the 
> > "to" basis.  Can we come up with more descriptive property names here?
> > 
> > Is the rewrite always just prefix substitution/replacement?
> > What if there was some generic path rewrite needed? (KISS? :-))
> 
> I just tried this:
> 
> $ git config rewrite."master.kernel.org:/pub".url \
> 	git://git.kernel.org/pub/
> 
> and it worked.  IOW, the config will look like this:
> 
> [rewrite "master.kernel.org:/pub"]
>         url = git://git.kernel.org/pub/
> 
> So maybe this is easier to grasp?

If you've also got people using http:// URLs, this would require giving the preferred URL twice. And it doesn't help with possible future host-based configuration, or a non-prefix pattern, if that came up some day.

I suppose perhaps:
[host "git://git.kernel.org/pub/"]
	aka = "master.kernel.org:/pub/"
	aka = "http://www.kernel.org/pub/"

I'm a little worried about putting URLs in keys, but I don't know if that's a reasonable worry.

	-Daniel
*This .sig left intentionally blank*
Jakub Narebski· Feb 20, 2008, 19:44 UTC · re: Johannes Schindelin · lore

Re: [PATCH 2/2] Add support for host aliases in config files

Johannes Schindelin wrotr:
 
Show 11 quoted lines
> I just tried this:
> 
> $ git config rewrite."master.kernel.org:/pub".url \
> 	git://git.kernel.org/pub/
> 
> and it worked.  IOW, the config will look like this:
> 
> [rewrite "master.kernel.org:/pub"]
>         url = git://git.kernel.org/pub/
> 
> So maybe this is easier to grasp?

Perhaps that it is easier to grasp, but imho it is backwards, as it is inherently 1 -> 1 in <original> -> <rewritten>, while I think it is more common to have n -> 1, i.e. multiple URLs (aliases), multiple alternative "spellings" with one tried way to access repository.

-- 
Jakub Narebski
Poland
Daniel Barkalow· Feb 19, 2008, 16:45 UTC · re: Jon Loeliger · lore

Re: [PATCH 2/2] Add support for host aliases in config files

On Tue, 19 Feb 2008, Jon Loeliger wrote:
Show 33 quoted lines
> Daniel Barkalow wrote:
> > This allows users with different preferences for access methods to the
> > same remote repositories to rewrite each other's URLs by pattern
> > matching across a large set of similiarly set up repositories to each
> > get the desired access.
> > 
> > For example, if you don't have a kernel.org account, you might want
> > settings like:
> > 
> > [host "kernel.org"]
> >       base = git://git.kernel.org/pub/
> >       rewritebase = master.kernel.org:/pub
> > 
> > Then, if you give git a URL like:
> > 
> >   master.kernel.org:/pub/scm/linux/kernel/git/linville/wireless-2.6.git
> > 
> > it will act like you gave it:
> > 
> >   git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-2.6.git
> > 
> > and you can cut-and-paste pull requests in email without fixing them
> > by hand, for example.
> > 
> > Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
> > ---
> 
> So, I may be dense, but I'm having a hard time distinguishing
> the names "base" and "rewritebase" just from a User Interface
> perspective.  It's not clear to me which of those is the A and
> which is the B part in s/A/B/.  "Rewritebase" might be the "from"
> basis, or it might be the "to" basis.  Can we come up with more
> descriptive property names here?

rewritebase is the base you rewrite, which follows apache usage (which is where somebody got the term to suggest in an earlier round). It seems like it should be clear; am I obfuscating it by showing the options in the more backwards order in the examples? If you've got an idea that people find more clear, I'd be happy to change it (again).

> Is the rewrite always just prefix substitution/replacement?
> What if there was some generic path rewrite needed? (KISS? :-))

It's just prefix substitution. I suppose a future version could do something more clever (but it would probably also use variables named "pattern" instead of "base", so there shouldn't be any issues).

> Also, is "host" really the right config key?  Maybe "rewritehost"?
> Or is this thought to be the basis of other (perhaps not rewrite
> related) properties of the same host as well?

I think it's likely that other properties of the host could show up. I suspect that uploadpack and receivepack would be useful host-wide if you were using a lot of ad-hoc repository names on a big site. Rewriting the URLs is just the itch I had with this scope.

	-Daniel
*This .sig left intentionally blank*
Junio C Hamano· Feb 19, 2008, 21:35 UTC · re: Daniel Barkalow · lore

Re: [PATCH 2/2] Add support for host aliases in config files

Daniel Barkalow <barkalow@iabervon.org> writes:
Show 10 quoted lines
> On Tue, 19 Feb 2008, Jon Loeliger wrote:
>
>> Also, is "host" really the right config key?  Maybe "rewritehost"?
>> Or is this thought to be the basis of other (perhaps not rewrite
>> related) properties of the same host as well?
>
> I think it's likely that other properties of the host could show up. I 
> suspect that uploadpack and receivepack would be useful host-wide if you 
> were using a lot of ad-hoc repository names on a big site. Rewriting the 
> URLs is just the itch I had with this scope.

I'd agree that per-host configuration may be useful, and your "paths to programs" may be a good example. But I think it is plausible that a host needs multiple <rewritefrom,rewriteto> tuples. Either your '[host "$this_url"] aka = $that_url' or Dscho's earlier '[rewrite "$this_url_to"] url = $that_url' shows that this is not about host but "prefix part of URL".

Daniel Barkalow· Feb 20, 2008, 00:02 UTC · re: Junio C Hamano · lore

Re: [PATCH 2/2] Add support for host aliases in config files

On Tue, 19 Feb 2008, Junio C Hamano wrote:
Show 17 quoted lines
> Daniel Barkalow <barkalow@iabervon.org> writes:
> 
> > On Tue, 19 Feb 2008, Jon Loeliger wrote:
> >
> >> Also, is "host" really the right config key?  Maybe "rewritehost"?
> >> Or is this thought to be the basis of other (perhaps not rewrite
> >> related) properties of the same host as well?
> >
> > I think it's likely that other properties of the host could show up. I 
> > suspect that uploadpack and receivepack would be useful host-wide if you 
> > were using a lot of ad-hoc repository names on a big site. Rewriting the 
> > URLs is just the itch I had with this scope.
> 
> I'd agree that per-host configuration may be useful, and your
> "paths to programs" may be a good example.  But I think it is
> plausible that a host needs multiple <rewritefrom,rewriteto>
> tuples.

That's probably true, even taking into account the option of having multiple froms for the same to.

> Either your '[host "$this_url"] aka = $that_url' or
> Dscho's earlier '[rewrite "$this_url_to"] url = $that_url' shows
> that this is not about host but "prefix part of URL".

Well, it shows that it's *possible* to do this feature that way, although I'm not entirely convinced that it's best to do it like that. Are there character set concerns with having urls in config keys?

	-Daniel
*This .sig left intentionally blank*
Johannes Schindelin· Feb 20, 2008, 01:01 UTC · re: Daniel Barkalow · lore

Re: [PATCH 2/2] Add support for host aliases in config files

Hi,
On Tue, 19 Feb 2008, Daniel Barkalow wrote:
Show 8 quoted lines
> On Tue, 19 Feb 2008, Junio C Hamano wrote:
> 
> > Either your '[host "$this_url"] aka = $that_url' or
> > Dscho's earlier '[rewrite "$this_url_to"] url = $that_url' shows
> > that this is not about host but "prefix part of URL".
> 
> Well, it shows that it's *possible* to do this feature that way, 
> although I'm not entirely convinced that it's best to do it like that. 

I think in the interest of simplicity, you do not want to have an extra (arbitrary) key.

> Are there character set concerns with having urls in config keys?

I think not. AFAICT the "extended base var" is allowed anything except for newlines. Everything else you can get at least escaped with '\'. If in doubt, consult get_extended_base_var() in config.c.

Hth, Dscho

Junio C Hamano· Feb 20, 2008, 01:04 UTC · re: Johannes Schindelin · lore

Re: [PATCH 2/2] Add support for host aliases in config files

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> ...  If 
> in doubt, consult get_extended_base_var() in config.c.

Perhaps it is time for Documentation/technical/api-config.txt ;-)?

Johannes Schindelin· Feb 20, 2008, 01:14 UTC · re: Junio C Hamano · lore

Re: [PATCH 2/2] Add support for host aliases in config files

Hi,
On Tue, 19 Feb 2008, Junio C Hamano wrote:
Show 5 quoted lines
> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> 
> > ...  If in doubt, consult get_extended_base_var() in config.c.
> 
> Perhaps it is time for Documentation/technical/api-config.txt ;-)?
*winces* I try to find the time...

Sorry, Dscho

← back to recent threads