threads / patch / 22796

patch, 3 partsDifferent views on a repository: HEAD mapping

Subject: [PATCH 3/3] Different views on a repository: HEAD mapping

## tl;dr

16 messages between Feb 24, 2010 and Feb 26, 2010. Diffs are folded; open one to read it.

replies: 15people: 6as markdown or json

Andreas Gruenbacher· Feb 24, 2010, 16:41 UTC · lore

[RFC][PATCH 0/3] Different views on a repository

Hello,

we have a use case with groups of repositories which share lots of objects, but which are logically independent. There is no strict hierarchy between the repositories, the development modl is arbitrary. The alternates mechanism for sharig objects between repositories won't work.

The best idea I came up with so far to solve this was to keep everything in the same repository on the server. Then, to keep the logically independent repositories separate, directories are used below refs/heads and refs/tags. Receive-pack and upload-pack are modified to hide this directory structure from clients so that repositories will continue to look "normal" to users. For example, the following structure on the server:

	refs/heads/one/master
	refs/tags/one/tag1
	refs/heads/two/master
	refs/heads/two/branch2
would appear as two independent repositories to different clients:
	refs/heads/master
	refs/tags/tag1
and:
	refs/heads/master
	refs/heads/branch2

The following three patches implement this. What do you guys think -- does the basic idea and implementation look sensible, or am I overlooking a way to solve this kind of problem with other means?

Thanks!
  receive-pack: Two small code cleanups
  Different views on a repository
  Different views on a repository: HEAD mapping
 Documentation/git-receive-pack.txt |    9 ++++-
 Documentation/git-upload-pack.txt  |   10 +++++-
 builtin-receive-pack.c             |   32 ++++++++++++++---
 refs.c                             |   65 ++++++++++++++++++++++++++++++++++--
 refs.h                             |    4 ++
 upload-pack.c                      |   20 ++++++++++-
 6 files changed, 127 insertions(+), 13 deletions(-)
Andreas Gruenbacher· Feb 24, 2010, 15:33 UTC · re: Andreas Gruenbacher · lore

[PATCH 1/3] receive-pack: Two small code cleanups

Rename show_ref()'s path parameter to refname.

In read_head_info(), lines may have trailing capability strings. Throw away such strings after evaluation; they are not needed in the command structs.

Signed-off-by: Andreas Gruenbacher <agruen@suse.de>
---
 builtin-receive-pack.c |   12 ++++++------
 1 files changed, 6 insertions(+), 6 deletions(-)
Show changes to builtin-receive-pack.c +6 −6
diff --git a/builtin-receive-pack.c b/builtin-receive-pack.c
index 0559fcc..77cbc2a 100644
--- a/builtin-receive-pack.c
+++ b/builtin-receive-pack.c
@@ -105,13 +105,13 @@ static int receive_pack_config(const char *var, const char *value, void *cb)
 	return git_default_config(var, value, cb);
 }
 
-static int show_ref(const char *path, const unsigned char *sha1, int flag, void *cb_data)
+static int show_ref(const char *refname, const unsigned char *sha1, int flag, void *cb_data)
 {
 	if (sent_capabilities)
-		packet_write(1, "%s %s\n", sha1_to_hex(sha1), path);
+		packet_write(1, "%s %s\n", sha1_to_hex(sha1), refname);
 	else
 		packet_write(1, "%s %s%c%s%s\n",
-			     sha1_to_hex(sha1), path, 0,
+			     sha1_to_hex(sha1), refname, 0,
 			     " report-status delete-refs side-band-64k",
 			     prefer_ofs_delta ? " ofs-delta" : "");
 	sent_capabilities = 1;
@@ -524,7 +524,7 @@ static void read_head_info(void)
 		static char line[1000];
 		unsigned char old_sha1[20], new_sha1[20];
 		struct command *cmd;
-		char *refname;
+		const char *refname;
 		int len, reflen;
 
 		len = packet_read_line(0, line, sizeof(line));
@@ -548,10 +548,10 @@ static void read_head_info(void)
 			if (strstr(refname + reflen + 1, "side-band-64k"))
 				use_sideband = LARGE_PACKET_MAX;
 		}
-		cmd = xmalloc(sizeof(struct command) + len - 80);
+		cmd = xmalloc(sizeof(struct command) + reflen + 1);
 		hashcpy(cmd->old_sha1, old_sha1);
 		hashcpy(cmd->new_sha1, new_sha1);
-		memcpy(cmd->ref_name, line + 82, len - 81);
+		memcpy(cmd->ref_name, refname, reflen + 1);
 		cmd->error_string = NULL;
 		cmd->next = NULL;
 		*p = cmd;
-- 
1.6.6.243.gff6d2
Andreas Gruenbacher· Feb 24, 2010, 15:57 UTC · re: Andreas Gruenbacher · lore

[PATCH 2/3] Different views on a repository

Add --view options in upload-pack and receive-pack so that a repository on the server side can be made to look like several independent repositories on the client side.

This is implemented by transforming ref names: for example, with --view=one/, refs/heads/one/master on the server will look like refs/heads/master to the client, refs/tags/one/v1 will look like refs/tags/v1, etc.

This allows to transparently share repositories on the server which have a lot of objects in common without complicating things for the client, and without breaking garbage collection.

Signed-off-by: Andreas Gruenbacher <agruen@suse.de>
---
 Documentation/git-receive-pack.txt |    8 +++++-
 Documentation/git-upload-pack.txt  |    9 ++++++-
 builtin-receive-pack.c             |   20 ++++++++++++++++
 refs.c                             |   44 ++++++++++++++++++++++++++++++++++++
 refs.h                             |    3 ++
 upload-pack.c                      |   11 +++++++++
 6 files changed, 93 insertions(+), 2 deletions(-)
Show changes to 6 files +93 −2

Documentation/git-receive-pack.txt, Documentation/git-upload-pack.txt, builtin-receive-pack.c, refs.c, refs.h, upload-pack.c

diff --git a/Documentation/git-receive-pack.txt b/Documentation/git-receive-pack.txt
index 2790eeb..09d7d0c 100644
--- a/Documentation/git-receive-pack.txt
+++ b/Documentation/git-receive-pack.txt
@@ -8,7 +8,7 @@ git-receive-pack - Receive what is pushed into the repository
 
 SYNOPSIS
 --------
-'git-receive-pack' <directory>
+'git-receive-pack' [--view=<prefix>] <directory>
 
 DESCRIPTION
 -----------
@@ -34,6 +34,12 @@ are not fast-forwards.
 
 OPTIONS
 -------
+--view=<prefix>::
+	Prepend <prefix> to all ref names.  For example, --view=one/ will
+	turn refs/tags/v1 into refs/tags/one/v1 on the receiving end.  Together
+	with the --view option of linkgit:git-upload-pack[1], this allows to
+	make one respository look like multiple independent repositories.
+
 <directory>::
 	The repository to sync into.
 
diff --git a/Documentation/git-upload-pack.txt b/Documentation/git-upload-pack.txt
index 71ca4ef..0eee0ba 100644
--- a/Documentation/git-upload-pack.txt
+++ b/Documentation/git-upload-pack.txt
@@ -8,7 +8,7 @@ git-upload-pack - Send objects packed back to git-fetch-pack
 
 SYNOPSIS
 --------
-'git-upload-pack' [--strict] [--timeout=<n>] <directory>
+'git-upload-pack' [--strict] [--timeout=<n>] [--view=<prefix>] <directory>
 
 DESCRIPTION
 -----------
@@ -30,6 +30,13 @@ OPTIONS
 --timeout=<n>::
 	Interrupt transfer after <n> seconds of inactivity.
 
+--view=<prefix>::
+	Only upload refs which start with <prefix>, and hide <prefix> from the
+	remote side.  For example, --view=one/ will skip refs/heads/master
+	and turn refs/tags/one/v1 into refs/tags/v1.  Together with the --view
+	option of linkgit:git-receive-pack, this allows to make one respository
+	look like multiple independent repositories.
+
 <directory>::
 	The repository to sync from.
 
diff --git a/builtin-receive-pack.c b/builtin-receive-pack.c
index 77cbc2a..44d7055 100644
--- a/builtin-receive-pack.c
+++ b/builtin-receive-pack.c
@@ -32,6 +32,7 @@ static int use_sideband;
 static int prefer_ofs_delta = 1;
 static int auto_update_server_info;
 static int auto_gc = 1;
+static const char *view;
 static const char *head_name;
 static int sent_capabilities;
 
@@ -107,6 +108,12 @@ static int receive_pack_config(const char *var, const char *value, void *cb)
 
 static int show_ref(const char *refname, const unsigned char *sha1, int flag, void *cb_data)
 {
+	if (view) {
+		refname = ref_to_view(refname, view);
+		if (!refname)
+			return 0;
+	}
+
 	if (sent_capabilities)
 		packet_write(1, "%s %s\n", sha1_to_hex(sha1), refname);
 	else
@@ -548,6 +555,15 @@ static void read_head_info(void)
 			if (strstr(refname + reflen + 1, "side-band-64k"))
 				use_sideband = LARGE_PACKET_MAX;
 		}
+		if (view) {
+			const char *r;
+
+			r = view_to_ref(refname, view);
+			if (r) {
+				refname = r;
+				reflen = strlen(refname);
+			}
+		}
 		cmd = xmalloc(sizeof(struct command) + reflen + 1);
 		hashcpy(cmd->old_sha1, old_sha1);
 		hashcpy(cmd->new_sha1, new_sha1);
@@ -736,6 +752,10 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 				stateless_rpc = 1;
 				continue;
 			}
+			if (!prefixcmp(arg, "--view=")) {
+				view = arg + 7;
+				continue;
+			}
 
 			usage(receive_pack_usage);
 		}
diff --git a/refs.c b/refs.c
index f3fcbe0..b1f3951 100644
--- a/refs.c
+++ b/refs.c
@@ -1829,3 +1829,47 @@ char *shorten_unambiguous_ref(const char *ref, int strict)
 	free(short_name);
 	return xstrdup(ref);
 }
+
+const char *ref_to_view(const char *refname, const char *view)
+{
+	static char *buffer;
+	int prefix_len, view_len, suffix_len;
+	const char *r, *suffix;
+
+	if (prefixcmp(refname, "refs/"))
+		return NULL;
+	r = strchr(refname + 5, '/');
+	if (!r)
+		return NULL;
+	r++;
+	view_len = strlen(view);
+	if (strncmp(r, view, view_len))
+		return NULL;
+	suffix = r + view_len;
+	prefix_len = r - refname;
+	suffix_len = strlen(suffix);
+	buffer = xrealloc(buffer, prefix_len + suffix_len + 1);
+	sprintf(buffer, "%.*s%s", prefix_len, refname, suffix);
+	return buffer;
+}
+
+const char *view_to_ref(const char *refname, const char *view)
+{
+	static char *buffer;
+	int prefix_len, view_len, suffix_len;
+	const char *r, *suffix;
+
+	view_len = strlen(view);
+	if (prefixcmp(refname, "refs/"))
+		return NULL;
+	r = strchr(refname + 5, '/');
+	if (!r)
+		return NULL;
+	r++;
+	prefix_len = r - refname;
+	suffix = r + view_len;
+	suffix_len = strlen(suffix);
+	buffer = xrealloc(buffer, prefix_len + view_len + suffix_len + 1);
+	sprintf(buffer, "%.*s%s%s", prefix_len, refname, view, suffix);
+	return buffer;
+}
diff --git a/refs.h b/refs.h
index f7648b9..390e812 100644
--- a/refs.h
+++ b/refs.h
@@ -98,4 +98,7 @@ int update_ref(const char *action, const char *refname,
 		const unsigned char *sha1, const unsigned char *oldval,
 		int flags, enum action_on_err onerr);
 
+extern const char *ref_to_view(const char *refname, const char *view);
+extern const char *view_to_ref(const char *refname, const char *view);
+
 #endif /* REFS_H */
diff --git a/upload-pack.c b/upload-pack.c
index dc464d7..bc72471 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -34,6 +34,7 @@ static struct object_array have_obj;
 static struct object_array want_obj;
 static struct object_array extra_edge_obj;
 static unsigned int timeout;
+static const char *view;
 /* 0 for no sideband,
  * otherwise maximum packet size (up to 65520 bytes).
  */
@@ -629,6 +630,12 @@ static int send_ref(const char *refname, const unsigned char *sha1, int flag, vo
 	if (!o)
 		die("git upload-pack: cannot find object %s:", sha1_to_hex(sha1));
 
+	if (view) {
+		refname = ref_to_view(refname, view);
+		if (!refname)
+			return 0;
+	}
+
 	if (capabilities)
 		packet_write(1, "%s %s%c%s\n", sha1_to_hex(sha1), refname,
 			0, capabilities);
@@ -711,6 +718,10 @@ int main(int argc, char **argv)
 			daemon_mode = 1;
 			continue;
 		}
+		if (!prefixcmp(arg, "--view=")) {
+			view = arg + 7;
+			continue;
+		}
 		if (!strcmp(arg, "--")) {
 			i++;
 			break;
-- 
1.6.6.243.gff6d2
Andreas Gruenbacher· Feb 24, 2010, 16:14 UTC · re: Andreas Gruenbacher · lore

The HEAD ref is not located under .git/refs/heads/, so the trivial view mapping doesn't work. Fix this by making refs/heads/<view>HEAD appear as HEAD on the client when --view=<view> is used in upload-pack and receive-pack.

Signed-off-by: Andreas Gruenbacher <agruen@suse.de>
---
 Documentation/git-receive-pack.txt |    7 ++++---
 Documentation/git-upload-pack.txt  |    9 +++++----
 refs.c                             |   21 ++++++++++++++++++---
 refs.h                             |    1 +
 upload-pack.c                      |    9 +++++++--
 5 files changed, 35 insertions(+), 12 deletions(-)
Show changes to 5 files +35 −12

Documentation/git-receive-pack.txt, Documentation/git-upload-pack.txt, refs.c, refs.h, upload-pack.c

diff --git a/Documentation/git-receive-pack.txt b/Documentation/git-receive-pack.txt
index 09d7d0c..07e0159 100644
--- a/Documentation/git-receive-pack.txt
+++ b/Documentation/git-receive-pack.txt
@@ -36,9 +36,10 @@ OPTIONS
 -------
 --view=<prefix>::
 	Prepend <prefix> to all ref names.  For example, --view=one/ will
-	turn refs/tags/v1 into refs/tags/one/v1 on the receiving end.  Together
-	with the --view option of linkgit:git-upload-pack[1], this allows to
-	make one respository look like multiple independent repositories.
+	turn refs/tags/v1 into refs/tags/one/v1 and HEAD into refs/heads/one/HEAD
+	on the receiving end.  Together with the --view option of
+	linkgit:git-upload-pack[1], this allows to make one respository look like
+	multiple independent repositories.
 
 <directory>::
 	The repository to sync into.
diff --git a/Documentation/git-upload-pack.txt b/Documentation/git-upload-pack.txt
index 0eee0ba..1e8b76b 100644
--- a/Documentation/git-upload-pack.txt
+++ b/Documentation/git-upload-pack.txt
@@ -32,10 +32,11 @@ OPTIONS
 
 --view=<prefix>::
 	Only upload refs which start with <prefix>, and hide <prefix> from the
-	remote side.  For example, --view=one/ will skip refs/heads/master
-	and turn refs/tags/one/v1 into refs/tags/v1.  Together with the --view
-	option of linkgit:git-receive-pack, this allows to make one respository
-	look like multiple independent repositories.
+	remote side.  For example, --view=one/ will skip refs/heads/master,
+	turn refs/tags/one/v1 into refs/tags/v1, and refs/heads/one/HEAD into
+	HEAD.  Together with the --view option of linkgit:git-receive-pack,
+	this allows to make one respository look like multiple independent
+	repositories.
 
 <directory>::
 	The repository to sync from.
diff --git a/refs.c b/refs.c
index b1f3951..77a6267 100644
--- a/refs.c
+++ b/refs.c
@@ -650,16 +650,21 @@ end_each:
 	return retval;
 }
 
-int head_ref(each_ref_fn fn, void *cb_data)
+int one_ref(each_ref_fn fn, void *cb_data, const char *refname)
 {
 	unsigned char sha1[20];
 	int flag;
 
-	if (resolve_ref("HEAD", sha1, 1, &flag))
-		return fn("HEAD", sha1, flag, cb_data);
+	if (resolve_ref(refname, sha1, 1, &flag))
+		return fn(refname, sha1, flag, cb_data);
 	return 0;
 }
 
+int head_ref(each_ref_fn fn, void *cb_data)
+{
+	return one_ref(fn, cb_data, "HEAD");
+}
+
 int for_each_ref(each_ref_fn fn, void *cb_data)
 {
 	return do_for_each_ref("refs/", fn, 0, 0, cb_data);
@@ -1846,6 +1851,10 @@ const char *ref_to_view(const char *refname, const char *view)
 	if (strncmp(r, view, view_len))
 		return NULL;
 	suffix = r + view_len;
+	if (!strncmp(refname + 5, "heads/", 6) &&
+	    !strcmp(suffix, "HEAD"))
+		return "HEAD";
+
 	prefix_len = r - refname;
 	suffix_len = strlen(suffix);
 	buffer = xrealloc(buffer, prefix_len + suffix_len + 1);
@@ -1860,6 +1869,12 @@ const char *view_to_ref(const char *refname, const char *view)
 	const char *r, *suffix;
 
 	view_len = strlen(view);
+	if (!strcmp(refname, "HEAD")) {
+		buffer = xrealloc(buffer, view_len + 16);
+		sprintf(buffer, "refs/heads/%sHEAD", view);
+		return buffer;
+	}
+
 	if (prefixcmp(refname, "refs/"))
 		return NULL;
 	r = strchr(refname + 5, '/');
diff --git a/refs.h b/refs.h
index 390e812..addcc2d 100644
--- a/refs.h
+++ b/refs.h
@@ -18,6 +18,7 @@ struct ref_lock {
  * and returns the value
  */
 typedef int each_ref_fn(const char *refname, const unsigned char *sha1, int flags, void *cb_data);
+extern int one_ref(each_ref_fn, void *, const char *);
 extern int head_ref(each_ref_fn, void *);
 extern int for_each_ref(each_ref_fn, void *);
 extern int for_each_ref_in(const char *, each_ref_fn, void *);
diff --git a/upload-pack.c b/upload-pack.c
index bc72471..b3bf20f 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -668,13 +668,18 @@ static int mark_our_ref(const char *refname, const unsigned char *sha1, int flag
 
 static void upload_pack(void)
 {
+	const char *head = "HEAD";
+
+	if (view)
+		head = view_to_ref(head, view);
+
 	if (advertise_refs || !stateless_rpc) {
 		reset_timeout();
-		head_ref(send_ref, NULL);
+		one_ref(send_ref, NULL, head);
 		for_each_ref(send_ref, NULL);
 		packet_flush(1);
 	} else {
-		head_ref(mark_our_ref, NULL);
+		one_ref(mark_our_ref, NULL, head);
 		for_each_ref(mark_our_ref, NULL);
 	}
 	if (advertise_refs)
-- 
1.6.6.243.gff6d2
Shawn O. Pearce· Feb 24, 2010, 17:42 UTC · re: Andreas Gruenbacher · lore

Re: [PATCH 2/3] Different views on a repository

Andreas Gruenbacher <agruen@suse.de> wrote:
> Add --view options in upload-pack and receive-pack so that a repository
> on the server side can be made to look like several independent
> repositories on the client side.

Before saying this is good... I'd like to know how a repository owner is supposed to set these options on the user started invocations of other remote side program.

Right now, I don't see how this is too different from just doing the following on a client:

  git init
  git remote add origin URL
  git config remote.origin.fetch refs/heads/one/*:refs/remotes/origin/*

and therefore shouldn't just be handled on the *client* side of the connection, as part of the remote setup and push matching refs rules.

(Of course, the push matching ref logic is messy too... adding yet more into that pile might also be ugly.)

> +const char *view_to_ref(const char *refname, const char *view)
> +{
> +	static char *buffer;
...
> +	buffer = xrealloc(buffer, prefix_len + view_len + suffix_len + 1);
> +	sprintf(buffer, "%.*s%s%s", prefix_len, refname, view, suffix);
> +	return buffer;

I'd rather not use a static buffer like this. Why not alloc and let the caller free? Or have the caller pass in a strbuf you populate for them?

-- 
Shawn.
Michael J Gruber· Feb 25, 2010, 09:01 UTC · re: Andreas Gruenbacher · lore

Re: [PATCH 2/3] Different views on a repository

Andreas Gruenbacher venit, vidit, dixit 24.02.2010 16:57:
Show 12 quoted lines
> Add --view options in upload-pack and receive-pack so that a repository
> on the server side can be made to look like several independent
> repositories on the client side.
> 
> This is implemented by transforming ref names: for example, with
> --view=one/, refs/heads/one/master on the server will look like
> refs/heads/master to the client, refs/tags/one/v1 will look like
> refs/tags/v1, etc.
> 
> This allows to transparently share repositories on the server which
> have a lot of objects in common without complicating things for the
> client, and without breaking garbage collection.

Just from this description, I can't see why the same can't be done with appropriate refspecs. (A helper for doing that would be more welcome, of course.)

Maybe a few tests and documentation (i.e. examples, not just the option description) would clear this up?

Michael
Andreas Gruenbacher· Feb 25, 2010, 09:25 UTC · re: Michael J Gruber · lore

Re: [PATCH 2/3] Different views on a repository

On Thursday 25 February 2010 10:01:43 Michael J Gruber wrote:
Show 17 quoted lines
> Andreas Gruenbacher venit, vidit, dixit 24.02.2010 16:57:
> > Add --view options in upload-pack and receive-pack so that a repository
> > on the server side can be made to look like several independent
> > repositories on the client side.
> >
> > This is implemented by transforming ref names: for example, with
> > --view=one/, refs/heads/one/master on the server will look like
> > refs/heads/master to the client, refs/tags/one/v1 will look like
> > refs/tags/v1, etc.
> >
> > This allows to transparently share repositories on the server which
> > have a lot of objects in common without complicating things for the
> > client, and without breaking garbage collection.
> 
> Just from this description, I can't see why the same can't be done with
> appropriate refspecs. (A helper for doing that would be more welcome, of
> course.)

You mean on the client side? The problem then is that a simple "git clone" would not do the right thing anymore; you would still expose server-side implementation details to clients. Clients shouldn't have to bother with this added complexity. (They might not even have access to some of the views.) When you do the mapping server-side, you can split or merge repositories as needed without the clients even noticing.

> Maybe a few tests and documentation (i.e. examples, not just the option
> description) would clear this up?
Indeed, I should add some more background info.

Thanks, Andreas

Michael J Gruber· Feb 25, 2010, 12:30 UTC · re: Andreas Gruenbacher · lore

Re: [PATCH 2/3] Different views on a repository

Andreas Gruenbacher venit, vidit, dixit 25.02.2010 10:25:
Show 25 quoted lines
> On Thursday 25 February 2010 10:01:43 Michael J Gruber wrote:
>> Andreas Gruenbacher venit, vidit, dixit 24.02.2010 16:57:
>>> Add --view options in upload-pack and receive-pack so that a repository
>>> on the server side can be made to look like several independent
>>> repositories on the client side.
>>>
>>> This is implemented by transforming ref names: for example, with
>>> --view=one/, refs/heads/one/master on the server will look like
>>> refs/heads/master to the client, refs/tags/one/v1 will look like
>>> refs/tags/v1, etc.
>>>
>>> This allows to transparently share repositories on the server which
>>> have a lot of objects in common without complicating things for the
>>> client, and without breaking garbage collection.
>>
>> Just from this description, I can't see why the same can't be done with
>> appropriate refspecs. (A helper for doing that would be more welcome, of
>> course.)
> 
> You mean on the client side? The problem then is that a simple "git clone" 
> would not do the right thing anymore; you would still expose server-side 
> implementation details to clients. Clients shouldn't have to bother with this 
> added complexity. (They might not even have access to some of the views.) When 
> you do the mapping server-side, you can split or merge repositories as needed 
> without the clients even noticing.

But the client has to request a specific view, doesn't it? You have to tell all clients "don't just clone, use...", where the "..." don't seem to be part of the series yet. [I could see 0/3 on gmane only now, by the way.]

I just can't help the impression that this is a use case which does not need a new feature, at least not upload/receive-pack wise. It's more a matter of ensuring that all clients use a specific configuration (which you would have to with your patch as well, AFAICT), and this more general issue is creeping up again and again, with no agreeable solution so far.

Cheers, Michael

Andreas Gruenbacher· Feb 25, 2010, 14:35 UTC · re: Michael J Gruber · lore

Re: [PATCH 2/3] Different views on a repository

On Thursday 25 February 2010 13:30:24 Michael J Gruber wrote:
Show 28 quoted lines
> Andreas Gruenbacher venit, vidit, dixit 25.02.2010 10:25:
> > On Thursday 25 February 2010 10:01:43 Michael J Gruber wrote:
> >> Andreas Gruenbacher venit, vidit, dixit 24.02.2010 16:57:
> >>> Add --view options in upload-pack and receive-pack so that a repository
> >>> on the server side can be made to look like several independent
> >>> repositories on the client side.
> >>>
> >>> This is implemented by transforming ref names: for example, with
> >>> --view=one/, refs/heads/one/master on the server will look like
> >>> refs/heads/master to the client, refs/tags/one/v1 will look like
> >>> refs/tags/v1, etc.
> >>>
> >>> This allows to transparently share repositories on the server which
> >>> have a lot of objects in common without complicating things for the
> >>> client, and without breaking garbage collection.
> >>
> >> Just from this description, I can't see why the same can't be done with
> >> appropriate refspecs. (A helper for doing that would be more welcome, of
> >> course.)
> >
> > You mean on the client side? The problem then is that a simple "git
> > clone" would not do the right thing anymore; you would still expose
> > server-side implementation details to clients. Clients shouldn't have to
> > bother with this added complexity. (They might not even have access to
> > some of the views.) When you do the mapping server-side, you can split or
> > merge repositories as needed without the clients even noticing.
> 
> But the client has to request a specific view, doesn't it?

No, it's a server side thing. The git commands affected are upload-pack and receice-pack, and those run on the remote end of a fetch. For example, the client would ask the server for repository /foo/one or /foo/two, and the server would map that to different views of /bar/shared: when the client asks the server to run "git-upload-pack /foo/one", the server runs "git-upload-pack --view=one/ /bar/shared" instead.

This is relatively easy to set up over ssh using a simple script; for direct git access, a small wrapper daemon would be needed. I'm not sure how this could be hacked into http access, but it doesn't seem all that hard, either.

> I just can't help the impression that this is a use case which does not
> need a new feature, at least not upload/receive-pack wise.
Still, even with the additional explanation above?
> It's more a matter of ensuring that all clients use a specific configuration
> (which you would have to with your patch as well, AFAICT), and this more
> general issue is creeping up again and again, with no agreeable solution
> so far.

Well that's another problem which we indeed also have for enabling things like local consistency checks and merge drivers. I don't have a good answer here :)

Thanks, Andreas

Junio C Hamano· Feb 25, 2010, 17:28 UTC · re: Andreas Gruenbacher · lore

Re: [PATCH 2/3] Different views on a repository

Andreas Gruenbacher <agruen@suse.de> writes:
> No, it's a server side thing.

If it were a server side thing, then I would expect no change to send/receive pack. Instead your clients will access distinct URL as if they are different repositories.

    git clone git://example.com/pub/scm/git/A
    git push example.com:/pub/scm/git/B master
    git pull http://example.com/pub/scm/git/C

They should not have to care that the server is cheating to save disk space, and they should be able to access your server with Git v1.6.0.

Instead, the server side would:
 - have separate repositories, A, B and C, as normal repositories;
 - these repositories share their object stores by having their
   .git/objects pointing at a shared location via a symlink;
 - on the server side, gc/prune/fsck will have to be updated so that when
   the object store of a repository (say A) is shared with something else,
   they will consider refs in other repositories (B and C) also as the
   root of traversal.

So if this were a server side solution, I would expect the series would add:

 - a way to set up a shared object store;
 - a way to maintain a list of backlinks to repositories that share an
   object store;
 - a way to create a new repository that shares the object store
   (e.g. create a symlink to the shared store instead of having its own
   .git/objects/, and add itself to the list of backlinks for the shared
   object store);
 - a way to retire an existing such repository (rm -rf and remove itself
   from the list of backlinks);
 - update gc/prune/fsck to honor such a list of backlinks.

This would help a "forks" setup commonly seen at places like repo.or.cz and github.com among others.

One thing that is missing from the above handwaving outline that your "different views" offers is a "consolidated view", a pseudo-repository that allows you to see refs from individual real (from the client's and project participant's point of view) repositories as if they are in individual subhierarchies of the ref namespace.

I however suspect that you didn't want such a view in the first place if there weren't issues around reachability. In other words, I suspect that you invented it merely as one possible solution to the reachability issue, and it was not your goal to have such a consolidated view by itself.

Andreas Gruenbacher· Feb 26, 2010, 00:45 UTC · re: Junio C Hamano · lore

Re: [PATCH 2/3] Different views on a repository

On Thursday 25 February 2010 18:28:46 Junio C Hamano wrote:
Show 27 quoted lines
> Andreas Gruenbacher <agruen@suse.de> writes:
> > No, it's a server side thing.
> 
> If it were a server side thing, then I would expect no change to
> send/receive pack.
>
> Instead your clients will access distinct URL as if they are different
> repositories.
> 
>     git clone git://example.com/pub/scm/git/A
>     git push example.com:/pub/scm/git/B master
>     git pull http://example.com/pub/scm/git/C
>
> They should not have to care that the server is cheating to save disk
> space, and they should be able to access your server with Git v1.6.0.
> 
> Instead, the server side would:
> 
>  - have separate repositories, A, B and C, as normal repositories;
> 
>  - these repositories share their object stores by having their
>    .git/objects pointing at a shared location via a symlink;
> 
>  - on the server side, gc/prune/fsck will have to be updated so that when
>    the object store of a repository (say A) is shared with something else,
>    they will consider refs in other repositories (B and C) also as the
>    root of traversal.

I was proposing to change receive-pack and upload-pack, both of which are running on the server; there was no mention of send-pack. What I've proposed is not a complete solution, but it should suffice to show the idea.

Your alternative proposal would also solve my problem, in a different way. With either approach, what looks like separate repositories A, B, and C to clients looks like one repository to gc/prune/fsck.

Show 20 quoted lines
> So if this were a server side solution, I would expect the series would
> add:
> 
>  - a way to set up a shared object store;
> 
>  - a way to maintain a list of backlinks to repositories that share an
>    object store;
> 
>  - a way to create a new repository that shares the object store
>    (e.g. create a symlink to the shared store instead of having its own
>    .git/objects/, and add itself to the list of backlinks for the shared
>    object store);
> 
>  - a way to retire an existing such repository (rm -rf and remove itself
>    from the list of backlinks);
> 
>  - update gc/prune/fsck to honor such a list of backlinks.
> 
> This would help a "forks" setup commonly seen at places like repo.or.cz
> and github.com among others.

Yes. I'm don't know how big a problem this is for those kinds of hosters; in our case, it is a big problem.

Show 5 quoted lines
> One thing that is missing from the above handwaving outline that your
> "different views" offers is a "consolidated view", a pseudo-repository
> that allows you to see refs from individual real (from the client's and
> project participant's point of view) repositories as if they are in
> individual subhierarchies of the ref namespace.

I have been talking about a repository and different subsets or views of that repository; you call the former a consolidated view and the latter a repository. Those are really just two sides of the same coin.

> I however suspect that you didn't want such a view in the first place if
> there weren't issues around reachability.  In other words, I suspect that
> you invented it merely as one possible solution to the reachability issue,
> and it was not your goal to have such a consolidated view by itself.

I'm actually not sure. The "consolidated view" as you put it may be useful all by itself; it would be a proper, self sufficient git repository -- a really nice property. it may be too painful to maintain this view though.

When sharing objects across repositories, the worst-case scenario is that something goes wrong with the backlinks. You will eventually lose objects, but it may take a while until it happens and until you notice, with a lot of damage. That's nasty.

A combination of the two approaches would be to "link forward" instead of "linking back", so that the consolidated view would maintain itself, with a server repo setup like this:

	/repos/ABC:
		objects
		refs/tags/A/
		refs/tags/B/
		refs/heads/A/
		refs/heads/B/
	/repos/A:
		refs/tags -> /repos/ABC/refs/tags/A/
		refs/heads -> /repos/ABC/refs/heads/A/
		objects -> /repos/ABC/objects/
	/repos/B:
		refs/tags -> /repos/ABC/refs/tags/B/
		refs/heads -> /repos/ABC/refs/heads/B/
		objects -> /repos/ABC/objects/

This could be made safe by not doing garbage collection if objects is a symlink instead of a directory. (The ABC repo could be garbage collected as usual.) Am I overlooking anything why this can't work?

Thanks, Andreas

Andreas Gruenbacher· Feb 26, 2010, 21:35 UTC · re: Andreas Gruenbacher · lore

Re: [PATCH 2/3] Different views on a repository

On Friday 26 February 2010 01:45:33 Andreas Gruenbacher wrote:
Show 24 quoted lines
> A combination of the two approaches would be to "link forward" instead of
> "linking back", so that the consolidated view would maintain itself, with a
> server repo setup like this:
> 
> 	/repos/ABC:
> 		objects
> 		refs/tags/A/
> 		refs/tags/B/
> 		refs/heads/A/
> 		refs/heads/B/
> 
> 	/repos/A:
> 		refs/tags -> /repos/ABC/refs/tags/A/
> 		refs/heads -> /repos/ABC/refs/heads/A/
> 		objects -> /repos/ABC/objects/
> 
> 	/repos/B:
> 		refs/tags -> /repos/ABC/refs/tags/B/
> 		refs/heads -> /repos/ABC/refs/heads/B/
> 		objects -> /repos/ABC/objects/
> 
> This could be made safe by not doing garbage collection if objects is a
> symlink instead of a directory.  (The ABC repo could be garbage collected
>  as usual.)  Am I overlooking anything why this can't work?
Self reply: reference packing breaks this kind of setup.  Crap.
Andreas
Shawn O. Pearce· Feb 24, 2010, 17:29 UTC · re: Andreas Gruenbacher · lore

Re: [PATCH 1/3] receive-pack: Two small code cleanups

Andreas Gruenbacher <agruen@suse.de> wrote:
Show 7 quoted lines
> Rename show_ref()'s path parameter to refname.
> 
> In read_head_info(), lines may have trailing capability strings.  Throw
> away such strings after evaluation; they are not needed in the command
> structs.
> 
> Signed-off-by: Andreas Gruenbacher <agruen@suse.de>
Acked-by: Shawn O. Pearce <spearce@spearce.org>
-- 
Shawn.
James Pickens· Feb 25, 2010, 20:13 UTC · re: Andreas Gruenbacher · lore

Re: [RFC][PATCH 0/3] Different views on a repository

On Wed, Feb 24, 2010, Andreas Gruenbacher <agruen@suse.de> wrote:
Show 5 quoted lines
> we have a use case with groups of repositories which share lots of
> objects, but which are logically independent.  There is no strict
> hierarchy between the repositories, the development modl is arbitrary.
> The alternates mechanism for sharig objects between repositories won't
> work.
Can you elaborate on why alternates won't work?
> The best idea I came up with so far to solve this

Solve what? You didn't mention any specific problem. Are you just trying to save disk space by not storing multiple copies of the same objects?

James
Adam Brewster· Feb 26, 2010, 04:30 UTC · re: Andreas Gruenbacher · lore

Re: [RFC][PATCH 0/3] Different views on a repository

On Wed, Feb 24, 2010 at 11:41 AM, Andreas Gruenbacher <agruen@suse.de>wrote:
Show 8 quoted lines
> Hello,
>
> we have a use case with groups of repositories which share lots of
> objects, but which are logically independent.  There is no strict
> hierarchy between the repositories, the development modl is arbitrary.
> The alternates mechanism for sharig objects between repositories won't
> work.
>

I don't know what you're trying to accomplish, but for what it's worth, I do something similar with a couple of shell scripts.

My goal was to make bundles (basically thin packs) smaller by taking advantage of files I knew were available on the far side of the air-gap even if they weren't in a the repository I was bundling that particular day.

The idea was to push from all of my repositories into a super repository with a fancy (and auto-generated) refspec. The actual code is impenetrable, but reconstructing it in everybody's favorite IDE, gmail, I came up with

#!/bin/bash PROJECTS=$HOME/projects SUPER=$HOME/projects/.git-super-repo

[[ -d "$SUPER" ]] || \
 (mkdir "$SUPER"; git --git-dir="$SUPER" init)
for i in $PROJECTS/*/.git; do
 name=$(basename "$(dirname "$0")")
 echo "$SUPER/objects" > $i/.git/objects/info/alternates
 git --git-dir="$i" push -f "$SUPER" "*:refs/$name/*"
done
git --git-dir="$SUPER" gc --aggressive
for i in $PROJECTS/*/.git; do
 git --git-dir="$i" repack -Ad #unnecessary?
 git --git-dir="$i" gc --aggressive
done

Clearly I can lose data if I try to rebase $SUPER or something, but I think it's pretty safe for normal use.

In your case, the "projects" are called "views".
Adam
Andreas Gruenbacher· Feb 26, 2010, 12:01 UTC · lore

Re: [RFC][PATCH 0/3] Different views on a repository

On Friday 26 February 2010 05:12:45 Adam Brewster wrote:
Show 24 quoted lines
> The idea was to push from all of my repositories into a super repository
> with a fancy (and auto-generated) refspec.  The actual code is
> impenetrable, but reconstructing it in everybody's favorite IDE, gmail, I
> came up with
> 
> #!/bin/bash
> PROJECTS=$HOME/projects
> SUPER=$HOME/projects/.git-super-repo
> 
> [[ -d "$SUPER" ]] || \
>   (mkdir "$SUPER"; git --git-dir="$SUPER" init)
> 
> for i in $PROJECTS/*/.git; do
>   name=$(basename "$(dirname "$0")")
>   echo "$SUPER/objects" > $i/.git/objects/info/alternates
>   git --git-dir="$i" push -f "$SUPER" "*:refs/$name/*"
> done
> 
> git --git-dir="$SUPER" gc --aggressive
> 
> for i in $PROJECTS/*/.git; do
>   git --git-dir="$i" repack -Ad #unnecessary?
>   git --git-dir="$i" gc --aggressive
> done

I see, when multiple projects share the same objects, you push them into those projects independently first, and the script will later move them to $SUPER.

> Clearly I can lose data if I try to rebase $SUPER or something, but I think
> it's pretty safe for normal use.

It looks safe unless somebody messes with $SUPER. A lot of repacking will still occur as part of moving stuff to $SUPER, though.

I was trying to set things up so that this extra work won't be necessary in the first place.

Thanks!
Andreas

← back to recent threads