{"thread":{"id":"27449","subject":"[PATCH v3 2/3] Support virtual repositories in smart http-backend, specified by environment","startedAt":"2011-05-25T00:46:30Z","lastAt":"2011-05-26T18:28:33Z","messageCount":11,"participants":["Jamey Sharp","Junio C Hamano","Jeff King","Shawn Pearce","Josh Triplett"],"isPatch":true,"patchVersion":3,"patchTotal":3},"messages":[{"id":"168617","messageId":"1306284392-12034-1-git-send-email-jamey@minilop.net","threadId":"27449","inReplyTo":null,"subject":"[PATCH v3 1/3] Support multiple virtual repositories with a single object store and refs","fromName":"Jamey Sharp","fromEmail":"jamey@minilop.net","sentAt":"2011-05-25T00:46:30Z","receivedAt":"2011-05-25T00:46:30Z","isPatch":true,"sender":{"key":"jamey@minilop.net","avatar":"https://gravatar.com/avatar/979ed4f9190c19c13a84945afe9a6cf141e8884ef6ddaae342db7a74ae28828e?d=mp&s=160"},"body":"From: Josh Triplett <josh@joshtriplett.org>\n\nGiven many repositories with copies of the same objects (such as branches of\nthe same source), sharing a common object store will avoid duplication.\nAlternates provide a single baseline, but don't handle ongoing activity in the\nvarious repositories.  Furthermore, operations such as git-gc need to know\nabout all of the refs.\n\nGit supports storing multiple virtual repositories within the object store and\nreferences of a single underlying repository.  The underlying repository\nstores the objects for all of the virtual repositories, and includes all the\nrefs and heads of the virtual repositories using prefixed names.\n\ngit-upload-pack and git-receive-pack rewrite the names of refs and heads as\nspecified by the --ref-prefix and --head options.  For instance,\n--ref-prefix=virtual/reponame/ will use refs/virtual/reponame/heads/* and\nrefs/virtual/reponame/tags/*.  git-upload-pack and git-receive-pack will\nignore any references that do not match the specified prefix.\n\nThese options implement the underlying mechanism for virtual\nrepositories; the higher-level protocol handler (such as http-backend or\na custom server) can pass these options when invoking upload-pack or\nreceive-pack, providing values based on components of the repository\npath.  For a simple local test, git-remote-ext works:\n\ngit clone ext::'git %s --ref-prefix=virtual/reponame/ --head=virtual-HEAD/reponame storage.git'\n\nCommit by Josh Triplett and Jamey Sharp.\n\nSigned-off-by: Josh Triplett <josh@joshtriplett.org>\nSigned-off-by: Jamey Sharp <jamey@minilop.net>\n---\nv2: remove accidentally-included debug message, and add patch 2/2 for\n    git-http-backend.\nv3: add patch 3/3 with documentation for virtual repositories, and\n    incorporated feedback from Jeff King and Junio C Hamano.\n\n builtin/receive-pack.c |   36 +++++++++++++++++++++++++++---------\n upload-pack.c          |   34 +++++++++++++++++++++++++++-------\n 2 files changed, 54 insertions(+), 16 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex e1ba4dc..76dacd0 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -34,6 +34,8 @@ static int prefer_ofs_delta = 1;\n static int auto_update_server_info;\n static int auto_gc = 1;\n static const char *head_name;\n+static const char *head_path = \"HEAD\";\n+static const char *ref_prefix = \"refs/\";\n static int sent_capabilities;\n \n static enum deny_action parse_deny_action(const char *var, const char *value)\n@@ -108,11 +110,12 @@ static int receive_pack_config(const char *var, const char *value, void *cb)\n \n static int show_ref(const char *path, const unsigned char *sha1, int flag, void *cb_data)\n {\n+\tconst char *refnameprefix = cb_data;\n \tif (sent_capabilities)\n-\t\tpacket_write(1, \"%s %s\\n\", sha1_to_hex(sha1), path);\n+\t\tpacket_write(1, \"%s %s%s\\n\", sha1_to_hex(sha1), refnameprefix, path);\n \telse\n-\t\tpacket_write(1, \"%s %s%c%s%s\\n\",\n-\t\t\t     sha1_to_hex(sha1), path, 0,\n+\t\tpacket_write(1, \"%s %s%s%c%s%s\\n\",\n+\t\t\t     sha1_to_hex(sha1), refnameprefix, path, 0,\n \t\t\t     \" report-status delete-refs side-band-64k\",\n \t\t\t     prefer_ofs_delta ? \" ofs-delta\" : \"\");\n \tsent_capabilities = 1;\n@@ -121,9 +124,9 @@ static int show_ref(const char *path, const unsigned char *sha1, int flag, void\n \n static void write_head_info(void)\n {\n-\tfor_each_ref(show_ref, NULL);\n+\tfor_each_ref_in(ref_prefix, show_ref, \"refs/\");\n \tif (!sent_capabilities)\n-\t\tshow_ref(\"capabilities^{}\", null_sha1, 0, NULL);\n+\t\tshow_ref(\"capabilities^{}\", null_sha1, 0, \"\");\n \n }\n \n@@ -332,6 +335,8 @@ static void refuse_unconfigured_deny_delete_current(void)\n static const char *update(struct command *cmd)\n {\n \tconst char *name = cmd->ref_name;\n+\tstruct strbuf prefixed_name_buf = STRBUF_INIT;\n+\tconst char *prefixed_name;\n \tunsigned char *old_sha1 = cmd->old_sha1;\n \tunsigned char *new_sha1 = cmd->new_sha1;\n \tstruct ref_lock *lock;\n@@ -342,7 +347,10 @@ static const char *update(struct command *cmd)\n \t\treturn \"funny refname\";\n \t}\n \n-\tif (is_ref_checked_out(name)) {\n+\tstrbuf_addf(&prefixed_name_buf, \"%s%s\", ref_prefix, name + 5);\n+\tprefixed_name = strbuf_detach(&prefixed_name_buf, NULL);\n+\n+\tif (is_ref_checked_out(prefixed_name)) {\n \t\tswitch (deny_current_branch) {\n \t\tcase DENY_IGNORE:\n \t\t\tbreak;\n@@ -370,7 +378,7 @@ static const char *update(struct command *cmd)\n \t\t\treturn \"deletion prohibited\";\n \t\t}\n \n-\t\tif (!strcmp(name, head_name)) {\n+\t\tif (!strcmp(prefixed_name, head_name)) {\n \t\t\tswitch (deny_delete_current) {\n \t\t\tcase DENY_IGNORE:\n \t\t\t\tbreak;\n@@ -426,14 +434,14 @@ static const char *update(struct command *cmd)\n \t\t\trp_warning(\"Allowing deletion of corrupt ref.\");\n \t\t\told_sha1 = NULL;\n \t\t}\n-\t\tif (delete_ref(name, old_sha1, 0)) {\n+\t\tif (delete_ref(prefixed_name, old_sha1, 0)) {\n \t\t\trp_error(\"failed to delete %s\", name);\n \t\t\treturn \"failed to delete\";\n \t\t}\n \t\treturn NULL; /* good */\n \t}\n \telse {\n-\t\tlock = lock_any_ref_for_update(name, old_sha1, 0);\n+\t\tlock = lock_any_ref_for_update(prefixed_name, old_sha1, 0);\n \t\tif (!lock) {\n \t\t\trp_error(\"failed to lock %s\", name);\n \t\t\treturn \"failed to lock\";\n@@ -760,6 +768,16 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\tadvertise_refs = 1;\n \t\t\t\tcontinue;\n \t\t\t}\n+\t\t\tif (!prefixcmp(arg, \"--head=\")) {\n+\t\t\t\thead_path = arg+7;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tif (!prefixcmp(arg, \"--ref-prefix=\")) {\n+\t\t\t\tstruct strbuf prefixbuf = STRBUF_INIT;\n+\t\t\t\tstrbuf_addf(&prefixbuf, \"refs/%s\", arg+13);\n+\t\t\t\tref_prefix = strbuf_detach(&prefixbuf, NULL);\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t\tif (!strcmp(arg, \"--stateless-rpc\")) {\n \t\t\t\tstateless_rpc = 1;\n \t\t\t\tcontinue;\ndiff --git a/upload-pack.c b/upload-pack.c\nindex ce5cbbe..a1e495f 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -34,6 +34,8 @@ static int shallow_nr;\n static struct object_array have_obj;\n static struct object_array want_obj;\n static struct object_array extra_edge_obj;\n+static const char *head_path = \"HEAD\";\n+static const char *ref_prefix = \"\";\n static unsigned int timeout;\n /* 0 for no sideband,\n  * otherwise maximum packet size (up to 65520 bytes).\n@@ -640,17 +642,18 @@ static int send_ref(const char *refname, const unsigned char *sha1, int flag, vo\n \tstatic const char *capabilities = \"multi_ack thin-pack side-band\"\n \t\t\" side-band-64k ofs-delta shallow no-progress\"\n \t\t\" include-tag multi_ack_detailed\";\n+\tconst char *refnameprefix = cb_data;\n \tstruct object *o = parse_object(sha1);\n \n \tif (!o)\n \t\tdie(\"git upload-pack: cannot find object %s:\", sha1_to_hex(sha1));\n \n \tif (capabilities)\n-\t\tpacket_write(1, \"%s %s%c%s%s\\n\", sha1_to_hex(sha1), refname,\n+\t\tpacket_write(1, \"%s %s%s%c%s%s\\n\", sha1_to_hex(sha1), refnameprefix, refname,\n \t\t\t     0, capabilities,\n \t\t\t     stateless_rpc ? \" no-done\" : \"\");\n \telse\n-\t\tpacket_write(1, \"%s %s\\n\", sha1_to_hex(sha1), refname);\n+\t\tpacket_write(1, \"%s %s%s\\n\", sha1_to_hex(sha1), refnameprefix, refname);\n \tcapabilities = NULL;\n \tif (!(o->flags & OUR_REF)) {\n \t\to->flags |= OUR_REF;\n@@ -659,7 +662,7 @@ static int send_ref(const char *refname, const unsigned char *sha1, int flag, vo\n \tif (o->type == OBJ_TAG) {\n \t\to = deref_tag(o, refname, 0);\n \t\tif (o)\n-\t\t\tpacket_write(1, \"%s %s^{}\\n\", sha1_to_hex(o->sha1), refname);\n+\t\t\tpacket_write(1, \"%s %s%s^{}\\n\", sha1_to_hex(o->sha1), refnameprefix, refname);\n \t}\n \treturn 0;\n }\n@@ -678,15 +681,24 @@ static int mark_our_ref(const char *refname, const unsigned char *sha1, int flag\n \n static void upload_pack(void)\n {\n+\tstruct strbuf prefix = STRBUF_INIT;\n+\tunsigned char sha1[20];\n+\tint flag;\n+\n+\tstrbuf_addf(&prefix, \"refs/%s\", ref_prefix);\n \tif (advertise_refs || !stateless_rpc) {\n \t\treset_timeout();\n-\t\thead_ref(send_ref, NULL);\n-\t\tfor_each_ref(send_ref, NULL);\n+\t\tif (resolve_ref(head_path, sha1, 1, &flag))\n+\t\t\tsend_ref(\"HEAD\", sha1, flag, \"\");\n+\t\tfor_each_ref_in(prefix.buf, send_ref, \"refs/\");\n \t\tpacket_flush(1);\n \t} else {\n-\t\thead_ref(mark_our_ref, NULL);\n-\t\tfor_each_ref(mark_our_ref, NULL);\n+\t\tif (resolve_ref(head_path, sha1, 1, &flag))\n+\t\t\tmark_our_ref(\"HEAD\", sha1, flag, NULL);\n+\t\tfor_each_ref_in(prefix.buf, mark_our_ref, NULL);\n \t}\n+\tstrbuf_release(&prefix);\n+\n \tif (advertise_refs)\n \t\treturn;\n \n@@ -716,6 +728,14 @@ int main(int argc, char **argv)\n \t\t\tadvertise_refs = 1;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!prefixcmp(arg, \"--head=\")) {\n+\t\t\thead_path = arg+7;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!prefixcmp(arg, \"--ref-prefix=\")) {\n+\t\t\tref_prefix = arg+13;\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!strcmp(arg, \"--stateless-rpc\")) {\n \t\t\tstateless_rpc = 1;\n \t\t\tcontinue;\n-- \n1.7.5.1\n"},{"id":"168616","messageId":"1306284392-12034-2-git-send-email-jamey@minilop.net","threadId":"27449","inReplyTo":"1306284392-12034-1-git-send-email-jamey@minilop.net","subject":"[PATCH v3 2/3] Support virtual repositories in smart http-backend, specified by environment","fromName":"Jamey Sharp","fromEmail":"jamey@minilop.net","sentAt":"2011-05-25T00:46:31Z","receivedAt":"2011-05-25T00:46:31Z","isPatch":true,"sender":{"key":"jamey@minilop.net","avatar":"https://gravatar.com/avatar/979ed4f9190c19c13a84945afe9a6cf141e8884ef6ddaae342db7a74ae28828e?d=mp&s=160"},"body":"From: Josh Triplett <josh@joshtriplett.org>\n\nTranslate the GIT_REF_PREFIX and GIT_HEAD environment variables to the\n--ref-prefix= and --head= options to upload-pack and receive-pack.\n\nAdd documentation, including a sample Apache configuration snippet.\n\nCommit by Josh Triplett and Jamey Sharp.\n\nSigned-off-by: Josh Triplett <josh@joshtriplett.org>\nSigned-off-by: Jamey Sharp <jamey@minilop.net>\n---\nv2: add patch 2/2 for git-http-backend.\nv3: add patch 3/3 with documentation for virtual repositories, and\n    incorporated feedback from Jeff King and Junio C Hamano.\n\n Documentation/git-http-backend.txt |   14 ++++++++++++\n http-backend.c                     |   39 +++++++++++++++++++++++++++++------\n 2 files changed, 46 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/git-http-backend.txt b/Documentation/git-http-backend.txt\nindex 277d9e1..4e0b243 100644\n--- a/Documentation/git-http-backend.txt\n+++ b/Documentation/git-http-backend.txt\n@@ -119,6 +119,14 @@ ScriptAliasMatch \\\n \n ScriptAlias /git/ /var/www/cgi-bin/gitweb.cgi/\n ----------------------------------------------------------------\n++\n+To serve multiple virtual repositories from a single storage\n+repository:\n++\n+----------------------------------------------------------------\n+SetEnvIf Request_URI \"^/git/([^/]*)\" GIT_REF_PREFIX=$1/ GIT_HEAD=$1-HEAD\n+ScriptAliasMatch ^/git/[^/]*(.*) /usr/libexec/git-core/git-http-backend/storage.git$1\n+----------------------------------------------------------------\n \n Accelerated static Apache 2.x::\n \tSimilar to the above, but Apache can be used to return static\n@@ -167,6 +175,12 @@ The GIT_HTTP_EXPORT_ALL environmental variable may be passed to\n 'git-http-backend' to bypass the check for the \"git-daemon-export-ok\"\n file in each repository before allowing export of that repository.\n \n+The GIT_REF_PREFIX and GIT_HEAD environment variables allow\n+http-backend to support multiple virtual repositories within a single\n+storage repository.  If set, 'git http-backend' will operate on\n+'refs/${GIT_REF_PREFIX}' rather than 'refs/', and '$GIT_HEAD' rather\n+than HEAD.\n+\n The backend process sets GIT_COMMITTER_NAME to '$REMOTE_USER' and\n GIT_COMMITTER_EMAIL to '$\\{REMOTE_USER}@http.$\\{REMOTE_ADDR\\}',\n ensuring that any reflogs created by 'git-receive-pack' contain some\ndiff --git a/http-backend.c b/http-backend.c\nindex 8501504..3d9e3b1 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -315,16 +315,23 @@ done:\n \tclose(out);\n }\n \n-static void run_service(const char **argv)\n+#define RUN_SERVICE_LAST_EXTRA_ARG 2\n+#define RUN_SERVICE_EXTRA_ARGS NULL, NULL, NULL\n+\n+static void run_service(const char *argv0, const char **argv)\n {\n \tconst char *encoding = getenv(\"HTTP_CONTENT_ENCODING\");\n \tconst char *user = getenv(\"REMOTE_USER\");\n \tconst char *host = getenv(\"REMOTE_ADDR\");\n+\tchar *ref_prefix = getenv(\"GIT_REF_PREFIX\");\n+\tchar *head = getenv(\"GIT_HEAD\");\n \tchar *env[3];\n \tstruct strbuf buf = STRBUF_INIT;\n \tint gzipped_request = 0;\n \tstruct child_process cld;\n \n+\targv += RUN_SERVICE_LAST_EXTRA_ARG;\n+\n \tif (encoding && !strcmp(encoding, \"gzip\"))\n \t\tgzipped_request = 1;\n \telse if (encoding && !strcmp(encoding, \"x-gzip\"))\n@@ -343,6 +350,24 @@ static void run_service(const char **argv)\n \tenv[1] = strbuf_detach(&buf, NULL);\n \tenv[2] = NULL;\n \n+\tif (ref_prefix && !*ref_prefix)\n+\t\tref_prefix = NULL;\n+\tif (ref_prefix) {\n+\t\tstrbuf_addf(&buf, \"--ref-prefix=%s\", ref_prefix);\n+\t\tref_prefix = strbuf_detach(&buf, NULL);\n+\t\t*argv-- = ref_prefix;\n+\t}\n+\n+\tif (head && !*head)\n+\t\thead = NULL;\n+\tif (head) {\n+\t\tstrbuf_addf(&buf, \"--head=%s\", head);\n+\t\thead = strbuf_detach(&buf, NULL);\n+\t\t*argv-- = head;\n+\t}\n+\n+\t*argv = argv0;\n+\n \tmemset(&cld, 0, sizeof(cld));\n \tcld.argv = argv;\n \tcld.env = (const char *const *)env;\n@@ -362,6 +387,8 @@ static void run_service(const char **argv)\n \t\texit(1);\n \tfree(env[0]);\n \tfree(env[1]);\n+\tfree(ref_prefix);\n+\tfree(head);\n \tstrbuf_release(&buf);\n }\n \n@@ -391,7 +418,7 @@ static void get_info_refs(char *arg)\n \thdr_nocache();\n \n \tif (service_name) {\n-\t\tconst char *argv[] = {NULL /* service name */,\n+\t\tconst char *argv[] = {RUN_SERVICE_EXTRA_ARGS,\n \t\t\t\"--stateless-rpc\", \"--advertise-refs\",\n \t\t\t\".\", NULL};\n \t\tstruct rpc_service *svc = select_service(service_name);\n@@ -404,8 +431,7 @@ static void get_info_refs(char *arg)\n \t\tpacket_write(1, \"# service=git-%s\\n\", svc->name);\n \t\tpacket_flush(1);\n \n-\t\targv[0] = svc->name;\n-\t\trun_service(argv);\n+\t\trun_service(svc->name, argv);\n \n \t} else {\n \t\tselect_getanyfile();\n@@ -462,7 +488,7 @@ static void check_content_type(const char *accepted_type)\n \n static void service_rpc(char *service_name)\n {\n-\tconst char *argv[] = {NULL, \"--stateless-rpc\", \".\", NULL};\n+\tconst char *argv[] = {RUN_SERVICE_EXTRA_ARGS, \"--stateless-rpc\", \".\", NULL};\n \tstruct rpc_service *svc = select_service(service_name);\n \tstruct strbuf buf = STRBUF_INIT;\n \n@@ -478,8 +504,7 @@ static void service_rpc(char *service_name)\n \n \tend_headers();\n \n-\targv[0] = svc->name;\n-\trun_service(argv);\n+\trun_service(svc->name, argv);\n \tstrbuf_release(&buf);\n }\n \n-- \n1.7.5.1\n"},{"id":"168618","messageId":"1306284392-12034-3-git-send-email-jamey@minilop.net","threadId":"27449","inReplyTo":"1306284392-12034-1-git-send-email-jamey@minilop.net","subject":"[PATCH v3 3/3] Add documentation for virtual repositories","fromName":"Jamey Sharp","fromEmail":"jamey@minilop.net","sentAt":"2011-05-25T00:46:32Z","receivedAt":"2011-05-25T00:46:32Z","isPatch":true,"sender":{"key":"jamey@minilop.net","avatar":"https://gravatar.com/avatar/979ed4f9190c19c13a84945afe9a6cf141e8884ef6ddaae342db7a74ae28828e?d=mp&s=160"},"body":"From: Josh Triplett <josh@joshtriplett.org>\n\nAdd a new gitvirtual(5) document, and cross-reference it from\ngit-http-backend(1).\n\nThanks to Jeff King for providing the material for the SECURITY section\nand inspiring the CONVENTIONS section.\n\nCommit by Josh Triplett and Jamey Sharp.\n\nSigned-off-by: Josh Triplett <josh@joshtriplett.org>\nSigned-off-by: Jamey Sharp <jamey@minilop.net>\n---\nv3: add patch 3/3 with documentation for virtual repositories, and\n    incorporated feedback from Jeff King and Junio C Hamano.\n\n Documentation/Makefile                 |    2 +-\n Documentation/git-http-backend.txt     |    4 +-\n Documentation/gitvirtual.txt           |   76 ++++++++++++++++++++++++++++++++\n contrib/completion/git-completion.bash |    2 +-\n 4 files changed, 80 insertions(+), 4 deletions(-)\n create mode 100644 Documentation/gitvirtual.txt\n\ndiff --git a/Documentation/Makefile b/Documentation/Makefile\nindex 36989b7..4b4bd2f 100644\n--- a/Documentation/Makefile\n+++ b/Documentation/Makefile\n@@ -6,7 +6,7 @@ MAN5_TXT=gitattributes.txt gitignore.txt gitmodules.txt githooks.txt \\\n \tgitrepository-layout.txt\n MAN7_TXT=gitcli.txt gittutorial.txt gittutorial-2.txt \\\n \tgitcvs-migration.txt gitcore-tutorial.txt gitglossary.txt \\\n-\tgitdiffcore.txt gitrevisions.txt gitworkflows.txt\n+\tgitdiffcore.txt gitrevisions.txt gitvirtual.txt gitworkflows.txt\n \n MAN_TXT = $(MAN1_TXT) $(MAN5_TXT) $(MAN7_TXT)\n MAN_XML=$(patsubst %.txt,%.xml,$(MAN_TXT))\ndiff --git a/Documentation/git-http-backend.txt b/Documentation/git-http-backend.txt\nindex 4e0b243..4a7c42b 100644\n--- a/Documentation/git-http-backend.txt\n+++ b/Documentation/git-http-backend.txt\n@@ -120,8 +120,8 @@ ScriptAliasMatch \\\n ScriptAlias /git/ /var/www/cgi-bin/gitweb.cgi/\n ----------------------------------------------------------------\n +\n-To serve multiple virtual repositories from a single storage\n-repository:\n+To serve multiple virtual repositories (linkgit:gitvirtual[7]) from a\n+single storage repository:\n +\n ----------------------------------------------------------------\n SetEnvIf Request_URI \"^/git/([^/]*)\" GIT_REF_PREFIX=$1/ GIT_HEAD=$1-HEAD\ndiff --git a/Documentation/gitvirtual.txt b/Documentation/gitvirtual.txt\nnew file mode 100644\nindex 0000000..ed5a4c5\n--- /dev/null\n+++ b/Documentation/gitvirtual.txt\n@@ -0,0 +1,76 @@\n+gitvirtual(7)\n+=============\n+\n+NAME\n+----\n+gitvirtual - Git virtual repositories\n+\n+DESCRIPTION\n+-----------\n+\n+Given many repositories with copies of the same objects (such as\n+branches of the same source), sharing a common object store will avoid\n+duplication.  Alternates provide a single baseline, but don't handle\n+ongoing activity in the various repositories.  Furthermore, operations\n+such as linkgit:git-gc[1] need to know about all of the refs.\n+\n+Git supports storing multiple virtual repositories within the object\n+store and references of a single underlying repository.  The underlying\n+repository stores the objects for all of the virtual repositories, and\n+includes all the refs and heads of the virtual repositories using\n+prefixed names.\n+\n+linkgit:git-upload-pack[1] and linkgit:git-receive-pack[1] rewrite the\n+names of refs and heads as specified by the --ref-prefix and --head\n+options.  For instance, --ref-prefix=`virtual/reponame/` will use\n++pass:[refs/virtual/reponame/heads/*]+ and\n++pass:[refs/virtual/reponame/tags/*]+.  git-upload-pack and\n+git-receive-pack will ignore any references that do not match the\n+specified prefix.\n+\n+These options implement the underlying mechanism for virtual\n+repositories.  The smart HTTP server, linkgit:git-http-backend[1], can\n+accept a ref prefix and HEAD path via environment variables, and pass\n+them to the backend programs.  For a simple local test, you can use\n+git-remote-ext:\n+\n+----------\n+git clone ext::'git %s --ref-prefix=virtual/prefix/ --head=prefix-HEAD /tmp/prefixed.git'\n+----------\n+\n+CONVENTIONS\n+-----------\n+\n+The --ref-prefix and --head options provide quite a bit of flexibility\n+in organizing the refs of virtual repositories within those of the\n+underlying repository.  In the absence of a strong reason to do\n+otherwise, consider following these conventions:\n+\n+--ref-prefix=`virtual/reponame/`::\n+\tThis puts refs under `refs/virtual/reponame/`, which avoids a\n+\tnamespace conflict between `reponame` and built-in ref\n+\tdirectories such as `heads` and `tags`.\n+\n+--head=`virtual-HEAD/reponame`::\n+\tThis puts HEADs under `virtual-HEAD/` to avoid namespace\n+\tconflicts with top-level filenames in a git repository.\n+\n+SECURITY\n+--------\n+\n+Anyone with access to any virtual repository can potentially access\n+objects from any other virtual repository stored in the same underlying\n+repository.  You can't directly say \"give me object ABCD\" if you don't\n+have a ref to it, but you can do some other sneaky things like:\n+\n+. Claiming to push ABCD, at which point the server will optimize out the\n+  need for you to actually send it. Now you have a ref to ABCD and can\n+  fetch it (claiming not to have it, of course).\n+\n+. Requesting other refs, claiming that you have ABCD, at which point the\n+  server may generate deltas against ABCD.\n+\n+None of this causes a problem if you only host public repositories, or\n+if everyone who may read one virtual repo may also read everything in\n+every other virtual repo (for instance, if everyone in an organization\n+has read permission to every repository).\ndiff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash\nindex bb8d7d0..a58d27e 100755\n--- a/contrib/completion/git-completion.bash\n+++ b/contrib/completion/git-completion.bash\n@@ -1469,7 +1469,7 @@ _git_help ()\n \t\tattributes cli core-tutorial cvs-migration\n \t\tdiffcore gitk glossary hooks ignore modules\n \t\trepository-layout tutorial tutorial-2\n-\t\tworkflows\n+\t\tvirtual workflows\n \t\t\"\n }\n \n-- \n1.7.5.1\n"},{"id":"168621","messageId":"7vr57n60eb.fsf@alter.siamese.dyndns.org","threadId":"27449","inReplyTo":"1306284392-12034-1-git-send-email-jamey@minilop.net","subject":"Re: [PATCH v3 1/3] Support multiple virtual repositories with a single object store and refs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-25T01:21:00Z","receivedAt":"2011-05-25T01:21:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jamey Sharp <jamey@minilop.net> writes:\n\n> From: Josh Triplett <josh@joshtriplett.org>\n>\n> Given many repositories with copies of the same objects (such as branches of\n> the same source), sharing a common object store will avoid duplication.\n> Alternates provide a single baseline, but don't handle ongoing activity in the\n> various repositories.  Furthermore, operations such as git-gc need to know\n> about all of the refs.\n>\n> Git supports storing multiple virtual repositories within the object store and\n> references of a single underlying repository.  The underlying repository\n> stores the objects for all of the virtual repositories, and includes all the\n> refs and heads of the virtual repositories using prefixed names.\n\nI do not see anything changed up to this point since the previous\nround... sent a wrong patch?\n\nIn any case, I _think_ what you are trying to say is:\n\n - Implemented in the most naïve way, you can host multiple instances of\n   related projects, but that is wasteful; their object stores will have\n   duplicated objects without sharing. (This is the crucial part missing\n   from your description that confused me when trying to _guess_ what\n   problem you are trying to solve in the first place).\n\n - You _could_ use alternates mechanism to alleviate that problem, but it\n   has issues, e.g. gc needs to be aware of other repositories (This is in\n   your first paragraph).\n\n - Instead, we could store a single, large, repository and carve out its\n   refs namespaces into multiple hierarchies, to make it look as if there\n   are multiple repositories. (The first sentence of the second paragraph\n   also confused me, as you said \"Git supports storing multiple ...\" in\n   present tense).\n\nOne thing you would want to be careful with is what to do with the HEAD\nsymrefs, which should appear to read \"ref: refs/heads/<some-branch>\" from\nthe point of view of the clients that are under the illusion that they are\ninteracting with one specific repository among others, while for the\npurpose of gc and things in the huge single repository they should be\npointing at something like \"refs/hosted-1-project/heads/<that-branch>\",\nbut other than that, after a lot of guesswork, the problem you are trying\nto solve seems clearer to me.\n\nBut please do not make me guess.\n"},{"id":"168674","messageId":"20110525160708.GE8795@sigill.intra.peff.net","threadId":"27449","inReplyTo":"1306284392-12034-3-git-send-email-jamey@minilop.net","subject":"Re: [PATCH v3 3/3] Add documentation for virtual repositories","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-05-25T16:07:08Z","receivedAt":"2011-05-25T16:07:08Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, May 24, 2011 at 05:46:32PM -0700, Jamey Sharp wrote:\n\n>  Documentation/Makefile                 |    2 +-\n>  Documentation/git-http-backend.txt     |    4 +-\n>  Documentation/gitvirtual.txt           |   76 ++++++++++++++++++++++++++++++++\n>  contrib/completion/git-completion.bash |    2 +-\n\nMaybe it would make sense to mention your new options to upload-pack and\nreceive-pack in their manpages; the description can be short, but refer\nthe user to gitvirtual.\n\n> +Given many repositories with copies of the same objects (such as\n> +branches of the same source), sharing a common object store will avoid\n> +duplication.  Alternates provide a single baseline, but don't handle\n> +ongoing activity in the various repositories.  Furthermore, operations\n> +such as linkgit:git-gc[1] need to know about all of the refs.\n\nIt's not quite true that alternates provide only a single baseline. They\ncan be updated and objects consolidated over time (e.g., with a nightly\nrepack). The problem is that they require management to do so (this is\nalso a benefit, if you want a sharing policy besides \"all repos have all\nobjects\").\n\n> +linkgit:git-upload-pack[1] and linkgit:git-receive-pack[1] rewrite the\n> +names of refs and heads as specified by the --ref-prefix and --head\n> +options.  For instance, --ref-prefix=`virtual/reponame/` will use\n> ++pass:[refs/virtual/reponame/heads/*]+ and\n> ++pass:[refs/virtual/reponame/tags/*]+.  git-upload-pack and\n> +git-receive-pack will ignore any references that do not match the\n> +specified prefix.\n\nThinking on the whole idea a bit more, is there a reason to restrict\nthis to upload-pack and receive-pack? Sure, they are the most obvious\nplaces to use it for hosting, but might I not want to be able to do:\n\n  cd /path/to/mega-repository.git\n  git --ref-prefix=virtual/repo1 log master\n\nto do server-side scripting inside the virtual repos (or more likely,\nsetting GIT_REF_PREFIX at the top of your script).\n\n> +The --ref-prefix and --head options provide quite a bit of flexibility\n> +in organizing the refs of virtual repositories within those of the\n> +underlying repository.  In the absence of a strong reason to do\n> +otherwise, consider following these conventions:\n> +\n> +--ref-prefix=`virtual/reponame/`::\n> +\tThis puts refs under `refs/virtual/reponame/`, which avoids a\n> +\tnamespace conflict between `reponame` and built-in ref\n> +\tdirectories such as `heads` and `tags`.\n> +\n> +--head=`virtual-HEAD/reponame`::\n> +\tThis puts HEADs under `virtual-HEAD/` to avoid namespace\n> +\tconflicts with top-level filenames in a git repository.\n\nI'm curious if you have a use for this much flexibility. In particular,\nwhy do the HEAD and refs prefixes need the ability to be separate? Also,\nwhat about other non-HEAD top-level refs? IOW, a true \"virtual\nrepository\" to me would just be:\n\n  GIT_REF_PREFIX=refs/virtual/repo1\n\nand then _every_ ref resolution would just prefix that, whether it was\nin refs/ or not. So you would have:\n\n  .git/refs/virtual/repo1/HEAD\n  .git/refs/virtual/repo1/refs/heads/master\n  .git/refs/virtual/repo1/refs/tags/v1.0\n\nand so on. And this fits in with the idea of it not just being an\nupload-pack and receive-pack thing. I could do:\n\n  GIT_REF_PREFIX=refs/virtual/repo1; export GIT_REF_PREFIX\n  git fetch some-remote\n\nand it would write to .git/refs/virtual/repo1/FETCH_HEAD.\n\nSo the virtual repository is basically just a \"chroot\" of the ref\nnamespace. And it's dirt simple to implement, because you do the\ntranslation at the refs.c layer.\n\n> +SECURITY\n> +--------\n> +\n> +Anyone with access to any virtual repository can potentially access\n> +objects from any other virtual repository stored in the same underlying\n> +repository.  You can't directly say \"give me object ABCD\" if you don't\n> +have a ref to it, but you can do some other sneaky things like:\n> +\n> +. Claiming to push ABCD, at which point the server will optimize out the\n> +  need for you to actually send it. Now you have a ref to ABCD and can\n> +  fetch it (claiming not to have it, of course).\n> +\n> +. Requesting other refs, claiming that you have ABCD, at which point the\n> +  server may generate deltas against ABCD.\n> +\n> +None of this causes a problem if you only host public repositories, or\n> +if everyone who may read one virtual repo may also read everything in\n> +every other virtual repo (for instance, if everyone in an organization\n> +has read permission to every repository).\n\nWell, this text is obviously correct and written by a very smart person.\n;)\n\nYou might want to mention that if you do need to handle these security\nconcerns, then the alternates route, even though it creates more\nmanagement headache, is going to be more flexible with respect to which\nobjects are shared.\n\nIn fact, given what I said at the very top of the email, I wonder if the\ndocumentation would be better structured as \"here are two methods for\nsharing objects, here are reasons why you might choose one or the other,\nand here is how to use each\".\n\n-Peff\n"},{"id":"168675","messageId":"20110525160816.GB4839@oh.minilop.net","threadId":"27449","inReplyTo":"7vr57n60eb.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH v3 1/3] Support multiple virtual repositories with a single object store and refs","fromName":"Jamey Sharp","fromEmail":"jamey@minilop.net","sentAt":"2011-05-25T16:08:16Z","receivedAt":"2011-05-25T16:08:16Z","isPatch":true,"sender":{"key":"jamey@minilop.net","avatar":"https://gravatar.com/avatar/979ed4f9190c19c13a84945afe9a6cf141e8884ef6ddaae342db7a74ae28828e?d=mp&s=160"},"body":"On Tue, May 24, 2011 at 06:21:00PM -0700, Junio C Hamano wrote:\n> Jamey Sharp <jamey@minilop.net> writes:\n> \n> > From: Josh Triplett <josh@joshtriplett.org>\n> >\n> > Given many repositories with copies of the same objects (such as branches of\n> > the same source), sharing a common object store will avoid duplication.\n> > Alternates provide a single baseline, but don't handle ongoing activity in the\n> > various repositories.  Furthermore, operations such as git-gc need to know\n> > about all of the refs.\n> >\n> > Git supports storing multiple virtual repositories within the object store and\n> > references of a single underlying repository.  The underlying repository\n> > stores the objects for all of the virtual repositories, and includes all the\n> > refs and heads of the virtual repositories using prefixed names.\n> \n> I do not see anything changed up to this point since the previous\n> round... sent a wrong patch?\n\nApparently so. I watched Josh fix up that commit message, and then I\ndon't know where it went.\n\n> In any case, I _think_ what you are trying to say is:\n> \n>  - Implemented in the most naïve way, you can host multiple instances of\n>    related projects, but that is wasteful; their object stores will have\n>    duplicated objects without sharing. (This is the crucial part missing\n>    from your description that confused me when trying to _guess_ what\n>    problem you are trying to solve in the first place).\n> \n>  - You _could_ use alternates mechanism to alleviate that problem, but it\n>    has issues, e.g. gc needs to be aware of other repositories (This is in\n>    your first paragraph).\n> \n>  - Instead, we could store a single, large, repository and carve out its\n>    refs namespaces into multiple hierarchies, to make it look as if there\n>    are multiple repositories. (The first sentence of the second paragraph\n>    also confused me, as you said \"Git supports storing multiple ...\" in\n>    present tense).\n\nYes. I hope you won't mind if we blatantly steal this description. :-)\n\n> One thing you would want to be careful with is what to do with the HEAD\n> symrefs, which should appear to read \"ref: refs/heads/<some-branch>\" from\n> the point of view of the clients that are under the illusion that they are\n> interacting with one specific repository among others, while for the\n> purpose of gc and things in the huge single repository they should be\n> pointing at something like \"refs/hosted-1-project/heads/<that-branch>\",\n\nAs far as I can tell, that isn't true. Judging by the pack-protocol\ndocumentation, my reading of the implementation, and the results of some\ntests I ran, symrefs are resolved to hashes before being sent over the\nwire, and then HEAD is magically re-inferred back into a symref on the\nother end.\n\n(This has the odd property that if you create a repository containing\ntwo branches with identical heads, then clone that repository, the\nclone's origin/HEAD will point to a randomly-selected one of the two\nbranches. Tested in version 1.7.4.4, and seems to be a necessary\nconsequence of the protocol design.)\n\nAs a result, symrefs only need to be valid in the underlying repository;\nthere's no mapping needed for the protocol. However, you probably do\nwant a different HEAD for each virtual repository, which is why we added\nthe --head option.\n\nWe didn't actually think about impact of these virtual HEADs on gc. As\nlong as they're all symrefs, they can't matter for gc, right? The head\nthey reference is already a suitable gc root. If the virtual HEADs do\nneed to participate in gc, then I guess we should update the conventions\ndocumentation to recommend that they live somewhere under refs/.\n\n> but other than that, after a lot of guesswork, the problem you are trying\n> to solve seems clearer to me.\n> \n> But please do not make me guess.\n\nIndeed. We'll get that right next round, honest this time. :-/\n\nNow that you have the problem statement down, is the proposed solution\nacceptable for merge?\n\nJamey\n"},{"id":"168683","messageId":"BANLkTikwxiBTVdqnQtdvr-VTCm2hSOcRjw@mail.gmail.com","threadId":"27449","inReplyTo":"20110525160708.GE8795@sigill.intra.peff.net","subject":"Re: [PATCH v3 3/3] Add documentation for virtual repositories","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2011-05-25T17:01:20Z","receivedAt":"2011-05-25T17:01:20Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Wed, May 25, 2011 at 09:07, Jeff King <peff@peff.net> wrote:\n>\n> Thinking on the whole idea a bit more, is there a reason to restrict\n> this to upload-pack and receive-pack? Sure, they are the most obvious\n> places to use it for hosting, but might I not want to be able to do:\n\nNo, there isn't. I had the same impression reading this series... that\ndoing it in upload-pack receive-pack was wrong, but I couldn't put my\nfinger on why. I think you did (below), so thank you.\n\n>  cd /path/to/mega-repository.git\n>  git --ref-prefix=virtual/repo1 log master\n>\n> to do server-side scripting inside the virtual repos (or more likely,\n> setting GIT_REF_PREFIX at the top of your script).\n>\n>> +The --ref-prefix and --head options provide quite a bit of flexibility\n...\n> I'm curious if you have a use for this much flexibility. In particular,\n> why do the HEAD and refs prefixes need the ability to be separate? Also,\n> what about other non-HEAD top-level refs? IOW, a true \"virtual\n> repository\" to me would just be:\n>\n>  GIT_REF_PREFIX=refs/virtual/repo1\n>\n> and then _every_ ref resolution would just prefix that, whether it was\n> in refs/ or not. So you would have:\n>\n>  .git/refs/virtual/repo1/HEAD\n>  .git/refs/virtual/repo1/refs/heads/master\n>  .git/refs/virtual/repo1/refs/tags/v1.0\n>\n> and so on. And this fits in with the idea of it not just being an\n> upload-pack and receive-pack thing. I could do:\n>\n>  GIT_REF_PREFIX=refs/virtual/repo1; export GIT_REF_PREFIX\n>  git fetch some-remote\n\n+1 * 1000. This should be a single environment variable / top level\noption. HEAD should also use the GIT_REF_PREFIX, like any other ref.\n\nFETCH_HEAD and MERGE_HEAD probably should as well if GIT_REF_PREFIX is\nset, however these are going to be a bit harder to move. Not all tools\nthat read them are GIT_REF_PREFIX aware, or go through C code that can\nbe modified to be GIT_REF_PREFIX aware. (git-gui and EGit, I'm talking\nabout you here!)  They can obviously be fixed, but until then using a\nworking directory with GIT_REF_PREFIX set will be slightly\ninteresting.\n\n> So the virtual repository is basically just a \"chroot\" of the ref\n> namespace. And it's dirt simple to implement, because you do the\n> translation at the refs.c layer.\n\nYes, exactly.\n\n-- \nShawn.\n"},{"id":"168685","messageId":"20110525171021.GA24038@sigill.intra.peff.net","threadId":"27449","inReplyTo":"BANLkTikwxiBTVdqnQtdvr-VTCm2hSOcRjw@mail.gmail.com","subject":"Re: [PATCH v3 3/3] Add documentation for virtual repositories","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-05-25T17:10:22Z","receivedAt":"2011-05-25T17:10:22Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, May 25, 2011 at 10:01:20AM -0700, Shawn O. Pearce wrote:\n\n> > and so on. And this fits in with the idea of it not just being an\n> > upload-pack and receive-pack thing. I could do:\n> >\n> >  GIT_REF_PREFIX=refs/virtual/repo1; export GIT_REF_PREFIX\n> >  git fetch some-remote\n> \n> +1 * 1000. This should be a single environment variable / top level\n> option. HEAD should also use the GIT_REF_PREFIX, like any other ref.\n\nGood, I am glad I'm not the only one thinking this. :)\n\n> FETCH_HEAD and MERGE_HEAD probably should as well if GIT_REF_PREFIX is\n> set, however these are going to be a bit harder to move. Not all tools\n> that read them are GIT_REF_PREFIX aware, or go through C code that can\n> be modified to be GIT_REF_PREFIX aware. (git-gui and EGit, I'm talking\n> about you here!)  They can obviously be fixed, but until then using a\n> working directory with GIT_REF_PREFIX set will be slightly\n> interesting.\n\nYeah, most scripts these days will need to go through the C programs to\nhandle packed-refs. But the top-level pseudo-refs are a bit more\nmagical. That is perhaps why they split HEAD and refs handling in the\noriginal patch. Still, I don't think it's insurmountable.\n\n> > So the virtual repository is basically just a \"chroot\" of the ref\n> > namespace. And it's dirt simple to implement, because you do the\n> > translation at the refs.c layer.\n> \n> Yes, exactly.\n\nLike chroots, there is a sticky point with symbolic links. What should\n\"refs/virtual/repo1/HEAD\" have in it? Either:\n\n  ref: refs/virtual/repo1/refs/heads/master\n\nor\n\n  ref: refs/heads/master\n\n?\n\nIf the former, then we will have to make sure the ref is inside our\nprefix, and strip it out. If the latter, then you will get different\nresults for:\n\n  git show refs/virtual/repo1/HEAD\n\nversus\n\n  GIT_REF_PREFIX=refs/virtual/repo1 git show HEAD\n\nwhich I think is a bad thing.\n\n-Peff\n"},{"id":"168688","messageId":"7vsjs24rzm.fsf@alter.siamese.dyndns.org","threadId":"27449","inReplyTo":"20110525160708.GE8795@sigill.intra.peff.net","subject":"Re: [PATCH v3 3/3] Add documentation for virtual repositories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-25T17:20:13Z","receivedAt":"2011-05-25T17:20:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> I'm curious if you have a use for this much flexibility. In particular,\n> why do the HEAD and refs prefixes need the ability to be separate?...\n> ...\n> So the virtual repository is basically just a \"chroot\" of the ref\n> namespace.\n\nYes, that makes much more sense than singling HEAD out.\n"},{"id":"168694","messageId":"20110525180341.GA2324@leaf","threadId":"27449","inReplyTo":"20110525160708.GE8795@sigill.intra.peff.net","subject":"Re: [PATCH v3 3/3] Add documentation for virtual repositories","fromName":"Josh Triplett","fromEmail":"josh@joshtriplett.org","sentAt":"2011-05-25T18:03:42Z","receivedAt":"2011-05-25T18:03:42Z","isPatch":true,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"On Wed, May 25, 2011 at 12:07:08PM -0400, Jeff King wrote:\n> On Tue, May 24, 2011 at 05:46:32PM -0700, Jamey Sharp wrote:\n> \n> >  Documentation/Makefile                 |    2 +-\n> >  Documentation/git-http-backend.txt     |    4 +-\n> >  Documentation/gitvirtual.txt           |   76 ++++++++++++++++++++++++++++++++\n> >  contrib/completion/git-completion.bash |    2 +-\n> \n> Maybe it would make sense to mention your new options to upload-pack and\n> receive-pack in their manpages; the description can be short, but refer\n> the user to gitvirtual.\n\nFair enough.  We'll go ahead and document them (in patch 1/3 with a\nreference to gitvirtual added in patch 3/3), and avoid making the pile\nof undocumented upload-pack and receive-pack options larger. :)\n\n> > +Given many repositories with copies of the same objects (such as\n> > +branches of the same source), sharing a common object store will avoid\n> > +duplication.  Alternates provide a single baseline, but don't handle\n> > +ongoing activity in the various repositories.  Furthermore, operations\n> > +such as linkgit:git-gc[1] need to know about all of the refs.\n> \n> It's not quite true that alternates provide only a single baseline. They\n> can be updated and objects consolidated over time (e.g., with a nightly\n> repack). The problem is that they require management to do so (this is\n> also a benefit, if you want a sharing policy besides \"all repos have all\n> objects\").\n\nTrue enough.  We wanted something that automatically worked without\nbackground maintenance, but alternates can help if you keep moving\ncommon objects to the alternate repository.\n\n> > +linkgit:git-upload-pack[1] and linkgit:git-receive-pack[1] rewrite the\n> > +names of refs and heads as specified by the --ref-prefix and --head\n> > +options.  For instance, --ref-prefix=`virtual/reponame/` will use\n> > ++pass:[refs/virtual/reponame/heads/*]+ and\n> > ++pass:[refs/virtual/reponame/tags/*]+.  git-upload-pack and\n> > +git-receive-pack will ignore any references that do not match the\n> > +specified prefix.\n> \n> Thinking on the whole idea a bit more, is there a reason to restrict\n> this to upload-pack and receive-pack? Sure, they are the most obvious\n> places to use it for hosting, but might I not want to be able to do:\n> \n>   cd /path/to/mega-repository.git\n>   git --ref-prefix=virtual/repo1 log master\n> \n> to do server-side scripting inside the virtual repos (or more likely,\n> setting GIT_REF_PREFIX at the top of your script).\n\nMany git commands will need special handling for this, though.  For\ninstance, gc needs to know about all refs, not just a prefix of refs;\notherwise it will break the repository.  Or, for an example within a\nsingle command, the checks for updating a currently-checked-out ref in a\nrepository need to use the repository's HEAD, not the virtual HEAD.\nAnd similarly, git checkout with a ref-prefix set would construct a\nrepository where HEAD doesn't match the workdir.\n\nHaving this handled \"transparently\" for all git commands seems likely to\nrun into this kind of corner case, where parts of a git command run\ncorrectly with ref-prefix but other parts or other invoked git commands\nmust not run with ref-prefix.\n\nI do agree that some other git programs could learn to use ref-prefix,\nand it makes sense to move the functionality into refs.c as a general\nmechanism for those programs to use.  However, I don't think it makes\nsense to transparently make all git programs use ref-prefix without\nchecking them individually to see if it makes sense.\n\n> > +The --ref-prefix and --head options provide quite a bit of flexibility\n> > +in organizing the refs of virtual repositories within those of the\n> > +underlying repository.  In the absence of a strong reason to do\n> > +otherwise, consider following these conventions:\n> > +\n> > +--ref-prefix=`virtual/reponame/`::\n> > +\tThis puts refs under `refs/virtual/reponame/`, which avoids a\n> > +\tnamespace conflict between `reponame` and built-in ref\n> > +\tdirectories such as `heads` and `tags`.\n> > +\n> > +--head=`virtual-HEAD/reponame`::\n> > +\tThis puts HEADs under `virtual-HEAD/` to avoid namespace\n> > +\tconflicts with top-level filenames in a git repository.\n> \n> I'm curious if you have a use for this much flexibility. In particular,\n> why do the HEAD and refs prefixes need the ability to be separate? Also,\n> what about other non-HEAD top-level refs? IOW, a true \"virtual\n> repository\" to me would just be:\n> \n>   GIT_REF_PREFIX=refs/virtual/repo1\n> \n> and then _every_ ref resolution would just prefix that, whether it was\n> in refs/ or not. So you would have:\n> \n>   .git/refs/virtual/repo1/HEAD\n>   .git/refs/virtual/repo1/refs/heads/master\n>   .git/refs/virtual/repo1/refs/tags/v1.0\n\nAh, *now* I see what you meant by including the repeated \"refs/\", and\nusing that to allow putting HEAD in the same namespace makes sense.\n\nWe don't actually need the flexibility of putting HEAD in a different\nplace, and this layout makes sense, so we can change the ref-prefix\nmechanism to drop the separate --head entirely.\n\n> > +SECURITY\n> > +--------\n> > +\n> > +Anyone with access to any virtual repository can potentially access\n> > +objects from any other virtual repository stored in the same underlying\n> > +repository.  You can't directly say \"give me object ABCD\" if you don't\n> > +have a ref to it, but you can do some other sneaky things like:\n> > +\n> > +. Claiming to push ABCD, at which point the server will optimize out the\n> > +  need for you to actually send it. Now you have a ref to ABCD and can\n> > +  fetch it (claiming not to have it, of course).\n> > +\n> > +. Requesting other refs, claiming that you have ABCD, at which point the\n> > +  server may generate deltas against ABCD.\n> > +\n> > +None of this causes a problem if you only host public repositories, or\n> > +if everyone who may read one virtual repo may also read everything in\n> > +every other virtual repo (for instance, if everyone in an organization\n> > +has read permission to every repository).\n> \n> Well, this text is obviously correct and written by a very smart person.\n> ;)\n> \n> You might want to mention that if you do need to handle these security\n> concerns, then the alternates route, even though it creates more\n> management headache, is going to be more flexible with respect to which\n> objects are shared.\n> \n> In fact, given what I said at the very top of the email, I wonder if the\n> documentation would be better structured as \"here are two methods for\n> sharing objects, here are reasons why you might choose one or the other,\n> and here is how to use each\".\n\nI think it makes sense to reference alternates in the gitvirtual page, but\nI don't think it makes sense to put the full documentation for both in\nthe same page.\n\n- Josh Triplett\n"},{"id":"168810","messageId":"BANLkTimQO3AFeSmmFZOBLLGwEDgQk33Euw@mail.gmail.com","threadId":"27449","inReplyTo":"20110525171021.GA24038@sigill.intra.peff.net","subject":"Re: [PATCH v3 3/3] Add documentation for virtual repositories","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2011-05-26T18:28:33Z","receivedAt":"2011-05-26T18:28:33Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Wed, May 25, 2011 at 10:10, Jeff King <peff@peff.net> wrote:\n> Like chroots, there is a sticky point with symbolic links. What should\n> \"refs/virtual/repo1/HEAD\" have in it? Either:\n>\n>  ref: refs/virtual/repo1/refs/heads/master\n...\n> If the former, then we will have to make sure the ref is inside our\n> prefix, and strip it out.\n\nYes. And if its outside of our chroot space, we have to hide the\nsymbolic reference as though it is broken. This is fairly simple since\nsymbolic references are supposed to be \"absolute\", we can do a quick\nprefix test and either strip the prefix on read (or add it on write),\nor pretend like the reference is broken and hide it if its outside the\nprefix.\n\n> If the latter, then you will get different\n> results for:\n>\n>  git show refs/virtual/repo1/HEAD\n>\n> versus\n>\n>  GIT_REF_PREFIX=refs/virtual/repo1 git show HEAD\n>\n> which I think is a bad thing.\n\nYes, I agree, this would be bad. Hence we should store the real path\nin the symbolic reference, but strip/add the prefix when\nGIT_REF_PREFIX is set.\n\n-- \nShawn.\n"}]}