{"thread":{"id":"27440","subject":"[PATCH] Support multiple virtual repositories with a single object store and refs","startedAt":"2011-05-24T01:02:52Z","lastAt":"2011-05-25T16:56:47Z","messageCount":4,"participants":["Josh Triplett","Jeff King","Shawn Pearce"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"168561","messageId":"20110524010252.GA5368@leaf","threadId":"27440","inReplyTo":null,"subject":"[PATCH] Support multiple virtual repositories with a single object store and refs","fromName":"Josh Triplett","fromEmail":"josh@joshtriplett.org","sentAt":"2011-05-24T01:02:52Z","receivedAt":"2011-05-24T01:02:52Z","isPatch":true,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"Given many repositories with copies of the same objects (such as\nbranches of the same source), sharing a common object store will avoid\nduplication.  Alternates provide a single baseline, but don't handle\nongoing activity in the various repositories.  Git safely handles\nconcurrent accesses to the same object store across repositories, but\noperations such as gc need to know about all of the refs.\n\nThis change adds support in upload-pack and receive-pack to simulate\nmultiple virtual repositories within the object store and references of\na single underlying repository.  The refs and heads of the virtual\nrepositories get stored in the underlying repository using prefixed\nnames specified by the --ref-prefix and --head options; for instance,\n--ref-prefix=repo1/ will use refs/repo1/heads/* and refs/repo1/tags/*.\nupload-pack and receive-pack will not expose any references that do not\nmatch the specified prefix.\n\nThese options implement the underlying mechanism for virtual\nrepositories; the higher-level protocol handler (such as http-backend or\na custom server) can pass these options when invoking upload-pack or\nreceive-pack, providing values based on components of the repository\npath.  For a simple local test, git-remote-ext works:\n\ngit clone ext::'git %s --ref-prefix=prefix/ --head=prefix-HEAD /tmp/prefixed.git'\n\nCommit by Josh Triplett and Jamey Sharp.\nSigned-off-by: Josh Triplett <josh@joshtriplett.org>\nSigned-off-by: Jamey Sharp <jamey@minilop.net>\n---\n builtin/receive-pack.c |   38 +++++++++++++++++++++++++++++---------\n upload-pack.c          |   34 +++++++++++++++++++++++++++-------\n 2 files changed, 56 insertions(+), 16 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex e1ba4dc..45d0b35 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -34,6 +34,8 @@ static int prefer_ofs_delta = 1;\n static int auto_update_server_info;\n static int auto_gc = 1;\n static const char *head_name;\n+static const char *head_path = \"HEAD\";\n+static const char *ref_prefix = \"refs/\";\n static int sent_capabilities;\n \n static enum deny_action parse_deny_action(const char *var, const char *value)\n@@ -108,11 +110,12 @@ static int receive_pack_config(const char *var, const char *value, void *cb)\n \n static int show_ref(const char *path, const unsigned char *sha1, int flag, void *cb_data)\n {\n+\tconst char *refnameprefix = cb_data;\n \tif (sent_capabilities)\n-\t\tpacket_write(1, \"%s %s\\n\", sha1_to_hex(sha1), path);\n+\t\tpacket_write(1, \"%s %s%s\\n\", sha1_to_hex(sha1), refnameprefix, path);\n \telse\n-\t\tpacket_write(1, \"%s %s%c%s%s\\n\",\n-\t\t\t     sha1_to_hex(sha1), path, 0,\n+\t\tpacket_write(1, \"%s %s%s%c%s%s\\n\",\n+\t\t\t     sha1_to_hex(sha1), refnameprefix, path, 0,\n \t\t\t     \" report-status delete-refs side-band-64k\",\n \t\t\t     prefer_ofs_delta ? \" ofs-delta\" : \"\");\n \tsent_capabilities = 1;\n@@ -121,9 +124,9 @@ static int show_ref(const char *path, const unsigned char *sha1, int flag, void\n \n static void write_head_info(void)\n {\n-\tfor_each_ref(show_ref, NULL);\n+\tfor_each_ref_in(ref_prefix, show_ref, \"refs/\");\n \tif (!sent_capabilities)\n-\t\tshow_ref(\"capabilities^{}\", null_sha1, 0, NULL);\n+\t\tshow_ref(\"capabilities^{}\", null_sha1, 0, \"\");\n \n }\n \n@@ -332,6 +335,8 @@ static void refuse_unconfigured_deny_delete_current(void)\n static const char *update(struct command *cmd)\n {\n \tconst char *name = cmd->ref_name;\n+\tstruct strbuf prefixed_name_buf = STRBUF_INIT;\n+\tconst char *prefixed_name;\n \tunsigned char *old_sha1 = cmd->old_sha1;\n \tunsigned char *new_sha1 = cmd->new_sha1;\n \tstruct ref_lock *lock;\n@@ -342,7 +347,12 @@ static const char *update(struct command *cmd)\n \t\treturn \"funny refname\";\n \t}\n \n-\tif (is_ref_checked_out(name)) {\n+\tstrbuf_addf(&prefixed_name_buf, \"%s%s\", ref_prefix, name + 5);\n+\tprefixed_name = strbuf_detach(&prefixed_name_buf, NULL);\n+\n+\trp_warning(\"name \\\"%s\\\", prefixed_name \\\"%s\\\"\", name, prefixed_name);\n+\n+\tif (is_ref_checked_out(prefixed_name)) {\n \t\tswitch (deny_current_branch) {\n \t\tcase DENY_IGNORE:\n \t\t\tbreak;\n@@ -370,7 +380,7 @@ static const char *update(struct command *cmd)\n \t\t\treturn \"deletion prohibited\";\n \t\t}\n \n-\t\tif (!strcmp(name, head_name)) {\n+\t\tif (!strcmp(prefixed_name, head_name)) {\n \t\t\tswitch (deny_delete_current) {\n \t\t\tcase DENY_IGNORE:\n \t\t\t\tbreak;\n@@ -426,14 +436,14 @@ static const char *update(struct command *cmd)\n \t\t\trp_warning(\"Allowing deletion of corrupt ref.\");\n \t\t\told_sha1 = NULL;\n \t\t}\n-\t\tif (delete_ref(name, old_sha1, 0)) {\n+\t\tif (delete_ref(prefixed_name, old_sha1, 0)) {\n \t\t\trp_error(\"failed to delete %s\", name);\n \t\t\treturn \"failed to delete\";\n \t\t}\n \t\treturn NULL; /* good */\n \t}\n \telse {\n-\t\tlock = lock_any_ref_for_update(name, old_sha1, 0);\n+\t\tlock = lock_any_ref_for_update(prefixed_name, old_sha1, 0);\n \t\tif (!lock) {\n \t\t\trp_error(\"failed to lock %s\", name);\n \t\t\treturn \"failed to lock\";\n@@ -760,6 +770,16 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\tadvertise_refs = 1;\n \t\t\t\tcontinue;\n \t\t\t}\n+\t\t\tif (!prefixcmp(arg, \"--head=\")) {\n+\t\t\t\thead_path = arg+7;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tif (!prefixcmp(arg, \"--ref-prefix=\")) {\n+\t\t\t\tstruct strbuf prefixbuf = STRBUF_INIT;\n+\t\t\t\tstrbuf_addf(&prefixbuf, \"refs/%s\", arg+13);\n+\t\t\t\tref_prefix = strbuf_detach(&prefixbuf, NULL);\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t\tif (!strcmp(arg, \"--stateless-rpc\")) {\n \t\t\t\tstateless_rpc = 1;\n \t\t\t\tcontinue;\ndiff --git a/upload-pack.c b/upload-pack.c\nindex ce5cbbe..a1e495f 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -34,6 +34,8 @@ static int shallow_nr;\n static struct object_array have_obj;\n static struct object_array want_obj;\n static struct object_array extra_edge_obj;\n+static const char *head_path = \"HEAD\";\n+static const char *ref_prefix = \"\";\n static unsigned int timeout;\n /* 0 for no sideband,\n  * otherwise maximum packet size (up to 65520 bytes).\n@@ -640,17 +642,18 @@ static int send_ref(const char *refname, const unsigned char *sha1, int flag, vo\n \tstatic const char *capabilities = \"multi_ack thin-pack side-band\"\n \t\t\" side-band-64k ofs-delta shallow no-progress\"\n \t\t\" include-tag multi_ack_detailed\";\n+\tconst char *refnameprefix = cb_data;\n \tstruct object *o = parse_object(sha1);\n \n \tif (!o)\n \t\tdie(\"git upload-pack: cannot find object %s:\", sha1_to_hex(sha1));\n \n \tif (capabilities)\n-\t\tpacket_write(1, \"%s %s%c%s%s\\n\", sha1_to_hex(sha1), refname,\n+\t\tpacket_write(1, \"%s %s%s%c%s%s\\n\", sha1_to_hex(sha1), refnameprefix, refname,\n \t\t\t     0, capabilities,\n \t\t\t     stateless_rpc ? \" no-done\" : \"\");\n \telse\n-\t\tpacket_write(1, \"%s %s\\n\", sha1_to_hex(sha1), refname);\n+\t\tpacket_write(1, \"%s %s%s\\n\", sha1_to_hex(sha1), refnameprefix, refname);\n \tcapabilities = NULL;\n \tif (!(o->flags & OUR_REF)) {\n \t\to->flags |= OUR_REF;\n@@ -659,7 +662,7 @@ static int send_ref(const char *refname, const unsigned char *sha1, int flag, vo\n \tif (o->type == OBJ_TAG) {\n \t\to = deref_tag(o, refname, 0);\n \t\tif (o)\n-\t\t\tpacket_write(1, \"%s %s^{}\\n\", sha1_to_hex(o->sha1), refname);\n+\t\t\tpacket_write(1, \"%s %s%s^{}\\n\", sha1_to_hex(o->sha1), refnameprefix, refname);\n \t}\n \treturn 0;\n }\n@@ -678,15 +681,24 @@ static int mark_our_ref(const char *refname, const unsigned char *sha1, int flag\n \n static void upload_pack(void)\n {\n+\tstruct strbuf prefix = STRBUF_INIT;\n+\tunsigned char sha1[20];\n+\tint flag;\n+\n+\tstrbuf_addf(&prefix, \"refs/%s\", ref_prefix);\n \tif (advertise_refs || !stateless_rpc) {\n \t\treset_timeout();\n-\t\thead_ref(send_ref, NULL);\n-\t\tfor_each_ref(send_ref, NULL);\n+\t\tif (resolve_ref(head_path, sha1, 1, &flag))\n+\t\t\tsend_ref(\"HEAD\", sha1, flag, \"\");\n+\t\tfor_each_ref_in(prefix.buf, send_ref, \"refs/\");\n \t\tpacket_flush(1);\n \t} else {\n-\t\thead_ref(mark_our_ref, NULL);\n-\t\tfor_each_ref(mark_our_ref, NULL);\n+\t\tif (resolve_ref(head_path, sha1, 1, &flag))\n+\t\t\tmark_our_ref(\"HEAD\", sha1, flag, NULL);\n+\t\tfor_each_ref_in(prefix.buf, mark_our_ref, NULL);\n \t}\n+\tstrbuf_release(&prefix);\n+\n \tif (advertise_refs)\n \t\treturn;\n \n@@ -716,6 +728,14 @@ int main(int argc, char **argv)\n \t\t\tadvertise_refs = 1;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!prefixcmp(arg, \"--head=\")) {\n+\t\t\thead_path = arg+7;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!prefixcmp(arg, \"--ref-prefix=\")) {\n+\t\t\tref_prefix = arg+13;\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!strcmp(arg, \"--stateless-rpc\")) {\n \t\t\tstateless_rpc = 1;\n \t\t\tcontinue;\n-- \n1.7.5.1\n"},{"id":"168593","messageId":"20110524195937.GE584@sigill.intra.peff.net","threadId":"27440","inReplyTo":"20110524010252.GA5368@leaf","subject":"Re: [PATCH] Support multiple virtual repositories with a single object store and refs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-05-24T19:59:37Z","receivedAt":"2011-05-24T19:59:37Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, May 23, 2011 at 06:02:52PM -0700, Josh Triplett wrote:\n\n> Given many repositories with copies of the same objects (such as\n> branches of the same source), sharing a common object store will avoid\n> duplication.  Alternates provide a single baseline, but don't handle\n> ongoing activity in the various repositories.  Git safely handles\n> concurrent accesses to the same object store across repositories, but\n> operations such as gc need to know about all of the refs.\n> \n> This change adds support in upload-pack and receive-pack to simulate\n> multiple virtual repositories within the object store and references of\n> a single underlying repository.\n\nNeat idea. It is important to note, though, that it is possible to leak\ninformation between virtual repos that share the same object store. You\ncan't directly say \"give me object ABCD\" if you don't have a ref to it,\nbut you can do some other sneaky things like:\n\n  1. Claiming to push ABCD, at which point the server will optimize out\n     the need for you to actually send it. Now you have a ref to ABCD\n     and can fetch it (claiming not to have it, of course).\n\n  2. Requesting other refs, claiming that you have ABCD, at which point\n     the server may generates deltas against ABCD.\n\nBoth are problems with alternates, too, of course. But in the case of\nalternates, you can share only a subset of the objects. So every day or\nso, you could pack all of the objects that _all_ repos can see into one\nbig alternates repo, and then each \"leaf\" repo contains any objects\nprivate to itself.\n\nOf course none of this is a concern if you are just hosting public\nrepositories, or everyone who gets to see one virtual repo can see\nwhat's in other ones (e.g., everybody is sharing objects within one\norganization).\n\nBut it may make sense to touch on these issues in the documentation\n(which also needs to be written at all :) ).\n\n> The refs and heads of the virtual repositories get stored in the\n> underlying repository using prefixed names specified by the\n> --ref-prefix and --head options; for instance, --ref-prefix=repo1/\n> will use refs/repo1/heads/* and refs/repo1/tags/*.  upload-pack and\n> receive-pack will not expose any references that do not match the\n> specified prefix.\n\nYou have a namespace clash if a repo is named \"heads\" or \"tags\" or\n\"remotes\". Should we give it its own namespace, like:\n\n  refs/virtual/repo1/heads/*\n\n?\n\nAlso, it seems conceptually simpler to me if it's a straight prefix.\nIOW, \"refs/heads/foo\" in repo1 becomes:\n\n  refs/virtual/repo1/refs/heads/foo\n\nThen if we are operating in the virtual repo1 space, then:\n\n  1. It is an easy test to know whether we are allowed to see a ref:\n     \"does it start with refs/virtual/$repo/ ?\"\n\n  2. Converting back and forth is simple. You just prepend or strip the\n     refs/virtual/$repo prefix.\n\n-Peff\n"},{"id":"168608","messageId":"20110524224711.GA2527@leaf","threadId":"27440","inReplyTo":"20110524195937.GE584@sigill.intra.peff.net","subject":"Re: [PATCH] Support multiple virtual repositories with a single object store and refs","fromName":"Josh Triplett","fromEmail":"josh@joshtriplett.org","sentAt":"2011-05-24T22:47:11Z","receivedAt":"2011-05-24T22:47:11Z","isPatch":true,"sender":{"key":"josh@joshtriplett.org","avatar":"https://avatars.githubusercontent.com/u/162737?v=4"},"body":"Thanks for your feedback!  We just sent v2 of the patch (with a new patch\nto http-backend) before seeing your mail, so we'll send out a v3\nincorporating your feedback.\n\nOn Tue, May 24, 2011 at 03:59:37PM -0400, Jeff King wrote:\n> On Mon, May 23, 2011 at 06:02:52PM -0700, Josh Triplett wrote:\n> > Given many repositories with copies of the same objects (such as\n> > branches of the same source), sharing a common object store will avoid\n> > duplication.  Alternates provide a single baseline, but don't handle\n> > ongoing activity in the various repositories.  Git safely handles\n> > concurrent accesses to the same object store across repositories, but\n> > operations such as gc need to know about all of the refs.\n> > \n> > This change adds support in upload-pack and receive-pack to simulate\n> > multiple virtual repositories within the object store and references of\n> > a single underlying repository.\n> \n> Neat idea. It is important to note, though, that it is possible to leak\n> information between virtual repos that share the same object store. You\n> can't directly say \"give me object ABCD\" if you don't have a ref to it,\n> but you can do some other sneaky things like:\n> \n>   1. Claiming to push ABCD, at which point the server will optimize out\n>      the need for you to actually send it. Now you have a ref to ABCD\n>      and can fetch it (claiming not to have it, of course).\n> \n>   2. Requesting other refs, claiming that you have ABCD, at which point\n>      the server may generates deltas against ABCD.\n> \n> Both are problems with alternates, too, of course. But in the case of\n> alternates, you can share only a subset of the objects. So every day or\n> so, you could pack all of the objects that _all_ repos can see into one\n> big alternates repo, and then each \"leaf\" repo contains any objects\n> private to itself.\n> \n> Of course none of this is a concern if you are just hosting public\n> repositories, or everyone who gets to see one virtual repo can see\n> what's in other ones (e.g., everybody is sharing objects within one\n> organization).\n\nWe hadn't thought of those ways to access objects from another virtual\nrepository.  We had already planned to use separate storage repositories\nfor separate security domains for exactly such reasons, though.  Fixing\nthose issues seems possible if someone cares about security models other\nthan the everyone-can-read model you mentioned, but given that our use\ncase fits in that model we'd like to leave that as Someone Else's\nProblem(tm). :)\n\n> But it may make sense to touch on these issues in the documentation\n> (which also needs to be written at all :) ).\n\nv2 of the patch includes some documentation in the http-backend manpage,\nthough we probably should have a separate manpage documenting the whole\nconcept and reference that from any backends which implement it.  How\ndoes gitvirtual(1) sound?\n\nWe didn't document the new upload-pack and receive-pack options, but\nthose programs already seem to have a pile of undocumented options. :)\n\nv3 will include additional documentation, and we'll make sure to mention\nthe security implications.\n\n> > The refs and heads of the virtual repositories get stored in the\n> > underlying repository using prefixed names specified by the\n> > --ref-prefix and --head options; for instance, --ref-prefix=repo1/\n> > will use refs/repo1/heads/* and refs/repo1/tags/*.  upload-pack and\n> > receive-pack will not expose any references that do not match the\n> > specified prefix.\n> \n> You have a namespace clash if a repo is named \"heads\" or \"tags\" or\n> \"remotes\". Should we give it its own namespace, like:\n> \n>   refs/virtual/repo1/heads/*\n> \n> ?\n> \n> Also, it seems conceptually simpler to me if it's a straight prefix.\n> IOW, \"refs/heads/foo\" in repo1 becomes:\n> \n>   refs/virtual/repo1/refs/heads/foo\n>\n> Then if we are operating in the virtual repo1 space, then:\n> \n>   1. It is an easy test to know whether we are allowed to see a ref:\n>      \"does it start with refs/virtual/$repo/ ?\"\n>\n>   2. Converting back and forth is simple. You just prepend or strip the\n>      refs/virtual/$repo prefix.\n\nBoth of the namespaces you suggested work with our current patch:\n--ref-prefix=virtual/repo1/, or --ref-prefix=virtual/repo1/refs/.  We'd\nlike to leave the exact choice of paths up to the policies of the host,\nbut your suggestion does seem like a good general namespacing policy.\n\n- Josh Triplett and Jamey Sharp\n"},{"id":"168682","messageId":"BANLkTi=WTbEbRoR7pEzja9e5hfoApFhw_w@mail.gmail.com","threadId":"27440","inReplyTo":"20110524224711.GA2527@leaf","subject":"Re: [PATCH] Support multiple virtual repositories with a single object store and refs","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2011-05-25T16:56:47Z","receivedAt":"2011-05-25T16:56:47Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Tue, May 24, 2011 at 15:47, Josh Triplett <josh@joshtriplett.org> wrote:\n> On Tue, May 24, 2011 at 03:59:37PM -0400, Jeff King wrote:\n>> You have a namespace clash if a repo is named \"heads\" or \"tags\" or\n>> \"remotes\". Should we give it its own namespace, like:\n>>\n>>   refs/virtual/repo1/heads/*\n\nYes, I strongly agree with Peff here. We should \"standardize\" the\nprefix of \"refs/virtual/\" for these things, to keep them from\ninteferring with the other \"standard\" namespaces of refs/heads,\nrefs/remotes, refs/tags, refs/notes, and if Gerrit Code Review is\nused, refs/changes.\n\n>> Also, it seems conceptually simpler to me if it's a straight prefix.\n>> IOW, \"refs/heads/foo\" in repo1 becomes:\n>>\n>>   refs/virtual/repo1/refs/heads/foo\n\nI also think this is a great idea. It vastly simplifies the operations\ninvolved and allows each virtual namespace to have its own HEAD within\nthe virtual namespace, as a sibling of \"refs\", just like in a normal\nrepository. It may seem a little ugly to put two refs in there, but I\nthink this is easily understood by repository owners/administrators\nand will keep the implementation much more simple.\n\n>> Then if we are operating in the virtual repo1 space, then:\n>>\n>>   1. It is an easy test to know whether we are allowed to see a ref:\n>>      \"does it start with refs/virtual/$repo/ ?\"\n>>\n>>   2. Converting back and forth is simple. You just prepend or strip the\n>>      refs/virtual/$repo prefix.\n>\n> Both of the namespaces you suggested work with our current patch:\n> --ref-prefix=virtual/repo1/, or --ref-prefix=virtual/repo1/refs/.  We'd\n> like to leave the exact choice of paths up to the policies of the host,\n> but your suggestion does seem like a good general namespacing policy.\n\nIdeally your implementation would only add/remove the prefix and\nwouldn't muck around with the \"refs/\" part. Then step 1 and step 2 are\ntrivial.\n\nDon't forget that a lot of Git usage comes from shell scripts. Being\nable to use git for-each-ref with a simple sed script to process these\nvirtual namespaces is really important. If the sed script just needs\nto remove a prefix, this easy. If it needs to remove part of the\nprefix and replace with something else, its a bit more complicated for\nthe script writer to work with.\n\n-- \nShawn.\n"}]}