{"thread":{"id":"32664","subject":"[PATCH 0/2] Hiding some refs in ls-remote","startedAt":"2013-01-19T00:37:04Z","lastAt":"2013-01-21T23:45:49Z","messageCount":16,"participants":["Junio C Hamano","Duy Nguyen","Michael Haggerty","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"207238","messageId":"1358555826-11883-1-git-send-email-gitster@pobox.com","threadId":"32664","inReplyTo":null,"subject":"[PATCH 0/2] Hiding some refs in ls-remote","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-19T00:37:04Z","receivedAt":"2013-01-19T00:37:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This is an early preview of reducing the network cost while talking\nwith a repository with tons of refs, most of which are of use by\nvery narrow audiences (e.g. refs under Gerrit's refs/changes/ are\nuseful only for people who are interested in the changes under\nreview).  As long as these narrow audiences have a way to learn the\nnames of refs or objects pointed at by the refs out-of-band, it is\nnot necessary to advertise these refs.\n\nOn the server end, you tell upload-pack that some refs do not have\nto be advertised with the uploadPack.hiderefs multi-valued\nconfiguration variable:\n\n\t[uploadPack]\n\t\thiderefs = refs/changes\n\nThe changes necessary on the client side to allow fetching objects\nat the tip of a ref in hidden hierarchies are much more involved and\nnot part of this early preview, but the end user UI is expected to\nbe like these:\n\n\t$ git fetch $there refs/changes/72/41672/1\n\t$ git fetch $there 9598d59cdc098c5d9094d68024475e2430343182\n\nThat is, you ask for a refname as usual even though it is not part\nof ls-remote response, or you ask for the commit object that is at\nthe tip of whatever hidden ref you are interested in.\n\nJunio C Hamano (2):\n  upload-pack: share more code\n  upload-pack: allow hiding ref hiearchies\n\n t/t5512-ls-remote.sh |  9 ++++++\n upload-pack.c        | 86 ++++++++++++++++++++++++++++++++++++++++++----------\n 2 files changed, 79 insertions(+), 16 deletions(-)\n\n-- \n1.8.1.1.454.g48d39c0\n"},{"id":"207239","messageId":"1358555826-11883-2-git-send-email-gitster@pobox.com","threadId":"32664","inReplyTo":"1358555826-11883-1-git-send-email-gitster@pobox.com","subject":"[PATCH 1/2] upload-pack: share more code","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-19T00:37:05Z","receivedAt":"2013-01-19T00:37:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"We mark the objects pointed at our refs with \"OUR_REF\" flag in two\nfunctions (mark_our_ref() and send_ref()), but we can just use the\nformer as a helper for the latter.\n\nUpdate the way mark_our_ref() prepares in-core object to use\nlookup_unknown_object() to delay reading the actual object data,\njust like we did in 435c833 (upload-pack: use peel_ref for ref\nadvertisements, 2012-10-04).\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n upload-pack.c | 31 ++++++++++++++-----------------\n 1 file changed, 14 insertions(+), 17 deletions(-)\n\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 95d8313..3dd220d 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -722,15 +722,28 @@ static void receive_needs(void)\n \tfree(shallows.objects);\n }\n \n+static int mark_our_ref(const char *refname, const unsigned char *sha1, int flag, void *cb_data)\n+{\n+\tstruct object *o = lookup_unknown_object(sha1);\n+\tif (!o)\n+\t\tdie(\"git upload-pack: cannot find object %s:\", sha1_to_hex(sha1));\n+\tif (!(o->flags & OUR_REF)) {\n+\t\to->flags |= OUR_REF;\n+\t\tnr_our_refs++;\n+\t}\n+\treturn 0;\n+}\n+\n static int send_ref(const char *refname, const unsigned char *sha1, int flag, void *cb_data)\n {\n \tstatic const char *capabilities = \"multi_ack thin-pack side-band\"\n \t\t\" side-band-64k ofs-delta shallow no-progress\"\n \t\t\" include-tag multi_ack_detailed\";\n-\tstruct object *o = lookup_unknown_object(sha1);\n \tconst char *refname_nons = strip_namespace(refname);\n \tunsigned char peeled[20];\n \n+\tmark_our_ref(refname, sha1, flag, cb_data);\n+\n \tif (capabilities)\n \t\tpacket_write(1, \"%s %s%c%s%s agent=%s\\n\",\n \t\t\t     sha1_to_hex(sha1), refname_nons,\n@@ -740,27 +753,11 @@ static int send_ref(const char *refname, const unsigned char *sha1, int flag, vo\n \telse\n \t\tpacket_write(1, \"%s %s\\n\", sha1_to_hex(sha1), refname_nons);\n \tcapabilities = NULL;\n-\tif (!(o->flags & OUR_REF)) {\n-\t\to->flags |= OUR_REF;\n-\t\tnr_our_refs++;\n-\t}\n \tif (!peel_ref(refname, peeled))\n \t\tpacket_write(1, \"%s %s^{}\\n\", sha1_to_hex(peeled), refname_nons);\n \treturn 0;\n }\n \n-static int mark_our_ref(const char *refname, const unsigned char *sha1, int flag, void *cb_data)\n-{\n-\tstruct object *o = parse_object(sha1);\n-\tif (!o)\n-\t\tdie(\"git upload-pack: cannot find object %s:\", sha1_to_hex(sha1));\n-\tif (!(o->flags & OUR_REF)) {\n-\t\to->flags |= OUR_REF;\n-\t\tnr_our_refs++;\n-\t}\n-\treturn 0;\n-}\n-\n static void upload_pack(void)\n {\n \tif (advertise_refs || !stateless_rpc) {\n-- \n1.8.1.1.454.g48d39c0\n"},{"id":"207240","messageId":"1358555826-11883-3-git-send-email-gitster@pobox.com","threadId":"32664","inReplyTo":"1358555826-11883-1-git-send-email-gitster@pobox.com","subject":"[PATCH 2/2] upload-pack: allow hiding ref hiearchies","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-19T00:37:06Z","receivedAt":"2013-01-19T00:37:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Teach upload-pack to omit some refs from the initial advertisement\nby introducing the uploadPack.hiderefs multivalued configuration\nvariable.  Any ref that is under the hierarchies listed on the value\nof this variable is excluded from responses to \"ls-remote\", \"fetch\"\nor \"clone\" requests.  One typical use case may be\n\n\t[uploadPack]\n\t\thiderefs = refs/changes\n\nNote that the underlying protocol allows a request to fetch objects\nat the tip of any ref, including the hidden ones, but on the client\nside (e.g. fetch-pack), the requests are checked against the\nls-remote response, so it cannot ask for refs/changes/72/41672/1, or\nfor the object name (which is what eventually goes over the wire on\n\"want\" line) the user may obtain out of band (e.g. Gerrit\ndashboard).  A new capability \"allow-tip-sha1-in-want\" is returned\nby upload-pack to signal updated clients that it may be OK to ask\nfor objects that were not advertised.\n\nIf we want to allow fetching refname that is hidden, e.g.\n\n\t$ git fetch $there refs/changes/72/41672/1\n\nwe need to further update the server side to understand a message\nthat has the refname instead of object name on \"want\" line.  The\nnecessary change to the server side to support that is not in this\nstep.\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n t/t5512-ls-remote.sh |  9 ++++++++\n upload-pack.c        | 61 ++++++++++++++++++++++++++++++++++++++++++++++++++--\n 2 files changed, 68 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t5512-ls-remote.sh b/t/t5512-ls-remote.sh\nindex d16e5d3..9ee3d2a 100755\n--- a/t/t5512-ls-remote.sh\n+++ b/t/t5512-ls-remote.sh\n@@ -126,4 +126,13 @@ test_expect_success 'Report match with --exit-code' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'Hide some refs' '\n+\ttest_config uploadPack.hiderefs refs/tags &&\n+\tgit ls-remote . >actual &&\n+\ttest_unconfig uploadPack.hiderefs &&\n+\tgit ls-remote . |\n+\tsed -e \"/\trefs\\/tags\\//d\" >expect &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 3dd220d..54c304d 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -12,6 +12,7 @@\n #include \"run-command.h\"\n #include \"sigchain.h\"\n #include \"version.h\"\n+#include \"string-list.h\"\n \n static const char upload_pack_usage[] = \"git upload-pack [--strict] [--timeout=<n>] <dir>\";\n \n@@ -28,10 +29,13 @@ static const char upload_pack_usage[] = \"git upload-pack [--strict] [--timeout=<\n \n static unsigned long oldest_have;\n \n-static int multi_ack, nr_our_refs;\n+static int multi_ack;\n+static int nr_our_refs; /* This counts both advertised and unadvertised */\n static int no_done;\n static int use_thin_pack, use_ofs_delta, use_include_tag;\n static int no_progress, daemon_mode;\n+static struct string_list *hide_refs;\n+static int allow_tip_sha1_in_want;\n static int shallow_nr;\n static struct object_array have_obj;\n static struct object_array want_obj;\n@@ -722,6 +726,23 @@ static void receive_needs(void)\n \tfree(shallows.objects);\n }\n \n+static int ref_is_hidden(const char *refname)\n+{\n+\tstruct string_list_item *item;\n+\n+\tif (!hide_refs)\n+\t\treturn 0;\n+\tfor_each_string_list_item(item, hide_refs) {\n+\t\tint len;\n+\t\tif (prefixcmp(refname, item->string))\n+\t\t\tcontinue;\n+\t\tlen = strlen(item->string);\n+\t\tif (!refname[len] || refname[len] == '/')\n+\t\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n static int mark_our_ref(const char *refname, const unsigned char *sha1, int flag, void *cb_data)\n {\n \tstruct object *o = lookup_unknown_object(sha1);\n@@ -743,11 +764,14 @@ static int send_ref(const char *refname, const unsigned char *sha1, int flag, vo\n \tunsigned char peeled[20];\n \n \tmark_our_ref(refname, sha1, flag, cb_data);\n+\tif (allow_tip_sha1_in_want && ref_is_hidden(refname))\n+\t\treturn 0;\n \n \tif (capabilities)\n-\t\tpacket_write(1, \"%s %s%c%s%s agent=%s\\n\",\n+\t\tpacket_write(1, \"%s %s%c%s%s%s agent=%s\\n\",\n \t\t\t     sha1_to_hex(sha1), refname_nons,\n \t\t\t     0, capabilities,\n+\t\t\t     allow_tip_sha1_in_want ? \" allow-tip-sha1-in-want\" : \"\",\n \t\t\t     stateless_rpc ? \" no-done\" : \"\",\n \t\t\t     git_user_agent_sanitized());\n \telse\n@@ -758,11 +782,21 @@ static int send_ref(const char *refname, const unsigned char *sha1, int flag, vo\n \treturn 0;\n }\n \n+static int check_hidden_ref(const char *refname, const unsigned char *sha1, int flag, void *cb_data)\n+{\n+\tif (!ref_is_hidden(refname))\n+\t\treturn 0;\n+\tallow_tip_sha1_in_want = 1;\n+\treturn 1; /* terminate iteration over refs early */\n+}\n+\n static void upload_pack(void)\n {\n \tif (advertise_refs || !stateless_rpc) {\n \t\treset_timeout();\n \t\thead_ref_namespaced(send_ref, NULL);\n+\t\tif (hide_refs)\n+\t\t\tfor_each_namespaced_ref(check_hidden_ref, NULL);\n \t\tfor_each_namespaced_ref(send_ref, NULL);\n \t\tpacket_flush(1);\n \t} else {\n@@ -779,6 +813,28 @@ static void upload_pack(void)\n \t}\n }\n \n+static int upload_pack_config(const char *var, const char *value, void *unused)\n+{\n+\tif (!strcmp(\"uploadpack.hiderefs\", var)) {\n+\t\tchar *ref;\n+\t\tint len;\n+\n+\t\tif (!value)\n+\t\t\treturn config_error_nonbool(var);\n+\t\tref = xstrdup(value);\n+\t\tlen = strlen(ref);\n+\t\twhile (len && ref[len - 1] == '/')\n+\t\t\tref[--len] = '\\0';\n+\t\tif (!hide_refs) {\n+\t\t\thide_refs = xcalloc(1, sizeof(*hide_refs));\n+\t\t\thide_refs->strdup_strings = 1;\n+\t\t}\n+\t\tstring_list_append(hide_refs, ref);\n+\t\treturn 0;\n+\t}\n+\treturn 0;\n+}\n+\n int main(int argc, char **argv)\n {\n \tchar *dir;\n@@ -830,6 +886,7 @@ int main(int argc, char **argv)\n \t\tdie(\"'%s' does not appear to be a git repository\", dir);\n \tif (is_repository_shallow())\n \t\tdie(\"attempt to fetch/clone from a shallow repository\");\n+\tgit_config(upload_pack_config, NULL);\n \tif (getenv(\"GIT_DEBUG_SEND_PACK\"))\n \t\tdebug_fd = atoi(getenv(\"GIT_DEBUG_SEND_PACK\"));\n \tupload_pack();\n-- \n1.8.1.1.454.g48d39c0\n"},{"id":"207243","messageId":"CACsJy8C4qx0P621imj5B+HdoJkow0_jaGLVDRvdCDw3YRnK98g@mail.gmail.com","threadId":"32664","inReplyTo":"1358555826-11883-1-git-send-email-gitster@pobox.com","subject":"Re: [PATCH 0/2] Hiding some refs in ls-remote","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-01-19T05:50:10Z","receivedAt":"2013-01-19T05:50:10Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Jan 19, 2013 at 7:37 AM, Junio C Hamano <gitster@pobox.com> wrote:\n> This is an early preview of reducing the network cost while talking\n> with a repository with tons of refs, most of which are of use by\n> very narrow audiences (e.g. refs under Gerrit's refs/changes/ are\n> useful only for people who are interested in the changes under\n> review).  As long as these narrow audiences have a way to learn the\n> names of refs or objects pointed at by the refs out-of-band, it is\n> not necessary to advertise these refs.\n>\n> On the server end, you tell upload-pack that some refs do not have\n> to be advertised with the uploadPack.hiderefs multi-valued\n> configuration variable:\n>\n>         [uploadPack]\n>                 hiderefs = refs/changes\n>\n> The changes necessary on the client side to allow fetching objects\n> at the tip of a ref in hidden hierarchies are much more involved and\n> not part of this early preview, but the end user UI is expected to\n> be like these:\n>\n>         $ git fetch $there refs/changes/72/41672/1\n>         $ git fetch $there 9598d59cdc098c5d9094d68024475e2430343182\n>\n> That is, you ask for a refname as usual even though it is not part\n> of ls-remote response, or you ask for the commit object that is at\n> the tip of whatever hidden ref you are interested in.\n\nShould the client side learn how to list hidden refs too? I'm thinking\nof an extreme case where upload-pack advertises nothing (or maybe just\nrefs/heads/master) and it's up to the client to ask for the ref\nselection it's interested in. upload-pack may need more updates to do\nthat, I think.\n-- \nDuy\n"},{"id":"207244","messageId":"50FA3ACC.6070909@alum.mit.edu","threadId":"32664","inReplyTo":"1358555826-11883-1-git-send-email-gitster@pobox.com","subject":"Re: [PATCH 0/2] Hiding some refs in ls-remote","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2013-01-19T06:18:52Z","receivedAt":"2013-01-19T06:18:52Z","isPatch":true,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 01/19/2013 01:37 AM, Junio C Hamano wrote:\n> This is an early preview of reducing the network cost while talking\n> with a repository with tons of refs, most of which are of use by\n> very narrow audiences (e.g. refs under Gerrit's refs/changes/ are\n> useful only for people who are interested in the changes under\n> review).  As long as these narrow audiences have a way to learn the\n> names of refs or objects pointed at by the refs out-of-band, it is\n> not necessary to advertise these refs.\n> \n> On the server end, you tell upload-pack that some refs do not have\n> to be advertised with the uploadPack.hiderefs multi-valued\n> configuration variable:\n> \n> \t[uploadPack]\n> \t\thiderefs = refs/changes\n> \n> The changes necessary on the client side to allow fetching objects\n> at the tip of a ref in hidden hierarchies are much more involved and\n> not part of this early preview, but the end user UI is expected to\n> be like these:\n> \n> \t$ git fetch $there refs/changes/72/41672/1\n> \t$ git fetch $there 9598d59cdc098c5d9094d68024475e2430343182\n> \n> That is, you ask for a refname as usual even though it is not part\n> of ls-remote response, or you ask for the commit object that is at\n> the tip of whatever hidden ref you are interested in.\n\nAlthough I can understand the pain of slow network performance, somehow\nthis proposal gives me the feeling of being expeditious rather than elegant.\n\nCould the problem be solved in some other way?  Maybe such references\ncould be stored in a second repository or in a separate namespace (in\nthe sense of gitnamespaces(7)) to prevent their creating overhead when\nthey are unneeded?\n\nAnd *if* reference hiding makes sense, it seems to me that the client,\nnot the server, should be the one who decides which server references it\nis interested in (though I understand that would require a protocol\nchange).  Otherwise the git repository *relies* on out-of-band channels\nfor its functionality.  If I understand correctly, a user would have *no\nway* to discover, via git, what hidden references are contained in a\nremote repository, or indeed even that the repo contains a hidden\nnamespace.  For example this would make it impossible to clean up\nobsolete \"hidden\" references on a remote repository without the\nsupplementary information stored elsewhere.  And if anybody accidentally\ncreates a reference in a hidden namespace by hand, it will just sit\nthere undetectably, forever.\n\nI assume (though I've never checked) that a server does not let a client\nask for a SHA1 that is not currently reachable from a server-side\nreference, and I assume that that you are not proposing to change this\npolicy.  But allowing objects to be fetched from a hidden reference\nopens up some \"interesting\" possibilities:\n\n* A pusher could upload arbitrary content to a public git server under a\ncryptic hidden reference name.  Most people would be completely unable\nto see this content, unless given the SHA1 or the reference name by the\npusher.  Thus this mechanism could be used as a dark channel to exchange\narbitrary data relatively secretly.\n\n* Somebody could push a trojan version of code to a hidden reference in\na project, then pass the SHA1 to a victim.  The victim might trust the\ncode because it comes from a known project website, even though the code\nwould be invisible to other project developers and thus impossible for\nthem to audit.  And even if they learned about the trojan's SHA1 they\nwould be unable to remove it from their repository because they have no\nway to find out the name of the hidden reference!\n\nObviously these hacks would only be possible for a bad guy with push\nprivileges to a repository that has turned on hidden references, but I\nthink they are sobering nevertheless.\n\nThese worries would go away if reference hiding were configured on the\nclient rather than on the server.\n\nA second point: currently, the output of \"git show-ref -d\" and \"git\nls-remote .\" are almost identical.  Under your proposal, I believe that\nthe hiderefs would only be omitted from the latter.  Would it be useful\nto add an option to \"git show-ref\" to make it omit the \"hiderefs\" refs?\n And maybe another option to make it display *only* the hideref refs?\n\nAnd in the bikeshedding department, I wonder if \"hiderefs\" is the best\nname for the config setting.  \"hiderefs\", implies to me that the refs\nare actively hidden and not available to the client in any way.  But in\nfact they are just not advertised; they can be fetched normally.  Maybe\nanother name would be more suggestive of its true effect, for example\n\"quietrefs\" or \"noadvertiserefs\".\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\nhttp://softwareswirl.blogspot.com/\n"},{"id":"207269","messageId":"20130119165042.GB12307@sigill.intra.peff.net","threadId":"32664","inReplyTo":"1358555826-11883-1-git-send-email-gitster@pobox.com","subject":"Re: [PATCH 0/2] Hiding some refs in ls-remote","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-01-19T16:50:42Z","receivedAt":"2013-01-19T16:50:42Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 18, 2013 at 04:37:04PM -0800, Junio C Hamano wrote:\n\n> This is an early preview of reducing the network cost while talking\n> with a repository with tons of refs, most of which are of use by\n> very narrow audiences (e.g. refs under Gerrit's refs/changes/ are\n> useful only for people who are interested in the changes under\n> review).  As long as these narrow audiences have a way to learn the\n> names of refs or objects pointed at by the refs out-of-band, it is\n> not necessary to advertise these refs.\n> \n> On the server end, you tell upload-pack that some refs do not have\n> to be advertised with the uploadPack.hiderefs multi-valued\n> configuration variable:\n> \n> \t[uploadPack]\n> \t\thiderefs = refs/changes\n\nWould you want to do the same thing on receive-pack? It could benefit\nfrom the same reduction in network cost (although it tends to be invoked\nless frequently than upload-pack).\n\nAt GitHub, we have a similar patch (we even call it hiderefs), but we do\nit only for receive-pack. In our case, it is not about network traffic,\nbut rather that we provide a set of read-only refs in the refs/pull\nhierarchy. These are generated upstream by the creation of pull\nrequests, and we reject any updates to them via the git protocol using a\npre-receive hook.\n\nHowever, if a client without these refs uses \"git push --mirror\", it\nwill attempt to delete them (which will fail). Meaning that a mirror\npush will always report failure, because it will always fail to push the\nrefs/pull deletions.\n\nI don't know much about Gerrit's inner workings. Are refs/changes also\nread-only?\n\n-Peff\n"},{"id":"207270","messageId":"7v38xxnfv3.fsf@alter.siamese.dyndns.org","threadId":"32664","inReplyTo":"CACsJy8C4qx0P621imj5B+HdoJkow0_jaGLVDRvdCDw3YRnK98g@mail.gmail.com","subject":"Re: [PATCH 0/2] Hiding some refs in ls-remote","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-19T19:16:00Z","receivedAt":"2013-01-19T19:16:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen <pclouds@gmail.com> writes:\n\n> Should the client side learn how to list hidden refs too? I'm thinking\n> of an extreme case where upload-pack advertises nothing (or maybe just\n> refs/heads/master) and it's up to the client to ask for the ref\n> selection it's interested in. upload-pack may need more updates to do\n> that, I think.\n\nWhat you are talking about is a different goal.\n\nLook at this as a mechanism for the repository owner to control the\nclutter in what is shown to the intended audience of what s/he\npublishes in the repository.  Network bandwidth reduction of\nadvertisement is a side effect of clutter reduction, and not\nnecessarily the primary goal.\n\nIt lets me as the repository owner to say \"do not list them to make\nthese discoverable\"; letting the client side learn defeats the whole\npoint of this mechanism.\n\nFor example, if you mirror-clone from either of my repositories from\nGitHub:\n\n    git clone --mirror git://github.com/git/git/\n    git clone --mirror git://github.com/gitster/git/\n\nyou will see stuff that does not belong to the project; objects that\nare only reachable from refs/pull/* are not something I approved to\nbe placed in my repository; I as the project owner do not want to\ngive these objects to a cloner as if they are part of my project.\n\nThe server side can hide refs/pull/ and refs/pull-request-tags/ so\nthat clones (and ls-remote) will not see clutter, and nobody gets\nhurt.\n\nThese refs are there only to support GitHub \"pull requests\"; the\nadvertisement of them to ls-remote and appearance of them in mirror\nclones are undesirable side effects of how GitHub happened to\nimplement the feature.  GitHub has its way to notify of these pull\nrequests, and it makes these pull requests discoverable out of band\nwithout involving Git.\n\nAbility to say \"git fetch origin $SHA1\" (or \"git fetch origin\npull/1\") is the only thing lacking, in order to get rid of these\nclutter.  The patches do the \"get rid of clutter\" part by letting\nyou hide these refs from ls-remote and clone; allowing \"fetch by tip\nSHA1\" needs to come latter.\n\nAnother example.  If you run ls-remote against the above two\nrepositories, you will notice that the latter has tons more\nbranches.  The former are to publish only the primary integration\nbranches, while the latter are to show individual topics.\n\nI wish I didn't have to do this if I could.\n\nWe have periodical \"What's cooking\" postings that let interested\nparties learn topics out-of-band.  If I can hide refs/heads/??/*\nfrom normal users' clones while actually keeping individual topics\nthere at the same place, I do not have to push into two places.\n"},{"id":"207310","messageId":"7vpq0zn2za.fsf@alter.siamese.dyndns.org","threadId":"32664","inReplyTo":"20130119165042.GB12307@sigill.intra.peff.net","subject":"Re: [PATCH 0/2] Hiding some refs in ls-remote","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-20T18:06:33Z","receivedAt":"2013-01-20T18:06:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n>> \t[uploadPack]\n>> \t\thiderefs = refs/changes\n>\n> Would you want to do the same thing on receive-pack? It could benefit\n> from the same reduction in network cost (although it tends to be invoked\n> less frequently than upload-pack).\n\nDo *I* want to do?  Not when there already is a patch exists; I'd\nrather take existing and tested patch ;-)\n\nAs I said, I view this primarily as solving the cluttering issue.\nThe use of hiderefs to hide these refs that point at objects I do\nnot consider belong to my repository from the pushers equally makes\nsense as such, I think.\n"},{"id":"207311","messageId":"7vip6rn2ea.fsf@alter.siamese.dyndns.org","threadId":"32664","inReplyTo":"7v38xxnfv3.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 0/2] Hiding some refs in ls-remote","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-20T18:19:09Z","receivedAt":"2013-01-20T18:19:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> Duy Nguyen <pclouds@gmail.com> writes:\n>\n>> Should the client side learn how to list hidden refs too? I'm thinking\n>> of an extreme case where upload-pack advertises nothing (or maybe just\n>> refs/heads/master) and it's up to the client to ask for the ref\n>> selection it's interested in. upload-pack may need more updates to do\n>> that, I think.\n>\n> What you are talking about is a different goal.\n>\n> Look at this as a mechanism for the repository owner to control the\n> clutter in what is shown to the intended audience of what s/he\n> publishes in the repository.  Network bandwidth reduction of\n> advertisement is a side effect of clutter reduction, and not\n> necessarily the primary goal.\n\nAs a separate and follow-up topic, I can see a future where we also\nhave .delayref (in addition to .hideref) configuration, that causes\nthe upload-pack to:\n\n * omit refs that are marked as \"delayed\";\n\n * advertise \"allow-expand-ref=<value>\" where the value is the\n   pattern used to specify which refs are omitted via this mechanism\n   (e.g. \"refs/*\" in your extreme example, or perhaps \"refs/tags/\"\n   if you only advertise branches in order to limit the bandwidth);\n\nand then a fetch-pack that is interested in fetching \"refs/foo/bar\",\nafter seeing that it matches one of the \"allow-expand-ref\" patterns,\nto ask \"expand-ref refs/foo/bar\", expect the upload-pack to respond\nwith \"refs/foo/bar <value of that ref>\" (or \"refs/foo/bar does not\nexist\"), before going into the usual \"want\" exchange (\"ls-remote\"\nwould of course do the same, and pattern-less \"ls-remote\" may even\nask to expand everything by saying \"expand-ref refs/*\" (where the\ncapability said \"allow-expand-ref=refs/*\" in your extreme example)\nto grab everything discoverable).\n\nThat is _primarily_ for trading network bandwidth with initial\nlatency; as far as the end-result is concerned, delayed ones to be\nexpanded on demand are just as discoverable as the ones advertised\ninitially.\n\nI think we'd want to have both in the end.  It just happens I just\nposted the beginning of clutter-reduction one, as I think developing\nboth in parallel would be messy and hideref is probably less\ninvolved than the delayref.\n"},{"id":"207340","messageId":"7vip6rjyn3.fsf@alter.siamese.dyndns.org","threadId":"32664","inReplyTo":"7vpq0zn2za.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 0/2] Hiding some refs in ls-remote","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-20T22:08:32Z","receivedAt":"2013-01-20T22:08:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> Jeff King <peff@peff.net> writes:\n>\n>>> \t[uploadPack]\n>>> \t\thiderefs = refs/changes\n>>\n>> Would you want to do the same thing on receive-pack? It could benefit\n>> from the same reduction in network cost (although it tends to be invoked\n>> less frequently than upload-pack).\n> ...\n> As I said, I view this primarily as solving the cluttering issue.\n> The use of hiderefs to hide these refs that point at objects I do\n> not consider belong to my repository from the pushers equally makes\n> sense as such, I think.\n\nNow that raises one question.  Should this be transfer.hiderefs that\ngoverns both upload-pack and receive-pack?  I tend to think the\nanswer is yes.\n\nIt may even make sense to have \"git push\" honor it, excluding them\nfrom \"git push --mirror\" (or \"git push --all\" if some of the\nbranches are hidden); I haven't thought things through, though.\n"},{"id":"207355","messageId":"CACsJy8AunW_qWEimNhdgzw9xzgs4T+uBZm8o51Kfur0e9pv+HA@mail.gmail.com","threadId":"32664","inReplyTo":"7v38xxnfv3.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 0/2] Hiding some refs in ls-remote","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-01-21T01:46:22Z","receivedAt":"2013-01-21T01:46:22Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Jan 20, 2013 at 2:16 AM, Junio C Hamano <gitster@pobox.com> wrote:\n> Duy Nguyen <pclouds@gmail.com> writes:\n>\n>> Should the client side learn how to list hidden refs too? I'm thinking\n>> of an extreme case where upload-pack advertises nothing (or maybe just\n>> refs/heads/master) and it's up to the client to ask for the ref\n>> selection it's interested in. upload-pack may need more updates to do\n>> that, I think.\n>\n> What you are talking about is a different goal.\n>\n> Look at this as a mechanism for the repository owner to control the\n> clutter in what is shown to the intended audience of what s/he\n> publishes in the repository.  Network bandwidth reduction of\n> advertisement is a side effect of clutter reduction, and not\n> necessarily the primary goal.\n\nProbably stupid question: does gitnamespaces.txt attempt to achieve\nthe same? The document says smart http supports passing namespace,\nnothing about git protocol so I guess we need some extension in\nupload-pack (or git-daemon) for specifying namespace over git\nprotocol.\n-- \nDuy\n"},{"id":"207447","messageId":"20130121225626.GA17156@sigill.intra.peff.net","threadId":"32664","inReplyTo":"7v38xxnfv3.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 0/2] Hiding some refs in ls-remote","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-01-21T22:56:27Z","receivedAt":"2013-01-21T22:56:27Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Jan 19, 2013 at 11:16:00AM -0800, Junio C Hamano wrote:\n\n> For example, if you mirror-clone from either of my repositories from\n> GitHub:\n> \n>     git clone --mirror git://github.com/git/git/\n>     git clone --mirror git://github.com/gitster/git/\n> \n> you will see stuff that does not belong to the project; objects that\n> are only reachable from refs/pull/* are not something I approved to\n> be placed in my repository; I as the project owner do not want to\n> give these objects to a cloner as if they are part of my project.\n> \n> The server side can hide refs/pull/ and refs/pull-request-tags/ so\n> that clones (and ls-remote) will not see clutter, and nobody gets\n> hurt.\n\nI think it is unfortunately not so simple as that. You do not want them\nto be part of your \"clone --mirror\", but some people do (because they\nwill inspect them when evaluating the pull request). And that decision\nis, I think, in the eye of the cloner, not the eye of the repo owner. I\nthink in your case it is a little harder to see, in that you do not care\nabout the PR mechanism for git.git at all, but let us think for a minute\nabout a project that actually uses PRs.\n\nFor an ordinary developer, they would be content to fetch the branch\ntips and tags (and that is what we do by default with \"git clone). They\ndo not need to fetch all of refs/pull. If they learn out-of-band that PR\n123 exists, they can in theory ask for refs/pull/123/head. That works\neven with your proposal, because it is just about dropping the\nadvertisement, not the availability of refs.\n\nBut what about entities which really do want all of refs/pull?  I can\nthink of two good reasons to want them:\n\n  1. You really do want a mirror, because you are creating a backup or\n     alternate distribution channel. IOW, you are using \"git clone\n     --mirror\", but it is missing those refs.\n\n  2. You are a developer who is sometimes disconnected. You want to\n     fetch all of the PRs, and then examine them at your leisure.\n\nWithout advertising, how do they ask for the wildcard of `refs/pull/*`?\nThey're stuck massaging some out-of-band data into a set of distinct\nfetch refspecs.\n\nI don't know much about what's in Gerrit's refs/changes, but I imagine\none could make a similar argument that their value is in the eye of the\nclient. And later you give this example:\n\n> Another example.  If you run ls-remote against the above two\n> repositories, you will notice that the latter has tons more\n> branches.  The former are to publish only the primary integration\n> branches, while the latter are to show individual topics.\n> \n> I wish I didn't have to do this if I could.\n> \n> We have periodical \"What's cooking\" postings that let interested\n> parties learn topics out-of-band.  If I can hide refs/heads/??/*\n> from normal users' clones while actually keeping individual topics\n> there at the same place, I do not have to push into two places.\n\nMost people do not want to see those heads. But some people (like me)\ndo, and it is a great convenience to be able to fetch them all with a\nwildcard refspec, which cannot work if they are not advertised. I would\nhave to parse what's cooking and fetch them each individually. That's\nnot a technologically insurmountable problem, but it means git is being\na lot less helpful than it could be.\n\n\nSo I think in these cases of \"extra refs\", some clients would want them,\nand some would not. And only they can know which camp they are in, not\nthe server side. Which is why the current system works pretty well: we\nadvertise everything and let the client decide what it wants. Clone very\nsanely fetches only refs/heads/* (and associated tags), and you can put\nwhatever extra junk you want elsewhere in the hierarchy. A mirrored\nclone will fetch it, but to me that is the point of --mirror. And if you\nwant some arbitrary subset, then you can implement that, too, by the\nuse of fetch refspecs[1].\n\nThe downside, of course, is that the ref advertisement is bigger than\nmany clients will want. But dealing with that is a separate issue from\n\"these refs should never be shown\", as solutions for one may not work\nfrom the other (e.g., if we delayed ref advertisement until the client\nhad spoken, the client could tell us what refspecs they are interested\nin).\n\nFor your topic branch example, why don't you push to refs/topics of the\nmain git repository? Then normal cloners wouldn't see it, but anybody\ninterested could add the appropriate fetch refspec.\n\n-Peff\n\n[1] It may be that refspecs are not as expressive as we would like,\n    but that is a client side problem we can deal with. For instance,\n    you cannot currently say \"give me everything except refs/foo/*\".\n"},{"id":"207448","messageId":"20130121230108.GB17156@sigill.intra.peff.net","threadId":"32664","inReplyTo":"7vip6rjyn3.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 0/2] Hiding some refs in ls-remote","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-01-21T23:01:08Z","receivedAt":"2013-01-21T23:01:08Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Jan 20, 2013 at 02:08:32PM -0800, Junio C Hamano wrote:\n\n> Junio C Hamano <gitster@pobox.com> writes:\n> \n> > Jeff King <peff@peff.net> writes:\n> >\n> >>> \t[uploadPack]\n> >>> \t\thiderefs = refs/changes\n> >>\n> >> Would you want to do the same thing on receive-pack? It could benefit\n> >> from the same reduction in network cost (although it tends to be invoked\n> >> less frequently than upload-pack).\n> > ...\n> > As I said, I view this primarily as solving the cluttering issue.\n> > The use of hiderefs to hide these refs that point at objects I do\n> > not consider belong to my repository from the pushers equally makes\n> > sense as such, I think.\n> \n> Now that raises one question.  Should this be transfer.hiderefs that\n> governs both upload-pack and receive-pack?  I tend to think the\n> answer is yes.\n\nYes, that is what I was getting at (and it should have individual\nhiderefs for each side that override the transfer.*, in case somebody\nwants to make the distinction).\n\n> It may even make sense to have \"git push\" honor it, excluding them\n> from \"git push --mirror\" (or \"git push --all\" if some of the\n> branches are hidden); I haven't thought things through, though.\n\nThat is harder, as that is something that happens on the client. How\ndoes the client learn about the transfer.hiderefs setting on the remote?\nIt has to be out-of-band, at which point calling it by the same name has\nlittle benefit.\n\n-Peff\n"},{"id":"207449","messageId":"20130121230339.GC17156@sigill.intra.peff.net","threadId":"32664","inReplyTo":"7vpq0zn2za.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 0/2] Hiding some refs in ls-remote","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-01-21T23:03:39Z","receivedAt":"2013-01-21T23:03:39Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Jan 20, 2013 at 10:06:33AM -0800, Junio C Hamano wrote:\n\n> Jeff King <peff@peff.net> writes:\n> \n> >> \t[uploadPack]\n> >> \t\thiderefs = refs/changes\n> >\n> > Would you want to do the same thing on receive-pack? It could benefit\n> > from the same reduction in network cost (although it tends to be invoked\n> > less frequently than upload-pack).\n> \n> Do *I* want to do?  Not when there already is a patch exists; I'd\n> rather take existing and tested patch ;-)\n\nThe patch we have is below, but I do not think you want it, as it is\nmissing several things:\n\n  - docs and tests\n\n  - handling multiple values\n\n  - anything but raw prefix matching (you even have to put in the \"/\"\n    yourself).\n\nIt was not my patch originally, and I never bothered to clean and\nupstream it because I didn't think it was something anybody else would\nbe interested in. I'm happy to do so, but if you are working in the area\nanyway, it would make sense to be part of your series.\n\n-Peff\n\n---\ndiff --git b/builtin/receive-pack.c a/builtin/receive-pack.c\nindex 0afb8b2..b22670c 100644\n--- b/builtin/receive-pack.c\n+++ a/builtin/receive-pack.c\n@@ -39,6 +39,7 @@ static void *head_name_to_free;\n static int auto_gc = 1;\n static const char *head_name;\n static void *head_name_to_free;\n+static const char *hide_refs;\n static int sent_capabilities;\n \n static enum deny_action parse_deny_action(const char *var, const char *value)\n@@ -113,11 +114,19 @@ static void show_ref(const char *path, const unsigned char *sha1)\n \t\treturn 0;\n \t}\n \n+\tif (strcmp(var, \"receive.hiderefs\") == 0) {\n+\t\tgit_config_string(&hide_refs, var, value);\n+\t\treturn 0;\n+\t}\n+\n \treturn git_default_config(var, value, cb);\n }\n \n static void show_ref(const char *path, const unsigned char *sha1)\n {\n+\tif (hide_refs && strncmp(path, hide_refs, strlen(hide_refs)) == 0)\n+\t\treturn 0;\n+\n \tif (sent_capabilities)\n \t\tpacket_write(1, \"%s %s\\n\", sha1_to_hex(sha1), path);\n \telse\n"},{"id":"207454","messageId":"7v38xuf6w5.fsf@alter.siamese.dyndns.org","threadId":"32664","inReplyTo":"20130121230108.GB17156@sigill.intra.peff.net","subject":"Re: [PATCH 0/2] Hiding some refs in ls-remote","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-21T23:33:46Z","receivedAt":"2013-01-21T23:33:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n>> It may even make sense to have \"git push\" honor it, excluding them\n>> from \"git push --mirror\" (or \"git push --all\" if some of the\n>> branches are hidden); I haven't thought things through, though.\n>\n> That is harder, as that is something that happens on the client. How\n> does the client learn about the transfer.hiderefs setting on the remote?\n\nNo, I was talking about running \"git push\" from a repository that\nhas the transfer.hiderefs set, emulating a fetch from a client by\npushing out in the reverse direction.\n"},{"id":"207456","messageId":"20130121234549.GF17156@sigill.intra.peff.net","threadId":"32664","inReplyTo":"7v38xuf6w5.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 0/2] Hiding some refs in ls-remote","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-01-21T23:45:49Z","receivedAt":"2013-01-21T23:45:49Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jan 21, 2013 at 03:33:46PM -0800, Junio C Hamano wrote:\n\n> Jeff King <peff@peff.net> writes:\n> \n> >> It may even make sense to have \"git push\" honor it, excluding them\n> >> from \"git push --mirror\" (or \"git push --all\" if some of the\n> >> branches are hidden); I haven't thought things through, though.\n> >\n> > That is harder, as that is something that happens on the client. How\n> > does the client learn about the transfer.hiderefs setting on the remote?\n> \n> No, I was talking about running \"git push\" from a repository that\n> has the transfer.hiderefs set, emulating a fetch from a client by\n> pushing out in the reverse direction.\n\nAh. That seems a bit more questionable to me, as you are assuming that\nthe face that the repository shows to network clients is the same as it\nwould show when it is the client itself. That wouldn't be true, for\nexample, when pushing to a backup repository which would expect to get\neverything.\n\nOf course the same problem comes from a backup repository which wants to\nfetch from you. Which again comes down to the fact that I think this\nref-hiding is really in the eye of the receiver, not the sender.\n\n-Peff\n"}]}