{"thread":{"id":"46240","subject":"[RFC PATCH] proposal for refs/tracking/* hierarchy","startedAt":"2017-06-23T13:52:50Z","lastAt":"2017-06-26T19:28:45Z","messageCount":6,"participants":["Jacob Keller","Junio C Hamano","Marc Branchaud"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"323094","messageId":"20170623135230.10534-1-jacob.e.keller@intel.com","threadId":"46240","inReplyTo":null,"subject":"[RFC PATCH] proposal for refs/tracking/* hierarchy","fromName":"Jacob Keller","fromEmail":"jacob.e.keller@intel.com","sentAt":"2017-06-23T13:52:30Z","receivedAt":"2017-06-23T13:52:50Z","isPatch":true,"sender":{"key":"jacob.e.keller@intel.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"From: Jacob Keller <jacob.keller@gmail.com>\n\nHistorically, git has tracked the status of remote branches (heads) in\nrefs/remotes/<name>/*. This is necessary and useful as it allows users\nto track the difference between their local work and the last known\nstatus of the remote work.\n\nUnfortunately this hierarchy is limited in that it only tracks branches.\nAdditionally, it is not feasible to extend it directly, because the\nlayout would become ambiguous. For example, if a user happened to have\na local branch named \"notes\" it could be confusing if you tried to add\n\"refs/remotes/origin/notes/<remote notes refs\" as this would collide\nwith the already existing refs/remotes/origin/notes branch that existed.\n\nInstead, lets add support for a new refs/tracking/* hierarchy which is\nlaid out in such a way to avoid this inconsistency. All refs in\n\"refs/tracking/<remote>/*\" will include the complete ref, such that\ndropping the \"tracking/<remote>\" part will give the exact ref name as it\nis found in the upstream. Thus, we can track any ref here by simply\nfetching it into refs/tracking/<remote>/*.\n\nAdd support to tell a new remote to start tracking remote hierarchies\nvia \"--follow\" which will track all refs under that section, ie:\n\ngit remote add --follow notes origin <url> will cause\n\n+refs/notes/*:refs/tracking/origin/notes/* to be added as a fetch\nrefspec for this remote.\n\nThis ensures that any future refs which want a method of sharing the\ncurrent remote status separate from local status could use\nrefs/tracking\n\nA long term goal might be to deprecate refs/remotes/ in favor of\nrefs/tracking (possibly adding in magic to convert refs/remotes directly\ninto refs/tracking so that old stuff still works?).\n\nThings not yet thought through:\n\n1) maybe we should create a configurable default so that some known set\n   of default refs get pulled (ie: notes, grafts, replace, etc?)\n   Possibly with some sort of easy way to add or subtract from the list\n   in config...\n\n2) so far, there's no work done to figure out how to remove\n   refs/tracking when a remote is dropped. I *think* we can just delete\n   all refs under refs/tracking/<name> but I'm not completely certain\n\n3) adding magic to complete refs under tracking, such as for git-notes\n   or similar may wish to understand shorthand for referencing the\n   remote version of notes\n\n4) adding support for clone.. (this is weird because git-clone and\n   git-remote don't both use the same flow for setup.. oops??)\n\n5) tests, documentation etc. (This is primarily an RFC, with the goal of\n   providing a known location for remote references such as notes to\n   reside)\n\n6) we probably want to enable notes and grafts and other similar things\n   to use the remote tracking data if its available.\n\n7) what about tags? Currently we fetch all tags into refs/tags directly,\n   which is a bit awkward, if for example you create a local tag and\n   a remote creates a tag with the same name, you simply don't get that\n   new version. This is good, but now you have no idea how to tell if\n   your tags are out of date or not. refs/tracking can partially resolve\n   this since remote tags will always be \"exactly\" what the remote has.\n   Unfortunately, I don't know how we'd resolve them into local tags...\n\nProbably other things missing too...\n\nSigned-off-by: Jacob Keller <jacob.keller@gmail.com>\n---\n builtin/remote.c | 18 ++++++++++++++++++\n 1 file changed, 18 insertions(+)\n\ndiff --git a/builtin/remote.c b/builtin/remote.c\nindex f1a88fe26589..dffe3892be11 100644\n--- a/builtin/remote.c\n+++ b/builtin/remote.c\n@@ -122,6 +122,16 @@ static void add_branch(const char *key, const char *branchname,\n \tgit_config_set_multivar(key, tmp->buf, \"^$\", 0);\n }\n \n+static void add_tracking(const char *key, const char *namespace,\n+\t\t\t const char *remotename, struct strbuf *tmp)\n+{\n+\tstrbuf_reset(tmp);\n+\tstrbuf_addch(tmp, '+');\n+\tstrbuf_addf(tmp, \"refs/%s/*:refs/tracking/%s/%s/*\",\n+\t\t    namespace, remotename, namespace);\n+\tgit_config_set_multivar(key, tmp->buf, \"^$\", 0);\n+}\n+\n static const char mirror_advice[] =\n N_(\"--mirror is dangerous and deprecated; please\\n\"\n    \"\\t use --mirror=fetch or --mirror=push instead\");\n@@ -149,6 +159,7 @@ static int add(int argc, const char **argv)\n \tint fetch = 0, fetch_tags = TAGS_DEFAULT;\n \tunsigned mirror = MIRROR_NONE;\n \tstruct string_list track = STRING_LIST_INIT_NODUP;\n+\tstruct string_list follow = STRING_LIST_INIT_NODUP;\n \tconst char *master = NULL;\n \tstruct remote *remote;\n \tstruct strbuf buf = STRBUF_INIT, buf2 = STRBUF_INIT;\n@@ -164,6 +175,8 @@ static int add(int argc, const char **argv)\n \t\t\t    N_(\"or do not fetch any tag at all (--no-tags)\"), TAGS_UNSET),\n \t\tOPT_STRING_LIST('t', \"track\", &track, N_(\"branch\"),\n \t\t\t\tN_(\"branch(es) to track\")),\n+\t\tOPT_STRING_LIST(0, \"follow\", &follow, N_(\"namespace\"),\n+\t\t\t\tN_(\"refs namespaces to follow in refs/tracking\")),\n \t\tOPT_STRING('m', \"master\", &master, N_(\"branch\"), N_(\"master branch\")),\n \t\t{ OPTION_CALLBACK, 0, \"mirror\", &mirror, N_(\"push|fetch\"),\n \t\t\tN_(\"set up remote as a mirror to push to or fetch from\"),\n@@ -207,6 +220,11 @@ static int add(int argc, const char **argv)\n \t\t}\n \t}\n \n+\tfor (i = 0; i < follow.nr; i++) {\n+\t\tadd_tracking(buf.buf, follow.items[i].string,\n+\t\t\t     name, &buf2);\n+\t}\n+\n \tif (mirror & MIRROR_PUSH) {\n \t\tstrbuf_reset(&buf);\n \t\tstrbuf_addf(&buf, \"remote.%s.mirror\", name);\n-- \n2.13.0.615.gb09ed6e59a40\n\n"},{"id":"323095","messageId":"CA+P7+xov17vsoVZSvSim0psX72NJ+arsc_5LPSuxpqXHEg4egg@mail.gmail.com","threadId":"46240","inReplyTo":"20170623135230.10534-1-jacob.e.keller@intel.com","subject":"Re: [RFC PATCH] proposal for refs/tracking/* hierarchy","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2017-06-23T13:55:46Z","receivedAt":"2017-06-23T13:56:20Z","isPatch":true,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Fri, Jun 23, 2017 at 6:52 AM, Jacob Keller <jacob.e.keller@intel.com> wrote:\n> From: Jacob Keller <jacob.keller@gmail.com>\n>\n> Historically, git has tracked the status of remote branches (heads) in\n> refs/remotes/<name>/*. This is necessary and useful as it allows users\n> to track the difference between their local work and the last known\n> status of the remote work.\n>\n> Unfortunately this hierarchy is limited in that it only tracks branches.\n> Additionally, it is not feasible to extend it directly, because the\n> layout would become ambiguous. For example, if a user happened to have\n> a local branch named \"notes\" it could be confusing if you tried to add\n> \"refs/remotes/origin/notes/<remote notes refs\" as this would collide\n> with the already existing refs/remotes/origin/notes branch that existed.\n>\n> Instead, lets add support for a new refs/tracking/* hierarchy which is\n> laid out in such a way to avoid this inconsistency. All refs in\n> \"refs/tracking/<remote>/*\" will include the complete ref, such that\n> dropping the \"tracking/<remote>\" part will give the exact ref name as it\n> is found in the upstream. Thus, we can track any ref here by simply\n> fetching it into refs/tracking/<remote>/*.\n>\n> Add support to tell a new remote to start tracking remote hierarchies\n> via \"--follow\" which will track all refs under that section, ie:\n>\n> git remote add --follow notes origin <url> will cause\n>\n> +refs/notes/*:refs/tracking/origin/notes/* to be added as a fetch\n> refspec for this remote.\n>\n> This ensures that any future refs which want a method of sharing the\n> current remote status separate from local status could use\n> refs/tracking\n>\n> A long term goal might be to deprecate refs/remotes/ in favor of\n> refs/tracking (possibly adding in magic to convert refs/remotes directly\n> into refs/tracking so that old stuff still works?).\n>\n> Things not yet thought through:\n>\n> 1) maybe we should create a configurable default so that some known set\n>    of default refs get pulled (ie: notes, grafts, replace, etc?)\n>    Possibly with some sort of easy way to add or subtract from the list\n>    in config...\n>\n> 2) so far, there's no work done to figure out how to remove\n>    refs/tracking when a remote is dropped. I *think* we can just delete\n>    all refs under refs/tracking/<name> but I'm not completely certain\n>\n> 3) adding magic to complete refs under tracking, such as for git-notes\n>    or similar may wish to understand shorthand for referencing the\n>    remote version of notes\n>\n> 4) adding support for clone.. (this is weird because git-clone and\n>    git-remote don't both use the same flow for setup.. oops??)\n>\n> 5) tests, documentation etc. (This is primarily an RFC, with the goal of\n>    providing a known location for remote references such as notes to\n>    reside)\n>\n> 6) we probably want to enable notes and grafts and other similar things\n>    to use the remote tracking data if its available.\n>\n> 7) what about tags? Currently we fetch all tags into refs/tags directly,\n>    which is a bit awkward, if for example you create a local tag and\n>    a remote creates a tag with the same name, you simply don't get that\n>    new version. This is good, but now you have no idea how to tell if\n>    your tags are out of date or not. refs/tracking can partially resolve\n>    this since remote tags will always be \"exactly\" what the remote has.\n>    Unfortunately, I don't know how we'd resolve them into local tags...\n>\n\nOops:\n\n8) if we decided to go with \"all refs always get tracked in\nrefs/tracking\" we probably want to either add a way to say \"all refs\nexcept refs/tracking ones\" or we want a way for servers to (by\ndefault) not advertise refs/tracking when clients fetch from them.\nThat's partially why I went with the easier \"only some hierarchies\nwill get pulled by default\" Otherwise, two remotes that fetch from\neach other could create a never ending cycle of\nrefs/tracking/origin/tracking/origin/tracking/origin/ adding a new\nlayer every time they fetched.\n\nThanks,\nJake\n"},{"id":"323154","messageId":"xmqqvanmctmb.fsf@gitster.mtv.corp.google.com","threadId":"46240","inReplyTo":"20170623135230.10534-1-jacob.e.keller@intel.com","subject":"Re: [RFC PATCH] proposal for refs/tracking/* hierarchy","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-06-23T20:54:36Z","receivedAt":"2017-06-23T20:54:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jacob Keller <jacob.e.keller@intel.com> writes:\n\n> Instead, lets add support for a new refs/tracking/* hierarchy which is\n> laid out in such a way to avoid this inconsistency. All refs in\n> \"refs/tracking/<remote>/*\" will include the complete ref, such that\n> dropping the \"tracking/<remote>\" part will give the exact ref name as it\n> is found in the upstream. Thus, we can track any ref here by simply\n> fetching it into refs/tracking/<remote>/*.\n\nI do think this overall is a good goal to aim for wrt \"tracking\",\ni.e.  keeping a pristine copy of what we observed from the outside\nworld.  In addition to what you listed as \"undecided\" below,\nhowever, I expect that this will highlight a lot of trouble in\n\"working together\", i.e. reconciling the differences of various\nworld views and moving the project forward, in two orthogonal axes:\n\n - Things pointed at by some refs (e.g. notes/, but I think the\n   \".gitmodules equivalent that is not tied to any particular commit\n   in the superproject\" Jonathan Nieder and friends advocate falls\n   into the same category) do not correspond to the working tree\n   contents, and merging their contents will always pose challenge\n   when we need to prepare for conflict resolution.\n\n - Things pointed at by some other refs (e.g. tags/) do not make\n   sense to be merged.  You may locally call a commit with a tag\n   \"wip\", while your friends may use the same \"wip\" tag to point at\n   a different one.  Both are valid, and more importantly, there is\n   no point even trying to reconcile what the \"wip\" tag means in the\n   project globally.\n\nFor the former, I expect that merging these non-working tree\ncontents will all have to require some specific tool that is\ntailored for the meaning each hierarchy has, just like \"git notes\nmerge\" gives a solution that is very specific to the notes refs (I\ndo not know how well \"notes merge\" works in practice, though).\n\nFor the latter, having a separate tracking hierarchy is a strict\nimprovement from \"tracking\" point of view, but I think \"working\ntogether\" also needs a well designed set of new look-up rules, and a\nnew convention.  For example, some tags may not want to be shared\n(e.g. \"wip\" example above) even though they should be easy to access\nby those who already have them (e.g. \"git log ..wip\" should work\nwithout having to say \"git log ..refs/local-tags/wip\", even if we\ndecide to implement the \"some tags may not want to be shared\"\nsegregation by introducing a new hierarchy).  And also a shared tag\nthat is copied to \"refs/tracking/origin/tags/v1.0\" would need a way\nbetter than \"git log tracking/origin/tags/v1.0\" for this mechanism\nto be useful in everyday workflow.\n\nThanks.\n\n"},{"id":"323157","messageId":"CA+P7+xoohKiCvZWQmGmEzWahEevg2KEuFRiM0mmYN6vtnrrnkA@mail.gmail.com","threadId":"46240","inReplyTo":"xmqqvanmctmb.fsf@gitster.mtv.corp.google.com","subject":"Re: [RFC PATCH] proposal for refs/tracking/* hierarchy","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2017-06-23T21:30:19Z","receivedAt":"2017-06-23T21:30:48Z","isPatch":true,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Fri, Jun 23, 2017 at 1:54 PM, Junio C Hamano <gitster@pobox.com> wrote:\n> Jacob Keller <jacob.e.keller@intel.com> writes:\n>\n>> Instead, lets add support for a new refs/tracking/* hierarchy which is\n>> laid out in such a way to avoid this inconsistency. All refs in\n>> \"refs/tracking/<remote>/*\" will include the complete ref, such that\n>> dropping the \"tracking/<remote>\" part will give the exact ref name as it\n>> is found in the upstream. Thus, we can track any ref here by simply\n>> fetching it into refs/tracking/<remote>/*.\n>\n> I do think this overall is a good goal to aim for wrt \"tracking\",\n> i.e.  keeping a pristine copy of what we observed from the outside\n> world.  In addition to what you listed as \"undecided\" below,\n> however, I expect that this will highlight a lot of trouble in\n> \"working together\", i.e. reconciling the differences of various\n> world views and moving the project forward, in two orthogonal axes:\n>\n\nI agree, I think this is definitely a problem that we'd want/have to solve.\n\n>  - Things pointed at by some refs (e.g. notes/, but I think the\n>    \".gitmodules equivalent that is not tied to any particular commit\n>    in the superproject\" Jonathan Nieder and friends advocate falls\n>    into the same category) do not correspond to the working tree\n>    contents, and merging their contents will always pose challenge\n>    when we need to prepare for conflict resolution.\n\nThis is part of why I started by looking at notes, since there is\nalready some code for notes merge, it just doesn't have a standard\nplace to store \"remote\" notes. I've wanted that ability to enable more\nease of sharing.\n\n>\n>  - Things pointed at by some other refs (e.g. tags/) do not make\n>    sense to be merged.  You may locally call a commit with a tag\n>    \"wip\", while your friends may use the same \"wip\" tag to point at\n>    a different one.  Both are valid, and more importantly, there is\n>    no point even trying to reconcile what the \"wip\" tag means in the\n>    project globally.\n>\n> For the former, I expect that merging these non-working tree\n> contents will all have to require some specific tool that is\n> tailored for the meaning each hierarchy has, just like \"git notes\n> merge\" gives a solution that is very specific to the notes refs (I\n> do not know how well \"notes merge\" works in practice, though).\n\nAgreed. That's also part of why I'm looking at doing it as a partial\n\"one hierarchy at a time\" proposal, I've got a somewhat better format\nfor that coming Soon(TM)\n\n>\n> For the latter, having a separate tracking hierarchy is a strict\n> improvement from \"tracking\" point of view, but I think \"working\n> together\" also needs a well designed set of new look-up rules, and a\n> new convention.  For example, some tags may not want to be shared\n> (e.g. \"wip\" example above) even though they should be easy to access\n> by those who already have them (e.g. \"git log ..wip\" should work\n> without having to say \"git log ..refs/local-tags/wip\", even if we\n> decide to implement the \"some tags may not want to be shared\"\n> segregation by introducing a new hierarchy).  And also a shared tag\n> that is copied to \"refs/tracking/origin/tags/v1.0\" would need a way\n> better than \"git log tracking/origin/tags/v1.0\" for this mechanism\n> to be useful in everyday workflow.\n>\n> Thanks.\n>\n\nMy thoughts on tags are sort of the following:\n\nkeep the default semantics of tags get copied into our local\nrefs/tags, (if we're fetching them). However, create some mechanism\nfor easily showing when you have a local tag with the same name as a\nremote tag which don't point at the same thing, by using the\nrefs/tracking. Possibly, any time you grab a tag, check and see if the\nremote version is different and \"do something\" like complain, or show\nboth, or give some advice (possibly with a new tool to \"pull in the\nremote tag if you verify it's accurate, etc\".\n\nThis way, you generally work with tags the same way but get added\nprotection for knowing when your local tags differ from what remotes\nshow?\n\nThanks,\nJake\n"},{"id":"323284","messageId":"aa8a7ff6-2068-645d-061e-3142760a4b2a@xiplink.com","threadId":"46240","inReplyTo":"xmqqvanmctmb.fsf@gitster.mtv.corp.google.com","subject":"Re: [RFC PATCH] proposal for refs/tracking/* hierarchy","fromName":"Marc Branchaud","fromEmail":"marcnarc@xiplink.com","sentAt":"2017-06-26T18:06:15Z","receivedAt":"2017-06-26T18:06:26Z","isPatch":true,"sender":{"key":"marcnarc@xiplink.com","avatar":"https://avatars.githubusercontent.com/u/14980203?v=4"},"body":"On 2017-06-23 04:54 PM, Junio C Hamano wrote:\n> Jacob Keller <jacob.e.keller@intel.com> writes:\n> \n>> Instead, lets add support for a new refs/tracking/* hierarchy which is\n>> laid out in such a way to avoid this inconsistency. All refs in\n>> \"refs/tracking/<remote>/*\" will include the complete ref, such that\n>> dropping the \"tracking/<remote>\" part will give the exact ref name as it\n>> is found in the upstream. Thus, we can track any ref here by simply\n>> fetching it into refs/tracking/<remote>/*.\n> \n> I do think this overall is a good goal to aim for wrt \"tracking\",\n> i.e.  keeping a pristine copy of what we observed from the outside\n> world.  In addition to what you listed as \"undecided\" below,\n> however, I expect that this will highlight a lot of trouble in\n> \"working together\", i.e. reconciling the differences of various\n> world views and moving the project forward, in two orthogonal axes:\n> \n>   - Things pointed at by some refs (e.g. notes/, but I think the\n>     \".gitmodules equivalent that is not tied to any particular commit\n>     in the superproject\" Jonathan Nieder and friends advocate falls\n>     into the same category) do not correspond to the working tree\n>     contents, and merging their contents will always pose challenge\n>     when we need to prepare for conflict resolution.\n\nOTOH, we shouldn't need to do any conflict resolution on these \n\"tracking\" refs:  The remote side should update the refs properly. \nNobody should make local changes to these refs.\n\nI guess I'm advocating that git should only allow \"tracking\" refs to be \nupdated by a fetch, or a successful push of a local, non-\"tracking\" ref.\n\n\t\tM.\n\n\n>   - Things pointed at by some other refs (e.g. tags/) do not make\n>     sense to be merged.  You may locally call a commit with a tag\n>     \"wip\", while your friends may use the same \"wip\" tag to point at\n>     a different one.  Both are valid, and more importantly, there is\n>     no point even trying to reconcile what the \"wip\" tag means in the\n>     project globally.\n> \n> For the former, I expect that merging these non-working tree\n> contents will all have to require some specific tool that is\n> tailored for the meaning each hierarchy has, just like \"git notes\n> merge\" gives a solution that is very specific to the notes refs (I\n> do not know how well \"notes merge\" works in practice, though).\n> \n> For the latter, having a separate tracking hierarchy is a strict\n> improvement from \"tracking\" point of view, but I think \"working\n> together\" also needs a well designed set of new look-up rules, and a\n> new convention.  For example, some tags may not want to be shared\n> (e.g. \"wip\" example above) even though they should be easy to access\n> by those who already have them (e.g. \"git log ..wip\" should work\n> without having to say \"git log ..refs/local-tags/wip\", even if we\n> decide to implement the \"some tags may not want to be shared\"\n> segregation by introducing a new hierarchy).  And also a shared tag\n> that is copied to \"refs/tracking/origin/tags/v1.0\" would need a way\n> better than \"git log tracking/origin/tags/v1.0\" for this mechanism\n> to be useful in everyday workflow.\n> \n> Thanks.\n> \n"},{"id":"323290","messageId":"xmqqvani4kgs.fsf@gitster.mtv.corp.google.com","threadId":"46240","inReplyTo":"aa8a7ff6-2068-645d-061e-3142760a4b2a@xiplink.com","subject":"Re: [RFC PATCH] proposal for refs/tracking/* hierarchy","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-06-26T19:28:35Z","receivedAt":"2017-06-26T19:28:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Marc Branchaud <marcnarc@xiplink.com> writes:\n\n> OTOH, we shouldn't need to do any conflict resolution on these\n> \"tracking\" refs:  The remote side should update the refs\n> properly. Nobody should make local changes to these refs.\n>\n> I guess I'm advocating that git should only allow \"tracking\" refs to\n> be updated by a fetch, or a successful push of a local, non-\"tracking\"\n> ref.\n\nI do not think anybody is imagining new things to happen inside\nrefs/tracking other than by fetching, just like refs/remotes/ refs\nbehave like so.\n\nWhat I was discussing was mostly the step next to the introduction\nof tracking/.  Some things are useful merely by existing (e.g. tags\nyou got from remote, as long as you can easily refer to them, are\nuseful to you).  Some other things are cumbersome to use until you\nmanage to consolidate your views with theirs in order to make\ncollective progress, and they require merging (e.g. notes).\n"}]}