{"thread":{"id":"42142","subject":"[RFC/PATCH] Ordering of remotes for fetch --all","startedAt":"2016-04-25T21:15:05Z","lastAt":"2016-04-28T15:24:20Z","messageCount":4,"participants":["Guido Martínez","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"284483","messageId":"20160425211506.8421-1-mtzguido@gmail.com","threadId":"42142","inReplyTo":null,"subject":"[RFC/PATCH] Ordering of remotes for fetch --all","fromName":"Guido Martínez","fromEmail":"mtzguido@gmail.com","sentAt":"2016-04-25T21:15:05Z","receivedAt":"2016-04-25T21:15:05Z","isPatch":true,"sender":{"key":"mtzguido@gmail.com","avatar":null},"body":"Hi all,\n\nI run a server with several git mirrors, that are updated every hour. On\nthat same server, users clone those projects and work on them. We use\nthe local mirrors to reduce network load: the users can fetch from the\nmirror first (to get most of the objects with zero network cost) and\nthen fetch the real remote (to make sure they're completely up to date).\n\nI would like this to be configurable in each git working directory,\nso users can just configure the order they want and then just do \"git\nremote update\".\n\nI'm aware one can get this behavior by editing .git/config and\nordering the remotes as one wishes, but I find that very hacky and not\nscripting-friendly.\n\nThis patch introduces a fetch priority for each remote, at a default of\n50 and modifiable via git config. This new order will only matter when\ndoing fetch --all.\n\nDo you think this is a useful feature? Hopefully you don't consider this\nas just noise :)\n\n(As a side note: for ordering the remotes a stable sort would be best,\nto have the least impact possible on current behavior. I believe\ngit_qsort is stable but a confirmation would be nice.)\n\nThanks!\nGuido\n\nGuido Martínez (1):\n  remote: add a fetching priority to each remote\n\n Documentation/config.txt |  5 +++++\n builtin/fetch.c          |  2 +-\n remote.c                 | 43 +++++++++++++++++++++++++++++++++++++++----\n remote.h                 |  2 ++\n 4 files changed, 47 insertions(+), 5 deletions(-)\n\n-- \n2.8.1.281.g0994585\n"},{"id":"284484","messageId":"20160425211506.8421-2-mtzguido@gmail.com","threadId":"42142","inReplyTo":"20160425211506.8421-1-mtzguido@gmail.com","subject":"[RFC/PATCH] remote: add a fetching priority to each remote","fromName":"Guido Martínez","fromEmail":"mtzguido@gmail.com","sentAt":"2016-04-25T21:15:06Z","receivedAt":"2016-04-25T21:15:06Z","isPatch":true,"sender":{"key":"mtzguido@gmail.com","avatar":null},"body":"Add an int that allows for a way of setting a fetch order for remotes,\nmainly for the use case of \"git remote update\" which updates every\nremote.\n\nThis way, users can set local mirrors of repositories first to quickly\nget a bunch of objects, and later the upstream repo to make sure that\nthey pulled the latest commit.\n\nSigned-off-by: Guido Martínez <mtzguido@gmail.com>\n---\n Documentation/config.txt |  5 +++++\n builtin/fetch.c          |  2 +-\n remote.c                 | 43 +++++++++++++++++++++++++++++++++++++++----\n remote.h                 |  2 ++\n 4 files changed, 47 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 42d2b50..5ca199c 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2489,6 +2489,11 @@ remote.<name>.fetch::\n \tThe default set of \"refspec\" for linkgit:git-fetch[1]. See\n \tlinkgit:git-fetch[1].\n \n+remote.<name>.fetchprio::\n+\tSet a priority for fetching this remote, to allow you to set\n+\ta custom order when doing \"git fetch --all\" (thus also when\n+\trunning \"git remote update\"). Default value is 50.\n+\n remote.<name>.push::\n \tThe default set of \"refspec\" for linkgit:git-push[1]. See\n \tlinkgit:git-push[1].\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex f8455bd..44f42bf 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1187,7 +1187,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t\t\tdie(_(\"fetch --all does not take a repository argument\"));\n \t\telse if (argc > 1)\n \t\t\tdie(_(\"fetch --all does not make sense with refspecs\"));\n-\t\t(void) for_each_remote(get_one_remote_for_fetch, &list);\n+\t\t(void) for_each_sorted_remote(get_one_remote_for_fetch, &list);\n \t\tresult = fetch_multiple(&list);\n \t} else if (argc == 0) {\n \t\t/* No arguments -- use default remote */\ndiff --git a/remote.c b/remote.c\nindex 28fd676..3b9f20a 100644\n--- a/remote.c\n+++ b/remote.c\n@@ -168,6 +168,7 @@ static struct remote *make_remote(const char *name, int len)\n \tALLOC_GROW(remotes, remotes_nr + 1, remotes_alloc);\n \tremotes[remotes_nr++] = ret;\n \tret->name = xstrndup(name, len);\n+\tret->fetch_prio = 50;\n \n \thashmap_entry_init(ret, lookup_entry.hash);\n \treplaced = hashmap_put(&remotes_hash, ret);\n@@ -375,6 +376,8 @@ static int handle_config(const char *key, const char *value, void *cb)\n \t\tremote->mirror = git_config_bool(key, value);\n \telse if (!strcmp(subkey, \"skipdefaultupdate\"))\n \t\tremote->skip_default_update = git_config_bool(key, value);\n+\telse if (!strcmp(subkey, \"fetchprio\"))\n+\t\tremote->fetch_prio = git_config_int(key, value);\n \telse if (!strcmp(subkey, \"skipfetchall\"))\n \t\tremote->skip_default_update = git_config_bool(key, value);\n \telse if (!strcmp(subkey, \"prune\"))\n@@ -719,12 +722,12 @@ int remote_is_configured(struct remote *remote)\n \treturn remote && remote->origin;\n }\n \n-int for_each_remote(each_remote_fn fn, void *priv)\n+static int for_each_remote_do(struct remote **rlist, int len,\n+\t\t\t      each_remote_fn fn, void *priv)\n {\n \tint i, result = 0;\n-\tread_config();\n-\tfor (i = 0; i < remotes_nr && !result; i++) {\n-\t\tstruct remote *r = remotes[i];\n+\tfor (i = 0; i < len && !result; i++) {\n+\t\tstruct remote *r = rlist[i];\n \t\tif (!r)\n \t\t\tcontinue;\n \t\tif (!r->fetch)\n@@ -738,6 +741,38 @@ int for_each_remote(each_remote_fn fn, void *priv)\n \treturn result;\n }\n \n+int for_each_remote(each_remote_fn fn, void *priv)\n+{\n+\tread_config();\n+\treturn for_each_remote_do(remotes, remotes_nr, fn, priv);\n+}\n+\n+int compare_fetch_prio(const void *p, const void *q)\n+{\n+\tconst struct remote *pp = *(struct remote**)p;\n+\tconst struct remote *qq = *(struct remote**)q;\n+\n+\treturn pp->fetch_prio - qq->fetch_prio;\n+}\n+\n+int for_each_sorted_remote(each_remote_fn fn, void *priv)\n+{\n+\tstruct remote **sr;\n+\tint i, rc;\n+\n+\tread_config();\n+\n+\tsr = xmalloc(sizeof (struct remote*) * remotes_nr);\n+\tfor (i = 0; i < remotes_nr; i++)\n+\t\tsr[i] = remotes[i];\n+\n+\tqsort(sr, remotes_nr, sizeof (struct remote*), compare_fetch_prio);\n+\n+\trc = for_each_remote_do(sr, remotes_nr, fn, priv);\n+\tfree(sr);\n+\treturn rc;\n+}\n+\n static void handle_duplicate(struct ref *ref1, struct ref *ref2)\n {\n \tif (strcmp(ref1->name, ref2->name)) {\ndiff --git a/remote.h b/remote.h\nindex c21fd37..09b48e4 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -47,6 +47,7 @@ struct remote {\n \tint skip_default_update;\n \tint mirror;\n \tint prune;\n+\tint fetch_prio;\n \n \tconst char *receivepack;\n \tconst char *uploadpack;\n@@ -64,6 +65,7 @@ int remote_is_configured(struct remote *remote);\n \n typedef int each_remote_fn(struct remote *remote, void *priv);\n int for_each_remote(each_remote_fn fn, void *priv);\n+int for_each_sorted_remote(each_remote_fn fn, void *priv);\n \n int remote_has_url(struct remote *remote, const char *url);\n \n-- \n2.8.1.281.g0994585\n"},{"id":"284490","messageId":"20160425213720.GB11227@sigill.intra.peff.net","threadId":"42142","inReplyTo":"20160425211506.8421-1-mtzguido@gmail.com","subject":"Re: [RFC/PATCH] Ordering of remotes for fetch --all","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-04-25T21:37:20Z","receivedAt":"2016-04-25T21:37:20Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Apr 25, 2016 at 11:15:05PM +0200, Guido Martínez wrote:\n\n> I run a server with several git mirrors, that are updated every hour. On\n> that same server, users clone those projects and work on them. We use\n> the local mirrors to reduce network load: the users can fetch from the\n> mirror first (to get most of the objects with zero network cost) and\n> then fetch the real remote (to make sure they're completely up to date).\n> \n> I would like this to be configurable in each git working directory,\n> so users can just configure the order they want and then just do \"git\n> remote update\".\n> \n> I'm aware one can get this behavior by editing .git/config and\n> ordering the remotes as one wishes, but I find that very hacky and not\n> scripting-friendly.\n\nYou can also define your own ordered groups, like:\n\n  $ git config remotes.foo \"one two three\"\n  $ git fetch foo 2>&1 | grep ^Fetching\n  Fetching one\n  Fetching two\n  Fetching three\n\nThat's not _exactly_ the same, because you can't give a partial ordering\nof one high-priority remote and then say \"all the rest, in whatever\norder you want\", because there's no way to say \"all the rest\".\n\nYou _can_ say:\n\n  git config remotes.foo \"high-priority --all\"\n\nbut the final \"--all\" will fetch from high-priority again. An\nalternative feature would be to teach remotes.* groups to cull\nduplicates, if that's not acceptable.\n\nI don't have a strong opinion against your approach, though. Just\nexploring alternatives.\n\n-Peff\n"},{"id":"284817","messageId":"CA++DQUnWnVfCWHZ1Jhzxr2BpdmXj2aChez+ozxW7FUPkpQuoNw@mail.gmail.com","threadId":"42142","inReplyTo":"20160425213720.GB11227@sigill.intra.peff.net","subject":"Re: [RFC/PATCH] Ordering of remotes for fetch --all","fromName":"Guido Martínez","fromEmail":"mtzguido@gmail.com","sentAt":"2016-04-28T15:24:20Z","receivedAt":"2016-04-28T15:24:20Z","isPatch":true,"sender":{"key":"mtzguido@gmail.com","avatar":null},"body":"Hi Jeff, thanks for your comments.\n\nOn Mon, Apr 25, 2016 at 11:37 PM, Jeff King <peff@peff.net> wrote:\n> On Mon, Apr 25, 2016 at 11:15:05PM +0200, Guido Martínez wrote:\n>\n>> I run a server with several git mirrors, that are updated every hour. On\n>> that same server, users clone those projects and work on them. We use\n>> the local mirrors to reduce network load: the users can fetch from the\n>> mirror first (to get most of the objects with zero network cost) and\n>> then fetch the real remote (to make sure they're completely up to date).\n>>\n>> I would like this to be configurable in each git working directory,\n>> so users can just configure the order they want and then just do \"git\n>> remote update\".\n>>\n>> I'm aware one can get this behavior by editing .git/config and\n>> ordering the remotes as one wishes, but I find that very hacky and not\n>> scripting-friendly.\n>\n> You can also define your own ordered groups, like:\n>\n>   $ git config remotes.foo \"one two three\"\n>   $ git fetch foo 2>&1 | grep ^Fetching\n>   Fetching one\n>   Fetching two\n>   Fetching three\n>\n> That's not _exactly_ the same, because you can't give a partial ordering\n> of one high-priority remote and then say \"all the rest, in whatever\n> order you want\", because there's no way to say \"all the rest\".\n>\n> You _can_ say:\n>\n>   git config remotes.foo \"high-priority --all\"\n>\n> but the final \"--all\" will fetch from high-priority again. An\n> alternative feature would be to teach remotes.* groups to cull\n> duplicates, if that's not acceptable.\nThese are good, but my main drive was to be able to just \"git remote\nupdate\" without any more information. In your cases I need to call\nupdate \"foo\". Also as you mention you either need to edit foo when\nadding a repo, or duplicating the pull from the high-prio one.\n\nAnother approach would be to add a \"fetchdep\" pointing to another\nremote, and then do a topological sort on fetch --all. This can also\nbe used on \"git pull\", to first pull from the mirror without any extra\ncommand.\n\nMaybe it's not such a big deal, but I think it's a nice feature to\nhave. It allows for a stupidly simple mirroring/prefetch scheme,\nwithout any proxy or anything fancy.\n\nNot sure if it suits the needs of anyone else, though... Would there\nbe interest in me implementing the \"fetchdep\" alternative?\n\nThanks!\nGuido\n\n>\n> I don't have a strong opinion against your approach, though. Just\n> exploring alternatives.\n>\n> -Peff\n"}]}