{"thread":{"id":"48881","subject":"[PATCH v3 0/9] Introducing remote ODBs","startedAt":"2018-07-13T17:50:21Z","lastAt":"2018-07-22T09:05:29Z","messageCount":11,"participants":["Christian Couder","Duy Nguyen"],"isPatch":true,"patchVersion":3,"patchTotal":9},"messages":[{"id":"352508","messageId":"20180713174959.16748-1-chriscool@tuxfamily.org","threadId":"48881","inReplyTo":null,"subject":"[PATCH v3 0/9] Introducing remote ODBs","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-07-13T17:49:50Z","receivedAt":"2018-07-13T17:50:21Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This path series is a follow up from the patch series called \"odb\nremote\" that I sent earlier this year, which was itself a follow up\nfrom previous series. See the links section for more information.\n\nAs with the previous \"odb remote\" series, this series is only about\nintegrating with the promisor/narrow clone work and showing that it\nmakes it possible to use more than one promisor remote. Everything\nthat is not necessary for that integration has been removed for now\n(though you can still find it in one of my branches on GitHub if you\nwant).\n\nThere is one test in patch 8/9 that shows that more than one promisor\nremote can now be used, but I still feel that it could be interesting\nto add other such tests, so I am open to ideas in this area.\n\nChanges compared to V2 of this patch series\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n  - fix unused variable in patch 2/9: the \"SQUASH???\" patch that Junio\n    had created (thanks!) has been squashed\n\nHigh level overview of this patch series\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nAll the patches except 2/9 are unchanged compared to V2.\n\n  - Patch 1/9:\n\nThis makes functions in fetch-object.c return an error code, which is\nnecessary to later tell that they failed and try another remote odb\nwhen there is more than one. This could also just be seen as a fix to\nthese functions.\n\n  - Patch 2/9:\n\nThis introduces the minimum infrastructure for remote odbs.\n\n  - Patches 3/9 and 4/9:\n\nThese patches implement remote_odb_get_direct() and\nremote_odb_get_many_direct() using the functions from fetch-object.c.\nThese new functions will be used in following patches to replace the\nfunctions from fetch-object.c.\n\n  - Patch 5/9:\n\nThis implement remote_odb_reinit() which will be needed to reparse the\nremote odb configuration.\n\n  - Patches 6/9 and 7/9:\n\nThese patches integrate the remote odb mechanism into the\npromisor/narrow clone code. The \"extensions.partialClone\" config\noption is replaced by \"odb.<name>.promisorRemote\" and\n\"core.partialCloneFilter\" is replaced by\n\"odb.<name>.partialCloneFilter\".\n\n  - Patch 8/9:\n\nThis adds a test case that shows that now more than one promisor\nremote can be used.\n\n  - Patch 9/9:\n\nThis starts documenting the remote odb mechanism.\n\nDiscussion\n~~~~~~~~~~\n\nI am not sure that it is ok to completely replace the\n\"extensions.partialclone\" config option. Even if it is fully replaced,\nno \"extensions.remoteodb\" is implemented in these patches, as maybe\nthe \"extensions.partialclone\" name could be kept even if the\nunderlying mechanism is the remote odb mechanism.\n\nAnyway I think that the remote odb mechanism is much more extensible,\nso I think using \"extensions.partialclone\" to specify a promisor\nremote should be at least deprecated.\n\nLinks\n~~~~~\n\nThis patch series on GitHub:\n\nV3: https://github.com/chriscool/git/commits/remote-odb\nV2: https://github.com/chriscool/git/commits/remote-odb2\nV1: https://github.com/chriscool/git/commits/remote-odb1\n\nDiscussions related to previous versions:\n\nV2: https://public-inbox.org/git/20180630083542.20347-1-chriscool@tuxfamily.org/\nV1: https://public-inbox.org/git/20180623121846.19750-1-chriscool@tuxfamily.org/\n\nPrevious \"odb remote\" series:\n\nhttps://public-inbox.org/git/20180513103232.17514-1-chriscool@tuxfamily.org/\nhttps://github.com/chriscool/git/commits/odb-remote\n\nVersion 1 and 2 of the \"Promisor remotes and external ODB support\" series:\n\nhttps://public-inbox.org/git/20180103163403.11303-1-chriscool@tuxfamily.org/\nhttps://public-inbox.org/git/20180319133147.15413-1-chriscool@tuxfamily.org/\n\nVersion 1 and 2 of the \"Promisor remotes and external ODB support\" series on GitHub:\n\nhttps://github.com/chriscool/git/commits/gl-small-promisor-external-odb12\nhttps://github.com/chriscool/git/commits/gl-small-promisor-external-odb71\n\nChristian Couder (9):\n  fetch-object: make functions return an error code\n  Add initial remote odb support\n  remote-odb: implement remote_odb_get_direct()\n  remote-odb: implement remote_odb_get_many_direct()\n  remote-odb: add remote_odb_reinit()\n  Use remote_odb_get_direct() and has_remote_odb()\n  Use odb.origin.partialclonefilter instead of core.partialclonefilter\n  t0410: test fetching from many promisor remotes\n  Documentation/config: add odb.<name>.promisorRemote\n\n Documentation/config.txt      |   5 ++\n Makefile                      |   2 +\n builtin/cat-file.c            |   5 +-\n builtin/fetch.c               |  13 ++--\n builtin/gc.c                  |   3 +-\n builtin/repack.c              |   3 +-\n cache.h                       |   2 -\n connected.c                   |   3 +-\n environment.c                 |   1 -\n fetch-object.c                |  15 ++--\n fetch-object.h                |   6 +-\n list-objects-filter-options.c |  51 +++++++------\n list-objects-filter-options.h |   3 +-\n odb-helper.c                  |  45 +++++++++++\n odb-helper.h                  |  24 ++++++\n packfile.c                    |   3 +-\n remote-odb.c                  | 137 ++++++++++++++++++++++++++++++++++\n remote-odb.h                  |  10 +++\n setup.c                       |   7 +-\n sha1-file.c                   |  14 ++--\n t/t0410-partial-clone.sh      |  58 +++++++++-----\n t/t5500-fetch-pack.sh         |   4 +-\n t/t5601-clone.sh              |   2 +-\n t/t5616-partial-clone.sh      |   4 +-\n t/t5702-protocol-v2.sh        |   2 +-\n unpack-trees.c                |   6 +-\n 26 files changed, 342 insertions(+), 86 deletions(-)\n create mode 100644 odb-helper.c\n create mode 100644 odb-helper.h\n create mode 100644 remote-odb.c\n create mode 100644 remote-odb.h\n\n-- \n2.18.0.171.ge260395bfe\n\n"},{"id":"352509","messageId":"20180713174959.16748-2-chriscool@tuxfamily.org","threadId":"48881","inReplyTo":"20180713174959.16748-1-chriscool@tuxfamily.org","subject":"[PATCH v3 1/9] fetch-object: make functions return an error code","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-07-13T17:49:51Z","receivedAt":"2018-07-13T17:50:23Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: Christian Couder <christian.couder@gmail.com>\n\nThe callers of the fetch_object() and fetch_objects() might\nbe interested in knowing if these functions succeeded or not.\n\nSigned-off-by: Christian Couder <chriscool@tuxfamily.org>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n fetch-object.c | 15 +++++++++------\n fetch-object.h |  6 +++---\n sha1-file.c    |  4 ++--\n 3 files changed, 14 insertions(+), 11 deletions(-)\n\ndiff --git a/fetch-object.c b/fetch-object.c\nindex 853624f811..ccc4ea7f1a 100644\n--- a/fetch-object.c\n+++ b/fetch-object.c\n@@ -5,11 +5,12 @@\n #include \"transport.h\"\n #include \"fetch-object.h\"\n \n-static void fetch_refs(const char *remote_name, struct ref *ref)\n+static int fetch_refs(const char *remote_name, struct ref *ref)\n {\n \tstruct remote *remote;\n \tstruct transport *transport;\n \tint original_fetch_if_missing = fetch_if_missing;\n+\tint res;\n \n \tfetch_if_missing = 0;\n \tremote = remote_get(remote_name);\n@@ -19,18 +20,20 @@ static void fetch_refs(const char *remote_name, struct ref *ref)\n \n \ttransport_set_option(transport, TRANS_OPT_FROM_PROMISOR, \"1\");\n \ttransport_set_option(transport, TRANS_OPT_NO_DEPENDENTS, \"1\");\n-\ttransport_fetch_refs(transport, ref);\n+\tres = transport_fetch_refs(transport, ref);\n \tfetch_if_missing = original_fetch_if_missing;\n+\n+\treturn res;\n }\n \n-void fetch_object(const char *remote_name, const unsigned char *sha1)\n+int fetch_object(const char *remote_name, const unsigned char *sha1)\n {\n \tstruct ref *ref = alloc_ref(sha1_to_hex(sha1));\n \thashcpy(ref->old_oid.hash, sha1);\n-\tfetch_refs(remote_name, ref);\n+\treturn fetch_refs(remote_name, ref);\n }\n \n-void fetch_objects(const char *remote_name, const struct oid_array *to_fetch)\n+int fetch_objects(const char *remote_name, const struct oid_array *to_fetch)\n {\n \tstruct ref *ref = NULL;\n \tint i;\n@@ -41,5 +44,5 @@ void fetch_objects(const char *remote_name, const struct oid_array *to_fetch)\n \t\tnew_ref->next = ref;\n \t\tref = new_ref;\n \t}\n-\tfetch_refs(remote_name, ref);\n+\treturn fetch_refs(remote_name, ref);\n }\ndiff --git a/fetch-object.h b/fetch-object.h\nindex 4b269d07ed..12e1f9ee70 100644\n--- a/fetch-object.h\n+++ b/fetch-object.h\n@@ -3,9 +3,9 @@\n \n #include \"sha1-array.h\"\n \n-extern void fetch_object(const char *remote_name, const unsigned char *sha1);\n+extern int fetch_object(const char *remote_name, const unsigned char *sha1);\n \n-extern void fetch_objects(const char *remote_name,\n-\t\t\t  const struct oid_array *to_fetch);\n+extern int fetch_objects(const char *remote_name,\n+\t\t\t const struct oid_array *to_fetch);\n \n #endif\ndiff --git a/sha1-file.c b/sha1-file.c\nindex de4839e634..c099f5584d 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -1312,8 +1312,8 @@ int oid_object_info_extended(struct repository *r, const struct object_id *oid,\n \t\tif (fetch_if_missing && repository_format_partial_clone &&\n \t\t    !already_retried && r == the_repository) {\n \t\t\t/*\n-\t\t\t * TODO Investigate having fetch_object() return\n-\t\t\t * TODO error/success and stopping the music here.\n+\t\t\t * TODO Investigate checking fetch_object() return\n+\t\t\t * TODO value and stopping on error here.\n \t\t\t * TODO Pass a repository struct through fetch_object,\n \t\t\t * such that arbitrary repositories work.\n \t\t\t */\n-- \n2.18.0.171.ge260395bfe\n\n"},{"id":"352510","messageId":"20180713174959.16748-3-chriscool@tuxfamily.org","threadId":"48881","inReplyTo":"20180713174959.16748-1-chriscool@tuxfamily.org","subject":"[PATCH v3 2/9] Add initial remote odb support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-07-13T17:49:52Z","receivedAt":"2018-07-13T17:50:24Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: Christian Couder <christian.couder@gmail.com>\n\nThe remote-odb.{c,h} files will contain the functions\nthat are called by the rest of Git mostly from\n\"sha1-file.c\" to access the objects managed by the\nremote odbs.\n\nThe odb-helper.{c,h} files will contain the functions to\nactually implement communication with either the internal\nfunctions or the external scripts or processes that will\nmanage and provide remote git objects.\n\nFor now only infrastructure to create helpers from the\nconfig and to check if there are remote odbs and odb\nhelpers is implemented.\n\nWe expect that there will not be a lot of helpers, so it\nis ok to use a simple linked list to manage them.\n\nHelped-by: Jeff King <peff@peff.net>\nSigned-off-by: Christian Couder <chriscool@tuxfamily.org>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n Makefile     |  2 ++\n odb-helper.c | 16 +++++++++\n odb-helper.h | 19 +++++++++++\n remote-odb.c | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n remote-odb.h |  7 ++++\n 5 files changed, 135 insertions(+)\n create mode 100644 odb-helper.c\n create mode 100644 odb-helper.h\n create mode 100644 remote-odb.c\n create mode 100644 remote-odb.h\n\ndiff --git a/Makefile b/Makefile\nindex 0cb6590f24..5af048ddd5 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -896,6 +896,8 @@ LIB_OBJS += notes-cache.o\n LIB_OBJS += notes-merge.o\n LIB_OBJS += notes-utils.o\n LIB_OBJS += object.o\n+LIB_OBJS += odb-helper.o\n+LIB_OBJS += remote-odb.o\n LIB_OBJS += oidmap.o\n LIB_OBJS += oidset.o\n LIB_OBJS += packfile.o\ndiff --git a/odb-helper.c b/odb-helper.c\nnew file mode 100644\nindex 0000000000..b4d403ffa9\n--- /dev/null\n+++ b/odb-helper.c\n@@ -0,0 +1,16 @@\n+#include \"cache.h\"\n+#include \"object.h\"\n+#include \"argv-array.h\"\n+#include \"odb-helper.h\"\n+#include \"run-command.h\"\n+#include \"sha1-lookup.h\"\n+\n+struct odb_helper *odb_helper_new(const char *name, int namelen)\n+{\n+\tstruct odb_helper *o;\n+\n+\to = xcalloc(1, sizeof(*o));\n+\to->name = xmemdupz(name, namelen);\n+\n+\treturn o;\n+}\ndiff --git a/odb-helper.h b/odb-helper.h\nnew file mode 100644\nindex 0000000000..4b792a3896\n--- /dev/null\n+++ b/odb-helper.h\n@@ -0,0 +1,19 @@\n+#ifndef ODB_HELPER_H\n+#define ODB_HELPER_H\n+\n+/*\n+ * An odb helper is a way to access a remote odb.\n+ *\n+ * Information in its fields comes from the config file [odb \"NAME\"]\n+ * entries.\n+ */\n+struct odb_helper {\n+\tconst char *name;                 /* odb.<NAME>.* */\n+\tconst char *remote;               /* odb.<NAME>.promisorRemote */\n+\n+\tstruct odb_helper *next;\n+};\n+\n+extern struct odb_helper *odb_helper_new(const char *name, int namelen);\n+\n+#endif /* ODB_HELPER_H */\ndiff --git a/remote-odb.c b/remote-odb.c\nnew file mode 100644\nindex 0000000000..03be54ba2e\n--- /dev/null\n+++ b/remote-odb.c\n@@ -0,0 +1,91 @@\n+#include \"cache.h\"\n+#include \"remote-odb.h\"\n+#include \"odb-helper.h\"\n+#include \"config.h\"\n+\n+static struct odb_helper *helpers;\n+static struct odb_helper **helpers_tail = &helpers;\n+\n+static struct odb_helper *find_or_create_helper(const char *name, int len)\n+{\n+\tstruct odb_helper *o;\n+\n+\tfor (o = helpers; o; o = o->next)\n+\t\tif (!strncmp(o->name, name, len) && !o->name[len])\n+\t\t\treturn o;\n+\n+\to = odb_helper_new(name, len);\n+\t*helpers_tail = o;\n+\thelpers_tail = &o->next;\n+\n+\treturn o;\n+}\n+\n+static struct odb_helper *do_find_odb_helper(const char *remote)\n+{\n+\tstruct odb_helper *o;\n+\n+\tfor (o = helpers; o; o = o->next)\n+\t\tif (o->remote && !strcmp(o->remote, remote))\n+\t\t\treturn o;\n+\n+\treturn NULL;\n+}\n+\n+static int remote_odb_config(const char *var, const char *value, void *data)\n+{\n+\tstruct odb_helper *o;\n+\tconst char *name;\n+\tint namelen;\n+\tconst char *subkey;\n+\n+\tif (parse_config_key(var, \"odb\", &name, &namelen, &subkey) < 0)\n+\t\treturn 0;\n+\n+\to = find_or_create_helper(name, namelen);\n+\n+\tif (!strcmp(subkey, \"promisorremote\")) {\n+\t\tconst char *remote;\n+\t\tint res = git_config_string(&remote, var, value);\n+\n+\t\tif (res)\n+\t\t\treturn res;\n+\n+\t\tif (do_find_odb_helper(remote))\n+\t\t\treturn error(_(\"when parsing config key '%s' \"\n+\t\t\t\t       \"helper for remote '%s' already exists\"),\n+\t\t\t\t     var, remote);\n+\n+\t\to->remote = remote;\n+\n+\t\treturn 0;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void remote_odb_init(void)\n+{\n+\tstatic int initialized;\n+\n+\tif (initialized)\n+\t\treturn;\n+\tinitialized = 1;\n+\n+\tgit_config(remote_odb_config, NULL);\n+}\n+\n+struct odb_helper *find_odb_helper(const char *remote)\n+{\n+\tremote_odb_init();\n+\n+\tif (!remote)\n+\t\treturn helpers;\n+\n+\treturn do_find_odb_helper(remote);\n+}\n+\n+int has_remote_odb(void)\n+{\n+\treturn !!find_odb_helper(NULL);\n+}\ndiff --git a/remote-odb.h b/remote-odb.h\nnew file mode 100644\nindex 0000000000..4c7b13695f\n--- /dev/null\n+++ b/remote-odb.h\n@@ -0,0 +1,7 @@\n+#ifndef REMOTE_ODB_H\n+#define REMOTE_ODB_H\n+\n+extern struct odb_helper *find_odb_helper(const char *remote);\n+extern int has_remote_odb(void);\n+\n+#endif /* REMOTE_ODB_H */\n-- \n2.18.0.171.ge260395bfe\n\n"},{"id":"352511","messageId":"20180713174959.16748-4-chriscool@tuxfamily.org","threadId":"48881","inReplyTo":"20180713174959.16748-1-chriscool@tuxfamily.org","subject":"[PATCH v3 3/9] remote-odb: implement remote_odb_get_direct()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-07-13T17:49:53Z","receivedAt":"2018-07-13T17:50:26Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: Christian Couder <christian.couder@gmail.com>\n\nThis is implemented only in the promisor remote mode\nfor now by calling fetch_object().\n\nSigned-off-by: Christian Couder <chriscool@tuxfamily.org>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n odb-helper.c | 14 ++++++++++++++\n odb-helper.h |  3 ++-\n remote-odb.c | 17 +++++++++++++++++\n remote-odb.h |  1 +\n 4 files changed, 34 insertions(+), 1 deletion(-)\n\ndiff --git a/odb-helper.c b/odb-helper.c\nindex b4d403ffa9..99b5a345e8 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -4,6 +4,7 @@\n #include \"odb-helper.h\"\n #include \"run-command.h\"\n #include \"sha1-lookup.h\"\n+#include \"fetch-object.h\"\n \n struct odb_helper *odb_helper_new(const char *name, int namelen)\n {\n@@ -14,3 +15,16 @@ struct odb_helper *odb_helper_new(const char *name, int namelen)\n \n \treturn o;\n }\n+\n+int odb_helper_get_direct(struct odb_helper *o,\n+\t\t\t  const unsigned char *sha1)\n+{\n+\tint res;\n+\tuint64_t start = getnanotime();\n+\n+\tres = fetch_object(o->remote, sha1);\n+\n+\ttrace_performance_since(start, \"odb_helper_get_direct\");\n+\n+\treturn res;\n+}\ndiff --git a/odb-helper.h b/odb-helper.h\nindex 4b792a3896..4c52e81ce8 100644\n--- a/odb-helper.h\n+++ b/odb-helper.h\n@@ -15,5 +15,6 @@ struct odb_helper {\n };\n \n extern struct odb_helper *odb_helper_new(const char *name, int namelen);\n-\n+extern int odb_helper_get_direct(struct odb_helper *o,\n+\t\t\t\t const unsigned char *sha1);\n #endif /* ODB_HELPER_H */\ndiff --git a/remote-odb.c b/remote-odb.c\nindex 03be54ba2e..7f815c7ebb 100644\n--- a/remote-odb.c\n+++ b/remote-odb.c\n@@ -89,3 +89,20 @@ int has_remote_odb(void)\n {\n \treturn !!find_odb_helper(NULL);\n }\n+\n+int remote_odb_get_direct(const unsigned char *sha1)\n+{\n+\tstruct odb_helper *o;\n+\n+\ttrace_printf(\"trace: remote_odb_get_direct: %s\", sha1_to_hex(sha1));\n+\n+\tremote_odb_init();\n+\n+\tfor (o = helpers; o; o = o->next) {\n+\t\tif (odb_helper_get_direct(o, sha1) < 0)\n+\t\t\tcontinue;\n+\t\treturn 0;\n+\t}\n+\n+\treturn -1;\n+}\ndiff --git a/remote-odb.h b/remote-odb.h\nindex 4c7b13695f..c5384c922d 100644\n--- a/remote-odb.h\n+++ b/remote-odb.h\n@@ -3,5 +3,6 @@\n \n extern struct odb_helper *find_odb_helper(const char *remote);\n extern int has_remote_odb(void);\n+extern int remote_odb_get_direct(const unsigned char *sha1);\n \n #endif /* REMOTE_ODB_H */\n-- \n2.18.0.171.ge260395bfe\n\n"},{"id":"352512","messageId":"20180713174959.16748-5-chriscool@tuxfamily.org","threadId":"48881","inReplyTo":"20180713174959.16748-1-chriscool@tuxfamily.org","subject":"[PATCH v3 4/9] remote-odb: implement remote_odb_get_many_direct()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-07-13T17:49:54Z","receivedAt":"2018-07-13T17:50:27Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: Christian Couder <christian.couder@gmail.com>\n\nThis function will be used to get many objects from a promisor\nremote.\n\nSigned-off-by: Christian Couder <chriscool@tuxfamily.org>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n odb-helper.c | 15 +++++++++++++++\n odb-helper.h |  3 +++\n remote-odb.c | 17 +++++++++++++++++\n remote-odb.h |  1 +\n 4 files changed, 36 insertions(+)\n\ndiff --git a/odb-helper.c b/odb-helper.c\nindex 99b5a345e8..246ebf8f7a 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -28,3 +28,18 @@ int odb_helper_get_direct(struct odb_helper *o,\n \n \treturn res;\n }\n+\n+int odb_helper_get_many_direct(struct odb_helper *o,\n+\t\t\t       const struct oid_array *to_get)\n+{\n+\tint res;\n+\tuint64_t start;\n+\n+\tstart = getnanotime();\n+\n+\tres = fetch_objects(o->remote, to_get);\n+\n+\ttrace_performance_since(start, \"odb_helper_get_many_direct\");\n+\n+\treturn res;\n+}\ndiff --git a/odb-helper.h b/odb-helper.h\nindex 4c52e81ce8..154ce4c7e4 100644\n--- a/odb-helper.h\n+++ b/odb-helper.h\n@@ -17,4 +17,7 @@ struct odb_helper {\n extern struct odb_helper *odb_helper_new(const char *name, int namelen);\n extern int odb_helper_get_direct(struct odb_helper *o,\n \t\t\t\t const unsigned char *sha1);\n+extern int odb_helper_get_many_direct(struct odb_helper *o,\n+\t\t\t\t      const struct oid_array *to_get);\n+\n #endif /* ODB_HELPER_H */\ndiff --git a/remote-odb.c b/remote-odb.c\nindex 7f815c7ebb..09dfc2e16f 100644\n--- a/remote-odb.c\n+++ b/remote-odb.c\n@@ -106,3 +106,20 @@ int remote_odb_get_direct(const unsigned char *sha1)\n \n \treturn -1;\n }\n+\n+int remote_odb_get_many_direct(const struct oid_array *to_get)\n+{\n+\tstruct odb_helper *o;\n+\n+\ttrace_printf(\"trace: remote_odb_get_many_direct: nr: %d\", to_get->nr);\n+\n+\tremote_odb_init();\n+\n+\tfor (o = helpers; o; o = o->next) {\n+\t\tif (odb_helper_get_many_direct(o, to_get) < 0)\n+\t\t\tcontinue;\n+\t\treturn 0;\n+\t}\n+\n+\treturn -1;\n+}\ndiff --git a/remote-odb.h b/remote-odb.h\nindex c5384c922d..e10a8bf855 100644\n--- a/remote-odb.h\n+++ b/remote-odb.h\n@@ -4,5 +4,6 @@\n extern struct odb_helper *find_odb_helper(const char *remote);\n extern int has_remote_odb(void);\n extern int remote_odb_get_direct(const unsigned char *sha1);\n+extern int remote_odb_get_many_direct(const struct oid_array *to_get);\n \n #endif /* REMOTE_ODB_H */\n-- \n2.18.0.171.ge260395bfe\n\n"},{"id":"352513","messageId":"20180713174959.16748-6-chriscool@tuxfamily.org","threadId":"48881","inReplyTo":"20180713174959.16748-1-chriscool@tuxfamily.org","subject":"[PATCH v3 5/9] remote-odb: add remote_odb_reinit()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-07-13T17:49:55Z","receivedAt":"2018-07-13T17:50:29Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: Christian Couder <christian.couder@gmail.com>\n\nWe will need to reinitialize the remote odb configuration\nas we will make some changes to it in a later commit when\nwe will detect that a remote is also a remote odb.\n\nSigned-off-by: Christian Couder <chriscool@tuxfamily.org>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n remote-odb.c | 14 ++++++++++++--\n remote-odb.h |  1 +\n 2 files changed, 13 insertions(+), 2 deletions(-)\n\ndiff --git a/remote-odb.c b/remote-odb.c\nindex 09dfc2e16f..53a72b4ba5 100644\n--- a/remote-odb.c\n+++ b/remote-odb.c\n@@ -64,17 +64,27 @@ static int remote_odb_config(const char *var, const char *value, void *data)\n \treturn 0;\n }\n \n-static void remote_odb_init(void)\n+static void remote_odb_do_init(int force)\n {\n \tstatic int initialized;\n \n-\tif (initialized)\n+\tif (!force && initialized)\n \t\treturn;\n \tinitialized = 1;\n \n \tgit_config(remote_odb_config, NULL);\n }\n \n+static inline void remote_odb_init(void)\n+{\n+\tremote_odb_do_init(0);\n+}\n+\n+inline void remote_odb_reinit(void)\n+{\n+\tremote_odb_do_init(1);\n+}\n+\n struct odb_helper *find_odb_helper(const char *remote)\n {\n \tremote_odb_init();\ndiff --git a/remote-odb.h b/remote-odb.h\nindex e10a8bf855..79803782af 100644\n--- a/remote-odb.h\n+++ b/remote-odb.h\n@@ -1,6 +1,7 @@\n #ifndef REMOTE_ODB_H\n #define REMOTE_ODB_H\n \n+extern void remote_odb_reinit(void);\n extern struct odb_helper *find_odb_helper(const char *remote);\n extern int has_remote_odb(void);\n extern int remote_odb_get_direct(const unsigned char *sha1);\n-- \n2.18.0.171.ge260395bfe\n\n"},{"id":"352514","messageId":"20180713174959.16748-7-chriscool@tuxfamily.org","threadId":"48881","inReplyTo":"20180713174959.16748-1-chriscool@tuxfamily.org","subject":"[PATCH v3 6/9] Use remote_odb_get_direct() and has_remote_odb()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-07-13T17:49:56Z","receivedAt":"2018-07-13T17:50:32Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: Christian Couder <christian.couder@gmail.com>\n\nInstead of using the repository_format_partial_clone global\nand fetch_object() directly, let's use has_remote_odb() and\nremote_odb_get_direct().\n\nSigned-off-by: Christian Couder <chriscool@tuxfamily.org>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n builtin/cat-file.c            |  5 +++--\n builtin/fetch.c               | 11 ++++++-----\n builtin/gc.c                  |  3 ++-\n builtin/repack.c              |  3 ++-\n cache.h                       |  2 --\n connected.c                   |  3 ++-\n environment.c                 |  1 -\n list-objects-filter-options.c | 27 +++++++++++++++------------\n packfile.c                    |  3 ++-\n setup.c                       |  7 +------\n sha1-file.c                   | 14 ++++++++------\n t/t0410-partial-clone.sh      | 34 +++++++++++++++++-----------------\n t/t5500-fetch-pack.sh         |  4 ++--\n t/t5601-clone.sh              |  2 +-\n t/t5616-partial-clone.sh      |  2 +-\n t/t5702-protocol-v2.sh        |  2 +-\n unpack-trees.c                |  6 +++---\n 17 files changed, 66 insertions(+), 63 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 665b581949..3b24c1b9b7 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -13,6 +13,7 @@\n #include \"tree-walk.h\"\n #include \"sha1-array.h\"\n #include \"packfile.h\"\n+#include \"remote-odb.h\"\n \n struct batch_options {\n \tint enabled;\n@@ -477,8 +478,8 @@ static int batch_objects(struct batch_options *opt)\n \n \t\tfor_each_loose_object(batch_loose_object, &sa, 0);\n \t\tfor_each_packed_object(batch_packed_object, &sa, 0);\n-\t\tif (repository_format_partial_clone)\n-\t\t\twarning(\"This repository has extensions.partialClone set. Some objects may not be loaded.\");\n+\t\tif (has_remote_odb())\n+\t\t\twarning(\"This repository uses an odb. Some objects may not be loaded.\");\n \n \t\tcb.opt = opt;\n \t\tcb.expand = &data;\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex ea5b9669ad..1fd4dfd024 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -21,6 +21,7 @@\n #include \"utf8.h\"\n #include \"packfile.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"remote-odb.h\"\n \n static const char * const builtin_fetch_usage[] = {\n \tN_(\"git fetch [<options>] [<repository> [<refspec>...]]\"),\n@@ -1313,7 +1314,7 @@ static inline void fetch_one_setup_partial(struct remote *remote)\n \t * If no prior partial clone/fetch and the current fetch DID NOT\n \t * request a partial-fetch, do a normal fetch.\n \t */\n-\tif (!repository_format_partial_clone && !filter_options.choice)\n+\tif (!has_remote_odb() && !filter_options.choice)\n \t\treturn;\n \n \t/*\n@@ -1321,7 +1322,7 @@ static inline void fetch_one_setup_partial(struct remote *remote)\n \t * on this repo and remember the given filter-spec as the default\n \t * for subsequent fetches to this remote.\n \t */\n-\tif (!repository_format_partial_clone && filter_options.choice) {\n+\tif (!has_remote_odb() && filter_options.choice) {\n \t\tpartial_clone_register(remote->name, &filter_options);\n \t\treturn;\n \t}\n@@ -1330,7 +1331,7 @@ static inline void fetch_one_setup_partial(struct remote *remote)\n \t * We are currently limited to only ONE promisor remote and only\n \t * allow partial-fetches from the promisor remote.\n \t */\n-\tif (strcmp(remote->name, repository_format_partial_clone)) {\n+\tif (!find_odb_helper(remote->name)) {\n \t\tif (filter_options.choice)\n \t\t\tdie(_(\"--filter can only be used with the remote configured in core.partialClone\"));\n \t\treturn;\n@@ -1461,7 +1462,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tif (depth || deepen_since || deepen_not.nr)\n \t\tdeepen = 1;\n \n-\tif (filter_options.choice && !repository_format_partial_clone)\n+\tif (filter_options.choice && !has_remote_odb())\n \t\tdie(\"--filter can only be used when extensions.partialClone is set\");\n \n \tif (all) {\n@@ -1495,7 +1496,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (remote) {\n-\t\tif (filter_options.choice || repository_format_partial_clone)\n+\t\tif (filter_options.choice || has_remote_odb())\n \t\t\tfetch_one_setup_partial(remote);\n \t\tresult = fetch_one(remote, argc, argv, prune_tags_ok);\n \t} else {\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex ccfb1ceaeb..02c783b514 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -26,6 +26,7 @@\n #include \"pack-objects.h\"\n #include \"blob.h\"\n #include \"tree.h\"\n+#include \"remote-odb.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -619,7 +620,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\targv_array_push(&prune, prune_expire);\n \t\t\tif (quiet)\n \t\t\t\targv_array_push(&prune, \"--no-progress\");\n-\t\t\tif (repository_format_partial_clone)\n+\t\t\tif (has_remote_odb())\n \t\t\t\targv_array_push(&prune,\n \t\t\t\t\t\t\"--exclude-promisor-objects\");\n \t\t\tif (run_command_v_opt(prune.argv, RUN_GIT_CMD))\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 6c636e159e..72b1e2d94f 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -8,6 +8,7 @@\n #include \"strbuf.h\"\n #include \"string-list.h\"\n #include \"argv-array.h\"\n+#include \"remote-odb.h\"\n \n static int delta_base_offset = 1;\n static int pack_kept_objects = -1;\n@@ -249,7 +250,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \targv_array_push(&cmd.args, \"--all\");\n \targv_array_push(&cmd.args, \"--reflog\");\n \targv_array_push(&cmd.args, \"--indexed-objects\");\n-\tif (repository_format_partial_clone)\n+\tif (has_remote_odb())\n \t\targv_array_push(&cmd.args, \"--exclude-promisor-objects\");\n \tif (window)\n \t\targv_array_pushf(&cmd.args, \"--window=%s\", window);\ndiff --git a/cache.h b/cache.h\nindex d49092d94d..af689abd19 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -904,13 +904,11 @@ extern int grafts_replace_parents;\n #define GIT_REPO_VERSION 0\n #define GIT_REPO_VERSION_READ 1\n extern int repository_format_precious_objects;\n-extern char *repository_format_partial_clone;\n extern const char *core_partial_clone_filter_default;\n \n struct repository_format {\n \tint version;\n \tint precious_objects;\n-\tchar *partial_clone; /* value of extensions.partialclone */\n \tint is_bare;\n \tint hash_algo;\n \tchar *work_tree;\ndiff --git a/connected.c b/connected.c\nindex 91feb78815..07154a358d 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -4,6 +4,7 @@\n #include \"connected.h\"\n #include \"transport.h\"\n #include \"packfile.h\"\n+#include \"remote-odb.h\"\n \n /*\n  * If we feed all the commits we want to verify to this command\n@@ -56,7 +57,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n \targv_array_push(&rev_list.args,\"rev-list\");\n \targv_array_push(&rev_list.args, \"--objects\");\n \targv_array_push(&rev_list.args, \"--stdin\");\n-\tif (repository_format_partial_clone)\n+\tif (has_remote_odb())\n \t\targv_array_push(&rev_list.args, \"--exclude-promisor-objects\");\n \targv_array_push(&rev_list.args, \"--not\");\n \targv_array_push(&rev_list.args, \"--all\");\ndiff --git a/environment.c b/environment.c\nindex 2a6de2330b..f8048f4a5d 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -30,7 +30,6 @@ int warn_ambiguous_refs = 1;\n int warn_on_object_refname_ambiguity = 1;\n int ref_paranoia = -1;\n int repository_format_precious_objects;\n-char *repository_format_partial_clone;\n const char *core_partial_clone_filter_default;\n const char *git_commit_encoding;\n const char *git_log_output_encoding;\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex c0e2bd6a06..60452c8f36 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -6,6 +6,7 @@\n #include \"list-objects.h\"\n #include \"list-objects-filter.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"remote-odb.h\"\n \n /*\n  * Parse value of the argument to the \"filter\" keyword.\n@@ -114,30 +115,32 @@ void partial_clone_register(\n \tconst char *remote,\n \tconst struct list_objects_filter_options *filter_options)\n {\n-\t/*\n-\t * Record the name of the partial clone remote in the\n-\t * config and in the global variable -- the latter is\n-\t * used throughout to indicate that partial clone is\n-\t * enabled and to expect missing objects.\n-\t */\n-\tif (repository_format_partial_clone &&\n-\t    *repository_format_partial_clone &&\n-\t    strcmp(remote, repository_format_partial_clone))\n-\t\tdie(_(\"cannot change partial clone promisor remote\"));\n+\tchar *cfg_name;\n+\n+\t/* Check if it is already registered */\n+\tif (find_odb_helper(remote))\n+\t\treturn;\n \n \tgit_config_set(\"core.repositoryformatversion\", \"1\");\n-\tgit_config_set(\"extensions.partialclone\", remote);\n \n-\trepository_format_partial_clone = xstrdup(remote);\n+\t/* Add odb config for the remote */\n+\tcfg_name = xstrfmt(\"odb.%s.promisorRemote\", remote);\n+\tgit_config_set(cfg_name, remote);\n+\tfree(cfg_name);\n \n \t/*\n \t * Record the initial filter-spec in the config as\n \t * the default for subsequent fetches from this remote.\n+\t *\n+\t * TODO: move core.partialclonefilter into odb.<name>\n \t */\n \tcore_partial_clone_filter_default =\n \t\txstrdup(filter_options->filter_spec);\n \tgit_config_set(\"core.partialclonefilter\",\n \t\t       core_partial_clone_filter_default);\n+\n+\t/* Make sure the config info are reset */\n+\tremote_odb_reinit();\n }\n \n void partial_clone_get_default_filter_spec(\ndiff --git a/packfile.c b/packfile.c\nindex 7cd45aa4b2..9cbc974612 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -15,6 +15,7 @@\n #include \"tree-walk.h\"\n #include \"tree.h\"\n #include \"object-store.h\"\n+#include \"remote-odb.h\"\n \n char *odb_pack_name(struct strbuf *buf,\n \t\t    const unsigned char *sha1,\n@@ -1976,7 +1977,7 @@ int is_promisor_object(const struct object_id *oid)\n \tstatic int promisor_objects_prepared;\n \n \tif (!promisor_objects_prepared) {\n-\t\tif (repository_format_partial_clone) {\n+\t\tif (has_remote_odb()) {\n \t\t\tfor_each_packed_object(add_promisor_object,\n \t\t\t\t\t       &promisor_objects,\n \t\t\t\t\t       FOR_EACH_OBJECT_PROMISOR_ONLY);\ndiff --git a/setup.c b/setup.c\nindex b24c811c1c..4eeb7fe073 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -419,11 +419,7 @@ static int check_repo_format(const char *var, const char *value, void *vdata)\n \t\t\t;\n \t\telse if (!strcmp(ext, \"preciousobjects\"))\n \t\t\tdata->precious_objects = git_config_bool(var, value);\n-\t\telse if (!strcmp(ext, \"partialclone\")) {\n-\t\t\tif (!value)\n-\t\t\t\treturn config_error_nonbool(var);\n-\t\t\tdata->partial_clone = xstrdup(value);\n-\t\t} else\n+\t\telse\n \t\t\tstring_list_append(&data->unknown_extensions, ext);\n \t} else if (strcmp(var, \"core.bare\") == 0) {\n \t\tdata->is_bare = git_config_bool(var, value);\n@@ -465,7 +461,6 @@ static int check_repository_format_gently(const char *gitdir, struct repository_\n \t}\n \n \trepository_format_precious_objects = candidate->precious_objects;\n-\trepository_format_partial_clone = candidate->partial_clone;\n \tstring_list_clear(&candidate->unknown_extensions, 0);\n \tif (!has_common) {\n \t\tif (candidate->is_bare != -1) {\ndiff --git a/sha1-file.c b/sha1-file.c\nindex c099f5584d..5e183d2cd4 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -32,6 +32,7 @@\n #include \"packfile.h\"\n #include \"fetch-object.h\"\n #include \"object-store.h\"\n+#include \"remote-odb.h\"\n \n /* The maximum size for an object header. */\n #define MAX_HEADER_LEN 32\n@@ -1309,15 +1310,16 @@ int oid_object_info_extended(struct repository *r, const struct object_id *oid,\n \t\t}\n \n \t\t/* Check if it is a missing object */\n-\t\tif (fetch_if_missing && repository_format_partial_clone &&\n+\t\tif (fetch_if_missing && has_remote_odb() &&\n \t\t    !already_retried && r == the_repository) {\n \t\t\t/*\n-\t\t\t * TODO Investigate checking fetch_object() return\n-\t\t\t * TODO value and stopping on error here.\n-\t\t\t * TODO Pass a repository struct through fetch_object,\n-\t\t\t * such that arbitrary repositories work.\n+\t\t\t * TODO Investigate checking remote_odb_get_direct()\n+\t\t\t * TODO return value and stopping on error here.\n+\t\t\t * TODO Pass a repository struct through\n+\t\t\t * remote_odb_get_direct(), such that arbitrary\n+\t\t\t * repositories work.\n \t\t\t */\n-\t\t\tfetch_object(repository_format_partial_clone, real->hash);\n+\t\t\tremote_odb_get_direct(real->hash);\n \t\t\talready_retried = 1;\n \t\t\tcontinue;\n \t\t}\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex 4984ca583d..71a9b9a3e8 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -23,10 +23,10 @@ promise_and_delete () {\n \tdelete_object repo \"$HASH\"\n }\n \n-test_expect_success 'extensions.partialclone without filter' '\n+test_expect_success 'promisor remote without filter' '\n \ttest_create_repo server &&\n \tgit clone --filter=\"blob:none\" \"file://$(pwd)/server\" client &&\n-\tgit -C client config --unset core.partialclonefilter &&\n+\tgit -C client config --unset odb.origin.partialclonefilter &&\n \tgit -C client fetch origin\n '\n \n@@ -51,7 +51,7 @@ test_expect_success 'missing reflog object, but promised by a commit, passes fsc\n \n \t# But with the extension, it succeeds\n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"arbitrary string\" &&\n \tgit -C repo fsck\n '\n \n@@ -74,7 +74,7 @@ test_expect_success 'missing reflog object, but promised by a tag, passes fsck'\n \tprintf \"$T\\n\" | pack_as_from_promisor &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"arbitrary string\" &&\n \tgit -C repo fsck\n '\n \n@@ -92,7 +92,7 @@ test_expect_success 'missing reflog object alone fails fsck, even with extension\n \tdelete_object repo \"$A\" &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"arbitrary string\" &&\n \ttest_must_fail git -C repo fsck\n '\n \n@@ -108,7 +108,7 @@ test_expect_success 'missing ref object, but promised, passes fsck' '\n \tpromise_and_delete \"$A\" &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"arbitrary string\" &&\n \tgit -C repo fsck\n '\n \n@@ -131,7 +131,7 @@ test_expect_success 'missing object, but promised, passes fsck' '\n \tpromise_and_delete \"$AT\" &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"arbitrary string\" &&\n \tgit -C repo fsck\n '\n \n@@ -144,7 +144,7 @@ test_expect_success 'missing CLI object, but promised, passes fsck' '\n \tpromise_and_delete \"$A\" &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"arbitrary string\" &&\n \tgit -C repo fsck \"$A\"\n '\n \n@@ -159,7 +159,7 @@ test_expect_success 'fetching of missing objects' '\n \trm -rf repo/.git/objects/* &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"origin\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"origin\" &&\n \tgit -C repo cat-file -p \"$HASH\" &&\n \n \t# Ensure that the .promisor file is written, and check that its\n@@ -180,7 +180,7 @@ test_expect_success 'rev-list stops traversal at missing and promised commit' '\n \tpromise_and_delete \"$FOO\" &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"arbitrary string\" &&\n \tgit -C repo rev-list --exclude-promisor-objects --objects bar >out &&\n \tgrep $(git -C repo rev-parse bar) out &&\n \t! grep $FOO out\n@@ -205,7 +205,7 @@ test_expect_success 'rev-list stops traversal at missing and promised tree' '\n \tpromise_and_delete \"$TREE2\" &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"arbitrary string\" &&\n \tgit -C repo rev-list --exclude-promisor-objects --objects HEAD >out &&\n \tgrep $(git -C repo rev-parse foo) out &&\n \t! grep $TREE out &&\n@@ -224,7 +224,7 @@ test_expect_success 'rev-list stops traversal at missing and promised blob' '\n \tpromise_and_delete \"$BLOB\" &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"arbitrary string\" &&\n \tgit -C repo rev-list --exclude-promisor-objects --objects HEAD >out &&\n \tgrep $(git -C repo rev-parse HEAD) out &&\n \t! grep $BLOB out\n@@ -243,7 +243,7 @@ test_expect_success 'rev-list stops traversal at promisor commit, tree, and blob\n \tprintf \"%s\\n%s\\n%s\\n\" $COMMIT $TREE $BLOB | pack_as_from_promisor &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"arbitrary string\" &&\n \tgit -C repo rev-list --exclude-promisor-objects --objects HEAD >out &&\n \t! grep $COMMIT out &&\n \t! grep $TREE out &&\n@@ -267,7 +267,7 @@ test_expect_success 'rev-list accepts missing and promised objects on command li\n \tpromise_and_delete $BLOB &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"arbitrary string\" &&\n \tgit -C repo rev-list --exclude-promisor-objects --objects \"$COMMIT\" \"$TREE\" \"$BLOB\"\n '\n \n@@ -280,7 +280,7 @@ test_expect_success 'gc does not repack promisor objects' '\n \tHASH=$(printf \"$TREE_HASH\\n\" | pack_as_from_promisor) &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"arbitrary string\" &&\n \tgit -C repo gc &&\n \n \t# Ensure that the promisor packfile still exists, and remove it\n@@ -304,7 +304,7 @@ test_expect_success 'gc stops traversal when a missing but promised object is re\n \tHASH=$(promise_and_delete $TREE_HASH) &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"arbitrary string\" &&\n \tgit -C repo gc &&\n \n \t# Ensure that the promisor packfile still exists, and remove it\n@@ -335,7 +335,7 @@ test_expect_success 'fetching of missing objects from an HTTP server' '\n \trm -rf repo/.git/objects/* &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"origin\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"origin\" &&\n \tgit -C repo cat-file -p \"$HASH\" &&\n \n \t# Ensure that the .promisor file is written, and check that its\ndiff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh\nindex ea6570e819..72654b24c4 100755\n--- a/t/t5500-fetch-pack.sh\n+++ b/t/t5500-fetch-pack.sh\n@@ -844,14 +844,14 @@ fetch_filter_blob_limit_zero () {\n \ttest_config -C \"$SERVER\" uploadpack.allowfilter 1 &&\n \n \tgit clone \"$URL\" client &&\n-\ttest_config -C client extensions.partialclone origin &&\n+\ttest_config -C client odb.magic.promisorRemote origin &&\n \n \ttest_commit -C \"$SERVER\" two &&\n \n \tgit -C client fetch --filter=blob:limit=0 origin HEAD:somewhere &&\n \n \t# Ensure that commit is fetched, but blob is not\n-\ttest_config -C client extensions.partialclone \"arbitrary string\" &&\n+\ttest_config -C client odb.magic.promisorRemote \"arbitrary string\" &&\n \tgit -C client cat-file -e $(git -C \"$SERVER\" rev-parse two) &&\n \ttest_must_fail git -C client cat-file -e $(git hash-object \"$SERVER/two.t\")\n }\ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex 0b62037744..022c65a7ae 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -647,7 +647,7 @@ partial_clone () {\n \tgit -C client fsck &&\n \n \t# Ensure that unneeded blobs are not inadvertently fetched.\n-\ttest_config -C client extensions.partialclone \"not a remote\" &&\n+\ttest_config -C client odb.origin.promisorRemote \"not a remote\" &&\n \ttest_must_fail git -C client cat-file -e \"$HASH1\" &&\n \n \t# But this blob was fetched, because clone performs an initial checkout\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex cee5565367..5ddd1e011c 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -40,7 +40,7 @@ test_expect_success 'do partial clone 1' '\n \t\t| sort >observed.oids &&\n \ttest_cmp expect_1.oids observed.oids &&\n \ttest \"$(git -C pc1 config --local core.repositoryformatversion)\" = \"1\" &&\n-\ttest \"$(git -C pc1 config --local extensions.partialclone)\" = \"origin\" &&\n+\ttest \"$(git -C pc1 config --local odb.origin.promisorRemote)\" = \"origin\" &&\n \ttest \"$(git -C pc1 config --local core.partialclonefilter)\" = \"blob:none\"\n '\n \ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex a4fe6508bd..0c47599568 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -285,7 +285,7 @@ test_expect_success 'partial fetch' '\n \trm -rf client \"$(pwd)/trace\" &&\n \tgit init client &&\n \tSERVER=\"file://$(pwd)/server\" &&\n-\ttest_config -C client extensions.partialClone \"$SERVER\" &&\n+\ttest_config -C client odb.magic.promisorRemote \"$SERVER\" &&\n \n \tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -C client -c protocol.version=2 \\\n \t\tfetch --filter=blob:none \"$SERVER\" master:refs/heads/other &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 3a85a02a77..e6ac3e6d10 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -17,6 +17,7 @@\n #include \"submodule-config.h\"\n #include \"fsmonitor.h\"\n #include \"fetch-object.h\"\n+#include \"remote-odb.h\"\n \n /*\n  * Error messages expected by scripts out of plumbing commands such as\n@@ -382,7 +383,7 @@ static int check_updates(struct unpack_trees_options *o)\n \t\tload_gitmodules_file(index, &state);\n \n \tenable_delayed_checkout(&state);\n-\tif (repository_format_partial_clone && o->update && !o->dry_run) {\n+\tif (has_remote_odb() && o->update && !o->dry_run) {\n \t\t/*\n \t\t * Prefetch the objects that are to be checked out in the loop\n \t\t * below.\n@@ -399,8 +400,7 @@ static int check_updates(struct unpack_trees_options *o)\n \t\t\t}\n \t\t}\n \t\tif (to_fetch.nr)\n-\t\t\tfetch_objects(repository_format_partial_clone,\n-\t\t\t\t      &to_fetch);\n+\t\t\tremote_odb_get_many_direct(&to_fetch);\n \t\tfetch_if_missing = fetch_if_missing_store;\n \t\toid_array_clear(&to_fetch);\n \t}\n-- \n2.18.0.171.ge260395bfe\n\n"},{"id":"352515","messageId":"20180713174959.16748-8-chriscool@tuxfamily.org","threadId":"48881","inReplyTo":"20180713174959.16748-1-chriscool@tuxfamily.org","subject":"[PATCH v3 7/9] Use odb.origin.partialclonefilter instead of core.partialclonefilter","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-07-13T17:49:57Z","receivedAt":"2018-07-13T17:50:33Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: Christian Couder <christian.couder@gmail.com>\n\nLet's make the partial clone filter specific to one odb\ninstead of general to all the odbs.\n\nThis makes it possible to have different partial clone\nfilters for different odbs.\n\nSigned-off-by: Christian Couder <chriscool@tuxfamily.org>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n builtin/fetch.c               |  2 +-\n list-objects-filter-options.c | 28 ++++++++++++++++------------\n list-objects-filter-options.h |  3 ++-\n odb-helper.h                  |  1 +\n remote-odb.c                  |  2 ++\n t/t5616-partial-clone.sh      |  2 +-\n 6 files changed, 23 insertions(+), 15 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 1fd4dfd024..52ec2fd200 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1343,7 +1343,7 @@ static inline void fetch_one_setup_partial(struct remote *remote)\n \t * the config.\n \t */\n \tif (!filter_options.choice)\n-\t\tpartial_clone_get_default_filter_spec(&filter_options);\n+\t\tpartial_clone_get_default_filter_spec(&filter_options, remote->name);\n \treturn;\n }\n \ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex 60452c8f36..755a793664 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -7,6 +7,7 @@\n #include \"list-objects-filter.h\"\n #include \"list-objects-filter-options.h\"\n #include \"remote-odb.h\"\n+#include \"odb-helper.h\"\n \n /*\n  * Parse value of the argument to the \"filter\" keyword.\n@@ -29,6 +30,9 @@ static int gently_parse_list_objects_filter(\n {\n \tconst char *v0;\n \n+\tif (!arg)\n+\t\treturn 0;\n+\n \tif (filter_options->choice) {\n \t\tif (errbuf) {\n \t\t\tstrbuf_init(errbuf, 0);\n@@ -116,6 +120,7 @@ void partial_clone_register(\n \tconst struct list_objects_filter_options *filter_options)\n {\n \tchar *cfg_name;\n+\tchar *filter_name;\n \n \t/* Check if it is already registered */\n \tif (find_odb_helper(remote))\n@@ -131,27 +136,26 @@ void partial_clone_register(\n \t/*\n \t * Record the initial filter-spec in the config as\n \t * the default for subsequent fetches from this remote.\n-\t *\n-\t * TODO: move core.partialclonefilter into odb.<name>\n \t */\n-\tcore_partial_clone_filter_default =\n-\t\txstrdup(filter_options->filter_spec);\n-\tgit_config_set(\"core.partialclonefilter\",\n-\t\t       core_partial_clone_filter_default);\n+\tfilter_name = xstrfmt(\"odb.%s.partialclonefilter\", remote);\n+\tgit_config_set(filter_name, filter_options->filter_spec);\n+\tfree(filter_name);\n \n \t/* Make sure the config info are reset */\n \tremote_odb_reinit();\n }\n \n void partial_clone_get_default_filter_spec(\n-\tstruct list_objects_filter_options *filter_options)\n+\tstruct list_objects_filter_options *filter_options,\n+\tconst char *remote)\n {\n+\tstruct odb_helper *helper = find_odb_helper(remote);\n+\n \t/*\n \t * Parse default value, but silently ignore it if it is invalid.\n \t */\n-\tif (!core_partial_clone_filter_default)\n-\t\treturn;\n-\tgently_parse_list_objects_filter(filter_options,\n-\t\t\t\t\t core_partial_clone_filter_default,\n-\t\t\t\t\t NULL);\n+\tif (helper)\n+\t\tgently_parse_list_objects_filter(filter_options,\n+\t\t\t\t\t\t helper->partial_clone_filter,\n+\t\t\t\t\t\t NULL);\n }\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 0000a61f82..12ceef3230 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -74,6 +74,7 @@ void partial_clone_register(\n \tconst char *remote,\n \tconst struct list_objects_filter_options *filter_options);\n void partial_clone_get_default_filter_spec(\n-\tstruct list_objects_filter_options *filter_options);\n+\tstruct list_objects_filter_options *filter_options,\n+\tconst char *remote);\n \n #endif /* LIST_OBJECTS_FILTER_OPTIONS_H */\ndiff --git a/odb-helper.h b/odb-helper.h\nindex 154ce4c7e4..4af75ef55c 100644\n--- a/odb-helper.h\n+++ b/odb-helper.h\n@@ -10,6 +10,7 @@\n struct odb_helper {\n \tconst char *name;                 /* odb.<NAME>.* */\n \tconst char *remote;               /* odb.<NAME>.promisorRemote */\n+\tconst char *partial_clone_filter; /* odb.<NAME>.partialCloneFilter */\n \n \tstruct odb_helper *next;\n };\ndiff --git a/remote-odb.c b/remote-odb.c\nindex 53a72b4ba5..847a865057 100644\n--- a/remote-odb.c\n+++ b/remote-odb.c\n@@ -60,6 +60,8 @@ static int remote_odb_config(const char *var, const char *value, void *data)\n \n \t\treturn 0;\n \t}\n+\tif (!strcmp(subkey, \"partialclonefilter\"))\n+\t\treturn git_config_string(&o->partial_clone_filter, var, value);\n \n \treturn 0;\n }\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex 5ddd1e011c..4231121181 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -41,7 +41,7 @@ test_expect_success 'do partial clone 1' '\n \ttest_cmp expect_1.oids observed.oids &&\n \ttest \"$(git -C pc1 config --local core.repositoryformatversion)\" = \"1\" &&\n \ttest \"$(git -C pc1 config --local odb.origin.promisorRemote)\" = \"origin\" &&\n-\ttest \"$(git -C pc1 config --local core.partialclonefilter)\" = \"blob:none\"\n+\ttest \"$(git -C pc1 config --local odb.origin.partialclonefilter)\" = \"blob:none\"\n '\n \n # checkout master to force dynamic object fetch of blobs at HEAD.\n-- \n2.18.0.171.ge260395bfe\n\n"},{"id":"352516","messageId":"20180713174959.16748-9-chriscool@tuxfamily.org","threadId":"48881","inReplyTo":"20180713174959.16748-1-chriscool@tuxfamily.org","subject":"[PATCH v3 8/9] t0410: test fetching from many promisor remotes","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-07-13T17:49:58Z","receivedAt":"2018-07-13T17:50:33Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: Christian Couder <christian.couder@gmail.com>\n\nSigned-off-by: Christian Couder <chriscool@tuxfamily.org>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n t/t0410-partial-clone.sh | 24 +++++++++++++++++++++++-\n 1 file changed, 23 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex 71a9b9a3e8..9d513ebf57 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -170,8 +170,30 @@ test_expect_success 'fetching of missing objects' '\n \tgit verify-pack --verbose \"$IDX\" | grep \"$HASH\"\n '\n \n+test_expect_success 'fetching of missing objects from another odb remote' '\n+\tgit clone \"file://$(pwd)/server\" server2 &&\n+\ttest_commit -C server2 bar &&\n+\tgit -C server2 repack -a -d --write-bitmap-index &&\n+\tHASH2=$(git -C server2 rev-parse bar) &&\n+\n+\tgit -C repo remote add server2 \"file://$(pwd)/server2\" &&\n+\tgit -C repo config odb.magic2.promisorRemote server2 &&\n+\tgit -C repo cat-file -p \"$HASH2\" &&\n+\n+\tgit -C repo fetch server2 &&\n+\trm -rf repo/.git/objects/* &&\n+\tgit -C repo cat-file -p \"$HASH2\" &&\n+\n+\t# Ensure that the .promisor file is written, and check that its\n+\t# associated packfile contains the object\n+\tls repo/.git/objects/pack/pack-*.promisor >promisorlist &&\n+\ttest_line_count = 1 promisorlist &&\n+\tIDX=$(cat promisorlist | sed \"s/promisor$/idx/\") &&\n+\tgit verify-pack --verbose \"$IDX\" | grep \"$HASH2\"\n+'\n+\n test_expect_success 'rev-list stops traversal at missing and promised commit' '\n-\trm -rf repo &&\n+\trm -rf repo server server2 &&\n \ttest_create_repo repo &&\n \ttest_commit -C repo foo &&\n \ttest_commit -C repo bar &&\n-- \n2.18.0.171.ge260395bfe\n\n"},{"id":"352517","messageId":"20180713174959.16748-10-chriscool@tuxfamily.org","threadId":"48881","inReplyTo":"20180713174959.16748-1-chriscool@tuxfamily.org","subject":"[PATCH v3 9/9] Documentation/config: add odb.<name>.promisorRemote","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-07-13T17:49:59Z","receivedAt":"2018-07-13T17:50:35Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: Christian Couder <christian.couder@gmail.com>\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n Documentation/config.txt | 5 +++++\n 1 file changed, 5 insertions(+)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 1cc18a828c..066858886b 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2505,6 +2505,11 @@ This setting can be overridden with the `GIT_NOTES_REWRITE_REF`\n environment variable, which must be a colon separated list of refs or\n globs.\n \n+odb.<name>.promisorRemote::\n+\tThe name of a promisor remote. For now promisor remotes are\n+\tthe only kind of remote object database (odb) that is\n+\tsupported.\n+\n pack.window::\n \tThe size of the window used by linkgit:git-pack-objects[1] when no\n \twindow size is given on the command line. Defaults to 10.\n-- \n2.18.0.171.ge260395bfe\n\n"},{"id":"353301","messageId":"CACsJy8B+NBgKFZbrN2etgkecv6yH4NGWogVqzK0K=01MSO2naQ@mail.gmail.com","threadId":"48881","inReplyTo":"20180713174959.16748-7-chriscool@tuxfamily.org","subject":"Re: [PATCH v3 6/9] Use remote_odb_get_direct() and has_remote_odb()","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-07-22T09:05:01Z","receivedAt":"2018-07-22T09:05:29Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Jul 13, 2018 at 7:50 PM Christian Couder\n<christian.couder@gmail.com> wrote:\n> @@ -477,8 +478,8 @@ static int batch_objects(struct batch_options *opt)\n>\n>                 for_each_loose_object(batch_loose_object, &sa, 0);\n>                 for_each_packed_object(batch_packed_object, &sa, 0);\n> -               if (repository_format_partial_clone)\n> -                       warning(\"This repository has extensions.partialClone set. Some objects may not be loaded.\");\n> +               if (has_remote_odb())\n> +                       warning(\"This repository uses an odb. Some objects may not be loaded.\");\n\nI can see the string is not marked for translation before, but since\nyou're updating it, please consider _() if needed. Maybe also say\n\"remote object database\" instead of odb\n-- \nDuy\n"}]}