{"thread":{"id":"48083","subject":"[PATCH v2 00/36] Promisor remotes and external ODB support","startedAt":"2018-03-19T13:32:17Z","lastAt":"2018-03-19T19:13:58Z","messageCount":38,"participants":["Christian Couder","Junio C Hamano"],"isPatch":true,"patchVersion":2,"patchTotal":36},"messages":[{"id":"342151","messageId":"20180319133147.15413-1-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":null,"subject":"[PATCH v2 00/36] Promisor remotes and external ODB support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:11Z","receivedAt":"2018-03-19T13:32:17Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This is a follow up from an early patch series at the beginning of the\nyear that started to integrate the jh/fsck-promisors patch series with\nthe external odb patch series.\n\nA lot of things are different because the jh/fsck-promisors and\njh/partial-clone have been merged into master since the v1. So the\nintegration is much more complete now (though not fully complete), and\nthis integration happens quite early in the patch series.\n\nThis integration makes it possible to have many promisor and partial\nclone remotes (instead of just one) with possibly different partial\nclone filters, though that is not tested yet.\n\nI am not sure that the \"external odb\" name is still the best, as the\npromisor remotes are not quite external. So I am open to suggestions\nabout a new name.\n\nThis patch series does not include the last part of the original\nexternal odb series which was about adding an `--inital-refspec`\noption to `git clone`, as this might not be needed anymore and this\ncurrent series is long and complex enough.\n\nHigh level overview of this patch series\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nIn general the external odb tests are now numbered 05XX instead of\n04XX, so that they don't conflict with promisor objects/partial clone\ntests. Also there are a lot of changes since v1, so I don't think an\ninterdiff would be useful.\n\n - Patches 1/36 to 12/36:\n \nThese patches implement the external odb infrastructure as well as the\nsupport for promisor objects and partial clone.\n\n - Patches 13/36 and 14/36:\n\nThese patches move over the promisor objects and partial clone code to\nuse the external odb mechanism. The result of 13/36 is that instead of\nthe \"extensions.partialclone\" config variable, a\n\"odb..promisorRemote\" config variable is now used. The result of\n14/36 is that instead of the \"core.partialclonefilter\" config\nvariable, a \"odb..partialclonefilter\" config variable is now\nused.\n\n - Patches 15/36 to 36/36:\n\nThese patches implement the rest of the external odb mechanism. Patch\n36/36 is a documentation patch that explains how this mechanism work,\nthough this patch needs to be updated.\n\nFuture work\n~~~~~~~~~~~\n\n - add tests that show that one can now use more than one promisor remote\n - add fsck, gc and repack tests for script and process modes\n - see if partial clone filters could be used for other modes\n - update documentation patch\n - fix memory leaks\n - take another look at prepare_helper_command() [1]\n - take a look at changing the order of the have object lines [1]\n - take a look at what happens when a server creates a pack [2]\n - add tests with different kinds of external odbs\n\n[1] https://public-inbox.org/git/ebf67bcc-3e17-3fda-9f56-dd152e7bf3af@jeffhostetler.com/\n[2] https://public-inbox.org/git/8de3597a-01bd-a721-fffb-20769701d0af@jeffhostetler.com/\n\nLinks\n~~~~~\n\nThis patch series on GitHub:\n\nhttps://github.com/chriscool/git/commits/gl-small-promisor-external-odb-next\n\nPeff started to work on external odbs some years ago:\n\nhttp://thread.gmane.org/gmane.comp.version-control.git/206886/focus=207040\nhttp://thread.gmane.org/gmane.comp.version-control.git/247171\nhttp://thread.gmane.org/gmane.comp.version-control.git/202902/focus=203020\n\nHis work, which is not compile-tested any more:\n\nhttps://github.com/peff/git/commits/jk/external-odb-wip\n\nInitial discussions about external odbs:\n\nhttp://thread.gmane.org/gmane.comp.version-control.git/288151/focus=295160\n\nVersion 1, 2, 3, 4, 5 and 6 of the external odbs series:\n\nhttps://public-inbox.org/git/20160613085546.11784-1-chriscool@tuxfamily.org/\nhttps://public-inbox.org/git/20160628181933.24620-1-chriscool@tuxfamily.org/\nhttps://public-inbox.org/git/20161130210420.15982-1-chriscool@tuxfamily.org/\nhttps://public-inbox.org/git/20170620075523.26961-1-chriscool@tuxfamily.org/\nhttps://public-inbox.org/git/20170803091926.1755-1-chriscool@tuxfamily.org/\nhttps://public-inbox.org/git/20170916080731.13925-1-chriscool@tuxfamily.org/\n\nSome of the discussions related to Ben Peart's work that is used by\nthis series:\n\nhttps://public-inbox.org/git/20170113155253.1644-1-benpeart@microsoft.com/\nhttps://public-inbox.org/git/20170322165220.5660-1-benpeart@microsoft.com/\nhttps://public-inbox.org/git/20170714132651.170708-1-benpeart@microsoft.com/\n\nVersion 1, 2, 3, 4, 5 and 6 of the external odbs series on GitHub:\n\nhttps://github.com/chriscool/git/commits/gl-external-odb12\nhttps://github.com/chriscool/git/commits/gl-external-odb22\nhttps://github.com/chriscool/git/commits/gl-external-odb61\nhttps://github.com/chriscool/git/commits/gl-external-odb239\nhttps://github.com/chriscool/git/commits/gl-external-odb373\nhttps://github.com/chriscool/git/commits/gl-external-odb411\n\nVersion 1 of this \"Promisor remotes and external ODB support\" series:\n\nhttps://public-inbox.org/git/20180103163403.11303-1-chriscool@tuxfamily.org/\n\nVersion 1 of this \"Promisor remotes and external ODB support\" series on GitHub:\n\nhttps://github.com/chriscool/git/commits/gl-small-promisor-external-odb12\n\n\nBen Peart (1):\n Add t0550 to test 'get_direct' mechanism\n\nChristian Couder (35):\n Add initial external odb support\n sha1_file: add prepare_external_alt_odb()\n Add GIT_NO_EXTERNAL_ODB env variable\n external-odb: add has_external_odb()\n external-odb: implement external_odb_get_direct\n sha1_file: prepare for external odbs\n odb-helper: add 'enum odb_helper_type'\n external-odb: add external_odb_reinit()\n external-odb: add script mode support\n odb-helper: add 'script_mode' to 'struct odb_helper'\n odb-helper: add odb_helper_init() to send 'init' instruction\n external-odb: add external_odb_get_many_direct()\n Use external_odb_get_direct() and has_external_odb()\n Use odb.origin.partialclonefilter instead of core.partialclonefilter\n external-odb: add 'get_direct' support\n t0500: add 'put_raw_obj' instruction to odb-helper script\n external odb: add 'put_raw_obj' support\n external-odb: accept only blobs for now\n t0500: add test for external odb write support\n Add t0510 to test external ODB transfer\n lib-httpd: pass config file to start_httpd()\n lib-httpd: add upload.sh\n lib-httpd: add list.sh\n lib-httpd: add apache-e-odb.conf\n odb-helper: add odb_helper_get_raw_object()\n pack-objects: don't pack objects in external odbs\n Add t0520 to test transfer to HTTP external odb\n odb-helper: add init_object_process()\n Add t0560 to test passing git objects\n odb-helper: add put_object_process()\n Add t0570 to test passing raw objects\n odb-helper: add have_object_process()\n Add t0580 to test \"have\" capability and raw objects\n external-odb: use 'odb=magic' attribute to mark odb blobs\n Add Documentation/technical/external-odb.txt\n\n Documentation/technical/external-odb.txt | 342 +++++++\n Makefile | 2 +\n builtin/cat-file.c | 5 +-\n builtin/fetch.c | 13 +-\n builtin/gc.c | 3 +-\n builtin/pack-objects.c | 4 +\n builtin/repack.c | 3 +-\n cache.h | 20 +-\n connected.c | 3 +-\n environment.c | 5 +-\n external-odb.c | 250 +++++\n external-odb.h | 25 +\n list-objects-filter-options.c | 46 +-\n list-objects-filter-options.h | 3 +-\n odb-helper.c | 1109 ++++++++++++++++++++++\n odb-helper.h | 49 +\n packfile.c | 3 +-\n setup.c | 7 +-\n sha1_file.c | 120 ++-\n t/lib-httpd.sh | 8 +-\n t/lib-httpd/apache-e-odb.conf | 214 +++++\n t/lib-httpd/list.sh | 41 +\n t/lib-httpd/upload.sh | 45 +\n t/t0410-partial-clone.sh | 30 +-\n t/t0500-external-odb.sh | 83 ++\n t/t0510-transfer-e-odb.sh | 147 +++\n t/t0520-transfer-http-e-odb.sh | 145 +++\n t/t0550-read-object.sh | 28 +\n t/t0550/read-object | 68 ++\n t/t0560-read-object-git.sh | 28 +\n t/t0560/read-object-git | 78 ++\n t/t0570-read-object-http-e-odb.sh | 112 +++\n t/t0570/read-object-plain | 83 ++\n t/t0580-read-object-have-http-e-odb.sh | 112 +++\n t/t0580/read-object-plain-have | 103 ++\n t/t5500-fetch-pack.sh | 4 +-\n t/t5601-clone.sh | 2 +-\n t/t5616-partial-clone.sh | 4 +-\n unpack-trees.c | 6 +-\n 39 files changed, 3254 insertions(+), 99 deletions(-)\n create mode 100644 Documentation/technical/external-odb.txt\n create mode 100644 external-odb.c\n create mode 100644 external-odb.h\n create mode 100644 odb-helper.c\n create mode 100644 odb-helper.h\n create mode 100644 t/lib-httpd/apache-e-odb.conf\n create mode 100644 t/lib-httpd/list.sh\n create mode 100644 t/lib-httpd/upload.sh\n create mode 100755 t/t0500-external-odb.sh\n create mode 100755 t/t0510-transfer-e-odb.sh\n create mode 100755 t/t0520-transfer-http-e-odb.sh\n create mode 100755 t/t0550-read-object.sh\n create mode 100755 t/t0550/read-object\n create mode 100755 t/t0560-read-object-git.sh\n create mode 100755 t/t0560/read-object-git\n create mode 100755 t/t0570-read-object-http-e-odb.sh\n create mode 100755 t/t0570/read-object-plain\n create mode 100755 t/t0580-read-object-have-http-e-odb.sh\n create mode 100755 t/t0580/read-object-plain-have\n\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342152","messageId":"20180319133147.15413-2-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 01/36] Add initial external odb support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:12Z","receivedAt":"2018-03-19T13:32:21Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"The external-odb.{c,h} files will contain the functions\nthat are called by the rest of Git mostly from\n\"sha1_file.c\" to access the objects managed by the\nexternal odbs.\n\nThe odb-helper.{c,h} files will contain the functions to\nactually implement communication with either the internal\nfunctions or the external scripts or processes that will\nmanage and provide external git objects.\n\nFor now only infrastructure to create helpers from the\nconfig and to manage a cache for the 'have' command is\nimplemented.\n\nHelped-by: Jeff King \nSigned-off-by: Christian Couder \n---\n Makefile | 2 ++\n external-odb.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++\n external-odb.h | 7 +++++\n odb-helper.c | 54 +++++++++++++++++++++++++++++++++++++\n odb-helper.h | 24 +++++++++++++++++\n 5 files changed, 159 insertions(+)\n create mode 100644 external-odb.c\n create mode 100644 external-odb.h\n create mode 100644 odb-helper.c\n create mode 100644 odb-helper.h\n\ndiff --git a/Makefile b/Makefile\nindex a1d8775adb..11dd620155 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -808,6 +808,7 @@ LIB_OBJS += ewah/ewah_bitmap.o\n LIB_OBJS += ewah/ewah_io.o\n LIB_OBJS += ewah/ewah_rlw.o\n LIB_OBJS += exec_cmd.o\n+LIB_OBJS += external-odb.o\n LIB_OBJS += fetch-object.o\n LIB_OBJS += fetch-pack.o\n LIB_OBJS += fsck.o\n@@ -843,6 +844,7 @@ LIB_OBJS += notes-cache.o\n LIB_OBJS += notes-merge.o\n LIB_OBJS += notes-utils.o\n LIB_OBJS += object.o\n+LIB_OBJS += odb-helper.o\n LIB_OBJS += oidmap.o\n LIB_OBJS += oidset.o\n LIB_OBJS += packfile.o\ndiff --git a/external-odb.c b/external-odb.c\nnew file mode 100644\nindex 0000000000..f3ea491333\n--- /dev/null\n+++ b/external-odb.c\n@@ -0,0 +1,72 @@\n+#include \"cache.h\"\n+#include \"external-odb.h\"\n+#include \"odb-helper.h\"\n+#include \"config.h\"\n+\n+static struct odb_helper *helpers;\n+static struct odb_helper **helpers_tail = &helpers;\n+\n+static struct odb_helper *find_or_create_helper(const char *name, int len)\n+{\n+\tstruct odb_helper *o;\n+\n+\tfor (o = helpers; o; o = o->next)\n+\t\tif (!strncmp(o->name, name, len) && !o->name[len])\n+\t\t\treturn o;\n+\n+\to = odb_helper_new(name, len);\n+\t*helpers_tail = o;\n+\thelpers_tail = &o->next;\n+\n+\treturn o;\n+}\n+\n+static int external_odb_config(const char *var, const char *value, void *data)\n+{\n+\tstruct odb_helper *o;\n+\tconst char *name;\n+\tint namelen;\n+\tconst char *subkey;\n+\n+\tif (parse_config_key(var, \"odb\", &name, &namelen, &subkey) < 0)\n+\t\treturn 0;\n+\n+\to = find_or_create_helper(name, namelen);\n+\n+\tif (!strcmp(subkey, \"promisorremote\"))\n+\t\treturn git_config_string(&o->dealer, var, value);\n+\n+\treturn 0;\n+}\n+\n+static void external_odb_init(void)\n+{\n+\tstatic int initialized;\n+\n+\tif (initialized)\n+\t\treturn;\n+\tinitialized = 1;\n+\n+\tgit_config(external_odb_config, NULL);\n+}\n+\n+const char *external_odb_root(void)\n+{\n+\tstatic const char *root;\n+\tif (!root)\n+\t\troot = git_pathdup(\"objects/external\");\n+\treturn root;\n+}\n+\n+int external_odb_has_object(const unsigned char *sha1)\n+{\n+\tstruct odb_helper *o;\n+\n+\texternal_odb_init();\n+\n+\tfor (o = helpers; o; o = o->next)\n+\t\tif (odb_helper_has_object(o, sha1))\n+\t\t\treturn 1;\n+\treturn 0;\n+}\n+\ndiff --git a/external-odb.h b/external-odb.h\nnew file mode 100644\nindex 0000000000..ae2b228792\n--- /dev/null\n+++ b/external-odb.h\n@@ -0,0 +1,7 @@\n+#ifndef EXTERNAL_ODB_H\n+#define EXTERNAL_ODB_H\n+\n+extern const char *external_odb_root(void);\n+extern int external_odb_has_object(const unsigned char *sha1);\n+\n+#endif /* EXTERNAL_ODB_H */\ndiff --git a/odb-helper.c b/odb-helper.c\nnew file mode 100644\nindex 0000000000..67196e6317\n--- /dev/null\n+++ b/odb-helper.c\n@@ -0,0 +1,54 @@\n+#include \"cache.h\"\n+#include \"object.h\"\n+#include \"argv-array.h\"\n+#include \"odb-helper.h\"\n+#include \"run-command.h\"\n+#include \"sha1-lookup.h\"\n+\n+struct odb_helper *odb_helper_new(const char *name, int namelen)\n+{\n+\tstruct odb_helper *o;\n+\n+\to = xcalloc(1, sizeof(*o));\n+\to->name = xmemdupz(name, namelen);\n+\n+\treturn o;\n+}\n+\n+struct odb_helper_cmd {\n+\tstruct argv_array argv;\n+\tstruct child_process child;\n+};\n+\n+static void odb_helper_load_have(struct odb_helper *o)\n+{\n+\tif (o->have_valid)\n+\t\treturn;\n+\to->have_valid = 1;\n+\n+\t/* TODO */\n+}\n+\n+static const unsigned char *have_sha1_access(size_t index, void *table)\n+{\n+\tstruct odb_helper_object *have = table;\n+\treturn have[index].oid.hash;\n+}\n+\n+static struct odb_helper_object *odb_helper_lookup(struct odb_helper *o,\n+\t\t\t\t\t\t const unsigned char *sha1)\n+{\n+\tint idx;\n+\n+\todb_helper_load_have(o);\n+\tidx = sha1_pos(sha1, o->have, o->have_nr, have_sha1_access);\n+\tif (idx < 0)\n+\t\treturn NULL;\n+\treturn &o->have[idx];\n+}\n+\n+int odb_helper_has_object(struct odb_helper *o, const unsigned char *sha1)\n+{\n+\treturn !!odb_helper_lookup(o, sha1);\n+}\n+\ndiff --git a/odb-helper.h b/odb-helper.h\nnew file mode 100644\nindex 0000000000..e3c9c1cfe4\n--- /dev/null\n+++ b/odb-helper.h\n@@ -0,0 +1,24 @@\n+#ifndef ODB_HELPER_H\n+#define ODB_HELPER_H\n+\n+struct odb_helper {\n+\tconst char *name;\n+\tconst char *dealer;\n+\n+\tstruct odb_helper_object {\n+\t\tstruct object_id oid;\n+\t\tunsigned long size;\n+\t\tenum object_type type;\n+\t} *have;\n+\tint have_nr;\n+\tint have_alloc;\n+\tint have_valid;\n+\n+\tstruct odb_helper *next;\n+};\n+\n+extern struct odb_helper *odb_helper_new(const char *name, int namelen);\n+extern int odb_helper_has_object(struct odb_helper *o,\n+\t\t\t\t const unsigned char *sha1);\n+\n+#endif /* ODB_HELPER_H */\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342153","messageId":"20180319133147.15413-3-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 02/36] sha1_file: add prepare_external_alt_odb()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:13Z","receivedAt":"2018-03-19T13:32:24Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This new function adds the external odb cache to all\nthe other odbs.\n\nSigned-off-by: Christian Couder \n---\n cache.h | 1 +\n sha1_file.c | 17 +++++++++++++++++\n 2 files changed, 18 insertions(+)\n\ndiff --git a/cache.h b/cache.h\nindex d06932ed0b..2ac7d63e5c 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1583,6 +1583,7 @@ extern void prepare_alt_odb(void);\n extern char *compute_alternate_path(const char *path, struct strbuf *err);\n typedef int alt_odb_fn(struct alternate_object_database *, void *);\n extern int foreach_alt_odb(alt_odb_fn, void*);\n+extern void prepare_external_alt_odb(void);\n \n /*\n * Allocate a \"struct alternate_object_database\" but do _not_ actually\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 1b94f39c4c..3f00fc716e 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -29,6 +29,7 @@\n #include \"quote.h\"\n #include \"packfile.h\"\n #include \"fetch-object.h\"\n+#include \"external-odb.h\"\n \n const unsigned char null_sha1[GIT_MAX_RAWSZ];\n const struct object_id null_oid;\n@@ -663,6 +664,21 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)\n \treturn r;\n }\n \n+void prepare_external_alt_odb(void)\n+{\n+\tstatic int linked_external;\n+\tconst char *path;\n+\n+\tif (linked_external)\n+\t\treturn;\n+\n+\tpath = external_odb_root();\n+\tif (!access(path, F_OK)) {\n+\t\tlink_alt_odb_entry(path, NULL, 0, \"\");\n+\t\tlinked_external = 1;\n+\t}\n+}\n+\n void prepare_alt_odb(void)\n {\n \tconst char *alt;\n@@ -676,6 +692,7 @@ void prepare_alt_odb(void)\n \tlink_alt_odb_entries(alt, PATH_SEP, NULL, 0);\n \n \tread_info_alternates(get_object_directory(), 0);\n+\tprepare_external_alt_odb();\n }\n \n /* Returns 1 if we have successfully freshened the file, 0 otherwise. */\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342154","messageId":"20180319133147.15413-7-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 06/36] sha1_file: prepare for external odbs","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:17Z","receivedAt":"2018-03-19T13:32:27Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"In the following commits we will need some functions that were\ninternal to sha1_file.c, so let's first make them non static\nand declare them in \"cache.h\". While at it, let's rename\n'create_tmpfile()' to 'create_object_tmpfile()' to make its\nname less generic.\n\nLet's also split out 'sha1_file_name_alt()' from\n'sha1_file_name()' and 'open_sha1_file_alt()' from\n'open_sha1_file()', as we will need both of these new\nfunctions too.\n\nHelped-by: Jeff King \nSigned-off-by: Christian Couder \n---\n cache.h | 8 ++++++++\n sha1_file.c | 49 ++++++++++++++++++++++++++++++-------------------\n 2 files changed, 38 insertions(+), 19 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 4aa0129cf8..e6fd9b97ec 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -951,6 +951,12 @@ extern void check_repository_format(void);\n */\n extern void sha1_file_name(struct strbuf *buf, const unsigned char *sha1);\n \n+/*\n+ * Like sha1_file_name, but put in `buf` the filename within a\n+ * specific alternate object directory.\n+ */\n+extern void sha1_file_name_alt(struct strbuf *buf, const char *objdir, const unsigned char *sha1);\n+\n /*\n * Return an abbreviated sha1 unique within this repository's object database.\n * The result will be at least `len` characters long, and will be NUL\n@@ -1247,6 +1253,8 @@ extern int parse_sha1_header(const char *hdr, unsigned long *sizep);\n \n extern int check_sha1_signature(const unsigned char *sha1, void *buf, unsigned long size, const char *type);\n \n+extern int create_object_tmpfile(struct strbuf *tmp, const char *filename);\n+extern void close_sha1_file(int fd);\n extern int finalize_object_file(const char *tmpfile, const char *filename);\n \n /*\ndiff --git a/sha1_file.c b/sha1_file.c\nindex f2e078e840..c460eb2658 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -320,13 +320,18 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)\n \t}\n }\n \n-void sha1_file_name(struct strbuf *buf, const unsigned char *sha1)\n+void sha1_file_name_alt(struct strbuf *buf, const char *objdir, const unsigned char *sha1)\n {\n-\tstrbuf_addstr(buf, get_object_directory());\n+\tstrbuf_addstr(buf, objdir);\n \tstrbuf_addch(buf, '/');\n \tfill_sha1_path(buf, sha1);\n }\n \n+void sha1_file_name(struct strbuf *buf, const unsigned char *sha1)\n+{\n+\treturn sha1_file_name_alt(buf, get_object_directory(), sha1);\n+}\n+\n struct strbuf *alt_scratch_buf(struct alternate_object_database *alt)\n {\n \tstrbuf_setlen(&alt->scratch, alt->base_len);\n@@ -905,6 +910,25 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,\n \treturn -1;\n }\n \n+static int open_sha1_file_alt(const unsigned char *sha1, const char **path)\n+{\n+\tstruct alternate_object_database *alt;\n+\tint most_interesting_errno = errno;\n+\n+\tprepare_alt_odb();\n+\tfor (alt = alt_odb_list; alt; alt = alt->next) {\n+\t\tint fd;\n+\t\t*path = alt_sha1_path(alt, sha1);\n+\t\tfd = git_open(*path);\n+\t\tif (fd >= 0)\n+\t\t\treturn fd;\n+\t\tif (most_interesting_errno == ENOENT)\n+\t\t\tmost_interesting_errno = errno;\n+\t}\n+\terrno = most_interesting_errno;\n+\treturn -1;\n+}\n+\n /*\n * Like stat_sha1_file(), but actually open the object and return the\n * descriptor. See the caveats on the \"path\" parameter above.\n@@ -912,8 +936,6 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,\n static int open_sha1_file(const unsigned char *sha1, const char **path)\n {\n \tint fd;\n-\tstruct alternate_object_database *alt;\n-\tint most_interesting_errno;\n \tstatic struct strbuf buf = STRBUF_INIT;\n \n \tstrbuf_reset(&buf);\n@@ -923,19 +945,8 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)\n \tfd = git_open(*path);\n \tif (fd >= 0)\n \t\treturn fd;\n-\tmost_interesting_errno = errno;\n \n-\tprepare_alt_odb();\n-\tfor (alt = alt_odb_list; alt; alt = alt->next) {\n-\t\t*path = alt_sha1_path(alt, sha1);\n-\t\tfd = git_open(*path);\n-\t\tif (fd >= 0)\n-\t\t\treturn fd;\n-\t\tif (most_interesting_errno == ENOENT)\n-\t\t\tmost_interesting_errno = errno;\n-\t}\n-\terrno = most_interesting_errno;\n-\treturn -1;\n+\treturn open_sha1_file_alt(sha1, path);\n }\n \n /*\n@@ -1537,7 +1548,7 @@ int hash_object_file(const void *buf, unsigned long len, const char *type,\n }\n \n /* Finalize a file on disk, and close it. */\n-static void close_sha1_file(int fd)\n+void close_sha1_file(int fd)\n {\n \tif (fsync_object_files)\n \t\tfsync_or_die(fd, \"sha1 file\");\n@@ -1561,7 +1572,7 @@ static inline int directory_size(const char *filename)\n * We want to avoid cross-directory filename renames, because those\n * can have problems on various filesystems (FAT, NFS, Coda).\n */\n-static int create_tmpfile(struct strbuf *tmp, const char *filename)\n+int create_object_tmpfile(struct strbuf *tmp, const char *filename)\n {\n \tint fd, dirlen = directory_size(filename);\n \n@@ -1605,7 +1616,7 @@ static int write_loose_object(const struct object_id *oid, char *hdr,\n \tstrbuf_reset(&filename);\n \tsha1_file_name(&filename, oid->hash);\n \n-\tfd = create_tmpfile(&tmp_file, filename.buf);\n+\tfd = create_object_tmpfile(&tmp_file, filename.buf);\n \tif (fd < 0) {\n \t\tif (errno == EACCES)\n \t\t\treturn error(\"insufficient permission for adding an object to repository database %s\", get_object_directory());\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342155","messageId":"20180319133147.15413-6-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 05/36] external-odb: implement external_odb_get_direct","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:16Z","receivedAt":"2018-03-19T13:32:29Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This is implemented only in the promisor remote mode\nfor now by calling fetch_object().\n\nSigned-off-by: Christian Couder \n---\n external-odb.c | 15 +++++++++++++++\n external-odb.h | 1 +\n odb-helper.c | 13 +++++++++++++\n odb-helper.h | 3 ++-\n 4 files changed, 31 insertions(+), 1 deletion(-)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex d26e63d8b1..5d0afb9762 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -76,3 +76,18 @@ int external_odb_has_object(const unsigned char *sha1)\n \t\t\treturn 1;\n \treturn 0;\n }\n+\n+int external_odb_get_direct(const unsigned char *sha1)\n+{\n+\tstruct odb_helper *o;\n+\n+\texternal_odb_init();\n+\n+\tfor (o = helpers; o; o = o->next) {\n+\t\tif (odb_helper_get_direct(o, sha1) < 0)\n+\t\t\tcontinue;\n+\t\treturn 0;\n+\t}\n+\n+\treturn -1;\n+}\ndiff --git a/external-odb.h b/external-odb.h\nindex 9a3c2f01b3..fd6708163e 100644\n--- a/external-odb.h\n+++ b/external-odb.h\n@@ -4,5 +4,6 @@\n extern int has_external_odb(void);\n extern const char *external_odb_root(void);\n extern int external_odb_has_object(const unsigned char *sha1);\n+extern int external_odb_get_direct(const unsigned char *sha1);\n \n #endif /* EXTERNAL_ODB_H */\ndiff --git a/odb-helper.c b/odb-helper.c\nindex 67196e6317..c6127ce81f 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -4,6 +4,7 @@\n #include \"odb-helper.h\"\n #include \"run-command.h\"\n #include \"sha1-lookup.h\"\n+#include \"fetch-object.h\"\n \n struct odb_helper *odb_helper_new(const char *name, int namelen)\n {\n@@ -52,3 +53,15 @@ int odb_helper_has_object(struct odb_helper *o, const unsigned char *sha1)\n \treturn !!odb_helper_lookup(o, sha1);\n }\n \n+int odb_helper_get_direct(struct odb_helper *o,\n+\t\t\t const unsigned char *sha1)\n+{\n+\tint res = 0;\n+\tuint64_t start = getnanotime();\n+\n+\tfetch_object(o->dealer, sha1);\n+\n+\ttrace_performance_since(start, \"odb_helper_get_direct\");\n+\n+\treturn res;\n+}\ndiff --git a/odb-helper.h b/odb-helper.h\nindex e3c9c1cfe4..57b14fe814 100644\n--- a/odb-helper.h\n+++ b/odb-helper.h\n@@ -20,5 +20,6 @@ struct odb_helper {\n extern struct odb_helper *odb_helper_new(const char *name, int namelen);\n extern int odb_helper_has_object(struct odb_helper *o,\n \t\t\t\t const unsigned char *sha1);\n-\n+extern int odb_helper_get_direct(struct odb_helper *o,\n+\t\t\t\t const unsigned char *sha1);\n #endif /* ODB_HELPER_H */\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342156","messageId":"20180319133147.15413-8-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 07/36] odb-helper: add 'enum odb_helper_type'","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:18Z","receivedAt":"2018-03-19T13:32:32Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"As there will be different kinds of helpers, let's add\nan \"enum odb_helper_type\" to tell between the different\nkinds.\n\nLet's add a field with this type in \"struct odb_helper\",\nand set it when reading the config file.\n\nWhile at it let's also make it possible to find an helper\nof a specific kind by adding a new find_odb_helper()\nfunction.\n\nSigned-off-by: Christian Couder \n---\n external-odb.c | 13 +++++++++++++\n external-odb.h | 10 ++++++++++\n odb-helper.h | 3 +++\n 3 files changed, 26 insertions(+)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex 5d0afb9762..9c77180d6c 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -57,6 +57,19 @@ int has_external_odb(void)\n \treturn !!helpers;\n }\n \n+struct odb_helper *find_odb_helper(const char *dealer, enum odb_helper_type type)\n+{\n+\tstruct odb_helper *o;\n+\n+\texternal_odb_init();\n+\n+\tfor (o = helpers; o; o = o->next)\n+\t\tif (!strcmp(o->dealer, dealer) && o->type == type)\n+\t\t\treturn o;\n+\n+\treturn NULL;\n+}\n+\n const char *external_odb_root(void)\n {\n \tstatic const char *root;\ndiff --git a/external-odb.h b/external-odb.h\nindex fd6708163e..27c3d39c1b 100644\n--- a/external-odb.h\n+++ b/external-odb.h\n@@ -1,7 +1,17 @@\n #ifndef EXTERNAL_ODB_H\n #define EXTERNAL_ODB_H\n \n+enum odb_helper_type {\n+\tODB_HELPER_NONE = 0,\n+\tODB_HELPER_GIT_REMOTE,\n+\tODB_HELPER_SCRIPT_CMD,\n+\tODB_HELPER_SUBPROCESS_CMD,\n+\tOBJ_HELPER_MAX\n+};\n+\n extern int has_external_odb(void);\n+extern struct odb_helper *find_odb_helper(const char *dealer,\n+\t\t\t\t\t enum odb_helper_type type);\n extern const char *external_odb_root(void);\n extern int external_odb_has_object(const unsigned char *sha1);\n extern int external_odb_get_direct(const unsigned char *sha1);\ndiff --git a/odb-helper.h b/odb-helper.h\nindex 57b14fe814..d4b968693f 100644\n--- a/odb-helper.h\n+++ b/odb-helper.h\n@@ -1,9 +1,12 @@\n #ifndef ODB_HELPER_H\n #define ODB_HELPER_H\n \n+#include \"external-odb.h\"\n+\n struct odb_helper {\n \tconst char *name;\n \tconst char *dealer;\n+\tenum odb_helper_type type;\n \n \tstruct odb_helper_object {\n \t\tstruct object_id oid;\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342157","messageId":"20180319133147.15413-5-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 04/36] external-odb: add has_external_odb()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:15Z","receivedAt":"2018-03-19T13:32:34Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This function will be used to check if the external odb\nmechanism is actually used.\n\nSigned-off-by: Christian Couder \n---\n external-odb.c | 7 +++++++\n external-odb.h | 1 +\n 2 files changed, 8 insertions(+)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex 390958dbfe..d26e63d8b1 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -50,6 +50,13 @@ static void external_odb_init(void)\n \tgit_config(external_odb_config, NULL);\n }\n \n+int has_external_odb(void)\n+{\n+\texternal_odb_init();\n+\n+\treturn !!helpers;\n+}\n+\n const char *external_odb_root(void)\n {\n \tstatic const char *root;\ndiff --git a/external-odb.h b/external-odb.h\nindex ae2b228792..9a3c2f01b3 100644\n--- a/external-odb.h\n+++ b/external-odb.h\n@@ -1,6 +1,7 @@\n #ifndef EXTERNAL_ODB_H\n #define EXTERNAL_ODB_H\n \n+extern int has_external_odb(void);\n extern const char *external_odb_root(void);\n extern int external_odb_has_object(const unsigned char *sha1);\n \n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342158","messageId":"20180319133147.15413-4-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 03/36] Add GIT_NO_EXTERNAL_ODB env variable","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:14Z","receivedAt":"2018-03-19T13:32:38Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This new environment variable will be used to perform git\ncommands without involving any external odb mechanism.\n\nThis makes it possible for example to create new blobs that\nwill not be sent to an external odb even if the external odb\nsupports \"put_*\" instructions.\n\nSigned-off-by: Christian Couder \n---\n cache.h | 9 +++++++++\n environment.c | 4 ++++\n external-odb.c | 3 +--\n sha1_file.c | 3 +++\n 4 files changed, 17 insertions(+), 2 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 2ac7d63e5c..4aa0129cf8 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -413,6 +413,7 @@ static inline enum object_type object_type(unsigned int mode)\n #define CEILING_DIRECTORIES_ENVIRONMENT \"GIT_CEILING_DIRECTORIES\"\n #define NO_REPLACE_OBJECTS_ENVIRONMENT \"GIT_NO_REPLACE_OBJECTS\"\n #define GIT_REPLACE_REF_BASE_ENVIRONMENT \"GIT_REPLACE_REF_BASE\"\n+#define NO_EXTERNAL_ODB_ENVIRONMENT \"GIT_NO_EXTERNAL_ODB\"\n #define GITATTRIBUTES_FILE \".gitattributes\"\n #define INFOATTRIBUTES_FILE \"info/attributes\"\n #define ATTRIBUTE_MACRO_PREFIX \"[attr]\"\n@@ -799,6 +800,14 @@ void reset_shared_repository(void);\n extern int check_replace_refs;\n extern char *git_replace_ref_base;\n \n+/*\n+ * Do external odbs need to be used this run? This variable is\n+ * initialized to true unless $GIT_NO_EXTERNAL_ODB is set, but it\n+ * maybe set to false by some commands that do not want external\n+ * odbs to be active.\n+ */\n+extern int use_external_odb;\n+\n extern int fsync_object_files;\n extern int core_preload_index;\n extern int core_apply_sparse_checkout;\ndiff --git a/environment.c b/environment.c\nindex d6dd64662c..92f682cb9b 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -50,6 +50,7 @@ const char *excludes_file;\n enum auto_crlf auto_crlf = AUTO_CRLF_FALSE;\n int check_replace_refs = 1;\n char *git_replace_ref_base;\n+int use_external_odb = 1;\n enum eol core_eol = EOL_UNSET;\n int global_conv_flags_eol = CONV_EOL_RNDTRP_WARN;\n unsigned whitespace_rule_cfg = WS_DEFAULT_RULE;\n@@ -119,6 +120,7 @@ const char * const local_repo_env[] = {\n \tINDEX_ENVIRONMENT,\n \tNO_REPLACE_OBJECTS_ENVIRONMENT,\n \tGIT_REPLACE_REF_BASE_ENVIRONMENT,\n+\tNO_EXTERNAL_ODB_ENVIRONMENT,\n \tGIT_PREFIX_ENVIRONMENT,\n \tGIT_SUPER_PREFIX_ENVIRONMENT,\n \tGIT_SHALLOW_FILE_ENVIRONMENT,\n@@ -158,6 +160,8 @@ void setup_git_env(void)\n \tfree(git_replace_ref_base);\n \tgit_replace_ref_base = xstrdup(replace_ref_base ? replace_ref_base\n \t\t\t\t\t\t\t : \"refs/replace/\");\n+\tif (getenv(NO_EXTERNAL_ODB_ENVIRONMENT))\n+\t\tuse_external_odb = 0;\n \tfree(git_namespace);\n \tgit_namespace = expand_namespace(getenv(GIT_NAMESPACE_ENVIRONMENT));\n \tshallow_file = getenv(GIT_SHALLOW_FILE_ENVIRONMENT);\ndiff --git a/external-odb.c b/external-odb.c\nindex f3ea491333..390958dbfe 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -43,7 +43,7 @@ static void external_odb_init(void)\n {\n \tstatic int initialized;\n \n-\tif (initialized)\n+\tif (initialized || !use_external_odb)\n \t\treturn;\n \tinitialized = 1;\n \n@@ -69,4 +69,3 @@ int external_odb_has_object(const unsigned char *sha1)\n \t\t\treturn 1;\n \treturn 0;\n }\n-\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 3f00fc716e..f2e078e840 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -669,6 +669,9 @@ void prepare_external_alt_odb(void)\n \tstatic int linked_external;\n \tconst char *path;\n \n+\tif (!use_external_odb)\n+\t\treturn;\n+\n \tif (linked_external)\n \t\treturn;\n \n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342159","messageId":"20180319133147.15413-13-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 12/36] external-odb: add external_odb_get_many_direct()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:23Z","receivedAt":"2018-03-19T13:32:40Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n external-odb.c | 17 +++++++++++++++++\n external-odb.h | 1 +\n odb-helper.c | 16 ++++++++++++++++\n odb-helper.h | 2 ++\n 4 files changed, 36 insertions(+)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex ae6bc017fe..8be92277d2 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -182,3 +182,20 @@ int external_odb_get_direct(const unsigned char *sha1)\n \n \treturn -1;\n }\n+\n+int external_odb_get_many_direct(const struct oid_array *to_get)\n+{\n+\tstruct odb_helper *o;\n+\n+\texternal_odb_init();\n+\n+\tfor (o = helpers; o; o = o->next) {\n+\t\tif (!(o->supported_capabilities & ODB_HELPER_CAP_GET_DIRECT))\n+\t\t\tcontinue;\n+\t\tif (odb_helper_get_many_direct(o, to_get) < 0)\n+\t\t\tcontinue;\n+\t\treturn 0;\n+\t}\n+\n+\treturn -1;\n+}\ndiff --git a/external-odb.h b/external-odb.h\nindex c5ac071995..a4eda95b09 100644\n--- a/external-odb.h\n+++ b/external-odb.h\n@@ -17,5 +17,6 @@ extern const char *external_odb_root(void);\n extern int external_odb_has_object(const unsigned char *sha1);\n extern int external_odb_get_object(const unsigned char *sha1);\n extern int external_odb_get_direct(const unsigned char *sha1);\n+extern int external_odb_get_many_direct(const struct oid_array *to_get);\n \n #endif /* EXTERNAL_ODB_H */\ndiff --git a/odb-helper.c b/odb-helper.c\nindex a99f4a1299..7e5eab94cf 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -345,3 +345,19 @@ int odb_helper_get_direct(struct odb_helper *o,\n \n \treturn res;\n }\n+\n+int odb_helper_get_many_direct(struct odb_helper *o,\n+\t\t\t const struct oid_array *to_get)\n+{\n+\tint res = 0;\n+\tuint64_t start;\n+\n+\tstart = getnanotime();\n+\n+\tif (o->type == ODB_HELPER_GIT_REMOTE)\n+\t\tfetch_objects(o->dealer, to_get);\n+\n+\ttrace_performance_since(start, \"odb_helper_get_many_direct\");\n+\n+\treturn res;\n+}\ndiff --git a/odb-helper.h b/odb-helper.h\nindex 7720684744..2c529af6af 100644\n--- a/odb-helper.h\n+++ b/odb-helper.h\n@@ -39,5 +39,7 @@ extern int odb_helper_get_object(struct odb_helper *o,\n \t\t\t\t int fd);\n extern int odb_helper_get_direct(struct odb_helper *o,\n \t\t\t\t const unsigned char *sha1);\n+extern int odb_helper_get_many_direct(struct odb_helper *o,\n+\t\t\t\t const struct oid_array *to_get);\n \n #endif /* ODB_HELPER_H */\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342160","messageId":"20180319133147.15413-17-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 16/36] t0500: add 'put_raw_obj' instruction to odb-helper script","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:27Z","receivedAt":"2018-03-19T13:32:44Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"To properly test passing objects from Git to an external odb\nwe need an odb-helper script that supports a 'put'\ncapability/instruction.\n\nFor now we will support only sending raw blobs, so the\nsupported capability/instruction will be 'put_raw_obj'.\n\nWhile at it let's add a test to check that our odb-helper\nscript works well.\n\nSigned-off-by: Christian Couder \n---\n t/t0500-external-odb.sh | 24 ++++++++++++++++++++++++\n 1 file changed, 24 insertions(+)\n\ndiff --git a/t/t0500-external-odb.sh b/t/t0500-external-odb.sh\nindex 977fea852d..4ccca1e965 100755\n--- a/t/t0500-external-odb.sh\n+++ b/t/t0500-external-odb.sh\n@@ -7,10 +7,15 @@ test_description='basic tests for external object databases'\n ALT_SOURCE=\"$PWD/alt-repo/.git\"\n export ALT_SOURCE\n write_script odb-helper <<\\EOF\n+die() {\n+\tprintf >&2 \"%s\\n\" \"$@\"\n+\texit 1\n+}\n GIT_DIR=$ALT_SOURCE; export GIT_DIR\n case \"$1\" in\n init)\n \techo \"capability=get_git_obj\"\n+\techo \"capability=put_raw_obj\"\n \techo \"capability=have\"\n \t;;\n have)\n@@ -20,6 +25,16 @@ have)\n get_git_obj)\n \tcat \"$GIT_DIR\"/objects/$(echo $2 | sed 's#..#&/#')\n \t;;\n+put_raw_obj)\n+\tsha1=\"$2\"\n+\tsize=\"$3\"\n+\tkind=\"$4\"\n+\twritten=$(git hash-object -w -t \"$kind\" --stdin)\n+\ttest \"$written\" = \"$sha1\" || die \"bad sha1 passed '$sha1' vs written '$written'\"\n+\t;;\n+*)\n+\tdie \"unknown command '$1'\"\n+\t;;\n esac\n EOF\n HELPER=\"\\\"$PWD\\\"/odb-helper\"\n@@ -45,4 +60,13 @@ test_expect_success 'helper can retrieve alt objects' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'helper can add objects to alt repo' '\n+\thash=$(echo \"Hello odb!\" | git hash-object -w -t blob --stdin) &&\n+\ttest -f .git/objects/$(echo $hash | sed \"s#..#&/#\") &&\n+\tsize=$(git cat-file -s \"$hash\") &&\n+\tgit cat-file blob \"$hash\" | ./odb-helper put_raw_obj \"$hash\" \"$size\" blob &&\n+\talt_size=$(git -C alt-repo cat-file -s \"$hash\") &&\n+\ttest \"$size\" -eq \"$alt_size\"\n+'\n+\n test_done\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342161","messageId":"20180319133147.15413-19-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 18/36] external-odb: accept only blobs for now","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:29Z","receivedAt":"2018-03-19T13:32:51Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"The mechanism to decide which blobs should be sent to which\nexternal object database will be very simple for now.\nIf the external odb helper support any \"put_*\" instruction\nall the new blobs will be sent to it.\n\nSigned-off-by: Christian Couder \n---\n external-odb.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex 806dc8e699..de2ae70db1 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -210,6 +210,10 @@ int external_odb_put_object(const void *buf, size_t len,\n \n \texternal_odb_init();\n \n+\t/* For now accept only blobs */\n+\tif (strcmp(type, \"blob\"))\n+\t\treturn 1;\n+\n \tfor (o = helpers; o; o = o->next) {\n \t\tint r = odb_helper_put_object(o, buf, len, type, sha1);\n \t\tif (r <= 0)\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342162","messageId":"20180319133147.15413-18-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 17/36] external odb: add 'put_raw_obj' support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:28Z","receivedAt":"2018-03-19T13:32:54Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Add support for a 'put_raw_obj' capability/instruction to send new\nobjects to an external odb. Objects will be sent as they are (in\ntheir 'raw' format). They will not be converted to Git objects.\n\nFor now any new Git object (blob, tree, commit, ...) would be sent\nif 'put_raw_obj' is supported by an odb helper. This is not a great\ndefault, but let's leave it to following commits to tweak that.\n\nSigned-off-by: Christian Couder \n---\n external-odb.c | 15 +++++++++++++++\n external-odb.h | 2 ++\n odb-helper.c | 51 ++++++++++++++++++++++++++++++++++++++++++++------\n odb-helper.h | 3 +++\n sha1_file.c | 2 ++\n 5 files changed, 67 insertions(+), 6 deletions(-)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex 192b09e843..806dc8e699 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -202,3 +202,18 @@ int external_odb_get_many_direct(const struct oid_array *to_get)\n \n \treturn -1;\n }\n+\n+int external_odb_put_object(const void *buf, size_t len,\n+\t\t\t const char *type, unsigned char *sha1)\n+{\n+\tstruct odb_helper *o;\n+\n+\texternal_odb_init();\n+\n+\tfor (o = helpers; o; o = o->next) {\n+\t\tint r = odb_helper_put_object(o, buf, len, type, sha1);\n+\t\tif (r <= 0)\n+\t\t\treturn r;\n+\t}\n+\treturn 1;\n+}\ndiff --git a/external-odb.h b/external-odb.h\nindex a4eda95b09..60da8ed3d4 100644\n--- a/external-odb.h\n+++ b/external-odb.h\n@@ -18,5 +18,7 @@ extern int external_odb_has_object(const unsigned char *sha1);\n extern int external_odb_get_object(const unsigned char *sha1);\n extern int external_odb_get_direct(const unsigned char *sha1);\n extern int external_odb_get_many_direct(const struct oid_array *to_get);\n+extern int external_odb_put_object(const void *buf, size_t len,\n+\t\t\t\t const char *type, unsigned char *sha1);\n \n #endif /* EXTERNAL_ODB_H */\ndiff --git a/odb-helper.c b/odb-helper.c\nindex 30cfe689de..2734af793b 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -72,9 +72,10 @@ static void prepare_helper_command(struct argv_array *argv, const char *cmd,\n \tstrbuf_release(&buf);\n }\n \n-__attribute__((format (printf,3,4)))\n+__attribute__((format (printf,4,5)))\n static int odb_helper_start(struct odb_helper *o,\n \t\t\t struct odb_helper_cmd *cmd,\n+\t\t\t int use_stdin,\n \t\t\t const char *fmt, ...)\n {\n \tva_list ap;\n@@ -91,7 +92,10 @@ static int odb_helper_start(struct odb_helper *o,\n \n \tcmd->child.argv = cmd->argv.argv;\n \tcmd->child.use_shell = 1;\n-\tcmd->child.no_stdin = 1;\n+\tif (use_stdin)\n+\t\tcmd->child.in = -1;\n+\telse\n+\t\tcmd->child.no_stdin = 1;\n \tcmd->child.out = -1;\n \n \tif (start_command(&cmd->child) < 0) {\n@@ -124,7 +128,10 @@ int odb_helper_init(struct odb_helper *o)\n \t\treturn 0;\n \to->initialized = 1;\n \n-\tif (odb_helper_start(o, &cmd, \"init\") < 0)\n+\tif (o->type != ODB_HELPER_SCRIPT_CMD)\n+\t\treturn 0;\n+\n+\tif (odb_helper_start(o, &cmd, 0, \"init\") < 0)\n \t\treturn -1;\n \n \tfh = xfdopen(cmd.child.out, \"r\");\n@@ -180,7 +187,7 @@ static void have_object_script(struct odb_helper *o)\n \tFILE *fh;\n \tstruct strbuf line = STRBUF_INIT;\n \n-\tif (odb_helper_start(o, &cmd, \"have\") < 0)\n+\tif (odb_helper_start(o, &cmd, 0, \"have\") < 0)\n \t\treturn;\n \n \tfh = xfdopen(cmd.child.out, \"r\");\n@@ -245,7 +252,7 @@ int odb_helper_get_object(struct odb_helper *o, const unsigned char *sha1,\n \tif (!obj)\n \t\treturn -1;\n \n-\tif (odb_helper_start(o, &cmd, \"get_git_obj %s\", sha1_to_hex(sha1)) < 0)\n+\tif (odb_helper_start(o, &cmd, 0, \"get_git_obj %s\", sha1_to_hex(sha1)) < 0)\n \t\treturn -1;\n \n \tmemset(&stream, 0, sizeof(stream));\n@@ -348,7 +355,7 @@ int odb_helper_get_direct(struct odb_helper *o,\n \t\tif (!obj)\n \t\t\treturn -1;\n \n-\t\tif (odb_helper_start(o, &cmd, \"get_direct %s\", sha1_to_hex(sha1)) < 0)\n+\t\tif (odb_helper_start(o, &cmd, 0, \"get_direct %s\", sha1_to_hex(sha1)) < 0)\n \t\t\treturn -1;\n \n \t\tif (odb_helper_finish(o, &cmd))\n@@ -375,3 +382,35 @@ int odb_helper_get_many_direct(struct odb_helper *o,\n \n \treturn res;\n }\n+\n+int odb_helper_put_object(struct odb_helper *o,\n+\t\t\t const void *buf, size_t len,\n+\t\t\t const char *type, unsigned char *sha1)\n+{\n+\tstruct odb_helper_cmd cmd;\n+\n+\tif (o->type != ODB_HELPER_SCRIPT_CMD)\n+\t\treturn 1;\n+\n+\tif (odb_helper_start(o, &cmd, 1, \"put_raw_obj %s %\"PRIuMAX\" %s\",\n+\t\t\t sha1_to_hex(sha1), (uintmax_t)len, type) < 0)\n+\t\treturn -1;\n+\n+\tdo {\n+\t\tint w = xwrite(cmd.child.in, buf, len);\n+\t\tif (w < 0) {\n+\t\t\terror(\"unable to write to odb helper '%s': %s\",\n+\t\t\t o->name, strerror(errno));\n+\t\t\tclose(cmd.child.in);\n+\t\t\tclose(cmd.child.out);\n+\t\t\todb_helper_finish(o, &cmd);\n+\t\t\treturn -1;\n+\t\t}\n+\t\tlen -= w;\n+\t} while (len > 0);\n+\n+\tclose(cmd.child.in);\n+\tclose(cmd.child.out);\n+\todb_helper_finish(o, &cmd);\n+\treturn 0;\n+}\ndiff --git a/odb-helper.h b/odb-helper.h\nindex 3ec2d8448f..b6a1498662 100644\n--- a/odb-helper.h\n+++ b/odb-helper.h\n@@ -42,5 +42,8 @@ extern int odb_helper_get_direct(struct odb_helper *o,\n \t\t\t\t const unsigned char *sha1);\n extern int odb_helper_get_many_direct(struct odb_helper *o,\n \t\t\t\t const struct oid_array *to_get);\n+extern int odb_helper_put_object(struct odb_helper *o,\n+\t\t\t\t const void *buf, size_t len,\n+\t\t\t\t const char *type, unsigned char *sha1);\n \n #endif /* ODB_HELPER_H */\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 99adfceede..64679fae36 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1709,6 +1709,8 @@ int write_object_file(const void *buf, unsigned long len, const char *type,\n \t * it out into .git/objects/??/?{38} file.\n \t */\n \twrite_object_file_prepare(buf, len, type, oid, hdr, &hdrlen);\n+\tif (!external_odb_put_object(buf, len, type, oid->hash))\n+\t\treturn 0;\n \tif (freshen_packed_object(oid->hash) || freshen_loose_object(oid->hash))\n \t\treturn 0;\n \treturn write_loose_object(oid, hdr, hdrlen, buf, len, 0);\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342163","messageId":"20180319133147.15413-15-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 14/36] Use odb.origin.partialclonefilter instead of core.partialclonefilter","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:25Z","receivedAt":"2018-03-19T13:32:58Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Let's make the partial clone filter specific to one odb\ninstead of general to all the odbs.\n\nThis makes it possible to have different partial clone\nfilters for different odbs.\n\nSigned-off-by: Christian Couder \n---\n builtin/fetch.c | 2 +-\n external-odb.c | 2 ++\n list-objects-filter-options.c | 23 +++++++++++++----------\n list-objects-filter-options.h | 3 ++-\n odb-helper.h | 1 +\n t/t5616-partial-clone.sh | 2 +-\n 6 files changed, 20 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex f4a138acc8..11c9561bb4 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1333,7 +1333,7 @@ static inline void fetch_one_setup_partial(struct remote *remote)\n \t * the config.\n \t */\n \tif (!filter_options.choice)\n-\t\tpartial_clone_get_default_filter_spec(&filter_options);\n+\t\tpartial_clone_get_default_filter_spec(&filter_options, remote->name);\n \treturn;\n }\n \ndiff --git a/external-odb.c b/external-odb.c\nindex 8be92277d2..c3d90f39fb 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -47,6 +47,8 @@ static int external_odb_config(const char *var, const char *value, void *data)\n \t\to->type = ODB_HELPER_SUBPROCESS_CMD;\n \t\treturn git_config_string(&o->dealer, var, value);\n \t}\n+\tif (!strcmp(subkey, \"partialclonefilter\"))\n+\t\treturn git_config_string(&o->partial_clone_filter, var, value);\n \n \treturn 0;\n }\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex 12e52bf659..ff493b9163 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -7,6 +7,7 @@\n #include \"list-objects-filter.h\"\n #include \"list-objects-filter-options.h\"\n #include \"external-odb.h\"\n+#include \"odb-helper.h\"\n \n /*\n * Parse value of the argument to the \"filter\" keyword.\n@@ -116,6 +117,7 @@ void partial_clone_register(\n \tconst struct list_objects_filter_options *filter_options)\n {\n \tchar *cfg_name;\n+\tchar *filter_name;\n \n \t/* Check if it is already registered */\n \tif (find_odb_helper(remote, ODB_HELPER_GIT_REMOTE))\n@@ -131,25 +133,26 @@ void partial_clone_register(\n \t/*\n \t * Record the initial filter-spec in the config as\n \t * the default for subsequent fetches from this remote.\n-\t *\n-\t * TODO: move core.partialclonefilter into odb.\n \t */\n-\tcore_partial_clone_filter_default =\n-\t\txstrdup(filter_options->filter_spec);\n-\tgit_config_set(\"core.partialclonefilter\",\n-\t\t core_partial_clone_filter_default);\n+\tfilter_name = xstrfmt(\"odb.%s.partialclonefilter\", remote);\n+\tgit_config_set(filter_name, filter_options->filter_spec);\n+\tfree(filter_name);\n \n \t/* Make sure the config info are reset */\n \texternal_odb_reinit();\n }\n \n void partial_clone_get_default_filter_spec(\n-\tstruct list_objects_filter_options *filter_options)\n+\tstruct list_objects_filter_options *filter_options,\n+\tconst char *remote)\n {\n+\tstruct odb_helper *helper = find_odb_helper(remote, ODB_HELPER_GIT_REMOTE);\n+\n \t/*\n \t * Parse default value, but silently ignore it if it is invalid.\n \t */\n-\tgently_parse_list_objects_filter(filter_options,\n-\t\t\t\t\t core_partial_clone_filter_default,\n-\t\t\t\t\t NULL);\n+\tif (helper)\n+\t\tgently_parse_list_objects_filter(filter_options,\n+\t\t\t\t\t\t helper->partial_clone_filter,\n+\t\t\t\t\t\t NULL);\n }\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 0000a61f82..12ceef3230 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -74,6 +74,7 @@ void partial_clone_register(\n \tconst char *remote,\n \tconst struct list_objects_filter_options *filter_options);\n void partial_clone_get_default_filter_spec(\n-\tstruct list_objects_filter_options *filter_options);\n+\tstruct list_objects_filter_options *filter_options,\n+\tconst char *remote);\n \n #endif /* LIST_OBJECTS_FILTER_OPTIONS_H */\ndiff --git a/odb-helper.h b/odb-helper.h\nindex 2c529af6af..3ec2d8448f 100644\n--- a/odb-helper.h\n+++ b/odb-helper.h\n@@ -17,6 +17,7 @@ struct odb_helper {\n \tenum odb_helper_type type;\n \tunsigned int supported_capabilities;\n \tint initialized;\n+\tconst char *partial_clone_filter;\n \n \tstruct odb_helper_object {\n \t\tstruct object_id oid;\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex 5c84b0f95b..d9b8418275 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -41,7 +41,7 @@ test_expect_success 'do partial clone 1' '\n \ttest_cmp expect_1.oids observed.oids &&\n \ttest \"$(git -C pc1 config --local core.repositoryformatversion)\" = \"1\" &&\n \ttest \"$(git -C pc1 config --local odb.origin.promisorRemote)\" = \"origin\" &&\n-\ttest \"$(git -C pc1 config --local core.partialclonefilter)\" = \"blob:none\"\n+\ttest \"$(git -C pc1 config --local odb.origin.partialclonefilter)\" = \"blob:none\"\n '\n \n # checkout master to force dynamic object fetch of blobs at HEAD.\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342164","messageId":"20180319133147.15413-14-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 13/36] Use external_odb_get_direct() and has_external_odb()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:24Z","receivedAt":"2018-03-19T13:33:03Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Instead of using the repository_format_partial_clone global\nand fetch_object() directly, let's use has_external_odb() and\nexternal_odb_get_direct().\n\nSigned-off-by: Christian Couder \n---\n builtin/cat-file.c | 5 +++--\n builtin/fetch.c | 11 ++++++-----\n builtin/gc.c | 3 ++-\n builtin/repack.c | 3 ++-\n cache.h | 2 --\n connected.c | 3 ++-\n environment.c | 1 -\n list-objects-filter-options.c | 27 +++++++++++++++------------\n packfile.c | 3 ++-\n setup.c | 7 +------\n sha1_file.c | 8 ++------\n t/t0410-partial-clone.sh | 30 +++++++++++++++---------------\n t/t5500-fetch-pack.sh | 4 ++--\n t/t5601-clone.sh | 2 +-\n t/t5616-partial-clone.sh | 2 +-\n unpack-trees.c | 6 +++---\n 16 files changed, 57 insertions(+), 60 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex d90170f070..fb7b85502f 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -13,6 +13,7 @@\n #include \"tree-walk.h\"\n #include \"sha1-array.h\"\n #include \"packfile.h\"\n+#include \"external-odb.h\"\n \n struct batch_options {\n \tint enabled;\n@@ -475,8 +476,8 @@ static int batch_objects(struct batch_options *opt)\n \n \t\tfor_each_loose_object(batch_loose_object, &sa, 0);\n \t\tfor_each_packed_object(batch_packed_object, &sa, 0);\n-\t\tif (repository_format_partial_clone)\n-\t\t\twarning(\"This repository has extensions.partialClone set. Some objects may not be loaded.\");\n+\t\tif (has_external_odb())\n+\t\t\twarning(\"This repository uses an odb. Some objects may not be loaded.\");\n \n \t\tcb.opt = opt;\n \t\tcb.expand = &data;\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 6d73656a48..f4a138acc8 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -20,6 +20,7 @@\n #include \"utf8.h\"\n #include \"packfile.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"external-odb.h\"\n \n static const char * const builtin_fetch_usage[] = {\n \tN_(\"git fetch [] [ [...]]\"),\n@@ -1303,7 +1304,7 @@ static inline void fetch_one_setup_partial(struct remote *remote)\n \t * If no prior partial clone/fetch and the current fetch DID NOT\n \t * request a partial-fetch, do a normal fetch.\n \t */\n-\tif (!repository_format_partial_clone && !filter_options.choice)\n+\tif (!has_external_odb() && !filter_options.choice)\n \t\treturn;\n \n \t/*\n@@ -1311,7 +1312,7 @@ static inline void fetch_one_setup_partial(struct remote *remote)\n \t * on this repo and remember the given filter-spec as the default\n \t * for subsequent fetches to this remote.\n \t */\n-\tif (!repository_format_partial_clone && filter_options.choice) {\n+\tif (!has_external_odb() && filter_options.choice) {\n \t\tpartial_clone_register(remote->name, &filter_options);\n \t\treturn;\n \t}\n@@ -1320,7 +1321,7 @@ static inline void fetch_one_setup_partial(struct remote *remote)\n \t * We are currently limited to only ONE promisor remote and only\n \t * allow partial-fetches from the promisor remote.\n \t */\n-\tif (strcmp(remote->name, repository_format_partial_clone)) {\n+\tif (!find_odb_helper(remote->name, ODB_HELPER_GIT_REMOTE)) {\n \t\tif (filter_options.choice)\n \t\t\tdie(_(\"--filter can only be used with the remote configured in core.partialClone\"));\n \t\treturn;\n@@ -1456,7 +1457,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tif (depth || deepen_since || deepen_not.nr)\n \t\tdeepen = 1;\n \n-\tif (filter_options.choice && !repository_format_partial_clone)\n+\tif (filter_options.choice && !has_external_odb())\n \t\tdie(\"--filter can only be used when extensions.partialClone is set\");\n \n \tif (all) {\n@@ -1490,7 +1491,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (remote) {\n-\t\tif (filter_options.choice || repository_format_partial_clone)\n+\t\tif (filter_options.choice || has_external_odb())\n \t\t\tfetch_one_setup_partial(remote);\n \t\tresult = fetch_one(remote, argc, argv, prune_tags_ok);\n \t} else {\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex f51e5a6500..ff6e9805b2 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -20,6 +20,7 @@\n #include \"argv-array.h\"\n #include \"commit.h\"\n #include \"packfile.h\"\n+#include \"external-odb.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -461,7 +462,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\targv_array_push(&prune, prune_expire);\n \t\t\tif (quiet)\n \t\t\t\targv_array_push(&prune, \"--no-progress\");\n-\t\t\tif (repository_format_partial_clone)\n+\t\t\tif (has_external_odb())\n \t\t\t\targv_array_push(&prune,\n \t\t\t\t\t\t\"--exclude-promisor-objects\");\n \t\t\tif (run_command_v_opt(prune.argv, RUN_GIT_CMD))\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 7bdb40142f..a6d9cfb92c 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -8,6 +8,7 @@\n #include \"strbuf.h\"\n #include \"string-list.h\"\n #include \"argv-array.h\"\n+#include \"external-odb.h\"\n \n static int delta_base_offset = 1;\n static int pack_kept_objects = -1;\n@@ -234,7 +235,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \targv_array_push(&cmd.args, \"--all\");\n \targv_array_push(&cmd.args, \"--reflog\");\n \targv_array_push(&cmd.args, \"--indexed-objects\");\n-\tif (repository_format_partial_clone)\n+\tif (has_external_odb())\n \t\targv_array_push(&cmd.args, \"--exclude-promisor-objects\");\n \tif (window)\n \t\targv_array_pushf(&cmd.args, \"--window=%s\", window);\ndiff --git a/cache.h b/cache.h\nindex e6fd9b97ec..da967a5cdc 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -900,13 +900,11 @@ extern int grafts_replace_parents;\n #define GIT_REPO_VERSION 0\n #define GIT_REPO_VERSION_READ 1\n extern int repository_format_precious_objects;\n-extern char *repository_format_partial_clone;\n extern const char *core_partial_clone_filter_default;\n \n struct repository_format {\n \tint version;\n \tint precious_objects;\n-\tchar *partial_clone; /* value of extensions.partialclone */\n \tint is_bare;\n \tint hash_algo;\n \tchar *work_tree;\ndiff --git a/connected.c b/connected.c\nindex 91feb78815..a336c69a2d 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -4,6 +4,7 @@\n #include \"connected.h\"\n #include \"transport.h\"\n #include \"packfile.h\"\n+#include \"external-odb.h\"\n \n /*\n * If we feed all the commits we want to verify to this command\n@@ -56,7 +57,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n \targv_array_push(&rev_list.args,\"rev-list\");\n \targv_array_push(&rev_list.args, \"--objects\");\n \targv_array_push(&rev_list.args, \"--stdin\");\n-\tif (repository_format_partial_clone)\n+\tif (has_external_odb())\n \t\targv_array_push(&rev_list.args, \"--exclude-promisor-objects\");\n \targv_array_push(&rev_list.args, \"--not\");\n \targv_array_push(&rev_list.args, \"--all\");\ndiff --git a/environment.c b/environment.c\nindex 92f682cb9b..2ff9dc9218 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -27,7 +27,6 @@ int warn_ambiguous_refs = 1;\n int warn_on_object_refname_ambiguity = 1;\n int ref_paranoia = -1;\n int repository_format_precious_objects;\n-char *repository_format_partial_clone;\n const char *core_partial_clone_filter_default;\n const char *git_commit_encoding;\n const char *git_log_output_encoding;\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex 6a3cc985c4..12e52bf659 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -6,6 +6,7 @@\n #include \"list-objects.h\"\n #include \"list-objects-filter.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"external-odb.h\"\n \n /*\n * Parse value of the argument to the \"filter\" keyword.\n@@ -114,30 +115,32 @@ void partial_clone_register(\n \tconst char *remote,\n \tconst struct list_objects_filter_options *filter_options)\n {\n-\t/*\n-\t * Record the name of the partial clone remote in the\n-\t * config and in the global variable -- the latter is\n-\t * used throughout to indicate that partial clone is\n-\t * enabled and to expect missing objects.\n-\t */\n-\tif (repository_format_partial_clone &&\n-\t *repository_format_partial_clone &&\n-\t strcmp(remote, repository_format_partial_clone))\n-\t\tdie(_(\"cannot change partial clone promisor remote\"));\n+\tchar *cfg_name;\n+\n+\t/* Check if it is already registered */\n+\tif (find_odb_helper(remote, ODB_HELPER_GIT_REMOTE))\n+\t\treturn;\n \n \tgit_config_set(\"core.repositoryformatversion\", \"1\");\n-\tgit_config_set(\"extensions.partialclone\", remote);\n \n-\trepository_format_partial_clone = xstrdup(remote);\n+\t/* Add odb config for the remote */\n+\tcfg_name = xstrfmt(\"odb.%s.promisorRemote\", remote);\n+\tgit_config_set(cfg_name, remote);\n+\tfree(cfg_name);\n \n \t/*\n \t * Record the initial filter-spec in the config as\n \t * the default for subsequent fetches from this remote.\n+\t *\n+\t * TODO: move core.partialclonefilter into odb.\n \t */\n \tcore_partial_clone_filter_default =\n \t\txstrdup(filter_options->filter_spec);\n \tgit_config_set(\"core.partialclonefilter\",\n \t\t core_partial_clone_filter_default);\n+\n+\t/* Make sure the config info are reset */\n+\texternal_odb_reinit();\n }\n \n void partial_clone_get_default_filter_spec(\ndiff --git a/packfile.c b/packfile.c\nindex 7c1a2519fc..3e7c0213f9 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -13,6 +13,7 @@\n #include \"tag.h\"\n #include \"tree-walk.h\"\n #include \"tree.h\"\n+#include \"external-odb.h\"\n \n char *odb_pack_name(struct strbuf *buf,\n \t\t const unsigned char *sha1,\n@@ -1941,7 +1942,7 @@ int is_promisor_object(const struct object_id *oid)\n \tstatic int promisor_objects_prepared;\n \n \tif (!promisor_objects_prepared) {\n-\t\tif (repository_format_partial_clone) {\n+\t\tif (has_external_odb()) {\n \t\t\tfor_each_packed_object(add_promisor_object,\n \t\t\t\t\t &promisor_objects,\n \t\t\t\t\t FOR_EACH_OBJECT_PROMISOR_ONLY);\ndiff --git a/setup.c b/setup.c\nindex 7287779642..16e7a6bb4c 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -422,11 +422,7 @@ static int check_repo_format(const char *var, const char *value, void *vdata)\n \t\t\t;\n \t\telse if (!strcmp(ext, \"preciousobjects\"))\n \t\t\tdata->precious_objects = git_config_bool(var, value);\n-\t\telse if (!strcmp(ext, \"partialclone\")) {\n-\t\t\tif (!value)\n-\t\t\t\treturn config_error_nonbool(var);\n-\t\t\tdata->partial_clone = xstrdup(value);\n-\t\t} else\n+\t\telse\n \t\t\tstring_list_append(&data->unknown_extensions, ext);\n \t} else if (strcmp(var, \"core.bare\") == 0) {\n \t\tdata->is_bare = git_config_bool(var, value);\n@@ -468,7 +464,6 @@ static int check_repository_format_gently(const char *gitdir, struct repository_\n \t}\n \n \trepository_format_precious_objects = candidate->precious_objects;\n-\trepository_format_partial_clone = candidate->partial_clone;\n \tstring_list_clear(&candidate->unknown_extensions, 0);\n \tif (!has_common) {\n \t\tif (candidate->is_bare != -1) {\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 847883f261..99adfceede 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1313,13 +1313,9 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\t\tbreak;\n \n \t\t/* Check if it is a missing object */\n-\t\tif (fetch_if_missing && repository_format_partial_clone &&\n+\t\tif (fetch_if_missing && has_external_odb() &&\n \t\t !already_retried) {\n-\t\t\t/*\n-\t\t\t * TODO Investigate haveing fetch_object() return\n-\t\t\t * TODO error/success and stopping the music here.\n-\t\t\t */\n-\t\t\tfetch_object(repository_format_partial_clone, real);\n+\t\t\texternal_odb_get_direct(real);\n \t\t\talready_retried = 1;\n \t\t\tcontinue;\n \t\t}\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex cc18b75c03..6af4712da8 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -43,7 +43,7 @@ test_expect_success 'missing reflog object, but promised by a commit, passes fsc\n \n \t# But with the extension, it succeeds\n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"arbitrary string\" &&\n \tgit -C repo fsck\n '\n \n@@ -66,7 +66,7 @@ test_expect_success 'missing reflog object, but promised by a tag, passes fsck'\n \tprintf \"$T\\n\" | pack_as_from_promisor &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"arbitrary string\" &&\n \tgit -C repo fsck\n '\n \n@@ -84,7 +84,7 @@ test_expect_success 'missing reflog object alone fails fsck, even with extension\n \tdelete_object repo \"$A\" &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"arbitrary string\" &&\n \ttest_must_fail git -C repo fsck\n '\n \n@@ -100,7 +100,7 @@ test_expect_success 'missing ref object, but promised, passes fsck' '\n \tpromise_and_delete \"$A\" &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"arbitrary string\" &&\n \tgit -C repo fsck\n '\n \n@@ -123,7 +123,7 @@ test_expect_success 'missing object, but promised, passes fsck' '\n \tpromise_and_delete \"$AT\" &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"arbitrary string\" &&\n \tgit -C repo fsck\n '\n \n@@ -136,7 +136,7 @@ test_expect_success 'missing CLI object, but promised, passes fsck' '\n \tpromise_and_delete \"$A\" &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"arbitrary string\" &&\n \tgit -C repo fsck \"$A\"\n '\n \n@@ -151,7 +151,7 @@ test_expect_success 'fetching of missing objects' '\n \trm -rf repo/.git/objects/* &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"origin\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"origin\" &&\n \tgit -C repo cat-file -p \"$HASH\" &&\n \n \t# Ensure that the .promisor file is written, and check that its\n@@ -172,7 +172,7 @@ test_expect_success 'rev-list stops traversal at missing and promised commit' '\n \tpromise_and_delete \"$FOO\" &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"arbitrary string\" &&\n \tgit -C repo rev-list --exclude-promisor-objects --objects bar >out &&\n \tgrep $(git -C repo rev-parse bar) out &&\n \t! grep $FOO out\n@@ -197,7 +197,7 @@ test_expect_success 'rev-list stops traversal at missing and promised tree' '\n \tpromise_and_delete \"$TREE2\" &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"arbitrary string\" &&\n \tgit -C repo rev-list --exclude-promisor-objects --objects HEAD >out &&\n \tgrep $(git -C repo rev-parse foo) out &&\n \t! grep $TREE out &&\n@@ -216,7 +216,7 @@ test_expect_success 'rev-list stops traversal at missing and promised blob' '\n \tpromise_and_delete \"$BLOB\" &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"arbitrary string\" &&\n \tgit -C repo rev-list --exclude-promisor-objects --objects HEAD >out &&\n \tgrep $(git -C repo rev-parse HEAD) out &&\n \t! grep $BLOB out\n@@ -235,7 +235,7 @@ test_expect_success 'rev-list stops traversal at promisor commit, tree, and blob\n \tprintf \"%s\\n%s\\n%s\\n\" $COMMIT $TREE $BLOB | pack_as_from_promisor &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"arbitrary string\" &&\n \tgit -C repo rev-list --exclude-promisor-objects --objects HEAD >out &&\n \t! grep $COMMIT out &&\n \t! grep $TREE out &&\n@@ -259,7 +259,7 @@ test_expect_success 'rev-list accepts missing and promised objects on command li\n \tpromise_and_delete $BLOB &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"arbitrary string\" &&\n \tgit -C repo rev-list --exclude-promisor-objects --objects \"$COMMIT\" \"$TREE\" \"$BLOB\"\n '\n \n@@ -272,7 +272,7 @@ test_expect_success 'gc does not repack promisor objects' '\n \tHASH=$(printf \"$TREE_HASH\\n\" | pack_as_from_promisor) &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"arbitrary string\" &&\n \tgit -C repo gc &&\n \n \t# Ensure that the promisor packfile still exists, and remove it\n@@ -296,7 +296,7 @@ test_expect_success 'gc stops traversal when a missing but promised object is re\n \tHASH=$(promise_and_delete $TREE_HASH) &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"arbitrary string\" &&\n \tgit -C repo gc &&\n \n \t# Ensure that the promisor packfile still exists, and remove it\n@@ -327,7 +327,7 @@ test_expect_success 'fetching of missing objects from an HTTP server' '\n \trm -rf repo/.git/objects/* &&\n \n \tgit -C repo config core.repositoryformatversion 1 &&\n-\tgit -C repo config extensions.partialclone \"origin\" &&\n+\tgit -C repo config odb.magic.promisorRemote \"origin\" &&\n \tgit -C repo cat-file -p \"$HASH\" &&\n \n \t# Ensure that the .promisor file is written, and check that its\ndiff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh\nindex 0680dec808..3bb20b0566 100755\n--- a/t/t5500-fetch-pack.sh\n+++ b/t/t5500-fetch-pack.sh\n@@ -792,14 +792,14 @@ fetch_filter_blob_limit_zero () {\n \ttest_config -C \"$SERVER\" uploadpack.allowfilter 1 &&\n \n \tgit clone \"$URL\" client &&\n-\ttest_config -C client extensions.partialclone origin &&\n+\ttest_config -C client odb.magic.promisorRemote origin &&\n \n \ttest_commit -C \"$SERVER\" two &&\n \n \tgit -C client fetch --filter=blob:limit=0 origin HEAD:somewhere &&\n \n \t# Ensure that commit is fetched, but blob is not\n-\ttest_config -C client extensions.partialclone \"arbitrary string\" &&\n+\ttest_config -C client odb.magic.promisorRemote \"arbitrary string\" &&\n \tgit -C client cat-file -e $(git -C \"$SERVER\" rev-parse two) &&\n \ttest_must_fail git -C client cat-file -e $(git hash-object \"$SERVER/two.t\")\n }\ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex 0b62037744..022c65a7ae 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -647,7 +647,7 @@ partial_clone () {\n \tgit -C client fsck &&\n \n \t# Ensure that unneeded blobs are not inadvertently fetched.\n-\ttest_config -C client extensions.partialclone \"not a remote\" &&\n+\ttest_config -C client odb.origin.promisorRemote \"not a remote\" &&\n \ttest_must_fail git -C client cat-file -e \"$HASH1\" &&\n \n \t# But this blob was fetched, because clone performs an initial checkout\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex 29d8631184..5c84b0f95b 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -40,7 +40,7 @@ test_expect_success 'do partial clone 1' '\n \t\t| sort >observed.oids &&\n \ttest_cmp expect_1.oids observed.oids &&\n \ttest \"$(git -C pc1 config --local core.repositoryformatversion)\" = \"1\" &&\n-\ttest \"$(git -C pc1 config --local extensions.partialclone)\" = \"origin\" &&\n+\ttest \"$(git -C pc1 config --local odb.origin.promisorRemote)\" = \"origin\" &&\n \ttest \"$(git -C pc1 config --local core.partialclonefilter)\" = \"blob:none\"\n '\n \ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex d5685891a5..38124e233d 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -16,6 +16,7 @@\n #include \"submodule-config.h\"\n #include \"fsmonitor.h\"\n #include \"fetch-object.h\"\n+#include \"external-odb.h\"\n \n /*\n * Error messages expected by scripts out of plumbing commands such as\n@@ -371,7 +372,7 @@ static int check_updates(struct unpack_trees_options *o)\n \t\tload_gitmodules_file(index, &state);\n \n \tenable_delayed_checkout(&state);\n-\tif (repository_format_partial_clone && o->update && !o->dry_run) {\n+\tif (has_external_odb() && o->update && !o->dry_run) {\n \t\t/*\n \t\t * Prefetch the objects that are to be checked out in the loop\n \t\t * below.\n@@ -388,8 +389,7 @@ static int check_updates(struct unpack_trees_options *o)\n \t\t\t}\n \t\t}\n \t\tif (to_fetch.nr)\n-\t\t\tfetch_objects(repository_format_partial_clone,\n-\t\t\t\t &to_fetch);\n+\t\t\texternal_odb_get_many_direct(&to_fetch);\n \t\tfetch_if_missing = fetch_if_missing_store;\n \t}\n \tfor (i = 0; i < index->cache_nr; i++) {\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342165","messageId":"20180319133147.15413-23-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 22/36] lib-httpd: add upload.sh","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:33Z","receivedAt":"2018-03-19T13:33:06Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This cgi will be used to upload objects to, or to delete\nobjects from, an apache web server.\n\nThis way the apache server can work as an external object\ndatabase.\n\nSigned-off-by: Christian Couder \n---\n t/lib-httpd.sh | 1 +\n t/lib-httpd/upload.sh | 45 +++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 46 insertions(+)\n create mode 100644 t/lib-httpd/upload.sh\n\ndiff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\nindex 2e659a8ee2..d80b004549 100644\n--- a/t/lib-httpd.sh\n+++ b/t/lib-httpd.sh\n@@ -132,6 +132,7 @@ prepare_httpd() {\n \tcp \"$TEST_PATH\"/passwd \"$HTTPD_ROOT_PATH\"\n \tinstall_script broken-smart-http.sh\n \tinstall_script error.sh\n+\tinstall_script upload.sh\n \n \tln -s \"$LIB_HTTPD_MODULE_PATH\" \"$HTTPD_ROOT_PATH/modules\"\n \ndiff --git a/t/lib-httpd/upload.sh b/t/lib-httpd/upload.sh\nnew file mode 100644\nindex 0000000000..64d3f31c31\n--- /dev/null\n+++ b/t/lib-httpd/upload.sh\n@@ -0,0 +1,45 @@\n+#!/bin/sh\n+\n+# In part from http://codereview.stackexchange.com/questions/79549/bash-cgi-upload-file\n+\n+FILES_DIR=\"www/files\"\n+\n+OLDIFS=\"$IFS\"\n+IFS='&'\n+set -- $QUERY_STRING\n+IFS=\"$OLDIFS\"\n+\n+while test $# -gt 0\n+do\n+\tkey=${1%%=*}\n+\tval=${1#*=}\n+\n+\tcase \"$key\" in\n+\t\"sha1\") sha1=\"$val\" ;;\n+\t\"type\") type=\"$val\" ;;\n+\t\"size\") size=\"$val\" ;;\n+\t\"delete\") delete=1 ;;\n+\t*) echo >&2 \"unknown key '$key'\" ;;\n+\tesac\n+\n+\tshift\n+done\n+\n+case \"$REQUEST_METHOD\" in\n+POST)\n+\tif test \"$delete\" = \"1\"\n+\tthen\n+\t\trm -f \"$FILES_DIR/$sha1-$size-$type\"\n+\telse\n+\t\tmkdir -p \"$FILES_DIR\"\n+\t\tcat >\"$FILES_DIR/$sha1-$size-$type\"\n+\tfi\n+\n+\techo 'Status: 204 No Content'\n+\techo\n+\t;;\n+\n+*)\n+\techo 'Status: 405 Method Not Allowed'\n+\techo\n+esac\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342166","messageId":"20180319133147.15413-22-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 21/36] lib-httpd: pass config file to start_httpd()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:32Z","receivedAt":"2018-03-19T13:33:07Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This makes it possible to start an apache web server with different\nconfig files.\n\nThis will be used in a later patch to pass a config file that makes\napache store external objects.\n\nSigned-off-by: Christian Couder \n---\n t/lib-httpd.sh | 6 ++++--\n 1 file changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\nindex 435a37465a..2e659a8ee2 100644\n--- a/t/lib-httpd.sh\n+++ b/t/lib-httpd.sh\n@@ -171,12 +171,14 @@ prepare_httpd() {\n }\n \n start_httpd() {\n+\tAPACHE_CONF_FILE=${1-apache.conf}\n+\n \tprepare_httpd >&3 2>&4\n \n \ttrap 'code=$?; stop_httpd; (exit $code); die' EXIT\n \n \t\"$LIB_HTTPD_PATH\" -d \"$HTTPD_ROOT_PATH\" \\\n-\t\t-f \"$TEST_PATH/apache.conf\" $HTTPD_PARA \\\n+\t\t-f \"$TEST_PATH/$APACHE_CONF_FILE\" $HTTPD_PARA \\\n \t\t-c \"Listen 127.0.0.1:$LIB_HTTPD_PORT\" -k start \\\n \t\t>&3 2>&4\n \tif test $? -ne 0\n@@ -191,7 +193,7 @@ stop_httpd() {\n \ttrap 'die' EXIT\n \n \t\"$LIB_HTTPD_PATH\" -d \"$HTTPD_ROOT_PATH\" \\\n-\t\t-f \"$TEST_PATH/apache.conf\" $HTTPD_PARA -k stop\n+\t\t-f \"$TEST_PATH/$APACHE_CONF_FILE\" $HTTPD_PARA -k stop\n }\n \n test_http_push_nonff () {\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342167","messageId":"20180319133147.15413-27-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 26/36] pack-objects: don't pack objects in external odbs","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:37Z","receivedAt":"2018-03-19T13:33:09Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Objects managed by an external ODB should not be put into\npack files. They should be transfered using other mechanism\nthat can be specific to the external odb.\n\nSigned-off-by: Christian Couder \n---\n builtin/pack-objects.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex e9d3cfb9e3..59e12151e8 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -28,6 +28,7 @@\n #include \"argv-array.h\"\n #include \"list.h\"\n #include \"packfile.h\"\n+#include \"external-odb.h\"\n \n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [...] [< | < ]\"),\n@@ -1026,6 +1027,9 @@ static int want_object_in_pack(const struct object_id *oid,\n \t\t\treturn want;\n \t}\n \n+\tif (external_odb_has_object(oid->hash))\n+\t\treturn 0;\n+\n \tlist_for_each(pos, &packed_git_mru) {\n \t\tstruct packed_git *p = list_entry(pos, struct packed_git, mru);\n \t\toff_t offset;\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342168","messageId":"20180319133147.15413-28-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 27/36] Add t0520 to test transfer to HTTP external odb","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:38Z","receivedAt":"2018-03-19T13:33:10Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This tests that an apache web server can be used as an\nexternal object database and store files in their native\nformat instead of converting them to a Git object.\n\nSigned-off-by: Christian Couder \n---\n t/t0520-transfer-http-e-odb.sh | 142 +++++++++++++++++++++++++++++++++\n 1 file changed, 142 insertions(+)\n create mode 100755 t/t0520-transfer-http-e-odb.sh\n\ndiff --git a/t/t0520-transfer-http-e-odb.sh b/t/t0520-transfer-http-e-odb.sh\nnew file mode 100755\nindex 0000000000..f84fe950ec\n--- /dev/null\n+++ b/t/t0520-transfer-http-e-odb.sh\n@@ -0,0 +1,142 @@\n+#!/bin/sh\n+\n+test_description='tests for transfering external objects to an HTTPD server'\n+\n+. ./test-lib.sh\n+\n+# If we don't specify a port, the current test number will be used\n+# which will not work as it is less than 1024, so it can only be used by root.\n+LIB_HTTPD_PORT=$(expr ${this_test#t} + 12000)\n+\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+\n+start_httpd apache-e-odb.conf\n+\n+# odb helper script must see this\n+export HTTPD_URL\n+\n+write_script odb-http-helper <<\\EOF\n+die() {\n+\tprintf >&2 \"%s\\n\" \"$@\"\n+\texit 1\n+}\n+echo >&2 \"odb-http-helper args:\" \"$@\"\n+case \"$1\" in\n+init)\n+\techo \"capability=get_raw_obj\"\n+\techo \"capability=put_raw_obj\"\n+\techo \"capability=have\"\n+\t;;\n+have)\n+\tlist_url=\"$HTTPD_URL/list/\"\n+\tcurl \"$list_url\" ||\n+\tdie \"curl '$list_url' failed\"\n+\t;;\n+get_raw_obj)\n+\tget_url=\"$HTTPD_URL/list/?sha1=$2\"\n+\tcurl \"$get_url\" ||\n+\tdie \"curl '$get_url' failed\"\n+\t;;\n+put_raw_obj)\n+\tsha1=\"$2\"\n+\tsize=\"$3\"\n+\tkind=\"$4\"\n+\tupload_url=\"$HTTPD_URL/upload/?sha1=$sha1&size=$size&type=$kind\"\n+\tcurl --data-binary @- --include \"$upload_url\" >out ||\n+\tdie \"curl '$upload_url' failed\"\n+\tref_hash=$(echo \"$sha1 $size $kind\" | GIT_NO_EXTERNAL_ODB=1 git hash-object -w -t blob --stdin) || exit\n+\tgit update-ref refs/odbs/magic/\"$sha1\" \"$ref_hash\"\n+\t;;\n+*)\n+\tdie \"unknown command '$1'\"\n+\t;;\n+esac\n+EOF\n+HELPER=\"\\\"$PWD\\\"/odb-http-helper\"\n+\n+test_expect_success 'setup repo with a root commit and the helper' '\n+\ttest_commit zero &&\n+\tgit config odb.magic.scriptCommand \"$HELPER\"\n+'\n+\n+test_expect_success 'setup another repo from the first one' '\n+\tgit init other-repo &&\n+\t(cd other-repo &&\n+\t git remote add origin .. &&\n+\t git pull origin master &&\n+\t git checkout master &&\n+\t git log)\n+'\n+\n+UPLOADFILENAME=\"hello_apache_upload.txt\"\n+\n+UPLOAD_URL=\"$HTTPD_URL/upload/?sha1=$UPLOADFILENAME&size=123&type=blob\"\n+\n+test_expect_success 'can upload a file' '\n+\techo \"Hello Apache World!\" >hello_to_send.txt &&\n+\techo \"How are you?\" >>hello_to_send.txt &&\n+\tcurl --data-binary @hello_to_send.txt --include \"$UPLOAD_URL\" >out_upload\n+'\n+\n+LIST_URL=\"$HTTPD_URL/list/\"\n+\n+test_expect_success 'can list uploaded files' '\n+\tcurl --include \"$LIST_URL\" >out_list &&\n+\tgrep \"$UPLOADFILENAME\" out_list\n+'\n+\n+test_expect_success 'can delete uploaded files' '\n+\tcurl --data \"delete\" --include \"$UPLOAD_URL&delete=1\" >out_delete &&\n+\tcurl --include \"$LIST_URL\" >out_list2 &&\n+\t! grep \"$UPLOADFILENAME\" out_list2\n+'\n+\n+FILES_DIR=\"httpd/www/files\"\n+\n+test_expect_success 'new blobs are transfered to the http server' '\n+\ttest_commit one &&\n+\thash1=$(git ls-tree HEAD | grep one.t | cut -f1 | cut -d\\ -f3) &&\n+\techo \"$hash1-4-blob\" >expected &&\n+\tls \"$FILES_DIR\" >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'blobs can be retrieved from the http server' '\n+\tgit cat-file blob \"$hash1\" &&\n+\tgit log -p >expected\n+'\n+\n+test_expect_success 'update other repo from the first one' '\n+\t(cd other-repo &&\n+\t git fetch origin \"refs/odbs/magic/*:refs/odbs/magic/*\" &&\n+\t test_must_fail git cat-file blob \"$hash1\" &&\n+\t git config odb.magic.scriptCommand \"$HELPER\" &&\n+\t git cat-file blob \"$hash1\" &&\n+\t git pull origin master)\n+'\n+\n+test_expect_success 'local clone from the first repo' '\n+\tmkdir my-clone &&\n+\t(cd my-clone &&\n+\t git clone .. . &&\n+\t git cat-file blob \"$hash1\")\n+'\n+\n+test_expect_success 'no-local clone from the first repo fails' '\n+\tmkdir my-other-clone &&\n+\t(cd my-other-clone &&\n+\t test_must_fail git clone --no-local .. .) &&\n+\trm -rf my-other-clone\n+'\n+\n+test_expect_success 'no-local clone from the first repo with helper succeeds' '\n+\tmkdir my-other-clone &&\n+\t(cd my-other-clone &&\n+\t git clone -c odb.magic.scriptCommand=\"$HELPER\" \\\n+\t\t--no-local .. .) &&\n+\trm -rf my-other-clone\n+'\n+\n+stop_httpd\n+\n+test_done\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342169","messageId":"20180319133147.15413-31-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 30/36] Add t0560 to test passing git objects","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:41Z","receivedAt":"2018-03-19T13:33:13Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n t/t0560-read-object-git.sh | 28 ++++++++++++++\n t/t0560/read-object-git | 78 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 106 insertions(+)\n create mode 100755 t/t0560-read-object-git.sh\n create mode 100755 t/t0560/read-object-git\n\ndiff --git a/t/t0560-read-object-git.sh b/t/t0560-read-object-git.sh\nnew file mode 100755\nindex 0000000000..4e8fc04dfd\n--- /dev/null\n+++ b/t/t0560-read-object-git.sh\n@@ -0,0 +1,28 @@\n+#!/bin/sh\n+\n+test_description='tests for long running read-object process passing git objects'\n+\n+. ./test-lib.sh\n+\n+PATH=\"$PATH:$TEST_DIRECTORY/t0560\"\n+\n+test_expect_success 'setup host repo with a root commit' '\n+\ttest_commit zero &&\n+\thash1=$(git ls-tree HEAD | grep zero.t | cut -f1 | cut -d\\ -f3)\n+'\n+\n+HELPER=\"read-object-git\"\n+\n+test_expect_success 'blobs can be retrieved from the host repo' '\n+\tgit init guest-repo &&\n+\t(cd guest-repo &&\n+\t git config odb.magic.subprocessCommand \"$HELPER\" &&\n+\t git cat-file blob \"$hash1\" >/dev/null)\n+'\n+\n+test_expect_success 'invalid blobs generate errors' '\n+\tcd guest-repo &&\n+\ttest_must_fail git cat-file blob \"invalid\"\n+'\n+\n+test_done\ndiff --git a/t/t0560/read-object-git b/t/t0560/read-object-git\nnew file mode 100755\nindex 0000000000..4b3ca0948b\n--- /dev/null\n+++ b/t/t0560/read-object-git\n@@ -0,0 +1,78 @@\n+#!/usr/bin/perl\n+#\n+# Example implementation for the Git read-object protocol version 1\n+# See Documentation/technical/read-object-protocol.txt\n+#\n+# Allows you to test the ability for blobs to be pulled from a host git repo\n+# \"on demand.\" Called when git needs a blob it couldn't find locally due to\n+# a lazy clone that only cloned the commits and trees.\n+#\n+# A lazy clone can be simulated via the following commands from the host repo\n+# you wish to create a lazy clone of:\n+#\n+# cd /host_repo\n+# git rev-parse HEAD\n+# git init /guest_repo\n+# git cat-file --batch-check --batch-all-objects | grep -v 'blob' |\n+#\tcut -d' ' -f1 | git pack-objects /e/guest_repo/.git/objects/pack/noblobs\n+# cd /guest_repo\n+# git config core.virtualizeobjects true\n+# git reset --hard \n+#\n+# Please note, this sample is a minimal skeleton. No proper error handling \n+# was implemented.\n+#\n+\n+use 5.008;\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n+use strict;\n+use warnings;\n+use Git::Packet;\n+\n+#\n+# Point $DIR to the folder where your host git repo is located so we can pull\n+# missing objects from it\n+#\n+my $DIR = \"../.git/\";\n+\n+packet_initialize(\"git-read-object\", 1);\n+\n+my %remote_caps = packet_read_and_check_capabilities(\"get_git_obj\");\n+packet_check_and_write_capabilities(\\%remote_caps, \"get_git_obj\");\n+\n+while (1) {\n+\tmy ($res, $command) = packet_txt_read();\n+\n+\tif ( $res == -1 ) {\n+\t\texit 0;\n+\t}\n+\n+\t$command =~ s/^command=//;\n+\n+\tif ( $command eq \"init\" ) {\n+\t\tpacket_bin_read();\n+\n+\t\tpacket_txt_write(\"status=success\");\n+\t\tpacket_flush();\n+\t} elsif ( $command eq \"get_git_obj\" ) {\n+\t\tmy ($sha1) = packet_txt_read() =~ /^sha1=([0-9a-f]{40})$/;\n+\t\tpacket_bin_read();\n+\n+\t\tmy $path = $sha1;\n+\t\t$path =~ s{..}{$&/};\n+\t\t$path = $DIR . \"/objects/\" . $path;\n+\n+\t\tmy $contents = do {\n+\t\t local $/;\n+\t\t open my $fh, $path or die \"Can't open '$path': $!\";\n+\t\t <$fh>\n+\t\t};\n+\n+\t\tpacket_bin_write($contents);\n+\t\tpacket_flush();\n+\t\tpacket_txt_write(\"status=success\");\n+\t\tpacket_flush();\n+\t} else {\n+\t\tdie \"bad command '$command'\";\n+\t}\n+}\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342170","messageId":"20180319133147.15413-37-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 36/36] Add Documentation/technical/external-odb.txt","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:47Z","receivedAt":"2018-03-19T13:33:15Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This describes the external odb mechanism's purpose and\nhow it works.\n\nHelped-by: Ben Peart \nSigned-off-by: Christian Couder \n---\n Documentation/technical/external-odb.txt | 342 +++++++++++++++++++++++\n 1 file changed, 342 insertions(+)\n create mode 100644 Documentation/technical/external-odb.txt\n\ndiff --git a/Documentation/technical/external-odb.txt b/Documentation/technical/external-odb.txt\nnew file mode 100644\nindex 0000000000..58ec8a8145\n--- /dev/null\n+++ b/Documentation/technical/external-odb.txt\n@@ -0,0 +1,342 @@\n+External ODBs\n+^^^^^^^^^^^^^\n+\n+The External ODB mechanism makes it possible for Git objects, only\n+blobs for now though, to be stored in an \"external object database\"\n+(External ODB).\n+\n+An External ODB can be any object store as long as there is an helper\n+program called an \"odb helper\" that can communicate with Git to\n+transfer objects to/from the external odb and to retrieve information\n+about available objects in the external odb.\n+\n+Purpose\n+=======\n+\n+The purpose of this mechanism is to make possible to handle Git\n+objects, especially blobs, in much more flexible ways.\n+\n+Currently Git can store its objects only in the form of loose objects\n+in separate files or packed objects in a pack file. These existing\n+object stores cannot be easily optimized for many different kind of\n+contents.\n+\n+So the current stores are not flexible enough for some important use\n+cases like handling really big binary files or handling a really big\n+number of files that are fetched only as needed. And it is not\n+realistic to expect that Git could fully natively handle many of such\n+use cases. Git would need to natively implement different internal\n+stores which would be a huge burden and which could lead to\n+re-implement things like HTTP servers, Docker registries or artifact\n+stores that already exist outside Git.\n+\n+Furthermore many improvements that are dependent on specific setups\n+could be implemented in the way Git objects are managed if it was\n+possible to customize how the Git objects are handled. For example a\n+restartable clone using the bundle mechanism has often been requested,\n+but implementing that would go against the current strict rules under\n+which the Git objects are currently handled.\n+\n+What Git needs is a mechanism to make it possible to customize in a\n+lot of different ways how the Git objects are handled. Though this\n+mechanism should try as much as possible to avoid interfering with the\n+usual way in which Git handle its objects.\n+\n+Helpers\n+=======\n+\n+ODB helpers are commands that have to be registered using either the\n+\"odb..subprocessCommand\" or the \"odb..scriptCommand\"\n+config variables.\n+\n+Registering such a command tells Git that an external odb called\n+ exists and that the registered command should be used to\n+communicate with it.\n+\n+The communication happens through instructions that are sent by Git\n+and that the commands should answer. If it makes sense, Git can send\n+the same instruction to many commands in the order in which they are\n+configured.\n+\n+There are 2 kinds of commands. Commands registered using the\n+\"odb..subprocessCommand\" config variable are called \"process\n+commands\" and the associated mode is called \"process mode\". Commands\n+registered using the \"odb..scriptCommand\" config variables\n+are called \"script commands\" and the associated mode is called \"script\n+mode\".\n+\n+Early on git commands send an 'init' instruction to the registered\n+commands. A capability negociation will take place during this\n+request/response exchange which will let Git and the helpers know how\n+they can further collaborate. The attribute system can also be used to\n+tell Git which objects should be handled by which helper.\n+\n+Process Mode\n+============\n+\n+In process mode the command is started as a single process invocation\n+that should last for the entire life of the single Git command that\n+started it.\n+\n+A packet format (pkt-line, see technical/protocol-common.txt) based\n+protocol over standard input and standard output is used for\n+communication between Git and the helper command.\n+\n+After the process command is started, Git sends a welcome message\n+(\"git-read-object-client\"), a list of supported protocol version\n+numbers, and a flush packet. Git expects to read a welcome response\n+message (\"git-read-object-server\"), exactly one protocol version\n+number from the previously sent list, and a flush packet. All further\n+communication will be based on the selected version.\n+\n+The remaining protocol description below documents \"version=1\". Please\n+note that \"version=42\" in the example below does not exist and is only\n+there to illustrate how the protocol would look with more than one\n+version.\n+\n+After the version negotiation Git sends a list of all capabilities\n+that it supports and a flush packet. Git expects to read a list of\n+desired capabilities, which must be a subset of the supported\n+capabilities list, and a flush packet as response:\n+\n+------------------------\n+packet: git> git-read-object-client\n+packet: git> version=1\n+packet: git> version=42\n+packet: git> 0000\n+packet: git< git-read-object-server\n+packet: git< version=1\n+packet: git< 0000\n+packet: git> capability=get_raw_obj\n+packet: git> capability=have\n+packet: git> capability=put_raw_obj\n+packet: git> capability=not-yet-invented\n+packet: git> 0000\n+packet: git< capability=get_raw_obj\n+packet: git< 0000\n+------------------------\n+\n+Afterwards Git sends a list of \"key=value\" pairs terminated with a\n+flush packet. The list will contain at least the instruction (based on\n+the supported capabilities) and the arguments for the\n+instruction. Please note, that the process must not send any response\n+before it received the final flush packet.\n+\n+In general any response from the helper should end with a status\n+packet. See the documentation of the 'get_*' instructions below for\n+examples of status packets.\n+\n+After the helper has processed an instruction, it is expected to wait\n+for the next \"key=value\" list containing another instruction.\n+\n+On exit Git will close the pipe to the helper. The helper is then\n+expected to detect EOF and exit gracefully on its own. Git will wait\n+until the process has stopped.\n+\n+Script Mode\n+===========\n+\n+In this mode Git launches the script command each time it wants to\n+communicates with the helper. There is no welcome message and no\n+protocol version in this mode.\n+\n+The instruction and associated arguments are passed as arguments when\n+launching the script command and if needed further information is\n+passed between Git and the command through stdin and stdout.\n+\n+Capabilities/Instructions\n+=========================\n+\n+The following instructions are currently supported by Git:\n+\n+- init\n+- get_git_obj\n+- get_raw_obj\n+- get_direct\n+- put_raw_obj\n+- have\n+\n+The plan is to also support 'put_git_obj' and 'put_direct' soon, for\n+consistency with the 'get_*' instructions.\n+\n+ - 'init'\n+\n+All the process and script commands must accept the 'init'\n+instruction. It should be the first instruction sent to a command. It\n+should not be advertised in the capability exchange. Any argument\n+should be ignored.\n+\n+In process mode, after receiving the 'init' instruction and a flush\n+packet, the helper should just send a status packet and then a flush\n+packet. See the 'get_*' instructions below for examples of status\n+packets.\n+\n+In script mode the command should print on stdout the capabilities\n+that it supports if any. This is the only time in script mode when a\n+capability exchange happens.\n+\n+For example a script command could use the following shell code\n+snippet to handle the 'init' instruction:\n+\n+------------------------\n+case \"$1\" in\n+init)\n+\techo \"capability=get_git_obj\"\n+\techo \"capability=put_raw_obj\"\n+\techo \"capability=have\"\n+\t;;\n+------------------------\n+\n+ - 'get_git_obj ' and 'get_raw_obj '\n+\n+These instructions should have a hexadecimal argument to tell\n+which object the helper should send to git.\n+\n+In process mode the sha1 argument should be followed by a flush packet\n+like this:\n+\n+------------------------\n+packet: git> command=get_git_obj\n+packet: git> sha1=0a214a649e1b3d5011e14a3dc227753f2bd2be05\n+packet: git> 0000\n+------------------------\n+\n+After reading that the helper should send the requested object to Git in a\n+packet series followed by a flush packet. If the helper does not experience\n+problems then the helper must send a \"success\" status like the following:\n+\n+------------------------\n+packet: git< status=success\n+packet: git< 0000\n+------------------------\n+\n+In case the helper cannot or does not want to send the requested\n+object as well as any other object for the lifetime of the Git\n+process, then it is expected to respond with an \"abort\" status at any\n+point in the protocol:\n+\n+------------------------\n+packet: git< status=abort\n+packet: git< 0000\n+------------------------\n+\n+Git neither stops nor restarts the helper in case a\n+\"notfound\"/\"error\"/\"abort\" status is set. An \"error\" status means a\n+possibly more transient error than an abort. The helper should also\n+send a \"notfound\" error in case of a \"get_*\" instruction, which means\n+that the requested object cannot be found.\n+\n+If the helper dies during the communication or does not adhere to the\n+protocol then Git will stop and restart it with the next instruction.\n+\n+In script mode the helper should just send the requested object to Git\n+by writing it to stdout and should then exit. The exit code should\n+signal to Git if a problem occured or not.\n+\n+The only difference between 'get_git_obj' and 'get_raw_obj' is that in\n+case of 'get_git_obj' the requested object should be sent as a Git\n+object (that is in the same format as loose object files). In case of\n+'get_raw_obj' the object should be sent in its raw format (that is the\n+same output as `git cat-file `).\n+\n+ - 'get_direct '\n+\n+This instruction is similar as the other 'get_*' instructions except\n+that no object should be sent from the helper to Git. Instead the\n+helper should directly write the requested object into a loose object\n+file in the \".git/objects\" directory.\n+\n+After the helper has sent the \"status=success\" packet and the\n+following flush packet in process mode, or after it has exited in the\n+script mode, Git will lookup again for the requested sha1 in its loose\n+object files and pack files.\n+\n+ - 'put_raw_obj '\n+\n+This instruction should be following by three arguments to tell which\n+object the helper will receive from git: , and\n+. The hexadecimal argument describes the object that will\n+be sent from Git to the helper. The is the object type (\"blob\",\n+\"tree\", \"commit\" or \"tag\") of this object. The is the size in\n+bytes of the (decompressed) object content.\n+\n+In process mode the last argument (the type) should be followed by a\n+flush packet.\n+\n+After reading that the helper should read the announced object from\n+Git in a packet series followed by a flush packet.\n+\n+If the helper does not experience problems when receiving and storing\n+or processing the object, then the helper must send a \"success\" status\n+as described for the 'get_*' instructions.\n+\n+In script mode the helper should just receive the announced object\n+from its standard input. After receiving and processing the object,\n+the helper should exit and its exit code should signal to Git if a\n+problem occured or not.\n+\n+- 'have'\n+\n+In process mode this instruction should be followed by a flush\n+packet. After receiving this packet the helper should send the sha1,\n+size and type, in this order, of all the objects it can provide to Git\n+(through a 'get_*' instruction). There should be a space character\n+between the sha1 and the size and between the size and the type, and\n+then a new line character after the type.\n+\n+If many packets are needed to send back all this information, the\n+split between packets should be made after the new line characters.\n+\n+If the helper does not experience problems, then it must then send a\n+\"success\" status as described for the 'get_*' instructions.\n+\n+In script mode the helper should send to its standard output the sha1,\n+size and type, in this order of all the objects it can provide to\n+Git. There should also be a space character between the sha1 and the\n+size and between the size and the type, and then a new line character\n+after the type.\n+\n+After sending this, the script helper should exit and its exit code\n+should signal to Git if a problem occured or not.\n+\n+Order of instructions\n+=====================\n+\n+For get_*_object instructions the regular code to find objects is\n+called before the odb helpers.\n+\n+For put_*_object instructions the regular code to store the objects is\n+called after the odb helpers.\n+\n+For now this order is not configurable.\n+\n+Object caching\n+==============\n+\n+If a helper returns the object data as requested by get_git_obj or\n+get_raw_obj, then Git will itself store the object locally in its\n+regular object store, so it is redundant for the helper to also store\n+or try to store the object in the regular object store.\n+\n+Yeah, this seems to defeat the goal of enabling specialized object\n+handlers to handle large or other \"unusual\" objects that git normally\n+doesn't deal well with. So in the long run there should be a way to\n+make this configurable.\n+\n+Selecting objects\n+=================\n+\n+To select objects that should be handled by an external odb, one can\n+use the git attributes system. For now this will only work with blobs\n+and this will only work along with the 'put_raw_obj' instruction.\n+\n+For example if one has an external odb called \"magic\" and has\n+registered an associated a process command helper that supports the\n+'put_raw_obj' instruction, then one can tell Git that all the .jpg\n+files should be handled by the \"magic\" odb using a .gitattributes file\n+can that contains:\n+\n+------------------------\n+*.jpg odb=magic\n+------------------------\n+\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342171","messageId":"20180319133147.15413-12-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 11/36] odb-helper: add odb_helper_init() to send 'init' instruction","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:22Z","receivedAt":"2018-03-19T13:33:21Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Let's add an odb_helper_init() function to send an 'init'\ninstruction to the helpers. This 'init' instruction is\nespecially useful to get the capabilities that are supported\nby the helpers.\n\nSo while at it, let's also add a parse_capabilities()\nfunction to parse them and a supported_capabilities\nvariable in struct odb_helper to store them.\n\nSigned-off-by: Christian Couder \n---\n external-odb.c | 13 ++++++++-\n odb-helper.c | 58 +++++++++++++++++++++++++++++++++++++++++\n odb-helper.h | 11 ++++++++\n t/t0500-external-odb.sh | 4 +++\n 4 files changed, 85 insertions(+), 1 deletion(-)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex 9f797d66f4..ae6bc017fe 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -35,6 +35,8 @@ static int external_odb_config(const char *var, const char *value, void *data)\n \n \tif (!strcmp(subkey, \"promisorremote\")) {\n \t\to->type = ODB_HELPER_GIT_REMOTE;\n+\t\to->supported_capabilities |= ODB_HELPER_CAP_HAVE;\n+\t\to->supported_capabilities |= ODB_HELPER_CAP_GET_DIRECT;\n \t\treturn git_config_string(&o->dealer, var, value);\n \t}\n \tif (!strcmp(subkey, \"scriptcommand\")) {\n@@ -52,12 +54,16 @@ static int external_odb_config(const char *var, const char *value, void *data)\n static void external_odb_do_init(int force)\n {\n \tstatic int initialized;\n+\tstruct odb_helper *o;\n \n \tif ((!force && initialized) || !use_external_odb)\n \t\treturn;\n \tinitialized = 1;\n \n \tgit_config(external_odb_config, NULL);\n+\n+\tfor (o = helpers; o; o = o->next)\n+\t\todb_helper_init(o);\n }\n \n static inline void external_odb_init(void)\n@@ -104,9 +110,12 @@ int external_odb_has_object(const unsigned char *sha1)\n \n \texternal_odb_init();\n \n-\tfor (o = helpers; o; o = o->next)\n+\tfor (o = helpers; o; o = o->next) {\n+\t\tif (!(o->supported_capabilities & ODB_HELPER_CAP_HAVE))\n+\t\t\treturn 1;\n \t\tif (odb_helper_has_object(o, sha1))\n \t\t\treturn 1;\n+\t}\n \treturn 0;\n }\n \n@@ -164,6 +173,8 @@ int external_odb_get_direct(const unsigned char *sha1)\n \texternal_odb_init();\n \n \tfor (o = helpers; o; o = o->next) {\n+\t\tif (!(o->supported_capabilities & ODB_HELPER_CAP_GET_DIRECT))\n+\t\t\tcontinue;\n \t\tif (odb_helper_get_direct(o, sha1) < 0)\n \t\t\tcontinue;\n \t\treturn 0;\ndiff --git a/odb-helper.c b/odb-helper.c\nindex bf3c1ad8ac..a99f4a1299 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -6,6 +6,40 @@\n #include \"sha1-lookup.h\"\n #include \"fetch-object.h\"\n \n+static void parse_capabilities(char *cap_buf,\n+\t\t\t unsigned int *supported_capabilities,\n+\t\t\t const char *process_name)\n+{\n+\tstruct string_list cap_list = STRING_LIST_INIT_NODUP;\n+\n+\tstring_list_split_in_place(&cap_list, cap_buf, '=', 1);\n+\n+\tif (cap_list.nr == 2 && !strcmp(cap_list.items[0].string, \"capability\")) {\n+\t\tconst char *cap_name = cap_list.items[1].string;\n+\n+\t\tif (!strcmp(cap_name, \"get_git_obj\")) {\n+\t\t\t*supported_capabilities |= ODB_HELPER_CAP_GET_GIT_OBJ;\n+\t\t} else if (!strcmp(cap_name, \"get_raw_obj\")) {\n+\t\t\t*supported_capabilities |= ODB_HELPER_CAP_GET_RAW_OBJ;\n+\t\t} else if (!strcmp(cap_name, \"get_direct\")) {\n+\t\t\t*supported_capabilities |= ODB_HELPER_CAP_GET_DIRECT;\n+\t\t} else if (!strcmp(cap_name, \"put_git_obj\")) {\n+\t\t\t*supported_capabilities |= ODB_HELPER_CAP_PUT_GIT_OBJ;\n+\t\t} else if (!strcmp(cap_name, \"put_raw_obj\")) {\n+\t\t\t*supported_capabilities |= ODB_HELPER_CAP_PUT_RAW_OBJ;\n+\t\t} else if (!strcmp(cap_name, \"put_direct\")) {\n+\t\t\t*supported_capabilities |= ODB_HELPER_CAP_PUT_DIRECT;\n+\t\t} else if (!strcmp(cap_name, \"have\")) {\n+\t\t\t*supported_capabilities |= ODB_HELPER_CAP_HAVE;\n+\t\t} else {\n+\t\t\twarning(\"external process '%s' requested unsupported read-object capability '%s'\",\n+\t\t\t\tprocess_name, cap_name);\n+\t\t}\n+\t}\n+\n+\tstring_list_clear(&cap_list, 0);\n+}\n+\n struct odb_helper *odb_helper_new(const char *name, int namelen)\n {\n \tstruct odb_helper *o;\n@@ -80,6 +114,30 @@ static int odb_helper_finish(struct odb_helper *o,\n \treturn 0;\n }\n \n+int odb_helper_init(struct odb_helper *o)\n+{\n+\tstruct odb_helper_cmd cmd;\n+\tFILE *fh;\n+\tstruct strbuf line = STRBUF_INIT;\n+\n+\tif (o->initialized)\n+\t\treturn 0;\n+\to->initialized = 1;\n+\n+\tif (odb_helper_start(o, &cmd, \"init\") < 0)\n+\t\treturn -1;\n+\n+\tfh = xfdopen(cmd.child.out, \"r\");\n+\twhile (strbuf_getline(&line, fh) != EOF)\n+\t\tparse_capabilities(line.buf, &o->supported_capabilities, o->name);\n+\n+\tstrbuf_release(&line);\n+\tfclose(fh);\n+\todb_helper_finish(o, &cmd);\n+\n+\treturn 0;\n+}\n+\n static int parse_object_line(struct odb_helper_object *o, const char *line)\n {\n \tchar *end;\ndiff --git a/odb-helper.h b/odb-helper.h\nindex 819182f76a..7720684744 100644\n--- a/odb-helper.h\n+++ b/odb-helper.h\n@@ -3,10 +3,20 @@\n \n #include \"external-odb.h\"\n \n+#define ODB_HELPER_CAP_GET_GIT_OBJ (1u<<0)\n+#define ODB_HELPER_CAP_GET_RAW_OBJ (1u<<1)\n+#define ODB_HELPER_CAP_GET_DIRECT (1u<<2)\n+#define ODB_HELPER_CAP_PUT_GIT_OBJ (1u<<3)\n+#define ODB_HELPER_CAP_PUT_RAW_OBJ (1u<<4)\n+#define ODB_HELPER_CAP_PUT_DIRECT (1u<<5)\n+#define ODB_HELPER_CAP_HAVE (1u<<6)\n+\n struct odb_helper {\n \tconst char *name;\n \tconst char *dealer;\n \tenum odb_helper_type type;\n+\tunsigned int supported_capabilities;\n+\tint initialized;\n \n \tstruct odb_helper_object {\n \t\tstruct object_id oid;\n@@ -21,6 +31,7 @@ struct odb_helper {\n };\n \n extern struct odb_helper *odb_helper_new(const char *name, int namelen);\n+extern int odb_helper_init(struct odb_helper *o);\n extern int odb_helper_has_object(struct odb_helper *o,\n \t\t\t\t const unsigned char *sha1);\n extern int odb_helper_get_object(struct odb_helper *o,\ndiff --git a/t/t0500-external-odb.sh b/t/t0500-external-odb.sh\nindex eaa3688a31..977fea852d 100755\n--- a/t/t0500-external-odb.sh\n+++ b/t/t0500-external-odb.sh\n@@ -9,6 +9,10 @@ export ALT_SOURCE\n write_script odb-helper <<\\EOF\n GIT_DIR=$ALT_SOURCE; export GIT_DIR\n case \"$1\" in\n+init)\n+\techo \"capability=get_git_obj\"\n+\techo \"capability=have\"\n+\t;;\n have)\n \tgit cat-file --batch-check --batch-all-objects |\n \tawk '{print $1 \" \" $3 \" \" $2}'\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342172","messageId":"20180319133147.15413-35-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 34/36] Add t0580 to test \"have\" capability and raw objects","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:45Z","receivedAt":"2018-03-19T13:33:27Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n t/t0580-read-object-have-http-e-odb.sh | 109 +++++++++++++++++++++++++\n t/t0580/read-object-plain-have | 103 +++++++++++++++++++++++\n 2 files changed, 212 insertions(+)\n create mode 100755 t/t0580-read-object-have-http-e-odb.sh\n create mode 100755 t/t0580/read-object-plain-have\n\ndiff --git a/t/t0580-read-object-have-http-e-odb.sh b/t/t0580-read-object-have-http-e-odb.sh\nnew file mode 100755\nindex 0000000000..9cbfb18d28\n--- /dev/null\n+++ b/t/t0580-read-object-have-http-e-odb.sh\n@@ -0,0 +1,109 @@\n+#!/bin/sh\n+\n+test_description='tests for read-object process with \"have\" cap and plain objects'\n+\n+. ./test-lib.sh\n+\n+# If we don't specify a port, the current test number will be used\n+# which will not work as it is less than 1024, so it can only be used by root.\n+LIB_HTTPD_PORT=$(expr ${this_test#t} + 12000)\n+\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+\n+start_httpd apache-e-odb.conf\n+\n+PATH=\"$PATH:$TEST_DIRECTORY/t0580\"\n+\n+# odb helper script must see this\n+export HTTPD_URL\n+\n+HELPER=\"read-object-plain-have\"\n+\n+test_expect_success 'setup repo with a root commit' '\n+\ttest_commit zero\n+'\n+\n+test_expect_success 'setup another repo from the first one' '\n+\tgit init other-repo &&\n+\t(cd other-repo &&\n+\t git remote add origin .. &&\n+\t git pull origin master &&\n+\t git checkout master &&\n+\t git log)\n+'\n+\n+test_expect_success 'setup the helper in the root repo' '\n+\tgit config odb.magic.subprocessCommand \"$HELPER\"\n+'\n+\n+UPLOADFILENAME=\"hello_apache_upload.txt\"\n+\n+UPLOAD_URL=\"$HTTPD_URL/upload/?sha1=$UPLOADFILENAME&size=123&type=blob\"\n+\n+test_expect_success 'can upload a file' '\n+\techo \"Hello Apache World!\" >hello_to_send.txt &&\n+\techo \"How are you?\" >>hello_to_send.txt &&\n+\tcurl --data-binary @hello_to_send.txt --include \"$UPLOAD_URL\" >out_upload\n+'\n+\n+LIST_URL=\"$HTTPD_URL/list/\"\n+\n+test_expect_success 'can list uploaded files' '\n+\tcurl --include \"$LIST_URL\" >out_list &&\n+\tgrep \"$UPLOADFILENAME\" out_list\n+'\n+\n+test_expect_success 'can delete uploaded files' '\n+\tcurl --data \"delete\" --include \"$UPLOAD_URL&delete=1\" >out_delete &&\n+\tcurl --include \"$LIST_URL\" >out_list2 &&\n+\t! grep \"$UPLOADFILENAME\" out_list2\n+'\n+\n+FILES_DIR=\"httpd/www/files\"\n+\n+test_expect_success 'new blobs are transfered to the http server' '\n+\ttest_commit one &&\n+\thash1=$(git ls-tree HEAD | grep one.t | cut -f1 | cut -d\\ -f3) &&\n+\techo \"$hash1-4-blob\" >expected &&\n+\tls \"$FILES_DIR\" >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'blobs can be retrieved from the http server' '\n+\tgit cat-file blob \"$hash1\" &&\n+\tgit log -p >expected\n+'\n+\n+test_expect_success 'update other repo from the first one' '\n+\t(cd other-repo &&\n+\t git fetch origin \"refs/odbs/magic/*:refs/odbs/magic/*\" &&\n+\t test_must_fail git cat-file blob \"$hash1\" &&\n+\t git config odb.magic.subprocessCommand \"$HELPER\" &&\n+\t git cat-file blob \"$hash1\" &&\n+\t git pull origin master)\n+'\n+\n+test_expect_success 'local clone from the first repo' '\n+\tmkdir my-clone &&\n+\t(cd my-clone &&\n+\t git clone .. . &&\n+\t git cat-file blob \"$hash1\")\n+'\n+\n+test_expect_success 'no-local clone from the first repo fails' '\n+\tmkdir my-other-clone &&\n+\t(cd my-other-clone &&\n+\t test_must_fail git clone --no-local .. .) &&\n+\trm -rf my-other-clone\n+'\n+\n+test_expect_success 'no-local clone from the first repo with helper succeeds' '\n+\tmkdir my-other-clone &&\n+\t(cd my-other-clone &&\n+\t git clone -c odb.magic.subprocessCommand=\"$HELPER\" --no-local .. .) &&\n+\trm -rf my-other-clone\n+'\n+\n+stop_httpd\n+\n+test_done\ndiff --git a/t/t0580/read-object-plain-have b/t/t0580/read-object-plain-have\nnew file mode 100755\nindex 0000000000..f230cbd5eb\n--- /dev/null\n+++ b/t/t0580/read-object-plain-have\n@@ -0,0 +1,103 @@\n+#!/usr/bin/perl\n+#\n+\n+use 5.008;\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n+use strict;\n+use warnings;\n+use Git::Packet;\n+use LWP::UserAgent;\n+use HTTP::Request::Common;\n+\n+packet_initialize(\"git-read-object\", 1);\n+\n+my %remote_caps = packet_read_and_check_capabilities(\"get_raw_obj\", \"put_raw_obj\", \"have\");\n+packet_check_and_write_capabilities(\\%remote_caps, \"get_raw_obj\", \"put_raw_obj\", \"have\");\n+\n+my $http_url = $ENV{HTTPD_URL};\n+\n+while (1) {\n+\tmy ($res, $command) = packet_txt_read();\n+\n+\tif ( $res == -1 ) {\n+\t\texit 0;\n+\t}\n+\n+\t$command =~ s/^command=//;\n+\n+\tif ( $command eq \"init\" ) {\n+\t\tpacket_bin_read();\n+\n+\t\tpacket_txt_write(\"status=success\");\n+\t\tpacket_flush();\n+\t} elsif ( $command eq \"have\" ) {\n+\t\t# read the flush after the command\n+\t\tpacket_bin_read();\n+\n+\t\tmy $have_url = $http_url . \"/list/\";\n+\n+\t\tmy $userAgent = LWP::UserAgent->new();\n+\t\tmy $response = $userAgent->get( $have_url );\n+\n+\t\tif ($response->is_error) {\n+\t\t\tpacket_bin_write(\"\");\n+\t\t\tpacket_flush();\n+\t\t\tpacket_txt_write(\"status=failure\");\n+\t\t} else {\n+\t\t\tpacket_bin_write($response->content);\n+\t\t\tpacket_flush();\n+\t\t\tpacket_txt_write(\"status=success\");\n+\t\t}\n+\t\tpacket_flush();\n+\t} elsif ( $command eq \"get_raw_obj\" ) {\n+\t\tmy ($sha1) = packet_txt_read() =~ /^sha1=([0-9a-f]{40})$/;\n+\t\tpacket_bin_read();\n+\n+\t\tmy $get_url = $http_url . \"/list/?sha1=\" . $sha1;\n+\n+\t\tmy $userAgent = LWP::UserAgent->new();\n+\n+\t\tmy $response = $userAgent->get( $get_url );\n+\n+\t\tif ($response->is_error) {\n+\t\t\tpacket_txt_write(\"size=0\");\n+\t\t\tpacket_txt_write(\"kind=none\");\t \n+\t\t\tpacket_txt_write(\"status=notfound\");\n+\t\t} else {\n+\t\t\tpacket_txt_write(\"size=\" . length($response->content));\n+\t\t\tpacket_txt_write(\"kind=blob\");\n+\t\t\tpacket_bin_write($response->content);\n+\t\t\tpacket_flush();\n+\t\t\tpacket_txt_write(\"status=success\");\n+\t\t}\n+\n+\t\tpacket_flush();\n+\t} elsif ( $command eq \"put_raw_obj\" ) {\n+\t\tmy ($sha1) = packet_txt_read() =~ /^sha1=([0-9a-f]{40})$/;\n+\t\tmy ($size) = packet_txt_read() =~ /^size=([0-9]+)$/;\n+\t\tmy ($kind) = packet_txt_read() =~ /^kind=(\\w+)$/;\n+\n+\t\tpacket_bin_read();\n+\n+\t\t# We must read the content we are sent and send it to the right url\n+\t\tmy ($res, $buf) = packet_bin_read();\n+\t\tdie \"bad packet_bin_read res ($res)\" unless ($res eq 0);\n+\t\t( packet_bin_read() eq ( 1, \"\" ) ) || die \"bad send end\";\t\t\n+\n+\t\tmy $upload_url = $http_url . \"/upload/?sha1=\" . $sha1 . \"&size=\" . $size . \"&type=blob\";\n+\n+\t\tmy $userAgent = LWP::UserAgent->new();\n+\t\tmy $request = POST $upload_url, Content_Type => 'multipart/form-data', Content => $buf;\n+\n+\t\tmy $response = $userAgent->request($request);\n+\n+\t\tif ($response->is_error) {\n+\t\t\tpacket_txt_write(\"status=failure\");\n+\t\t} else {\n+\t\t\tpacket_txt_write(\"status=success\");\n+\t\t}\n+\t\tpacket_flush();\n+\t} else {\n+\t\tdie \"bad command '$command'\";\n+\t}\n+}\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342173","messageId":"20180319133147.15413-34-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 33/36] odb-helper: add have_object_process()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:44Z","receivedAt":"2018-03-19T13:33:29Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This adds the infrastructure to handle 'have' instructions in\nprocess mode.\n\nThe answer from the helper sub-process should be like the\noutput in script mode, that is lines like this:\n\nsha1 SPACE size SPACE type NEWLINE\n\nSigned-off-by: Christian Couder \n---\n odb-helper.c | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 73 insertions(+), 1 deletion(-)\n\ndiff --git a/odb-helper.c b/odb-helper.c\nindex 83f8cce3b1..5ec21e9d1f 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -649,6 +649,70 @@ static int odb_helper_object_cmp(const void *va, const void *vb)\n \treturn oidcmp(&a->oid, &b->oid);\n }\n \n+static int send_have_packets(struct odb_helper *o,\n+\t\t\t struct object_process *entry,\n+\t\t\t struct strbuf *status)\n+{\n+\tint packet_len;\n+\tint total_got = 0;\n+\tstruct child_process *process = &entry->subprocess.process;\n+\tint err = packet_write_fmt_gently(process->in, \"command=have\\n\");\n+\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = packet_flush_gently(process->in);\n+\tif (err)\n+\t\treturn err;\n+\n+\tfor (;;) {\n+\t\t/* packet_read() writes a '\\0' extra byte at the end */\n+\t\tchar buf[LARGE_PACKET_DATA_MAX + 1];\n+\t\tchar *p = buf;\n+\t\tint more;\n+\n+\t\tpacket_len = packet_read(process->out, NULL, NULL,\n+\t\t\tbuf, LARGE_PACKET_DATA_MAX + 1,\n+\t\t\tPACKET_READ_GENTLE_ON_EOF);\n+\n+\t\tif (packet_len <= 0)\n+\t\t\tbreak;\n+\n+\t\ttotal_got += packet_len;\n+\n+\t\t/* 'have' packets should end with '\\n' or '\\0' */\n+\t\tdo {\n+\t\t\tchar *eol = strchrnul(p, '\\n');\n+\t\t\tmore = (*eol == '\\n');\n+\t\t\t*eol = '\\0';\n+\t\t\tif (add_have_entry(o, p))\n+\t\t\t\tbreak;\n+\t\t\tp = eol + 1;\n+\t\t} while (more && *p);\n+\t}\n+\n+\tif (packet_len < 0)\n+\t\treturn packet_len;\n+\n+\treturn check_object_process_status(process->out, status);\n+}\n+\n+static int have_object_process(struct odb_helper *o)\n+{\n+\tint err;\n+\tstruct object_process *entry;\n+\tstruct strbuf status = STRBUF_INIT;\n+\n+\tentry = launch_object_process(o, ODB_HELPER_CAP_HAVE);\n+\tif (!entry)\n+\t\treturn -1;\n+\n+\terr = send_have_packets(o, entry, &status);\n+\n+\treturn check_object_process_error(err, status.buf, entry, o->dealer,\n+\t\t\t\t\t ODB_HELPER_CAP_HAVE);\n+}\n+\n static void have_object_script(struct odb_helper *o)\n {\n \tstruct odb_helper_cmd cmd;\n@@ -670,12 +734,20 @@ static void have_object_script(struct odb_helper *o)\n \n static void odb_helper_load_have(struct odb_helper *o)\n {\n+\tuint64_t start;\n+\n \tif (o->have_valid)\n \t\treturn;\n \to->have_valid = 1;\n \n+\tstart = getnanotime();\n+\n \tif (o->type == ODB_HELPER_SCRIPT_CMD)\n \t\thave_object_script(o);\n+\telse if (o->type == ODB_HELPER_SUBPROCESS_CMD)\n+\t\thave_object_process(o);\n+\n+\ttrace_performance_since(start, \"odb_helper_load_have\");\n \n \tqsort(o->have, o->have_nr, sizeof(*o->have), odb_helper_object_cmp);\n }\n@@ -937,7 +1009,7 @@ int odb_helper_get_direct(struct odb_helper *o,\n \t\tfetch_object(o->dealer, sha1);\n \telse if (o->type == ODB_HELPER_SCRIPT_CMD)\n \t\tres = get_direct_script(o, sha1);\n-\telse\n+\telse if (o->type == ODB_HELPER_SUBPROCESS_CMD)\n \t\tres = get_object_process(o, sha1, -1);\n \n \ttrace_performance_since(start, \"odb_helper_get_direct\");\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342174","messageId":"20180319133147.15413-36-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 35/36] external-odb: use 'odb=magic' attribute to mark odb blobs","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:46Z","receivedAt":"2018-03-19T13:33:43Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"To tell which blobs should be sent to the \"magic\" external odb,\nlet's require that the blobs be marked using the 'odb=magic'\nattribute.\n\nSigned-off-by: Christian Couder \n---\n external-odb.c | 25 ++++++++++++++++++++++---\n external-odb.h | 3 ++-\n sha1_file.c | 20 ++++++++++++++------\n t/t0500-external-odb.sh | 3 +++\n t/t0510-transfer-e-odb.sh | 3 +++\n t/t0520-transfer-http-e-odb.sh | 3 +++\n t/t0570-read-object-http-e-odb.sh | 3 +++\n t/t0580-read-object-have-http-e-odb.sh | 3 +++\n 8 files changed, 53 insertions(+), 10 deletions(-)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex 5e5871c933..af2fc4a120 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -2,6 +2,7 @@\n #include \"external-odb.h\"\n #include \"odb-helper.h\"\n #include \"config.h\"\n+#include \"attr.h\"\n \n static struct odb_helper *helpers;\n static struct odb_helper **helpers_tail = &helpers;\n@@ -211,19 +212,37 @@ int external_odb_get_object(const unsigned char *sha1)\n \treturn external_odb_do_get_object(sha1);\n }\n \n+static int has_odb_attrs(struct odb_helper *o, const char *path)\n+{\n+\tstatic struct attr_check *check;\n+\n+\tif (!check)\n+\t\tcheck = attr_check_initl(\"odb\", NULL);\n+\n+\tif (!git_check_attr(path, check)) {\n+\t\tconst char *value = check->items[0].value;\n+\t\treturn value ? !strcmp(o->name, value) : 0;\n+\t}\n+\treturn 0;\n+}\n+\n int external_odb_put_object(const void *buf, size_t len,\n-\t\t\t const char *type, unsigned char *sha1)\n+\t\t\t const char *type, unsigned char *sha1,\n+\t\t\t const char *path)\n {\n \tstruct odb_helper *o;\n \n \texternal_odb_init();\n \n \t/* For now accept only blobs */\n-\tif (strcmp(type, \"blob\"))\n+\tif (!path || strcmp(type, \"blob\"))\n \t\treturn 1;\n \n \tfor (o = helpers; o; o = o->next) {\n-\t\tint r = odb_helper_put_object(o, buf, len, type, sha1);\n+\t\tint r;\n+\t\tif (!has_odb_attrs(o, path))\n+\t\t\tcontinue;\n+\t\tr = odb_helper_put_object(o, buf, len, type, sha1);\n \t\tif (r <= 0)\n \t\t\treturn r;\n \t}\ndiff --git a/external-odb.h b/external-odb.h\nindex 60da8ed3d4..a7f4d7eec0 100644\n--- a/external-odb.h\n+++ b/external-odb.h\n@@ -19,6 +19,7 @@ extern int external_odb_get_object(const unsigned char *sha1);\n extern int external_odb_get_direct(const unsigned char *sha1);\n extern int external_odb_get_many_direct(const struct oid_array *to_get);\n extern int external_odb_put_object(const void *buf, size_t len,\n-\t\t\t\t const char *type, unsigned char *sha1);\n+\t\t\t\t const char *type, unsigned char *sha1,\n+\t\t\t\t const char *path);\n \n #endif /* EXTERNAL_ODB_H */\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 9ccdae35bc..b8e5af3cb8 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1708,8 +1708,9 @@ static int freshen_packed_object(const unsigned char *sha1)\n \treturn 1;\n }\n \n-int write_object_file(const void *buf, unsigned long len, const char *type,\n-\t\t struct object_id *oid)\n+static int write_object_file_with_path(const void *buf, unsigned long len,\n+\t\t\t\t const char *type, struct object_id *oid,\n+\t\t\t\t const char *path)\n {\n \tchar hdr[32];\n \tint hdrlen = sizeof(hdr);\n@@ -1718,13 +1719,19 @@ int write_object_file(const void *buf, unsigned long len, const char *type,\n \t * it out into .git/objects/??/?{38} file.\n \t */\n \twrite_object_file_prepare(buf, len, type, oid, hdr, &hdrlen);\n-\tif (!external_odb_put_object(buf, len, type, oid->hash))\n+\tif (!external_odb_put_object(buf, len, type, oid->hash, path))\n \t\treturn 0;\n \tif (freshen_packed_object(oid->hash) || freshen_loose_object(oid->hash))\n \t\treturn 0;\n \treturn write_loose_object(oid, hdr, hdrlen, buf, len, 0);\n }\n \n+int write_object_file(const void *buf, unsigned long len,\n+\t\t const char *type, struct object_id *oid)\n+{\n+\treturn write_object_file_with_path(buf, len, type, oid, NULL);\n+}\n+\n int hash_object_file_literally(const void *buf, unsigned long len,\n \t\t\t const char *type, struct object_id *oid,\n \t\t\t unsigned flags)\n@@ -1846,7 +1853,8 @@ static int index_mem(struct object_id *oid, void *buf, size_t size,\n \t}\n \n \tif (write_object)\n-\t\tret = write_object_file(buf, size, type_name(type), oid);\n+\t\tret = write_object_file_with_path(buf, size, type_name(type),\n+\t\t\t\t\t\t oid, path);\n \telse\n \t\tret = hash_object_file(buf, size, type_name(type), oid);\n \tif (re_allocated)\n@@ -1868,8 +1876,8 @@ static int index_stream_convert_blob(struct object_id *oid, int fd,\n \t\t\t\t get_conv_flags(flags));\n \n \tif (write_object)\n-\t\tret = write_object_file(sbuf.buf, sbuf.len, type_name(OBJ_BLOB),\n-\t\t\t\t\toid);\n+\t\tret = write_object_file_with_path(sbuf.buf, sbuf.len, type_name(OBJ_BLOB),\n+\t\t\t\t\t\t oid, path);\n \telse\n \t\tret = hash_object_file(sbuf.buf, sbuf.len, type_name(OBJ_BLOB),\n \t\t\t\t oid);\ndiff --git a/t/t0500-external-odb.sh b/t/t0500-external-odb.sh\nindex f924de870f..944b7bb28f 100755\n--- a/t/t0500-external-odb.sh\n+++ b/t/t0500-external-odb.sh\n@@ -71,6 +71,9 @@ test_expect_success 'helper can add objects to alt repo' '\n \n test_expect_success 'commit adds objects to alt repo' '\n \ttest_config odb.magic.scriptCommand \"$HELPER\" &&\n+\techo \"* odb=magic\" >.gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git add .gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git commit -m \"Add .gitattributes\" &&\n \ttest_commit three &&\n \thash3=$(git ls-tree HEAD | grep three.t | cut -f1 | cut -d\\ -f3) &&\n \tcontent=$(git -C alt-repo show \"$hash3\") &&\ndiff --git a/t/t0510-transfer-e-odb.sh b/t/t0510-transfer-e-odb.sh\nindex 065ec7d759..fd3e37918c 100755\n--- a/t/t0510-transfer-e-odb.sh\n+++ b/t/t0510-transfer-e-odb.sh\n@@ -111,6 +111,9 @@ test_expect_success 'setup other repo and its alternate repo' '\n '\n \n test_expect_success 'new blobs are put in first object store' '\n+\techo \"* odb=magic\" >.gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git add .gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git commit -m \"Add .gitattributes\" &&\n \ttest_commit one &&\n \thash1=$(git ls-tree HEAD | grep one.t | cut -f1 | cut -d\\ -f3) &&\n \tcontent=$(cd alt-repo1 && git show \"$hash1\") &&\ndiff --git a/t/t0520-transfer-http-e-odb.sh b/t/t0520-transfer-http-e-odb.sh\nindex f84fe950ec..d307af0457 100755\n--- a/t/t0520-transfer-http-e-odb.sh\n+++ b/t/t0520-transfer-http-e-odb.sh\n@@ -94,6 +94,9 @@ test_expect_success 'can delete uploaded files' '\n FILES_DIR=\"httpd/www/files\"\n \n test_expect_success 'new blobs are transfered to the http server' '\n+\techo \"* odb=magic\" >.gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git add .gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git commit -m \"Add .gitattributes\" &&\n \ttest_commit one &&\n \thash1=$(git ls-tree HEAD | grep one.t | cut -f1 | cut -d\\ -f3) &&\n \techo \"$hash1-4-blob\" >expected &&\ndiff --git a/t/t0570-read-object-http-e-odb.sh b/t/t0570-read-object-http-e-odb.sh\nindex 1b2205e273..56980b700a 100755\n--- a/t/t0570-read-object-http-e-odb.sh\n+++ b/t/t0570-read-object-http-e-odb.sh\n@@ -62,6 +62,9 @@ test_expect_success 'can delete uploaded files' '\n FILES_DIR=\"httpd/www/files\"\n \n test_expect_success 'new blobs are transfered to the http server' '\n+\techo \"* odb=magic\" >.gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git add .gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git commit -m \"Add .gitattributes\" &&\n \ttest_commit one &&\n \thash1=$(git ls-tree HEAD | grep one.t | cut -f1 | cut -d\\ -f3) &&\n \techo \"$hash1-4-blob\" >expected &&\ndiff --git a/t/t0580-read-object-have-http-e-odb.sh b/t/t0580-read-object-have-http-e-odb.sh\nindex 9cbfb18d28..fea0205e2e 100755\n--- a/t/t0580-read-object-have-http-e-odb.sh\n+++ b/t/t0580-read-object-have-http-e-odb.sh\n@@ -62,6 +62,9 @@ test_expect_success 'can delete uploaded files' '\n FILES_DIR=\"httpd/www/files\"\n \n test_expect_success 'new blobs are transfered to the http server' '\n+\techo \"* odb=magic\" >.gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git add .gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git commit -m \"Add .gitattributes\" &&\n \ttest_commit one &&\n \thash1=$(git ls-tree HEAD | grep one.t | cut -f1 | cut -d\\ -f3) &&\n \techo \"$hash1-4-blob\" >expected &&\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342175","messageId":"20180319133147.15413-16-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 15/36] external-odb: add 'get_direct' support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:26Z","receivedAt":"2018-03-19T13:33:54Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This implements the 'get_direct' capability/instruction that makes\nit possible for external odb helper scripts to pass blobs to Git\nby directly writing them as loose objects files.\n\nIt is better to call this a \"direct\" mode rather than a \"fault-in\"\nmode as we could have the same kind of mechanism to \"put\" objects\ninto an external odb, where the odb helper would access blobs it\nwants to send to an external odb directly from files, but it\nwould be strange to call that a fault-in mode too.\n\nSigned-off-by: Christian Couder \n---\n external-odb.c | 3 ++-\n odb-helper.c | 20 +++++++++++++++++---\n 2 files changed, 19 insertions(+), 4 deletions(-)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex c3d90f39fb..192b09e843 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -138,7 +138,8 @@ int external_odb_get_object(const unsigned char *sha1)\n \t\tint ret;\n \t\tint fd;\n \n-\t\tif (!odb_helper_has_object(o, sha1))\n+\t\tif (!(o->supported_capabilities & ODB_HELPER_CAP_GET_RAW_OBJ) &&\n+\t\t !(o->supported_capabilities & ODB_HELPER_CAP_GET_GIT_OBJ))\n \t\t\tcontinue;\n \n \t\tfd = create_object_tmpfile(&tmpfile, pathbuf.buf);\ndiff --git a/odb-helper.c b/odb-helper.c\nindex 7e5eab94cf..30cfe689de 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -336,14 +336,28 @@ int odb_helper_get_object(struct odb_helper *o, const unsigned char *sha1,\n int odb_helper_get_direct(struct odb_helper *o,\n \t\t\t const unsigned char *sha1)\n {\n-\tint res = 0;\n \tuint64_t start = getnanotime();\n \n-\tfetch_object(o->dealer, sha1);\n+\tif (o->type == ODB_HELPER_GIT_REMOTE) {\n+\t\tfetch_object(o->dealer, sha1);\n+\t} else {\n+\t\tstruct odb_helper_object *obj;\n+\t\tstruct odb_helper_cmd cmd;\n+\n+\t\tobj = odb_helper_lookup(o, sha1);\n+\t\tif (!obj)\n+\t\t\treturn -1;\n+\n+\t\tif (odb_helper_start(o, &cmd, \"get_direct %s\", sha1_to_hex(sha1)) < 0)\n+\t\t\treturn -1;\n+\n+\t\tif (odb_helper_finish(o, &cmd))\n+\t\t\treturn -1;\n+\t}\n \n \ttrace_performance_since(start, \"odb_helper_get_direct\");\n \n-\treturn res;\n+\treturn 0;\n }\n \n int odb_helper_get_many_direct(struct odb_helper *o,\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342176","messageId":"20180319133147.15413-33-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 32/36] Add t0570 to test passing raw objects","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:43Z","receivedAt":"2018-03-19T13:33:56Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n t/t0570-read-object-http-e-odb.sh | 109 ++++++++++++++++++++++++++++++\n t/t0570/read-object-plain | 83 +++++++++++++++++++++++\n 2 files changed, 192 insertions(+)\n create mode 100755 t/t0570-read-object-http-e-odb.sh\n create mode 100755 t/t0570/read-object-plain\n\ndiff --git a/t/t0570-read-object-http-e-odb.sh b/t/t0570-read-object-http-e-odb.sh\nnew file mode 100755\nindex 0000000000..1b2205e273\n--- /dev/null\n+++ b/t/t0570-read-object-http-e-odb.sh\n@@ -0,0 +1,109 @@\n+#!/bin/sh\n+\n+test_description='tests for read-object process passing plain objects to an HTTPD server'\n+\n+. ./test-lib.sh\n+\n+# If we don't specify a port, the current test number will be used\n+# which will not work as it is less than 1024, so it can only be used by root.\n+LIB_HTTPD_PORT=$(expr ${this_test#t} + 12000)\n+\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+\n+start_httpd apache-e-odb.conf\n+\n+PATH=\"$PATH:$TEST_DIRECTORY/t0570\"\n+\n+# odb helper script must see this\n+export HTTPD_URL\n+\n+HELPER=\"read-object-plain\"\n+\n+test_expect_success 'setup repo with a root commit' '\n+\ttest_commit zero\n+'\n+\n+test_expect_success 'setup another repo from the first one' '\n+\tgit init other-repo &&\n+\t(cd other-repo &&\n+\t git remote add origin .. &&\n+\t git pull origin master &&\n+\t git checkout master &&\n+\t git log)\n+'\n+\n+test_expect_success 'setup the helper in the root repo' '\n+\tgit config odb.magic.subprocessCommand \"$HELPER\"\n+'\n+\n+UPLOADFILENAME=\"hello_apache_upload.txt\"\n+\n+UPLOAD_URL=\"$HTTPD_URL/upload/?sha1=$UPLOADFILENAME&size=123&type=blob\"\n+\n+test_expect_success 'can upload a file' '\n+\techo \"Hello Apache World!\" >hello_to_send.txt &&\n+\techo \"How are you?\" >>hello_to_send.txt &&\n+\tcurl --data-binary @hello_to_send.txt --include \"$UPLOAD_URL\" >out_upload\n+'\n+\n+LIST_URL=\"$HTTPD_URL/list/\"\n+\n+test_expect_success 'can list uploaded files' '\n+\tcurl --include \"$LIST_URL\" >out_list &&\n+\tgrep \"$UPLOADFILENAME\" out_list\n+'\n+\n+test_expect_success 'can delete uploaded files' '\n+\tcurl --data \"delete\" --include \"$UPLOAD_URL&delete=1\" >out_delete &&\n+\tcurl --include \"$LIST_URL\" >out_list2 &&\n+\t! grep \"$UPLOADFILENAME\" out_list2\n+'\n+\n+FILES_DIR=\"httpd/www/files\"\n+\n+test_expect_success 'new blobs are transfered to the http server' '\n+\ttest_commit one &&\n+\thash1=$(git ls-tree HEAD | grep one.t | cut -f1 | cut -d\\ -f3) &&\n+\techo \"$hash1-4-blob\" >expected &&\n+\tls \"$FILES_DIR\" >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'blobs can be retrieved from the http server' '\n+\tgit cat-file blob \"$hash1\" &&\n+\tgit log -p >expected\n+'\n+\n+test_expect_success 'update other repo from the first one' '\n+\t(cd other-repo &&\n+\t git fetch origin \"refs/odbs/magic/*:refs/odbs/magic/*\" &&\n+\t test_must_fail git cat-file blob \"$hash1\" &&\n+\t git config odb.magic.subprocesscommand \"$HELPER\" &&\n+\t git cat-file blob \"$hash1\" &&\n+\t git pull origin master)\n+'\n+\n+test_expect_success 'local clone from the first repo' '\n+\tmkdir my-clone &&\n+\t(cd my-clone &&\n+\t git clone .. . &&\n+\t git cat-file blob \"$hash1\")\n+'\n+\n+test_expect_success 'no-local clone from the first repo fails' '\n+\tmkdir my-other-clone &&\n+\t(cd my-other-clone &&\n+\t test_must_fail git clone --no-local .. .) &&\n+\trm -rf my-other-clone\n+'\n+\n+test_expect_success 'no-local clone from the first repo with helper succeeds' '\n+\tmkdir my-other-clone &&\n+\t(cd my-other-clone &&\n+\t git clone -c odb.magic.subprocessCommand=\"$HELPER\" --no-local .. .) &&\n+\trm -rf my-other-clone\n+'\n+\n+stop_httpd\n+\n+test_done\ndiff --git a/t/t0570/read-object-plain b/t/t0570/read-object-plain\nnew file mode 100755\nindex 0000000000..0766e16032\n--- /dev/null\n+++ b/t/t0570/read-object-plain\n@@ -0,0 +1,83 @@\n+#!/usr/bin/perl\n+#\n+\n+use 5.008;\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n+use strict;\n+use warnings;\n+use Git::Packet;\n+use LWP::UserAgent;\n+use HTTP::Request::Common;\n+\n+packet_initialize(\"git-read-object\", 1);\n+\n+my %remote_caps = packet_read_and_check_capabilities(\"get_raw_obj\", \"put_raw_obj\");\n+packet_check_and_write_capabilities(\\%remote_caps, \"get_raw_obj\", \"put_raw_obj\");\n+\n+my $http_url = $ENV{HTTPD_URL};\n+\n+while (1) {\n+\tmy ($res, $command) = packet_txt_read();\n+\n+\tif ( $res == -1 ) {\n+\t\texit 0;\n+\t}\n+\n+\t$command =~ s/^command=//;\n+\n+\tif ( $command eq \"init\" ) {\n+\t\tpacket_bin_read();\n+\n+\t\tpacket_txt_write(\"status=success\");\n+\t\tpacket_flush();\n+\t} elsif ( $command eq \"get_raw_obj\" ) {\n+\t\tmy ($sha1) = packet_txt_read() =~ /^sha1=([0-9a-f]{40})$/;\n+\t\tpacket_bin_read();\n+\n+\t\tmy $get_url = $http_url . \"/list/?sha1=\" . $sha1;\n+\n+\t\tmy $userAgent = LWP::UserAgent->new();\n+\n+\t\tmy $response = $userAgent->get( $get_url );\n+\n+\t\tif ($response->is_error) {\n+\t\t packet_txt_write(\"size=0\");\n+\t\t packet_txt_write(\"kind=none\");\t \n+\t\t packet_txt_write(\"status=notfound\");\n+\t\t} else {\n+\t\t packet_txt_write(\"size=\" . length($response->content));\n+\t\t packet_txt_write(\"kind=blob\");\n+\t\t packet_bin_write($response->content);\n+\t\t packet_flush();\n+\t\t packet_txt_write(\"status=success\");\n+\t\t}\n+\n+\t\tpacket_flush();\n+\t} elsif ( $command eq \"put_raw_obj\" ) {\n+\t\tmy ($sha1) = packet_txt_read() =~ /^sha1=([0-9a-f]{40})$/;\n+\t\tmy ($size) = packet_txt_read() =~ /^size=([0-9]+)$/;\n+\t\tmy ($kind) = packet_txt_read() =~ /^kind=(\\w+)$/;\n+\t\tpacket_bin_read();\n+\n+\t\t# We must read the content we are sent and send it to the right url\n+\t\tmy ($res, $buf) = packet_bin_read();\n+\t\tdie \"bad packet_bin_read res ($res)\" unless ($res eq 0);\n+\t\t( packet_bin_read() eq ( 1, \"\" ) ) || die \"bad send end\";\t\t\n+\n+\t\tmy $upload_url = $http_url . \"/upload/?sha1=\" . $sha1 . \"&size=\" . $size . \"&type=blob\";\n+\n+\t\tmy $userAgent = LWP::UserAgent->new();\n+\t\tmy $request = POST $upload_url, Content_Type => 'multipart/form-data', Content => $buf;\n+\n+\t\tmy $response = $userAgent->request($request);\n+\n+\t\tif ($response->is_error) {\n+\t\t\tpacket_txt_write(\"status=failure\");\n+\t\t} else {\n+\t\t\tpacket_txt_write(\"status=success\");\n+\t\t}\n+\t\tpacket_flush();\n+\t} else {\n+\t\tdie \"bad command '$command'\";\n+\t}\n+}\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342177","messageId":"20180319133147.15413-32-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 31/36] odb-helper: add put_object_process()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:42Z","receivedAt":"2018-03-19T13:34:01Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This adds the infrastructure to send objects to a sub-process\nhandling the communication with an external odb.\n\nFor now we only handle sending raw blobs using the 'put_raw_obj'\ninstruction.\n\nSigned-off-by: Christian Couder \n---\n odb-helper.c | 78 ++++++++++++++++++++++++++++++++++++++++++++++++----\n 1 file changed, 72 insertions(+), 6 deletions(-)\n\ndiff --git a/odb-helper.c b/odb-helper.c\nindex cad3d1005c..83f8cce3b1 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -442,6 +442,58 @@ static int get_object_process(struct odb_helper *o, const unsigned char *sha1, i\n \t\t\t\t\t o->dealer, cur_cap);\n }\n \n+static int send_put_packets(struct object_process *entry,\n+\t\t\t const unsigned char *sha1,\n+\t\t\t const void *buf,\n+\t\t\t size_t len,\n+\t\t\t struct strbuf *status)\n+{\n+\tstruct child_process *process = &entry->subprocess.process;\n+\tint err = packet_write_fmt_gently(process->in, \"command=put_raw_obj\\n\");\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = packet_write_fmt_gently(process->in, \"sha1=%s\\n\", sha1_to_hex(sha1));\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = packet_write_fmt_gently(process->in, \"size=%\"PRIuMAX\"\\n\", len);\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = packet_write_fmt_gently(process->in, \"kind=blob\\n\");\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = packet_flush_gently(process->in);\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = write_packetized_from_buf(buf, len, process->in);\n+\tif (err)\n+\t\treturn err;\n+\n+\treturn check_object_process_status(process->out, status);\n+}\n+\n+static int put_object_process(struct odb_helper *o,\n+\t\t\t const void *buf, size_t len,\n+\t\t\t const char *type, unsigned char *sha1)\n+{\n+\tint err;\n+\tstruct object_process *entry;\n+\tstruct strbuf status = STRBUF_INIT;\n+\n+\tentry = launch_object_process(o, ODB_HELPER_CAP_PUT_RAW_OBJ);\n+\tif (!entry)\n+\t\treturn -1;\n+\n+\terr = send_put_packets(entry, sha1, buf, len, &status);\n+\n+\treturn check_object_process_error(err, status.buf, entry, o->dealer,\n+\t\t\t\t\t ODB_HELPER_CAP_PUT_RAW_OBJ);\n+}\n+\n struct odb_helper *odb_helper_new(const char *name, int namelen)\n {\n \tstruct odb_helper *o;\n@@ -938,15 +990,12 @@ int odb_helper_get_object(struct odb_helper *o,\n \treturn res;\n }\n \n-int odb_helper_put_object(struct odb_helper *o,\n-\t\t\t const void *buf, size_t len,\n-\t\t\t const char *type, unsigned char *sha1)\n+static int put_raw_object_script(struct odb_helper *o,\n+\t\t\t\t const void *buf, size_t len,\n+\t\t\t\t const char *type, unsigned char *sha1)\n {\n \tstruct odb_helper_cmd cmd;\n \n-\tif (o->type != ODB_HELPER_SCRIPT_CMD)\n-\t\treturn 1;\n-\n \tif (odb_helper_start(o, &cmd, 1, \"put_raw_obj %s %\"PRIuMAX\" %s\",\n \t\t\t sha1_to_hex(sha1), (uintmax_t)len, type) < 0)\n \t\treturn -1;\n@@ -969,3 +1018,20 @@ int odb_helper_put_object(struct odb_helper *o,\n \todb_helper_finish(o, &cmd);\n \treturn 0;\n }\n+\n+int odb_helper_put_object(struct odb_helper *o,\n+\t\t\t const void *buf, size_t len,\n+\t\t\t const char *type, unsigned char *sha1)\n+{\n+\tint res = 1;\n+\tuint64_t start = getnanotime();\n+\n+\tif (o->type == ODB_HELPER_SCRIPT_CMD)\n+\t\tres = put_raw_object_script(o, buf, len, type, sha1);\n+\telse if (o->type == ODB_HELPER_SUBPROCESS_CMD)\n+\t\tres = put_object_process(o, buf, len, type, sha1);\n+\n+\ttrace_performance_since(start, \"odb_helper_put_object\");\n+\n+\treturn res;\n+}\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342178","messageId":"20180319133147.15413-30-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 29/36] Add t0550 to test 'get_direct' mechanism","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:40Z","receivedAt":"2018-03-19T13:34:06Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: Ben Peart \n\nSigned-off-by: Ben Peart \nSigned-off-by: Christian Couder \n---\n t/t0550-read-object.sh | 28 +++++++++++++++++\n t/t0550/read-object | 68 ++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 96 insertions(+)\n create mode 100755 t/t0550-read-object.sh\n create mode 100755 t/t0550/read-object\n\ndiff --git a/t/t0550-read-object.sh b/t/t0550-read-object.sh\nnew file mode 100755\nindex 0000000000..93c8eea3f9\n--- /dev/null\n+++ b/t/t0550-read-object.sh\n@@ -0,0 +1,28 @@\n+#!/bin/sh\n+\n+test_description='tests for long running read-object process'\n+\n+. ./test-lib.sh\n+\n+PATH=\"$PATH:$TEST_DIRECTORY/t0550\"\n+\n+test_expect_success 'setup host repo with a root commit' '\n+\ttest_commit zero &&\n+\thash1=$(git ls-tree HEAD | grep zero.t | cut -f1 | cut -d\\ -f3)\n+'\n+\n+HELPER=\"read-object\"\n+\n+test_expect_success 'blobs can be retrieved from the host repo' '\n+\tgit init guest-repo &&\n+\t(cd guest-repo &&\n+\t git config odb.magic.subprocessCommand \"$HELPER\" &&\n+\t git cat-file blob \"$hash1\" >/dev/null)\n+'\n+\n+test_expect_success 'invalid blobs generate errors' '\n+\tcd guest-repo &&\n+\ttest_must_fail git cat-file blob \"invalid\"\n+'\n+\n+test_done\ndiff --git a/t/t0550/read-object b/t/t0550/read-object\nnew file mode 100755\nindex 0000000000..004e9368c9\n--- /dev/null\n+++ b/t/t0550/read-object\n@@ -0,0 +1,68 @@\n+#!/usr/bin/perl\n+#\n+# Example implementation for the Git read-object protocol version 1\n+# See Documentation/technical/read-object-protocol.txt\n+#\n+# Allows you to test the ability for blobs to be pulled from a host git repo\n+# \"on demand.\" Called when git needs a blob it couldn't find locally due to\n+# a lazy clone that only cloned the commits and trees.\n+#\n+# A lazy clone can be simulated via the following commands from the host repo\n+# you wish to create a lazy clone of:\n+#\n+# cd /host_repo\n+# git rev-parse HEAD\n+# git init /guest_repo\n+# git cat-file --batch-check --batch-all-objects | grep -v 'blob' |\n+#\tcut -d' ' -f1 | git pack-objects /e/guest_repo/.git/objects/pack/noblobs\n+# cd /guest_repo\n+# git config core.virtualizeobjects true\n+# git reset --hard \n+#\n+# Please note, this sample is a minimal skeleton. No proper error handling \n+# was implemented.\n+#\n+\n+use 5.008;\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n+use strict;\n+use warnings;\n+use Git::Packet;\n+\n+#\n+# Point $DIR to the folder where your host git repo is located so we can pull\n+# missing objects from it\n+#\n+my $DIR = \"../.git/\";\n+\n+packet_initialize(\"git-read-object\", 1);\n+\n+my %remote_caps = packet_read_and_check_capabilities(\"get_direct\");\n+packet_check_and_write_capabilities(\\%remote_caps, \"get_direct\");\n+\n+while (1) {\n+\tmy ($res, $command) = packet_txt_read();\n+\n+\tif ( $res == -1 ) {\n+\t\texit 0;\n+\t}\n+\n+\t$command =~ s/^command=//;\n+\n+\tif ( $command eq \"init\" ) {\n+\t\tpacket_bin_read();\n+\n+\t\tpacket_txt_write(\"status=success\");\n+\t\tpacket_flush();\n+\t} elsif ( $command eq \"get_direct\" ) {\n+\t\tmy ($sha1) = packet_txt_read() =~ /^sha1=([0-9a-f]{40})$/;\n+\t\tpacket_bin_read();\n+\n+\t\tsystem ('git --git-dir=\"' . $DIR . '\" cat-file blob ' . $sha1 . ' | GIT_NO_EXTERNAL_ODB=1 git hash-object -w --stdin >/dev/null 2>&1');\n+\n+\t\tpacket_txt_write(($?) ? \"status=error\" : \"status=success\");\n+\t\tpacket_flush();\n+\t} else {\n+\t\tdie \"bad command '$command'\";\n+\t}\n+}\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342179","messageId":"20180319133147.15413-29-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 28/36] odb-helper: add init_object_process()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:39Z","receivedAt":"2018-03-19T13:34:08Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This adds the infrastructure to launch and use long running\nsub-processes as external odb helpers.\n\nFor now only the 'init' and 'get_direct' capabilities are\nsupported with sub-processes.\n\nSigned-off-by: Christian Couder \n---\n external-odb.c | 46 +++--\n odb-helper.c | 498 ++++++++++++++++++++++++++++++++++++++++++++++---\n sha1_file.c | 13 +-\n 3 files changed, 510 insertions(+), 47 deletions(-)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex de2ae70db1..5e5871c933 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -106,29 +106,11 @@ const char *external_odb_root(void)\n \treturn root;\n }\n \n-int external_odb_has_object(const unsigned char *sha1)\n-{\n-\tstruct odb_helper *o;\n-\n-\texternal_odb_init();\n-\n-\tfor (o = helpers; o; o = o->next) {\n-\t\tif (!(o->supported_capabilities & ODB_HELPER_CAP_HAVE))\n-\t\t\treturn 1;\n-\t\tif (odb_helper_has_object(o, sha1))\n-\t\t\treturn 1;\n-\t}\n-\treturn 0;\n-}\n-\n-int external_odb_get_object(const unsigned char *sha1)\n+static int external_odb_do_get_object(const unsigned char *sha1)\n {\n \tstruct odb_helper *o;\n \tstruct strbuf pathbuf = STRBUF_INIT;\n \n-\tif (!external_odb_has_object(sha1))\n-\t\treturn -1;\n-\n \tsha1_file_name_alt(&pathbuf, external_odb_root(), sha1);\n \tsafe_create_leading_directories_const(pathbuf.buf);\n \tprepare_external_alt_odb();\n@@ -203,6 +185,32 @@ int external_odb_get_many_direct(const struct oid_array *to_get)\n \treturn -1;\n }\n \n+int external_odb_has_object(const unsigned char *sha1)\n+{\n+\tstruct odb_helper *o;\n+\n+\texternal_odb_init();\n+\n+\tfor (o = helpers; o; o = o->next) {\n+\t\tif (!(o->supported_capabilities & ODB_HELPER_CAP_HAVE)) {\n+\t\t\tif (o->supported_capabilities & ODB_HELPER_CAP_GET_DIRECT)\n+\t\t\t\treturn 1;\n+\t\t\treturn !external_odb_do_get_object(sha1);\n+\t\t}\n+\t\tif (odb_helper_has_object(o, sha1))\n+\t\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+int external_odb_get_object(const unsigned char *sha1)\n+{\n+\tif (!external_odb_has_object(sha1))\n+\t\treturn -1;\n+\n+\treturn external_odb_do_get_object(sha1);\n+}\n+\n int external_odb_put_object(const void *buf, size_t len,\n \t\t\t const char *type, unsigned char *sha1)\n {\ndiff --git a/odb-helper.c b/odb-helper.c\nindex fdd062ec8d..cad3d1005c 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -5,6 +5,22 @@\n #include \"run-command.h\"\n #include \"sha1-lookup.h\"\n #include \"fetch-object.h\"\n+#include \"sub-process.h\"\n+#include \"pkt-line.h\"\n+#include \"sigchain.h\"\n+\n+struct object_process {\n+\tstruct subprocess_entry subprocess;\n+\tunsigned int supported_capabilities;\n+};\n+\n+static struct hashmap subprocess_map;\n+\n+static int check_object_process_status(int fd, struct strbuf *status)\n+{\n+\tsubprocess_read_status(fd, status);\n+\treturn strcmp(status->buf, \"success\");\n+}\n \n static void parse_capabilities(char *cap_buf,\n \t\t\t unsigned int *supported_capabilities,\n@@ -40,6 +56,392 @@ static void parse_capabilities(char *cap_buf,\n \tstring_list_clear(&cap_list, 0);\n }\n \n+static int start_object_process_fn(struct subprocess_entry *subprocess)\n+{\n+\tstatic int versions[] = {1, 0};\n+\tstatic struct subprocess_capability capabilities[] = {\n+\t\t{ \"get_git_obj\", ODB_HELPER_CAP_GET_GIT_OBJ },\n+\t\t{ \"get_raw_obj\", ODB_HELPER_CAP_GET_RAW_OBJ },\n+\t\t{ \"get_direct\", ODB_HELPER_CAP_GET_DIRECT },\n+\t\t{ \"put_git_obj\", ODB_HELPER_CAP_PUT_GIT_OBJ },\n+\t\t{ \"put_raw_obj\", ODB_HELPER_CAP_PUT_RAW_OBJ },\n+\t\t{ \"put_direct\", ODB_HELPER_CAP_PUT_DIRECT },\n+\t\t{ \"have\", ODB_HELPER_CAP_HAVE },\n+\t\t{ NULL, 0 }\n+\t};\n+\tstruct object_process *entry = (struct object_process *)subprocess;\n+\treturn subprocess_handshake(subprocess, \"git-read-object\", versions, NULL,\n+\t\t\t\t capabilities,\n+\t\t\t\t &entry->supported_capabilities);\n+}\n+\n+static struct object_process *launch_object_process(struct odb_helper *o,\n+\t\t\t\t\t\t unsigned int capability)\n+{\n+\tstruct object_process *entry = NULL;\n+\n+\tif (!subprocess_map.tablesize)\n+\t\thashmap_init(&subprocess_map, (hashmap_cmp_fn) cmd2process_cmp, NULL, 0);\n+\telse\n+\t\tentry = (struct object_process *)subprocess_find_entry(&subprocess_map, o->dealer);\n+\n+\tfflush(NULL);\n+\n+\tif (!entry) {\n+\t\tentry = xmalloc(sizeof(*entry));\n+\t\tentry->supported_capabilities = 0;\n+\n+\t\tif (subprocess_start(&subprocess_map, &entry->subprocess, o->dealer, start_object_process_fn)) {\n+\t\t\terror(\"Could not launch process for cmd '%s'\", o->dealer);\n+\t\t\tfree(entry);\n+\t\t\treturn NULL;\n+\t\t}\n+\t}\n+\n+\to->supported_capabilities = entry->supported_capabilities;\n+\n+\tif (capability && !(capability & entry->supported_capabilities)) {\n+\t\terror(\"The cmd '%s' does not support capability '%d'\", o->dealer, capability);\n+\t\treturn NULL;\n+\t}\n+\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\n+\treturn entry;\n+}\n+\n+static int check_object_process_error(int err,\n+\t\t\t\t const char *status,\n+\t\t\t\t struct object_process *entry,\n+\t\t\t\t const char *cmd,\n+\t\t\t\t unsigned int capability)\n+{\n+\tsigchain_pop(SIGPIPE);\n+\n+\tif (!err)\n+\t\treturn 0;\n+\n+\tif (!strcmp(status, \"error\")) {\n+\t\t/* The process signaled a problem with the file. */\n+\t} else if (!strcmp(status, \"notfound\")) {\n+\t\t/* Object was not found */\n+\t\terr = -1;\n+\t} else if (!strcmp(status, \"abort\")) {\n+\t\t/*\n+\t\t * The process signaled a permanent problem. Don't try to read\n+\t\t * objects with the same command for the lifetime of the current\n+\t\t * Git process.\n+\t\t */\n+\t\tif (capability)\n+\t\t\tentry->supported_capabilities &= ~capability;\n+\t} else {\n+\t\t/*\n+\t\t * Something went wrong with the read-object process.\n+\t\t * Force shutdown and restart if needed.\n+\t\t */\n+\t\terror(\"external object process '%s' failed\", cmd);\n+\t\tsubprocess_stop(&subprocess_map, &entry->subprocess);\n+\t\tfree(entry);\n+\t}\n+\n+\treturn err;\n+}\n+\n+static int send_init_packets(struct object_process *entry,\n+\t\t\t struct strbuf *status)\n+{\n+\tstruct child_process *process = &entry->subprocess.process;\n+\n+\treturn packet_write_fmt_gently(process->in, \"command=init\\n\") ||\n+\t\tpacket_flush_gently(process->in) ||\n+\t\tcheck_object_process_status(process->out, status);\n+}\n+\n+static int init_object_process(struct odb_helper *o)\n+{\n+\tint err;\n+\tstruct strbuf status = STRBUF_INIT;\n+\tstruct object_process *entry = launch_object_process(o, 0);\n+\tif (!entry)\n+\t\treturn -1;\n+\n+\terr = send_init_packets(entry, &status);\n+\n+\treturn check_object_process_error(err, status.buf, entry,\n+\t\t\t\t\t o->dealer, 0);\n+}\n+\n+static ssize_t read_packetized_raw_object_to_fd(struct odb_helper *o,\n+\t\t\t\t\t\tconst unsigned char *sha1,\n+\t\t\t\t\t\tint fd_in, int fd_out)\n+{\n+\tssize_t total_read = 0;\n+\tunsigned long total_got = 0;\n+\tint packet_len;\n+\n+\tchar hdr[32];\n+\tint hdrlen;\n+\n+\tint ret = Z_STREAM_END;\n+\tunsigned char compressed[4096];\n+\tgit_zstream stream;\n+\tgit_SHA_CTX hash;\n+\tunsigned char real_sha1[20];\n+\n+\toff_t size;\n+\tenum object_type type;\n+\tconst char *s;\n+\tint pkt_size;\n+\tchar *size_buf;\n+\n+\tsize_buf = packet_read_line(fd_in, &pkt_size);\n+\tif (!skip_prefix(size_buf, \"size=\", &s))\n+\t\treturn error(\"odb helper '%s' did not send size of plain object\", o->name);\n+\tsize = strtoumax(s, NULL, 10);\n+\tif (!skip_prefix(packet_read_line(fd_in, NULL), \"kind=\", &s))\n+\t\treturn error(\"odb helper '%s' did not send kind of plain object\", o->name);\n+\t/* Check if the object is not available */\n+\tif (!strcmp(s, \"none\"))\n+\t\treturn -1;\n+\ttype = type_from_string_gently(s, strlen(s), 1);\n+\tif (type < 0)\n+\t\treturn error(\"odb helper '%s' sent bad type '%s'\", o->name, s);\n+\n+\t/* Set it up */\n+\tgit_deflate_init(&stream, zlib_compression_level);\n+\tstream.next_out = compressed;\n+\tstream.avail_out = sizeof(compressed);\n+\tgit_SHA1_Init(&hash);\n+\n+\t/* First header.. */\n+\thdrlen = xsnprintf(hdr, sizeof(hdr), \"%s %lu\", type_name(type), size) + 1;\n+\tstream.next_in = (unsigned char *)hdr;\n+\tstream.avail_in = hdrlen;\n+\twhile (git_deflate(&stream, 0) == Z_OK)\n+\t\t; /* nothing */\n+\tgit_SHA1_Update(&hash, hdr, hdrlen);\n+\n+\tfor (;;) {\n+\t\t/* packet_read() writes a '\\0' extra byte at the end */\n+\t\tchar buf[LARGE_PACKET_DATA_MAX + 1];\n+\n+\t\tpacket_len = packet_read(fd_in, NULL, NULL,\n+\t\t\tbuf, LARGE_PACKET_DATA_MAX + 1,\n+\t\t\tPACKET_READ_GENTLE_ON_EOF);\n+\n+\t\tif (packet_len <= 0)\n+\t\t\tbreak;\n+\n+\t\ttotal_got += packet_len;\n+\n+\t\t/* Then the data itself.. */\n+\t\tstream.next_in = (void *)buf;\n+\t\tstream.avail_in = packet_len;\n+\t\tdo {\n+\t\t\tunsigned char *in0 = stream.next_in;\n+\t\t\tret = git_deflate(&stream, Z_FINISH);\n+\t\t\tgit_SHA1_Update(&hash, in0, stream.next_in - in0);\n+\t\t\twrite_or_die(fd_out, compressed, stream.next_out - compressed);\n+\t\t\tstream.next_out = compressed;\n+\t\t\tstream.avail_out = sizeof(compressed);\n+\t\t} while (ret == Z_OK);\n+\n+\t\ttotal_read += packet_len;\n+\t}\n+\n+\tif (packet_len < 0) {\n+\t\terror(\"unable to read from odb helper '%s': %s\",\n+\t\t o->name, strerror(errno));\n+\t\tgit_deflate_end(&stream);\n+\t\treturn packet_len;\n+\t}\n+\n+\tif (ret != Z_STREAM_END) {\n+\t\twarning(\"bad zlib data from odb helper '%s' for %s\",\n+\t\t\to->name, sha1_to_hex(sha1));\n+\t\treturn -1;\n+\t}\n+\n+\tret = git_deflate_end_gently(&stream);\n+\tif (ret != Z_OK) {\n+\t\twarning(\"deflateEnd on object %s from odb helper '%s' failed (%d)\",\n+\t\t\tsha1_to_hex(sha1), o->name, ret);\n+\t\treturn -1;\n+\t}\n+\tgit_SHA1_Final(real_sha1, &hash);\n+\tif (hashcmp(sha1, real_sha1)) {\n+\t\twarning(\"sha1 mismatch from odb helper '%s' for %s (got %s)\",\n+\t\t\to->name, sha1_to_hex(sha1), sha1_to_hex(real_sha1));\n+\t\treturn -1;\n+\t}\n+\tif (total_got != size) {\n+\t\twarning(\"size mismatch from odb helper '%s' for %s (%lu != %lu)\",\n+\t\t\to->name, sha1_to_hex(sha1), total_got, size);\n+\t\treturn -1;\n+\t}\n+\n+\treturn total_read;\n+}\n+\n+static ssize_t read_packetized_git_object_to_fd(struct odb_helper *o,\n+\t\t\t\t\t\tconst unsigned char *sha1,\n+\t\t\t\t\t\tint fd_in, int fd_out)\n+{\n+\tssize_t total_read = 0;\n+\tunsigned long total_got = 0;\n+\tint packet_len;\n+\tgit_zstream stream;\n+\tint zret = Z_STREAM_END;\n+\tgit_SHA_CTX hash;\n+\tunsigned char real_sha1[20];\n+\tstruct strbuf header = STRBUF_INIT;\n+\tunsigned long hdr_size;\n+\n+\tmemset(&stream, 0, sizeof(stream));\n+\tgit_inflate_init(&stream);\n+\tgit_SHA1_Init(&hash);\n+\n+\tfor (;;) {\n+\t\t/* packet_read() writes a '\\0' extra byte at the end */\n+\t\tchar buf[LARGE_PACKET_DATA_MAX + 1];\n+\n+\t\tpacket_len = packet_read(fd_in, NULL, NULL,\n+\t\t\tbuf, LARGE_PACKET_DATA_MAX + 1,\n+\t\t\tPACKET_READ_GENTLE_ON_EOF);\n+\n+\t\tif (packet_len <= 0)\n+\t\t\tbreak;\n+\n+\t\twrite_or_die(fd_out, buf, packet_len);\n+\n+\t\tstream.next_in = (unsigned char *)buf;\n+\t\tstream.avail_in = packet_len;\n+\t\tdo {\n+\t\t\tunsigned char inflated[4096];\n+\t\t\tunsigned long got;\n+\n+\t\t\tstream.next_out = inflated;\n+\t\t\tstream.avail_out = sizeof(inflated);\n+\t\t\tzret = git_inflate(&stream, Z_SYNC_FLUSH);\n+\t\t\tgot = sizeof(inflated) - stream.avail_out;\n+\n+\t\t\tgit_SHA1_Update(&hash, inflated, got);\n+\t\t\t/* skip header when counting size */\n+\t\t\tif (!total_got) {\n+\t\t\t\tconst unsigned char *p = memchr(inflated, '\\0', got);\n+\t\t\t\tif (p) {\n+\t\t\t\t\tunsigned long hdr_last = p - inflated + 1;\n+\t\t\t\t\tstrbuf_add(&header, inflated, hdr_last);\n+\t\t\t\t\tgot -= hdr_last;\n+\t\t\t\t} else {\n+\t\t\t\t\tstrbuf_add(&header, inflated, got);\n+\t\t\t\t\tgot = 0;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t\ttotal_got += got;\n+\t\t} while (stream.avail_in && zret == Z_OK);\n+\n+\t\ttotal_read += packet_len;\n+\t}\n+\n+\tgit_inflate_end(&stream);\n+\n+\tif (packet_len < 0) {\n+\t\tstrbuf_release(&header);\n+\t\treturn packet_len;\n+\t}\n+\n+\tgit_SHA1_Final(real_sha1, &hash);\n+\n+\tif (zret != Z_STREAM_END) {\n+\t\twarning(\"bad zlib data from odb helper '%s' for %s\",\n+\t\t\to->name, sha1_to_hex(sha1));\n+\t\tstrbuf_release(&header);\n+\t\treturn -1;\n+\t}\n+\tif (hashcmp(real_sha1, sha1)) {\n+\t\twarning(\"sha1 mismatch from odb helper '%s' for %s (got %s)\",\n+\t\t\to->name, sha1_to_hex(sha1), sha1_to_hex(real_sha1));\n+\t\tstrbuf_release(&header);\n+\t\treturn -1;\n+\t}\n+\tif (parse_sha1_header(header.buf, &hdr_size) < 0) {\n+\t\twarning(\"could not parse header from odb helper '%s' for %s\",\n+\t\t\to->name, sha1_to_hex(sha1));\n+\t\tstrbuf_release(&header);\n+\t\treturn -1;\n+\t}\n+\tif (total_got != hdr_size) {\n+\t\twarning(\"size mismatch from odb helper '%s' for %s (%lu != %lu)\",\n+\t\t\to->name, sha1_to_hex(sha1), total_got, hdr_size);\n+\t\tstrbuf_release(&header);\n+\t\treturn -1;\n+\t}\n+\n+\tstrbuf_release(&header);\n+\n+\treturn total_read;\n+}\n+\n+static int send_get_packets(struct odb_helper *o,\n+\t\t\t struct object_process *entry,\n+\t\t\t const unsigned char *sha1,\n+\t\t\t int fd,\n+\t\t\t unsigned int *cur_cap,\n+\t\t\t struct strbuf *status)\n+{\n+\tconst char *instruction;\n+\tint err;\n+\tstruct child_process *process = &entry->subprocess.process;\n+\n+\tif (entry->supported_capabilities & ODB_HELPER_CAP_GET_GIT_OBJ) {\n+\t\t*cur_cap = ODB_HELPER_CAP_GET_GIT_OBJ;\n+\t\tinstruction = \"get_git_obj\";\n+\t} else if (entry->supported_capabilities & ODB_HELPER_CAP_GET_RAW_OBJ) {\n+\t\t*cur_cap = ODB_HELPER_CAP_GET_RAW_OBJ;\n+\t\tinstruction = \"get_raw_obj\";\n+\t} else if (entry->supported_capabilities & ODB_HELPER_CAP_GET_DIRECT) {\n+\t\t*cur_cap = ODB_HELPER_CAP_GET_DIRECT;\n+\t\tinstruction = \"get_direct\";\n+\t} else {\n+\t\tBUG(\"No known ODB_HELPER_CAP_GET_XXX capability!\");\n+\t}\n+\n+\terr = packet_write_fmt_gently(process->in, \"command=%s\\n\", instruction);\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = packet_write_fmt_gently(process->in, \"sha1=%s\\n\", sha1_to_hex(sha1));\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = packet_flush_gently(process->in);\n+\tif (err)\n+\t\treturn err;\n+\n+\tif (entry->supported_capabilities & ODB_HELPER_CAP_GET_RAW_OBJ)\n+\t\terr = read_packetized_raw_object_to_fd(o, sha1, process->out, fd) < 0;\n+\telse if (entry->supported_capabilities & ODB_HELPER_CAP_GET_GIT_OBJ)\n+\t\terr = read_packetized_git_object_to_fd(o, sha1, process->out, fd) < 0;\n+\n+\treturn check_object_process_status(process->out, status);\n+}\n+\n+static int get_object_process(struct odb_helper *o, const unsigned char *sha1, int fd)\n+{\n+\tint err;\n+\tstruct strbuf status = STRBUF_INIT;\n+\tunsigned int cur_cap = 0;\n+\tstruct object_process *entry = launch_object_process(o, 0);\n+\tif (!entry)\n+\t\treturn -1;\n+\n+\terr = send_get_packets(o, entry, sha1, fd, &cur_cap, &status);\n+\n+\treturn check_object_process_error(err, status.buf, entry,\n+\t\t\t\t\t o->dealer, cur_cap);\n+}\n+\n struct odb_helper *odb_helper_new(const char *name, int namelen)\n {\n \tstruct odb_helper *o;\n@@ -118,19 +520,12 @@ static int odb_helper_finish(struct odb_helper *o,\n \treturn 0;\n }\n \n-int odb_helper_init(struct odb_helper *o)\n+static int init_object_script(struct odb_helper *o)\n {\n \tstruct odb_helper_cmd cmd;\n \tFILE *fh;\n \tstruct strbuf line = STRBUF_INIT;\n \n-\tif (o->initialized)\n-\t\treturn 0;\n-\to->initialized = 1;\n-\n-\tif (o->type != ODB_HELPER_SCRIPT_CMD)\n-\t\treturn 0;\n-\n \tif (odb_helper_start(o, &cmd, 0, \"init\") < 0)\n \t\treturn -1;\n \n@@ -145,6 +540,27 @@ int odb_helper_init(struct odb_helper *o)\n \treturn 0;\n }\n \n+int odb_helper_init(struct odb_helper *o)\n+{\n+\tint res = 0;\n+\tuint64_t start;\n+\n+\tif (o->initialized)\n+\t\treturn 0;\n+\to->initialized = 1;\n+\n+\tstart = getnanotime();\n+\n+\tif (o->type == ODB_HELPER_SCRIPT_CMD)\n+\t\tres = init_object_script(o);\n+\telse if (o->type == ODB_HELPER_SUBPROCESS_CMD)\n+\t\tres = init_object_process(o);\n+\n+\ttrace_performance_since(start, \"odb_helper_init\");\n+\n+\treturn res;\n+}\n+\n static int parse_object_line(struct odb_helper_object *o, const char *line)\n {\n \tchar *end;\n@@ -439,31 +855,42 @@ static int odb_helper_get_git_object(struct odb_helper *o,\n \treturn 0;\n }\n \n+static int get_direct_script(struct odb_helper *o, const unsigned char *sha1)\n+{\n+\tstruct odb_helper_cmd cmd;\n+\n+\tif (odb_helper_start(o, &cmd, 0, \"get_direct %s\", sha1_to_hex(sha1)) < 0)\n+\t\treturn -1;\n+\tif (odb_helper_finish(o, &cmd))\n+\t\treturn -1;\n+\treturn 0;\n+}\n+\n int odb_helper_get_direct(struct odb_helper *o,\n \t\t\t const unsigned char *sha1)\n {\n-\tuint64_t start = getnanotime();\n-\n-\tif (o->type == ODB_HELPER_GIT_REMOTE) {\n-\t\tfetch_object(o->dealer, sha1);\n-\t} else {\n-\t\tstruct odb_helper_object *obj;\n-\t\tstruct odb_helper_cmd cmd;\n+\tint res = 0;\n+\tuint64_t start;\n \n-\t\tobj = odb_helper_lookup(o, sha1);\n+\tif (o->supported_capabilities & ODB_HELPER_CAP_HAVE &&\n+\t o->type != ODB_HELPER_GIT_REMOTE) {\n+\t\tstruct odb_helper_object *obj = odb_helper_lookup(o, sha1);\n \t\tif (!obj)\n \t\t\treturn -1;\n+\t}\n \n-\t\tif (odb_helper_start(o, &cmd, 0, \"get_direct %s\", sha1_to_hex(sha1)) < 0)\n-\t\t\treturn -1;\n+\tstart = getnanotime();\n \n-\t\tif (odb_helper_finish(o, &cmd))\n-\t\t\treturn -1;\n-\t}\n+\tif (o->type == ODB_HELPER_GIT_REMOTE)\n+\t\tfetch_object(o->dealer, sha1);\n+\telse if (o->type == ODB_HELPER_SCRIPT_CMD)\n+\t\tres = get_direct_script(o, sha1);\n+\telse\n+\t\tres = get_object_process(o, sha1, -1);\n \n \ttrace_performance_since(start, \"odb_helper_get_direct\");\n \n-\treturn 0;\n+\treturn res;\n }\n \n int odb_helper_get_many_direct(struct odb_helper *o,\n@@ -482,14 +909,33 @@ int odb_helper_get_many_direct(struct odb_helper *o,\n \treturn res;\n }\n \n+static int get_object_script(struct odb_helper *o, const unsigned char *sha1, int fd)\n+{\n+\tif (o->supported_capabilities & ODB_HELPER_CAP_GET_GIT_OBJ)\n+\t\treturn odb_helper_get_git_object(o, sha1, fd);\n+\tif (o->supported_capabilities & ODB_HELPER_CAP_GET_RAW_OBJ)\n+\t\treturn odb_helper_get_raw_object(o, sha1, fd);\n+\tif (o->supported_capabilities & ODB_HELPER_CAP_GET_DIRECT)\n+\t\treturn 0;\n+\n+\tBUG(\"invalid get capability (capabilities: '%d')\", o->supported_capabilities);\n+}\n+\n int odb_helper_get_object(struct odb_helper *o,\n \t\t\t const unsigned char *sha1,\n \t\t\t int fd)\n {\n-\tif (o->supported_capabilities & ODB_HELPER_CAP_GET_RAW_OBJ)\n-\t\treturn odb_helper_get_raw_object(o, sha1, fd);\n-\telse\n-\t\treturn odb_helper_get_git_object(o, sha1, fd);\n+\tint res = 0;\n+\tuint64_t start = getnanotime();\n+\n+\tif (o->type == ODB_HELPER_SCRIPT_CMD)\n+\t\tres = get_object_script(o, sha1, fd);\n+\telse if (o->type == ODB_HELPER_SUBPROCESS_CMD)\n+\t\tres = get_object_process(o, sha1, fd);\n+\n+\ttrace_performance_since(start, \"odb_helper_get_object\");\n+\n+\treturn res;\n }\n \n int odb_helper_put_object(struct odb_helper *o,\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 64679fae36..9ccdae35bc 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -730,11 +730,20 @@ int check_and_freshen_file(const char *fn, int freshen)\n static int check_and_freshen_local(const unsigned char *sha1, int freshen)\n {\n \tstatic struct strbuf buf = STRBUF_INIT;\n+\tint ret;\n+\tint tried_hook = 0;\n \n \tstrbuf_reset(&buf);\n \tsha1_file_name(&buf, sha1);\n \n-\treturn check_and_freshen_file(buf.buf, freshen);\n+retry:\n+\tret = check_and_freshen_file(buf.buf, freshen);\n+\tif (!ret && !tried_hook) {\n+\t\ttried_hook = 1;\n+\t\tif (!external_odb_get_direct(sha1))\n+\t\t\tgoto retry;\n+\t}\n+\treturn ret;\n }\n \n static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)\n@@ -746,7 +755,7 @@ static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)\n \t\tif (check_and_freshen_file(path, freshen))\n \t\t\treturn 1;\n \t}\n-\treturn 0;\n+\treturn external_odb_has_object(sha1);\n }\n \n static int check_and_freshen(const unsigned char *sha1, int freshen)\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342180","messageId":"20180319133147.15413-26-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 25/36] odb-helper: add odb_helper_get_raw_object()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:36Z","receivedAt":"2018-03-19T13:34:11Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"The existing odb_helper_get_object() is renamed\nodb_helper_get_git_object() and a new odb_helper_get_raw_object()\nis introduced to deal with external objects that are not in Git format.\n\nSigned-off-by: Christian Couder \n---\n odb-helper.c | 113 ++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 111 insertions(+), 2 deletions(-)\n\ndiff --git a/odb-helper.c b/odb-helper.c\nindex 2734af793b..fdd062ec8d 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -235,8 +235,107 @@ int odb_helper_has_object(struct odb_helper *o, const unsigned char *sha1)\n \treturn !!odb_helper_lookup(o, sha1);\n }\n \n-int odb_helper_get_object(struct odb_helper *o, const unsigned char *sha1,\n-\t\t\t int fd)\n+static int odb_helper_get_raw_object(struct odb_helper *o,\n+\t\t\t\t const unsigned char *sha1,\n+\t\t\t\t int fd)\n+{\n+\tstruct odb_helper_object *obj;\n+\tstruct odb_helper_cmd cmd;\n+\tunsigned long total_got = 0;\n+\n+\tchar hdr[32];\n+\tint hdrlen;\n+\n+\tint ret = Z_STREAM_END;\n+\tunsigned char compressed[4096];\n+\tgit_zstream stream;\n+\tgit_SHA_CTX hash;\n+\tunsigned char real_sha1[20];\n+\n+\tobj = odb_helper_lookup(o, sha1);\n+\tif (!obj)\n+\t\treturn -1;\n+\n+\tif (odb_helper_start(o, &cmd, 0, \"get_raw_obj %s\", sha1_to_hex(sha1)) < 0)\n+\t\treturn -1;\n+\n+\t/* Set it up */\n+\tgit_deflate_init(&stream, zlib_compression_level);\n+\tstream.next_out = compressed;\n+\tstream.avail_out = sizeof(compressed);\n+\tgit_SHA1_Init(&hash);\n+\n+\t/* First header.. */\n+\thdrlen = xsnprintf(hdr, sizeof(hdr), \"%s %lu\", type_name(obj->type), obj->size) + 1;\n+\tstream.next_in = (unsigned char *)hdr;\n+\tstream.avail_in = hdrlen;\n+\twhile (git_deflate(&stream, 0) == Z_OK)\n+\t\t; /* nothing */\n+\tgit_SHA1_Update(&hash, hdr, hdrlen);\n+\n+\tfor (;;) {\n+\t\tunsigned char buf[4096];\n+\t\tint r;\n+\n+\t\tr = xread(cmd.child.out, buf, sizeof(buf));\n+\t\tif (r < 0) {\n+\t\t\terror(\"unable to read from odb helper '%s': %s\",\n+\t\t\t o->name, strerror(errno));\n+\t\t\tclose(cmd.child.out);\n+\t\t\todb_helper_finish(o, &cmd);\n+\t\t\tgit_deflate_end(&stream);\n+\t\t\treturn -1;\n+\t\t}\n+\t\tif (r == 0)\n+\t\t\tbreak;\n+\n+\t\ttotal_got += r;\n+\n+\t\t/* Then the data itself.. */\n+\t\tstream.next_in = (void *)buf;\n+\t\tstream.avail_in = r;\n+\t\tdo {\n+\t\t\tunsigned char *in0 = stream.next_in;\n+\t\t\tret = git_deflate(&stream, Z_FINISH);\n+\t\t\tgit_SHA1_Update(&hash, in0, stream.next_in - in0);\n+\t\t\twrite_or_die(fd, compressed, stream.next_out - compressed);\n+\t\t\tstream.next_out = compressed;\n+\t\t\tstream.avail_out = sizeof(compressed);\n+\t\t} while (ret == Z_OK);\n+\t}\n+\n+\tclose(cmd.child.out);\n+\tif (ret != Z_STREAM_END) {\n+\t\twarning(\"bad zlib data from odb helper '%s' for %s\",\n+\t\t\to->name, sha1_to_hex(sha1));\n+\t\treturn -1;\n+\t}\n+\tret = git_deflate_end_gently(&stream);\n+\tif (ret != Z_OK) {\n+\t\twarning(\"deflateEnd on object %s from odb helper '%s' failed (%d)\",\n+\t\t\tsha1_to_hex(sha1), o->name, ret);\n+\t\treturn -1;\n+\t}\n+\tgit_SHA1_Final(real_sha1, &hash);\n+\tif (hashcmp(sha1, real_sha1)) {\n+\t\twarning(\"sha1 mismatch from odb helper '%s' for %s (got %s)\",\n+\t\t\to->name, sha1_to_hex(sha1), sha1_to_hex(real_sha1));\n+\t\treturn -1;\n+\t}\n+\tif (odb_helper_finish(o, &cmd))\n+\t\treturn -1;\n+\tif (total_got != obj->size) {\n+\t\twarning(\"size mismatch from odb helper '%s' for %s (%lu != %lu)\",\n+\t\t\to->name, sha1_to_hex(sha1), total_got, obj->size);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int odb_helper_get_git_object(struct odb_helper *o,\n+\t\t\t\t const unsigned char *sha1,\n+\t\t\t\t int fd)\n {\n \tstruct odb_helper_object *obj;\n \tstruct odb_helper_cmd cmd;\n@@ -383,6 +482,16 @@ int odb_helper_get_many_direct(struct odb_helper *o,\n \treturn res;\n }\n \n+int odb_helper_get_object(struct odb_helper *o,\n+\t\t\t const unsigned char *sha1,\n+\t\t\t int fd)\n+{\n+\tif (o->supported_capabilities & ODB_HELPER_CAP_GET_RAW_OBJ)\n+\t\treturn odb_helper_get_raw_object(o, sha1, fd);\n+\telse\n+\t\treturn odb_helper_get_git_object(o, sha1, fd);\n+}\n+\n int odb_helper_put_object(struct odb_helper *o,\n \t\t\t const void *buf, size_t len,\n \t\t\t const char *type, unsigned char *sha1)\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342181","messageId":"20180319133147.15413-21-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 20/36] Add t0510 to test external ODB transfer","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:31Z","receivedAt":"2018-03-19T13:34:14Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n t/t0510-transfer-e-odb.sh | 144 ++++++++++++++++++++++++++++++++++++++\n 1 file changed, 144 insertions(+)\n create mode 100755 t/t0510-transfer-e-odb.sh\n\ndiff --git a/t/t0510-transfer-e-odb.sh b/t/t0510-transfer-e-odb.sh\nnew file mode 100755\nindex 0000000000..065ec7d759\n--- /dev/null\n+++ b/t/t0510-transfer-e-odb.sh\n@@ -0,0 +1,144 @@\n+#!/bin/sh\n+\n+test_description='basic tests for transfering external ODBs'\n+\n+. ./test-lib.sh\n+\n+ORIG_SOURCE=\"$PWD/.git\"\n+export ORIG_SOURCE\n+\n+ALT_SOURCE1=\"$PWD/alt-repo1/.git\"\n+export ALT_SOURCE1\n+write_script odb-helper1 <<\\EOF\n+die() {\n+\tprintf >&2 \"%s\\n\" \"$@\"\n+\texit 1\n+}\n+GIT_DIR=$ALT_SOURCE1; export GIT_DIR\n+case \"$1\" in\n+init)\n+\techo \"capability=get_git_obj\"\n+\techo \"capability=have\"\n+\t;;\n+have)\n+\tgit cat-file --batch-check --batch-all-objects |\n+\tawk '{print $1 \" \" $3 \" \" $2}'\n+\t;;\n+get_git_obj)\n+\tcat \"$GIT_DIR\"/objects/$(echo $2 | sed 's#..#&/#')\n+\t;;\n+put_raw_obj)\n+\tsha1=\"$2\"\n+\tsize=\"$3\"\n+\tkind=\"$4\"\n+\twriten=$(git hash-object -w -t \"$kind\" --stdin)\n+\ttest \"$writen\" = \"$sha1\" || die \"bad sha1 passed '$sha1' vs writen '$writen'\"\n+\tref_hash=$(echo \"$sha1 $size $kind\" | GIT_DIR=$ORIG_SOURCE GIT_NO_EXTERNAL_ODB=1 git hash-object -w -t blob --stdin) || exit\n+\tGIT_DIR=$ORIG_SOURCE git update-ref refs/odbs/magic/\"$sha1\" \"$ref_hash\"\n+\t;;\n+*)\n+\tdie \"unknown command '$1'\"\n+\t;;\n+esac\n+EOF\n+HELPER1=\"\\\"$PWD\\\"/odb-helper1\"\n+\n+OTHER_SOURCE=\"$PWD/.git\"\n+export OTHER_SOURCE\n+\n+ALT_SOURCE2=\"$PWD/alt-repo2/.git\"\n+export ALT_SOURCE2\n+write_script odb-helper2 <<\\EOF\n+die() {\n+\tprintf >&2 \"%s\\n\" \"$@\"\n+\texit 1\n+}\n+GIT_DIR=$ALT_SOURCE2; export GIT_DIR\n+case \"$1\" in\n+init)\n+\techo \"capability=get_git_obj\"\n+\techo \"capability=have\"\n+\t;;\n+have)\n+\tGIT_DIR=$OTHER_SOURCE git for-each-ref --format='%(objectname)' refs/odbs/magic/ | GIT_DIR=$OTHER_SOURCE xargs git show\n+\t;;\n+get_git_obj)\n+\tOBJ_FILE=\"$GIT_DIR\"/objects/$(echo $2 | sed 's#..#&/#')\n+\tif ! test -f \"$OBJ_FILE\"\n+\tthen\n+\t\t# \"Download\" the missing object by copying it from alt-repo1\n+\t\tOBJ_DIR=$(echo $2 | sed 's/\\(..\\).*/\\1/')\n+\t\tOBJ_BASE=$(basename \"$OBJ_FILE\")\n+\t\tALT_OBJ_DIR1=\"$ALT_SOURCE1/objects/$OBJ_DIR\"\n+\t\tALT_OBJ_DIR2=\"$ALT_SOURCE2/objects/$OBJ_DIR\"\n+\t\tmkdir -p \"$ALT_OBJ_DIR2\" || die \"Could not mkdir '$ALT_OBJ_DIR2'\"\n+\t\tOBJ_SRC=\"$ALT_OBJ_DIR1/$OBJ_BASE\"\n+\t\tcp \"$OBJ_SRC\" \"$ALT_OBJ_DIR2\" ||\n+\t\tdie \"Could not cp '$OBJ_SRC' into '$ALT_OBJ_DIR2'\"\n+\tfi\n+\tcat \"$OBJ_FILE\" || die \"Could not cat '$OBJ_FILE'\"\n+\t;;\n+put_raw_obj)\n+\tsha1=\"$2\"\n+\tsize=\"$3\"\n+\tkind=\"$4\"\n+\twriten=$(git hash-object -w -t \"$kind\" --stdin)\n+\ttest \"$writen\" = \"$sha1\" || die \"bad sha1 passed '$sha1' vs writen '$writen'\"\n+\tref_hash=$(echo \"$sha1 $size $kind\" | GIT_DIR=$OTHER_SOURCE GIT_NO_EXTERNAL_ODB=1 git hash-object -w -t blob --stdin) || exit\n+\tGIT_DIR=$OTHER_SOURCE git update-ref refs/odbs/magic/\"$sha1\" \"$ref_hash\"\n+\t;;\n+*)\n+\tdie \"unknown command '$1'\"\n+\t;;\n+esac\n+EOF\n+HELPER2=\"\\\"$PWD\\\"/odb-helper2\"\n+\n+test_expect_success 'setup first alternate repo' '\n+\tgit init alt-repo1 &&\n+\ttest_commit zero &&\n+\tgit config odb.magic.scriptCommand \"$HELPER1\"\n+'\n+\n+test_expect_success 'setup other repo and its alternate repo' '\n+\tgit init other-repo &&\n+\tgit init alt-repo2 &&\n+\t(cd other-repo &&\n+\t git remote add origin .. &&\n+\t git pull origin master &&\n+\t git checkout master &&\n+\t git log)\n+'\n+\n+test_expect_success 'new blobs are put in first object store' '\n+\ttest_commit one &&\n+\thash1=$(git ls-tree HEAD | grep one.t | cut -f1 | cut -d\\ -f3) &&\n+\tcontent=$(cd alt-repo1 && git show \"$hash1\") &&\n+\ttest \"$content\" = \"one\" &&\n+\ttest_commit two &&\n+\thash2=$(git ls-tree HEAD | grep two.t | cut -f1 | cut -d\\ -f3) &&\n+\tcontent=$(cd alt-repo1 && git show \"$hash2\") &&\n+\ttest \"$content\" = \"two\"\n+'\n+\n+test_expect_success 'other repo gets the blobs from object store' '\n+\t(cd other-repo &&\n+\t git fetch origin \"refs/odbs/magic/*:refs/odbs/magic/*\" &&\n+\t test_must_fail git cat-file blob \"$hash1\" &&\n+\t test_must_fail git cat-file blob \"$hash2\" &&\n+\t git config odb.magic.scriptCommand \"$HELPER2\" &&\n+\t git cat-file blob \"$hash1\" &&\n+\t git cat-file blob \"$hash2\"\n+\t)\n+'\n+\n+test_expect_success 'other repo gets everything else' '\n+\t(cd other-repo &&\n+\t git fetch origin &&\n+\t content=$(git show \"$hash1\") &&\n+\t test \"$content\" = \"one\" &&\n+\t content=$(git show \"$hash2\") &&\n+\t test \"$content\" = \"two\")\n+'\n+\n+test_done\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342182","messageId":"20180319133147.15413-20-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 19/36] t0500: add test for external odb write support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:30Z","receivedAt":"2018-03-19T13:34:15Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n t/t0500-external-odb.sh | 8 ++++++++\n 1 file changed, 8 insertions(+)\n\ndiff --git a/t/t0500-external-odb.sh b/t/t0500-external-odb.sh\nindex 4ccca1e965..f924de870f 100755\n--- a/t/t0500-external-odb.sh\n+++ b/t/t0500-external-odb.sh\n@@ -69,4 +69,12 @@ test_expect_success 'helper can add objects to alt repo' '\n \ttest \"$size\" -eq \"$alt_size\"\n '\n \n+test_expect_success 'commit adds objects to alt repo' '\n+\ttest_config odb.magic.scriptCommand \"$HELPER\" &&\n+\ttest_commit three &&\n+\thash3=$(git ls-tree HEAD | grep three.t | cut -f1 | cut -d\\ -f3) &&\n+\tcontent=$(git -C alt-repo show \"$hash3\") &&\n+\ttest \"$content\" = \"three\"\n+'\n+\n test_done\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342183","messageId":"20180319133147.15413-25-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 24/36] lib-httpd: add apache-e-odb.conf","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:35Z","receivedAt":"2018-03-19T13:34:18Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This is an apache config file to test external object databases.\nIt uses the upload.sh and list.sh cgi that have been added\npreviously to make apache store external objects.\n\nSigned-off-by: Christian Couder \n---\n t/lib-httpd/apache-e-odb.conf | 214 ++++++++++++++++++++++++++++++++++\n 1 file changed, 214 insertions(+)\n create mode 100644 t/lib-httpd/apache-e-odb.conf\n\ndiff --git a/t/lib-httpd/apache-e-odb.conf b/t/lib-httpd/apache-e-odb.conf\nnew file mode 100644\nindex 0000000000..19a1540c82\n--- /dev/null\n+++ b/t/lib-httpd/apache-e-odb.conf\n@@ -0,0 +1,214 @@\n+ServerName dummy\n+PidFile httpd.pid\n+DocumentRoot www\n+LogFormat \"%h %l %u %t \\\"%r\\\" %>s %b\" common\n+CustomLog access.log common\n+ErrorLog error.log\n+\n+\tLoadModule log_config_module modules/mod_log_config.so\n+\n+\n+\tLoadModule alias_module modules/mod_alias.so\n+\n+\n+\tLoadModule cgi_module modules/mod_cgi.so\n+\n+\n+\tLoadModule env_module modules/mod_env.so\n+\n+\n+\tLoadModule rewrite_module modules/mod_rewrite.so\n+\n+\n+\tLoadModule version_module modules/mod_version.so\n+\n+\n+\tLoadModule headers_module modules/mod_headers.so\n+\n+\n+\n+LockFile accept.lock\n+\n+\n+\n+\n+\tLoadModule auth_module modules/mod_auth.so\n+\n+\n+\n+= 2.1>\n+\n+\tLoadModule auth_basic_module modules/mod_auth_basic.so\n+\n+\n+\tLoadModule authn_file_module modules/mod_authn_file.so\n+\n+\n+\tLoadModule authz_user_module modules/mod_authz_user.so\n+\n+\n+\tLoadModule authz_host_module modules/mod_authz_host.so\n+\n+\n+\n+= 2.4>\n+\n+\tLoadModule authn_core_module modules/mod_authn_core.so\n+\n+\n+\tLoadModule authz_core_module modules/mod_authz_core.so\n+\n+\n+\tLoadModule access_compat_module modules/mod_access_compat.so\n+\n+\n+\tLoadModule mpm_prefork_module modules/mod_mpm_prefork.so\n+\n+\n+\tLoadModule unixd_module modules/mod_unixd.so\n+\n+\n+\n+PassEnv GIT_VALGRIND\n+PassEnv GIT_VALGRIND_OPTIONS\n+PassEnv GNUPGHOME\n+PassEnv ASAN_OPTIONS\n+PassEnv GIT_TRACE\n+PassEnv GIT_CONFIG_NOSYSTEM\n+\n+Alias /dumb/ www/\n+Alias /auth/dumb/ www/auth/dumb/\n+\n+\n+\tSetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}\n+\tSetEnv GIT_HTTP_EXPORT_ALL\n+\n+\n+\tSetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}\n+\n+\n+\tSetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}\n+\tSetEnv GIT_HTTP_EXPORT_ALL\n+\tSetEnv GIT_COMMITTER_NAME \"Custom User\"\n+\tSetEnv GIT_COMMITTER_EMAIL custom@example.com\n+\n+\n+\tSetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}\n+\tSetEnv GIT_HTTP_EXPORT_ALL\n+\tSetEnv GIT_NAMESPACE ns\n+\n+\n+\tSetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}\n+\tSetEnv GIT_HTTP_EXPORT_ALL\n+\tHeader set Set-Cookie name=value\n+\n+\n+\tSetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}\n+\tSetEnv GIT_HTTP_EXPORT_ALL\n+\n+ScriptAlias /upload/ upload.sh/\n+ScriptAlias /list/ list.sh/\n+\n+\tOptions FollowSymlinks\n+\n+\n+ Options ExecCGI\n+\n+\n+ Options ExecCGI\n+\n+\n+\tOptions ExecCGI\n+\n+\n+RewriteEngine on\n+RewriteRule ^/smart-redir-perm/(.*)$ /smart/$1 [R=301]\n+RewriteRule ^/smart-redir-temp/(.*)$ /smart/$1 [R=302]\n+RewriteRule ^/smart-redir-auth/(.*)$ /auth/smart/$1 [R=301]\n+RewriteRule ^/smart-redir-limited/(.*)/info/refs$ /smart/$1/info/refs [R=301]\n+RewriteRule ^/ftp-redir/(.*)$ ftp://localhost:1000/$1 [R=302]\n+\n+RewriteRule ^/loop-redir/x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-(.*) /$1 [R=302]\n+RewriteRule ^/loop-redir/(.*)$ /loop-redir/x-$1 [R=302]\n+\n+# Apache 2.2 does not understand , so we use RewriteCond.\n+# And as RewriteCond does not allow testing for non-matches, we match\n+# the desired case first (one has abra, two has cadabra), and let it\n+# pass by marking the RewriteRule as [L], \"last rule, do not process\n+# any other matching RewriteRules after this\"), and then have another\n+# RewriteRule that matches all other cases and lets them fail via '[F]',\n+# \"fail the request\".\n+RewriteCond %{HTTP:x-magic-one} =abra\n+RewriteCond %{HTTP:x-magic-two} =cadabra\n+RewriteRule ^/smart_headers/.* - [L]\n+RewriteRule ^/smart_headers/.* - [F]\n+\n+\n+LoadModule ssl_module modules/mod_ssl.so\n+\n+SSLCertificateFile httpd.pem\n+SSLCertificateKeyFile httpd.pem\n+SSLRandomSeed startup file:/dev/urandom 512\n+SSLRandomSeed connect file:/dev/urandom 512\n+SSLSessionCache none\n+SSLMutex file:ssl_mutex\n+SSLEngine On\n+\n+\n+\n+\tAuthType Basic\n+\tAuthName \"git-auth\"\n+\tAuthUserFile passwd\n+\tRequire valid-user\n+\n+\n+\n+\tAuthType Basic\n+\tAuthName \"git-auth\"\n+\tAuthUserFile passwd\n+\tRequire valid-user\n+\n+\n+\n+\tAuthType Basic\n+\tAuthName \"git-auth\"\n+\tAuthUserFile passwd\n+\tRequire valid-user\n+\n+\n+RewriteCond %{QUERY_STRING} service=git-receive-pack [OR]\n+RewriteCond %{REQUEST_URI} /git-receive-pack$\n+RewriteRule ^/half-auth-complete/ - [E=AUTHREQUIRED:yes]\n+\n+\n+ Order Deny,Allow\n+ Deny from env=AUTHREQUIRED\n+\n+ AuthType Basic\n+ AuthName \"Git Access\"\n+ AuthUserFile passwd\n+ Require valid-user\n+ Satisfy Any\n+\n+\n+\n+\tLoadModule dav_module modules/mod_dav.so\n+\tLoadModule dav_fs_module modules/mod_dav_fs.so\n+\n+\tDAVLockDB DAVLock\n+\t\n+\t\tDav on\n+\t\n+\t\n+\t\tDav on\n+\t\n+\n+\n+\n+\tLoadModule dav_svn_module modules/mod_dav_svn.so\n+\n+\t\n+\t\tDAV svn\n+\t\tSVNPath \"${LIB_HTTPD_SVNPATH}\"\n+\t\n+\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342184","messageId":"20180319133147.15413-24-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 23/36] lib-httpd: add list.sh","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:34Z","receivedAt":"2018-03-19T13:34:20Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This cgi script can list Git objects that have been uploaded as\nfiles to an apache web server. This script can also retrieve\nthe content of each of these files.\n\nThis will help make apache work as an external object database.\n\nSigned-off-by: Christian Couder \n---\n t/lib-httpd.sh | 1 +\n t/lib-httpd/list.sh | 41 +++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 42 insertions(+)\n create mode 100644 t/lib-httpd/list.sh\n\ndiff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\nindex d80b004549..f31ea261f5 100644\n--- a/t/lib-httpd.sh\n+++ b/t/lib-httpd.sh\n@@ -133,6 +133,7 @@ prepare_httpd() {\n \tinstall_script broken-smart-http.sh\n \tinstall_script error.sh\n \tinstall_script upload.sh\n+\tinstall_script list.sh\n \n \tln -s \"$LIB_HTTPD_MODULE_PATH\" \"$HTTPD_ROOT_PATH/modules\"\n \ndiff --git a/t/lib-httpd/list.sh b/t/lib-httpd/list.sh\nnew file mode 100644\nindex 0000000000..b6d6c29a2f\n--- /dev/null\n+++ b/t/lib-httpd/list.sh\n@@ -0,0 +1,41 @@\n+#!/bin/sh\n+\n+FILES_DIR=\"www/files\"\n+\n+OLDIFS=\"$IFS\"\n+IFS='&'\n+set -- $QUERY_STRING\n+IFS=\"$OLDIFS\"\n+\n+while test $# -gt 0\n+do\n+\tkey=${1%%=*}\n+\tval=${1#*=}\n+\n+\tcase \"$key\" in\n+\t\"sha1\") sha1=\"$val\" ;;\n+\t*) echo >&2 \"unknown key '$key'\" ;;\n+\tesac\n+\n+\tshift\n+done\n+\n+if test -d \"$FILES_DIR\"\n+then\n+\tif test -z \"$sha1\"\n+\tthen\n+\t\techo 'Status: 200 OK'\n+\t\techo\n+\t\tls \"$FILES_DIR\" | tr '-' ' '\n+\telse\n+\t\tif test -f \"$FILES_DIR/$sha1\"-*\n+\t\tthen\n+\t\t\techo 'Status: 200 OK'\n+\t\t\techo\n+\t\t\tcat \"$FILES_DIR/$sha1\"-*\n+\t\telse\n+\t\t\techo 'Status: 404 Not Found'\n+\t\t\techo\n+\t\tfi\n+\tfi\n+fi\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342185","messageId":"20180319133147.15413-11-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 10/36] odb-helper: add 'script_mode' to 'struct odb_helper'","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:21Z","receivedAt":"2018-03-19T13:34:26Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"to prepare for having a long running odb helper sub-process\nhandling the communication between Git and an external odb.\n\nWe introduce \"odb..subprocesscommand\" to make it\npossible to define such a sub-process, and we mark such odb\nhelpers with the new 'script_mode' field set to 0.\n\nHelpers defined using the existing \"odb..scriptcommand\"\nare marked with the 'script_mode' field set to 1.\n\nImplementation of the different capabilities/instructions in\nthe new (sub-)process mode is left for following commits.\n\nSigned-off-by: Christian Couder \n---\n external-odb.c | 4 ++++\n odb-helper.c | 16 +++++++++++-----\n 2 files changed, 15 insertions(+), 5 deletions(-)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex ec1ec82dd9..9f797d66f4 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -41,6 +41,10 @@ static int external_odb_config(const char *var, const char *value, void *data)\n \t\to->type = ODB_HELPER_SCRIPT_CMD;\n \t\treturn git_config_string(&o->dealer, var, value);\n \t}\n+\tif (!strcmp(subkey, \"subprocesscommand\")) {\n+\t\to->type = ODB_HELPER_SUBPROCESS_CMD;\n+\t\treturn git_config_string(&o->dealer, var, value);\n+\t}\n \n \treturn 0;\n }\ndiff --git a/odb-helper.c b/odb-helper.c\nindex 677ea850a5..bf3c1ad8ac 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -116,16 +116,12 @@ static int odb_helper_object_cmp(const void *va, const void *vb)\n \treturn oidcmp(&a->oid, &b->oid);\n }\n \n-static void odb_helper_load_have(struct odb_helper *o)\n+static void have_object_script(struct odb_helper *o)\n {\n \tstruct odb_helper_cmd cmd;\n \tFILE *fh;\n \tstruct strbuf line = STRBUF_INIT;\n \n-\tif (o->have_valid)\n-\t\treturn;\n-\to->have_valid = 1;\n-\n \tif (odb_helper_start(o, &cmd, \"have\") < 0)\n \t\treturn;\n \n@@ -137,6 +133,16 @@ static void odb_helper_load_have(struct odb_helper *o)\n \tstrbuf_release(&line);\n \tfclose(fh);\n \todb_helper_finish(o, &cmd);\n+}\n+\n+static void odb_helper_load_have(struct odb_helper *o)\n+{\n+\tif (o->have_valid)\n+\t\treturn;\n+\to->have_valid = 1;\n+\n+\tif (o->type == ODB_HELPER_SCRIPT_CMD)\n+\t\thave_object_script(o);\n \n \tqsort(o->have, o->have_nr, sizeof(*o->have), odb_helper_object_cmp);\n }\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342186","messageId":"20180319133147.15413-10-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 09/36] external-odb: add script mode support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:20Z","receivedAt":"2018-03-19T13:34:29Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This adds support for the script command mode where\nan helper script or command is called to retrieve or\nmanage objects.\n\nThis implements the 'have' and 'get_git_obj'\ninstructions for the script mode.\n\nSigned-off-by: Christian Couder \n---\n external-odb.c | 55 +++++++++-\n external-odb.h | 1 +\n odb-helper.c | 218 +++++++++++++++++++++++++++++++++++++++-\n odb-helper.h | 4 +\n sha1_file.c | 12 ++-\n t/t0500-external-odb.sh | 44 ++++++++\n 6 files changed, 331 insertions(+), 3 deletions(-)\n create mode 100755 t/t0500-external-odb.sh\n\ndiff --git a/external-odb.c b/external-odb.c\nindex 82ac80aa04..ec1ec82dd9 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -33,8 +33,14 @@ static int external_odb_config(const char *var, const char *value, void *data)\n \n \to = find_or_create_helper(name, namelen);\n \n-\tif (!strcmp(subkey, \"promisorremote\"))\n+\tif (!strcmp(subkey, \"promisorremote\")) {\n+\t\to->type = ODB_HELPER_GIT_REMOTE;\n \t\treturn git_config_string(&o->dealer, var, value);\n+\t}\n+\tif (!strcmp(subkey, \"scriptcommand\")) {\n+\t\to->type = ODB_HELPER_SCRIPT_CMD;\n+\t\treturn git_config_string(&o->dealer, var, value);\n+\t}\n \n \treturn 0;\n }\n@@ -100,6 +106,53 @@ int external_odb_has_object(const unsigned char *sha1)\n \treturn 0;\n }\n \n+int external_odb_get_object(const unsigned char *sha1)\n+{\n+\tstruct odb_helper *o;\n+\tstruct strbuf pathbuf = STRBUF_INIT;\n+\n+\tif (!external_odb_has_object(sha1))\n+\t\treturn -1;\n+\n+\tsha1_file_name_alt(&pathbuf, external_odb_root(), sha1);\n+\tsafe_create_leading_directories_const(pathbuf.buf);\n+\tprepare_external_alt_odb();\n+\n+\tfor (o = helpers; o; o = o->next) {\n+\t\tstruct strbuf tmpfile = STRBUF_INIT;\n+\t\tint ret;\n+\t\tint fd;\n+\n+\t\tif (!odb_helper_has_object(o, sha1))\n+\t\t\tcontinue;\n+\n+\t\tfd = create_object_tmpfile(&tmpfile, pathbuf.buf);\n+\t\tif (fd < 0) {\n+\t\t\tstrbuf_release(&tmpfile);\n+\t\t\tstrbuf_release(&pathbuf);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tif (odb_helper_get_object(o, sha1, fd) < 0) {\n+\t\t\tclose(fd);\n+\t\t\tunlink(tmpfile.buf);\n+\t\t\tstrbuf_release(&tmpfile);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tclose_sha1_file(fd);\n+\t\tret = finalize_object_file(tmpfile.buf, pathbuf.buf);\n+\t\tstrbuf_release(&tmpfile);\n+\t\tstrbuf_release(&pathbuf);\n+\t\tif (!ret)\n+\t\t\treturn 0;\n+\t}\n+\n+\tstrbuf_release(&pathbuf);\n+\n+\treturn -1;\n+}\n+\n int external_odb_get_direct(const unsigned char *sha1)\n {\n \tstruct odb_helper *o;\ndiff --git a/external-odb.h b/external-odb.h\nindex 254070994a..c5ac071995 100644\n--- a/external-odb.h\n+++ b/external-odb.h\n@@ -15,6 +15,7 @@ extern struct odb_helper *find_odb_helper(const char *dealer,\n \t\t\t\t\t enum odb_helper_type type);\n extern const char *external_odb_root(void);\n extern int external_odb_has_object(const unsigned char *sha1);\n+extern int external_odb_get_object(const unsigned char *sha1);\n extern int external_odb_get_direct(const unsigned char *sha1);\n \n #endif /* EXTERNAL_ODB_H */\ndiff --git a/odb-helper.c b/odb-helper.c\nindex c6127ce81f..677ea850a5 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -21,13 +21,124 @@ struct odb_helper_cmd {\n \tstruct child_process child;\n };\n \n+/*\n+ * Callers are responsible to ensure that the result of vaddf(fmt, ap)\n+ * is properly shell-quoted.\n+ */\n+static void prepare_helper_command(struct argv_array *argv, const char *cmd,\n+\t\t\t\t const char *fmt, va_list ap)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tstrbuf_addstr(&buf, cmd);\n+\tstrbuf_addch(&buf, ' ');\n+\tstrbuf_vaddf(&buf, fmt, ap);\n+\n+\targv_array_push(argv, buf.buf);\n+\tstrbuf_release(&buf);\n+}\n+\n+__attribute__((format (printf,3,4)))\n+static int odb_helper_start(struct odb_helper *o,\n+\t\t\t struct odb_helper_cmd *cmd,\n+\t\t\t const char *fmt, ...)\n+{\n+\tva_list ap;\n+\n+\tmemset(cmd, 0, sizeof(*cmd));\n+\targv_array_init(&cmd->argv);\n+\n+\tif (!o->dealer)\n+\t\treturn -1;\n+\n+\tva_start(ap, fmt);\n+\tprepare_helper_command(&cmd->argv, o->dealer, fmt, ap);\n+\tva_end(ap);\n+\n+\tcmd->child.argv = cmd->argv.argv;\n+\tcmd->child.use_shell = 1;\n+\tcmd->child.no_stdin = 1;\n+\tcmd->child.out = -1;\n+\n+\tif (start_command(&cmd->child) < 0) {\n+\t\targv_array_clear(&cmd->argv);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int odb_helper_finish(struct odb_helper *o,\n+\t\t\t struct odb_helper_cmd *cmd)\n+{\n+\tint ret = finish_command(&cmd->child);\n+\targv_array_clear(&cmd->argv);\n+\tif (ret) {\n+\t\twarning(\"odb helper '%s' reported failure\", o->name);\n+\t\treturn -1;\n+\t}\n+\treturn 0;\n+}\n+\n+static int parse_object_line(struct odb_helper_object *o, const char *line)\n+{\n+\tchar *end;\n+\tif (get_oid_hex(line, &o->oid) < 0)\n+\t\treturn -1;\n+\n+\tline += 40;\n+\tif (*line++ != ' ')\n+\t\treturn -1;\n+\n+\to->size = strtoul(line, &end, 10);\n+\tif (line == end || *end++ != ' ')\n+\t\treturn -1;\n+\n+\to->type = type_from_string(end);\n+\treturn 0;\n+}\n+\n+static int add_have_entry(struct odb_helper *o, const char *line)\n+{\n+\tALLOC_GROW(o->have, o->have_nr+1, o->have_alloc);\n+\tif (parse_object_line(&o->have[o->have_nr], line) < 0) {\n+\t\twarning(\"bad 'have' input from odb helper '%s': %s\",\n+\t\t\to->name, line);\n+\t\treturn 1;\n+\t}\n+\to->have_nr++;\n+\treturn 0;\n+}\n+\n+static int odb_helper_object_cmp(const void *va, const void *vb)\n+{\n+\tconst struct odb_helper_object *a = va, *b = vb;\n+\treturn oidcmp(&a->oid, &b->oid);\n+}\n+\n static void odb_helper_load_have(struct odb_helper *o)\n {\n+\tstruct odb_helper_cmd cmd;\n+\tFILE *fh;\n+\tstruct strbuf line = STRBUF_INIT;\n+\n \tif (o->have_valid)\n \t\treturn;\n \to->have_valid = 1;\n \n-\t/* TODO */\n+\tif (odb_helper_start(o, &cmd, \"have\") < 0)\n+\t\treturn;\n+\n+\tfh = xfdopen(cmd.child.out, \"r\");\n+\twhile (strbuf_getline(&line, fh) != EOF)\n+\t\tif (add_have_entry(o, line.buf))\n+\t\t\tbreak;\n+\n+\tstrbuf_release(&line);\n+\tfclose(fh);\n+\todb_helper_finish(o, &cmd);\n+\n+\tqsort(o->have, o->have_nr, sizeof(*o->have), odb_helper_object_cmp);\n }\n \n static const unsigned char *have_sha1_access(size_t index, void *table)\n@@ -53,6 +164,111 @@ int odb_helper_has_object(struct odb_helper *o, const unsigned char *sha1)\n \treturn !!odb_helper_lookup(o, sha1);\n }\n \n+int odb_helper_get_object(struct odb_helper *o, const unsigned char *sha1,\n+\t\t\t int fd)\n+{\n+\tstruct odb_helper_object *obj;\n+\tstruct odb_helper_cmd cmd;\n+\tunsigned long total_got;\n+\tgit_zstream stream;\n+\tint zret = Z_STREAM_END;\n+\tgit_SHA_CTX hash;\n+\tunsigned char real_sha1[20];\n+\tstruct strbuf header = STRBUF_INIT;\n+\tunsigned long hdr_size;\n+\n+\tobj = odb_helper_lookup(o, sha1);\n+\tif (!obj)\n+\t\treturn -1;\n+\n+\tif (odb_helper_start(o, &cmd, \"get_git_obj %s\", sha1_to_hex(sha1)) < 0)\n+\t\treturn -1;\n+\n+\tmemset(&stream, 0, sizeof(stream));\n+\tgit_inflate_init(&stream);\n+\tgit_SHA1_Init(&hash);\n+\ttotal_got = 0;\n+\n+\tfor (;;) {\n+\t\tunsigned char buf[4096];\n+\t\tint r;\n+\n+\t\tr = xread(cmd.child.out, buf, sizeof(buf));\n+\t\tif (r < 0) {\n+\t\t\terror(\"unable to read from odb helper '%s': %s\",\n+\t\t\t o->name, strerror(errno));\n+\t\t\tclose(cmd.child.out);\n+\t\t\todb_helper_finish(o, &cmd);\n+\t\t\tgit_inflate_end(&stream);\n+\t\t\treturn -1;\n+\t\t}\n+\t\tif (r == 0)\n+\t\t\tbreak;\n+\n+\t\twrite_or_die(fd, buf, r);\n+\n+\t\tstream.next_in = buf;\n+\t\tstream.avail_in = r;\n+\t\tdo {\n+\t\t\tunsigned char inflated[4096];\n+\t\t\tunsigned long got;\n+\n+\t\t\tstream.next_out = inflated;\n+\t\t\tstream.avail_out = sizeof(inflated);\n+\t\t\tzret = git_inflate(&stream, Z_SYNC_FLUSH);\n+\t\t\tgot = sizeof(inflated) - stream.avail_out;\n+\n+\t\t\tgit_SHA1_Update(&hash, inflated, got);\n+\t\t\t/* skip header when counting size */\n+\t\t\tif (!total_got) {\n+\t\t\t\tconst unsigned char *p = memchr(inflated, '\\0', got);\n+\t\t\t\tif (p) {\n+\t\t\t\t\tunsigned long hdr_last = p - inflated + 1;\n+\t\t\t\t\tstrbuf_add(&header, inflated, hdr_last);\n+\t\t\t\t\tgot -= hdr_last;\n+\t\t\t\t} else {\n+\t\t\t\t\tstrbuf_add(&header, inflated, got);\n+\t\t\t\t\tgot = 0;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t\ttotal_got += got;\n+\t\t} while (stream.avail_in && zret == Z_OK);\n+\t}\n+\n+\tclose(cmd.child.out);\n+\tgit_inflate_end(&stream);\n+\tgit_SHA1_Final(real_sha1, &hash);\n+\tif (odb_helper_finish(o, &cmd))\n+\t\treturn -1;\n+\tif (zret != Z_STREAM_END) {\n+\t\twarning(\"bad zlib data from odb helper '%s' for %s\",\n+\t\t\to->name, sha1_to_hex(sha1));\n+\t\treturn -1;\n+\t}\n+\tif (total_got != obj->size) {\n+\t\twarning(\"size mismatch from odb helper '%s' for %s (%lu != %lu)\",\n+\t\t\to->name, sha1_to_hex(sha1), total_got, obj->size);\n+\t\treturn -1;\n+\t}\n+\tif (hashcmp(real_sha1, sha1)) {\n+\t\twarning(\"sha1 mismatch from odb helper '%s' for %s (got %s)\",\n+\t\t\to->name, sha1_to_hex(sha1), sha1_to_hex(real_sha1));\n+\t\treturn -1;\n+\t}\n+\tif (parse_sha1_header(header.buf, &hdr_size) < 0) {\n+\t\twarning(\"could not parse header from odb helper '%s' for %s\",\n+\t\t\to->name, sha1_to_hex(sha1));\n+\t\treturn -1;\n+\t}\n+\tif (total_got != hdr_size) {\n+\t\twarning(\"size mismatch from odb helper '%s' for %s (%lu != %lu)\",\n+\t\t\to->name, sha1_to_hex(sha1), total_got, hdr_size);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n int odb_helper_get_direct(struct odb_helper *o,\n \t\t\t const unsigned char *sha1)\n {\ndiff --git a/odb-helper.h b/odb-helper.h\nindex d4b968693f..819182f76a 100644\n--- a/odb-helper.h\n+++ b/odb-helper.h\n@@ -23,6 +23,10 @@ struct odb_helper {\n extern struct odb_helper *odb_helper_new(const char *name, int namelen);\n extern int odb_helper_has_object(struct odb_helper *o,\n \t\t\t\t const unsigned char *sha1);\n+extern int odb_helper_get_object(struct odb_helper *o,\n+\t\t\t\t const unsigned char *sha1,\n+\t\t\t\t int fd);\n extern int odb_helper_get_direct(struct odb_helper *o,\n \t\t\t\t const unsigned char *sha1);\n+\n #endif /* ODB_HELPER_H */\ndiff --git a/sha1_file.c b/sha1_file.c\nindex c460eb2658..847883f261 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -907,6 +907,9 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,\n \t\t\treturn 0;\n \t}\n \n+\tif (!external_odb_get_object(sha1) && !lstat(*path, st))\n+\t\treturn 0;\n+\n \treturn -1;\n }\n \n@@ -946,7 +949,14 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)\n \tif (fd >= 0)\n \t\treturn fd;\n \n-\treturn open_sha1_file_alt(sha1, path);\n+\tfd = open_sha1_file_alt(sha1, path);\n+\tif (fd >= 0)\n+\t\treturn fd;\n+\n+\tif (!external_odb_get_object(sha1))\n+\t\tfd = open_sha1_file_alt(sha1, path);\n+\n+\treturn fd;\n }\n \n /*\ndiff --git a/t/t0500-external-odb.sh b/t/t0500-external-odb.sh\nnew file mode 100755\nindex 0000000000..eaa3688a31\n--- /dev/null\n+++ b/t/t0500-external-odb.sh\n@@ -0,0 +1,44 @@\n+#!/bin/sh\n+\n+test_description='basic tests for external object databases'\n+\n+. ./test-lib.sh\n+\n+ALT_SOURCE=\"$PWD/alt-repo/.git\"\n+export ALT_SOURCE\n+write_script odb-helper <<\\EOF\n+GIT_DIR=$ALT_SOURCE; export GIT_DIR\n+case \"$1\" in\n+have)\n+\tgit cat-file --batch-check --batch-all-objects |\n+\tawk '{print $1 \" \" $3 \" \" $2}'\n+\t;;\n+get_git_obj)\n+\tcat \"$GIT_DIR\"/objects/$(echo $2 | sed 's#..#&/#')\n+\t;;\n+esac\n+EOF\n+HELPER=\"\\\"$PWD\\\"/odb-helper\"\n+\n+test_expect_success 'setup alternate repo' '\n+\tgit init alt-repo &&\n+\ttest_commit -C alt-repo one &&\n+\ttest_commit -C alt-repo two &&\n+\talt_head=$(git -C alt-repo rev-parse HEAD)\n+'\n+\n+test_expect_success 'alt objects are missing' '\n+\ttest_must_fail git log --format=%s $alt_head\n+'\n+\n+test_expect_success 'helper can retrieve alt objects' '\n+\ttest_config odb.magic.scriptCommand \"$HELPER\" &&\n+\tcat >expect <<-\\EOF &&\n+\ttwo\n+\tone\n+\tEOF\n+\tgit log --format=%s $alt_head >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342187","messageId":"20180319133147.15413-9-chriscool@tuxfamily.org","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"[PATCH v2 08/36] external-odb: add external_odb_reinit()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-03-19T13:31:19Z","receivedAt":"2018-03-19T13:34:31Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"As we will need to reinitialize the list of odb helpers\nfrom the config file, let's add external_odb_reinit() for\nthat purpose.\n\nSigned-off-by: Christian Couder \n---\n external-odb.c | 14 ++++++++++++--\n external-odb.h | 1 +\n 2 files changed, 13 insertions(+), 2 deletions(-)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex 9c77180d6c..82ac80aa04 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -39,17 +39,27 @@ static int external_odb_config(const char *var, const char *value, void *data)\n \treturn 0;\n }\n \n-static void external_odb_init(void)\n+static void external_odb_do_init(int force)\n {\n \tstatic int initialized;\n \n-\tif (initialized || !use_external_odb)\n+\tif ((!force && initialized) || !use_external_odb)\n \t\treturn;\n \tinitialized = 1;\n \n \tgit_config(external_odb_config, NULL);\n }\n \n+static inline void external_odb_init(void)\n+{\n+\texternal_odb_do_init(0);\n+}\n+\n+inline void external_odb_reinit(void)\n+{\n+\texternal_odb_do_init(1);\n+}\n+\n int has_external_odb(void)\n {\n \texternal_odb_init();\ndiff --git a/external-odb.h b/external-odb.h\nindex 27c3d39c1b..254070994a 100644\n--- a/external-odb.h\n+++ b/external-odb.h\n@@ -9,6 +9,7 @@ enum odb_helper_type {\n \tOBJ_HELPER_MAX\n };\n \n+extern void external_odb_reinit(void);\n extern int has_external_odb(void);\n extern struct odb_helper *find_odb_helper(const char *dealer,\n \t\t\t\t\t enum odb_helper_type type);\n-- \n2.17.0.rc0.37.g8f476fabe9\n\n"},{"id":"342232","messageId":"xmqqh8pbvpxu.fsf@gitster-ct.c.googlers.com","threadId":"48083","inReplyTo":"20180319133147.15413-1-chriscool@tuxfamily.org","subject":"Re: [PATCH v2 00/36] Promisor remotes and external ODB support","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-19T19:13:49Z","receivedAt":"2018-03-19T19:13:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> A lot of things are different because the jh/fsck-promisors and\n> jh/partial-clone have been merged into master since the v1. So the\n> integration is much more complete now (though not fully complete), and\n> this integration happens quite early in the patch series.\n>\n> This integration makes it possible to have many promisor and partial\n> clone remotes (instead of just one) with possibly different partial\n> clone filters, though that is not tested yet.\n>\n> I am not sure that the \"external odb\" name is still the best, as the\n> promisor remotes are not quite external. So I am open to suggestions\n> about a new name.\n\nSo,... so far we have a way to make an incomplete clone of a project\nfrom a remote that promises that objects (deliberately) missing from\nthe resulting repository are available later by an on-demand request.\n\nWe do not yet have code to actually make an on-demand request, and\nthe other side of the request to fulfill the promise. \n\nAnd that is what these patches want to do?\n\nThat sounds like a \"lazy backfill\" mechanism; I know others are\nbetter in naming things than me, though ;-)\n\nOn the other hand, if the code updated with these patches do not\ncooperate with the promise mechansim (e.g. request is made to any\nmissing objects, instead of \"this object was promised by that\nremote, so let's go there and ask\" and \"this object is simply\nmissing, without promise by anybody, so let's not bother the\npromisor remote\"), then it is not even \"back\"-filling, but is a\nmechanism to access remote object database over a protocol, so\na minimum s/ext/remote/ would clarify what it is.\n\nI guess \"lazy backfill\" would be more preferrable than a pile of\nindependent, competing and uncooperative features ;-)\n\n> - Patches 13/36 and 14/36:\n>\n> These patches move over the promisor objects and partial clone code to\n> use the external odb mechanism. The result of 13/36 is that instead of\n> the \"extensions.partialclone\" config variable, a\n> \"odb..promisorRemote\" config variable is now used. The result of\n> 14/36 is that instead of the \"core.partialclonefilter\" config\n> variable, a \"odb..partialclonefilter\" config variable is now\n> used.\n\nThe use of \"extensions\" was to protect the repository from versions\nof Git that are unaware of the \"promise\" mechanism to even attempt\ntouching it. Will we keep the same degree of safety with these\nchanges, I wonder?\n\n"}]}