{"thread":{"id":"42589","subject":"[RFC/PATCH 0/8] Add initial experimental external ODB support","startedAt":"2016-06-16T02:19:53Z","lastAt":"2016-06-16T02:19:53Z","messageCount":10,"participants":["Christian Couder","Duy Nguyen"],"isPatch":true,"patchVersion":1,"patchTotal":8},"messages":[{"id":"289085","messageId":"20160613085546.11784-1-chriscool@tuxfamily.org","threadId":"42589","inReplyTo":null,"subject":"[RFC/PATCH 0/8] Add initial experimental external ODB support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-06-13T08:55:38Z","receivedAt":"2016-06-16T02:19:53Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Goal\n~~~~\n\nGit can store its objects only in the form of loose objects in\nseparate files or packed objects in a pack file.\nTo be able to better handle some kind of objects, for example big\nblobs, it would be nice if Git could store its objects in other object\ndatabases (ODB).\nTo do that, this patch series makes it possible to register commands,\nusing \"odb.<odbname>.command\" config variables, to access external\nODBs where objects can be stored and retrieved.\n\nDesign\n~~~~~~\n\nEach registered command manages access to one external ODB and will be\ncalled the following ways:\n\n  - \"<command> have\": the command should output the sha1, size and\ntype of all the objects the external ODB contains, one object per\nline.\n\n  - \"<command> get <sha1>\": the command should then read from the\nexternal ODB the content of the object corresponding to <sha1> and\noutput it on stdout.\n\n  - \"<command> put <sha1> <size> <type>\": the command should then read\nfrom stdin an object and store it in the external ODB.\n\nThis RFC patch series for now does not address the following important\nparts of a complete solution:\n\n  - There is no way to transfer external ODB content using Git.\n\n  - No real external ODB has been interfaced with Git. The tests use\nanother git repo in a separate directory for this purpose which is\nprobably useless in the real world.\n\nDesign discussion about performance\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nYeah, it is not efficient to fork/exec a command to just read or write\none object to or from the external ODB. Batch calls and/or using a\ndaemon and/or RPC should be used instead to be able to store regular\nobjects in an external ODB. But for now the external ODB would be all\nabout really big files, where the cost of a fork+exec should not\nmatter much. If we later want to extend usage of external ODBs, yeah\nwe will probably need to design other mechanisms.\n\nHere are some related explanations from Peff:\n\n{{{\nBecause this \"external odb\" essentially acts as a git alternate, we\nwould hit it only when we couldn't find an object through regular means.\nGit would then make the object available in the usual on-disk format\n(probably as a loose object).\n\nSo in most processes, we would not need to consult the odb command at\nall. And when we do, the first thing would be to get its \"have\" list,\nwhich would at most run once per process.\n\nSo the per-object cost is really calling \"get\", and my assumption there\nwas that the cost of actually retrieving the object over the network\nwould dwarf the fork/exec cost.\n\nI also waffled on having git cache the output of \"<command> have\" in\nsome fast-lookup format to save even the single fork/exec. But I figured\nthat was something that could be added later if needed.\n\nYou'll note that this is sort of a \"fault-in\" model. Another model would\nbe to treat external odb updates similar to fetches. I.e., we touch the\nnetwork only during a special update operation, and then try to work\nlocally with whatever the external odb has. IMHO this policy could\nactually be up to the external odb itself (i.e., its \"have\" command\ncould serve from a local cache if it likes).\n}}}\n\nImplementation\n~~~~~~~~~~~~~~\n\nThis series adds a set of function in external-odb.{c,h} that are\ncalled by the rest of Git to manage all the external ODBs.\n\nThese functions use 'struct odb_helper' and its associated functions\ndefined in odb-helper.{c,h} to talk to the different external ODBs by\nlaunching the configured \"odb.<odbname>.command\" commands and writing\nto or reading from them.\n\nThe tests in this series creates an odb-helper script that is\nregistered using the \"odb.magic.command\" config variable, and then\ncalled to read from and write to the external ODB.\n\nHighlevel view of the patches in the series\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n    - Patches 01/08 and 02/08 are Peff's initial work fixed a little\n      bit so that it compiles and pass tests.\n\n    - Patches 03/08 is an optimization in the odb-helper script that\n      is used for testing. I will probably squash it into 01/08.\n\n    - Patches 04/08 and 05/08 are adding \"put\" support in the\n      odb-helper script and testing that.\n\n    - Patches 06/08 and 08/08 are enhancing external-odb.{c,h} and\n      odb-helper.{c,h}, so that Git can write into an external ODB.\n\n    - Patches 07/08 limits write support to \"blobs\" for now to\n      simplify things.\n\nFuture work\n~~~~~~~~~~~\n\n>From the discussions it appear that using the bundle v3 mechanism to\ntranfer external ODB data could work, but only if the server has access\nto its own external ODB.\n\nAnother possible mechanism to transfer external ODB data would be some\nkind of replace refs. This would be slower but the mechanism for the\ntransfer already fully exists.\n\nSo I think I am going to experiment with some kind of replace refs.\n\nOne interesting thing also would be to use the streaming api when\nreading from or writing to the external ODB.\n\nPrevious work and discussions\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nPeff started to work on this and discuss this some years ago:\n\nhttp://thread.gmane.org/gmane.comp.version-control.git/206886/focus=207040\nhttp://thread.gmane.org/gmane.comp.version-control.git/247171\nhttp://thread.gmane.org/gmane.comp.version-control.git/202902/focus=203020\n\nHis work, which is not compile-tested any more, is still there:\n\nhttps://github.com/peff/git/commits/jk/external-odb-wip\n\nInitial discussions about this new series are there:\n\nhttp://thread.gmane.org/gmane.comp.version-control.git/288151/focus=295160\n\nLinks\n~~~~~\n\nThis patch series is available here:\n\nhttps://github.com/chriscool/git/commits/external-odb\n\n\nChristian Couder (6):\n  t0400: use --batch-all-objects to get all objects\n  t0400: add 'put' command to odb-helper script\n  t0400: add test for 'put' command\n  external odb: add write support\n  external-odb: accept only blobs for now\n  t0400: add test for external odb write support\n\nJeff King (2):\n  Add initial external odb support\n  external odb foreach\n\n Makefile                |   2 +\n cache.h                 |   9 ++\n external-odb.c          | 148 +++++++++++++++++++++++++\n external-odb.h          |  16 +++\n odb-helper.c            | 287 ++++++++++++++++++++++++++++++++++++++++++++++++\n odb-helper.h            |  32 ++++++\n sha1_file.c             |  66 ++++++++---\n t/t0400-external-odb.sh |  77 +++++++++++++\n 8 files changed, 622 insertions(+), 15 deletions(-)\n create mode 100644 external-odb.c\n create mode 100644 external-odb.h\n create mode 100644 odb-helper.c\n create mode 100644 odb-helper.h\n create mode 100755 t/t0400-external-odb.sh\n\n-- \n2.9.0.rc2.362.g3cd93d0\n"},{"id":"289086","messageId":"20160613085546.11784-3-chriscool@tuxfamily.org","threadId":"42589","inReplyTo":"20160613085546.11784-1-chriscool@tuxfamily.org","subject":"[RFC/PATCH 2/8] external odb foreach","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-06-13T08:55:40Z","receivedAt":"2016-06-16T02:19:53Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: Jeff King <peff@peff.net>\n\n---\n external-odb.c | 14 ++++++++++++++\n external-odb.h |  6 ++++++\n odb-helper.c   | 15 +++++++++++++++\n odb-helper.h   |  4 ++++\n 4 files changed, 39 insertions(+)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex 1ccfa99..42978a3 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -113,3 +113,17 @@ int external_odb_fetch_object(const unsigned char *sha1)\n \n \treturn -1;\n }\n+\n+int external_odb_for_each_object(each_external_object_fn fn, void *data)\n+{\n+\tstruct odb_helper *o;\n+\n+\texternal_odb_init();\n+\n+\tfor (o = helpers; o; o = o->next) {\n+\t\tint r = odb_helper_for_each_object(o, fn, data);\n+\t\tif (r)\n+\t\t\treturn r;\n+\t}\n+\treturn 0;\n+}\ndiff --git a/external-odb.h b/external-odb.h\nindex 2397477..cea8570 100644\n--- a/external-odb.h\n+++ b/external-odb.h\n@@ -5,4 +5,10 @@ const char *external_odb_root(void);\n int external_odb_has_object(const unsigned char *sha1);\n int external_odb_fetch_object(const unsigned char *sha1);\n \n+typedef int (*each_external_object_fn)(const unsigned char *sha1,\n+\t\t\t\t       enum object_type type,\n+\t\t\t\t       unsigned long size,\n+\t\t\t\t       void *data);\n+int external_odb_for_each_object(each_external_object_fn, void *);\n+\n #endif /* EXTERNAL_ODB_H */\ndiff --git a/odb-helper.c b/odb-helper.c\nindex 7029a59..045cf6f 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -235,5 +235,20 @@ int odb_helper_fetch_object(struct odb_helper *o, const unsigned char *sha1,\n \t\treturn -1;\n \t}\n \n+\treturn 0;\n+}\n+\n+int odb_helper_for_each_object(struct odb_helper *o,\n+\t\t\t       each_external_object_fn fn,\n+\t\t\t       void *data)\n+{\n+\tint i;\n+\tfor (i = 0; i < o->have_nr; i++) {\n+\t\tstruct odb_helper_object *obj = &o->have[i];\n+\t\tint r = fn(obj->sha1, obj->type, obj->size, data);\n+\t\tif (r)\n+\t\t\treturn r;\n+\t}\n+\n \treturn 0;\n }\ndiff --git a/odb-helper.h b/odb-helper.h\nindex 0f704f9..8c3916d 100644\n--- a/odb-helper.h\n+++ b/odb-helper.h\n@@ -1,6 +1,8 @@\n #ifndef ODB_HELPER_H\n #define ODB_HELPER_H\n \n+#include \"external-odb.h\"\n+\n struct odb_helper {\n \tconst char *name;\n \tconst char *cmd;\n@@ -21,5 +23,7 @@ struct odb_helper *odb_helper_new(const char *name, int namelen);\n int odb_helper_has_object(struct odb_helper *o, const unsigned char *sha1);\n int odb_helper_fetch_object(struct odb_helper *o, const unsigned char *sha1,\n \t\t\t    int fd);\n+int odb_helper_for_each_object(struct odb_helper *o,\n+\t\t\t       each_external_object_fn, void *);\n \n #endif /* ODB_HELPER_H */\n-- \n2.9.0.rc2.362.g3cd93d0\n"},{"id":"289087","messageId":"20160613085546.11784-5-chriscool@tuxfamily.org","threadId":"42589","inReplyTo":"20160613085546.11784-1-chriscool@tuxfamily.org","subject":"[RFC/PATCH 4/8] t0400: add 'put' command to odb-helper script","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-06-13T08:55:42Z","receivedAt":"2016-06-16T02:19:53Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n t/t0400-external-odb.sh | 14 ++++++++++++++\n 1 file changed, 14 insertions(+)\n\ndiff --git a/t/t0400-external-odb.sh b/t/t0400-external-odb.sh\nindex fe85413..0f1bb97 100755\n--- a/t/t0400-external-odb.sh\n+++ b/t/t0400-external-odb.sh\n@@ -7,6 +7,10 @@ test_description='basic tests for external object databases'\n ALT_SOURCE=\"$PWD/alt-repo/.git\"\n export ALT_SOURCE\n write_script odb-helper <<\\EOF\n+die() {\n+\tprintf >&2 \"%s\\n\" \"$@\"\n+\texit 1\n+}\n GIT_DIR=$ALT_SOURCE; export GIT_DIR\n case \"$1\" in\n have)\n@@ -16,6 +20,16 @@ have)\n get)\n \tcat \"$GIT_DIR\"/objects/$(echo $2 | sed 's#..#&/#')\n \t;;\n+put)\n+\tsha1=\"$2\"\n+\tsize=\"$3\"\n+\tkind=\"$4\"\n+\twriten=$(git hash-object -w -t \"$kind\" --stdin)\n+\ttest \"$writen\" = \"$sha1\" || die \"bad sha1 passed '$sha1' vs writen '$writen'\"\n+\t;;\n+*)\n+\tdie \"unknown command '$1'\"\n+\t;;\n esac\n EOF\n HELPER=\"\\\"$PWD\\\"/odb-helper\"\n-- \n2.9.0.rc2.362.g3cd93d0\n"},{"id":"289088","messageId":"20160613085546.11784-2-chriscool@tuxfamily.org","threadId":"42589","inReplyTo":"20160613085546.11784-1-chriscool@tuxfamily.org","subject":"[RFC/PATCH 1/8] Add initial external odb support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-06-13T08:55:39Z","receivedAt":"2016-06-16T02:19:53Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: Jeff King <peff@peff.net>\n\nSigned-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n Makefile                |   2 +\n cache.h                 |   9 ++\n external-odb.c          | 115 +++++++++++++++++++++++\n external-odb.h          |   8 ++\n odb-helper.c            | 239 ++++++++++++++++++++++++++++++++++++++++++++++++\n odb-helper.h            |  25 +++++\n sha1_file.c             |  64 ++++++++++---\n t/t0400-external-odb.sh |  48 ++++++++++\n 8 files changed, 495 insertions(+), 15 deletions(-)\n create mode 100644 external-odb.c\n create mode 100644 external-odb.h\n create mode 100644 odb-helper.c\n create mode 100644 odb-helper.h\n create mode 100755 t/t0400-external-odb.sh\n\ndiff --git a/Makefile b/Makefile\nindex de5a030..8641318 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -726,6 +726,7 @@ LIB_OBJS += ewah/ewah_bitmap.o\n LIB_OBJS += ewah/ewah_io.o\n LIB_OBJS += ewah/ewah_rlw.o\n LIB_OBJS += exec_cmd.o\n+LIB_OBJS += external-odb.o\n LIB_OBJS += fetch-pack.o\n LIB_OBJS += fsck.o\n LIB_OBJS += gettext.o\n@@ -757,6 +758,7 @@ LIB_OBJS += notes-cache.o\n LIB_OBJS += notes-merge.o\n LIB_OBJS += notes-utils.o\n LIB_OBJS += object.o\n+LIB_OBJS += odb-helper.o\n LIB_OBJS += pack-bitmap.o\n LIB_OBJS += pack-bitmap-write.o\n LIB_OBJS += pack-check.o\ndiff --git a/cache.h b/cache.h\nindex 6049f86..cc0a934 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -864,6 +864,12 @@ const char *git_path_shallow(void);\n  */\n extern const char *sha1_file_name(const unsigned char *sha1);\n \n+/*\n+ * Like sha1_file_name, but return the filename within a specific alternate\n+ * object directory. Shares the same static buffer with sha1_file_name.\n+ */\n+extern const char *sha1_file_name_alt(const char *objdir, const unsigned char *sha1);\n+\n /*\n  * Return the name of the (local) packfile with the specified sha1 in\n  * its name.  The return value is a pointer to memory that is\n@@ -1093,6 +1099,8 @@ extern int do_check_packed_object_crc;\n \n extern int check_sha1_signature(const unsigned char *sha1, void *buf, unsigned long size, const char *type);\n \n+extern int create_object_tmpfile(struct strbuf *tmp, const char *filename);\n+extern void close_sha1_file(int fd);\n extern int finalize_object_file(const char *tmpfile, const char *filename);\n \n extern int has_sha1_pack(const unsigned char *sha1);\n@@ -1338,6 +1346,7 @@ extern void read_info_alternates(const char * relative_base, int depth);\n extern void add_to_alternates_file(const char *reference);\n typedef int alt_odb_fn(struct alternate_object_database *, void *);\n extern int foreach_alt_odb(alt_odb_fn, void*);\n+extern void prepare_external_alt_odb(void);\n \n struct pack_window {\n \tstruct pack_window *next;\ndiff --git a/external-odb.c b/external-odb.c\nnew file mode 100644\nindex 0000000..1ccfa99\n--- /dev/null\n+++ b/external-odb.c\n@@ -0,0 +1,115 @@\n+#include \"cache.h\"\n+#include \"external-odb.h\"\n+#include \"odb-helper.h\"\n+\n+static struct odb_helper *helpers;\n+static struct odb_helper **helpers_tail = &helpers;\n+\n+static struct odb_helper *find_or_create_helper(const char *name, int len)\n+{\n+\tstruct odb_helper *o;\n+\n+\tfor (o = helpers; o; o = o->next)\n+\t\tif (!strncmp(o->name, name, len) && !o->name[len])\n+\t\t\treturn o;\n+\n+\to = odb_helper_new(name, len);\n+\t*helpers_tail = o;\n+\thelpers_tail = &o->next;\n+\n+\treturn o;\n+}\n+\n+static int external_odb_config(const char *var, const char *value, void *data)\n+{\n+\tstruct odb_helper *o;\n+\tconst char *key, *dot;\n+\n+\tif (!skip_prefix(var, \"odb.\", &key))\n+\t\treturn 0;\n+\tdot = strrchr(key, '.');\n+\tif (!dot)\n+\t\treturn 0;\n+\n+\to = find_or_create_helper(key, dot - key);\n+\tkey = dot + 1;\n+\n+\tif (!strcmp(key, \"command\"))\n+\t\treturn git_config_string(&o->cmd, var, value);\n+\n+\treturn 0;\n+}\n+\n+static void external_odb_init(void)\n+{\n+\tstatic int initialized;\n+\n+\tif (initialized)\n+\t\treturn;\n+\tinitialized = 1;\n+\n+\tgit_config(external_odb_config, NULL);\n+}\n+\n+const char *external_odb_root(void)\n+{\n+\tstatic const char *root;\n+\tif (!root)\n+\t\troot = git_pathdup(\"objects/external\");\n+\treturn root;\n+}\n+\n+int external_odb_has_object(const unsigned char *sha1)\n+{\n+\tstruct odb_helper *o;\n+\n+\texternal_odb_init();\n+\n+\tfor (o = helpers; o; o = o->next)\n+\t\tif (odb_helper_has_object(o, sha1))\n+\t\t\treturn 1;\n+\treturn 0;\n+}\n+\n+int external_odb_fetch_object(const unsigned char *sha1)\n+{\n+\tstruct odb_helper *o;\n+\tconst char *path;\n+\n+\tif (!external_odb_has_object(sha1))\n+\t\treturn -1;\n+\n+\tpath = sha1_file_name_alt(external_odb_root(), sha1);\n+\tsafe_create_leading_directories_const(path);\n+\tprepare_external_alt_odb();\n+\n+\tfor (o = helpers; o; o = o->next) {\n+\t\tstruct strbuf tmpfile = STRBUF_INIT;\n+\t\tint ret;\n+\t\tint fd;\n+\n+\t\tif (!odb_helper_has_object(o, sha1))\n+\t\t\tcontinue;\n+\n+\t\tfd = create_object_tmpfile(&tmpfile, path);\n+\t\tif (fd < 0) {\n+\t\t\tstrbuf_release(&tmpfile);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tif (odb_helper_fetch_object(o, sha1, fd) < 0) {\n+\t\t\tclose(fd);\n+\t\t\tunlink(tmpfile.buf);\n+\t\t\tstrbuf_release(&tmpfile);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tclose_sha1_file(fd);\n+\t\tret = finalize_object_file(tmpfile.buf, path);\n+\t\tstrbuf_release(&tmpfile);\n+\t\tif (!ret)\n+\t\t\treturn 0;\n+\t}\n+\n+\treturn -1;\n+}\ndiff --git a/external-odb.h b/external-odb.h\nnew file mode 100644\nindex 0000000..2397477\n--- /dev/null\n+++ b/external-odb.h\n@@ -0,0 +1,8 @@\n+#ifndef EXTERNAL_ODB_H\n+#define EXTERNAL_ODB_H\n+\n+const char *external_odb_root(void);\n+int external_odb_has_object(const unsigned char *sha1);\n+int external_odb_fetch_object(const unsigned char *sha1);\n+\n+#endif /* EXTERNAL_ODB_H */\ndiff --git a/odb-helper.c b/odb-helper.c\nnew file mode 100644\nindex 0000000..7029a59\n--- /dev/null\n+++ b/odb-helper.c\n@@ -0,0 +1,239 @@\n+#include \"cache.h\"\n+#include \"object.h\"\n+#include \"argv-array.h\"\n+#include \"odb-helper.h\"\n+#include \"run-command.h\"\n+#include \"sha1-lookup.h\"\n+\n+struct odb_helper *odb_helper_new(const char *name, int namelen)\n+{\n+\tstruct odb_helper *o;\n+\n+\to = xcalloc(1, sizeof(*o));\n+\to->name = xmemdupz(name, namelen);\n+\n+\treturn o;\n+}\n+\n+struct odb_helper_cmd {\n+\tstruct argv_array argv;\n+\tstruct child_process child;\n+};\n+\n+static void prepare_helper_command(struct argv_array *argv, const char *cmd,\n+\t\t\t\t   const char *fmt, va_list ap)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tstrbuf_addstr(&buf, cmd);\n+\tstrbuf_addch(&buf, ' ');\n+\tstrbuf_vaddf(&buf, fmt, ap);\n+\n+\targv_array_push(argv, buf.buf);\n+\tstrbuf_release(&buf);\n+}\n+\n+__attribute__((format (printf,3,4)))\n+static int odb_helper_start(struct odb_helper *o,\n+\t\t\t    struct odb_helper_cmd *cmd,\n+\t\t\t    const char *fmt, ...)\n+{\n+\tva_list ap;\n+\n+\tmemset(cmd, 0, sizeof(*cmd));\n+\targv_array_init(&cmd->argv);\n+\n+\tif (!o->cmd)\n+\t\treturn -1;\n+\n+\tva_start(ap, fmt);\n+\tprepare_helper_command(&cmd->argv, o->cmd, fmt, ap);\n+\tva_end(ap);\n+\n+\tcmd->child.argv = cmd->argv.argv;\n+\tcmd->child.use_shell = 1;\n+\tcmd->child.no_stdin = 1;\n+\tcmd->child.out = -1;\n+\n+\tif (start_command(&cmd->child) < 0) {\n+\t\targv_array_clear(&cmd->argv);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int odb_helper_finish(struct odb_helper *o,\n+\t\t\t     struct odb_helper_cmd *cmd)\n+{\n+\tint ret = finish_command(&cmd->child);\n+\targv_array_clear(&cmd->argv);\n+\tif (ret) {\n+\t\twarning(\"odb helper '%s' reported failure\", o->name);\n+\t\treturn -1;\n+\t}\n+\treturn 0;\n+}\n+\n+static int parse_object_line(struct odb_helper_object *o, const char *line)\n+{\n+\tchar *end;\n+\tif (get_sha1_hex(line, o->sha1) < 0)\n+\t\treturn -1;\n+\n+\tline += 40;\n+\tif (*line++ != ' ')\n+\t\treturn -1;\n+\n+\to->size = strtoul(line, &end, 10);\n+\tif (line == end || *end++ != ' ')\n+\t\treturn -1;\n+\n+\to->type = type_from_string(end);\n+\treturn 0;\n+}\n+\n+static int odb_helper_object_cmp(const void *va, const void *vb)\n+{\n+\tconst struct odb_helper_object *a = va, *b = vb;\n+\treturn hashcmp(a->sha1, b->sha1);\n+}\n+\n+static void odb_helper_load_have(struct odb_helper *o)\n+{\n+\tstruct odb_helper_cmd cmd;\n+\tFILE *fh;\n+\tstruct strbuf line = STRBUF_INIT;\n+\n+\tif (o->have_valid)\n+\t\treturn;\n+\to->have_valid = 1;\n+\n+\tif (odb_helper_start(o, &cmd, \"have\") < 0)\n+\t\treturn;\n+\n+\tfh = xfdopen(cmd.child.out, \"r\");\n+\twhile (strbuf_getline(&line, fh) != EOF) {\n+\t\tALLOC_GROW(o->have, o->have_nr+1, o->have_alloc);\n+\t\tif (parse_object_line(&o->have[o->have_nr], line.buf) < 0) {\n+\t\t\twarning(\"bad 'have' input from odb helper '%s': %s\",\n+\t\t\t\to->name, line.buf);\n+\t\t\tbreak;\n+\t\t}\n+\t\to->have_nr++;\n+\t}\n+\n+\tstrbuf_release(&line);\n+\tfclose(fh);\n+\todb_helper_finish(o, &cmd);\n+\n+\tqsort(o->have, o->have_nr, sizeof(*o->have), odb_helper_object_cmp);\n+}\n+\n+static struct odb_helper_object *odb_helper_lookup(struct odb_helper *o,\n+\t\t\t\t\t\t   const unsigned char *sha1)\n+{\n+\tint idx;\n+\n+\todb_helper_load_have(o);\n+\tidx = sha1_entry_pos(o->have, sizeof(*o->have), 0,\n+\t\t\t     0, o->have_nr, o->have_nr,\n+\t\t\t     sha1);\n+\tif (idx < 0)\n+\t\treturn NULL;\n+\treturn &o->have[idx];\n+}\n+\n+int odb_helper_has_object(struct odb_helper *o, const unsigned char *sha1)\n+{\n+\treturn !!odb_helper_lookup(o, sha1);\n+}\n+\n+int odb_helper_fetch_object(struct odb_helper *o, const unsigned char *sha1,\n+\t\t\t    int fd)\n+{\n+\tstruct odb_helper_object *obj;\n+\tstruct odb_helper_cmd cmd;\n+\tunsigned long total_got;\n+\tgit_zstream stream;\n+\tint zret;\n+\tgit_SHA_CTX hash;\n+\tunsigned char real_sha1[20];\n+\n+\tobj = odb_helper_lookup(o, sha1);\n+\tif (!obj)\n+\t\treturn -1;\n+\n+\tif (odb_helper_start(o, &cmd, \"get %s\", sha1_to_hex(sha1)) < 0)\n+\t\treturn -1;\n+\n+\tmemset(&stream, 0, sizeof(stream));\n+\tgit_inflate_init(&stream);\n+\tgit_SHA1_Init(&hash);\n+\ttotal_got = 0;\n+\n+\tfor (;;) {\n+\t\tunsigned char buf[4096];\n+\t\tint r;\n+\n+\t\tr = xread(cmd.child.out, buf, sizeof(buf));\n+\t\tif (r < 0) {\n+\t\t\terror(\"unable to read from odb helper '%s': %s\",\n+\t\t\t      o->name, strerror(errno));\n+\t\t\tclose(cmd.child.out);\n+\t\t\todb_helper_finish(o, &cmd);\n+\t\t\tgit_inflate_end(&stream);\n+\t\t\treturn -1;\n+\t\t}\n+\t\tif (r == 0)\n+\t\t\tbreak;\n+\n+\t\twrite_or_die(fd, buf, r);\n+\n+\t\tstream.next_in = buf;\n+\t\tstream.avail_in = r;\n+\t\tdo {\n+\t\t\tunsigned char inflated[4096];\n+\t\t\tunsigned long got;\n+\n+\t\t\tstream.next_out = inflated;\n+\t\t\tstream.avail_out = sizeof(inflated);\n+\t\t\tzret = git_inflate(&stream, Z_SYNC_FLUSH);\n+\t\t\tgot = sizeof(inflated) - stream.avail_out;\n+\n+\t\t\tgit_SHA1_Update(&hash, inflated, got);\n+\t\t\t/* skip header when counting size */\n+\t\t\tif (!total_got) {\n+\t\t\t\tconst unsigned char *p = memchr(inflated, '\\0', got);\n+\t\t\t\tif (p)\n+\t\t\t\t\tgot -= p - inflated + 1;\n+\t\t\t\telse\n+\t\t\t\t\tgot = 0;\n+\t\t\t}\n+\t\t\ttotal_got += got;\n+\t\t} while (stream.avail_in && zret == Z_OK);\n+\t}\n+\n+\tclose(cmd.child.out);\n+\tgit_inflate_end(&stream);\n+\tgit_SHA1_Final(real_sha1, &hash);\n+\tif (odb_helper_finish(o, &cmd))\n+\t\treturn -1;\n+\tif (zret != Z_STREAM_END) {\n+\t\twarning(\"bad zlib data from odb helper '%s' for %s\",\n+\t\t\to->name, sha1_to_hex(sha1));\n+\t\treturn -1;\n+\t}\n+\tif (total_got != obj->size) {\n+\t\twarning(\"size mismatch from odb helper '%s' for %s (%lu != %lu)\",\n+\t\t\to->name, sha1_to_hex(sha1), total_got, obj->size);\n+\t\treturn -1;\n+\t}\n+\tif (hashcmp(real_sha1, sha1)) {\n+\t\twarning(\"sha1 mismatch from odb helper '%s' for %s (got %s)\",\n+\t\t\to->name, sha1_to_hex(sha1), sha1_to_hex(real_sha1));\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/odb-helper.h b/odb-helper.h\nnew file mode 100644\nindex 0000000..0f704f9\n--- /dev/null\n+++ b/odb-helper.h\n@@ -0,0 +1,25 @@\n+#ifndef ODB_HELPER_H\n+#define ODB_HELPER_H\n+\n+struct odb_helper {\n+\tconst char *name;\n+\tconst char *cmd;\n+\n+\tstruct odb_helper_object {\n+\t\tunsigned char sha1[20];\n+\t\tunsigned long size;\n+\t\tenum object_type type;\n+\t} *have;\n+\tint have_nr;\n+\tint have_alloc;\n+\tint have_valid;\n+\n+\tstruct odb_helper *next;\n+};\n+\n+struct odb_helper *odb_helper_new(const char *name, int namelen);\n+int odb_helper_has_object(struct odb_helper *o, const unsigned char *sha1);\n+int odb_helper_fetch_object(struct odb_helper *o, const unsigned char *sha1,\n+\t\t\t    int fd);\n+\n+#endif /* ODB_HELPER_H */\ndiff --git a/sha1_file.c b/sha1_file.c\nindex d5e1121..a707bc1 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -23,6 +23,7 @@\n #include \"bulk-checkin.h\"\n #include \"streaming.h\"\n #include \"dir.h\"\n+#include \"external-odb.h\"\n \n #ifndef O_NOATIME\n #if defined(__linux__) && (defined(__i386__) || defined(__PPC__))\n@@ -183,13 +184,11 @@ static void fill_sha1_path(char *pathbuf, const unsigned char *sha1)\n \t}\n }\n \n-const char *sha1_file_name(const unsigned char *sha1)\n+const char *sha1_file_name_alt(const char *objdir, const unsigned char *sha1)\n {\n \tstatic char buf[PATH_MAX];\n-\tconst char *objdir;\n \tint len;\n \n-\tobjdir = get_object_directory();\n \tlen = strlen(objdir);\n \n \t/* '/' + sha1(2) + '/' + sha1(38) + '\\0' */\n@@ -203,6 +202,11 @@ const char *sha1_file_name(const unsigned char *sha1)\n \treturn buf;\n }\n \n+const char *sha1_file_name(const unsigned char *sha1)\n+{\n+\treturn sha1_file_name_alt(get_object_directory(), sha1);\n+}\n+\n /*\n  * Return the name of the pack or index file with the specified sha1\n  * in its filename.  *base and *name are scratch space that must be\n@@ -439,6 +443,21 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)\n \treturn r;\n }\n \n+void prepare_external_alt_odb(void)\n+{\n+\tstatic int linked_external;\n+\tconst char *path;\n+\n+\tif (linked_external)\n+\t\treturn;\n+\n+\tpath = external_odb_root();\n+\tif (!access(path, F_OK)) {\n+\t\tlink_alt_odb_entry(path, NULL, 0, \"\");\n+\t\tlinked_external = 1;\n+\t}\n+}\n+\n void prepare_alt_odb(void)\n {\n \tconst char *alt;\n@@ -453,6 +472,7 @@ void prepare_alt_odb(void)\n \tlink_alt_odb_entries(alt, strlen(alt), PATH_SEP, NULL, 0);\n \n \tread_info_alternates(get_object_directory(), 0);\n+\tprepare_external_alt_odb();\n }\n \n /* Returns 1 if we have successfully freshened the file, 0 otherwise. */\n@@ -493,7 +513,7 @@ static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)\n \t\tif (check_and_freshen_file(alt->base, freshen))\n \t\t\treturn 1;\n \t}\n-\treturn 0;\n+\treturn external_odb_has_object(sha1);\n }\n \n static int check_and_freshen(const unsigned char *sha1, int freshen)\n@@ -1523,19 +1543,15 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st)\n \treturn -1;\n }\n \n-static int open_sha1_file(const unsigned char *sha1)\n+static int open_sha1_file_alt(const unsigned char *sha1)\n {\n-\tint fd;\n \tstruct alternate_object_database *alt;\n-\tint most_interesting_errno;\n-\n-\tfd = git_open_noatime(sha1_file_name(sha1));\n-\tif (fd >= 0)\n-\t\treturn fd;\n-\tmost_interesting_errno = errno;\n+\tint most_interesting_errno = errno;\n \n \tprepare_alt_odb();\n \tfor (alt = alt_odb_list; alt; alt = alt->next) {\n+\t\tint fd;\n+\n \t\tfill_sha1_path(alt->name, sha1);\n \t\tfd = git_open_noatime(alt->base);\n \t\tif (fd >= 0)\n@@ -1547,6 +1563,24 @@ static int open_sha1_file(const unsigned char *sha1)\n \treturn -1;\n }\n \n+static int open_sha1_file(const unsigned char *sha1)\n+{\n+\tint fd;\n+\n+\tfd = git_open_noatime(sha1_file_name(sha1));\n+\tif (fd >= 0)\n+\t\treturn fd;\n+\n+\tfd = open_sha1_file_alt(sha1);\n+\tif (fd >= 0)\n+\t\treturn fd;\n+\n+\tif (!external_odb_fetch_object(sha1))\n+\t\tfd = open_sha1_file_alt(sha1);\n+\n+\treturn fd;\n+}\n+\n void *map_sha1_file(const unsigned char *sha1, unsigned long *size)\n {\n \tvoid *map;\n@@ -3011,7 +3045,7 @@ int hash_sha1_file(const void *buf, unsigned long len, const char *type,\n }\n \n /* Finalize a file on disk, and close it. */\n-static void close_sha1_file(int fd)\n+void close_sha1_file(int fd)\n {\n \tif (fsync_object_files)\n \t\tfsync_or_die(fd, \"sha1 file\");\n@@ -3035,7 +3069,7 @@ static inline int directory_size(const char *filename)\n  * We want to avoid cross-directory filename renames, because those\n  * can have problems on various filesystems (FAT, NFS, Coda).\n  */\n-static int create_tmpfile(struct strbuf *tmp, const char *filename)\n+int create_object_tmpfile(struct strbuf *tmp, const char *filename)\n {\n \tint fd, dirlen = directory_size(filename);\n \n@@ -3075,7 +3109,7 @@ static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,\n \tstatic struct strbuf tmp_file = STRBUF_INIT;\n \tconst char *filename = sha1_file_name(sha1);\n \n-\tfd = create_tmpfile(&tmp_file, filename);\n+\tfd = create_object_tmpfile(&tmp_file, filename);\n \tif (fd < 0) {\n \t\tif (errno == EACCES)\n \t\t\treturn error(\"insufficient permission for adding an object to repository database %s\", get_object_directory());\ndiff --git a/t/t0400-external-odb.sh b/t/t0400-external-odb.sh\nnew file mode 100755\nindex 0000000..2b01617\n--- /dev/null\n+++ b/t/t0400-external-odb.sh\n@@ -0,0 +1,48 @@\n+#!/bin/sh\n+\n+test_description='basic tests for external object databases'\n+\n+. ./test-lib.sh\n+\n+ALT_SOURCE=\"$PWD/alt-repo/.git\"\n+export ALT_SOURCE\n+write_script odb-helper <<\\EOF\n+GIT_DIR=$ALT_SOURCE; export GIT_DIR\n+case \"$1\" in\n+have)\n+\tgit rev-list --all --objects |\n+\tcut -d' ' -f1 |\n+\tgit cat-file --batch-check |\n+\tawk '{print $1 \" \" $3 \" \" $2}'\n+\t;;\n+get)\n+\tcat \"$GIT_DIR\"/objects/$(echo $2 | sed 's#..#&/#')\n+\t;;\n+esac\n+EOF\n+HELPER=\"\\\"$PWD\\\"/odb-helper\"\n+\n+test_expect_success 'setup alternate repo' '\n+\tgit init alt-repo &&\n+\t(cd alt-repo &&\n+\t test_commit one &&\n+\t test_commit two\n+\t) &&\n+\talt_head=`cd alt-repo && git rev-parse HEAD`\n+'\n+\n+test_expect_success 'alt objects are missing' '\n+\ttest_must_fail git log --format=%s $alt_head\n+'\n+\n+test_expect_success 'helper can retrieve alt objects' '\n+\ttest_config odb.magic.command \"$HELPER\" &&\n+\tcat >expect <<-\\EOF &&\n+\ttwo\n+\tone\n+\tEOF\n+\tgit log --format=%s $alt_head >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n-- \n2.9.0.rc2.362.g3cd93d0\n"},{"id":"289089","messageId":"20160613085546.11784-7-chriscool@tuxfamily.org","threadId":"42589","inReplyTo":"20160613085546.11784-1-chriscool@tuxfamily.org","subject":"[RFC/PATCH 6/8] external odb: add write support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-06-13T08:55:44Z","receivedAt":"2016-06-16T02:19:53Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n external-odb.c | 15 +++++++++++++++\n external-odb.h |  2 ++\n odb-helper.c   | 41 +++++++++++++++++++++++++++++++++++++----\n odb-helper.h   |  3 +++\n sha1_file.c    |  2 ++\n 5 files changed, 59 insertions(+), 4 deletions(-)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex 42978a3..bb70fe3 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -127,3 +127,18 @@ int external_odb_for_each_object(each_external_object_fn fn, void *data)\n \t}\n \treturn 0;\n }\n+\n+int external_odb_write_object(const void *buf, unsigned long len,\n+\t\t\t      const char *type, unsigned char *sha1)\n+{\n+\tstruct odb_helper *o;\n+\n+\texternal_odb_init();\n+\n+\tfor (o = helpers; o; o = o->next) {\n+\t\tint r = odb_helper_write_object(o, buf, len, type, sha1);\n+\t\tif (r <= 0)\n+\t\t\treturn r;\n+\t}\n+\treturn 1;\n+}\ndiff --git a/external-odb.h b/external-odb.h\nindex cea8570..55d291d 100644\n--- a/external-odb.h\n+++ b/external-odb.h\n@@ -10,5 +10,7 @@ typedef int (*each_external_object_fn)(const unsigned char *sha1,\n \t\t\t\t       unsigned long size,\n \t\t\t\t       void *data);\n int external_odb_for_each_object(each_external_object_fn, void *);\n+int external_odb_write_object(const void *buf, unsigned long len,\n+\t\t\t      const char *type, unsigned char *sha1);\n \n #endif /* EXTERNAL_ODB_H */\ndiff --git a/odb-helper.c b/odb-helper.c\nindex 045cf6f..677a5e7 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -33,9 +33,10 @@ static void prepare_helper_command(struct argv_array *argv, const char *cmd,\n \tstrbuf_release(&buf);\n }\n \n-__attribute__((format (printf,3,4)))\n+__attribute__((format (printf,4,5)))\n static int odb_helper_start(struct odb_helper *o,\n \t\t\t    struct odb_helper_cmd *cmd,\n+\t\t\t    int use_stdin,\n \t\t\t    const char *fmt, ...)\n {\n \tva_list ap;\n@@ -52,7 +53,10 @@ static int odb_helper_start(struct odb_helper *o,\n \n \tcmd->child.argv = cmd->argv.argv;\n \tcmd->child.use_shell = 1;\n-\tcmd->child.no_stdin = 1;\n+\tif (use_stdin)\n+\t\tcmd->child.in = -1;\n+\telse\n+\t\tcmd->child.no_stdin = 1;\n \tcmd->child.out = -1;\n \n \tif (start_command(&cmd->child) < 0) {\n@@ -109,7 +113,7 @@ static void odb_helper_load_have(struct odb_helper *o)\n \t\treturn;\n \to->have_valid = 1;\n \n-\tif (odb_helper_start(o, &cmd, \"have\") < 0)\n+\tif (odb_helper_start(o, &cmd, 0, \"have\") < 0)\n \t\treturn;\n \n \tfh = xfdopen(cmd.child.out, \"r\");\n@@ -164,7 +168,7 @@ int odb_helper_fetch_object(struct odb_helper *o, const unsigned char *sha1,\n \tif (!obj)\n \t\treturn -1;\n \n-\tif (odb_helper_start(o, &cmd, \"get %s\", sha1_to_hex(sha1)) < 0)\n+\tif (odb_helper_start(o, &cmd, 0, \"get %s\", sha1_to_hex(sha1)) < 0)\n \t\treturn -1;\n \n \tmemset(&stream, 0, sizeof(stream));\n@@ -252,3 +256,32 @@ int odb_helper_for_each_object(struct odb_helper *o,\n \n \treturn 0;\n }\n+\n+int odb_helper_write_object(struct odb_helper *o,\n+\t\t\t    const void *buf, unsigned long len,\n+\t\t\t    const char *type, unsigned char *sha1)\n+{\n+\tstruct odb_helper_cmd cmd;\n+\n+\tif (odb_helper_start(o, &cmd, 1, \"put %s %lu %s\",\n+\t\t\t     sha1_to_hex(sha1), len, type) < 0)\n+\t\treturn -1;\n+\n+\tdo {\n+\t\tint w = xwrite(cmd.child.in, buf, len);\n+\t\tif (w < 0) {\n+\t\t\terror(\"unable to write to odb helper '%s': %s\",\n+\t\t\t      o->name, strerror(errno));\n+\t\t\tclose(cmd.child.in);\n+\t\t\tclose(cmd.child.out);\n+\t\t\todb_helper_finish(o, &cmd);\n+\t\t\treturn -1;\n+\t\t}\n+\t\tlen -= w;\n+\t} while (len > 0);\n+\n+\tclose(cmd.child.in);\n+\tclose(cmd.child.out);\n+\todb_helper_finish(o, &cmd);\n+\treturn 0;\n+}\ndiff --git a/odb-helper.h b/odb-helper.h\nindex 8c3916d..af31cc2 100644\n--- a/odb-helper.h\n+++ b/odb-helper.h\n@@ -25,5 +25,8 @@ int odb_helper_fetch_object(struct odb_helper *o, const unsigned char *sha1,\n \t\t\t    int fd);\n int odb_helper_for_each_object(struct odb_helper *o,\n \t\t\t       each_external_object_fn, void *);\n+int odb_helper_write_object(struct odb_helper *o,\n+\t\t\t    const void *buf, unsigned long len,\n+\t\t\t    const char *type, unsigned char *sha1);\n \n #endif /* ODB_HELPER_H */\ndiff --git a/sha1_file.c b/sha1_file.c\nindex a707bc1..90f19de 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -3192,6 +3192,8 @@ int write_sha1_file(const void *buf, unsigned long len, const char *type, unsign\n \t * it out into .git/objects/??/?{38} file.\n \t */\n \twrite_sha1_file_prepare(buf, len, type, sha1, hdr, &hdrlen);\n+\tif (!external_odb_write_object(buf, len, type, sha1))\n+\t\treturn 0;\n \tif (freshen_packed_object(sha1) || freshen_loose_object(sha1))\n \t\treturn 0;\n \treturn write_loose_object(sha1, hdr, hdrlen, buf, len, 0);\n-- \n2.9.0.rc2.362.g3cd93d0\n"},{"id":"289090","messageId":"20160613085546.11784-8-chriscool@tuxfamily.org","threadId":"42589","inReplyTo":"20160613085546.11784-1-chriscool@tuxfamily.org","subject":"[RFC/PATCH 7/8] external-odb: accept only blobs for now","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-06-13T08:55:45Z","receivedAt":"2016-06-16T02:19:53Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n external-odb.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex bb70fe3..6dd7b25 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -133,6 +133,10 @@ int external_odb_write_object(const void *buf, unsigned long len,\n {\n \tstruct odb_helper *o;\n \n+\t/* For now accept only blobs */\n+\tif (strcmp(type, \"blob\"))\n+\t\treturn 1;\n+\n \texternal_odb_init();\n \n \tfor (o = helpers; o; o = o->next) {\n-- \n2.9.0.rc2.362.g3cd93d0\n"},{"id":"289091","messageId":"20160613085546.11784-4-chriscool@tuxfamily.org","threadId":"42589","inReplyTo":"20160613085546.11784-1-chriscool@tuxfamily.org","subject":"[RFC/PATCH 3/8] t0400: use --batch-all-objects to get all objects","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-06-13T08:55:41Z","receivedAt":"2016-06-16T02:19:53Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n t/t0400-external-odb.sh | 4 +---\n 1 file changed, 1 insertion(+), 3 deletions(-)\n\ndiff --git a/t/t0400-external-odb.sh b/t/t0400-external-odb.sh\nindex 2b01617..fe85413 100755\n--- a/t/t0400-external-odb.sh\n+++ b/t/t0400-external-odb.sh\n@@ -10,9 +10,7 @@ write_script odb-helper <<\\EOF\n GIT_DIR=$ALT_SOURCE; export GIT_DIR\n case \"$1\" in\n have)\n-\tgit rev-list --all --objects |\n-\tcut -d' ' -f1 |\n-\tgit cat-file --batch-check |\n+\tgit cat-file --batch-check --batch-all-objects |\n \tawk '{print $1 \" \" $3 \" \" $2}'\n \t;;\n get)\n-- \n2.9.0.rc2.362.g3cd93d0\n"},{"id":"289092","messageId":"20160613085546.11784-9-chriscool@tuxfamily.org","threadId":"42589","inReplyTo":"20160613085546.11784-1-chriscool@tuxfamily.org","subject":"[RFC/PATCH 8/8] t0400: add test for external odb write support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-06-13T08:55:46Z","receivedAt":"2016-06-16T02:19:53Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n t/t0400-external-odb.sh | 8 ++++++++\n 1 file changed, 8 insertions(+)\n\ndiff --git a/t/t0400-external-odb.sh b/t/t0400-external-odb.sh\nindex 6c6da5c..3c868ca 100755\n--- a/t/t0400-external-odb.sh\n+++ b/t/t0400-external-odb.sh\n@@ -66,4 +66,12 @@ test_expect_success 'helper can add objects to alt repo' '\n \ttest \"$size\" -eq \"$alt_size\"\n '\n \n+test_expect_success 'commit adds objects to alt repo' '\n+\ttest_config odb.magic.command \"$HELPER\" &&\n+\ttest_commit three &&\n+\thash3=$(git ls-tree HEAD | grep three.t | cut -f1 | cut -d\\  -f3) &&\n+\tcontent=$(cd alt-repo && git show \"$hash3\") &&\n+\ttest \"$content\" = \"three\"\n+'\n+\n test_done\n-- \n2.9.0.rc2.362.g3cd93d0\n"},{"id":"289093","messageId":"20160613085546.11784-6-chriscool@tuxfamily.org","threadId":"42589","inReplyTo":"20160613085546.11784-1-chriscool@tuxfamily.org","subject":"[RFC/PATCH 5/8] t0400: add test for 'put' command","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-06-13T08:55:43Z","receivedAt":"2016-06-16T02:19:53Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n t/t0400-external-odb.sh | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/t/t0400-external-odb.sh b/t/t0400-external-odb.sh\nindex 0f1bb97..6c6da5c 100755\n--- a/t/t0400-external-odb.sh\n+++ b/t/t0400-external-odb.sh\n@@ -57,4 +57,13 @@ test_expect_success 'helper can retrieve alt objects' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'helper can add objects to alt repo' '\n+\thash=$(echo \"Hello odb!\" | git hash-object -w -t blob --stdin) &&\n+\ttest -f .git/objects/$(echo $hash | sed \"s#..#&/#\") &&\n+\tsize=$(git cat-file -s \"$hash\") &&\n+\tgit cat-file blob \"$hash\" | ./odb-helper put \"$hash\" \"$size\" blob &&\n+\talt_size=$(cd alt-repo && git cat-file -s \"$hash\") &&\n+\ttest \"$size\" -eq \"$alt_size\"\n+'\n+\n test_done\n-- \n2.9.0.rc2.362.g3cd93d0\n"},{"id":"289098","messageId":"CACsJy8ARuqKX1z4zze6Xj47X14tJ1eApCWivDwwuQh2gqRBYMw@mail.gmail.com","threadId":"42589","inReplyTo":"20160613085546.11784-1-chriscool@tuxfamily.org","subject":"Re: [RFC/PATCH 0/8] Add initial experimental external ODB support","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-13T10:10:37Z","receivedAt":"2016-06-16T02:19:53Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Jun 13, 2016 at 3:55 PM, Christian Couder\n<christian.couder@gmail.com> wrote:\n> Future work\n> ~~~~~~~~~~~\n>\n> From the discussions it appear that using the bundle v3 mechanism to\n> tranfer external ODB data could work, but only if the server has access\n> to its own external ODB.\n>\n> Another possible mechanism to transfer external ODB data would be some\n> kind of replace refs. This would be slower but the mechanism for the\n> transfer already fully exists.\n>\n> So I think I am going to experiment with some kind of replace refs.\n\nOr we can go \"fault-in\" all the way.\n\n1) Extend rev-list to support large blob exclusion, e.g. blobs whose\nsize above a certain limit are excluded, or blobs in certain paths are\nexcluded (maybe except those at the tip)\n\n2) Add a new command, similar to \"shallow\", in the git protocol for\nthe client to tell the server that it does not want the server to send\ncertain large blobs (and the server probably should acknowledge with\n\"ok you can get them from this url...\")\n\n3) upload-pack passes the relevant rev-list blob filtering arguments\nto pack-objects, no big blobs are transferred\n\n4) External odb setting must be done (temporarily) before index-pack\nis run (as part of the clone/fetch process). index-pack will notice\nthat certain blobs are missing from the received pack, but it will\nalso notice that it already has them via standard has_sha1_file() (we\njust need to make sure it does not read blob content back, i think\nwe're ok here). I'm assuming here that EODB in fact contains _all_\nblobs, so no extra setup is required at the server side.\n\n5) If index-pack says \"ok\" then you can make this new external odb\npermanent, else remove it.\n\nSubsequent fetches need to send the same \"do not include these blobs\"\ninstructions, of course. Not sure how well it interacts with\npack-bitmap, but I guess when you use this, you probably already use\nshallow clone (which disables pack bitmap).\n\n> One interesting thing also would be to use the streaming api when\n> reading from or writing to the external ODB.\n\nIf EODB is about large blobs only, you probably want to stream them\ndirectly to a pack as you get them and not keep in loose object form.\nInfrastructure is already in place (I think it's used by git-add), but\nhooking it at sha1_file.c layer might be tricky. This is just cherry\non top, not strictly needed of course.\n-- \nDuy\n"}]}