{"thread":{"id":"42736","subject":"[RFC/PATCH v2 00/10] Add initial experimental external ODB support","startedAt":"2016-06-28T18:19:53Z","lastAt":"2016-06-30T10:04:01Z","messageCount":13,"participants":["Christian Couder","Eric Wong"],"isPatch":true,"patchVersion":2,"patchTotal":10},"messages":[{"id":"290408","messageId":"20160628181933.24620-1-chriscool@tuxfamily.org","threadId":"42736","inReplyTo":null,"subject":"[RFC/PATCH v2 00/10] Add initial experimental external ODB support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-06-28T18:19:23Z","receivedAt":"2016-06-28T18:19:53Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Goal\n~~~~\n\nGit can store its objects only in the form of loose objects in\nseparate files or packed objects in a pack file.\n\nTo be able to better handle some kind of objects, for example big\nblobs, it would be nice if Git could store its objects in other object\ndatabases (ODB).\n\nTo do that, this patch series makes it possible to register commands,\nusing \"odb..command\" config variables, to access external\nODBs where objects can be stored and retrieved.\n\nExternal ODBs should be able to tranfer information about the blobs\nthey store. This patch series show how this is possible using kind of\nreplace refs.\n\nDesign\n~~~~~~\n\n* Registered command\n\nEach registered command manages access to one external ODB and will be\ncalled the following ways:\n\n - \" have\": the command should output the sha1, size and\ntype of all the objects the external ODB contains, one object per\nline.\n\n - \" get \": the command should then read from the\nexternal ODB the content of the object corresponding to and\noutput it on stdout.\n\n - \" put \": the command should then read\nfrom stdin an object and store it in the external ODB.\n\n* Transfer\n\nTo tranfer information about the blobs stored in external ODB, some\nspecial refs, called \"odb ref\", similar as replace refs, are used.\n\nFor now there should be one odb ref per blob. Each ref name should be\nrefs/odbs// where is the sha1 of the blob stored\nin the external odb named .\n\nThese odb refs should all point to a blob that should be stored in the\nGit repository and contain information about the blob stored in the\nexternal odb. This information can be specific to the external odb.\nThe repos can then share this information using commands like:\n\n`git fetch origin \"refs/odbs//*:refs/odbs//*\"`\n\n* Other\n\nThis RFC patch series for now does not address the following important\npart of a complete solution:\n\n - No real external ODB has been interfaced with Git. The tests use\nanother git repo in a separate directory for this purpose which is\nprobably useless in the real world.\n\nDesign discussion about performance\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nYeah, it is not efficient to fork/exec a command to just read or write\none object to or from the external ODB. Batch calls and/or using a\ndaemon and/or RPC should be used instead to be able to store regular\nobjects in an external ODB. But for now the external ODB would be all\nabout really big files, where the cost of a fork+exec should not\nmatter much. If we later want to extend usage of external ODBs, yeah\nwe will probably need to design other mechanisms.\n\nHere are some related explanations from Peff:\n\n{{{\nBecause this \"external odb\" essentially acts as a git alternate, we\nwould hit it only when we couldn't find an object through regular means.\nGit would then make the object available in the usual on-disk format\n(probably as a loose object).\n\nSo in most processes, we would not need to consult the odb command at\nall. And when we do, the first thing would be to get its \"have\" list,\nwhich would at most run once per process.\n\nSo the per-object cost is really calling \"get\", and my assumption there\nwas that the cost of actually retrieving the object over the network\nwould dwarf the fork/exec cost.\n\nI also waffled on having git cache the output of \" have\" in\nsome fast-lookup format to save even the single fork/exec. But I figured\nthat was something that could be added later if needed.\n\nYou'll note that this is sort of a \"fault-in\" model. Another model would\nbe to treat external odb updates similar to fetches. I.e., we touch the\nnetwork only during a special update operation, and then try to work\nlocally with whatever the external odb has. IMHO this policy could\nactually be up to the external odb itself (i.e., its \"have\" command\ncould serve from a local cache if it likes).\n}}}\n\nImplementation\n~~~~~~~~~~~~~~\n\n* Mechanism to call the registered commands\n\nThis series adds a set of function in external-odb.{c,h} that are\ncalled by the rest of Git to manage all the external ODBs.\n\nThese functions use 'struct odb_helper' and its associated functions\ndefined in odb-helper.{c,h} to talk to the different external ODBs by\nlaunching the configured \"odb..command\" commands and writing\nto or reading from them.\n\nThe tests in this series creates an odb-helper script that is\nregistered using the \"odb.magic.command\" config variable, and then\ncalled to read from and write to the external ODB.\n\n* ODB refs\n\nFor now odb ref management is only implemented in a registered command\nin t0410, but maybe this or some parts of it could be done by Git\nitself.\n\nWhen a new blob is added to an external odb, its sha1, size and type\nare writen in another new blob and the odb ref is created.\n\nWhen the list of existing blobs is requested from the external odb,\nthe content of the blobs pointed to by the odb refs can also be used\nby the odb to claim that it can get the objects.\n\nWhen a blob is actually requested from the external odb, it can use\nthe content stored in the blobs pointed to by the odb refs to get the\nactual blobs and then pass them.\n\nHighlevel view of the patches in the series\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n - Patches 01/10 and 02/10 are Peff's initial work. They are not\n changed since v1.\n\n - Patches 03/10 is an optimization in the odb-helper script that\n is used for testing. I will probably squash it into 01/08, but\n didn't yet. So there is no change since v1.\n\n - Patches 04/10 and 05/10 are adding \"put\" support in the\n odb-helper script and testing that. They are not changed since\n v1.\n\n - Patches 06/10 and 08/10 are enhancing external-odb.{c,h} and\n odb-helper.{c,h}, so that Git can write into an external\n ODB. They are not changed since v1.\n\n - Patch 07/10 limits write support to \"blobs\" for now to\n simplify things. It did not change since v1.\n\n - Patch 09/10 add a GIT_NO_EXTERNAL_ODB env variable to disable\n using the external database. It is new in this series.\n\n - Patch 10/10 add test t0410 that shows how odb refs can be used\n to transfer information about blobs managed by an external\n odb. It is new in this series.\n\nFuture work\n~~~~~~~~~~~\n\nI think that the odb refs don't prevent a regular fetch or push from\nwanting to send the objects that are managed by an external odb. So I\nam interested in suggestions about this problem. I will take a look at\nprevious discussions and how other mechanisms (shallow clone, bundle\nv3, ...) handle this.\n\nOne interesting thing also would be to use the streaming api when\nreading from or writing to the external ODB. (If it is not\nautomatically used already when the blob is bigger than\ncore.bigFileThreshold.)\n\nAnother interesting thing would be to use an http server as an\nexperimental external odb.\n\nPrevious work and discussions\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nPeff started to work on this and discuss this some years ago:\n\nhttp://thread.gmane.org/gmane.comp.version-control.git/206886/focus=207040\nhttp://thread.gmane.org/gmane.comp.version-control.git/247171\nhttp://thread.gmane.org/gmane.comp.version-control.git/202902/focus=203020\n\nHis work, which is not compile-tested any more, is still there:\n\nhttps://github.com/peff/git/commits/jk/external-odb-wip\n\nInitial discussions about this new series are there:\n\nhttp://thread.gmane.org/gmane.comp.version-control.git/288151/focus=295160\n\nVersion 1 of this RFC/PATCH series is here:\n\nhttp://thread.gmane.org/gmane.comp.version-control.git/297164\n\nLinks\n~~~~~\n\nThis patch series is available here:\n\nhttps://github.com/chriscool/git/commits/external-odb\n\nVersion 1 is here:\n\nhttps://github.com/chriscool/git/commits/gl-external-odb12\n\n\nChristian Couder (8):\n t0400: use --batch-all-objects to get all objects\n t0400: add 'put' command to odb-helper script\n t0400: add test for 'put' command\n external odb: add write support\n external-odb: accept only blobs for now\n t0400: add test for external odb write support\n Add GIT_NO_EXTERNAL_ODB env variable\n Add t0410 to test external ODB transfer\n\nJeff King (2):\n Add initial external odb support\n external odb foreach\n\n Makefile | 2 +\n cache.h | 18 +++\n environment.c | 4 +\n external-odb.c | 154 +++++++++++++++++++++++++\n external-odb.h | 16 +++\n odb-helper.c | 287 ++++++++++++++++++++++++++++++++++++++++++++++\n odb-helper.h | 32 ++++++\n sha1_file.c | 69 ++++++++---\n t/t0400-external-odb.sh | 77 +++++++++++++\n t/t0410-transfer-e-odb.sh | 136 ++++++++++++++++++++++\n 10 files changed, 780 insertions(+), 15 deletions(-)\n create mode 100644 external-odb.c\n create mode 100644 external-odb.h\n create mode 100644 odb-helper.c\n create mode 100644 odb-helper.h\n create mode 100755 t/t0400-external-odb.sh\n create mode 100755 t/t0410-transfer-e-odb.sh\n\n-- \n2.9.0.rc2.11.g990c140\n\n"},{"id":"290409","messageId":"20160628181933.24620-2-chriscool@tuxfamily.org","threadId":"42736","inReplyTo":"20160628181933.24620-1-chriscool@tuxfamily.org","subject":"[RFC/PATCH v2 01/10] Add initial external odb support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-06-28T18:19:24Z","receivedAt":"2016-06-28T18:20:00Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: Jeff King \n\nSigned-off-by: Christian Couder \n---\n Makefile | 2 +\n cache.h | 9 ++\n external-odb.c | 115 +++++++++++++++++++++++\n external-odb.h | 8 ++\n odb-helper.c | 239 ++++++++++++++++++++++++++++++++++++++++++++++++\n odb-helper.h | 25 +++++\n sha1_file.c | 64 ++++++++++---\n t/t0400-external-odb.sh | 48 ++++++++++\n 8 files changed, 495 insertions(+), 15 deletions(-)\n create mode 100644 external-odb.c\n create mode 100644 external-odb.h\n create mode 100644 odb-helper.c\n create mode 100644 odb-helper.h\n create mode 100755 t/t0400-external-odb.sh\n\ndiff --git a/Makefile b/Makefile\nindex de5a030..8641318 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -726,6 +726,7 @@ LIB_OBJS += ewah/ewah_bitmap.o\n LIB_OBJS += ewah/ewah_io.o\n LIB_OBJS += ewah/ewah_rlw.o\n LIB_OBJS += exec_cmd.o\n+LIB_OBJS += external-odb.o\n LIB_OBJS += fetch-pack.o\n LIB_OBJS += fsck.o\n LIB_OBJS += gettext.o\n@@ -757,6 +758,7 @@ LIB_OBJS += notes-cache.o\n LIB_OBJS += notes-merge.o\n LIB_OBJS += notes-utils.o\n LIB_OBJS += object.o\n+LIB_OBJS += odb-helper.o\n LIB_OBJS += pack-bitmap.o\n LIB_OBJS += pack-bitmap-write.o\n LIB_OBJS += pack-check.o\ndiff --git a/cache.h b/cache.h\nindex 6049f86..cc0a934 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -864,6 +864,12 @@ const char *git_path_shallow(void);\n */\n extern const char *sha1_file_name(const unsigned char *sha1);\n \n+/*\n+ * Like sha1_file_name, but return the filename within a specific alternate\n+ * object directory. Shares the same static buffer with sha1_file_name.\n+ */\n+extern const char *sha1_file_name_alt(const char *objdir, const unsigned char *sha1);\n+\n /*\n * Return the name of the (local) packfile with the specified sha1 in\n * its name. The return value is a pointer to memory that is\n@@ -1093,6 +1099,8 @@ extern int do_check_packed_object_crc;\n \n extern int check_sha1_signature(const unsigned char *sha1, void *buf, unsigned long size, const char *type);\n \n+extern int create_object_tmpfile(struct strbuf *tmp, const char *filename);\n+extern void close_sha1_file(int fd);\n extern int finalize_object_file(const char *tmpfile, const char *filename);\n \n extern int has_sha1_pack(const unsigned char *sha1);\n@@ -1338,6 +1346,7 @@ extern void read_info_alternates(const char * relative_base, int depth);\n extern void add_to_alternates_file(const char *reference);\n typedef int alt_odb_fn(struct alternate_object_database *, void *);\n extern int foreach_alt_odb(alt_odb_fn, void*);\n+extern void prepare_external_alt_odb(void);\n \n struct pack_window {\n \tstruct pack_window *next;\ndiff --git a/external-odb.c b/external-odb.c\nnew file mode 100644\nindex 0000000..1ccfa99\n--- /dev/null\n+++ b/external-odb.c\n@@ -0,0 +1,115 @@\n+#include \"cache.h\"\n+#include \"external-odb.h\"\n+#include \"odb-helper.h\"\n+\n+static struct odb_helper *helpers;\n+static struct odb_helper **helpers_tail = &helpers;\n+\n+static struct odb_helper *find_or_create_helper(const char *name, int len)\n+{\n+\tstruct odb_helper *o;\n+\n+\tfor (o = helpers; o; o = o->next)\n+\t\tif (!strncmp(o->name, name, len) && !o->name[len])\n+\t\t\treturn o;\n+\n+\to = odb_helper_new(name, len);\n+\t*helpers_tail = o;\n+\thelpers_tail = &o->next;\n+\n+\treturn o;\n+}\n+\n+static int external_odb_config(const char *var, const char *value, void *data)\n+{\n+\tstruct odb_helper *o;\n+\tconst char *key, *dot;\n+\n+\tif (!skip_prefix(var, \"odb.\", &key))\n+\t\treturn 0;\n+\tdot = strrchr(key, '.');\n+\tif (!dot)\n+\t\treturn 0;\n+\n+\to = find_or_create_helper(key, dot - key);\n+\tkey = dot + 1;\n+\n+\tif (!strcmp(key, \"command\"))\n+\t\treturn git_config_string(&o->cmd, var, value);\n+\n+\treturn 0;\n+}\n+\n+static void external_odb_init(void)\n+{\n+\tstatic int initialized;\n+\n+\tif (initialized)\n+\t\treturn;\n+\tinitialized = 1;\n+\n+\tgit_config(external_odb_config, NULL);\n+}\n+\n+const char *external_odb_root(void)\n+{\n+\tstatic const char *root;\n+\tif (!root)\n+\t\troot = git_pathdup(\"objects/external\");\n+\treturn root;\n+}\n+\n+int external_odb_has_object(const unsigned char *sha1)\n+{\n+\tstruct odb_helper *o;\n+\n+\texternal_odb_init();\n+\n+\tfor (o = helpers; o; o = o->next)\n+\t\tif (odb_helper_has_object(o, sha1))\n+\t\t\treturn 1;\n+\treturn 0;\n+}\n+\n+int external_odb_fetch_object(const unsigned char *sha1)\n+{\n+\tstruct odb_helper *o;\n+\tconst char *path;\n+\n+\tif (!external_odb_has_object(sha1))\n+\t\treturn -1;\n+\n+\tpath = sha1_file_name_alt(external_odb_root(), sha1);\n+\tsafe_create_leading_directories_const(path);\n+\tprepare_external_alt_odb();\n+\n+\tfor (o = helpers; o; o = o->next) {\n+\t\tstruct strbuf tmpfile = STRBUF_INIT;\n+\t\tint ret;\n+\t\tint fd;\n+\n+\t\tif (!odb_helper_has_object(o, sha1))\n+\t\t\tcontinue;\n+\n+\t\tfd = create_object_tmpfile(&tmpfile, path);\n+\t\tif (fd < 0) {\n+\t\t\tstrbuf_release(&tmpfile);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tif (odb_helper_fetch_object(o, sha1, fd) < 0) {\n+\t\t\tclose(fd);\n+\t\t\tunlink(tmpfile.buf);\n+\t\t\tstrbuf_release(&tmpfile);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tclose_sha1_file(fd);\n+\t\tret = finalize_object_file(tmpfile.buf, path);\n+\t\tstrbuf_release(&tmpfile);\n+\t\tif (!ret)\n+\t\t\treturn 0;\n+\t}\n+\n+\treturn -1;\n+}\ndiff --git a/external-odb.h b/external-odb.h\nnew file mode 100644\nindex 0000000..2397477\n--- /dev/null\n+++ b/external-odb.h\n@@ -0,0 +1,8 @@\n+#ifndef EXTERNAL_ODB_H\n+#define EXTERNAL_ODB_H\n+\n+const char *external_odb_root(void);\n+int external_odb_has_object(const unsigned char *sha1);\n+int external_odb_fetch_object(const unsigned char *sha1);\n+\n+#endif /* EXTERNAL_ODB_H */\ndiff --git a/odb-helper.c b/odb-helper.c\nnew file mode 100644\nindex 0000000..7029a59\n--- /dev/null\n+++ b/odb-helper.c\n@@ -0,0 +1,239 @@\n+#include \"cache.h\"\n+#include \"object.h\"\n+#include \"argv-array.h\"\n+#include \"odb-helper.h\"\n+#include \"run-command.h\"\n+#include \"sha1-lookup.h\"\n+\n+struct odb_helper *odb_helper_new(const char *name, int namelen)\n+{\n+\tstruct odb_helper *o;\n+\n+\to = xcalloc(1, sizeof(*o));\n+\to->name = xmemdupz(name, namelen);\n+\n+\treturn o;\n+}\n+\n+struct odb_helper_cmd {\n+\tstruct argv_array argv;\n+\tstruct child_process child;\n+};\n+\n+static void prepare_helper_command(struct argv_array *argv, const char *cmd,\n+\t\t\t\t const char *fmt, va_list ap)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tstrbuf_addstr(&buf, cmd);\n+\tstrbuf_addch(&buf, ' ');\n+\tstrbuf_vaddf(&buf, fmt, ap);\n+\n+\targv_array_push(argv, buf.buf);\n+\tstrbuf_release(&buf);\n+}\n+\n+__attribute__((format (printf,3,4)))\n+static int odb_helper_start(struct odb_helper *o,\n+\t\t\t struct odb_helper_cmd *cmd,\n+\t\t\t const char *fmt, ...)\n+{\n+\tva_list ap;\n+\n+\tmemset(cmd, 0, sizeof(*cmd));\n+\targv_array_init(&cmd->argv);\n+\n+\tif (!o->cmd)\n+\t\treturn -1;\n+\n+\tva_start(ap, fmt);\n+\tprepare_helper_command(&cmd->argv, o->cmd, fmt, ap);\n+\tva_end(ap);\n+\n+\tcmd->child.argv = cmd->argv.argv;\n+\tcmd->child.use_shell = 1;\n+\tcmd->child.no_stdin = 1;\n+\tcmd->child.out = -1;\n+\n+\tif (start_command(&cmd->child) < 0) {\n+\t\targv_array_clear(&cmd->argv);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int odb_helper_finish(struct odb_helper *o,\n+\t\t\t struct odb_helper_cmd *cmd)\n+{\n+\tint ret = finish_command(&cmd->child);\n+\targv_array_clear(&cmd->argv);\n+\tif (ret) {\n+\t\twarning(\"odb helper '%s' reported failure\", o->name);\n+\t\treturn -1;\n+\t}\n+\treturn 0;\n+}\n+\n+static int parse_object_line(struct odb_helper_object *o, const char *line)\n+{\n+\tchar *end;\n+\tif (get_sha1_hex(line, o->sha1) < 0)\n+\t\treturn -1;\n+\n+\tline += 40;\n+\tif (*line++ != ' ')\n+\t\treturn -1;\n+\n+\to->size = strtoul(line, &end, 10);\n+\tif (line == end || *end++ != ' ')\n+\t\treturn -1;\n+\n+\to->type = type_from_string(end);\n+\treturn 0;\n+}\n+\n+static int odb_helper_object_cmp(const void *va, const void *vb)\n+{\n+\tconst struct odb_helper_object *a = va, *b = vb;\n+\treturn hashcmp(a->sha1, b->sha1);\n+}\n+\n+static void odb_helper_load_have(struct odb_helper *o)\n+{\n+\tstruct odb_helper_cmd cmd;\n+\tFILE *fh;\n+\tstruct strbuf line = STRBUF_INIT;\n+\n+\tif (o->have_valid)\n+\t\treturn;\n+\to->have_valid = 1;\n+\n+\tif (odb_helper_start(o, &cmd, \"have\") < 0)\n+\t\treturn;\n+\n+\tfh = xfdopen(cmd.child.out, \"r\");\n+\twhile (strbuf_getline(&line, fh) != EOF) {\n+\t\tALLOC_GROW(o->have, o->have_nr+1, o->have_alloc);\n+\t\tif (parse_object_line(&o->have[o->have_nr], line.buf) < 0) {\n+\t\t\twarning(\"bad 'have' input from odb helper '%s': %s\",\n+\t\t\t\to->name, line.buf);\n+\t\t\tbreak;\n+\t\t}\n+\t\to->have_nr++;\n+\t}\n+\n+\tstrbuf_release(&line);\n+\tfclose(fh);\n+\todb_helper_finish(o, &cmd);\n+\n+\tqsort(o->have, o->have_nr, sizeof(*o->have), odb_helper_object_cmp);\n+}\n+\n+static struct odb_helper_object *odb_helper_lookup(struct odb_helper *o,\n+\t\t\t\t\t\t const unsigned char *sha1)\n+{\n+\tint idx;\n+\n+\todb_helper_load_have(o);\n+\tidx = sha1_entry_pos(o->have, sizeof(*o->have), 0,\n+\t\t\t 0, o->have_nr, o->have_nr,\n+\t\t\t sha1);\n+\tif (idx < 0)\n+\t\treturn NULL;\n+\treturn &o->have[idx];\n+}\n+\n+int odb_helper_has_object(struct odb_helper *o, const unsigned char *sha1)\n+{\n+\treturn !!odb_helper_lookup(o, sha1);\n+}\n+\n+int odb_helper_fetch_object(struct odb_helper *o, const unsigned char *sha1,\n+\t\t\t int fd)\n+{\n+\tstruct odb_helper_object *obj;\n+\tstruct odb_helper_cmd cmd;\n+\tunsigned long total_got;\n+\tgit_zstream stream;\n+\tint zret;\n+\tgit_SHA_CTX hash;\n+\tunsigned char real_sha1[20];\n+\n+\tobj = odb_helper_lookup(o, sha1);\n+\tif (!obj)\n+\t\treturn -1;\n+\n+\tif (odb_helper_start(o, &cmd, \"get %s\", sha1_to_hex(sha1)) < 0)\n+\t\treturn -1;\n+\n+\tmemset(&stream, 0, sizeof(stream));\n+\tgit_inflate_init(&stream);\n+\tgit_SHA1_Init(&hash);\n+\ttotal_got = 0;\n+\n+\tfor (;;) {\n+\t\tunsigned char buf[4096];\n+\t\tint r;\n+\n+\t\tr = xread(cmd.child.out, buf, sizeof(buf));\n+\t\tif (r < 0) {\n+\t\t\terror(\"unable to read from odb helper '%s': %s\",\n+\t\t\t o->name, strerror(errno));\n+\t\t\tclose(cmd.child.out);\n+\t\t\todb_helper_finish(o, &cmd);\n+\t\t\tgit_inflate_end(&stream);\n+\t\t\treturn -1;\n+\t\t}\n+\t\tif (r == 0)\n+\t\t\tbreak;\n+\n+\t\twrite_or_die(fd, buf, r);\n+\n+\t\tstream.next_in = buf;\n+\t\tstream.avail_in = r;\n+\t\tdo {\n+\t\t\tunsigned char inflated[4096];\n+\t\t\tunsigned long got;\n+\n+\t\t\tstream.next_out = inflated;\n+\t\t\tstream.avail_out = sizeof(inflated);\n+\t\t\tzret = git_inflate(&stream, Z_SYNC_FLUSH);\n+\t\t\tgot = sizeof(inflated) - stream.avail_out;\n+\n+\t\t\tgit_SHA1_Update(&hash, inflated, got);\n+\t\t\t/* skip header when counting size */\n+\t\t\tif (!total_got) {\n+\t\t\t\tconst unsigned char *p = memchr(inflated, '\\0', got);\n+\t\t\t\tif (p)\n+\t\t\t\t\tgot -= p - inflated + 1;\n+\t\t\t\telse\n+\t\t\t\t\tgot = 0;\n+\t\t\t}\n+\t\t\ttotal_got += got;\n+\t\t} while (stream.avail_in && zret == Z_OK);\n+\t}\n+\n+\tclose(cmd.child.out);\n+\tgit_inflate_end(&stream);\n+\tgit_SHA1_Final(real_sha1, &hash);\n+\tif (odb_helper_finish(o, &cmd))\n+\t\treturn -1;\n+\tif (zret != Z_STREAM_END) {\n+\t\twarning(\"bad zlib data from odb helper '%s' for %s\",\n+\t\t\to->name, sha1_to_hex(sha1));\n+\t\treturn -1;\n+\t}\n+\tif (total_got != obj->size) {\n+\t\twarning(\"size mismatch from odb helper '%s' for %s (%lu != %lu)\",\n+\t\t\to->name, sha1_to_hex(sha1), total_got, obj->size);\n+\t\treturn -1;\n+\t}\n+\tif (hashcmp(real_sha1, sha1)) {\n+\t\twarning(\"sha1 mismatch from odb helper '%s' for %s (got %s)\",\n+\t\t\to->name, sha1_to_hex(sha1), sha1_to_hex(real_sha1));\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/odb-helper.h b/odb-helper.h\nnew file mode 100644\nindex 0000000..0f704f9\n--- /dev/null\n+++ b/odb-helper.h\n@@ -0,0 +1,25 @@\n+#ifndef ODB_HELPER_H\n+#define ODB_HELPER_H\n+\n+struct odb_helper {\n+\tconst char *name;\n+\tconst char *cmd;\n+\n+\tstruct odb_helper_object {\n+\t\tunsigned char sha1[20];\n+\t\tunsigned long size;\n+\t\tenum object_type type;\n+\t} *have;\n+\tint have_nr;\n+\tint have_alloc;\n+\tint have_valid;\n+\n+\tstruct odb_helper *next;\n+};\n+\n+struct odb_helper *odb_helper_new(const char *name, int namelen);\n+int odb_helper_has_object(struct odb_helper *o, const unsigned char *sha1);\n+int odb_helper_fetch_object(struct odb_helper *o, const unsigned char *sha1,\n+\t\t\t int fd);\n+\n+#endif /* ODB_HELPER_H */\ndiff --git a/sha1_file.c b/sha1_file.c\nindex d5e1121..a707bc1 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -23,6 +23,7 @@\n #include \"bulk-checkin.h\"\n #include \"streaming.h\"\n #include \"dir.h\"\n+#include \"external-odb.h\"\n \n #ifndef O_NOATIME\n #if defined(__linux__) && (defined(__i386__) || defined(__PPC__))\n@@ -183,13 +184,11 @@ static void fill_sha1_path(char *pathbuf, const unsigned char *sha1)\n \t}\n }\n \n-const char *sha1_file_name(const unsigned char *sha1)\n+const char *sha1_file_name_alt(const char *objdir, const unsigned char *sha1)\n {\n \tstatic char buf[PATH_MAX];\n-\tconst char *objdir;\n \tint len;\n \n-\tobjdir = get_object_directory();\n \tlen = strlen(objdir);\n \n \t/* '/' + sha1(2) + '/' + sha1(38) + '\\0' */\n@@ -203,6 +202,11 @@ const char *sha1_file_name(const unsigned char *sha1)\n \treturn buf;\n }\n \n+const char *sha1_file_name(const unsigned char *sha1)\n+{\n+\treturn sha1_file_name_alt(get_object_directory(), sha1);\n+}\n+\n /*\n * Return the name of the pack or index file with the specified sha1\n * in its filename. *base and *name are scratch space that must be\n@@ -439,6 +443,21 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)\n \treturn r;\n }\n \n+void prepare_external_alt_odb(void)\n+{\n+\tstatic int linked_external;\n+\tconst char *path;\n+\n+\tif (linked_external)\n+\t\treturn;\n+\n+\tpath = external_odb_root();\n+\tif (!access(path, F_OK)) {\n+\t\tlink_alt_odb_entry(path, NULL, 0, \"\");\n+\t\tlinked_external = 1;\n+\t}\n+}\n+\n void prepare_alt_odb(void)\n {\n \tconst char *alt;\n@@ -453,6 +472,7 @@ void prepare_alt_odb(void)\n \tlink_alt_odb_entries(alt, strlen(alt), PATH_SEP, NULL, 0);\n \n \tread_info_alternates(get_object_directory(), 0);\n+\tprepare_external_alt_odb();\n }\n \n /* Returns 1 if we have successfully freshened the file, 0 otherwise. */\n@@ -493,7 +513,7 @@ static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)\n \t\tif (check_and_freshen_file(alt->base, freshen))\n \t\t\treturn 1;\n \t}\n-\treturn 0;\n+\treturn external_odb_has_object(sha1);\n }\n \n static int check_and_freshen(const unsigned char *sha1, int freshen)\n@@ -1523,19 +1543,15 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st)\n \treturn -1;\n }\n \n-static int open_sha1_file(const unsigned char *sha1)\n+static int open_sha1_file_alt(const unsigned char *sha1)\n {\n-\tint fd;\n \tstruct alternate_object_database *alt;\n-\tint most_interesting_errno;\n-\n-\tfd = git_open_noatime(sha1_file_name(sha1));\n-\tif (fd >= 0)\n-\t\treturn fd;\n-\tmost_interesting_errno = errno;\n+\tint most_interesting_errno = errno;\n \n \tprepare_alt_odb();\n \tfor (alt = alt_odb_list; alt; alt = alt->next) {\n+\t\tint fd;\n+\n \t\tfill_sha1_path(alt->name, sha1);\n \t\tfd = git_open_noatime(alt->base);\n \t\tif (fd >= 0)\n@@ -1547,6 +1563,24 @@ static int open_sha1_file(const unsigned char *sha1)\n \treturn -1;\n }\n \n+static int open_sha1_file(const unsigned char *sha1)\n+{\n+\tint fd;\n+\n+\tfd = git_open_noatime(sha1_file_name(sha1));\n+\tif (fd >= 0)\n+\t\treturn fd;\n+\n+\tfd = open_sha1_file_alt(sha1);\n+\tif (fd >= 0)\n+\t\treturn fd;\n+\n+\tif (!external_odb_fetch_object(sha1))\n+\t\tfd = open_sha1_file_alt(sha1);\n+\n+\treturn fd;\n+}\n+\n void *map_sha1_file(const unsigned char *sha1, unsigned long *size)\n {\n \tvoid *map;\n@@ -3011,7 +3045,7 @@ int hash_sha1_file(const void *buf, unsigned long len, const char *type,\n }\n \n /* Finalize a file on disk, and close it. */\n-static void close_sha1_file(int fd)\n+void close_sha1_file(int fd)\n {\n \tif (fsync_object_files)\n \t\tfsync_or_die(fd, \"sha1 file\");\n@@ -3035,7 +3069,7 @@ static inline int directory_size(const char *filename)\n * We want to avoid cross-directory filename renames, because those\n * can have problems on various filesystems (FAT, NFS, Coda).\n */\n-static int create_tmpfile(struct strbuf *tmp, const char *filename)\n+int create_object_tmpfile(struct strbuf *tmp, const char *filename)\n {\n \tint fd, dirlen = directory_size(filename);\n \n@@ -3075,7 +3109,7 @@ static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,\n \tstatic struct strbuf tmp_file = STRBUF_INIT;\n \tconst char *filename = sha1_file_name(sha1);\n \n-\tfd = create_tmpfile(&tmp_file, filename);\n+\tfd = create_object_tmpfile(&tmp_file, filename);\n \tif (fd < 0) {\n \t\tif (errno == EACCES)\n \t\t\treturn error(\"insufficient permission for adding an object to repository database %s\", get_object_directory());\ndiff --git a/t/t0400-external-odb.sh b/t/t0400-external-odb.sh\nnew file mode 100755\nindex 0000000..2b01617\n--- /dev/null\n+++ b/t/t0400-external-odb.sh\n@@ -0,0 +1,48 @@\n+#!/bin/sh\n+\n+test_description='basic tests for external object databases'\n+\n+. ./test-lib.sh\n+\n+ALT_SOURCE=\"$PWD/alt-repo/.git\"\n+export ALT_SOURCE\n+write_script odb-helper <<\\EOF\n+GIT_DIR=$ALT_SOURCE; export GIT_DIR\n+case \"$1\" in\n+have)\n+\tgit rev-list --all --objects |\n+\tcut -d' ' -f1 |\n+\tgit cat-file --batch-check |\n+\tawk '{print $1 \" \" $3 \" \" $2}'\n+\t;;\n+get)\n+\tcat \"$GIT_DIR\"/objects/$(echo $2 | sed 's#..#&/#')\n+\t;;\n+esac\n+EOF\n+HELPER=\"\\\"$PWD\\\"/odb-helper\"\n+\n+test_expect_success 'setup alternate repo' '\n+\tgit init alt-repo &&\n+\t(cd alt-repo &&\n+\t test_commit one &&\n+\t test_commit two\n+\t) &&\n+\talt_head=`cd alt-repo && git rev-parse HEAD`\n+'\n+\n+test_expect_success 'alt objects are missing' '\n+\ttest_must_fail git log --format=%s $alt_head\n+'\n+\n+test_expect_success 'helper can retrieve alt objects' '\n+\ttest_config odb.magic.command \"$HELPER\" &&\n+\tcat >expect <<-\\EOF &&\n+\ttwo\n+\tone\n+\tEOF\n+\tgit log --format=%s $alt_head >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n-- \n2.9.0.rc2.11.g990c140\n\n"},{"id":"290411","messageId":"20160628181933.24620-3-chriscool@tuxfamily.org","threadId":"42736","inReplyTo":"20160628181933.24620-1-chriscool@tuxfamily.org","subject":"[RFC/PATCH v2 02/10] external odb foreach","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-06-28T18:19:25Z","receivedAt":"2016-06-28T18:20:10Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: Jeff King \n\n---\n external-odb.c | 14 ++++++++++++++\n external-odb.h | 6 ++++++\n odb-helper.c | 15 +++++++++++++++\n odb-helper.h | 4 ++++\n 4 files changed, 39 insertions(+)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex 1ccfa99..42978a3 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -113,3 +113,17 @@ int external_odb_fetch_object(const unsigned char *sha1)\n \n \treturn -1;\n }\n+\n+int external_odb_for_each_object(each_external_object_fn fn, void *data)\n+{\n+\tstruct odb_helper *o;\n+\n+\texternal_odb_init();\n+\n+\tfor (o = helpers; o; o = o->next) {\n+\t\tint r = odb_helper_for_each_object(o, fn, data);\n+\t\tif (r)\n+\t\t\treturn r;\n+\t}\n+\treturn 0;\n+}\ndiff --git a/external-odb.h b/external-odb.h\nindex 2397477..cea8570 100644\n--- a/external-odb.h\n+++ b/external-odb.h\n@@ -5,4 +5,10 @@ const char *external_odb_root(void);\n int external_odb_has_object(const unsigned char *sha1);\n int external_odb_fetch_object(const unsigned char *sha1);\n \n+typedef int (*each_external_object_fn)(const unsigned char *sha1,\n+\t\t\t\t enum object_type type,\n+\t\t\t\t unsigned long size,\n+\t\t\t\t void *data);\n+int external_odb_for_each_object(each_external_object_fn, void *);\n+\n #endif /* EXTERNAL_ODB_H */\ndiff --git a/odb-helper.c b/odb-helper.c\nindex 7029a59..045cf6f 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -235,5 +235,20 @@ int odb_helper_fetch_object(struct odb_helper *o, const unsigned char *sha1,\n \t\treturn -1;\n \t}\n \n+\treturn 0;\n+}\n+\n+int odb_helper_for_each_object(struct odb_helper *o,\n+\t\t\t each_external_object_fn fn,\n+\t\t\t void *data)\n+{\n+\tint i;\n+\tfor (i = 0; i < o->have_nr; i++) {\n+\t\tstruct odb_helper_object *obj = &o->have[i];\n+\t\tint r = fn(obj->sha1, obj->type, obj->size, data);\n+\t\tif (r)\n+\t\t\treturn r;\n+\t}\n+\n \treturn 0;\n }\ndiff --git a/odb-helper.h b/odb-helper.h\nindex 0f704f9..8c3916d 100644\n--- a/odb-helper.h\n+++ b/odb-helper.h\n@@ -1,6 +1,8 @@\n #ifndef ODB_HELPER_H\n #define ODB_HELPER_H\n \n+#include \"external-odb.h\"\n+\n struct odb_helper {\n \tconst char *name;\n \tconst char *cmd;\n@@ -21,5 +23,7 @@ struct odb_helper *odb_helper_new(const char *name, int namelen);\n int odb_helper_has_object(struct odb_helper *o, const unsigned char *sha1);\n int odb_helper_fetch_object(struct odb_helper *o, const unsigned char *sha1,\n \t\t\t int fd);\n+int odb_helper_for_each_object(struct odb_helper *o,\n+\t\t\t each_external_object_fn, void *);\n \n #endif /* ODB_HELPER_H */\n-- \n2.9.0.rc2.11.g990c140\n\n"},{"id":"290412","messageId":"20160628181933.24620-6-chriscool@tuxfamily.org","threadId":"42736","inReplyTo":"20160628181933.24620-1-chriscool@tuxfamily.org","subject":"[RFC/PATCH v2 05/10] t0400: add test for 'put' command","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-06-28T18:19:28Z","receivedAt":"2016-06-28T18:20:13Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n t/t0400-external-odb.sh | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/t/t0400-external-odb.sh b/t/t0400-external-odb.sh\nindex 0f1bb97..6c6da5c 100755\n--- a/t/t0400-external-odb.sh\n+++ b/t/t0400-external-odb.sh\n@@ -57,4 +57,13 @@ test_expect_success 'helper can retrieve alt objects' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'helper can add objects to alt repo' '\n+\thash=$(echo \"Hello odb!\" | git hash-object -w -t blob --stdin) &&\n+\ttest -f .git/objects/$(echo $hash | sed \"s#..#&/#\") &&\n+\tsize=$(git cat-file -s \"$hash\") &&\n+\tgit cat-file blob \"$hash\" | ./odb-helper put \"$hash\" \"$size\" blob &&\n+\talt_size=$(cd alt-repo && git cat-file -s \"$hash\") &&\n+\ttest \"$size\" -eq \"$alt_size\"\n+'\n+\n test_done\n-- \n2.9.0.rc2.11.g990c140\n\n"},{"id":"290413","messageId":"20160628181933.24620-7-chriscool@tuxfamily.org","threadId":"42736","inReplyTo":"20160628181933.24620-1-chriscool@tuxfamily.org","subject":"[RFC/PATCH v2 06/10] external odb: add write support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-06-28T18:19:29Z","receivedAt":"2016-06-28T18:20:15Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n external-odb.c | 15 +++++++++++++++\n external-odb.h | 2 ++\n odb-helper.c | 41 +++++++++++++++++++++++++++++++++++++----\n odb-helper.h | 3 +++\n sha1_file.c | 2 ++\n 5 files changed, 59 insertions(+), 4 deletions(-)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex 42978a3..bb70fe3 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -127,3 +127,18 @@ int external_odb_for_each_object(each_external_object_fn fn, void *data)\n \t}\n \treturn 0;\n }\n+\n+int external_odb_write_object(const void *buf, unsigned long len,\n+\t\t\t const char *type, unsigned char *sha1)\n+{\n+\tstruct odb_helper *o;\n+\n+\texternal_odb_init();\n+\n+\tfor (o = helpers; o; o = o->next) {\n+\t\tint r = odb_helper_write_object(o, buf, len, type, sha1);\n+\t\tif (r <= 0)\n+\t\t\treturn r;\n+\t}\n+\treturn 1;\n+}\ndiff --git a/external-odb.h b/external-odb.h\nindex cea8570..55d291d 100644\n--- a/external-odb.h\n+++ b/external-odb.h\n@@ -10,5 +10,7 @@ typedef int (*each_external_object_fn)(const unsigned char *sha1,\n \t\t\t\t unsigned long size,\n \t\t\t\t void *data);\n int external_odb_for_each_object(each_external_object_fn, void *);\n+int external_odb_write_object(const void *buf, unsigned long len,\n+\t\t\t const char *type, unsigned char *sha1);\n \n #endif /* EXTERNAL_ODB_H */\ndiff --git a/odb-helper.c b/odb-helper.c\nindex 045cf6f..677a5e7 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -33,9 +33,10 @@ static void prepare_helper_command(struct argv_array *argv, const char *cmd,\n \tstrbuf_release(&buf);\n }\n \n-__attribute__((format (printf,3,4)))\n+__attribute__((format (printf,4,5)))\n static int odb_helper_start(struct odb_helper *o,\n \t\t\t struct odb_helper_cmd *cmd,\n+\t\t\t int use_stdin,\n \t\t\t const char *fmt, ...)\n {\n \tva_list ap;\n@@ -52,7 +53,10 @@ static int odb_helper_start(struct odb_helper *o,\n \n \tcmd->child.argv = cmd->argv.argv;\n \tcmd->child.use_shell = 1;\n-\tcmd->child.no_stdin = 1;\n+\tif (use_stdin)\n+\t\tcmd->child.in = -1;\n+\telse\n+\t\tcmd->child.no_stdin = 1;\n \tcmd->child.out = -1;\n \n \tif (start_command(&cmd->child) < 0) {\n@@ -109,7 +113,7 @@ static void odb_helper_load_have(struct odb_helper *o)\n \t\treturn;\n \to->have_valid = 1;\n \n-\tif (odb_helper_start(o, &cmd, \"have\") < 0)\n+\tif (odb_helper_start(o, &cmd, 0, \"have\") < 0)\n \t\treturn;\n \n \tfh = xfdopen(cmd.child.out, \"r\");\n@@ -164,7 +168,7 @@ int odb_helper_fetch_object(struct odb_helper *o, const unsigned char *sha1,\n \tif (!obj)\n \t\treturn -1;\n \n-\tif (odb_helper_start(o, &cmd, \"get %s\", sha1_to_hex(sha1)) < 0)\n+\tif (odb_helper_start(o, &cmd, 0, \"get %s\", sha1_to_hex(sha1)) < 0)\n \t\treturn -1;\n \n \tmemset(&stream, 0, sizeof(stream));\n@@ -252,3 +256,32 @@ int odb_helper_for_each_object(struct odb_helper *o,\n \n \treturn 0;\n }\n+\n+int odb_helper_write_object(struct odb_helper *o,\n+\t\t\t const void *buf, unsigned long len,\n+\t\t\t const char *type, unsigned char *sha1)\n+{\n+\tstruct odb_helper_cmd cmd;\n+\n+\tif (odb_helper_start(o, &cmd, 1, \"put %s %lu %s\",\n+\t\t\t sha1_to_hex(sha1), len, type) < 0)\n+\t\treturn -1;\n+\n+\tdo {\n+\t\tint w = xwrite(cmd.child.in, buf, len);\n+\t\tif (w < 0) {\n+\t\t\terror(\"unable to write to odb helper '%s': %s\",\n+\t\t\t o->name, strerror(errno));\n+\t\t\tclose(cmd.child.in);\n+\t\t\tclose(cmd.child.out);\n+\t\t\todb_helper_finish(o, &cmd);\n+\t\t\treturn -1;\n+\t\t}\n+\t\tlen -= w;\n+\t} while (len > 0);\n+\n+\tclose(cmd.child.in);\n+\tclose(cmd.child.out);\n+\todb_helper_finish(o, &cmd);\n+\treturn 0;\n+}\ndiff --git a/odb-helper.h b/odb-helper.h\nindex 8c3916d..af31cc2 100644\n--- a/odb-helper.h\n+++ b/odb-helper.h\n@@ -25,5 +25,8 @@ int odb_helper_fetch_object(struct odb_helper *o, const unsigned char *sha1,\n \t\t\t int fd);\n int odb_helper_for_each_object(struct odb_helper *o,\n \t\t\t each_external_object_fn, void *);\n+int odb_helper_write_object(struct odb_helper *o,\n+\t\t\t const void *buf, unsigned long len,\n+\t\t\t const char *type, unsigned char *sha1);\n \n #endif /* ODB_HELPER_H */\ndiff --git a/sha1_file.c b/sha1_file.c\nindex a707bc1..90f19de 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -3192,6 +3192,8 @@ int write_sha1_file(const void *buf, unsigned long len, const char *type, unsign\n \t * it out into .git/objects/??/?{38} file.\n \t */\n \twrite_sha1_file_prepare(buf, len, type, sha1, hdr, &hdrlen);\n+\tif (!external_odb_write_object(buf, len, type, sha1))\n+\t\treturn 0;\n \tif (freshen_packed_object(sha1) || freshen_loose_object(sha1))\n \t\treturn 0;\n \treturn write_loose_object(sha1, hdr, hdrlen, buf, len, 0);\n-- \n2.9.0.rc2.11.g990c140\n\n"},{"id":"290414","messageId":"20160628181933.24620-8-chriscool@tuxfamily.org","threadId":"42736","inReplyTo":"20160628181933.24620-1-chriscool@tuxfamily.org","subject":"[RFC/PATCH v2 07/10] external-odb: accept only blobs for now","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-06-28T18:19:30Z","receivedAt":"2016-06-28T18:20:18Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n external-odb.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex bb70fe3..6dd7b25 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -133,6 +133,10 @@ int external_odb_write_object(const void *buf, unsigned long len,\n {\n \tstruct odb_helper *o;\n \n+\t/* For now accept only blobs */\n+\tif (strcmp(type, \"blob\"))\n+\t\treturn 1;\n+\n \texternal_odb_init();\n \n \tfor (o = helpers; o; o = o->next) {\n-- \n2.9.0.rc2.11.g990c140\n\n"},{"id":"290415","messageId":"20160628181933.24620-4-chriscool@tuxfamily.org","threadId":"42736","inReplyTo":"20160628181933.24620-1-chriscool@tuxfamily.org","subject":"[RFC/PATCH v2 03/10] t0400: use --batch-all-objects to get all objects","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-06-28T18:19:26Z","receivedAt":"2016-06-28T18:20:20Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n t/t0400-external-odb.sh | 4 +---\n 1 file changed, 1 insertion(+), 3 deletions(-)\n\ndiff --git a/t/t0400-external-odb.sh b/t/t0400-external-odb.sh\nindex 2b01617..fe85413 100755\n--- a/t/t0400-external-odb.sh\n+++ b/t/t0400-external-odb.sh\n@@ -10,9 +10,7 @@ write_script odb-helper <<\\EOF\n GIT_DIR=$ALT_SOURCE; export GIT_DIR\n case \"$1\" in\n have)\n-\tgit rev-list --all --objects |\n-\tcut -d' ' -f1 |\n-\tgit cat-file --batch-check |\n+\tgit cat-file --batch-check --batch-all-objects |\n \tawk '{print $1 \" \" $3 \" \" $2}'\n \t;;\n get)\n-- \n2.9.0.rc2.11.g990c140\n\n"},{"id":"290416","messageId":"20160628181933.24620-9-chriscool@tuxfamily.org","threadId":"42736","inReplyTo":"20160628181933.24620-1-chriscool@tuxfamily.org","subject":"[RFC/PATCH v2 08/10] t0400: add test for external odb write support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-06-28T18:19:31Z","receivedAt":"2016-06-28T18:20:23Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n t/t0400-external-odb.sh | 8 ++++++++\n 1 file changed, 8 insertions(+)\n\ndiff --git a/t/t0400-external-odb.sh b/t/t0400-external-odb.sh\nindex 6c6da5c..3c868ca 100755\n--- a/t/t0400-external-odb.sh\n+++ b/t/t0400-external-odb.sh\n@@ -66,4 +66,12 @@ test_expect_success 'helper can add objects to alt repo' '\n \ttest \"$size\" -eq \"$alt_size\"\n '\n \n+test_expect_success 'commit adds objects to alt repo' '\n+\ttest_config odb.magic.command \"$HELPER\" &&\n+\ttest_commit three &&\n+\thash3=$(git ls-tree HEAD | grep three.t | cut -f1 | cut -d\\ -f3) &&\n+\tcontent=$(cd alt-repo && git show \"$hash3\") &&\n+\ttest \"$content\" = \"three\"\n+'\n+\n test_done\n-- \n2.9.0.rc2.11.g990c140\n\n"},{"id":"290417","messageId":"20160628181933.24620-11-chriscool@tuxfamily.org","threadId":"42736","inReplyTo":"20160628181933.24620-1-chriscool@tuxfamily.org","subject":"[RFC/PATCH v2 10/10] Add t0410 to test external ODB transfer","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-06-28T18:19:33Z","receivedAt":"2016-06-28T18:20:26Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n t/t0410-transfer-e-odb.sh | 136 ++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 136 insertions(+)\n create mode 100755 t/t0410-transfer-e-odb.sh\n\ndiff --git a/t/t0410-transfer-e-odb.sh b/t/t0410-transfer-e-odb.sh\nnew file mode 100755\nindex 0000000..868b55d\n--- /dev/null\n+++ b/t/t0410-transfer-e-odb.sh\n@@ -0,0 +1,136 @@\n+#!/bin/sh\n+\n+test_description='basic tests for transfering external ODBs'\n+\n+. ./test-lib.sh\n+\n+ORIG_SOURCE=\"$PWD/.git\"\n+export ORIG_SOURCE\n+\n+ALT_SOURCE1=\"$PWD/alt-repo1/.git\"\n+export ALT_SOURCE1\n+write_script odb-helper1 <<\\EOF\n+die() {\n+\tprintf >&2 \"%s\\n\" \"$@\"\n+\texit 1\n+}\n+GIT_DIR=$ALT_SOURCE1; export GIT_DIR\n+case \"$1\" in\n+have)\n+\tgit cat-file --batch-check --batch-all-objects |\n+\tawk '{print $1 \" \" $3 \" \" $2}'\n+\t;;\n+get)\n+\tcat \"$GIT_DIR\"/objects/$(echo $2 | sed 's#..#&/#')\n+\t;;\n+put)\n+\tsha1=\"$2\"\n+\tsize=\"$3\"\n+\tkind=\"$4\"\n+\twriten=$(git hash-object -w -t \"$kind\" --stdin)\n+\ttest \"$writen\" = \"$sha1\" || die \"bad sha1 passed '$sha1' vs writen '$writen'\"\n+\tref_hash=$(echo \"$sha1 $size $kind\" | GIT_DIR=$ORIG_SOURCE GIT_NO_EXTERNAL_ODB=1 git hash-object -w -t blob --stdin) || exit\n+\tGIT_DIR=$ORIG_SOURCE git update-ref refs/odbs/magic/\"$sha1\" \"$ref_hash\"\n+\t;;\n+*)\n+\tdie \"unknown command '$1'\"\n+\t;;\n+esac\n+EOF\n+HELPER1=\"\\\"$PWD\\\"/odb-helper1\"\n+\n+OTHER_SOURCE=\"$PWD/.git\"\n+export OTHER_SOURCE\n+\n+ALT_SOURCE2=\"$PWD/alt-repo2/.git\"\n+export ALT_SOURCE2\n+write_script odb-helper2 <<\\EOF\n+die() {\n+\tprintf >&2 \"%s\\n\" \"$@\"\n+\texit 1\n+}\n+GIT_DIR=$ALT_SOURCE2; export GIT_DIR\n+case \"$1\" in\n+have)\n+\tGIT_DIR=$OTHER_SOURCE git for-each-ref --format='%(objectname)' refs/odbs/magic/ | GIT_DIR=$OTHER_SOURCE xargs git show\n+\t;;\n+get)\n+\tOBJ_FILE=\"$GIT_DIR\"/objects/$(echo $2 | sed 's#..#&/#')\n+\tif ! test -f \"$OBJ_FILE\"\n+\tthen\n+\t\t# \"Download\" the missing object by copying it from alt-repo1\n+\t\tOBJ_DIR=$(echo $2 | sed 's/\\(..\\).*/\\1/')\n+\t\tOBJ_BASE=$(basename \"$OBJ_FILE\")\n+\t\tALT_OBJ_DIR1=\"$ALT_SOURCE1/objects/$OBJ_DIR\"\n+\t\tALT_OBJ_DIR2=\"$ALT_SOURCE2/objects/$OBJ_DIR\"\n+\t\tmkdir -p \"$ALT_OBJ_DIR2\" || die \"Could not mkdir '$ALT_OBJ_DIR2'\"\n+\t\tOBJ_SRC=\"$ALT_OBJ_DIR1/$OBJ_BASE\"\n+\t\tcp \"$OBJ_SRC\" \"$ALT_OBJ_DIR2\" ||\n+\t\tdie \"Could not cp '$OBJ_SRC' into '$ALT_OBJ_DIR2'\"\n+\tfi\n+\tcat \"$OBJ_FILE\" || die \"Could not cat '$OBJ_FILE'\"\n+\t;;\n+put)\n+\tsha1=\"$2\"\n+\tsize=\"$3\"\n+\tkind=\"$4\"\n+\twriten=$(git hash-object -w -t \"$kind\" --stdin)\n+\ttest \"$writen\" = \"$sha1\" || die \"bad sha1 passed '$sha1' vs writen '$writen'\"\n+\tref_hash=$(echo \"$sha1 $size $kind\" | GIT_DIR=$OTHER_SOURCE GIT_NO_EXTERNAL_ODB=1 git hash-object -w -t blob --stdin) || exit\n+\tGIT_DIR=$OTHER_SOURCE git update-ref refs/odbs/magic/\"$sha1\" \"$ref_hash\"\n+\t;;\n+*)\n+\tdie \"unknown command '$1'\"\n+\t;;\n+esac\n+EOF\n+HELPER2=\"\\\"$PWD\\\"/odb-helper2\"\n+\n+test_expect_success 'setup first alternate repo' '\n+\tgit init alt-repo1 &&\n+\ttest_commit zero &&\n+\tgit config odb.magic.command \"$HELPER1\"\n+'\n+\n+test_expect_success 'setup other repo and its alternate repo' '\n+\tgit init other-repo &&\n+\tgit init alt-repo2 &&\n+\t(cd other-repo &&\n+\t git remote add origin .. &&\n+\t git pull origin master &&\n+\t git checkout master &&\n+\t git log)\n+'\n+\n+test_expect_success 'new blobs are put in first object store' '\n+\ttest_commit one &&\n+\thash1=$(git ls-tree HEAD | grep one.t | cut -f1 | cut -d\\ -f3) &&\n+\tcontent=$(cd alt-repo1 && git show \"$hash1\") &&\n+\ttest \"$content\" = \"one\" &&\n+\ttest_commit two &&\n+\thash2=$(git ls-tree HEAD | grep two.t | cut -f1 | cut -d\\ -f3) &&\n+\tcontent=$(cd alt-repo1 && git show \"$hash2\") &&\n+\ttest \"$content\" = \"two\"\n+'\n+\n+test_expect_success 'other repo gets the blobs from object store' '\n+\t(cd other-repo &&\n+\t git fetch origin \"refs/odbs/magic/*:refs/odbs/magic/*\" &&\n+\t test_must_fail git cat-file blob \"$hash1\" &&\n+\t test_must_fail git cat-file blob \"$hash2\" &&\n+\t git config odb.magic.command \"$HELPER2\" &&\n+\t git cat-file blob \"$hash1\" &&\n+\t git cat-file blob \"$hash2\"\n+\t)\n+'\n+\n+test_expect_success 'other repo gets everything else' '\n+\t(cd other-repo &&\n+\t git fetch origin &&\n+\t content=$(git show \"$hash1\") &&\n+\t test \"$content\" = \"one\" &&\n+\t content=$(git show \"$hash2\") &&\n+\t test \"$content\" = \"two\")\n+'\n+\n+test_done\n-- \n2.9.0.rc2.11.g990c140\n\n"},{"id":"290418","messageId":"20160628181933.24620-5-chriscool@tuxfamily.org","threadId":"42736","inReplyTo":"20160628181933.24620-1-chriscool@tuxfamily.org","subject":"[RFC/PATCH v2 04/10] t0400: add 'put' command to odb-helper script","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-06-28T18:19:27Z","receivedAt":"2016-06-28T18:20:30Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n t/t0400-external-odb.sh | 14 ++++++++++++++\n 1 file changed, 14 insertions(+)\n\ndiff --git a/t/t0400-external-odb.sh b/t/t0400-external-odb.sh\nindex fe85413..0f1bb97 100755\n--- a/t/t0400-external-odb.sh\n+++ b/t/t0400-external-odb.sh\n@@ -7,6 +7,10 @@ test_description='basic tests for external object databases'\n ALT_SOURCE=\"$PWD/alt-repo/.git\"\n export ALT_SOURCE\n write_script odb-helper <<\\EOF\n+die() {\n+\tprintf >&2 \"%s\\n\" \"$@\"\n+\texit 1\n+}\n GIT_DIR=$ALT_SOURCE; export GIT_DIR\n case \"$1\" in\n have)\n@@ -16,6 +20,16 @@ have)\n get)\n \tcat \"$GIT_DIR\"/objects/$(echo $2 | sed 's#..#&/#')\n \t;;\n+put)\n+\tsha1=\"$2\"\n+\tsize=\"$3\"\n+\tkind=\"$4\"\n+\twriten=$(git hash-object -w -t \"$kind\" --stdin)\n+\ttest \"$writen\" = \"$sha1\" || die \"bad sha1 passed '$sha1' vs writen '$writen'\"\n+\t;;\n+*)\n+\tdie \"unknown command '$1'\"\n+\t;;\n esac\n EOF\n HELPER=\"\\\"$PWD\\\"/odb-helper\"\n-- \n2.9.0.rc2.11.g990c140\n\n"},{"id":"290419","messageId":"20160628181933.24620-10-chriscool@tuxfamily.org","threadId":"42736","inReplyTo":"20160628181933.24620-1-chriscool@tuxfamily.org","subject":"[RFC/PATCH v2 09/10] Add GIT_NO_EXTERNAL_ODB env variable","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-06-28T18:19:32Z","receivedAt":"2016-06-28T18:20:34Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n cache.h | 9 +++++++++\n environment.c | 4 ++++\n external-odb.c | 6 ++++++\n sha1_file.c | 3 +++\n 4 files changed, 22 insertions(+)\n\ndiff --git a/cache.h b/cache.h\nindex cc0a934..b0fe2bc 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -420,6 +420,7 @@ static inline enum object_type object_type(unsigned int mode)\n #define CEILING_DIRECTORIES_ENVIRONMENT \"GIT_CEILING_DIRECTORIES\"\n #define NO_REPLACE_OBJECTS_ENVIRONMENT \"GIT_NO_REPLACE_OBJECTS\"\n #define GIT_REPLACE_REF_BASE_ENVIRONMENT \"GIT_REPLACE_REF_BASE\"\n+#define NO_EXTERNAL_ODB_ENVIRONMENT \"GIT_NO_EXTERNAL_ODB\"\n #define GITATTRIBUTES_FILE \".gitattributes\"\n #define INFOATTRIBUTES_FILE \"info/attributes\"\n #define ATTRIBUTE_MACRO_PREFIX \"[attr]\"\n@@ -678,6 +679,14 @@ int get_shared_repository(void);\n extern int check_replace_refs;\n extern char *git_replace_ref_base;\n \n+/*\n+ * Do external odbs need to be used this run? This variable is\n+ * initialized to true unless $GIT_NO_EXTERNAL_ODB is set, but it\n+ * maybe set to false by some commands that do not want external\n+ * odbs to be active.\n+ */\n+extern int use_external_odb;\n+\n extern int fsync_object_files;\n extern int core_preload_index;\n extern int core_apply_sparse_checkout;\ndiff --git a/environment.c b/environment.c\nindex ca72464..1799339 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -48,6 +48,7 @@ const char *excludes_file;\n enum auto_crlf auto_crlf = AUTO_CRLF_FALSE;\n int check_replace_refs = 1;\n char *git_replace_ref_base;\n+int use_external_odb = 1;\n enum eol core_eol = EOL_UNSET;\n enum safe_crlf safe_crlf = SAFE_CRLF_WARN;\n unsigned whitespace_rule_cfg = WS_DEFAULT_RULE;\n@@ -119,6 +120,7 @@ const char * const local_repo_env[] = {\n \tINDEX_ENVIRONMENT,\n \tNO_REPLACE_OBJECTS_ENVIRONMENT,\n \tGIT_REPLACE_REF_BASE_ENVIRONMENT,\n+\tNO_EXTERNAL_ODB_ENVIRONMENT,\n \tGIT_PREFIX_ENVIRONMENT,\n \tGIT_SHALLOW_FILE_ENVIRONMENT,\n \tGIT_COMMON_DIR_ENVIRONMENT,\n@@ -183,6 +185,8 @@ static void setup_git_env(void)\n \treplace_ref_base = getenv(GIT_REPLACE_REF_BASE_ENVIRONMENT);\n \tgit_replace_ref_base = xstrdup(replace_ref_base ? replace_ref_base\n \t\t\t\t\t\t\t : \"refs/replace/\");\n+\tif (getenv(NO_EXTERNAL_ODB_ENVIRONMENT))\n+\t\tuse_external_odb = 0;\n \tnamespace = expand_namespace(getenv(GIT_NAMESPACE_ENVIRONMENT));\n \tnamespace_len = strlen(namespace);\n \tshallow_file = getenv(GIT_SHALLOW_FILE_ENVIRONMENT);\ndiff --git a/external-odb.c b/external-odb.c\nindex 6dd7b25..a980fbf 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -63,6 +63,9 @@ int external_odb_has_object(const unsigned char *sha1)\n {\n \tstruct odb_helper *o;\n \n+\tif (!use_external_odb)\n+\t\treturn 0;\n+\n \texternal_odb_init();\n \n \tfor (o = helpers; o; o = o->next)\n@@ -133,6 +136,9 @@ int external_odb_write_object(const void *buf, unsigned long len,\n {\n \tstruct odb_helper *o;\n \n+\tif (!use_external_odb)\n+\t\treturn 1;\n+\n \t/* For now accept only blobs */\n \tif (strcmp(type, \"blob\"))\n \t\treturn 1;\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 90f19de..13d4d75 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -448,6 +448,9 @@ void prepare_external_alt_odb(void)\n \tstatic int linked_external;\n \tconst char *path;\n \n+\tif (!use_external_odb)\n+\t\treturn;\n+\n \tif (linked_external)\n \t\treturn;\n \n-- \n2.9.0.rc2.11.g990c140\n\n"},{"id":"290526","messageId":"20160629200142.GA17878@dcvr.yhbt.net","threadId":"42736","inReplyTo":"20160628181933.24620-1-chriscool@tuxfamily.org","subject":"Re: [RFC/PATCH v2 00/10] Add initial experimental external ODB support","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2016-06-29T20:01:42Z","receivedAt":"2016-06-29T20:01:48Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Christian Couder wrote:\n> Design discussion about performance\n> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n> \n> Yeah, it is not efficient to fork/exec a command to just read or write\n> one object to or from the external ODB. Batch calls and/or using a\n> daemon and/or RPC should be used instead to be able to store regular\n> objects in an external ODB. But for now the external ODB would be all\n> about really big files, where the cost of a fork+exec should not\n> matter much. If we later want to extend usage of external ODBs, yeah\n> we will probably need to design other mechanisms.\n\nI would also investigate switching run_command to use vfork+exec\nor posix_spawn for performance (keeping in mind vfork\ncaveats documented at https://ewontfix.com/7/ )\n\nposix_spawn in future glibc (probably 2.24) will use CLONE_VFORK\nin all cases under Linux, and posix_spawn may help with\nportability, too. I think the only thing we can't support\nwith posix_spawn which run_command supports is chdir;\nall the redirections/closing FDs should be fine.\n\nWith only 10MB malloc-ed, the following shows vfork performance\nbeing noticeably faster than plain fork:\n\n/* gcc -o vfork-test -Wall -O2 vfork-test.c */\n#include \n#include \n#include \n#include \n#include \n\nint main(int argc, char *argv[], char *envp[])\n{\n\tint i;\n\tint do_vfork = argc > 1 && !strcmp(argv[1], \"vfork\");\n\tchar * const cmd[] = { \"/bin/true\", 0 };\n\tsize_t n = 1024 * 1024 * 10;\n\tchar *mem = malloc(n);\n\n\tmemset(mem, 'a', n); /* make sure it's really allocated */\n\n\tfor (i = 0; i < 10000; i++) {\n\t\tpid_t pid = do_vfork ? vfork() : fork();\n\n\t\tif (pid == 0) {\n\t\t\texecve(cmd[0], cmd, envp);\n\t\t\twrite(2, \"exec error\\n\", 11);\n\t\t\t_exit(1);\n\t\t}\n\t\twaitpid(pid, 0, 0);\n\t}\n\treturn 0;\n}\n"},{"id":"290585","messageId":"CAP8UFD03=fy2hXrZcT_7VP630CqAOHKz61T5GDKsT2f+v5Rnsg@mail.gmail.com","threadId":"42736","inReplyTo":"20160629200142.GA17878@dcvr.yhbt.net","subject":"Re: [RFC/PATCH v2 00/10] Add initial experimental external ODB support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-06-30T09:49:21Z","receivedAt":"2016-06-30T10:04:01Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Jun 29, 2016 at 10:01 PM, Eric Wong wrote:\n> Christian Couder wrote:\n>> Design discussion about performance\n>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n>>\n>> Yeah, it is not efficient to fork/exec a command to just read or write\n>> one object to or from the external ODB. Batch calls and/or using a\n>> daemon and/or RPC should be used instead to be able to store regular\n>> objects in an external ODB. But for now the external ODB would be all\n>> about really big files, where the cost of a fork+exec should not\n>> matter much. If we later want to extend usage of external ODBs, yeah\n>> we will probably need to design other mechanisms.\n>\n> I would also investigate switching run_command to use vfork+exec\n> or posix_spawn for performance (keeping in mind vfork\n> caveats documented at https://ewontfix.com/7/ )\n\nThanks Eric for this idea and the test!\n"}]}