{"thread":{"id":"46384","subject":"[RFC/PATCH v2 0/1] Add support for downloading blobs on demand","startedAt":"2017-07-14T13:27:39Z","lastAt":"2017-07-17T23:24:42Z","messageCount":6,"participants":["Ben Peart","Christian Couder","Jonathan Tan"],"isPatch":true,"patchVersion":2,"patchTotal":1},"messages":[{"id":"324501","messageId":"20170714132651.170708-1-benpeart@microsoft.com","threadId":"46384","inReplyTo":null,"subject":"[RFC/PATCH v2 0/1] Add support for downloading blobs on demand","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-07-14T13:26:50Z","receivedAt":"2017-07-14T13:27:39Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"This patch series enables git to request missing objects when they are\nnot found in the object store. This is a fault-in model where the\n\"read-object\" sub-process will fetch the missing object and store it in\nthe object store as a loose, alternate, or pack file. On success, git\nwill retry the operation and find the requested object.\n\nIt utilizes the recent sub-process refactoring to spawn a \"read-object\"\nhook as a sub-process on the first request and then all subsequent\nrequests are made to that existing sub-process. This significantly\nreduces the cost of making multiple request within a single git command.\n\nAn earlier version [1] of this patch series is pulled into [2] which\nenables registering multiple ODB handlers which can be a simple command\nor a sub-process. My hope is that this patch series can be completed to\nmeet the needs of the various efforts where faulting in missing objects\nis required.\n\nIn the meantime, now that the sub-process refactoring has made it to\nmaster, I�ve refactored this patch series to be as small and focused as\npossible so it can be easily incorporated in other patch series where it\nmakes sense.\n\n[3] has a need for faulting in missing objects and will be reworked to\ntake advantage of this patch series for its next iteration.\n\n[4] has a similar function that uses a registered command instead of a\nsub-process. Spawning a separate process for every missing object only\nworks if there are very few missing objects.  It does not scale well\nwhen there are many missing objects (especially small objects like\ncommits and trees).\n\nPatch is available here:\nhttps://github.com/benpeart/git-for-windows/commits/read-object-process\n\n\n[RFC] Add support for downloading blobs on demand \n[1] https://public-inbox.org/git/20170113155253.1644-1-benpeart@microsoft.com/\n\n[RFC/PATCH v4 00/49] Add initial experimental external ODB support \n[2] https://public-inbox.org/git/20170620075523.26961-1-chriscool@tuxfamily.org/\n\n[PATCH v2 00/19] WIP object filtering for partial clone\n[3] https://public-inbox.org/git/20170713173459.3559-1-git@jeffhostetler.com/\n\n[RFC PATCH 0/3] Partial clone: promised blobs (formerly \"missing blobs\")\n[4] https://public-inbox.org/git/cover.1499800530.git.jonathantanmy@google.com/\n\nBen Peart (1):\n  sha1_file: Add support for downloading blobs on demand\n\n Documentation/technical/read-object-protocol.txt | 102 ++++++++++++\n cache.h                                          |   1 +\n config.c                                         |   5 +\n contrib/long-running-read-object/example.pl      | 114 +++++++++++++\n environment.c                                    |   1 +\n sha1_file.c                                      | 193 ++++++++++++++++++++++-\n t/t0410-read-object.sh                           |  27 ++++\n t/t0410/read-object                              | 114 +++++++++++++\n 8 files changed, 550 insertions(+), 7 deletions(-)\n create mode 100644 Documentation/technical/read-object-protocol.txt\n create mode 100755 contrib/long-running-read-object/example.pl\n create mode 100755 t/t0410-read-object.sh\n create mode 100755 t/t0410/read-object\n\n-- \n2.13.2.windows.1\n\n"},{"id":"324502","messageId":"20170714132651.170708-2-benpeart@microsoft.com","threadId":"46384","inReplyTo":"20170714132651.170708-1-benpeart@microsoft.com","subject":"[PATCH v2 1/1] sha1_file: Add support for downloading blobs on demand","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-07-14T13:26:51Z","receivedAt":"2017-07-14T13:27:41Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"This patch series enables git to request missing objects when they are\nnot found in the object store. This is a fault-in model where the\n\"read-object\" sub-process will fetch the missing object and store it in\nthe object store as a loose, alternate, or pack file. On success, git\nwill retry the operation and find the requested object.\n\nIt utilizes the recent sub-process refactoring to spawn a \"read-object\"\nhook as a sub-process on the first request and then all subsequent\nrequests are made to that existing sub-process. This significantly\nreduces the cost of making multiple request within a single git command.\n\nSigned-off-by: Ben Peart <benpeart@microsoft.com>\n---\n Documentation/technical/read-object-protocol.txt | 102 ++++++++++++\n cache.h                                          |   1 +\n config.c                                         |   5 +\n contrib/long-running-read-object/example.pl      | 114 +++++++++++++\n environment.c                                    |   1 +\n sha1_file.c                                      | 193 ++++++++++++++++++++++-\n t/t0410-read-object.sh                           |  27 ++++\n t/t0410/read-object                              | 114 +++++++++++++\n 8 files changed, 550 insertions(+), 7 deletions(-)\n create mode 100644 Documentation/technical/read-object-protocol.txt\n create mode 100755 contrib/long-running-read-object/example.pl\n create mode 100755 t/t0410-read-object.sh\n create mode 100755 t/t0410/read-object\n\ndiff --git a/Documentation/technical/read-object-protocol.txt b/Documentation/technical/read-object-protocol.txt\nnew file mode 100644\nindex 0000000000..a893b46e7c\n--- /dev/null\n+++ b/Documentation/technical/read-object-protocol.txt\n@@ -0,0 +1,102 @@\n+Read Object Process\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+The read-object process enables Git to read all missing blobs with a\n+single process invocation for the entire life of a single Git command.\n+This is achieved by using a packet format (pkt-line, see technical/\n+protocol-common.txt) based protocol over standard input and standard\n+output as follows. All packets, except for the \"*CONTENT\" packets and\n+the \"0000\" flush packet, are considered text and therefore are\n+terminated by a LF.\n+\n+Git starts the process when it encounters the first missing object that\n+needs to be retrieved. After the process is started, Git sends a welcome\n+message (\"git-read-object-client\"), a list of supported protocol version\n+numbers, and a flush packet. Git expects to read a welcome response\n+message (\"git-read-object-server\"), exactly one protocol version number\n+from the previously sent list, and a flush packet. All further\n+communication will be based on the selected version.\n+\n+The remaining protocol description below documents \"version=1\". Please\n+note that \"version=42\" in the example below does not exist and is only\n+there to illustrate how the protocol would look with more than one\n+version.\n+\n+After the version negotiation Git sends a list of all capabilities that\n+it supports and a flush packet. Git expects to read a list of desired\n+capabilities, which must be a subset of the supported capabilities list,\n+and a flush packet as response:\n+------------------------\n+packet: git> git-read-object-client\n+packet: git> version=1\n+packet: git> version=42\n+packet: git> 0000\n+packet: git< git-read-object-server\n+packet: git< version=1\n+packet: git< 0000\n+packet: git> capability=get\n+packet: git> capability=have\n+packet: git> capability=put\n+packet: git> capability=not-yet-invented\n+packet: git> 0000\n+packet: git< capability=get\n+packet: git< 0000\n+------------------------\n+The only supported capability in version 1 is \"get\".\n+\n+Afterwards Git sends a list of \"key=value\" pairs terminated with a flush\n+packet. The list will contain at least the command (based on the\n+supported capabilities) and the sha1 of the object to retrieve. Please\n+note, that the process must not send any response before it received the\n+final flush packet.\n+\n+When the process receives the \"get\" command, it should make the requested\n+object available in the git object store and then return success. Git will\n+then check the object store again and this time find it and proceed.\n+------------------------\n+packet: git> command=get\n+packet: git> sha1=0a214a649e1b3d5011e14a3dc227753f2bd2be05\n+packet: git> 0000\n+------------------------\n+\n+The process is expected to respond with a list of \"key=value\" pairs\n+terminated with a flush packet. If the process does not experience\n+problems then the list must contain a \"success\" status.\n+------------------------\n+packet: git< status=success\n+packet: git< 0000\n+------------------------\n+\n+In case the process cannot or does not want to process the content, it\n+is expected to respond with an \"error\" status.\n+------------------------\n+packet: git< status=error\n+packet: git< 0000\n+------------------------\n+\n+In case the process cannot or does not want to process the content as\n+well as any future content for the lifetime of the Git process, then it\n+is expected to respond with an \"abort\" status at any point in the\n+protocol.\n+------------------------\n+packet: git< status=abort\n+packet: git< 0000\n+------------------------\n+\n+Git neither stops nor restarts the process in case the \"error\"/\"abort\"\n+status is set.\n+\n+If the process dies during the communication or does not adhere to the\n+protocol then Git will stop the process and restart it with the next\n+object that needs to be processed.\n+\n+After the read-object process has processed an object it is expected to\n+wait for the next \"key=value\" list containing a command. Git will close\n+the command pipe on exit. The process is expected to detect EOF and exit\n+gracefully on its own. Git will wait until the process has stopped.\n+\n+A long running read-object process demo implementation can be found in\n+`contrib/long-running-read-object/example.pl` located in the Git core\n+repository. If you develop your own long running process then the\n+`GIT_TRACE_PACKET` environment variables can be very helpful for\n+debugging (see linkgit:git[1]).\ndiff --git a/cache.h b/cache.h\nindex 71fe092644..914379724f 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -804,6 +804,7 @@ enum log_refs_config {\n \tLOG_REFS_ALWAYS\n };\n extern enum log_refs_config log_all_ref_updates;\n+extern int core_virtualize_objects;\n \n enum branch_track {\n \tBRANCH_TRACK_UNSPECIFIED = -1,\ndiff --git a/config.c b/config.c\nindex a9356c1383..cc6e8f3237 100644\n--- a/config.c\n+++ b/config.c\n@@ -1241,6 +1241,11 @@ static int git_default_core_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.virtualizeobjects\")) {\n+\t\tcore_virtualize_objects = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n \t/* Add other config variables here and to Documentation/config.txt. */\n \treturn 0;\n }\ndiff --git a/contrib/long-running-read-object/example.pl b/contrib/long-running-read-object/example.pl\nnew file mode 100755\nindex 0000000000..b8f37f836a\n--- /dev/null\n+++ b/contrib/long-running-read-object/example.pl\n@@ -0,0 +1,114 @@\n+#!/usr/bin/perl\n+#\n+# Example implementation for the Git read-object protocol version 1\n+# See Documentation/technical/read-object-protocol.txt\n+#\n+# Allows you to test the ability for blobs to be pulled from a host git repo\n+# \"on demand.\"  Called when git needs a blob it couldn't find locally due to\n+# a lazy clone that only cloned the commits and trees.\n+#\n+# A lazy clone can be simulated via the following commands from the host repo\n+# you wish to create a lazy clone of:\n+#\n+# cd /host_repo\n+# git rev-parse HEAD\n+# git init /guest_repo\n+# git cat-file --batch-check --batch-all-objects | grep -v 'blob' |\n+#\tcut -d' ' -f1 | git pack-objects /guest_repo/.git/objects/pack/noblobs\n+# cd /guest_repo\n+# git config core.virtualizeobjects true\n+# git reset --hard <sha from rev-parse call above>\n+#\n+# Please note, this sample is a minimal skeleton. No proper error handling\n+# was implemented.\n+#\n+\n+use strict;\n+use warnings;\n+\n+#\n+# Point $DIR to the folder where your host git repo is located so we can pull\n+# missing objects from it\n+#\n+my $DIR = \"/host_repo/.git/\";\n+\n+sub packet_bin_read {\n+\tmy $buffer;\n+\tmy $bytes_read = read STDIN, $buffer, 4;\n+\tif ( $bytes_read == 0 ) {\n+\n+\t\t# EOF - Git stopped talking to us!\n+\t\texit();\n+\t}\n+\telsif ( $bytes_read != 4 ) {\n+\t\tdie \"invalid packet: '$buffer'\";\n+\t}\n+\tmy $pkt_size = hex($buffer);\n+\tif ( $pkt_size == 0 ) {\n+\t\treturn ( 1, \"\" );\n+\t}\n+\telsif ( $pkt_size > 4 ) {\n+\t\tmy $content_size = $pkt_size - 4;\n+\t\t$bytes_read = read STDIN, $buffer, $content_size;\n+\t\tif ( $bytes_read != $content_size ) {\n+\t\t\tdie \"invalid packet ($content_size bytes expected; $bytes_read bytes read)\";\n+\t\t}\n+\t\treturn ( 0, $buffer );\n+\t}\n+\telse {\n+\t\tdie \"invalid packet size: $pkt_size\";\n+\t}\n+}\n+\n+sub packet_txt_read {\n+\tmy ( $res, $buf ) = packet_bin_read();\n+\tunless ( $buf =~ s/\\n$// ) {\n+\t\tdie \"A non-binary line MUST be terminated by an LF.\";\n+\t}\n+\treturn ( $res, $buf );\n+}\n+\n+sub packet_bin_write {\n+\tmy $buf = shift;\n+\tprint STDOUT sprintf( \"%04x\", length($buf) + 4 );\n+\tprint STDOUT $buf;\n+\tSTDOUT->flush();\n+}\n+\n+sub packet_txt_write {\n+\tpacket_bin_write( $_[0] . \"\\n\" );\n+}\n+\n+sub packet_flush {\n+\tprint STDOUT sprintf( \"%04x\", 0 );\n+\tSTDOUT->flush();\n+}\n+\n+( packet_txt_read() eq ( 0, \"git-read-object-client\" ) ) || die \"bad initialize\";\n+( packet_txt_read() eq ( 0, \"version=1\" ) )\t\t\t\t || die \"bad version\";\n+( packet_bin_read() eq ( 1, \"\" ) )                       || die \"bad version end\";\n+\n+packet_txt_write(\"git-read-object-server\");\n+packet_txt_write(\"version=1\");\n+packet_flush();\n+\n+( packet_txt_read() eq ( 0, \"capability=get\" ) )    || die \"bad capability\";\n+( packet_bin_read() eq ( 1, \"\" ) )                  || die \"bad capability end\";\n+\n+packet_txt_write(\"capability=get\");\n+packet_flush();\n+\n+while (1) {\n+\tmy ($command) = packet_txt_read() =~ /^command=([^=]+)$/;\n+\n+\tif ( $command eq \"get\" ) {\n+\t\tmy ($sha1) = packet_txt_read() =~ /^sha1=([0-9a-f]{40})$/;\n+\t\tpacket_bin_read();\n+\n+\t\tsystem ('git --git-dir=\"' . $DIR . '\" cat-file blob ' . $sha1 . ' | git -c core.virtualizeobjects=false hash-object -w --stdin >/dev/null 2>&1');\n+\t\tpacket_txt_write(($?) ? \"status=error\" : \"status=success\");\n+\t\tpacket_flush();\n+\t} else {\n+\t\tdie \"bad command '$command'\";\n+\t}\n+}\ndiff --git a/environment.c b/environment.c\nindex 3fd4b10845..ad9403ed6c 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -66,6 +66,7 @@ int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n unsigned long pack_size_limit_cfg;\n enum hide_dotfiles_type hide_dotfiles = HIDE_DOTFILES_DOTGITONLY;\n enum log_refs_config log_all_ref_updates = LOG_REFS_UNSET;\n+int core_virtualize_objects;\n \n #ifndef PROTECT_HFS_DEFAULT\n #define PROTECT_HFS_DEFAULT 0\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 5862386cd0..06290f8647 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -28,6 +28,9 @@\n #include \"list.h\"\n #include \"mergesort.h\"\n #include \"quote.h\"\n+#include \"sigchain.h\"\n+#include \"sub-process.h\"\n+#include \"pkt-line.h\"\n \n #define SZ_FMT PRIuMAX\n static inline uintmax_t sz_fmt(size_t s) { return s; }\n@@ -647,6 +650,162 @@ void prepare_alt_odb(void)\n \tread_info_alternates(get_object_directory(), 0);\n }\n \n+#define CAP_GET    (1u<<0)\n+\n+static int subprocess_map_initialized;\n+static struct hashmap subprocess_map;\n+\n+struct read_object_process {\n+\tstruct subprocess_entry subprocess;\n+\tunsigned int supported_capabilities;\n+};\n+\n+int start_read_object_fn(struct subprocess_entry *subprocess)\n+{\n+\tint err;\n+\tstruct read_object_process *entry = (struct read_object_process *)subprocess;\n+\tstruct child_process *process;\n+\tstruct string_list cap_list = STRING_LIST_INIT_NODUP;\n+\tchar *cap_buf;\n+\tconst char *cap_name;\n+\n+\tprocess = subprocess_get_child_process(&entry->subprocess);\n+\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\n+\terr = packet_writel(process->in, \"git-read-object-client\", \"version=1\", NULL);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = strcmp(packet_read_line(process->out, NULL), \"git-read-object-server\");\n+\tif (err) {\n+\t\terror(\"external process '%s' does not support read-object protocol version 1\", subprocess->cmd);\n+\t\tgoto done;\n+\t}\n+\terr = strcmp(packet_read_line(process->out, NULL), \"version=1\");\n+\tif (err)\n+\t\tgoto done;\n+\terr = packet_read_line(process->out, NULL) != NULL;\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = packet_writel(process->in, \"capability=get\", NULL);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tfor (;;) {\n+\t\tcap_buf = packet_read_line(process->out, NULL);\n+\t\tif (!cap_buf)\n+\t\t\tbreak;\n+\t\tstring_list_split_in_place(&cap_list, cap_buf, '=', 1);\n+\n+\t\tif (cap_list.nr != 2 || strcmp(cap_list.items[0].string, \"capability\"))\n+\t\t\tcontinue;\n+\n+\t\tcap_name = cap_list.items[1].string;\n+\t\tif (!strcmp(cap_name, \"get\")) {\n+\t\t\tentry->supported_capabilities |= CAP_GET;\n+\t\t}\n+\t\telse {\n+\t\t\twarning(\n+\t\t\t\t\"external process '%s' requested unsupported read-object capability '%s'\",\n+\t\t\t\tsubprocess->cmd, cap_name\n+\t\t\t);\n+\t\t}\n+\n+\t\tstring_list_clear(&cap_list, 0);\n+\t}\n+\n+done:\n+\tsigchain_pop(SIGPIPE);\n+\n+\tif (err || errno == EPIPE)\n+\t\treturn err ? err : errno;\n+\n+\treturn 0;\n+}\n+\n+static int read_object_process(const unsigned char *sha1)\n+{\n+\tint err;\n+\tstruct read_object_process *entry;\n+\tstruct child_process *process;\n+\tstruct strbuf status = STRBUF_INIT;\n+\tconst char *cmd = find_hook(\"read-object\");\n+\tuint64_t start;\n+\n+\tstart = getnanotime();\n+\n+\tif (!subprocess_map_initialized) {\n+\t\tsubprocess_map_initialized = 1;\n+\t\thashmap_init(&subprocess_map, (hashmap_cmp_fn)cmd2process_cmp, 0);\n+\t\tentry = NULL;\n+\t} else {\n+\t\tentry = (struct cmd2process *)subprocess_find_entry(&subprocess_map, cmd);\n+\t}\n+\tif (!entry) {\n+\t\tentry = xmalloc(sizeof(*entry));\n+\t\tentry->supported_capabilities = 0;\n+\n+\t\tif (subprocess_start(&subprocess_map, &entry->subprocess, cmd, start_read_object_fn)) {\n+\t\t\tfree(entry);\n+\t\t\treturn -1;\n+\t\t}\n+\t}\n+\tprocess = subprocess_get_child_process(&entry->subprocess);\n+\n+\tif (!(CAP_GET & entry->supported_capabilities))\n+\t\treturn -1;\n+\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\n+\terr = packet_write_fmt_gently(process->in, \"command=get\\n\");\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = packet_write_fmt_gently(process->in, \"sha1=%s\\n\", sha1_to_hex(sha1));\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = packet_flush_gently(process->in);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = subprocess_read_status(process->out, &status);\n+\terr = err ? err : strcmp(status.buf, \"success\");\n+\n+done:\n+\tsigchain_pop(SIGPIPE);\n+\n+\tif (err || errno == EPIPE) {\n+\t\terr = err ? err : errno;\n+\t\tif (!strcmp(status.buf, \"error\")) {\n+\t\t\t/* The process signaled a problem with the file. */\n+\t\t}\n+\t\telse if (!strcmp(status.buf, \"abort\")) {\n+\t\t\t/*\n+\t\t\t * The process signaled a permanent problem. Don't try to read\n+\t\t\t * objects with the same command for the lifetime of the current\n+\t\t\t * Git process.\n+\t\t\t */\n+\t\t\tentry->supported_capabilities &= ~CAP_GET;\n+\t\t}\n+\t\telse {\n+\t\t\t/*\n+\t\t\t * Something went wrong with the read-object process.\n+\t\t\t * Force shutdown and restart if needed.\n+\t\t\t */\n+\t\t\terror(\"external process '%s' failed\", cmd);\n+\t\t\tsubprocess_stop(&subprocess_map, (struct subprocess_entry *)entry);\n+\t\t\tfree(entry);\n+\t\t}\n+\t}\n+\n+\ttrace_performance_since(start, \"read_object_process\");\n+\n+\treturn err;\n+}\n+\n /* Returns 1 if we have successfully freshened the file, 0 otherwise. */\n static int freshen_file(const char *fn)\n {\n@@ -690,8 +849,19 @@ static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)\n \n static int check_and_freshen(const unsigned char *sha1, int freshen)\n {\n-\treturn check_and_freshen_local(sha1, freshen) ||\n-\t       check_and_freshen_nonlocal(sha1, freshen);\n+\tint ret;\n+\tint already_retried = 0;\n+\n+retry:\n+\tret = check_and_freshen_local(sha1, freshen) ||\n+\t\tcheck_and_freshen_nonlocal(sha1, freshen);\n+\tif (!ret && core_virtualize_objects && !already_retried) {\n+\t\talready_retried = 1;\n+\t\tif (!read_object_process(sha1))\n+\t\t\tgoto retry;\n+\t}\n+\n+\treturn ret;\n }\n \n int has_loose_object_nonlocal(const unsigned char *sha1)\n@@ -2983,6 +3153,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \tconst unsigned char *real = (flags & OBJECT_INFO_LOOKUP_REPLACE) ?\n \t\t\t\t    lookup_replace_object(sha1) :\n \t\t\t\t    sha1;\n+\tint already_retried = 0;\n \n \tif (!oi)\n \t\toi = &blank_oi;\n@@ -3007,6 +3178,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\t}\n \t}\n \n+retry:\n \tif (!find_pack_entry(real, &e)) {\n \t\t/* Most likely it's a loose object. */\n \t\tif (!sha1_loose_object_info(real, oi, flags)) {\n@@ -3015,13 +3187,19 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\t}\n \n \t\t/* Not a loose object; someone else may have just packed it. */\n-\t\tif (flags & OBJECT_INFO_QUICK) {\n-\t\t\treturn -1;\n-\t\t} else {\n+\t\tif (!(flags & OBJECT_INFO_QUICK)) {\n \t\t\treprepare_packed_git();\n-\t\t\tif (!find_pack_entry(real, &e))\n-\t\t\t\treturn -1;\n+\t\t\tif (find_pack_entry(real, &e))\n+\t\t\t\tgoto found_packed;\n+\t\t}\n+\n+\t\t/* Request the object be retrieved */\n+\t\tif (core_virtualize_objects && !already_retried) {\n+\t\t\talready_retried = 1;\n+\t\t\tif (!read_object_process(sha1))\n+\t\t\t\tgoto retry;\n \t\t}\n+\t\treturn -1;\n \t}\n \n \tif (oi == &blank_oi)\n@@ -3031,6 +3209,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\t */\n \t\treturn 0;\n \n+found_packed:\n \trtype = packed_object_info(e.p, e.offset, oi);\n \tif (rtype < 0) {\n \t\tmark_bad_packed_object(e.p, real);\ndiff --git a/t/t0410-read-object.sh b/t/t0410-read-object.sh\nnew file mode 100755\nindex 0000000000..b8d7521c2c\n--- /dev/null\n+++ b/t/t0410-read-object.sh\n@@ -0,0 +1,27 @@\n+#!/bin/sh\n+\n+test_description='tests for long running read-object process'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup host repo with a root commit' '\n+\ttest_commit zero &&\n+\thash1=$(git ls-tree HEAD | grep zero.t | cut -f1 | cut -d\\  -f3)\n+'\n+\n+test_expect_success 'blobs can be retrieved from the host repo' '\n+\tgit init guest-repo &&\n+\t(cd guest-repo &&\n+\t mkdir -p .git/hooks &&\n+\t cp $TEST_DIRECTORY/t0410/read-object .git/hooks/ &&\n+\t git config core.virtualizeobjects true &&\n+\t git cat-file blob \"$hash1\")\n+'\n+\n+test_expect_success 'invalid blobs generate errors' '\n+\t(cd guest-repo &&\n+\t test_must_fail git cat-file blob \"invalid\")\n+'\n+\n+\n+test_done\ndiff --git a/t/t0410/read-object b/t/t0410/read-object\nnew file mode 100755\nindex 0000000000..85e997c930\n--- /dev/null\n+++ b/t/t0410/read-object\n@@ -0,0 +1,114 @@\n+#!/usr/bin/perl\n+#\n+# Example implementation for the Git read-object protocol version 1\n+# See Documentation/technical/read-object-protocol.txt\n+#\n+# Allows you to test the ability for blobs to be pulled from a host git repo\n+# \"on demand.\"  Called when git needs a blob it couldn't find locally due to\n+# a lazy clone that only cloned the commits and trees.\n+#\n+# A lazy clone can be simulated via the following commands from the host repo\n+# you wish to create a lazy clone of:\n+#\n+# cd /host_repo\n+# git rev-parse HEAD\n+# git init /guest_repo\n+# git cat-file --batch-check --batch-all-objects | grep -v 'blob' |\n+#\tcut -d' ' -f1 | git pack-objects /guest_repo/.git/objects/pack/noblobs\n+# cd /guest_repo\n+# git config core.virtualizeobjects true\n+# git reset --hard <sha from rev-parse call above>\n+#\n+# Please note, this sample is a minimal skeleton. No proper error handling\n+# was implemented.\n+#\n+\n+use strict;\n+use warnings;\n+\n+#\n+# Point $DIR to the folder where your host git repo is located so we can pull\n+# missing objects from it\n+#\n+my $DIR = \"../.git/\";\n+\n+sub packet_bin_read {\n+\tmy $buffer;\n+\tmy $bytes_read = read STDIN, $buffer, 4;\n+\tif ( $bytes_read == 0 ) {\n+\n+\t\t# EOF - Git stopped talking to us!\n+\t\texit();\n+\t}\n+\telsif ( $bytes_read != 4 ) {\n+\t\tdie \"invalid packet: '$buffer'\";\n+\t}\n+\tmy $pkt_size = hex($buffer);\n+\tif ( $pkt_size == 0 ) {\n+\t\treturn ( 1, \"\" );\n+\t}\n+\telsif ( $pkt_size > 4 ) {\n+\t\tmy $content_size = $pkt_size - 4;\n+\t\t$bytes_read = read STDIN, $buffer, $content_size;\n+\t\tif ( $bytes_read != $content_size ) {\n+\t\t\tdie \"invalid packet ($content_size bytes expected; $bytes_read bytes read)\";\n+\t\t}\n+\t\treturn ( 0, $buffer );\n+\t}\n+\telse {\n+\t\tdie \"invalid packet size: $pkt_size\";\n+\t}\n+}\n+\n+sub packet_txt_read {\n+\tmy ( $res, $buf ) = packet_bin_read();\n+\tunless ( $buf =~ s/\\n$// ) {\n+\t\tdie \"A non-binary line MUST be terminated by an LF.\";\n+\t}\n+\treturn ( $res, $buf );\n+}\n+\n+sub packet_bin_write {\n+\tmy $buf = shift;\n+\tprint STDOUT sprintf( \"%04x\", length($buf) + 4 );\n+\tprint STDOUT $buf;\n+\tSTDOUT->flush();\n+}\n+\n+sub packet_txt_write {\n+\tpacket_bin_write( $_[0] . \"\\n\" );\n+}\n+\n+sub packet_flush {\n+\tprint STDOUT sprintf( \"%04x\", 0 );\n+\tSTDOUT->flush();\n+}\n+\n+( packet_txt_read() eq ( 0, \"git-read-object-client\" ) ) || die \"bad initialize\";\n+( packet_txt_read() eq ( 0, \"version=1\" ) )\t\t\t\t || die \"bad version\";\n+( packet_bin_read() eq ( 1, \"\" ) )                       || die \"bad version end\";\n+\n+packet_txt_write(\"git-read-object-server\");\n+packet_txt_write(\"version=1\");\n+packet_flush();\n+\n+( packet_txt_read() eq ( 0, \"capability=get\" ) )    || die \"bad capability\";\n+( packet_bin_read() eq ( 1, \"\" ) )                  || die \"bad capability end\";\n+\n+packet_txt_write(\"capability=get\");\n+packet_flush();\n+\n+while (1) {\n+\tmy ($command) = packet_txt_read() =~ /^command=([^=]+)$/;\n+\n+\tif ( $command eq \"get\" ) {\n+\t\tmy ($sha1) = packet_txt_read() =~ /^sha1=([0-9a-f]{40})$/;\n+\t\tpacket_bin_read();\n+\n+\t\tsystem ('git --git-dir=\"' . $DIR . '\" cat-file blob ' . $sha1 . ' | git -c core.virtualizeobjects=false hash-object -w --stdin >/dev/null 2>&1');\n+\t\tpacket_txt_write(($?) ? \"status=error\" : \"status=success\");\n+\t\tpacket_flush();\n+\t} else {\n+\t\tdie \"bad command '$command'\";\n+\t}\n+}\n-- \n2.13.2.windows.1\n\n"},{"id":"324528","messageId":"CAP8UFD2tQQ3Dyx64LMtt_+WRoDS9V+AbMtvdCcngqiL5kCBFTg@mail.gmail.com","threadId":"46384","inReplyTo":"20170714132651.170708-2-benpeart@microsoft.com","subject":"Re: [PATCH v2 1/1] sha1_file: Add support for downloading blobs on demand","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-07-14T15:18:27Z","receivedAt":"2017-07-14T15:18:39Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Fri, Jul 14, 2017 at 3:26 PM, Ben Peart <peartben@gmail.com> wrote:\n\n> diff --git a/contrib/long-running-read-object/example.pl b/contrib/long-running-read-object/example.pl\n> new file mode 100755\n> index 0000000000..b8f37f836a\n> --- /dev/null\n> +++ b/contrib/long-running-read-object/example.pl\n\n[...]\n\n> +sub packet_bin_read {\n> +       my $buffer;\n> +       my $bytes_read = read STDIN, $buffer, 4;\n> +       if ( $bytes_read == 0 ) {\n> +\n> +               # EOF - Git stopped talking to us!\n> +               exit();\n> +       }\n> +       elsif ( $bytes_read != 4 ) {\n> +               die \"invalid packet: '$buffer'\";\n> +       }\n> +       my $pkt_size = hex($buffer);\n> +       if ( $pkt_size == 0 ) {\n> +               return ( 1, \"\" );\n> +       }\n> +       elsif ( $pkt_size > 4 ) {\n> +               my $content_size = $pkt_size - 4;\n> +               $bytes_read = read STDIN, $buffer, $content_size;\n> +               if ( $bytes_read != $content_size ) {\n> +                       die \"invalid packet ($content_size bytes expected; $bytes_read bytes read)\";\n> +               }\n> +               return ( 0, $buffer );\n> +       }\n> +       else {\n> +               die \"invalid packet size: $pkt_size\";\n> +       }\n> +}\n> +\n> +sub packet_txt_read {\n> +       my ( $res, $buf ) = packet_bin_read();\n> +       unless ( $buf =~ s/\\n$// ) {\n> +               die \"A non-binary line MUST be terminated by an LF.\";\n> +       }\n> +       return ( $res, $buf );\n> +}\n> +\n> +sub packet_bin_write {\n> +       my $buf = shift;\n> +       print STDOUT sprintf( \"%04x\", length($buf) + 4 );\n> +       print STDOUT $buf;\n> +       STDOUT->flush();\n> +}\n> +\n> +sub packet_txt_write {\n> +       packet_bin_write( $_[0] . \"\\n\" );\n> +}\n> +\n> +sub packet_flush {\n> +       print STDOUT sprintf( \"%04x\", 0 );\n> +       STDOUT->flush();\n> +}\n\nThe above could reuse the refactoring of t0021/rot13-filter.pl into\nperl/Git/Packet.pm that is at the beginning of my patch series.\n\n> diff --git a/t/t0410/read-object b/t/t0410/read-object\n> new file mode 100755\n> index 0000000000..85e997c930\n> --- /dev/null\n> +++ b/t/t0410/read-object\n\n[...]\n\n> +sub packet_bin_read {\n> +       my $buffer;\n> +       my $bytes_read = read STDIN, $buffer, 4;\n> +       if ( $bytes_read == 0 ) {\n> +\n> +               # EOF - Git stopped talking to us!\n> +               exit();\n> +       }\n> +       elsif ( $bytes_read != 4 ) {\n> +               die \"invalid packet: '$buffer'\";\n> +       }\n> +       my $pkt_size = hex($buffer);\n> +       if ( $pkt_size == 0 ) {\n> +               return ( 1, \"\" );\n> +       }\n> +       elsif ( $pkt_size > 4 ) {\n> +               my $content_size = $pkt_size - 4;\n> +               $bytes_read = read STDIN, $buffer, $content_size;\n> +               if ( $bytes_read != $content_size ) {\n> +                       die \"invalid packet ($content_size bytes expected; $bytes_read bytes read)\";\n> +               }\n> +               return ( 0, $buffer );\n> +       }\n> +       else {\n> +               die \"invalid packet size: $pkt_size\";\n> +       }\n> +}\n> +\n> +sub packet_txt_read {\n> +       my ( $res, $buf ) = packet_bin_read();\n> +       unless ( $buf =~ s/\\n$// ) {\n> +               die \"A non-binary line MUST be terminated by an LF.\";\n> +       }\n> +       return ( $res, $buf );\n> +}\n> +\n> +sub packet_bin_write {\n> +       my $buf = shift;\n> +       print STDOUT sprintf( \"%04x\", length($buf) + 4 );\n> +       print STDOUT $buf;\n> +       STDOUT->flush();\n> +}\n> +\n> +sub packet_txt_write {\n> +       packet_bin_write( $_[0] . \"\\n\" );\n> +}\n> +\n> +sub packet_flush {\n> +       print STDOUT sprintf( \"%04x\", 0 );\n> +       STDOUT->flush();\n> +}\n\nSame thing about the above and perl/Git/Packet.pm.\n\nOtherwise thanks for updating this!\n"},{"id":"324657","messageId":"20170717110602.6fac89ea@twelve2.svl.corp.google.com","threadId":"46384","inReplyTo":"20170714132651.170708-2-benpeart@microsoft.com","subject":"Re: [PATCH v2 1/1] sha1_file: Add support for downloading blobs on demand","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-07-17T18:06:02Z","receivedAt":"2017-07-17T18:06:12Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"About the difference between this patch and my patch set [1], besides\nthe fact that this patch does not spawn separate processes for each\nmissing object, which does seem like an improvement to me, this patch\n(i) does not use a list of promised objects (but instead communicates\nwith the hook for each missing object), and (ii) provides backwards\ncompatibility with other Git code (that does not know about promised\nobjects) in a different way.\n\nThe costs and benefits of (i) are being discussed here [2]. As for (ii),\nI still think that my approach is better - I have commented more about\nthis below.\n\nMaybe the best approach is a combination of both our approaches.\n\n[1] https://public-inbox.org/git/34efd9e9936fdab331655f5a33a098a72dc134f4.1499800530.git.jonathantanmy@google.com/\n\n[2] https://public-inbox.org/git/20170713123951.5cab1adc@twelve2.svl.corp.google.com/\n\nOn Fri, 14 Jul 2017 09:26:51 -0400\nBen Peart <peartben@gmail.com> wrote:\n\n> +------------------------\n> +packet: git> command=get\n> +packet: git> sha1=0a214a649e1b3d5011e14a3dc227753f2bd2be05\n> +packet: git> 0000\n> +------------------------\n\nIt would be useful to have this command support more than one SHA-1, so\nthat hooks that know how to batch can do so.\n\n> +static int subprocess_map_initialized;\n> +static struct hashmap subprocess_map;\n\nThe documentation of \"tablesize\" in \"struct hashmap\" states that it can\nbe used to check if the hashmap is initialized, so\nsubprocess_map_initialized is probably unnecessary.\n\n>  static int check_and_freshen(const unsigned char *sha1, int freshen)\n>  {\n> -\treturn check_and_freshen_local(sha1, freshen) ||\n> -\t       check_and_freshen_nonlocal(sha1, freshen);\n> +\tint ret;\n> +\tint already_retried = 0;\n> +\n> +retry:\n> +\tret = check_and_freshen_local(sha1, freshen) ||\n> +\t\tcheck_and_freshen_nonlocal(sha1, freshen);\n> +\tif (!ret && core_virtualize_objects && !already_retried) {\n> +\t\talready_retried = 1;\n> +\t\tif (!read_object_process(sha1))\n> +\t\t\tgoto retry;\n> +\t}\n> +\n> +\treturn ret;\n>  }\n\nIs this change meant to ensure that Git code that operates on loose\nobjects directly (bypassing storage-agnostic functions such as\nsha1_object_info_extended() and has_sha1_file()) still work? If yes,\nthis patch appears incomplete (for example, read_loose_object() needs to\nbe changed too), and this seems like a difficult task - in my patch set\n[1], I ended up deciding to create a separate type of storage and\ninstead looked at the code that operates on *packed* objects directly\n(because there were fewer such methods) to ensure that they would work\ncorrectly in the presence of a separate type of storage.\n\n[1] https://public-inbox.org/git/34efd9e9936fdab331655f5a33a098a72dc134f4.1499800530.git.jonathantanmy@google.com/\n"},{"id":"324668","messageId":"d3f1f884-7b8a-885f-47cb-eca2b8ef0ecf@gmail.com","threadId":"46384","inReplyTo":"20170717110602.6fac89ea@twelve2.svl.corp.google.com","subject":"Re: [PATCH v2 1/1] sha1_file: Add support for downloading blobs on demand","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-07-17T20:09:17Z","receivedAt":"2017-07-17T20:09:25Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 7/17/2017 2:06 PM, Jonathan Tan wrote:\n> About the difference between this patch and my patch set [1], besides\n> the fact that this patch does not spawn separate processes for each\n> missing object, which does seem like an improvement to me, this patch\n> (i) does not use a list of promised objects (but instead communicates\n> with the hook for each missing object), and (ii) provides backwards\n> compatibility with other Git code (that does not know about promised\n> objects) in a different way.\n> \n> The costs and benefits of (i) are being discussed here [2]. As for (ii),\n> I still think that my approach is better - I have commented more about\n> this below.\n> \n> Maybe the best approach is a combination of both our approaches.\n\nYes, in the context of the promised objects model patch series, the \nvalue of this patch series is primarily as a sample of how to use the \nsub-process mechanism to create a versioned interface for retrieving \nobjects.\n\n> \n> [1] https://public-inbox.org/git/34efd9e9936fdab331655f5a33a098a72dc134f4.1499800530.git.jonathantanmy@google.com/\n> \n> [2] https://public-inbox.org/git/20170713123951.5cab1adc@twelve2.svl.corp.google.com/\n> \n> On Fri, 14 Jul 2017 09:26:51 -0400\n> Ben Peart <peartben@gmail.com> wrote:\n> \n>> +------------------------\n>> +packet: git> command=get\n>> +packet: git> sha1=0a214a649e1b3d5011e14a3dc227753f2bd2be05\n>> +packet: git> 0000\n>> +------------------------\n> \n> It would be useful to have this command support more than one SHA-1, so\n> that hooks that know how to batch can do so.\n> \n\nI agree.  Since nothing was using that capability yet, I decided to keep \nit simple and not add support for a feature that wasn't being used. The \nreason the interface is versioned is so that if/when something does need \nthat capability, it can be added.\n\n>> +static int subprocess_map_initialized;\n>> +static struct hashmap subprocess_map;\n> \n> The documentation of \"tablesize\" in \"struct hashmap\" states that it can\n> be used to check if the hashmap is initialized, so\n> subprocess_map_initialized is probably unnecessary.\n> \n\nNice.  That will make things a little simpler.\n\n>>   static int check_and_freshen(const unsigned char *sha1, int freshen)\n>>   {\n>> -\treturn check_and_freshen_local(sha1, freshen) ||\n>> -\t       check_and_freshen_nonlocal(sha1, freshen);\n>> +\tint ret;\n>> +\tint already_retried = 0;\n>> +\n>> +retry:\n>> +\tret = check_and_freshen_local(sha1, freshen) ||\n>> +\t\tcheck_and_freshen_nonlocal(sha1, freshen);\n>> +\tif (!ret && core_virtualize_objects && !already_retried) {\n>> +\t\talready_retried = 1;\n>> +\t\tif (!read_object_process(sha1))\n>> +\t\t\tgoto retry;\n>> +\t}\n>> +\n>> +\treturn ret;\n>>   }\n> \n> Is this change meant to ensure that Git code that operates on loose\n> objects directly (bypassing storage-agnostic functions such as\n> sha1_object_info_extended() and has_sha1_file()) still work? If yes,\n> this patch appears incomplete (for example, read_loose_object() needs to\n> be changed too), and this seems like a difficult task - in my patch set\n> [1], I ended up deciding to create a separate type of storage and\n> instead looked at the code that operates on *packed* objects directly\n> (because there were fewer such methods) to ensure that they would work\n> correctly in the presence of a separate type of storage.\n> \n\nYes, with this set of patches, we've been running successfully on \ncompletely sparse clones (no commits, trees, or blobs) for several \nmonths.  read_loose_object() is only called by fsck when it is \nenumerating existing loose objects so does not need to be updated.\n\nWe have a few thousand developers making ~100K commits per week so in \nour particular usage, I'm fairly confident it works correctly.  That \nsaid, it is possible there is some code path I've missed. :)\n\n> [1] https://public-inbox.org/git/34efd9e9936fdab331655f5a33a098a72dc134f4.1499800530.git.jonathantanmy@google.com/\n> \n"},{"id":"324692","messageId":"20170717162431.1e11737d@twelve2.svl.corp.google.com","threadId":"46384","inReplyTo":"d3f1f884-7b8a-885f-47cb-eca2b8ef0ecf@gmail.com","subject":"Re: [PATCH v2 1/1] sha1_file: Add support for downloading blobs on demand","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-07-17T23:24:31Z","receivedAt":"2017-07-17T23:24:42Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Mon, 17 Jul 2017 16:09:17 -0400\nBen Peart <peartben@gmail.com> wrote:\n\n> > Is this change meant to ensure that Git code that operates on loose\n> > objects directly (bypassing storage-agnostic functions such as\n> > sha1_object_info_extended() and has_sha1_file()) still work? If yes,\n> > this patch appears incomplete (for example, read_loose_object() needs to\n> > be changed too), and this seems like a difficult task - in my patch set\n> > [1], I ended up deciding to create a separate type of storage and\n> > instead looked at the code that operates on *packed* objects directly\n> > (because there were fewer such methods) to ensure that they would work\n> > correctly in the presence of a separate type of storage.\n> > \n> \n> Yes, with this set of patches, we've been running successfully on \n> completely sparse clones (no commits, trees, or blobs) for several \n> months.  read_loose_object() is only called by fsck when it is \n> enumerating existing loose objects so does not need to be updated.\n\nAh, that's good to know. I think such an analysis (of the other\nloose-related functions) in the commit message would be useful, like I\ndid for the packed-related functions [1].\n\n[1] https://public-inbox.org/git/34efd9e9936fdab331655f5a33a098a72dc134f4.1499800530.git.jonathantanmy@google.com/\n\n> We have a few thousand developers making ~100K commits per week so in \n> our particular usage, I'm fairly confident it works correctly.  That \n> said, it is possible there is some code path I've missed. :)\n\nI think that code paths like the history-searching ones (\"git log -S\",\nfor example) should still work offline if possible - one of my ideas is\nto have these commands take a size threshold parameter so that we do not\nneed to fetch large blobs during the invocation. (Hence my preference\nfor size information to be already available to the repo.)\n\nAside from that, does fsck of a partial repo work? Ability to fsck seems\nquite important (or, at least, useful). I tried updating fsck to support\nomitting commits and trees (in addition to blobs), and it seems\nrelatively involved (although I didn't look into it deeply yet).\n\n(Also, that investigation also made me realize that, in my patch set, I\ndidn't handle the case where a tag references a blob - fsck doesn't work\nwhen the blob is missing, even if it is promised. That's something for\nme to look into.)\n"}]}