{"thread":{"id":"46764","subject":"[PATCH v6 01/40] builtin/clone: get rid of 'value' strbuf","startedAt":"2017-09-16T08:07:54Z","lastAt":"2017-10-04T00:15:55Z","messageCount":49,"participants":["Christian Couder","Jonathan Tan","Ben Peart"],"isPatch":true,"patchVersion":6,"patchTotal":40},"messages":[{"id":"328187","messageId":"20170916080731.13925-2-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 01/40] builtin/clone: get rid of 'value' strbuf","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:06:52Z","receivedAt":"2017-09-16T08:07:54Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This makes the code simpler by removing a few lines, and getting\nrid of one variable.\n\nSigned-off-by: Christian Couder \n---\n builtin/clone.c | 9 +++------\n 1 file changed, 3 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 8d11b570a1..dcd5b878f1 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -875,7 +875,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tconst struct ref *our_head_points_at;\n \tstruct ref *mapped_refs;\n \tconst struct ref *ref;\n-\tstruct strbuf key = STRBUF_INIT, value = STRBUF_INIT;\n+\tstruct strbuf key = STRBUF_INIT;\n \tstruct strbuf branch_top = STRBUF_INIT, reflog_msg = STRBUF_INIT;\n \tstruct transport *transport = NULL;\n \tconst char *src_ref_prefix = \"refs/heads/\";\n@@ -1040,7 +1040,6 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\tstrbuf_addf(&branch_top, \"refs/remotes/%s/\", option_origin);\n \t}\n \n-\tstrbuf_addf(&value, \"+%s*:%s*\", src_ref_prefix, branch_top.buf);\n \tstrbuf_addf(&key, \"remote.%s.url\", option_origin);\n \tgit_config_set(key.buf, repo);\n \tstrbuf_reset(&key);\n@@ -1054,10 +1053,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (option_required_reference.nr || option_optional_reference.nr)\n \t\tsetup_reference();\n \n-\tfetch_pattern = value.buf;\n+\tfetch_pattern = xstrfmt(\"+%s*:%s*\", src_ref_prefix, branch_top.buf);\n \trefspec = parse_fetch_refspec(1, &fetch_pattern);\n-\n-\tstrbuf_reset(&value);\n+\tfree((char *)fetch_pattern);\n \n \tremote = remote_get(option_origin);\n \ttransport = transport_get(remote, remote->url[0]);\n@@ -1196,7 +1194,6 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tstrbuf_release(&reflog_msg);\n \tstrbuf_release(&branch_top);\n \tstrbuf_release(&key);\n-\tstrbuf_release(&value);\n \tjunk_mode = JUNK_LEAVE_ALL;\n \n \tfree(refspec);\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328188","messageId":"20170916080731.13925-5-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 04/40] t0021/rot13-filter: improve error message","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:06:55Z","receivedAt":"2017-09-16T08:07:57Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"If there is no new line at the end of something it receives,\nthe packet_txt_read() function die()s, but it's difficult to\ndebug without much context.\n\nLet's give a bit more information when that happens.\n\nSigned-off-by: Christian Couder \n---\n t/t0021/rot13-filter.pl | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\nindex 82882392ae..3b3da8a03d 100644\n--- a/t/t0021/rot13-filter.pl\n+++ b/t/t0021/rot13-filter.pl\n@@ -82,7 +82,8 @@ sub packet_bin_read {\n sub packet_txt_read {\n \tmy ( $res, $buf ) = packet_bin_read();\n \tunless ( $res == -1 or $buf eq '' or $buf =~ s/\\n$// ) {\n-\t\tdie \"A non-binary line MUST be terminated by an LF.\";\n+\t\tdie \"A non-binary line MUST be terminated by an LF.\\n\"\n+\t\t . \"Received: '$buf'\";\n \t}\n \treturn ( $res, $buf );\n }\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328189","messageId":"20170916080731.13925-6-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 05/40] t0021/rot13-filter: add packet_initialize()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:06:56Z","receivedAt":"2017-09-16T08:08:03Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Let's refactor the code to initialize communication into its own\npacket_initialize() function, so that we can reuse this\nfunctionality in following patches.\n\nSigned-off-by: Christian Couder \n---\n t/t0021/rot13-filter.pl | 20 +++++++++++++-------\n 1 file changed, 13 insertions(+), 7 deletions(-)\n\ndiff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\nindex 3b3da8a03d..278fc6f534 100644\n--- a/t/t0021/rot13-filter.pl\n+++ b/t/t0021/rot13-filter.pl\n@@ -104,16 +104,22 @@ sub packet_flush {\n \tSTDOUT->flush();\n }\n \n+sub packet_initialize {\n+\tmy ($name, $version) = @_;\n+\n+\t( packet_txt_read() eq ( 0, $name . \"-client\" ) ) || die \"bad initialize\";\n+\t( packet_txt_read() eq ( 0, \"version=\" . $version ) ) || die \"bad version\";\n+\t( packet_bin_read() eq ( 1, \"\" ) ) || die \"bad version end\";\n+\n+\tpacket_txt_write( $name . \"-server\" );\n+\tpacket_txt_write( \"version=\" . $version );\n+\tpacket_flush();\n+}\n+\n print $debug \"START\\n\";\n $debug->flush();\n \n-( packet_txt_read() eq ( 0, \"git-filter-client\" ) ) || die \"bad initialize\";\n-( packet_txt_read() eq ( 0, \"version=2\" ) ) || die \"bad version\";\n-( packet_bin_read() eq ( 1, \"\" ) ) || die \"bad version end\";\n-\n-packet_txt_write(\"git-filter-server\");\n-packet_txt_write(\"version=2\");\n-packet_flush();\n+packet_initialize(\"git-filter\", 2);\n \n ( packet_txt_read() eq ( 0, \"capability=clean\" ) ) || die \"bad capability\";\n ( packet_txt_read() eq ( 0, \"capability=smudge\" ) ) || die \"bad capability\";\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328190","messageId":"20170916080731.13925-8-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 07/40] Add Git/Packet.pm from parts of t0021/rot13-filter.pl","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:06:58Z","receivedAt":"2017-09-16T08:08:06Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"And while at it let's simplify t0021/rot13-filter.pl by\nusing Git/Packet.pm.\n\nThis will make it possible to reuse packet related\nfunctions in other test scripts.\n\nSigned-off-by: Christian Couder \n---\n perl/Git/Packet.pm | 118 ++++++++++++++++++++++++++++++++++++++++++++++++\n t/t0021/rot13-filter.pl | 94 ++------------------------------------\n 2 files changed, 121 insertions(+), 91 deletions(-)\n create mode 100644 perl/Git/Packet.pm\n\ndiff --git a/perl/Git/Packet.pm b/perl/Git/Packet.pm\nnew file mode 100644\nindex 0000000000..b1e67477a0\n--- /dev/null\n+++ b/perl/Git/Packet.pm\n@@ -0,0 +1,118 @@\n+package Git::Packet;\n+use 5.008;\n+use strict;\n+use warnings;\n+BEGIN {\n+\trequire Exporter;\n+\tif ($] < 5.008003) {\n+\t\t*import = \\&Exporter::import;\n+\t} else {\n+\t\t# Exporter 5.57 which supports this invocation was\n+\t\t# released with perl 5.8.3\n+\t\tExporter->import('import');\n+\t}\n+}\n+\n+our @EXPORT = qw(\n+\t\t\tpacket_bin_read\n+\t\t\tpacket_txt_read\n+\t\t\tpacket_bin_write\n+\t\t\tpacket_txt_write\n+\t\t\tpacket_flush\n+\t\t\tpacket_initialize\n+\t\t\tpacket_read_capabilities\n+\t\t\tpacket_write_capabilities\n+\t\t\tpacket_read_and_check_capabilities\n+\t\t);\n+our @EXPORT_OK = @EXPORT;\n+\n+sub packet_bin_read {\n+\tmy $buffer;\n+\tmy $bytes_read = read STDIN, $buffer, 4;\n+\tif ( $bytes_read == 0 ) {\n+\t\t# EOF - Git stopped talking to us!\n+\t\treturn ( -1, \"\" );\n+\t} elsif ( $bytes_read != 4 ) {\n+\t\tdie \"invalid packet: '$buffer'\";\n+\t}\n+\tmy $pkt_size = hex($buffer);\n+\tif ( $pkt_size == 0 ) {\n+\t\treturn ( 1, \"\" );\n+\t} elsif ( $pkt_size > 4 ) {\n+\t\tmy $content_size = $pkt_size - 4;\n+\t\t$bytes_read = read STDIN, $buffer, $content_size;\n+\t\tif ( $bytes_read != $content_size ) {\n+\t\t\tdie \"invalid packet ($content_size bytes expected; $bytes_read bytes read)\";\n+\t\t}\n+\t\treturn ( 0, $buffer );\n+\t} else {\n+\t\tdie \"invalid packet size: $pkt_size\";\n+\t}\n+}\n+\n+sub packet_txt_read {\n+\tmy ( $res, $buf ) = packet_bin_read();\n+\tunless ( $res == -1 or $buf eq '' or $buf =~ s/\\n$// ) {\n+\t\tdie \"A non-binary line MUST be terminated by an LF.\\n\"\n+\t\t . \"Received: '$buf'\";\n+\t}\n+\treturn ( $res, $buf );\n+}\n+\n+sub packet_bin_write {\n+\tmy $buf = shift;\n+\tprint STDOUT sprintf( \"%04x\", length($buf) + 4 );\n+\tprint STDOUT $buf;\n+\tSTDOUT->flush();\n+}\n+\n+sub packet_txt_write {\n+\tpacket_bin_write( $_[0] . \"\\n\" );\n+}\n+\n+sub packet_flush {\n+\tprint STDOUT sprintf( \"%04x\", 0 );\n+\tSTDOUT->flush();\n+}\n+\n+sub packet_initialize {\n+\tmy ($name, $version) = @_;\n+\n+\t( packet_txt_read() eq ( 0, $name . \"-client\" ) )\t|| die \"bad initialize\";\n+\t( packet_txt_read() eq ( 0, \"version=\" . $version ) )\t|| die \"bad version\";\n+\t( packet_bin_read() eq ( 1, \"\" ) )\t\t\t|| die \"bad version end\";\n+\n+\tpacket_txt_write( $name . \"-server\" );\n+\tpacket_txt_write( \"version=\" . $version );\n+\tpacket_flush();\n+}\n+\n+sub packet_read_capabilities {\n+\tmy @cap;\n+\twhile (1) {\n+\t\tmy ( $res, $buf ) = packet_bin_read();\n+\t\treturn ( $res, @cap ) if ( $res != 0 );\n+\t\tunless ( $buf =~ s/\\n$// ) {\n+\t\t\tdie \"A non-binary line MUST be terminated by an LF.\\n\"\n+\t\t\t . \"Received: '$buf'\";\n+\t\t}\n+\t\tdie \"bad capability buf: '$buf'\" unless ( $buf =~ s/capability=// );\n+\t\tpush @cap, $buf;\n+\t}\n+}\n+\n+sub packet_read_and_check_capabilities {\n+\tmy @local_caps = @_;\n+\tmy @remote_res_caps = packet_read_capabilities();\n+\tmy $res = shift @remote_res_caps;\n+\tmy %remote_caps = map { $_ => 1 } @remote_res_caps;\n+\tforeach (@local_caps) {\n+\t\tdie \"'$_' capability not available\" unless (exists($remote_caps{$_}));\n+\t}\n+\treturn $res;\n+}\n+\n+sub packet_write_capabilities {\n+\tpacket_txt_write( \"capability=\" . $_ ) foreach (@_);\n+\tpacket_flush();\n+}\ndiff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\nindex ba18b207c6..2e8ad4d496 100644\n--- a/t/t0021/rot13-filter.pl\n+++ b/t/t0021/rot13-filter.pl\n@@ -30,9 +30,12 @@\n # to the \"list_available_blobs\" response.\n #\n \n+use 5.008;\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n use strict;\n use warnings;\n use IO::File;\n+use Git::Packet;\n \n my $MAX_PACKET_CONTENT_SIZE = 65516;\n my $log_file = shift @ARGV;\n@@ -55,97 +58,6 @@ sub rot13 {\n \treturn $str;\n }\n \n-sub packet_bin_read {\n-\tmy $buffer;\n-\tmy $bytes_read = read STDIN, $buffer, 4;\n-\tif ( $bytes_read == 0 ) {\n-\t\t# EOF - Git stopped talking to us!\n-\t\treturn ( -1, \"\" );\n-\t} elsif ( $bytes_read != 4 ) {\n-\t\tdie \"invalid packet: '$buffer'\";\n-\t}\n-\tmy $pkt_size = hex($buffer);\n-\tif ( $pkt_size == 0 ) {\n-\t\treturn ( 1, \"\" );\n-\t} elsif ( $pkt_size > 4 ) {\n-\t\tmy $content_size = $pkt_size - 4;\n-\t\t$bytes_read = read STDIN, $buffer, $content_size;\n-\t\tif ( $bytes_read != $content_size ) {\n-\t\t\tdie \"invalid packet ($content_size bytes expected; $bytes_read bytes read)\";\n-\t\t}\n-\t\treturn ( 0, $buffer );\n-\t} else {\n-\t\tdie \"invalid packet size: $pkt_size\";\n-\t}\n-}\n-\n-sub packet_txt_read {\n-\tmy ( $res, $buf ) = packet_bin_read();\n-\tunless ( $res == -1 or $buf eq '' or $buf =~ s/\\n$// ) {\n-\t\tdie \"A non-binary line MUST be terminated by an LF.\\n\"\n-\t\t . \"Received: '$buf'\";\n-\t}\n-\treturn ( $res, $buf );\n-}\n-\n-sub packet_bin_write {\n-\tmy $buf = shift;\n-\tprint STDOUT sprintf( \"%04x\", length($buf) + 4 );\n-\tprint STDOUT $buf;\n-\tSTDOUT->flush();\n-}\n-\n-sub packet_txt_write {\n-\tpacket_bin_write( $_[0] . \"\\n\" );\n-}\n-\n-sub packet_flush {\n-\tprint STDOUT sprintf( \"%04x\", 0 );\n-\tSTDOUT->flush();\n-}\n-\n-sub packet_initialize {\n-\tmy ($name, $version) = @_;\n-\n-\t( packet_txt_read() eq ( 0, $name . \"-client\" ) ) || die \"bad initialize\";\n-\t( packet_txt_read() eq ( 0, \"version=\" . $version ) ) || die \"bad version\";\n-\t( packet_bin_read() eq ( 1, \"\" ) ) || die \"bad version end\";\n-\n-\tpacket_txt_write( $name . \"-server\" );\n-\tpacket_txt_write( \"version=\" . $version );\n-\tpacket_flush();\n-}\n-\n-sub packet_read_capabilities {\n-\tmy @cap;\n-\twhile (1) {\n-\t\tmy ( $res, $buf ) = packet_bin_read();\n-\t\treturn ( $res, @cap ) if ( $res != 0 );\n-\t\tunless ( $buf =~ s/\\n$// ) {\n-\t\t\tdie \"A non-binary line MUST be terminated by an LF.\\n\"\n-\t\t\t . \"Received: '$buf'\";\n-\t\t}\n-\t\tdie \"bad capability buf: '$buf'\" unless ( $buf =~ s/capability=// );\n-\t\tpush @cap, $buf;\n-\t}\n-}\n-\n-sub packet_read_and_check_capabilities {\n-\tmy @local_caps = @_;\n-\tmy @remote_res_caps = packet_read_capabilities();\n-\tmy $res = shift @remote_res_caps;\n-\tmy %remote_caps = map { $_ => 1 } @remote_res_caps;\n-\tforeach (@local_caps) {\n- \tdie \"'$_' capability not available\" unless (exists($remote_caps{$_}));\n-\t}\n-\treturn $res;\n-}\n-\n-sub packet_write_capabilities {\n-\tpacket_txt_write( \"capability=\" . $_ ) foreach (@_);\n-\tpacket_flush();\n-}\n-\n print $debug \"START\\n\";\n $debug->flush();\n \n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328191","messageId":"20170916080731.13925-12-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 11/40] t0400: add 'put_raw_obj' instruction to odb-helper script","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:02Z","receivedAt":"2017-09-16T08:08:10Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"To properly test passing objects from Git to an external odb\nwe need an odb-helper script that supports a 'put'\ncapability/instruction.\n\nFor now we will support only sending raw blobs, so the\nsupported capability/instruction will be 'put_raw_obj'.\n\nWhile at it let's add a test to check that our odb-helper\nscript works well.\n\nSigned-off-by: Christian Couder \n---\n t/t0400-external-odb.sh | 24 ++++++++++++++++++++++++\n 1 file changed, 24 insertions(+)\n\ndiff --git a/t/t0400-external-odb.sh b/t/t0400-external-odb.sh\nindex ed89f3ab40..f9e6ea1015 100755\n--- a/t/t0400-external-odb.sh\n+++ b/t/t0400-external-odb.sh\n@@ -7,10 +7,15 @@ test_description='basic tests for external object databases'\n ALT_SOURCE=\"$PWD/alt-repo/.git\"\n export ALT_SOURCE\n write_script odb-helper <<\\EOF\n+die() {\n+\tprintf >&2 \"%s\\n\" \"$@\"\n+\texit 1\n+}\n GIT_DIR=$ALT_SOURCE; export GIT_DIR\n case \"$1\" in\n init)\n \techo \"capability=get_git_obj\"\n+\techo \"capability=put_raw_obj\"\n \techo \"capability=have\"\n \t;;\n have)\n@@ -20,6 +25,16 @@ have)\n get_git_obj)\n \tcat \"$GIT_DIR\"/objects/$(echo $2 | sed 's#..#&/#')\n \t;;\n+put_raw_obj)\n+\tsha1=\"$2\"\n+\tsize=\"$3\"\n+\tkind=\"$4\"\n+\twritten=$(git hash-object -w -t \"$kind\" --stdin)\n+\ttest \"$written\" = \"$sha1\" || die \"bad sha1 passed '$sha1' vs written '$written'\"\n+\t;;\n+*)\n+\tdie \"unknown command '$1'\"\n+\t;;\n esac\n EOF\n HELPER=\"\\\"$PWD\\\"/odb-helper\"\n@@ -47,4 +62,13 @@ test_expect_success 'helper can retrieve alt objects' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'helper can add objects to alt repo' '\n+\thash=$(echo \"Hello odb!\" | git hash-object -w -t blob --stdin) &&\n+\ttest -f .git/objects/$(echo $hash | sed \"s#..#&/#\") &&\n+\tsize=$(git cat-file -s \"$hash\") &&\n+\tgit cat-file blob \"$hash\" | ./odb-helper put_raw_obj \"$hash\" \"$size\" blob &&\n+\talt_size=$(cd alt-repo && git cat-file -s \"$hash\") &&\n+\ttest \"$size\" -eq \"$alt_size\"\n+'\n+\n test_done\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328192","messageId":"20170916080731.13925-11-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 10/40] odb-helper: add odb_helper_init() to send 'init' instruction","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:01Z","receivedAt":"2017-09-16T08:08:17Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Let's add an odb_helper_init() function to send an 'init'\ninstruction to the helpers. This 'init' instruction is\nespecially useful to get the capabilities that are supported\nby the helpers.\n\nSo while at it, let's also add a parse_capabilities()\nfunction to parse them and a supported_capabilities\nvariable in struct odb_helper to store them.\n\nSigned-off-by: Christian Couder \n---\n external-odb.c | 9 ++++++++-\n odb-helper.c | 54 +++++++++++++++++++++++++++++++++++++++++++++++++\n odb-helper.h | 12 +++++++++++\n t/t0400-external-odb.sh | 4 ++++\n 4 files changed, 78 insertions(+), 1 deletion(-)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex e9c3f11666..0f0de170b8 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -41,12 +41,16 @@ static int external_odb_config(const char *var, const char *value, void *data)\n static void external_odb_init(void)\n {\n \tstatic int initialized;\n+\tstruct odb_helper *o;\n \n \tif (initialized)\n \t\treturn;\n \tinitialized = 1;\n \n \tgit_config(external_odb_config, NULL);\n+\n+\tfor (o = helpers; o; o = o->next)\n+\t\todb_helper_init(o);\n }\n \n const char *external_odb_root(void)\n@@ -63,9 +67,12 @@ int external_odb_has_object(const unsigned char *sha1)\n \n \texternal_odb_init();\n \n-\tfor (o = helpers; o; o = o->next)\n+\tfor (o = helpers; o; o = o->next) {\n+\t\tif (!(o->supported_capabilities & ODB_HELPER_CAP_HAVE))\n+\t\t\treturn 1;\n \t\tif (odb_helper_has_object(o, sha1))\n \t\t\treturn 1;\n+\t}\n \treturn 0;\n }\n \ndiff --git a/odb-helper.c b/odb-helper.c\nindex 5e91044872..9375eca58f 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -5,6 +5,40 @@\n #include \"run-command.h\"\n #include \"sha1-lookup.h\"\n \n+static void parse_capabilities(char *cap_buf,\n+\t\t\t unsigned int *supported_capabilities,\n+\t\t\t const char *process_name)\n+{\n+\tstruct string_list cap_list = STRING_LIST_INIT_NODUP;\n+\n+\tstring_list_split_in_place(&cap_list, cap_buf, '=', 1);\n+\n+\tif (cap_list.nr == 2 && !strcmp(cap_list.items[0].string, \"capability\")) {\n+\t\tconst char *cap_name = cap_list.items[1].string;\n+\n+\t\tif (!strcmp(cap_name, \"get_git_obj\")) {\n+\t\t\t*supported_capabilities |= ODB_HELPER_CAP_GET_GIT_OBJ;\n+\t\t} else if (!strcmp(cap_name, \"get_raw_obj\")) {\n+\t\t\t*supported_capabilities |= ODB_HELPER_CAP_GET_RAW_OBJ;\n+\t\t} else if (!strcmp(cap_name, \"get_direct\")) {\n+\t\t\t*supported_capabilities |= ODB_HELPER_CAP_GET_DIRECT;\n+\t\t} else if (!strcmp(cap_name, \"put_git_obj\")) {\n+\t\t\t*supported_capabilities |= ODB_HELPER_CAP_PUT_GIT_OBJ;\n+\t\t} else if (!strcmp(cap_name, \"put_raw_obj\")) {\n+\t\t\t*supported_capabilities |= ODB_HELPER_CAP_PUT_RAW_OBJ;\n+\t\t} else if (!strcmp(cap_name, \"put_direct\")) {\n+\t\t\t*supported_capabilities |= ODB_HELPER_CAP_PUT_DIRECT;\n+\t\t} else if (!strcmp(cap_name, \"have\")) {\n+\t\t\t*supported_capabilities |= ODB_HELPER_CAP_HAVE;\n+\t\t} else {\n+\t\t\twarning(\"external process '%s' requested unsupported read-object capability '%s'\",\n+\t\t\t\tprocess_name, cap_name);\n+\t\t}\n+\t}\n+\n+\tstring_list_clear(&cap_list, 0);\n+}\n+\n struct odb_helper *odb_helper_new(const char *name, int namelen)\n {\n \tstruct odb_helper *o;\n@@ -79,6 +113,26 @@ static int odb_helper_finish(struct odb_helper *o,\n \treturn 0;\n }\n \n+int odb_helper_init(struct odb_helper *o)\n+{\n+\tstruct odb_helper_cmd cmd;\n+\tFILE *fh;\n+\tstruct strbuf line = STRBUF_INIT;\n+\n+\tif (odb_helper_start(o, &cmd, \"init\") < 0)\n+\t\treturn -1;\n+\n+\tfh = xfdopen(cmd.child.out, \"r\");\n+\twhile (strbuf_getline(&line, fh) != EOF)\n+\t\tparse_capabilities(line.buf, &o->supported_capabilities, o->name);\n+\n+\tstrbuf_release(&line);\n+\tfclose(fh);\n+\todb_helper_finish(o, &cmd);\n+\n+\treturn 0;\n+}\n+\n static int parse_object_line(struct odb_helper_object *o, const char *line)\n {\n \tchar *end;\ndiff --git a/odb-helper.h b/odb-helper.h\nindex fb25ad579e..5f28a6e512 100644\n--- a/odb-helper.h\n+++ b/odb-helper.h\n@@ -1,9 +1,20 @@\n #ifndef ODB_HELPER_H\n #define ODB_HELPER_H\n \n+#include \"external-odb.h\"\n+\n+#define ODB_HELPER_CAP_GET_GIT_OBJ (1u<<0)\n+#define ODB_HELPER_CAP_GET_RAW_OBJ (1u<<1)\n+#define ODB_HELPER_CAP_GET_DIRECT (1u<<2)\n+#define ODB_HELPER_CAP_PUT_GIT_OBJ (1u<<3)\n+#define ODB_HELPER_CAP_PUT_RAW_OBJ (1u<<4)\n+#define ODB_HELPER_CAP_PUT_DIRECT (1u<<5)\n+#define ODB_HELPER_CAP_HAVE (1u<<6)\n+\n struct odb_helper {\n \tconst char *name;\n \tconst char *cmd;\n+\tunsigned int supported_capabilities;\n \n \tstruct odb_helper_object {\n \t\tunsigned char sha1[20];\n@@ -18,6 +29,7 @@ struct odb_helper {\n };\n \n extern struct odb_helper *odb_helper_new(const char *name, int namelen);\n+extern int odb_helper_init(struct odb_helper *o);\n extern int odb_helper_has_object(struct odb_helper *o,\n \t\t\t\t const unsigned char *sha1);\n extern int odb_helper_get_object(struct odb_helper *o,\ndiff --git a/t/t0400-external-odb.sh b/t/t0400-external-odb.sh\nindex 2f4749fab1..ed89f3ab40 100755\n--- a/t/t0400-external-odb.sh\n+++ b/t/t0400-external-odb.sh\n@@ -9,6 +9,10 @@ export ALT_SOURCE\n write_script odb-helper <<\\EOF\n GIT_DIR=$ALT_SOURCE; export GIT_DIR\n case \"$1\" in\n+init)\n+\techo \"capability=get_git_obj\"\n+\techo \"capability=have\"\n+\t;;\n have)\n \tgit cat-file --batch-check --batch-all-objects |\n \tawk '{print $1 \" \" $3 \" \" $2}'\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328193","messageId":"20170916080731.13925-17-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 16/40] Add t0410 to test external ODB transfer","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:07Z","receivedAt":"2017-09-16T08:08:20Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n t/t0410-transfer-e-odb.sh | 144 ++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 144 insertions(+)\n create mode 100755 t/t0410-transfer-e-odb.sh\n\ndiff --git a/t/t0410-transfer-e-odb.sh b/t/t0410-transfer-e-odb.sh\nnew file mode 100755\nindex 0000000000..065ec7d759\n--- /dev/null\n+++ b/t/t0410-transfer-e-odb.sh\n@@ -0,0 +1,144 @@\n+#!/bin/sh\n+\n+test_description='basic tests for transfering external ODBs'\n+\n+. ./test-lib.sh\n+\n+ORIG_SOURCE=\"$PWD/.git\"\n+export ORIG_SOURCE\n+\n+ALT_SOURCE1=\"$PWD/alt-repo1/.git\"\n+export ALT_SOURCE1\n+write_script odb-helper1 <<\\EOF\n+die() {\n+\tprintf >&2 \"%s\\n\" \"$@\"\n+\texit 1\n+}\n+GIT_DIR=$ALT_SOURCE1; export GIT_DIR\n+case \"$1\" in\n+init)\n+\techo \"capability=get_git_obj\"\n+\techo \"capability=have\"\n+\t;;\n+have)\n+\tgit cat-file --batch-check --batch-all-objects |\n+\tawk '{print $1 \" \" $3 \" \" $2}'\n+\t;;\n+get_git_obj)\n+\tcat \"$GIT_DIR\"/objects/$(echo $2 | sed 's#..#&/#')\n+\t;;\n+put_raw_obj)\n+\tsha1=\"$2\"\n+\tsize=\"$3\"\n+\tkind=\"$4\"\n+\twriten=$(git hash-object -w -t \"$kind\" --stdin)\n+\ttest \"$writen\" = \"$sha1\" || die \"bad sha1 passed '$sha1' vs writen '$writen'\"\n+\tref_hash=$(echo \"$sha1 $size $kind\" | GIT_DIR=$ORIG_SOURCE GIT_NO_EXTERNAL_ODB=1 git hash-object -w -t blob --stdin) || exit\n+\tGIT_DIR=$ORIG_SOURCE git update-ref refs/odbs/magic/\"$sha1\" \"$ref_hash\"\n+\t;;\n+*)\n+\tdie \"unknown command '$1'\"\n+\t;;\n+esac\n+EOF\n+HELPER1=\"\\\"$PWD\\\"/odb-helper1\"\n+\n+OTHER_SOURCE=\"$PWD/.git\"\n+export OTHER_SOURCE\n+\n+ALT_SOURCE2=\"$PWD/alt-repo2/.git\"\n+export ALT_SOURCE2\n+write_script odb-helper2 <<\\EOF\n+die() {\n+\tprintf >&2 \"%s\\n\" \"$@\"\n+\texit 1\n+}\n+GIT_DIR=$ALT_SOURCE2; export GIT_DIR\n+case \"$1\" in\n+init)\n+\techo \"capability=get_git_obj\"\n+\techo \"capability=have\"\n+\t;;\n+have)\n+\tGIT_DIR=$OTHER_SOURCE git for-each-ref --format='%(objectname)' refs/odbs/magic/ | GIT_DIR=$OTHER_SOURCE xargs git show\n+\t;;\n+get_git_obj)\n+\tOBJ_FILE=\"$GIT_DIR\"/objects/$(echo $2 | sed 's#..#&/#')\n+\tif ! test -f \"$OBJ_FILE\"\n+\tthen\n+\t\t# \"Download\" the missing object by copying it from alt-repo1\n+\t\tOBJ_DIR=$(echo $2 | sed 's/\\(..\\).*/\\1/')\n+\t\tOBJ_BASE=$(basename \"$OBJ_FILE\")\n+\t\tALT_OBJ_DIR1=\"$ALT_SOURCE1/objects/$OBJ_DIR\"\n+\t\tALT_OBJ_DIR2=\"$ALT_SOURCE2/objects/$OBJ_DIR\"\n+\t\tmkdir -p \"$ALT_OBJ_DIR2\" || die \"Could not mkdir '$ALT_OBJ_DIR2'\"\n+\t\tOBJ_SRC=\"$ALT_OBJ_DIR1/$OBJ_BASE\"\n+\t\tcp \"$OBJ_SRC\" \"$ALT_OBJ_DIR2\" ||\n+\t\tdie \"Could not cp '$OBJ_SRC' into '$ALT_OBJ_DIR2'\"\n+\tfi\n+\tcat \"$OBJ_FILE\" || die \"Could not cat '$OBJ_FILE'\"\n+\t;;\n+put_raw_obj)\n+\tsha1=\"$2\"\n+\tsize=\"$3\"\n+\tkind=\"$4\"\n+\twriten=$(git hash-object -w -t \"$kind\" --stdin)\n+\ttest \"$writen\" = \"$sha1\" || die \"bad sha1 passed '$sha1' vs writen '$writen'\"\n+\tref_hash=$(echo \"$sha1 $size $kind\" | GIT_DIR=$OTHER_SOURCE GIT_NO_EXTERNAL_ODB=1 git hash-object -w -t blob --stdin) || exit\n+\tGIT_DIR=$OTHER_SOURCE git update-ref refs/odbs/magic/\"$sha1\" \"$ref_hash\"\n+\t;;\n+*)\n+\tdie \"unknown command '$1'\"\n+\t;;\n+esac\n+EOF\n+HELPER2=\"\\\"$PWD\\\"/odb-helper2\"\n+\n+test_expect_success 'setup first alternate repo' '\n+\tgit init alt-repo1 &&\n+\ttest_commit zero &&\n+\tgit config odb.magic.scriptCommand \"$HELPER1\"\n+'\n+\n+test_expect_success 'setup other repo and its alternate repo' '\n+\tgit init other-repo &&\n+\tgit init alt-repo2 &&\n+\t(cd other-repo &&\n+\t git remote add origin .. &&\n+\t git pull origin master &&\n+\t git checkout master &&\n+\t git log)\n+'\n+\n+test_expect_success 'new blobs are put in first object store' '\n+\ttest_commit one &&\n+\thash1=$(git ls-tree HEAD | grep one.t | cut -f1 | cut -d\\ -f3) &&\n+\tcontent=$(cd alt-repo1 && git show \"$hash1\") &&\n+\ttest \"$content\" = \"one\" &&\n+\ttest_commit two &&\n+\thash2=$(git ls-tree HEAD | grep two.t | cut -f1 | cut -d\\ -f3) &&\n+\tcontent=$(cd alt-repo1 && git show \"$hash2\") &&\n+\ttest \"$content\" = \"two\"\n+'\n+\n+test_expect_success 'other repo gets the blobs from object store' '\n+\t(cd other-repo &&\n+\t git fetch origin \"refs/odbs/magic/*:refs/odbs/magic/*\" &&\n+\t test_must_fail git cat-file blob \"$hash1\" &&\n+\t test_must_fail git cat-file blob \"$hash2\" &&\n+\t git config odb.magic.scriptCommand \"$HELPER2\" &&\n+\t git cat-file blob \"$hash1\" &&\n+\t git cat-file blob \"$hash2\"\n+\t)\n+'\n+\n+test_expect_success 'other repo gets everything else' '\n+\t(cd other-repo &&\n+\t git fetch origin &&\n+\t content=$(git show \"$hash1\") &&\n+\t test \"$content\" = \"one\" &&\n+\t content=$(git show \"$hash2\") &&\n+\t test \"$content\" = \"two\")\n+'\n+\n+test_done\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328194","messageId":"20170916080731.13925-15-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 14/40] t0400: add test for external odb write support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:05Z","receivedAt":"2017-09-16T08:08:23Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n t/t0400-external-odb.sh | 8 ++++++++\n 1 file changed, 8 insertions(+)\n\ndiff --git a/t/t0400-external-odb.sh b/t/t0400-external-odb.sh\nindex f9e6ea1015..03df030461 100755\n--- a/t/t0400-external-odb.sh\n+++ b/t/t0400-external-odb.sh\n@@ -71,4 +71,12 @@ test_expect_success 'helper can add objects to alt repo' '\n \ttest \"$size\" -eq \"$alt_size\"\n '\n \n+test_expect_success 'commit adds objects to alt repo' '\n+\ttest_config odb.magic.scriptCommand \"$HELPER\" &&\n+\ttest_commit three &&\n+\thash3=$(git ls-tree HEAD | grep three.t | cut -f1 | cut -d\\ -f3) &&\n+\tcontent=$(cd alt-repo && git show \"$hash3\") &&\n+\ttest \"$content\" = \"three\"\n+'\n+\n test_done\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328195","messageId":"20170916080731.13925-20-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 19/40] lib-httpd: add list.sh","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:10Z","receivedAt":"2017-09-16T08:08:26Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This cgi script can list Git objects that have been uploaded as\nfiles to an apache web server. This script can also retrieve\nthe content of each of these files.\n\nThis will help make apache work as an external object database.\n\nSigned-off-by: Christian Couder \n---\n t/lib-httpd.sh | 1 +\n t/lib-httpd/list.sh | 41 +++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 42 insertions(+)\n create mode 100644 t/lib-httpd/list.sh\n\ndiff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\nindex d80b004549..f31ea261f5 100644\n--- a/t/lib-httpd.sh\n+++ b/t/lib-httpd.sh\n@@ -133,6 +133,7 @@ prepare_httpd() {\n \tinstall_script broken-smart-http.sh\n \tinstall_script error.sh\n \tinstall_script upload.sh\n+\tinstall_script list.sh\n \n \tln -s \"$LIB_HTTPD_MODULE_PATH\" \"$HTTPD_ROOT_PATH/modules\"\n \ndiff --git a/t/lib-httpd/list.sh b/t/lib-httpd/list.sh\nnew file mode 100644\nindex 0000000000..b6d6c29a2f\n--- /dev/null\n+++ b/t/lib-httpd/list.sh\n@@ -0,0 +1,41 @@\n+#!/bin/sh\n+\n+FILES_DIR=\"www/files\"\n+\n+OLDIFS=\"$IFS\"\n+IFS='&'\n+set -- $QUERY_STRING\n+IFS=\"$OLDIFS\"\n+\n+while test $# -gt 0\n+do\n+\tkey=${1%%=*}\n+\tval=${1#*=}\n+\n+\tcase \"$key\" in\n+\t\"sha1\") sha1=\"$val\" ;;\n+\t*) echo >&2 \"unknown key '$key'\" ;;\n+\tesac\n+\n+\tshift\n+done\n+\n+if test -d \"$FILES_DIR\"\n+then\n+\tif test -z \"$sha1\"\n+\tthen\n+\t\techo 'Status: 200 OK'\n+\t\techo\n+\t\tls \"$FILES_DIR\" | tr '-' ' '\n+\telse\n+\t\tif test -f \"$FILES_DIR/$sha1\"-*\n+\t\tthen\n+\t\t\techo 'Status: 200 OK'\n+\t\t\techo\n+\t\t\tcat \"$FILES_DIR/$sha1\"-*\n+\t\telse\n+\t\t\techo 'Status: 404 Not Found'\n+\t\t\techo\n+\t\tfi\n+\tfi\n+fi\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328196","messageId":"20170916080731.13925-23-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 22/40] pack-objects: don't pack objects in external odbs","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:13Z","receivedAt":"2017-09-16T08:08:29Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Objects managed by an external ODB should not be put into\npack files. They should be transfered using other mechanism\nthat can be specific to the external odb.\n\nSigned-off-by: Christian Couder \n---\n builtin/pack-objects.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex a57b4f058d..db5e225d5a 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -26,6 +26,7 @@\n #include \"argv-array.h\"\n #include \"mru.h\"\n #include \"packfile.h\"\n+#include \"external-odb.h\"\n \n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [...] [< | < ]\"),\n@@ -1012,6 +1013,9 @@ static int want_object_in_pack(const unsigned char *sha1,\n \t\t\treturn want;\n \t}\n \n+\tif (external_odb_has_object(sha1))\n+\t\treturn 0;\n+\n \tfor (entry = packed_git_mru->head; entry; entry = entry->next) {\n \t\tstruct packed_git *p = entry->item;\n \t\toff_t offset;\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328197","messageId":"20170916080731.13925-27-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 26/40] odb-helper: add init_object_process()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:17Z","receivedAt":"2017-09-16T08:08:35Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: Ben Peart \n\nThis adds the infrastructure to launch and use long running\nsub-processes as external odb helpers.\n\nFor now only the 'init' and 'get_direct' capabilities are\nsupported with sub-processes.\n\nSigned-off-by: Christian Couder \n---\n external-odb.c | 52 ++++---\n odb-helper.c | 469 +++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n sha1_file.c | 56 +++++--\n 3 files changed, 523 insertions(+), 54 deletions(-)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex ccca67eff5..084cd32e0b 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -67,32 +67,11 @@ const char *external_odb_root(void)\n \treturn root;\n }\n \n-int external_odb_has_object(const unsigned char *sha1)\n-{\n-\tstruct odb_helper *o;\n-\n-\tif (!use_external_odb)\n-\t\treturn 0;\n-\n-\texternal_odb_init();\n-\n-\tfor (o = helpers; o; o = o->next) {\n-\t\tif (!(o->supported_capabilities & ODB_HELPER_CAP_HAVE))\n-\t\t\treturn 1;\n-\t\tif (odb_helper_has_object(o, sha1))\n-\t\t\treturn 1;\n-\t}\n-\treturn 0;\n-}\n-\n-int external_odb_get_object(const unsigned char *sha1)\n+static int external_odb_do_get_object(const unsigned char *sha1)\n {\n \tstruct odb_helper *o;\n \tconst char *path;\n \n-\tif (!external_odb_has_object(sha1))\n-\t\treturn -1;\n-\n \tpath = sha1_file_name_alt(external_odb_root(), sha1);\n \tsafe_create_leading_directories_const(path);\n \tprepare_external_alt_odb();\n@@ -147,6 +126,35 @@ int external_odb_get_direct(const unsigned char *sha1)\n \treturn -1;\n }\n \n+int external_odb_has_object(const unsigned char *sha1)\n+{\n+\tstruct odb_helper *o;\n+\n+\tif (!use_external_odb)\n+\t\treturn 0;\n+\n+\texternal_odb_init();\n+\n+\tfor (o = helpers; o; o = o->next) {\n+\t\tif (!(o->supported_capabilities & ODB_HELPER_CAP_HAVE)) {\n+\t\t\tif (o->supported_capabilities & ODB_HELPER_CAP_GET_DIRECT)\n+\t\t\t\treturn 1;\n+\t\t\treturn !external_odb_do_get_object(sha1);\n+\t\t}\n+\t\tif (odb_helper_has_object(o, sha1))\n+\t\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+int external_odb_get_object(const unsigned char *sha1)\n+{\n+\tif (!external_odb_has_object(sha1))\n+\t\treturn -1;\n+\n+\treturn external_odb_do_get_object(sha1);\n+}\n+\n int external_odb_put_object(const void *buf, size_t len,\n \t\t\t const char *type, unsigned char *sha1)\n {\ndiff --git a/odb-helper.c b/odb-helper.c\nindex 22728200e3..3148bcfa15 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -4,6 +4,22 @@\n #include \"odb-helper.h\"\n #include \"run-command.h\"\n #include \"sha1-lookup.h\"\n+#include \"sub-process.h\"\n+#include \"pkt-line.h\"\n+#include \"sigchain.h\"\n+\n+struct object_process {\n+\tstruct subprocess_entry subprocess;\n+\tunsigned int supported_capabilities;\n+};\n+\n+static struct hashmap subprocess_map;\n+\n+static int check_object_process_status(int fd, struct strbuf *status)\n+{\n+\tsubprocess_read_status(fd, status);\n+\treturn strcmp(status->buf, \"success\");\n+}\n \n static void parse_capabilities(char *cap_buf,\n \t\t\t unsigned int *supported_capabilities,\n@@ -39,6 +55,384 @@ static void parse_capabilities(char *cap_buf,\n \tstring_list_clear(&cap_list, 0);\n }\n \n+static int start_object_process_fn(struct subprocess_entry *subprocess)\n+{\n+\tstatic int versions[] = {1, 0};\n+\tstatic struct subprocess_capability capabilities[] = {\n+\t\t{ \"get_git_obj\", ODB_HELPER_CAP_GET_GIT_OBJ },\n+\t\t{ \"get_raw_obj\", ODB_HELPER_CAP_GET_RAW_OBJ },\n+\t\t{ \"get_direct\", ODB_HELPER_CAP_GET_DIRECT },\n+\t\t{ \"put_git_obj\", ODB_HELPER_CAP_PUT_GIT_OBJ },\n+\t\t{ \"put_raw_obj\", ODB_HELPER_CAP_PUT_RAW_OBJ },\n+\t\t{ \"put_direct\", ODB_HELPER_CAP_PUT_DIRECT },\n+\t\t{ \"have\", ODB_HELPER_CAP_HAVE },\n+\t\t{ NULL, 0 }\n+\t};\n+\tstruct object_process *entry = (struct object_process *)subprocess;\n+\treturn subprocess_handshake(subprocess, \"git-read-object\", versions, NULL,\n+\t\t\t\t capabilities,\n+\t\t\t\t &entry->supported_capabilities);\n+}\n+\n+static struct object_process *launch_object_process(struct odb_helper *o,\n+\t\t\t\t\t\t unsigned int capability)\n+{\n+\tstruct object_process *entry = NULL;\n+\n+\tif (!subprocess_map.tablesize)\n+\t\thashmap_init(&subprocess_map, (hashmap_cmp_fn) cmd2process_cmp, NULL, 0);\n+\telse\n+\t\tentry = (struct object_process *)subprocess_find_entry(&subprocess_map, o->cmd);\n+\n+\tfflush(NULL);\n+\n+\tif (!entry) {\n+\t\tentry = xmalloc(sizeof(*entry));\n+\t\tentry->supported_capabilities = 0;\n+\n+\t\tif (subprocess_start(&subprocess_map, &entry->subprocess, o->cmd, start_object_process_fn)) {\n+\t\t\terror(\"Could not launch process for cmd '%s'\", o->cmd);\n+\t\t\tfree(entry);\n+\t\t\treturn NULL;\n+\t\t}\n+\t}\n+\n+\to->supported_capabilities = entry->supported_capabilities;\n+\n+\tif (capability && !(capability & entry->supported_capabilities)) {\n+\t\terror(\"The cmd '%s' does not support capability '%d'\", o->cmd, capability);\n+\t\treturn NULL;\n+\t}\n+\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\n+\treturn entry;\n+}\n+\n+static int check_object_process_error(int err,\n+\t\t\t\t const char *status,\n+\t\t\t\t struct object_process *entry,\n+\t\t\t\t const char *cmd,\n+\t\t\t\t unsigned int capability)\n+{\n+\tsigchain_pop(SIGPIPE);\n+\n+\tif (!err)\n+\t\treturn 0;\n+\n+\tif (!strcmp(status, \"error\")) {\n+\t\t/* The process signaled a problem with the file. */\n+\t} else if (!strcmp(status, \"notfound\")) {\n+\t\t/* Object was not found */\n+\t\terr = -1;\n+\t} else if (!strcmp(status, \"abort\")) {\n+\t\t/*\n+\t\t * The process signaled a permanent problem. Don't try to read\n+\t\t * objects with the same command for the lifetime of the current\n+\t\t * Git process.\n+\t\t */\n+\t\tif (capability)\n+\t\t\tentry->supported_capabilities &= ~capability;\n+\t} else {\n+\t\t/*\n+\t\t * Something went wrong with the read-object process.\n+\t\t * Force shutdown and restart if needed.\n+\t\t */\n+\t\terror(\"external object process '%s' failed\", cmd);\n+\t\tsubprocess_stop(&subprocess_map, &entry->subprocess);\n+\t\tfree(entry);\n+\t}\n+\n+\treturn err;\n+}\n+\n+static int send_init_packets(struct object_process *entry,\n+\t\t\t struct strbuf *status)\n+{\n+\tstruct child_process *process = &entry->subprocess.process;\n+\n+\treturn packet_write_fmt_gently(process->in, \"command=init\\n\") ||\n+\t\tpacket_flush_gently(process->in) ||\n+\t\tcheck_object_process_status(process->out, status);\n+}\n+\n+static int init_object_process(struct odb_helper *o)\n+{\n+\tint err;\n+\tstruct strbuf status = STRBUF_INIT;\n+\tstruct object_process *entry = launch_object_process(o, 0);\n+\tif (!entry)\n+\t\treturn -1;\n+\n+\terr = send_init_packets(entry, &status);\n+\n+\treturn check_object_process_error(err, status.buf, entry,\n+\t\t\t\t\t o->cmd, 0);\n+}\n+\n+static ssize_t read_packetized_raw_object_to_fd(struct odb_helper *o,\n+\t\t\t\t\t\tconst unsigned char *sha1,\n+\t\t\t\t\t\tint fd_in, int fd_out)\n+{\n+\tssize_t total_read = 0;\n+\tunsigned long total_got = 0;\n+\tint packet_len;\n+\n+\tchar hdr[32];\n+\tint hdrlen;\n+\n+\tint ret = Z_STREAM_END;\n+\tunsigned char compressed[4096];\n+\tgit_zstream stream;\n+\tgit_SHA_CTX hash;\n+\tunsigned char real_sha1[20];\n+\n+\toff_t size;\n+\tenum object_type type;\n+\tconst char *s;\n+\tint pkt_size;\n+\tchar *size_buf;\n+\n+\tsize_buf = packet_read_line(fd_in, &pkt_size);\n+\tif (!skip_prefix(size_buf, \"size=\", &s))\n+\t\treturn error(\"odb helper '%s' did not send size of plain object\", o->name);\n+\tsize = strtoumax(s, NULL, 10);\n+\tif (!skip_prefix(packet_read_line(fd_in, NULL), \"kind=\", &s))\n+\t\treturn error(\"odb helper '%s' did not send kind of plain object\", o->name);\n+\t/* Check if the object is not available */\n+\tif (!strcmp(s, \"none\"))\n+\t\treturn -1;\n+\ttype = type_from_string_gently(s, strlen(s), 1);\n+\tif (type < 0)\n+\t\treturn error(\"odb helper '%s' sent bad type '%s'\", o->name, s);\n+\n+\t/* Set it up */\n+\tgit_deflate_init(&stream, zlib_compression_level);\n+\tstream.next_out = compressed;\n+\tstream.avail_out = sizeof(compressed);\n+\tgit_SHA1_Init(&hash);\n+\n+\t/* First header.. */\n+\thdrlen = xsnprintf(hdr, sizeof(hdr), \"%s %lu\", typename(type), size) + 1;\n+\tstream.next_in = (unsigned char *)hdr;\n+\tstream.avail_in = hdrlen;\n+\twhile (git_deflate(&stream, 0) == Z_OK)\n+\t\t; /* nothing */\n+\tgit_SHA1_Update(&hash, hdr, hdrlen);\n+\n+\tfor (;;) {\n+\t\t/* packet_read() writes a '\\0' extra byte at the end */\n+\t\tchar buf[LARGE_PACKET_DATA_MAX + 1];\n+\n+\t\tpacket_len = packet_read(fd_in, NULL, NULL,\n+\t\t\tbuf, LARGE_PACKET_DATA_MAX + 1,\n+\t\t\tPACKET_READ_GENTLE_ON_EOF);\n+\n+\t\tif (packet_len <= 0)\n+\t\t\tbreak;\n+\n+\t\ttotal_got += packet_len;\n+\n+\t\t/* Then the data itself.. */\n+\t\tstream.next_in = (void *)buf;\n+\t\tstream.avail_in = packet_len;\n+\t\tdo {\n+\t\t\tunsigned char *in0 = stream.next_in;\n+\t\t\tret = git_deflate(&stream, Z_FINISH);\n+\t\t\tgit_SHA1_Update(&hash, in0, stream.next_in - in0);\n+\t\t\twrite_or_die(fd_out, compressed, stream.next_out - compressed);\n+\t\t\tstream.next_out = compressed;\n+\t\t\tstream.avail_out = sizeof(compressed);\n+\t\t} while (ret == Z_OK);\n+\n+\t\ttotal_read += packet_len;\n+\t}\n+\n+\tif (packet_len < 0) {\n+\t\terror(\"unable to read from odb helper '%s': %s\",\n+\t\t o->name, strerror(errno));\n+\t\tgit_deflate_end(&stream);\n+\t\treturn packet_len;\n+\t}\n+\n+\tif (ret != Z_STREAM_END) {\n+\t\twarning(\"bad zlib data from odb helper '%s' for %s\",\n+\t\t\to->name, sha1_to_hex(sha1));\n+\t\treturn -1;\n+\t}\n+\n+\tret = git_deflate_end_gently(&stream);\n+\tif (ret != Z_OK) {\n+\t\twarning(\"deflateEnd on object %s from odb helper '%s' failed (%d)\",\n+\t\t\tsha1_to_hex(sha1), o->name, ret);\n+\t\treturn -1;\n+\t}\n+\tgit_SHA1_Final(real_sha1, &hash);\n+\tif (hashcmp(sha1, real_sha1)) {\n+\t\twarning(\"sha1 mismatch from odb helper '%s' for %s (got %s)\",\n+\t\t\to->name, sha1_to_hex(sha1), sha1_to_hex(real_sha1));\n+\t\treturn -1;\n+\t}\n+\tif (total_got != size) {\n+\t\twarning(\"size mismatch from odb helper '%s' for %s (%lu != %lu)\",\n+\t\t\to->name, sha1_to_hex(sha1), total_got, size);\n+\t\treturn -1;\n+\t}\n+\n+\treturn total_read;\n+}\n+\n+static ssize_t read_packetized_git_object_to_fd(struct odb_helper *o,\n+\t\t\t\t\t\tconst unsigned char *sha1,\n+\t\t\t\t\t\tint fd_in, int fd_out)\n+{\n+\tssize_t total_read = 0;\n+\tunsigned long total_got = 0;\n+\tint packet_len;\n+\tgit_zstream stream;\n+\tint zret = Z_STREAM_END;\n+\tgit_SHA_CTX hash;\n+\tunsigned char real_sha1[20];\n+\tstruct strbuf header = STRBUF_INIT;\n+\tunsigned long hdr_size;\n+\n+\tmemset(&stream, 0, sizeof(stream));\n+\tgit_inflate_init(&stream);\n+\tgit_SHA1_Init(&hash);\n+\n+\tfor (;;) {\n+\t\t/* packet_read() writes a '\\0' extra byte at the end */\n+\t\tchar buf[LARGE_PACKET_DATA_MAX + 1];\n+\n+\t\tpacket_len = packet_read(fd_in, NULL, NULL,\n+\t\t\tbuf, LARGE_PACKET_DATA_MAX + 1,\n+\t\t\tPACKET_READ_GENTLE_ON_EOF);\n+\n+\t\tif (packet_len <= 0)\n+\t\t\tbreak;\n+\n+\t\twrite_or_die(fd_out, buf, packet_len);\n+\n+\t\tstream.next_in = (unsigned char *)buf;\n+\t\tstream.avail_in = packet_len;\n+\t\tdo {\n+\t\t\tunsigned char inflated[4096];\n+\t\t\tunsigned long got;\n+\n+\t\t\tstream.next_out = inflated;\n+\t\t\tstream.avail_out = sizeof(inflated);\n+\t\t\tzret = git_inflate(&stream, Z_SYNC_FLUSH);\n+\t\t\tgot = sizeof(inflated) - stream.avail_out;\n+\n+\t\t\tgit_SHA1_Update(&hash, inflated, got);\n+\t\t\t/* skip header when counting size */\n+\t\t\tif (!total_got) {\n+\t\t\t\tconst unsigned char *p = memchr(inflated, '\\0', got);\n+\t\t\t\tif (p) {\n+\t\t\t\t\tunsigned long hdr_last = p - inflated + 1;\n+\t\t\t\t\tstrbuf_add(&header, inflated, hdr_last);\n+\t\t\t\t\tgot -= hdr_last;\n+\t\t\t\t} else {\n+\t\t\t\t\tstrbuf_add(&header, inflated, got);\n+\t\t\t\t\tgot = 0;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t\ttotal_got += got;\n+\t\t} while (stream.avail_in && zret == Z_OK);\n+\n+\t\ttotal_read += packet_len;\n+\t}\n+\n+\tgit_inflate_end(&stream);\n+\n+\tif (packet_len < 0)\n+\t\treturn packet_len;\n+\n+\tgit_SHA1_Final(real_sha1, &hash);\n+\n+\tif (zret != Z_STREAM_END) {\n+\t\twarning(\"bad zlib data from odb helper '%s' for %s\",\n+\t\t\to->name, sha1_to_hex(sha1));\n+\t\treturn -1;\n+\t}\n+\tif (hashcmp(real_sha1, sha1)) {\n+\t\twarning(\"sha1 mismatch from odb helper '%s' for %s (got %s)\",\n+\t\t\to->name, sha1_to_hex(sha1), sha1_to_hex(real_sha1));\n+\t\treturn -1;\n+\t}\n+\tif (parse_sha1_header(header.buf, &hdr_size) < 0) {\n+\t\twarning(\"could not parse header from odb helper '%s' for %s\",\n+\t\t\to->name, sha1_to_hex(sha1));\n+\t\treturn -1;\n+\t}\n+\tif (total_got != hdr_size) {\n+\t\twarning(\"size mismatch from odb helper '%s' for %s (%lu != %lu)\",\n+\t\t\to->name, sha1_to_hex(sha1), total_got, hdr_size);\n+\t\treturn -1;\n+\t}\n+\n+\treturn total_read;\n+}\n+\n+static int send_get_packets(struct odb_helper *o,\n+\t\t\t struct object_process *entry,\n+\t\t\t const unsigned char *sha1,\n+\t\t\t int fd,\n+\t\t\t unsigned int *cur_cap,\n+\t\t\t struct strbuf *status)\n+{\n+\tconst char *instruction;\n+\tint err;\n+\tstruct child_process *process = &entry->subprocess.process;\n+\n+\tif (entry->supported_capabilities & ODB_HELPER_CAP_GET_GIT_OBJ) {\n+\t\t*cur_cap = ODB_HELPER_CAP_GET_GIT_OBJ;\n+\t\tinstruction = \"get_git_obj\";\n+\t} else if (entry->supported_capabilities & ODB_HELPER_CAP_GET_RAW_OBJ) {\n+\t\t*cur_cap = ODB_HELPER_CAP_GET_RAW_OBJ;\n+\t\tinstruction = \"get_raw_obj\";\n+\t} else if (entry->supported_capabilities & ODB_HELPER_CAP_GET_DIRECT) {\n+\t\t*cur_cap = ODB_HELPER_CAP_GET_DIRECT;\n+\t\tinstruction = \"get_direct\";\n+\t} else {\n+\t\tBUG(\"No known ODB_HELPER_CAP_GET_XXX capability!\");\n+\t}\n+\n+\terr = packet_write_fmt_gently(process->in, \"command=%s\\n\", instruction);\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = packet_write_fmt_gently(process->in, \"sha1=%s\\n\", sha1_to_hex(sha1));\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = packet_flush_gently(process->in);\n+\tif (err)\n+\t\treturn err;\n+\n+\tif (entry->supported_capabilities & ODB_HELPER_CAP_GET_RAW_OBJ)\n+\t\terr = read_packetized_raw_object_to_fd(o, sha1, process->out, fd) < 0;\n+\telse if (entry->supported_capabilities & ODB_HELPER_CAP_GET_GIT_OBJ)\n+\t\terr = read_packetized_git_object_to_fd(o, sha1, process->out, fd) < 0;\n+\n+\treturn check_object_process_status(process->out, status);\n+}\n+\n+static int get_object_process(struct odb_helper *o, const unsigned char *sha1, int fd)\n+{\n+\tint err;\n+\tstruct strbuf status = STRBUF_INIT;\n+\tunsigned int cur_cap = 0;\n+\tstruct object_process *entry = launch_object_process(o, 0);\n+\tif (!entry)\n+\t\treturn -1;\n+\n+\terr = send_get_packets(o, entry, sha1, fd, &cur_cap, &status);\n+\n+\treturn check_object_process_error(err, status.buf, entry,\n+\t\t\t\t\t o->cmd, cur_cap);\n+}\n+\n struct odb_helper *odb_helper_new(const char *name, int namelen)\n {\n \tstruct odb_helper *o;\n@@ -117,15 +511,12 @@ static int odb_helper_finish(struct odb_helper *o,\n \treturn 0;\n }\n \n-int odb_helper_init(struct odb_helper *o)\n+static int init_object_script(struct odb_helper *o)\n {\n \tstruct odb_helper_cmd cmd;\n \tFILE *fh;\n \tstruct strbuf line = STRBUF_INIT;\n \n-\tif (!o->script_mode)\n-\t\treturn 0;\n-\n \tif (odb_helper_start(o, &cmd, 0, \"init\") < 0)\n \t\treturn -1;\n \n@@ -140,6 +531,21 @@ int odb_helper_init(struct odb_helper *o)\n \treturn 0;\n }\n \n+int odb_helper_init(struct odb_helper *o)\n+{\n+\tint res;\n+\tuint64_t start = getnanotime();\n+\n+\tif (o->script_mode)\n+\t\tres = init_object_script(o);\n+\telse\n+\t\tres = init_object_process(o);\n+\n+\ttrace_performance_since(start, \"odb_helper_init\");\n+\n+\treturn 0;\n+}\n+\n static int parse_object_line(struct odb_helper_object *o, const char *line)\n {\n \tchar *end;\n@@ -434,28 +840,42 @@ static int odb_helper_get_git_object(struct odb_helper *o,\n \treturn 0;\n }\n \n-int odb_helper_get_direct(struct odb_helper *o,\n-\t\t\t const unsigned char *sha1)\n+static int get_direct_script(struct odb_helper *o, const unsigned char *sha1)\n {\n-\tstruct odb_helper_object *obj;\n \tstruct odb_helper_cmd cmd;\n \n-\tobj = odb_helper_lookup(o, sha1);\n-\tif (!obj)\n-\t\treturn -1;\n-\n \tif (odb_helper_start(o, &cmd, 0, \"get_direct %s\", sha1_to_hex(sha1)) < 0)\n \t\treturn -1;\n-\n \tif (odb_helper_finish(o, &cmd))\n \t\treturn -1;\n-\n \treturn 0;\n }\n \n-int odb_helper_get_object(struct odb_helper *o,\n-\t\t\t const unsigned char *sha1,\n-\t\t\t int fd)\n+int odb_helper_get_direct(struct odb_helper *o,\n+\t\t\t const unsigned char *sha1)\n+{\n+\tint res;\n+\tuint64_t start;\n+\n+\tif (o->supported_capabilities & ODB_HELPER_CAP_HAVE) {\n+\t\tstruct odb_helper_object *obj = odb_helper_lookup(o, sha1);\n+\t\tif (!obj)\n+\t\t\treturn -1;\n+\t}\n+\n+\tstart = getnanotime();\n+\n+\tif (o->script_mode)\n+\t\tres = get_direct_script(o, sha1);\n+\telse\n+\t\tres = get_object_process(o, sha1, -1);\n+\n+\ttrace_performance_since(start, \"odb_helper_get_direct\");\n+\n+\treturn res;\n+}\n+\n+static int get_object_script(struct odb_helper *o, const unsigned char *sha1, int fd)\n {\n \tif (o->supported_capabilities & ODB_HELPER_CAP_GET_GIT_OBJ)\n \t\treturn odb_helper_get_git_object(o, sha1, fd);\n@@ -467,6 +887,23 @@ int odb_helper_get_object(struct odb_helper *o,\n \tBUG(\"invalid get capability (capabilities: '%d')\", o->supported_capabilities);\n }\n \n+int odb_helper_get_object(struct odb_helper *o,\n+\t\t\t const unsigned char *sha1,\n+\t\t\t int fd)\n+{\n+\tint res;\n+\tuint64_t start = getnanotime();\n+\n+\tif (o->script_mode)\n+\t\tres = get_object_script(o, sha1, fd);\n+\telse\n+\t\tres = get_object_process(o, sha1, fd);\n+\n+\ttrace_performance_since(start, \"odb_helper_get_object\");\n+\n+\treturn res;\n+}\n+\n int odb_helper_put_object(struct odb_helper *o,\n \t\t\t const void *buf, size_t len,\n \t\t\t const char *type, unsigned char *sha1)\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 7b2a0f64fa..c5b6d89b97 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -675,7 +675,17 @@ int check_and_freshen_file(const char *fn, int freshen)\n \n static int check_and_freshen_local(const unsigned char *sha1, int freshen)\n {\n-\treturn check_and_freshen_file(sha1_file_name(sha1), freshen);\n+\tint ret;\n+\tint tried_hook = 0;\n+\n+retry:\n+\tret = check_and_freshen_file(sha1_file_name(sha1), freshen);\n+\tif (!ret && !tried_hook) {\n+\t\ttried_hook = 1;\n+\t\tif (!external_odb_get_direct(sha1))\n+\t\t\tgoto retry;\n+\t}\n+\treturn ret;\n }\n \n static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)\n@@ -1190,20 +1200,11 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \treturn (status < 0) ? status : 0;\n }\n \n-int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi, unsigned flags)\n+static int find_cached_or_packed(const unsigned char *sha1, struct object_info *oi,\n+\t\t\t\t unsigned flags, struct pack_entry *e, int retry)\n {\n-\tstatic struct object_info blank_oi = OBJECT_INFO_INIT;\n-\tstruct pack_entry e;\n-\tint rtype;\n-\tconst unsigned char *real = (flags & OBJECT_INFO_LOOKUP_REPLACE) ?\n-\t\t\t\t lookup_replace_object(sha1) :\n-\t\t\t\t sha1;\n-\n-\tif (!oi)\n-\t\toi = &blank_oi;\n-\n \tif (!(flags & OBJECT_INFO_SKIP_CACHED)) {\n-\t\tstruct cached_object *co = find_cached_object(real);\n+\t\tstruct cached_object *co = find_cached_object(sha1);\n \t\tif (co) {\n \t\t\tif (oi->typep)\n \t\t\t\t*(oi->typep) = co->type;\n@@ -1222,9 +1223,9 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\t}\n \t}\n \n-\tif (!find_pack_entry(real, &e)) {\n+\tif (!find_pack_entry(sha1, e)) {\n \t\t/* Most likely it's a loose object. */\n-\t\tif (!sha1_loose_object_info(real, oi, flags))\n+\t\tif (!sha1_loose_object_info(sha1, oi, flags))\n \t\t\treturn 0;\n \n \t\t/* Not a loose object; someone else may have just packed it. */\n@@ -1232,10 +1233,33 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\t\treturn -1;\n \t\t} else {\n \t\t\treprepare_packed_git();\n-\t\t\tif (!find_pack_entry(real, &e))\n+\t\t\tif (!find_pack_entry(sha1, e)) {\n+\t\t\t\tif (retry && !external_odb_get_direct(sha1))\n+\t\t\t\t\treturn find_cached_or_packed(sha1, oi, flags, e, 0);\n \t\t\t\treturn -1;\n+\t\t\t}\n \t\t}\n \t}\n+\treturn 1;\n+}\n+\n+int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi, unsigned flags)\n+{\n+\tstatic struct object_info blank_oi = OBJECT_INFO_INIT;\n+\tstruct pack_entry e;\n+\tint rtype;\n+\tenum object_type real_type;\n+\tint res;\n+\tconst unsigned char *real = (flags & OBJECT_INFO_LOOKUP_REPLACE) ?\n+\t\t\t\t lookup_replace_object(sha1) :\n+\t\t\t\t sha1;\n+\n+\tif (!oi)\n+\t\toi = &blank_oi;\n+\n+\tres = find_cached_or_packed(real, oi, flags, &e, 1);\n+\tif (res < 1)\n+\t\treturn res;\n \n \tif (oi == &blank_oi)\n \t\t/*\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328198","messageId":"20170916080731.13925-30-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 29/40] odb-helper: add put_object_process()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:20Z","receivedAt":"2017-09-16T08:08:38Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This adds the infrastructure to send objects to a sub-process\nhandling the communication with an external odb.\n\nFor now we only handle sending raw blobs using the 'put_raw_obj'\ninstruction.\n\nSigned-off-by: Christian Couder \n---\n odb-helper.c | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++---\n 1 file changed, 72 insertions(+), 3 deletions(-)\n\ndiff --git a/odb-helper.c b/odb-helper.c\nindex 3148bcfa15..356f6172d8 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -433,6 +433,58 @@ static int get_object_process(struct odb_helper *o, const unsigned char *sha1, i\n \t\t\t\t\t o->cmd, cur_cap);\n }\n \n+static int send_put_packets(struct object_process *entry,\n+\t\t\t const unsigned char *sha1,\n+\t\t\t const void *buf,\n+\t\t\t size_t len,\n+\t\t\t struct strbuf *status)\n+{\n+\tstruct child_process *process = &entry->subprocess.process;\n+\tint err = packet_write_fmt_gently(process->in, \"command=put_raw_obj\\n\");\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = packet_write_fmt_gently(process->in, \"sha1=%s\\n\", sha1_to_hex(sha1));\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = packet_write_fmt_gently(process->in, \"size=%\"PRIuMAX\"\\n\", len);\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = packet_write_fmt_gently(process->in, \"kind=blob\\n\");\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = packet_flush_gently(process->in);\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = write_packetized_from_buf(buf, len, process->in);\n+\tif (err)\n+\t\treturn err;\n+\n+\treturn check_object_process_status(process->out, status);\n+}\n+\n+static int put_object_process(struct odb_helper *o,\n+\t\t\t const void *buf, size_t len,\n+\t\t\t const char *type, unsigned char *sha1)\n+{\n+\tint err;\n+\tstruct object_process *entry;\n+\tstruct strbuf status = STRBUF_INIT;\n+\n+\tentry = launch_object_process(o, ODB_HELPER_CAP_PUT_RAW_OBJ);\n+\tif (!entry)\n+\t\treturn -1;\n+\n+\terr = send_put_packets(entry, sha1, buf, len, &status);\n+\n+\treturn check_object_process_error(err, status.buf, entry, o->cmd,\n+\t\t\t\t\t ODB_HELPER_CAP_PUT_RAW_OBJ);\n+}\n+\n struct odb_helper *odb_helper_new(const char *name, int namelen)\n {\n \tstruct odb_helper *o;\n@@ -904,9 +956,9 @@ int odb_helper_get_object(struct odb_helper *o,\n \treturn res;\n }\n \n-int odb_helper_put_object(struct odb_helper *o,\n-\t\t\t const void *buf, size_t len,\n-\t\t\t const char *type, unsigned char *sha1)\n+static int put_raw_object_script(struct odb_helper *o,\n+\t\t\t\t const void *buf, size_t len,\n+\t\t\t\t const char *type, unsigned char *sha1)\n {\n \tstruct odb_helper_cmd cmd;\n \n@@ -932,3 +984,20 @@ int odb_helper_put_object(struct odb_helper *o,\n \todb_helper_finish(o, &cmd);\n \treturn 0;\n }\n+\n+int odb_helper_put_object(struct odb_helper *o,\n+\t\t\t const void *buf, size_t len,\n+\t\t\t const char *type, unsigned char *sha1)\n+{\n+\tint res;\n+\tuint64_t start = getnanotime();\n+\n+\tif (o->script_mode)\n+\t\tres = put_raw_object_script(o, buf, len, type, sha1);\n+\telse\n+\t\tres = put_object_process(o, buf, len, type, sha1);\n+\n+\ttrace_performance_since(start, \"odb_helper_put_object\");\n+\n+\treturn res;\n+}\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328199","messageId":"20170916080731.13925-31-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 30/40] Add t0470 to test passing raw objects","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:21Z","receivedAt":"2017-09-16T08:08:41Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n t/t0470-read-object-http-e-odb.sh | 109 ++++++++++++++++++++++++++++++++++++++\n t/t0470/read-object-plain | 83 +++++++++++++++++++++++++++++\n 2 files changed, 192 insertions(+)\n create mode 100755 t/t0470-read-object-http-e-odb.sh\n create mode 100755 t/t0470/read-object-plain\n\ndiff --git a/t/t0470-read-object-http-e-odb.sh b/t/t0470-read-object-http-e-odb.sh\nnew file mode 100755\nindex 0000000000..774528c04f\n--- /dev/null\n+++ b/t/t0470-read-object-http-e-odb.sh\n@@ -0,0 +1,109 @@\n+#!/bin/sh\n+\n+test_description='tests for read-object process passing plain objects to an HTTPD server'\n+\n+. ./test-lib.sh\n+\n+# If we don't specify a port, the current test number will be used\n+# which will not work as it is less than 1024, so it can only be used by root.\n+LIB_HTTPD_PORT=$(expr ${this_test#t} + 12000)\n+\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+\n+start_httpd apache-e-odb.conf\n+\n+PATH=\"$PATH:$TEST_DIRECTORY/t0470\"\n+\n+# odb helper script must see this\n+export HTTPD_URL\n+\n+HELPER=\"read-object-plain\"\n+\n+test_expect_success 'setup repo with a root commit' '\n+\ttest_commit zero\n+'\n+\n+test_expect_success 'setup another repo from the first one' '\n+\tgit init other-repo &&\n+\t(cd other-repo &&\n+\t git remote add origin .. &&\n+\t git pull origin master &&\n+\t git checkout master &&\n+\t git log)\n+'\n+\n+test_expect_success 'setup the helper in the root repo' '\n+\tgit config odb.magic.subprocessCommand \"$HELPER\"\n+'\n+\n+UPLOADFILENAME=\"hello_apache_upload.txt\"\n+\n+UPLOAD_URL=\"$HTTPD_URL/upload/?sha1=$UPLOADFILENAME&size=123&type=blob\"\n+\n+test_expect_success 'can upload a file' '\n+\techo \"Hello Apache World!\" >hello_to_send.txt &&\n+\techo \"How are you?\" >>hello_to_send.txt &&\n+\tcurl --data-binary @hello_to_send.txt --include \"$UPLOAD_URL\" >out_upload\n+'\n+\n+LIST_URL=\"$HTTPD_URL/list/\"\n+\n+test_expect_success 'can list uploaded files' '\n+\tcurl --include \"$LIST_URL\" >out_list &&\n+\tgrep \"$UPLOADFILENAME\" out_list\n+'\n+\n+test_expect_success 'can delete uploaded files' '\n+\tcurl --data \"delete\" --include \"$UPLOAD_URL&delete=1\" >out_delete &&\n+\tcurl --include \"$LIST_URL\" >out_list2 &&\n+\t! grep \"$UPLOADFILENAME\" out_list2\n+'\n+\n+FILES_DIR=\"httpd/www/files\"\n+\n+test_expect_success 'new blobs are transfered to the http server' '\n+\ttest_commit one &&\n+\thash1=$(git ls-tree HEAD | grep one.t | cut -f1 | cut -d\\ -f3) &&\n+\techo \"$hash1-4-blob\" >expected &&\n+\tls \"$FILES_DIR\" >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'blobs can be retrieved from the http server' '\n+\tgit cat-file blob \"$hash1\" &&\n+\tgit log -p >expected\n+'\n+\n+test_expect_success 'update other repo from the first one' '\n+\t(cd other-repo &&\n+\t git fetch origin \"refs/odbs/magic/*:refs/odbs/magic/*\" &&\n+\t test_must_fail git cat-file blob \"$hash1\" &&\n+\t git config odb.magic.subprocesscommand \"$HELPER\" &&\n+\t git cat-file blob \"$hash1\" &&\n+\t git pull origin master)\n+'\n+\n+test_expect_success 'local clone from the first repo' '\n+\tmkdir my-clone &&\n+\t(cd my-clone &&\n+\t git clone .. . &&\n+\t git cat-file blob \"$hash1\")\n+'\n+\n+test_expect_success 'no-local clone from the first repo fails' '\n+\tmkdir my-other-clone &&\n+\t(cd my-other-clone &&\n+\t test_must_fail git clone --no-local .. .) &&\n+\trm -rf my-other-clone\n+'\n+\n+test_expect_success 'no-local clone from the first repo with helper succeeds' '\n+\tmkdir my-other-clone &&\n+\t(cd my-other-clone &&\n+\t git clone -c odb.magic.subprocessCommand=\"$HELPER\" --no-local .. .) &&\n+\trm -rf my-other-clone\n+'\n+\n+stop_httpd\n+\n+test_done\ndiff --git a/t/t0470/read-object-plain b/t/t0470/read-object-plain\nnew file mode 100755\nindex 0000000000..918e7b00b5\n--- /dev/null\n+++ b/t/t0470/read-object-plain\n@@ -0,0 +1,83 @@\n+#!/usr/bin/perl\n+#\n+\n+use 5.008;\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n+use strict;\n+use warnings;\n+use Git::Packet;\n+use LWP::UserAgent;\n+use HTTP::Request::Common;\n+\n+packet_initialize(\"git-read-object\", 1);\n+\n+packet_read_and_check_capabilities(\"get_raw_obj\", \"put_raw_obj\");\n+packet_write_capabilities(\"get_raw_obj\", \"put_raw_obj\");\n+\n+my $http_url = $ENV{HTTPD_URL};\n+\n+while (1) {\n+\tmy ($res, $command) = packet_txt_read();\n+\n+\tif ( $res == -1 ) {\n+\t\texit 0;\n+\t}\n+\n+\t$command =~ s/^command=//;\n+\n+\tif ( $command eq \"init\" ) {\n+\t\tpacket_bin_read();\n+\n+\t\tpacket_txt_write(\"status=success\");\n+\t\tpacket_flush();\n+\t} elsif ( $command eq \"get_raw_obj\" ) {\n+\t\tmy ($sha1) = packet_txt_read() =~ /^sha1=([0-9a-f]{40})$/;\n+\t\tpacket_bin_read();\n+\n+\t\tmy $get_url = $http_url . \"/list/?sha1=\" . $sha1;\n+\n+\t\tmy $userAgent = LWP::UserAgent->new();\n+\n+\t\tmy $response = $userAgent->get( $get_url );\n+\n+\t\tif ($response->is_error) {\n+\t\t packet_txt_write(\"size=0\");\n+\t\t packet_txt_write(\"kind=none\");\t \n+\t\t packet_txt_write(\"status=notfound\");\n+\t\t} else {\n+\t\t packet_txt_write(\"size=\" . length($response->content));\n+\t\t packet_txt_write(\"kind=blob\");\n+\t\t packet_bin_write($response->content);\n+\t\t packet_flush();\n+\t\t packet_txt_write(\"status=success\");\n+\t\t}\n+\n+\t\tpacket_flush();\n+\t} elsif ( $command eq \"put_raw_obj\" ) {\n+\t\tmy ($sha1) = packet_txt_read() =~ /^sha1=([0-9a-f]{40})$/;\n+\t\tmy ($size) = packet_txt_read() =~ /^size=([0-9]+)$/;\n+\t\tmy ($kind) = packet_txt_read() =~ /^kind=(\\w+)$/;\n+\t\tpacket_bin_read();\n+\n+\t\t# We must read the content we are sent and send it to the right url\n+\t\tmy ($res, $buf) = packet_bin_read();\n+\t\tdie \"bad packet_bin_read res ($res)\" unless ($res eq 0);\n+\t\t( packet_bin_read() eq ( 1, \"\" ) ) || die \"bad send end\";\t\t\n+\n+\t\tmy $upload_url = $http_url . \"/upload/?sha1=\" . $sha1 . \"&size=\" . $size . \"&type=blob\";\n+\n+\t\tmy $userAgent = LWP::UserAgent->new();\n+\t\tmy $request = POST $upload_url, Content_Type => 'multipart/form-data', Content => $buf;\n+\n+\t\tmy $response = $userAgent->request($request);\n+\n+\t\tif ($response->is_error) {\n+\t\t\tpacket_txt_write(\"status=failure\");\n+\t\t} else {\n+\t\t\tpacket_txt_write(\"status=success\");\n+\t\t}\n+\t\tpacket_flush();\n+\t} else {\n+\t\tdie \"bad command '$command'\";\n+\t}\n+}\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328200","messageId":"20170916080731.13925-32-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 31/40] odb-helper: add have_object_process()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:22Z","receivedAt":"2017-09-16T08:08:43Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This adds the infrastructure to handle 'have' instructions in\nprocess mode.\n\nThe answer from the helper sub-process should be like the\noutput in script mode, that is lines like this:\n\nsha1 SPACE size SPACE type NEWLINE\n\nSigned-off-by: Christian Couder \n---\n odb-helper.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 73 insertions(+)\n\ndiff --git a/odb-helper.c b/odb-helper.c\nindex 356f6172d8..7433a4bfc6 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -634,6 +634,71 @@ static int odb_helper_object_cmp(const void *va, const void *vb)\n \treturn hashcmp(a->sha1, b->sha1);\n }\n \n+static int send_have_packets(struct odb_helper *o,\n+\t\t\t struct object_process *entry,\n+\t\t\t struct strbuf *status)\n+{\n+\tchar *line;\n+\tint packet_len;\n+\tint total_got = 0;\n+\tstruct child_process *process = &entry->subprocess.process;\n+\tint err = packet_write_fmt_gently(process->in, \"command=have\\n\");\n+\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = packet_flush_gently(process->in);\n+\tif (err)\n+\t\treturn err;\n+\n+\tfor (;;) {\n+\t\t/* packet_read() writes a '\\0' extra byte at the end */\n+\t\tchar buf[LARGE_PACKET_DATA_MAX + 1];\n+\t\tchar *p = buf;\n+\t\tint more;\n+\n+\t\tpacket_len = packet_read(process->out, NULL, NULL,\n+\t\t\tbuf, LARGE_PACKET_DATA_MAX + 1,\n+\t\t\tPACKET_READ_GENTLE_ON_EOF);\n+\n+\t\tif (packet_len <= 0)\n+\t\t\tbreak;\n+\n+\t\ttotal_got += packet_len;\n+\n+\t\t/* 'have' packets should end with '\\n' or '\\0' */\n+\t\tdo {\n+\t\t\tchar *eol = strchrnul(p, '\\n');\n+\t\t\tmore = (*eol == '\\n');\n+\t\t\t*eol = '\\0';\n+\t\t\tif (add_have_entry(o, p))\n+\t\t\t\tbreak;\n+\t\t\tp = eol + 1;\n+\t\t} while (more && *p);\n+\t}\n+\n+\tif (packet_len < 0)\n+\t\treturn packet_len;\n+\n+\treturn check_object_process_status(process->out, status);\n+}\n+\n+static int have_object_process(struct odb_helper *o)\n+{\n+\tint err;\n+\tstruct object_process *entry;\n+\tstruct strbuf status = STRBUF_INIT;\n+\n+\tentry = launch_object_process(o, ODB_HELPER_CAP_HAVE);\n+\tif (!entry)\n+\t\treturn -1;\n+\n+\terr = send_have_packets(o, entry, &status);\n+\n+\treturn check_object_process_error(err, status.buf, entry, o->cmd,\n+\t\t\t\t\t ODB_HELPER_CAP_HAVE);\n+}\n+\n static void have_object_script(struct odb_helper *o)\n {\n \tstruct odb_helper_cmd cmd;\n@@ -655,12 +720,20 @@ static void have_object_script(struct odb_helper *o)\n \n static void odb_helper_load_have(struct odb_helper *o)\n {\n+\tuint64_t start;\n+\n \tif (o->have_valid)\n \t\treturn;\n \to->have_valid = 1;\n \n+\tstart = getnanotime();\n+\n \tif (o->script_mode)\n \t\thave_object_script(o);\n+\telse\n+\t\thave_object_process(o);\n+\n+\ttrace_performance_since(start, \"odb_helper_load_have\");\n \n \tqsort(o->have, o->have_nr, sizeof(*o->have), odb_helper_object_cmp);\n }\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328201","messageId":"20170916080731.13925-33-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 32/40] Add t0480 to test \"have\" capability and raw objects","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:23Z","receivedAt":"2017-09-16T08:08:47Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n t/t0480-read-object-have-http-e-odb.sh | 109 +++++++++++++++++++++++++++++++++\n t/t0480/read-object-plain-have | 103 +++++++++++++++++++++++++++++++\n 2 files changed, 212 insertions(+)\n create mode 100755 t/t0480-read-object-have-http-e-odb.sh\n create mode 100755 t/t0480/read-object-plain-have\n\ndiff --git a/t/t0480-read-object-have-http-e-odb.sh b/t/t0480-read-object-have-http-e-odb.sh\nnew file mode 100755\nindex 0000000000..056a40f2bb\n--- /dev/null\n+++ b/t/t0480-read-object-have-http-e-odb.sh\n@@ -0,0 +1,109 @@\n+#!/bin/sh\n+\n+test_description='tests for read-object process with \"have\" cap and plain objects'\n+\n+. ./test-lib.sh\n+\n+# If we don't specify a port, the current test number will be used\n+# which will not work as it is less than 1024, so it can only be used by root.\n+LIB_HTTPD_PORT=$(expr ${this_test#t} + 12000)\n+\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+\n+start_httpd apache-e-odb.conf\n+\n+PATH=\"$PATH:$TEST_DIRECTORY/t0480\"\n+\n+# odb helper script must see this\n+export HTTPD_URL\n+\n+HELPER=\"read-object-plain-have\"\n+\n+test_expect_success 'setup repo with a root commit' '\n+\ttest_commit zero\n+'\n+\n+test_expect_success 'setup another repo from the first one' '\n+\tgit init other-repo &&\n+\t(cd other-repo &&\n+\t git remote add origin .. &&\n+\t git pull origin master &&\n+\t git checkout master &&\n+\t git log)\n+'\n+\n+test_expect_success 'setup the helper in the root repo' '\n+\tgit config odb.magic.subprocessCommand \"$HELPER\"\n+'\n+\n+UPLOADFILENAME=\"hello_apache_upload.txt\"\n+\n+UPLOAD_URL=\"$HTTPD_URL/upload/?sha1=$UPLOADFILENAME&size=123&type=blob\"\n+\n+test_expect_success 'can upload a file' '\n+\techo \"Hello Apache World!\" >hello_to_send.txt &&\n+\techo \"How are you?\" >>hello_to_send.txt &&\n+\tcurl --data-binary @hello_to_send.txt --include \"$UPLOAD_URL\" >out_upload\n+'\n+\n+LIST_URL=\"$HTTPD_URL/list/\"\n+\n+test_expect_success 'can list uploaded files' '\n+\tcurl --include \"$LIST_URL\" >out_list &&\n+\tgrep \"$UPLOADFILENAME\" out_list\n+'\n+\n+test_expect_success 'can delete uploaded files' '\n+\tcurl --data \"delete\" --include \"$UPLOAD_URL&delete=1\" >out_delete &&\n+\tcurl --include \"$LIST_URL\" >out_list2 &&\n+\t! grep \"$UPLOADFILENAME\" out_list2\n+'\n+\n+FILES_DIR=\"httpd/www/files\"\n+\n+test_expect_success 'new blobs are transfered to the http server' '\n+\ttest_commit one &&\n+\thash1=$(git ls-tree HEAD | grep one.t | cut -f1 | cut -d\\ -f3) &&\n+\techo \"$hash1-4-blob\" >expected &&\n+\tls \"$FILES_DIR\" >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'blobs can be retrieved from the http server' '\n+\tgit cat-file blob \"$hash1\" &&\n+\tgit log -p >expected\n+'\n+\n+test_expect_success 'update other repo from the first one' '\n+\t(cd other-repo &&\n+\t git fetch origin \"refs/odbs/magic/*:refs/odbs/magic/*\" &&\n+\t test_must_fail git cat-file blob \"$hash1\" &&\n+\t git config odb.magic.subprocessCommand \"$HELPER\" &&\n+\t git cat-file blob \"$hash1\" &&\n+\t git pull origin master)\n+'\n+\n+test_expect_success 'local clone from the first repo' '\n+\tmkdir my-clone &&\n+\t(cd my-clone &&\n+\t git clone .. . &&\n+\t git cat-file blob \"$hash1\")\n+'\n+\n+test_expect_success 'no-local clone from the first repo fails' '\n+\tmkdir my-other-clone &&\n+\t(cd my-other-clone &&\n+\t test_must_fail git clone --no-local .. .) &&\n+\trm -rf my-other-clone\n+'\n+\n+test_expect_success 'no-local clone from the first repo with helper succeeds' '\n+\tmkdir my-other-clone &&\n+\t(cd my-other-clone &&\n+\t git clone -c odb.magic.subprocessCommand=\"$HELPER\" --no-local .. .) &&\n+\trm -rf my-other-clone\n+'\n+\n+stop_httpd\n+\n+test_done\ndiff --git a/t/t0480/read-object-plain-have b/t/t0480/read-object-plain-have\nnew file mode 100755\nindex 0000000000..d63e327f33\n--- /dev/null\n+++ b/t/t0480/read-object-plain-have\n@@ -0,0 +1,103 @@\n+#!/usr/bin/perl\n+#\n+\n+use 5.008;\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n+use strict;\n+use warnings;\n+use Git::Packet;\n+use LWP::UserAgent;\n+use HTTP::Request::Common;\n+\n+packet_initialize(\"git-read-object\", 1);\n+\n+packet_read_and_check_capabilities(\"get_raw_obj\", \"put_raw_obj\", \"have\");\n+packet_write_capabilities(\"get_raw_obj\", \"put_raw_obj\", \"have\");\n+\n+my $http_url = $ENV{HTTPD_URL};\n+\n+while (1) {\n+\tmy ($res, $command) = packet_txt_read();\n+\n+\tif ( $res == -1 ) {\n+\t\texit 0;\n+\t}\n+\n+\t$command =~ s/^command=//;\n+\n+\tif ( $command eq \"init\" ) {\n+\t\tpacket_bin_read();\n+\n+\t\tpacket_txt_write(\"status=success\");\n+\t\tpacket_flush();\n+\t} elsif ( $command eq \"have\" ) {\n+\t\t# read the flush after the command\n+\t\tpacket_bin_read();\n+\n+\t\tmy $have_url = $http_url . \"/list/\";\n+\n+\t\tmy $userAgent = LWP::UserAgent->new();\n+\t\tmy $response = $userAgent->get( $have_url );\n+\n+\t\tif ($response->is_error) {\n+\t\t\tpacket_bin_write(\"\");\n+\t\t\tpacket_flush();\n+\t\t\tpacket_txt_write(\"status=failure\");\n+\t\t} else {\n+\t\t\tpacket_bin_write($response->content);\n+\t\t\tpacket_flush();\n+\t\t\tpacket_txt_write(\"status=success\");\n+\t\t}\n+\t\tpacket_flush();\n+\t} elsif ( $command eq \"get_raw_obj\" ) {\n+\t\tmy ($sha1) = packet_txt_read() =~ /^sha1=([0-9a-f]{40})$/;\n+\t\tpacket_bin_read();\n+\n+\t\tmy $get_url = $http_url . \"/list/?sha1=\" . $sha1;\n+\n+\t\tmy $userAgent = LWP::UserAgent->new();\n+\n+\t\tmy $response = $userAgent->get( $get_url );\n+\n+\t\tif ($response->is_error) {\n+\t\t\tpacket_txt_write(\"size=0\");\n+\t\t\tpacket_txt_write(\"kind=none\");\t \n+\t\t\tpacket_txt_write(\"status=notfound\");\n+\t\t} else {\n+\t\t\tpacket_txt_write(\"size=\" . length($response->content));\n+\t\t\tpacket_txt_write(\"kind=blob\");\n+\t\t\tpacket_bin_write($response->content);\n+\t\t\tpacket_flush();\n+\t\t\tpacket_txt_write(\"status=success\");\n+\t\t}\n+\n+\t\tpacket_flush();\n+\t} elsif ( $command eq \"put_raw_obj\" ) {\n+\t\tmy ($sha1) = packet_txt_read() =~ /^sha1=([0-9a-f]{40})$/;\n+\t\tmy ($size) = packet_txt_read() =~ /^size=([0-9]+)$/;\n+\t\tmy ($kind) = packet_txt_read() =~ /^kind=(\\w+)$/;\n+\n+\t\tpacket_bin_read();\n+\n+\t\t# We must read the content we are sent and send it to the right url\n+\t\tmy ($res, $buf) = packet_bin_read();\n+\t\tdie \"bad packet_bin_read res ($res)\" unless ($res eq 0);\n+\t\t( packet_bin_read() eq ( 1, \"\" ) ) || die \"bad send end\";\t\t\n+\n+\t\tmy $upload_url = $http_url . \"/upload/?sha1=\" . $sha1 . \"&size=\" . $size . \"&type=blob\";\n+\n+\t\tmy $userAgent = LWP::UserAgent->new();\n+\t\tmy $request = POST $upload_url, Content_Type => 'multipart/form-data', Content => $buf;\n+\n+\t\tmy $response = $userAgent->request($request);\n+\n+\t\tif ($response->is_error) {\n+\t\t\tpacket_txt_write(\"status=failure\");\n+\t\t} else {\n+\t\t\tpacket_txt_write(\"status=success\");\n+\t\t}\n+\t\tpacket_flush();\n+\t} else {\n+\t\tdie \"bad command '$command'\";\n+\t}\n+}\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328202","messageId":"20170916080731.13925-34-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 33/40] external-odb: use 'odb=magic' attribute to mark odb blobs","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:24Z","receivedAt":"2017-09-16T08:08:50Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"To tell which blobs should be sent to the \"magic\" external odb,\nlet's require that the blobs be marked using the 'odb=magic'\nattribute.\n\nSigned-off-by: Christian Couder \n---\n external-odb.c | 22 ++++++++++++++++++++--\n external-odb.h | 3 ++-\n sha1_file.c | 20 +++++++++++++++-----\n t/t0400-external-odb.sh | 3 +++\n t/t0410-transfer-e-odb.sh | 3 +++\n t/t0420-transfer-http-e-odb.sh | 3 +++\n t/t0470-read-object-http-e-odb.sh | 3 +++\n t/t0480-read-object-have-http-e-odb.sh | 3 +++\n 8 files changed, 52 insertions(+), 8 deletions(-)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex 084cd32e0b..e103514a46 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -1,6 +1,7 @@\n #include \"cache.h\"\n #include \"external-odb.h\"\n #include \"odb-helper.h\"\n+#include \"attr.h\"\n \n static struct odb_helper *helpers;\n static struct odb_helper **helpers_tail = &helpers;\n@@ -155,8 +156,23 @@ int external_odb_get_object(const unsigned char *sha1)\n \treturn external_odb_do_get_object(sha1);\n }\n \n+static int has_odb_attrs(struct odb_helper *o, const char *path)\n+{\n+\tstatic struct attr_check *check;\n+\n+\tif (!check)\n+\t\tcheck = attr_check_initl(\"odb\", NULL);\n+\n+\tif (!git_check_attr(path, check)) {\n+\t\tconst char *value = check->items[0].value;\n+\t\treturn value ? !strcmp(o->name, value) : 0;\n+\t}\n+\treturn 0;\n+}\n+\n int external_odb_put_object(const void *buf, size_t len,\n-\t\t\t const char *type, unsigned char *sha1)\n+\t\t\t const char *type, unsigned char *sha1,\n+\t\t\t const char *path)\n {\n \tstruct odb_helper *o;\n \n@@ -164,12 +180,14 @@ int external_odb_put_object(const void *buf, size_t len,\n \t\treturn 1;\n \n \t/* For now accept only blobs */\n-\tif (strcmp(type, \"blob\"))\n+\tif (!path || strcmp(type, \"blob\"))\n \t\treturn 1;\n \n \texternal_odb_init();\n \n \tfor (o = helpers; o; o = o->next) {\n+\t\tif (!has_odb_attrs(o, path))\n+\t\t\tcontinue;\n \t\tint r = odb_helper_put_object(o, buf, len, type, sha1);\n \t\tif (r <= 0)\n \t\t\treturn r;\ndiff --git a/external-odb.h b/external-odb.h\nindex 1fda08c0fb..93a3b35a04 100644\n--- a/external-odb.h\n+++ b/external-odb.h\n@@ -6,6 +6,7 @@ extern int external_odb_has_object(const unsigned char *sha1);\n extern int external_odb_get_object(const unsigned char *sha1);\n extern int external_odb_get_direct(const unsigned char *sha1);\n extern int external_odb_put_object(const void *buf, size_t len,\n-\t\t\t\t const char *type, unsigned char *sha1);\n+\t\t\t\t const char *type, unsigned char *sha1,\n+\t\t\t\t const char *path);\n \n #endif /* EXTERNAL_ODB_H */\ndiff --git a/sha1_file.c b/sha1_file.c\nindex c5b6d89b97..24fbc28eab 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1631,7 +1631,9 @@ static int freshen_packed_object(const unsigned char *sha1)\n \treturn 1;\n }\n \n-int write_sha1_file(const void *buf, unsigned long len, const char *type, unsigned char *sha1)\n+static int write_sha1_file_with_path(const void *buf, unsigned long len,\n+\t\t\t\t const char *type, unsigned char *sha1,\n+\t\t\t\t const char *path)\n {\n \tchar hdr[32];\n \tint hdrlen = sizeof(hdr);\n@@ -1640,13 +1642,19 @@ int write_sha1_file(const void *buf, unsigned long len, const char *type, unsign\n \t * it out into .git/objects/??/?{38} file.\n \t */\n \twrite_sha1_file_prepare(buf, len, type, sha1, hdr, &hdrlen);\n-\tif (!external_odb_put_object(buf, len, type, sha1))\n+\tif (!external_odb_put_object(buf, len, type, sha1, path))\n \t\treturn 0;\n \tif (freshen_packed_object(sha1) || freshen_loose_object(sha1))\n \t\treturn 0;\n \treturn write_loose_object(sha1, hdr, hdrlen, buf, len, 0);\n }\n \n+int write_sha1_file(const void *buf, unsigned long len,\n+\t\t const char *type, unsigned char *sha1)\n+{\n+\twrite_sha1_file_with_path(buf, len, type, sha1, NULL);\n+}\n+\n int hash_sha1_file_literally(const void *buf, unsigned long len, const char *type,\n \t\t\t struct object_id *oid, unsigned flags)\n {\n@@ -1767,7 +1775,8 @@ static int index_mem(unsigned char *sha1, void *buf, size_t size,\n \t}\n \n \tif (write_object)\n-\t\tret = write_sha1_file(buf, size, typename(type), sha1);\n+\t\tret = write_sha1_file_with_path(buf, size, typename(type),\n+\t\t\t\t\t\tsha1, path);\n \telse\n \t\tret = hash_sha1_file(buf, size, typename(type), sha1);\n \tif (re_allocated)\n@@ -1789,8 +1798,9 @@ static int index_stream_convert_blob(unsigned char *sha1, int fd,\n \t\t\t\t write_object ? safe_crlf : SAFE_CRLF_FALSE);\n \n \tif (write_object)\n-\t\tret = write_sha1_file(sbuf.buf, sbuf.len, typename(OBJ_BLOB),\n-\t\t\t\t sha1);\n+\t\tret = write_sha1_file_with_path(sbuf.buf, sbuf.len,\n+\t\t\t\t\t\ttypename(OBJ_BLOB),\n+\t\t\t\t\t\tsha1, path);\n \telse\n \t\tret = hash_sha1_file(sbuf.buf, sbuf.len, typename(OBJ_BLOB),\n \t\t\t\t sha1);\ndiff --git a/t/t0400-external-odb.sh b/t/t0400-external-odb.sh\nindex 03df030461..492c772076 100755\n--- a/t/t0400-external-odb.sh\n+++ b/t/t0400-external-odb.sh\n@@ -73,6 +73,9 @@ test_expect_success 'helper can add objects to alt repo' '\n \n test_expect_success 'commit adds objects to alt repo' '\n \ttest_config odb.magic.scriptCommand \"$HELPER\" &&\n+\techo \"* odb=magic\" >.gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git add .gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git commit -m \"Add .gitattributes\" &&\n \ttest_commit three &&\n \thash3=$(git ls-tree HEAD | grep three.t | cut -f1 | cut -d\\ -f3) &&\n \tcontent=$(cd alt-repo && git show \"$hash3\") &&\ndiff --git a/t/t0410-transfer-e-odb.sh b/t/t0410-transfer-e-odb.sh\nindex 065ec7d759..fd3e37918c 100755\n--- a/t/t0410-transfer-e-odb.sh\n+++ b/t/t0410-transfer-e-odb.sh\n@@ -111,6 +111,9 @@ test_expect_success 'setup other repo and its alternate repo' '\n '\n \n test_expect_success 'new blobs are put in first object store' '\n+\techo \"* odb=magic\" >.gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git add .gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git commit -m \"Add .gitattributes\" &&\n \ttest_commit one &&\n \thash1=$(git ls-tree HEAD | grep one.t | cut -f1 | cut -d\\ -f3) &&\n \tcontent=$(cd alt-repo1 && git show \"$hash1\") &&\ndiff --git a/t/t0420-transfer-http-e-odb.sh b/t/t0420-transfer-http-e-odb.sh\nindex f84fe950ec..d307af0457 100755\n--- a/t/t0420-transfer-http-e-odb.sh\n+++ b/t/t0420-transfer-http-e-odb.sh\n@@ -94,6 +94,9 @@ test_expect_success 'can delete uploaded files' '\n FILES_DIR=\"httpd/www/files\"\n \n test_expect_success 'new blobs are transfered to the http server' '\n+\techo \"* odb=magic\" >.gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git add .gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git commit -m \"Add .gitattributes\" &&\n \ttest_commit one &&\n \thash1=$(git ls-tree HEAD | grep one.t | cut -f1 | cut -d\\ -f3) &&\n \techo \"$hash1-4-blob\" >expected &&\ndiff --git a/t/t0470-read-object-http-e-odb.sh b/t/t0470-read-object-http-e-odb.sh\nindex 774528c04f..d814a43d59 100755\n--- a/t/t0470-read-object-http-e-odb.sh\n+++ b/t/t0470-read-object-http-e-odb.sh\n@@ -62,6 +62,9 @@ test_expect_success 'can delete uploaded files' '\n FILES_DIR=\"httpd/www/files\"\n \n test_expect_success 'new blobs are transfered to the http server' '\n+\techo \"* odb=magic\" >.gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git add .gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git commit -m \"Add .gitattributes\" &&\n \ttest_commit one &&\n \thash1=$(git ls-tree HEAD | grep one.t | cut -f1 | cut -d\\ -f3) &&\n \techo \"$hash1-4-blob\" >expected &&\ndiff --git a/t/t0480-read-object-have-http-e-odb.sh b/t/t0480-read-object-have-http-e-odb.sh\nindex 056a40f2bb..fe1fac5ef3 100755\n--- a/t/t0480-read-object-have-http-e-odb.sh\n+++ b/t/t0480-read-object-have-http-e-odb.sh\n@@ -62,6 +62,9 @@ test_expect_success 'can delete uploaded files' '\n FILES_DIR=\"httpd/www/files\"\n \n test_expect_success 'new blobs are transfered to the http server' '\n+\techo \"* odb=magic\" >.gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git add .gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git commit -m \"Add .gitattributes\" &&\n \ttest_commit one &&\n \thash1=$(git ls-tree HEAD | grep one.t | cut -f1 | cut -d\\ -f3) &&\n \techo \"$hash1-4-blob\" >expected &&\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328203","messageId":"20170916080731.13925-39-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 38/40] Add tests for 'clone --initial-refspec'","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:29Z","receivedAt":"2017-09-16T08:08:51Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n t/t0420-transfer-http-e-odb.sh | 7 +++++\n t/t0470-read-object-http-e-odb.sh | 7 +++++\n t/t0480-read-object-have-http-e-odb.sh | 7 +++++\n t/t5616-clone-initial-refspec.sh | 48 ++++++++++++++++++++++++++++++++++\n 4 files changed, 69 insertions(+)\n create mode 100755 t/t5616-clone-initial-refspec.sh\n\ndiff --git a/t/t0420-transfer-http-e-odb.sh b/t/t0420-transfer-http-e-odb.sh\nindex d307af0457..ed833850c3 100755\n--- a/t/t0420-transfer-http-e-odb.sh\n+++ b/t/t0420-transfer-http-e-odb.sh\n@@ -140,6 +140,13 @@ test_expect_success 'no-local clone from the first repo with helper succeeds' '\n \trm -rf my-other-clone\n '\n \n+test_expect_success 'no-local initial-refspec clone succeeds' '\n+\tmkdir my-other-clone &&\n+\t(cd my-other-clone &&\n+\t git -c odb.magic.scriptCommand=\"$HELPER\" \\\n+\t\tclone --no-local --initial-refspec \"refs/odbs/magic/*:refs/odbs/magic/*\" .. .)\n+'\n+\n stop_httpd\n \n test_done\ndiff --git a/t/t0470-read-object-http-e-odb.sh b/t/t0470-read-object-http-e-odb.sh\nindex d814a43d59..7355ca4d51 100755\n--- a/t/t0470-read-object-http-e-odb.sh\n+++ b/t/t0470-read-object-http-e-odb.sh\n@@ -107,6 +107,13 @@ test_expect_success 'no-local clone from the first repo with helper succeeds' '\n \trm -rf my-other-clone\n '\n \n+test_expect_success 'no-local initial-refspec clone succeeds' '\n+\tmkdir my-other-clone &&\n+\t(cd my-other-clone &&\n+\t git -c odb.magic.subprocessCommand=\"$HELPER\" \\\n+\t\tclone --no-local --initial-refspec \"refs/odbs/magic/*:refs/odbs/magic/*\" .. .)\n+'\n+\n stop_httpd\n \n test_done\ndiff --git a/t/t0480-read-object-have-http-e-odb.sh b/t/t0480-read-object-have-http-e-odb.sh\nindex fe1fac5ef3..c451d269a7 100755\n--- a/t/t0480-read-object-have-http-e-odb.sh\n+++ b/t/t0480-read-object-have-http-e-odb.sh\n@@ -107,6 +107,13 @@ test_expect_success 'no-local clone from the first repo with helper succeeds' '\n \trm -rf my-other-clone\n '\n \n+test_expect_success 'no-local initial-refspec clone succeeds' '\n+\tmkdir my-other-clone &&\n+\t(cd my-other-clone &&\n+\t git -c odb.magic.subprocessCommand=\"$HELPER\" \\\n+\t\tclone --no-local --initial-refspec \"refs/odbs/magic/*:refs/odbs/magic/*\" .. .)\n+'\n+\n stop_httpd\n \n test_done\ndiff --git a/t/t5616-clone-initial-refspec.sh b/t/t5616-clone-initial-refspec.sh\nnew file mode 100755\nindex 0000000000..ccbc27f83f\n--- /dev/null\n+++ b/t/t5616-clone-initial-refspec.sh\n@@ -0,0 +1,48 @@\n+#!/bin/sh\n+\n+test_description='test clone with --initial-refspec option'\n+. ./test-lib.sh\n+\n+\n+test_expect_success 'setup regular repo' '\n+\t# Make two branches, \"master\" and \"side\"\n+\techo one >file &&\n+\tgit add file &&\n+\tgit commit -m one &&\n+\techo two >file &&\n+\tgit commit -a -m two &&\n+\tgit tag two &&\n+\techo three >file &&\n+\tgit commit -a -m three &&\n+\tgit checkout -b side &&\n+\techo four >file &&\n+\tgit commit -a -m four &&\n+\tgit checkout master\n+'\n+\n+test_expect_success 'add a special ref pointing to a blob' '\n+\thash=$(echo \"Hello world!\" | git hash-object -w -t blob --stdin) &&\n+\tgit update-ref refs/special/hello \"$hash\"\n+'\n+\n+test_expect_success 'no-local clone from the first repo' '\n+\tmkdir my-clone &&\n+\t(cd my-clone &&\n+\t git clone --no-local .. . &&\n+\t test_must_fail git cat-file blob \"$hash\") &&\n+\trm -rf my-clone\n+'\n+\n+test_expect_success 'no-local clone with --initial-refspec' '\n+\tmkdir my-clone &&\n+\t(cd my-clone &&\n+\t git clone --no-local --initial-refspec \"refs/special/*:refs/special/*\" .. . &&\n+\t git cat-file blob \"$hash\" &&\n+\t git rev-parse refs/special/hello >actual &&\n+\t echo \"$hash\" >expected &&\n+\t test_cmp expected actual) &&\n+\trm -rf my-clone\n+'\n+\n+test_done\n+\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328204","messageId":"20170916080731.13925-40-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 39/40] Add t0430 to test cloning using bundles","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:30Z","receivedAt":"2017-09-16T08:08:53Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n t/t0430-clone-bundle-e-odb.sh | 85 +++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 85 insertions(+)\n create mode 100755 t/t0430-clone-bundle-e-odb.sh\n\ndiff --git a/t/t0430-clone-bundle-e-odb.sh b/t/t0430-clone-bundle-e-odb.sh\nnew file mode 100755\nindex 0000000000..ac38ae1be5\n--- /dev/null\n+++ b/t/t0430-clone-bundle-e-odb.sh\n@@ -0,0 +1,85 @@\n+#!/bin/sh\n+\n+test_description='tests for cloning using a bundle through e-odb'\n+\n+. ./test-lib.sh\n+\n+# If we don't specify a port, the current test number will be used\n+# which will not work as it is less than 1024, so it can only be used by root.\n+LIB_HTTPD_PORT=$(expr ${this_test#t} + 12000)\n+\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+\n+start_httpd apache-e-odb.conf\n+\n+# odb helper script must see this\n+export HTTPD_URL\n+\n+write_script odb-clone-bundle-helper <<\\EOF\n+die() {\n+\tprintf >&2 \"%s\\n\" \"$@\"\n+\texit 1\n+}\n+echo >&2 \"odb-clone-bundle-helper args:\" \"$@\"\n+case \"$1\" in\n+init)\n+\tref_hash=$(git rev-parse refs/odbs/magic/bundle) ||\n+\tdie \"couldn't find refs/odbs/magic/bundle\"\n+\tGIT_NO_EXTERNAL_ODB=1 git cat-file blob \"$ref_hash\" >bundle_info ||\n+\tdie \"couldn't get blob $ref_hash\"\n+\tbundle_url=$(sed -e 's/bundle url: //' bundle_info)\n+\techo >&2 \"bundle_url: '$bundle_url'\"\n+\tcurl \"$bundle_url\" -o bundle_file ||\n+\tdie \"curl '$bundle_url' failed\"\n+\tGIT_NO_EXTERNAL_ODB=1 git bundle unbundle bundle_file >unbundling_info ||\n+\tdie \"unbundling 'bundle_file' failed\"\n+\t;;\n+get*)\n+\tdie \"odb-clone-bundle-helper '$1' called\"\n+\t;;\n+put*)\n+\tdie \"odb-clone-bundle-helper '$1' called\"\n+\t;;\n+*)\n+\tdie \"unknown command '$1'\"\n+\t;;\n+esac\n+EOF\n+HELPER=\"\\\"$PWD\\\"/odb-clone-bundle-helper\"\n+\n+\n+test_expect_success 'setup repo with a few commits' '\n+\ttest_commit one &&\n+\ttest_commit two &&\n+\ttest_commit three &&\n+\ttest_commit four\n+'\n+\n+BUNDLE_FILE=\"file.bundle\"\n+FILES_DIR=\"httpd/www/files\"\n+GET_URL=\"$HTTPD_URL/files/$BUNDLE_FILE\"\n+\n+test_expect_success 'create a bundle for this repo and check that it can be downloaded' '\n+\tgit bundle create \"$BUNDLE_FILE\" master &&\n+\tmkdir \"$FILES_DIR\" &&\n+\tcp \"$BUNDLE_FILE\" \"$FILES_DIR/\" &&\n+\tcurl \"$GET_URL\" --output actual &&\n+\ttest_cmp \"$BUNDLE_FILE\" actual\n+'\n+\n+test_expect_success 'create an e-odb ref for this bundle' '\n+\tref_hash=$(echo \"bundle url: $GET_URL\" | GIT_NO_EXTERNAL_ODB=1 git hash-object -w -t blob --stdin) &&\n+\tgit update-ref refs/odbs/magic/bundle \"$ref_hash\"\n+'\n+\n+test_expect_success 'clone using the e-odb helper to download and install the bundle' '\n+\tmkdir my-clone &&\n+\t(cd my-clone &&\n+\t git clone --no-local \\\n+\t\t-c odb.magic.scriptCommand=\"$HELPER\" \\\n+\t\t--initial-refspec \"refs/odbs/magic/*:refs/odbs/magic/*\" .. .)\n+'\n+\n+stop_httpd\n+\n+test_done\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328205","messageId":"20170916080731.13925-41-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 40/40] Doc/external-odb: explain transfering objects and metadata","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:31Z","receivedAt":"2017-09-16T08:08:58Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n Documentation/technical/external-odb.txt | 105 +++++++++++++++++++++++++++++++\n 1 file changed, 105 insertions(+)\n\ndiff --git a/Documentation/technical/external-odb.txt b/Documentation/technical/external-odb.txt\nindex 58ec8a8145..76dd1e2e6c 100644\n--- a/Documentation/technical/external-odb.txt\n+++ b/Documentation/technical/external-odb.txt\n@@ -340,3 +340,108 @@ can that contains:\n *.jpg odb=magic\n ------------------------\n \n+Transfering objects\n+===================\n+\n+When an external odb helper is configured, the objects managed by the\n+external odb are not put in the pack file that is sent (when pushing\n+or answering clone and fetch requests), so the receiver should also\n+have configured an external odb helper that can get the missing\n+objects otherwise Git will error out complaining about missing\n+objects.\n+\n+This has some drawbacks of course, but at least it makes sure that\n+users' and admins' repositories are both properly configured to use a\n+common external ODB before they can talk to each other.\n+\n+Transfering meta information and restartable clone\n+==================================================\n+\n+There are different ways to make it possible for the external odb\n+helpers to know which services they should get the objects from (or\n+put them into), for example the information could be hardcoded into\n+the helpers, or the information could be computed from configuration\n+information like the url of the \"origin\" remote.\n+\n+The external odb mechanism itself doesn't really take care of this, so\n+helpers are free to do whatever they want.\n+\n+One interesting possibility though is to have this information as part\n+of the repository in special refs, for example refs/odb/magic/*, where\n+\"magic\" is the external odb name.\n+\n+This would especially make it possible to implement a restartable\n+clone using Git bundles (and an external odb helper) like this:\n+\n+\t1) At the very start of the clone, Git would fetch the refs\n+\tthat contain \"meta information\", for example refs/odb/magic/*\n+\t(where \"magic\" is the odb name). These refs would point to\n+\tsome blobs that contain lists of the bundles that are\n+\tavailable for fetching by the helper, along with enough\n+\tinformation for the helper to fetch them (for example HTTP\n+\turls of the bundles).\n+\n+\t2) After this first fetch of the refs/odb/magic/* refs, the\n+\thelper would be sent the 'init' instruction. At that time it\n+\tcan read all the blobs pointed to by these refs and download\n+\tthe bundles listed in the blobs.\n+\n+\tIf something goes wrong when the helper \"fetches\" a bundle,\n+\tthe helper could force the clone to error out (after maybe\n+\tretrying), and when the user (or the helper itself) tries\n+\tagain to clone, the helper would restart its bundle \"fetch\"\n+\t(using the restartable protocol, for example HTTP).\n+\n+\tWhen this \"fetch\" eventually succeeds, then the helper will\n+\tunbundle what it received, and then give back control to the\n+\tsecond regular part of the clone.\n+\n+\t3) This regular part of the clone will then try to fetch the\n+\tusual refs, but as the unbundling has already updated the\n+\tcontent of the usual refs as well as the object stores this\n+\tfetch will find that everything is up-to-date.\n+\n+\tOr if everything is not quite up-to-date and there are still\n+\tthings to fetch, another hopefully much small regular fetch\n+\twill happen.\n+\n+As this is an interesting use of the external odb mechanism, the\n+`--initial-refspec` option has been implemented in `git clone`. This\n+makes it possible to perform all the above steps using a single clone\n+command like:\n+\n+------------------------\n+$ git clone -c odb.magic.scriptCommand=\"$HELPER\" \\\n+ --initial-refspec \"refs/odbs/magic/*:refs/odbs/magic/*\" \"$URL\"\n+------------------------\n+\n+But note that the above could also be performed using:\n+\n+------------------------\n+$ git init\n+$ git remote add origin \"$URL\"\n+$ git fetch origin \"refs/odbs/magic/*:refs/odbs/magic/*\"\n+$ git config odb.magic.scriptCommand \"$HELPER\"\n+$ git fetch origin\n+------------------------\n+\n+So the `--initial-refspec` option can be seen as just a shortcut to\n+simplify external odb helped clones for users.\n+\n+Also note that this `--initial-refspec` approach could be slower than\n+a regular clone, so it is mostly interesting if one wants to fetch a\n+big number of objects or many big objects, like for an initial clone\n+of a big repo. In this use case a relatively small amount of time\n+spent in the initial fetch is an acceptable trade-off if the clone is\n+restartable.\n+\n+Though in some cases, as the `--initial-refspec` clone could alleviate\n+resource usage of the Git server, it could be even faster than a\n+regular clone.\n+\n+So admins and users should not blindly use the `--initial-refspec`\n+option all the time when an external odb is configured. But using an\n+external odb in the first place means that they have specific\n+requirements for handling objects which suggests that the regular way\n+to clone might not be very good for their use cases and for the\n+objects that are stored in their external ODBs.\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328206","messageId":"20170916080731.13925-38-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 37/40] clone: disable external odb before initial clone","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:28Z","receivedAt":"2017-09-16T08:09:00Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"To make it possible to have the external odb mechanism only kick in\nafter the initial part of a clone, we should disable it during the\ninitial part of the clone.\n\nLet's do that by saving and then restoring the value of the\n'use_external_odb' global variable.\n\nSigned-off-by: Christian Couder \n---\n builtin/clone.c | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 57cecd194c..323b73016e 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -938,6 +938,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \n \tstruct refspec *initial_refspecs;\n \tint is_initial;\n+\tint saved_use_external_odb;\n \n \tpacket_trace_identity(\"clone\");\n \targc = parse_options(argc, argv, prefix, builtin_clone_options,\n@@ -1083,6 +1084,10 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \n \tgit_config(git_default_config, NULL);\n \n+\t/* Temporarily disable external ODB before initial clone */\n+\tsaved_use_external_odb = use_external_odb;\n+\tuse_external_odb = 0;\n+\n \tif (option_bare) {\n \t\tif (option_mirror)\n \t\t\tsrc_ref_prefix = \"refs/\";\n@@ -1166,6 +1171,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\tfetch_initial_refs(transport, refs, initial_refspecs,\n \t\t\t\t branch_top.buf, reflog_msg.buf, is_local);\n \n+\t\tuse_external_odb = saved_use_external_odb;\n+\n \t\tmapped_refs = wanted_peer_refs(refs, refspec);\n \t\t/*\n \t\t * transport_get_remote_refs() may return refs with null sha-1\n@@ -1207,6 +1214,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\t\t\t\toption_branch, option_origin);\n \n \t\twarning(_(\"You appear to have cloned an empty repository.\"));\n+\n+\t\tuse_external_odb = saved_use_external_odb;\n+\n \t\tmapped_refs = NULL;\n \t\tour_head_points_at = NULL;\n \t\tremote_head_points_at = NULL;\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328207","messageId":"20170916080731.13925-36-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 35/40] clone: add 'initial' param to write_remote_refs()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:26Z","receivedAt":"2017-09-16T08:09:08Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"We want to make it possible to separate fetching remote refs into\nan initial part and a later part. To prepare for that, let's add\nan 'initial' boolean parameter to write_remote_refs() to tell this\nfunction if we are performing the initial part or not.\n\nSigned-off-by: Christian Couder \n---\n builtin/clone.c | 19 +++++++++++++------\n 1 file changed, 13 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex dcd5b878f1..2e5d60521d 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -574,7 +574,7 @@ static struct ref *wanted_peer_refs(const struct ref *refs,\n \treturn local_refs;\n }\n \n-static void write_remote_refs(const struct ref *local_refs)\n+static void write_remote_refs(const struct ref *local_refs, int initial)\n {\n \tconst struct ref *r;\n \n@@ -593,8 +593,13 @@ static void write_remote_refs(const struct ref *local_refs)\n \t\t\tdie(\"%s\", err.buf);\n \t}\n \n-\tif (initial_ref_transaction_commit(t, &err))\n-\t\tdie(\"%s\", err.buf);\n+\tif (initial) {\n+\t\tif (initial_ref_transaction_commit(t, &err))\n+\t\t\tdie(\"%s\", err.buf);\n+\t} else {\n+\t\tif (ref_transaction_commit(t, &err))\n+\t\t\tdie(\"%s\", err.buf);\n+\t}\n \n \tstrbuf_release(&err);\n \tref_transaction_free(t);\n@@ -641,7 +646,8 @@ static void update_remote_refs(const struct ref *refs,\n \t\t\t const char *branch_top,\n \t\t\t const char *msg,\n \t\t\t struct transport *transport,\n-\t\t\t int check_connectivity)\n+\t\t\t int check_connectivity,\n+\t\t\t int initial)\n {\n \tconst struct ref *rm = mapped_refs;\n \n@@ -656,7 +662,7 @@ static void update_remote_refs(const struct ref *refs,\n \t}\n \n \tif (refs) {\n-\t\twrite_remote_refs(mapped_refs);\n+\t\twrite_remote_refs(mapped_refs, initial);\n \t\tif (option_single_branch && !option_no_tags)\n \t\t\twrite_followtags(refs, msg);\n \t}\n@@ -1168,7 +1174,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\ttransport_fetch_refs(transport, mapped_refs);\n \n \tupdate_remote_refs(refs, mapped_refs, remote_head_points_at,\n-\t\t\t branch_top.buf, reflog_msg.buf, transport, !is_local);\n+\t\t\t branch_top.buf, reflog_msg.buf, transport,\n+\t\t\t !is_local, 0);\n \n \tupdate_head(our_head_points_at, remote_head, reflog_msg.buf);\n \n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328208","messageId":"20170916080731.13925-37-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 36/40] clone: add --initial-refspec option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:27Z","receivedAt":"2017-09-16T08:09:09Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This option makes it possible to separate fetching refs when cloning\nin two parts, an initial part and a later normal part.\n\nThis way after the initial part, mechanisms like the external odb\nmechanism can be used to prefetch some objects using information\nthat has been made available during the initial fetch.\n\nSigned-off-by: Christian Couder \n---\n builtin/clone.c | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 54 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 2e5d60521d..57cecd194c 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -57,6 +57,7 @@ static enum transport_family family;\n static struct string_list option_config = STRING_LIST_INIT_NODUP;\n static struct string_list option_required_reference = STRING_LIST_INIT_NODUP;\n static struct string_list option_optional_reference = STRING_LIST_INIT_NODUP;\n+static struct string_list option_initial_refspec = STRING_LIST_INIT_NODUP;\n static int option_dissociate;\n static int max_jobs = -1;\n static struct string_list option_recurse_submodules = STRING_LIST_INIT_NODUP;\n@@ -107,6 +108,8 @@ static struct option builtin_clone_options[] = {\n \t\t\tN_(\"reference repository\")),\n \tOPT_STRING_LIST(0, \"reference-if-able\", &option_optional_reference,\n \t\t\tN_(\"repo\"), N_(\"reference repository\")),\n+\tOPT_STRING_LIST(0, \"initial-refspec\", &option_initial_refspec,\n+\t\t\tN_(\"refspec\"), N_(\"fetch this refspec first\")),\n \tOPT_BOOL(0, \"dissociate\", &option_dissociate,\n \t\t N_(\"use --reference only while cloning\")),\n \tOPT_STRING('o', \"origin\", &option_origin, N_(\"name\"),\n@@ -869,6 +872,47 @@ static void dissociate_from_references(void)\n \tfree(alternates);\n }\n \n+static struct refspec *parse_initial_refspecs(void)\n+{\n+\tconst char **refspecs;\n+\tstruct refspec *initial_refspecs;\n+\tstruct string_list_item *rs;\n+\tint i = 0;\n+\n+\tif (!option_initial_refspec.nr)\n+\t\treturn NULL;\n+\n+\trefspecs = xcalloc(option_initial_refspec.nr, sizeof(const char *));\n+\n+\tfor_each_string_list_item(rs, &option_initial_refspec)\n+\t\trefspecs[i++] = rs->string;\n+\n+\tinitial_refspecs = parse_fetch_refspec(option_initial_refspec.nr, refspecs);\n+\n+\tfree(refspecs);\n+\n+\treturn initial_refspecs;\n+}\n+\n+static void fetch_initial_refs(struct transport *transport,\n+\t\t\t const struct ref *refs,\n+\t\t\t struct refspec *initial_refspecs,\n+\t\t\t const char *branch_top,\n+\t\t\t const char *reflog_msg,\n+\t\t\t int is_local)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < option_initial_refspec.nr; i++) {\n+\t\tstruct ref *init_refs = NULL;\n+\t\tstruct ref **tail = &init_refs;\n+\t\tget_fetch_map(refs, &initial_refspecs[i], &tail, 0);\n+\t\ttransport_fetch_refs(transport, init_refs);\n+\t\tupdate_remote_refs(refs, init_refs, NULL, branch_top, reflog_msg,\n+\t\t\t\t transport, !is_local, 1);\n+\t}\n+}\n+\n int cmd_clone(int argc, const char **argv, const char *prefix)\n {\n \tint is_bundle = 0, is_local;\n@@ -892,6 +936,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tstruct refspec *refspec;\n \tconst char *fetch_pattern;\n \n+\tstruct refspec *initial_refspecs;\n+\tint is_initial;\n+\n \tpacket_trace_identity(\"clone\");\n \targc = parse_options(argc, argv, prefix, builtin_clone_options,\n \t\t\t builtin_clone_usage, 0);\n@@ -1059,6 +1106,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (option_required_reference.nr || option_optional_reference.nr)\n \t\tsetup_reference();\n \n+\tinitial_refspecs = parse_initial_refspecs();\n+\n \tfetch_pattern = xstrfmt(\"+%s*:%s*\", src_ref_prefix, branch_top.buf);\n \trefspec = parse_fetch_refspec(1, &fetch_pattern);\n \tfree((char *)fetch_pattern);\n@@ -1114,6 +1163,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \trefs = transport_get_remote_refs(transport);\n \n \tif (refs) {\n+\t\tfetch_initial_refs(transport, refs, initial_refspecs,\n+\t\t\t\t branch_top.buf, reflog_msg.buf, is_local);\n+\n \t\tmapped_refs = wanted_peer_refs(refs, refspec);\n \t\t/*\n \t\t * transport_get_remote_refs() may return refs with null sha-1\n@@ -1173,9 +1225,10 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \telse if (refs && complete_refs_before_fetch)\n \t\ttransport_fetch_refs(transport, mapped_refs);\n \n+\tis_initial = !refs || option_initial_refspec.nr == 0;\n \tupdate_remote_refs(refs, mapped_refs, remote_head_points_at,\n \t\t\t branch_top.buf, reflog_msg.buf, transport,\n-\t\t\t !is_local, 0);\n+\t\t\t !is_local, is_initial);\n \n \tupdate_head(our_head_points_at, remote_head, reflog_msg.buf);\n \n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328209","messageId":"20170916080731.13925-35-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 34/40] Add Documentation/technical/external-odb.txt","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:25Z","receivedAt":"2017-09-16T08:09:12Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This describes the external odb mechanism's purpose and\nhow it works.\n\nHelped-by: Ben Peart \nSigned-off-by: Christian Couder \n---\n Documentation/technical/external-odb.txt | 342 +++++++++++++++++++++++++++++++\n 1 file changed, 342 insertions(+)\n create mode 100644 Documentation/technical/external-odb.txt\n\ndiff --git a/Documentation/technical/external-odb.txt b/Documentation/technical/external-odb.txt\nnew file mode 100644\nindex 0000000000..58ec8a8145\n--- /dev/null\n+++ b/Documentation/technical/external-odb.txt\n@@ -0,0 +1,342 @@\n+External ODBs\n+^^^^^^^^^^^^^\n+\n+The External ODB mechanism makes it possible for Git objects, only\n+blobs for now though, to be stored in an \"external object database\"\n+(External ODB).\n+\n+An External ODB can be any object store as long as there is an helper\n+program called an \"odb helper\" that can communicate with Git to\n+transfer objects to/from the external odb and to retrieve information\n+about available objects in the external odb.\n+\n+Purpose\n+=======\n+\n+The purpose of this mechanism is to make possible to handle Git\n+objects, especially blobs, in much more flexible ways.\n+\n+Currently Git can store its objects only in the form of loose objects\n+in separate files or packed objects in a pack file. These existing\n+object stores cannot be easily optimized for many different kind of\n+contents.\n+\n+So the current stores are not flexible enough for some important use\n+cases like handling really big binary files or handling a really big\n+number of files that are fetched only as needed. And it is not\n+realistic to expect that Git could fully natively handle many of such\n+use cases. Git would need to natively implement different internal\n+stores which would be a huge burden and which could lead to\n+re-implement things like HTTP servers, Docker registries or artifact\n+stores that already exist outside Git.\n+\n+Furthermore many improvements that are dependent on specific setups\n+could be implemented in the way Git objects are managed if it was\n+possible to customize how the Git objects are handled. For example a\n+restartable clone using the bundle mechanism has often been requested,\n+but implementing that would go against the current strict rules under\n+which the Git objects are currently handled.\n+\n+What Git needs is a mechanism to make it possible to customize in a\n+lot of different ways how the Git objects are handled. Though this\n+mechanism should try as much as possible to avoid interfering with the\n+usual way in which Git handle its objects.\n+\n+Helpers\n+=======\n+\n+ODB helpers are commands that have to be registered using either the\n+\"odb..subprocessCommand\" or the \"odb..scriptCommand\"\n+config variables.\n+\n+Registering such a command tells Git that an external odb called\n+ exists and that the registered command should be used to\n+communicate with it.\n+\n+The communication happens through instructions that are sent by Git\n+and that the commands should answer. If it makes sense, Git can send\n+the same instruction to many commands in the order in which they are\n+configured.\n+\n+There are 2 kinds of commands. Commands registered using the\n+\"odb..subprocessCommand\" config variable are called \"process\n+commands\" and the associated mode is called \"process mode\". Commands\n+registered using the \"odb..scriptCommand\" config variables\n+are called \"script commands\" and the associated mode is called \"script\n+mode\".\n+\n+Early on git commands send an 'init' instruction to the registered\n+commands. A capability negociation will take place during this\n+request/response exchange which will let Git and the helpers know how\n+they can further collaborate. The attribute system can also be used to\n+tell Git which objects should be handled by which helper.\n+\n+Process Mode\n+============\n+\n+In process mode the command is started as a single process invocation\n+that should last for the entire life of the single Git command that\n+started it.\n+\n+A packet format (pkt-line, see technical/protocol-common.txt) based\n+protocol over standard input and standard output is used for\n+communication between Git and the helper command.\n+\n+After the process command is started, Git sends a welcome message\n+(\"git-read-object-client\"), a list of supported protocol version\n+numbers, and a flush packet. Git expects to read a welcome response\n+message (\"git-read-object-server\"), exactly one protocol version\n+number from the previously sent list, and a flush packet. All further\n+communication will be based on the selected version.\n+\n+The remaining protocol description below documents \"version=1\". Please\n+note that \"version=42\" in the example below does not exist and is only\n+there to illustrate how the protocol would look with more than one\n+version.\n+\n+After the version negotiation Git sends a list of all capabilities\n+that it supports and a flush packet. Git expects to read a list of\n+desired capabilities, which must be a subset of the supported\n+capabilities list, and a flush packet as response:\n+\n+------------------------\n+packet: git> git-read-object-client\n+packet: git> version=1\n+packet: git> version=42\n+packet: git> 0000\n+packet: git< git-read-object-server\n+packet: git< version=1\n+packet: git< 0000\n+packet: git> capability=get_raw_obj\n+packet: git> capability=have\n+packet: git> capability=put_raw_obj\n+packet: git> capability=not-yet-invented\n+packet: git> 0000\n+packet: git< capability=get_raw_obj\n+packet: git< 0000\n+------------------------\n+\n+Afterwards Git sends a list of \"key=value\" pairs terminated with a\n+flush packet. The list will contain at least the instruction (based on\n+the supported capabilities) and the arguments for the\n+instruction. Please note, that the process must not send any response\n+before it received the final flush packet.\n+\n+In general any response from the helper should end with a status\n+packet. See the documentation of the 'get_*' instructions below for\n+examples of status packets.\n+\n+After the helper has processed an instruction, it is expected to wait\n+for the next \"key=value\" list containing another instruction.\n+\n+On exit Git will close the pipe to the helper. The helper is then\n+expected to detect EOF and exit gracefully on its own. Git will wait\n+until the process has stopped.\n+\n+Script Mode\n+===========\n+\n+In this mode Git launches the script command each time it wants to\n+communicates with the helper. There is no welcome message and no\n+protocol version in this mode.\n+\n+The instruction and associated arguments are passed as arguments when\n+launching the script command and if needed further information is\n+passed between Git and the command through stdin and stdout.\n+\n+Capabilities/Instructions\n+=========================\n+\n+The following instructions are currently supported by Git:\n+\n+- init\n+- get_git_obj\n+- get_raw_obj\n+- get_direct\n+- put_raw_obj\n+- have\n+\n+The plan is to also support 'put_git_obj' and 'put_direct' soon, for\n+consistency with the 'get_*' instructions.\n+\n+ - 'init'\n+\n+All the process and script commands must accept the 'init'\n+instruction. It should be the first instruction sent to a command. It\n+should not be advertised in the capability exchange. Any argument\n+should be ignored.\n+\n+In process mode, after receiving the 'init' instruction and a flush\n+packet, the helper should just send a status packet and then a flush\n+packet. See the 'get_*' instructions below for examples of status\n+packets.\n+\n+In script mode the command should print on stdout the capabilities\n+that it supports if any. This is the only time in script mode when a\n+capability exchange happens.\n+\n+For example a script command could use the following shell code\n+snippet to handle the 'init' instruction:\n+\n+------------------------\n+case \"$1\" in\n+init)\n+\techo \"capability=get_git_obj\"\n+\techo \"capability=put_raw_obj\"\n+\techo \"capability=have\"\n+\t;;\n+------------------------\n+\n+ - 'get_git_obj ' and 'get_raw_obj '\n+\n+These instructions should have a hexadecimal argument to tell\n+which object the helper should send to git.\n+\n+In process mode the sha1 argument should be followed by a flush packet\n+like this:\n+\n+------------------------\n+packet: git> command=get_git_obj\n+packet: git> sha1=0a214a649e1b3d5011e14a3dc227753f2bd2be05\n+packet: git> 0000\n+------------------------\n+\n+After reading that the helper should send the requested object to Git in a\n+packet series followed by a flush packet. If the helper does not experience\n+problems then the helper must send a \"success\" status like the following:\n+\n+------------------------\n+packet: git< status=success\n+packet: git< 0000\n+------------------------\n+\n+In case the helper cannot or does not want to send the requested\n+object as well as any other object for the lifetime of the Git\n+process, then it is expected to respond with an \"abort\" status at any\n+point in the protocol:\n+\n+------------------------\n+packet: git< status=abort\n+packet: git< 0000\n+------------------------\n+\n+Git neither stops nor restarts the helper in case a\n+\"notfound\"/\"error\"/\"abort\" status is set. An \"error\" status means a\n+possibly more transient error than an abort. The helper should also\n+send a \"notfound\" error in case of a \"get_*\" instruction, which means\n+that the requested object cannot be found.\n+\n+If the helper dies during the communication or does not adhere to the\n+protocol then Git will stop and restart it with the next instruction.\n+\n+In script mode the helper should just send the requested object to Git\n+by writing it to stdout and should then exit. The exit code should\n+signal to Git if a problem occured or not.\n+\n+The only difference between 'get_git_obj' and 'get_raw_obj' is that in\n+case of 'get_git_obj' the requested object should be sent as a Git\n+object (that is in the same format as loose object files). In case of\n+'get_raw_obj' the object should be sent in its raw format (that is the\n+same output as `git cat-file `).\n+\n+ - 'get_direct '\n+\n+This instruction is similar as the other 'get_*' instructions except\n+that no object should be sent from the helper to Git. Instead the\n+helper should directly write the requested object into a loose object\n+file in the \".git/objects\" directory.\n+\n+After the helper has sent the \"status=success\" packet and the\n+following flush packet in process mode, or after it has exited in the\n+script mode, Git will lookup again for the requested sha1 in its loose\n+object files and pack files.\n+\n+ - 'put_raw_obj '\n+\n+This instruction should be following by three arguments to tell which\n+object the helper will receive from git: , and\n+. The hexadecimal argument describes the object that will\n+be sent from Git to the helper. The is the object type (\"blob\",\n+\"tree\", \"commit\" or \"tag\") of this object. The is the size in\n+bytes of the (decompressed) object content.\n+\n+In process mode the last argument (the type) should be followed by a\n+flush packet.\n+\n+After reading that the helper should read the announced object from\n+Git in a packet series followed by a flush packet.\n+\n+If the helper does not experience problems when receiving and storing\n+or processing the object, then the helper must send a \"success\" status\n+as described for the 'get_*' instructions.\n+\n+In script mode the helper should just receive the announced object\n+from its standard input. After receiving and processing the object,\n+the helper should exit and its exit code should signal to Git if a\n+problem occured or not.\n+\n+- 'have'\n+\n+In process mode this instruction should be followed by a flush\n+packet. After receiving this packet the helper should send the sha1,\n+size and type, in this order, of all the objects it can provide to Git\n+(through a 'get_*' instruction). There should be a space character\n+between the sha1 and the size and between the size and the type, and\n+then a new line character after the type.\n+\n+If many packets are needed to send back all this information, the\n+split between packets should be made after the new line characters.\n+\n+If the helper does not experience problems, then it must then send a\n+\"success\" status as described for the 'get_*' instructions.\n+\n+In script mode the helper should send to its standard output the sha1,\n+size and type, in this order of all the objects it can provide to\n+Git. There should also be a space character between the sha1 and the\n+size and between the size and the type, and then a new line character\n+after the type.\n+\n+After sending this, the script helper should exit and its exit code\n+should signal to Git if a problem occured or not.\n+\n+Order of instructions\n+=====================\n+\n+For get_*_object instructions the regular code to find objects is\n+called before the odb helpers.\n+\n+For put_*_object instructions the regular code to store the objects is\n+called after the odb helpers.\n+\n+For now this order is not configurable.\n+\n+Object caching\n+==============\n+\n+If a helper returns the object data as requested by get_git_obj or\n+get_raw_obj, then Git will itself store the object locally in its\n+regular object store, so it is redundant for the helper to also store\n+or try to store the object in the regular object store.\n+\n+Yeah, this seems to defeat the goal of enabling specialized object\n+handlers to handle large or other \"unusual\" objects that git normally\n+doesn't deal well with. So in the long run there should be a way to\n+make this configurable.\n+\n+Selecting objects\n+=================\n+\n+To select objects that should be handled by an external odb, one can\n+use the git attributes system. For now this will only work with blobs\n+and this will only work along with the 'put_raw_obj' instruction.\n+\n+For example if one has an external odb called \"magic\" and has\n+registered an associated a process command helper that supports the\n+'put_raw_obj' instruction, then one can tell Git that all the .jpg\n+files should be handled by the \"magic\" odb using a .gitattributes file\n+can that contains:\n+\n+------------------------\n+*.jpg odb=magic\n+------------------------\n+\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328210","messageId":"20170916080731.13925-26-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 25/40] odb-helper: add 'script_mode' to 'struct odb_helper'","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:16Z","receivedAt":"2017-09-16T08:09:20Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"to prepare for having a long running odb helper sub-process\nhandling the communication between Git and an external odb.\n\nWe introduce \"odb..subprocesscommand\" to make it\npossible to define such a sub-process, and we mark such odb\nhelpers with the new 'script_mode' field set to 0.\n\nHelpers defined using the existing \"odb..scriptcommand\"\nare marked with the 'script_mode' field set to 1.\n\nImplementation of the different capabilities/instructions in\nthe new (sub-)process mode is left for following commits.\n\nSigned-off-by: Christian Couder \n---\n external-odb.c | 8 +++++++-\n odb-helper.c | 19 ++++++++++++++-----\n odb-helper.h | 1 +\n 3 files changed, 22 insertions(+), 6 deletions(-)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex 31d21bfe04..ccca67eff5 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -32,8 +32,14 @@ static int external_odb_config(const char *var, const char *value, void *data)\n \n \to = find_or_create_helper(name, namelen);\n \n-\tif (!strcmp(subkey, \"scriptcommand\"))\n+\tif (!strcmp(subkey, \"scriptcommand\")) {\n+\t\to->script_mode = 1;\n \t\treturn git_config_string(&o->cmd, var, value);\n+\t}\n+\tif (!strcmp(subkey, \"subprocesscommand\")) {\n+\t\to->script_mode = 0;\n+\t\treturn git_config_string(&o->cmd, var, value);\n+\t}\n \n \treturn 0;\n }\ndiff --git a/odb-helper.c b/odb-helper.c\nindex 3d940a3171..22728200e3 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -123,6 +123,9 @@ int odb_helper_init(struct odb_helper *o)\n \tFILE *fh;\n \tstruct strbuf line = STRBUF_INIT;\n \n+\tif (!o->script_mode)\n+\t\treturn 0;\n+\n \tif (odb_helper_start(o, &cmd, 0, \"init\") < 0)\n \t\treturn -1;\n \n@@ -173,16 +176,12 @@ static int odb_helper_object_cmp(const void *va, const void *vb)\n \treturn hashcmp(a->sha1, b->sha1);\n }\n \n-static void odb_helper_load_have(struct odb_helper *o)\n+static void have_object_script(struct odb_helper *o)\n {\n \tstruct odb_helper_cmd cmd;\n \tFILE *fh;\n \tstruct strbuf line = STRBUF_INIT;\n \n-\tif (o->have_valid)\n-\t\treturn;\n-\to->have_valid = 1;\n-\n \tif (odb_helper_start(o, &cmd, 0, \"have\") < 0)\n \t\treturn;\n \n@@ -194,6 +193,16 @@ static void odb_helper_load_have(struct odb_helper *o)\n \tstrbuf_release(&line);\n \tfclose(fh);\n \todb_helper_finish(o, &cmd);\n+}\n+\n+static void odb_helper_load_have(struct odb_helper *o)\n+{\n+\tif (o->have_valid)\n+\t\treturn;\n+\to->have_valid = 1;\n+\n+\tif (o->script_mode)\n+\t\thave_object_script(o);\n \n \tqsort(o->have, o->have_nr, sizeof(*o->have), odb_helper_object_cmp);\n }\ndiff --git a/odb-helper.h b/odb-helper.h\nindex fbb6d333ee..ee2b09b182 100644\n--- a/odb-helper.h\n+++ b/odb-helper.h\n@@ -15,6 +15,7 @@ struct odb_helper {\n \tconst char *name;\n \tconst char *cmd;\n \tunsigned int supported_capabilities;\n+\tint script_mode;\n \n \tstruct odb_helper_object {\n \t\tunsigned char sha1[20];\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328211","messageId":"20170916080731.13925-29-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 28/40] Add t0460 to test passing git objects","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:19Z","receivedAt":"2017-09-16T08:09:22Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n t/t0460-read-object-git.sh | 28 +++++++++++++++++\n t/t0460/read-object-git | 78 ++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 106 insertions(+)\n create mode 100755 t/t0460-read-object-git.sh\n create mode 100755 t/t0460/read-object-git\n\ndiff --git a/t/t0460-read-object-git.sh b/t/t0460-read-object-git.sh\nnew file mode 100755\nindex 0000000000..2873b445f3\n--- /dev/null\n+++ b/t/t0460-read-object-git.sh\n@@ -0,0 +1,28 @@\n+#!/bin/sh\n+\n+test_description='tests for long running read-object process passing git objects'\n+\n+. ./test-lib.sh\n+\n+PATH=\"$PATH:$TEST_DIRECTORY/t0460\"\n+\n+test_expect_success 'setup host repo with a root commit' '\n+\ttest_commit zero &&\n+\thash1=$(git ls-tree HEAD | grep zero.t | cut -f1 | cut -d\\ -f3)\n+'\n+\n+HELPER=\"read-object-git\"\n+\n+test_expect_success 'blobs can be retrieved from the host repo' '\n+\tgit init guest-repo &&\n+\t(cd guest-repo &&\n+\t git config odb.magic.subprocessCommand \"$HELPER\" &&\n+\t git cat-file blob \"$hash1\" >/dev/null)\n+'\n+\n+test_expect_success 'invalid blobs generate errors' '\n+\tcd guest-repo &&\n+\ttest_must_fail git cat-file blob \"invalid\"\n+'\n+\n+test_done\ndiff --git a/t/t0460/read-object-git b/t/t0460/read-object-git\nnew file mode 100755\nindex 0000000000..38529e622e\n--- /dev/null\n+++ b/t/t0460/read-object-git\n@@ -0,0 +1,78 @@\n+#!/usr/bin/perl\n+#\n+# Example implementation for the Git read-object protocol version 1\n+# See Documentation/technical/read-object-protocol.txt\n+#\n+# Allows you to test the ability for blobs to be pulled from a host git repo\n+# \"on demand.\" Called when git needs a blob it couldn't find locally due to\n+# a lazy clone that only cloned the commits and trees.\n+#\n+# A lazy clone can be simulated via the following commands from the host repo\n+# you wish to create a lazy clone of:\n+#\n+# cd /host_repo\n+# git rev-parse HEAD\n+# git init /guest_repo\n+# git cat-file --batch-check --batch-all-objects | grep -v 'blob' |\n+#\tcut -d' ' -f1 | git pack-objects /e/guest_repo/.git/objects/pack/noblobs\n+# cd /guest_repo\n+# git config core.virtualizeobjects true\n+# git reset --hard \n+#\n+# Please note, this sample is a minimal skeleton. No proper error handling \n+# was implemented.\n+#\n+\n+use 5.008;\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n+use strict;\n+use warnings;\n+use Git::Packet;\n+\n+#\n+# Point $DIR to the folder where your host git repo is located so we can pull\n+# missing objects from it\n+#\n+my $DIR = \"../.git/\";\n+\n+packet_initialize(\"git-read-object\", 1);\n+\n+packet_read_and_check_capabilities(\"get_git_obj\");\n+packet_write_capabilities(\"get_git_obj\");\n+\n+while (1) {\n+\tmy ($res, $command) = packet_txt_read();\n+\n+\tif ( $res == -1 ) {\n+\t\texit 0;\n+\t}\n+\n+\t$command =~ s/^command=//;\n+\n+\tif ( $command eq \"init\" ) {\n+\t\tpacket_bin_read();\n+\n+\t\tpacket_txt_write(\"status=success\");\n+\t\tpacket_flush();\n+\t} elsif ( $command eq \"get_git_obj\" ) {\n+\t\tmy ($sha1) = packet_txt_read() =~ /^sha1=([0-9a-f]{40})$/;\n+\t\tpacket_bin_read();\n+\n+\t\tmy $path = $sha1;\n+\t\t$path =~ s{..}{$&/};\n+\t\t$path = $DIR . \"/objects/\" . $path;\n+\n+\t\tmy $contents = do {\n+\t\t local $/;\n+\t\t open my $fh, $path or die \"Can't open '$path': $!\";\n+\t\t <$fh>\n+\t\t};\n+\n+\t\tpacket_bin_write($contents);\n+\t\tpacket_flush();\n+\t\tpacket_txt_write(\"status=success\");\n+\t\tpacket_flush();\n+\t} else {\n+\t\tdie \"bad command '$command'\";\n+\t}\n+}\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328212","messageId":"20170916080731.13925-24-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 23/40] Add t0420 to test transfer to HTTP external odb","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:14Z","receivedAt":"2017-09-16T08:09:30Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This tests that an apache web server can be used as an\nexternal object database and store files in their native\nformat instead of converting them to a Git object.\n\nSigned-off-by: Christian Couder \n---\n t/t0420-transfer-http-e-odb.sh | 142 +++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 142 insertions(+)\n create mode 100755 t/t0420-transfer-http-e-odb.sh\n\ndiff --git a/t/t0420-transfer-http-e-odb.sh b/t/t0420-transfer-http-e-odb.sh\nnew file mode 100755\nindex 0000000000..f84fe950ec\n--- /dev/null\n+++ b/t/t0420-transfer-http-e-odb.sh\n@@ -0,0 +1,142 @@\n+#!/bin/sh\n+\n+test_description='tests for transfering external objects to an HTTPD server'\n+\n+. ./test-lib.sh\n+\n+# If we don't specify a port, the current test number will be used\n+# which will not work as it is less than 1024, so it can only be used by root.\n+LIB_HTTPD_PORT=$(expr ${this_test#t} + 12000)\n+\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+\n+start_httpd apache-e-odb.conf\n+\n+# odb helper script must see this\n+export HTTPD_URL\n+\n+write_script odb-http-helper <<\\EOF\n+die() {\n+\tprintf >&2 \"%s\\n\" \"$@\"\n+\texit 1\n+}\n+echo >&2 \"odb-http-helper args:\" \"$@\"\n+case \"$1\" in\n+init)\n+\techo \"capability=get_raw_obj\"\n+\techo \"capability=put_raw_obj\"\n+\techo \"capability=have\"\n+\t;;\n+have)\n+\tlist_url=\"$HTTPD_URL/list/\"\n+\tcurl \"$list_url\" ||\n+\tdie \"curl '$list_url' failed\"\n+\t;;\n+get_raw_obj)\n+\tget_url=\"$HTTPD_URL/list/?sha1=$2\"\n+\tcurl \"$get_url\" ||\n+\tdie \"curl '$get_url' failed\"\n+\t;;\n+put_raw_obj)\n+\tsha1=\"$2\"\n+\tsize=\"$3\"\n+\tkind=\"$4\"\n+\tupload_url=\"$HTTPD_URL/upload/?sha1=$sha1&size=$size&type=$kind\"\n+\tcurl --data-binary @- --include \"$upload_url\" >out ||\n+\tdie \"curl '$upload_url' failed\"\n+\tref_hash=$(echo \"$sha1 $size $kind\" | GIT_NO_EXTERNAL_ODB=1 git hash-object -w -t blob --stdin) || exit\n+\tgit update-ref refs/odbs/magic/\"$sha1\" \"$ref_hash\"\n+\t;;\n+*)\n+\tdie \"unknown command '$1'\"\n+\t;;\n+esac\n+EOF\n+HELPER=\"\\\"$PWD\\\"/odb-http-helper\"\n+\n+test_expect_success 'setup repo with a root commit and the helper' '\n+\ttest_commit zero &&\n+\tgit config odb.magic.scriptCommand \"$HELPER\"\n+'\n+\n+test_expect_success 'setup another repo from the first one' '\n+\tgit init other-repo &&\n+\t(cd other-repo &&\n+\t git remote add origin .. &&\n+\t git pull origin master &&\n+\t git checkout master &&\n+\t git log)\n+'\n+\n+UPLOADFILENAME=\"hello_apache_upload.txt\"\n+\n+UPLOAD_URL=\"$HTTPD_URL/upload/?sha1=$UPLOADFILENAME&size=123&type=blob\"\n+\n+test_expect_success 'can upload a file' '\n+\techo \"Hello Apache World!\" >hello_to_send.txt &&\n+\techo \"How are you?\" >>hello_to_send.txt &&\n+\tcurl --data-binary @hello_to_send.txt --include \"$UPLOAD_URL\" >out_upload\n+'\n+\n+LIST_URL=\"$HTTPD_URL/list/\"\n+\n+test_expect_success 'can list uploaded files' '\n+\tcurl --include \"$LIST_URL\" >out_list &&\n+\tgrep \"$UPLOADFILENAME\" out_list\n+'\n+\n+test_expect_success 'can delete uploaded files' '\n+\tcurl --data \"delete\" --include \"$UPLOAD_URL&delete=1\" >out_delete &&\n+\tcurl --include \"$LIST_URL\" >out_list2 &&\n+\t! grep \"$UPLOADFILENAME\" out_list2\n+'\n+\n+FILES_DIR=\"httpd/www/files\"\n+\n+test_expect_success 'new blobs are transfered to the http server' '\n+\ttest_commit one &&\n+\thash1=$(git ls-tree HEAD | grep one.t | cut -f1 | cut -d\\ -f3) &&\n+\techo \"$hash1-4-blob\" >expected &&\n+\tls \"$FILES_DIR\" >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'blobs can be retrieved from the http server' '\n+\tgit cat-file blob \"$hash1\" &&\n+\tgit log -p >expected\n+'\n+\n+test_expect_success 'update other repo from the first one' '\n+\t(cd other-repo &&\n+\t git fetch origin \"refs/odbs/magic/*:refs/odbs/magic/*\" &&\n+\t test_must_fail git cat-file blob \"$hash1\" &&\n+\t git config odb.magic.scriptCommand \"$HELPER\" &&\n+\t git cat-file blob \"$hash1\" &&\n+\t git pull origin master)\n+'\n+\n+test_expect_success 'local clone from the first repo' '\n+\tmkdir my-clone &&\n+\t(cd my-clone &&\n+\t git clone .. . &&\n+\t git cat-file blob \"$hash1\")\n+'\n+\n+test_expect_success 'no-local clone from the first repo fails' '\n+\tmkdir my-other-clone &&\n+\t(cd my-other-clone &&\n+\t test_must_fail git clone --no-local .. .) &&\n+\trm -rf my-other-clone\n+'\n+\n+test_expect_success 'no-local clone from the first repo with helper succeeds' '\n+\tmkdir my-other-clone &&\n+\t(cd my-other-clone &&\n+\t git clone -c odb.magic.scriptCommand=\"$HELPER\" \\\n+\t\t--no-local .. .) &&\n+\trm -rf my-other-clone\n+'\n+\n+stop_httpd\n+\n+test_done\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328213","messageId":"20170916080731.13925-22-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 21/40] odb-helper: add odb_helper_get_raw_object()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:12Z","receivedAt":"2017-09-16T08:09:32Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"The existing odb_helper_get_object() is renamed\nodb_helper_get_git_object() and a new odb_helper_get_raw_object()\nis introduced to deal with external objects that are not in Git format.\n\nSigned-off-by: Christian Couder \n---\n odb-helper.c | 113 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 111 insertions(+), 2 deletions(-)\n\ndiff --git a/odb-helper.c b/odb-helper.c\nindex 39d20fdfd7..1f4666b349 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -221,8 +221,107 @@ int odb_helper_has_object(struct odb_helper *o, const unsigned char *sha1)\n \treturn !!odb_helper_lookup(o, sha1);\n }\n \n-int odb_helper_get_object(struct odb_helper *o, const unsigned char *sha1,\n-\t\t\t int fd)\n+static int odb_helper_get_raw_object(struct odb_helper *o,\n+\t\t\t\t const unsigned char *sha1,\n+\t\t\t\t int fd)\n+{\n+\tstruct odb_helper_object *obj;\n+\tstruct odb_helper_cmd cmd;\n+\tunsigned long total_got = 0;\n+\n+\tchar hdr[32];\n+\tint hdrlen;\n+\n+\tint ret = Z_STREAM_END;\n+\tunsigned char compressed[4096];\n+\tgit_zstream stream;\n+\tgit_SHA_CTX hash;\n+\tunsigned char real_sha1[20];\n+\n+\tobj = odb_helper_lookup(o, sha1);\n+\tif (!obj)\n+\t\treturn -1;\n+\n+\tif (odb_helper_start(o, &cmd, 0, \"get_raw_obj %s\", sha1_to_hex(sha1)) < 0)\n+\t\treturn -1;\n+\n+\t/* Set it up */\n+\tgit_deflate_init(&stream, zlib_compression_level);\n+\tstream.next_out = compressed;\n+\tstream.avail_out = sizeof(compressed);\n+\tgit_SHA1_Init(&hash);\n+\n+\t/* First header.. */\n+\thdrlen = xsnprintf(hdr, sizeof(hdr), \"%s %lu\", typename(obj->type), obj->size) + 1;\n+\tstream.next_in = (unsigned char *)hdr;\n+\tstream.avail_in = hdrlen;\n+\twhile (git_deflate(&stream, 0) == Z_OK)\n+\t\t; /* nothing */\n+\tgit_SHA1_Update(&hash, hdr, hdrlen);\n+\n+\tfor (;;) {\n+\t\tunsigned char buf[4096];\n+\t\tint r;\n+\n+\t\tr = xread(cmd.child.out, buf, sizeof(buf));\n+\t\tif (r < 0) {\n+\t\t\terror(\"unable to read from odb helper '%s': %s\",\n+\t\t\t o->name, strerror(errno));\n+\t\t\tclose(cmd.child.out);\n+\t\t\todb_helper_finish(o, &cmd);\n+\t\t\tgit_deflate_end(&stream);\n+\t\t\treturn -1;\n+\t\t}\n+\t\tif (r == 0)\n+\t\t\tbreak;\n+\n+\t\ttotal_got += r;\n+\n+\t\t/* Then the data itself.. */\n+\t\tstream.next_in = (void *)buf;\n+\t\tstream.avail_in = r;\n+\t\tdo {\n+\t\t\tunsigned char *in0 = stream.next_in;\n+\t\t\tret = git_deflate(&stream, Z_FINISH);\n+\t\t\tgit_SHA1_Update(&hash, in0, stream.next_in - in0);\n+\t\t\twrite_or_die(fd, compressed, stream.next_out - compressed);\n+\t\t\tstream.next_out = compressed;\n+\t\t\tstream.avail_out = sizeof(compressed);\n+\t\t} while (ret == Z_OK);\n+\t}\n+\n+\tclose(cmd.child.out);\n+\tif (ret != Z_STREAM_END) {\n+\t\twarning(\"bad zlib data from odb helper '%s' for %s\",\n+\t\t\to->name, sha1_to_hex(sha1));\n+\t\treturn -1;\n+\t}\n+\tret = git_deflate_end_gently(&stream);\n+\tif (ret != Z_OK) {\n+\t\twarning(\"deflateEnd on object %s from odb helper '%s' failed (%d)\",\n+\t\t\tsha1_to_hex(sha1), o->name, ret);\n+\t\treturn -1;\n+\t}\n+\tgit_SHA1_Final(real_sha1, &hash);\n+\tif (hashcmp(sha1, real_sha1)) {\n+\t\twarning(\"sha1 mismatch from odb helper '%s' for %s (got %s)\",\n+\t\t\to->name, sha1_to_hex(sha1), sha1_to_hex(real_sha1));\n+\t\treturn -1;\n+\t}\n+\tif (odb_helper_finish(o, &cmd))\n+\t\treturn -1;\n+\tif (total_got != obj->size) {\n+\t\twarning(\"size mismatch from odb helper '%s' for %s (%lu != %lu)\",\n+\t\t\to->name, sha1_to_hex(sha1), total_got, obj->size);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int odb_helper_get_git_object(struct odb_helper *o,\n+\t\t\t\t const unsigned char *sha1,\n+\t\t\t\t int fd)\n {\n \tstruct odb_helper_object *obj;\n \tstruct odb_helper_cmd cmd;\n@@ -326,6 +425,16 @@ int odb_helper_get_object(struct odb_helper *o, const unsigned char *sha1,\n \treturn 0;\n }\n \n+int odb_helper_get_object(struct odb_helper *o,\n+\t\t\t const unsigned char *sha1,\n+\t\t\t int fd)\n+{\n+\tif (o->supported_capabilities & ODB_HELPER_CAP_GET_RAW_OBJ)\n+\t\treturn odb_helper_get_raw_object(o, sha1, fd);\n+\telse\n+\t\treturn odb_helper_get_git_object(o, sha1, fd);\n+}\n+\n int odb_helper_put_object(struct odb_helper *o,\n \t\t\t const void *buf, size_t len,\n \t\t\t const char *type, unsigned char *sha1)\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328214","messageId":"20170916080731.13925-21-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 20/40] lib-httpd: add apache-e-odb.conf","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:11Z","receivedAt":"2017-09-16T08:09:34Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This is an apache config file to test external object databases.\nIt uses the upload.sh and list.sh cgi that have been added\npreviously to make apache store external objects.\n\nSigned-off-by: Christian Couder \n---\n t/lib-httpd/apache-e-odb.conf | 214 ++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 214 insertions(+)\n create mode 100644 t/lib-httpd/apache-e-odb.conf\n\ndiff --git a/t/lib-httpd/apache-e-odb.conf b/t/lib-httpd/apache-e-odb.conf\nnew file mode 100644\nindex 0000000000..19a1540c82\n--- /dev/null\n+++ b/t/lib-httpd/apache-e-odb.conf\n@@ -0,0 +1,214 @@\n+ServerName dummy\n+PidFile httpd.pid\n+DocumentRoot www\n+LogFormat \"%h %l %u %t \\\"%r\\\" %>s %b\" common\n+CustomLog access.log common\n+ErrorLog error.log\n+\n+\tLoadModule log_config_module modules/mod_log_config.so\n+\n+\n+\tLoadModule alias_module modules/mod_alias.so\n+\n+\n+\tLoadModule cgi_module modules/mod_cgi.so\n+\n+\n+\tLoadModule env_module modules/mod_env.so\n+\n+\n+\tLoadModule rewrite_module modules/mod_rewrite.so\n+\n+\n+\tLoadModule version_module modules/mod_version.so\n+\n+\n+\tLoadModule headers_module modules/mod_headers.so\n+\n+\n+\n+LockFile accept.lock\n+\n+\n+\n+\n+\tLoadModule auth_module modules/mod_auth.so\n+\n+\n+\n+= 2.1>\n+\n+\tLoadModule auth_basic_module modules/mod_auth_basic.so\n+\n+\n+\tLoadModule authn_file_module modules/mod_authn_file.so\n+\n+\n+\tLoadModule authz_user_module modules/mod_authz_user.so\n+\n+\n+\tLoadModule authz_host_module modules/mod_authz_host.so\n+\n+\n+\n+= 2.4>\n+\n+\tLoadModule authn_core_module modules/mod_authn_core.so\n+\n+\n+\tLoadModule authz_core_module modules/mod_authz_core.so\n+\n+\n+\tLoadModule access_compat_module modules/mod_access_compat.so\n+\n+\n+\tLoadModule mpm_prefork_module modules/mod_mpm_prefork.so\n+\n+\n+\tLoadModule unixd_module modules/mod_unixd.so\n+\n+\n+\n+PassEnv GIT_VALGRIND\n+PassEnv GIT_VALGRIND_OPTIONS\n+PassEnv GNUPGHOME\n+PassEnv ASAN_OPTIONS\n+PassEnv GIT_TRACE\n+PassEnv GIT_CONFIG_NOSYSTEM\n+\n+Alias /dumb/ www/\n+Alias /auth/dumb/ www/auth/dumb/\n+\n+\n+\tSetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}\n+\tSetEnv GIT_HTTP_EXPORT_ALL\n+\n+\n+\tSetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}\n+\n+\n+\tSetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}\n+\tSetEnv GIT_HTTP_EXPORT_ALL\n+\tSetEnv GIT_COMMITTER_NAME \"Custom User\"\n+\tSetEnv GIT_COMMITTER_EMAIL custom@example.com\n+\n+\n+\tSetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}\n+\tSetEnv GIT_HTTP_EXPORT_ALL\n+\tSetEnv GIT_NAMESPACE ns\n+\n+\n+\tSetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}\n+\tSetEnv GIT_HTTP_EXPORT_ALL\n+\tHeader set Set-Cookie name=value\n+\n+\n+\tSetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}\n+\tSetEnv GIT_HTTP_EXPORT_ALL\n+\n+ScriptAlias /upload/ upload.sh/\n+ScriptAlias /list/ list.sh/\n+\n+\tOptions FollowSymlinks\n+\n+\n+ Options ExecCGI\n+\n+\n+ Options ExecCGI\n+\n+\n+\tOptions ExecCGI\n+\n+\n+RewriteEngine on\n+RewriteRule ^/smart-redir-perm/(.*)$ /smart/$1 [R=301]\n+RewriteRule ^/smart-redir-temp/(.*)$ /smart/$1 [R=302]\n+RewriteRule ^/smart-redir-auth/(.*)$ /auth/smart/$1 [R=301]\n+RewriteRule ^/smart-redir-limited/(.*)/info/refs$ /smart/$1/info/refs [R=301]\n+RewriteRule ^/ftp-redir/(.*)$ ftp://localhost:1000/$1 [R=302]\n+\n+RewriteRule ^/loop-redir/x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-(.*) /$1 [R=302]\n+RewriteRule ^/loop-redir/(.*)$ /loop-redir/x-$1 [R=302]\n+\n+# Apache 2.2 does not understand , so we use RewriteCond.\n+# And as RewriteCond does not allow testing for non-matches, we match\n+# the desired case first (one has abra, two has cadabra), and let it\n+# pass by marking the RewriteRule as [L], \"last rule, do not process\n+# any other matching RewriteRules after this\"), and then have another\n+# RewriteRule that matches all other cases and lets them fail via '[F]',\n+# \"fail the request\".\n+RewriteCond %{HTTP:x-magic-one} =abra\n+RewriteCond %{HTTP:x-magic-two} =cadabra\n+RewriteRule ^/smart_headers/.* - [L]\n+RewriteRule ^/smart_headers/.* - [F]\n+\n+\n+LoadModule ssl_module modules/mod_ssl.so\n+\n+SSLCertificateFile httpd.pem\n+SSLCertificateKeyFile httpd.pem\n+SSLRandomSeed startup file:/dev/urandom 512\n+SSLRandomSeed connect file:/dev/urandom 512\n+SSLSessionCache none\n+SSLMutex file:ssl_mutex\n+SSLEngine On\n+\n+\n+\n+\tAuthType Basic\n+\tAuthName \"git-auth\"\n+\tAuthUserFile passwd\n+\tRequire valid-user\n+\n+\n+\n+\tAuthType Basic\n+\tAuthName \"git-auth\"\n+\tAuthUserFile passwd\n+\tRequire valid-user\n+\n+\n+\n+\tAuthType Basic\n+\tAuthName \"git-auth\"\n+\tAuthUserFile passwd\n+\tRequire valid-user\n+\n+\n+RewriteCond %{QUERY_STRING} service=git-receive-pack [OR]\n+RewriteCond %{REQUEST_URI} /git-receive-pack$\n+RewriteRule ^/half-auth-complete/ - [E=AUTHREQUIRED:yes]\n+\n+\n+ Order Deny,Allow\n+ Deny from env=AUTHREQUIRED\n+\n+ AuthType Basic\n+ AuthName \"Git Access\"\n+ AuthUserFile passwd\n+ Require valid-user\n+ Satisfy Any\n+\n+\n+\n+\tLoadModule dav_module modules/mod_dav.so\n+\tLoadModule dav_fs_module modules/mod_dav_fs.so\n+\n+\tDAVLockDB DAVLock\n+\t\n+\t\tDav on\n+\t\n+\t\n+\t\tDav on\n+\t\n+\n+\n+\n+\tLoadModule dav_svn_module modules/mod_dav_svn.so\n+\n+\t\n+\t\tDAV svn\n+\t\tSVNPath \"${LIB_HTTPD_SVNPATH}\"\n+\t\n+\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328215","messageId":"20170916080731.13925-19-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 18/40] lib-httpd: add upload.sh","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:09Z","receivedAt":"2017-09-16T08:09:36Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This cgi will be used to upload objects to, or to delete\nobjects from, an apache web server.\n\nThis way the apache server can work as an external object\ndatabase.\n\nSigned-off-by: Christian Couder \n---\n t/lib-httpd.sh | 1 +\n t/lib-httpd/upload.sh | 45 +++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 46 insertions(+)\n create mode 100644 t/lib-httpd/upload.sh\n\ndiff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\nindex 2e659a8ee2..d80b004549 100644\n--- a/t/lib-httpd.sh\n+++ b/t/lib-httpd.sh\n@@ -132,6 +132,7 @@ prepare_httpd() {\n \tcp \"$TEST_PATH\"/passwd \"$HTTPD_ROOT_PATH\"\n \tinstall_script broken-smart-http.sh\n \tinstall_script error.sh\n+\tinstall_script upload.sh\n \n \tln -s \"$LIB_HTTPD_MODULE_PATH\" \"$HTTPD_ROOT_PATH/modules\"\n \ndiff --git a/t/lib-httpd/upload.sh b/t/lib-httpd/upload.sh\nnew file mode 100644\nindex 0000000000..64d3f31c31\n--- /dev/null\n+++ b/t/lib-httpd/upload.sh\n@@ -0,0 +1,45 @@\n+#!/bin/sh\n+\n+# In part from http://codereview.stackexchange.com/questions/79549/bash-cgi-upload-file\n+\n+FILES_DIR=\"www/files\"\n+\n+OLDIFS=\"$IFS\"\n+IFS='&'\n+set -- $QUERY_STRING\n+IFS=\"$OLDIFS\"\n+\n+while test $# -gt 0\n+do\n+\tkey=${1%%=*}\n+\tval=${1#*=}\n+\n+\tcase \"$key\" in\n+\t\"sha1\") sha1=\"$val\" ;;\n+\t\"type\") type=\"$val\" ;;\n+\t\"size\") size=\"$val\" ;;\n+\t\"delete\") delete=1 ;;\n+\t*) echo >&2 \"unknown key '$key'\" ;;\n+\tesac\n+\n+\tshift\n+done\n+\n+case \"$REQUEST_METHOD\" in\n+POST)\n+\tif test \"$delete\" = \"1\"\n+\tthen\n+\t\trm -f \"$FILES_DIR/$sha1-$size-$type\"\n+\telse\n+\t\tmkdir -p \"$FILES_DIR\"\n+\t\tcat >\"$FILES_DIR/$sha1-$size-$type\"\n+\tfi\n+\n+\techo 'Status: 204 No Content'\n+\techo\n+\t;;\n+\n+*)\n+\techo 'Status: 405 Method Not Allowed'\n+\techo\n+esac\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328216","messageId":"20170916080731.13925-25-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 24/40] external-odb: add 'get_direct' support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:15Z","receivedAt":"2017-09-16T08:09:45Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This implements the 'get_direct' capability/instruction that makes\nit possible for external odb helper scripts to pass blobs to Git\nby directly writing them as loose objects files.\n\nIt is better to call this a \"direct\" mode rather than a \"fault-in\"\nmode as we could have the same kind of mechanism to \"put\" objects\ninto an external odb, where the odb helper would access blobs it\nwants to send to an external odb directly from files, but it\nwould be strange to call that a fault-in mode too.\n\nSigned-off-by: Christian Couder \n---\n external-odb.c | 21 ++++++++++++++++++++-\n external-odb.h | 1 +\n odb-helper.c | 27 +++++++++++++++++++++++++--\n odb-helper.h | 2 ++\n 4 files changed, 48 insertions(+), 3 deletions(-)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex 52cb448d01..31d21bfe04 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -96,7 +96,8 @@ int external_odb_get_object(const unsigned char *sha1)\n \t\tint ret;\n \t\tint fd;\n \n-\t\tif (!odb_helper_has_object(o, sha1))\n+\t\tif (!(o->supported_capabilities & ODB_HELPER_CAP_GET_RAW_OBJ) &&\n+\t\t !(o->supported_capabilities & ODB_HELPER_CAP_GET_GIT_OBJ))\n \t\t\tcontinue;\n \n \t\tfd = create_object_tmpfile(&tmpfile, path);\n@@ -122,6 +123,24 @@ int external_odb_get_object(const unsigned char *sha1)\n \treturn -1;\n }\n \n+int external_odb_get_direct(const unsigned char *sha1)\n+{\n+\tstruct odb_helper *o;\n+\n+\tif (!external_odb_has_object(sha1))\n+\t\treturn -1;\n+\n+\tfor (o = helpers; o; o = o->next) {\n+\t\tif (!(o->supported_capabilities & ODB_HELPER_CAP_GET_DIRECT))\n+\t\t\tcontinue;\n+\t\tif (odb_helper_get_direct(o, sha1) < 0)\n+\t\t\tcontinue;\n+\t\treturn 0;\n+\t}\n+\n+\treturn -1;\n+}\n+\n int external_odb_put_object(const void *buf, size_t len,\n \t\t\t const char *type, unsigned char *sha1)\n {\ndiff --git a/external-odb.h b/external-odb.h\nindex d369dfdf6f..1fda08c0fb 100644\n--- a/external-odb.h\n+++ b/external-odb.h\n@@ -4,6 +4,7 @@\n extern const char *external_odb_root(void);\n extern int external_odb_has_object(const unsigned char *sha1);\n extern int external_odb_get_object(const unsigned char *sha1);\n+extern int external_odb_get_direct(const unsigned char *sha1);\n extern int external_odb_put_object(const void *buf, size_t len,\n \t\t\t\t const char *type, unsigned char *sha1);\n \ndiff --git a/odb-helper.c b/odb-helper.c\nindex 1f4666b349..3d940a3171 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -425,14 +425,37 @@ static int odb_helper_get_git_object(struct odb_helper *o,\n \treturn 0;\n }\n \n+int odb_helper_get_direct(struct odb_helper *o,\n+\t\t\t const unsigned char *sha1)\n+{\n+\tstruct odb_helper_object *obj;\n+\tstruct odb_helper_cmd cmd;\n+\n+\tobj = odb_helper_lookup(o, sha1);\n+\tif (!obj)\n+\t\treturn -1;\n+\n+\tif (odb_helper_start(o, &cmd, 0, \"get_direct %s\", sha1_to_hex(sha1)) < 0)\n+\t\treturn -1;\n+\n+\tif (odb_helper_finish(o, &cmd))\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n int odb_helper_get_object(struct odb_helper *o,\n \t\t\t const unsigned char *sha1,\n \t\t\t int fd)\n {\n+\tif (o->supported_capabilities & ODB_HELPER_CAP_GET_GIT_OBJ)\n+\t\treturn odb_helper_get_git_object(o, sha1, fd);\n \tif (o->supported_capabilities & ODB_HELPER_CAP_GET_RAW_OBJ)\n \t\treturn odb_helper_get_raw_object(o, sha1, fd);\n-\telse\n-\t\treturn odb_helper_get_git_object(o, sha1, fd);\n+\tif (o->supported_capabilities & ODB_HELPER_CAP_GET_DIRECT)\n+\t\treturn 0;\n+\n+\tBUG(\"invalid get capability (capabilities: '%d')\", o->supported_capabilities);\n }\n \n int odb_helper_put_object(struct odb_helper *o,\ndiff --git a/odb-helper.h b/odb-helper.h\nindex 0571ba09cb..fbb6d333ee 100644\n--- a/odb-helper.h\n+++ b/odb-helper.h\n@@ -35,6 +35,8 @@ extern int odb_helper_has_object(struct odb_helper *o,\n extern int odb_helper_get_object(struct odb_helper *o,\n \t\t\t\t const unsigned char *sha1,\n \t\t\t\t int fd);\n+extern int odb_helper_get_direct(struct odb_helper *o,\n+\t\t\t\t const unsigned char *sha1);\n extern int odb_helper_put_object(struct odb_helper *o,\n \t\t\t\t const void *buf, size_t len,\n \t\t\t\t const char *type, unsigned char *sha1);\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328217","messageId":"20170916080731.13925-28-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 27/40] Add t0450 to test 'get_direct' mechanism","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:18Z","receivedAt":"2017-09-16T08:09:48Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: Ben Peart \n\nSigned-off-by: Ben Peart \nSigned-off-by: Christian Couder \n---\n t/t0450-read-object.sh | 28 +++++++++++++++++++++\n t/t0450/read-object | 68 ++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 96 insertions(+)\n create mode 100755 t/t0450-read-object.sh\n create mode 100755 t/t0450/read-object\n\ndiff --git a/t/t0450-read-object.sh b/t/t0450-read-object.sh\nnew file mode 100755\nindex 0000000000..6b97305452\n--- /dev/null\n+++ b/t/t0450-read-object.sh\n@@ -0,0 +1,28 @@\n+#!/bin/sh\n+\n+test_description='tests for long running read-object process'\n+\n+. ./test-lib.sh\n+\n+PATH=\"$PATH:$TEST_DIRECTORY/t0450\"\n+\n+test_expect_success 'setup host repo with a root commit' '\n+\ttest_commit zero &&\n+\thash1=$(git ls-tree HEAD | grep zero.t | cut -f1 | cut -d\\ -f3)\n+'\n+\n+HELPER=\"read-object\"\n+\n+test_expect_success 'blobs can be retrieved from the host repo' '\n+\tgit init guest-repo &&\n+\t(cd guest-repo &&\n+\t git config odb.magic.subprocessCommand \"$HELPER\" &&\n+\t git cat-file blob \"$hash1\" >/dev/null)\n+'\n+\n+test_expect_success 'invalid blobs generate errors' '\n+\tcd guest-repo &&\n+\ttest_must_fail git cat-file blob \"invalid\"\n+'\n+\n+test_done\ndiff --git a/t/t0450/read-object b/t/t0450/read-object\nnew file mode 100755\nindex 0000000000..cf22e2f581\n--- /dev/null\n+++ b/t/t0450/read-object\n@@ -0,0 +1,68 @@\n+#!/usr/bin/perl\n+#\n+# Example implementation for the Git read-object protocol version 1\n+# See Documentation/technical/read-object-protocol.txt\n+#\n+# Allows you to test the ability for blobs to be pulled from a host git repo\n+# \"on demand.\" Called when git needs a blob it couldn't find locally due to\n+# a lazy clone that only cloned the commits and trees.\n+#\n+# A lazy clone can be simulated via the following commands from the host repo\n+# you wish to create a lazy clone of:\n+#\n+# cd /host_repo\n+# git rev-parse HEAD\n+# git init /guest_repo\n+# git cat-file --batch-check --batch-all-objects | grep -v 'blob' |\n+#\tcut -d' ' -f1 | git pack-objects /e/guest_repo/.git/objects/pack/noblobs\n+# cd /guest_repo\n+# git config core.virtualizeobjects true\n+# git reset --hard \n+#\n+# Please note, this sample is a minimal skeleton. No proper error handling \n+# was implemented.\n+#\n+\n+use 5.008;\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n+use strict;\n+use warnings;\n+use Git::Packet;\n+\n+#\n+# Point $DIR to the folder where your host git repo is located so we can pull\n+# missing objects from it\n+#\n+my $DIR = \"../.git/\";\n+\n+packet_initialize(\"git-read-object\", 1);\n+\n+packet_read_and_check_capabilities(\"get_direct\");\n+packet_write_capabilities(\"get_direct\");\n+\n+while (1) {\n+\tmy ($res, $command) = packet_txt_read();\n+\n+\tif ( $res == -1 ) {\n+\t\texit 0;\n+\t}\n+\n+\t$command =~ s/^command=//;\n+\n+\tif ( $command eq \"init\" ) {\n+\t\tpacket_bin_read();\n+\n+\t\tpacket_txt_write(\"status=success\");\n+\t\tpacket_flush();\n+\t} elsif ( $command eq \"get_direct\" ) {\n+\t\tmy ($sha1) = packet_txt_read() =~ /^sha1=([0-9a-f]{40})$/;\n+\t\tpacket_bin_read();\n+\n+\t\tsystem ('git --git-dir=\"' . $DIR . '\" cat-file blob ' . $sha1 . ' | GIT_NO_EXTERNAL_ODB=1 git hash-object -w --stdin >/dev/null 2>&1');\n+\n+\t\tpacket_txt_write(($?) ? \"status=error\" : \"status=success\");\n+\t\tpacket_flush();\n+\t} else {\n+\t\tdie \"bad command '$command'\";\n+\t}\n+}\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328218","messageId":"20170916080731.13925-16-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 15/40] Add GIT_NO_EXTERNAL_ODB env variable","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:06Z","receivedAt":"2017-09-16T08:09:53Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This new environment variable will be used to perform git\ncommands without involving any external odb mechanism.\n\nThis makes it possible for example to create new blobs that\nwill not be sent to an external odb even if the external odb\nsupports \"put_*\" instructions.\n\nSigned-off-by: Christian Couder \n---\n cache.h | 9 +++++++++\n environment.c | 4 ++++\n external-odb.c | 6 ++++++\n sha1_file.c | 3 +++\n 4 files changed, 22 insertions(+)\n\ndiff --git a/cache.h b/cache.h\nindex 6c22bd0525..a2bd2090b3 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -429,6 +429,7 @@ static inline enum object_type object_type(unsigned int mode)\n #define CEILING_DIRECTORIES_ENVIRONMENT \"GIT_CEILING_DIRECTORIES\"\n #define NO_REPLACE_OBJECTS_ENVIRONMENT \"GIT_NO_REPLACE_OBJECTS\"\n #define GIT_REPLACE_REF_BASE_ENVIRONMENT \"GIT_REPLACE_REF_BASE\"\n+#define NO_EXTERNAL_ODB_ENVIRONMENT \"GIT_NO_EXTERNAL_ODB\"\n #define GITATTRIBUTES_FILE \".gitattributes\"\n #define INFOATTRIBUTES_FILE \"info/attributes\"\n #define ATTRIBUTE_MACRO_PREFIX \"[attr]\"\n@@ -767,6 +768,14 @@ void reset_shared_repository(void);\n extern int check_replace_refs;\n extern char *git_replace_ref_base;\n \n+/*\n+ * Do external odbs need to be used this run? This variable is\n+ * initialized to true unless $GIT_NO_EXTERNAL_ODB is set, but it\n+ * maybe set to false by some commands that do not want external\n+ * odbs to be active.\n+ */\n+extern int use_external_odb;\n+\n extern int fsync_object_files;\n extern int core_preload_index;\n extern int core_apply_sparse_checkout;\ndiff --git a/environment.c b/environment.c\nindex 3fd4b10845..bbccabef6b 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -48,6 +48,7 @@ const char *excludes_file;\n enum auto_crlf auto_crlf = AUTO_CRLF_FALSE;\n int check_replace_refs = 1;\n char *git_replace_ref_base;\n+int use_external_odb = 1;\n enum eol core_eol = EOL_UNSET;\n enum safe_crlf safe_crlf = SAFE_CRLF_WARN;\n unsigned whitespace_rule_cfg = WS_DEFAULT_RULE;\n@@ -116,6 +117,7 @@ const char * const local_repo_env[] = {\n \tINDEX_ENVIRONMENT,\n \tNO_REPLACE_OBJECTS_ENVIRONMENT,\n \tGIT_REPLACE_REF_BASE_ENVIRONMENT,\n+\tNO_EXTERNAL_ODB_ENVIRONMENT,\n \tGIT_PREFIX_ENVIRONMENT,\n \tGIT_SUPER_PREFIX_ENVIRONMENT,\n \tGIT_SHALLOW_FILE_ENVIRONMENT,\n@@ -154,6 +156,8 @@ void setup_git_env(void)\n \treplace_ref_base = getenv(GIT_REPLACE_REF_BASE_ENVIRONMENT);\n \tgit_replace_ref_base = xstrdup(replace_ref_base ? replace_ref_base\n \t\t\t\t\t\t\t : \"refs/replace/\");\n+\tif (getenv(NO_EXTERNAL_ODB_ENVIRONMENT))\n+\t\tuse_external_odb = 0;\n \tnamespace = expand_namespace(getenv(GIT_NAMESPACE_ENVIRONMENT));\n \tshallow_file = getenv(GIT_SHALLOW_FILE_ENVIRONMENT);\n \tif (shallow_file)\ndiff --git a/external-odb.c b/external-odb.c\nindex a4f8c72e1c..52cb448d01 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -65,6 +65,9 @@ int external_odb_has_object(const unsigned char *sha1)\n {\n \tstruct odb_helper *o;\n \n+\tif (!use_external_odb)\n+\t\treturn 0;\n+\n \texternal_odb_init();\n \n \tfor (o = helpers; o; o = o->next) {\n@@ -124,6 +127,9 @@ int external_odb_put_object(const void *buf, size_t len,\n {\n \tstruct odb_helper *o;\n \n+\tif (!use_external_odb)\n+\t\treturn 1;\n+\n \t/* For now accept only blobs */\n \tif (strcmp(type, \"blob\"))\n \t\treturn 1;\ndiff --git a/sha1_file.c b/sha1_file.c\nindex d0155e392f..7b2a0f64fa 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -619,6 +619,9 @@ void prepare_external_alt_odb(void)\n \tstatic int linked_external;\n \tconst char *path;\n \n+\tif (!use_external_odb)\n+\t\treturn;\n+\n \tif (linked_external)\n \t\treturn;\n \n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328219","messageId":"20170916080731.13925-14-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 13/40] external-odb: accept only blobs for now","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:04Z","receivedAt":"2017-09-16T08:10:02Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"The mechanism to decide which blobs should be sent to which\nexternal object database will be very simple for now.\nIf the external odb helper support any \"put_*\" instruction\nall the new blobs will be sent to it.\n\nSigned-off-by: Christian Couder \n---\n external-odb.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex 82fac702e8..a4f8c72e1c 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -124,6 +124,10 @@ int external_odb_put_object(const void *buf, size_t len,\n {\n \tstruct odb_helper *o;\n \n+\t/* For now accept only blobs */\n+\tif (strcmp(type, \"blob\"))\n+\t\treturn 1;\n+\n \texternal_odb_init();\n \n \tfor (o = helpers; o; o = o->next) {\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328220","messageId":"20170916080731.13925-18-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 17/40] lib-httpd: pass config file to start_httpd()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:08Z","receivedAt":"2017-09-16T08:10:04Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This makes it possible to start an apache web server with different\nconfig files.\n\nThis will be used in a later patch to pass a config file that makes\napache store external objects.\n\nSigned-off-by: Christian Couder \n---\n t/lib-httpd.sh | 6 ++++--\n 1 file changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\nindex 435a37465a..2e659a8ee2 100644\n--- a/t/lib-httpd.sh\n+++ b/t/lib-httpd.sh\n@@ -171,12 +171,14 @@ prepare_httpd() {\n }\n \n start_httpd() {\n+\tAPACHE_CONF_FILE=${1-apache.conf}\n+\n \tprepare_httpd >&3 2>&4\n \n \ttrap 'code=$?; stop_httpd; (exit $code); die' EXIT\n \n \t\"$LIB_HTTPD_PATH\" -d \"$HTTPD_ROOT_PATH\" \\\n-\t\t-f \"$TEST_PATH/apache.conf\" $HTTPD_PARA \\\n+\t\t-f \"$TEST_PATH/$APACHE_CONF_FILE\" $HTTPD_PARA \\\n \t\t-c \"Listen 127.0.0.1:$LIB_HTTPD_PORT\" -k start \\\n \t\t>&3 2>&4\n \tif test $? -ne 0\n@@ -191,7 +193,7 @@ stop_httpd() {\n \ttrap 'die' EXIT\n \n \t\"$LIB_HTTPD_PATH\" -d \"$HTTPD_ROOT_PATH\" \\\n-\t\t-f \"$TEST_PATH/apache.conf\" $HTTPD_PARA -k stop\n+\t\t-f \"$TEST_PATH/$APACHE_CONF_FILE\" $HTTPD_PARA -k stop\n }\n \n test_http_push_nonff () {\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328221","messageId":"20170916080731.13925-13-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 12/40] external odb: add 'put_raw_obj' support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:03Z","receivedAt":"2017-09-16T08:10:10Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Add support for a 'put_raw_obj' capability/instruction to send new\nobjects to an external odb. Objects will be sent as they are (in\ntheir 'raw' format). They will not be converted to Git objects.\n\nFor now any new Git object (blob, tree, commit, ...) would be sent\nif 'put_raw_obj' is supported by an odb helper. This is not a great\ndefault, but let's leave it to following commits to tweak that.\n\nSigned-off-by: Christian Couder \n---\n external-odb.c | 15 +++++++++++++++\n external-odb.h | 2 ++\n odb-helper.c | 43 ++++++++++++++++++++++++++++++++++++++-----\n odb-helper.h | 3 +++\n sha1_file.c | 2 ++\n 5 files changed, 60 insertions(+), 5 deletions(-)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex 0f0de170b8..82fac702e8 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -118,3 +118,18 @@ int external_odb_get_object(const unsigned char *sha1)\n \n \treturn -1;\n }\n+\n+int external_odb_put_object(const void *buf, size_t len,\n+\t\t\t const char *type, unsigned char *sha1)\n+{\n+\tstruct odb_helper *o;\n+\n+\texternal_odb_init();\n+\n+\tfor (o = helpers; o; o = o->next) {\n+\t\tint r = odb_helper_put_object(o, buf, len, type, sha1);\n+\t\tif (r <= 0)\n+\t\t\treturn r;\n+\t}\n+\treturn 1;\n+}\ndiff --git a/external-odb.h b/external-odb.h\nindex dc5635f452..d369dfdf6f 100644\n--- a/external-odb.h\n+++ b/external-odb.h\n@@ -4,5 +4,7 @@\n extern const char *external_odb_root(void);\n extern int external_odb_has_object(const unsigned char *sha1);\n extern int external_odb_get_object(const unsigned char *sha1);\n+extern int external_odb_put_object(const void *buf, size_t len,\n+\t\t\t\t const char *type, unsigned char *sha1);\n \n #endif /* EXTERNAL_ODB_H */\ndiff --git a/odb-helper.c b/odb-helper.c\nindex 9375eca58f..39d20fdfd7 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -71,9 +71,10 @@ static void prepare_helper_command(struct argv_array *argv, const char *cmd,\n \tstrbuf_release(&buf);\n }\n \n-__attribute__((format (printf,3,4)))\n+__attribute__((format (printf,4,5)))\n static int odb_helper_start(struct odb_helper *o,\n \t\t\t struct odb_helper_cmd *cmd,\n+\t\t\t int use_stdin,\n \t\t\t const char *fmt, ...)\n {\n \tva_list ap;\n@@ -90,7 +91,10 @@ static int odb_helper_start(struct odb_helper *o,\n \n \tcmd->child.argv = cmd->argv.argv;\n \tcmd->child.use_shell = 1;\n-\tcmd->child.no_stdin = 1;\n+\tif (use_stdin)\n+\t\tcmd->child.in = -1;\n+\telse\n+\t\tcmd->child.no_stdin = 1;\n \tcmd->child.out = -1;\n \n \tif (start_command(&cmd->child) < 0) {\n@@ -119,7 +123,7 @@ int odb_helper_init(struct odb_helper *o)\n \tFILE *fh;\n \tstruct strbuf line = STRBUF_INIT;\n \n-\tif (odb_helper_start(o, &cmd, \"init\") < 0)\n+\tif (odb_helper_start(o, &cmd, 0, \"init\") < 0)\n \t\treturn -1;\n \n \tfh = xfdopen(cmd.child.out, \"r\");\n@@ -179,7 +183,7 @@ static void odb_helper_load_have(struct odb_helper *o)\n \t\treturn;\n \to->have_valid = 1;\n \n-\tif (odb_helper_start(o, &cmd, \"have\") < 0)\n+\tif (odb_helper_start(o, &cmd, 0, \"have\") < 0)\n \t\treturn;\n \n \tfh = xfdopen(cmd.child.out, \"r\");\n@@ -234,7 +238,7 @@ int odb_helper_get_object(struct odb_helper *o, const unsigned char *sha1,\n \tif (!obj)\n \t\treturn -1;\n \n-\tif (odb_helper_start(o, &cmd, \"get_git_obj %s\", sha1_to_hex(sha1)) < 0)\n+\tif (odb_helper_start(o, &cmd, 0, \"get_git_obj %s\", sha1_to_hex(sha1)) < 0)\n \t\treturn -1;\n \n \tmemset(&stream, 0, sizeof(stream));\n@@ -321,3 +325,32 @@ int odb_helper_get_object(struct odb_helper *o, const unsigned char *sha1,\n \n \treturn 0;\n }\n+\n+int odb_helper_put_object(struct odb_helper *o,\n+\t\t\t const void *buf, size_t len,\n+\t\t\t const char *type, unsigned char *sha1)\n+{\n+\tstruct odb_helper_cmd cmd;\n+\n+\tif (odb_helper_start(o, &cmd, 1, \"put_raw_obj %s %\"PRIuMAX\" %s\",\n+\t\t\t sha1_to_hex(sha1), (uintmax_t)len, type) < 0)\n+\t\treturn -1;\n+\n+\tdo {\n+\t\tint w = xwrite(cmd.child.in, buf, len);\n+\t\tif (w < 0) {\n+\t\t\terror(\"unable to write to odb helper '%s': %s\",\n+\t\t\t o->name, strerror(errno));\n+\t\t\tclose(cmd.child.in);\n+\t\t\tclose(cmd.child.out);\n+\t\t\todb_helper_finish(o, &cmd);\n+\t\t\treturn -1;\n+\t\t}\n+\t\tlen -= w;\n+\t} while (len > 0);\n+\n+\tclose(cmd.child.in);\n+\tclose(cmd.child.out);\n+\todb_helper_finish(o, &cmd);\n+\treturn 0;\n+}\ndiff --git a/odb-helper.h b/odb-helper.h\nindex 5f28a6e512..0571ba09cb 100644\n--- a/odb-helper.h\n+++ b/odb-helper.h\n@@ -35,5 +35,8 @@ extern int odb_helper_has_object(struct odb_helper *o,\n extern int odb_helper_get_object(struct odb_helper *o,\n \t\t\t\t const unsigned char *sha1,\n \t\t\t\t int fd);\n+extern int odb_helper_put_object(struct odb_helper *o,\n+\t\t\t\t const void *buf, size_t len,\n+\t\t\t\t const char *type, unsigned char *sha1);\n \n #endif /* ODB_HELPER_H */\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 4a4f5df5ec..d0155e392f 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1613,6 +1613,8 @@ int write_sha1_file(const void *buf, unsigned long len, const char *type, unsign\n \t * it out into .git/objects/??/?{38} file.\n \t */\n \twrite_sha1_file_prepare(buf, len, type, sha1, hdr, &hdrlen);\n+\tif (!external_odb_put_object(buf, len, type, sha1))\n+\t\treturn 0;\n \tif (freshen_packed_object(sha1) || freshen_loose_object(sha1))\n \t\treturn 0;\n \treturn write_loose_object(sha1, hdr, hdrlen, buf, len, 0);\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328222","messageId":"20170916080731.13925-7-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 06/40] t0021/rot13-filter: add capability functions","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:06:57Z","receivedAt":"2017-09-16T08:10:14Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Add functions to help read and write capabilities.\nThese functions will be reused in following patches.\n\nSigned-off-by: Christian Couder \n---\n t/t0021/rot13-filter.pl | 40 ++++++++++++++++++++++++++++++++--------\n 1 file changed, 32 insertions(+), 8 deletions(-)\n\ndiff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\nindex 278fc6f534..ba18b207c6 100644\n--- a/t/t0021/rot13-filter.pl\n+++ b/t/t0021/rot13-filter.pl\n@@ -116,20 +116,44 @@ sub packet_initialize {\n \tpacket_flush();\n }\n \n+sub packet_read_capabilities {\n+\tmy @cap;\n+\twhile (1) {\n+\t\tmy ( $res, $buf ) = packet_bin_read();\n+\t\treturn ( $res, @cap ) if ( $res != 0 );\n+\t\tunless ( $buf =~ s/\\n$// ) {\n+\t\t\tdie \"A non-binary line MUST be terminated by an LF.\\n\"\n+\t\t\t . \"Received: '$buf'\";\n+\t\t}\n+\t\tdie \"bad capability buf: '$buf'\" unless ( $buf =~ s/capability=// );\n+\t\tpush @cap, $buf;\n+\t}\n+}\n+\n+sub packet_read_and_check_capabilities {\n+\tmy @local_caps = @_;\n+\tmy @remote_res_caps = packet_read_capabilities();\n+\tmy $res = shift @remote_res_caps;\n+\tmy %remote_caps = map { $_ => 1 } @remote_res_caps;\n+\tforeach (@local_caps) {\n+ \tdie \"'$_' capability not available\" unless (exists($remote_caps{$_}));\n+\t}\n+\treturn $res;\n+}\n+\n+sub packet_write_capabilities {\n+\tpacket_txt_write( \"capability=\" . $_ ) foreach (@_);\n+\tpacket_flush();\n+}\n+\n print $debug \"START\\n\";\n $debug->flush();\n \n packet_initialize(\"git-filter\", 2);\n \n-( packet_txt_read() eq ( 0, \"capability=clean\" ) ) || die \"bad capability\";\n-( packet_txt_read() eq ( 0, \"capability=smudge\" ) ) || die \"bad capability\";\n-( packet_txt_read() eq ( 0, \"capability=delay\" ) ) || die \"bad capability\";\n-( packet_bin_read() eq ( 1, \"\" ) ) || die \"bad capability end\";\n+packet_read_and_check_capabilities(\"clean\", \"smudge\", \"delay\");\n+packet_write_capabilities(@capabilities);\n \n-foreach (@capabilities) {\n-\tpacket_txt_write( \"capability=\" . $_ );\n-}\n-packet_flush();\n print $debug \"init handshake complete\\n\";\n $debug->flush();\n \n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328223","messageId":"20170916080731.13925-10-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 09/40] Add initial external odb support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:07:00Z","receivedAt":"2017-09-16T08:10:17Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"The external-odb.{c,h} files contains the functions that are\ncalled by the rest of Git from \"sha1_file.c\".\n\nThe odb-helper.{c,h} files contains the functions to\nactually implement communication with the external scripts or\nprocesses that will manage external git objects.\n\nFor now only script mode is supported, and only the 'have' and\n'get_git_obj' instructions are supported.\n\nHelped-by: Jeff King \nSigned-off-by: Christian Couder \n---\n Makefile | 2 +\n cache.h | 1 +\n external-odb.c | 113 ++++++++++++++++++++\n external-odb.h | 8 ++\n odb-helper.c | 269 ++++++++++++++++++++++++++++++++++++++++++++++++\n odb-helper.h | 27 +++++\n sha1_file.c | 31 +++++-\n t/t0400-external-odb.sh | 46 +++++++++\n 8 files changed, 495 insertions(+), 2 deletions(-)\n create mode 100644 external-odb.c\n create mode 100644 external-odb.h\n create mode 100644 odb-helper.c\n create mode 100644 odb-helper.h\n create mode 100755 t/t0400-external-odb.sh\n\ndiff --git a/Makefile b/Makefile\nindex f2bb7f2f63..24aab8ace3 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -784,6 +784,7 @@ LIB_OBJS += ewah/ewah_bitmap.o\n LIB_OBJS += ewah/ewah_io.o\n LIB_OBJS += ewah/ewah_rlw.o\n LIB_OBJS += exec_cmd.o\n+LIB_OBJS += external-odb.o\n LIB_OBJS += fetch-pack.o\n LIB_OBJS += fsck.o\n LIB_OBJS += gettext.o\n@@ -816,6 +817,7 @@ LIB_OBJS += notes-cache.o\n LIB_OBJS += notes-merge.o\n LIB_OBJS += notes-utils.o\n LIB_OBJS += object.o\n+LIB_OBJS += odb-helper.o\n LIB_OBJS += oidset.o\n LIB_OBJS += packfile.o\n LIB_OBJS += pack-bitmap.o\ndiff --git a/cache.h b/cache.h\nindex 00d89568f3..6c22bd0525 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1534,6 +1534,7 @@ extern void prepare_alt_odb(void);\n extern char *compute_alternate_path(const char *path, struct strbuf *err);\n typedef int alt_odb_fn(struct alternate_object_database *, void *);\n extern int foreach_alt_odb(alt_odb_fn, void*);\n+extern void prepare_external_alt_odb(void);\n \n /*\n * Allocate a \"struct alternate_object_database\" but do _not_ actually\ndiff --git a/external-odb.c b/external-odb.c\nnew file mode 100644\nindex 0000000000..e9c3f11666\n--- /dev/null\n+++ b/external-odb.c\n@@ -0,0 +1,113 @@\n+#include \"cache.h\"\n+#include \"external-odb.h\"\n+#include \"odb-helper.h\"\n+\n+static struct odb_helper *helpers;\n+static struct odb_helper **helpers_tail = &helpers;\n+\n+static struct odb_helper *find_or_create_helper(const char *name, int len)\n+{\n+\tstruct odb_helper *o;\n+\n+\tfor (o = helpers; o; o = o->next)\n+\t\tif (!strncmp(o->name, name, len) && !o->name[len])\n+\t\t\treturn o;\n+\n+\to = odb_helper_new(name, len);\n+\t*helpers_tail = o;\n+\thelpers_tail = &o->next;\n+\n+\treturn o;\n+}\n+\n+static int external_odb_config(const char *var, const char *value, void *data)\n+{\n+\tstruct odb_helper *o;\n+\tconst char *name;\n+\tint namelen;\n+\tconst char *subkey;\n+\n+\tif (parse_config_key(var, \"odb\", &name, &namelen, &subkey) < 0)\n+\t\treturn 0;\n+\n+\to = find_or_create_helper(name, namelen);\n+\n+\tif (!strcmp(subkey, \"scriptcommand\"))\n+\t\treturn git_config_string(&o->cmd, var, value);\n+\n+\treturn 0;\n+}\n+\n+static void external_odb_init(void)\n+{\n+\tstatic int initialized;\n+\n+\tif (initialized)\n+\t\treturn;\n+\tinitialized = 1;\n+\n+\tgit_config(external_odb_config, NULL);\n+}\n+\n+const char *external_odb_root(void)\n+{\n+\tstatic const char *root;\n+\tif (!root)\n+\t\troot = git_pathdup(\"objects/external\");\n+\treturn root;\n+}\n+\n+int external_odb_has_object(const unsigned char *sha1)\n+{\n+\tstruct odb_helper *o;\n+\n+\texternal_odb_init();\n+\n+\tfor (o = helpers; o; o = o->next)\n+\t\tif (odb_helper_has_object(o, sha1))\n+\t\t\treturn 1;\n+\treturn 0;\n+}\n+\n+int external_odb_get_object(const unsigned char *sha1)\n+{\n+\tstruct odb_helper *o;\n+\tconst char *path;\n+\n+\tif (!external_odb_has_object(sha1))\n+\t\treturn -1;\n+\n+\tpath = sha1_file_name_alt(external_odb_root(), sha1);\n+\tsafe_create_leading_directories_const(path);\n+\tprepare_external_alt_odb();\n+\n+\tfor (o = helpers; o; o = o->next) {\n+\t\tstruct strbuf tmpfile = STRBUF_INIT;\n+\t\tint ret;\n+\t\tint fd;\n+\n+\t\tif (!odb_helper_has_object(o, sha1))\n+\t\t\tcontinue;\n+\n+\t\tfd = create_object_tmpfile(&tmpfile, path);\n+\t\tif (fd < 0) {\n+\t\t\tstrbuf_release(&tmpfile);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tif (odb_helper_get_object(o, sha1, fd) < 0) {\n+\t\t\tclose(fd);\n+\t\t\tunlink(tmpfile.buf);\n+\t\t\tstrbuf_release(&tmpfile);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tclose_sha1_file(fd);\n+\t\tret = finalize_object_file(tmpfile.buf, path);\n+\t\tstrbuf_release(&tmpfile);\n+\t\tif (!ret)\n+\t\t\treturn 0;\n+\t}\n+\n+\treturn -1;\n+}\ndiff --git a/external-odb.h b/external-odb.h\nnew file mode 100644\nindex 0000000000..dc5635f452\n--- /dev/null\n+++ b/external-odb.h\n@@ -0,0 +1,8 @@\n+#ifndef EXTERNAL_ODB_H\n+#define EXTERNAL_ODB_H\n+\n+extern const char *external_odb_root(void);\n+extern int external_odb_has_object(const unsigned char *sha1);\n+extern int external_odb_get_object(const unsigned char *sha1);\n+\n+#endif /* EXTERNAL_ODB_H */\ndiff --git a/odb-helper.c b/odb-helper.c\nnew file mode 100644\nindex 0000000000..5e91044872\n--- /dev/null\n+++ b/odb-helper.c\n@@ -0,0 +1,269 @@\n+#include \"cache.h\"\n+#include \"object.h\"\n+#include \"argv-array.h\"\n+#include \"odb-helper.h\"\n+#include \"run-command.h\"\n+#include \"sha1-lookup.h\"\n+\n+struct odb_helper *odb_helper_new(const char *name, int namelen)\n+{\n+\tstruct odb_helper *o;\n+\n+\to = xcalloc(1, sizeof(*o));\n+\to->name = xmemdupz(name, namelen);\n+\n+\treturn o;\n+}\n+\n+struct odb_helper_cmd {\n+\tstruct argv_array argv;\n+\tstruct child_process child;\n+};\n+\n+/*\n+ * Callers are responsible to ensure that the result of vaddf(fmt, ap)\n+ * is properly shell-quoted.\n+ */\n+static void prepare_helper_command(struct argv_array *argv, const char *cmd,\n+\t\t\t\t const char *fmt, va_list ap)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tstrbuf_addstr(&buf, cmd);\n+\tstrbuf_addch(&buf, ' ');\n+\tstrbuf_vaddf(&buf, fmt, ap);\n+\n+\targv_array_push(argv, buf.buf);\n+\tstrbuf_release(&buf);\n+}\n+\n+__attribute__((format (printf,3,4)))\n+static int odb_helper_start(struct odb_helper *o,\n+\t\t\t struct odb_helper_cmd *cmd,\n+\t\t\t const char *fmt, ...)\n+{\n+\tva_list ap;\n+\n+\tmemset(cmd, 0, sizeof(*cmd));\n+\targv_array_init(&cmd->argv);\n+\n+\tif (!o->cmd)\n+\t\treturn -1;\n+\n+\tva_start(ap, fmt);\n+\tprepare_helper_command(&cmd->argv, o->cmd, fmt, ap);\n+\tva_end(ap);\n+\n+\tcmd->child.argv = cmd->argv.argv;\n+\tcmd->child.use_shell = 1;\n+\tcmd->child.no_stdin = 1;\n+\tcmd->child.out = -1;\n+\n+\tif (start_command(&cmd->child) < 0) {\n+\t\targv_array_clear(&cmd->argv);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int odb_helper_finish(struct odb_helper *o,\n+\t\t\t struct odb_helper_cmd *cmd)\n+{\n+\tint ret = finish_command(&cmd->child);\n+\targv_array_clear(&cmd->argv);\n+\tif (ret) {\n+\t\twarning(\"odb helper '%s' reported failure\", o->name);\n+\t\treturn -1;\n+\t}\n+\treturn 0;\n+}\n+\n+static int parse_object_line(struct odb_helper_object *o, const char *line)\n+{\n+\tchar *end;\n+\tif (get_sha1_hex(line, o->sha1) < 0)\n+\t\treturn -1;\n+\n+\tline += 40;\n+\tif (*line++ != ' ')\n+\t\treturn -1;\n+\n+\to->size = strtoul(line, &end, 10);\n+\tif (line == end || *end++ != ' ')\n+\t\treturn -1;\n+\n+\to->type = type_from_string(end);\n+\treturn 0;\n+}\n+\n+static int add_have_entry(struct odb_helper *o, const char *line)\n+{\n+\tALLOC_GROW(o->have, o->have_nr+1, o->have_alloc);\n+\tif (parse_object_line(&o->have[o->have_nr], line) < 0) {\n+\t\twarning(\"bad 'have' input from odb helper '%s': %s\",\n+\t\t\to->name, line);\n+\t\treturn 1;\n+\t}\n+\to->have_nr++;\n+\treturn 0;\n+}\n+\n+static int odb_helper_object_cmp(const void *va, const void *vb)\n+{\n+\tconst struct odb_helper_object *a = va, *b = vb;\n+\treturn hashcmp(a->sha1, b->sha1);\n+}\n+\n+static void odb_helper_load_have(struct odb_helper *o)\n+{\n+\tstruct odb_helper_cmd cmd;\n+\tFILE *fh;\n+\tstruct strbuf line = STRBUF_INIT;\n+\n+\tif (o->have_valid)\n+\t\treturn;\n+\to->have_valid = 1;\n+\n+\tif (odb_helper_start(o, &cmd, \"have\") < 0)\n+\t\treturn;\n+\n+\tfh = xfdopen(cmd.child.out, \"r\");\n+\twhile (strbuf_getline(&line, fh) != EOF)\n+\t\tif (add_have_entry(o, line.buf))\n+\t\t\tbreak;\n+\n+\tstrbuf_release(&line);\n+\tfclose(fh);\n+\todb_helper_finish(o, &cmd);\n+\n+\tqsort(o->have, o->have_nr, sizeof(*o->have), odb_helper_object_cmp);\n+}\n+\n+static const unsigned char *have_sha1_access(size_t index, void *table)\n+{\n+\tstruct odb_helper_object *have = table;\n+\treturn have[index].sha1;\n+}\n+\n+static struct odb_helper_object *odb_helper_lookup(struct odb_helper *o,\n+\t\t\t\t\t\t const unsigned char *sha1)\n+{\n+\tint idx;\n+\n+\todb_helper_load_have(o);\n+\tidx = sha1_pos(sha1, o->have, o->have_nr, have_sha1_access);\n+\tif (idx < 0)\n+\t\treturn NULL;\n+\treturn &o->have[idx];\n+}\n+\n+int odb_helper_has_object(struct odb_helper *o, const unsigned char *sha1)\n+{\n+\treturn !!odb_helper_lookup(o, sha1);\n+}\n+\n+int odb_helper_get_object(struct odb_helper *o, const unsigned char *sha1,\n+\t\t\t int fd)\n+{\n+\tstruct odb_helper_object *obj;\n+\tstruct odb_helper_cmd cmd;\n+\tunsigned long total_got;\n+\tgit_zstream stream;\n+\tint zret = Z_STREAM_END;\n+\tgit_SHA_CTX hash;\n+\tunsigned char real_sha1[20];\n+\tstruct strbuf header = STRBUF_INIT;\n+\tunsigned long hdr_size;\n+\n+\tobj = odb_helper_lookup(o, sha1);\n+\tif (!obj)\n+\t\treturn -1;\n+\n+\tif (odb_helper_start(o, &cmd, \"get_git_obj %s\", sha1_to_hex(sha1)) < 0)\n+\t\treturn -1;\n+\n+\tmemset(&stream, 0, sizeof(stream));\n+\tgit_inflate_init(&stream);\n+\tgit_SHA1_Init(&hash);\n+\ttotal_got = 0;\n+\n+\tfor (;;) {\n+\t\tunsigned char buf[4096];\n+\t\tint r;\n+\n+\t\tr = xread(cmd.child.out, buf, sizeof(buf));\n+\t\tif (r < 0) {\n+\t\t\terror(\"unable to read from odb helper '%s': %s\",\n+\t\t\t o->name, strerror(errno));\n+\t\t\tclose(cmd.child.out);\n+\t\t\todb_helper_finish(o, &cmd);\n+\t\t\tgit_inflate_end(&stream);\n+\t\t\treturn -1;\n+\t\t}\n+\t\tif (r == 0)\n+\t\t\tbreak;\n+\n+\t\twrite_or_die(fd, buf, r);\n+\n+\t\tstream.next_in = buf;\n+\t\tstream.avail_in = r;\n+\t\tdo {\n+\t\t\tunsigned char inflated[4096];\n+\t\t\tunsigned long got;\n+\n+\t\t\tstream.next_out = inflated;\n+\t\t\tstream.avail_out = sizeof(inflated);\n+\t\t\tzret = git_inflate(&stream, Z_SYNC_FLUSH);\n+\t\t\tgot = sizeof(inflated) - stream.avail_out;\n+\n+\t\t\tgit_SHA1_Update(&hash, inflated, got);\n+\t\t\t/* skip header when counting size */\n+\t\t\tif (!total_got) {\n+\t\t\t\tconst unsigned char *p = memchr(inflated, '\\0', got);\n+\t\t\t\tif (p) {\n+\t\t\t\t\tunsigned long hdr_last = p - inflated + 1;\n+\t\t\t\t\tstrbuf_add(&header, inflated, hdr_last);\n+\t\t\t\t\tgot -= hdr_last;\n+\t\t\t\t} else {\n+\t\t\t\t\tstrbuf_add(&header, inflated, got);\n+\t\t\t\t\tgot = 0;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t\ttotal_got += got;\n+\t\t} while (stream.avail_in && zret == Z_OK);\n+\t}\n+\n+\tclose(cmd.child.out);\n+\tgit_inflate_end(&stream);\n+\tgit_SHA1_Final(real_sha1, &hash);\n+\tif (odb_helper_finish(o, &cmd))\n+\t\treturn -1;\n+\tif (zret != Z_STREAM_END) {\n+\t\twarning(\"bad zlib data from odb helper '%s' for %s\",\n+\t\t\to->name, sha1_to_hex(sha1));\n+\t\treturn -1;\n+\t}\n+\tif (total_got != obj->size) {\n+\t\twarning(\"size mismatch from odb helper '%s' for %s (%lu != %lu)\",\n+\t\t\to->name, sha1_to_hex(sha1), total_got, obj->size);\n+\t\treturn -1;\n+\t}\n+\tif (hashcmp(real_sha1, sha1)) {\n+\t\twarning(\"sha1 mismatch from odb helper '%s' for %s (got %s)\",\n+\t\t\to->name, sha1_to_hex(sha1), sha1_to_hex(real_sha1));\n+\t\treturn -1;\n+\t}\n+\tif (parse_sha1_header(header.buf, &hdr_size) < 0) {\n+\t\twarning(\"could not parse header from odb helper '%s' for %s\",\n+\t\t\to->name, sha1_to_hex(sha1));\n+\t\treturn -1;\n+\t}\n+\tif (total_got != hdr_size) {\n+\t\twarning(\"size mismatch from odb helper '%s' for %s (%lu != %lu)\",\n+\t\t\to->name, sha1_to_hex(sha1), total_got, hdr_size);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/odb-helper.h b/odb-helper.h\nnew file mode 100644\nindex 0000000000..fb25ad579e\n--- /dev/null\n+++ b/odb-helper.h\n@@ -0,0 +1,27 @@\n+#ifndef ODB_HELPER_H\n+#define ODB_HELPER_H\n+\n+struct odb_helper {\n+\tconst char *name;\n+\tconst char *cmd;\n+\n+\tstruct odb_helper_object {\n+\t\tunsigned char sha1[20];\n+\t\tunsigned long size;\n+\t\tenum object_type type;\n+\t} *have;\n+\tint have_nr;\n+\tint have_alloc;\n+\tint have_valid;\n+\n+\tstruct odb_helper *next;\n+};\n+\n+extern struct odb_helper *odb_helper_new(const char *name, int namelen);\n+extern int odb_helper_has_object(struct odb_helper *o,\n+\t\t\t\t const unsigned char *sha1);\n+extern int odb_helper_get_object(struct odb_helper *o,\n+\t\t\t\t const unsigned char *sha1,\n+\t\t\t\t int fd);\n+\n+#endif /* ODB_HELPER_H */\ndiff --git a/sha1_file.c b/sha1_file.c\nindex bea1ae6afb..4a4f5df5ec 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -29,6 +29,7 @@\n #include \"mergesort.h\"\n #include \"quote.h\"\n #include \"packfile.h\"\n+#include \"external-odb.h\"\n \n const unsigned char null_sha1[GIT_MAX_RAWSZ];\n const struct object_id null_oid;\n@@ -613,6 +614,21 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)\n \treturn r;\n }\n \n+void prepare_external_alt_odb(void)\n+{\n+\tstatic int linked_external;\n+\tconst char *path;\n+\n+\tif (linked_external)\n+\t\treturn;\n+\n+\tpath = external_odb_root();\n+\tif (!access(path, F_OK)) {\n+\t\tlink_alt_odb_entry(path, NULL, 0, \"\");\n+\t\tlinked_external = 1;\n+\t}\n+}\n+\n void prepare_alt_odb(void)\n {\n \tconst char *alt;\n@@ -627,6 +643,7 @@ void prepare_alt_odb(void)\n \tlink_alt_odb_entries(alt, strlen(alt), PATH_SEP, NULL, 0);\n \n \tread_info_alternates(get_object_directory(), 0);\n+\tprepare_external_alt_odb();\n }\n \n /* Returns 1 if we have successfully freshened the file, 0 otherwise. */\n@@ -667,7 +684,7 @@ static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)\n \t\tif (check_and_freshen_file(path, freshen))\n \t\t\treturn 1;\n \t}\n-\treturn 0;\n+\treturn external_odb_has_object(sha1);\n }\n \n static int check_and_freshen(const unsigned char *sha1, int freshen)\n@@ -824,6 +841,9 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,\n \t\t\treturn 0;\n \t}\n \n+\tif (!external_odb_get_object(sha1) && !lstat(*path, st))\n+\t\treturn 0;\n+\n \treturn -1;\n }\n \n@@ -859,7 +879,14 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)\n \tif (fd >= 0)\n \t\treturn fd;\n \n-\treturn open_sha1_file_alt(sha1, path);\n+\tfd = open_sha1_file_alt(sha1, path);\n+\tif (fd >= 0)\n+\t\treturn fd;\n+\n+\tif (!external_odb_get_object(sha1))\n+\t\tfd = open_sha1_file_alt(sha1, path);\n+\n+\treturn fd;\n }\n \n /*\ndiff --git a/t/t0400-external-odb.sh b/t/t0400-external-odb.sh\nnew file mode 100755\nindex 0000000000..2f4749fab1\n--- /dev/null\n+++ b/t/t0400-external-odb.sh\n@@ -0,0 +1,46 @@\n+#!/bin/sh\n+\n+test_description='basic tests for external object databases'\n+\n+. ./test-lib.sh\n+\n+ALT_SOURCE=\"$PWD/alt-repo/.git\"\n+export ALT_SOURCE\n+write_script odb-helper <<\\EOF\n+GIT_DIR=$ALT_SOURCE; export GIT_DIR\n+case \"$1\" in\n+have)\n+\tgit cat-file --batch-check --batch-all-objects |\n+\tawk '{print $1 \" \" $3 \" \" $2}'\n+\t;;\n+get_git_obj)\n+\tcat \"$GIT_DIR\"/objects/$(echo $2 | sed 's#..#&/#')\n+\t;;\n+esac\n+EOF\n+HELPER=\"\\\"$PWD\\\"/odb-helper\"\n+\n+test_expect_success 'setup alternate repo' '\n+\tgit init alt-repo &&\n+\t(cd alt-repo &&\n+\t test_commit one &&\n+\t test_commit two\n+\t) &&\n+\talt_head=`cd alt-repo && git rev-parse HEAD`\n+'\n+\n+test_expect_success 'alt objects are missing' '\n+\ttest_must_fail git log --format=%s $alt_head\n+'\n+\n+test_expect_success 'helper can retrieve alt objects' '\n+\ttest_config odb.magic.scriptCommand \"$HELPER\" &&\n+\tcat >expect <<-\\EOF &&\n+\ttwo\n+\tone\n+\tEOF\n+\tgit log --format=%s $alt_head >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328224","messageId":"20170916080731.13925-1-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":null,"subject":"[PATCH v6 00/40] Add initial experimental external ODB support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:06:51Z","receivedAt":"2017-09-16T08:10:22Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Note: a lot of information about the goals, the design and how things\nwork is now in the followin patches:\n\n - [PATCH v6 34/40] Add Documentation/technical/external-odb.txt\n - [PATCH v6 40/40] Doc/external-odb: explain transfering objects and metadata\n\nGoal\n~~~~\n\nGit can store its objects only in the form of loose objects in\nseparate files or packed objects in a pack file.\n\nTo be able to better handle some kind of objects, for example big\nblobs, it would be nice if Git could store its objects in other object\ndatabases (ODB).\n\nTo do that, this patch series makes it possible to register commands,\nalso called \"helpers\", using \"odb..scriptCommand\" or\n\"odb..subprocessCommand\" config variables, to access external\nODBs where objects can be stored and retrieved.\n\nDesign\n~~~~~~\n\n* The \"helpers\" (registered commands)\n\nEach helper manages access to one external ODB.\n\nThere are 2 different modes for helper:\n\n - Helpers configured using \"odb..scriptCommand\" are\n launched each time Git wants to communicate with the \n external ODB. This is called \"script mode\".\n\n - Helpers configured using \"odb..subprocessCommand\" are\n launched launched once as a sub-process (using sub-process.h), and\n Git communicates with them using packet lines. This is called\n \"process mode\".\n\nA helper can be given different instructions by Git. The instructions\nthat are supported are negociated at the beginning of the\ncommunication using a capability mechanism.\n\nSee patch 34/40 and 40/40 (the documentation patchs) for more\ninformation about the different instructions and their arguments.\n\n* Performance\n\nThe process mode has been implemented using the refactoring that Ben\nPeart did on top of Lars Schneider's work on using sub-processes and\npacket lines in the smudge/clean filters for git-lfs.\n\nThis also uses further work from Ben Peart called \"read object\nprocess\".\n\nSee:\n\nhttps://public-inbox.org/git/20170113155253.1644-1-benpeart@microsoft.com/\nhttps://public-inbox.org/git/20170322165220.5660-1-benpeart@microsoft.com/\n\nBen recently sent an update of this work but this update has not been\nintegrated into the current patch series. See:\n\nhttps://public-inbox.org/git/20170714132651.170708-1-benpeart@microsoft.com/\n\nAnyway thanks to this, the external ODB mechanism should in the end perform\nas well as the git-lfs mechanism when many objects should be\ntransfered.\n\nImplementation\n~~~~~~~~~~~~~~\n\n* Mechanism to call the registered commands\n\nA set of function in external-odb.{c,h} are called by the rest of Git\nto manage all the external ODBs.\n\nThese functions use 'struct odb_helper' and its associated functions\ndefined in odb-helper.{c,h} to talk to the different external ODBs by\nlaunching the configured \"odb..*command\" commands and writing\nto or reading from them.\n\n* Transfering information\n\nTo tranfer information about the blobs stored in external ODB, some\nspecial refs, called \"odb ref\", similar as replace refs, are used in\nthe tests of this series, but in general nothing forces the helper to\nuse that mechanism.\n\nThe external odb helper is responsible for using and creating the refs\nin refs/odbs//, if it wants to do that. It is free for example\nto just create one ref, as it is also free to create many refs. Git\nwould just transmit the refs that have been created by this helper, if\nGit is asked to do so.\n\nFor now in the tests there is one odb ref per blob, as it is simple\nand as it is similar to what git-lfs does. Each ref name is\nrefs/odbs// where is the sha1 of the blob stored\nin the external odb named .\n\nThese odb refs point to a blob that is stored in the Git\nrepository and contain information about the blob stored in the\nexternal odb. This information can be specific to the external odb.\nThe repos can then share this information using commands like:\n\n`git fetch origin \"refs/odbs//*:refs/odbs//*\"`\n\nAt the end of the current patch series, \"git clone\" is teached a\n\"--initial-refspec\" option, that asks it to first fetch some specified\nrefs. This is used in the tests to fetch the odb refs first.\n\nThis way only one \"git clone\" command can setup a repo using the\nexternal ODB mechanism as long as the right helper is installed on the\nmachine and as long as the following options are used:\n\n - \"--initial-refspec \" to fetch the odb refspec\n - \"-c odb..command=\" to configure the helper\n\nThere is also a test script (t0430) that shows that the\n\"--initial-refspec\" option along with the external ODB mechanism can\nbe used to implement cloning using bundles.\n\n* ODB refs\n\nFor now odb ref management is only implemented in a helper in t0410.\n\nWhen a new blob is added to an external odb, its sha1, size and type\nare writen in another new blob and the odb ref is created.\n\nWhen the list of existing blobs is requested from the external odb,\nthe content of the blobs pointed to by the odb refs can also be used\nby the odb to claim that it can get the objects.\n\nWhen a blob is actually requested from the external odb, it can use\nthe content stored in the blobs pointed to by the odb refs to get the\nactual blobs and then pass them.\n\nHighlevel view of the patches in the series\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n - Patch 1/40 is a small code cleanup that I already sent to the\n mailing list but may be removed in the end due to ongoing work\n on \"git clone\".\n\n - Patches 02/40 to 07/40 create a \"Git/Packet.pm\" module by\n refactoring \"t0021/rot13-filter.pl\". Functions from this new\n module will be used later in test scripts. According to Junio's\n suggestion compared to v5 we now first fully refactor\n \"t0021/rot13-filter.pl\" before creating the \"Git/Packet.pm\"\n module.\n\n - Patches 08/40 to 16/40 create the external ODB insfrastructure\n in external-odb.{c,h} and odb-helper.{c,h} for the script mode.\n The main changes compared to v5 are the following:\n - we mark as \"extern\" functions in *.h files\n\t- we use sha1_pos() instead of sha1_entry_pos()\n\t- we check the size in the header when we 'get' a Git object\n\n - Patches 17/40 to 23/40 improve lib-http to make it possible to\n use it as an external ODB to test storing blobs in an HTTP\n server. The \"upload.sh\" and \"list.sh\" files are now properly\n indented and they use %% instead of % in parameter\n substitutions compared to v5.\n\n - Patches 24/40 to 32/40 improve the external ODB insfrastructure\n to support sub-processes and make everything work using\n them. The main changes compared to v5 are the following:\n - we mark as \"extern\" functions in *.h files\n\t- we use the new subprocess_handshake() function\n\t- we check the size in the header when we 'get' a Git object\n\n - Patch 33/40 uses attributes to mark blobs that should be handled\n by an external odb.\n\n - Patch 34/40 adds documentation about the external odb\n mechanism. This patch has been much improved since v5.\n\n - Patches 35/40 to 39/40 add the --initial-refspec to git clone\n along with tests.\n\n - Patch 40/40 adds documentation about transfering objects and\n metadata when using the external odb mechanism. This patch is\n new since v5.\n\nFuture work\n~~~~~~~~~~~\n\nThere are still things that could be cleaned or improved. I think I\nmay work on:\n\n - Integrate changes in recent \"read-object-process\" work by Ben Peart.\n\n - Better test all the combinations of the different modes with and\n without \"have\" and \"put_*\" instructions.\n\n - Maybe implement the missing kinds of 'put' ('put_git_obj' and\n 'put_direct'), so that Git could pass either a git object a plain\n object or ask the helper to retreive it directly from Git's object\n database.\n\n - Add more long running tests and improve tests in general.\n\nPrevious work and discussions\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n(Sorry for the old Gmane links, I hope I will try to replace them with\npublic-inbox.org at one point.)\n\nPeff started to work on this and discuss this some years ago:\n\nhttp://thread.gmane.org/gmane.comp.version-control.git/206886/focus=207040\nhttp://thread.gmane.org/gmane.comp.version-control.git/247171\nhttp://thread.gmane.org/gmane.comp.version-control.git/202902/focus=203020\n\nHis work, which is not compile-tested any more, is still there:\n\nhttps://github.com/peff/git/commits/jk/external-odb-wip\n\nInitial discussions about this new series are there:\n\nhttp://thread.gmane.org/gmane.comp.version-control.git/288151/focus=295160\n\nVersion 1, 2, 3, 4 and 5 of this series are here:\n\nhttps://public-inbox.org/git/20160613085546.11784-1-chriscool@tuxfamily.org/\nhttps://public-inbox.org/git/20160628181933.24620-1-chriscool@tuxfamily.org/\nhttps://public-inbox.org/git/20161130210420.15982-1-chriscool@tuxfamily.org/\nhttps://public-inbox.org/git/20170620075523.26961-1-chriscool@tuxfamily.org/\nhttps://public-inbox.org/git/20170803091926.1755-1-chriscool@tuxfamily.org/\n\nSome of the discussions related to Ben Peart's work that is used by\nthis series are here:\n\nhttps://public-inbox.org/git/20170113155253.1644-1-benpeart@microsoft.com/\nhttps://public-inbox.org/git/20170322165220.5660-1-benpeart@microsoft.com/\nhttps://public-inbox.org/git/20170714132651.170708-1-benpeart@microsoft.com/\n\nLinks\n~~~~~\n\nThis patch series is available here:\n\nhttps://github.com/chriscool/git/commits/external-odb\n\nVersion 1, 2, 3, 4 and 5 are here:\n\nhttps://github.com/chriscool/git/commits/gl-external-odb12\nhttps://github.com/chriscool/git/commits/gl-external-odb22\nhttps://github.com/chriscool/git/commits/gl-external-odb61\nhttps://github.com/chriscool/git/commits/gl-external-odb239\nhttps://github.com/chriscool/git/commits/gl-external-odb373\n\n\nBen Peart (2):\n odb-helper: add init_object_process()\n Add t0450 to test 'get_direct' mechanism\n\nChristian Couder (38):\n builtin/clone: get rid of 'value' strbuf\n t0021/rot13-filter: refactor packet reading functions\n t0021/rot13-filter: improve 'if .. elsif .. else' style\n t0021/rot13-filter: improve error message\n t0021/rot13-filter: add packet_initialize()\n t0021/rot13-filter: add capability functions\n Add Git/Packet.pm from parts of t0021/rot13-filter.pl\n sha1_file: prepare for external odbs\n Add initial external odb support\n odb-helper: add odb_helper_init() to send 'init' instruction\n t0400: add 'put_raw_obj' instruction to odb-helper script\n external odb: add 'put_raw_obj' support\n external-odb: accept only blobs for now\n t0400: add test for external odb write support\n Add GIT_NO_EXTERNAL_ODB env variable\n Add t0410 to test external ODB transfer\n lib-httpd: pass config file to start_httpd()\n lib-httpd: add upload.sh\n lib-httpd: add list.sh\n lib-httpd: add apache-e-odb.conf\n odb-helper: add odb_helper_get_raw_object()\n pack-objects: don't pack objects in external odbs\n Add t0420 to test transfer to HTTP external odb\n external-odb: add 'get_direct' support\n odb-helper: add 'script_mode' to 'struct odb_helper'\n Add t0460 to test passing git objects\n odb-helper: add put_object_process()\n Add t0470 to test passing raw objects\n odb-helper: add have_object_process()\n Add t0480 to test \"have\" capability and raw objects\n external-odb: use 'odb=magic' attribute to mark odb blobs\n Add Documentation/technical/external-odb.txt\n clone: add 'initial' param to write_remote_refs()\n clone: add --initial-refspec option\n clone: disable external odb before initial clone\n Add tests for 'clone --initial-refspec'\n Add t0430 to test cloning using bundles\n Doc/external-odb: explain transfering objects and metadata\n\n Documentation/technical/external-odb.txt | 447 +++++++++++++\n Makefile | 2 +\n builtin/clone.c | 91 ++-\n builtin/pack-objects.c | 4 +\n cache.h | 18 +\n environment.c | 4 +\n external-odb.c | 196 ++++++\n external-odb.h | 12 +\n odb-helper.c | 1076 ++++++++++++++++++++++++++++++\n odb-helper.h | 45 ++\n perl/Git/Packet.pm | 118 ++++\n sha1_file.c | 155 +++--\n t/lib-httpd.sh | 8 +-\n t/lib-httpd/apache-e-odb.conf | 214 ++++++\n t/lib-httpd/list.sh | 41 ++\n t/lib-httpd/upload.sh | 45 ++\n t/t0021/rot13-filter.pl | 110 +--\n t/t0400-external-odb.sh | 85 +++\n t/t0410-transfer-e-odb.sh | 147 ++++\n t/t0420-transfer-http-e-odb.sh | 152 +++++\n t/t0430-clone-bundle-e-odb.sh | 85 +++\n t/t0450-read-object.sh | 28 +\n t/t0450/read-object | 68 ++\n t/t0460-read-object-git.sh | 28 +\n t/t0460/read-object-git | 78 +++\n t/t0470-read-object-http-e-odb.sh | 119 ++++\n t/t0470/read-object-plain | 83 +++\n t/t0480-read-object-have-http-e-odb.sh | 119 ++++\n t/t0480/read-object-plain-have | 103 +++\n t/t5616-clone-initial-refspec.sh | 48 ++\n 30 files changed, 3588 insertions(+), 141 deletions(-)\n create mode 100644 Documentation/technical/external-odb.txt\n create mode 100644 external-odb.c\n create mode 100644 external-odb.h\n create mode 100644 odb-helper.c\n create mode 100644 odb-helper.h\n create mode 100644 perl/Git/Packet.pm\n create mode 100644 t/lib-httpd/apache-e-odb.conf\n create mode 100644 t/lib-httpd/list.sh\n create mode 100644 t/lib-httpd/upload.sh\n create mode 100755 t/t0400-external-odb.sh\n create mode 100755 t/t0410-transfer-e-odb.sh\n create mode 100755 t/t0420-transfer-http-e-odb.sh\n create mode 100755 t/t0430-clone-bundle-e-odb.sh\n create mode 100755 t/t0450-read-object.sh\n create mode 100755 t/t0450/read-object\n create mode 100755 t/t0460-read-object-git.sh\n create mode 100755 t/t0460/read-object-git\n create mode 100755 t/t0470-read-object-http-e-odb.sh\n create mode 100755 t/t0470/read-object-plain\n create mode 100755 t/t0480-read-object-have-http-e-odb.sh\n create mode 100755 t/t0480/read-object-plain-have\n create mode 100755 t/t5616-clone-initial-refspec.sh\n\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328225","messageId":"20170916080731.13925-4-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 03/40] t0021/rot13-filter: improve 'if .. elsif .. else' style","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:06:54Z","receivedAt":"2017-09-16T08:10:25Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Before further refactoring the \"t0021/rot13-filter.pl\" script,\nlet's modernize the style of its 'if .. elsif .. else' clauses\nto improve its readability by making it more similar to our\nother perl scripts.\n\nSigned-off-by: Christian Couder \n---\n t/t0021/rot13-filter.pl | 39 +++++++++++++--------------------------\n 1 file changed, 13 insertions(+), 26 deletions(-)\n\ndiff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\nindex e4495a52f3..82882392ae 100644\n--- a/t/t0021/rot13-filter.pl\n+++ b/t/t0021/rot13-filter.pl\n@@ -61,23 +61,20 @@ sub packet_bin_read {\n \tif ( $bytes_read == 0 ) {\n \t\t# EOF - Git stopped talking to us!\n \t\treturn ( -1, \"\" );\n-\t}\n-\telsif ( $bytes_read != 4 ) {\n+\t} elsif ( $bytes_read != 4 ) {\n \t\tdie \"invalid packet: '$buffer'\";\n \t}\n \tmy $pkt_size = hex($buffer);\n \tif ( $pkt_size == 0 ) {\n \t\treturn ( 1, \"\" );\n-\t}\n-\telsif ( $pkt_size > 4 ) {\n+\t} elsif ( $pkt_size > 4 ) {\n \t\tmy $content_size = $pkt_size - 4;\n \t\t$bytes_read = read STDIN, $buffer, $content_size;\n \t\tif ( $bytes_read != $content_size ) {\n \t\t\tdie \"invalid packet ($content_size bytes expected; $bytes_read bytes read)\";\n \t\t}\n \t\treturn ( 0, $buffer );\n-\t}\n-\telse {\n+\t} else {\n \t\tdie \"invalid packet size: $pkt_size\";\n \t}\n }\n@@ -165,8 +162,7 @@ while (1) {\n \t\t$debug->flush();\n \t\tpacket_txt_write(\"status=success\");\n \t\tpacket_flush();\n-\t}\n-\telse {\n+\t} else {\n \t\tmy ( $pathname ) = packet_txt_read() =~ /^pathname=(.+)$/;\n \t\tprint $debug \" $pathname\";\n \t\t$debug->flush();\n@@ -205,17 +201,13 @@ while (1) {\n \t\tmy $output;\n \t\tif ( exists $DELAY{$pathname} and exists $DELAY{$pathname}{\"output\"} ) {\n \t\t\t$output = $DELAY{$pathname}{\"output\"}\n-\t\t}\n-\t\telsif ( $pathname eq \"error.r\" or $pathname eq \"abort.r\" ) {\n+\t\t} elsif ( $pathname eq \"error.r\" or $pathname eq \"abort.r\" ) {\n \t\t\t$output = \"\";\n-\t\t}\n-\t\telsif ( $command eq \"clean\" and grep( /^clean$/, @capabilities ) ) {\n+\t\t} elsif ( $command eq \"clean\" and grep( /^clean$/, @capabilities ) ) {\n \t\t\t$output = rot13($input);\n-\t\t}\n-\t\telsif ( $command eq \"smudge\" and grep( /^smudge$/, @capabilities ) ) {\n+\t\t} elsif ( $command eq \"smudge\" and grep( /^smudge$/, @capabilities ) ) {\n \t\t\t$output = rot13($input);\n-\t\t}\n-\t\telse {\n+\t\t} else {\n \t\t\tdie \"bad command '$command'\";\n \t\t}\n \n@@ -224,25 +216,21 @@ while (1) {\n \t\t\t$debug->flush();\n \t\t\tpacket_txt_write(\"status=error\");\n \t\t\tpacket_flush();\n-\t\t}\n-\t\telsif ( $pathname eq \"abort.r\" ) {\n+\t\t} elsif ( $pathname eq \"abort.r\" ) {\n \t\t\tprint $debug \"[ABORT]\\n\";\n \t\t\t$debug->flush();\n \t\t\tpacket_txt_write(\"status=abort\");\n \t\t\tpacket_flush();\n-\t\t}\n-\t\telsif ( $command eq \"smudge\" and\n+\t\t} elsif ( $command eq \"smudge\" and\n \t\t\texists $DELAY{$pathname} and\n-\t\t\t$DELAY{$pathname}{\"requested\"} == 1\n-\t\t) {\n+\t\t\t$DELAY{$pathname}{\"requested\"} == 1 ) {\n \t\t\tprint $debug \"[DELAYED]\\n\";\n \t\t\t$debug->flush();\n \t\t\tpacket_txt_write(\"status=delayed\");\n \t\t\tpacket_flush();\n \t\t\t$DELAY{$pathname}{\"requested\"} = 2;\n \t\t\t$DELAY{$pathname}{\"output\"} = $output;\n-\t\t}\n-\t\telse {\n+\t\t} else {\n \t\t\tpacket_txt_write(\"status=success\");\n \t\t\tpacket_flush();\n \n@@ -262,8 +250,7 @@ while (1) {\n \t\t\t\tprint $debug \".\";\n \t\t\t\tif ( length($output) > $MAX_PACKET_CONTENT_SIZE ) {\n \t\t\t\t\t$output = substr( $output, $MAX_PACKET_CONTENT_SIZE );\n-\t\t\t\t}\n-\t\t\t\telse {\n+\t\t\t\t} else {\n \t\t\t\t\t$output = \"\";\n \t\t\t\t}\n \t\t\t}\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328226","messageId":"20170916080731.13925-9-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 08/40] sha1_file: prepare for external odbs","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:06:59Z","receivedAt":"2017-09-16T08:10:28Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"In the following commits we will need some functions that were\ninternal to sha1_file.c, so let's first make them non static\nand declare them in \"cache.h\". While at it, let's rename\n'create_tmpfile()' to 'create_object_tmpfile()' to make its\nname less generic.\n\nLet's also split out 'sha1_file_name_alt()' from\n'sha1_file_name()' and 'open_sha1_file_alt()' from\n'open_sha1_file()', as we will need both of these new\nfunctions too.\n\nHelped-by: Jeff King \nSigned-off-by: Christian Couder \n---\n cache.h | 8 ++++++++\n sha1_file.c | 47 +++++++++++++++++++++++++++++------------------\n 2 files changed, 37 insertions(+), 18 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex a916bc79e3..00d89568f3 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -902,6 +902,12 @@ extern void check_repository_format(void);\n */\n extern const char *sha1_file_name(const unsigned char *sha1);\n \n+/*\n+ * Like sha1_file_name, but return the filename within a specific alternate\n+ * object directory. Shares the same static buffer with sha1_file_name.\n+ */\n+extern const char *sha1_file_name_alt(const char *objdir, const unsigned char *sha1);\n+\n /*\n * Return an abbreviated sha1 unique within this repository's object database.\n * The result will be at least `len` characters long, and will be NUL\n@@ -1189,6 +1195,8 @@ extern int parse_sha1_header(const char *hdr, unsigned long *sizep);\n \n extern int check_sha1_signature(const unsigned char *sha1, void *buf, unsigned long size, const char *type);\n \n+extern int create_object_tmpfile(struct strbuf *tmp, const char *filename);\n+extern void close_sha1_file(int fd);\n extern int finalize_object_file(const char *tmpfile, const char *filename);\n \n /*\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 5f71bbac3e..bea1ae6afb 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -251,17 +251,22 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)\n \t}\n }\n \n-const char *sha1_file_name(const unsigned char *sha1)\n+const char *sha1_file_name_alt(const char *objdir, const unsigned char *sha1)\n {\n \tstatic struct strbuf buf = STRBUF_INIT;\n \n \tstrbuf_reset(&buf);\n-\tstrbuf_addf(&buf, \"%s/\", get_object_directory());\n+\tstrbuf_addf(&buf, \"%s/\", objdir);\n \n \tfill_sha1_path(&buf, sha1);\n \treturn buf.buf;\n }\n \n+const char *sha1_file_name(const unsigned char *sha1)\n+{\n+\treturn sha1_file_name_alt(get_object_directory(), sha1);\n+}\n+\n struct strbuf *alt_scratch_buf(struct alternate_object_database *alt)\n {\n \tstrbuf_setlen(&alt->scratch, alt->base_len);\n@@ -822,24 +827,14 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,\n \treturn -1;\n }\n \n-/*\n- * Like stat_sha1_file(), but actually open the object and return the\n- * descriptor. See the caveats on the \"path\" parameter above.\n- */\n-static int open_sha1_file(const unsigned char *sha1, const char **path)\n+static int open_sha1_file_alt(const unsigned char *sha1, const char **path)\n {\n-\tint fd;\n \tstruct alternate_object_database *alt;\n-\tint most_interesting_errno;\n-\n-\t*path = sha1_file_name(sha1);\n-\tfd = git_open(*path);\n-\tif (fd >= 0)\n-\t\treturn fd;\n-\tmost_interesting_errno = errno;\n+\tint most_interesting_errno = errno;\n \n \tprepare_alt_odb();\n \tfor (alt = alt_odb_list; alt; alt = alt->next) {\n+\t\tint fd;\n \t\t*path = alt_sha1_path(alt, sha1);\n \t\tfd = git_open(*path);\n \t\tif (fd >= 0)\n@@ -851,6 +846,22 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)\n \treturn -1;\n }\n \n+/*\n+ * Like stat_sha1_file(), but actually open the object and return the\n+ * descriptor. See the caveats on the \"path\" parameter above.\n+ */\n+static int open_sha1_file(const unsigned char *sha1, const char **path)\n+{\n+\tint fd;\n+\n+\t*path = sha1_file_name(sha1);\n+\tfd = git_open(*path);\n+\tif (fd >= 0)\n+\t\treturn fd;\n+\n+\treturn open_sha1_file_alt(sha1, path);\n+}\n+\n /*\n * Map the loose object at \"path\" if it is not NULL, or the path found by\n * searching for a loose object named \"sha1\".\n@@ -1428,7 +1439,7 @@ int hash_sha1_file(const void *buf, unsigned long len, const char *type,\n }\n \n /* Finalize a file on disk, and close it. */\n-static void close_sha1_file(int fd)\n+void close_sha1_file(int fd)\n {\n \tif (fsync_object_files)\n \t\tfsync_or_die(fd, \"sha1 file\");\n@@ -1452,7 +1463,7 @@ static inline int directory_size(const char *filename)\n * We want to avoid cross-directory filename renames, because those\n * can have problems on various filesystems (FAT, NFS, Coda).\n */\n-static int create_tmpfile(struct strbuf *tmp, const char *filename)\n+int create_object_tmpfile(struct strbuf *tmp, const char *filename)\n {\n \tint fd, dirlen = directory_size(filename);\n \n@@ -1492,7 +1503,7 @@ static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,\n \tstatic struct strbuf tmp_file = STRBUF_INIT;\n \tconst char *filename = sha1_file_name(sha1);\n \n-\tfd = create_tmpfile(&tmp_file, filename);\n+\tfd = create_object_tmpfile(&tmp_file, filename);\n \tif (fd < 0) {\n \t\tif (errno == EACCES)\n \t\t\treturn error(\"insufficient permission for adding an object to repository database %s\", get_object_directory());\n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328227","messageId":"20170916080731.13925-3-chriscool@tuxfamily.org","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"[PATCH v6 02/40] t0021/rot13-filter: refactor packet reading functions","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-16T08:06:53Z","receivedAt":"2017-09-16T08:10:35Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"To make it possible in a following commit to move packet\nreading and writing functions into a Packet.pm module,\nlet's refactor these functions, so they don't handle\nprinting debug output and exiting.\n\nSigned-off-by: Christian Couder \n---\n t/t0021/rot13-filter.pl | 12 ++++++++----\n 1 file changed, 8 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\nindex ad685d92f8..e4495a52f3 100644\n--- a/t/t0021/rot13-filter.pl\n+++ b/t/t0021/rot13-filter.pl\n@@ -60,8 +60,7 @@ sub packet_bin_read {\n \tmy $bytes_read = read STDIN, $buffer, 4;\n \tif ( $bytes_read == 0 ) {\n \t\t# EOF - Git stopped talking to us!\n-\t\tprint $debug \"STOP\\n\";\n-\t\texit();\n+\t\treturn ( -1, \"\" );\n \t}\n \telsif ( $bytes_read != 4 ) {\n \t\tdie \"invalid packet: '$buffer'\";\n@@ -85,7 +84,7 @@ sub packet_bin_read {\n \n sub packet_txt_read {\n \tmy ( $res, $buf ) = packet_bin_read();\n-\tunless ( $buf eq '' or $buf =~ s/\\n$// ) {\n+\tunless ( $res == -1 or $buf eq '' or $buf =~ s/\\n$// ) {\n \t\tdie \"A non-binary line MUST be terminated by an LF.\";\n \t}\n \treturn ( $res, $buf );\n@@ -131,7 +130,12 @@ print $debug \"init handshake complete\\n\";\n $debug->flush();\n \n while (1) {\n-\tmy ( $command ) = packet_txt_read() =~ /^command=(.+)$/;\n+\tmy ( $res, $command ) = packet_txt_read();\n+\tif ( $res == -1 ) {\n+\t\tprint $debug \"STOP\\n\";\n+\t\texit();\n+\t}\n+\t$command =~ s/^command=//;\n \tprint $debug \"IN: $command\";\n \t$debug->flush();\n \n-- \n2.14.1.576.g3f707d88cd\n\n"},{"id":"328396","messageId":"20170919104553.494d7d3b@twelve2.svl.corp.google.com","threadId":"46764","inReplyTo":"20170916080731.13925-10-chriscool@tuxfamily.org","subject":"Re: [PATCH v6 09/40] Add initial external odb support","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-09-19T17:45:53Z","receivedAt":"2017-09-19T17:46:03Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"I wonder if it's better to get a change like this (PATCH v6 09/40 and\nany of the previous patches that this depends on) in and then build on\nit rather than to review the whole patch set at a time. This would\nreduce ripple effects (of needing to change later patches in a patch set\nmultiple times unnecessarily) and help collaboration (in that multiple\npeople can write patches, since the foundation would already be laid).\n\nThe same concerns about fsck apply, but that shouldn't be a problem,\nsince this patch provides the same internal API as mine (\"get\" function\ntaking in a single hash, and \"have\" function taking in a single hash) so\nit shouldn't be too difficult to adapt my fsck and gc patches [1]. (I\ncan do that if necessary.)\n\n[1] https://public-inbox.org/git/20170915134343.3814dc38@twelve2.svl.corp.google.com/\n\nOne possible issue (with committing something early) is that later work\n(for example, a fast long-running process protocol) will make the\nearlier work (for example, here, a simple single-shot protocol)\nobsolete, while saddling us with the necessity of maintaining the\nearlier one. To that end, if we want to start with the support for a\nhook, a better approach might be to only code the fast long-running\nprocess protocol, and put a wrapper script in contrib/ that can wrap a\nsingle-shot process in a long-running process.\n\nAnd another possible issue is that we design ourselves into a corner.\nThinking about the use cases that I know about (the Android use case and\nthe Microsoft GVFS use case), I don't think we are doing that - for\nAndroid, this means that large blob metadata needs to be part of the\ndesign (and this patch series does provide for that), and for Microsoft\nGVFS, \"get\" is relatively cheap, so a configuration option to not invoke\n\"have\" first when loading a missing object might be sufficient.\n\nAs for the design itself (including fetch and clone), it differs from my\npatches (linked above as [1]) in that mine is self-contained (requiring\nonly an updated Git server and Git client) whereas this, as far as I can\ntell, requires an external process and some measure of coordination\nbetween the administrator of the server and the client user (for\nexample, the client must have the same ODB mechanism as the server, if\nnot, the server might omit certain blobs that the client does not know\nhow to fetch).\n\nAnd I think that my design can be extended to support a use case in\nwhich, for example, blobs corresponding to a certain type of filename\n(defined by a glob like in gitattributes) can be excluded during\nfetch/clone, much like --blob-max-bytes, and they can be fetched either\nthrough the built-in mechanism or through a custom hook.\n\nFor those reasons, I still lean towards my design, but if we do want to\ngo with this design, here are my comments about this patch...\n\nFirst of all:\n - You'll probably need to add a repository extension.\n - I get compile errors when I \"git am\" these onto master. I think\n '#include \"config.h\"' is needed in some places.\n\nOn Sat, 16 Sep 2017 10:07:00 +0200\nChristian Couder wrote:\n\n> The external-odb.{c,h} files contains the functions that are\n> called by the rest of Git from \"sha1_file.c\".\n> \n> The odb-helper.{c,h} files contains the functions to\n> actually implement communication with the external scripts or\n> processes that will manage external git objects.\n> \n> For now only script mode is supported, and only the 'have' and\n> 'get_git_obj' instructions are supported.\n\nThis \"have\", as I see from this commit, is more like a \"list\" command in\nthat it lists all hashes that it knows about, and does not check if a\ngiven hash exists.\n\n> +static struct odb_helper *helpers;\n> +static struct odb_helper **helpers_tail = &helpers;\n\nThis could be done with the helpers in list.h instead.\n\n> +int external_odb_get_object(const unsigned char *sha1)\n> +{\n> +\tstruct odb_helper *o;\n> +\tconst char *path;\n> +\n> +\tif (!external_odb_has_object(sha1))\n> +\t\treturn -1;\n> +\n> +\tpath = sha1_file_name_alt(external_odb_root(), sha1);\n\nIf the purpose of making these functions global in the previous patch is\njust for temporary names, I don't think it's necessary for them to be\nglobal. Just concatenate the hex SHA1 to external_odb_root()?\n\n> /* Returns 1 if we have successfully freshened the file, 0 otherwise. */\n> @@ -667,7 +684,7 @@ static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)\n> \t\tif (check_and_freshen_file(path, freshen))\n> \t\t\treturn 1;\n> \t}\n> -\treturn 0;\n> +\treturn external_odb_has_object(sha1);\n> }\n> \n> static int check_and_freshen(const unsigned char *sha1, int freshen)\n> @@ -824,6 +841,9 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,\n> \t\t\treturn 0;\n> \t}\n> \n> +\tif (!external_odb_get_object(sha1) && !lstat(*path, st))\n> +\t\treturn 0;\n> +\n> \treturn -1;\n> }\n> \n> @@ -859,7 +879,14 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)\n> \tif (fd >= 0)\n> \t\treturn fd;\n> \n> -\treturn open_sha1_file_alt(sha1, path);\n> +\tfd = open_sha1_file_alt(sha1, path);\n> +\tif (fd >= 0)\n> +\t\treturn fd;\n> +\n> +\tif (!external_odb_get_object(sha1))\n> +\t\tfd = open_sha1_file_alt(sha1, path);\n> +\n> +\treturn fd;\n> }\n\nAny reason why you prefer to update the loose object functions than to\nupdate the generic one (sha1_object_info_extended)? My concern with just\nupdating the loose object functions was that a caller might have\nobtained the path by iterating through the loose object dirs, and in\nthat case we shouldn't query the external ODB for anything.\n\n> +ALT_SOURCE=\"$PWD/alt-repo/.git\"\n> +export ALT_SOURCE\n> +write_script odb-helper <<\\EOF\n> +GIT_DIR=$ALT_SOURCE; export GIT_DIR\n> +case \"$1\" in\n> +have)\n> +\tgit cat-file --batch-check --batch-all-objects |\n> +\tawk '{print $1 \" \" $3 \" \" $2}'\n> +\t;;\n> +get_git_obj)\n> +\tcat \"$GIT_DIR\"/objects/$(echo $2 | sed 's#..#&/#')\n> +\t;;\n> +esac\n> +EOF\n> +HELPER=\"\\\"$PWD\\\"/odb-helper\"\n\nThanks for the clear test. It is very obvious that \"have\" returns a list\nof objects, and \"get_git_obj\" returns the compressed loose object with\nthe Git loose object header included.\n\n> +\n> +test_expect_success 'setup alternate repo' '\n> +\tgit init alt-repo &&\n> +\t(cd alt-repo &&\n> +\t test_commit one &&\n\nProbably better written as \"test_commit -C alt-repo one\".\n\n> +\t test_commit two\n> +\t) &&\n> +\talt_head=`cd alt-repo && git rev-parse HEAD`\n\nI think the style is to use $() and \"git -C alt-repo rev-parse HEAD\".\n"},{"id":"329041","messageId":"CAP8UFD2D2NkKKy9oT-1Mwe0Gq6=UG+15t9GENKKbV-PbRf3Kkw@mail.gmail.com","threadId":"46764","inReplyTo":"20170919104553.494d7d3b@twelve2.svl.corp.google.com","subject":"Re: [PATCH v6 09/40] Add initial external odb support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-27T16:46:30Z","receivedAt":"2017-09-27T16:46:37Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Tue, Sep 19, 2017 at 7:45 PM, Jonathan Tan wrote:\n> I wonder if it's better to get a change like this (PATCH v6 09/40 and\n> any of the previous patches that this depends on) in and then build on\n> it rather than to review the whole patch set at a time. This would\n> reduce ripple effects (of needing to change later patches in a patch set\n> multiple times unnecessarily) and help collaboration (in that multiple\n> people can write patches, since the foundation would already be laid).\n\nI am ok to split the patch series, but I am not sure that 01/40 to\n09/40 is the right range for the first patch series.\nI would say that 01/40 to 07/40 is better as it can be seen as a\nseparate refactoring.\n\n> The same concerns about fsck apply, but that shouldn't be a problem,\n> since this patch provides the same internal API as mine (\"get\" function\n> taking in a single hash, and \"have\" function taking in a single hash) so\n> it shouldn't be too difficult to adapt my fsck and gc patches [1]. (I\n> can do that if necessary.)\n>\n> [1] https://public-inbox.org/git/20170915134343.3814dc38@twelve2.svl.corp.google.com/\n\nGreat! I would be happy with such an outcome.\n\n> One possible issue (with committing something early) is that later work\n> (for example, a fast long-running process protocol) will make the\n> earlier work (for example, here, a simple single-shot protocol)\n> obsolete, while saddling us with the necessity of maintaining the\n> earlier one. To that end, if we want to start with the support for a\n> hook, a better approach might be to only code the fast long-running\n> process protocol, and put a wrapper script in contrib/ that can wrap a\n> single-shot process in a long-running process.\n\nI don't think single-shot processes would be a huge burden, because\nthe code is simpler, and because for example for filters we already\nhave single shot and long-running processes and no one complains about\nthat. It's code that is useful as it makes it much easier for people\nto do some things (see the clone bundle example).\n\nIn fact in Git development we usually start to by first implementing\nsimpler single-shot solutions, before thinking, when the need arise,\nto make it faster. So a perhaps an equally valid opinion could be to\nfirst only submit the patches for the single-shot protocol and later\nsubmit the rest of the series when we start getting feedback about how\nexternal odbs are used.\n\nAnd yeah I could change the order of the patch series to implement the\nlong-running processes first and the single-shot process last, so that\nit could be possible to first get feedback about the long-running\nprocesses, before we decide to merge or not the single-shot stuff, but\nI don't think it would look like the most logical order.\n\n> And another possible issue is that we design ourselves into a corner.\n> Thinking about the use cases that I know about (the Android use case and\n> the Microsoft GVFS use case), I don't think we are doing that - for\n> Android, this means that large blob metadata needs to be part of the\n> design (and this patch series does provide for that), and for Microsoft\n> GVFS, \"get\" is relatively cheap, so a configuration option to not invoke\n> \"have\" first when loading a missing object might be sufficient.\n\nIf the helper does not advertise the \"have\" capability, the \"have\"\ninstruction will not be sent to the helper, so the current design is\nalready working for that case.\n\n> As for the design itself (including fetch and clone), it differs from my\n> patches (linked above as [1]) in that mine is self-contained (requiring\n> only an updated Git server and Git client) whereas this, as far as I can\n> tell, requires an external process and some measure of coordination\n> between the administrator of the server and the client user (for\n> example, the client must have the same ODB mechanism as the server, if\n> not, the server might omit certain blobs that the client does not know\n> how to fetch).\n\nYeah, your design is more self contained, but it doesn't handle as\nmany use cases.\n\n> And I think that my design can be extended to support a use case in\n> which, for example, blobs corresponding to a certain type of filename\n> (defined by a glob like in gitattributes) can be excluded during\n> fetch/clone, much like --blob-max-bytes, and they can be fetched either\n> through the built-in mechanism or through a custom hook.\n\nSure, we could probably rebuild something equivalent to what I did on\ntop of your design.\nMy opinion though is that if we want to eventually get to the same\ngoal, it is better to first merge something that get us very close to\nthe end goal and then add some improvements on top of it.\n\n> For those reasons, I still lean towards my design, but if we do want to\n> go with this design, here are my comments about this patch...\n>\n> First of all:\n> - You'll probably need to add a repository extension.\n\nI am ok to add one if this is needed, but I am not sure it is, as the\nrepository format does not really change.\n\n> - I get compile errors when I \"git am\" these onto master. I think\n> '#include \"config.h\"' is needed in some places.\n\nIt's strange because I get no compile errors even after a \"make clean\"\nfrom my branch.\nCould you show the actual errors?\n\n> On Sat, 16 Sep 2017 10:07:00 +0200\n> Christian Couder wrote:\n>\n>> The external-odb.{c,h} files contains the functions that are\n>> called by the rest of Git from \"sha1_file.c\".\n>>\n>> The odb-helper.{c,h} files contains the functions to\n>> actually implement communication with the external scripts or\n>> processes that will manage external git objects.\n>>\n>> For now only script mode is supported, and only the 'have' and\n>> 'get_git_obj' instructions are supported.\n>\n> This \"have\", as I see from this commit, is more like a \"list\" command in\n> that it lists all hashes that it knows about, and does not check if a\n> given hash exists.\n\nYes.\n\n>> +static struct odb_helper *helpers;\n>> +static struct odb_helper **helpers_tail = &helpers;\n>\n> This could be done with the helpers in list.h instead.\n\nYeah, but list.h is for a doubly-linked list and I am not sure we need that.\n\n>> +int external_odb_get_object(const unsigned char *sha1)\n>> +{\n>> + struct odb_helper *o;\n>> + const char *path;\n>> +\n>> + if (!external_odb_has_object(sha1))\n>> + return -1;\n>> +\n>> + path = sha1_file_name_alt(external_odb_root(), sha1);\n>\n> If the purpose of making these functions global in the previous patch is\n> just for temporary names, I don't think it's necessary for them to be\n> global. Just concatenate the hex SHA1 to external_odb_root()?\n\nIn my opinion it is cleaner to make a few functions that are needed\nglobal rather than have some custom code, even if the custom code is\nsmall.\n(And I guess that Peff who wrote the above agrees.)\n\n>> /* Returns 1 if we have successfully freshened the file, 0 otherwise. */\n>> @@ -667,7 +684,7 @@ static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)\n>> if (check_and_freshen_file(path, freshen))\n>> return 1;\n>> }\n>> - return 0;\n>> + return external_odb_has_object(sha1);\n>> }\n>>\n>> static int check_and_freshen(const unsigned char *sha1, int freshen)\n>> @@ -824,6 +841,9 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,\n>> return 0;\n>> }\n>>\n>> + if (!external_odb_get_object(sha1) && !lstat(*path, st))\n>> + return 0;\n>> +\n>> return -1;\n>> }\n>>\n>> @@ -859,7 +879,14 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)\n>> if (fd >= 0)\n>> return fd;\n>>\n>> - return open_sha1_file_alt(sha1, path);\n>> + fd = open_sha1_file_alt(sha1, path);\n>> + if (fd >= 0)\n>> + return fd;\n>> +\n>> + if (!external_odb_get_object(sha1))\n>> + fd = open_sha1_file_alt(sha1, path);\n>> +\n>> + return fd;\n>> }\n>\n> Any reason why you prefer to update the loose object functions than to\n> update the generic one (sha1_object_info_extended)? My concern with just\n> updating the loose object functions was that a caller might have\n> obtained the path by iterating through the loose object dirs, and in\n> that case we shouldn't query the external ODB for anything.\n\nYou are thinking about fsck or gc?\nOtherwise I don't think it would be clean to iterate through loose object dirs.\n\n>> +ALT_SOURCE=\"$PWD/alt-repo/.git\"\n>> +export ALT_SOURCE\n>> +write_script odb-helper <<\\EOF\n>> +GIT_DIR=$ALT_SOURCE; export GIT_DIR\n>> +case \"$1\" in\n>> +have)\n>> + git cat-file --batch-check --batch-all-objects |\n>> + awk '{print $1 \" \" $3 \" \" $2}'\n>> + ;;\n>> +get_git_obj)\n>> + cat \"$GIT_DIR\"/objects/$(echo $2 | sed 's#..#&/#')\n>> + ;;\n>> +esac\n>> +EOF\n>> +HELPER=\"\\\"$PWD\\\"/odb-helper\"\n>\n> Thanks for the clear test. It is very obvious that \"have\" returns a list\n> of objects, and \"get_git_obj\" returns the compressed loose object with\n> the Git loose object header included.\n\nHappy that you like how simple helpers can be with single-shot processes ;-)\n\n>> +test_expect_success 'setup alternate repo' '\n>> + git init alt-repo &&\n>> + (cd alt-repo &&\n>> + test_commit one &&\n>\n> Probably better written as \"test_commit -C alt-repo one\".\n\nOk, I will use that .\n\n>> + test_commit two\n>> + ) &&\n>> + alt_head=`cd alt-repo && git rev-parse HEAD`\n>\n> I think the style is to use $() and \"git -C alt-repo rev-parse HEAD\".\n\nOk, I will change that.\n\nThanks,\nChristian.\n"},{"id":"329213","messageId":"20170929133614.1c0cd68ad80139fafdf68b86@google.com","threadId":"46764","inReplyTo":"CAP8UFD2D2NkKKy9oT-1Mwe0Gq6=UG+15t9GENKKbV-PbRf3Kkw@mail.gmail.com","subject":"Re: [PATCH v6 09/40] Add initial external odb support","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-09-29T20:36:14Z","receivedAt":"2017-09-29T20:36:21Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Wed, 27 Sep 2017 18:46:30 +0200\nChristian Couder wrote:\n\n> I am ok to split the patch series, but I am not sure that 01/40 to\n> 09/40 is the right range for the first patch series.\n> I would say that 01/40 to 07/40 is better as it can be seen as a\n> separate refactoring.\n\nI mentioned 09/40 because this is (as far as I can tell) the first one\nthat introduces a new design.\n\n> I don't think single-shot processes would be a huge burden, because\n> the code is simpler, and because for example for filters we already\n> have single shot and long-running processes and no one complains about\n> that. It's code that is useful as it makes it much easier for people\n> to do some things (see the clone bundle example).\n> \n> In fact in Git development we usually start to by first implementing\n> simpler single-shot solutions, before thinking, when the need arise,\n> to make it faster. So a perhaps an equally valid opinion could be to\n> first only submit the patches for the single-shot protocol and later\n> submit the rest of the series when we start getting feedback about how\n> external odbs are used.\n\nMy concern is that, as far as I understand about the Microsoft use case,\nwe already know that we need the faster solution, so the need has\nalready arisen.\n\n> And yeah I could change the order of the patch series to implement the\n> long-running processes first and the single-shot process last, so that\n> it could be possible to first get feedback about the long-running\n> processes, before we decide to merge or not the single-shot stuff, but\n> I don't think it would look like the most logical order.\n\nMy thinking was that we would just implement the long-running process\nand not implement the single-shot process at all (besides maybe a script\nin contrib/). If we are going to do both anyway, I agree that we should\ndo the single-shot process first.\n\n> > And another possible issue is that we design ourselves into a corner.\n> > Thinking about the use cases that I know about (the Android use case and\n> > the Microsoft GVFS use case), I don't think we are doing that - for\n> > Android, this means that large blob metadata needs to be part of the\n> > design (and this patch series does provide for that), and for Microsoft\n> > GVFS, \"get\" is relatively cheap, so a configuration option to not invoke\n> > \"have\" first when loading a missing object might be sufficient.\n> \n> If the helper does not advertise the \"have\" capability, the \"have\"\n> instruction will not be sent to the helper, so the current design is\n> already working for that case.\n\nAh, that's good to know.\n\n> > And I think that my design can be extended to support a use case in\n> > which, for example, blobs corresponding to a certain type of filename\n> > (defined by a glob like in gitattributes) can be excluded during\n> > fetch/clone, much like --blob-max-bytes, and they can be fetched either\n> > through the built-in mechanism or through a custom hook.\n> \n> Sure, we could probably rebuild something equivalent to what I did on\n> top of your design.\n> My opinion though is that if we want to eventually get to the same\n> goal, it is better to first merge something that get us very close to\n> the end goal and then add some improvements on top of it.\n\nI agree - I mentioned that because I personally prefer to review smaller\npatch sets at a time, and my patch set already includes a lot of the\nsame infrastructure needed by yours - for example, the places in the\ncode to dynamically fetch objects, exclusion of objects when fetching or\ncloning, configuring the cloned repo when cloning, fsck, and gc.\n\n> > - I get compile errors when I \"git am\" these onto master. I think\n> > '#include \"config.h\"' is needed in some places.\n> \n> It's strange because I get no compile errors even after a \"make clean\"\n> from my branch.\n> Could you show the actual errors?\n\nI don't have the error messages with me now, but it was something about\na function being implicitly declared. You will probably get these errors\nif you sync past commit e67a57f (\"config: create config.h\", 2017-06-15).\n\n> > Any reason why you prefer to update the loose object functions than to\n> > update the generic one (sha1_object_info_extended)? My concern with just\n> > updating the loose object functions was that a caller might have\n> > obtained the path by iterating through the loose object dirs, and in\n> > that case we shouldn't query the external ODB for anything.\n> \n> You are thinking about fsck or gc?\n> Otherwise I don't think it would be clean to iterate through loose object dirs.\n\nYes, fsck and gc (well, prune, I think) do that. I agree that Git\ntypically doesn't do that (except for exceptional cases like fsck and\ngc), but I was thinking about supporting existing code that does that\niteration, not introducing new code that does that.\n"},{"id":"329446","messageId":"1d53569f-4bc8-0d8e-0182-eec0eb6634cd@gmail.com","threadId":"46764","inReplyTo":"20170916080731.13925-1-chriscool@tuxfamily.org","subject":"Re: [PATCH v6 00/40] Add initial experimental external ODB support","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-10-02T14:18:15Z","receivedAt":"2017-10-02T14:18:25Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 9/16/2017 4:06 AM, Christian Couder wrote:\n\n> Highlevel view of the patches in the series\n> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n> \n\nThis is a massive patch series and IMO keeping it a single monolithic \nset of patches makes it difficult to review and unwieldy to make \nprogress on as an \"all or nothing\" series.\n\nI highly recommend breaking it up into multiple smaller patch series \nthat can be reviewed and accepted individually. I have to admit I've \nskipped reviewing the last couple of iterations simply because of the \ntime investment required and difficulty separating out the various pieces.\n\nI think using your division of the patches below is a good place to \nstart. Many of these would be good changes even if they weren't part of \nthe larger external ODB effort.\n\n - Patch 1/40 is a small code cleanup that I already sent to the\n> mailing list but may be removed in the end due to ongoing work\n> on \"git clone\".\n> \n> - Patches 02/40 to 07/40 create a \"Git/Packet.pm\" module by\n> refactoring \"t0021/rot13-filter.pl\". Functions from this new\n> module will be used later in test scripts. According to Junio's\n> suggestion compared to v5 we now first fully refactor\n> \"t0021/rot13-filter.pl\" before creating the \"Git/Packet.pm\"\n> module.\n> \n\nThis seems like a very logical thing to do and should be split out so \nthat progress can be made independently.\n\n> - Patches 08/40 to 16/40 create the external ODB insfrastructure\n> in external-odb.{c,h} and odb-helper.{c,h} for the script mode.\n> The main changes compared to v5 are the following:\n> - we mark as \"extern\" functions in *.h files\n> \t- we use sha1_pos() instead of sha1_entry_pos()\n> \t- we check the size in the header when we 'get' a Git object\n> \n\nThis is the heart of the ODB infrastructure series. I'll respond in \nmore detail in the specific patches.\n\n> - Patches 17/40 to 23/40 improve lib-http to make it possible to\n> use it as an external ODB to test storing blobs in an HTTP\n> server. The \"upload.sh\" and \"list.sh\" files are now properly\n> indented and they use %% instead of % in parameter\n> substitutions compared to v5.\n> \n> - Patches 24/40 to 32/40 improve the external ODB insfrastructure\n> to support sub-processes and make everything work using\n> them. The main changes compared to v5 are the following:\n> - we mark as \"extern\" functions in *.h files\n> \t- we use the new subprocess_handshake() function\n> \t- we check the size in the header when we 'get' a Git object\n> \n> - Patch 33/40 uses attributes to mark blobs that should be handled\n> by an external odb.\n> \n> - Patch 34/40 adds documentation about the external odb\n> mechanism. This patch has been much improved since v5.\n> \n> - Patches 35/40 to 39/40 add the --initial-refspec to git clone\n> along with tests.\n> \n> - Patch 40/40 adds documentation about transfering objects and\n> metadata when using the external odb mechanism. This patch is\n> new since v5.\n> \n> Future work\n> ~~~~~~~~~~~\n> \n> There are still things that could be cleaned or improved. I think I\n> may work on:\n> \n> - Integrate changes in recent \"read-object-process\" work by Ben Peart.\n> \n> - Better test all the combinations of the different modes with and\n> without \"have\" and \"put_*\" instructions.\n> \n> - Maybe implement the missing kinds of 'put' ('put_git_obj' and\n> 'put_direct'), so that Git could pass either a git object a plain\n> object or ask the helper to retreive it directly from Git's object\n> database.\n> \n> - Add more long running tests and improve tests in general.\n> \n> Previous work and discussions\n> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n> \n> (Sorry for the old Gmane links, I hope I will try to replace them with\n> public-inbox.org at one point.)\n> \n> Peff started to work on this and discuss this some years ago:\n> \n> http://thread.gmane.org/gmane.comp.version-control.git/206886/focus=207040\n> http://thread.gmane.org/gmane.comp.version-control.git/247171\n> http://thread.gmane.org/gmane.comp.version-control.git/202902/focus=203020\n> \n> His work, which is not compile-tested any more, is still there:\n> \n> https://github.com/peff/git/commits/jk/external-odb-wip\n> \n> Initial discussions about this new series are there:\n> \n> http://thread.gmane.org/gmane.comp.version-control.git/288151/focus=295160\n> \n> Version 1, 2, 3, 4 and 5 of this series are here:\n> \n> https://public-inbox.org/git/20160613085546.11784-1-chriscool@tuxfamily.org/\n> https://public-inbox.org/git/20160628181933.24620-1-chriscool@tuxfamily.org/\n> https://public-inbox.org/git/20161130210420.15982-1-chriscool@tuxfamily.org/\n> https://public-inbox.org/git/20170620075523.26961-1-chriscool@tuxfamily.org/\n> https://public-inbox.org/git/20170803091926.1755-1-chriscool@tuxfamily.org/\n> \n> Some of the discussions related to Ben Peart's work that is used by\n> this series are here:\n> \n> https://public-inbox.org/git/20170113155253.1644-1-benpeart@microsoft.com/\n> https://public-inbox.org/git/20170322165220.5660-1-benpeart@microsoft.com/\n> https://public-inbox.org/git/20170714132651.170708-1-benpeart@microsoft.com/\n> \n> Links\n> ~~~~~\n> \n> This patch series is available here:\n> \n> https://github.com/chriscool/git/commits/external-odb\n> \n> Version 1, 2, 3, 4 and 5 are here:\n> \n> https://github.com/chriscool/git/commits/gl-external-odb12\n> https://github.com/chriscool/git/commits/gl-external-odb22\n> https://github.com/chriscool/git/commits/gl-external-odb61\n> https://github.com/chriscool/git/commits/gl-external-odb239\n> https://github.com/chriscool/git/commits/gl-external-odb373\n> \n> \n> Ben Peart (2):\n> odb-helper: add init_object_process()\n> Add t0450 to test 'get_direct' mechanism\n> \n> Christian Couder (38):\n> builtin/clone: get rid of 'value' strbuf\n> t0021/rot13-filter: refactor packet reading functions\n> t0021/rot13-filter: improve 'if .. elsif .. else' style\n> t0021/rot13-filter: improve error message\n> t0021/rot13-filter: add packet_initialize()\n> t0021/rot13-filter: add capability functions\n> Add Git/Packet.pm from parts of t0021/rot13-filter.pl\n> sha1_file: prepare for external odbs\n> Add initial external odb support\n> odb-helper: add odb_helper_init() to send 'init' instruction\n> t0400: add 'put_raw_obj' instruction to odb-helper script\n> external odb: add 'put_raw_obj' support\n> external-odb: accept only blobs for now\n> t0400: add test for external odb write support\n> Add GIT_NO_EXTERNAL_ODB env variable\n> Add t0410 to test external ODB transfer\n> lib-httpd: pass config file to start_httpd()\n> lib-httpd: add upload.sh\n> lib-httpd: add list.sh\n> lib-httpd: add apache-e-odb.conf\n> odb-helper: add odb_helper_get_raw_object()\n> pack-objects: don't pack objects in external odbs\n> Add t0420 to test transfer to HTTP external odb\n> external-odb: add 'get_direct' support\n> odb-helper: add 'script_mode' to 'struct odb_helper'\n> Add t0460 to test passing git objects\n> odb-helper: add put_object_process()\n> Add t0470 to test passing raw objects\n> odb-helper: add have_object_process()\n> Add t0480 to test \"have\" capability and raw objects\n> external-odb: use 'odb=magic' attribute to mark odb blobs\n> Add Documentation/technical/external-odb.txt\n> clone: add 'initial' param to write_remote_refs()\n> clone: add --initial-refspec option\n> clone: disable external odb before initial clone\n> Add tests for 'clone --initial-refspec'\n> Add t0430 to test cloning using bundles\n> Doc/external-odb: explain transfering objects and metadata\n> \n> Documentation/technical/external-odb.txt | 447 +++++++++++++\n> Makefile | 2 +\n> builtin/clone.c | 91 ++-\n> builtin/pack-objects.c | 4 +\n> cache.h | 18 +\n> environment.c | 4 +\n> external-odb.c | 196 ++++++\n> external-odb.h | 12 +\n> odb-helper.c | 1076 ++++++++++++++++++++++++++++++\n> odb-helper.h | 45 ++\n> perl/Git/Packet.pm | 118 ++++\n> sha1_file.c | 155 +++--\n> t/lib-httpd.sh | 8 +-\n> t/lib-httpd/apache-e-odb.conf | 214 ++++++\n> t/lib-httpd/list.sh | 41 ++\n> t/lib-httpd/upload.sh | 45 ++\n> t/t0021/rot13-filter.pl | 110 +--\n> t/t0400-external-odb.sh | 85 +++\n> t/t0410-transfer-e-odb.sh | 147 ++++\n> t/t0420-transfer-http-e-odb.sh | 152 +++++\n> t/t0430-clone-bundle-e-odb.sh | 85 +++\n> t/t0450-read-object.sh | 28 +\n> t/t0450/read-object | 68 ++\n> t/t0460-read-object-git.sh | 28 +\n> t/t0460/read-object-git | 78 +++\n> t/t0470-read-object-http-e-odb.sh | 119 ++++\n> t/t0470/read-object-plain | 83 +++\n> t/t0480-read-object-have-http-e-odb.sh | 119 ++++\n> t/t0480/read-object-plain-have | 103 +++\n> t/t5616-clone-initial-refspec.sh | 48 ++\n> 30 files changed, 3588 insertions(+), 141 deletions(-)\n> create mode 100644 Documentation/technical/external-odb.txt\n> create mode 100644 external-odb.c\n> create mode 100644 external-odb.h\n> create mode 100644 odb-helper.c\n> create mode 100644 odb-helper.h\n> create mode 100644 perl/Git/Packet.pm\n> create mode 100644 t/lib-httpd/apache-e-odb.conf\n> create mode 100644 t/lib-httpd/list.sh\n> create mode 100644 t/lib-httpd/upload.sh\n> create mode 100755 t/t0400-external-odb.sh\n> create mode 100755 t/t0410-transfer-e-odb.sh\n> create mode 100755 t/t0420-transfer-http-e-odb.sh\n> create mode 100755 t/t0430-clone-bundle-e-odb.sh\n> create mode 100755 t/t0450-read-object.sh\n> create mode 100755 t/t0450/read-object\n> create mode 100755 t/t0460-read-object-git.sh\n> create mode 100755 t/t0460/read-object-git\n> create mode 100755 t/t0470-read-object-http-e-odb.sh\n> create mode 100755 t/t0470/read-object-plain\n> create mode 100755 t/t0480-read-object-have-http-e-odb.sh\n> create mode 100755 t/t0480/read-object-plain-have\n> create mode 100755 t/t5616-clone-initial-refspec.sh\n> \n"},{"id":"329448","messageId":"903cb956-464e-fdab-fac1-00a2b48b942c@gmail.com","threadId":"46764","inReplyTo":"20170929133614.1c0cd68ad80139fafdf68b86@google.com","subject":"Re: [PATCH v6 09/40] Add initial external odb support","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-10-02T14:34:39Z","receivedAt":"2017-10-02T14:34:47Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 9/29/2017 4:36 PM, Jonathan Tan wrote:\n> On Wed, 27 Sep 2017 18:46:30 +0200\n> Christian Couder wrote:\n> \n>> I am ok to split the patch series, but I am not sure that 01/40 to\n>> 09/40 is the right range for the first patch series.\n>> I would say that 01/40 to 07/40 is better as it can be seen as a\n>> separate refactoring.\n> \n> I mentioned 09/40 because this is (as far as I can tell) the first one\n> that introduces a new design.\n> \n>> I don't think single-shot processes would be a huge burden, because\n>> the code is simpler, and because for example for filters we already\n>> have single shot and long-running processes and no one complains about\n>> that. It's code that is useful as it makes it much easier for people\n>> to do some things (see the clone bundle example).\n>>\n>> In fact in Git development we usually start to by first implementing\n>> simpler single-shot solutions, before thinking, when the need arise,\n>> to make it faster. So a perhaps an equally valid opinion could be to\n>> first only submit the patches for the single-shot protocol and later\n>> submit the rest of the series when we start getting feedback about how\n>> external odbs are used.\n> \n> My concern is that, as far as I understand about the Microsoft use case,\n> we already know that we need the faster solution, so the need has\n> already arisen.\n> \n>> And yeah I could change the order of the patch series to implement the\n>> long-running processes first and the single-shot process last, so that\n>> it could be possible to first get feedback about the long-running\n>> processes, before we decide to merge or not the single-shot stuff, but\n>> I don't think it would look like the most logical order.\n> \n> My thinking was that we would just implement the long-running process\n> and not implement the single-shot process at all (besides maybe a script\n> in contrib/). If we are going to do both anyway, I agree that we should\n> do the single-shot process first.\n> \n\nI agree with Jonathan's feedback. We already know the performance of \nsingle shot requests is insufficient as there are scenarios where there \nwill potentially be many missing objects that need to be retrieved to \ncomplete a git operation (ie checkout). As a results, we will need the \nlong running process model so, overall, it will be simpler to focus \nentirely on that model and skip the single-shot model.\n\nIf the complexity of the process model is considered to be too high, we \ncan provide helper code in both script and native code that can be used \nto reduce the cost/complexity. I believe we have most of this already \nwith the existing sub-process.c/h module and the packet.pm refactoring \nyou have done earlier in this series.\n\nProviding high quality working samples of both is another way to reduce \nthe cost and improve the quality.\n\n>>> And another possible issue is that we design ourselves into a corner.\n>>> Thinking about the use cases that I know about (the Android use case and\n>>> the Microsoft GVFS use case), I don't think we are doing that - for\n>>> Android, this means that large blob metadata needs to be part of the\n>>> design (and this patch series does provide for that), and for Microsoft\n>>> GVFS, \"get\" is relatively cheap, so a configuration option to not invoke\n>>> \"have\" first when loading a missing object might be sufficient.\n>>\n>> If the helper does not advertise the \"have\" capability, the \"have\"\n>> instruction will not be sent to the helper, so the current design is\n>> already working for that case.\n> \n> Ah, that's good to know.\n> \n>>> And I think that my design can be extended to support a use case in\n>>> which, for example, blobs corresponding to a certain type of filename\n>>> (defined by a glob like in gitattributes) can be excluded during\n>>> fetch/clone, much like --blob-max-bytes, and they can be fetched either\n>>> through the built-in mechanism or through a custom hook.\n>>\n>> Sure, we could probably rebuild something equivalent to what I did on\n>> top of your design.\n>> My opinion though is that if we want to eventually get to the same\n>> goal, it is better to first merge something that get us very close to\n>> the end goal and then add some improvements on top of it.\n> \n> I agree - I mentioned that because I personally prefer to review smaller\n> patch sets at a time, and my patch set already includes a lot of the\n> same infrastructure needed by yours - for example, the places in the\n> code to dynamically fetch objects, exclusion of objects when fetching or\n> cloning, configuring the cloned repo when cloning, fsck, and gc.\n> \n\nI agree here has well. I think smaller patch sets that we can \nreview/approve independently will be more effective.\n\nI think Jonathan has a lot of the infrastructure support in his partial \nclone series. I'd like to take that work, add your external ODB work + \nJeff's filtering work and come up with the best of all three solution. :)\n\n>>> - I get compile errors when I \"git am\" these onto master. I think\n>>> '#include \"config.h\"' is needed in some places.\n>>\n>> It's strange because I get no compile errors even after a \"make clean\"\n>> from my branch.\n>> Could you show the actual errors?\n> \n> I don't have the error messages with me now, but it was something about\n> a function being implicitly declared. You will probably get these errors\n> if you sync past commit e67a57f (\"config: create config.h\", 2017-06-15).\n> \n>>> Any reason why you prefer to update the loose object functions than to\n>>> update the generic one (sha1_object_info_extended)? My concern with just\n>>> updating the loose object functions was that a caller might have\n>>> obtained the path by iterating through the loose object dirs, and in\n>>> that case we shouldn't query the external ODB for anything.\n>>\n>> You are thinking about fsck or gc?\n>> Otherwise I don't think it would be clean to iterate through loose object dirs.\n> \n> Yes, fsck and gc (well, prune, I think) do that. I agree that Git\n> typically doesn't do that (except for exceptional cases like fsck and\n> gc), but I was thinking about supporting existing code that does that\n> iteration, not introducing new code that does that.\n> \n"},{"id":"329533","messageId":"CAP8UFD0tfei69b1BOCoAynWKtz8MJ3v2NF6iXc=N-x-C6YCiYA@mail.gmail.com","threadId":"46764","inReplyTo":"1d53569f-4bc8-0d8e-0182-eec0eb6634cd@gmail.com","subject":"Re: [PATCH v6 00/40] Add initial experimental external ODB support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-10-03T06:32:57Z","receivedAt":"2017-10-03T06:33:03Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Mon, Oct 2, 2017 at 4:18 PM, Ben Peart wrote:\n>\n> On 9/16/2017 4:06 AM, Christian Couder wrote:\n>\n> - Patch 1/40 is a small code cleanup that I already sent to the\n>>\n>> mailing list but may be removed in the end due to ongoing work\n>> on \"git clone\".\n>>\n>> - Patches 02/40 to 07/40 create a \"Git/Packet.pm\" module by\n>> refactoring \"t0021/rot13-filter.pl\". Functions from this new\n>> module will be used later in test scripts. According to Junio's\n>> suggestion compared to v5 we now first fully refactor\n>> \"t0021/rot13-filter.pl\" before creating the \"Git/Packet.pm\"\n>> module.\n>\n> This seems like a very logical thing to do and should be split out so that\n> progress can be made independently.\n\nOk, I will split according to my division of the patches.\n"},{"id":"329545","messageId":"CAP8UFD05QzCn53SjbbgaToVWyd=3N+HayDvR4OZNaP7YKbEBfg@mail.gmail.com","threadId":"46764","inReplyTo":"20170929133614.1c0cd68ad80139fafdf68b86@google.com","subject":"Re: [PATCH v6 09/40] Add initial external odb support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-10-03T09:45:02Z","receivedAt":"2017-10-03T09:45:10Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Fri, Sep 29, 2017 at 10:36 PM, Jonathan Tan wrote:\n> On Wed, 27 Sep 2017 18:46:30 +0200\n> Christian Couder wrote:\n\n>> I don't think single-shot processes would be a huge burden, because\n>> the code is simpler, and because for example for filters we already\n>> have single shot and long-running processes and no one complains about\n>> that. It's code that is useful as it makes it much easier for people\n>> to do some things (see the clone bundle example).\n>>\n>> In fact in Git development we usually start to by first implementing\n>> simpler single-shot solutions, before thinking, when the need arise,\n>> to make it faster. So a perhaps an equally valid opinion could be to\n>> first only submit the patches for the single-shot protocol and later\n>> submit the rest of the series when we start getting feedback about how\n>> external odbs are used.\n>\n> My concern is that, as far as I understand about the Microsoft use case,\n> we already know that we need the faster solution, so the need has\n> already arisen.\n\nYeah, some people need the faster solution, but my opinion is that\nmany other people would prefer the single shot protocol.\nIf all you want to do is a simple resumable clone using bundles for\nexample, then the long running process solution is very much overkill.\n\nFor example with filters there are people using them to do keyword\nexpansion (maybe to emulate the way Subversion and CVS substitutes\nkeywords like $Id$, $Author$ and so on). It would be really bad to\ndeprecate the single shot filters and tell those people they now have\nto use long running processes because we don't want to maintain the\nsmall code that make single shot filters work.\n\nThe Microsoft GVFS use case is just one use case that is very far from\nwhat most people need. And my opinion is that many more people could\nbenefit from the single shot protocol. For example many people and\nadmins could benefit from resumable clones using bundles and, if I\nremove the single shot protocol, this use case will be unnecessarily\nmore difficult to implement in the same way as keyword expansion would\nbe unnecessarily more difficult to implement if we removed the single\nshot filters.\n\nSee the first article in\nhttps://git.github.io/rev_news/2016/03/16/edition-13/ about resumable\nclone if you are not convinced that resumable clones are not an old\nand important problem.\n\n>> And yeah I could change the order of the patch series to implement the\n>> long-running processes first and the single-shot process last, so that\n>> it could be possible to first get feedback about the long-running\n>> processes, before we decide to merge or not the single-shot stuff, but\n>> I don't think it would look like the most logical order.\n>\n> My thinking was that we would just implement the long-running process\n> and not implement the single-shot process at all (besides maybe a script\n> in contrib/). If we are going to do both anyway, I agree that we should\n> do the single-shot process first.\n\nNice to hear that!\n\n>> > And I think that my design can be extended to support a use case in\n>> > which, for example, blobs corresponding to a certain type of filename\n>> > (defined by a glob like in gitattributes) can be excluded during\n>> > fetch/clone, much like --blob-max-bytes, and they can be fetched either\n>> > through the built-in mechanism or through a custom hook.\n>>\n>> Sure, we could probably rebuild something equivalent to what I did on\n>> top of your design.\n>> My opinion though is that if we want to eventually get to the same\n>> goal, it is better to first merge something that get us very close to\n>> the end goal and then add some improvements on top of it.\n>\n> I agree\n\nSo are you ok to rebase your patch series on top of my patch series?\n\nMy opinion is that my patch series is trying to get to the end goal,\nand succeeding to a very large extent, with the smallest amount of\ndeep technical changes as possible, and that it is the right way to\napproach this problem for the following reasons:\n\n1) The root problem is that the current object stores (packfiles and\nloose object files) are not good ways to store some objects,\nespecially some blobs.\n\n2) This root problem cannot be dealt with by Git itself without any\nhelp from external programs, because Git cannot realistically\nimplement many different object stores (like http servers, artifact\nstores, etc). So Git must be improved so that it becomes capable of\ncommunicating with external object stores.\n\n3) As the Git protocol uses packfiles to send objects and is not very\nflexible, it might be better if external stores can also be used to\ntransfer objects that are not stored any more in the current object\nstores. (As packfiles are not good for storing some objects, they are\nprobably also not a good format for sending them. Also, as the Git\nprotocol is not resumable, we might easily be able to implement\nresumable clones if we let external stores handle some transfer.)\n\n4) Making it easy and flexible to exchange objects (and maybe meta\ninformation) with the external stores is very important.\n\n5) Protocol changes are more difficult than many other code changes,\nso we should care a lot about the protocol between Git and external\nstores.\n\n> - I mentioned that because I personally prefer to review smaller\n> patch sets at a time,\n\nI am ok with sending small patch sets, and I will send smaller patch\nsets about this from now on.\n\n> and my patch set already includes a lot of the\n> same infrastructure needed by yours - for example, the places in the\n> code to dynamically fetch objects, exclusion of objects when fetching or\n> cloning, configuring the cloned repo when cloning, fsck, and gc.\n\nI agree that your patch set already includes some infrastructure that\ncould be used by my work, and your patch sets are perhaps implementing\nsome of this infrastructure better than in my work (I haven't taken a\ndeep look). But I really think that the right approach is to focus\nfirst on designing a flexible protocol between Git and external\nstores. Then the infrastructure work should be related to improving or\nenabling the flexible protocol and the communication between Git and\nexternal stores.\n\nDoing infrastructure work first and improving things on top of this\nnew infrastructure without relying first on a design of the protocol\nbetween Git and external stores is not the best approach as I think we\nmight over engineer some infrastructure work or base some user\ninterfaces on the infrastructure work and not on the end goal.\n\nFor example if we improve the current protocol, which is not\nnecessarily a bad thing in itself, we might forget that for resumable\nclone it is much better if we just let external stores and helpers\nhandle the transfer.\n\nI am not saying that doing infrastructure work is bad or will not in\nthe end let us reach our goals, but I see it as something that is\npotentially distracting, or misleading, from focusing first on the\nprotocol between Git and external stores.\n\n>> > - I get compile errors when I \"git am\" these onto master. I think\n>> > '#include \"config.h\"' is needed in some places.\n>>\n>> It's strange because I get no compile errors even after a \"make clean\"\n>> from my branch.\n>> Could you show the actual errors?\n>\n> I don't have the error messages with me now, but it was something about\n> a function being implicitly declared. You will probably get these errors\n> if you sync past commit e67a57f (\"config: create config.h\", 2017-06-15).\n\nI am past this commit and I get no errors.\nI rebased on top of: ea220ee40c \"The eleventh batch for 2.15\"\n\n>> > Any reason why you prefer to update the loose object functions than to\n>> > update the generic one (sha1_object_info_extended)? My concern with just\n>> > updating the loose object functions was that a caller might have\n>> > obtained the path by iterating through the loose object dirs, and in\n>> > that case we shouldn't query the external ODB for anything.\n>>\n>> You are thinking about fsck or gc?\n>> Otherwise I don't think it would be clean to iterate through loose object dirs.\n>\n> Yes, fsck and gc (well, prune, I think) do that. I agree that Git\n> typically doesn't do that (except for exceptional cases like fsck and\n> gc), but I was thinking about supporting existing code that does that\n> iteration, not introducing new code that does that.\n\nI haven't taken a look at how fsck and prune work and this is still\ncode that Peff wrote (though a long time ago), so I tend to trust it.\nBut I will take a look, and if it is indeed better for them, I am ok\nto update sha1_object_info_extended() instead of loose object\nfunctions.\n\nThanks.\n"},{"id":"329627","messageId":"CAGf8dgLD3JR4_oqEM09D6j+izrnXoj8YPr7JfBwKrx2qj=Vkkw@mail.gmail.com","threadId":"46764","inReplyTo":"CAP8UFD05QzCn53SjbbgaToVWyd=3N+HayDvR4OZNaP7YKbEBfg@mail.gmail.com","subject":"Re: [PATCH v6 09/40] Add initial external odb support","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-10-04T00:15:49Z","receivedAt":"2017-10-04T00:15:55Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, Oct 3, 2017 at 2:45 AM, Christian Couder\n wrote:\n> Yeah, some people need the faster solution, but my opinion is that\n> many other people would prefer the single shot protocol.\n> If all you want to do is a simple resumable clone using bundles for\n> example, then the long running process solution is very much overkill.\n>\n> For example with filters there are people using them to do keyword\n> expansion (maybe to emulate the way Subversion and CVS substitutes\n> keywords like $Id$, $Author$ and so on). It would be really bad to\n> deprecate the single shot filters and tell those people they now have\n> to use long running processes because we don't want to maintain the\n> small code that make single shot filters work.\n>\n> The Microsoft GVFS use case is just one use case that is very far from\n> what most people need. And my opinion is that many more people could\n> benefit from the single shot protocol. For example many people and\n> admins could benefit from resumable clones using bundles and, if I\n> remove the single shot protocol, this use case will be unnecessarily\n> more difficult to implement in the same way as keyword expansion would\n> be unnecessarily more difficult to implement if we removed the single\n> shot filters.\n\nThe idea that some users will prefer writing to the single-shot\nprotocol is reasonable to me, but I think that providing a contrib/\nPerl script that wraps something that speaks the single-shot protocol\nis sufficient. This results in less C code, and a better separation of\nconcerns (I prefer 1 exit point and 1 adapter over 2 exit points).\n\n> I agree that your patch set already includes some infrastructure that\n> could be used by my work, and your patch sets are perhaps implementing\n> some of this infrastructure better than in my work (I haven't taken a\n> deep look). But I really think that the right approach is to focus\n> first on designing a flexible protocol between Git and external\n> stores. Then the infrastructure work should be related to improving or\n> enabling the flexible protocol and the communication between Git and\n> external stores.\n>\n> Doing infrastructure work first and improving things on top of this\n> new infrastructure without relying first on a design of the protocol\n> between Git and external stores is not the best approach as I think we\n> might over engineer some infrastructure work or base some user\n> interfaces on the infrastructure work and not on the end goal.\n>\n> For example if we improve the current protocol, which is not\n> necessarily a bad thing in itself, we might forget that for resumable\n> clone it is much better if we just let external stores and helpers\n> handle the transfer.\n>\n> I am not saying that doing infrastructure work is bad or will not in\n> the end let us reach our goals, but I see it as something that is\n> potentially distracting, or misleading, from focusing first on the\n> protocol between Git and external stores.\n\nI think that the infrastructure really needs to be considered when\ndesigning the protocol. In particular, we had to consider the needs of\nthe connectivity check in fsck and the repacking in GC when designing\nwhat the promisor remote (or ODB, in this case) needs to tell us and\nwhat, if any, postprocessing needs to be done. In the end, I settled\non tracking which objects came from the promisor remote and which did\nnot, which works in my design (which I have tried to ensure that it\nfits in our and Microsoft's use case). But that design won't work in\nwhat I understand to be the ODB case, at least from what I understand,\nbecause (at least) (i) you can have multiple ODBs, and (ii) Git does\nnot have direct access to the objects stored within the ODBs. So some\nmore design needs to be done.\n"}]}