{"thread":{"id":"46506","subject":"[PATCH v5 00/40] Add initial experimental external ODB support","startedAt":"2017-08-03T09:19:51Z","lastAt":"2017-09-15T20:55:00Z","messageCount":73,"participants":["Christian Couder","Stefan Beller","Junio C Hamano","Jeff King","Jonathan Tan","Ben Peart","Lars Schneider"],"isPatch":true,"patchVersion":5,"patchTotal":40},"messages":[{"id":"325504","messageId":"20170803091926.1755-1-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":null,"subject":"[PATCH v5 00/40] Add initial experimental external ODB support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:18:46Z","receivedAt":"2017-08-03T09:19:51Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Note: a lot of information about the goals, the design and how things\nwork is now in patch 35/40 \"Add\nDocumentation/technical/external-odb.txt\" of this v5 series.\n\nGoal\n~~~~\n\nGit can store its objects only in the form of loose objects in\nseparate files or packed objects in a pack file.\n\nTo be able to better handle some kind of objects, for example big\nblobs, it would be nice if Git could store its objects in other object\ndatabases (ODB).\n\nTo do that, this patch series makes it possible to register commands,\nalso called \"helpers\", using \"odb..scriptCommand\" or\n\"odb..subprocessCommand\" config variables, to access external\nODBs where objects can be stored and retrieved.\n\nDesign\n~~~~~~\n\n* The \"helpers\" (registered commands)\n\nEach helper manages access to one external ODB.\n\nThere are 2 different modes for helper:\n\n - Helpers configured using \"odb..scriptCommand\" are\n launched each time Git wants to communicate with the \n external ODB. This is called \"script mode\".\n\n - Helpers configured using \"odb..subprocessCommand\" are\n launched launched once as a sub-process (using sub-process.h), and\n Git communicates with them using packet lines. This is called\n \"process mode\".\n\nA helper can be given different instructions by Git. The instructions\nthat are supported are negociated at the beginning of the\ncommunication using a capability mechanism.\n\nSee patch 35/40 (the documentation patch) for more information about\nthe different instructions and their arguments.\n\n* Performance\n\nThe process mode has been implemented using the refactoring that Ben\nPeart did on top of Lars Schneider's work on using sub-processes and\npacket lines in the smudge/clean filters for git-lfs.\n\nThis also uses further work from Ben Peart called \"read object\nprocess\".\n\nSee:\n\nhttps://public-inbox.org/git/20170113155253.1644-1-benpeart@microsoft.com/\nhttps://public-inbox.org/git/20170322165220.5660-1-benpeart@microsoft.com/\n\nBen recently sent an update of this work but this update has not been\nintegrated into the current patch series. See:\n\nhttps://public-inbox.org/git/20170714132651.170708-1-benpeart@microsoft.com/\n\nAnyway thanks to this, the external ODB mechanism should in the end perform\nas well as the git-lfs mechanism when many objects should be\ntransfered.\n\nImplementation\n~~~~~~~~~~~~~~\n\n* Mechanism to call the registered commands\n\nA set of function in external-odb.{c,h} are called by the rest of Git\nto manage all the external ODBs.\n\nThese functions use 'struct odb_helper' and its associated functions\ndefined in odb-helper.{c,h} to talk to the different external ODBs by\nlaunching the configured \"odb..*command\" commands and writing\nto or reading from them.\n\n* Transfering information\n\nTo tranfer information about the blobs stored in external ODB, some\nspecial refs, called \"odb ref\", similar as replace refs, are used in\nthe tests of this series, but in general nothing forces the helper to\nuse that mechanism.\n\nThe external odb helper is responsible for using and creating the refs\nin refs/odbs//, if it wants to do that. It is free for example\nto just create one ref, as it is also free to create many refs. Git\nwould just transmit the refs that have been created by this helper, if\nGit is asked to do so.\n\nFor now in the tests there is one odb ref per blob, as it is simple\nand as it is similar to what git-lfs does. Each ref name is\nrefs/odbs// where is the sha1 of the blob stored\nin the external odb named .\n\nThese odb refs point to a blob that is stored in the Git\nrepository and contain information about the blob stored in the\nexternal odb. This information can be specific to the external odb.\nThe repos can then share this information using commands like:\n\n`git fetch origin \"refs/odbs//*:refs/odbs//*\"`\n\nAt the end of the current patch series, \"git clone\" is teached a\n\"--initial-refspec\" option, that asks it to first fetch some specified\nrefs. This is used in the tests to fetch the odb refs first.\n\nThis way only one \"git clone\" command can setup a repo using the\nexternal ODB mechanism as long as the right helper is installed on the\nmachine and as long as the following options are used:\n\n - \"--initial-refspec \" to fetch the odb refspec\n - \"-c odb..command=\" to configure the helper\n\nThere is also a test script that shows that the \"--initial-refspec\"\noption along with the external ODB mechanism can be used to implement\ncloning using bundles.\n\n* ODB refs\n\nFor now odb ref management is only implemented in a helper in t0410.\n\nWhen a new blob is added to an external odb, its sha1, size and type\nare writen in another new blob and the odb ref is created.\n\nWhen the list of existing blobs is requested from the external odb,\nthe content of the blobs pointed to by the odb refs can also be used\nby the odb to claim that it can get the objects.\n\nWhen a blob is actually requested from the external odb, it can use\nthe content stored in the blobs pointed to by the odb refs to get the\nactual blobs and then pass them.\n\nHighlevel view of the patches in the series\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n - Patch 1/40 is a small code cleanup that I already sent to the\n mailing list but will probably be removed in the end due to\n ongoing work on \"git clone\".\n\n - Patches 02/40 to 08/40 create a Git/Packet.pm module by\n refactoring \"t0021/rot13-filter.pl\". Functions from this new\n module will be used later in test scripts.\n\n - Patches 09/40 to 17/40 create the external ODB insfrastructure\n in external-odb.{c,h} and odb-helper.{c,h} for the script mode.\n\n - Patches 18/40 to 24/40 improve lib-http to make it possible to\n use it as an external ODB to test storing blobs in an HTTP\n server.\n\n - Patches 25/40 to 33/40 improve the external ODB insfrastructure\n to support sub-processes and make everything work using them.\n\n - Patch 34/40 uses attributes to mark blobs that should be handled\n by an external odb.\n\n - Patch 35/40 adds documentation about the external odb mechanism.\n\n - Patches 36/40 to 40/40 add the --initial-refspec to git clone\n along with tests.\n\nBig changes since the previous patch series\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n - The \"odb..scriptMode\" and \"odb..command\" options\n have been replaced with \"odb..scriptCommand\" and\n \"odb..subprocessCommand\".\n\n - Capabilities are used to decide which kind of \"get\" will be\n used. So there is now 'get_raw_obj', 'get_git_obj' and\n 'get_direct' instead of just 'get' and \"odb..fetchKind\".\n\n - An \"init\" instruction has been added and is the only required\n instruction for any helper to implement. It replaces the \"get_cap\"\n instruction that was only available in script node, and it helps\n the process mode too, as it makes it possible for Git to know the\n capabilities before trying to send any instruction (that might not\n be supported by the helper).\n\n - An attributes based mechanism has been added to mark files that\n should be handled by an external odb. See patch 34/40\n \"external-odb: use 'odb=magic' attribute to mark odb blobs\"\n\n - A lot of functions, structures and variables have been\n renamed. The \"read-object-process\" mechanism and related names\n that came from Ben's work have been renamed or prefixed using just\n \"object-process\" or just \"process\" as this is not about reading\n only and the instructions are called 'get_*' not 'read_*'. Names\n related to script mode have been renamed or prefixed using just\n \"object-script\" or just \"script\".\n\n - Documentation and commit messages are much improved.\n\nFuture work\n~~~~~~~~~~~\n\nThere are still many things that could be cleaned or improved, but I\nthink that now the series is in a good enough state to not be RFC\nanymore.\n\nThings I think I may work on:\n\n - Integrate recent \"read-object-process\" work by Ben Peart.\n\n - Look at possible short-read and better checking objects Git\n receives.\n\n - Better test all the combinations of the different modes with and\n without \"have\" and \"put_*\" instructions.\n\n - Maybe implement the missing kinds of 'put' ('put_git_obj' and\n 'put_direct'), so that Git could pass either a git object a plain\n object or ask the helper to retreive it directly from Git's object\n database.\n\n - Add more long running tests and improve tests in general.\n\nPrevious work and discussions\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n(Sorry for the old Gmane links, I hope I will try to replace them with\npublic-inbox.org at one point.)\n\nPeff started to work on this and discuss this some years ago:\n\nhttp://thread.gmane.org/gmane.comp.version-control.git/206886/focus=207040\nhttp://thread.gmane.org/gmane.comp.version-control.git/247171\nhttp://thread.gmane.org/gmane.comp.version-control.git/202902/focus=203020\n\nHis work, which is not compile-tested any more, is still there:\n\nhttps://github.com/peff/git/commits/jk/external-odb-wip\n\nInitial discussions about this new series are there:\n\nhttp://thread.gmane.org/gmane.comp.version-control.git/288151/focus=295160\n\nVersion 1, 2, 3 and 4 of this series are here:\n\nhttps://public-inbox.org/git/20160613085546.11784-1-chriscool@tuxfamily.org/\nhttps://public-inbox.org/git/20160628181933.24620-1-chriscool@tuxfamily.org/\nhttps://public-inbox.org/git/20161130210420.15982-1-chriscool@tuxfamily.org/\nhttps://public-inbox.org/git/20170620075523.26961-1-chriscool@tuxfamily.org/\n\nSome of the discussions related to Ben Peart's work that is used by\nthis series are here:\n\nhttps://public-inbox.org/git/20170113155253.1644-1-benpeart@microsoft.com/\nhttps://public-inbox.org/git/20170322165220.5660-1-benpeart@microsoft.com/\nhttps://public-inbox.org/git/20170714132651.170708-1-benpeart@microsoft.com/\n\nLinks\n~~~~~\n\nThis patch series is available here:\n\nhttps://github.com/chriscool/git/commits/external-odb\n\nVersion 1, 2, 3 and 4 are here:\n\nhttps://github.com/chriscool/git/commits/gl-external-odb12\nhttps://github.com/chriscool/git/commits/gl-external-odb22\nhttps://github.com/chriscool/git/commits/gl-external-odb61\nhttps://github.com/chriscool/git/commits/gl-external-odb239\n\nBen Peart (2):\n odb-helper: add init_object_process()\n Add t0450 to test 'get_direct' mechanism\n\nChristian Couder (38):\n builtin/clone: get rid of 'value' strbuf\n t0021/rot13-filter: refactor packet reading functions\n t0021/rot13-filter: improve 'if .. elsif .. else' style\n Add Git/Packet.pm from parts of t0021/rot13-filter.pl\n t0021/rot13-filter: use Git/Packet.pm\n Git/Packet.pm: improve error message\n Git/Packet.pm: add packet_initialize()\n Git/Packet.pm: add capability functions\n sha1_file: prepare for external odbs\n Add initial external odb support\n odb-helper: add odb_helper_init() to send 'init' instruction\n t0400: add 'put_raw_obj' instruction to odb-helper script\n external odb: add 'put_raw_obj' support\n external-odb: accept only blobs for now\n t0400: add test for external odb write support\n Add GIT_NO_EXTERNAL_ODB env variable\n Add t0410 to test external ODB transfer\n lib-httpd: pass config file to start_httpd()\n lib-httpd: add upload.sh\n lib-httpd: add list.sh\n lib-httpd: add apache-e-odb.conf\n odb-helper: add odb_helper_get_raw_object()\n pack-objects: don't pack objects in external odbs\n Add t0420 to test transfer to HTTP external odb\n external-odb: add 'get_direct' support\n odb-helper: add 'script_mode' to 'struct odb_helper'\n Add t0460 to test passing git objects\n odb-helper: add put_object_process()\n Add t0470 to test passing raw objects\n odb-helper: add have_object_process()\n Add t0480 to test \"have\" capability and raw objects\n external-odb: use 'odb=magic' attribute to mark odb blobs\n Add Documentation/technical/external-odb.txt\n clone: add 'initial' param to write_remote_refs()\n clone: add --initial-refspec option\n clone: disable external odb before initial clone\n Add tests for 'clone --initial-refspec'\n Add t0430 to test cloning using bundles\n\n Documentation/technical/external-odb.txt | 295 +++++++++\n Makefile | 2 +\n builtin/clone.c | 91 ++-\n builtin/pack-objects.c | 4 +\n cache.h | 18 +\n environment.c | 4 +\n external-odb.c | 196 ++++++\n external-odb.h | 12 +\n odb-helper.c | 1067 ++++++++++++++++++++++++++++++\n odb-helper.h | 42 ++\n perl/Git/Packet.pm | 118 ++++\n sha1_file.c | 161 +++--\n t/lib-httpd.sh | 8 +-\n t/lib-httpd/apache-e-odb.conf | 214 ++++++\n t/lib-httpd/list.sh | 41 ++\n t/lib-httpd/upload.sh | 45 ++\n t/t0021/rot13-filter.pl | 97 +--\n t/t0400-external-odb.sh | 85 +++\n t/t0410-transfer-e-odb.sh | 147 ++++\n t/t0420-transfer-http-e-odb.sh | 152 +++++\n t/t0430-clone-bundle-e-odb.sh | 85 +++\n t/t0450-read-object.sh | 28 +\n t/t0450/read-object | 68 ++\n t/t0460-read-object-git.sh | 28 +\n t/t0460/read-object-git | 78 +++\n t/t0470-read-object-http-e-odb.sh | 119 ++++\n t/t0470/read-object-plain | 83 +++\n t/t0480-read-object-have-http-e-odb.sh | 119 ++++\n t/t0480/read-object-plain-have | 103 +++\n t/t5616-clone-initial-refspec.sh | 48 ++\n 30 files changed, 3423 insertions(+), 135 deletions(-)\n create mode 100644 Documentation/technical/external-odb.txt\n create mode 100644 external-odb.c\n create mode 100644 external-odb.h\n create mode 100644 odb-helper.c\n create mode 100644 odb-helper.h\n create mode 100644 perl/Git/Packet.pm\n create mode 100644 t/lib-httpd/apache-e-odb.conf\n create mode 100644 t/lib-httpd/list.sh\n create mode 100644 t/lib-httpd/upload.sh\n create mode 100755 t/t0400-external-odb.sh\n create mode 100755 t/t0410-transfer-e-odb.sh\n create mode 100755 t/t0420-transfer-http-e-odb.sh\n create mode 100755 t/t0430-clone-bundle-e-odb.sh\n create mode 100755 t/t0450-read-object.sh\n create mode 100755 t/t0450/read-object\n create mode 100755 t/t0460-read-object-git.sh\n create mode 100755 t/t0460/read-object-git\n create mode 100755 t/t0470-read-object-http-e-odb.sh\n create mode 100755 t/t0470/read-object-plain\n create mode 100755 t/t0480-read-object-have-http-e-odb.sh\n create mode 100755 t/t0480/read-object-plain-have\n create mode 100755 t/t5616-clone-initial-refspec.sh\n\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325505","messageId":"20170803091926.1755-4-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 03/40] t0021/rot13-filter: improve 'if .. elsif .. else' style","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:18:49Z","receivedAt":"2017-08-03T09:20:00Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Before further refactoring the \"t0021/rot13-filter.pl\" script,\nlet's modernize the style of its 'if .. elsif .. else' clauses\nto improve its readability by making it more similar to our\nother perl scripts.\n\nSigned-off-by: Christian Couder \n---\n t/t0021/rot13-filter.pl | 27 +++++++++------------------\n 1 file changed, 9 insertions(+), 18 deletions(-)\n\ndiff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\nindex d6411ca523..1fc581c814 100644\n--- a/t/t0021/rot13-filter.pl\n+++ b/t/t0021/rot13-filter.pl\n@@ -40,23 +40,20 @@ sub packet_bin_read {\n \tif ( $bytes_read == 0 ) {\n \t\t# EOF - Git stopped talking to us!\n \t\treturn ( -1, \"\" );\n-\t}\n-\telsif ( $bytes_read != 4 ) {\n+\t} elsif ( $bytes_read != 4 ) {\n \t\tdie \"invalid packet: '$buffer'\";\n \t}\n \tmy $pkt_size = hex($buffer);\n \tif ( $pkt_size == 0 ) {\n \t\treturn ( 1, \"\" );\n-\t}\n-\telsif ( $pkt_size > 4 ) {\n+\t} elsif ( $pkt_size > 4 ) {\n \t\tmy $content_size = $pkt_size - 4;\n \t\t$bytes_read = read STDIN, $buffer, $content_size;\n \t\tif ( $bytes_read != $content_size ) {\n \t\t\tdie \"invalid packet ($content_size bytes expected; $bytes_read bytes read)\";\n \t\t}\n \t\treturn ( 0, $buffer );\n-\t}\n-\telse {\n+\t} else {\n \t\tdie \"invalid packet size: $pkt_size\";\n \t}\n }\n@@ -144,14 +141,11 @@ while (1) {\n \tmy $output;\n \tif ( $pathname eq \"error.r\" or $pathname eq \"abort.r\" ) {\n \t\t$output = \"\";\n-\t}\n-\telsif ( $command eq \"clean\" and grep( /^clean$/, @capabilities ) ) {\n+\t} elsif ( $command eq \"clean\" and grep( /^clean$/, @capabilities ) ) {\n \t\t$output = rot13($input);\n-\t}\n-\telsif ( $command eq \"smudge\" and grep( /^smudge$/, @capabilities ) ) {\n+\t} elsif ( $command eq \"smudge\" and grep( /^smudge$/, @capabilities ) ) {\n \t\t$output = rot13($input);\n-\t}\n-\telse {\n+\t} else {\n \t\tdie \"bad command '$command'\";\n \t}\n \n@@ -163,14 +157,12 @@ while (1) {\n \t\t$debug->flush();\n \t\tpacket_txt_write(\"status=error\");\n \t\tpacket_flush();\n-\t}\n-\telsif ( $pathname eq \"abort.r\" ) {\n+\t} elsif ( $pathname eq \"abort.r\" ) {\n \t\tprint $debug \"[ABORT]\\n\";\n \t\t$debug->flush();\n \t\tpacket_txt_write(\"status=abort\");\n \t\tpacket_flush();\n-\t}\n-\telse {\n+\t} else {\n \t\tpacket_txt_write(\"status=success\");\n \t\tpacket_flush();\n \n@@ -187,8 +179,7 @@ while (1) {\n \t\t\tprint $debug \".\";\n \t\t\tif ( length($output) > $MAX_PACKET_CONTENT_SIZE ) {\n \t\t\t\t$output = substr( $output, $MAX_PACKET_CONTENT_SIZE );\n-\t\t\t}\n-\t\t\telse {\n+\t\t\t} else {\n \t\t\t\t$output = \"\";\n \t\t\t}\n \t\t}\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325506","messageId":"20170803091926.1755-5-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 04/40] Add Git/Packet.pm from parts of t0021/rot13-filter.pl","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:18:50Z","receivedAt":"2017-08-03T09:20:04Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This will make it possible to reuse packet reading and writing\nfunctions in other test scripts.\n\nSigned-off-by: Christian Couder \n---\n perl/Git/Packet.pm | 71 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 71 insertions(+)\n create mode 100644 perl/Git/Packet.pm\n\ndiff --git a/perl/Git/Packet.pm b/perl/Git/Packet.pm\nnew file mode 100644\nindex 0000000000..aaffecbe2a\n--- /dev/null\n+++ b/perl/Git/Packet.pm\n@@ -0,0 +1,71 @@\n+package Git::Packet;\n+use 5.008;\n+use strict;\n+use warnings;\n+BEGIN {\n+\trequire Exporter;\n+\tif ($] < 5.008003) {\n+\t\t*import = \\&Exporter::import;\n+\t} else {\n+\t\t# Exporter 5.57 which supports this invocation was\n+\t\t# released with perl 5.8.3\n+\t\tExporter->import('import');\n+\t}\n+}\n+\n+our @EXPORT = qw(\n+\t\t\tpacket_bin_read\n+\t\t\tpacket_txt_read\n+\t\t\tpacket_bin_write\n+\t\t\tpacket_txt_write\n+\t\t\tpacket_flush\n+\t\t);\n+our @EXPORT_OK = @EXPORT;\n+\n+sub packet_bin_read {\n+\tmy $buffer;\n+\tmy $bytes_read = read STDIN, $buffer, 4;\n+\tif ( $bytes_read == 0 ) {\n+\t\t# EOF - Git stopped talking to us!\n+\t\treturn ( -1, \"\" );\n+\t} elsif ( $bytes_read != 4 ) {\n+\t\tdie \"invalid packet: '$buffer'\";\n+\t}\n+\tmy $pkt_size = hex($buffer);\n+\tif ( $pkt_size == 0 ) {\n+\t\treturn ( 1, \"\" );\n+\t} elsif ( $pkt_size > 4 ) {\n+\t\tmy $content_size = $pkt_size - 4;\n+\t\t$bytes_read = read STDIN, $buffer, $content_size;\n+\t\tif ( $bytes_read != $content_size ) {\n+\t\t\tdie \"invalid packet ($content_size bytes expected; $bytes_read bytes read)\";\n+\t\t}\n+\t\treturn ( 0, $buffer );\n+\t} else {\n+\t\tdie \"invalid packet size: $pkt_size\";\n+\t}\n+}\n+\n+sub packet_txt_read {\n+\tmy ( $res, $buf ) = packet_bin_read();\n+\tunless ( $res == -1 || $buf =~ s/\\n$// ) {\n+\t\tdie \"A non-binary line MUST be terminated by an LF.\";\n+\t}\n+\treturn ( $res, $buf );\n+}\n+\n+sub packet_bin_write {\n+\tmy $buf = shift;\n+\tprint STDOUT sprintf( \"%04x\", length($buf) + 4 );\n+\tprint STDOUT $buf;\n+\tSTDOUT->flush();\n+}\n+\n+sub packet_txt_write {\n+\tpacket_bin_write( $_[0] . \"\\n\" );\n+}\n+\n+sub packet_flush {\n+\tprint STDOUT sprintf( \"%04x\", 0 );\n+\tSTDOUT->flush();\n+}\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325507","messageId":"20170803091926.1755-8-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 07/40] Git/Packet.pm: add packet_initialize()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:18:53Z","receivedAt":"2017-08-03T09:20:08Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Add a function to initialize the communication. And use this\nfunction in 't/t0021/rot13-filter.pl'.\n\nSigned-off-by: Christian Couder \n---\n perl/Git/Packet.pm | 13 +++++++++++++\n t/t0021/rot13-filter.pl | 8 +-------\n 2 files changed, 14 insertions(+), 7 deletions(-)\n\ndiff --git a/perl/Git/Packet.pm b/perl/Git/Packet.pm\nindex 2ad6b00d6c..b0233caf37 100644\n--- a/perl/Git/Packet.pm\n+++ b/perl/Git/Packet.pm\n@@ -19,6 +19,7 @@ our @EXPORT = qw(\n \t\t\tpacket_bin_write\n \t\t\tpacket_txt_write\n \t\t\tpacket_flush\n+\t\t\tpacket_initialize\n \t\t);\n our @EXPORT_OK = @EXPORT;\n \n@@ -70,3 +71,15 @@ sub packet_flush {\n \tprint STDOUT sprintf( \"%04x\", 0 );\n \tSTDOUT->flush();\n }\n+\n+sub packet_initialize {\n+\tmy ($name, $version) = @_;\n+\n+\t( packet_txt_read() eq ( 0, $name . \"-client\" ) )\t|| die \"bad initialize\";\n+\t( packet_txt_read() eq ( 0, \"version=\" . $version ) )\t|| die \"bad version\";\n+\t( packet_bin_read() eq ( 1, \"\" ) )\t\t\t|| die \"bad version end\";\n+\n+\tpacket_txt_write( $name . \"-server\" );\n+\tpacket_txt_write( \"version=\" . $version );\n+\tpacket_flush();\n+}\ndiff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\nindex 36a9eb3608..5b05518640 100644\n--- a/t/t0021/rot13-filter.pl\n+++ b/t/t0021/rot13-filter.pl\n@@ -40,13 +40,7 @@ sub rot13 {\n print $debug \"START\\n\";\n $debug->flush();\n \n-( packet_txt_read() eq ( 0, \"git-filter-client\" ) ) || die \"bad initialize\";\n-( packet_txt_read() eq ( 0, \"version=2\" ) ) || die \"bad version\";\n-( packet_bin_read() eq ( 1, \"\" ) ) || die \"bad version end\";\n-\n-packet_txt_write(\"git-filter-server\");\n-packet_txt_write(\"version=2\");\n-packet_flush();\n+packet_initialize(\"git-filter\", 2);\n \n ( packet_txt_read() eq ( 0, \"capability=clean\" ) ) || die \"bad capability\";\n ( packet_txt_read() eq ( 0, \"capability=smudge\" ) ) || die \"bad capability\";\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325508","messageId":"20170803091926.1755-10-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 09/40] sha1_file: prepare for external odbs","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:18:55Z","receivedAt":"2017-08-03T09:20:12Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"In the following commits we will need some functions that were\ninternal to sha1_file.c, so let's first make them non static\nand declare them in \"cache.h\". While at it, let's rename\n'create_tmpfile()' to 'create_object_tmpfile()' to make its\nname less generic.\n\nLet's also split out 'sha1_file_name_alt()' from\n'sha1_file_name()' and 'open_sha1_file_alt()' from\n'open_sha1_file()', as we will need both of these new\nfunctions too.\n\nHelped-by: Jeff King \nSigned-off-by: Christian Couder \n---\n cache.h | 8 ++++++++\n sha1_file.c | 57 ++++++++++++++++++++++++++++++++++++---------------------\n 2 files changed, 44 insertions(+), 21 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 71fe092644..06da3d8a3f 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -902,6 +902,12 @@ extern void check_repository_format(void);\n */\n extern const char *sha1_file_name(const unsigned char *sha1);\n \n+/*\n+ * Like sha1_file_name, but return the filename within a specific alternate\n+ * object directory. Shares the same static buffer with sha1_file_name.\n+ */\n+extern const char *sha1_file_name_alt(const char *objdir, const unsigned char *sha1);\n+\n /*\n * Return the name of the (local) packfile with the specified sha1 in\n * its name. The return value is a pointer to memory that is\n@@ -1205,6 +1211,8 @@ extern int do_check_packed_object_crc;\n \n extern int check_sha1_signature(const unsigned char *sha1, void *buf, unsigned long size, const char *type);\n \n+extern int create_object_tmpfile(struct strbuf *tmp, const char *filename);\n+extern void close_sha1_file(int fd);\n extern int finalize_object_file(const char *tmpfile, const char *filename);\n \n extern int has_sha1_pack(const unsigned char *sha1);\ndiff --git a/sha1_file.c b/sha1_file.c\nindex b60ae15f70..d330996bc4 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -253,12 +253,12 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)\n \t}\n }\n \n-const char *sha1_file_name(const unsigned char *sha1)\n+const char *sha1_file_name_alt(const char *objdir, const unsigned char *sha1)\n {\n \tstatic struct strbuf buf = STRBUF_INIT;\n \n \tstrbuf_reset(&buf);\n-\tstrbuf_addf(&buf, \"%s/\", get_object_directory());\n+\tstrbuf_addf(&buf, \"%s/\", objdir);\n \n \tfill_sha1_path(&buf, sha1);\n \treturn buf.buf;\n@@ -278,9 +278,14 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,\n \treturn buf->buf;\n }\n \n- char *odb_pack_name(struct strbuf *buf,\n-\t\t const unsigned char *sha1,\n-\t\t const char *ext)\n+const char *sha1_file_name(const unsigned char *sha1)\n+{\n+\treturn sha1_file_name_alt(get_object_directory(), sha1);\n+}\n+\n+char *odb_pack_name(struct strbuf *buf,\n+\t\t const unsigned char *sha1,\n+\t\t const char *ext)\n {\n \tstrbuf_reset(buf);\n \tstrbuf_addf(buf, \"%s/pack/pack-%s.%s\", get_object_directory(),\n@@ -1727,24 +1732,14 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,\n \treturn -1;\n }\n \n-/*\n- * Like stat_sha1_file(), but actually open the object and return the\n- * descriptor. See the caveats on the \"path\" parameter above.\n- */\n-static int open_sha1_file(const unsigned char *sha1, const char **path)\n+static int open_sha1_file_alt(const unsigned char *sha1, const char **path)\n {\n-\tint fd;\n \tstruct alternate_object_database *alt;\n-\tint most_interesting_errno;\n-\n-\t*path = sha1_file_name(sha1);\n-\tfd = git_open(*path);\n-\tif (fd >= 0)\n-\t\treturn fd;\n-\tmost_interesting_errno = errno;\n+\tint most_interesting_errno = errno;\n \n \tprepare_alt_odb();\n \tfor (alt = alt_odb_list; alt; alt = alt->next) {\n+\t\tint fd;\n \t\t*path = alt_sha1_path(alt, sha1);\n \t\tfd = git_open(*path);\n \t\tif (fd >= 0)\n@@ -1756,6 +1751,26 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)\n \treturn -1;\n }\n \n+/*\n+ * Like stat_sha1_file(), but actually open the object and return the\n+ * descriptor. See the caveats on the \"path\" parameter above.\n+ */\n+static int open_sha1_file(const unsigned char *sha1, const char **path)\n+{\n+\tint fd;\n+\n+\t*path = sha1_file_name(sha1);\n+\tfd = git_open(*path);\n+\tif (fd >= 0)\n+\t\treturn fd;\n+\n+\tfd = open_sha1_file_alt(sha1, path);\n+\tif (fd >= 0)\n+\t\treturn fd;\n+\n+\treturn fd;\n+}\n+\n /*\n * Map the loose object at \"path\" if it is not NULL, or the path found by\n * searching for a loose object named \"sha1\".\n@@ -3284,7 +3299,7 @@ int hash_sha1_file(const void *buf, unsigned long len, const char *type,\n }\n \n /* Finalize a file on disk, and close it. */\n-static void close_sha1_file(int fd)\n+void close_sha1_file(int fd)\n {\n \tif (fsync_object_files)\n \t\tfsync_or_die(fd, \"sha1 file\");\n@@ -3308,7 +3323,7 @@ static inline int directory_size(const char *filename)\n * We want to avoid cross-directory filename renames, because those\n * can have problems on various filesystems (FAT, NFS, Coda).\n */\n-static int create_tmpfile(struct strbuf *tmp, const char *filename)\n+int create_object_tmpfile(struct strbuf *tmp, const char *filename)\n {\n \tint fd, dirlen = directory_size(filename);\n \n@@ -3348,7 +3363,7 @@ static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,\n \tstatic struct strbuf tmp_file = STRBUF_INIT;\n \tconst char *filename = sha1_file_name(sha1);\n \n-\tfd = create_tmpfile(&tmp_file, filename);\n+\tfd = create_object_tmpfile(&tmp_file, filename);\n \tif (fd < 0) {\n \t\tif (errno == EACCES)\n \t\t\treturn error(\"insufficient permission for adding an object to repository database %s\", get_object_directory());\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325509","messageId":"20170803091926.1755-12-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 11/40] odb-helper: add odb_helper_init() to send 'init' instruction","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:18:57Z","receivedAt":"2017-08-03T09:20:15Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Let's add an odb_helper_init() function to send an 'init'\ninstruction to the helpers. This 'init' instruction is\nespecially useful to get the capabilities that are supported\nby the helpers.\n\nSo while at it, let's also add a parse_capabilities()\nfunction to parse them and a supported_capabilities\nvariable in struct odb_helper to store them.\n\nSigned-off-by: Christian Couder \n---\n external-odb.c | 9 ++++++++-\n odb-helper.c | 54 +++++++++++++++++++++++++++++++++++++++++++++++++\n odb-helper.h | 12 +++++++++++\n t/t0400-external-odb.sh | 4 ++++\n 4 files changed, 78 insertions(+), 1 deletion(-)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex e9c3f11666..0f0de170b8 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -41,12 +41,16 @@ static int external_odb_config(const char *var, const char *value, void *data)\n static void external_odb_init(void)\n {\n \tstatic int initialized;\n+\tstruct odb_helper *o;\n \n \tif (initialized)\n \t\treturn;\n \tinitialized = 1;\n \n \tgit_config(external_odb_config, NULL);\n+\n+\tfor (o = helpers; o; o = o->next)\n+\t\todb_helper_init(o);\n }\n \n const char *external_odb_root(void)\n@@ -63,9 +67,12 @@ int external_odb_has_object(const unsigned char *sha1)\n \n \texternal_odb_init();\n \n-\tfor (o = helpers; o; o = o->next)\n+\tfor (o = helpers; o; o = o->next) {\n+\t\tif (!(o->supported_capabilities & ODB_HELPER_CAP_HAVE))\n+\t\t\treturn 1;\n \t\tif (odb_helper_has_object(o, sha1))\n \t\t\treturn 1;\n+\t}\n \treturn 0;\n }\n \ndiff --git a/odb-helper.c b/odb-helper.c\nindex 0e6f824e4a..c6e16b938c 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -5,6 +5,40 @@\n #include \"run-command.h\"\n #include \"sha1-lookup.h\"\n \n+static void parse_capabilities(char *cap_buf,\n+\t\t\t unsigned int *supported_capabilities,\n+\t\t\t const char *process_name)\n+{\n+\tstruct string_list cap_list = STRING_LIST_INIT_NODUP;\n+\n+\tstring_list_split_in_place(&cap_list, cap_buf, '=', 1);\n+\n+\tif (cap_list.nr == 2 && !strcmp(cap_list.items[0].string, \"capability\")) {\n+\t\tconst char *cap_name = cap_list.items[1].string;\n+\n+\t\tif (!strcmp(cap_name, \"get_git_obj\")) {\n+\t\t\t*supported_capabilities |= ODB_HELPER_CAP_GET_GIT_OBJ;\n+\t\t} else if (!strcmp(cap_name, \"get_raw_obj\")) {\n+\t\t\t*supported_capabilities |= ODB_HELPER_CAP_GET_RAW_OBJ;\n+\t\t} else if (!strcmp(cap_name, \"get_direct\")) {\n+\t\t\t*supported_capabilities |= ODB_HELPER_CAP_GET_DIRECT;\n+\t\t} else if (!strcmp(cap_name, \"put_git_obj\")) {\n+\t\t\t*supported_capabilities |= ODB_HELPER_CAP_PUT_GIT_OBJ;\n+\t\t} else if (!strcmp(cap_name, \"put_raw_obj\")) {\n+\t\t\t*supported_capabilities |= ODB_HELPER_CAP_PUT_RAW_OBJ;\n+\t\t} else if (!strcmp(cap_name, \"put_direct\")) {\n+\t\t\t*supported_capabilities |= ODB_HELPER_CAP_PUT_DIRECT;\n+\t\t} else if (!strcmp(cap_name, \"have\")) {\n+\t\t\t*supported_capabilities |= ODB_HELPER_CAP_HAVE;\n+\t\t} else {\n+\t\t\twarning(\"external process '%s' requested unsupported read-object capability '%s'\",\n+\t\t\t\tprocess_name, cap_name);\n+\t\t}\n+\t}\n+\n+\tstring_list_clear(&cap_list, 0);\n+}\n+\n struct odb_helper *odb_helper_new(const char *name, int namelen)\n {\n \tstruct odb_helper *o;\n@@ -79,6 +113,26 @@ static int odb_helper_finish(struct odb_helper *o,\n \treturn 0;\n }\n \n+int odb_helper_init(struct odb_helper *o)\n+{\n+\tstruct odb_helper_cmd cmd;\n+\tFILE *fh;\n+\tstruct strbuf line = STRBUF_INIT;\n+\n+\tif (odb_helper_start(o, &cmd, \"init\") < 0)\n+\t\treturn -1;\n+\n+\tfh = xfdopen(cmd.child.out, \"r\");\n+\twhile (strbuf_getline(&line, fh) != EOF)\n+\t\tparse_capabilities(line.buf, &o->supported_capabilities, o->name);\n+\n+\tstrbuf_release(&line);\n+\tfclose(fh);\n+\todb_helper_finish(o, &cmd);\n+\n+\treturn 0;\n+}\n+\n static int parse_object_line(struct odb_helper_object *o, const char *line)\n {\n \tchar *end;\ndiff --git a/odb-helper.h b/odb-helper.h\nindex 5800661704..8e0b9dd9cb 100644\n--- a/odb-helper.h\n+++ b/odb-helper.h\n@@ -1,9 +1,20 @@\n #ifndef ODB_HELPER_H\n #define ODB_HELPER_H\n \n+#include \"external-odb.h\"\n+\n+#define ODB_HELPER_CAP_GET_GIT_OBJ (1u<<0)\n+#define ODB_HELPER_CAP_GET_RAW_OBJ (1u<<1)\n+#define ODB_HELPER_CAP_GET_DIRECT (1u<<2)\n+#define ODB_HELPER_CAP_PUT_GIT_OBJ (1u<<3)\n+#define ODB_HELPER_CAP_PUT_RAW_OBJ (1u<<4)\n+#define ODB_HELPER_CAP_PUT_DIRECT (1u<<5)\n+#define ODB_HELPER_CAP_HAVE (1u<<6)\n+\n struct odb_helper {\n \tconst char *name;\n \tconst char *cmd;\n+\tunsigned int supported_capabilities;\n \n \tstruct odb_helper_object {\n \t\tunsigned char sha1[20];\n@@ -18,6 +29,7 @@ struct odb_helper {\n };\n \n struct odb_helper *odb_helper_new(const char *name, int namelen);\n+int odb_helper_init(struct odb_helper *o);\n int odb_helper_has_object(struct odb_helper *o, const unsigned char *sha1);\n int odb_helper_get_object(struct odb_helper *o, const unsigned char *sha1,\n \t\t\t int fd);\ndiff --git a/t/t0400-external-odb.sh b/t/t0400-external-odb.sh\nindex 2f4749fab1..ed89f3ab40 100755\n--- a/t/t0400-external-odb.sh\n+++ b/t/t0400-external-odb.sh\n@@ -9,6 +9,10 @@ export ALT_SOURCE\n write_script odb-helper <<\\EOF\n GIT_DIR=$ALT_SOURCE; export GIT_DIR\n case \"$1\" in\n+init)\n+\techo \"capability=get_git_obj\"\n+\techo \"capability=have\"\n+\t;;\n have)\n \tgit cat-file --batch-check --batch-all-objects |\n \tawk '{print $1 \" \" $3 \" \" $2}'\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325510","messageId":"20170803091926.1755-15-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 14/40] external-odb: accept only blobs for now","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:19:00Z","receivedAt":"2017-08-03T09:20:20Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"The mechanism to decide which blobs should be sent to which\nexternal object database will be very simple for now.\nIf the external odb helper support any \"put_*\" instruction\nall the new blobs will be sent to it.\n\nSigned-off-by: Christian Couder \n---\n external-odb.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex 82fac702e8..a4f8c72e1c 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -124,6 +124,10 @@ int external_odb_put_object(const void *buf, size_t len,\n {\n \tstruct odb_helper *o;\n \n+\t/* For now accept only blobs */\n+\tif (strcmp(type, \"blob\"))\n+\t\treturn 1;\n+\n \texternal_odb_init();\n \n \tfor (o = helpers; o; o = o->next) {\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325511","messageId":"20170803091926.1755-13-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 12/40] t0400: add 'put_raw_obj' instruction to odb-helper script","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:18:58Z","receivedAt":"2017-08-03T09:20:23Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"To properly test passing objects from Git to an external odb\nwe need an odb-helper script that supports a 'put'\ncapability/instruction.\n\nFor now we will support only sending raw blobs, so the\nsupported capability/instruction will be 'put_raw_obj'.\n\nWhile at it let's add a test to check that our odb-helper\nscript works well.\n\nSigned-off-by: Christian Couder \n---\n t/t0400-external-odb.sh | 24 ++++++++++++++++++++++++\n 1 file changed, 24 insertions(+)\n\ndiff --git a/t/t0400-external-odb.sh b/t/t0400-external-odb.sh\nindex ed89f3ab40..3fa0449883 100755\n--- a/t/t0400-external-odb.sh\n+++ b/t/t0400-external-odb.sh\n@@ -7,10 +7,15 @@ test_description='basic tests for external object databases'\n ALT_SOURCE=\"$PWD/alt-repo/.git\"\n export ALT_SOURCE\n write_script odb-helper <<\\EOF\n+die() {\n+\tprintf >&2 \"%s\\n\" \"$@\"\n+\texit 1\n+}\n GIT_DIR=$ALT_SOURCE; export GIT_DIR\n case \"$1\" in\n init)\n \techo \"capability=get_git_obj\"\n+\techo \"capability=put_raw_obj\"\n \techo \"capability=have\"\n \t;;\n have)\n@@ -20,6 +25,16 @@ have)\n get_git_obj)\n \tcat \"$GIT_DIR\"/objects/$(echo $2 | sed 's#..#&/#')\n \t;;\n+put_raw_obj)\n+\tsha1=\"$2\"\n+\tsize=\"$3\"\n+\tkind=\"$4\"\n+\twriten=$(git hash-object -w -t \"$kind\" --stdin)\n+\ttest \"$writen\" = \"$sha1\" || die \"bad sha1 passed '$sha1' vs writen '$writen'\"\n+\t;;\n+*)\n+\tdie \"unknown command '$1'\"\n+\t;;\n esac\n EOF\n HELPER=\"\\\"$PWD\\\"/odb-helper\"\n@@ -47,4 +62,13 @@ test_expect_success 'helper can retrieve alt objects' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'helper can add objects to alt repo' '\n+\thash=$(echo \"Hello odb!\" | git hash-object -w -t blob --stdin) &&\n+\ttest -f .git/objects/$(echo $hash | sed \"s#..#&/#\") &&\n+\tsize=$(git cat-file -s \"$hash\") &&\n+\tgit cat-file blob \"$hash\" | ./odb-helper put_raw_obj \"$hash\" \"$size\" blob &&\n+\talt_size=$(cd alt-repo && git cat-file -s \"$hash\") &&\n+\ttest \"$size\" -eq \"$alt_size\"\n+'\n+\n test_done\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325512","messageId":"20170803091926.1755-21-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 20/40] lib-httpd: add list.sh","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:19:06Z","receivedAt":"2017-08-03T09:20:28Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This cgi script can list Git objects that have been uploaded as\nfiles to an apache web server. This script can also retrieve\nthe content of each of these files.\n\nThis will help make apache work as an external object database.\n\nSigned-off-by: Christian Couder \n---\n t/lib-httpd.sh | 1 +\n t/lib-httpd/list.sh | 41 +++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 42 insertions(+)\n create mode 100644 t/lib-httpd/list.sh\n\ndiff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\nindex d80b004549..f31ea261f5 100644\n--- a/t/lib-httpd.sh\n+++ b/t/lib-httpd.sh\n@@ -133,6 +133,7 @@ prepare_httpd() {\n \tinstall_script broken-smart-http.sh\n \tinstall_script error.sh\n \tinstall_script upload.sh\n+\tinstall_script list.sh\n \n \tln -s \"$LIB_HTTPD_MODULE_PATH\" \"$HTTPD_ROOT_PATH/modules\"\n \ndiff --git a/t/lib-httpd/list.sh b/t/lib-httpd/list.sh\nnew file mode 100644\nindex 0000000000..7e520e507a\n--- /dev/null\n+++ b/t/lib-httpd/list.sh\n@@ -0,0 +1,41 @@\n+#!/bin/sh\n+\n+FILES_DIR=\"www/files\"\n+\n+OLDIFS=\"$IFS\"\n+IFS='&'\n+set -- $QUERY_STRING\n+IFS=\"$OLDIFS\"\n+\n+while test $# -gt 0\n+do\n+ key=${1%=*}\n+ val=${1#*=}\n+\n+ case \"$key\" in\n+\t\"sha1\") sha1=\"$val\" ;;\n+\t*) echo >&2 \"unknown key '$key'\" ;;\n+ esac\n+\n+ shift\n+done\n+\n+if test -d \"$FILES_DIR\"\n+then\n+ if test -z \"$sha1\"\n+ then\n+\techo 'Status: 200 OK'\n+\techo\n+\tls \"$FILES_DIR\" | tr '-' ' '\n+ else\n+\tif test -f \"$FILES_DIR/$sha1\"-*\n+\tthen\n+\t echo 'Status: 200 OK'\n+\t echo\n+\t cat \"$FILES_DIR/$sha1\"-*\n+\telse\n+\t echo 'Status: 404 Not Found'\n+\t echo\n+\tfi\n+ fi\n+fi\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325513","messageId":"20170803091926.1755-14-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 13/40] external odb: add 'put_raw_obj' support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:18:59Z","receivedAt":"2017-08-03T09:20:31Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Add support for a 'put_raw_obj' capability/instruction to send new\nobjects to an external odb. Objects will be sent as they are (in\ntheir 'raw' format). They will not be converted to Git objects.\n\nFor now any new Git object (blob, tree, commit, ...) would be sent\nif 'put_raw_obj' is supported by an odb helper. This is not a great\ndefault, but let's leave it to following commits to tweak that.\n\nSigned-off-by: Christian Couder \n---\n external-odb.c | 15 +++++++++++++++\n external-odb.h | 2 ++\n odb-helper.c | 43 ++++++++++++++++++++++++++++++++++++++-----\n odb-helper.h | 3 +++\n sha1_file.c | 2 ++\n 5 files changed, 60 insertions(+), 5 deletions(-)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex 0f0de170b8..82fac702e8 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -118,3 +118,18 @@ int external_odb_get_object(const unsigned char *sha1)\n \n \treturn -1;\n }\n+\n+int external_odb_put_object(const void *buf, size_t len,\n+\t\t\t const char *type, unsigned char *sha1)\n+{\n+\tstruct odb_helper *o;\n+\n+\texternal_odb_init();\n+\n+\tfor (o = helpers; o; o = o->next) {\n+\t\tint r = odb_helper_put_object(o, buf, len, type, sha1);\n+\t\tif (r <= 0)\n+\t\t\treturn r;\n+\t}\n+\treturn 1;\n+}\ndiff --git a/external-odb.h b/external-odb.h\nindex 9989490c9e..3e0e6d0165 100644\n--- a/external-odb.h\n+++ b/external-odb.h\n@@ -4,5 +4,7 @@\n const char *external_odb_root(void);\n int external_odb_has_object(const unsigned char *sha1);\n int external_odb_get_object(const unsigned char *sha1);\n+int external_odb_put_object(const void *buf, size_t len,\n+\t\t\t const char *type, unsigned char *sha1);\n \n #endif /* EXTERNAL_ODB_H */\ndiff --git a/odb-helper.c b/odb-helper.c\nindex c6e16b938c..1be4461158 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -71,9 +71,10 @@ static void prepare_helper_command(struct argv_array *argv, const char *cmd,\n \tstrbuf_release(&buf);\n }\n \n-__attribute__((format (printf,3,4)))\n+__attribute__((format (printf,4,5)))\n static int odb_helper_start(struct odb_helper *o,\n \t\t\t struct odb_helper_cmd *cmd,\n+\t\t\t int use_stdin,\n \t\t\t const char *fmt, ...)\n {\n \tva_list ap;\n@@ -90,7 +91,10 @@ static int odb_helper_start(struct odb_helper *o,\n \n \tcmd->child.argv = cmd->argv.argv;\n \tcmd->child.use_shell = 1;\n-\tcmd->child.no_stdin = 1;\n+\tif (use_stdin)\n+\t\tcmd->child.in = -1;\n+\telse\n+\t\tcmd->child.no_stdin = 1;\n \tcmd->child.out = -1;\n \n \tif (start_command(&cmd->child) < 0) {\n@@ -119,7 +123,7 @@ int odb_helper_init(struct odb_helper *o)\n \tFILE *fh;\n \tstruct strbuf line = STRBUF_INIT;\n \n-\tif (odb_helper_start(o, &cmd, \"init\") < 0)\n+\tif (odb_helper_start(o, &cmd, 0, \"init\") < 0)\n \t\treturn -1;\n \n \tfh = xfdopen(cmd.child.out, \"r\");\n@@ -179,7 +183,7 @@ static void odb_helper_load_have(struct odb_helper *o)\n \t\treturn;\n \to->have_valid = 1;\n \n-\tif (odb_helper_start(o, &cmd, \"have\") < 0)\n+\tif (odb_helper_start(o, &cmd, 0, \"have\") < 0)\n \t\treturn;\n \n \tfh = xfdopen(cmd.child.out, \"r\");\n@@ -228,7 +232,7 @@ int odb_helper_get_object(struct odb_helper *o, const unsigned char *sha1,\n \tif (!obj)\n \t\treturn -1;\n \n-\tif (odb_helper_start(o, &cmd, \"get_git_obj %s\", sha1_to_hex(sha1)) < 0)\n+\tif (odb_helper_start(o, &cmd, 0, \"get_git_obj %s\", sha1_to_hex(sha1)) < 0)\n \t\treturn -1;\n \n \tmemset(&stream, 0, sizeof(stream));\n@@ -301,3 +305,32 @@ int odb_helper_get_object(struct odb_helper *o, const unsigned char *sha1,\n \n \treturn 0;\n }\n+\n+int odb_helper_put_object(struct odb_helper *o,\n+\t\t\t const void *buf, size_t len,\n+\t\t\t const char *type, unsigned char *sha1)\n+{\n+\tstruct odb_helper_cmd cmd;\n+\n+\tif (odb_helper_start(o, &cmd, 1, \"put_raw_obj %s %\"PRIuMAX\" %s\",\n+\t\t\t sha1_to_hex(sha1), (uintmax_t)len, type) < 0)\n+\t\treturn -1;\n+\n+\tdo {\n+\t\tint w = xwrite(cmd.child.in, buf, len);\n+\t\tif (w < 0) {\n+\t\t\terror(\"unable to write to odb helper '%s': %s\",\n+\t\t\t o->name, strerror(errno));\n+\t\t\tclose(cmd.child.in);\n+\t\t\tclose(cmd.child.out);\n+\t\t\todb_helper_finish(o, &cmd);\n+\t\t\treturn -1;\n+\t\t}\n+\t\tlen -= w;\n+\t} while (len > 0);\n+\n+\tclose(cmd.child.in);\n+\tclose(cmd.child.out);\n+\todb_helper_finish(o, &cmd);\n+\treturn 0;\n+}\ndiff --git a/odb-helper.h b/odb-helper.h\nindex 8e0b9dd9cb..318e0d48dc 100644\n--- a/odb-helper.h\n+++ b/odb-helper.h\n@@ -33,5 +33,8 @@ int odb_helper_init(struct odb_helper *o);\n int odb_helper_has_object(struct odb_helper *o, const unsigned char *sha1);\n int odb_helper_get_object(struct odb_helper *o, const unsigned char *sha1,\n \t\t\t int fd);\n+int odb_helper_put_object(struct odb_helper *o,\n+\t\t\t const void *buf, size_t len,\n+\t\t\t const char *type, unsigned char *sha1);\n \n #endif /* ODB_HELPER_H */\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 4bd790f6f8..6f6406fb36 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -3469,6 +3469,8 @@ int write_sha1_file(const void *buf, unsigned long len, const char *type, unsign\n \t * it out into .git/objects/??/?{38} file.\n \t */\n \twrite_sha1_file_prepare(buf, len, type, sha1, hdr, &hdrlen);\n+\tif (!external_odb_put_object(buf, len, type, sha1))\n+\t\treturn 0;\n \tif (freshen_packed_object(sha1) || freshen_loose_object(sha1))\n \t\treturn 0;\n \treturn write_loose_object(sha1, hdr, hdrlen, buf, len, 0);\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325514","messageId":"20170803091926.1755-23-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 22/40] odb-helper: add odb_helper_get_raw_object()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:19:08Z","receivedAt":"2017-08-03T09:20:34Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"The existing odb_helper_get_object() is renamed\nodb_helper_get_git_object() and a new odb_helper_get_raw_object()\nis introduced to deal with external objects that are not in Git format.\n\nSigned-off-by: Christian Couder \n---\n odb-helper.c | 113 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 111 insertions(+), 2 deletions(-)\n\ndiff --git a/odb-helper.c b/odb-helper.c\nindex 1be4461158..0603993057 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -217,8 +217,107 @@ int odb_helper_has_object(struct odb_helper *o, const unsigned char *sha1)\n \treturn !!odb_helper_lookup(o, sha1);\n }\n \n-int odb_helper_get_object(struct odb_helper *o, const unsigned char *sha1,\n-\t\t\t int fd)\n+static int odb_helper_get_raw_object(struct odb_helper *o,\n+\t\t\t\t const unsigned char *sha1,\n+\t\t\t\t int fd)\n+{\n+\tstruct odb_helper_object *obj;\n+\tstruct odb_helper_cmd cmd;\n+\tunsigned long total_got = 0;\n+\n+\tchar hdr[32];\n+\tint hdrlen;\n+\n+\tint ret = Z_STREAM_END;\n+\tunsigned char compressed[4096];\n+\tgit_zstream stream;\n+\tgit_SHA_CTX hash;\n+\tunsigned char real_sha1[20];\n+\n+\tobj = odb_helper_lookup(o, sha1);\n+\tif (!obj)\n+\t\treturn -1;\n+\n+\tif (odb_helper_start(o, &cmd, 0, \"get_raw_obj %s\", sha1_to_hex(sha1)) < 0)\n+\t\treturn -1;\n+\n+\t/* Set it up */\n+\tgit_deflate_init(&stream, zlib_compression_level);\n+\tstream.next_out = compressed;\n+\tstream.avail_out = sizeof(compressed);\n+\tgit_SHA1_Init(&hash);\n+\n+\t/* First header.. */\n+\thdrlen = xsnprintf(hdr, sizeof(hdr), \"%s %lu\", typename(obj->type), obj->size) + 1;\n+\tstream.next_in = (unsigned char *)hdr;\n+\tstream.avail_in = hdrlen;\n+\twhile (git_deflate(&stream, 0) == Z_OK)\n+\t\t; /* nothing */\n+\tgit_SHA1_Update(&hash, hdr, hdrlen);\n+\n+\tfor (;;) {\n+\t\tunsigned char buf[4096];\n+\t\tint r;\n+\n+\t\tr = xread(cmd.child.out, buf, sizeof(buf));\n+\t\tif (r < 0) {\n+\t\t\terror(\"unable to read from odb helper '%s': %s\",\n+\t\t\t o->name, strerror(errno));\n+\t\t\tclose(cmd.child.out);\n+\t\t\todb_helper_finish(o, &cmd);\n+\t\t\tgit_deflate_end(&stream);\n+\t\t\treturn -1;\n+\t\t}\n+\t\tif (r == 0)\n+\t\t\tbreak;\n+\n+\t\ttotal_got += r;\n+\n+\t\t/* Then the data itself.. */\n+\t\tstream.next_in = (void *)buf;\n+\t\tstream.avail_in = r;\n+\t\tdo {\n+\t\t\tunsigned char *in0 = stream.next_in;\n+\t\t\tret = git_deflate(&stream, Z_FINISH);\n+\t\t\tgit_SHA1_Update(&hash, in0, stream.next_in - in0);\n+\t\t\twrite_or_die(fd, compressed, stream.next_out - compressed);\n+\t\t\tstream.next_out = compressed;\n+\t\t\tstream.avail_out = sizeof(compressed);\n+\t\t} while (ret == Z_OK);\n+\t}\n+\n+\tclose(cmd.child.out);\n+\tif (ret != Z_STREAM_END) {\n+\t\twarning(\"bad zlib data from odb helper '%s' for %s\",\n+\t\t\to->name, sha1_to_hex(sha1));\n+\t\treturn -1;\n+\t}\n+\tret = git_deflate_end_gently(&stream);\n+\tif (ret != Z_OK) {\n+\t\twarning(\"deflateEnd on object %s from odb helper '%s' failed (%d)\",\n+\t\t\tsha1_to_hex(sha1), o->name, ret);\n+\t\treturn -1;\n+\t}\n+\tgit_SHA1_Final(real_sha1, &hash);\n+\tif (hashcmp(sha1, real_sha1)) {\n+\t\twarning(\"sha1 mismatch from odb helper '%s' for %s (got %s)\",\n+\t\t\to->name, sha1_to_hex(sha1), sha1_to_hex(real_sha1));\n+\t\treturn -1;\n+\t}\n+\tif (odb_helper_finish(o, &cmd))\n+\t\treturn -1;\n+\tif (total_got != obj->size) {\n+\t\twarning(\"size mismatch from odb helper '%s' for %s (%lu != %lu)\",\n+\t\t\to->name, sha1_to_hex(sha1), total_got, obj->size);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int odb_helper_get_git_object(struct odb_helper *o,\n+\t\t\t\t const unsigned char *sha1,\n+\t\t\t\t int fd)\n {\n \tstruct odb_helper_object *obj;\n \tstruct odb_helper_cmd cmd;\n@@ -306,6 +405,16 @@ int odb_helper_get_object(struct odb_helper *o, const unsigned char *sha1,\n \treturn 0;\n }\n \n+int odb_helper_get_object(struct odb_helper *o,\n+\t\t\t const unsigned char *sha1,\n+\t\t\t int fd)\n+{\n+\tif (o->supported_capabilities & ODB_HELPER_CAP_GET_RAW_OBJ)\n+\t\treturn odb_helper_get_raw_object(o, sha1, fd);\n+\telse\n+\t\treturn odb_helper_get_git_object(o, sha1, fd);\n+}\n+\n int odb_helper_put_object(struct odb_helper *o,\n \t\t\t const void *buf, size_t len,\n \t\t\t const char *type, unsigned char *sha1)\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325515","messageId":"20170803091926.1755-20-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 19/40] lib-httpd: add upload.sh","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:19:05Z","receivedAt":"2017-08-03T09:20:37Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This cgi will be used to upload objects to, or to delete\nobjects from, an apache web server.\n\nThis way the apache server can work as an external object\ndatabase.\n\nSigned-off-by: Christian Couder \n---\n t/lib-httpd.sh | 1 +\n t/lib-httpd/upload.sh | 45 +++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 46 insertions(+)\n create mode 100644 t/lib-httpd/upload.sh\n\ndiff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\nindex 2e659a8ee2..d80b004549 100644\n--- a/t/lib-httpd.sh\n+++ b/t/lib-httpd.sh\n@@ -132,6 +132,7 @@ prepare_httpd() {\n \tcp \"$TEST_PATH\"/passwd \"$HTTPD_ROOT_PATH\"\n \tinstall_script broken-smart-http.sh\n \tinstall_script error.sh\n+\tinstall_script upload.sh\n \n \tln -s \"$LIB_HTTPD_MODULE_PATH\" \"$HTTPD_ROOT_PATH/modules\"\n \ndiff --git a/t/lib-httpd/upload.sh b/t/lib-httpd/upload.sh\nnew file mode 100644\nindex 0000000000..172be0f73f\n--- /dev/null\n+++ b/t/lib-httpd/upload.sh\n@@ -0,0 +1,45 @@\n+#!/bin/sh\n+\n+# In part from http://codereview.stackexchange.com/questions/79549/bash-cgi-upload-file\n+\n+FILES_DIR=\"www/files\"\n+\n+OLDIFS=\"$IFS\"\n+IFS='&'\n+set -- $QUERY_STRING\n+IFS=\"$OLDIFS\"\n+\n+while test $# -gt 0\n+do\n+ key=${1%=*}\n+ val=${1#*=}\n+\n+ case \"$key\" in\n+\t\"sha1\") sha1=\"$val\" ;;\n+\t\"type\") type=\"$val\" ;;\n+\t\"size\") size=\"$val\" ;;\n+\t\"delete\") delete=1 ;;\n+\t*) echo >&2 \"unknown key '$key'\" ;;\n+ esac\n+\n+ shift\n+done\n+\n+case \"$REQUEST_METHOD\" in\n+ POST)\n+ if test \"$delete\" = \"1\"\n+ then\n+\trm -f \"$FILES_DIR/$sha1-$size-$type\"\n+ else\n+\tmkdir -p \"$FILES_DIR\"\n+\tcat >\"$FILES_DIR/$sha1-$size-$type\"\n+ fi\n+\n+ echo 'Status: 204 No Content'\n+ echo\n+ ;;\n+\n+ *)\n+ echo 'Status: 405 Method Not Allowed'\n+ echo\n+esac\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325516","messageId":"20170803091926.1755-22-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 21/40] lib-httpd: add apache-e-odb.conf","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:19:07Z","receivedAt":"2017-08-03T09:20:40Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This is an apache config file to test external object databases.\nIt uses the upload.sh and list.sh cgi that have been added\npreviously to make apache store external objects.\n\nSigned-off-by: Christian Couder \n---\n t/lib-httpd/apache-e-odb.conf | 214 ++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 214 insertions(+)\n create mode 100644 t/lib-httpd/apache-e-odb.conf\n\ndiff --git a/t/lib-httpd/apache-e-odb.conf b/t/lib-httpd/apache-e-odb.conf\nnew file mode 100644\nindex 0000000000..19a1540c82\n--- /dev/null\n+++ b/t/lib-httpd/apache-e-odb.conf\n@@ -0,0 +1,214 @@\n+ServerName dummy\n+PidFile httpd.pid\n+DocumentRoot www\n+LogFormat \"%h %l %u %t \\\"%r\\\" %>s %b\" common\n+CustomLog access.log common\n+ErrorLog error.log\n+\n+\tLoadModule log_config_module modules/mod_log_config.so\n+\n+\n+\tLoadModule alias_module modules/mod_alias.so\n+\n+\n+\tLoadModule cgi_module modules/mod_cgi.so\n+\n+\n+\tLoadModule env_module modules/mod_env.so\n+\n+\n+\tLoadModule rewrite_module modules/mod_rewrite.so\n+\n+\n+\tLoadModule version_module modules/mod_version.so\n+\n+\n+\tLoadModule headers_module modules/mod_headers.so\n+\n+\n+\n+LockFile accept.lock\n+\n+\n+\n+\n+\tLoadModule auth_module modules/mod_auth.so\n+\n+\n+\n+= 2.1>\n+\n+\tLoadModule auth_basic_module modules/mod_auth_basic.so\n+\n+\n+\tLoadModule authn_file_module modules/mod_authn_file.so\n+\n+\n+\tLoadModule authz_user_module modules/mod_authz_user.so\n+\n+\n+\tLoadModule authz_host_module modules/mod_authz_host.so\n+\n+\n+\n+= 2.4>\n+\n+\tLoadModule authn_core_module modules/mod_authn_core.so\n+\n+\n+\tLoadModule authz_core_module modules/mod_authz_core.so\n+\n+\n+\tLoadModule access_compat_module modules/mod_access_compat.so\n+\n+\n+\tLoadModule mpm_prefork_module modules/mod_mpm_prefork.so\n+\n+\n+\tLoadModule unixd_module modules/mod_unixd.so\n+\n+\n+\n+PassEnv GIT_VALGRIND\n+PassEnv GIT_VALGRIND_OPTIONS\n+PassEnv GNUPGHOME\n+PassEnv ASAN_OPTIONS\n+PassEnv GIT_TRACE\n+PassEnv GIT_CONFIG_NOSYSTEM\n+\n+Alias /dumb/ www/\n+Alias /auth/dumb/ www/auth/dumb/\n+\n+\n+\tSetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}\n+\tSetEnv GIT_HTTP_EXPORT_ALL\n+\n+\n+\tSetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}\n+\n+\n+\tSetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}\n+\tSetEnv GIT_HTTP_EXPORT_ALL\n+\tSetEnv GIT_COMMITTER_NAME \"Custom User\"\n+\tSetEnv GIT_COMMITTER_EMAIL custom@example.com\n+\n+\n+\tSetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}\n+\tSetEnv GIT_HTTP_EXPORT_ALL\n+\tSetEnv GIT_NAMESPACE ns\n+\n+\n+\tSetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}\n+\tSetEnv GIT_HTTP_EXPORT_ALL\n+\tHeader set Set-Cookie name=value\n+\n+\n+\tSetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}\n+\tSetEnv GIT_HTTP_EXPORT_ALL\n+\n+ScriptAlias /upload/ upload.sh/\n+ScriptAlias /list/ list.sh/\n+\n+\tOptions FollowSymlinks\n+\n+\n+ Options ExecCGI\n+\n+\n+ Options ExecCGI\n+\n+\n+\tOptions ExecCGI\n+\n+\n+RewriteEngine on\n+RewriteRule ^/smart-redir-perm/(.*)$ /smart/$1 [R=301]\n+RewriteRule ^/smart-redir-temp/(.*)$ /smart/$1 [R=302]\n+RewriteRule ^/smart-redir-auth/(.*)$ /auth/smart/$1 [R=301]\n+RewriteRule ^/smart-redir-limited/(.*)/info/refs$ /smart/$1/info/refs [R=301]\n+RewriteRule ^/ftp-redir/(.*)$ ftp://localhost:1000/$1 [R=302]\n+\n+RewriteRule ^/loop-redir/x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-(.*) /$1 [R=302]\n+RewriteRule ^/loop-redir/(.*)$ /loop-redir/x-$1 [R=302]\n+\n+# Apache 2.2 does not understand , so we use RewriteCond.\n+# And as RewriteCond does not allow testing for non-matches, we match\n+# the desired case first (one has abra, two has cadabra), and let it\n+# pass by marking the RewriteRule as [L], \"last rule, do not process\n+# any other matching RewriteRules after this\"), and then have another\n+# RewriteRule that matches all other cases and lets them fail via '[F]',\n+# \"fail the request\".\n+RewriteCond %{HTTP:x-magic-one} =abra\n+RewriteCond %{HTTP:x-magic-two} =cadabra\n+RewriteRule ^/smart_headers/.* - [L]\n+RewriteRule ^/smart_headers/.* - [F]\n+\n+\n+LoadModule ssl_module modules/mod_ssl.so\n+\n+SSLCertificateFile httpd.pem\n+SSLCertificateKeyFile httpd.pem\n+SSLRandomSeed startup file:/dev/urandom 512\n+SSLRandomSeed connect file:/dev/urandom 512\n+SSLSessionCache none\n+SSLMutex file:ssl_mutex\n+SSLEngine On\n+\n+\n+\n+\tAuthType Basic\n+\tAuthName \"git-auth\"\n+\tAuthUserFile passwd\n+\tRequire valid-user\n+\n+\n+\n+\tAuthType Basic\n+\tAuthName \"git-auth\"\n+\tAuthUserFile passwd\n+\tRequire valid-user\n+\n+\n+\n+\tAuthType Basic\n+\tAuthName \"git-auth\"\n+\tAuthUserFile passwd\n+\tRequire valid-user\n+\n+\n+RewriteCond %{QUERY_STRING} service=git-receive-pack [OR]\n+RewriteCond %{REQUEST_URI} /git-receive-pack$\n+RewriteRule ^/half-auth-complete/ - [E=AUTHREQUIRED:yes]\n+\n+\n+ Order Deny,Allow\n+ Deny from env=AUTHREQUIRED\n+\n+ AuthType Basic\n+ AuthName \"Git Access\"\n+ AuthUserFile passwd\n+ Require valid-user\n+ Satisfy Any\n+\n+\n+\n+\tLoadModule dav_module modules/mod_dav.so\n+\tLoadModule dav_fs_module modules/mod_dav_fs.so\n+\n+\tDAVLockDB DAVLock\n+\t\n+\t\tDav on\n+\t\n+\t\n+\t\tDav on\n+\t\n+\n+\n+\n+\tLoadModule dav_svn_module modules/mod_dav_svn.so\n+\n+\t\n+\t\tDAV svn\n+\t\tSVNPath \"${LIB_HTTPD_SVNPATH}\"\n+\t\n+\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325517","messageId":"20170803091926.1755-30-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 29/40] Add t0460 to test passing git objects","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:19:15Z","receivedAt":"2017-08-03T09:20:43Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n t/t0460-read-object-git.sh | 28 +++++++++++++++++\n t/t0460/read-object-git | 78 ++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 106 insertions(+)\n create mode 100755 t/t0460-read-object-git.sh\n create mode 100755 t/t0460/read-object-git\n\ndiff --git a/t/t0460-read-object-git.sh b/t/t0460-read-object-git.sh\nnew file mode 100755\nindex 0000000000..2873b445f3\n--- /dev/null\n+++ b/t/t0460-read-object-git.sh\n@@ -0,0 +1,28 @@\n+#!/bin/sh\n+\n+test_description='tests for long running read-object process passing git objects'\n+\n+. ./test-lib.sh\n+\n+PATH=\"$PATH:$TEST_DIRECTORY/t0460\"\n+\n+test_expect_success 'setup host repo with a root commit' '\n+\ttest_commit zero &&\n+\thash1=$(git ls-tree HEAD | grep zero.t | cut -f1 | cut -d\\ -f3)\n+'\n+\n+HELPER=\"read-object-git\"\n+\n+test_expect_success 'blobs can be retrieved from the host repo' '\n+\tgit init guest-repo &&\n+\t(cd guest-repo &&\n+\t git config odb.magic.subprocessCommand \"$HELPER\" &&\n+\t git cat-file blob \"$hash1\" >/dev/null)\n+'\n+\n+test_expect_success 'invalid blobs generate errors' '\n+\tcd guest-repo &&\n+\ttest_must_fail git cat-file blob \"invalid\"\n+'\n+\n+test_done\ndiff --git a/t/t0460/read-object-git b/t/t0460/read-object-git\nnew file mode 100755\nindex 0000000000..38529e622e\n--- /dev/null\n+++ b/t/t0460/read-object-git\n@@ -0,0 +1,78 @@\n+#!/usr/bin/perl\n+#\n+# Example implementation for the Git read-object protocol version 1\n+# See Documentation/technical/read-object-protocol.txt\n+#\n+# Allows you to test the ability for blobs to be pulled from a host git repo\n+# \"on demand.\" Called when git needs a blob it couldn't find locally due to\n+# a lazy clone that only cloned the commits and trees.\n+#\n+# A lazy clone can be simulated via the following commands from the host repo\n+# you wish to create a lazy clone of:\n+#\n+# cd /host_repo\n+# git rev-parse HEAD\n+# git init /guest_repo\n+# git cat-file --batch-check --batch-all-objects | grep -v 'blob' |\n+#\tcut -d' ' -f1 | git pack-objects /e/guest_repo/.git/objects/pack/noblobs\n+# cd /guest_repo\n+# git config core.virtualizeobjects true\n+# git reset --hard \n+#\n+# Please note, this sample is a minimal skeleton. No proper error handling \n+# was implemented.\n+#\n+\n+use 5.008;\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n+use strict;\n+use warnings;\n+use Git::Packet;\n+\n+#\n+# Point $DIR to the folder where your host git repo is located so we can pull\n+# missing objects from it\n+#\n+my $DIR = \"../.git/\";\n+\n+packet_initialize(\"git-read-object\", 1);\n+\n+packet_read_and_check_capabilities(\"get_git_obj\");\n+packet_write_capabilities(\"get_git_obj\");\n+\n+while (1) {\n+\tmy ($res, $command) = packet_txt_read();\n+\n+\tif ( $res == -1 ) {\n+\t\texit 0;\n+\t}\n+\n+\t$command =~ s/^command=//;\n+\n+\tif ( $command eq \"init\" ) {\n+\t\tpacket_bin_read();\n+\n+\t\tpacket_txt_write(\"status=success\");\n+\t\tpacket_flush();\n+\t} elsif ( $command eq \"get_git_obj\" ) {\n+\t\tmy ($sha1) = packet_txt_read() =~ /^sha1=([0-9a-f]{40})$/;\n+\t\tpacket_bin_read();\n+\n+\t\tmy $path = $sha1;\n+\t\t$path =~ s{..}{$&/};\n+\t\t$path = $DIR . \"/objects/\" . $path;\n+\n+\t\tmy $contents = do {\n+\t\t local $/;\n+\t\t open my $fh, $path or die \"Can't open '$path': $!\";\n+\t\t <$fh>\n+\t\t};\n+\n+\t\tpacket_bin_write($contents);\n+\t\tpacket_flush();\n+\t\tpacket_txt_write(\"status=success\");\n+\t\tpacket_flush();\n+\t} else {\n+\t\tdie \"bad command '$command'\";\n+\t}\n+}\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325518","messageId":"20170803091926.1755-29-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 28/40] Add t0450 to test 'get_direct' mechanism","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:19:14Z","receivedAt":"2017-08-03T09:20:45Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: Ben Peart \n\nSigned-off-by: Ben Peart \nSigned-off-by: Christian Couder \n---\n t/t0450-read-object.sh | 28 +++++++++++++++++++++\n t/t0450/read-object | 68 ++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 96 insertions(+)\n create mode 100755 t/t0450-read-object.sh\n create mode 100755 t/t0450/read-object\n\ndiff --git a/t/t0450-read-object.sh b/t/t0450-read-object.sh\nnew file mode 100755\nindex 0000000000..6b97305452\n--- /dev/null\n+++ b/t/t0450-read-object.sh\n@@ -0,0 +1,28 @@\n+#!/bin/sh\n+\n+test_description='tests for long running read-object process'\n+\n+. ./test-lib.sh\n+\n+PATH=\"$PATH:$TEST_DIRECTORY/t0450\"\n+\n+test_expect_success 'setup host repo with a root commit' '\n+\ttest_commit zero &&\n+\thash1=$(git ls-tree HEAD | grep zero.t | cut -f1 | cut -d\\ -f3)\n+'\n+\n+HELPER=\"read-object\"\n+\n+test_expect_success 'blobs can be retrieved from the host repo' '\n+\tgit init guest-repo &&\n+\t(cd guest-repo &&\n+\t git config odb.magic.subprocessCommand \"$HELPER\" &&\n+\t git cat-file blob \"$hash1\" >/dev/null)\n+'\n+\n+test_expect_success 'invalid blobs generate errors' '\n+\tcd guest-repo &&\n+\ttest_must_fail git cat-file blob \"invalid\"\n+'\n+\n+test_done\ndiff --git a/t/t0450/read-object b/t/t0450/read-object\nnew file mode 100755\nindex 0000000000..cf22e2f581\n--- /dev/null\n+++ b/t/t0450/read-object\n@@ -0,0 +1,68 @@\n+#!/usr/bin/perl\n+#\n+# Example implementation for the Git read-object protocol version 1\n+# See Documentation/technical/read-object-protocol.txt\n+#\n+# Allows you to test the ability for blobs to be pulled from a host git repo\n+# \"on demand.\" Called when git needs a blob it couldn't find locally due to\n+# a lazy clone that only cloned the commits and trees.\n+#\n+# A lazy clone can be simulated via the following commands from the host repo\n+# you wish to create a lazy clone of:\n+#\n+# cd /host_repo\n+# git rev-parse HEAD\n+# git init /guest_repo\n+# git cat-file --batch-check --batch-all-objects | grep -v 'blob' |\n+#\tcut -d' ' -f1 | git pack-objects /e/guest_repo/.git/objects/pack/noblobs\n+# cd /guest_repo\n+# git config core.virtualizeobjects true\n+# git reset --hard \n+#\n+# Please note, this sample is a minimal skeleton. No proper error handling \n+# was implemented.\n+#\n+\n+use 5.008;\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n+use strict;\n+use warnings;\n+use Git::Packet;\n+\n+#\n+# Point $DIR to the folder where your host git repo is located so we can pull\n+# missing objects from it\n+#\n+my $DIR = \"../.git/\";\n+\n+packet_initialize(\"git-read-object\", 1);\n+\n+packet_read_and_check_capabilities(\"get_direct\");\n+packet_write_capabilities(\"get_direct\");\n+\n+while (1) {\n+\tmy ($res, $command) = packet_txt_read();\n+\n+\tif ( $res == -1 ) {\n+\t\texit 0;\n+\t}\n+\n+\t$command =~ s/^command=//;\n+\n+\tif ( $command eq \"init\" ) {\n+\t\tpacket_bin_read();\n+\n+\t\tpacket_txt_write(\"status=success\");\n+\t\tpacket_flush();\n+\t} elsif ( $command eq \"get_direct\" ) {\n+\t\tmy ($sha1) = packet_txt_read() =~ /^sha1=([0-9a-f]{40})$/;\n+\t\tpacket_bin_read();\n+\n+\t\tsystem ('git --git-dir=\"' . $DIR . '\" cat-file blob ' . $sha1 . ' | GIT_NO_EXTERNAL_ODB=1 git hash-object -w --stdin >/dev/null 2>&1');\n+\n+\t\tpacket_txt_write(($?) ? \"status=error\" : \"status=success\");\n+\t\tpacket_flush();\n+\t} else {\n+\t\tdie \"bad command '$command'\";\n+\t}\n+}\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325519","messageId":"20170803091926.1755-34-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 33/40] Add t0480 to test \"have\" capability and raw objects","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:19:19Z","receivedAt":"2017-08-03T09:20:59Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n t/t0480-read-object-have-http-e-odb.sh | 109 +++++++++++++++++++++++++++++++++\n t/t0480/read-object-plain-have | 103 +++++++++++++++++++++++++++++++\n 2 files changed, 212 insertions(+)\n create mode 100755 t/t0480-read-object-have-http-e-odb.sh\n create mode 100755 t/t0480/read-object-plain-have\n\ndiff --git a/t/t0480-read-object-have-http-e-odb.sh b/t/t0480-read-object-have-http-e-odb.sh\nnew file mode 100755\nindex 0000000000..056a40f2bb\n--- /dev/null\n+++ b/t/t0480-read-object-have-http-e-odb.sh\n@@ -0,0 +1,109 @@\n+#!/bin/sh\n+\n+test_description='tests for read-object process with \"have\" cap and plain objects'\n+\n+. ./test-lib.sh\n+\n+# If we don't specify a port, the current test number will be used\n+# which will not work as it is less than 1024, so it can only be used by root.\n+LIB_HTTPD_PORT=$(expr ${this_test#t} + 12000)\n+\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+\n+start_httpd apache-e-odb.conf\n+\n+PATH=\"$PATH:$TEST_DIRECTORY/t0480\"\n+\n+# odb helper script must see this\n+export HTTPD_URL\n+\n+HELPER=\"read-object-plain-have\"\n+\n+test_expect_success 'setup repo with a root commit' '\n+\ttest_commit zero\n+'\n+\n+test_expect_success 'setup another repo from the first one' '\n+\tgit init other-repo &&\n+\t(cd other-repo &&\n+\t git remote add origin .. &&\n+\t git pull origin master &&\n+\t git checkout master &&\n+\t git log)\n+'\n+\n+test_expect_success 'setup the helper in the root repo' '\n+\tgit config odb.magic.subprocessCommand \"$HELPER\"\n+'\n+\n+UPLOADFILENAME=\"hello_apache_upload.txt\"\n+\n+UPLOAD_URL=\"$HTTPD_URL/upload/?sha1=$UPLOADFILENAME&size=123&type=blob\"\n+\n+test_expect_success 'can upload a file' '\n+\techo \"Hello Apache World!\" >hello_to_send.txt &&\n+\techo \"How are you?\" >>hello_to_send.txt &&\n+\tcurl --data-binary @hello_to_send.txt --include \"$UPLOAD_URL\" >out_upload\n+'\n+\n+LIST_URL=\"$HTTPD_URL/list/\"\n+\n+test_expect_success 'can list uploaded files' '\n+\tcurl --include \"$LIST_URL\" >out_list &&\n+\tgrep \"$UPLOADFILENAME\" out_list\n+'\n+\n+test_expect_success 'can delete uploaded files' '\n+\tcurl --data \"delete\" --include \"$UPLOAD_URL&delete=1\" >out_delete &&\n+\tcurl --include \"$LIST_URL\" >out_list2 &&\n+\t! grep \"$UPLOADFILENAME\" out_list2\n+'\n+\n+FILES_DIR=\"httpd/www/files\"\n+\n+test_expect_success 'new blobs are transfered to the http server' '\n+\ttest_commit one &&\n+\thash1=$(git ls-tree HEAD | grep one.t | cut -f1 | cut -d\\ -f3) &&\n+\techo \"$hash1-4-blob\" >expected &&\n+\tls \"$FILES_DIR\" >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'blobs can be retrieved from the http server' '\n+\tgit cat-file blob \"$hash1\" &&\n+\tgit log -p >expected\n+'\n+\n+test_expect_success 'update other repo from the first one' '\n+\t(cd other-repo &&\n+\t git fetch origin \"refs/odbs/magic/*:refs/odbs/magic/*\" &&\n+\t test_must_fail git cat-file blob \"$hash1\" &&\n+\t git config odb.magic.subprocessCommand \"$HELPER\" &&\n+\t git cat-file blob \"$hash1\" &&\n+\t git pull origin master)\n+'\n+\n+test_expect_success 'local clone from the first repo' '\n+\tmkdir my-clone &&\n+\t(cd my-clone &&\n+\t git clone .. . &&\n+\t git cat-file blob \"$hash1\")\n+'\n+\n+test_expect_success 'no-local clone from the first repo fails' '\n+\tmkdir my-other-clone &&\n+\t(cd my-other-clone &&\n+\t test_must_fail git clone --no-local .. .) &&\n+\trm -rf my-other-clone\n+'\n+\n+test_expect_success 'no-local clone from the first repo with helper succeeds' '\n+\tmkdir my-other-clone &&\n+\t(cd my-other-clone &&\n+\t git clone -c odb.magic.subprocessCommand=\"$HELPER\" --no-local .. .) &&\n+\trm -rf my-other-clone\n+'\n+\n+stop_httpd\n+\n+test_done\ndiff --git a/t/t0480/read-object-plain-have b/t/t0480/read-object-plain-have\nnew file mode 100755\nindex 0000000000..d63e327f33\n--- /dev/null\n+++ b/t/t0480/read-object-plain-have\n@@ -0,0 +1,103 @@\n+#!/usr/bin/perl\n+#\n+\n+use 5.008;\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n+use strict;\n+use warnings;\n+use Git::Packet;\n+use LWP::UserAgent;\n+use HTTP::Request::Common;\n+\n+packet_initialize(\"git-read-object\", 1);\n+\n+packet_read_and_check_capabilities(\"get_raw_obj\", \"put_raw_obj\", \"have\");\n+packet_write_capabilities(\"get_raw_obj\", \"put_raw_obj\", \"have\");\n+\n+my $http_url = $ENV{HTTPD_URL};\n+\n+while (1) {\n+\tmy ($res, $command) = packet_txt_read();\n+\n+\tif ( $res == -1 ) {\n+\t\texit 0;\n+\t}\n+\n+\t$command =~ s/^command=//;\n+\n+\tif ( $command eq \"init\" ) {\n+\t\tpacket_bin_read();\n+\n+\t\tpacket_txt_write(\"status=success\");\n+\t\tpacket_flush();\n+\t} elsif ( $command eq \"have\" ) {\n+\t\t# read the flush after the command\n+\t\tpacket_bin_read();\n+\n+\t\tmy $have_url = $http_url . \"/list/\";\n+\n+\t\tmy $userAgent = LWP::UserAgent->new();\n+\t\tmy $response = $userAgent->get( $have_url );\n+\n+\t\tif ($response->is_error) {\n+\t\t\tpacket_bin_write(\"\");\n+\t\t\tpacket_flush();\n+\t\t\tpacket_txt_write(\"status=failure\");\n+\t\t} else {\n+\t\t\tpacket_bin_write($response->content);\n+\t\t\tpacket_flush();\n+\t\t\tpacket_txt_write(\"status=success\");\n+\t\t}\n+\t\tpacket_flush();\n+\t} elsif ( $command eq \"get_raw_obj\" ) {\n+\t\tmy ($sha1) = packet_txt_read() =~ /^sha1=([0-9a-f]{40})$/;\n+\t\tpacket_bin_read();\n+\n+\t\tmy $get_url = $http_url . \"/list/?sha1=\" . $sha1;\n+\n+\t\tmy $userAgent = LWP::UserAgent->new();\n+\n+\t\tmy $response = $userAgent->get( $get_url );\n+\n+\t\tif ($response->is_error) {\n+\t\t\tpacket_txt_write(\"size=0\");\n+\t\t\tpacket_txt_write(\"kind=none\");\t \n+\t\t\tpacket_txt_write(\"status=notfound\");\n+\t\t} else {\n+\t\t\tpacket_txt_write(\"size=\" . length($response->content));\n+\t\t\tpacket_txt_write(\"kind=blob\");\n+\t\t\tpacket_bin_write($response->content);\n+\t\t\tpacket_flush();\n+\t\t\tpacket_txt_write(\"status=success\");\n+\t\t}\n+\n+\t\tpacket_flush();\n+\t} elsif ( $command eq \"put_raw_obj\" ) {\n+\t\tmy ($sha1) = packet_txt_read() =~ /^sha1=([0-9a-f]{40})$/;\n+\t\tmy ($size) = packet_txt_read() =~ /^size=([0-9]+)$/;\n+\t\tmy ($kind) = packet_txt_read() =~ /^kind=(\\w+)$/;\n+\n+\t\tpacket_bin_read();\n+\n+\t\t# We must read the content we are sent and send it to the right url\n+\t\tmy ($res, $buf) = packet_bin_read();\n+\t\tdie \"bad packet_bin_read res ($res)\" unless ($res eq 0);\n+\t\t( packet_bin_read() eq ( 1, \"\" ) ) || die \"bad send end\";\t\t\n+\n+\t\tmy $upload_url = $http_url . \"/upload/?sha1=\" . $sha1 . \"&size=\" . $size . \"&type=blob\";\n+\n+\t\tmy $userAgent = LWP::UserAgent->new();\n+\t\tmy $request = POST $upload_url, Content_Type => 'multipart/form-data', Content => $buf;\n+\n+\t\tmy $response = $userAgent->request($request);\n+\n+\t\tif ($response->is_error) {\n+\t\t\tpacket_txt_write(\"status=failure\");\n+\t\t} else {\n+\t\t\tpacket_txt_write(\"status=success\");\n+\t\t}\n+\t\tpacket_flush();\n+\t} else {\n+\t\tdie \"bad command '$command'\";\n+\t}\n+}\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325520","messageId":"20170803091926.1755-35-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 34/40] external-odb: use 'odb=magic' attribute to mark odb blobs","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:19:20Z","receivedAt":"2017-08-03T09:21:01Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"To tell which blobs should be sent to the \"magic\" external odb,\nlet's require that the blobs be marked using the 'odb=magic'\nattribute.\n\nSigned-off-by: Christian Couder \n---\n external-odb.c | 22 ++++++++++++++++++++--\n external-odb.h | 3 ++-\n sha1_file.c | 20 +++++++++++++++-----\n t/t0400-external-odb.sh | 3 +++\n t/t0410-transfer-e-odb.sh | 3 +++\n t/t0420-transfer-http-e-odb.sh | 3 +++\n t/t0470-read-object-http-e-odb.sh | 3 +++\n t/t0480-read-object-have-http-e-odb.sh | 3 +++\n 8 files changed, 52 insertions(+), 8 deletions(-)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex 084cd32e0b..e103514a46 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -1,6 +1,7 @@\n #include \"cache.h\"\n #include \"external-odb.h\"\n #include \"odb-helper.h\"\n+#include \"attr.h\"\n \n static struct odb_helper *helpers;\n static struct odb_helper **helpers_tail = &helpers;\n@@ -155,8 +156,23 @@ int external_odb_get_object(const unsigned char *sha1)\n \treturn external_odb_do_get_object(sha1);\n }\n \n+static int has_odb_attrs(struct odb_helper *o, const char *path)\n+{\n+\tstatic struct attr_check *check;\n+\n+\tif (!check)\n+\t\tcheck = attr_check_initl(\"odb\", NULL);\n+\n+\tif (!git_check_attr(path, check)) {\n+\t\tconst char *value = check->items[0].value;\n+\t\treturn value ? !strcmp(o->name, value) : 0;\n+\t}\n+\treturn 0;\n+}\n+\n int external_odb_put_object(const void *buf, size_t len,\n-\t\t\t const char *type, unsigned char *sha1)\n+\t\t\t const char *type, unsigned char *sha1,\n+\t\t\t const char *path)\n {\n \tstruct odb_helper *o;\n \n@@ -164,12 +180,14 @@ int external_odb_put_object(const void *buf, size_t len,\n \t\treturn 1;\n \n \t/* For now accept only blobs */\n-\tif (strcmp(type, \"blob\"))\n+\tif (!path || strcmp(type, \"blob\"))\n \t\treturn 1;\n \n \texternal_odb_init();\n \n \tfor (o = helpers; o; o = o->next) {\n+\t\tif (!has_odb_attrs(o, path))\n+\t\t\tcontinue;\n \t\tint r = odb_helper_put_object(o, buf, len, type, sha1);\n \t\tif (r <= 0)\n \t\t\treturn r;\ndiff --git a/external-odb.h b/external-odb.h\nindex 247b131fd5..9bd7856b60 100644\n--- a/external-odb.h\n+++ b/external-odb.h\n@@ -6,6 +6,7 @@ int external_odb_has_object(const unsigned char *sha1);\n int external_odb_get_object(const unsigned char *sha1);\n int external_odb_get_direct(const unsigned char *sha1);\n int external_odb_put_object(const void *buf, size_t len,\n-\t\t\t const char *type, unsigned char *sha1);\n+\t\t\t const char *type, unsigned char *sha1,\n+\t\t\t const char *path);\n \n #endif /* EXTERNAL_ODB_H */\ndiff --git a/sha1_file.c b/sha1_file.c\nindex fb34f0b18d..75188850da 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -3487,7 +3487,9 @@ static int freshen_packed_object(const unsigned char *sha1)\n \treturn 1;\n }\n \n-int write_sha1_file(const void *buf, unsigned long len, const char *type, unsigned char *sha1)\n+static int write_sha1_file_with_path(const void *buf, unsigned long len,\n+\t\t\t\t const char *type, unsigned char *sha1,\n+\t\t\t\t const char *path)\n {\n \tchar hdr[32];\n \tint hdrlen = sizeof(hdr);\n@@ -3496,13 +3498,19 @@ int write_sha1_file(const void *buf, unsigned long len, const char *type, unsign\n \t * it out into .git/objects/??/?{38} file.\n \t */\n \twrite_sha1_file_prepare(buf, len, type, sha1, hdr, &hdrlen);\n-\tif (!external_odb_put_object(buf, len, type, sha1))\n+\tif (!external_odb_put_object(buf, len, type, sha1, path))\n \t\treturn 0;\n \tif (freshen_packed_object(sha1) || freshen_loose_object(sha1))\n \t\treturn 0;\n \treturn write_loose_object(sha1, hdr, hdrlen, buf, len, 0);\n }\n \n+int write_sha1_file(const void *buf, unsigned long len,\n+\t\t const char *type, unsigned char *sha1)\n+{\n+\twrite_sha1_file_with_path(buf, len, type, sha1, NULL);\n+}\n+\n int hash_sha1_file_literally(const void *buf, unsigned long len, const char *type,\n \t\t\t unsigned char *sha1, unsigned flags)\n {\n@@ -3637,7 +3645,8 @@ static int index_mem(unsigned char *sha1, void *buf, size_t size,\n \t}\n \n \tif (write_object)\n-\t\tret = write_sha1_file(buf, size, typename(type), sha1);\n+\t\tret = write_sha1_file_with_path(buf, size, typename(type),\n+\t\t\t\t\t\tsha1, path);\n \telse\n \t\tret = hash_sha1_file(buf, size, typename(type), sha1);\n \tif (re_allocated)\n@@ -3659,8 +3668,9 @@ static int index_stream_convert_blob(unsigned char *sha1, int fd,\n \t\t\t\t write_object ? safe_crlf : SAFE_CRLF_FALSE);\n \n \tif (write_object)\n-\t\tret = write_sha1_file(sbuf.buf, sbuf.len, typename(OBJ_BLOB),\n-\t\t\t\t sha1);\n+\t\tret = write_sha1_file_with_path(sbuf.buf, sbuf.len,\n+\t\t\t\t\t\ttypename(OBJ_BLOB),\n+\t\t\t\t\t\tsha1, path);\n \telse\n \t\tret = hash_sha1_file(sbuf.buf, sbuf.len, typename(OBJ_BLOB),\n \t\t\t\t sha1);\ndiff --git a/t/t0400-external-odb.sh b/t/t0400-external-odb.sh\nindex fa355bd7bb..f2c45a625c 100755\n--- a/t/t0400-external-odb.sh\n+++ b/t/t0400-external-odb.sh\n@@ -73,6 +73,9 @@ test_expect_success 'helper can add objects to alt repo' '\n \n test_expect_success 'commit adds objects to alt repo' '\n \ttest_config odb.magic.scriptCommand \"$HELPER\" &&\n+\techo \"* odb=magic\" >.gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git add .gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git commit -m \"Add .gitattributes\" &&\n \ttest_commit three &&\n \thash3=$(git ls-tree HEAD | grep three.t | cut -f1 | cut -d\\ -f3) &&\n \tcontent=$(cd alt-repo && git show \"$hash3\") &&\ndiff --git a/t/t0410-transfer-e-odb.sh b/t/t0410-transfer-e-odb.sh\nindex 065ec7d759..fd3e37918c 100755\n--- a/t/t0410-transfer-e-odb.sh\n+++ b/t/t0410-transfer-e-odb.sh\n@@ -111,6 +111,9 @@ test_expect_success 'setup other repo and its alternate repo' '\n '\n \n test_expect_success 'new blobs are put in first object store' '\n+\techo \"* odb=magic\" >.gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git add .gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git commit -m \"Add .gitattributes\" &&\n \ttest_commit one &&\n \thash1=$(git ls-tree HEAD | grep one.t | cut -f1 | cut -d\\ -f3) &&\n \tcontent=$(cd alt-repo1 && git show \"$hash1\") &&\ndiff --git a/t/t0420-transfer-http-e-odb.sh b/t/t0420-transfer-http-e-odb.sh\nindex f84fe950ec..d307af0457 100755\n--- a/t/t0420-transfer-http-e-odb.sh\n+++ b/t/t0420-transfer-http-e-odb.sh\n@@ -94,6 +94,9 @@ test_expect_success 'can delete uploaded files' '\n FILES_DIR=\"httpd/www/files\"\n \n test_expect_success 'new blobs are transfered to the http server' '\n+\techo \"* odb=magic\" >.gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git add .gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git commit -m \"Add .gitattributes\" &&\n \ttest_commit one &&\n \thash1=$(git ls-tree HEAD | grep one.t | cut -f1 | cut -d\\ -f3) &&\n \techo \"$hash1-4-blob\" >expected &&\ndiff --git a/t/t0470-read-object-http-e-odb.sh b/t/t0470-read-object-http-e-odb.sh\nindex 774528c04f..d814a43d59 100755\n--- a/t/t0470-read-object-http-e-odb.sh\n+++ b/t/t0470-read-object-http-e-odb.sh\n@@ -62,6 +62,9 @@ test_expect_success 'can delete uploaded files' '\n FILES_DIR=\"httpd/www/files\"\n \n test_expect_success 'new blobs are transfered to the http server' '\n+\techo \"* odb=magic\" >.gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git add .gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git commit -m \"Add .gitattributes\" &&\n \ttest_commit one &&\n \thash1=$(git ls-tree HEAD | grep one.t | cut -f1 | cut -d\\ -f3) &&\n \techo \"$hash1-4-blob\" >expected &&\ndiff --git a/t/t0480-read-object-have-http-e-odb.sh b/t/t0480-read-object-have-http-e-odb.sh\nindex 056a40f2bb..fe1fac5ef3 100755\n--- a/t/t0480-read-object-have-http-e-odb.sh\n+++ b/t/t0480-read-object-have-http-e-odb.sh\n@@ -62,6 +62,9 @@ test_expect_success 'can delete uploaded files' '\n FILES_DIR=\"httpd/www/files\"\n \n test_expect_success 'new blobs are transfered to the http server' '\n+\techo \"* odb=magic\" >.gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git add .gitattributes &&\n+\tGIT_NO_EXTERNAL_ODB=1 git commit -m \"Add .gitattributes\" &&\n \ttest_commit one &&\n \thash1=$(git ls-tree HEAD | grep one.t | cut -f1 | cut -d\\ -f3) &&\n \techo \"$hash1-4-blob\" >expected &&\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325521","messageId":"20170803091926.1755-38-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 37/40] clone: add --initial-refspec option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:19:23Z","receivedAt":"2017-08-03T09:21:02Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This option makes it possible to separate fetching refs when cloning\nin two parts, an initial part and a later normal part.\n\nThis way after the initial part, mechanisms like the external odb\nmechanism can be used to prefetch some objects using information\nthat has been made available during the initial fetch.\n\nSigned-off-by: Christian Couder \n---\n builtin/clone.c | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 54 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 2362dda880..76e561534d 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -56,6 +56,7 @@ static enum transport_family family;\n static struct string_list option_config = STRING_LIST_INIT_NODUP;\n static struct string_list option_required_reference = STRING_LIST_INIT_NODUP;\n static struct string_list option_optional_reference = STRING_LIST_INIT_NODUP;\n+static struct string_list option_initial_refspec = STRING_LIST_INIT_NODUP;\n static int option_dissociate;\n static int max_jobs = -1;\n static struct string_list option_recurse_submodules = STRING_LIST_INIT_NODUP;\n@@ -106,6 +107,8 @@ static struct option builtin_clone_options[] = {\n \t\t\tN_(\"reference repository\")),\n \tOPT_STRING_LIST(0, \"reference-if-able\", &option_optional_reference,\n \t\t\tN_(\"repo\"), N_(\"reference repository\")),\n+\tOPT_STRING_LIST(0, \"initial-refspec\", &option_initial_refspec,\n+\t\t\tN_(\"refspec\"), N_(\"fetch this refspec first\")),\n \tOPT_BOOL(0, \"dissociate\", &option_dissociate,\n \t\t N_(\"use --reference only while cloning\")),\n \tOPT_STRING('o', \"origin\", &option_origin, N_(\"name\"),\n@@ -865,6 +868,47 @@ static void dissociate_from_references(void)\n \tfree(alternates);\n }\n \n+static struct refspec *parse_initial_refspecs(void)\n+{\n+\tconst char **refspecs;\n+\tstruct refspec *initial_refspecs;\n+\tstruct string_list_item *rs;\n+\tint i = 0;\n+\n+\tif (!option_initial_refspec.nr)\n+\t\treturn NULL;\n+\n+\trefspecs = xcalloc(option_initial_refspec.nr, sizeof(const char *));\n+\n+\tfor_each_string_list_item(rs, &option_initial_refspec)\n+\t\trefspecs[i++] = rs->string;\n+\n+\tinitial_refspecs = parse_fetch_refspec(option_initial_refspec.nr, refspecs);\n+\n+\tfree(refspecs);\n+\n+\treturn initial_refspecs;\n+}\n+\n+static void fetch_initial_refs(struct transport *transport,\n+\t\t\t const struct ref *refs,\n+\t\t\t struct refspec *initial_refspecs,\n+\t\t\t const char *branch_top,\n+\t\t\t const char *reflog_msg,\n+\t\t\t int is_local)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < option_initial_refspec.nr; i++) {\n+\t\tstruct ref *init_refs = NULL;\n+\t\tstruct ref **tail = &init_refs;\n+\t\tget_fetch_map(refs, &initial_refspecs[i], &tail, 0);\n+\t\ttransport_fetch_refs(transport, init_refs);\n+\t\tupdate_remote_refs(refs, init_refs, NULL, branch_top, reflog_msg,\n+\t\t\t\t transport, !is_local, 1);\n+\t}\n+}\n+\n int cmd_clone(int argc, const char **argv, const char *prefix)\n {\n \tint is_bundle = 0, is_local;\n@@ -888,6 +932,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tstruct refspec *refspec;\n \tconst char *fetch_pattern;\n \n+\tstruct refspec *initial_refspecs;\n+\tint is_initial;\n+\n \tpacket_trace_identity(\"clone\");\n \targc = parse_options(argc, argv, prefix, builtin_clone_options,\n \t\t\t builtin_clone_usage, 0);\n@@ -1055,6 +1102,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (option_required_reference.nr || option_optional_reference.nr)\n \t\tsetup_reference();\n \n+\tinitial_refspecs = parse_initial_refspecs();\n+\n \tfetch_pattern = xstrfmt(\"+%s*:%s*\", src_ref_prefix, branch_top.buf);\n \trefspec = parse_fetch_refspec(1, &fetch_pattern);\n \tfree((char *)fetch_pattern);\n@@ -1110,6 +1159,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \trefs = transport_get_remote_refs(transport);\n \n \tif (refs) {\n+\t\tfetch_initial_refs(transport, refs, initial_refspecs,\n+\t\t\t\t branch_top.buf, reflog_msg.buf, is_local);\n+\n \t\tmapped_refs = wanted_peer_refs(refs, refspec);\n \t\t/*\n \t\t * transport_get_remote_refs() may return refs with null sha-1\n@@ -1169,9 +1221,10 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \telse if (refs && complete_refs_before_fetch)\n \t\ttransport_fetch_refs(transport, mapped_refs);\n \n+\tis_initial = !refs || option_initial_refspec.nr == 0;\n \tupdate_remote_refs(refs, mapped_refs, remote_head_points_at,\n \t\t\t branch_top.buf, reflog_msg.buf, transport,\n-\t\t\t !is_local, 0);\n+\t\t\t !is_local, is_initial);\n \n \tupdate_head(our_head_points_at, remote_head, reflog_msg.buf);\n \n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325522","messageId":"20170803091926.1755-37-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 36/40] clone: add 'initial' param to write_remote_refs()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:19:22Z","receivedAt":"2017-08-03T09:21:04Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"We want to make it possible to separate fetching remote refs into\nan initial part and a later part. To prepare for that, let's add\nan 'initial' boolean parameter to write_remote_refs() to tell this\nfunction if we are performing the initial part or not.\n\nSigned-off-by: Christian Couder \n---\n builtin/clone.c | 19 +++++++++++++------\n 1 file changed, 13 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 4b5340c55f..2362dda880 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -573,7 +573,7 @@ static struct ref *wanted_peer_refs(const struct ref *refs,\n \treturn local_refs;\n }\n \n-static void write_remote_refs(const struct ref *local_refs)\n+static void write_remote_refs(const struct ref *local_refs, int initial)\n {\n \tconst struct ref *r;\n \n@@ -592,8 +592,13 @@ static void write_remote_refs(const struct ref *local_refs)\n \t\t\tdie(\"%s\", err.buf);\n \t}\n \n-\tif (initial_ref_transaction_commit(t, &err))\n-\t\tdie(\"%s\", err.buf);\n+\tif (initial) {\n+\t\tif (initial_ref_transaction_commit(t, &err))\n+\t\t\tdie(\"%s\", err.buf);\n+\t} else {\n+\t\tif (ref_transaction_commit(t, &err))\n+\t\t\tdie(\"%s\", err.buf);\n+\t}\n \n \tstrbuf_release(&err);\n \tref_transaction_free(t);\n@@ -640,7 +645,8 @@ static void update_remote_refs(const struct ref *refs,\n \t\t\t const char *branch_top,\n \t\t\t const char *msg,\n \t\t\t struct transport *transport,\n-\t\t\t int check_connectivity)\n+\t\t\t int check_connectivity,\n+\t\t\t int initial)\n {\n \tconst struct ref *rm = mapped_refs;\n \n@@ -655,7 +661,7 @@ static void update_remote_refs(const struct ref *refs,\n \t}\n \n \tif (refs) {\n-\t\twrite_remote_refs(mapped_refs);\n+\t\twrite_remote_refs(mapped_refs, initial);\n \t\tif (option_single_branch && !option_no_tags)\n \t\t\twrite_followtags(refs, msg);\n \t}\n@@ -1164,7 +1170,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\ttransport_fetch_refs(transport, mapped_refs);\n \n \tupdate_remote_refs(refs, mapped_refs, remote_head_points_at,\n-\t\t\t branch_top.buf, reflog_msg.buf, transport, !is_local);\n+\t\t\t branch_top.buf, reflog_msg.buf, transport,\n+\t\t\t !is_local, 0);\n \n \tupdate_head(our_head_points_at, remote_head, reflog_msg.buf);\n \n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325523","messageId":"20170803091926.1755-36-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 35/40] Add Documentation/technical/external-odb.txt","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:19:21Z","receivedAt":"2017-08-03T09:21:07Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This describes the external odb mechanism's purpose and\nhow it works.\n\nHelped-by: Ben Peart \nSigned-off-by: Christian Couder \n---\n Documentation/technical/external-odb.txt | 295 +++++++++++++++++++++++++++++++\n 1 file changed, 295 insertions(+)\n create mode 100644 Documentation/technical/external-odb.txt\n\ndiff --git a/Documentation/technical/external-odb.txt b/Documentation/technical/external-odb.txt\nnew file mode 100644\nindex 0000000000..5991221fd5\n--- /dev/null\n+++ b/Documentation/technical/external-odb.txt\n@@ -0,0 +1,295 @@\n+External ODBs\n+^^^^^^^^^^^^^\n+\n+The External ODB mechanism makes it possible for Git objects, mostly\n+blobs for now though, to be stored in an \"external object database\"\n+(External ODB).\n+\n+An External ODB can be any object store as long as there is an helper\n+program called an \"odb helper\" that can communicate with Git to\n+transfer objects to/from the external odb and to retrieve information\n+about available objects in the external odb.\n+\n+Purpose\n+=======\n+\n+The purpose of this mechanism is to make possible to handle Git\n+objects, especially blobs, in much more flexible ways.\n+\n+Currently Git can store its objects only in the form of loose objects\n+in separate files or packed objects in a pack file.\n+\n+This is not flexible enough for some important use cases like handling\n+really big binary files or handling a really big number of files that\n+are fetched only as needed. And it is not realistic to expect that Git\n+could fully natively handle many of such use cases.\n+\n+Furthermore many improvements that are dependent on specific setups\n+could be implemented in the way Git objects are managed if it was\n+possible to customize how the Git objects are handled. For example a\n+restartable clone using the bundle mechanism has often been requested,\n+but implementing that would go against the current strict rules under\n+which the Git objects are currently handled.\n+\n+What Git needs a mechanism to make it possible to customize in a lot\n+of different ways how the Git objects are handled. Though this\n+mechanism should try as much as possible to avoid interfering with the\n+usual way in which Git handle its objects.\n+\n+Helpers\n+=======\n+\n+ODB helpers are commands that have to be registered using either the\n+\"odb..subprocessCommand\" or the \"odb..scriptCommand\"\n+config variables.\n+\n+Registering such a command tells Git that an external odb called\n+ exists and that the registered command should be used to\n+communicate with it.\n+\n+There are 2 kinds of commands. Commands registered using the\n+\"odb..subprocessCommand\" config variable are called \"process\n+commands\" and the associated mode is called \"process mode\". Commands\n+registered using the \"odb..scriptCommand\" config variables\n+are called \"script commands\" and the associated mode is called \"script\n+mode\".\n+\n+Process Mode\n+============\n+\n+In process mode the command is started as a single process invocation\n+that should last for the entire life of the single Git command that\n+started it.\n+\n+A packet format (pkt-line, see technical/protocol-common.txt) based\n+protocol over standard input and standard output is used for\n+communication between Git and the helper command.\n+\n+After the process command is started, Git sends a welcome message\n+(\"git-read-object-client\"), a list of supported protocol version\n+numbers, and a flush packet. Git expects to read a welcome response\n+message (\"git-read-object-server\"), exactly one protocol version\n+number from the previously sent list, and a flush packet. All further\n+communication will be based on the selected version.\n+\n+The remaining protocol description below documents \"version=1\". Please\n+note that \"version=42\" in the example below does not exist and is only\n+there to illustrate how the protocol would look with more than one\n+version.\n+\n+After the version negotiation Git sends a list of all capabilities\n+that it supports and a flush packet. Git expects to read a list of\n+desired capabilities, which must be a subset of the supported\n+capabilities list, and a flush packet as response:\n+\n+------------------------\n+packet: git> git-read-object-client\n+packet: git> version=1\n+packet: git> version=42\n+packet: git> 0000\n+packet: git< git-read-object-server\n+packet: git< version=1\n+packet: git< 0000\n+packet: git> capability=get_raw_obj\n+packet: git> capability=have\n+packet: git> capability=put_raw_obj\n+packet: git> capability=not-yet-invented\n+packet: git> 0000\n+packet: git< capability=get_raw_obj\n+packet: git< 0000\n+------------------------\n+\n+Afterwards Git sends a list of \"key=value\" pairs terminated with a\n+flush packet. The list will contain at least the instruction (based on\n+the supported capabilities) and the arguments for the\n+instruction. Please note, that the process must not send any response\n+before it received the final flush packet.\n+\n+In general any response from the helper should end with a status\n+packet. See the documentation of the 'get_*' instructions below for\n+examples of status packets.\n+\n+After the helper has processed an instruction, it is expected to wait\n+for the next \"key=value\" list containing another instruction.\n+\n+On exit Git will close the pipe to the helper. The helper is then\n+expected to detect EOF and exit gracefully on its own. Git will wait\n+until the process has stopped.\n+\n+Script Mode\n+===========\n+\n+In this mode Git launches the script command each time it wants to\n+communicates with the helper. There is no welcome message and no\n+protocol version in this mode.\n+\n+The instruction and associated arguments are passed as arguments when\n+launching the script command and if needed further information is\n+passed between Git and the command through stdin and stdout.\n+\n+Capabilities/Instructions\n+=========================\n+\n+The following instructions are currently supported by Git:\n+\n+- init\n+- get_git_obj\n+- get_raw_obj\n+- get_direct\n+- put_raw_obj\n+- have\n+\n+The plan is to also support 'put_git_obj' and 'put_direct' soon, for\n+consistency with the 'get_*' instructions.\n+\n+ - 'init'\n+\n+All the process and script commands must accept the 'init'\n+instruction. It should be the first instruction sent to a command. It\n+should not be advertised in the capability exchange. Any argument\n+should be ignored.\n+\n+In process mode, after receiving the 'init' instruction and a flush\n+packet, the helper should just send a status packet and then a flush\n+packet. See the 'get_*' instructions below for examples of status\n+packets.\n+\n+In script mode the command should print on stdout the capabilities\n+that it supports if any. This is the only time in script mode when a\n+capability exchange happens.\n+\n+For example a script command could use the following shell code\n+snippet to handle the 'init' instruction:\n+\n+------------------------\n+case \"$1\" in\n+init)\n+\techo \"capability=get_git_obj\"\n+\techo \"capability=put_raw_obj\"\n+\techo \"capability=have\"\n+\t;;\n+------------------------\n+\n+ - 'get_git_obj ' and 'get_raw_obj '\n+\n+These instructions should have a hexadecimal argument to tell\n+which object the helper should send to git.\n+\n+In process mode the sha1 argument should be followed by a flush packet\n+like this:\n+\n+------------------------\n+packet: git> command=get_git_obj\n+packet: git> sha1=0a214a649e1b3d5011e14a3dc227753f2bd2be05\n+packet: git> 0000\n+------------------------\n+\n+After reading that the helper should send the requested object to Git in a\n+packet series followed by a flush packet. If the helper does not experience\n+problems then the helper must send a \"success\" status like the following:\n+\n+------------------------\n+packet: git< status=success\n+packet: git< 0000\n+------------------------\n+\n+In case the helper cannot or does not want to send the requested\n+object as well as any other object for the lifetime of the Git\n+process, then it is expected to respond with an \"abort\" status at any\n+point in the protocol:\n+\n+------------------------\n+packet: git< status=abort\n+packet: git< 0000\n+------------------------\n+\n+Git neither stops nor restarts the helper in case the \"error\"/\"abort\"\n+status is set.\n+\n+If the helper dies during the communication or does not adhere to the\n+protocol then Git will stop and restart it with the next instruction.\n+\n+In script mode the helper should just send the requested object to Git\n+by writing it to stdout and should then exit. The exit code should\n+signal to Git if a problem occured or not.\n+\n+The only difference between 'get_git_obj' and 'get_raw_obj' is that in\n+case of 'get_git_obj' the requested object should be sent as a Git\n+object (that is in the same format as loose object files). In case of\n+'get_raw_obj' the object should be sent in its raw format (that is the\n+same output as `git cat-file `).\n+\n+ - 'get_direct '\n+\n+This instruction is similar as the other 'get_*' instructions except\n+that no object should be sent from the helper to Git. Instead the\n+helper should directly write the requested object into a loose object\n+file in the \".git/objects\" directory.\n+\n+After the helper has sent the \"status=success\" packet and the\n+following flush packet in process mode, or after it has exited in the\n+script mode, Git should lookup again for a loose object file with the\n+requested sha1.\n+\n+ - 'put_raw_obj '\n+\n+This instruction should be following by three arguments to tell which\n+object the helper will receive from git: , and\n+. The hexadecimal argument describes the object that will\n+be sent from Git to the helper. The is the object type (blob,\n+tree, commit or tag) of this object. The is the size of the\n+(decompressed) object content.\n+\n+In process mode the last argument (the type) should be followed by a\n+flush packet.\n+\n+After reading that the helper should read the announced object from\n+Git in a packet series followed by a flush packet.\n+\n+If the helper does not experience problems when receiving and storing\n+or processing the object, then the helper must send a \"success\" status\n+as described for the 'get_*' instructions.\n+\n+In script mode the helper should just receive the announced object\n+from its standard input. After receiving and processing the object,\n+the helper should exit and its exit code should signal to Git if a\n+problem occured or not.\n+\n+- 'have'\n+\n+In process mode this instruction should be followed by a flush\n+packet. After receiving this packet the helper should send the sha1,\n+size and type, in this order, of all the objects it can provide to Git\n+(through a 'get_*' instruction). There should be a space character\n+between the sha1 and the size and between the size and the type, and\n+then a new line character after the type.\n+\n+If the helper does not experience problems, then it must then send a\n+\"success\" status as described for the 'get_*' instructions.\n+\n+In script mode the helper should send to its standard output the sha1,\n+size and type, in this order of all the objects it can provide to\n+Git. There should also be a space character between the sha1 and the\n+size and between the size and the type, and then a new line character\n+after the type.\n+\n+After sending this, the script helper should exit and its exit code\n+should signal to Git if a problem occured or not.\n+\n+Selecting objects\n+=================\n+\n+To select objects that should be handled by an external odb, one can\n+use the git attributes system. For now this will only work will blobs\n+and this will only work along with the 'put_raw_obj' instruction.\n+\n+For example if one has an external odb called \"magic\" and has\n+registered an associated a process command helper that supports the\n+'put_raw_obj' instruction, then one can tell Git that all the .jpg\n+files should be handled by the \"magic\" odb using a .gitattributes file\n+can that contains:\n+\n+------------------------\n+*.jpg odb=magic\n+------------------------\n+\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325524","messageId":"20170803091926.1755-31-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 30/40] odb-helper: add put_object_process()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:19:16Z","receivedAt":"2017-08-03T09:21:09Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This adds the infrastructure to send objects to a sub-process\nhandling the communication with an external odb.\n\nFor now we only handle sending raw blobs using the 'put_raw_obj'\ninstruction.\n\nSigned-off-by: Christian Couder \n---\n odb-helper.c | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++---\n 1 file changed, 72 insertions(+), 3 deletions(-)\n\ndiff --git a/odb-helper.c b/odb-helper.c\nindex fce1dff501..db90c0a004 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -445,6 +445,58 @@ static int get_object_process(struct odb_helper *o, const unsigned char *sha1, i\n \t\t\t\t\t o->cmd, cur_cap);\n }\n \n+static int send_put_packets(struct object_process *entry,\n+\t\t\t const unsigned char *sha1,\n+\t\t\t const void *buf,\n+\t\t\t size_t len,\n+\t\t\t struct strbuf *status)\n+{\n+\tstruct child_process *process = &entry->subprocess.process;\n+\tint err = packet_write_fmt_gently(process->in, \"command=put_raw_obj\\n\");\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = packet_write_fmt_gently(process->in, \"sha1=%s\\n\", sha1_to_hex(sha1));\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = packet_write_fmt_gently(process->in, \"size=%\"PRIuMAX\"\\n\", len);\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = packet_write_fmt_gently(process->in, \"kind=blob\\n\");\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = packet_flush_gently(process->in);\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = write_packetized_from_buf(buf, len, process->in);\n+\tif (err)\n+\t\treturn err;\n+\n+\treturn check_object_process_status(process->out, status);\n+}\n+\n+static int put_object_process(struct odb_helper *o,\n+\t\t\t const void *buf, size_t len,\n+\t\t\t const char *type, unsigned char *sha1)\n+{\n+\tint err;\n+\tstruct object_process *entry;\n+\tstruct strbuf status = STRBUF_INIT;\n+\n+\tentry = launch_object_process(o, ODB_HELPER_CAP_PUT_RAW_OBJ);\n+\tif (!entry)\n+\t\treturn -1;\n+\n+\terr = send_put_packets(entry, sha1, buf, len, &status);\n+\n+\treturn check_object_process_error(err, status.buf, entry, o->cmd,\n+\t\t\t\t\t ODB_HELPER_CAP_PUT_RAW_OBJ);\n+}\n+\n struct odb_helper *odb_helper_new(const char *name, int namelen)\n {\n \tstruct odb_helper *o;\n@@ -896,9 +948,9 @@ int odb_helper_get_object(struct odb_helper *o,\n \treturn res;\n }\n \n-int odb_helper_put_object(struct odb_helper *o,\n-\t\t\t const void *buf, size_t len,\n-\t\t\t const char *type, unsigned char *sha1)\n+static int put_raw_object_script(struct odb_helper *o,\n+\t\t\t\t const void *buf, size_t len,\n+\t\t\t\t const char *type, unsigned char *sha1)\n {\n \tstruct odb_helper_cmd cmd;\n \n@@ -924,3 +976,20 @@ int odb_helper_put_object(struct odb_helper *o,\n \todb_helper_finish(o, &cmd);\n \treturn 0;\n }\n+\n+int odb_helper_put_object(struct odb_helper *o,\n+\t\t\t const void *buf, size_t len,\n+\t\t\t const char *type, unsigned char *sha1)\n+{\n+\tint res;\n+\tuint64_t start = getnanotime();\n+\n+\tif (o->script_mode)\n+\t\tres = put_raw_object_script(o, buf, len, type, sha1);\n+\telse\n+\t\tres = put_object_process(o, buf, len, type, sha1);\n+\n+\ttrace_performance_since(start, \"odb_helper_put_object\");\n+\n+\treturn res;\n+}\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325525","messageId":"20170803091926.1755-33-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 32/40] odb-helper: add have_object_process()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:19:18Z","receivedAt":"2017-08-03T09:21:15Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This adds the infrastructure to handle 'have' instructions in\nprocess mode.\n\nThe answer from the helper sub-process should be like the\noutput in script mode, that is lines like this:\n\nsha1 SPACE size SPACE type NEWLINE\n\nSigned-off-by: Christian Couder \n---\n odb-helper.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 72 insertions(+)\n\ndiff --git a/odb-helper.c b/odb-helper.c\nindex db90c0a004..31fb398469 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -646,6 +646,70 @@ static int odb_helper_object_cmp(const void *va, const void *vb)\n \treturn hashcmp(a->sha1, b->sha1);\n }\n \n+static int send_have_packets(struct odb_helper *o,\n+\t\t\t struct object_process *entry,\n+\t\t\t struct strbuf *status)\n+{\n+\tchar *line;\n+\tint packet_len;\n+\tint total_got = 0;\n+\tstruct child_process *process = &entry->subprocess.process;\n+\tint err = packet_write_fmt_gently(process->in, \"command=have\\n\");\n+\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = packet_flush_gently(process->in);\n+\tif (err)\n+\t\treturn err;\n+\n+\tfor (;;) {\n+\t\t/* packet_read() writes a '\\0' extra byte at the end */\n+\t\tchar buf[LARGE_PACKET_DATA_MAX + 1];\n+\t\tchar *p = buf;\n+\t\tint more;\n+\n+\t\tpacket_len = packet_read(process->out, NULL, NULL,\n+\t\t\tbuf, LARGE_PACKET_DATA_MAX + 1,\n+\t\t\tPACKET_READ_GENTLE_ON_EOF);\n+\n+\t\tif (packet_len <= 0)\n+\t\t\tbreak;\n+\n+\t\ttotal_got += packet_len;\n+\n+\t\tdo {\n+\t\t\tchar *eol = strchrnul(p, '\\n');\n+\t\t\tmore = (*eol == '\\n');\n+\t\t\t*eol = '\\0';\n+\t\t\tif (add_have_entry(o, p))\n+\t\t\t\tbreak;\n+\t\t\tp = eol + 1;\n+\t\t} while (more);\n+\t}\n+\n+\tif (packet_len < 0)\n+\t\treturn packet_len;\n+\n+\treturn check_object_process_status(process->out, status);\n+}\n+\n+static int have_object_process(struct odb_helper *o)\n+{\n+\tint err;\n+\tstruct object_process *entry;\n+\tstruct strbuf status = STRBUF_INIT;\n+\n+\tentry = launch_object_process(o, ODB_HELPER_CAP_HAVE);\n+\tif (!entry)\n+\t\treturn -1;\n+\n+\terr = send_have_packets(o, entry, &status);\n+\n+\treturn check_object_process_error(err, status.buf, entry, o->cmd,\n+\t\t\t\t\t ODB_HELPER_CAP_HAVE);\n+}\n+\n static void have_object_script(struct odb_helper *o)\n {\n \tstruct odb_helper_cmd cmd;\n@@ -667,12 +731,20 @@ static void have_object_script(struct odb_helper *o)\n \n static void odb_helper_load_have(struct odb_helper *o)\n {\n+\tuint64_t start;\n+\n \tif (o->have_valid)\n \t\treturn;\n \to->have_valid = 1;\n \n+\tstart = getnanotime();\n+\n \tif (o->script_mode)\n \t\thave_object_script(o);\n+\telse\n+\t\thave_object_process(o);\n+\n+\ttrace_performance_since(start, \"odb_helper_load_have\");\n \n \tqsort(o->have, o->have_nr, sizeof(*o->have), odb_helper_object_cmp);\n }\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325526","messageId":"20170803091926.1755-39-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 38/40] clone: disable external odb before initial clone","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:19:24Z","receivedAt":"2017-08-03T09:21:17Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"To make it possible to have the external odb mechanism only kick in\nafter the initial part of a clone, we should disable it during the\ninitial part of the clone.\n\nLet's do that by saving and then restoring the value of the\n'use_external_odb' global variable.\n\nSigned-off-by: Christian Couder \n---\n builtin/clone.c | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 76e561534d..dc57eabd40 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -934,6 +934,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \n \tstruct refspec *initial_refspecs;\n \tint is_initial;\n+\tint saved_use_external_odb;\n \n \tpacket_trace_identity(\"clone\");\n \targc = parse_options(argc, argv, prefix, builtin_clone_options,\n@@ -1079,6 +1080,10 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \n \tgit_config(git_default_config, NULL);\n \n+\t/* Temporarily disable external ODB before initial clone */\n+\tsaved_use_external_odb = use_external_odb;\n+\tuse_external_odb = 0;\n+\n \tif (option_bare) {\n \t\tif (option_mirror)\n \t\t\tsrc_ref_prefix = \"refs/\";\n@@ -1162,6 +1167,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\tfetch_initial_refs(transport, refs, initial_refspecs,\n \t\t\t\t branch_top.buf, reflog_msg.buf, is_local);\n \n+\t\tuse_external_odb = saved_use_external_odb;\n+\n \t\tmapped_refs = wanted_peer_refs(refs, refspec);\n \t\t/*\n \t\t * transport_get_remote_refs() may return refs with null sha-1\n@@ -1203,6 +1210,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\t\t\t\toption_branch, option_origin);\n \n \t\twarning(_(\"You appear to have cloned an empty repository.\"));\n+\n+\t\tuse_external_odb = saved_use_external_odb;\n+\n \t\tmapped_refs = NULL;\n \t\tour_head_points_at = NULL;\n \t\tremote_head_points_at = NULL;\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325527","messageId":"20170803091926.1755-40-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 39/40] Add tests for 'clone --initial-refspec'","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:19:25Z","receivedAt":"2017-08-03T09:21:19Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n t/t0420-transfer-http-e-odb.sh | 7 +++++\n t/t0470-read-object-http-e-odb.sh | 7 +++++\n t/t0480-read-object-have-http-e-odb.sh | 7 +++++\n t/t5616-clone-initial-refspec.sh | 48 ++++++++++++++++++++++++++++++++++\n 4 files changed, 69 insertions(+)\n create mode 100755 t/t5616-clone-initial-refspec.sh\n\ndiff --git a/t/t0420-transfer-http-e-odb.sh b/t/t0420-transfer-http-e-odb.sh\nindex d307af0457..ed833850c3 100755\n--- a/t/t0420-transfer-http-e-odb.sh\n+++ b/t/t0420-transfer-http-e-odb.sh\n@@ -140,6 +140,13 @@ test_expect_success 'no-local clone from the first repo with helper succeeds' '\n \trm -rf my-other-clone\n '\n \n+test_expect_success 'no-local initial-refspec clone succeeds' '\n+\tmkdir my-other-clone &&\n+\t(cd my-other-clone &&\n+\t git -c odb.magic.scriptCommand=\"$HELPER\" \\\n+\t\tclone --no-local --initial-refspec \"refs/odbs/magic/*:refs/odbs/magic/*\" .. .)\n+'\n+\n stop_httpd\n \n test_done\ndiff --git a/t/t0470-read-object-http-e-odb.sh b/t/t0470-read-object-http-e-odb.sh\nindex d814a43d59..7355ca4d51 100755\n--- a/t/t0470-read-object-http-e-odb.sh\n+++ b/t/t0470-read-object-http-e-odb.sh\n@@ -107,6 +107,13 @@ test_expect_success 'no-local clone from the first repo with helper succeeds' '\n \trm -rf my-other-clone\n '\n \n+test_expect_success 'no-local initial-refspec clone succeeds' '\n+\tmkdir my-other-clone &&\n+\t(cd my-other-clone &&\n+\t git -c odb.magic.subprocessCommand=\"$HELPER\" \\\n+\t\tclone --no-local --initial-refspec \"refs/odbs/magic/*:refs/odbs/magic/*\" .. .)\n+'\n+\n stop_httpd\n \n test_done\ndiff --git a/t/t0480-read-object-have-http-e-odb.sh b/t/t0480-read-object-have-http-e-odb.sh\nindex fe1fac5ef3..c451d269a7 100755\n--- a/t/t0480-read-object-have-http-e-odb.sh\n+++ b/t/t0480-read-object-have-http-e-odb.sh\n@@ -107,6 +107,13 @@ test_expect_success 'no-local clone from the first repo with helper succeeds' '\n \trm -rf my-other-clone\n '\n \n+test_expect_success 'no-local initial-refspec clone succeeds' '\n+\tmkdir my-other-clone &&\n+\t(cd my-other-clone &&\n+\t git -c odb.magic.subprocessCommand=\"$HELPER\" \\\n+\t\tclone --no-local --initial-refspec \"refs/odbs/magic/*:refs/odbs/magic/*\" .. .)\n+'\n+\n stop_httpd\n \n test_done\ndiff --git a/t/t5616-clone-initial-refspec.sh b/t/t5616-clone-initial-refspec.sh\nnew file mode 100755\nindex 0000000000..ccbc27f83f\n--- /dev/null\n+++ b/t/t5616-clone-initial-refspec.sh\n@@ -0,0 +1,48 @@\n+#!/bin/sh\n+\n+test_description='test clone with --initial-refspec option'\n+. ./test-lib.sh\n+\n+\n+test_expect_success 'setup regular repo' '\n+\t# Make two branches, \"master\" and \"side\"\n+\techo one >file &&\n+\tgit add file &&\n+\tgit commit -m one &&\n+\techo two >file &&\n+\tgit commit -a -m two &&\n+\tgit tag two &&\n+\techo three >file &&\n+\tgit commit -a -m three &&\n+\tgit checkout -b side &&\n+\techo four >file &&\n+\tgit commit -a -m four &&\n+\tgit checkout master\n+'\n+\n+test_expect_success 'add a special ref pointing to a blob' '\n+\thash=$(echo \"Hello world!\" | git hash-object -w -t blob --stdin) &&\n+\tgit update-ref refs/special/hello \"$hash\"\n+'\n+\n+test_expect_success 'no-local clone from the first repo' '\n+\tmkdir my-clone &&\n+\t(cd my-clone &&\n+\t git clone --no-local .. . &&\n+\t test_must_fail git cat-file blob \"$hash\") &&\n+\trm -rf my-clone\n+'\n+\n+test_expect_success 'no-local clone with --initial-refspec' '\n+\tmkdir my-clone &&\n+\t(cd my-clone &&\n+\t git clone --no-local --initial-refspec \"refs/special/*:refs/special/*\" .. . &&\n+\t git cat-file blob \"$hash\" &&\n+\t git rev-parse refs/special/hello >actual &&\n+\t echo \"$hash\" >expected &&\n+\t test_cmp expected actual) &&\n+\trm -rf my-clone\n+'\n+\n+test_done\n+\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325528","messageId":"20170803091926.1755-41-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 40/40] Add t0430 to test cloning using bundles","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:19:26Z","receivedAt":"2017-08-03T09:21:20Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n t/t0430-clone-bundle-e-odb.sh | 85 +++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 85 insertions(+)\n create mode 100755 t/t0430-clone-bundle-e-odb.sh\n\ndiff --git a/t/t0430-clone-bundle-e-odb.sh b/t/t0430-clone-bundle-e-odb.sh\nnew file mode 100755\nindex 0000000000..ac38ae1be5\n--- /dev/null\n+++ b/t/t0430-clone-bundle-e-odb.sh\n@@ -0,0 +1,85 @@\n+#!/bin/sh\n+\n+test_description='tests for cloning using a bundle through e-odb'\n+\n+. ./test-lib.sh\n+\n+# If we don't specify a port, the current test number will be used\n+# which will not work as it is less than 1024, so it can only be used by root.\n+LIB_HTTPD_PORT=$(expr ${this_test#t} + 12000)\n+\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+\n+start_httpd apache-e-odb.conf\n+\n+# odb helper script must see this\n+export HTTPD_URL\n+\n+write_script odb-clone-bundle-helper <<\\EOF\n+die() {\n+\tprintf >&2 \"%s\\n\" \"$@\"\n+\texit 1\n+}\n+echo >&2 \"odb-clone-bundle-helper args:\" \"$@\"\n+case \"$1\" in\n+init)\n+\tref_hash=$(git rev-parse refs/odbs/magic/bundle) ||\n+\tdie \"couldn't find refs/odbs/magic/bundle\"\n+\tGIT_NO_EXTERNAL_ODB=1 git cat-file blob \"$ref_hash\" >bundle_info ||\n+\tdie \"couldn't get blob $ref_hash\"\n+\tbundle_url=$(sed -e 's/bundle url: //' bundle_info)\n+\techo >&2 \"bundle_url: '$bundle_url'\"\n+\tcurl \"$bundle_url\" -o bundle_file ||\n+\tdie \"curl '$bundle_url' failed\"\n+\tGIT_NO_EXTERNAL_ODB=1 git bundle unbundle bundle_file >unbundling_info ||\n+\tdie \"unbundling 'bundle_file' failed\"\n+\t;;\n+get*)\n+\tdie \"odb-clone-bundle-helper '$1' called\"\n+\t;;\n+put*)\n+\tdie \"odb-clone-bundle-helper '$1' called\"\n+\t;;\n+*)\n+\tdie \"unknown command '$1'\"\n+\t;;\n+esac\n+EOF\n+HELPER=\"\\\"$PWD\\\"/odb-clone-bundle-helper\"\n+\n+\n+test_expect_success 'setup repo with a few commits' '\n+\ttest_commit one &&\n+\ttest_commit two &&\n+\ttest_commit three &&\n+\ttest_commit four\n+'\n+\n+BUNDLE_FILE=\"file.bundle\"\n+FILES_DIR=\"httpd/www/files\"\n+GET_URL=\"$HTTPD_URL/files/$BUNDLE_FILE\"\n+\n+test_expect_success 'create a bundle for this repo and check that it can be downloaded' '\n+\tgit bundle create \"$BUNDLE_FILE\" master &&\n+\tmkdir \"$FILES_DIR\" &&\n+\tcp \"$BUNDLE_FILE\" \"$FILES_DIR/\" &&\n+\tcurl \"$GET_URL\" --output actual &&\n+\ttest_cmp \"$BUNDLE_FILE\" actual\n+'\n+\n+test_expect_success 'create an e-odb ref for this bundle' '\n+\tref_hash=$(echo \"bundle url: $GET_URL\" | GIT_NO_EXTERNAL_ODB=1 git hash-object -w -t blob --stdin) &&\n+\tgit update-ref refs/odbs/magic/bundle \"$ref_hash\"\n+'\n+\n+test_expect_success 'clone using the e-odb helper to download and install the bundle' '\n+\tmkdir my-clone &&\n+\t(cd my-clone &&\n+\t git clone --no-local \\\n+\t\t-c odb.magic.scriptCommand=\"$HELPER\" \\\n+\t\t--initial-refspec \"refs/odbs/magic/*:refs/odbs/magic/*\" .. .)\n+'\n+\n+stop_httpd\n+\n+test_done\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325529","messageId":"20170803091926.1755-32-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 31/40] Add t0470 to test passing raw objects","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:19:17Z","receivedAt":"2017-08-03T09:21:26Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n t/t0470-read-object-http-e-odb.sh | 109 ++++++++++++++++++++++++++++++++++++++\n t/t0470/read-object-plain | 83 +++++++++++++++++++++++++++++\n 2 files changed, 192 insertions(+)\n create mode 100755 t/t0470-read-object-http-e-odb.sh\n create mode 100755 t/t0470/read-object-plain\n\ndiff --git a/t/t0470-read-object-http-e-odb.sh b/t/t0470-read-object-http-e-odb.sh\nnew file mode 100755\nindex 0000000000..774528c04f\n--- /dev/null\n+++ b/t/t0470-read-object-http-e-odb.sh\n@@ -0,0 +1,109 @@\n+#!/bin/sh\n+\n+test_description='tests for read-object process passing plain objects to an HTTPD server'\n+\n+. ./test-lib.sh\n+\n+# If we don't specify a port, the current test number will be used\n+# which will not work as it is less than 1024, so it can only be used by root.\n+LIB_HTTPD_PORT=$(expr ${this_test#t} + 12000)\n+\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+\n+start_httpd apache-e-odb.conf\n+\n+PATH=\"$PATH:$TEST_DIRECTORY/t0470\"\n+\n+# odb helper script must see this\n+export HTTPD_URL\n+\n+HELPER=\"read-object-plain\"\n+\n+test_expect_success 'setup repo with a root commit' '\n+\ttest_commit zero\n+'\n+\n+test_expect_success 'setup another repo from the first one' '\n+\tgit init other-repo &&\n+\t(cd other-repo &&\n+\t git remote add origin .. &&\n+\t git pull origin master &&\n+\t git checkout master &&\n+\t git log)\n+'\n+\n+test_expect_success 'setup the helper in the root repo' '\n+\tgit config odb.magic.subprocessCommand \"$HELPER\"\n+'\n+\n+UPLOADFILENAME=\"hello_apache_upload.txt\"\n+\n+UPLOAD_URL=\"$HTTPD_URL/upload/?sha1=$UPLOADFILENAME&size=123&type=blob\"\n+\n+test_expect_success 'can upload a file' '\n+\techo \"Hello Apache World!\" >hello_to_send.txt &&\n+\techo \"How are you?\" >>hello_to_send.txt &&\n+\tcurl --data-binary @hello_to_send.txt --include \"$UPLOAD_URL\" >out_upload\n+'\n+\n+LIST_URL=\"$HTTPD_URL/list/\"\n+\n+test_expect_success 'can list uploaded files' '\n+\tcurl --include \"$LIST_URL\" >out_list &&\n+\tgrep \"$UPLOADFILENAME\" out_list\n+'\n+\n+test_expect_success 'can delete uploaded files' '\n+\tcurl --data \"delete\" --include \"$UPLOAD_URL&delete=1\" >out_delete &&\n+\tcurl --include \"$LIST_URL\" >out_list2 &&\n+\t! grep \"$UPLOADFILENAME\" out_list2\n+'\n+\n+FILES_DIR=\"httpd/www/files\"\n+\n+test_expect_success 'new blobs are transfered to the http server' '\n+\ttest_commit one &&\n+\thash1=$(git ls-tree HEAD | grep one.t | cut -f1 | cut -d\\ -f3) &&\n+\techo \"$hash1-4-blob\" >expected &&\n+\tls \"$FILES_DIR\" >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'blobs can be retrieved from the http server' '\n+\tgit cat-file blob \"$hash1\" &&\n+\tgit log -p >expected\n+'\n+\n+test_expect_success 'update other repo from the first one' '\n+\t(cd other-repo &&\n+\t git fetch origin \"refs/odbs/magic/*:refs/odbs/magic/*\" &&\n+\t test_must_fail git cat-file blob \"$hash1\" &&\n+\t git config odb.magic.subprocesscommand \"$HELPER\" &&\n+\t git cat-file blob \"$hash1\" &&\n+\t git pull origin master)\n+'\n+\n+test_expect_success 'local clone from the first repo' '\n+\tmkdir my-clone &&\n+\t(cd my-clone &&\n+\t git clone .. . &&\n+\t git cat-file blob \"$hash1\")\n+'\n+\n+test_expect_success 'no-local clone from the first repo fails' '\n+\tmkdir my-other-clone &&\n+\t(cd my-other-clone &&\n+\t test_must_fail git clone --no-local .. .) &&\n+\trm -rf my-other-clone\n+'\n+\n+test_expect_success 'no-local clone from the first repo with helper succeeds' '\n+\tmkdir my-other-clone &&\n+\t(cd my-other-clone &&\n+\t git clone -c odb.magic.subprocessCommand=\"$HELPER\" --no-local .. .) &&\n+\trm -rf my-other-clone\n+'\n+\n+stop_httpd\n+\n+test_done\ndiff --git a/t/t0470/read-object-plain b/t/t0470/read-object-plain\nnew file mode 100755\nindex 0000000000..918e7b00b5\n--- /dev/null\n+++ b/t/t0470/read-object-plain\n@@ -0,0 +1,83 @@\n+#!/usr/bin/perl\n+#\n+\n+use 5.008;\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n+use strict;\n+use warnings;\n+use Git::Packet;\n+use LWP::UserAgent;\n+use HTTP::Request::Common;\n+\n+packet_initialize(\"git-read-object\", 1);\n+\n+packet_read_and_check_capabilities(\"get_raw_obj\", \"put_raw_obj\");\n+packet_write_capabilities(\"get_raw_obj\", \"put_raw_obj\");\n+\n+my $http_url = $ENV{HTTPD_URL};\n+\n+while (1) {\n+\tmy ($res, $command) = packet_txt_read();\n+\n+\tif ( $res == -1 ) {\n+\t\texit 0;\n+\t}\n+\n+\t$command =~ s/^command=//;\n+\n+\tif ( $command eq \"init\" ) {\n+\t\tpacket_bin_read();\n+\n+\t\tpacket_txt_write(\"status=success\");\n+\t\tpacket_flush();\n+\t} elsif ( $command eq \"get_raw_obj\" ) {\n+\t\tmy ($sha1) = packet_txt_read() =~ /^sha1=([0-9a-f]{40})$/;\n+\t\tpacket_bin_read();\n+\n+\t\tmy $get_url = $http_url . \"/list/?sha1=\" . $sha1;\n+\n+\t\tmy $userAgent = LWP::UserAgent->new();\n+\n+\t\tmy $response = $userAgent->get( $get_url );\n+\n+\t\tif ($response->is_error) {\n+\t\t packet_txt_write(\"size=0\");\n+\t\t packet_txt_write(\"kind=none\");\t \n+\t\t packet_txt_write(\"status=notfound\");\n+\t\t} else {\n+\t\t packet_txt_write(\"size=\" . length($response->content));\n+\t\t packet_txt_write(\"kind=blob\");\n+\t\t packet_bin_write($response->content);\n+\t\t packet_flush();\n+\t\t packet_txt_write(\"status=success\");\n+\t\t}\n+\n+\t\tpacket_flush();\n+\t} elsif ( $command eq \"put_raw_obj\" ) {\n+\t\tmy ($sha1) = packet_txt_read() =~ /^sha1=([0-9a-f]{40})$/;\n+\t\tmy ($size) = packet_txt_read() =~ /^size=([0-9]+)$/;\n+\t\tmy ($kind) = packet_txt_read() =~ /^kind=(\\w+)$/;\n+\t\tpacket_bin_read();\n+\n+\t\t# We must read the content we are sent and send it to the right url\n+\t\tmy ($res, $buf) = packet_bin_read();\n+\t\tdie \"bad packet_bin_read res ($res)\" unless ($res eq 0);\n+\t\t( packet_bin_read() eq ( 1, \"\" ) ) || die \"bad send end\";\t\t\n+\n+\t\tmy $upload_url = $http_url . \"/upload/?sha1=\" . $sha1 . \"&size=\" . $size . \"&type=blob\";\n+\n+\t\tmy $userAgent = LWP::UserAgent->new();\n+\t\tmy $request = POST $upload_url, Content_Type => 'multipart/form-data', Content => $buf;\n+\n+\t\tmy $response = $userAgent->request($request);\n+\n+\t\tif ($response->is_error) {\n+\t\t\tpacket_txt_write(\"status=failure\");\n+\t\t} else {\n+\t\t\tpacket_txt_write(\"status=success\");\n+\t\t}\n+\t\tpacket_flush();\n+\t} else {\n+\t\tdie \"bad command '$command'\";\n+\t}\n+}\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325530","messageId":"20170803091926.1755-28-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 27/40] odb-helper: add init_object_process()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:19:13Z","receivedAt":"2017-08-03T09:21:45Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: Ben Peart \n\nThis adds the infrastructure to launch and use long running\nsub-processes as external odb helpers.\n\nFor now only the 'init' and 'get_direct' capabilities are\nsupported with sub-processes.\n\nSigned-off-by: Christian Couder \n---\n external-odb.c | 52 ++++---\n odb-helper.c | 481 +++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n sha1_file.c | 56 +++++--\n 3 files changed, 535 insertions(+), 54 deletions(-)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex ccca67eff5..084cd32e0b 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -67,32 +67,11 @@ const char *external_odb_root(void)\n \treturn root;\n }\n \n-int external_odb_has_object(const unsigned char *sha1)\n-{\n-\tstruct odb_helper *o;\n-\n-\tif (!use_external_odb)\n-\t\treturn 0;\n-\n-\texternal_odb_init();\n-\n-\tfor (o = helpers; o; o = o->next) {\n-\t\tif (!(o->supported_capabilities & ODB_HELPER_CAP_HAVE))\n-\t\t\treturn 1;\n-\t\tif (odb_helper_has_object(o, sha1))\n-\t\t\treturn 1;\n-\t}\n-\treturn 0;\n-}\n-\n-int external_odb_get_object(const unsigned char *sha1)\n+static int external_odb_do_get_object(const unsigned char *sha1)\n {\n \tstruct odb_helper *o;\n \tconst char *path;\n \n-\tif (!external_odb_has_object(sha1))\n-\t\treturn -1;\n-\n \tpath = sha1_file_name_alt(external_odb_root(), sha1);\n \tsafe_create_leading_directories_const(path);\n \tprepare_external_alt_odb();\n@@ -147,6 +126,35 @@ int external_odb_get_direct(const unsigned char *sha1)\n \treturn -1;\n }\n \n+int external_odb_has_object(const unsigned char *sha1)\n+{\n+\tstruct odb_helper *o;\n+\n+\tif (!use_external_odb)\n+\t\treturn 0;\n+\n+\texternal_odb_init();\n+\n+\tfor (o = helpers; o; o = o->next) {\n+\t\tif (!(o->supported_capabilities & ODB_HELPER_CAP_HAVE)) {\n+\t\t\tif (o->supported_capabilities & ODB_HELPER_CAP_GET_DIRECT)\n+\t\t\t\treturn 1;\n+\t\t\treturn !external_odb_do_get_object(sha1);\n+\t\t}\n+\t\tif (odb_helper_has_object(o, sha1))\n+\t\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+int external_odb_get_object(const unsigned char *sha1)\n+{\n+\tif (!external_odb_has_object(sha1))\n+\t\treturn -1;\n+\n+\treturn external_odb_do_get_object(sha1);\n+}\n+\n int external_odb_put_object(const void *buf, size_t len,\n \t\t\t const char *type, unsigned char *sha1)\n {\ndiff --git a/odb-helper.c b/odb-helper.c\nindex 4c16dd297a..fce1dff501 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -4,6 +4,22 @@\n #include \"odb-helper.h\"\n #include \"run-command.h\"\n #include \"sha1-lookup.h\"\n+#include \"sub-process.h\"\n+#include \"pkt-line.h\"\n+#include \"sigchain.h\"\n+\n+struct object_process {\n+\tstruct subprocess_entry subprocess;\n+\tunsigned int supported_capabilities;\n+};\n+\n+static struct hashmap subprocess_map;\n+\n+static int check_object_process_status(int fd, struct strbuf *status)\n+{\n+\tsubprocess_read_status(fd, status);\n+\treturn strcmp(status->buf, \"success\");\n+}\n \n static void parse_capabilities(char *cap_buf,\n \t\t\t unsigned int *supported_capabilities,\n@@ -39,6 +55,396 @@ static void parse_capabilities(char *cap_buf,\n \tstring_list_clear(&cap_list, 0);\n }\n \n+static int send_start_packets(struct child_process *process, const char *cmd)\n+{\n+\tint err = packet_writel(process->in, \"git-read-object-client\", \"version=1\", NULL);\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = strcmp(packet_read_line(process->out, NULL), \"git-read-object-server\");\n+\tif (err) {\n+\t\terror(\"external process '%s' does not support read-object protocol version 1\", cmd);\n+\t\treturn err;\n+\t}\n+\terr = strcmp(packet_read_line(process->out, NULL), \"version=1\");\n+\tif (err)\n+\t\treturn err;\n+\terr = packet_read_line(process->out, NULL) != NULL;\n+\tif (err)\n+\t\treturn err;\n+\n+\treturn packet_writel(process->in,\n+\t\t\t \"capability=get_git_obj\",\n+\t\t\t \"capability=get_raw_obj\",\n+\t\t\t \"capability=get_direct\",\n+\t\t\t \"capability=put_raw_obj\",\n+\t\t\t \"capability=have\",\n+\t\t\t NULL);\n+}\n+\n+static int start_object_process_fn(struct subprocess_entry *subprocess)\n+{\n+\tint err;\n+\tstruct object_process *entry = (struct object_process *)subprocess;\n+\tstruct child_process *process = &subprocess->process;\n+\tchar *cap_buf;\n+\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\n+\terr = send_start_packets(process, subprocess->cmd);\n+\n+\tif (!err)\n+\t\twhile ((cap_buf = packet_read_line(process->out, NULL)))\n+\t\t\tparse_capabilities(cap_buf, &entry->supported_capabilities, subprocess->cmd);\n+\n+\tsigchain_pop(SIGPIPE);\n+\n+\treturn err;\n+}\n+\n+static struct object_process *launch_object_process(struct odb_helper *o,\n+\t\t\t\t\t\t unsigned int capability)\n+{\n+\tstruct object_process *entry = NULL;\n+\n+\tif (!subprocess_map.tablesize)\n+\t\thashmap_init(&subprocess_map, (hashmap_cmp_fn) cmd2process_cmp, NULL, 0);\n+\telse\n+\t\tentry = (struct object_process *)subprocess_find_entry(&subprocess_map, o->cmd);\n+\n+\tfflush(NULL);\n+\n+\tif (!entry) {\n+\t\tentry = xmalloc(sizeof(*entry));\n+\t\tentry->supported_capabilities = 0;\n+\n+\t\tif (subprocess_start(&subprocess_map, &entry->subprocess, o->cmd, start_object_process_fn)) {\n+\t\t\terror(\"Could not launch process for cmd '%s'\", o->cmd);\n+\t\t\tfree(entry);\n+\t\t\treturn NULL;\n+\t\t}\n+\t}\n+\n+\to->supported_capabilities = entry->supported_capabilities;\n+\n+\tif (capability && !(capability & entry->supported_capabilities)) {\n+\t\terror(\"The cmd '%s' does not support capability '%d'\", o->cmd, capability);\n+\t\treturn NULL;\n+\t}\n+\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\n+\treturn entry;\n+}\n+\n+static int check_object_process_error(int err,\n+\t\t\t\t const char *status,\n+\t\t\t\t struct object_process *entry,\n+\t\t\t\t const char *cmd,\n+\t\t\t\t unsigned int capability)\n+{\n+\tsigchain_pop(SIGPIPE);\n+\n+\tif (!err)\n+\t\treturn 0;\n+\n+\tif (!strcmp(status, \"error\")) {\n+\t\t/* The process signaled a problem with the file. */\n+\t} else if (!strcmp(status, \"notfound\")) {\n+\t\t/* Object was not found */\n+\t\terr = -1;\n+\t} else if (!strcmp(status, \"abort\")) {\n+\t\t/*\n+\t\t * The process signaled a permanent problem. Don't try to read\n+\t\t * objects with the same command for the lifetime of the current\n+\t\t * Git process.\n+\t\t */\n+\t\tif (capability)\n+\t\t\tentry->supported_capabilities &= ~capability;\n+\t} else {\n+\t\t/*\n+\t\t * Something went wrong with the read-object process.\n+\t\t * Force shutdown and restart if needed.\n+\t\t */\n+\t\terror(\"external object process '%s' failed\", cmd);\n+\t\tsubprocess_stop(&subprocess_map, &entry->subprocess);\n+\t\tfree(entry);\n+\t}\n+\n+\treturn err;\n+}\n+\n+static int send_init_packets(struct object_process *entry,\n+\t\t\t struct strbuf *status)\n+{\n+\tstruct child_process *process = &entry->subprocess.process;\n+\n+\treturn packet_write_fmt_gently(process->in, \"command=init\\n\") ||\n+\t\tpacket_flush_gently(process->in) ||\n+\t\tcheck_object_process_status(process->out, status);\n+}\n+\n+static int init_object_process(struct odb_helper *o)\n+{\n+\tint err;\n+\tstruct strbuf status = STRBUF_INIT;\n+\tstruct object_process *entry = launch_object_process(o, 0);\n+\tif (!entry)\n+\t\treturn -1;\n+\n+\terr = send_init_packets(entry, &status);\n+\n+\treturn check_object_process_error(err, status.buf, entry,\n+\t\t\t\t\t o->cmd, 0);\n+}\n+\n+static ssize_t read_packetized_raw_object_to_fd(struct odb_helper *o,\n+\t\t\t\t\t\tconst unsigned char *sha1,\n+\t\t\t\t\t\tint fd_in, int fd_out)\n+{\n+\tssize_t total_read = 0;\n+\tunsigned long total_got = 0;\n+\tint packet_len;\n+\n+\tchar hdr[32];\n+\tint hdrlen;\n+\n+\tint ret = Z_STREAM_END;\n+\tunsigned char compressed[4096];\n+\tgit_zstream stream;\n+\tgit_SHA_CTX hash;\n+\tunsigned char real_sha1[20];\n+\n+\toff_t size;\n+\tenum object_type type;\n+\tconst char *s;\n+\tint pkt_size;\n+\tchar *size_buf;\n+\n+\tsize_buf = packet_read_line(fd_in, &pkt_size);\n+\tif (!skip_prefix(size_buf, \"size=\", &s))\n+\t\treturn error(\"odb helper '%s' did not send size of plain object\", o->name);\n+\tsize = strtoumax(s, NULL, 10);\n+\tif (!skip_prefix(packet_read_line(fd_in, NULL), \"kind=\", &s))\n+\t\treturn error(\"odb helper '%s' did not send kind of plain object\", o->name);\n+\t/* Check if the object is not available */\n+\tif (!strcmp(s, \"none\"))\n+\t\treturn -1;\n+\ttype = type_from_string_gently(s, strlen(s), 1);\n+\tif (type < 0)\n+\t\treturn error(\"odb helper '%s' sent bad type '%s'\", o->name, s);\n+\n+\t/* Set it up */\n+\tgit_deflate_init(&stream, zlib_compression_level);\n+\tstream.next_out = compressed;\n+\tstream.avail_out = sizeof(compressed);\n+\tgit_SHA1_Init(&hash);\n+\n+\t/* First header.. */\n+\thdrlen = xsnprintf(hdr, sizeof(hdr), \"%s %lu\", typename(type), size) + 1;\n+\tstream.next_in = (unsigned char *)hdr;\n+\tstream.avail_in = hdrlen;\n+\twhile (git_deflate(&stream, 0) == Z_OK)\n+\t\t; /* nothing */\n+\tgit_SHA1_Update(&hash, hdr, hdrlen);\n+\n+\tfor (;;) {\n+\t\t/* packet_read() writes a '\\0' extra byte at the end */\n+\t\tchar buf[LARGE_PACKET_DATA_MAX + 1];\n+\n+\t\tpacket_len = packet_read(fd_in, NULL, NULL,\n+\t\t\tbuf, LARGE_PACKET_DATA_MAX + 1,\n+\t\t\tPACKET_READ_GENTLE_ON_EOF);\n+\n+\t\tif (packet_len <= 0)\n+\t\t\tbreak;\n+\n+\t\ttotal_got += packet_len;\n+\n+\t\t/* Then the data itself.. */\n+\t\tstream.next_in = (void *)buf;\n+\t\tstream.avail_in = packet_len;\n+\t\tdo {\n+\t\t\tunsigned char *in0 = stream.next_in;\n+\t\t\tret = git_deflate(&stream, Z_FINISH);\n+\t\t\tgit_SHA1_Update(&hash, in0, stream.next_in - in0);\n+\t\t\twrite_or_die(fd_out, compressed, stream.next_out - compressed);\n+\t\t\tstream.next_out = compressed;\n+\t\t\tstream.avail_out = sizeof(compressed);\n+\t\t} while (ret == Z_OK);\n+\n+\t\ttotal_read += packet_len;\n+\t}\n+\n+\tif (packet_len < 0) {\n+\t\terror(\"unable to read from odb helper '%s': %s\",\n+\t\t o->name, strerror(errno));\n+\t\tgit_deflate_end(&stream);\n+\t\treturn packet_len;\n+\t}\n+\n+\tif (ret != Z_STREAM_END) {\n+\t\twarning(\"bad zlib data from odb helper '%s' for %s\",\n+\t\t\to->name, sha1_to_hex(sha1));\n+\t\treturn -1;\n+\t}\n+\n+\tret = git_deflate_end_gently(&stream);\n+\tif (ret != Z_OK) {\n+\t\twarning(\"deflateEnd on object %s from odb helper '%s' failed (%d)\",\n+\t\t\tsha1_to_hex(sha1), o->name, ret);\n+\t\treturn -1;\n+\t}\n+\tgit_SHA1_Final(real_sha1, &hash);\n+\tif (hashcmp(sha1, real_sha1)) {\n+\t\twarning(\"sha1 mismatch from odb helper '%s' for %s (got %s)\",\n+\t\t\to->name, sha1_to_hex(sha1), sha1_to_hex(real_sha1));\n+\t\treturn -1;\n+\t}\n+\tif (total_got != size) {\n+\t\twarning(\"size mismatch from odb helper '%s' for %s (%lu != %lu)\",\n+\t\t\to->name, sha1_to_hex(sha1), total_got, size);\n+\t\treturn -1;\n+\t}\n+\n+\treturn total_read;\n+}\n+\n+static ssize_t read_packetized_git_object_to_fd(struct odb_helper *o,\n+\t\t\t\t\t\tconst unsigned char *sha1,\n+\t\t\t\t\t\tint fd_in, int fd_out)\n+{\n+\tssize_t total_read = 0;\n+\tunsigned long total_got = 0;\n+\tint packet_len;\n+\tgit_zstream stream;\n+\tint zret = Z_STREAM_END;\n+\tgit_SHA_CTX hash;\n+\tunsigned char real_sha1[20];\n+\n+\tmemset(&stream, 0, sizeof(stream));\n+\tgit_inflate_init(&stream);\n+\tgit_SHA1_Init(&hash);\n+\n+\tfor (;;) {\n+\t\t/* packet_read() writes a '\\0' extra byte at the end */\n+\t\tchar buf[LARGE_PACKET_DATA_MAX + 1];\n+\n+\t\tpacket_len = packet_read(fd_in, NULL, NULL,\n+\t\t\tbuf, LARGE_PACKET_DATA_MAX + 1,\n+\t\t\tPACKET_READ_GENTLE_ON_EOF);\n+\n+\t\tif (packet_len <= 0)\n+\t\t\tbreak;\n+\n+\t\twrite_or_die(fd_out, buf, packet_len);\n+\n+\t\tstream.next_in = (unsigned char *)buf;\n+\t\tstream.avail_in = packet_len;\n+\t\tdo {\n+\t\t\tunsigned char inflated[4096];\n+\t\t\tunsigned long got;\n+\n+\t\t\tstream.next_out = inflated;\n+\t\t\tstream.avail_out = sizeof(inflated);\n+\t\t\tzret = git_inflate(&stream, Z_SYNC_FLUSH);\n+\t\t\tgot = sizeof(inflated) - stream.avail_out;\n+\n+\t\t\tgit_SHA1_Update(&hash, inflated, got);\n+\t\t\t/* skip header when counting size */\n+\t\t\tif (!total_got) {\n+\t\t\t\tconst unsigned char *p = memchr(inflated, '\\0', got);\n+\t\t\t\tif (p)\n+\t\t\t\t\tgot -= p - inflated + 1;\n+\t\t\t\telse\n+\t\t\t\t\tgot = 0;\n+\t\t\t}\n+\t\t\ttotal_got += got;\n+\t\t} while (stream.avail_in && zret == Z_OK);\n+\n+\t\ttotal_read += packet_len;\n+\t}\n+\n+\tgit_inflate_end(&stream);\n+\n+\tif (packet_len < 0)\n+\t\treturn packet_len;\n+\n+\tgit_SHA1_Final(real_sha1, &hash);\n+\n+\tif (zret != Z_STREAM_END) {\n+\t\twarning(\"bad zlib data from odb helper '%s' for %s\",\n+\t\t\to->name, sha1_to_hex(sha1));\n+\t\treturn -1;\n+\t}\n+\tif (hashcmp(real_sha1, sha1)) {\n+\t\twarning(\"sha1 mismatch from odb helper '%s' for %s (got %s)\",\n+\t\t\to->name, sha1_to_hex(sha1), sha1_to_hex(real_sha1));\n+\t\treturn -1;\n+\t}\n+\n+\treturn total_read;\n+}\n+\n+static int send_get_packets(struct odb_helper *o,\n+\t\t\t struct object_process *entry,\n+\t\t\t const unsigned char *sha1,\n+\t\t\t int fd,\n+\t\t\t unsigned int *cur_cap,\n+\t\t\t struct strbuf *status)\n+{\n+\tconst char *instruction;\n+\tint err;\n+\tstruct child_process *process = &entry->subprocess.process;\n+\n+\tif (entry->supported_capabilities & ODB_HELPER_CAP_GET_GIT_OBJ) {\n+\t\t*cur_cap = ODB_HELPER_CAP_GET_GIT_OBJ;\n+\t\tinstruction = \"get_git_obj\";\n+\t} else if (entry->supported_capabilities & ODB_HELPER_CAP_GET_RAW_OBJ) {\n+\t\t*cur_cap = ODB_HELPER_CAP_GET_RAW_OBJ;\n+\t\tinstruction = \"get_raw_obj\";\n+\t} else if (entry->supported_capabilities & ODB_HELPER_CAP_GET_DIRECT) {\n+\t\t*cur_cap = ODB_HELPER_CAP_GET_DIRECT;\n+\t\tinstruction = \"get_direct\";\n+\t} else {\n+\t\tBUG(\"No known ODB_HELPER_CAP_GET_XXX capability!\");\n+\t}\n+\n+\terr = packet_write_fmt_gently(process->in, \"command=%s\\n\", instruction);\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = packet_write_fmt_gently(process->in, \"sha1=%s\\n\", sha1_to_hex(sha1));\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = packet_flush_gently(process->in);\n+\tif (err)\n+\t\treturn err;\n+\n+\tif (entry->supported_capabilities & ODB_HELPER_CAP_GET_RAW_OBJ)\n+\t\terr = read_packetized_raw_object_to_fd(o, sha1, process->out, fd) < 0;\n+\telse if (entry->supported_capabilities & ODB_HELPER_CAP_GET_GIT_OBJ)\n+\t\terr = read_packetized_git_object_to_fd(o, sha1, process->out, fd) < 0;\n+\n+\treturn check_object_process_status(process->out, status);\n+}\n+\n+static int get_object_process(struct odb_helper *o, const unsigned char *sha1, int fd)\n+{\n+\tint err;\n+\tstruct strbuf status = STRBUF_INIT;\n+\tunsigned int cur_cap = 0;\n+\tstruct object_process *entry = launch_object_process(o, 0);\n+\tif (!entry)\n+\t\treturn -1;\n+\n+\terr = send_get_packets(o, entry, sha1, fd, &cur_cap, &status);\n+\n+\treturn check_object_process_error(err, status.buf, entry,\n+\t\t\t\t\t o->cmd, cur_cap);\n+}\n+\n struct odb_helper *odb_helper_new(const char *name, int namelen)\n {\n \tstruct odb_helper *o;\n@@ -117,15 +523,12 @@ static int odb_helper_finish(struct odb_helper *o,\n \treturn 0;\n }\n \n-int odb_helper_init(struct odb_helper *o)\n+static int init_object_script(struct odb_helper *o)\n {\n \tstruct odb_helper_cmd cmd;\n \tFILE *fh;\n \tstruct strbuf line = STRBUF_INIT;\n \n-\tif (!o->script_mode)\n-\t\treturn 0;\n-\n \tif (odb_helper_start(o, &cmd, 0, \"init\") < 0)\n \t\treturn -1;\n \n@@ -140,6 +543,21 @@ int odb_helper_init(struct odb_helper *o)\n \treturn 0;\n }\n \n+int odb_helper_init(struct odb_helper *o)\n+{\n+\tint res;\n+\tuint64_t start = getnanotime();\n+\n+\tif (o->script_mode)\n+\t\tres = init_object_script(o);\n+\telse\n+\t\tres = init_object_process(o);\n+\n+\ttrace_performance_since(start, \"odb_helper_init\");\n+\n+\treturn 0;\n+}\n+\n static int parse_object_line(struct odb_helper_object *o, const char *line)\n {\n \tchar *end;\n@@ -414,28 +832,42 @@ static int odb_helper_get_git_object(struct odb_helper *o,\n \treturn 0;\n }\n \n-int odb_helper_get_direct(struct odb_helper *o,\n-\t\t\t const unsigned char *sha1)\n+static int get_direct_script(struct odb_helper *o, const unsigned char *sha1)\n {\n-\tstruct odb_helper_object *obj;\n \tstruct odb_helper_cmd cmd;\n \n-\tobj = odb_helper_lookup(o, sha1);\n-\tif (!obj)\n-\t\treturn -1;\n-\n \tif (odb_helper_start(o, &cmd, 0, \"get_direct %s\", sha1_to_hex(sha1)) < 0)\n \t\treturn -1;\n-\n \tif (odb_helper_finish(o, &cmd))\n \t\treturn -1;\n-\n \treturn 0;\n }\n \n-int odb_helper_get_object(struct odb_helper *o,\n-\t\t\t const unsigned char *sha1,\n-\t\t\t int fd)\n+int odb_helper_get_direct(struct odb_helper *o,\n+\t\t\t const unsigned char *sha1)\n+{\n+\tint res;\n+\tuint64_t start;\n+\n+\tif (o->supported_capabilities & ODB_HELPER_CAP_HAVE) {\n+\t\tstruct odb_helper_object *obj = odb_helper_lookup(o, sha1);\n+\t\tif (!obj)\n+\t\t\treturn -1;\n+\t}\n+\n+\tstart = getnanotime();\n+\n+\tif (o->script_mode)\n+\t\tres = get_direct_script(o, sha1);\n+\telse\n+\t\tres = get_object_process(o, sha1, -1);\n+\n+\ttrace_performance_since(start, \"odb_helper_get_direct\");\n+\n+\treturn res;\n+}\n+\n+static int get_object_script(struct odb_helper *o, const unsigned char *sha1, int fd)\n {\n \tif (o->supported_capabilities & ODB_HELPER_CAP_GET_GIT_OBJ)\n \t\treturn odb_helper_get_git_object(o, sha1, fd);\n@@ -447,6 +879,23 @@ int odb_helper_get_object(struct odb_helper *o,\n \tBUG(\"invalid get capability (capabilities: '%d')\", o->supported_capabilities);\n }\n \n+int odb_helper_get_object(struct odb_helper *o,\n+\t\t\t const unsigned char *sha1,\n+\t\t\t int fd)\n+{\n+\tint res;\n+\tuint64_t start = getnanotime();\n+\n+\tif (o->script_mode)\n+\t\tres = get_object_script(o, sha1, fd);\n+\telse\n+\t\tres = get_object_process(o, sha1, fd);\n+\n+\ttrace_performance_since(start, \"odb_helper_get_object\");\n+\n+\treturn res;\n+}\n+\n int odb_helper_put_object(struct odb_helper *o,\n \t\t\t const void *buf, size_t len,\n \t\t\t const char *type, unsigned char *sha1)\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 3735720bfc..fb34f0b18d 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -698,7 +698,17 @@ int check_and_freshen_file(const char *fn, int freshen)\n \n static int check_and_freshen_local(const unsigned char *sha1, int freshen)\n {\n-\treturn check_and_freshen_file(sha1_file_name(sha1), freshen);\n+\tint ret;\n+\tint tried_hook = 0;\n+\n+retry:\n+\tret = check_and_freshen_file(sha1_file_name(sha1), freshen);\n+\tif (!ret && !tried_hook) {\n+\t\ttried_hook = 1;\n+\t\tif (!external_odb_get_direct(sha1))\n+\t\t\tgoto retry;\n+\t}\n+\treturn ret;\n }\n \n static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)\n@@ -3017,20 +3027,11 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \treturn (status < 0) ? status : 0;\n }\n \n-int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi, unsigned flags)\n+static int find_cached_or_packed(const unsigned char *sha1, struct object_info *oi,\n+\t\t\t\t unsigned flags, struct pack_entry *e, int retry)\n {\n-\tstatic struct object_info blank_oi = OBJECT_INFO_INIT;\n-\tstruct pack_entry e;\n-\tint rtype;\n-\tconst unsigned char *real = (flags & OBJECT_INFO_LOOKUP_REPLACE) ?\n-\t\t\t\t lookup_replace_object(sha1) :\n-\t\t\t\t sha1;\n-\n-\tif (!oi)\n-\t\toi = &blank_oi;\n-\n \tif (!(flags & OBJECT_INFO_SKIP_CACHED)) {\n-\t\tstruct cached_object *co = find_cached_object(real);\n+\t\tstruct cached_object *co = find_cached_object(sha1);\n \t\tif (co) {\n \t\t\tif (oi->typep)\n \t\t\t\t*(oi->typep) = co->type;\n@@ -3049,9 +3050,9 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\t}\n \t}\n \n-\tif (!find_pack_entry(real, &e)) {\n+\tif (!find_pack_entry(sha1, e)) {\n \t\t/* Most likely it's a loose object. */\n-\t\tif (!sha1_loose_object_info(real, oi, flags)) {\n+\t\tif (!sha1_loose_object_info(sha1, oi, flags)) {\n \t\t\toi->whence = OI_LOOSE;\n \t\t\treturn 0;\n \t\t}\n@@ -3061,10 +3062,33 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\t\treturn -1;\n \t\t} else {\n \t\t\treprepare_packed_git();\n-\t\t\tif (!find_pack_entry(real, &e))\n+\t\t\tif (!find_pack_entry(sha1, e)) {\n+\t\t\t\tif (retry && !external_odb_get_direct(sha1))\n+\t\t\t\t\treturn find_cached_or_packed(sha1, oi, flags, e, 0);\n \t\t\t\treturn -1;\n+\t\t\t}\n \t\t}\n \t}\n+\treturn 1;\n+}\n+\n+int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi, unsigned flags)\n+{\n+\tstatic struct object_info blank_oi = OBJECT_INFO_INIT;\n+\tstruct pack_entry e;\n+\tint rtype;\n+\tenum object_type real_type;\n+\tint res;\n+\tconst unsigned char *real = (flags & OBJECT_INFO_LOOKUP_REPLACE) ?\n+\t\t\t\t lookup_replace_object(sha1) :\n+\t\t\t\t sha1;\n+\n+\tif (!oi)\n+\t\toi = &blank_oi;\n+\n+\tres = find_cached_or_packed(real, oi, flags, &e, 1);\n+\tif (res < 1)\n+\t\treturn res;\n \n \tif (oi == &blank_oi)\n \t\t/*\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325531","messageId":"20170803091926.1755-24-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 23/40] pack-objects: don't pack objects in external odbs","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:19:09Z","receivedAt":"2017-08-03T09:21:47Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Objects managed by an external ODB should not be put into\npack files. They should be transfered using other mechanism\nthat can be specific to the external odb.\n\nSigned-off-by: Christian Couder \n---\n builtin/pack-objects.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex f4a8441fe9..8283d15408 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -25,6 +25,7 @@\n #include \"sha1-array.h\"\n #include \"argv-array.h\"\n #include \"mru.h\"\n+#include \"external-odb.h\"\n \n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [...] [< | < ]\"),\n@@ -1011,6 +1012,9 @@ static int want_object_in_pack(const unsigned char *sha1,\n \t\t\treturn want;\n \t}\n \n+\tif (external_odb_has_object(sha1))\n+\t\treturn 0;\n+\n \tfor (entry = packed_git_mru->head; entry; entry = entry->next) {\n \t\tstruct packed_git *p = entry->item;\n \t\toff_t offset;\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325532","messageId":"20170803091926.1755-27-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 26/40] odb-helper: add 'script_mode' to 'struct odb_helper'","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:19:12Z","receivedAt":"2017-08-03T09:21:48Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"to prepare for having a long running odb helper sub-process\nhandling the communication between Git and an external odb.\n\nWe introduce \"odb..subprocesscommand\" to make it\npossible to define such a sub-process, and we mark such odb\nhelpers with the new 'script_mode' field set to 0.\n\nHelpers defined using the existing \"odb..scriptcommand\"\nare marked with the 'script_mode' field set to 1.\n\nImplementation of the different capabilities/instructions in\nthe new (sub-)process mode is left for following commits.\n\nSigned-off-by: Christian Couder \n---\n external-odb.c | 8 +++++++-\n odb-helper.c | 19 ++++++++++++++-----\n odb-helper.h | 1 +\n 3 files changed, 22 insertions(+), 6 deletions(-)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex 31d21bfe04..ccca67eff5 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -32,8 +32,14 @@ static int external_odb_config(const char *var, const char *value, void *data)\n \n \to = find_or_create_helper(name, namelen);\n \n-\tif (!strcmp(subkey, \"scriptcommand\"))\n+\tif (!strcmp(subkey, \"scriptcommand\")) {\n+\t\to->script_mode = 1;\n \t\treturn git_config_string(&o->cmd, var, value);\n+\t}\n+\tif (!strcmp(subkey, \"subprocesscommand\")) {\n+\t\to->script_mode = 0;\n+\t\treturn git_config_string(&o->cmd, var, value);\n+\t}\n \n \treturn 0;\n }\ndiff --git a/odb-helper.c b/odb-helper.c\nindex b1f5464214..4c16dd297a 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -123,6 +123,9 @@ int odb_helper_init(struct odb_helper *o)\n \tFILE *fh;\n \tstruct strbuf line = STRBUF_INIT;\n \n+\tif (!o->script_mode)\n+\t\treturn 0;\n+\n \tif (odb_helper_start(o, &cmd, 0, \"init\") < 0)\n \t\treturn -1;\n \n@@ -173,16 +176,12 @@ static int odb_helper_object_cmp(const void *va, const void *vb)\n \treturn hashcmp(a->sha1, b->sha1);\n }\n \n-static void odb_helper_load_have(struct odb_helper *o)\n+static void have_object_script(struct odb_helper *o)\n {\n \tstruct odb_helper_cmd cmd;\n \tFILE *fh;\n \tstruct strbuf line = STRBUF_INIT;\n \n-\tif (o->have_valid)\n-\t\treturn;\n-\to->have_valid = 1;\n-\n \tif (odb_helper_start(o, &cmd, 0, \"have\") < 0)\n \t\treturn;\n \n@@ -194,6 +193,16 @@ static void odb_helper_load_have(struct odb_helper *o)\n \tstrbuf_release(&line);\n \tfclose(fh);\n \todb_helper_finish(o, &cmd);\n+}\n+\n+static void odb_helper_load_have(struct odb_helper *o)\n+{\n+\tif (o->have_valid)\n+\t\treturn;\n+\to->have_valid = 1;\n+\n+\tif (o->script_mode)\n+\t\thave_object_script(o);\n \n \tqsort(o->have, o->have_nr, sizeof(*o->have), odb_helper_object_cmp);\n }\ndiff --git a/odb-helper.h b/odb-helper.h\nindex f2fd2b7c9c..04b85f1d02 100644\n--- a/odb-helper.h\n+++ b/odb-helper.h\n@@ -15,6 +15,7 @@ struct odb_helper {\n \tconst char *name;\n \tconst char *cmd;\n \tunsigned int supported_capabilities;\n+\tint script_mode;\n \n \tstruct odb_helper_object {\n \t\tunsigned char sha1[20];\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325533","messageId":"20170803091926.1755-25-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 24/40] Add t0420 to test transfer to HTTP external odb","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:19:10Z","receivedAt":"2017-08-03T09:21:54Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This tests that an apache web server can be used as an\nexternal object database and store files in their native\nformat instead of converting them to a Git object.\n\nSigned-off-by: Christian Couder \n---\n t/t0420-transfer-http-e-odb.sh | 142 +++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 142 insertions(+)\n create mode 100755 t/t0420-transfer-http-e-odb.sh\n\ndiff --git a/t/t0420-transfer-http-e-odb.sh b/t/t0420-transfer-http-e-odb.sh\nnew file mode 100755\nindex 0000000000..f84fe950ec\n--- /dev/null\n+++ b/t/t0420-transfer-http-e-odb.sh\n@@ -0,0 +1,142 @@\n+#!/bin/sh\n+\n+test_description='tests for transfering external objects to an HTTPD server'\n+\n+. ./test-lib.sh\n+\n+# If we don't specify a port, the current test number will be used\n+# which will not work as it is less than 1024, so it can only be used by root.\n+LIB_HTTPD_PORT=$(expr ${this_test#t} + 12000)\n+\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+\n+start_httpd apache-e-odb.conf\n+\n+# odb helper script must see this\n+export HTTPD_URL\n+\n+write_script odb-http-helper <<\\EOF\n+die() {\n+\tprintf >&2 \"%s\\n\" \"$@\"\n+\texit 1\n+}\n+echo >&2 \"odb-http-helper args:\" \"$@\"\n+case \"$1\" in\n+init)\n+\techo \"capability=get_raw_obj\"\n+\techo \"capability=put_raw_obj\"\n+\techo \"capability=have\"\n+\t;;\n+have)\n+\tlist_url=\"$HTTPD_URL/list/\"\n+\tcurl \"$list_url\" ||\n+\tdie \"curl '$list_url' failed\"\n+\t;;\n+get_raw_obj)\n+\tget_url=\"$HTTPD_URL/list/?sha1=$2\"\n+\tcurl \"$get_url\" ||\n+\tdie \"curl '$get_url' failed\"\n+\t;;\n+put_raw_obj)\n+\tsha1=\"$2\"\n+\tsize=\"$3\"\n+\tkind=\"$4\"\n+\tupload_url=\"$HTTPD_URL/upload/?sha1=$sha1&size=$size&type=$kind\"\n+\tcurl --data-binary @- --include \"$upload_url\" >out ||\n+\tdie \"curl '$upload_url' failed\"\n+\tref_hash=$(echo \"$sha1 $size $kind\" | GIT_NO_EXTERNAL_ODB=1 git hash-object -w -t blob --stdin) || exit\n+\tgit update-ref refs/odbs/magic/\"$sha1\" \"$ref_hash\"\n+\t;;\n+*)\n+\tdie \"unknown command '$1'\"\n+\t;;\n+esac\n+EOF\n+HELPER=\"\\\"$PWD\\\"/odb-http-helper\"\n+\n+test_expect_success 'setup repo with a root commit and the helper' '\n+\ttest_commit zero &&\n+\tgit config odb.magic.scriptCommand \"$HELPER\"\n+'\n+\n+test_expect_success 'setup another repo from the first one' '\n+\tgit init other-repo &&\n+\t(cd other-repo &&\n+\t git remote add origin .. &&\n+\t git pull origin master &&\n+\t git checkout master &&\n+\t git log)\n+'\n+\n+UPLOADFILENAME=\"hello_apache_upload.txt\"\n+\n+UPLOAD_URL=\"$HTTPD_URL/upload/?sha1=$UPLOADFILENAME&size=123&type=blob\"\n+\n+test_expect_success 'can upload a file' '\n+\techo \"Hello Apache World!\" >hello_to_send.txt &&\n+\techo \"How are you?\" >>hello_to_send.txt &&\n+\tcurl --data-binary @hello_to_send.txt --include \"$UPLOAD_URL\" >out_upload\n+'\n+\n+LIST_URL=\"$HTTPD_URL/list/\"\n+\n+test_expect_success 'can list uploaded files' '\n+\tcurl --include \"$LIST_URL\" >out_list &&\n+\tgrep \"$UPLOADFILENAME\" out_list\n+'\n+\n+test_expect_success 'can delete uploaded files' '\n+\tcurl --data \"delete\" --include \"$UPLOAD_URL&delete=1\" >out_delete &&\n+\tcurl --include \"$LIST_URL\" >out_list2 &&\n+\t! grep \"$UPLOADFILENAME\" out_list2\n+'\n+\n+FILES_DIR=\"httpd/www/files\"\n+\n+test_expect_success 'new blobs are transfered to the http server' '\n+\ttest_commit one &&\n+\thash1=$(git ls-tree HEAD | grep one.t | cut -f1 | cut -d\\ -f3) &&\n+\techo \"$hash1-4-blob\" >expected &&\n+\tls \"$FILES_DIR\" >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'blobs can be retrieved from the http server' '\n+\tgit cat-file blob \"$hash1\" &&\n+\tgit log -p >expected\n+'\n+\n+test_expect_success 'update other repo from the first one' '\n+\t(cd other-repo &&\n+\t git fetch origin \"refs/odbs/magic/*:refs/odbs/magic/*\" &&\n+\t test_must_fail git cat-file blob \"$hash1\" &&\n+\t git config odb.magic.scriptCommand \"$HELPER\" &&\n+\t git cat-file blob \"$hash1\" &&\n+\t git pull origin master)\n+'\n+\n+test_expect_success 'local clone from the first repo' '\n+\tmkdir my-clone &&\n+\t(cd my-clone &&\n+\t git clone .. . &&\n+\t git cat-file blob \"$hash1\")\n+'\n+\n+test_expect_success 'no-local clone from the first repo fails' '\n+\tmkdir my-other-clone &&\n+\t(cd my-other-clone &&\n+\t test_must_fail git clone --no-local .. .) &&\n+\trm -rf my-other-clone\n+'\n+\n+test_expect_success 'no-local clone from the first repo with helper succeeds' '\n+\tmkdir my-other-clone &&\n+\t(cd my-other-clone &&\n+\t git clone -c odb.magic.scriptCommand=\"$HELPER\" \\\n+\t\t--no-local .. .) &&\n+\trm -rf my-other-clone\n+'\n+\n+stop_httpd\n+\n+test_done\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325534","messageId":"20170803091926.1755-26-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 25/40] external-odb: add 'get_direct' support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:19:11Z","receivedAt":"2017-08-03T09:21:57Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This implements the 'get_direct' capability/instruction that makes\nit possible for external odb helper scripts to pass blobs to Git\nby directly writing them as loose objects files.\n\nIt is better to call this a \"direct\" mode rather than a \"fault-in\"\nmode as we could have the same kind of mechanism to \"put\" objects\ninto an external odb, where the odb helper would access blobs it\nwants to send to an external odb directly from files, but it\nwould be strange to call that a fault-in mode too.\n\nSigned-off-by: Christian Couder \n---\n external-odb.c | 21 ++++++++++++++++++++-\n external-odb.h | 1 +\n odb-helper.c | 27 +++++++++++++++++++++++++--\n odb-helper.h | 1 +\n 4 files changed, 47 insertions(+), 3 deletions(-)\n\ndiff --git a/external-odb.c b/external-odb.c\nindex 52cb448d01..31d21bfe04 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -96,7 +96,8 @@ int external_odb_get_object(const unsigned char *sha1)\n \t\tint ret;\n \t\tint fd;\n \n-\t\tif (!odb_helper_has_object(o, sha1))\n+\t\tif (!(o->supported_capabilities & ODB_HELPER_CAP_GET_RAW_OBJ) &&\n+\t\t !(o->supported_capabilities & ODB_HELPER_CAP_GET_GIT_OBJ))\n \t\t\tcontinue;\n \n \t\tfd = create_object_tmpfile(&tmpfile, path);\n@@ -122,6 +123,24 @@ int external_odb_get_object(const unsigned char *sha1)\n \treturn -1;\n }\n \n+int external_odb_get_direct(const unsigned char *sha1)\n+{\n+\tstruct odb_helper *o;\n+\n+\tif (!external_odb_has_object(sha1))\n+\t\treturn -1;\n+\n+\tfor (o = helpers; o; o = o->next) {\n+\t\tif (!(o->supported_capabilities & ODB_HELPER_CAP_GET_DIRECT))\n+\t\t\tcontinue;\n+\t\tif (odb_helper_get_direct(o, sha1) < 0)\n+\t\t\tcontinue;\n+\t\treturn 0;\n+\t}\n+\n+\treturn -1;\n+}\n+\n int external_odb_put_object(const void *buf, size_t len,\n \t\t\t const char *type, unsigned char *sha1)\n {\ndiff --git a/external-odb.h b/external-odb.h\nindex 3e0e6d0165..247b131fd5 100644\n--- a/external-odb.h\n+++ b/external-odb.h\n@@ -4,6 +4,7 @@\n const char *external_odb_root(void);\n int external_odb_has_object(const unsigned char *sha1);\n int external_odb_get_object(const unsigned char *sha1);\n+int external_odb_get_direct(const unsigned char *sha1);\n int external_odb_put_object(const void *buf, size_t len,\n \t\t\t const char *type, unsigned char *sha1);\n \ndiff --git a/odb-helper.c b/odb-helper.c\nindex 0603993057..b1f5464214 100644\n--- a/odb-helper.c\n+++ b/odb-helper.c\n@@ -405,14 +405,37 @@ static int odb_helper_get_git_object(struct odb_helper *o,\n \treturn 0;\n }\n \n+int odb_helper_get_direct(struct odb_helper *o,\n+\t\t\t const unsigned char *sha1)\n+{\n+\tstruct odb_helper_object *obj;\n+\tstruct odb_helper_cmd cmd;\n+\n+\tobj = odb_helper_lookup(o, sha1);\n+\tif (!obj)\n+\t\treturn -1;\n+\n+\tif (odb_helper_start(o, &cmd, 0, \"get_direct %s\", sha1_to_hex(sha1)) < 0)\n+\t\treturn -1;\n+\n+\tif (odb_helper_finish(o, &cmd))\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n int odb_helper_get_object(struct odb_helper *o,\n \t\t\t const unsigned char *sha1,\n \t\t\t int fd)\n {\n+\tif (o->supported_capabilities & ODB_HELPER_CAP_GET_GIT_OBJ)\n+\t\treturn odb_helper_get_git_object(o, sha1, fd);\n \tif (o->supported_capabilities & ODB_HELPER_CAP_GET_RAW_OBJ)\n \t\treturn odb_helper_get_raw_object(o, sha1, fd);\n-\telse\n-\t\treturn odb_helper_get_git_object(o, sha1, fd);\n+\tif (o->supported_capabilities & ODB_HELPER_CAP_GET_DIRECT)\n+\t\treturn 0;\n+\n+\tBUG(\"invalid get capability (capabilities: '%d')\", o->supported_capabilities);\n }\n \n int odb_helper_put_object(struct odb_helper *o,\ndiff --git a/odb-helper.h b/odb-helper.h\nindex 318e0d48dc..f2fd2b7c9c 100644\n--- a/odb-helper.h\n+++ b/odb-helper.h\n@@ -33,6 +33,7 @@ int odb_helper_init(struct odb_helper *o);\n int odb_helper_has_object(struct odb_helper *o, const unsigned char *sha1);\n int odb_helper_get_object(struct odb_helper *o, const unsigned char *sha1,\n \t\t\t int fd);\n+int odb_helper_get_direct(struct odb_helper *o, const unsigned char *sha1);\n int odb_helper_put_object(struct odb_helper *o,\n \t\t\t const void *buf, size_t len,\n \t\t\t const char *type, unsigned char *sha1);\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325535","messageId":"20170803091926.1755-16-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 15/40] t0400: add test for external odb write support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:19:01Z","receivedAt":"2017-08-03T09:22:01Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n t/t0400-external-odb.sh | 8 ++++++++\n 1 file changed, 8 insertions(+)\n\ndiff --git a/t/t0400-external-odb.sh b/t/t0400-external-odb.sh\nindex 3fa0449883..fa355bd7bb 100755\n--- a/t/t0400-external-odb.sh\n+++ b/t/t0400-external-odb.sh\n@@ -71,4 +71,12 @@ test_expect_success 'helper can add objects to alt repo' '\n \ttest \"$size\" -eq \"$alt_size\"\n '\n \n+test_expect_success 'commit adds objects to alt repo' '\n+\ttest_config odb.magic.scriptCommand \"$HELPER\" &&\n+\ttest_commit three &&\n+\thash3=$(git ls-tree HEAD | grep three.t | cut -f1 | cut -d\\ -f3) &&\n+\tcontent=$(cd alt-repo && git show \"$hash3\") &&\n+\ttest \"$content\" = \"three\"\n+'\n+\n test_done\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325536","messageId":"20170803091926.1755-18-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 17/40] Add t0410 to test external ODB transfer","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:19:03Z","receivedAt":"2017-08-03T09:22:03Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n t/t0410-transfer-e-odb.sh | 144 ++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 144 insertions(+)\n create mode 100755 t/t0410-transfer-e-odb.sh\n\ndiff --git a/t/t0410-transfer-e-odb.sh b/t/t0410-transfer-e-odb.sh\nnew file mode 100755\nindex 0000000000..065ec7d759\n--- /dev/null\n+++ b/t/t0410-transfer-e-odb.sh\n@@ -0,0 +1,144 @@\n+#!/bin/sh\n+\n+test_description='basic tests for transfering external ODBs'\n+\n+. ./test-lib.sh\n+\n+ORIG_SOURCE=\"$PWD/.git\"\n+export ORIG_SOURCE\n+\n+ALT_SOURCE1=\"$PWD/alt-repo1/.git\"\n+export ALT_SOURCE1\n+write_script odb-helper1 <<\\EOF\n+die() {\n+\tprintf >&2 \"%s\\n\" \"$@\"\n+\texit 1\n+}\n+GIT_DIR=$ALT_SOURCE1; export GIT_DIR\n+case \"$1\" in\n+init)\n+\techo \"capability=get_git_obj\"\n+\techo \"capability=have\"\n+\t;;\n+have)\n+\tgit cat-file --batch-check --batch-all-objects |\n+\tawk '{print $1 \" \" $3 \" \" $2}'\n+\t;;\n+get_git_obj)\n+\tcat \"$GIT_DIR\"/objects/$(echo $2 | sed 's#..#&/#')\n+\t;;\n+put_raw_obj)\n+\tsha1=\"$2\"\n+\tsize=\"$3\"\n+\tkind=\"$4\"\n+\twriten=$(git hash-object -w -t \"$kind\" --stdin)\n+\ttest \"$writen\" = \"$sha1\" || die \"bad sha1 passed '$sha1' vs writen '$writen'\"\n+\tref_hash=$(echo \"$sha1 $size $kind\" | GIT_DIR=$ORIG_SOURCE GIT_NO_EXTERNAL_ODB=1 git hash-object -w -t blob --stdin) || exit\n+\tGIT_DIR=$ORIG_SOURCE git update-ref refs/odbs/magic/\"$sha1\" \"$ref_hash\"\n+\t;;\n+*)\n+\tdie \"unknown command '$1'\"\n+\t;;\n+esac\n+EOF\n+HELPER1=\"\\\"$PWD\\\"/odb-helper1\"\n+\n+OTHER_SOURCE=\"$PWD/.git\"\n+export OTHER_SOURCE\n+\n+ALT_SOURCE2=\"$PWD/alt-repo2/.git\"\n+export ALT_SOURCE2\n+write_script odb-helper2 <<\\EOF\n+die() {\n+\tprintf >&2 \"%s\\n\" \"$@\"\n+\texit 1\n+}\n+GIT_DIR=$ALT_SOURCE2; export GIT_DIR\n+case \"$1\" in\n+init)\n+\techo \"capability=get_git_obj\"\n+\techo \"capability=have\"\n+\t;;\n+have)\n+\tGIT_DIR=$OTHER_SOURCE git for-each-ref --format='%(objectname)' refs/odbs/magic/ | GIT_DIR=$OTHER_SOURCE xargs git show\n+\t;;\n+get_git_obj)\n+\tOBJ_FILE=\"$GIT_DIR\"/objects/$(echo $2 | sed 's#..#&/#')\n+\tif ! test -f \"$OBJ_FILE\"\n+\tthen\n+\t\t# \"Download\" the missing object by copying it from alt-repo1\n+\t\tOBJ_DIR=$(echo $2 | sed 's/\\(..\\).*/\\1/')\n+\t\tOBJ_BASE=$(basename \"$OBJ_FILE\")\n+\t\tALT_OBJ_DIR1=\"$ALT_SOURCE1/objects/$OBJ_DIR\"\n+\t\tALT_OBJ_DIR2=\"$ALT_SOURCE2/objects/$OBJ_DIR\"\n+\t\tmkdir -p \"$ALT_OBJ_DIR2\" || die \"Could not mkdir '$ALT_OBJ_DIR2'\"\n+\t\tOBJ_SRC=\"$ALT_OBJ_DIR1/$OBJ_BASE\"\n+\t\tcp \"$OBJ_SRC\" \"$ALT_OBJ_DIR2\" ||\n+\t\tdie \"Could not cp '$OBJ_SRC' into '$ALT_OBJ_DIR2'\"\n+\tfi\n+\tcat \"$OBJ_FILE\" || die \"Could not cat '$OBJ_FILE'\"\n+\t;;\n+put_raw_obj)\n+\tsha1=\"$2\"\n+\tsize=\"$3\"\n+\tkind=\"$4\"\n+\twriten=$(git hash-object -w -t \"$kind\" --stdin)\n+\ttest \"$writen\" = \"$sha1\" || die \"bad sha1 passed '$sha1' vs writen '$writen'\"\n+\tref_hash=$(echo \"$sha1 $size $kind\" | GIT_DIR=$OTHER_SOURCE GIT_NO_EXTERNAL_ODB=1 git hash-object -w -t blob --stdin) || exit\n+\tGIT_DIR=$OTHER_SOURCE git update-ref refs/odbs/magic/\"$sha1\" \"$ref_hash\"\n+\t;;\n+*)\n+\tdie \"unknown command '$1'\"\n+\t;;\n+esac\n+EOF\n+HELPER2=\"\\\"$PWD\\\"/odb-helper2\"\n+\n+test_expect_success 'setup first alternate repo' '\n+\tgit init alt-repo1 &&\n+\ttest_commit zero &&\n+\tgit config odb.magic.scriptCommand \"$HELPER1\"\n+'\n+\n+test_expect_success 'setup other repo and its alternate repo' '\n+\tgit init other-repo &&\n+\tgit init alt-repo2 &&\n+\t(cd other-repo &&\n+\t git remote add origin .. &&\n+\t git pull origin master &&\n+\t git checkout master &&\n+\t git log)\n+'\n+\n+test_expect_success 'new blobs are put in first object store' '\n+\ttest_commit one &&\n+\thash1=$(git ls-tree HEAD | grep one.t | cut -f1 | cut -d\\ -f3) &&\n+\tcontent=$(cd alt-repo1 && git show \"$hash1\") &&\n+\ttest \"$content\" = \"one\" &&\n+\ttest_commit two &&\n+\thash2=$(git ls-tree HEAD | grep two.t | cut -f1 | cut -d\\ -f3) &&\n+\tcontent=$(cd alt-repo1 && git show \"$hash2\") &&\n+\ttest \"$content\" = \"two\"\n+'\n+\n+test_expect_success 'other repo gets the blobs from object store' '\n+\t(cd other-repo &&\n+\t git fetch origin \"refs/odbs/magic/*:refs/odbs/magic/*\" &&\n+\t test_must_fail git cat-file blob \"$hash1\" &&\n+\t test_must_fail git cat-file blob \"$hash2\" &&\n+\t git config odb.magic.scriptCommand \"$HELPER2\" &&\n+\t git cat-file blob \"$hash1\" &&\n+\t git cat-file blob \"$hash2\"\n+\t)\n+'\n+\n+test_expect_success 'other repo gets everything else' '\n+\t(cd other-repo &&\n+\t git fetch origin &&\n+\t content=$(git show \"$hash1\") &&\n+\t test \"$content\" = \"one\" &&\n+\t content=$(git show \"$hash2\") &&\n+\t test \"$content\" = \"two\")\n+'\n+\n+test_done\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325537","messageId":"20170803091926.1755-19-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 18/40] lib-httpd: pass config file to start_httpd()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:19:04Z","receivedAt":"2017-08-03T09:22:07Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This makes it possible to start an apache web server with different\nconfig files.\n\nThis will be used in a later patch to pass a config file that makes\napache store external objects.\n\nSigned-off-by: Christian Couder \n---\n t/lib-httpd.sh | 6 ++++--\n 1 file changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\nindex 435a37465a..2e659a8ee2 100644\n--- a/t/lib-httpd.sh\n+++ b/t/lib-httpd.sh\n@@ -171,12 +171,14 @@ prepare_httpd() {\n }\n \n start_httpd() {\n+\tAPACHE_CONF_FILE=${1-apache.conf}\n+\n \tprepare_httpd >&3 2>&4\n \n \ttrap 'code=$?; stop_httpd; (exit $code); die' EXIT\n \n \t\"$LIB_HTTPD_PATH\" -d \"$HTTPD_ROOT_PATH\" \\\n-\t\t-f \"$TEST_PATH/apache.conf\" $HTTPD_PARA \\\n+\t\t-f \"$TEST_PATH/$APACHE_CONF_FILE\" $HTTPD_PARA \\\n \t\t-c \"Listen 127.0.0.1:$LIB_HTTPD_PORT\" -k start \\\n \t\t>&3 2>&4\n \tif test $? -ne 0\n@@ -191,7 +193,7 @@ stop_httpd() {\n \ttrap 'die' EXIT\n \n \t\"$LIB_HTTPD_PATH\" -d \"$HTTPD_ROOT_PATH\" \\\n-\t\t-f \"$TEST_PATH/apache.conf\" $HTTPD_PARA -k stop\n+\t\t-f \"$TEST_PATH/$APACHE_CONF_FILE\" $HTTPD_PARA -k stop\n }\n \n test_http_push_nonff () {\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325538","messageId":"20170803091926.1755-17-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 16/40] Add GIT_NO_EXTERNAL_ODB env variable","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:19:02Z","receivedAt":"2017-08-03T09:22:11Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This new environment variable will be used to perform git\ncommands without involving any external odb mechanism.\n\nThis makes it possible for example to create new blobs that\nwill not be sent to an external odb even if the external odb\nsupports \"put_*\" instructions.\n\nSigned-off-by: Christian Couder \n---\n cache.h | 9 +++++++++\n environment.c | 4 ++++\n external-odb.c | 6 ++++++\n sha1_file.c | 3 +++\n 4 files changed, 22 insertions(+)\n\ndiff --git a/cache.h b/cache.h\nindex 13694069b1..73ebd99830 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -430,6 +430,7 @@ static inline enum object_type object_type(unsigned int mode)\n #define CEILING_DIRECTORIES_ENVIRONMENT \"GIT_CEILING_DIRECTORIES\"\n #define NO_REPLACE_OBJECTS_ENVIRONMENT \"GIT_NO_REPLACE_OBJECTS\"\n #define GIT_REPLACE_REF_BASE_ENVIRONMENT \"GIT_REPLACE_REF_BASE\"\n+#define NO_EXTERNAL_ODB_ENVIRONMENT \"GIT_NO_EXTERNAL_ODB\"\n #define GITATTRIBUTES_FILE \".gitattributes\"\n #define INFOATTRIBUTES_FILE \"info/attributes\"\n #define ATTRIBUTE_MACRO_PREFIX \"[attr]\"\n@@ -767,6 +768,14 @@ void reset_shared_repository(void);\n extern int check_replace_refs;\n extern char *git_replace_ref_base;\n \n+/*\n+ * Do external odbs need to be used this run? This variable is\n+ * initialized to true unless $GIT_NO_EXTERNAL_ODB is set, but it\n+ * maybe set to false by some commands that do not want external\n+ * odbs to be active.\n+ */\n+extern int use_external_odb;\n+\n extern int fsync_object_files;\n extern int core_preload_index;\n extern int core_apply_sparse_checkout;\ndiff --git a/environment.c b/environment.c\nindex 3fd4b10845..bbccabef6b 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -48,6 +48,7 @@ const char *excludes_file;\n enum auto_crlf auto_crlf = AUTO_CRLF_FALSE;\n int check_replace_refs = 1;\n char *git_replace_ref_base;\n+int use_external_odb = 1;\n enum eol core_eol = EOL_UNSET;\n enum safe_crlf safe_crlf = SAFE_CRLF_WARN;\n unsigned whitespace_rule_cfg = WS_DEFAULT_RULE;\n@@ -116,6 +117,7 @@ const char * const local_repo_env[] = {\n \tINDEX_ENVIRONMENT,\n \tNO_REPLACE_OBJECTS_ENVIRONMENT,\n \tGIT_REPLACE_REF_BASE_ENVIRONMENT,\n+\tNO_EXTERNAL_ODB_ENVIRONMENT,\n \tGIT_PREFIX_ENVIRONMENT,\n \tGIT_SUPER_PREFIX_ENVIRONMENT,\n \tGIT_SHALLOW_FILE_ENVIRONMENT,\n@@ -154,6 +156,8 @@ void setup_git_env(void)\n \treplace_ref_base = getenv(GIT_REPLACE_REF_BASE_ENVIRONMENT);\n \tgit_replace_ref_base = xstrdup(replace_ref_base ? replace_ref_base\n \t\t\t\t\t\t\t : \"refs/replace/\");\n+\tif (getenv(NO_EXTERNAL_ODB_ENVIRONMENT))\n+\t\tuse_external_odb = 0;\n \tnamespace = expand_namespace(getenv(GIT_NAMESPACE_ENVIRONMENT));\n \tshallow_file = getenv(GIT_SHALLOW_FILE_ENVIRONMENT);\n \tif (shallow_file)\ndiff --git a/external-odb.c b/external-odb.c\nindex a4f8c72e1c..52cb448d01 100644\n--- a/external-odb.c\n+++ b/external-odb.c\n@@ -65,6 +65,9 @@ int external_odb_has_object(const unsigned char *sha1)\n {\n \tstruct odb_helper *o;\n \n+\tif (!use_external_odb)\n+\t\treturn 0;\n+\n \texternal_odb_init();\n \n \tfor (o = helpers; o; o = o->next) {\n@@ -124,6 +127,9 @@ int external_odb_put_object(const void *buf, size_t len,\n {\n \tstruct odb_helper *o;\n \n+\tif (!use_external_odb)\n+\t\treturn 1;\n+\n \t/* For now accept only blobs */\n \tif (strcmp(type, \"blob\"))\n \t\treturn 1;\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 6f6406fb36..3735720bfc 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -642,6 +642,9 @@ void prepare_external_alt_odb(void)\n \tstatic int linked_external;\n \tconst char *path;\n \n+\tif (!use_external_odb)\n+\t\treturn;\n+\n \tif (linked_external)\n \t\treturn;\n \n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325539","messageId":"20170803091926.1755-11-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 10/40] Add initial external odb support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:18:56Z","receivedAt":"2017-08-03T09:22:13Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"The external-odb.{c,h} files contains the functions that are\ncalled by the rest of Git from \"sha1_file.c\".\n\nThe odb-helper.{c,h} files contains the functions to\nactually implement communication with the external scripts or\nprocesses that will manage external git objects.\n\nFor now only script mode is supported, and only the 'have' and\n'get_git_obj' instructions are supported.\n\nHelped-by: Jeff King \nSigned-off-by: Christian Couder \n---\n Makefile | 2 +\n cache.h | 1 +\n external-odb.c | 113 ++++++++++++++++++++++\n external-odb.h | 8 ++\n odb-helper.c | 249 ++++++++++++++++++++++++++++++++++++++++++++++++\n odb-helper.h | 25 +++++\n sha1_file.c | 25 ++++-\n t/t0400-external-odb.sh | 46 +++++++++\n 8 files changed, 468 insertions(+), 1 deletion(-)\n create mode 100644 external-odb.c\n create mode 100644 external-odb.h\n create mode 100644 odb-helper.c\n create mode 100644 odb-helper.h\n create mode 100755 t/t0400-external-odb.sh\n\ndiff --git a/Makefile b/Makefile\nindex 461c845d33..dde4b0e868 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -783,6 +783,7 @@ LIB_OBJS += ewah/ewah_bitmap.o\n LIB_OBJS += ewah/ewah_io.o\n LIB_OBJS += ewah/ewah_rlw.o\n LIB_OBJS += exec_cmd.o\n+LIB_OBJS += external-odb.o\n LIB_OBJS += fetch-pack.o\n LIB_OBJS += fsck.o\n LIB_OBJS += gettext.o\n@@ -815,6 +816,7 @@ LIB_OBJS += notes-cache.o\n LIB_OBJS += notes-merge.o\n LIB_OBJS += notes-utils.o\n LIB_OBJS += object.o\n+LIB_OBJS += odb-helper.o\n LIB_OBJS += oidset.o\n LIB_OBJS += pack-bitmap.o\n LIB_OBJS += pack-bitmap-write.o\ndiff --git a/cache.h b/cache.h\nindex 06da3d8a3f..13694069b1 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1553,6 +1553,7 @@ extern void read_info_alternates(const char * relative_base, int depth);\n extern char *compute_alternate_path(const char *path, struct strbuf *err);\n typedef int alt_odb_fn(struct alternate_object_database *, void *);\n extern int foreach_alt_odb(alt_odb_fn, void*);\n+extern void prepare_external_alt_odb(void);\n \n /*\n * Allocate a \"struct alternate_object_database\" but do _not_ actually\ndiff --git a/external-odb.c b/external-odb.c\nnew file mode 100644\nindex 0000000000..e9c3f11666\n--- /dev/null\n+++ b/external-odb.c\n@@ -0,0 +1,113 @@\n+#include \"cache.h\"\n+#include \"external-odb.h\"\n+#include \"odb-helper.h\"\n+\n+static struct odb_helper *helpers;\n+static struct odb_helper **helpers_tail = &helpers;\n+\n+static struct odb_helper *find_or_create_helper(const char *name, int len)\n+{\n+\tstruct odb_helper *o;\n+\n+\tfor (o = helpers; o; o = o->next)\n+\t\tif (!strncmp(o->name, name, len) && !o->name[len])\n+\t\t\treturn o;\n+\n+\to = odb_helper_new(name, len);\n+\t*helpers_tail = o;\n+\thelpers_tail = &o->next;\n+\n+\treturn o;\n+}\n+\n+static int external_odb_config(const char *var, const char *value, void *data)\n+{\n+\tstruct odb_helper *o;\n+\tconst char *name;\n+\tint namelen;\n+\tconst char *subkey;\n+\n+\tif (parse_config_key(var, \"odb\", &name, &namelen, &subkey) < 0)\n+\t\treturn 0;\n+\n+\to = find_or_create_helper(name, namelen);\n+\n+\tif (!strcmp(subkey, \"scriptcommand\"))\n+\t\treturn git_config_string(&o->cmd, var, value);\n+\n+\treturn 0;\n+}\n+\n+static void external_odb_init(void)\n+{\n+\tstatic int initialized;\n+\n+\tif (initialized)\n+\t\treturn;\n+\tinitialized = 1;\n+\n+\tgit_config(external_odb_config, NULL);\n+}\n+\n+const char *external_odb_root(void)\n+{\n+\tstatic const char *root;\n+\tif (!root)\n+\t\troot = git_pathdup(\"objects/external\");\n+\treturn root;\n+}\n+\n+int external_odb_has_object(const unsigned char *sha1)\n+{\n+\tstruct odb_helper *o;\n+\n+\texternal_odb_init();\n+\n+\tfor (o = helpers; o; o = o->next)\n+\t\tif (odb_helper_has_object(o, sha1))\n+\t\t\treturn 1;\n+\treturn 0;\n+}\n+\n+int external_odb_get_object(const unsigned char *sha1)\n+{\n+\tstruct odb_helper *o;\n+\tconst char *path;\n+\n+\tif (!external_odb_has_object(sha1))\n+\t\treturn -1;\n+\n+\tpath = sha1_file_name_alt(external_odb_root(), sha1);\n+\tsafe_create_leading_directories_const(path);\n+\tprepare_external_alt_odb();\n+\n+\tfor (o = helpers; o; o = o->next) {\n+\t\tstruct strbuf tmpfile = STRBUF_INIT;\n+\t\tint ret;\n+\t\tint fd;\n+\n+\t\tif (!odb_helper_has_object(o, sha1))\n+\t\t\tcontinue;\n+\n+\t\tfd = create_object_tmpfile(&tmpfile, path);\n+\t\tif (fd < 0) {\n+\t\t\tstrbuf_release(&tmpfile);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tif (odb_helper_get_object(o, sha1, fd) < 0) {\n+\t\t\tclose(fd);\n+\t\t\tunlink(tmpfile.buf);\n+\t\t\tstrbuf_release(&tmpfile);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tclose_sha1_file(fd);\n+\t\tret = finalize_object_file(tmpfile.buf, path);\n+\t\tstrbuf_release(&tmpfile);\n+\t\tif (!ret)\n+\t\t\treturn 0;\n+\t}\n+\n+\treturn -1;\n+}\ndiff --git a/external-odb.h b/external-odb.h\nnew file mode 100644\nindex 0000000000..9989490c9e\n--- /dev/null\n+++ b/external-odb.h\n@@ -0,0 +1,8 @@\n+#ifndef EXTERNAL_ODB_H\n+#define EXTERNAL_ODB_H\n+\n+const char *external_odb_root(void);\n+int external_odb_has_object(const unsigned char *sha1);\n+int external_odb_get_object(const unsigned char *sha1);\n+\n+#endif /* EXTERNAL_ODB_H */\ndiff --git a/odb-helper.c b/odb-helper.c\nnew file mode 100644\nindex 0000000000..0e6f824e4a\n--- /dev/null\n+++ b/odb-helper.c\n@@ -0,0 +1,249 @@\n+#include \"cache.h\"\n+#include \"object.h\"\n+#include \"argv-array.h\"\n+#include \"odb-helper.h\"\n+#include \"run-command.h\"\n+#include \"sha1-lookup.h\"\n+\n+struct odb_helper *odb_helper_new(const char *name, int namelen)\n+{\n+\tstruct odb_helper *o;\n+\n+\to = xcalloc(1, sizeof(*o));\n+\to->name = xmemdupz(name, namelen);\n+\n+\treturn o;\n+}\n+\n+struct odb_helper_cmd {\n+\tstruct argv_array argv;\n+\tstruct child_process child;\n+};\n+\n+/*\n+ * Callers are responsible to ensure that the result of vaddf(fmt, ap)\n+ * is properly shell-quoted.\n+ */\n+static void prepare_helper_command(struct argv_array *argv, const char *cmd,\n+\t\t\t\t const char *fmt, va_list ap)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tstrbuf_addstr(&buf, cmd);\n+\tstrbuf_addch(&buf, ' ');\n+\tstrbuf_vaddf(&buf, fmt, ap);\n+\n+\targv_array_push(argv, buf.buf);\n+\tstrbuf_release(&buf);\n+}\n+\n+__attribute__((format (printf,3,4)))\n+static int odb_helper_start(struct odb_helper *o,\n+\t\t\t struct odb_helper_cmd *cmd,\n+\t\t\t const char *fmt, ...)\n+{\n+\tva_list ap;\n+\n+\tmemset(cmd, 0, sizeof(*cmd));\n+\targv_array_init(&cmd->argv);\n+\n+\tif (!o->cmd)\n+\t\treturn -1;\n+\n+\tva_start(ap, fmt);\n+\tprepare_helper_command(&cmd->argv, o->cmd, fmt, ap);\n+\tva_end(ap);\n+\n+\tcmd->child.argv = cmd->argv.argv;\n+\tcmd->child.use_shell = 1;\n+\tcmd->child.no_stdin = 1;\n+\tcmd->child.out = -1;\n+\n+\tif (start_command(&cmd->child) < 0) {\n+\t\targv_array_clear(&cmd->argv);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int odb_helper_finish(struct odb_helper *o,\n+\t\t\t struct odb_helper_cmd *cmd)\n+{\n+\tint ret = finish_command(&cmd->child);\n+\targv_array_clear(&cmd->argv);\n+\tif (ret) {\n+\t\twarning(\"odb helper '%s' reported failure\", o->name);\n+\t\treturn -1;\n+\t}\n+\treturn 0;\n+}\n+\n+static int parse_object_line(struct odb_helper_object *o, const char *line)\n+{\n+\tchar *end;\n+\tif (get_sha1_hex(line, o->sha1) < 0)\n+\t\treturn -1;\n+\n+\tline += 40;\n+\tif (*line++ != ' ')\n+\t\treturn -1;\n+\n+\to->size = strtoul(line, &end, 10);\n+\tif (line == end || *end++ != ' ')\n+\t\treturn -1;\n+\n+\to->type = type_from_string(end);\n+\treturn 0;\n+}\n+\n+static int add_have_entry(struct odb_helper *o, const char *line)\n+{\n+\tALLOC_GROW(o->have, o->have_nr+1, o->have_alloc);\n+\tif (parse_object_line(&o->have[o->have_nr], line) < 0) {\n+\t\twarning(\"bad 'have' input from odb helper '%s': %s\",\n+\t\t\to->name, line);\n+\t\treturn 1;\n+\t}\n+\to->have_nr++;\n+\treturn 0;\n+}\n+\n+static int odb_helper_object_cmp(const void *va, const void *vb)\n+{\n+\tconst struct odb_helper_object *a = va, *b = vb;\n+\treturn hashcmp(a->sha1, b->sha1);\n+}\n+\n+static void odb_helper_load_have(struct odb_helper *o)\n+{\n+\tstruct odb_helper_cmd cmd;\n+\tFILE *fh;\n+\tstruct strbuf line = STRBUF_INIT;\n+\n+\tif (o->have_valid)\n+\t\treturn;\n+\to->have_valid = 1;\n+\n+\tif (odb_helper_start(o, &cmd, \"have\") < 0)\n+\t\treturn;\n+\n+\tfh = xfdopen(cmd.child.out, \"r\");\n+\twhile (strbuf_getline(&line, fh) != EOF)\n+\t\tif (add_have_entry(o, line.buf))\n+\t\t\tbreak;\n+\n+\tstrbuf_release(&line);\n+\tfclose(fh);\n+\todb_helper_finish(o, &cmd);\n+\n+\tqsort(o->have, o->have_nr, sizeof(*o->have), odb_helper_object_cmp);\n+}\n+\n+static struct odb_helper_object *odb_helper_lookup(struct odb_helper *o,\n+\t\t\t\t\t\t const unsigned char *sha1)\n+{\n+\tint idx;\n+\n+\todb_helper_load_have(o);\n+\tidx = sha1_entry_pos(o->have, sizeof(*o->have), 0,\n+\t\t\t 0, o->have_nr, o->have_nr,\n+\t\t\t sha1);\n+\tif (idx < 0)\n+\t\treturn NULL;\n+\treturn &o->have[idx];\n+}\n+\n+int odb_helper_has_object(struct odb_helper *o, const unsigned char *sha1)\n+{\n+\treturn !!odb_helper_lookup(o, sha1);\n+}\n+\n+int odb_helper_get_object(struct odb_helper *o, const unsigned char *sha1,\n+\t\t\t int fd)\n+{\n+\tstruct odb_helper_object *obj;\n+\tstruct odb_helper_cmd cmd;\n+\tunsigned long total_got;\n+\tgit_zstream stream;\n+\tint zret = Z_STREAM_END;\n+\tgit_SHA_CTX hash;\n+\tunsigned char real_sha1[20];\n+\n+\tobj = odb_helper_lookup(o, sha1);\n+\tif (!obj)\n+\t\treturn -1;\n+\n+\tif (odb_helper_start(o, &cmd, \"get_git_obj %s\", sha1_to_hex(sha1)) < 0)\n+\t\treturn -1;\n+\n+\tmemset(&stream, 0, sizeof(stream));\n+\tgit_inflate_init(&stream);\n+\tgit_SHA1_Init(&hash);\n+\ttotal_got = 0;\n+\n+\tfor (;;) {\n+\t\tunsigned char buf[4096];\n+\t\tint r;\n+\n+\t\tr = xread(cmd.child.out, buf, sizeof(buf));\n+\t\tif (r < 0) {\n+\t\t\terror(\"unable to read from odb helper '%s': %s\",\n+\t\t\t o->name, strerror(errno));\n+\t\t\tclose(cmd.child.out);\n+\t\t\todb_helper_finish(o, &cmd);\n+\t\t\tgit_inflate_end(&stream);\n+\t\t\treturn -1;\n+\t\t}\n+\t\tif (r == 0)\n+\t\t\tbreak;\n+\n+\t\twrite_or_die(fd, buf, r);\n+\n+\t\tstream.next_in = buf;\n+\t\tstream.avail_in = r;\n+\t\tdo {\n+\t\t\tunsigned char inflated[4096];\n+\t\t\tunsigned long got;\n+\n+\t\t\tstream.next_out = inflated;\n+\t\t\tstream.avail_out = sizeof(inflated);\n+\t\t\tzret = git_inflate(&stream, Z_SYNC_FLUSH);\n+\t\t\tgot = sizeof(inflated) - stream.avail_out;\n+\n+\t\t\tgit_SHA1_Update(&hash, inflated, got);\n+\t\t\t/* skip header when counting size */\n+\t\t\tif (!total_got) {\n+\t\t\t\tconst unsigned char *p = memchr(inflated, '\\0', got);\n+\t\t\t\tif (p)\n+\t\t\t\t\tgot -= p - inflated + 1;\n+\t\t\t\telse\n+\t\t\t\t\tgot = 0;\n+\t\t\t}\n+\t\t\ttotal_got += got;\n+\t\t} while (stream.avail_in && zret == Z_OK);\n+\t}\n+\n+\tclose(cmd.child.out);\n+\tgit_inflate_end(&stream);\n+\tgit_SHA1_Final(real_sha1, &hash);\n+\tif (odb_helper_finish(o, &cmd))\n+\t\treturn -1;\n+\tif (zret != Z_STREAM_END) {\n+\t\twarning(\"bad zlib data from odb helper '%s' for %s\",\n+\t\t\to->name, sha1_to_hex(sha1));\n+\t\treturn -1;\n+\t}\n+\tif (total_got != obj->size) {\n+\t\twarning(\"size mismatch from odb helper '%s' for %s (%lu != %lu)\",\n+\t\t\to->name, sha1_to_hex(sha1), total_got, obj->size);\n+\t\treturn -1;\n+\t}\n+\tif (hashcmp(real_sha1, sha1)) {\n+\t\twarning(\"sha1 mismatch from odb helper '%s' for %s (got %s)\",\n+\t\t\to->name, sha1_to_hex(sha1), sha1_to_hex(real_sha1));\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/odb-helper.h b/odb-helper.h\nnew file mode 100644\nindex 0000000000..5800661704\n--- /dev/null\n+++ b/odb-helper.h\n@@ -0,0 +1,25 @@\n+#ifndef ODB_HELPER_H\n+#define ODB_HELPER_H\n+\n+struct odb_helper {\n+\tconst char *name;\n+\tconst char *cmd;\n+\n+\tstruct odb_helper_object {\n+\t\tunsigned char sha1[20];\n+\t\tunsigned long size;\n+\t\tenum object_type type;\n+\t} *have;\n+\tint have_nr;\n+\tint have_alloc;\n+\tint have_valid;\n+\n+\tstruct odb_helper *next;\n+};\n+\n+struct odb_helper *odb_helper_new(const char *name, int namelen);\n+int odb_helper_has_object(struct odb_helper *o, const unsigned char *sha1);\n+int odb_helper_get_object(struct odb_helper *o, const unsigned char *sha1,\n+\t\t\t int fd);\n+\n+#endif /* ODB_HELPER_H */\ndiff --git a/sha1_file.c b/sha1_file.c\nindex d330996bc4..4bd790f6f8 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -28,6 +28,7 @@\n #include \"list.h\"\n #include \"mergesort.h\"\n #include \"quote.h\"\n+#include \"external-odb.h\"\n \n #define SZ_FMT PRIuMAX\n static inline uintmax_t sz_fmt(size_t s) { return s; }\n@@ -636,6 +637,21 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)\n \treturn r;\n }\n \n+void prepare_external_alt_odb(void)\n+{\n+\tstatic int linked_external;\n+\tconst char *path;\n+\n+\tif (linked_external)\n+\t\treturn;\n+\n+\tpath = external_odb_root();\n+\tif (!access(path, F_OK)) {\n+\t\tlink_alt_odb_entry(path, NULL, 0, \"\");\n+\t\tlinked_external = 1;\n+\t}\n+}\n+\n void prepare_alt_odb(void)\n {\n \tconst char *alt;\n@@ -650,6 +666,7 @@ void prepare_alt_odb(void)\n \tlink_alt_odb_entries(alt, strlen(alt), PATH_SEP, NULL, 0);\n \n \tread_info_alternates(get_object_directory(), 0);\n+\tprepare_external_alt_odb();\n }\n \n /* Returns 1 if we have successfully freshened the file, 0 otherwise. */\n@@ -690,7 +707,7 @@ static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)\n \t\tif (check_and_freshen_file(path, freshen))\n \t\t\treturn 1;\n \t}\n-\treturn 0;\n+\treturn external_odb_has_object(sha1);\n }\n \n static int check_and_freshen(const unsigned char *sha1, int freshen)\n@@ -1729,6 +1746,9 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,\n \t\t\treturn 0;\n \t}\n \n+\tif (!external_odb_get_object(sha1) && !lstat(*path, st))\n+\t\treturn 0;\n+\n \treturn -1;\n }\n \n@@ -1768,6 +1788,9 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)\n \tif (fd >= 0)\n \t\treturn fd;\n \n+\tif (!external_odb_get_object(sha1))\n+\t\tfd = open_sha1_file_alt(sha1, path);\n+\n \treturn fd;\n }\n \ndiff --git a/t/t0400-external-odb.sh b/t/t0400-external-odb.sh\nnew file mode 100755\nindex 0000000000..2f4749fab1\n--- /dev/null\n+++ b/t/t0400-external-odb.sh\n@@ -0,0 +1,46 @@\n+#!/bin/sh\n+\n+test_description='basic tests for external object databases'\n+\n+. ./test-lib.sh\n+\n+ALT_SOURCE=\"$PWD/alt-repo/.git\"\n+export ALT_SOURCE\n+write_script odb-helper <<\\EOF\n+GIT_DIR=$ALT_SOURCE; export GIT_DIR\n+case \"$1\" in\n+have)\n+\tgit cat-file --batch-check --batch-all-objects |\n+\tawk '{print $1 \" \" $3 \" \" $2}'\n+\t;;\n+get_git_obj)\n+\tcat \"$GIT_DIR\"/objects/$(echo $2 | sed 's#..#&/#')\n+\t;;\n+esac\n+EOF\n+HELPER=\"\\\"$PWD\\\"/odb-helper\"\n+\n+test_expect_success 'setup alternate repo' '\n+\tgit init alt-repo &&\n+\t(cd alt-repo &&\n+\t test_commit one &&\n+\t test_commit two\n+\t) &&\n+\talt_head=`cd alt-repo && git rev-parse HEAD`\n+'\n+\n+test_expect_success 'alt objects are missing' '\n+\ttest_must_fail git log --format=%s $alt_head\n+'\n+\n+test_expect_success 'helper can retrieve alt objects' '\n+\ttest_config odb.magic.scriptCommand \"$HELPER\" &&\n+\tcat >expect <<-\\EOF &&\n+\ttwo\n+\tone\n+\tEOF\n+\tgit log --format=%s $alt_head >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325540","messageId":"20170803091926.1755-9-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 08/40] Git/Packet.pm: add capability functions","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:18:54Z","receivedAt":"2017-08-03T09:22:35Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Add functions to help read and write capabilities.\nUse these functions in 't/t0021/rot13-filter.pl'.\n\nSigned-off-by: Christian Couder \n---\n perl/Git/Packet.pm | 33 +++++++++++++++++++++++++++++++++\n t/t0021/rot13-filter.pl | 9 ++-------\n 2 files changed, 35 insertions(+), 7 deletions(-)\n\ndiff --git a/perl/Git/Packet.pm b/perl/Git/Packet.pm\nindex b0233caf37..4443b67724 100644\n--- a/perl/Git/Packet.pm\n+++ b/perl/Git/Packet.pm\n@@ -20,6 +20,9 @@ our @EXPORT = qw(\n \t\t\tpacket_txt_write\n \t\t\tpacket_flush\n \t\t\tpacket_initialize\n+\t\t\tpacket_read_capabilities\n+\t\t\tpacket_write_capabilities\n+\t\t\tpacket_read_and_check_capabilities\n \t\t);\n our @EXPORT_OK = @EXPORT;\n \n@@ -83,3 +86,33 @@ sub packet_initialize {\n \tpacket_txt_write( \"version=\" . $version );\n \tpacket_flush();\n }\n+\n+sub packet_read_capabilities {\n+\tmy @cap;\n+\twhile (1) {\n+\t\tmy ( $res, $buf ) = packet_bin_read();\n+\t\treturn ( $res, @cap ) if ( $res != 0 );\n+\t\tunless ( $buf =~ s/\\n$// ) {\n+\t\t\tdie \"A non-binary line MUST be terminated by an LF.\\n\"\n+\t\t\t . \"Received: '$buf'\";\n+\t\t}\n+\t\tdie \"bad capability buf: '$buf'\" unless ( $buf =~ s/capability=// );\n+\t\tpush @cap, $buf;\n+\t}\n+}\n+\n+sub packet_read_and_check_capabilities {\n+\tmy @local_caps = @_;\n+\tmy @remote_res_caps = packet_read_capabilities();\n+\tmy $res = shift @remote_res_caps;\n+\tmy %remote_caps = map { $_ => 1 } @remote_res_caps;\n+\tforeach (@local_caps) {\n+\t die \"'$_' capability not available\" unless (exists($remote_caps{$_}));\n+\t}\n+\treturn $res;\n+}\n+\n+sub packet_write_capabilities {\n+\tpacket_txt_write( \"capability=\" . $_ ) foreach (@_);\n+\tpacket_flush();\n+}\ndiff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\nindex 5b05518640..bbfd52619d 100644\n--- a/t/t0021/rot13-filter.pl\n+++ b/t/t0021/rot13-filter.pl\n@@ -42,14 +42,9 @@ $debug->flush();\n \n packet_initialize(\"git-filter\", 2);\n \n-( packet_txt_read() eq ( 0, \"capability=clean\" ) ) || die \"bad capability\";\n-( packet_txt_read() eq ( 0, \"capability=smudge\" ) ) || die \"bad capability\";\n-( packet_bin_read() eq ( 1, \"\" ) ) || die \"bad capability end\";\n+packet_read_and_check_capabilities(\"clean\", \"smudge\");\n+packet_write_capabilities(@capabilities);\n \n-foreach (@capabilities) {\n-\tpacket_txt_write( \"capability=\" . $_ );\n-}\n-packet_flush();\n print $debug \"init handshake complete\\n\";\n $debug->flush();\n \n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325541","messageId":"20170803091926.1755-2-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 01/40] builtin/clone: get rid of 'value' strbuf","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:18:47Z","receivedAt":"2017-08-03T09:22:39Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This makes the code simpler by removing a few lines, and getting\nrid of one variable.\n\nSigned-off-by: Christian Couder \n---\n builtin/clone.c | 9 +++------\n 1 file changed, 3 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 08b5cc433c..4b5340c55f 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -871,7 +871,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tconst struct ref *our_head_points_at;\n \tstruct ref *mapped_refs;\n \tconst struct ref *ref;\n-\tstruct strbuf key = STRBUF_INIT, value = STRBUF_INIT;\n+\tstruct strbuf key = STRBUF_INIT;\n \tstruct strbuf branch_top = STRBUF_INIT, reflog_msg = STRBUF_INIT;\n \tstruct transport *transport = NULL;\n \tconst char *src_ref_prefix = \"refs/heads/\";\n@@ -1036,7 +1036,6 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\tstrbuf_addf(&branch_top, \"refs/remotes/%s/\", option_origin);\n \t}\n \n-\tstrbuf_addf(&value, \"+%s*:%s*\", src_ref_prefix, branch_top.buf);\n \tstrbuf_addf(&key, \"remote.%s.url\", option_origin);\n \tgit_config_set(key.buf, repo);\n \tstrbuf_reset(&key);\n@@ -1050,10 +1049,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (option_required_reference.nr || option_optional_reference.nr)\n \t\tsetup_reference();\n \n-\tfetch_pattern = value.buf;\n+\tfetch_pattern = xstrfmt(\"+%s*:%s*\", src_ref_prefix, branch_top.buf);\n \trefspec = parse_fetch_refspec(1, &fetch_pattern);\n-\n-\tstrbuf_reset(&value);\n+\tfree((char *)fetch_pattern);\n \n \tremote = remote_get(option_origin);\n \ttransport = transport_get(remote, remote->url[0]);\n@@ -1192,7 +1190,6 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tstrbuf_release(&reflog_msg);\n \tstrbuf_release(&branch_top);\n \tstrbuf_release(&key);\n-\tstrbuf_release(&value);\n \tjunk_mode = JUNK_LEAVE_ALL;\n \n \tfree(refspec);\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325542","messageId":"20170803091926.1755-6-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 05/40] t0021/rot13-filter: use Git/Packet.pm","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:18:51Z","receivedAt":"2017-08-03T09:22:41Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"After creating Git/Packet.pm from part of t0021/rot13-filter.pl,\nwe can now simplify this script by using Git/Packet.pm.\n\nSigned-off-by: Christian Couder \n---\n t/t0021/rot13-filter.pl | 51 +++----------------------------------------------\n 1 file changed, 3 insertions(+), 48 deletions(-)\n\ndiff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\nindex 1fc581c814..36a9eb3608 100644\n--- a/t/t0021/rot13-filter.pl\n+++ b/t/t0021/rot13-filter.pl\n@@ -19,9 +19,12 @@\n # same command.\n #\n \n+use 5.008;\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n use strict;\n use warnings;\n use IO::File;\n+use Git::Packet;\n \n my $MAX_PACKET_CONTENT_SIZE = 65516;\n my @capabilities = @ARGV;\n@@ -34,54 +37,6 @@ sub rot13 {\n \treturn $str;\n }\n \n-sub packet_bin_read {\n-\tmy $buffer;\n-\tmy $bytes_read = read STDIN, $buffer, 4;\n-\tif ( $bytes_read == 0 ) {\n-\t\t# EOF - Git stopped talking to us!\n-\t\treturn ( -1, \"\" );\n-\t} elsif ( $bytes_read != 4 ) {\n-\t\tdie \"invalid packet: '$buffer'\";\n-\t}\n-\tmy $pkt_size = hex($buffer);\n-\tif ( $pkt_size == 0 ) {\n-\t\treturn ( 1, \"\" );\n-\t} elsif ( $pkt_size > 4 ) {\n-\t\tmy $content_size = $pkt_size - 4;\n-\t\t$bytes_read = read STDIN, $buffer, $content_size;\n-\t\tif ( $bytes_read != $content_size ) {\n-\t\t\tdie \"invalid packet ($content_size bytes expected; $bytes_read bytes read)\";\n-\t\t}\n-\t\treturn ( 0, $buffer );\n-\t} else {\n-\t\tdie \"invalid packet size: $pkt_size\";\n-\t}\n-}\n-\n-sub packet_txt_read {\n-\tmy ( $res, $buf ) = packet_bin_read();\n-\tunless ( $res == -1 || $buf =~ s/\\n$// ) {\n-\t\tdie \"A non-binary line MUST be terminated by an LF.\";\n-\t}\n-\treturn ( $res, $buf );\n-}\n-\n-sub packet_bin_write {\n-\tmy $buf = shift;\n-\tprint STDOUT sprintf( \"%04x\", length($buf) + 4 );\n-\tprint STDOUT $buf;\n-\tSTDOUT->flush();\n-}\n-\n-sub packet_txt_write {\n-\tpacket_bin_write( $_[0] . \"\\n\" );\n-}\n-\n-sub packet_flush {\n-\tprint STDOUT sprintf( \"%04x\", 0 );\n-\tSTDOUT->flush();\n-}\n-\n print $debug \"START\\n\";\n $debug->flush();\n \n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325543","messageId":"20170803091926.1755-7-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 06/40] Git/Packet.pm: improve error message","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:18:52Z","receivedAt":"2017-08-03T09:22:43Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Try to give a bit more information when we die()\nbecause there is no new line at the end of something\nwe receive.\n\nSigned-off-by: Christian Couder \n---\n perl/Git/Packet.pm | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/perl/Git/Packet.pm b/perl/Git/Packet.pm\nindex aaffecbe2a..2ad6b00d6c 100644\n--- a/perl/Git/Packet.pm\n+++ b/perl/Git/Packet.pm\n@@ -49,7 +49,8 @@ sub packet_bin_read {\n sub packet_txt_read {\n \tmy ( $res, $buf ) = packet_bin_read();\n \tunless ( $res == -1 || $buf =~ s/\\n$// ) {\n-\t\tdie \"A non-binary line MUST be terminated by an LF.\";\n+\t\tdie \"A non-binary line MUST be terminated by an LF.\\n\"\n+\t\t . \"Received: '$buf'\";\n \t}\n \treturn ( $res, $buf );\n }\n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325544","messageId":"20170803091926.1755-3-chriscool@tuxfamily.org","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"[PATCH v5 02/40] t0021/rot13-filter: refactor packet reading functions","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-03T09:18:48Z","receivedAt":"2017-08-03T09:22:47Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"To make it possible in a following commit to move packet\nreading and writing functions into a Packet.pm module,\nlet's refactor these functions, so they don't handle\nprinting debug output and exiting.\n\nSigned-off-by: Christian Couder \n---\n t/t0021/rot13-filter.pl | 12 ++++++++----\n 1 file changed, 8 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t0021/rot13-filter.pl b/t/t0021/rot13-filter.pl\nindex 617f581e56..d6411ca523 100644\n--- a/t/t0021/rot13-filter.pl\n+++ b/t/t0021/rot13-filter.pl\n@@ -39,8 +39,7 @@ sub packet_bin_read {\n \tmy $bytes_read = read STDIN, $buffer, 4;\n \tif ( $bytes_read == 0 ) {\n \t\t# EOF - Git stopped talking to us!\n-\t\tprint $debug \"STOP\\n\";\n-\t\texit();\n+\t\treturn ( -1, \"\" );\n \t}\n \telsif ( $bytes_read != 4 ) {\n \t\tdie \"invalid packet: '$buffer'\";\n@@ -64,7 +63,7 @@ sub packet_bin_read {\n \n sub packet_txt_read {\n \tmy ( $res, $buf ) = packet_bin_read();\n-\tunless ( $buf =~ s/\\n$// ) {\n+\tunless ( $res == -1 || $buf =~ s/\\n$// ) {\n \t\tdie \"A non-binary line MUST be terminated by an LF.\";\n \t}\n \treturn ( $res, $buf );\n@@ -109,7 +108,12 @@ print $debug \"init handshake complete\\n\";\n $debug->flush();\n \n while (1) {\n-\tmy ($command) = packet_txt_read() =~ /^command=(.+)$/;\n+\tmy ($res, $command) = packet_txt_read();\n+\tif ( $res == -1 ) {\n+\t\tprint $debug \"STOP\\n\";\n+\t\texit();\n+\t}\n+\t$command =~ s/^command=//;\n \tprint $debug \"IN: $command\";\n \t$debug->flush();\n \n-- \n2.14.0.rc1.52.gf02fb0ddac.dirty\n\n"},{"id":"325571","messageId":"CAGZ79kYhUJ5mmTEO3b9G7M6onuCusBUTSsD7KeCmeMpfaOvroQ@mail.gmail.com","threadId":"46506","inReplyTo":"20170803091926.1755-36-chriscool@tuxfamily.org","subject":"Re: [PATCH v5 35/40] Add Documentation/technical/external-odb.txt","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-08-03T18:38:05Z","receivedAt":"2017-08-03T18:38:14Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, Aug 3, 2017 at 2:19 AM, Christian Couder\n wrote:\n> This describes the external odb mechanism's purpose and\n> how it works.\n\nThanks for providing this documentation patch!\n\nI read through it sequentially, see questions that came to mind\nin between.\n\nIf the very last paragraph came earlier (or an example), it\nwould have helped me to understand the big picture better.\n\n>\n> Helped-by: Ben Peart \n> Signed-off-by: Christian Couder \n> ---\n> Documentation/technical/external-odb.txt | 295 +++++++++++++++++++++++++++++++\n> 1 file changed, 295 insertions(+)\n> create mode 100644 Documentation/technical/external-odb.txt\n>\n> diff --git a/Documentation/technical/external-odb.txt b/Documentation/technical/external-odb.txt\n> new file mode 100644\n> index 0000000000..5991221fd5\n> --- /dev/null\n> +++ b/Documentation/technical/external-odb.txt\n> @@ -0,0 +1,295 @@\n> +External ODBs\n> +^^^^^^^^^^^^^\n> +\n> +The External ODB mechanism makes it possible for Git objects, mostly\n> +blobs for now though, to be stored in an \"external object database\"\n> +(External ODB).\n> +\n> +An External ODB can be any object store as long as there is an helper\n> +program called an \"odb helper\" that can communicate with Git to\n> +transfer objects to/from the external odb and to retrieve information\n> +about available objects in the external odb.\n> +\n> +Purpose\n> +=======\n> +\n> +The purpose of this mechanism is to make possible to handle Git\n> +objects, especially blobs, in much more flexible ways.\n> +\n> +Currently Git can store its objects only in the form of loose objects\n> +in separate files or packed objects in a pack file.\n> +\n> +This is not flexible enough for some important use cases like handling\n> +really big binary files or handling a really big number of files that\n> +are fetched only as needed. And it is not realistic to expect that Git\n> +could fully natively handle many of such use cases.\n\nThis is a strong statement. Why is it not realistic? What are these\n\"many of such use cases\"?\n\n> +Furthermore many improvements that are dependent on specific setups\n> +could be implemented in the way Git objects are managed if it was\n> +possible to customize how the Git objects are handled. For example a\n> +restartable clone using the bundle mechanism has often been requested,\n> +but implementing that would go against the current strict rules under\n> +which the Git objects are currently handled.\n\nSo in this example, you would use todays git-clone to obtain a small version\nof the repo and then obtain other objects later?\n\n> +What Git needs a mechanism to make it possible to customize in a lot\n> +of different ways how the Git objects are handled.\n\nI do not understand why we need this. Is this aimed to support git LFS,\nwhich by its model has additional objects not natively tracked by Git, that\nare fetched later when needed?\n\n> Though this\n> +mechanism should try as much as possible to avoid interfering with the\n> +usual way in which Git handle its objects.\n> +\n> +Helpers\n> +=======\n> +\n> +ODB helpers are commands that have to be registered using either the\n> +\"odb..subprocessCommand\" or the \"odb..scriptCommand\"\n> +config variables.\n> +\n> +Registering such a command tells Git that an external odb called\n> + exists and that the registered command should be used to\n> +communicate with it.\n> +\n> +There are 2 kinds of commands. Commands registered using the\n> +\"odb..subprocessCommand\" config variable are called \"process\n> +commands\" and the associated mode is called \"process mode\". Commands\n> +registered using the \"odb..scriptCommand\" config variables\n> +are called \"script commands\" and the associated mode is called \"script\n> +mode\".\n\nSo there is the possibility for multiple ODBs by the nature of the config\nas we can have multiple sections. How does Git know which\nodb to talk to? (does it talk to all of them when asking for a random object?)\n\nWhen writing an object how does Git decide where to store an object\n(internally or in one of its ODB? Maybe in multiple ODBs? Does the user\ngive rules how to tackle the problem or will Git have some magic to do\nthe right thing? If so where can I read about that?)\n\nOne could think that one ODB is able to learn about objects out of band\ni.e. to replace the fetch/clone/push mechanism, whereas another ODB is\ncapable of efficient fast local storage and yet another one that is optimized\nfor storing large binary files.\n\n> +Process Mode\n> +============\n> +\n> +In process mode the command is started as a single process invocation\n> +that should last for the entire life of the single Git command that\n> +started it.\n> +\n> +A packet format (pkt-line, see technical/protocol-common.txt) based\n> +protocol over standard input and standard output is used for\n> +communication between Git and the helper command.\n> +\n> +After the process command is started, Git sends a welcome message\n> +(\"git-read-object-client\"), a list of supported protocol version\n> +numbers, and a flush packet. Git expects to read a welcome response\n> +message (\"git-read-object-server\"), exactly one protocol version\n> +number from the previously sent list, and a flush packet. All further\n> +communication will be based on the selected version.\n> +\n> +The remaining protocol description below documents \"version=1\". Please\n> +note that \"version=42\" in the example below does not exist and is only\n> +there to illustrate how the protocol would look with more than one\n> +version.\n> +\n> +After the version negotiation Git sends a list of all capabilities\n> +that it supports and a flush packet. Git expects to read a list of\n> +desired capabilities, which must be a subset of the supported\n> +capabilities list, and a flush packet as response:\n> +\n> +------------------------\n> +packet: git> git-read-object-client\n> +packet: git> version=1\n> +packet: git> version=42\n> +packet: git> 0000\n> +packet: git< git-read-object-server\n> +packet: git< version=1\n> +packet: git< 0000\n> +packet: git> capability=get_raw_obj\n> +packet: git> capability=have\n> +packet: git> capability=put_raw_obj\n> +packet: git> capability=not-yet-invented\n> +packet: git> 0000\n> +packet: git< capability=get_raw_obj\n> +packet: git< 0000\n> +------------------------\n> +\n> +Afterwards Git sends a list of \"key=value\" pairs terminated with a\n> +flush packet. The list will contain at least the instruction (based on\n> +the supported capabilities) and the arguments for the\n> +instruction. Please note, that the process must not send any response\n> +before it received the final flush packet.\n> +\n> +In general any response from the helper should end with a status\n> +packet. See the documentation of the 'get_*' instructions below for\n> +examples of status packets.\n> +\n> +After the helper has processed an instruction, it is expected to wait\n> +for the next \"key=value\" list containing another instruction.\n> +\n> +On exit Git will close the pipe to the helper. The helper is then\n> +expected to detect EOF and exit gracefully on its own. Git will wait\n> +until the process has stopped.\n> +\n> +Script Mode\n> +===========\n> +\n> +In this mode Git launches the script command each time it wants to\n> +communicates with the helper. There is no welcome message and no\n> +protocol version in this mode.\n> +\n> +The instruction and associated arguments are passed as arguments when\n> +launching the script command and if needed further information is\n> +passed between Git and the command through stdin and stdout.\n> +\n> +Capabilities/Instructions\n> +=========================\n> +\n> +The following instructions are currently supported by Git:\n> +\n> +- init\n> +- get_git_obj\n> +- get_raw_obj\n> +- get_direct\n> +- put_raw_obj\n> +- have\n> +\n> +The plan is to also support 'put_git_obj' and 'put_direct' soon, for\n> +consistency with the 'get_*' instructions.\n> +\n> + - 'init'\n> +\n> +All the process and script commands must accept the 'init'\n> +instruction. It should be the first instruction sent to a command. It\n> +should not be advertised in the capability exchange. Any argument\n> +should be ignored.\n> +\n> +In process mode, after receiving the 'init' instruction and a flush\n> +packet, the helper should just send a status packet and then a flush\n> +packet. See the 'get_*' instructions below for examples of status\n> +packets.\n> +\n> +In script mode the command should print on stdout the capabilities\n> +that it supports if any. This is the only time in script mode when a\n> +capability exchange happens.\n> +\n> +For example a script command could use the following shell code\n> +snippet to handle the 'init' instruction:\n> +\n> +------------------------\n> +case \"$1\" in\n> +init)\n> + echo \"capability=get_git_obj\"\n> + echo \"capability=put_raw_obj\"\n> + echo \"capability=have\"\n> + ;;\n> +------------------------\n\nI can see the rationale for script mode, but not quite for process mode\nas in process mode we could do the same init work that is needed after\nthe welcome message?\n\nIs it kept in process mode to keep consistent with script mode?\n\nI assume this is to setup the ODB, which then can also state things like\n\"I am not in a state to work, as the network connection is missing\"\nor ask the user for a password for the encrypted database?\n\n> + - 'get_git_obj ' and 'get_raw_obj '\n> +\n> +These instructions should have a hexadecimal argument to tell\n> +which object the helper should send to git.\n> +\n> +In process mode the sha1 argument should be followed by a flush packet\n> +like this:\n> +\n> +------------------------\n> +packet: git> command=get_git_obj\n> +packet: git> sha1=0a214a649e1b3d5011e14a3dc227753f2bd2be05\n> +packet: git> 0000\n> +------------------------\n> +\n> +After reading that the helper should send the requested object to Git in a\n> +packet series followed by a flush packet. If the helper does not experience\n> +problems then the helper must send a \"success\" status like the following:\n> +\n> +------------------------\n> +packet: git< status=success\n> +packet: git< 0000\n> +------------------------\n> +\n> +In case the helper cannot or does not want to send the requested\n> +object as well as any other object for the lifetime of the Git\n> +process, then it is expected to respond with an \"abort\" status at any\n> +point in the protocol:\n> +\n> +------------------------\n> +packet: git< status=abort\n> +packet: git< 0000\n> +------------------------\n> +\n> +Git neither stops nor restarts the helper in case the \"error\"/\"abort\"\n> +status is set.\n> +\n> +If the helper dies during the communication or does not adhere to the\n> +protocol then Git will stop and restart it with the next instruction.\n> +\n> +In script mode the helper should just send the requested object to Git\n> +by writing it to stdout and should then exit. The exit code should\n> +signal to Git if a problem occured or not.\n> +\n> +The only difference between 'get_git_obj' and 'get_raw_obj' is that in\n> +case of 'get_git_obj' the requested object should be sent as a Git\n> +object (that is in the same format as loose object files). In case of\n> +'get_raw_obj' the object should be sent in its raw format (that is the\n> +same output as `git cat-file `).\n\nIn case of abort, what are the implications for Git? How do we deliver the\nmessage to the user (should the helper print to stderr, or is there a way\nto relay it through Git such that we do not have racy output?)\n\n> + - 'get_direct '\n> +\n> +This instruction is similar as the other 'get_*' instructions except\n> +that no object should be sent from the helper to Git. Instead the\n> +helper should directly write the requested object into a loose object\n> +file in the \".git/objects\" directory.\n> +\n> +After the helper has sent the \"status=success\" packet and the\n> +following flush packet in process mode, or after it has exited in the\n> +script mode, Git should lookup again for a loose object file with the\n> +requested sha1.\n\nDoes it have to be a loose object or is the helper also allowed\nto put a packfile into $GIT_OBJECT_DIRECTORY/pack ?\nIf so, is it expected to also produce an idx file?\n\n> +\n> + - 'put_raw_obj '\n> +\n> +This instruction should be following by three arguments to tell which\n> +object the helper will receive from git: , and\n> +. The hexadecimal argument describes the object that will\n> +be sent from Git to the helper. The is the object type (blob,\n> +tree, commit or tag) of this object. The is the size of the\n> +(decompressed) object content.\n\nSo the type is encoded as strings \"blob\", \"tree\" ... Maybe quote them?\n\nThe size is \"in bytes\" (maybe add that unit?). I expect there is no fanciness\nallowed such as \"3.3MB\" as that is not precise enough.\n\n> +In process mode the last argument (the type) should be followed by a\n> +flush packet.\n> +\n> +After reading that the helper should read the announced object from\n> +Git in a packet series followed by a flush packet.\n> +\n> +If the helper does not experience problems when receiving and storing\n> +or processing the object, then the helper must send a \"success\" status\n> +as described for the 'get_*' instructions.\n\nOtherwise an abort is expected?\n\n> +\n> +In script mode the helper should just receive the announced object\n> +from its standard input. After receiving and processing the object,\n> +the helper should exit and its exit code should signal to Git if a\n> +problem occured or not.\n> +\n> +- 'have'\n> +\n> +In process mode this instruction should be followed by a flush\n> +packet. After receiving this packet the helper should send the sha1,\n> +size and type, in this order, of all the objects it can provide to Git\n> +(through a 'get_*' instruction). There should be a space character\n> +between the sha1 and the size and between the size and the type, and\n> +then a new line character after the type.\n\nAs this is also inside a packet, do we need to care about splitting\nup the payload? i.e. when we have a lot of objects such that we need\nmultiple packets to present all 'have's, are we allowed to split\nup anywhere or just after a '\\n' ?\n\n> +If the helper does not experience problems, then it must then send a\n> +\"success\" status as described for the 'get_*' instructions.\n> +\n> +In script mode the helper should send to its standard output the sha1,\n> +size and type, in this order of all the objects it can provide to\n> +Git. There should also be a space character between the sha1 and the\n> +size and between the size and the type, and then a new line character\n> +after the type.\n> +\n> +After sending this, the script helper should exit and its exit code\n> +should signal to Git if a problem occured or not.\n> +\n> +Selecting objects\n> +=================\n> +\n> +To select objects that should be handled by an external odb, one can\n> +use the git attributes system. For now this will only work will blobs\n> +and this will only work along with the 'put_raw_obj' instruction.\n> +\n> +For example if one has an external odb called \"magic\" and has\n> +registered an associated a process command helper that supports the\n> +'put_raw_obj' instruction, then one can tell Git that all the .jpg\n> +files should be handled by the \"magic\" odb using a .gitattributes file\n> +can that contains:\n> +\n> +------------------------\n> +*.jpg odb=magic\n> +------------------------\n\nHah that answers some questions that are asked earlier!\n\nWhat happens if I say\n\n *.jpg odb=my-magic-store,my-jpeg-store\n\n?\n\nMaybe relevant:\nhttps://public-inbox.org/git/20170725211300.vwlpioy5jes55273@sigill.intra.peff.net/\n\"Extend the .gitattributes file to also specify file sizes\"\n\n> +\n> --\n> 2.14.0.rc1.52.gf02fb0ddac.dirty\n>\n\nThanks,\nStefan\n"},{"id":"325576","messageId":"xmqqo9rwxy8k.fsf@gitster.mtv.corp.google.com","threadId":"46506","inReplyTo":"20170803091926.1755-5-chriscool@tuxfamily.org","subject":"Re: [PATCH v5 04/40] Add Git/Packet.pm from parts of t0021/rot13-filter.pl","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-08-03T19:11:39Z","receivedAt":"2017-08-03T19:11:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> This will make it possible to reuse packet reading and writing\n> functions in other test scripts.\n>\n> Signed-off-by: Christian Couder \n> ---\n> perl/Git/Packet.pm | 71 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n> 1 file changed, 71 insertions(+)\n> create mode 100644 perl/Git/Packet.pm\n>\n> diff --git a/perl/Git/Packet.pm b/perl/Git/Packet.pm\n> new file mode 100644\n> index 0000000000..aaffecbe2a\n> --- /dev/null\n> +++ b/perl/Git/Packet.pm\n> @@ -0,0 +1,71 @@\n> +package Git::Packet;\n> +use 5.008;\n> +use strict;\n> +use warnings;\n> +BEGIN {\n> +\trequire Exporter;\n> +\tif ($] < 5.008003) {\n> +\t\t*import = \\&Exporter::import;\n> +\t} else {\n> +\t\t# Exporter 5.57 which supports this invocation was\n> +\t\t# released with perl 5.8.3\n> +\t\tExporter->import('import');\n> +\t}\n> +}\n\nThis is merely me being curious, but do we want this boilerplate,\nwhich we do not use in perl/Git.pm but we do in perl/Git/I18N.pm?\n\n> +our @EXPORT = qw(\n> +\t\t\tpacket_bin_read\n> +\t\t\tpacket_txt_read\n> +\t\t\tpacket_bin_write\n> +\t\t\tpacket_txt_write\n> +\t\t\tpacket_flush\n> +\t\t);\n> +our @EXPORT_OK = @EXPORT;\n\nWe can see that you made sure that the only thing 05/40 needs to do\nis to use this package and remove the definition of these subs,\nwithout having to touch any caller by first updating the original\nimplementation in 03/40 and then exporting these names in 04/40.\nKnowing that the preparation is nicely done already, it is a bit\nirritating to see that 05/40 is a separate patch, as we need to\nswitch between the patches to see if there is any difference between\nthe original implementation of the subs, and the replacement\nimplemented in here. It would have been nicer to have changes in\n04/40 and 05/40 in a single patch.\n\n"},{"id":"325577","messageId":"xmqqk22kxy3m.fsf@gitster.mtv.corp.google.com","threadId":"46506","inReplyTo":"20170803091926.1755-9-chriscool@tuxfamily.org","subject":"Re: [PATCH v5 08/40] Git/Packet.pm: add capability functions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-08-03T19:14:37Z","receivedAt":"2017-08-03T19:14:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> Add functions to help read and write capabilities.\n> Use these functions in 't/t0021/rot13-filter.pl'.\n>\n> Signed-off-by: Christian Couder \n> ---\n\nSteps 06-08/40 all look sensible to me, but they probably fall into\nthe same bucket as step 03/40, i.e. better done before step 04-05/40\n(which should probably be a single patch, as I earlier said) as\npreparatory steps.\n"},{"id":"325579","messageId":"xmqqfud8xx6m.fsf@gitster.mtv.corp.google.com","threadId":"46506","inReplyTo":"20170803091926.1755-11-chriscool@tuxfamily.org","subject":"Re: [PATCH v5 10/40] Add initial external odb support","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-08-03T19:34:25Z","receivedAt":"2017-08-03T19:34:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> +int external_odb_has_object(const unsigned char *sha1)\n> +{\n> +\tstruct odb_helper *o;\n> +\n> +\texternal_odb_init();\n> +\n> +\tfor (o = helpers; o; o = o->next)\n> +\t\tif (odb_helper_has_object(o, sha1))\n> +\t\t\treturn 1;\n> +\treturn 0;\n> +}\n> +\n> +int external_odb_get_object(const unsigned char *sha1)\n> +{\n> +\tstruct odb_helper *o;\n> +\tconst char *path;\n> +\n> +\tif (!external_odb_has_object(sha1))\n> +\t\treturn -1;\n\nThis probably would not matter, as I do not expect one repository to\nconnect to and backed by very many external odb instances, but I\nwould have expected that the interaction would go more like \"ah, we\nneed this object that is lacking locally. let's see if there is\nanybody with the object. now we found who claims to have the object,\nlet's ask that guy (and nobody else) to give the object to us\".\n\nIOW, I would have expected two functions:\n\n - struct odb_helper *external_odb_with(struct object_id *oid);\n - int external_odb_get(struct object_id *oid, struct odb_helper *odb);\n\nwhere the latter may start like\n\n if (!odb) {\n\todb = external_odb_with(oid);\n\tif (!odb)\n\t return -1;\n }\n ... go ask that odb for the object ...\n\n> diff --git a/external-odb.h b/external-odb.h\n> new file mode 100644\n> index 0000000000..9989490c9e\n> --- /dev/null\n> +++ b/external-odb.h\n> @@ -0,0 +1,8 @@\n> +#ifndef EXTERNAL_ODB_H\n> +#define EXTERNAL_ODB_H\n> +\n> +const char *external_odb_root(void);\n> +int external_odb_has_object(const unsigned char *sha1);\n> +int external_odb_get_object(const unsigned char *sha1);\n\nEven though ancient codebase of ours deliberately omitted them, I\nthink our recent trend is to explicitly spell \"extern \" in headers.\n\n> diff --git a/odb-helper.h b/odb-helper.h\n> new file mode 100644\n> index 0000000000..5800661704\n> --- /dev/null\n> +++ b/odb-helper.h\n\nLikewise.\n"},{"id":"325580","messageId":"xmqqbmnwxwfw.fsf@gitster.mtv.corp.google.com","threadId":"46506","inReplyTo":"20170803091926.1755-14-chriscool@tuxfamily.org","subject":"Re: [PATCH v5 13/40] external odb: add 'put_raw_obj' support","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-08-03T19:50:27Z","receivedAt":"2017-08-03T19:50:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> Add support for a 'put_raw_obj' capability/instruction to send new\n> objects to an external odb. Objects will be sent as they are (in\n> their 'raw' format). They will not be converted to Git objects.\n>\n> For now any new Git object (blob, tree, commit, ...) would be sent\n> if 'put_raw_obj' is supported by an odb helper. This is not a great\n> default, but let's leave it to following commits to tweak that.\n>\n> Signed-off-by: Christian Couder \n> ---\n\nI thought in an earlier step that I saw this thing initialized in\nthe codepath that adds alternate object stores, which are read-only\nplaces we \"borrow\" from. Being able to write into it is good, but\nconceptually it no longer feels correct to initialize it from the\nalternate object database initialization codepath.\n\nAnother way to say it is that an object store, whether it is local\nor external, is not \"alt\" if it will result in storing new objects\nwe locally create. It's just an extension of our local object\nstore.\n\n"},{"id":"325581","messageId":"xmqq7eykxwbt.fsf@gitster.mtv.corp.google.com","threadId":"46506","inReplyTo":"20170803091926.1755-15-chriscool@tuxfamily.org","subject":"Re: [PATCH v5 14/40] external-odb: accept only blobs for now","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-08-03T19:52:54Z","receivedAt":"2017-08-03T19:53:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> The mechanism to decide which blobs should be sent to which\n> external object database will be very simple for now.\n> If the external odb helper support any \"put_*\" instruction\n> all the new blobs will be sent to it.\n>\n> Signed-off-by: Christian Couder \n> ---\n> external-odb.c | 4 ++++\n> 1 file changed, 4 insertions(+)\n>\n> diff --git a/external-odb.c b/external-odb.c\n> index 82fac702e8..a4f8c72e1c 100644\n> --- a/external-odb.c\n> +++ b/external-odb.c\n> @@ -124,6 +124,10 @@ int external_odb_put_object(const void *buf, size_t len,\n> {\n> \tstruct odb_helper *o;\n> \n> +\t/* For now accept only blobs */\n> +\tif (strcmp(type, \"blob\"))\n> +\t\treturn 1;\n> +\n\nI somehow doubt that a policy decision like this should be made at\nthis layer. Shouldn't it be encoded in the capability the other\nside supports, or determined at runtime per each individual object\nwhen a \"put\" is attempted (i.e. allow the other side to say \"You\ntell me that you want me to store an object of type X and size Y;\nI cannot do that, sorry\").\n\n> \texternal_odb_init();\n> \n> \tfor (o = helpers; o; o = o->next) {\n"},{"id":"325582","messageId":"xmqq3798xvns.fsf@gitster.mtv.corp.google.com","threadId":"46506","inReplyTo":"20170803091926.1755-20-chriscool@tuxfamily.org","subject":"Re: [PATCH v5 19/40] lib-httpd: add upload.sh","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-08-03T20:07:19Z","receivedAt":"2017-08-03T20:07:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> +OLDIFS=\"$IFS\"\n> +IFS='&'\n> +set -- $QUERY_STRING\n> +IFS=\"$OLDIFS\"\n> +\n> +while test $# -gt 0\n> +do\n> + key=${1%=*}\n> + val=${1#*=}\n\nWhen you see that ${V%X*} and ${V#*X} appear in a pair for the same\nvariable V and same delimiter X, it almost always indicates a bug\nwaiting to happen.\n\nWhat's the definition of \"key\" here? A member of known set of short\ntokens, all of which consists only of alphanumeric, or something?\nEven if you do not currently plan to deal with a value with '=' in\nit, it may be prudent to double '%' above (and do not double '#').\n\nStyle: indent your shell script with tabs.\n\n> +\n> + case \"$key\" in\n> +\t\"sha1\") sha1=\"$val\" ;;\n> +\t\"type\") type=\"$val\" ;;\n> +\t\"size\") size=\"$val\" ;;\n> +\t\"delete\") delete=1 ;;\n> +\t*) echo >&2 \"unknown key '$key'\" ;;\n> + esac\n\nIndent your shell script with tabs; case/esac and the labels used\nfor each case arms all align at the same column.\n\n> +\n> + shift\n> +done\n> +\n> +case \"$REQUEST_METHOD\" in\n> + POST)\n\nLikewise.\n\n> + if test \"$delete\" = \"1\"\n> + then\n> +\trm -f \"$FILES_DIR/$sha1-$size-$type\"\n> + else\n> +\tmkdir -p \"$FILES_DIR\"\n> +\tcat >\"$FILES_DIR/$sha1-$size-$type\"\n> + fi\n> +\n> + echo 'Status: 204 No Content'\n> + echo\n> + ;;\n> +\n> + *)\n> + echo 'Status: 405 Method Not Allowed'\n> + echo\n> +esac\n"},{"id":"325586","messageId":"20170803201736.b37ziyub3wdlh6iw@sigill.intra.peff.net","threadId":"46506","inReplyTo":"xmqqfud8xx6m.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v5 10/40] Add initial external odb support","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-08-03T20:17:37Z","receivedAt":"2017-08-03T20:17:44Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Aug 03, 2017 at 12:34:25PM -0700, Junio C Hamano wrote:\n\n> > +int external_odb_get_object(const unsigned char *sha1)\n> > +{\n> > +\tstruct odb_helper *o;\n> > +\tconst char *path;\n> > +\n> > +\tif (!external_odb_has_object(sha1))\n> > +\t\treturn -1;\n> \n> This probably would not matter, as I do not expect one repository to\n> connect to and backed by very many external odb instances, but I\n> would have expected that the interaction would go more like \"ah, we\n> need this object that is lacking locally. let's see if there is\n> anybody with the object. now we found who claims to have the object,\n> let's ask that guy (and nobody else) to give the object to us\".\n> \n> IOW, I would have expected two functions:\n> \n> - struct odb_helper *external_odb_with(struct object_id *oid);\n> - int external_odb_get(struct object_id *oid, struct odb_helper *odb);\n\nOne advantage of walking through them linearly and asking \"can you get\nit?\" is that it gracefully handles external odbs which aren't available.\nThat can be used for redundancy, or for situations where a preferred\nodb isn't always available (e.g., a fast server which is only available\nwhen you're on a particular network).\n\n-Peff\n"},{"id":"325596","messageId":"xmqq4ltowcrz.fsf@gitster.mtv.corp.google.com","threadId":"46506","inReplyTo":"20170803091926.1755-26-chriscool@tuxfamily.org","subject":"Re: [PATCH v5 25/40] external-odb: add 'get_direct' support","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-08-03T21:40:32Z","receivedAt":"2017-08-03T21:40:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> This implements the 'get_direct' capability/instruction that makes\n> it possible for external odb helper scripts to pass blobs to Git\n> by directly writing them as loose objects files.\n\nI am not sure if the assumption is made clear in this series, but I\nam (perhaps incorrectly) guessing that it is assumed that the\nintended use of this feature is to offload access to large blobs\nby not including them in the initial clone. So from that point of\nview, I think it makes tons of sense to let the external helper to\ndirectly populate the database bypassing Git (i.e. instead of\nfeeding data stream and have Git store it) like this \"direct\" method\ndoes.\n\nHow does this compare with (and how well does this work with) what\nJonathan Tan is doing recently?\n"},{"id":"325607","messageId":"CAP8UFD3KaTEWiPmdNGmMx_5Dc+UDTWbPgeV8-=7y26-26q0Mxw@mail.gmail.com","threadId":"46506","inReplyTo":"xmqqo9rwxy8k.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v5 04/40] Add Git/Packet.pm from parts of t0021/rot13-filter.pl","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-04T06:32:01Z","receivedAt":"2017-08-04T06:32:07Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, Aug 3, 2017 at 9:11 PM, Junio C Hamano wrote:\n\n>> diff --git a/perl/Git/Packet.pm b/perl/Git/Packet.pm\n>> new file mode 100644\n>> index 0000000000..aaffecbe2a\n>> --- /dev/null\n>> +++ b/perl/Git/Packet.pm\n>> @@ -0,0 +1,71 @@\n>> +package Git::Packet;\n>> +use 5.008;\n>> +use strict;\n>> +use warnings;\n>> +BEGIN {\n>> + require Exporter;\n>> + if ($] < 5.008003) {\n>> + *import = \\&Exporter::import;\n>> + } else {\n>> + # Exporter 5.57 which supports this invocation was\n>> + # released with perl 5.8.3\n>> + Exporter->import('import');\n>> + }\n>> +}\n>\n> This is merely me being curious, but do we want this boilerplate,\n> which we do not use in perl/Git.pm but we do in perl/Git/I18N.pm?\n\nI don't know. I copied it as I thought that we wanted to support Perl\nversions starting from 5.8.0, but I am ok to remove it or to leave it\ndepending on what the Perl experts think (CCing AEvar) and what we\ndecide.\n\n>> +our @EXPORT = qw(\n>> + packet_bin_read\n>> + packet_txt_read\n>> + packet_bin_write\n>> + packet_txt_write\n>> + packet_flush\n>> + );\n>> +our @EXPORT_OK = @EXPORT;\n>\n> We can see that you made sure that the only thing 05/40 needs to do\n> is to use this package and remove the definition of these subs,\n> without having to touch any caller by first updating the original\n> implementation in 03/40 and then exporting these names in 04/40.\n> Knowing that the preparation is nicely done already, it is a bit\n> irritating to see that 05/40 is a separate patch, as we need to\n> switch between the patches to see if there is any difference between\n> the original implementation of the subs, and the replacement\n> implemented in here. It would have been nicer to have changes in\n> 04/40 and 05/40 in a single patch.\n\nOk, I have squashed 04/40 and 05/40 together in my current version of\nthis series.\n"},{"id":"325625","messageId":"CAP8UFD1wMFPzRD76QfAH2L9kAZs592-_F0sr3oQm0DMF09jL9Q@mail.gmail.com","threadId":"46506","inReplyTo":"xmqqk22kxy3m.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v5 08/40] Git/Packet.pm: add capability functions","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-04T20:34:46Z","receivedAt":"2017-08-04T20:34:52Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, Aug 3, 2017 at 9:14 PM, Junio C Hamano wrote:\n> Christian Couder writes:\n>\n>> Add functions to help read and write capabilities.\n>> Use these functions in 't/t0021/rot13-filter.pl'.\n>>\n>> Signed-off-by: Christian Couder \n>> ---\n>\n> Steps 06-08/40 all look sensible to me, but they probably fall into\n> the same bucket as step 03/40, i.e. better done before step 04-05/40\n> (which should probably be a single patch, as I earlier said) as\n> preparatory steps.\n\nOk, I moved patches 06-08/40 before 04-05/40 in my current working version.\n"},{"id":"327172","messageId":"CAP8UFD1oONnj93UKf=nBzgOQtY2E+ZVvoLGDNGLsZVobfiN90Q@mail.gmail.com","threadId":"46506","inReplyTo":"CAGZ79kYhUJ5mmTEO3b9G7M6onuCusBUTSsD7KeCmeMpfaOvroQ@mail.gmail.com","subject":"Re: [PATCH v5 35/40] Add Documentation/technical/external-odb.txt","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-25T06:14:08Z","receivedAt":"2017-08-25T06:14:19Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, Aug 3, 2017 at 8:38 PM, Stefan Beller wrote:\n> On Thu, Aug 3, 2017 at 2:19 AM, Christian Couder\n> wrote:\n>> This describes the external odb mechanism's purpose and\n>> how it works.\n>\n> Thanks for providing this documentation patch!\n>\n> I read through it sequentially, see questions that came to mind\n> in between.\n\nThanks for your feedback!\n\n> If the very last paragraph came earlier (or an example), it\n> would have helped me to understand the big picture better.\n\nOk, I added the following at the end of the \"helpers\" section:\n\n\"Early on git commands send an 'init' instruction to the registered\ncommands. A capability negociation will take place during this\nrequest/response exchange which will let Git and the helpers know how\nthey can further collaborate. The attribute system can also be used to\ntell Git which objects should be handled by which helper.\"\n\n>> +Purpose\n>> +=======\n>> +\n>> +The purpose of this mechanism is to make possible to handle Git\n>> +objects, especially blobs, in much more flexible ways.\n>> +\n>> +Currently Git can store its objects only in the form of loose objects\n>> +in separate files or packed objects in a pack file.\n>> +\n>> +This is not flexible enough for some important use cases like handling\n>> +really big binary files or handling a really big number of files that\n>> +are fetched only as needed. And it is not realistic to expect that Git\n>> +could fully natively handle many of such use cases.\n>\n> This is a strong statement. Why is it not realistic? What are these\n> \"many of such use cases\"?\n\nWhat I mean is that the Git default storage (loose objects and packed\nobjects) cannot easily be optimized for many different kind of\ncontents. Currently it is optimized for a not huge number of not very\nbig text files, and it works quite well too when there are not too\nmany quite small binary files. And then there are tweaks that can be\nused to improve things in specific cases (for example if you have very\nbig text files, you can set \"core.bigfilethreshold\" to a size bigger\nthan your text files so that they will still be delta-compress as Peff\nexplained in a recent thread).\n\nAs Git is used by more and more by people having different needs, I\nthink it is not realistic to expect that we can optimize its object\nstorage for all these different needs. So a better strategy is to just\nlet them store objects in external stores.\n\nIf we wanted to optimize for different use cases without letting\npeople use external stores, we would anyway need to implement\ndifferent internal stores which would be a huge burden and which could\nlead us to re-implement things like HTTP servers that already exists\noutside Git.\n\nAbout these many use cases, I gave the \"really big binary files\"\nexample which is why Git LFS exists (and which GitLab is interested in\nbetter solving), and the \"really big number of files that are fetched\nonly as needed\" example which Microsoft is interested in solving. I\ncould also imagine that some people have both big text files and big\nbinary files in which case the \"core.bigfilethreshold\" might not work\nwell, or that some people already have blobs in some different stores\n(like HTTP servers, Docker registries, artifact stores, ...) and want\nto fetch them from there as much as possible. And then letting people\nuse different stores can make clones or fetches restartable which\nwould solve another problem people have long been complaining about...\n\nI will try to use the above explanations to better improve the\nstatement in the documentation though I don't want it to be as long as\nthe above. Do you have an idea about what the right balance should be?\n\n>> +Furthermore many improvements that are dependent on specific setups\n>> +could be implemented in the way Git objects are managed if it was\n>> +possible to customize how the Git objects are handled. For example a\n>> +restartable clone using the bundle mechanism has often been requested,\n>> +but implementing that would go against the current strict rules under\n>> +which the Git objects are currently handled.\n>\n> So in this example, you would use todays git-clone to obtain a small version\n> of the repo and then obtain other objects later?\n\nThe problem with explaining how it would work is that the\n--initial-refspec option is added to git clone later in the patch\nseries. And there could be changes in the later part of the patch\nseries. So I don't want to promise or explain too much here.\nBut maybe I could add another patch to better explain that at the end\nof the series.\n\n>> +What Git needs a mechanism to make it possible to customize in a lot\n>> +of different ways how the Git objects are handled.\n>\n> I do not understand why we need this. Is this aimed to support git LFS,\n> which by its model has additional objects not natively tracked by Git, that\n> are fetched later when needed?\n\nIt is aimed to support not just something like git LFS, but also many\ndifferent use cases (see my above explanations).\n\n>> Though this\n>> +mechanism should try as much as possible to avoid interfering with the\n>> +usual way in which Git handle its objects.\n>> +\n>> +Helpers\n>> +=======\n>> +\n>> +ODB helpers are commands that have to be registered using either the\n>> +\"odb..subprocessCommand\" or the \"odb..scriptCommand\"\n>> +config variables.\n>> +\n>> +Registering such a command tells Git that an external odb called\n>> + exists and that the registered command should be used to\n>> +communicate with it.\n>> +\n>> +There are 2 kinds of commands. Commands registered using the\n>> +\"odb..subprocessCommand\" config variable are called \"process\n>> +commands\" and the associated mode is called \"process mode\". Commands\n>> +registered using the \"odb..scriptCommand\" config variables\n>> +are called \"script commands\" and the associated mode is called \"script\n>> +mode\".\n>\n> So there is the possibility for multiple ODBs by the nature of the config\n> as we can have multiple sections. How does Git know which\n> odb to talk to? (does it talk to all of them when asking for a random object?)\n>\n> When writing an object how does Git decide where to store an object\n> (internally or in one of its ODB? Maybe in multiple ODBs? Does the user\n> give rules how to tackle the problem or will Git have some magic to do\n> the right thing? If so where can I read about that?)\n\nYeah, it's possible to configure many ODBs. In this case after the\n'init' instruction, Git will know what are the instructions supported\nby each ODB. If more than one ODB support a 'get_*' instruction, yeah,\nGit will ask the ODBs supporting a 'get_*' instruction in turn for\neach object it did not already find. If more than one ODB support a\n'put_*' instruction and if the attributes for a blob correspond to\nmore than one of these ODBs, yeah Git will try to \"put\" the blob into\nthese ODBs in turn until it succeeds.\n\n> One could think that one ODB is able to learn about objects out of band\n> i.e. to replace the fetch/clone/push mechanism, whereas another ODB is\n> capable of efficient fast local storage and yet another one that is optimized\n> for storing large binary files.\n\nYeah, all these things are possible.\n\nHopefully the following will clarify that:\n\n\"The communication happens through instructions that are sent by Git\nand that the commands should answer. If it makes sense, Git will send\nthe same instruction to many commands in the order in which they are\nconfigured.\"\n\n>> +Process Mode\n>> +============\n>> +\n>> +In process mode the command is started as a single process invocation\n>> +that should last for the entire life of the single Git command that\n>> +started it.\n>> +\n>> +A packet format (pkt-line, see technical/protocol-common.txt) based\n>> +protocol over standard input and standard output is used for\n>> +communication between Git and the helper command.\n>> +\n>> +After the process command is started, Git sends a welcome message\n>> +(\"git-read-object-client\"), a list of supported protocol version\n>> +numbers, and a flush packet. Git expects to read a welcome response\n>> +message (\"git-read-object-server\"), exactly one protocol version\n>> +number from the previously sent list, and a flush packet. All further\n>> +communication will be based on the selected version.\n>> +\n>> +The remaining protocol description below documents \"version=1\". Please\n>> +note that \"version=42\" in the example below does not exist and is only\n>> +there to illustrate how the protocol would look with more than one\n>> +version.\n>> +\n>> +After the version negotiation Git sends a list of all capabilities\n>> +that it supports and a flush packet. Git expects to read a list of\n>> +desired capabilities, which must be a subset of the supported\n>> +capabilities list, and a flush packet as response:\n>> +\n>> +------------------------\n>> +packet: git> git-read-object-client\n>> +packet: git> version=1\n>> +packet: git> version=42\n>> +packet: git> 0000\n>> +packet: git< git-read-object-server\n>> +packet: git< version=1\n>> +packet: git< 0000\n>> +packet: git> capability=get_raw_obj\n>> +packet: git> capability=have\n>> +packet: git> capability=put_raw_obj\n>> +packet: git> capability=not-yet-invented\n>> +packet: git> 0000\n>> +packet: git< capability=get_raw_obj\n>> +packet: git< 0000\n>> +------------------------\n>> +\n>> +Afterwards Git sends a list of \"key=value\" pairs terminated with a\n>> +flush packet. The list will contain at least the instruction (based on\n>> +the supported capabilities) and the arguments for the\n>> +instruction. Please note, that the process must not send any response\n>> +before it received the final flush packet.\n>> +\n>> +In general any response from the helper should end with a status\n>> +packet. See the documentation of the 'get_*' instructions below for\n>> +examples of status packets.\n>> +\n>> +After the helper has processed an instruction, it is expected to wait\n>> +for the next \"key=value\" list containing another instruction.\n>> +\n>> +On exit Git will close the pipe to the helper. The helper is then\n>> +expected to detect EOF and exit gracefully on its own. Git will wait\n>> +until the process has stopped.\n>> +\n>> +Script Mode\n>> +===========\n>> +\n>> +In this mode Git launches the script command each time it wants to\n>> +communicates with the helper. There is no welcome message and no\n>> +protocol version in this mode.\n>> +\n>> +The instruction and associated arguments are passed as arguments when\n>> +launching the script command and if needed further information is\n>> +passed between Git and the command through stdin and stdout.\n>> +\n>> +Capabilities/Instructions\n>> +=========================\n>> +\n>> +The following instructions are currently supported by Git:\n>> +\n>> +- init\n>> +- get_git_obj\n>> +- get_raw_obj\n>> +- get_direct\n>> +- put_raw_obj\n>> +- have\n>> +\n>> +The plan is to also support 'put_git_obj' and 'put_direct' soon, for\n>> +consistency with the 'get_*' instructions.\n>> +\n>> + - 'init'\n>> +\n>> +All the process and script commands must accept the 'init'\n>> +instruction. It should be the first instruction sent to a command. It\n>> +should not be advertised in the capability exchange. Any argument\n>> +should be ignored.\n>> +\n>> +In process mode, after receiving the 'init' instruction and a flush\n>> +packet, the helper should just send a status packet and then a flush\n>> +packet. See the 'get_*' instructions below for examples of status\n>> +packets.\n>> +\n>> +In script mode the command should print on stdout the capabilities\n>> +that it supports if any. This is the only time in script mode when a\n>> +capability exchange happens.\n>> +\n>> +For example a script command could use the following shell code\n>> +snippet to handle the 'init' instruction:\n>> +\n>> +------------------------\n>> +case \"$1\" in\n>> +init)\n>> + echo \"capability=get_git_obj\"\n>> + echo \"capability=put_raw_obj\"\n>> + echo \"capability=have\"\n>> + ;;\n>> +------------------------\n>\n> I can see the rationale for script mode, but not quite for process mode\n> as in process mode we could do the same init work that is needed after\n> the welcome message?\n>\n> Is it kept in process mode to keep consistent with script mode?\n\nYes and because I want only the 'init' instruction to be required.\nIn process mode, if there were no 'init' instruction how could Git\nknow if it is ok to send a 'get' instruction for example if it does\nnot yet know the helpers' capabilities, and how could a helper that\nonly setup things be called only once?\n\n> I assume this is to setup the ODB, which then can also state things like\n> \"I am not in a state to work, as the network connection is missing\"\n> or ask the user for a password for the encrypted database?\n\nYeah, the helper can also take advantage of 'init' to setup and check\neverything.\n\nDo you think I should clarify something?\n\n>> + - 'get_git_obj ' and 'get_raw_obj '\n>> +\n>> +These instructions should have a hexadecimal argument to tell\n>> +which object the helper should send to git.\n>> +\n>> +In process mode the sha1 argument should be followed by a flush packet\n>> +like this:\n>> +\n>> +------------------------\n>> +packet: git> command=get_git_obj\n>> +packet: git> sha1=0a214a649e1b3d5011e14a3dc227753f2bd2be05\n>> +packet: git> 0000\n>> +------------------------\n>> +\n>> +After reading that the helper should send the requested object to Git in a\n>> +packet series followed by a flush packet. If the helper does not experience\n>> +problems then the helper must send a \"success\" status like the following:\n>> +\n>> +------------------------\n>> +packet: git< status=success\n>> +packet: git< 0000\n>> +------------------------\n>> +\n>> +In case the helper cannot or does not want to send the requested\n>> +object as well as any other object for the lifetime of the Git\n>> +process, then it is expected to respond with an \"abort\" status at any\n>> +point in the protocol:\n>> +\n>> +------------------------\n>> +packet: git< status=abort\n>> +packet: git< 0000\n>> +------------------------\n>> +\n>> +Git neither stops nor restarts the helper in case the \"error\"/\"abort\"\n>> +status is set.\n>> +\n>> +If the helper dies during the communication or does not adhere to the\n>> +protocol then Git will stop and restart it with the next instruction.\n>> +\n>> +In script mode the helper should just send the requested object to Git\n>> +by writing it to stdout and should then exit. The exit code should\n>> +signal to Git if a problem occured or not.\n>> +\n>> +The only difference between 'get_git_obj' and 'get_raw_obj' is that in\n>> +case of 'get_git_obj' the requested object should be sent as a Git\n>> +object (that is in the same format as loose object files). In case of\n>> +'get_raw_obj' the object should be sent in its raw format (that is the\n>> +same output as `git cat-file `).\n>\n> In case of abort, what are the implications for Git? How do we deliver the\n> message to the user (should the helper print to stderr, or is there a way\n> to relay it through Git such that we do not have racy output?)\n\nThe helper can print something to stderr. Hopefully it will be printed\nusing one printf() call or something like that which will make it not\nso racy. (Or what kind of race are you talking about?) And Git will\nremove the current instruction from the helpers capabilities, so it\nwill not ask the same instruction again (for the duration of the\ncurrent git process).\n\n>> + - 'get_direct '\n>> +\n>> +This instruction is similar as the other 'get_*' instructions except\n>> +that no object should be sent from the helper to Git. Instead the\n>> +helper should directly write the requested object into a loose object\n>> +file in the \".git/objects\" directory.\n>> +\n>> +After the helper has sent the \"status=success\" packet and the\n>> +following flush packet in process mode, or after it has exited in the\n>> +script mode, Git should lookup again for a loose object file with the\n>> +requested sha1.\n>\n> Does it have to be a loose object or is the helper also allowed\n> to put a packfile into $GIT_OBJECT_DIRECTORY/pack ?\n> If so, is it expected to also produce an idx file?\n\nIt could also be a packfile and an idx file, but I expect most of\nthese kind of helpers will just create loose object files.\nI will clarify with:\n\n\"...Git will lookup again for the requested sha1 in its loose\nobject files and pack files.\"\n\n>> + - 'put_raw_obj '\n>> +\n>> +This instruction should be following by three arguments to tell which\n>> +object the helper will receive from git: , and\n>> +. The hexadecimal argument describes the object that will\n>> +be sent from Git to the helper. The is the object type (blob,\n>> +tree, commit or tag) of this object. The is the size of the\n>> +(decompressed) object content.\n>\n> So the type is encoded as strings \"blob\", \"tree\" ... Maybe quote them?\n\nOk, they will be quoted in the next version.\n\n> The size is \"in bytes\" (maybe add that unit?). I expect there is no fanciness\n> allowed such as \"3.3MB\" as that is not precise enough.\n\nYeah, I added \"in bytes\".\n\n>> +In process mode the last argument (the type) should be followed by a\n>> +flush packet.\n>> +\n>> +After reading that the helper should read the announced object from\n>> +Git in a packet series followed by a flush packet.\n>> +\n>> +If the helper does not experience problems when receiving and storing\n>> +or processing the object, then the helper must send a \"success\" status\n>> +as described for the 'get_*' instructions.\n>\n> Otherwise an abort is expected?\n\nThere are also \"notfound\" and \"error\" failures. I will clarify this\nwith the following:\n\n\"Git neither stops nor restarts the helper in case a\n\"notfound\"/\"error\"/\"abort\" status is set. An \"error\" status means a\npossibly more transient error than an abort. The helper should also\nsend a \"notfound\" error in case of a \"get_*\" instruction, which means\nthat the requested object cannot be found.\"\n\n>> +In script mode the helper should just receive the announced object\n>> +from its standard input. After receiving and processing the object,\n>> +the helper should exit and its exit code should signal to Git if a\n>> +problem occured or not.\n>> +\n>> +- 'have'\n>> +\n>> +In process mode this instruction should be followed by a flush\n>> +packet. After receiving this packet the helper should send the sha1,\n>> +size and type, in this order, of all the objects it can provide to Git\n>> +(through a 'get_*' instruction). There should be a space character\n>> +between the sha1 and the size and between the size and the type, and\n>> +then a new line character after the type.\n>\n> As this is also inside a packet, do we need to care about splitting\n> up the payload? i.e. when we have a lot of objects such that we need\n> multiple packets to present all 'have's, are we allowed to split\n> up anywhere or just after a '\\n' ?\n\nThe code only supports splitting just after a '\\n'. I will clarify with:\n\n\"If many packets are needed to send back all this information, the\nsplit between packets should be made after the new line characters.\"\n\n>> +If the helper does not experience problems, then it must then send a\n>> +\"success\" status as described for the 'get_*' instructions.\n>> +\n>> +In script mode the helper should send to its standard output the sha1,\n>> +size and type, in this order of all the objects it can provide to\n>> +Git. There should also be a space character between the sha1 and the\n>> +size and between the size and the type, and then a new line character\n>> +after the type.\n>> +\n>> +After sending this, the script helper should exit and its exit code\n>> +should signal to Git if a problem occured or not.\n>> +\n>> +Selecting objects\n>> +=================\n>> +\n>> +To select objects that should be handled by an external odb, one can\n>> +use the git attributes system. For now this will only work will blobs\n>> +and this will only work along with the 'put_raw_obj' instruction.\n>> +\n>> +For example if one has an external odb called \"magic\" and has\n>> +registered an associated a process command helper that supports the\n>> +'put_raw_obj' instruction, then one can tell Git that all the .jpg\n>> +files should be handled by the \"magic\" odb using a .gitattributes file\n>> +can that contains:\n>> +\n>> +------------------------\n>> +*.jpg odb=magic\n>> +------------------------\n>\n> Hah that answers some questions that are asked earlier!\n>\n> What happens if I say\n>\n> *.jpg odb=my-magic-store,my-jpeg-store\n>\n> ?\n\nI am not sure how the attributes system works but I think it should handle this.\nSo the above would mean that Git will try to send the .jpg files to\nboth the \"my-magic-store\" and the \"my-jpeg-store\" helpers. The order\ndepends on which of those appears first in the config files.\n\n> Maybe relevant:\n> https://public-inbox.org/git/20170725211300.vwlpioy5jes55273@sigill.intra.peff.net/\n> \"Extend the .gitattributes file to also specify file sizes\"\n\nYeah, it looks like this could help if some attributes could be set\ndepending on file sizes.\n\nThanks,\nChristian.\n"},{"id":"327223","messageId":"20170825142300.01b15d6b@twelve2.svl.corp.google.com","threadId":"46506","inReplyTo":"CAP8UFD1oONnj93UKf=nBzgOQtY2E+ZVvoLGDNGLsZVobfiN90Q@mail.gmail.com","subject":"Re: [PATCH v5 35/40] Add Documentation/technical/external-odb.txt","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-25T21:23:00Z","receivedAt":"2017-08-25T21:23:14Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Fri, 25 Aug 2017 08:14:08 +0200\nChristian Couder wrote:\n\n> As Git is used by more and more by people having different needs, I\n> think it is not realistic to expect that we can optimize its object\n> storage for all these different needs. So a better strategy is to just\n> let them store objects in external stores.\n[snip]\n> About these many use cases, I gave the \"really big binary files\"\n> example which is why Git LFS exists (and which GitLab is interested in\n> better solving), and the \"really big number of files that are fetched\n> only as needed\" example which Microsoft is interested in solving. I\n> could also imagine that some people have both big text files and big\n> binary files in which case the \"core.bigfilethreshold\" might not work\n> well, or that some people already have blobs in some different stores\n> (like HTTP servers, Docker registries, artifact stores, ...) and want\n> to fetch them from there as much as possible. \n\nThanks for explaining the use cases - this makes sense, especially the\nlast one which motivates the different modes for the \"get\" command\n(return raw bytes vs populating the Git repository with loose/packed\nobjects).\n\n> And then letting people\n> use different stores can make clones or fetches restartable which\n> would solve another problem people have long been complaining about...\n\nThis is unrelated to the rest of my e-mail, but out of curiosity, how\nwould a different store make clones or fetches restartable? Do you mean\nthat Git would invoke a \"fetch\" command through the ODB protocol instead\nof using its own native protocol?\n\n> >> +Furthermore many improvements that are dependent on specific setups\n> >> +could be implemented in the way Git objects are managed if it was\n> >> +possible to customize how the Git objects are handled. For example a\n> >> +restartable clone using the bundle mechanism has often been requested,\n> >> +but implementing that would go against the current strict rules under\n> >> +which the Git objects are currently handled.\n> >\n> > So in this example, you would use todays git-clone to obtain a small version\n> > of the repo and then obtain other objects later?\n> \n> The problem with explaining how it would work is that the\n> --initial-refspec option is added to git clone later in the patch\n> series. And there could be changes in the later part of the patch\n> series. So I don't want to promise or explain too much here.\n> But maybe I could add another patch to better explain that at the end\n> of the series.\n\nSuch an explanation, in whatever form (patch or e-mail) would be great,\nbecause I'm not sure of the interaction between fetches and the\nconnectivity check.\n\nThe approach I have taken in my own patches [1] is to (1) declare that\nif a lazy remote supplies an object, it promises to have everything\nreferred to by that object, and (2) we thus only need to check the\nobjects not from the lazy remote. Translated to the ODB world, (1) is\npossible in the Microsoft case and is trivial in all the cases where the\nODB provides only blobs (since blobs don't refer to any other object),\nand for (2), a \"list\" command should suffice.\n\nOne constraint is that we do not want to obtain (from the remote) or\nstore a separate list of what it has, to avoid the overhead. (I saw the\n--initial-refspec approach - that would not work if we want to avoid the\noverhead.)\n\nFor fetches, we remember the objects obtained from that specific remote\nby adding a special file, name to be determined (I used \".imported\" in\n[1]). (The same method is used to note objects lazily downloaded.) The\nrepack command understands the difference between these two types of\nobjects (patches for this are in progress).\n\nI'm not sure if this can be translated to the ODB world. The ODB can\ndeclare a special capability that fetch sends to the server in order to\ninform the server that it can exclude certain objects, and fetch can\ninform the ODB of the packfiles that it has written, but I'm not sure\nhow the ODB can \"remember\" what it has. The ODB could mark such packs\nwith \".managed\" to note that it is managed by that ODB, so Git shoudn't\ntouch it, but this means (for example) that Git can't GC them (and it\nseems also quite contradictory for an ODB to manage Git packfiles).\n\n[1] https://public-inbox.org/git/20170804145113.5ceafafa@twelve2.svl.corp.google.com/\n"},{"id":"327287","messageId":"e15513eb-aed9-8bd3-794d-009a91c1d20e@gmail.com","threadId":"46506","inReplyTo":"20170803091926.1755-36-chriscool@tuxfamily.org","subject":"Re: [PATCH v5 35/40] Add Documentation/technical/external-odb.txt","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-08-28T18:59:55Z","receivedAt":"2017-08-28T19:00:06Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 8/3/2017 5:19 AM, Christian Couder wrote:\n> This describes the external odb mechanism's purpose and\n> how it works.\n> \n> Helped-by: Ben Peart \n> Signed-off-by: Christian Couder \n> ---\n> Documentation/technical/external-odb.txt | 295 +++++++++++++++++++++++++++++++\n> 1 file changed, 295 insertions(+)\n> create mode 100644 Documentation/technical/external-odb.txt\n> \n> diff --git a/Documentation/technical/external-odb.txt b/Documentation/technical/external-odb.txt\n> new file mode 100644\n> index 0000000000..5991221fd5\n> --- /dev/null\n> +++ b/Documentation/technical/external-odb.txt\n> @@ -0,0 +1,295 @@\n> +External ODBs\n> +^^^^^^^^^^^^^\n> +\n> +The External ODB mechanism makes it possible for Git objects, mostly\n> +blobs for now though, to be stored in an \"external object database\"\n> +(External ODB).\n> +\n> +An External ODB can be any object store as long as there is an helper\n> +program called an \"odb helper\" that can communicate with Git to\n> +transfer objects to/from the external odb and to retrieve information\n> +about available objects in the external odb.\n> +\n> +Purpose\n> +=======\n> +\n> +The purpose of this mechanism is to make possible to handle Git\n> +objects, especially blobs, in much more flexible ways.\n> +\n> +Currently Git can store its objects only in the form of loose objects\n> +in separate files or packed objects in a pack file.\n> +\n> +This is not flexible enough for some important use cases like handling\n> +really big binary files or handling a really big number of files that\n> +are fetched only as needed. And it is not realistic to expect that Git\n> +could fully natively handle many of such use cases.\n> +\n> +Furthermore many improvements that are dependent on specific setups\n> +could be implemented in the way Git objects are managed if it was\n> +possible to customize how the Git objects are handled. For example a\n> +restartable clone using the bundle mechanism has often been requested,\n> +but implementing that would go against the current strict rules under\n> +which the Git objects are currently handled.\n> +\n> +What Git needs a mechanism to make it possible to customize in a lot\n> +of different ways how the Git objects are handled. Though this\n> +mechanism should try as much as possible to avoid interfering with the\n> +usual way in which Git handle its objects.\n> +\n> +Helpers\n> +=======\n> +\n> +ODB helpers are commands that have to be registered using either the\n> +\"odb..subprocessCommand\" or the \"odb..scriptCommand\"\n> +config variables.\n> +\n> +Registering such a command tells Git that an external odb called\n> + exists and that the registered command should be used to\n> +communicate with it.\n> +\n\nWhat order are the odb handlers called? Are they called before or after \nthe regular object store code for loose, pack and alternates? Is the \norder configurable?\n\n[...]\n> +\n> + - 'get_direct '\n> +\n> +This instruction is similar as the other 'get_*' instructions except\n> +that no object should be sent from the helper to Git. Instead the\n> +helper should directly write the requested object into a loose object\n> +file in the \".git/objects\" directory.\n> +\n> +After the helper has sent the \"status=success\" packet and the\n> +following flush packet in process mode, or after it has exited in the\n> +script mode, Git should lookup again for a loose object file with the\n> +requested sha1.\n\nWhen will git call get_direct vs one of the other get_* functions? Could \nthe functionality of enabling a helper to populate objects into the \nregular object store be provided by having a ODB helper that returned \nthe object data as requested by get_git_obj or get_raw_obj but also \nstored it in the regular object store as a loose object (or pack file) \nfor future calls?\n\n\n"},{"id":"327318","messageId":"CAP8UFD1J_kDL928J2YV_Me3JtymXWfrs2N98hsR7eSYbyU=-Qw@mail.gmail.com","threadId":"46506","inReplyTo":"20170825142300.01b15d6b@twelve2.svl.corp.google.com","subject":"Re: [PATCH v5 35/40] Add Documentation/technical/external-odb.txt","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-29T09:37:05Z","receivedAt":"2017-08-29T09:37:34Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Fri, Aug 25, 2017 at 11:23 PM, Jonathan Tan wrote:\n> On Fri, 25 Aug 2017 08:14:08 +0200\n> Christian Couder wrote:\n>\n>> As Git is used by more and more by people having different needs, I\n>> think it is not realistic to expect that we can optimize its object\n>> storage for all these different needs. So a better strategy is to just\n>> let them store objects in external stores.\n> [snip]\n>> About these many use cases, I gave the \"really big binary files\"\n>> example which is why Git LFS exists (and which GitLab is interested in\n>> better solving), and the \"really big number of files that are fetched\n>> only as needed\" example which Microsoft is interested in solving. I\n>> could also imagine that some people have both big text files and big\n>> binary files in which case the \"core.bigfilethreshold\" might not work\n>> well, or that some people already have blobs in some different stores\n>> (like HTTP servers, Docker registries, artifact stores, ...) and want\n>> to fetch them from there as much as possible.\n>\n> Thanks for explaining the use cases - this makes sense, especially the\n> last one which motivates the different modes for the \"get\" command\n> (return raw bytes vs populating the Git repository with loose/packed\n> objects).\n\nThanks for this comment. Now the beginning of the \"Purpose\" section\nlooks like this:\n\n--------\nThe purpose of this mechanism is to make possible to handle Git\nobjects, especially blobs, in much more flexible ways.\n\nCurrently Git can store its objects only in the form of loose objects\nin separate files or packed objects in a pack file. These existing\nobject stores cannot be easily optimized for many different kind of\ncontents.\n\nSo the current stores are not flexible enough for some important use\ncases like handling really big binary files or handling a really big\nnumber of files that are fetched only as needed. And it is not\nrealistic to expect that Git could fully natively handle many of such\nuse cases. Git would need to natively implement different internal\nstores which would be a huge burden and which could lead to\nre-implement things like HTTP servers, Docker registries or artifact\nstores that already exist outside Git.\n--------\n\n>> And then letting people\n>> use different stores can make clones or fetches restartable which\n>> would solve another problem people have long been complaining about...\n>\n> This is unrelated to the rest of my e-mail, but out of curiosity, how\n> would a different store make clones or fetches restartable? Do you mean\n> that Git would invoke a \"fetch\" command through the ODB protocol instead\n> of using its own native protocol?\n\nYeah, the idea is that during a clone (or a fetch), Git could first\nfetch some refs with \"meta information\", for example refs/odb/magic/*\n(where \"magic\" is the odb name) which would tell if some (new) bundles\nare available.\nIf there are (new) bundles available then during the 'init'\ninstruction, which takes place just after this first fetch, the\nexternal odb helper will notice that and \"fetch\" the (new) bundles\nusing a restartable protocol, for example HTTP.\n\nIf something goes wrong when the helper \"fetches\" a bundle, the helper\ncould force the clone (or the fetch) to error out (after maybe\nretrying), and when the user (or the helper itself) tries again to\nclone (or fetch), the helper would restart its bundle \"fetch\" (using\nthe restartable protocol).\n\nWhen this \"fetch\" eventually succeeds, then the helper will unbundle\nwhat it received, and then give back control to the second regular\npart of the clone (or fetch). This regular part of the clone (or\nfetch) will then try to fetch the usual refs, but as the unbundling\nhas already updated the content of the usual refs (as well as the\nobject stores) this fetch will find that everything is up-to-date.\n\nOk, maybe everything is not quite up-to-date and there are still\nthings to fetch, but anyway the biggest part of the clone (or fetch)\nhas already been made using a restartable protocol, so we are doing\nmuch better than if we are not restartable at all.\n\nThere are examples in t0430 at the end of the patch series of\nrestartable clones. They don't really test that one can indeed restart\na clone, but they show that things \"just work\" when `git clone` is\nsplit into an initial fetch (that updates \"meta information\" refs),\nthen the 'init' instruction sent to an helper (that fetches and\nunbundles a bundle based on the \"meta information\" refs) and then the\nregular part of the clone.\n\n>> >> +Furthermore many improvements that are dependent on specific setups\n>> >> +could be implemented in the way Git objects are managed if it was\n>> >> +possible to customize how the Git objects are handled. For example a\n>> >> +restartable clone using the bundle mechanism has often been requested,\n>> >> +but implementing that would go against the current strict rules under\n>> >> +which the Git objects are currently handled.\n>> >\n>> > So in this example, you would use todays git-clone to obtain a small version\n>> > of the repo and then obtain other objects later?\n>>\n>> The problem with explaining how it would work is that the\n>> --initial-refspec option is added to git clone later in the patch\n>> series. And there could be changes in the later part of the patch\n>> series. So I don't want to promise or explain too much here.\n>> But maybe I could add another patch to better explain that at the end\n>> of the series.\n>\n> Such an explanation, in whatever form (patch or e-mail) would be great,\n> because I'm not sure of the interaction between fetches and the\n> connectivity check.\n\nOk, I will add parts of the above explanations to a documentation\npatch at the end of the patch series.\n\n> The approach I have taken in my own patches [1] is to (1) declare that\n> if a lazy remote supplies an object, it promises to have everything\n> referred to by that object, and (2) we thus only need to check the\n> objects not from the lazy remote. Translated to the ODB world, (1) is\n> possible in the Microsoft case and is trivial in all the cases where the\n> ODB provides only blobs (since blobs don't refer to any other object),\n> and for (2), a \"list\" command should suffice.\n\nOk, the \"list\" command is a command that any lazy remote should\nimplement so that other repos can ask it which are all the objects it\nhas, right?\n\n> One constraint is that we do not want to obtain (from the remote) or\n> store a separate list of what it has, to avoid the overhead.\n\nSo the answer to the \"list\" command should be part of the answer which\nsends all the objects.\n\n> (I saw the\n> --initial-refspec approach - that would not work if we want to avoid the\n> overhead.)\n\nThe --initial-refspec approach is interesting if you want to fetch a\nbig number of objects or many big objects, like when you do an initial\nclone of a big repo. In this use case a relatively small amount of\ntime spent in the initial fetch is an acceptable trade-off if the\nclone or the fetch is restartable.\n\nAlso as the --initial-refspec clone or fetch could alleviate resource\nusage of the server, it could be even faster than a regular clone or\nfetch in this case.\n\nI don't think the --initial-refspec option should be used all the time\nwhen an external odb is configured. But using an external odb in the\nfirst place means that you have specific requirements which suggests\nthat the regular way to clone (or fetch or push) might not be very\ngood for your use cases and for the objects that are stored in the\nexternal ODB.\n\nWhen you don't use --intial-refspec and an external odb helper is\nconfigured, what happens is that the objects managed by the external\nodb are not put in the pack file that is sent, so the receiver should\nalso have configured an external odb helper that can get the missing\nobjects otherwise Git will error out complaining about missing\nobjects.\n\nThis has some drawbacks of course, but at least it makes sure that\nusers' repositories are properly configured before they can start\nworking with a server using an external ODB.\n\n> For fetches, we remember the objects obtained from that specific remote\n> by adding a special file, name to be determined (I used \".imported\" in\n> [1]). (The same method is used to note objects lazily downloaded.) The\n> repack command understands the difference between these two types of\n> objects (patches for this are in progress).\n>\n> I'm not sure if this can be translated to the ODB world. The ODB can\n> declare a special capability that fetch sends to the server in order to\n> inform the server that it can exclude certain objects, and fetch can\n> inform the ODB of the packfiles that it has written, but I'm not sure\n> how the ODB can \"remember\" what it has.\n\nThe current design of the \"ODB world\" doesn't require a new capability\nand I think that is a good thing.\nMaybe it will be (or it is already) possible to optimize it using a\nnew capability, but I think it is a good thing to separate the new\ncapability if possible.\n\nI know I have not answered all the previous emails and I will try to\nanswer soon, but I try to improve the documentation (and the code) at\nthe same time, so that hopefully it makes thing clearer for people who\nwill have similar questions later.\n\n> The ODB could mark such packs\n> with \".managed\" to note that it is managed by that ODB, so Git shoudn't\n> touch it, but this means (for example) that Git can't GC them (and it\n> seems also quite contradictory for an ODB to manage Git packfiles).\n\nI don't like very much ODB managing packs. Yeah, the \"get_direct\"\ninstruction and cloning using bundles require the helpers to kind of\nwrite loose files or pack files, but they should really use git\ncommands to do that and then stop taking care.\nThe more simple the helper can be, the better.\n"},{"id":"327322","messageId":"CAP8UFD2VFsJWyP1TWLS15wYz2vQxHd4hG5Bdbo9-B45Q-onG2w@mail.gmail.com","threadId":"46506","inReplyTo":"e15513eb-aed9-8bd3-794d-009a91c1d20e@gmail.com","subject":"Re: [PATCH v5 35/40] Add Documentation/technical/external-odb.txt","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-29T15:43:27Z","receivedAt":"2017-08-29T15:43:33Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Mon, Aug 28, 2017 at 8:59 PM, Ben Peart wrote:\n>\n> On 8/3/2017 5:19 AM, Christian Couder wrote:\n>>\n>> +Helpers\n>> +=======\n>> +\n>> +ODB helpers are commands that have to be registered using either the\n>> +\"odb..subprocessCommand\" or the \"odb..scriptCommand\"\n>> +config variables.\n>> +\n>> +Registering such a command tells Git that an external odb called\n>> + exists and that the registered command should be used to\n>> +communicate with it.\n>\n> What order are the odb handlers called? Are they called before or after the\n> regular object store code for loose, pack and alternates? Is the order\n> configurable?\n\nFor get_*_object instructions the regular code is called before the odb helpers.\n(So the odb helper code is at the end of stat_sha1_file() and of\nopen_sha1_file() in sha1_file.c.)\n\nFor put_*_object instructions the regular code is called after the odb helpers.\n(So the odb helper code is at the beginning of write_sha1_file() in\nsha1_file.c.)\n\nAnd no this order is not configurable, but of course it could be made\nconfigurable.\n\n>> + - 'get_direct '\n>> +\n>> +This instruction is similar as the other 'get_*' instructions except\n>> +that no object should be sent from the helper to Git. Instead the\n>> +helper should directly write the requested object into a loose object\n>> +file in the \".git/objects\" directory.\n>> +\n>> +After the helper has sent the \"status=success\" packet and the\n>> +following flush packet in process mode, or after it has exited in the\n>> +script mode, Git should lookup again for a loose object file with the\n>> +requested sha1.\n>\n> When will git call get_direct vs one of the other get_* functions?\n\nIt is called just before exiting when git cannot find an object.\nIt is not exactly at the same place as other get_* instructions as I\ntried to reuse your code and as it looks like it makes it easier to\nretry the regular code after the odb helper code.\n\n> Could the\n> functionality of enabling a helper to populate objects into the regular\n> object store be provided by having a ODB helper that returned the object\n> data as requested by get_git_obj or get_raw_obj but also stored it in the\n> regular object store as a loose object (or pack file) for future calls?\n\nI am not sure I understand what you mean.\nIf a helper returns the object data as requested by get_git_obj or\nget_raw_obj, then currently Git will itself store the object locally\nin its regular object store, so it is redundant for the helper to also\nstore or try to store the object in the regular object store.\n"},{"id":"327396","messageId":"dc9b7aed-c781-d0fc-688f-32d665ad61ab@gmail.com","threadId":"46506","inReplyTo":"CAP8UFD2VFsJWyP1TWLS15wYz2vQxHd4hG5Bdbo9-B45Q-onG2w@mail.gmail.com","subject":"Re: [PATCH v5 35/40] Add Documentation/technical/external-odb.txt","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-08-30T12:50:52Z","receivedAt":"2017-08-30T12:56:17Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 8/29/2017 11:43 AM, Christian Couder wrote:\n> On Mon, Aug 28, 2017 at 8:59 PM, Ben Peart wrote:\n>>\n>> On 8/3/2017 5:19 AM, Christian Couder wrote:\n>>>\n>>> +Helpers\n>>> +=======\n>>> +\n>>> +ODB helpers are commands that have to be registered using either the\n>>> +\"odb..subprocessCommand\" or the \"odb..scriptCommand\"\n>>> +config variables.\n>>> +\n>>> +Registering such a command tells Git that an external odb called\n>>> + exists and that the registered command should be used to\n>>> +communicate with it.\n>>\n>> What order are the odb handlers called? Are they called before or after the\n>> regular object store code for loose, pack and alternates? Is the order\n>> configurable?\n> \n> For get_*_object instructions the regular code is called before the odb helpers.\n> (So the odb helper code is at the end of stat_sha1_file() and of\n> open_sha1_file() in sha1_file.c.)\n> \n> For put_*_object instructions the regular code is called after the odb helpers.\n> (So the odb helper code is at the beginning of write_sha1_file() in\n> sha1_file.c.)\n> \n> And no this order is not configurable, but of course it could be made\n> configurable.\n> \n>>> + - 'get_direct '\n>>> +\n>>> +This instruction is similar as the other 'get_*' instructions except\n>>> +that no object should be sent from the helper to Git. Instead the\n>>> +helper should directly write the requested object into a loose object\n>>> +file in the \".git/objects\" directory.\n>>> +\n>>> +After the helper has sent the \"status=success\" packet and the\n>>> +following flush packet in process mode, or after it has exited in the\n>>> +script mode, Git should lookup again for a loose object file with the\n>>> +requested sha1.\n>>\n>> When will git call get_direct vs one of the other get_* functions?\n> \n> It is called just before exiting when git cannot find an object.\n> It is not exactly at the same place as other get_* instructions as I\n> tried to reuse your code and as it looks like it makes it easier to\n> retry the regular code after the odb helper code.\n> \n>> Could the\n>> functionality of enabling a helper to populate objects into the regular\n>> object store be provided by having a ODB helper that returned the object\n>> data as requested by get_git_obj or get_raw_obj but also stored it in the\n>> regular object store as a loose object (or pack file) for future calls?\n> \n> I am not sure I understand what you mean.\n> If a helper returns the object data as requested by get_git_obj or\n> get_raw_obj, then currently Git will itself store the object locally\n> in its regular object store, so it is redundant for the helper to also\n> store or try to store the object in the regular object store.\n> \n\nDoesn't this mean that objects will \"leak out\" into the regular object \nstore as they are used? For example, at checkout, all objects in the \nrequested commit would be retrieved from the various object stores and \nif they came from a \"large blob\" ODB handler, they would get retrieved \nfrom the ODB handler and then written to the regular object store \n(presumably as a loose object). From then on, the object would be \nretrieved from the regular object store.\n\nThis would seem to defeat the goal of enabling specialized object \nhandlers to handle large or other \"unusual\" objects that git normally \ndoesn't deal well with.\n"},{"id":"327402","messageId":"CAP8UFD3+4mtmfVB47CsbVzmOMGK-WR4GYzXu6MfvOdGwrqx9=g@mail.gmail.com","threadId":"46506","inReplyTo":"dc9b7aed-c781-d0fc-688f-32d665ad61ab@gmail.com","subject":"Re: [PATCH v5 35/40] Add Documentation/technical/external-odb.txt","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-30T14:15:04Z","receivedAt":"2017-08-30T14:15:10Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Aug 30, 2017 at 2:50 PM, Ben Peart wrote:\n>\n>\n> On 8/29/2017 11:43 AM, Christian Couder wrote:\n>>\n>> On Mon, Aug 28, 2017 at 8:59 PM, Ben Peart wrote:\n>>>\n>>>\n>>> On 8/3/2017 5:19 AM, Christian Couder wrote:\n>>>>\n>>>>\n>>>> +Helpers\n>>>> +=======\n>>>> +\n>>>> +ODB helpers are commands that have to be registered using either the\n>>>> +\"odb..subprocessCommand\" or the \"odb..scriptCommand\"\n>>>> +config variables.\n>>>> +\n>>>> +Registering such a command tells Git that an external odb called\n>>>> + exists and that the registered command should be used to\n>>>> +communicate with it.\n>>>\n>>>\n>>> What order are the odb handlers called? Are they called before or after\n>>> the\n>>> regular object store code for loose, pack and alternates? Is the order\n>>> configurable?\n>>\n>>\n>> For get_*_object instructions the regular code is called before the odb\n>> helpers.\n>> (So the odb helper code is at the end of stat_sha1_file() and of\n>> open_sha1_file() in sha1_file.c.)\n>>\n>> For put_*_object instructions the regular code is called after the odb\n>> helpers.\n>> (So the odb helper code is at the beginning of write_sha1_file() in\n>> sha1_file.c.)\n>>\n>> And no this order is not configurable, but of course it could be made\n>> configurable.\n>>\n>>>> + - 'get_direct '\n>>>> +\n>>>> +This instruction is similar as the other 'get_*' instructions except\n>>>> +that no object should be sent from the helper to Git. Instead the\n>>>> +helper should directly write the requested object into a loose object\n>>>> +file in the \".git/objects\" directory.\n>>>> +\n>>>> +After the helper has sent the \"status=success\" packet and the\n>>>> +following flush packet in process mode, or after it has exited in the\n>>>> +script mode, Git should lookup again for a loose object file with the\n>>>> +requested sha1.\n>>>\n>>>\n>>> When will git call get_direct vs one of the other get_* functions?\n>>\n>>\n>> It is called just before exiting when git cannot find an object.\n>> It is not exactly at the same place as other get_* instructions as I\n>> tried to reuse your code and as it looks like it makes it easier to\n>> retry the regular code after the odb helper code.\n>>\n>>> Could the\n>>> functionality of enabling a helper to populate objects into the regular\n>>> object store be provided by having a ODB helper that returned the object\n>>> data as requested by get_git_obj or get_raw_obj but also stored it in the\n>>> regular object store as a loose object (or pack file) for future calls?\n>>\n>>\n>> I am not sure I understand what you mean.\n>> If a helper returns the object data as requested by get_git_obj or\n>> get_raw_obj, then currently Git will itself store the object locally\n>> in its regular object store, so it is redundant for the helper to also\n>> store or try to store the object in the regular object store.\n>>\n>\n> Doesn't this mean that objects will \"leak out\" into the regular object store\n> as they are used? For example, at checkout, all objects in the requested\n> commit would be retrieved from the various object stores and if they came\n> from a \"large blob\" ODB handler, they would get retrieved from the ODB\n> handler and then written to the regular object store (presumably as a loose\n> object). From then on, the object would be retrieved from the regular\n> object store.\n>\n> This would seem to defeat the goal of enabling specialized object handlers\n> to handle large or other \"unusual\" objects that git normally doesn't deal\n> well with.\n\nYeah, I agree that storing the objects in the regular object store\nshould not be done in all the cases.\nThere should be a way to control that.\n"},{"id":"327830","messageId":"C327746B-3E57-4AA9-B29E-AA3C33722D3B@gmail.com","threadId":"46506","inReplyTo":"20170803091926.1755-12-chriscool@tuxfamily.org","subject":"Re: [PATCH v5 11/40] odb-helper: add odb_helper_init() to send 'init' instruction","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-09-10T12:12:27Z","receivedAt":"2017-09-10T12:12:36Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 03 Aug 2017, at 10:18, Christian Couder wrote:\n> \n> Let's add an odb_helper_init() function to send an 'init'\n> instruction to the helpers. This 'init' instruction is\n> especially useful to get the capabilities that are supported\n> by the helpers.\n> \n> So while at it, let's also add a parse_capabilities()\n> function to parse them and a supported_capabilities\n> variable in struct odb_helper to store them.\n> \n> Signed-off-by: Christian Couder \n> ---\n> ...\n> \n> +static void parse_capabilities(char *cap_buf,\n> +\t\t\t unsigned int *supported_capabilities,\n> +\t\t\t const char *process_name)\n> +{\n> +\tstruct string_list cap_list = STRING_LIST_INIT_NODUP;\n> +\n> +\tstring_list_split_in_place(&cap_list, cap_buf, '=', 1);\n> +\n> +\tif (cap_list.nr == 2 && !strcmp(cap_list.items[0].string, \"capability\")) {\n> +\t\tconst char *cap_name = cap_list.items[1].string;\n> +\n> +\t\tif (!strcmp(cap_name, \"get_git_obj\")) {\n> +\t\t\t*supported_capabilities |= ODB_HELPER_CAP_GET_GIT_OBJ;\n> +\t\t} else if (!strcmp(cap_name, \"get_raw_obj\")) {\n> +\t\t\t*supported_capabilities |= ODB_HELPER_CAP_GET_RAW_OBJ;\n> +\t\t} else if (!strcmp(cap_name, \"get_direct\")) {\n> +\t\t\t*supported_capabilities |= ODB_HELPER_CAP_GET_DIRECT;\n> +\t\t} else if (!strcmp(cap_name, \"put_git_obj\")) {\n> +\t\t\t*supported_capabilities |= ODB_HELPER_CAP_PUT_GIT_OBJ;\n> +\t\t} else if (!strcmp(cap_name, \"put_raw_obj\")) {\n> +\t\t\t*supported_capabilities |= ODB_HELPER_CAP_PUT_RAW_OBJ;\n> +\t\t} else if (!strcmp(cap_name, \"put_direct\")) {\n> +\t\t\t*supported_capabilities |= ODB_HELPER_CAP_PUT_DIRECT;\n> +\t\t} else if (!strcmp(cap_name, \"have\")) {\n> +\t\t\t*supported_capabilities |= ODB_HELPER_CAP_HAVE;\n> +\t\t} else {\n> +\t\t\twarning(\"external process '%s' requested unsupported read-object capability '%s'\",\n> +\t\t\t\tprocess_name, cap_name);\n> +\t\t}\n\nIn 1514c8ed (\"convert: refactor capabilities negotiation\", 2017-06-30) I introduced\na simpler version of the capabilities negotiation. Maybe useful for you here, too? \n\n- Lars"},{"id":"327831","messageId":"50C9AF15-26DF-48F5-926E-B5D57F492DD1@gmail.com","threadId":"46506","inReplyTo":"20170803091926.1755-13-chriscool@tuxfamily.org","subject":"Re: [PATCH v5 12/40] t0400: add 'put_raw_obj' instruction to odb-helper script","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-09-10T12:12:56Z","receivedAt":"2017-09-10T12:13:02Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 03 Aug 2017, at 10:18, Christian Couder wrote:\n> \n> To properly test passing objects from Git to an external odb\n> we need an odb-helper script that supports a 'put'\n> capability/instruction.\n> \n> For now we will support only sending raw blobs, so the\n> supported capability/instruction will be 'put_raw_obj'.\n\nWhat other kind of blobs do you imagine could we send?\n\n\n> While at it let's add a test to check that our odb-helper\n> script works well.\n> \n> Signed-off-by: Christian Couder \n> ---\n> t/t0400-external-odb.sh | 24 ++++++++++++++++++++++++\n> 1 file changed, 24 insertions(+)\n> \n> diff --git a/t/t0400-external-odb.sh b/t/t0400-external-odb.sh\n> index ed89f3ab40..3fa0449883 100755\n> --- a/t/t0400-external-odb.sh\n> +++ b/t/t0400-external-odb.sh\n> @@ -7,10 +7,15 @@ test_description='basic tests for external object databases'\n> ALT_SOURCE=\"$PWD/alt-repo/.git\"\n> export ALT_SOURCE\n> write_script odb-helper <<\\EOF\n> +die() {\n> +\tprintf >&2 \"%s\\n\" \"$@\"\n> +\texit 1\n> +}\n> GIT_DIR=$ALT_SOURCE; export GIT_DIR\n> case \"$1\" in\n> init)\n> \techo \"capability=get_git_obj\"\n> +\techo \"capability=put_raw_obj\"\n> \techo \"capability=have\"\n> \t;;\n> have)\n> @@ -20,6 +25,16 @@ have)\n> get_git_obj)\n> \tcat \"$GIT_DIR\"/objects/$(echo $2 | sed 's#..#&/#')\n> \t;;\n> +put_raw_obj)\n> +\tsha1=\"$2\"\n> +\tsize=\"$3\"\n> +\tkind=\"$4\"\n> +\twriten=$(git hash-object -w -t \"$kind\" --stdin)\n> +\ttest \"$writen\" = \"$sha1\" || die \"bad sha1 passed '$sha1' vs writen '$writen'\"\n\nTypo? Should it be \"written\"?\n\n- Lars"},{"id":"327833","messageId":"BE9DF8E9-3771-42DD-B1F6-EC77783380F7@gmail.com","threadId":"46506","inReplyTo":"20170803091926.1755-1-chriscool@tuxfamily.org","subject":"Re: [PATCH v5 00/40] Add initial experimental external ODB support","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-09-10T12:30:13Z","receivedAt":"2017-09-10T12:30:21Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 03 Aug 2017, at 10:18, Christian Couder wrote:\n> \n> ...\n> \n> * The \"helpers\" (registered commands)\n> \n> Each helper manages access to one external ODB.\n> \n> There are 2 different modes for helper:\n> \n> - Helpers configured using \"odb..scriptCommand\" are\n> launched each time Git wants to communicate with the \n> external ODB. This is called \"script mode\".\n> \n> - Helpers configured using \"odb..subprocessCommand\" are\n> launched launched once as a sub-process (using sub-process.h), and\n> Git communicates with them using packet lines. This is called\n> \"process mode\".\n\nI am curious, why would we support two modes? Wouldn't that increase\nthe maintenance cost? Wouldn't the subprocess command be superior?\nI imagine the script mode eases testing, right?!\n\n\n> ...\n> \n> These odb refs point to a blob that is stored in the Git\n> repository and contain information about the blob stored in the\n> external odb. This information can be specific to the external odb.\n> The repos can then share this information using commands like:\n> \n> `git fetch origin \"refs/odbs//*:refs/odbs//*\"`\n> \n> At the end of the current patch series, \"git clone\" is teached a\n> \"--initial-refspec\" option, that asks it to first fetch some specified\n> refs. This is used in the tests to fetch the odb refs first.\n> \n> This way only one \"git clone\" command can setup a repo using the\n> external ODB mechanism as long as the right helper is installed on the\n> machine and as long as the following options are used:\n> \n> - \"--initial-refspec \" to fetch the odb refspec\n> - \"-c odb..command=\" to configure the helper\n\nThe \"odb\" config could, of course, go into the global git config. \nThe odbrefspec is optional, right?\n\nI have the impression there are a number of topics on the list\nthat tackle the \"many/big objects in a Git repo\" problem. Is\nthere a write up about the status of them, how they relate\nto each other, and what the current problems are? \nI found the following but it looks abandoned:\nhttps://github.com/jrn/git-large-repositories\n\n- Lars"},{"id":"328025","messageId":"CAP8UFD3ZV4Ezucn+Tv-roY6vzDyk2j4ypRsNR1YbOqoQK_qr8A@mail.gmail.com","threadId":"46506","inReplyTo":"BE9DF8E9-3771-42DD-B1F6-EC77783380F7@gmail.com","subject":"Re: [PATCH v5 00/40] Add initial experimental external ODB support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-14T07:02:54Z","receivedAt":"2017-09-14T07:03:01Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Sun, Sep 10, 2017 at 2:30 PM, Lars Schneider\n wrote:\n>\n>> On 03 Aug 2017, at 10:18, Christian Couder wrote:\n>>\n>> ...\n>>\n>> * The \"helpers\" (registered commands)\n>>\n>> Each helper manages access to one external ODB.\n>>\n>> There are 2 different modes for helper:\n>>\n>> - Helpers configured using \"odb..scriptCommand\" are\n>> launched each time Git wants to communicate with the \n>> external ODB. This is called \"script mode\".\n>>\n>> - Helpers configured using \"odb..subprocessCommand\" are\n>> launched launched once as a sub-process (using sub-process.h), and\n>> Git communicates with them using packet lines. This is called\n>> \"process mode\".\n>\n> I am curious, why would we support two modes? Wouldn't that increase\n> the maintenance cost? Wouldn't the subprocess command be superior?\n> I imagine the script mode eases testing, right?!\n\nThe script mode makes it much easier to write some helpers. For\nexample, as shown in t0430 at the end of the patch series, a helper\nfor a restartable bundle based clone could be something like\nbasically:\n\ncase \"$1\" in\ninit)\n ref_hash=$(git rev-parse refs/odbs/magic/bundle) ||\n die \"couldn't find refs/odbs/magic/bundle\"\n GIT_NO_EXTERNAL_ODB=1 git cat-file blob \"$ref_hash\" >bundle_info ||\n die \"couldn't get blob $ref_hash\"\n bundle_url=$(sed -e 's/bundle url: //' bundle_info)\n curl \"$bundle_url\" -o bundle_file ||\n die \"curl '$bundle_url' failed\"\n GIT_NO_EXTERNAL_ODB=1 git bundle unbundle bundle_file >unbundling_info ||\n die \"unbundling 'bundle_file' failed\"\n ;;\n\n>> These odb refs point to a blob that is stored in the Git\n>> repository and contain information about the blob stored in the\n>> external odb. This information can be specific to the external odb.\n>> The repos can then share this information using commands like:\n>>\n>> `git fetch origin \"refs/odbs//*:refs/odbs//*\"`\n>>\n>> At the end of the current patch series, \"git clone\" is teached a\n>> \"--initial-refspec\" option, that asks it to first fetch some specified\n>> refs. This is used in the tests to fetch the odb refs first.\n>>\n>> This way only one \"git clone\" command can setup a repo using the\n>> external ODB mechanism as long as the right helper is installed on the\n>> machine and as long as the following options are used:\n>>\n>> - \"--initial-refspec \" to fetch the odb refspec\n>> - \"-c odb..command=\" to configure the helper\n>\n> The \"odb\" config could, of course, go into the global git config.\n\nSure.\n\n> The odbrefspec is optional, right?\n\nUsing \"--initial-refspec \" is optional. There will be more\ninformation in the documentation about this option in the next version\nof the series.\n\n> I have the impression there are a number of topics on the list\n> that tackle the \"many/big objects in a Git repo\" problem. Is\n> there a write up about the status of them, how they relate\n> to each other, and what the current problems are?\n> I found the following but it looks abandoned:\n> https://github.com/jrn/git-large-repositories\n\nYeah, it could be interesting to discuss all these topics together. On\nthe other hand people working on existing patch series, like me, have\nto work on them and post new versions, as just discussing the topics\nis not enough to move things forward.\nAnyway Junio and Jonathan Tan also asked me questions about how my\nwork relates to Jonathan's, so I will reply to them hopefully soon...\n"},{"id":"328026","messageId":"CAP8UFD04cNASyWGURb7wYMX2eyOqKAumVuFvHCQ-nWYoPzzJZw@mail.gmail.com","threadId":"46506","inReplyTo":"50C9AF15-26DF-48F5-926E-B5D57F492DD1@gmail.com","subject":"Re: [PATCH v5 12/40] t0400: add 'put_raw_obj' instruction to odb-helper script","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-14T07:09:45Z","receivedAt":"2017-09-14T07:09:51Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Sun, Sep 10, 2017 at 2:12 PM, Lars Schneider\n wrote:\n>\n>> On 03 Aug 2017, at 10:18, Christian Couder wrote:\n>>\n>> To properly test passing objects from Git to an external odb\n>> we need an odb-helper script that supports a 'put'\n>> capability/instruction.\n>>\n>> For now we will support only sending raw blobs, so the\n>> supported capability/instruction will be 'put_raw_obj'.\n>\n> What other kind of blobs do you imagine could we send?\n\nAs for the get instructions there could be 'put_git_obj' to send blobs\nin the Git format and 'put_direct' to have the helper read directly\nfrom the Git object store.\n\n>> +put_raw_obj)\n>> + sha1=\"$2\"\n>> + size=\"$3\"\n>> + kind=\"$4\"\n>> + writen=$(git hash-object -w -t \"$kind\" --stdin)\n>> + test \"$writen\" = \"$sha1\" || die \"bad sha1 passed '$sha1' vs writen '$writen'\"\n>\n> Typo? Should it be \"written\"?\n\nYeah, thanks. It's fixed in the current version.\n"},{"id":"328027","messageId":"CAP8UFD0roTFSt4Q9-_2fxDb9r0a5LxQ5rKbbbz2ZfPcEZqdTuQ@mail.gmail.com","threadId":"46506","inReplyTo":"C327746B-3E57-4AA9-B29E-AA3C33722D3B@gmail.com","subject":"Re: [PATCH v5 11/40] odb-helper: add odb_helper_init() to send 'init' instruction","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-14T07:18:33Z","receivedAt":"2017-09-14T07:18:39Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Sun, Sep 10, 2017 at 2:12 PM, Lars Schneider\n wrote:\n>\n>> On 03 Aug 2017, at 10:18, Christian Couder wrote:\n>>\n>> +static void parse_capabilities(char *cap_buf,\n>> + unsigned int *supported_capabilities,\n>> + const char *process_name)\n>> +{\n>> + struct string_list cap_list = STRING_LIST_INIT_NODUP;\n>> +\n>> + string_list_split_in_place(&cap_list, cap_buf, '=', 1);\n>> +\n>> + if (cap_list.nr == 2 && !strcmp(cap_list.items[0].string, \"capability\")) {\n>> + const char *cap_name = cap_list.items[1].string;\n>> +\n>> + if (!strcmp(cap_name, \"get_git_obj\")) {\n>> + *supported_capabilities |= ODB_HELPER_CAP_GET_GIT_OBJ;\n>> + } else if (!strcmp(cap_name, \"get_raw_obj\")) {\n>> + *supported_capabilities |= ODB_HELPER_CAP_GET_RAW_OBJ;\n>> + } else if (!strcmp(cap_name, \"get_direct\")) {\n>> + *supported_capabilities |= ODB_HELPER_CAP_GET_DIRECT;\n>> + } else if (!strcmp(cap_name, \"put_git_obj\")) {\n>> + *supported_capabilities |= ODB_HELPER_CAP_PUT_GIT_OBJ;\n>> + } else if (!strcmp(cap_name, \"put_raw_obj\")) {\n>> + *supported_capabilities |= ODB_HELPER_CAP_PUT_RAW_OBJ;\n>> + } else if (!strcmp(cap_name, \"put_direct\")) {\n>> + *supported_capabilities |= ODB_HELPER_CAP_PUT_DIRECT;\n>> + } else if (!strcmp(cap_name, \"have\")) {\n>> + *supported_capabilities |= ODB_HELPER_CAP_HAVE;\n>> + } else {\n>> + warning(\"external process '%s' requested unsupported read-object capability '%s'\",\n>> + process_name, cap_name);\n>> + }\n>\n> In 1514c8ed (\"convert: refactor capabilities negotiation\", 2017-06-30) I introduced\n> a simpler version of the capabilities negotiation. Maybe useful for you here, too?\n\nYeah, actually there is also fa64a2fdbe (sub-process: refactor\nhandshake to common function, 2017-07-26) that Jonathan Tan wrote on\ntop of your changes and that adds subprocess_handshake(). So the\ncurrent code is using it like that:\n\nstatic int start_object_process_fn(struct subprocess_entry *subprocess)\n{\n static int versions[] = {1, 0};\n static struct subprocess_capability capabilities[] = {\n { \"get_git_obj\", ODB_HELPER_CAP_GET_GIT_OBJ },\n { \"get_raw_obj\", ODB_HELPER_CAP_GET_RAW_OBJ },\n { \"get_direct\", ODB_HELPER_CAP_GET_DIRECT },\n { \"put_git_obj\", ODB_HELPER_CAP_PUT_GIT_OBJ },\n { \"put_raw_obj\", ODB_HELPER_CAP_PUT_RAW_OBJ },\n { \"put_direct\", ODB_HELPER_CAP_PUT_DIRECT },\n { \"have\", ODB_HELPER_CAP_HAVE },\n { NULL, 0 }\n };\n struct object_process *entry = (struct object_process *)subprocess;\n return subprocess_handshake(subprocess, \"git-read-object\", versions, NULL,\n capabilities,\n &entry->supported_capabilities);\n}\n"},{"id":"328029","messageId":"CAP8UFD2-V_2UOfp3zrTf0bSX7Y7g66FMAfhWW3==_GNd7oG_zw@mail.gmail.com","threadId":"46506","inReplyTo":"xmqq3798xvns.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v5 19/40] lib-httpd: add upload.sh","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-14T07:43:41Z","receivedAt":"2017-09-14T07:43:47Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, Aug 3, 2017 at 10:07 PM, Junio C Hamano wrote:\n> Christian Couder writes:\n>\n>> +OLDIFS=\"$IFS\"\n>> +IFS='&'\n>> +set -- $QUERY_STRING\n>> +IFS=\"$OLDIFS\"\n>> +\n>> +while test $# -gt 0\n>> +do\n>> + key=${1%=*}\n>> + val=${1#*=}\n>\n> When you see that ${V%X*} and ${V#*X} appear in a pair for the same\n> variable V and same delimiter X, it almost always indicates a bug\n> waiting to happen.\n>\n> What's the definition of \"key\" here? A member of known set of short\n> tokens, all of which consists only of alphanumeric, or something?\n\nYeah, the key can be only \"sha1\", \"type\", \"size\" or \"delete\" as can be\nseen later in the code.\n\n> Even if you do not currently plan to deal with a value with '=' in\n> it, it may be prudent to double '%' above (and do not double '#').\n\nYeah I agree. Thanks for spotting this!\n\n> Style: indent your shell script with tabs.\n\nSure.\n\n>> + case \"$key\" in\n>> + \"sha1\") sha1=\"$val\" ;;\n>> + \"type\") type=\"$val\" ;;\n>> + \"size\") size=\"$val\" ;;\n>> + \"delete\") delete=1 ;;\n>> + *) echo >&2 \"unknown key '$key'\" ;;\n>> + esac\n>\n> Indent your shell script with tabs; case/esac and the labels used\n> for each case arms all align at the same column.\n\nYeah, it will be properly indented in the version I will send soon.\n"},{"id":"328037","messageId":"CAP8UFD36yZEU4OM_+u6HkC=Ep1JifXm-ir0+CeJ3=dR44QceoQ@mail.gmail.com","threadId":"46506","inReplyTo":"xmqq4ltowcrz.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v5 25/40] external-odb: add 'get_direct' support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-14T08:39:35Z","receivedAt":"2017-09-14T08:39:42Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, Aug 3, 2017 at 11:40 PM, Junio C Hamano wrote:\n> Christian Couder writes:\n>\n>> This implements the 'get_direct' capability/instruction that makes\n>> it possible for external odb helper scripts to pass blobs to Git\n>> by directly writing them as loose objects files.\n>\n> I am not sure if the assumption is made clear in this series, but I\n> am (perhaps incorrectly) guessing that it is assumed that the\n> intended use of this feature is to offload access to large blobs\n> by not including them in the initial clone.\n\nYeah, it could be used for that, but that's not the only interesting use case.\n\nIt could also be used for example if the working tree contains a huge\nnumber of blobs and it is better to download only the blobs that are\nneeded when they are needed. In fact the code for 'get_direct' was\ntaken from Ben Peart's \"read-object\" patch series (actually from an\nearlier version of this patch series):\n\nhttps://public-inbox.org/git/20170714132651.170708-1-benpeart@microsoft.com/\n\n> So from that point of\n> view, I think it makes tons of sense to let the external helper to\n> directly populate the database bypassing Git (i.e. instead of\n> feeding data stream and have Git store it) like this \"direct\" method\n> does.\n>\n> How does this compare with (and how well does this work with) what\n> Jonathan Tan is doing recently?\n\nFrom the following email:\n\nhttps://public-inbox.org/git/20170804145113.5ceafafa@twelve2.svl.corp.google.com/\n\nit looks like his work is fundamentally about changing the rules of\nconnectivity checks. Objects are split between \"homegrown\" objects and\n\"imported\" objects which are in separate pack files. Then references\nto imported objects are not checked during connectivity check.\n\nI think changing connectivity rules is not necessary to make something\nlike external odb work. For example when fetching a pack that refers\nto objects that are in an external odb, if access this external odb\nhas been configured, then the connectivity check will pass as the\nmissing objects in the pack will be seen as already part of the repo.\n\nYeah, if some commands like fsck are used, then possibly all the\nobjects will have to be requested from the external odb, as it may not\nbe possible to fully check all the objects, especially the blobs,\nwithout accessing all their data. But I think this is a problem that\ncould be dealt with in different ways. For example we could develop\nspecific options in fsck so that it doesn't check the sha1 of objects\nthat are marked with some specific attributes, or that are stored in\nexternal odbs, or that are bigger than some size.\n"},{"id":"328038","messageId":"CAP8UFD2+-0DYRO+PN+Ertn3TvoXEpghWfv=-He9+0kVSA=TNxA@mail.gmail.com","threadId":"46506","inReplyTo":"xmqqbmnwxwfw.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v5 13/40] external odb: add 'put_raw_obj' support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-14T09:17:12Z","receivedAt":"2017-09-14T09:17:18Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, Aug 3, 2017 at 9:50 PM, Junio C Hamano wrote:\n> Christian Couder writes:\n>\n>> Add support for a 'put_raw_obj' capability/instruction to send new\n>> objects to an external odb. Objects will be sent as they are (in\n>> their 'raw' format). They will not be converted to Git objects.\n>>\n>> For now any new Git object (blob, tree, commit, ...) would be sent\n>> if 'put_raw_obj' is supported by an odb helper. This is not a great\n>> default, but let's leave it to following commits to tweak that.\n>>\n>> Signed-off-by: Christian Couder \n>> ---\n>\n> I thought in an earlier step that I saw this thing initialized in\n> the codepath that adds alternate object stores, which are read-only\n> places we \"borrow\" from. Being able to write into it is good, but\n> conceptually it no longer feels correct to initialize it from the\n> alternate object database initialization codepath.\n>\n> Another way to say it is that an object store, whether it is local\n> or external, is not \"alt\" if it will result in storing new objects\n> we locally create. It's just an extension of our local object\n> store.\n\nI guess you are talking about the following code in \"[PATCH v5 10/40]\nAdd initial external odb support\":\n\n+void prepare_external_alt_odb(void)\n+{\n+ static int linked_external;\n+ const char *path;\n+\n+ if (linked_external)\n+ return;\n+\n+ path = external_odb_root();\n+ if (!access(path, F_OK)) {\n+ link_alt_odb_entry(path, NULL, 0, \"\");\n+ linked_external = 1;\n+ }\n+}\n+\n void prepare_alt_odb(void)\n {\n const char *alt;\n@@ -650,6 +666,7 @@ void prepare_alt_odb(void)\n link_alt_odb_entries(alt, strlen(alt), PATH_SEP, NULL, 0);\n\n read_info_alternates(get_object_directory(), 0);\n+ prepare_external_alt_odb();\n }\n\nWould it be ok if I do the following:\n\n- rename prepare_external_alt_odb() to just prepare_external_odb(), as\nthis would avoid confusion between alt_odbs and external odbs\n- remove the call to prepare_external_odb() in prepare_alt_odb()\n- add a prepare_alt_and_external_odb() that just calls\nprepare_alt_odb() and then prepare_external_odb()\n- replace all the calls to prepare_alt_odb() with calls to\nprepare_alt_and_external_odb()\n\n?\n"},{"id":"328039","messageId":"CAP8UFD3=nuTRF24CLSoK4HSGm3nxGh8SbZVpMCg7cNcHj2zkBA@mail.gmail.com","threadId":"46506","inReplyTo":"xmqq7eykxwbt.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v5 14/40] external-odb: accept only blobs for now","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-14T09:59:17Z","receivedAt":"2017-09-14T09:59:23Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, Aug 3, 2017 at 9:52 PM, Junio C Hamano wrote:\n> Christian Couder writes:\n>\n>> The mechanism to decide which blobs should be sent to which\n>> external object database will be very simple for now.\n>> If the external odb helper support any \"put_*\" instruction\n>> all the new blobs will be sent to it.\n>>\n>> Signed-off-by: Christian Couder \n>> ---\n>> external-odb.c | 4 ++++\n>> 1 file changed, 4 insertions(+)\n>>\n>> diff --git a/external-odb.c b/external-odb.c\n>> index 82fac702e8..a4f8c72e1c 100644\n>> --- a/external-odb.c\n>> +++ b/external-odb.c\n>> @@ -124,6 +124,10 @@ int external_odb_put_object(const void *buf, size_t len,\n>> {\n>> struct odb_helper *o;\n>>\n>> + /* For now accept only blobs */\n>> + if (strcmp(type, \"blob\"))\n>> + return 1;\n>> +\n>\n> I somehow doubt that a policy decision like this should be made at\n> this layer. Shouldn't it be encoded in the capability the other\n> side supports, or determined at runtime per each individual object\n> when a \"put\" is attempted (i.e. allow the other side to say \"You\n> tell me that you want me to store an object of type X and size Y;\n> I cannot do that, sorry\").\n\nI agree that it would be conceptually better to be able to support\nother kind of objects in external odb, but realistically most use\ncases for 'get_*' and 'put_*' instructions are for storing/retrieving\nblobs as other kind of objects are in specific formats that are well\nsupported by the current object store.\n\nI also agree that it would be a nice feature if external odb could\ndecide by themselves which objects they accept and I really want to\nleave the door open to a future improvement implementing that using\nthe capability mechanism or perhaps another mechanism. For now though\nin \"[PATCH v5 34/40] external-odb: use 'odb=magic' attribute to mark\nodb blobs\" the attribute system is used to decide which blobs are put\ninto which external odb as it is probably good enough for many use\ncases. It is also simple to implement in Git and makes helpers simpler\nto implement.\n"},{"id":"328041","messageId":"CAP8UFD2F65Y+eTDO1=WFsrAMmO=8Wykes9bmGBUS5smULvAOtQ@mail.gmail.com","threadId":"46506","inReplyTo":"xmqqfud8xx6m.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v5 10/40] Add initial external odb support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-14T10:14:27Z","receivedAt":"2017-09-14T10:14:33Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, Aug 3, 2017 at 9:34 PM, Junio C Hamano wrote:\n> Christian Couder writes:\n>\n>> diff --git a/external-odb.h b/external-odb.h\n>> new file mode 100644\n>> index 0000000000..9989490c9e\n>> --- /dev/null\n>> +++ b/external-odb.h\n>> @@ -0,0 +1,8 @@\n>> +#ifndef EXTERNAL_ODB_H\n>> +#define EXTERNAL_ODB_H\n>> +\n>> +const char *external_odb_root(void);\n>> +int external_odb_has_object(const unsigned char *sha1);\n>> +int external_odb_get_object(const unsigned char *sha1);\n>\n> Even though ancient codebase of ours deliberately omitted them, I\n> think our recent trend is to explicitly spell \"extern \" in headers.\n>\n>> diff --git a/odb-helper.h b/odb-helper.h\n>> new file mode 100644\n>> index 0000000000..5800661704\n>> --- /dev/null\n>> +++ b/odb-helper.h\n>\n> Likewise.\n\nOk, I am adding \"extern \" to the headers.\n"},{"id":"328071","messageId":"20170914111945.62741763@twelve2.svl.corp.google.com","threadId":"46506","inReplyTo":"CAP8UFD36yZEU4OM_+u6HkC=Ep1JifXm-ir0+CeJ3=dR44QceoQ@mail.gmail.com","subject":"Re: [PATCH v5 25/40] external-odb: add 'get_direct' support","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-09-14T18:19:45Z","receivedAt":"2017-09-14T18:19:57Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Thu, 14 Sep 2017 10:39:35 +0200\nChristian Couder wrote:\n\n> From the following email:\n> \n> https://public-inbox.org/git/20170804145113.5ceafafa@twelve2.svl.corp.google.com/\n> \n> it looks like his work is fundamentally about changing the rules of\n> connectivity checks. Objects are split between \"homegrown\" objects and\n> \"imported\" objects which are in separate pack files. Then references\n> to imported objects are not checked during connectivity check.\n> \n> I think changing connectivity rules is not necessary to make something\n> like external odb work. For example when fetching a pack that refers\n> to objects that are in an external odb, if access this external odb\n> has been configured, then the connectivity check will pass as the\n> missing objects in the pack will be seen as already part of the repo.\n\nThere are still some nuances. For example, if an external ODB provides\nboth a tree and a blob that the tree references, do we fetch the tree in\norder to call \"have\" on all its blobs, or do we trust the ODB that if it\nhas the tree, it has all the other objects? In my design, I do the\nlatter, but in the general case where we have multiple ODBs, we might\nhave to do the former. (And if we do the former, it seems to me that the\nconnectivity check must be performed \"online\" - that is, with the ODBs\nbeing able to provide \"get\".)\n\n(Also, my work extends all the way to fetch/clone [1], but admittedly I\nhave been taking it a step at a time and recently have only been\ndiscussing how the local repo should handle the missing object\nsituation.)\n\n[1] https://public-inbox.org/git/cover.1499800530.git.jonathantanmy@google.com/\n\n> Yeah, if some commands like fsck are used, then possibly all the\n> objects will have to be requested from the external odb, as it may not\n> be possible to fully check all the objects, especially the blobs,\n> without accessing all their data. But I think this is a problem that\n> could be dealt with in different ways. For example we could develop\n> specific options in fsck so that it doesn't check the sha1 of objects\n> that are marked with some specific attributes, or that are stored in\n> external odbs, or that are bigger than some size.\n\nThe hard part is in dealing with missing commits and trees, I think, not\nblobs.\n"},{"id":"328117","messageId":"CAP8UFD0-myJuhBQhZoBiQ6cE4FEkzzuro-sZeDfCN-HK_LxriQ@mail.gmail.com","threadId":"46506","inReplyTo":"20170914111945.62741763@twelve2.svl.corp.google.com","subject":"Re: [PATCH v5 25/40] external-odb: add 'get_direct' support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-09-15T11:24:50Z","receivedAt":"2017-09-15T11:24:56Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, Sep 14, 2017 at 8:19 PM, Jonathan Tan wrote:\n> On Thu, 14 Sep 2017 10:39:35 +0200\n> Christian Couder wrote:\n>\n>> From the following email:\n>>\n>> https://public-inbox.org/git/20170804145113.5ceafafa@twelve2.svl.corp.google.com/\n>>\n>> it looks like his work is fundamentally about changing the rules of\n>> connectivity checks. Objects are split between \"homegrown\" objects and\n>> \"imported\" objects which are in separate pack files. Then references\n>> to imported objects are not checked during connectivity check.\n>>\n>> I think changing connectivity rules is not necessary to make something\n>> like external odb work. For example when fetching a pack that refers\n>> to objects that are in an external odb, if access this external odb\n>> has been configured, then the connectivity check will pass as the\n>> missing objects in the pack will be seen as already part of the repo.\n>\n> There are still some nuances. For example, if an external ODB provides\n> both a tree and a blob that the tree references, do we fetch the tree in\n> order to call \"have\" on all its blobs, or do we trust the ODB that if it\n> has the tree, it has all the other objects? In my design, I do the\n> latter, but in the general case where we have multiple ODBs, we might\n> have to do the former. (And if we do the former, it seems to me that the\n> connectivity check must be performed \"online\" - that is, with the ODBs\n> being able to provide \"get\".)\n\nYeah, I agree that the problem is more complex if there can be trees\nor all kind of objects in the external odb.\nBut as I explain in the following email to Junio, I don't think\nstoring other kind of objects is one of the most interesting use case:\n\nhttps://public-inbox.org/git/CAP8UFD3=nuTRF24CLSoK4HSGm3nxGh8SbZVpMCg7cNcHj2zkBA@mail.gmail.com/\n\n> (Also, my work extends all the way to fetch/clone [1], but admittedly I\n> have been taking it a step at a time and recently have only been\n> discussing how the local repo should handle the missing object\n> situation.)\n>\n> [1] https://public-inbox.org/git/cover.1499800530.git.jonathantanmy@google.com/\n\nYeah, I think your work is interesting and could perhaps be useful for\nexternal odbs as there could be situations that would be handled\nbetter using your work or something similar.\n"},{"id":"328169","messageId":"20170915135452.4289d591@twelve2.svl.corp.google.com","threadId":"46506","inReplyTo":"CAP8UFD0-myJuhBQhZoBiQ6cE4FEkzzuro-sZeDfCN-HK_LxriQ@mail.gmail.com","subject":"Re: [PATCH v5 25/40] external-odb: add 'get_direct' support","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-09-15T20:54:52Z","receivedAt":"2017-09-15T20:55:00Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Fri, 15 Sep 2017 13:24:50 +0200\nChristian Couder wrote:\n\n> > There are still some nuances. For example, if an external ODB provides\n> > both a tree and a blob that the tree references, do we fetch the tree in\n> > order to call \"have\" on all its blobs, or do we trust the ODB that if it\n> > has the tree, it has all the other objects? In my design, I do the\n> > latter, but in the general case where we have multiple ODBs, we might\n> > have to do the former. (And if we do the former, it seems to me that the\n> > connectivity check must be performed \"online\" - that is, with the ODBs\n> > being able to provide \"get\".)\n> \n> Yeah, I agree that the problem is more complex if there can be trees\n> or all kind of objects in the external odb.\n> But as I explain in the following email to Junio, I don't think\n> storing other kind of objects is one of the most interesting use case:\n> \n> https://public-inbox.org/git/CAP8UFD3=nuTRF24CLSoK4HSGm3nxGh8SbZVpMCg7cNcHj2zkBA@mail.gmail.com/\n\nIf we start with only blobs in the ODB, that makes sense (the ODB will\nneed to supply a fast enough \"list\" or \"have\", but, as you wrote before,\na mechanism like fetching an additional ref that contains all the\nnecessary information whenever we fetch refs would be enough). I agree\nthat it would work with existing use cases (including yours).\n\n> > (Also, my work extends all the way to fetch/clone [1], but admittedly I\n> > have been taking it a step at a time and recently have only been\n> > discussing how the local repo should handle the missing object\n> > situation.)\n> >\n> > [1] https://public-inbox.org/git/cover.1499800530.git.jonathantanmy@google.com/\n> \n> Yeah, I think your work is interesting and could perhaps be useful for\n> external odbs as there could be situations that would be handled\n> better using your work or something similar.\n\nThanks.\n"}]}