{"thread":{"id":"45320","subject":"[PATCH 00/10] RFC Partial Clone and Fetch","startedAt":"2017-03-08T18:51:05Z","lastAt":"2017-05-08T00:15:42Z","messageCount":24,"participants":["git@jeffhostetler.com","Jonathan Tan","Jeff Hostetler","ankostis","Johannes Schindelin","Jonathan Nieder","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":10},"messages":[{"id":"313534","messageId":"1488999039-37631-1-git-send-email-git@jeffhostetler.com","threadId":"45320","inReplyTo":null,"subject":"[PATCH 00/10] RFC Partial Clone and Fetch","fromName":"","fromEmail":"git@jeffhostetler.com","sentAt":"2017-03-08T18:50:29Z","receivedAt":"2017-03-08T18:51:05Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\n\n[RFC] Partial Clone and Fetch\n=============================\n\n\nThis is a WIP RFC for a partial clone and fetch feature wherein the client\ncan request that the server omit various blobs from the packfile during\nclone and fetch. Clients can later request omitted blobs (either from a\nmodified upload-pack-like request to the server or via a completely\nindependent mechanism).\n\nThe purpose here is to reduce the size of packfile downloads and help\ngit scale to extremely large repos.\n\nI use the term \"partial\" here to refer to a portion of one or more commits\nand to avoid use of loaded terms like \"sparse\", \"lazy\", \"narrow\", and \"skeleton\".\n\nThe concept of a partial clone/fetch is independent of and can complement\nthe existing shallow-clone, refspec, and limited-ref filtering mechanisms\nsince these all filter at the DAG level whereas the work described here\nworks *within* the set of commits already chosen for download.\n\n\nA. Requesting a Partial Clone/Fetch\n===================================\n\nClone, fetch, and fetch-pack will accept one or more new \"partial\"\ncommand line arguments as described below. The fetch-pack/upload-pack\nprotocol will be extended to include these new arguments. Upload-pack\nand pack-objects will be updated accordingly. Pack-objects will filter\nout the unwanted blobs as it is building the packfile. Rev-list and\nindex-pack will be updated to not complain when missing blobs are\ndetected in the received packfile.\n\n[1] \"--partial-by-size=[kmg]\"\nWhere is a non-negative integer with an optional unit.\n\nRequest that only blobs smaller than this be included in the packfile.\nThe client might use this to implement an alternate LFS or ODB mechanism\nfor large blobs, such as suggested in:\n https://public-inbox.org/git/20161130210420.15982-1-chriscool@tuxfamily.org/\n\nA special case of size zero would omit all blobs and is similar to the\ncommits-and-trees-only feature described in:\n https://public-inbox.org/git/20170113155253.1644-1-benpeart@microsoft.com/\n\n[2] \"--partial-special\"\nRequest that special files, such as \".gitignore\" and .gitattributes\",\nbe included.\n\n[3] *TODO* \"--partial-by-profile=\"\nWhere is verson-controlled file in the repository\n(either present in the requested commit or the default HEAD on the server).\n\n [I envision a \".gitsparse/\" hierarchy where teams can store\n common sparse-checkout profiles. And then they can reference\n them from their private \".git/info/sparse-checkout\" files.]\n\nPack-objects will use this file and the sparse-checkout rules to only\ninclude blobs in the packfile that would be needed to do the corresponding\nsparse-checkout (and let the client avoid having to demand-load their\nentire enlistment).\n\n\nWhen multiple \"partial\" options are given, they are treated as a simple OR\ngiving the union of the blobs selected.\n\nThe patch series describes the changes to the fetch-pack/upload-pack\nprotocol:\n Documentation/technical/pack-protocol.txt\n Documentation/technical/protocol-capabilities.txt\n\n\nB. Issues Backfilling Omitted Blobs\n===================================\n\nIdeally, if the client only does \"--partial-by-profile\" fetches, it\nshould not need to fetch individual missing blobs, but we have to allow\nfor it to handle the other commands and other unexpected issues.\n\nThere are 3 orthogonal concepts here: when, how and where?\n\n\n[1] When:\n(1a) a pre-command or hook to identify needed blobs and pre-fetch them\nbefore allowing the actual command to start;\n(1b) a dry-run mode for the command to likewise pre-fetch them; or\n(1c) \"fault\" them in as necessary in read_object() while the command is\nrunning and without any pre-fetch (either synchronously or asynchronously\nand with/without a helper process).\n\nIdeas for (1c) are being addressed in the following threads:\n https://public-inbox.org/git/20170113155253.1644-1-benpeart@microsoft.com/\n https://public-inbox.org/git/20170117184258.sd7h2hkv27w52gzt@sigill.intra.peff.net/\n https://public-inbox.org/git/20161130210420.15982-1-chriscool@tuxfamily.org/\nso I won't consider them here.\n\nIdeas (1a) and (1b) have the advantage that they try to obtain all\nrequired blobs before allowing an operation to start, so there is\nless opportunity to leave the user in a weird state.\n\nThe best solution may be a combination of (1a) and (1b) and may depend\non the individual command. However, (1b) will further complicate the\nsource in the existing commands, so in some cases it may be simpler to\njust take the ideas and implement stand-alone pre-commands.\n\nFor now, I'm going to limit this RFC to (1a).\n\n\n[2] How:\n(2a) augment the existing git protocols to include blob requests;\n(2b) local external process (such as a database client or a local bulk\nfetch daemon);\n\nIdeas for (2b) are being addressed in the above threads, so I won't\nconsider them here.\n\nSo I'm going to limit this RFC to (2a).\n\n\n[3] Where:\n(3a) the same remote server used for the partial clone/fetch;\n(3b) anywhere else, such as a proxy server or Azure or S3 blob store.\n\nThere's no reason that the client should be limited to going back to\nthe same server, but I'm not going to consider it here, so I'm going\nto limit this RFC to (3a).\n\n\n\nC. New Blob-Fetch Protocol (2a)\n===============================\n\n*TODO* A new pair of commands, such as fetch-blob-pack and upload-blob-pack,\nwill be created to let the client request a batch of blobs and receive a\npackfile. A protocol similar to the fetch-pack/upload-pack will be spoken\nbetween them. (This avoids complicating the existing protocol and the work\nof enumerating the refs.) Upload-blob-pack will use pack-objects to build\nthe packfile.\n\nIt is also more efficient than requesting a single blob at a time using\nthe existing fetch-pack/upload-pack mechanism (with the various allow\nunreachable options).\n\n*TODO* The new request protocol will be defined in the patch series.\nIt will include: a list of the desired blob SHAs. Possibly also the commit\nSHA, branch name, and pathname of each blob (or whatever is necessary to let\nthe server address the reachability concerns). Possibly also the last\nknown SHA for each blob to allow for deltafication in the packfile.\n\n\nD. Pre-fetching Blobs (1a)\n==========================\n\nOn the client side, one or more special commands will be created to assemble\nthe list of blobs needed for an operation and passed to fetch-blob-pack.\n\n\nCheckout Example: After running a command like:\n 'clone --partial-by-size=1m --no-checkout'\n\nand before doing an actual checkout, we need a command to essentially do:\n (1) \"ls-tree -r \",\n (2) filter that by the sparse-checkout currently in effect,\n (3) filter that for missing blobs,\n (4) and pass the resulting list to fetch-blob-pack.\n\nAfterwards, checkout should complete without faulting.\n\nA new \"git ls-partial \" command has been created to do\nsteps 1 thru 3 and print the resulting list of SHAs on stdout.\n\n\nE. Unresolved Thoughts\n======================\n\n*TODO* The server should optionally return (in a side-band?) a list \nof the blobs that it omitted from the packfile (and possibly the sizes\nor sha1_object_info() data for them) during the fetch-pack/upload-pack\noperation. This would allow the client to distinguish from invalid\nSHAs and missing ones. Size information would allow the client to\nmaybe choose between various servers.\n\n*TODO* The partial clone arguments should be recorded in \".git/info/\"\nso that subsequent fetch commands can inherit them and rev-list/index-pack\nknow to not complain by default.\n\n*TODO* Update GC like rev-list to not complain when there are missing blobs.\n\n*TODO* Extend ls-partial to include the \"-m\" and 3 tree-ish arguments\nlike read-tree, so we can pre-fetch for merges that may require file\nmerges (that may or may not be within our sparse-checkout).\n\n*TODO* I also need to review the RFC that Mark Thomas submitted over\nthe weekend:\n https://public-inbox.org/git/20170304191901.9622-1-markbt%40efaref.net/t\n\n\n\n\n\nJeff Hostetler (10):\n pack-objects: eat CR in addition to LF after fgets.\n pack-objects: add --partial-by-size=n --partial-special\n pack-objects: test for --partial-by-size --partial-special\n upload-pack: add partial (sparse) fetch\n fetch-pack: add partial-by-size and partial-special\n rev-list: add --allow-partial option to relax connectivity checks\n index-pack: add --allow-partial option to relax blob existence checks\n fetch: add partial-by-size and partial-special arguments\n clone: add partial-by-size and partial-special arguments\n ls-partial: created command to list missing blobs\n\n Documentation/technical/pack-protocol.txt | 14 ++\n Documentation/technical/protocol-capabilities.txt | 7 +\n Makefile | 2 +\n builtin.h | 1 +\n builtin/clone.c | 26 ++\n builtin/fetch-pack.c | 9 +\n builtin/fetch.c | 26 +-\n builtin/index-pack.c | 20 +-\n builtin/ls-partial.c | 110 +++++++++\n builtin/pack-objects.c | 64 ++++-\n builtin/rev-list.c | 22 +-\n connected.c | 3 +\n connected.h | 3 +\n fetch-pack.c | 17 ++\n fetch-pack.h | 2 +\n git.c | 1 +\n partial-utils.c | 279 ++++++++++++++++++++++\n partial-utils.h | 93 ++++++++\n t/5316-pack-objects-partial.sh | 72 ++++++\n transport.c | 8 +\n transport.h | 8 +\n upload-pack.c | 32 ++-\n 22 files changed, 813 insertions(+), 6 deletions(-)\n create mode 100644 builtin/ls-partial.c\n create mode 100644 partial-utils.c\n create mode 100644 partial-utils.h\n create mode 100644 t/5316-pack-objects-partial.sh\n\n-- \n2.7.4\n\n"},{"id":"313544","messageId":"1488999039-37631-9-git-send-email-git@jeffhostetler.com","threadId":"45320","inReplyTo":"1488999039-37631-1-git-send-email-git@jeffhostetler.com","subject":"[PATCH 08/10] fetch: add partial-by-size and partial-special arguments","fromName":"","fromEmail":"git@jeffhostetler.com","sentAt":"2017-03-08T18:50:37Z","receivedAt":"2017-03-08T20:47:34Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fetch to accept --partial-by-size=n and --partial-special\narguments and pass them to fetch-patch to request that the\nserver certain blobs from the generated packfile.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fetch.c | 26 +++++++++++++++++++++++++-\n connected.c | 3 +++\n connected.h | 3 +++\n 3 files changed, 31 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex b5ad09d..3d47107 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -52,6 +52,8 @@ static const char *recurse_submodules_default;\n static int shown_url = 0;\n static int refmap_alloc, refmap_nr;\n static const char **refmap_array;\n+static const char *partial_by_size;\n+static int partial_special;\n \n static int option_parse_recurse_submodules(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n@@ -141,6 +143,11 @@ static struct option builtin_fetch_options[] = {\n \t\t\tTRANSPORT_FAMILY_IPV4),\n \tOPT_SET_INT('6', \"ipv6\", &family, N_(\"use IPv6 addresses only\"),\n \t\t\tTRANSPORT_FAMILY_IPV6),\n+\tOPT_STRING(0, \"partial-by-size\", &partial_by_size,\n+\t\t\t N_(\"size\"),\n+\t\t\t N_(\"only include blobs smaller than this\")),\n+\tOPT_BOOL(0, \"partial-special\", &partial_special,\n+\t\t\t N_(\"only include blobs for git special files\")),\n \tOPT_END()\n };\n \n@@ -731,6 +738,10 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n \tconst char *filename = dry_run ? \"/dev/null\" : git_path_fetch_head();\n \tint want_status;\n \tint summary_width = transport_summary_width(ref_map);\n+\tstruct check_connected_options opt = CHECK_CONNECTED_INIT;\n+\n+\tif (partial_by_size || partial_special)\n+\t\topt.allow_partial = 1;\n \n \tfp = fopen(filename, \"a\");\n \tif (!fp)\n@@ -742,7 +753,7 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n \t\turl = xstrdup(\"foreign\");\n \n \trm = ref_map;\n-\tif (check_connected(iterate_ref_map, &rm, NULL)) {\n+\tif (check_connected(iterate_ref_map, &rm, &opt)) {\n \t\trc = error(_(\"%s did not send all necessary objects\\n\"), url);\n \t\tgoto abort;\n \t}\n@@ -882,6 +893,9 @@ static int quickfetch(struct ref *ref_map)\n \tstruct ref *rm = ref_map;\n \tstruct check_connected_options opt = CHECK_CONNECTED_INIT;\n \n+\tif (partial_by_size || partial_special)\n+\t\topt.allow_partial = 1;\n+\n \t/*\n \t * If we are deepening a shallow clone we already have these\n \t * objects reachable. Running rev-list here will return with\n@@ -1020,6 +1034,10 @@ static struct transport *prepare_transport(struct remote *remote, int deepen)\n \t\tset_option(transport, TRANS_OPT_DEEPEN_RELATIVE, \"yes\");\n \tif (update_shallow)\n \t\tset_option(transport, TRANS_OPT_UPDATE_SHALLOW, \"yes\");\n+\tif (partial_by_size)\n+\t\tset_option(transport, TRANS_OPT_PARTIAL_BY_SIZE, partial_by_size);\n+\tif (partial_special)\n+\t\tset_option(transport, TRANS_OPT_PARTIAL_SPECIAL, \"yes\");\n \treturn transport;\n }\n \n@@ -1314,6 +1332,12 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_fetch_options, builtin_fetch_usage, 0);\n \n+\tif (partial_by_size) {\n+\t\tunsigned long s;\n+\t\tif (!git_parse_ulong(partial_by_size, &s))\n+\t\t\tdie(_(\"invalid partial-by-size value\"));\n+\t}\n+\n \tif (deepen_relative) {\n \t\tif (deepen_relative < 0)\n \t\t\tdie(_(\"Negative depth in --deepen is not supported\"));\ndiff --git a/connected.c b/connected.c\nindex 136c2ac..b07cbb5 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -62,6 +62,9 @@ int check_connected(sha1_iterate_fn fn, void *cb_data,\n \t\targv_array_pushf(&rev_list.args, \"--progress=%s\",\n \t\t\t\t _(\"Checking connectivity\"));\n \n+\tif (opt->allow_partial)\n+\t\targv_array_push(&rev_list.args, \"--allow-partial\");\n+\n \trev_list.git_cmd = 1;\n \trev_list.env = opt->env;\n \trev_list.in = -1;\ndiff --git a/connected.h b/connected.h\nindex 4ca325f..756259e 100644\n--- a/connected.h\n+++ b/connected.h\n@@ -34,6 +34,9 @@ struct check_connected_options {\n \t/* If non-zero, show progress as we traverse the objects. */\n \tint progress;\n \n+\t/* A previous partial clone/fetch may have omitted some blobs. */\n+\tint allow_partial;\n+\n \t/*\n \t * Insert these variables into the environment of the child process.\n \t */\n-- \n2.7.4\n\n"},{"id":"313545","messageId":"1488999039-37631-4-git-send-email-git@jeffhostetler.com","threadId":"45320","inReplyTo":"1488999039-37631-1-git-send-email-git@jeffhostetler.com","subject":"[PATCH 03/10] pack-objects: test for --partial-by-size --partial-special","fromName":"","fromEmail":"git@jeffhostetler.com","sentAt":"2017-03-08T18:50:32Z","receivedAt":"2017-03-08T20:47:35Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSome simple tests for pack-objects with the new --partial-by-size\nand --partial-special options.\n\nSigned-off-by: Jeff Hostetler \n---\n t/5316-pack-objects-partial.sh | 72 ++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 72 insertions(+)\n create mode 100644 t/5316-pack-objects-partial.sh\n\ndiff --git a/t/5316-pack-objects-partial.sh b/t/5316-pack-objects-partial.sh\nnew file mode 100644\nindex 0000000..352de34\n--- /dev/null\n+++ b/t/5316-pack-objects-partial.sh\n@@ -0,0 +1,72 @@\n+#!/bin/sh\n+\n+test_description='pack-object partial'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\tperl -e \"print \\\"a\\\" x 11;\" > a &&\n+\tperl -e \"print \\\"a\\\" x 1100;\" > b &&\n+\tperl -e \"print \\\"a\\\" x 1100000;\" > c &&\n+\techo \"ignored\" > .gitignore &&\n+\tgit add a b c .gitignore &&\n+\tgit commit -m test\n+\t'\n+\n+test_expect_success 'all blobs' '\n+\tgit pack-objects --revs --thin --stdout >all.pack <<-EOF &&\n+\tmaster\n+\t\n+\tEOF\n+\tgit index-pack all.pack &&\n+\ttest 4 = $(git verify-pack -v all.pack | grep blob | wc -l)\n+\t'\n+\n+test_expect_success 'no blobs' '\n+\tgit pack-objects --revs --thin --stdout --partial-by-size=0 >none.pack <<-EOF &&\n+\tmaster\n+\t\n+\tEOF\n+\tgit index-pack none.pack &&\n+\ttest 0 = $(git verify-pack -v none.pack | grep blob | wc -l)\n+\t'\n+\n+test_expect_success 'small blobs' '\n+\tgit pack-objects --revs --thin --stdout --partial-by-size=1M >small.pack <<-EOF &&\n+\tmaster\n+\t\n+\tEOF\n+\tgit index-pack small.pack &&\n+\ttest 3 = $(git verify-pack -v small.pack | grep blob | wc -l)\n+\t'\n+\n+test_expect_success 'tiny blobs' '\n+\tgit pack-objects --revs --thin --stdout --partial-by-size=100 >tiny.pack <<-EOF &&\n+\tmaster\n+\t\n+\tEOF\n+\tgit index-pack tiny.pack &&\n+\ttest 2 = $(git verify-pack -v tiny.pack | grep blob | wc -l)\n+\t'\n+\n+test_expect_success 'special' '\n+\tgit pack-objects --revs --thin --stdout --partial-special >spec.pack <<-EOF &&\n+\tmaster\n+\t\n+\tEOF\n+\tgit index-pack spec.pack &&\n+\ttest 1 = $(git verify-pack -v spec.pack | grep blob | wc -l)\n+\t'\n+\n+test_expect_success 'union' '\n+\tgit pack-objects --revs --thin --stdout --partial-by-size=0 --partial-special >union.pack <<-EOF &&\n+\tmaster\n+\t\n+\tEOF\n+\tgit index-pack union.pack &&\n+\ttest 1 = $(git verify-pack -v union.pack | grep blob | wc -l)\n+\t'\n+\n+test_done\n+\n+\n-- \n2.7.4\n\n"},{"id":"313546","messageId":"1488999039-37631-10-git-send-email-git@jeffhostetler.com","threadId":"45320","inReplyTo":"1488999039-37631-1-git-send-email-git@jeffhostetler.com","subject":"[PATCH 09/10] clone: add partial-by-size and partial-special arguments","fromName":"","fromEmail":"git@jeffhostetler.com","sentAt":"2017-03-08T18:50:38Z","receivedAt":"2017-03-08T20:47:38Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach clone to accept --partial-by-size=n and --partial-special\narguments to request that the server omit certain blobs from\nthe generated packfile.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/clone.c | 26 ++++++++++++++++++++++++++\n 1 file changed, 26 insertions(+)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 3f63edb..e5a5904 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -56,6 +56,8 @@ static struct string_list option_required_reference = STRING_LIST_INIT_NODUP;\n static struct string_list option_optional_reference = STRING_LIST_INIT_NODUP;\n static int option_dissociate;\n static int max_jobs = -1;\n+static const char *partial_by_size;\n+static int partial_special;\n \n static struct option builtin_clone_options[] = {\n \tOPT__VERBOSITY(&option_verbosity),\n@@ -112,6 +114,11 @@ static struct option builtin_clone_options[] = {\n \t\t\tTRANSPORT_FAMILY_IPV4),\n \tOPT_SET_INT('6', \"ipv6\", &family, N_(\"use IPv6 addresses only\"),\n \t\t\tTRANSPORT_FAMILY_IPV6),\n+\tOPT_STRING(0, \"partial-by-size\", &partial_by_size,\n+\t\t\t N_(\"size\"),\n+\t\t\t N_(\"only include blobs smaller than this\")),\n+\tOPT_BOOL(0, \"partial-special\", &partial_special,\n+\t\t\t N_(\"only include blobs for git special files\")),\n \tOPT_END()\n };\n \n@@ -625,6 +632,9 @@ static void update_remote_refs(const struct ref *refs,\n \tif (check_connectivity) {\n \t\tstruct check_connected_options opt = CHECK_CONNECTED_INIT;\n \n+\t\tif (partial_by_size || partial_special)\n+\t\t\topt.allow_partial = 1;\n+\n \t\topt.transport = transport;\n \t\topt.progress = transport->progress;\n \n@@ -1021,6 +1031,10 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\t\twarning(_(\"--shallow-since is ignored in local clones; use file:// instead.\"));\n \t\tif (option_not.nr)\n \t\t\twarning(_(\"--shallow-exclude is ignored in local clones; use file:// instead.\"));\n+\t\tif (partial_by_size)\n+\t\t\twarning(_(\"--partial-by-size is ignored in local clones; use file:// instead.\"));\n+\t\tif (partial_special)\n+\t\t\twarning(_(\"--partial-special is ignored in local clones; use file:// instead.\"));\n \t\tif (!access(mkpath(\"%s/shallow\", path), F_OK)) {\n \t\t\tif (option_local > 0)\n \t\t\t\twarning(_(\"source repository is shallow, ignoring --local\"));\n@@ -1052,6 +1066,18 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK,\n \t\t\t\t option_upload_pack);\n \n+\tif (partial_by_size) {\n+\t\ttransport_set_option(transport, TRANS_OPT_PARTIAL_BY_SIZE,\n+\t\t\t\t partial_by_size);\n+\t\tif (transport->smart_options)\n+\t\t\ttransport->smart_options->partial_by_size = partial_by_size;\n+\t}\n+\tif (partial_special) {\n+\t\ttransport_set_option(transport, TRANS_OPT_PARTIAL_SPECIAL, \"yes\");\n+\t\tif (transport->smart_options)\n+\t\t\ttransport->smart_options->partial_special = 1;\n+\t}\n+\n \tif (transport->smart_options && !deepen)\n \t\ttransport->smart_options->check_self_contained_and_connected = 1;\n \n-- \n2.7.4\n\n"},{"id":"313547","messageId":"1488999039-37631-3-git-send-email-git@jeffhostetler.com","threadId":"45320","inReplyTo":"1488999039-37631-1-git-send-email-git@jeffhostetler.com","subject":"[PATCH 02/10] pack-objects: add --partial-by-size=n --partial-special","fromName":"","fromEmail":"git@jeffhostetler.com","sentAt":"2017-03-08T18:50:31Z","receivedAt":"2017-03-08T20:47:39Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach pack-objects to omit blobs from the generated packfile.\n\nWhen the --partial-by-size=n[kmg] argument is used, only blobs\nsmaller than the requested size are included. When n is zero,\nno blobs are included.\n\nWhen the --partial-special argument is used, git special files,\nsuch as \".gitattributes\" and \".gitignores\" are included.\n\nWhen both are given, the union of two are included.\n\nThis is intended to be used in a partial clone or fetch.\n(This has also been called sparse- or lazy-clone.)\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/pack-objects.c | 62 +++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 61 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7e052bb..2df2f49 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -77,6 +77,10 @@ static unsigned long cache_max_small_delta_size = 1000;\n \n static unsigned long window_memory_limit = 0;\n \n+static signed long partial_by_size = -1;\n+static int partial_special = 0;\n+static struct trace_key trace_partial = TRACE_KEY_INIT(PARTIAL);\n+\n /*\n * stats\n */\n@@ -2532,6 +2536,54 @@ static void show_object(struct object *obj, const char *name, void *data)\n \tobj->flags |= OBJECT_ADDED;\n }\n \n+/*\n+ * If ANY --partial-* option was given, we want to OMIT all\n+ * blobs UNLESS they match one of our patterns. We treat\n+ * the options as OR's so that we get the resulting UNION.\n+ */\n+static void show_object_partial(struct object *obj, const char *name, void *data)\n+{\n+\tunsigned long s = 0;\n+\n+\tif (obj->type != OBJ_BLOB)\n+\t\tgoto include_it;\n+\n+\t/*\n+\t * When (partial_by_size == 0), we want to OMIT all blobs.\n+\t * When (partial_by_size > 0), we want blobs smaller than that.\n+\t */\n+\tif (partial_by_size > 0) {\n+\t\tenum object_type t = sha1_object_info(obj->oid.hash, &s);\n+\t\tassert(t == OBJ_BLOB);\n+\t\tif (s < partial_by_size)\n+\t\t\tgoto include_it;\n+\t}\n+\n+\t/*\n+\t * When (partial_special), we want the .git* special files.\n+\t */\n+\tif (partial_special) {\n+\t\tif (strcmp(name, GITATTRIBUTES_FILE) == 0 ||\n+\t\t\tstrcmp(name, \".gitignore\") == 0)\n+\t\t\tgoto include_it;\n+\t\telse {\n+\t\t\tconst char *last_slash = strrchr(name, '/');\n+\t\t\tif (last_slash)\n+\t\t\t\tif (strcmp(last_slash+1, GITATTRIBUTES_FILE) == 0 ||\n+\t\t\t\t\tstrcmp(last_slash+1, \".gitignore\") == 0)\n+\t\t\t\t\tgoto include_it;\n+\t\t}\n+\t}\n+\n+\ttrace_printf_key(\n+\t\t&trace_partial, \"omitting blob '%s' %\"PRIuMAX\" '%s'\\n\",\n+\t\toid_to_hex(&obj->oid), (uintmax_t)s, name);\n+\treturn;\n+\n+include_it:\n+\tshow_object(obj, name, data);\n+}\n+\n static void show_edge(struct commit *commit)\n {\n \tadd_preferred_base(commit->object.oid.hash);\n@@ -2794,7 +2846,11 @@ static void get_object_list(int ac, const char **av)\n \tif (prepare_revision_walk(&revs))\n \t\tdie(\"revision walk setup failed\");\n \tmark_edges_uninteresting(&revs, show_edge);\n-\ttraverse_commit_list(&revs, show_commit, show_object, NULL);\n+\n+\tif (partial_by_size >= 0 || partial_special)\n+\t\ttraverse_commit_list(&revs, show_commit, show_object_partial, NULL);\n+\telse\n+\t\ttraverse_commit_list(&revs, show_commit, show_object, NULL);\n \n \tif (unpack_unreachable_expiration) {\n \t\trevs.ignore_missing_links = 1;\n@@ -2930,6 +2986,10 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"use a bitmap index if available to speed up counting objects\")),\n \t\tOPT_BOOL(0, \"write-bitmap-index\", &write_bitmap_index,\n \t\t\t N_(\"write a bitmap index together with the pack index\")),\n+\t\tOPT_MAGNITUDE(0, \"partial-by-size\", (unsigned long *)&partial_by_size,\n+\t\t\t N_(\"only include blobs smaller than size in result\")),\n+\t\tOPT_BOOL(0, \"partial-special\", &partial_special,\n+\t\t\t N_(\"only include blobs for git special files\")),\n \t\tOPT_END(),\n \t};\n \n-- \n2.7.4\n\n"},{"id":"313548","messageId":"1488999039-37631-2-git-send-email-git@jeffhostetler.com","threadId":"45320","inReplyTo":"1488999039-37631-1-git-send-email-git@jeffhostetler.com","subject":"[PATCH 01/10] pack-objects: eat CR in addition to LF after fgets.","fromName":"","fromEmail":"git@jeffhostetler.com","sentAt":"2017-03-08T18:50:30Z","receivedAt":"2017-03-08T20:47:44Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n builtin/pack-objects.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex f294dcf..7e052bb 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2764,6 +2764,8 @@ static void get_object_list(int ac, const char **av)\n \t\tint len = strlen(line);\n \t\tif (len && line[len - 1] == '\\n')\n \t\t\tline[--len] = 0;\n+\t\tif (len && line[len - 1] == '\\r')\n+\t\t\tline[--len] = 0;\n \t\tif (!len)\n \t\t\tbreak;\n \t\tif (*line == '-') {\n-- \n2.7.4\n\n"},{"id":"313549","messageId":"1488999039-37631-8-git-send-email-git@jeffhostetler.com","threadId":"45320","inReplyTo":"1488999039-37631-1-git-send-email-git@jeffhostetler.com","subject":"[PATCH 07/10] index-pack: add --allow-partial option to relax blob existence checks","fromName":"","fromEmail":"git@jeffhostetler.com","sentAt":"2017-03-08T18:50:36Z","receivedAt":"2017-03-08T20:47:46Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach index-pack to optionally not complain when there are missing\nblobs. This is for use following a partial clone or fetch when\nthe server omitted certain blobs.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/index-pack.c | 20 ++++++++++++++++++--\n 1 file changed, 18 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex f4b87c6..8f99408 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -13,7 +13,7 @@\n #include \"thread-utils.h\"\n \n static const char index_pack_usage[] =\n-\"git index-pack [-v] [-o ] [--keep | --keep=] [--verify] [--strict] ( | --stdin [--fix-thin] [])\";\n+\"git index-pack [-v] [-o ] [--keep | --keep=] [--verify] [--allow-partial] [--strict] ( | --stdin [--fix-thin] [])\";\n \n struct object_entry {\n \tstruct pack_idx_entry idx;\n@@ -81,6 +81,9 @@ static int show_resolving_progress;\n static int show_stat;\n static int check_self_contained_and_connected;\n \n+static int allow_partial;\n+static struct trace_key trace_partial = TRACE_KEY_INIT(PARTIAL);\n+\n static struct progress *progress;\n \n /* We always read in 4kB chunks. */\n@@ -220,9 +223,18 @@ static unsigned check_object(struct object *obj)\n \tif (!(obj->flags & FLAG_CHECKED)) {\n \t\tunsigned long size;\n \t\tint type = sha1_object_info(obj->oid.hash, &size);\n-\t\tif (type <= 0)\n+\t\tif (type <= 0) {\n+\t\t\tif (allow_partial > 0 && obj->type == OBJ_BLOB) {\n+\t\t\t\t/* Assume a previous partial clone/fetch omitted it. */\n+\t\t\t\ttrace_printf_key(\n+\t\t\t\t\t&trace_partial, \"omitted blob '%s'\\n\",\n+\t\t\t\t\toid_to_hex(&obj->oid));\n+\t\t\t\tobj->flags |= FLAG_CHECKED;\n+\t\t\t\treturn 0;\n+\t\t\t}\n \t\t\tdie(_(\"did not receive expected object %s\"),\n \t\t\t oid_to_hex(&obj->oid));\n+\t\t}\n \t\tif (type != obj->type)\n \t\t\tdie(_(\"object %s: expected type %s, found %s\"),\n \t\t\t oid_to_hex(&obj->oid),\n@@ -1718,6 +1730,10 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\t\tdie(_(\"bad %s\"), arg);\n \t\t\t} else if (skip_prefix(arg, \"--max-input-size=\", &arg)) {\n \t\t\t\tmax_input_size = strtoumax(arg, NULL, 10);\n+\t\t\t} else if (!strcmp(arg, \"--allow-partial\")) {\n+\t\t\t\tallow_partial = 1;\n+\t\t\t} else if (!strcmp(arg, \"--no-allow-partial\")) {\n+\t\t\t\tallow_partial = 0;\n \t\t\t} else\n \t\t\t\tusage(index_pack_usage);\n \t\t\tcontinue;\n-- \n2.7.4\n\n"},{"id":"313550","messageId":"1488999039-37631-11-git-send-email-git@jeffhostetler.com","threadId":"45320","inReplyTo":"1488999039-37631-1-git-send-email-git@jeffhostetler.com","subject":"[PATCH 10/10] ls-partial: created command to list missing blobs","fromName":"","fromEmail":"git@jeffhostetler.com","sentAt":"2017-03-08T18:50:39Z","receivedAt":"2017-03-08T20:47:55Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nAdded a command to list the missing blobs for a commit.\nThis can be used after a partial clone or fetch to list\nthe omitted blobs that the client would need to checkout\nthe given commit/branch. Optionally respecting or ignoring\nthe current sparse-checkout definition.\n\nThis command prints a simple list of blob SHAs. It is\nexpected that this would be piped into another command\nwith knowledge of the transport and/or blob store.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 2 +\n builtin.h | 1 +\n builtin/ls-partial.c | 110 ++++++++++++++++++++\n git.c | 1 +\n partial-utils.c | 279 +++++++++++++++++++++++++++++++++++++++++++++++++++\n partial-utils.h | 93 +++++++++++++++++\n 6 files changed, 486 insertions(+)\n create mode 100644 builtin/ls-partial.c\n create mode 100644 partial-utils.c\n create mode 100644 partial-utils.h\n\ndiff --git a/Makefile b/Makefile\nindex 9ec6065..96e9e1e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -791,6 +791,7 @@ LIB_OBJS += pack-write.o\n LIB_OBJS += pager.o\n LIB_OBJS += parse-options.o\n LIB_OBJS += parse-options-cb.o\n+LIB_OBJS += partial-utils.o\n LIB_OBJS += patch-delta.o\n LIB_OBJS += patch-ids.o\n LIB_OBJS += path.o\n@@ -908,6 +909,7 @@ BUILTIN_OBJS += builtin/init-db.o\n BUILTIN_OBJS += builtin/interpret-trailers.o\n BUILTIN_OBJS += builtin/log.o\n BUILTIN_OBJS += builtin/ls-files.o\n+BUILTIN_OBJS += builtin/ls-partial.o\n BUILTIN_OBJS += builtin/ls-remote.o\n BUILTIN_OBJS += builtin/ls-tree.o\n BUILTIN_OBJS += builtin/mailinfo.o\ndiff --git a/builtin.h b/builtin.h\nindex 9e4a898..df00c4b 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -79,6 +79,7 @@ extern int cmd_interpret_trailers(int argc, const char **argv, const char *prefi\n extern int cmd_log(int argc, const char **argv, const char *prefix);\n extern int cmd_log_reflog(int argc, const char **argv, const char *prefix);\n extern int cmd_ls_files(int argc, const char **argv, const char *prefix);\n+extern int cmd_ls_partial(int argc, const char **argv, const char *prefix);\n extern int cmd_ls_tree(int argc, const char **argv, const char *prefix);\n extern int cmd_ls_remote(int argc, const char **argv, const char *prefix);\n extern int cmd_mailinfo(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/ls-partial.c b/builtin/ls-partial.c\nnew file mode 100644\nindex 0000000..8ebf045\n--- /dev/null\n+++ b/builtin/ls-partial.c\n@@ -0,0 +1,110 @@\n+#include \"cache.h\"\n+#include \"blob.h\"\n+#include \"tree.h\"\n+#include \"commit.h\"\n+#include \"quote.h\"\n+#include \"builtin.h\"\n+#include \"parse-options.h\"\n+#include \"pathspec.h\"\n+#include \"dir.h\"\n+#include \"partial-utils.h\"\n+\n+static struct trace_key trace_partial = TRACE_KEY_INIT(PARTIAL);\n+\n+static int verbose;\n+static int ignore_sparse;\n+struct exclude_list el;\n+\n+static const char * const ls_partial_usage[] = {\n+\tN_(\"git ls-partial [] \"),\n+\tNULL\n+};\n+\n+/*\n+ * map arg into SHA1 and get the root treenode.\n+ */\n+static struct tree *lookup_tree_from_treeish(const char *arg)\n+{\n+\tunsigned char sha1[20];\n+\tstruct tree *tree;\n+\n+\tif (get_sha1(arg, sha1))\n+\t\tdie(\"not a valid object name '%s'\", arg);\n+\n+\ttrace_printf_key(\n+\t\t&trace_partial,\n+\t\t\"ls-partial: treeish '%s' '%s'\\n\",\n+\t\targ, sha1_to_hex(sha1));\n+\n+\tif (verbose) {\n+\t\tprintf(\"commit\\t%s\\n\", sha1_to_hex(sha1));\n+\t\tprintf(\"branch\\t%s\\n\", arg);\n+\t}\n+\t\n+\ttree = parse_tree_indirect(sha1);\n+\tif (!tree)\n+\t\tdie(\"not a tree object '%s'\", arg);\n+\n+\treturn tree;\n+}\n+\n+static void print_results(const struct pu_vec *vec)\n+{\n+\tint k;\n+\n+\tfor (k = 0; k < vec->data_nr; k++)\n+\t\tprintf(\"%s\\n\", oid_to_hex(&vec->data[k]->oid));\n+}\n+\n+static void print_results_verbose(const struct pu_vec *vec)\n+{\n+\tint k;\n+\n+\t/* TODO Consider -z version */\n+\n+\tfor (k = 0; k < vec->data_nr; k++)\n+\t\tprintf(\"%s\\t%s\\n\", oid_to_hex(&vec->data[k]->oid), vec->data[k]->fullpath.buf);\n+}\n+\n+int cmd_ls_partial(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct exclude_list el;\n+\tstruct tree *tree;\n+\tstruct pu_vec *vec;\n+\tstruct pu_vec *vec_all = NULL;\n+\tstruct pu_vec *vec_sparse = NULL;\n+\tstruct pu_vec *vec_missing = NULL;\n+\t\n+\tconst struct option ls_partial_options[] = {\n+\t\tOPT__VERBOSE(&verbose, N_(\"show verbose blob details\")),\n+\t\tOPT_BOOL(0, \"ignore-sparse\", &ignore_sparse,\n+\t\t\t\t N_(\"ignore sparse-checkout settings (scan whole tree)\")),\n+\t\tOPT_END()\n+\t};\n+\n+\tgit_config(git_default_config, NULL);\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t\t\t\t ls_partial_options, ls_partial_usage, 0);\n+\tif (argc < 1)\n+\t\tusage_with_options(ls_partial_usage, ls_partial_options);\n+\n+\ttree = lookup_tree_from_treeish(argv[0]);\n+\n+\tvec_all = pu_vec_ls_tree(tree, prefix, argv + 1);\n+\tif (ignore_sparse || pu_load_sparse_definitions(\"info/sparse-checkout\", &el) < 0)\n+\t\tvec = vec_all;\n+\telse {\n+\t\tvec_sparse = pu_vec_filter_sparse(vec_all, &el);\n+\t\tvec = vec_sparse;\n+\t}\n+\n+\tvec_missing = pu_vec_filter_missing(vec);\n+\tvec = vec_missing;\n+\n+\tif (verbose)\n+\t\tprint_results_verbose(vec);\n+\telse\n+\t\tprint_results(vec);\n+\n+\treturn 0;\n+}\ndiff --git a/git.c b/git.c\nindex 33f52ac..ef1e019 100644\n--- a/git.c\n+++ b/git.c\n@@ -444,6 +444,7 @@ static struct cmd_struct commands[] = {\n \t{ \"interpret-trailers\", cmd_interpret_trailers, RUN_SETUP_GENTLY },\n \t{ \"log\", cmd_log, RUN_SETUP },\n \t{ \"ls-files\", cmd_ls_files, RUN_SETUP | SUPPORT_SUPER_PREFIX },\n+\t{ \"ls-partial\", cmd_ls_partial, RUN_SETUP },\n \t{ \"ls-remote\", cmd_ls_remote, RUN_SETUP_GENTLY },\n \t{ \"ls-tree\", cmd_ls_tree, RUN_SETUP },\n \t{ \"mailinfo\", cmd_mailinfo, RUN_SETUP_GENTLY },\ndiff --git a/partial-utils.c b/partial-utils.c\nnew file mode 100644\nindex 0000000..b75e91e\n--- /dev/null\n+++ b/partial-utils.c\n@@ -0,0 +1,279 @@\n+#include \"cache.h\"\n+#include \"blob.h\"\n+#include \"tree.h\"\n+#include \"commit.h\"\n+#include \"quote.h\"\n+#include \"builtin.h\"\n+#include \"parse-options.h\"\n+#include \"pathspec.h\"\n+#include \"dir.h\"\n+#include \"partial-utils.h\"\n+\n+static struct trace_key trace_partial_utils = TRACE_KEY_INIT(PARTIAL_UTILS);\n+\n+void pu_row_trace(\n+\tconst struct pu_row *row,\n+\tconst char *label)\n+{\n+\ttrace_printf_key(\n+\t\t&trace_partial_utils,\n+\t\t\"%s: %06o %s %.*s\\n\",\n+\t\tlabel,\n+\t\trow->mode,\n+\t\toid_to_hex(&row->oid),\n+\t\t(int)row->fullpath.len,\n+\t\trow->fullpath.buf);\n+}\n+\n+struct pu_row *pu_row_alloc(\n+\tconst unsigned char *sha1,\n+\tconst struct strbuf *base,\n+\tconst char *entryname,\n+\tunsigned mode)\n+{\n+\tstruct pu_row *row = xcalloc(1, sizeof(struct pu_row));\n+\n+\thashcpy(row->oid.hash, sha1);\n+\tstrbuf_init(&row->fullpath, base->len + strlen(entryname) + 1);\n+\tif (base->len)\n+\t\tstrbuf_addbuf(&row->fullpath, base);\n+\tstrbuf_addstr(&row->fullpath, entryname);\n+\trow->mode = mode;\n+\trow->entryname_offset = base->len;\n+\n+\tpu_row_trace(row, \"alloc\");\n+\n+\treturn row;\n+}\n+\n+struct pu_vec *pu_vec_alloc(\n+\tunsigned int nr_pre_alloc)\n+{\n+\tstruct pu_vec *vec = xcalloc(1, sizeof(struct pu_vec));\n+\n+\tvec->data = xcalloc(nr_pre_alloc, sizeof(struct pu_row *));\n+\tvec->data_alloc = nr_pre_alloc;\n+\n+\treturn vec;\n+}\n+\n+void pu_vec_append(\n+\tstruct pu_vec *vec,\n+\tstruct pu_row *row)\n+{\n+\tALLOC_GROW(vec->data, vec->data_nr + 1, vec->data_alloc);\n+\tvec->data[vec->data_nr++] = row;\n+}\n+\n+static int ls_tree_cb(\n+\tconst unsigned char *sha1,\n+\tstruct strbuf *base,\n+\tconst char *pathname,\n+\tunsigned mode,\n+\tint stage,\n+\tvoid *context)\n+{\n+\tstruct pu_vec *vec = (struct pu_vec *)context;\n+\n+\t/* omit submodules */\n+\tif (S_ISGITLINK(mode))\n+\t\treturn 0;\n+\n+\tpu_vec_append(vec, pu_row_alloc(sha1, base, pathname, mode));\n+\n+\tif (S_ISDIR(mode))\n+\t\treturn READ_TREE_RECURSIVE;\n+\n+\treturn 0;\n+}\n+\n+struct pu_vec *pu_vec_ls_tree(\n+\tstruct tree *tree,\n+\tconst char *prefix,\n+\tconst char **argv)\n+{\n+\tstruct pu_vec *vec;\n+\tstruct pathspec pathspec;\n+\tint k;\n+\n+\tvec = pu_vec_alloc(PU_VEC_DEFAULT_SIZE);\n+\n+\tparse_pathspec(\n+\t\t&pathspec, PATHSPEC_GLOB | PATHSPEC_ICASE | PATHSPEC_EXCLUDE,\n+\t\tPATHSPEC_PREFER_CWD, prefix, argv);\n+\tfor (k = 0; k < pathspec.nr; k++)\n+\t\tpathspec.items[k].nowildcard_len = pathspec.items[k].len;\n+\tpathspec.has_wildcard = 0;\n+\n+\tif (read_tree_recursive(tree, \"\", 0, 0, &pathspec, ls_tree_cb, vec) != 0)\n+\t\tdie(\"Could not read tree\");\n+\n+\treturn vec;\n+}\n+\n+int pu_load_sparse_definitions(\n+\tconst char *path,\n+\tstruct exclude_list *pel)\n+{\n+\tint result;\n+\tchar *sparse = git_pathdup(\"info/sparse-checkout\");\n+\tmemset(pel, 0, sizeof(*pel));\n+\tresult = add_excludes_from_file_to_list(sparse, \"\", 0, pel, 0);\n+\tfree(sparse);\n+\treturn result;\n+}\n+\n+static int mode_to_dtype(unsigned mode)\n+{\n+\tif (S_ISREG(mode))\n+\t\treturn DT_REG;\n+\tif (S_ISDIR(mode) || S_ISGITLINK(mode))\n+\t\treturn DT_DIR;\n+\tif (S_ISLNK(mode))\n+\t\treturn DT_LNK;\n+\treturn DT_UNKNOWN;\n+}\n+\n+static int apply_excludes_1(\n+\tstruct pu_row **subset,\n+\tunsigned int nr,\n+\tstruct strbuf *prefix,\n+\tstruct exclude_list *pel,\n+\tint defval,\n+\tstruct pu_vec *vec_out);\n+\n+/* apply directory rules. based on clear_ce_flags_dir() */\n+static int apply_excludes_dir(\n+\tstruct pu_row **subset,\n+\tunsigned int nr,\n+\tstruct strbuf *prefix,\n+\tchar *basename,\n+\tstruct exclude_list *pel,\n+\tint defval,\n+\tstruct pu_vec *vec_out)\n+{\n+\tstruct pu_row **subset_end;\n+\tint dtype = DT_DIR;\n+\tint ret = is_excluded_from_list(\n+\t\tprefix->buf, prefix->len, basename, &dtype, pel);\n+\tint rc;\n+\n+\tstrbuf_addch(prefix, '/');\n+\n+\tif (ret < 0)\n+\t\tret = defval;\n+\n+\tfor (subset_end = subset; subset_end != subset + nr; subset_end++) {\n+\t\tstruct pu_row *row = *subset_end;\n+\t\tif (strncmp(row->fullpath.buf, prefix->buf, prefix->len))\n+\t\t\tbreak;\n+\t}\n+\n+\trc = apply_excludes_1(\n+\t\tsubset, subset_end - subset,\n+\t\tprefix, pel, ret,\n+\t\tvec_out);\n+\tstrbuf_setlen(prefix, prefix->len - 1);\n+\treturn rc;\n+}\n+\n+/* apply sparse rules to subset[0..nr). based on clear_ce_flags_1() */\n+static int apply_excludes_1(\n+\tstruct pu_row **subset,\n+\tunsigned int nr,\n+\tstruct strbuf *prefix,\n+\tstruct exclude_list *pel,\n+\tint defval,\n+\tstruct pu_vec *vec_out)\n+{\n+\tstruct pu_row **subset_end = subset + nr;\n+\n+\twhile (subset != subset_end) {\n+\t\tstruct pu_row *row = *subset;\n+\t\tconst char *name, *slash;\n+\t\tint len, dtype, val;\n+\n+\t\tif (prefix->len && strncmp(row->fullpath.buf, prefix->buf, prefix->len))\n+\t\t\tbreak;\n+\n+\t\tname = row->fullpath.buf + prefix->len;\n+\t\tslash = strchr(name, '/');\n+\n+\t\tif (slash) {\n+\t\t\tint processed;\n+\n+\t\t\tlen = slash - name;\n+\t\t\tstrbuf_add(prefix, name, len);\n+\n+\t\t\tprocessed = apply_excludes_dir(\n+\t\t\t\tsubset, subset_end - subset,\n+\t\t\t\tprefix, prefix->buf + prefix->len - len,\n+\t\t\t\tpel, defval,\n+\t\t\t\tvec_out);\n+\n+\t\t\tif (processed) {\n+\t\t\t\tsubset += processed;\n+\t\t\t\tstrbuf_setlen(prefix, prefix->len - len);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tstrbuf_addch(prefix, '/');\n+\t\t\tsubset += apply_excludes_1(\n+\t\t\t\tsubset, subset_end - subset,\n+\t\t\t\tprefix, pel, defval,\n+\t\t\t\tvec_out);\n+\t\t\tstrbuf_setlen(prefix, prefix->len - len - 1);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tdtype = mode_to_dtype(row->mode);\n+\t\tval = is_excluded_from_list(\n+\t\t\trow->fullpath.buf, row->fullpath.len, name, &dtype, pel);\n+\t\tif (val < 0)\n+\t\t\tval = defval;\n+\t\tif (val > 0) {\n+\t\t\tpu_row_trace(row, \"sparse\");\n+\t\t\tpu_vec_append(vec_out, row);\n+\t\t}\n+\t\tsubset++;\n+\t}\n+\n+\treturn nr - (subset_end - subset);\n+}\n+\n+struct pu_vec *pu_vec_filter_sparse(\n+\tconst struct pu_vec *vec_in,\n+\tstruct exclude_list *pel)\n+{\n+\tstruct pu_vec *vec_out;\n+\tstruct strbuf prefix = STRBUF_INIT;\n+\tint defval = 0;\n+\n+\tvec_out = pu_vec_alloc(vec_in->data_nr);\n+\n+\tapply_excludes_1(\n+\t\tvec_in->data, vec_in->data_nr,\n+\t\t&prefix, pel, defval,\n+\t\tvec_out);\n+\n+\treturn vec_out;\n+}\n+\n+struct pu_vec *pu_vec_filter_missing(\n+\tconst struct pu_vec *vec_in)\n+{\n+\tstruct pu_vec *vec_out;\n+\tint k;\n+\n+\tvec_out = pu_vec_alloc(vec_in->data_nr);\n+\n+\tfor (k = 0; k < vec_in->data_nr; k++) {\n+\t\tstruct pu_row *row = vec_in->data[k];\n+\t\tif (!has_sha1_file(row->oid.hash)) {\n+\t\t\tpu_row_trace(row, \"missing\");\n+\t\t\tpu_vec_append(vec_out, row);\n+\t\t}\n+\t}\n+\n+\treturn vec_out;\n+}\ndiff --git a/partial-utils.h b/partial-utils.h\nnew file mode 100644\nindex 0000000..3bdf2e4\n--- /dev/null\n+++ b/partial-utils.h\n@@ -0,0 +1,93 @@\n+#ifndef PARTIAL_UTILS_H\n+#define PARTIAL_UTILS_H\n+\n+/*\n+ * A 'partial-utils row' represents a single item in the tree.\n+ * This is conceptually equivalent to a cache_entry, but does\n+ * not require an index_state and lets us operate on any commit\n+ * and not be tied to the current worktree.\n+ */\n+struct pu_row\n+{\n+\tstruct strbuf fullpath;\n+\tstruct object_id oid;\n+\tunsigned mode;\n+\tunsigned entryname_offset;\n+};\n+\n+/*\n+ * A 'partial-utils vec' represents a vector of 'pu row'\n+ * values using the normal vector machinery.\n+ */\n+struct pu_vec\n+{\n+\tstruct pu_row **data;\n+\tunsigned int data_nr;\n+\tunsigned int data_alloc;\n+};\n+\n+#define PU_VEC_DEFAULT_SIZE (1024*1024)\n+\n+\n+void pu_row_trace(\n+\tconst struct pu_row *row,\n+\tconst char *label);\n+\n+struct pu_row *pu_row_alloc(\n+\tconst unsigned char *sha1,\n+\tconst struct strbuf *base,\n+\tconst char *entryname,\n+\tunsigned mode);\n+\n+struct pu_vec *pu_vec_alloc(\n+\tunsigned int nr_pre_alloc);\n+\n+/*\n+ * Append the given row onto the vector WITHOUT\n+ * assuming ownership of the pointer.\n+ */\n+void pu_vec_append(\n+\tstruct pu_vec *vec,\n+\tstruct pu_row *row);\n+\n+/*\n+ * Enumerate the contents of the tree (recursively) into\n+ * a vector of rows. This is essentially \"ls-tree -r -t\"\n+ * into a vector.\n+ */ \n+struct pu_vec *pu_vec_ls_tree(\n+\tstruct tree *tree,\n+\tconst char *prefix,\n+\tconst char **argv);\n+\n+/*\n+ * Load a sparse-checkout file into (*pel).\n+ * Returns -1 if none or error.\n+ */\n+int pu_load_sparse_definitions(\n+\tconst char *path,\n+\tstruct exclude_list *pel);\n+\n+/*\n+ * Filter the given vector using the sparse-checkout\n+ * definitions and return new vector of just the paths\n+ * that WOULD BE populated.\n+ *\n+ * The returned vector BORROWS rows from the input vector.\n+ *\n+ * This is loosely based upon clear_ce_flags() in unpack-trees.c\n+ */\n+struct pu_vec *pu_vec_filter_sparse(\n+\tconst struct pu_vec *vec_in,\n+\tstruct exclude_list *pel);\n+\n+/*\n+ * Filter the given vector and return the list of blobs\n+ * missing from the local ODB.\n+ *\n+ * The returned vector BORROWS rows from the input vector.\n+ */\n+struct pu_vec *pu_vec_filter_missing(\n+\tconst struct pu_vec *vec_in);\n+\n+#endif /* PARTIAL_UTILS_H */\n-- \n2.7.4\n\n"},{"id":"313551","messageId":"1488999039-37631-7-git-send-email-git@jeffhostetler.com","threadId":"45320","inReplyTo":"1488999039-37631-1-git-send-email-git@jeffhostetler.com","subject":"[PATCH 06/10] rev-list: add --allow-partial option to relax connectivity checks","fromName":"","fromEmail":"git@jeffhostetler.com","sentAt":"2017-03-08T18:50:35Z","receivedAt":"2017-03-08T20:48:01Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach rev-list to optionally not complain when there are missing\nblobs. This is for use following a partial clone or fetch when\nthe server omitted certain blobs.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/rev-list.c | 22 +++++++++++++++++++++-\n 1 file changed, 21 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 0aa93d5..50c49ba 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -45,6 +45,7 @@ static const char rev_list_usage[] =\n \" --left-right\\n\"\n \" --count\\n\"\n \" special purpose:\\n\"\n+\" --allow-partial\\n\"\n \" --bisect\\n\"\n \" --bisect-vars\\n\"\n \" --bisect-all\"\n@@ -53,6 +54,9 @@ static const char rev_list_usage[] =\n static struct progress *progress;\n static unsigned progress_counter;\n \n+static int allow_partial;\n+static struct trace_key trace_partial = TRACE_KEY_INIT(PARTIAL);\n+\n static void finish_commit(struct commit *commit, void *data);\n static void show_commit(struct commit *commit, void *data)\n {\n@@ -178,8 +182,16 @@ static void finish_commit(struct commit *commit, void *data)\n static void finish_object(struct object *obj, const char *name, void *cb_data)\n {\n \tstruct rev_list_info *info = cb_data;\n-\tif (obj->type == OBJ_BLOB && !has_object_file(&obj->oid))\n+\tif (obj->type == OBJ_BLOB && !has_object_file(&obj->oid)) {\n+\t\tif (allow_partial) {\n+\t\t\t/* Assume a previous partial clone/fetch omitted it. */\n+\t\t\ttrace_printf_key(\n+\t\t\t\t&trace_partial, \"omitted blob '%s' '%s'\\n\",\n+\t\t\t\toid_to_hex(&obj->oid), name);\n+\t\t\treturn;\n+\t\t}\n \t\tdie(\"missing blob object '%s'\", oid_to_hex(&obj->oid));\n+\t}\n \tif (info->revs->verify_objects && !obj->parsed && obj->type != OBJ_COMMIT)\n \t\tparse_object(obj->oid.hash);\n }\n@@ -329,6 +341,14 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\tshow_progress = arg;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!strcmp(arg, \"--allow-partial\")) {\n+\t\t\tallow_partial = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"--no-allow-partial\")) {\n+\t\t\tallow_partial = 0;\n+\t\t\tcontinue;\n+\t\t}\n \t\tusage(rev_list_usage);\n \n \t}\n-- \n2.7.4\n\n"},{"id":"313552","messageId":"1488999039-37631-6-git-send-email-git@jeffhostetler.com","threadId":"45320","inReplyTo":"1488999039-37631-1-git-send-email-git@jeffhostetler.com","subject":"[PATCH 05/10] fetch-pack: add partial-by-size and partial-special","fromName":"","fromEmail":"git@jeffhostetler.com","sentAt":"2017-03-08T18:50:34Z","receivedAt":"2017-03-08T20:48:09Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fetch-pack to take --partial-by-size and --partial-special\narguments and pass them via the transport to upload-pack to\nrequest that certain blobs be omitted from the resulting packfile.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fetch-pack.c | 9 +++++++++\n fetch-pack.c | 17 +++++++++++++++++\n fetch-pack.h | 2 ++\n transport.c | 8 ++++++++\n transport.h | 8 ++++++++\n 5 files changed, 44 insertions(+)\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex cfe9e44..324d7b2 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -8,6 +8,7 @@\n static const char fetch_pack_usage[] =\n \"git fetch-pack [--all] [--stdin] [--quiet | -q] [--keep | -k] [--thin] \"\n \"[--include-tag] [--upload-pack=] [--depth=] \"\n+\"[--partial-by-size=] [--partial-special] \"\n \"[--no-progress] [--diag-url] [-v] [:] [...]\";\n \n static void add_sought_entry(struct ref ***sought, int *nr, int *alloc,\n@@ -143,6 +144,14 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\targs.update_shallow = 1;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (skip_prefix(arg, \"--partial-by-size=\", &arg)) {\n+\t\t\targs.partial_by_size = xstrdup(arg);\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(\"--partial-special\", arg)) {\n+\t\t\targs.partial_special = 1;\n+\t\t\tcontinue;\n+\t\t}\n \t\tusage(fetch_pack_usage);\n \t}\n \tif (deepen_not.nr)\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex e0f5d5c..e355c38 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -372,6 +372,8 @@ static int find_common(struct fetch_pack_args *args,\n \t\t\tif (prefer_ofs_delta) strbuf_addstr(&c, \" ofs-delta\");\n \t\t\tif (deepen_since_ok) strbuf_addstr(&c, \" deepen-since\");\n \t\t\tif (deepen_not_ok) strbuf_addstr(&c, \" deepen-not\");\n+\t\t\tif (args->partial_by_size || args->partial_special)\n+\t\t\t\tstrbuf_addstr(&c, \" partial\");\n \t\t\tif (agent_supported) strbuf_addf(&c, \" agent=%s\",\n \t\t\t\t\t\t\t git_user_agent_sanitized());\n \t\t\tpacket_buf_write(&req_buf, \"want %s%s\\n\", remote_hex, c.buf);\n@@ -402,6 +404,12 @@ static int find_common(struct fetch_pack_args *args,\n \t\t\tpacket_buf_write(&req_buf, \"deepen-not %s\", s->string);\n \t\t}\n \t}\n+\n+\tif (args->partial_by_size)\n+\t\tpacket_buf_write(&req_buf, \"partial-by-size %s\", args->partial_by_size);\n+\tif (args->partial_special)\n+\t\tpacket_buf_write(&req_buf, \"partial-special\");\n+\n \tpacket_buf_flush(&req_buf);\n \tstate_len = req_buf.len;\n \n@@ -807,6 +815,10 @@ static int get_pack(struct fetch_pack_args *args,\n \t\t\t\t\t\"--keep=fetch-pack %\"PRIuMAX \" on %s\",\n \t\t\t\t\t(uintmax_t)getpid(), hostname);\n \t\t}\n+\n+\t\tif (args->partial_by_size || args->partial_special)\n+\t\t\targv_array_push(&cmd.args, \"--allow-partial\");\n+\n \t\tif (args->check_self_contained_and_connected)\n \t\t\targv_array_push(&cmd.args, \"--check-self-contained-and-connected\");\n \t}\n@@ -920,6 +932,11 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \telse\n \t\tprefer_ofs_delta = 0;\n \n+\tif (server_supports(\"partial\"))\n+\t\tprint_verbose(args, _(\"Server supports partial\"));\n+\telse if (args->partial_by_size || args->partial_special)\n+\t\tdie(_(\"Server does not support 'partial'\"));\n+\n \tif ((agent_feature = server_feature_value(\"agent\", &agent_len))) {\n \t\tagent_supported = 1;\n \t\tif (agent_len)\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex c912e3d..b8a26e0 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -12,6 +12,7 @@ struct fetch_pack_args {\n \tint depth;\n \tconst char *deepen_since;\n \tconst struct string_list *deepen_not;\n+\tconst char *partial_by_size;\n \tunsigned deepen_relative:1;\n \tunsigned quiet:1;\n \tunsigned keep_pack:1;\n@@ -29,6 +30,7 @@ struct fetch_pack_args {\n \tunsigned cloning:1;\n \tunsigned update_shallow:1;\n \tunsigned deepen:1;\n+\tunsigned partial_special:1;\n };\n \n /*\ndiff --git a/transport.c b/transport.c\nindex 5828e06..45f35a4 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -160,6 +160,12 @@ static int set_git_option(struct git_transport_options *opts,\n \t} else if (!strcmp(name, TRANS_OPT_DEEPEN_RELATIVE)) {\n \t\topts->deepen_relative = !!value;\n \t\treturn 0;\n+\t} else if (!strcmp(name, TRANS_OPT_PARTIAL_BY_SIZE)) {\n+\t\topts->partial_by_size = xstrdup(value);\n+\t\treturn 0;\n+\t} else if (!strcmp(name, TRANS_OPT_PARTIAL_SPECIAL)) {\n+\t\topts->partial_special = !!value;\n+\t\treturn 0;\n \t}\n \treturn 1;\n }\n@@ -227,6 +233,8 @@ static int fetch_refs_via_pack(struct transport *transport,\n \t\tdata->options.check_self_contained_and_connected;\n \targs.cloning = transport->cloning;\n \targs.update_shallow = data->options.update_shallow;\n+\targs.partial_by_size = data->options.partial_by_size;\n+\targs.partial_special = data->options.partial_special;\n \n \tif (!data->got_remote_heads) {\n \t\tconnect_setup(transport, 0);\ndiff --git a/transport.h b/transport.h\nindex bc55715..c3f2d52 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -15,12 +15,14 @@ struct git_transport_options {\n \tunsigned self_contained_and_connected : 1;\n \tunsigned update_shallow : 1;\n \tunsigned deepen_relative : 1;\n+\tunsigned partial_special : 1;\n \tint depth;\n \tconst char *deepen_since;\n \tconst struct string_list *deepen_not;\n \tconst char *uploadpack;\n \tconst char *receivepack;\n \tstruct push_cas_option *cas;\n+\tconst char *partial_by_size;\n };\n \n enum transport_family {\n@@ -210,6 +212,12 @@ void transport_check_allowed(const char *type);\n /* Send push certificates */\n #define TRANS_OPT_PUSH_CERT \"pushcert\"\n \n+/* Partial fetch to only include small files */\n+#define TRANS_OPT_PARTIAL_BY_SIZE \"partial-by-size\"\n+\n+/* Partial fetch to only include special files, like \".gitignore\" */\n+#define TRANS_OPT_PARTIAL_SPECIAL \"partial-special\"\n+\n /**\n * Returns 0 if the option was used, non-zero otherwise. Prints a\n * message to stderr if the option is not used.\n-- \n2.7.4\n\n"},{"id":"313553","messageId":"1488999039-37631-5-git-send-email-git@jeffhostetler.com","threadId":"45320","inReplyTo":"1488999039-37631-1-git-send-email-git@jeffhostetler.com","subject":"[PATCH 04/10] upload-pack: add partial (sparse) fetch","fromName":"","fromEmail":"git@jeffhostetler.com","sentAt":"2017-03-08T18:50:33Z","receivedAt":"2017-03-08T20:48:11Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach upload-pack to advertise the \"partial\" capability\nin the fetch-pack/upload-pack protocol header and to pass\nthe value of partial-by-size and partial-special on to\npack-objects.\n\nUpdate protocol documentation.\n\nThis might be used in conjunction with a partial (sparse) clone\nor fetch to omit various blobs from the generated packfile.\n\nSigned-off-by: Jeff Hostetler \n---\n Documentation/technical/pack-protocol.txt | 14 ++++++++++\n Documentation/technical/protocol-capabilities.txt | 7 +++++\n upload-pack.c | 32 ++++++++++++++++++++++-\n 3 files changed, 52 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt\nindex c59ac99..0032729 100644\n--- a/Documentation/technical/pack-protocol.txt\n+++ b/Documentation/technical/pack-protocol.txt\n@@ -212,6 +212,7 @@ out of what the server said it could do with the first 'want' line.\n upload-request = want-list\n \t\t *shallow-line\n \t\t *1depth-request\n+\t\t *partial\n \t\t flush-pkt\n \n want-list = first-want\n@@ -223,10 +224,15 @@ out of what the server said it could do with the first 'want' line.\n \t\t PKT-LINE(\"deepen-since\" SP timestamp) /\n \t\t PKT-LINE(\"deepen-not\" SP ref)\n \n+ partial = PKT-LINE(\"partial-by-size\" SP magnitude) /\n+\t\t PKT-LINE(\"partial-special) \n+\n first-want = PKT-LINE(\"want\" SP obj-id SP capability-list)\n additional-want = PKT-LINE(\"want\" SP obj-id)\n \n depth = 1*DIGIT\n+\n+ magnitude = 1*DIGIT [ \"k\" | \"m\" | \"g\" ]\n ----\n \n Clients MUST send all the obj-ids it wants from the reference\n@@ -249,6 +255,14 @@ complete those commits. Commits whose parents are not received as a\n result are defined as shallow and marked as such in the server. This\n information is sent back to the client in the next step.\n \n+The client can optionally request a partial packfile that omits\n+various blobs. The value of \"partial-by-size\" is a non-negative\n+integer with optional units and requests blobs smaller than this\n+value. The \"partial-special\" command requests git-special files,\n+such as \".gitignore\". Using both requests the union of the two.\n+These requests are only valid if the server advertises the \"partial\"\n+capability.\n+\n Once all the 'want's and 'shallow's (and optional 'deepen') are\n transferred, clients MUST send a flush-pkt, to tell the server side\n that it is done sending the list.\ndiff --git a/Documentation/technical/protocol-capabilities.txt b/Documentation/technical/protocol-capabilities.txt\nindex 26dcc6f..9aa2123 100644\n--- a/Documentation/technical/protocol-capabilities.txt\n+++ b/Documentation/technical/protocol-capabilities.txt\n@@ -309,3 +309,10 @@ to accept a signed push certificate, and asks the to be\n included in the push certificate. A send-pack client MUST NOT\n send a push-cert packet unless the receive-pack server advertises\n this capability.\n+\n+partial\n+-------\n+\n+If the upload-pack server advertises this capability, fetch-pack\n+may send various \"partial-*\" commands to request a partial clone\n+or fetch where the server omits certain blobs from the packfile.\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 7597ba3..74f9dfa 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -63,6 +63,11 @@ static int advertise_refs;\n static int stateless_rpc;\n static const char *pack_objects_hook;\n \n+static struct strbuf partial_by_size = STRBUF_INIT;\n+static int client_requested_partial_capability;\n+static int have_partial_by_size;\n+static int have_partial_special;\n+\n static void reset_timeout(void)\n {\n \talarm(timeout);\n@@ -130,6 +135,10 @@ static void create_pack_file(void)\n \t\targv_array_push(&pack_objects.args, \"--delta-base-offset\");\n \tif (use_include_tag)\n \t\targv_array_push(&pack_objects.args, \"--include-tag\");\n+\tif (have_partial_by_size)\n+\t\targv_array_push(&pack_objects.args, partial_by_size.buf);\n+\tif (have_partial_special)\n+\t\targv_array_push(&pack_objects.args, \"--partial-special\");\n \n \tpack_objects.in = -1;\n \tpack_objects.out = -1;\n@@ -793,6 +802,23 @@ static void receive_needs(void)\n \t\t\tdeepen_rev_list = 1;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (skip_prefix(line, \"partial-by-size \", &arg)) {\n+\t\t\tunsigned long s;\n+\t\t\tif (!client_requested_partial_capability)\n+\t\t\t\tdie(\"git upload-pack: 'partial-by-size' option requires 'partial' capability\");\n+\t\t\tif (!git_parse_ulong(arg, &s))\n+\t\t\t\tdie(\"git upload-pack: invalid partial-by-size value: %s\", line);\n+\t\t\tstrbuf_addstr(&partial_by_size, \"--partial-by-size=\");\n+\t\t\tstrbuf_addstr(&partial_by_size, arg);\n+\t\t\thave_partial_by_size = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(line, \"partial-special\", &arg)) {\n+\t\t\tif (!client_requested_partial_capability)\n+\t\t\t\tdie(\"git upload-pack: 'partial-special' option requires 'partial' capability\");\n+\t\t\thave_partial_special = 1;\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!skip_prefix(line, \"want \", &arg) ||\n \t\t get_sha1_hex(arg, sha1_buf))\n \t\t\tdie(\"git upload-pack: protocol error, \"\n@@ -820,6 +846,8 @@ static void receive_needs(void)\n \t\t\tno_progress = 1;\n \t\tif (parse_feature_request(features, \"include-tag\"))\n \t\t\tuse_include_tag = 1;\n+\t\tif (parse_feature_request(features, \"partial\"))\n+\t\t\tclient_requested_partial_capability = 1;\n \n \t\to = parse_object(sha1_buf);\n \t\tif (!o)\n@@ -924,7 +952,9 @@ static int send_ref(const char *refname, const struct object_id *oid,\n {\n \tstatic const char *capabilities = \"multi_ack thin-pack side-band\"\n \t\t\" side-band-64k ofs-delta shallow deepen-since deepen-not\"\n-\t\t\" deepen-relative no-progress include-tag multi_ack_detailed\";\n+\t\t\" deepen-relative no-progress include-tag multi_ack_detailed\"\n+\t\t\" partial\"\n+\t\t;\n \tconst char *refname_nons = strip_namespace(refname);\n \tstruct object_id peeled;\n \n-- \n2.7.4\n\n"},{"id":"313674","messageId":"9ef37430-17be-dfc2-bca5-e4e33bc68df8@google.com","threadId":"45320","inReplyTo":"1488999039-37631-1-git-send-email-git@jeffhostetler.com","subject":"Re: [PATCH 00/10] RFC Partial Clone and Fetch","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-03-09T20:18:34Z","receivedAt":"2017-03-09T20:18:40Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Overall, this fetch/clone approach seems reasonable to me, except \nperhaps some unanswered questions (some of which are also being \ndiscussed elsewhere):\n - does the server need to tell us of missing blobs?\n - if yes, does the server need to tell us their file sizes?\n - do we need to store the list of missing blobs somewhere (whether the\n server told it to us or whether we inferred it from the fetched\n trees)\n\nThe answers to this probably depend on the answers in \"B. Issues \nBackfilling Omitted Blobs\" (especially the additional concepts I listed \nbelow).\n\nAlso, do you have any plans to implement other functionality, e.g. \"git \ncheckout\" (which will allow fetches and clones to repositories with a \nworking directory)? (I don't know what the mailing list consensus would \nbe for the \"acceptance criteria\" for this patch set, but I would at \nleast include \"checkout\".)\n\nOn 03/08/2017 10:50 AM, git@jeffhostetler.com wrote:\n> B. Issues Backfilling Omitted Blobs\n> ===================================\n>\n> Ideally, if the client only does \"--partial-by-profile\" fetches, it\n> should not need to fetch individual missing blobs, but we have to allow\n> for it to handle the other commands and other unexpected issues.\n>\n> There are 3 orthogonal concepts here: when, how and where?\n\nAnother concept is \"how to determine if a blob is really omitted\" - do \nwe store a list somewhere or do we assume that all missing blobs are \npurposely omitted (like in this patch set)?\n\nYet another concept is \"whether to fetch\" - for example, a checkout \nshould almost certainly fetch, but a rev-list used by a connectivity \ncheck (like in patch 6 of this set) should not.\n\nFor example, for historical-blob-searching commands like \"git log -S\", \nshould we:\n a) fetch everything missing (so users should use date-limiting\n arguments)\n b) fetch nothing missing\n c) use the file size to automatically exclude big files, but fetch\n everything else\n\nFor a) and b), we wouldn't need file size information for missing blobs, \nbut for c), we do. This might determine if we need file size information \nin the fetch-pack/upload-pack protocol.\n\n> C. New Blob-Fetch Protocol (2a)\n> ===============================\n>\n> *TODO* A new pair of commands, such as fetch-blob-pack and upload-blob-pack,\n> will be created to let the client request a batch of blobs and receive a\n> packfile. A protocol similar to the fetch-pack/upload-pack will be spoken\n> between them. (This avoids complicating the existing protocol and the work\n> of enumerating the refs.) Upload-blob-pack will use pack-objects to build\n> the packfile.\n>\n> It is also more efficient than requesting a single blob at a time using\n> the existing fetch-pack/upload-pack mechanism (with the various allow\n> unreachable options).\n>\n> *TODO* The new request protocol will be defined in the patch series.\n> It will include: a list of the desired blob SHAs. Possibly also the commit\n> SHA, branch name, and pathname of each blob (or whatever is necessary to let\n> the server address the reachability concerns). Possibly also the last\n> known SHA for each blob to allow for deltafication in the packfile.\n\nContext (like the commit SHA-1) would help in reachability checks, but \nI'm not sure if we can rely on that. It is true that I can't think of a \nway that the client would dissociate a blob that is missing from its \ntree or commit (because it would first need to \"fault-in\" that blob to \ndo its operation). But clients operating on non-contextual SHA-1s (e.g. \n\"git cat-file\") and servers manipulating commits (so that the commit \nSHA-1 that the client had in its context is no longer reachable) are not \nuncommon, I think.\n\nHaving said that, it might be useful to include the context in the \nprotocol anyway as an optional \"hint\".\n\nI'm not sure what you mean by \"last known SHA for each blob\".\n\n(If we do store the file size of a blob somewhere, we could also store \nsome context there. I'm not sure how useful this is, though.)\n\n> E. Unresolved Thoughts\n> ======================\n\n\n\n> *TODO* The partial clone arguments should be recorded in \".git/info/\"\n> so that subsequent fetch commands can inherit them and rev-list/index-pack\n> know to not complain by default.\n>\n> *TODO* Update GC like rev-list to not complain when there are missing blobs.\n\nThese 2 points would be part of \"whether to fetch\" above.\n"},{"id":"314260","messageId":"7ce6e14f-b2b1-19f3-916a-9ed1307a3403@jeffhostetler.com","threadId":"45320","inReplyTo":"9ef37430-17be-dfc2-bca5-e4e33bc68df8@google.com","subject":"Re: [PATCH 00/10] RFC Partial Clone and Fetch","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-03-16T21:43:35Z","receivedAt":"2017-03-16T21:44:04Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/9/2017 3:18 PM, Jonathan Tan wrote:\n> Overall, this fetch/clone approach seems reasonable to me, except\n> perhaps some unanswered questions (some of which are also being\n> discussed elsewhere):\n> - does the server need to tell us of missing blobs?\n> - if yes, does the server need to tell us their file sizes?\n\nFile sizes are a nice addition. For example, with a virtual\nfile system, a \"ls -l\" can lie and tell you the sizes of the\nyet-to-be-populated files. Or if the client wants to distinguish\nbetween going back to the original remote or going to S3 for the\nblob, it could use the size to choose. (I'm not saying we actually\nbuild that yet, but others on the mailing list have spoken about\nparking large blobs in S3.)\n\nSo, not necessary, but might be nice to have.\n\n> - do we need to store the list of missing blobs somewhere (whether the\n> server told it to us or whether we inferred it from the fetched\n> trees)\n\nWe should be able to infer the list of missing blobs; I hadn't\nconsidered that. However, by doing so we will need to disable\nsome of the integrity checking (as I had to do with the\n\"--allow-partial\" option) and some concerns about that were\ndiscussed earlier in the thread. But if we do that inference\nduring clone/fetch and record it somewhere, we could get back\nthe integrity checking.\n\n>\n> The answers to this probably depend on the answers in \"B. Issues\n> Backfilling Omitted Blobs\" (especially the additional concepts I listed\n> below).\n>\n> Also, do you have any plans to implement other functionality, e.g. \"git\n> checkout\" (which will allow fetches and clones to repositories with a\n> working directory)? (I don't know what the mailing list consensus would\n> be for the \"acceptance criteria\" for this patch set, but I would at\n> least include \"checkout\".)\n\nYes, supporting \"checkout\" is essential. Commands like \"merge\", \"diff\",\nand etc. will come later. In Ben's RFC, he has been investigating\ndemand-loading blobs in read_object(). I've been focusing on\npre-fetching the missing blobs for a particular command. I need\nto make more progress on this topic.\n\n>\n> On 03/08/2017 10:50 AM, git@jeffhostetler.com wrote:\n>> B. Issues Backfilling Omitted Blobs\n>> ===================================\n>>\n>> Ideally, if the client only does \"--partial-by-profile\" fetches, it\n>> should not need to fetch individual missing blobs, but we have to allow\n>> for it to handle the other commands and other unexpected issues.\n>>\n>> There are 3 orthogonal concepts here: when, how and where?\n>\n> Another concept is \"how to determine if a blob is really omitted\" - do\n> we store a list somewhere or do we assume that all missing blobs are\n> purposely omitted (like in this patch set)?\n>\n> Yet another concept is \"whether to fetch\" - for example, a checkout\n> should almost certainly fetch, but a rev-list used by a connectivity\n> check (like in patch 6 of this set) should not.\n>\n> For example, for historical-blob-searching commands like \"git log -S\",\n> should we:\n> a) fetch everything missing (so users should use date-limiting\n> arguments)\n> b) fetch nothing missing\n> c) use the file size to automatically exclude big files, but fetch\n> everything else\n>\n> For a) and b), we wouldn't need file size information for missing blobs,\n> but for c), we do. This might determine if we need file size information\n> in the fetch-pack/upload-pack protocol.\n\ngood points.\n\n>\n>> C. New Blob-Fetch Protocol (2a)\n>> ===============================\n>>\n>> *TODO* A new pair of commands, such as fetch-blob-pack and\n>> upload-blob-pack,\n>> will be created to let the client request a batch of blobs and receive a\n>> packfile. A protocol similar to the fetch-pack/upload-pack will be\n>> spoken\n>> between them. (This avoids complicating the existing protocol and the\n>> work\n>> of enumerating the refs.) Upload-blob-pack will use pack-objects to\n>> build\n>> the packfile.\n>>\n>> It is also more efficient than requesting a single blob at a time using\n>> the existing fetch-pack/upload-pack mechanism (with the various allow\n>> unreachable options).\n>>\n>> *TODO* The new request protocol will be defined in the patch series.\n>> It will include: a list of the desired blob SHAs. Possibly also the\n>> commit\n>> SHA, branch name, and pathname of each blob (or whatever is necessary\n>> to let\n>> the server address the reachability concerns). Possibly also the last\n>> known SHA for each blob to allow for deltafication in the packfile.\n>\n> Context (like the commit SHA-1) would help in reachability checks, but\n> I'm not sure if we can rely on that. It is true that I can't think of a\n> way that the client would dissociate a blob that is missing from its\n> tree or commit (because it would first need to \"fault-in\" that blob to\n> do its operation). But clients operating on non-contextual SHA-1s (e.g.\n> \"git cat-file\") and servers manipulating commits (so that the commit\n> SHA-1 that the client had in its context is no longer reachable) are not\n> uncommon, I think.\n>\n> Having said that, it might be useful to include the context in the\n> protocol anyway as an optional \"hint\".\n\nThat is what I was thinking. A hint of the branch or commit the\nclient is referencing (wants to checkout, for example). Frankly,\nI'm not sure how secure some of the proposed/existing reachability\nconstraints are, so I'm hedging here a bit.\n\n>\n> I'm not sure what you mean by \"last known SHA for each blob\".\n\nWhat I meant was. If I'm missing the blob for a particular file\nin the commit I want to checkout, but I have the blob for the\nsame file in a recent ancestor commit, I could send the SHA-1 of\nthe file in the parent commit. The server would be free to deltafy\nthe missing blob relative to it, rather than sending the entire\nblob.\n\n>\n> (If we do store the file size of a blob somewhere, we could also store\n> some context there. I'm not sure how useful this is, though.)\n>\n>> E. Unresolved Thoughts\n>> ======================\n>\n> \n>\n>> *TODO* The partial clone arguments should be recorded in \".git/info/\"\n>> so that subsequent fetch commands can inherit them and\n>> rev-list/index-pack\n>> know to not complain by default.\n>>\n>> *TODO* Update GC like rev-list to not complain when there are missing\n>> blobs.\n>\n> These 2 points would be part of \"whether to fetch\" above.\n\nThanks for all of the comments.\nJeff\n"},{"id":"314335","messageId":"d8740193-9f12-84f1-1d9f-cebd6125640b@jeffhostetler.com","threadId":"45320","inReplyTo":"7ce6e14f-b2b1-19f3-916a-9ed1307a3403@jeffhostetler.com","subject":"Re: [PATCH 00/10] RFC Partial Clone and Fetch","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-03-17T14:13:56Z","receivedAt":"2017-03-17T14:16:00Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/16/2017 5:43 PM, Jeff Hostetler wrote:\n>\n>\n> On 3/9/2017 3:18 PM, Jonathan Tan wrote:\n>> Overall, this fetch/clone approach seems reasonable to me, except\n>> perhaps some unanswered questions (some of which are also being\n>> discussed elsewhere):\n>> - does the server need to tell us of missing blobs?\n>> - if yes, does the server need to tell us their file sizes?\n>\n> File sizes are a nice addition. For example, with a virtual\n> file system, a \"ls -l\" can lie and tell you the sizes of the\n> yet-to-be-populated files.\n\nNevermind the \"ls -l\" case, I forgot about the need for the\nclient to display the size of the (possibly) smudged file,\nrather than the actual blob size.\n"},{"id":"314908","messageId":"CA+dhYEWo3v+ns0zt_hWu-7i-=E0g_tFaXYcv7Q0j2ozx1SCVmQ@mail.gmail.com","threadId":"45320","inReplyTo":"1488999039-37631-1-git-send-email-git@jeffhostetler.com","subject":"Re: [PATCH 00/10] RFC Partial Clone and Fetch","fromName":"ankostis","fromEmail":"ankostis@gmail.com","sentAt":"2017-03-22T15:16:16Z","receivedAt":"2017-03-22T15:17:03Z","isPatch":true,"sender":{"key":"ankostis@gmail.com","avatar":"https://gravatar.com/avatar/1f3597ab8ad44cbb0fd782a2773e3aba94a52c03f9f6bbe5ccb7f1fe2581b12b?d=mp&s=160"},"body":"Dear Jeff\n\nI read most of the valuable references you provided\nbut could not find something along the lines describing inline.\n\n\nOn 8 March 2017 at 19:50, wrote:\n> From: Jeff Hostetler \n>\n>\n> [RFC] Partial Clone and Fetch\n> =============================\n>\n> This is a WIP RFC for a partial clone and fetch feature wherein the client\n> can request that the server omit various blobs from the packfile during\n> clone and fetch. Clients can later request omitted blobs (either from a\n> modified upload-pack-like request to the server or via a completely\n> independent mechanism).\n\nIs it foreseen the server to *decide* with partial objects to serve\nAnd the cloning-client still to work ok?\n\nMy case in mind is storing confidential files in Git (server)\nthat I want to publicize them to partial-cloning clients,\nfor non-repudiation, by sending out trees and commits alone\n(or any non-sensitive blobs).\n\nA possible UI would be to rely on a `.gitattributes` to specify\nwhich objects are to be upheld.\n\n\nApologies if I'm intruding with an unrelated feature requests.\n Kostis\n"},{"id":"314912","messageId":"alpine.DEB.2.20.1703221711180.3767@virtualbox","threadId":"45320","inReplyTo":"CA+dhYEWo3v+ns0zt_hWu-7i-=E0g_tFaXYcv7Q0j2ozx1SCVmQ@mail.gmail.com","subject":"Re: [PATCH 00/10] RFC Partial Clone and Fetch","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-03-22T16:21:11Z","receivedAt":"2017-03-22T16:22:43Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Kostis,\n\nOn Wed, 22 Mar 2017, ankostis wrote:\n\n> On 8 March 2017 at 19:50, wrote:\n> > From: Jeff Hostetler \n> >\n> > [RFC] Partial Clone and Fetch\n> > =============================\n> >\n> > This is a WIP RFC for a partial clone and fetch feature wherein the\n> > client can request that the server omit various blobs from the\n> > packfile during clone and fetch. Clients can later request omitted\n> > blobs (either from a modified upload-pack-like request to the server\n> > or via a completely independent mechanism).\n> \n> Is it foreseen the server to *decide* with partial objects to serve\n> And the cloning-client still to work ok?\n\nThe foreseeable use case will be to speed up clones of insanely large\nrepositories by omitting blobs that are not immediately required, and let\nthe client fetch them later on demand.\n\nThat is all, no additional permission model or anything. In fact, we do\nnot even need to ensure that blobs are reachable in our use case, as only\ntrusted parties are allowed to access the server to begin with.\n\nThat does not mean, of course, that there should not be an option to limit\naccess to objects that are reachable.\n\n> My case in mind is storing confidential files in Git (server)\n> that I want to publicize them to partial-cloning clients,\n> for non-repudiation, by sending out trees and commits alone\n> (or any non-sensitive blobs).\n> \n> A possible UI would be to rely on a `.gitattributes` to specify\n> which objects are to be upheld.\n> \n> \n> Apologies if I'm intruding with an unrelated feature requests.\n\nI think this is a valid use case, and Jeff's design certainly does not\nprevent future patches to that end.\n\nHowever, given that Jeff's use case does not require any such feature, I\nwould expect the people who want those features to do the heavy lifting on\ntop of his work. It is too different from the intended use case to\nreasonably ask of Jeff.\n\nCiao,\nJohannes\n"},{"id":"314931","messageId":"024c6643-89d7-ea89-3190-003223bb3227@jeffhostetler.com","threadId":"45320","inReplyTo":"alpine.DEB.2.20.1703221711180.3767@virtualbox","subject":"Re: [PATCH 00/10] RFC Partial Clone and Fetch","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-03-22T17:51:16Z","receivedAt":"2017-03-22T17:52:22Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/22/2017 12:21 PM, Johannes Schindelin wrote:\n> Hi Kostis,\n>\n> On Wed, 22 Mar 2017, ankostis wrote:\n>\n>> On 8 March 2017 at 19:50, wrote:\n>>> From: Jeff Hostetler \n>>>\n>>> [RFC] Partial Clone and Fetch\n>>> =============================\n>>>\n>>> This is a WIP RFC for a partial clone and fetch feature wherein the\n>>> client can request that the server omit various blobs from the\n>>> packfile during clone and fetch. Clients can later request omitted\n>>> blobs (either from a modified upload-pack-like request to the server\n>>> or via a completely independent mechanism).\n>>\n>> Is it foreseen the server to *decide* with partial objects to serve\n>> And the cloning-client still to work ok?\n>\n> The foreseeable use case will be to speed up clones of insanely large\n> repositories by omitting blobs that are not immediately required, and let\n> the client fetch them later on demand.\n>\n> That is all, no additional permission model or anything. In fact, we do\n> not even need to ensure that blobs are reachable in our use case, as only\n> trusted parties are allowed to access the server to begin with.\n>\n> That does not mean, of course, that there should not be an option to limit\n> access to objects that are reachable.\n>\n>> My case in mind is storing confidential files in Git (server)\n>> that I want to publicize them to partial-cloning clients,\n>> for non-repudiation, by sending out trees and commits alone\n>> (or any non-sensitive blobs).\n>>\n>> A possible UI would be to rely on a `.gitattributes` to specify\n>> which objects are to be upheld.\n>>\n>>\n>> Apologies if I'm intruding with an unrelated feature requests.\n>\n> I think this is a valid use case, and Jeff's design certainly does not\n> prevent future patches to that end.\n>\n> However, given that Jeff's use case does not require any such feature, I\n> would expect the people who want those features to do the heavy lifting on\n> top of his work. It is too different from the intended use case to\n> reasonably ask of Jeff.\n\nAs Johannes said, all I'm proposing is a way to limit the amount of\ndata the client receives to help git scale to extremely large\nrepositories. For example, I probably don't need 20 years of history\nor the entire source tree if I'm only working in a narrow subset of\nthe tree.\n\nI'm not sure how you would achieve the confidential file scenario\nthat you describe, but you might try to build on it and see if you\ncan make it work.\n\nJeff\n\n\n\n"},{"id":"318669","messageId":"777ab8f2-c31a-d07b-ffe3-f8333f408ea1@jeffhostetler.com","threadId":"45320","inReplyTo":"1488999039-37631-1-git-send-email-git@jeffhostetler.com","subject":"Re: [PATCH 00/10] RFC Partial Clone and Fetch","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-05-03T16:38:33Z","receivedAt":"2017-05-03T16:38:40Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/8/2017 1:50 PM, git@jeffhostetler.com wrote:\n> From: Jeff Hostetler \n>\n>\n> [RFC] Partial Clone and Fetch\n> =============================\n> [...]\n> E. Unresolved Thoughts\n> ======================\n>\n> *TODO* The server should optionally return (in a side-band?) a list\n> of the blobs that it omitted from the packfile (and possibly the sizes\n> or sha1_object_info() data for them) during the fetch-pack/upload-pack\n> operation. This would allow the client to distinguish from invalid\n> SHAs and missing ones. Size information would allow the client to\n> maybe choose between various servers.\n\nSince I first posted this, Jonathan Tan has started a related\ndiscussion on missing blob support.\nhttps://public-inbox.org/git/CAGf8dgK05+f4uX-8+iMFvQd0n2JP6YxJ18ag8uDaEH6qc6SgVQ@mail.gmail.com/T/\n\nI want to respond to both of these threads here.\n-------------------------------------------------\n\nMissing-Blob Support\n====================\n\nLet me offer up an alternative idea for representing\nmissing blobs. This is differs from both of our previous\nproposals. (I don't have any code for this new proposal,\nI just want to think out loud a bit and see if this is a\ndirection worth pursuing -- or a complete non-starter.)\n\nBoth proposals talk about detecting and adapting to a missing\nblob and ways to recover -- when we fail to find a blob.\nComments on the thread asked about:\n() being able to detect missing blobs vs corrupt repos\n() being unable to detect duplicate blobs\n() expense of blob search.\n\nSuppose we store \"positive\" information about missing blobs?\nThis would let us know that a blob is intentionally missing\nand possibly some meta-data about it.\n\n\n1. Suppose we update the .pack file format slightly.\n () We use the 5 value in \"enum object_type\" to mean a\n \"missing-blob\".\n () We update git-pack-object as I did in my RFC, but have it\n create type 5 entries for the blobs that are omitted,\n rather than nothing.\n () Hopefully, the same logic that currently keeps pack-object\n from sending unnecessary blobs on subsequent fetches can\n also be used to keep it from sending unnecessary missing-blob\n entries.\n () The type 5 missing-blob entry would contain the SHA-1 of the\n blob and some meta-data to be explained later.\n\n2. Make a similar change in the .idx format and git-index-pack\n to include them there. Then blob lookup operations could\n definitively determine that a blob exists and is just not\n present locally.\n\n3. With this, packfile-based blob-lookup operations can get a\n \"missing-blob\" result.\n () It should be possible to short-cut searching in other\n packfiles (because we don't have to assume that the blob\n was just misplaced in another packfile).\n () Lookup can still look for the corresponding loose blob\n (in case a previous lookup already \"faulted it in\").\n\n4. We can then think about dynamically fetching it.\n () Several techniques for this are currently being\n discussed on the mailing list in other threads,\n so I won't go into this here.\n () There has also been debate about whether this should\n yield a loose blob or a new packfile. I think both\n forms have merit and depend on whether we are limited\n to asking for a single blob or can make a batch request.\n () A dynamically-fetched loose blob is placed in the normal\n loose blob directory hierarchy so that subsequent\n lookups can find it as mentioned above.\n () A dynamically-fetched packfile (with one or more blobs)\n is written to the ODB and then the lookup operation\n completes.\n {} I want to isolate these packfiles from the main\n packfiles, so that they behave like a second-stage\n lookup and don't affect the caching/LRU nature of\n the existing first-stage packfile lookup.\n {} I also don't want the ambiguity of having 2 primary\n packfiles with a blob marked as missing in 1 and\n present in the other.\n\n5. git-repack should be updated to \"do the right thing\" and\n squash missing-blob entries.\n\n6. And etc.\n\n\nMissing-Blob Entry Data\n=======================\n\nA missing-blob entry needs to contain the SHA-1 value of\nthe blob (obviously). Other fields are nice to have, but\nare not necessary. Here are a few fields to consider.\n\nA. The SHA-1 (20 bytes)\n\nB. The raw size of the blob (5? bytes).\n () This is the cleaned size of the file as stored. The\n server does not (and should not) have any knowledge\n of the smudging that may happen.\n () This may be useful if whatever dynamic-fetch-hook\n wants to customize its behavior, such as individually\n fetching large blobs and batch fetching smaller ones\n from the same server.\n () GVFS found it necessary to create a custom server\n end-point to get blob size data so that \"ls -l\"\n could show file sizes for non-present virtualized\n files.\n () 5 bytes (uint:40) should be more than enough for this.\n\nC. A server \"hint\" (20 bytes)\n () Instructions to help the client fetch the blob.\n () If I have multiple remotes configured, a missing-blob\n should be fetched from the same server that created\n the missing-blob entry (since it may be the only\n one that has it).\n () If a blob is very large (and was omitted for this\n reason), the server may want to redirect the client\n to a geographically closer CDN.\n () This is the SHA-1 of a file in the repository of a\n hook (or a set of parameters to be used by a hook).\n {} This is a bit of *hand-wave* right now, but the\n idea is that you can use the information here to\n individually fetch a blob or batch fetch a set\n of blobs that have the same hint.\n {} Yes, there are security concerns here, so perhaps\n the hint file should just contain parameters for\n a stock git-fetch-pack or git-fetch-blob-pack or\n curl command (or wrapper script) that \"does the\n right thing\".\n {} I thought this would be more compact than listing\n detailed fetch data per-blob. And we don't have\n to define yet another syntax. For example, we can\n let the SHA-1 point to an administrator configured\n shell script and be done.\n () We assume that the SHA-1 file is present locally\n (not missing). This might refer to a pinned file\n in a special \".git*\" file (that we never omit) in\n HEAD. Or it might be in a branch that all clients\n are assumed to have.\n\n\nConcluding Thoughts\n===================\n\nCombining the ideas here with the partial clone/fetch\nparameters and the various blob back-filling proposals\ngives us the ability to create and work with sparse\nrepos.\n() Filtering can be based upon blob size; this could be\n seen as an alternative solution to LFS for repos with\n large objects.\n() Filtering could also be based upon pathnames (such as\n a sparse-checkout filter) and greatly help performance\n on very large repos where developers only work with\n small areas of the tree.\n\nThanks\nJeff\n\n\n\n\n\n\n\n"},{"id":"318685","messageId":"20170503182725.GC28740@aiede.svl.corp.google.com","threadId":"45320","inReplyTo":"777ab8f2-c31a-d07b-ffe3-f8333f408ea1@jeffhostetler.com","subject":"Re: [PATCH 00/10] RFC Partial Clone and Fetch","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-05-03T18:27:26Z","receivedAt":"2017-05-03T18:27:34Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nJeff Hostetler wrote:\n> On 3/8/2017 1:50 PM, git@jeffhostetler.com wrote:\n\n>> [RFC] Partial Clone and Fetch\n>> =============================\n>> [...]\n>> E. Unresolved Thoughts\n>> ======================\n>>\n>> *TODO* The server should optionally return (in a side-band?) a list\n>> of the blobs that it omitted from the packfile (and possibly the sizes\n>> or sha1_object_info() data for them) during the fetch-pack/upload-pack\n>> operation. This would allow the client to distinguish from invalid\n>> SHAs and missing ones. Size information would allow the client to\n>> maybe choose between various servers.\n>\n> Since I first posted this, Jonathan Tan has started a related\n> discussion on missing blob support.\n> https://public-inbox.org/git/CAGf8dgK05+f4uX-8+iMFvQd0n2JP6YxJ18ag8uDaEH6qc6SgVQ@mail.gmail.com/T/\n>\n> I want to respond to both of these threads here.\n\nThanks much for this.\n\n> Missing-Blob Support\n> ====================\n>\n> Let me offer up an alternative idea for representing\n> missing blobs. This is differs from both of our previous\n> proposals. (I don't have any code for this new proposal,\n> I just want to think out loud a bit and see if this is a\n> direction worth pursuing -- or a complete non-starter.)\n>\n> Both proposals talk about detecting and adapting to a missing\n> blob and ways to recover -- when we fail to find a blob.\n> Comments on the thread asked about:\n> () being able to detect missing blobs vs corrupt repos\n> () being unable to detect duplicate blobs\n> () expense of blob search.\n>\n> Suppose we store \"positive\" information about missing blobs?\n> This would let us know that a blob is intentionally missing\n> and possibly some meta-data about it.\n\nWe've discussed this a little informally but didn't go more into\nit, so I'm glad you brought it up.\n\nThere are two use cases I care about. I'll want names to refer to\nthem later, so describing them now:\n\n A. A p4 or svn style \"monorepo\" containing all code within a company.\n We want to make git scale well when working with such a\n repository. Disk usage, network usage, index size, and object\n lookup time are all issues for such a repository.\n\n A repository can creep up in size so it starts falling into this\n category even though git coped well with it before. Another way\n to end up in this category is a conversion from a version control\n system like p4 or svn.\n\n B. A more modestly sized repository with some large blobs in it. At\n clone time we want to omit unneeded large blobs to speed up the\n clone, save disk space, and save bandwidth.\n\n For this kind of repository, the idx file already contained all\n those blobs and that was not causing problems --- the only problem\n was the actual blob size.\n\n> 1. Suppose we update the .pack file format slightly.\n[...]\n> 2. Make a similar change in the .idx format and git-index-pack\n> to include them there. Then blob lookup operations could\n> definitively determine that a blob exists and is just not\n> present locally.\n\nSome nits:\n\n- there shouldn't be any need for the blobs to even be mentioned in\n the .pack stored locally. The .idx file maps from sha1 to offset\n within the packfile --- a special offset could mean \"this is a\n missing blob\".\n\n- Git protocol works by sending pack files over the wire. The idx\n file is not transmitted to the client --- the client instead\n reconstructs it from the pack file. I assume this is why you\n stored the SHA-1 of the object in the pack file, but it could\n equally well be sent through another stream (since this proposal\n involves a change to git protocol anyway).\n\n- However, the list of missing blobs can be inferred from the existing\n pack format, without a change to the pack format used in git\n protocol. As part of constructing the idx, \"git index-pack\"\n inflates every object in the pack file sent by the server. This\n means we know what blobs they reference, so we can easily produce a\n list for the idx file without changing the pack file format.\n\nIf this is done by only changing the idx file format and not the pack\nfile, then it does not affect the protocol. That is good for\nexperimentation --- it lets us try out different formats client-side\nwithout having to coordinate with server authors.\n\nIn case (A), this proposal means we get back some of the per-object\noverhead that we were trying to avoid. I would like to avoid that\nif possible.\n\nIn case (B), this proposal doesn't hurt.\n\nOne problem with proposals so far has been how to handle repositories\nwith multiple remotes. Having a local list of missing blobs is\nconvenient because it can be associated to the remote --- when a blob\nis referenced later, we know which remote to ask for for it. This is\na niche feature, but it's a nice bonus.\n\n[...]\n> 3. With this, packfile-based blob-lookup operations can get a\n> \"missing-blob\" result.\n> () It should be possible to short-cut searching in other\n> packfiles (because we don't have to assume that the blob\n> was just misplaced in another packfile).\n> () Lookup can still look for the corresponding loose blob\n> (in case a previous lookup already \"faulted it in\").\n\nThe number of packfiles to search is always a problem for object\nlookups --- we need to be able to keep the number of packfiles low.\nIf we solve that for missing blobs but not for the ones that are\npresent, then git's object access performance is still slow.\n\nI have some ideas for improving git's gc behavior to bound the number\nof packfiles better. I think that's mostly orthogonal to this\nproposal. The main point of overlap is the number of packfiles\nproduced if someone uses one packfile per large blob, but that is bad\nfor the performance in successful lookups, not just negative lookups,\nso it needs to be solved directly anyway.\n\n> 4. We can then think about dynamically fetching it.\n\nThis also seems orthogonal to whether the missing blobs list exists.\n\n[...]\n> A missing-blob entry needs to contain the SHA-1 value of\n> the blob (obviously). Other fields are nice to have, but\n> are not necessary. Here are a few fields to consider.\n[...]\n> B. The raw size of the blob (5? bytes).\n\nThis would be a very nice thing.\n\nIf we were starting over, would this belong in the tree object?\n(Part of the reason I ask is that we have an opportunity to sort\nof start over as part of the transition away from SHA-1.)\n\n[...]\n> C. A server \"hint\" (20 bytes)\n> () Instructions to help the client fetch the blob.\n> () If I have multiple remotes configured, a missing-blob\n> should be fetched from the same server that created\n> the missing-blob entry (since it may be the only\n> one that has it).\n\nI am worried about the implications of storing this kind of policy\ninformation in the pack file. There may be useful information along\nthese lines for a server to advertise, but I don't think it belongs in\nthe pack file. Git protocol relies on it being cheap to stream pack\nfiles as is.\n\n[...]\n> Combining the ideas here with the partial clone/fetch\n> parameters and the various blob back-filling proposals\n> gives us the ability to create and work with sparse\n> repos.\n> () Filtering can be based upon blob size; this could be\n> seen as an alternative solution to LFS for repos with\n> large objects.\n> () Filtering could also be based upon pathnames (such as\n> a sparse-checkout filter) and greatly help performance\n> on very large repos where developers only work with\n> small areas of the tree.\n\nTo summarize my thoughts:\n\n+ It is possible to construct a list of blobs missing from the\n packfile sent by the server, without changing the protocol.\n This is a very good thing, since it makes local experimentation\n possible.\n\n- Except in the multiple-remotes case, I am mostly unconvinced about\n the benefit of having the list of missing blobs around locally.\n For example:\n\n - \"git fsck\" can use the list of missing blobs to see whether a\n reachable missing blob is intentional or a violation of the\n repository's integrity. I don't think this is useful. In a\n partial clone repository, missing blobs are always permitted\n and do not indicate repository corruption, since git operations\n have a way of coping with them (by fetching them on demand using\n read-blob protocol).\n - In a partial clone repository that is acting as a server, the\n list of missing blobs can be used to deny a read-blob request\n without passing the request on to \"origin\". Without this\n check, such checks could be a DoS vector. Since there are other,\n simpler requests that have the potential to DoS \"origin\" (e.g.\n using plain clone --mirror) in such a setup, I think it best to\n disallow read-blob requests to partial clone repositories\n altogether for now.\n\n- I am also unconvinced about the server hint.\n\n+ Having the list of missing blob sizes locally seems very useful. I\n can imagine a client that wants this information for some blobs but\n not all --- e.g. they are only interested in a particular\n subdirectory so they want blob sizes from that directory.\n\nThe blob-size use case is compelling, but I still lean toward\nconstructing the list of blobs locally for now where it's useful,\ninstead of adding a missing-blob entry to the pack file.\n\nI'll think more about blob sizes. I agree that we need to do\nsomething to handle them.\n\nThanks and hope that helps,\nJonathan\n"},{"id":"318699","messageId":"2b1b504a-07c9-81b3-fed2-e9c029a5b284@google.com","threadId":"45320","inReplyTo":"777ab8f2-c31a-d07b-ffe3-f8333f408ea1@jeffhostetler.com","subject":"Re: [PATCH 00/10] RFC Partial Clone and Fetch","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-05-03T20:40:08Z","receivedAt":"2017-05-03T20:40:26Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On 05/03/2017 09:38 AM, Jeff Hostetler wrote:\n>\n>\n> On 3/8/2017 1:50 PM, git@jeffhostetler.com wrote:\n>> From: Jeff Hostetler \n>>\n>>\n>> [RFC] Partial Clone and Fetch\n>> =============================\n>> [...]\n>> E. Unresolved Thoughts\n>> ======================\n>>\n>> *TODO* The server should optionally return (in a side-band?) a list\n>> of the blobs that it omitted from the packfile (and possibly the sizes\n>> or sha1_object_info() data for them) during the fetch-pack/upload-pack\n>> operation. This would allow the client to distinguish from invalid\n>> SHAs and missing ones. Size information would allow the client to\n>> maybe choose between various servers.\n>\n> Since I first posted this, Jonathan Tan has started a related\n> discussion on missing blob support.\n> https://public-inbox.org/git/CAGf8dgK05+f4uX-8+iMFvQd0n2JP6YxJ18ag8uDaEH6qc6SgVQ@mail.gmail.com/T/\n>\n>\n> I want to respond to both of these threads here.\n> -------------------------------------------------\n\nThanks for your input. I see that you have explained both \"storing \n'positive' information about missing blobs\" and \"what to store with \nthose positive information\"; I'll just comment on the former for now.\n\n> Missing-Blob Support\n> ====================\n>\n> Let me offer up an alternative idea for representing\n> missing blobs. This is differs from both of our previous\n> proposals. (I don't have any code for this new proposal,\n> I just want to think out loud a bit and see if this is a\n> direction worth pursuing -- or a complete non-starter.)\n>\n> Both proposals talk about detecting and adapting to a missing\n> blob and ways to recover -- when we fail to find a blob.\n> Comments on the thread asked about:\n> () being able to detect missing blobs vs corrupt repos\n> () being unable to detect duplicate blobs\n> () expense of blob search.\n>\n> Suppose we store \"positive\" information about missing blobs?\n> This would let us know that a blob is intentionally missing\n> and possibly some meta-data about it.\n\nI thought about this (see \"Some alternative designs\" in [1]), listing \nsome similar benefits, but concluded that \"it is difficult to scale to \nlarge repos\".\n\nFirstly, to be clear, by large repos I meant (and mean) the svn-style \n\"monorepos\" that Jonathan Nieder mentions as use case \"A\" [2].\n\nMy concern is that such lists (whether in separate file(s) or in .idx \nfiles) would be too unwieldy to manipulate. Even if we design things to \navoid modifying such lists (for example, by adding a new list whenever \nwe fetch instead of trying to modify an existing one), we would at least \nneed to sort their contents (for example, when generating an .idx in the \nfirst place). For a repo with 10M-100M blobs [3], this might be doable \non today's computers, but I would be concerned if a repo would exceed \nsuch numbers.\n\n[1] <20170426221346.25337-1-jonathantanmy@google.com>\n[2] <20170503182725.GC28740@aiede.svl.corp.google.com>\n[3] In Microsoft's announcement of Git Virtual File System [4], they \nmentioned \"over 3.5 million files\" in the Windows codebase. I'm not sure \nif this refers to files in a snapshot (that is, working copy) or all \nhistorical versions.\n[4] \nhttps://blogs.msdn.microsoft.com/visualstudioalm/2017/02/03/announcing-gvfs-git-virtual-file-system/\n\n> 1. Suppose we update the .pack file format slightly.\n> () We use the 5 value in \"enum object_type\" to mean a\n> \"missing-blob\".\n> () We update git-pack-object as I did in my RFC, but have it\n> create type 5 entries for the blobs that are omitted,\n> rather than nothing.\n> () Hopefully, the same logic that currently keeps pack-object\n> from sending unnecessary blobs on subsequent fetches can\n> also be used to keep it from sending unnecessary missing-blob\n> entries.\n> () The type 5 missing-blob entry would contain the SHA-1 of the\n> blob and some meta-data to be explained later.\n\nMy original idea was to have sorted list(s) of hashes in separate \nfile(s) much like the currently existing shallow file; it would have the \nsemantics of \"a hash here might be present or absent; if it is absent, \nuse the hook\". (Initially I thought that one list would be sufficient, \nbut after reading your idea and considering it some more, multiple lists \nmight be better.)\n\nYour idea of storing them in an .idx (and possibly corresponding .pack \nfile) is similar, I think. Although mine is probably simpler - at least, \nwe wouldn't need a new object_type.\n\nAs described above, I don't think this list-of-hashes idea will work \n(because of the large numbers of blobs involved), but I'll compare it to \nyours anyway just in case we end up being convinced that this general \nidea works.\n\n> 2. Make a similar change in the .idx format and git-index-pack\n> to include them there. Then blob lookup operations could\n> definitively determine that a blob exists and is just not\n> present locally.\n>\n> 3. With this, packfile-based blob-lookup operations can get a\n> \"missing-blob\" result.\n> () It should be possible to short-cut searching in other\n> packfiles (because we don't have to assume that the blob\n> was just misplaced in another packfile).\n> () Lookup can still look for the corresponding loose blob\n> (in case a previous lookup already \"faulted it in\").\n\nThe binary search to lookup a packfile offset from a .idx file (which \ninvolves disk reads) would take longer for all lookups (not just lookups \nfor missing blobs) - I think I prefer keeping the lists separate, to \navoid pessimizing the (likely) usual case where the relevant blobs are \nall already in local repo storage.\n\n> 4. We can then think about dynamically fetching it.\n> () Several techniques for this are currently being\n> discussed on the mailing list in other threads,\n> so I won't go into this here.\n> () There has also been debate about whether this should\n> yield a loose blob or a new packfile. I think both\n> forms have merit and depend on whether we are limited\n> to asking for a single blob or can make a batch request.\n> () A dynamically-fetched loose blob is placed in the normal\n> loose blob directory hierarchy so that subsequent\n> lookups can find it as mentioned above.\n> () A dynamically-fetched packfile (with one or more blobs)\n> is written to the ODB and then the lookup operation\n> completes.\n> {} I want to isolate these packfiles from the main\n> packfiles, so that they behave like a second-stage\n> lookup and don't affect the caching/LRU nature of\n> the existing first-stage packfile lookup.\n> {} I also don't want the ambiguity of having 2 primary\n> packfiles with a blob marked as missing in 1 and\n> present in the other.\n\nWith my idea, the second-stage lookup is done on the list of missing \nhashes; there is no division between packfiles.\n\n> 5. git-repack should be updated to \"do the right thing\" and\n> squash missing-blob entries.\n>\n> 6. And etc.\n"},{"id":"318704","messageId":"20170503210850.GG28740@aiede.svl.corp.google.com","threadId":"45320","inReplyTo":"2b1b504a-07c9-81b3-fed2-e9c029a5b284@google.com","subject":"Re: [PATCH 00/10] RFC Partial Clone and Fetch","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-05-03T21:08:50Z","receivedAt":"2017-05-03T21:08:58Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nJonathan Tan wrote:\n\n> The binary search to lookup a packfile offset from a .idx file\n> (which involves disk reads) would take longer for all lookups (not\n> just lookups for missing blobs) - I think I prefer keeping the lists\n> separate, to avoid pessimizing the (likely) usual case where the\n> relevant blobs are all already in local repo storage.\n\nAnother relevant operation is looking up objects by offset or\nindex_nr. The current implementation involves building an in-memory\nreverse index on demand by reading the idx file and sorting it by\noffset --- see pack-revindex.c::create_pack_revindex. This takes O(n\nlog n) time where n is the size of the idx file.\n\nThat said, it could be avoided by storing an on-disk reverse index\nwith the pack. That's something we've been wanting to do anyway.\n\nThanks,\nJonathan\n"},{"id":"318792","messageId":"9b58bec9-dafb-aebc-d421-5327f2f607eb@jeffhostetler.com","threadId":"45320","inReplyTo":"20170503182725.GC28740@aiede.svl.corp.google.com","subject":"Re: [PATCH 00/10] RFC Partial Clone and Fetch","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-05-04T16:51:00Z","receivedAt":"2017-05-04T16:51:11Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 5/3/2017 2:27 PM, Jonathan Nieder wrote:\n> Hi,\n>\n> Jeff Hostetler wrote:\n>> Missing-Blob Support\n>> ====================\n>>\n>> Let me offer up an alternative idea for representing\n>> missing blobs. This is differs from both of our previous\n>> proposals. (I don't have any code for this new proposal,\n>> I just want to think out loud a bit and see if this is a\n>> direction worth pursuing -- or a complete non-starter.)\n>>\n>> Both proposals talk about detecting and adapting to a missing\n>> blob and ways to recover -- when we fail to find a blob.\n>> Comments on the thread asked about:\n>> () being able to detect missing blobs vs corrupt repos\n>> () being unable to detect duplicate blobs\n>> () expense of blob search.\n>>\n>> Suppose we store \"positive\" information about missing blobs?\n>> This would let us know that a blob is intentionally missing\n>> and possibly some meta-data about it.\n>\n> We've discussed this a little informally but didn't go more into\n> it, so I'm glad you brought it up.\n>\n> There are two use cases I care about. I'll want names to refer to\n> them later, so describing them now:\n>\n> A. A p4 or svn style \"monorepo\" containing all code within a company.\n> We want to make git scale well when working with such a\n> repository. Disk usage, network usage, index size, and object\n> lookup time are all issues for such a repository.\n>\n> A repository can creep up in size so it starts falling into this\n> category even though git coped well with it before. Another way\n> to end up in this category is a conversion from a version control\n> system like p4 or svn.\n>\n> B. A more modestly sized repository with some large blobs in it. At\n> clone time we want to omit unneeded large blobs to speed up the\n> clone, save disk space, and save bandwidth.\n>\n> For this kind of repository, the idx file already contained all\n> those blobs and that was not causing problems --- the only problem\n> was the actual blob size.\n\nYes, I've been primarily concerned with \"case A\" repos.\nI work with the team converting the Windows source repo\nto git. This was discussed in Brussels as part of the\nGVFS presentation.\n\nThe Windows tree has 3.5M files in the worktree for a simple checkout\nof HEAD. The index is 450MB. There are 500K trees/folders in\nthe commit. Multiply that by scale factor considering the number\nof trunk/release branches, number of developers, typical number of\ncommits per day per developer, and n years(decades) of history and\nwe get to a very large number....\n\nFWIW, there's also a \"case C\" which has both, but that\njust hurts to think about.\n\n>\n>> 1. Suppose we update the .pack file format slightly.\n> [...]\n>> 2. Make a similar change in the .idx format and git-index-pack\n>> to include them there. Then blob lookup operations could\n>> definitively determine that a blob exists and is just not\n>> present locally.\n>\n> Some nits:\n>\n> - there shouldn't be any need for the blobs to even be mentioned in\n> the .pack stored locally. The .idx file maps from sha1 to offset\n> within the packfile --- a special offset could mean \"this is a\n> missing blob\".\n>\n> - Git protocol works by sending pack files over the wire. The idx\n> file is not transmitted to the client --- the client instead\n> reconstructs it from the pack file. I assume this is why you\n> stored the SHA-1 of the object in the pack file, but it could\n> equally well be sent through another stream (since this proposal\n> involves a change to git protocol anyway).\n>\n> - However, the list of missing blobs can be inferred from the existing\n> pack format, without a change to the pack format used in git\n> protocol. As part of constructing the idx, \"git index-pack\"\n> inflates every object in the pack file sent by the server. This\n> means we know what blobs they reference, so we can easily produce a\n> list for the idx file without changing the pack file format.\n\nIn my original RFC there were comments/complaints that with\nmissing blobs we lose the ability to detect corruptions. My\nproposed changes to index-pack and rev-list (and suggestions\nfor other commands like fsck) just disabled those errors.\nPersonally, I'm OK with that, but I understand that others\nwould like to save the ability to distinguish between missing\nand corrupted.\n\nRight, only the .pack is sent over the wire. And that's why I\nsuggest listing the missing SHAs in it. I think we need the server\nto send a list of them -- whether in individual per-file type-5\nrecords as I suggested, or a single record containing a list of\nSHAs+data (which I think I prefer in hindsight).\n\nWRT being able to discover the missing blobs, is that true in\nall cases? I don't think it is for thin-packs -- where the server\nonly sends stuff you don't (supposedly) already have, right ?\n\nIf instead, we have pack-object indicate that it *would have*\nsent this blob normally, we don't change any of the semantics\nof how pack files are assembled. This gives the client a\ndefinitive list of what's missing.\n\n\n> If this is done by only changing the idx file format and not the pack\n> file, then it does not affect the protocol. That is good for\n> experimentation --- it lets us try out different formats client-side\n> without having to coordinate with server authors.\n>\n> In case (A), this proposal means we get back some of the per-object\n> overhead that we were trying to avoid. I would like to avoid that\n> if possible.\n\nIf thin-packs are used, the fetch only receives information about\nblobs that are new relative to the heads the client already has,\nright?\n\n>\n> In case (B), this proposal doesn't hurt.\n>\n> One problem with proposals so far has been how to handle repositories\n> with multiple remotes. Having a local list of missing blobs is\n> convenient because it can be associated to the remote --- when a blob\n> is referenced later, we know which remote to ask for for it. This is\n> a niche feature, but it's a nice bonus.\n\nThe more I think about it, I'd like to NOT put the list in the .idx\nfile. If we put it in a separate peer file next to the *.{pack,idx}\nwe could decorate it with the name of the remote used in the fetch\nor clone.\n\n>\n> [...]\n>> 3. With this, packfile-based blob-lookup operations can get a\n>> \"missing-blob\" result.\n>> () It should be possible to short-cut searching in other\n>> packfiles (because we don't have to assume that the blob\n>> was just misplaced in another packfile).\n>> () Lookup can still look for the corresponding loose blob\n>> (in case a previous lookup already \"faulted it in\").\n>\n> The number of packfiles to search is always a problem for object\n> lookups --- we need to be able to keep the number of packfiles low.\n> If we solve that for missing blobs but not for the ones that are\n> present, then git's object access performance is still slow.\n>\n> I have some ideas for improving git's gc behavior to bound the number\n> of packfiles better. I think that's mostly orthogonal to this\n> proposal. The main point of overlap is the number of packfiles\n> produced if someone uses one packfile per large blob, but that is bad\n> for the performance in successful lookups, not just negative lookups,\n> so it needs to be solved directly anyway.\n>\n\nI've always wondered if repack, fetch, and friends should build\na meta-idx that merges all of the current .idx files, but that\nis a different conversation....\n\n\n>> 4. We can then think about dynamically fetching it.\n>\n> This also seems orthogonal to whether the missing blobs list exists.\n>\n> [...]\n>> A missing-blob entry needs to contain the SHA-1 value of\n>> the blob (obviously). Other fields are nice to have, but\n>> are not necessary. Here are a few fields to consider.\n> [...]\n>> B. The raw size of the blob (5? bytes).\n>\n> This would be a very nice thing.\n>\n> If we were starting over, would this belong in the tree object?\n> (Part of the reason I ask is that we have an opportunity to sort\n> of start over as part of the transition away from SHA-1.)\n\nYes, putting the size in the tree would be nice. That does\nadd 5-8 bytes to every entry in every tree (on top of the\nlarger hash), but it would be useful.\n\nIf we're going there, we might just define the new hash\nas the concatenation of the SHA* and the length of the data\nhashed. So instead of a 32-byte SHA256, we'd have a (32 + 8)\nbyte value. (Or maybe a (32 + 5) if we want to squeeze it.)\n\n\n>\n> [...]\n>> C. A server \"hint\" (20 bytes)\n>> () Instructions to help the client fetch the blob.\n>> () If I have multiple remotes configured, a missing-blob\n>> should be fetched from the same server that created\n>> the missing-blob entry (since it may be the only\n>> one that has it).\n>\n> I am worried about the implications of storing this kind of policy\n> information in the pack file. There may be useful information along\n> these lines for a server to advertise, but I don't think it belongs in\n> the pack file. Git protocol relies on it being cheap to stream pack\n> files as is.\n\nPerhaps. I'm not sure it would be that expensive -- I'm thinking\nabout it being a server constant rather than a per-user/per-fetch\nfield. But maybe we don't need it. Assuming we can correctly\nassociate a missing blob with the server/remote that omitted it.\n\n>\n> [...]\n>\n> Thanks and hope that helps,\n> Jonathan\n>\n\nthanks!\nJeff\n"},{"id":"318797","messageId":"20170504184150.GA15203@aiede.svl.corp.google.com","threadId":"45320","inReplyTo":"9b58bec9-dafb-aebc-d421-5327f2f607eb@jeffhostetler.com","subject":"Re: [PATCH 00/10] RFC Partial Clone and Fetch","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-05-04T18:41:51Z","receivedAt":"2017-05-04T18:42:29Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi again,\n\nJeff Hostetler wrote:\n\n> In my original RFC there were comments/complaints that with\n> missing blobs we lose the ability to detect corruptions. My\n> proposed changes to index-pack and rev-list (and suggestions\n> for other commands like fsck) just disabled those errors.\n> Personally, I'm OK with that, but I understand that others\n> would like to save the ability to distinguish between missing\n> and corrupted.\n\nI'm also okay with it. In a partial clone, in the same way as a\nmissing ref represents a different valid state and thus passes fsck\nregardless of how it happened, a missing blob is a valid state and it\nis sensible for it to pass fsck.\n\nA person might object that previously a repository that passed \"git\nfsck\" was a repository where \"git fast-export --all\" would succeed,\nand if I omit a blob that is not present on the remote server then\nthat invariant is gone. But that problem exists even if we have a\nlist of missing blobs. The server could rewind history and garbage\ncollect, causing attempts on the client to fetch a previously\nadvertised missing blob to fail. Or the server can disappear\ncompletely, or it can lose all its data and have to be restored from\nan older backup that is missing newer blobs.\n\n> Right, only the .pack is sent over the wire. And that's why I\n> suggest listing the missing SHAs in it. I think we need the server\n> to send a list of them -- whether in individual per-file type-5\n> records as I suggested, or a single record containing a list of\n> SHAs+data (which I think I prefer in hindsight).\n\nA list of SHAs+data sounds sensible as a way of conveying additional\nper-blob information (such as size).\n\n> WRT being able to discover the missing blobs, is that true in\n> all cases? I don't think it is for thin-packs -- where the server\n> only sends stuff you don't (supposedly) already have, right ?\n\nGenerate the list of blobs referenced by trees in the pack, when you\nare inflating them in git index-pack. Omit any objects that you\nalready know about. The remainder is the set of missing blobs.\n\nOne thing this doesn't tell you is if the same missing blob is\navailable from multiple remotes. It associates each blob with a\nsingle remote, the first one it was discovered from.\n\n> If instead, we have pack-object indicate that it *would have*\n> sent this blob normally, we don't change any of the semantics\n> of how pack files are assembled. This gives the client a\n> definitive list of what's missing.\n\nIf there is additional information the server wants to convey about\nthe missing blobs, then this makes sense to me --- it has to send it\nsomewhere, and a separate list outside the pack seems like a good\nplace to put that.\n\nWhen there is no additional information beyond \"this is a blob I am\nomitting\", there is nothing the wire protocol needs to convey. But\nyou've convinced me that that's usually moot because the blob size\nis valuable information.\n\n[...]\n> The more I think about it, I'd like to NOT put the list in the .idx\n> file. If we put it in a separate peer file next to the *.{pack,idx}\n> we could decorate it with the name of the remote used in the fetch\n> or clone.\n\nI have no strong opinions about this in either direction.\n\nSince it only affects the local repository format and doesn't affect\nthe protocol, we can experiment without too much fuss. :)\n\n[...]\n> I've always wondered if repack, fetch, and friends should build\n> a meta-idx that merges all of the current .idx files, but that\n> is a different conversation....\n\nYes, we've been thinking about a one-index-for-many-packs facility\nto deal with the proliferation of packfiles with only one or a few\nlarge objects without having to waste I/O copying them into a\nconcatenated pack file.\n\nAnother thing we're looking into is incorporating something like\nMartin Fick's \"git exproll\" (\nhttp://public-inbox.org/git/1375756727-1275-1-git-send-email-artagnon@gmail.com/)\ninto \"git gc --auto\" to more aggressively keep the number of packs\ndown.\n\n> On 5/3/2017 2:27 PM, Jonathan Nieder wrote:\n\n>> If we were starting over, would this belong in the tree object?\n>> (Part of the reason I ask is that we have an opportunity to sort\n>> of start over as part of the transition away from SHA-1.)\n>\n> Yes, putting the size in the tree would be nice. That does\n> add 5-8 bytes to every entry in every tree (on top of the\n> larger hash), but it would be useful.\n>\n> If we're going there, we might just define the new hash\n> as the concatenation of the SHA* and the length of the data\n> hashed. So instead of a 32-byte SHA256, we'd have a (32 + 8)\n> byte value. (Or maybe a (32 + 5) if we want to squeeze it.)\n\nThanks --- that feedback helps.\n\nIt doesn't stop us from having to figure something else out in the\nshort term, of course.\n\n[...]\n>> I am worried about the implications of storing this kind of policy\n>> information in the pack file. There may be useful information along\n>> these lines for a server to advertise, but I don't think it belongs in\n>> the pack file. Git protocol relies on it being cheap to stream pack\n>> files as is.\n>\n> Perhaps. I'm not sure it would be that expensive -- I'm thinking\n> about it being a server constant rather than a per-user/per-fetch\n> field. But maybe we don't need it. Assuming we can correctly\n> associate a missing blob with the server/remote that omitted it.\n\nRight.\n\nI think we're heading toward a consensus:\n\n- that the server should (at least if requested?) inform the client of\n the blobs it is omitting and their sizes\n\n- that this would go in the same response as the packfile, but\n out-of-line from the pack\n\n- that it (at least optionally?) gets stored *somewhere* locally\n associated with the pack --- either through an extension to the idx\n file format or in a separate file alongside the pack file and idx\n file. This doesn't affect the protocol so it's not expensive to\n experiment\n\nThanks for the thoughtful replies.\n\nSincerely,\nJonathan\n"},{"id":"319025","messageId":"xmqqwp9scifu.fsf@gitster.mtv.corp.google.com","threadId":"45320","inReplyTo":"20170503182725.GC28740@aiede.svl.corp.google.com","subject":"Re: [PATCH 00/10] RFC Partial Clone and Fetch","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-05-08T00:15:33Z","receivedAt":"2017-05-08T00:15:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder writes:\n\n> - there shouldn't be any need for the blobs to even be mentioned in\n> the .pack stored locally. The .idx file maps from sha1 to offset\n> within the packfile --- a special offset could mean \"this is a\n> missing blob\".\n\nClever.\n\n> - However, the list of missing blobs can be inferred from the existing\n> pack format, without a change to the pack format used in git\n> protocol. As part of constructing the idx, \"git index-pack\"\n> inflates every object in the pack file sent by the server. This\n> means we know what blobs they reference, so we can easily produce a\n> list for the idx file without changing the pack file format.\n\nA minor wrinkle to keep in mind if you were to go this route is that\nyou'd need a way to tell the reason why a blob that is referenced by\na tree in the pack stream is not in the same pack stream. \n\nIf the resulting repository on the receiving side has that blob\nafter the transfer, it is likely that the reason why the blob does\nnot appear in the pack is because the want/have/ack exchange told\nthe sending side that the receiving side has a commit that contains\nthe blob. But when the blob does not exist in the receiving side\nafter the transfer, we cannot tell between two possible cases. The\nserver may have actively wanted to omit it (i.e. the case we are\ninterested in in this discussion thread). Or the receiving end said\nthat it has a commit that contains the blob, but due to preexisting\ncorruption, the receiving repository was missing the blob in\nreality.\n"}]}