{"thread":{"id":"45317","subject":"[PATCH 02/10] pack-objects: add --partial-by-size=n --partial-special","startedAt":"2017-03-08T17:46:01Z","lastAt":"2017-03-10T19:47:57Z","messageCount":31,"participants":["Jeff Hostetler","Junio C Hamano","Jeff King","Johannes Sixt","Brandon Williams"],"isPatch":true,"patchVersion":1,"patchTotal":10},"messages":[{"id":"313512","messageId":"1488994685-37403-3-git-send-email-jeffhost@microsoft.com","threadId":"45317","inReplyTo":"1488994685-37403-1-git-send-email-jeffhost@microsoft.com","subject":"[PATCH 02/10] pack-objects: add --partial-by-size=n --partial-special","fromName":"Jeff Hostetler","fromEmail":"jeffhost@microsoft.com","sentAt":"2017-03-08T17:37:57Z","receivedAt":"2017-03-08T17:46:01Z","isPatch":true,"sender":{"key":"jeffhost@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/7493832?v=4"},"body":"From: Jeff Hostetler \n\nTeach pack-objects to omit blobs from the generated packfile.\n\nWhen the --partial-by-size=n[kmg] argument is used, only blobs\nsmaller than the requested size are included. When n is zero,\nno blobs are included.\n\nWhen the --partial-special argument is used, git special files,\nsuch as \".gitattributes\" and \".gitignores\" are included.\n\nWhen both are given, the union of two are included.\n\nThis is intended to be used in a partial clone or fetch.\n(This has also been called sparse- or lazy-clone.)\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/pack-objects.c | 62 +++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 61 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7e052bb..2df2f49 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -77,6 +77,10 @@ static unsigned long cache_max_small_delta_size = 1000;\n \n static unsigned long window_memory_limit = 0;\n \n+static signed long partial_by_size = -1;\n+static int partial_special = 0;\n+static struct trace_key trace_partial = TRACE_KEY_INIT(PARTIAL);\n+\n /*\n * stats\n */\n@@ -2532,6 +2536,54 @@ static void show_object(struct object *obj, const char *name, void *data)\n \tobj->flags |= OBJECT_ADDED;\n }\n \n+/*\n+ * If ANY --partial-* option was given, we want to OMIT all\n+ * blobs UNLESS they match one of our patterns. We treat\n+ * the options as OR's so that we get the resulting UNION.\n+ */\n+static void show_object_partial(struct object *obj, const char *name, void *data)\n+{\n+\tunsigned long s = 0;\n+\n+\tif (obj->type != OBJ_BLOB)\n+\t\tgoto include_it;\n+\n+\t/*\n+\t * When (partial_by_size == 0), we want to OMIT all blobs.\n+\t * When (partial_by_size > 0), we want blobs smaller than that.\n+\t */\n+\tif (partial_by_size > 0) {\n+\t\tenum object_type t = sha1_object_info(obj->oid.hash, &s);\n+\t\tassert(t == OBJ_BLOB);\n+\t\tif (s < partial_by_size)\n+\t\t\tgoto include_it;\n+\t}\n+\n+\t/*\n+\t * When (partial_special), we want the .git* special files.\n+\t */\n+\tif (partial_special) {\n+\t\tif (strcmp(name, GITATTRIBUTES_FILE) == 0 ||\n+\t\t\tstrcmp(name, \".gitignore\") == 0)\n+\t\t\tgoto include_it;\n+\t\telse {\n+\t\t\tconst char *last_slash = strrchr(name, '/');\n+\t\t\tif (last_slash)\n+\t\t\t\tif (strcmp(last_slash+1, GITATTRIBUTES_FILE) == 0 ||\n+\t\t\t\t\tstrcmp(last_slash+1, \".gitignore\") == 0)\n+\t\t\t\t\tgoto include_it;\n+\t\t}\n+\t}\n+\n+\ttrace_printf_key(\n+\t\t&trace_partial, \"omitting blob '%s' %\"PRIuMAX\" '%s'\\n\",\n+\t\toid_to_hex(&obj->oid), (uintmax_t)s, name);\n+\treturn;\n+\n+include_it:\n+\tshow_object(obj, name, data);\n+}\n+\n static void show_edge(struct commit *commit)\n {\n \tadd_preferred_base(commit->object.oid.hash);\n@@ -2794,7 +2846,11 @@ static void get_object_list(int ac, const char **av)\n \tif (prepare_revision_walk(&revs))\n \t\tdie(\"revision walk setup failed\");\n \tmark_edges_uninteresting(&revs, show_edge);\n-\ttraverse_commit_list(&revs, show_commit, show_object, NULL);\n+\n+\tif (partial_by_size >= 0 || partial_special)\n+\t\ttraverse_commit_list(&revs, show_commit, show_object_partial, NULL);\n+\telse\n+\t\ttraverse_commit_list(&revs, show_commit, show_object, NULL);\n \n \tif (unpack_unreachable_expiration) {\n \t\trevs.ignore_missing_links = 1;\n@@ -2930,6 +2986,10 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"use a bitmap index if available to speed up counting objects\")),\n \t\tOPT_BOOL(0, \"write-bitmap-index\", &write_bitmap_index,\n \t\t\t N_(\"write a bitmap index together with the pack index\")),\n+\t\tOPT_MAGNITUDE(0, \"partial-by-size\", (unsigned long *)&partial_by_size,\n+\t\t\t N_(\"only include blobs smaller than size in result\")),\n+\t\tOPT_BOOL(0, \"partial-special\", &partial_special,\n+\t\t\t N_(\"only include blobs for git special files\")),\n \t\tOPT_END(),\n \t};\n \n-- \n2.7.4\n\n"},{"id":"313513","messageId":"1488994685-37403-8-git-send-email-jeffhost@microsoft.com","threadId":"45317","inReplyTo":"1488994685-37403-1-git-send-email-jeffhost@microsoft.com","subject":"[PATCH 07/10] index-pack: add --allow-partial option to relax blob existence checks","fromName":"Jeff Hostetler","fromEmail":"jeffhost@microsoft.com","sentAt":"2017-03-08T17:38:02Z","receivedAt":"2017-03-08T17:46:02Z","isPatch":true,"sender":{"key":"jeffhost@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/7493832?v=4"},"body":"From: Jeff Hostetler \n\nTeach index-pack to optionally not complain when there are missing\nblobs. This is for use following a partial clone or fetch when\nthe server omitted certain blobs.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/index-pack.c | 20 ++++++++++++++++++--\n 1 file changed, 18 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex f4b87c6..8f99408 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -13,7 +13,7 @@\n #include \"thread-utils.h\"\n \n static const char index_pack_usage[] =\n-\"git index-pack [-v] [-o ] [--keep | --keep=] [--verify] [--strict] ( | --stdin [--fix-thin] [])\";\n+\"git index-pack [-v] [-o ] [--keep | --keep=] [--verify] [--allow-partial] [--strict] ( | --stdin [--fix-thin] [])\";\n \n struct object_entry {\n \tstruct pack_idx_entry idx;\n@@ -81,6 +81,9 @@ static int show_resolving_progress;\n static int show_stat;\n static int check_self_contained_and_connected;\n \n+static int allow_partial;\n+static struct trace_key trace_partial = TRACE_KEY_INIT(PARTIAL);\n+\n static struct progress *progress;\n \n /* We always read in 4kB chunks. */\n@@ -220,9 +223,18 @@ static unsigned check_object(struct object *obj)\n \tif (!(obj->flags & FLAG_CHECKED)) {\n \t\tunsigned long size;\n \t\tint type = sha1_object_info(obj->oid.hash, &size);\n-\t\tif (type <= 0)\n+\t\tif (type <= 0) {\n+\t\t\tif (allow_partial > 0 && obj->type == OBJ_BLOB) {\n+\t\t\t\t/* Assume a previous partial clone/fetch omitted it. */\n+\t\t\t\ttrace_printf_key(\n+\t\t\t\t\t&trace_partial, \"omitted blob '%s'\\n\",\n+\t\t\t\t\toid_to_hex(&obj->oid));\n+\t\t\t\tobj->flags |= FLAG_CHECKED;\n+\t\t\t\treturn 0;\n+\t\t\t}\n \t\t\tdie(_(\"did not receive expected object %s\"),\n \t\t\t oid_to_hex(&obj->oid));\n+\t\t}\n \t\tif (type != obj->type)\n \t\t\tdie(_(\"object %s: expected type %s, found %s\"),\n \t\t\t oid_to_hex(&obj->oid),\n@@ -1718,6 +1730,10 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\t\tdie(_(\"bad %s\"), arg);\n \t\t\t} else if (skip_prefix(arg, \"--max-input-size=\", &arg)) {\n \t\t\t\tmax_input_size = strtoumax(arg, NULL, 10);\n+\t\t\t} else if (!strcmp(arg, \"--allow-partial\")) {\n+\t\t\t\tallow_partial = 1;\n+\t\t\t} else if (!strcmp(arg, \"--no-allow-partial\")) {\n+\t\t\t\tallow_partial = 0;\n \t\t\t} else\n \t\t\t\tusage(index_pack_usage);\n \t\t\tcontinue;\n-- \n2.7.4\n\n"},{"id":"313514","messageId":"1488994685-37403-6-git-send-email-jeffhost@microsoft.com","threadId":"45317","inReplyTo":"1488994685-37403-1-git-send-email-jeffhost@microsoft.com","subject":"[PATCH 05/10] fetch-pack: add partial-by-size and partial-special","fromName":"Jeff Hostetler","fromEmail":"jeffhost@microsoft.com","sentAt":"2017-03-08T17:38:00Z","receivedAt":"2017-03-08T17:46:03Z","isPatch":true,"sender":{"key":"jeffhost@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/7493832?v=4"},"body":"From: Jeff Hostetler \n\nTeach fetch-pack to take --partial-by-size and --partial-special\narguments and pass them via the transport to upload-pack to\nrequest that certain blobs be omitted from the resulting packfile.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fetch-pack.c | 9 +++++++++\n fetch-pack.c | 17 +++++++++++++++++\n fetch-pack.h | 2 ++\n transport.c | 8 ++++++++\n transport.h | 8 ++++++++\n 5 files changed, 44 insertions(+)\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex cfe9e44..324d7b2 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -8,6 +8,7 @@\n static const char fetch_pack_usage[] =\n \"git fetch-pack [--all] [--stdin] [--quiet | -q] [--keep | -k] [--thin] \"\n \"[--include-tag] [--upload-pack=] [--depth=] \"\n+\"[--partial-by-size=] [--partial-special] \"\n \"[--no-progress] [--diag-url] [-v] [:] [...]\";\n \n static void add_sought_entry(struct ref ***sought, int *nr, int *alloc,\n@@ -143,6 +144,14 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\targs.update_shallow = 1;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (skip_prefix(arg, \"--partial-by-size=\", &arg)) {\n+\t\t\targs.partial_by_size = xstrdup(arg);\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(\"--partial-special\", arg)) {\n+\t\t\targs.partial_special = 1;\n+\t\t\tcontinue;\n+\t\t}\n \t\tusage(fetch_pack_usage);\n \t}\n \tif (deepen_not.nr)\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex e0f5d5c..e355c38 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -372,6 +372,8 @@ static int find_common(struct fetch_pack_args *args,\n \t\t\tif (prefer_ofs_delta) strbuf_addstr(&c, \" ofs-delta\");\n \t\t\tif (deepen_since_ok) strbuf_addstr(&c, \" deepen-since\");\n \t\t\tif (deepen_not_ok) strbuf_addstr(&c, \" deepen-not\");\n+\t\t\tif (args->partial_by_size || args->partial_special)\n+\t\t\t\tstrbuf_addstr(&c, \" partial\");\n \t\t\tif (agent_supported) strbuf_addf(&c, \" agent=%s\",\n \t\t\t\t\t\t\t git_user_agent_sanitized());\n \t\t\tpacket_buf_write(&req_buf, \"want %s%s\\n\", remote_hex, c.buf);\n@@ -402,6 +404,12 @@ static int find_common(struct fetch_pack_args *args,\n \t\t\tpacket_buf_write(&req_buf, \"deepen-not %s\", s->string);\n \t\t}\n \t}\n+\n+\tif (args->partial_by_size)\n+\t\tpacket_buf_write(&req_buf, \"partial-by-size %s\", args->partial_by_size);\n+\tif (args->partial_special)\n+\t\tpacket_buf_write(&req_buf, \"partial-special\");\n+\n \tpacket_buf_flush(&req_buf);\n \tstate_len = req_buf.len;\n \n@@ -807,6 +815,10 @@ static int get_pack(struct fetch_pack_args *args,\n \t\t\t\t\t\"--keep=fetch-pack %\"PRIuMAX \" on %s\",\n \t\t\t\t\t(uintmax_t)getpid(), hostname);\n \t\t}\n+\n+\t\tif (args->partial_by_size || args->partial_special)\n+\t\t\targv_array_push(&cmd.args, \"--allow-partial\");\n+\n \t\tif (args->check_self_contained_and_connected)\n \t\t\targv_array_push(&cmd.args, \"--check-self-contained-and-connected\");\n \t}\n@@ -920,6 +932,11 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \telse\n \t\tprefer_ofs_delta = 0;\n \n+\tif (server_supports(\"partial\"))\n+\t\tprint_verbose(args, _(\"Server supports partial\"));\n+\telse if (args->partial_by_size || args->partial_special)\n+\t\tdie(_(\"Server does not support 'partial'\"));\n+\n \tif ((agent_feature = server_feature_value(\"agent\", &agent_len))) {\n \t\tagent_supported = 1;\n \t\tif (agent_len)\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex c912e3d..b8a26e0 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -12,6 +12,7 @@ struct fetch_pack_args {\n \tint depth;\n \tconst char *deepen_since;\n \tconst struct string_list *deepen_not;\n+\tconst char *partial_by_size;\n \tunsigned deepen_relative:1;\n \tunsigned quiet:1;\n \tunsigned keep_pack:1;\n@@ -29,6 +30,7 @@ struct fetch_pack_args {\n \tunsigned cloning:1;\n \tunsigned update_shallow:1;\n \tunsigned deepen:1;\n+\tunsigned partial_special:1;\n };\n \n /*\ndiff --git a/transport.c b/transport.c\nindex 5828e06..45f35a4 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -160,6 +160,12 @@ static int set_git_option(struct git_transport_options *opts,\n \t} else if (!strcmp(name, TRANS_OPT_DEEPEN_RELATIVE)) {\n \t\topts->deepen_relative = !!value;\n \t\treturn 0;\n+\t} else if (!strcmp(name, TRANS_OPT_PARTIAL_BY_SIZE)) {\n+\t\topts->partial_by_size = xstrdup(value);\n+\t\treturn 0;\n+\t} else if (!strcmp(name, TRANS_OPT_PARTIAL_SPECIAL)) {\n+\t\topts->partial_special = !!value;\n+\t\treturn 0;\n \t}\n \treturn 1;\n }\n@@ -227,6 +233,8 @@ static int fetch_refs_via_pack(struct transport *transport,\n \t\tdata->options.check_self_contained_and_connected;\n \targs.cloning = transport->cloning;\n \targs.update_shallow = data->options.update_shallow;\n+\targs.partial_by_size = data->options.partial_by_size;\n+\targs.partial_special = data->options.partial_special;\n \n \tif (!data->got_remote_heads) {\n \t\tconnect_setup(transport, 0);\ndiff --git a/transport.h b/transport.h\nindex bc55715..c3f2d52 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -15,12 +15,14 @@ struct git_transport_options {\n \tunsigned self_contained_and_connected : 1;\n \tunsigned update_shallow : 1;\n \tunsigned deepen_relative : 1;\n+\tunsigned partial_special : 1;\n \tint depth;\n \tconst char *deepen_since;\n \tconst struct string_list *deepen_not;\n \tconst char *uploadpack;\n \tconst char *receivepack;\n \tstruct push_cas_option *cas;\n+\tconst char *partial_by_size;\n };\n \n enum transport_family {\n@@ -210,6 +212,12 @@ void transport_check_allowed(const char *type);\n /* Send push certificates */\n #define TRANS_OPT_PUSH_CERT \"pushcert\"\n \n+/* Partial fetch to only include small files */\n+#define TRANS_OPT_PARTIAL_BY_SIZE \"partial-by-size\"\n+\n+/* Partial fetch to only include special files, like \".gitignore\" */\n+#define TRANS_OPT_PARTIAL_SPECIAL \"partial-special\"\n+\n /**\n * Returns 0 if the option was used, non-zero otherwise. Prints a\n * message to stderr if the option is not used.\n-- \n2.7.4\n\n"},{"id":"313515","messageId":"1488994685-37403-5-git-send-email-jeffhost@microsoft.com","threadId":"45317","inReplyTo":"1488994685-37403-1-git-send-email-jeffhost@microsoft.com","subject":"[PATCH 04/10] upload-pack: add partial (sparse) fetch","fromName":"Jeff Hostetler","fromEmail":"jeffhost@microsoft.com","sentAt":"2017-03-08T17:37:59Z","receivedAt":"2017-03-08T17:46:05Z","isPatch":true,"sender":{"key":"jeffhost@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/7493832?v=4"},"body":"From: Jeff Hostetler \n\nTeach upload-pack to advertise the \"partial\" capability\nin the fetch-pack/upload-pack protocol header and to pass\nthe value of partial-by-size and partial-special on to\npack-objects.\n\nUpdate protocol documentation.\n\nThis might be used in conjunction with a partial (sparse) clone\nor fetch to omit various blobs from the generated packfile.\n\nSigned-off-by: Jeff Hostetler \n---\n Documentation/technical/pack-protocol.txt | 14 ++++++++++\n Documentation/technical/protocol-capabilities.txt | 7 +++++\n upload-pack.c | 32 ++++++++++++++++++++++-\n 3 files changed, 52 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt\nindex c59ac99..0032729 100644\n--- a/Documentation/technical/pack-protocol.txt\n+++ b/Documentation/technical/pack-protocol.txt\n@@ -212,6 +212,7 @@ out of what the server said it could do with the first 'want' line.\n upload-request = want-list\n \t\t *shallow-line\n \t\t *1depth-request\n+\t\t *partial\n \t\t flush-pkt\n \n want-list = first-want\n@@ -223,10 +224,15 @@ out of what the server said it could do with the first 'want' line.\n \t\t PKT-LINE(\"deepen-since\" SP timestamp) /\n \t\t PKT-LINE(\"deepen-not\" SP ref)\n \n+ partial = PKT-LINE(\"partial-by-size\" SP magnitude) /\n+\t\t PKT-LINE(\"partial-special) \n+\n first-want = PKT-LINE(\"want\" SP obj-id SP capability-list)\n additional-want = PKT-LINE(\"want\" SP obj-id)\n \n depth = 1*DIGIT\n+\n+ magnitude = 1*DIGIT [ \"k\" | \"m\" | \"g\" ]\n ----\n \n Clients MUST send all the obj-ids it wants from the reference\n@@ -249,6 +255,14 @@ complete those commits. Commits whose parents are not received as a\n result are defined as shallow and marked as such in the server. This\n information is sent back to the client in the next step.\n \n+The client can optionally request a partial packfile that omits\n+various blobs. The value of \"partial-by-size\" is a non-negative\n+integer with optional units and requests blobs smaller than this\n+value. The \"partial-special\" command requests git-special files,\n+such as \".gitignore\". Using both requests the union of the two.\n+These requests are only valid if the server advertises the \"partial\"\n+capability.\n+\n Once all the 'want's and 'shallow's (and optional 'deepen') are\n transferred, clients MUST send a flush-pkt, to tell the server side\n that it is done sending the list.\ndiff --git a/Documentation/technical/protocol-capabilities.txt b/Documentation/technical/protocol-capabilities.txt\nindex 26dcc6f..9aa2123 100644\n--- a/Documentation/technical/protocol-capabilities.txt\n+++ b/Documentation/technical/protocol-capabilities.txt\n@@ -309,3 +309,10 @@ to accept a signed push certificate, and asks the to be\n included in the push certificate. A send-pack client MUST NOT\n send a push-cert packet unless the receive-pack server advertises\n this capability.\n+\n+partial\n+-------\n+\n+If the upload-pack server advertises this capability, fetch-pack\n+may send various \"partial-*\" commands to request a partial clone\n+or fetch where the server omits certain blobs from the packfile.\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 7597ba3..74f9dfa 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -63,6 +63,11 @@ static int advertise_refs;\n static int stateless_rpc;\n static const char *pack_objects_hook;\n \n+static struct strbuf partial_by_size = STRBUF_INIT;\n+static int client_requested_partial_capability;\n+static int have_partial_by_size;\n+static int have_partial_special;\n+\n static void reset_timeout(void)\n {\n \talarm(timeout);\n@@ -130,6 +135,10 @@ static void create_pack_file(void)\n \t\targv_array_push(&pack_objects.args, \"--delta-base-offset\");\n \tif (use_include_tag)\n \t\targv_array_push(&pack_objects.args, \"--include-tag\");\n+\tif (have_partial_by_size)\n+\t\targv_array_push(&pack_objects.args, partial_by_size.buf);\n+\tif (have_partial_special)\n+\t\targv_array_push(&pack_objects.args, \"--partial-special\");\n \n \tpack_objects.in = -1;\n \tpack_objects.out = -1;\n@@ -793,6 +802,23 @@ static void receive_needs(void)\n \t\t\tdeepen_rev_list = 1;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (skip_prefix(line, \"partial-by-size \", &arg)) {\n+\t\t\tunsigned long s;\n+\t\t\tif (!client_requested_partial_capability)\n+\t\t\t\tdie(\"git upload-pack: 'partial-by-size' option requires 'partial' capability\");\n+\t\t\tif (!git_parse_ulong(arg, &s))\n+\t\t\t\tdie(\"git upload-pack: invalid partial-by-size value: %s\", line);\n+\t\t\tstrbuf_addstr(&partial_by_size, \"--partial-by-size=\");\n+\t\t\tstrbuf_addstr(&partial_by_size, arg);\n+\t\t\thave_partial_by_size = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(line, \"partial-special\", &arg)) {\n+\t\t\tif (!client_requested_partial_capability)\n+\t\t\t\tdie(\"git upload-pack: 'partial-special' option requires 'partial' capability\");\n+\t\t\thave_partial_special = 1;\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!skip_prefix(line, \"want \", &arg) ||\n \t\t get_sha1_hex(arg, sha1_buf))\n \t\t\tdie(\"git upload-pack: protocol error, \"\n@@ -820,6 +846,8 @@ static void receive_needs(void)\n \t\t\tno_progress = 1;\n \t\tif (parse_feature_request(features, \"include-tag\"))\n \t\t\tuse_include_tag = 1;\n+\t\tif (parse_feature_request(features, \"partial\"))\n+\t\t\tclient_requested_partial_capability = 1;\n \n \t\to = parse_object(sha1_buf);\n \t\tif (!o)\n@@ -924,7 +952,9 @@ static int send_ref(const char *refname, const struct object_id *oid,\n {\n \tstatic const char *capabilities = \"multi_ack thin-pack side-band\"\n \t\t\" side-band-64k ofs-delta shallow deepen-since deepen-not\"\n-\t\t\" deepen-relative no-progress include-tag multi_ack_detailed\";\n+\t\t\" deepen-relative no-progress include-tag multi_ack_detailed\"\n+\t\t\" partial\"\n+\t\t;\n \tconst char *refname_nons = strip_namespace(refname);\n \tstruct object_id peeled;\n \n-- \n2.7.4\n\n"},{"id":"313516","messageId":"1488994685-37403-7-git-send-email-jeffhost@microsoft.com","threadId":"45317","inReplyTo":"1488994685-37403-1-git-send-email-jeffhost@microsoft.com","subject":"[PATCH 06/10] rev-list: add --allow-partial option to relax connectivity checks","fromName":"Jeff Hostetler","fromEmail":"jeffhost@microsoft.com","sentAt":"2017-03-08T17:38:01Z","receivedAt":"2017-03-08T17:46:13Z","isPatch":true,"sender":{"key":"jeffhost@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/7493832?v=4"},"body":"From: Jeff Hostetler \n\nTeach rev-list to optionally not complain when there are missing\nblobs. This is for use following a partial clone or fetch when\nthe server omitted certain blobs.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/rev-list.c | 22 +++++++++++++++++++++-\n 1 file changed, 21 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 0aa93d5..50c49ba 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -45,6 +45,7 @@ static const char rev_list_usage[] =\n \" --left-right\\n\"\n \" --count\\n\"\n \" special purpose:\\n\"\n+\" --allow-partial\\n\"\n \" --bisect\\n\"\n \" --bisect-vars\\n\"\n \" --bisect-all\"\n@@ -53,6 +54,9 @@ static const char rev_list_usage[] =\n static struct progress *progress;\n static unsigned progress_counter;\n \n+static int allow_partial;\n+static struct trace_key trace_partial = TRACE_KEY_INIT(PARTIAL);\n+\n static void finish_commit(struct commit *commit, void *data);\n static void show_commit(struct commit *commit, void *data)\n {\n@@ -178,8 +182,16 @@ static void finish_commit(struct commit *commit, void *data)\n static void finish_object(struct object *obj, const char *name, void *cb_data)\n {\n \tstruct rev_list_info *info = cb_data;\n-\tif (obj->type == OBJ_BLOB && !has_object_file(&obj->oid))\n+\tif (obj->type == OBJ_BLOB && !has_object_file(&obj->oid)) {\n+\t\tif (allow_partial) {\n+\t\t\t/* Assume a previous partial clone/fetch omitted it. */\n+\t\t\ttrace_printf_key(\n+\t\t\t\t&trace_partial, \"omitted blob '%s' '%s'\\n\",\n+\t\t\t\toid_to_hex(&obj->oid), name);\n+\t\t\treturn;\n+\t\t}\n \t\tdie(\"missing blob object '%s'\", oid_to_hex(&obj->oid));\n+\t}\n \tif (info->revs->verify_objects && !obj->parsed && obj->type != OBJ_COMMIT)\n \t\tparse_object(obj->oid.hash);\n }\n@@ -329,6 +341,14 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\tshow_progress = arg;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!strcmp(arg, \"--allow-partial\")) {\n+\t\t\tallow_partial = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"--no-allow-partial\")) {\n+\t\t\tallow_partial = 0;\n+\t\t\tcontinue;\n+\t\t}\n \t\tusage(rev_list_usage);\n \n \t}\n-- \n2.7.4\n\n"},{"id":"313517","messageId":"1488994685-37403-1-git-send-email-jeffhost@microsoft.com","threadId":"45317","inReplyTo":null,"subject":"[PATCH 00/10] RFC Partial Clone and Fetch","fromName":"Jeff Hostetler","fromEmail":"jeffhost@microsoft.com","sentAt":"2017-03-08T17:37:55Z","receivedAt":"2017-03-08T17:46:15Z","isPatch":true,"sender":{"key":"jeffhost@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/7493832?v=4"},"body":"From: Jeff Hostetler \n\n\n[RFC] Partial Clone and Fetch\n=============================\n\n\nThis is a WIP RFC for a partial clone and fetch feature wherein the client\ncan request that the server omit various blobs from the packfile during\nclone and fetch. Clients can later request omitted blobs (either from a\nmodified upload-pack-like request to the server or via a completely\nindependent mechanism).\n\nThe purpose here is to reduce the size of packfile downloads and help\ngit scale to extremely large repos.\n\nI use the term \"partial\" here to refer to a portion of one or more commits\nand to avoid use of loaded terms like \"sparse\", \"lazy\", \"narrow\", and \"skeleton\".\n\nThe concept of a partial clone/fetch is independent of and can complement\nthe existing shallow-clone, refspec, and limited-ref filtering mechanisms\nsince these all filter at the DAG level whereas the work described here\nworks *within* the set of commits already chosen for download.\n\n\nA. Requesting a Partial Clone/Fetch\n===================================\n\nClone, fetch, and fetch-pack will accept one or more new \"partial\"\ncommand line arguments as described below. The fetch-pack/upload-pack\nprotocol will be extended to include these new arguments. Upload-pack\nand pack-objects will be updated accordingly. Pack-objects will filter\nout the unwanted blobs as it is building the packfile. Rev-list and\nindex-pack will be updated to not complain when missing blobs are\ndetected in the received packfile.\n\n[1] \"--partial-by-size=[kmg]\"\nWhere is a non-negative integer with an optional unit.\n\nRequest that only blobs smaller than this be included in the packfile.\nThe client might use this to implement an alternate LFS or ODB mechanism\nfor large blobs, such as suggested in:\n https://public-inbox.org/git/20161130210420.15982-1-chriscool@tuxfamily.org/\n\nA special case of size zero would omit all blobs and is similar to the\ncommits-and-trees-only feature described in:\n https://public-inbox.org/git/20170113155253.1644-1-benpeart@microsoft.com/\n\n[2] \"--partial-special\"\nRequest that special files, such as \".gitignore\" and .gitattributes\",\nbe included.\n\n[3] *TODO* \"--partial-by-profile=\"\nWhere is verson-controlled file in the repository\n(either present in the requested commit or the default HEAD on the server).\n\n [I envision a \".gitsparse/\" hierarchy where teams can store\n common sparse-checkout profiles. And then they can reference\n them from their private \".git/info/sparse-checkout\" files.]\n\nPack-objects will use this file and the sparse-checkout rules to only\ninclude blobs in the packfile that would be needed to do the corresponding\nsparse-checkout (and let the client avoid having to demand-load their\nentire enlistment).\n\n\nWhen multiple \"partial\" options are given, they are treated as a simple OR\ngiving the union of the blobs selected.\n\nThe patch series describes the changes to the fetch-pack/upload-pack\nprotocol:\n Documentation/technical/pack-protocol.txt\n Documentation/technical/protocol-capabilities.txt\n\n\nB. Issues Backfilling Omitted Blobs\n===================================\n\nIdeally, if the client only does \"--partial-by-profile\" fetches, it\nshould not need to fetch individual missing blobs, but we have to allow\nfor it to handle the other commands and other unexpected issues.\n\nThere are 3 orthogonal concepts here: when, how and where?\n\n\n[1] When:\n(1a) a pre-command or hook to identify needed blobs and pre-fetch them\nbefore allowing the actual command to start;\n(1b) a dry-run mode for the command to likewise pre-fetch them; or\n(1c) \"fault\" them in as necessary in read_object() while the command is\nrunning and without any pre-fetch (either synchronously or asynchronously\nand with/without a helper process).\n\nIdeas for (1c) are being addressed in the following threads:\n https://public-inbox.org/git/20170113155253.1644-1-benpeart@microsoft.com/\n https://public-inbox.org/git/20170117184258.sd7h2hkv27w52gzt@sigill.intra.peff.net/\n https://public-inbox.org/git/20161130210420.15982-1-chriscool@tuxfamily.org/\nso I won't consider them here.\n\nIdeas (1a) and (1b) have the advantage that they try to obtain all\nrequired blobs before allowing an operation to start, so there is\nless opportunity to leave the user in a weird state.\n\nThe best solution may be a combination of (1a) and (1b) and may depend\non the individual command. However, (1b) will further complicate the\nsource in the existing commands, so in some cases it may be simpler to\njust take the ideas and implement stand-alone pre-commands.\n\nFor now, I'm going to limit this RFC to (1a).\n\n\n[2] How:\n(2a) augment the existing git protocols to include blob requests;\n(2b) local external process (such as a database client or a local bulk\nfetch daemon);\n\nIdeas for (2b) are being addressed in the above threads, so I won't\nconsider them here.\n\nSo I'm going to limit this RFC to (2a).\n\n\n[3] Where:\n(3a) the same remote server used for the partial clone/fetch;\n(3b) anywhere else, such as a proxy server or Azure or S3 blob store.\n\nThere's no reason that the client should be limited to going back to\nthe same server, but I'm not going to consider it here, so I'm going\nto limit this RFC to (3a).\n\n\n\nC. New Blob-Fetch Protocol (2a)\n===============================\n\n*TODO* A new pair of commands, such as fetch-blob-pack and upload-blob-pack,\nwill be created to let the client request a batch of blobs and receive a\npackfile. A protocol similar to the fetch-pack/upload-pack will be spoken\nbetween them. (This avoids complicating the existing protocol and the work\nof enumerating the refs.) Upload-blob-pack will use pack-objects to build\nthe packfile.\n\nIt is also more efficient than requesting a single blob at a time using\nthe existing fetch-pack/upload-pack mechanism (with the various allow\nunreachable options).\n\n*TODO* The new request protocol will be defined in the patch series.\nIt will include: a list of the desired blob SHAs. Possibly also the commit\nSHA, branch name, and pathname of each blob (or whatever is necessary to let\nthe server address the reachability concerns). Possibly also the last\nknown SHA for each blob to allow for deltafication in the packfile.\n\n\nD. Pre-fetching Blobs (1a)\n==========================\n\nOn the client side, one or more special commands will be created to assemble\nthe list of blobs needed for an operation and passed to fetch-blob-pack.\n\n\nCheckout Example: After running a command like:\n 'clone --partial-by-size=1m --no-checkout'\n\nand before doing an actual checkout, we need a command to essentially do:\n (1) \"ls-tree -r \",\n (2) filter that by the sparse-checkout currently in effect,\n (3) filter that for missing blobs,\n (4) and pass the resulting list to fetch-blob-pack.\n\nAfterwards, checkout should complete without faulting.\n\nA new \"git ls-partial \" command has been created to do\nsteps 1 thru 3 and print the resulting list of SHAs on stdout.\n\n\nE. Unresolved Thoughts\n======================\n\n*TODO* The server should optionally return (in a side-band?) a list \nof the blobs that it omitted from the packfile (and possibly the sizes\nor sha1_object_info() data for them) during the fetch-pack/upload-pack\noperation. This would allow the client to distinguish from invalid\nSHAs and missing ones. Size information would allow the client to\nmaybe choose between various servers.\n\n*TODO* The partial clone arguments should be recorded in \".git/info/\"\nso that subsequent fetch commands can inherit them and rev-list/index-pack\nknow to not complain by default.\n\n*TODO* Update GC like rev-list to not complain when there are missing blobs.\n\n*TODO* Extend ls-partial to include the \"-m\" and 3 tree-ish arguments\nlike read-tree, so we can pre-fetch for merges that may require file\nmerges (that may or may not be within our sparse-checkout).\n\n*TODO* I also need to review the RFC that Mark Thomas submitted over\nthe weekend:\n https://public-inbox.org/git/20170304191901.9622-1-markbt%40efaref.net/t\n\n\n\n\n\nJeff Hostetler (10):\n pack-objects: eat CR in addition to LF after fgets.\n pack-objects: add --partial-by-size=n --partial-special\n pack-objects: test for --partial-by-size --partial-special\n upload-pack: add partial (sparse) fetch\n fetch-pack: add partial-by-size and partial-special\n rev-list: add --allow-partial option to relax connectivity checks\n index-pack: add --allow-partial option to relax blob existence checks\n fetch: add partial-by-size and partial-special arguments\n clone: add partial-by-size and partial-special arguments\n ls-partial: created command to list missing blobs\n\n Documentation/technical/pack-protocol.txt | 14 ++\n Documentation/technical/protocol-capabilities.txt | 7 +\n Makefile | 2 +\n builtin.h | 1 +\n builtin/clone.c | 26 ++\n builtin/fetch-pack.c | 9 +\n builtin/fetch.c | 26 +-\n builtin/index-pack.c | 20 +-\n builtin/ls-partial.c | 110 +++++++++\n builtin/pack-objects.c | 64 ++++-\n builtin/rev-list.c | 22 +-\n connected.c | 3 +\n connected.h | 3 +\n fetch-pack.c | 17 ++\n fetch-pack.h | 2 +\n git.c | 1 +\n partial-utils.c | 279 ++++++++++++++++++++++\n partial-utils.h | 93 ++++++++\n t/5316-pack-objects-partial.sh | 72 ++++++\n transport.c | 8 +\n transport.h | 8 +\n upload-pack.c | 32 ++-\n 22 files changed, 813 insertions(+), 6 deletions(-)\n create mode 100644 builtin/ls-partial.c\n create mode 100644 partial-utils.c\n create mode 100644 partial-utils.h\n create mode 100644 t/5316-pack-objects-partial.sh\n\n-- \n2.7.4\n\n"},{"id":"313518","messageId":"1488994685-37403-4-git-send-email-jeffhost@microsoft.com","threadId":"45317","inReplyTo":"1488994685-37403-1-git-send-email-jeffhost@microsoft.com","subject":"[PATCH 03/10] pack-objects: test for --partial-by-size --partial-special","fromName":"Jeff Hostetler","fromEmail":"jeffhost@microsoft.com","sentAt":"2017-03-08T17:37:58Z","receivedAt":"2017-03-08T17:46:18Z","isPatch":true,"sender":{"key":"jeffhost@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/7493832?v=4"},"body":"From: Jeff Hostetler \n\nSome simple tests for pack-objects with the new --partial-by-size\nand --partial-special options.\n\nSigned-off-by: Jeff Hostetler \n---\n t/5316-pack-objects-partial.sh | 72 ++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 72 insertions(+)\n create mode 100644 t/5316-pack-objects-partial.sh\n\ndiff --git a/t/5316-pack-objects-partial.sh b/t/5316-pack-objects-partial.sh\nnew file mode 100644\nindex 0000000..352de34\n--- /dev/null\n+++ b/t/5316-pack-objects-partial.sh\n@@ -0,0 +1,72 @@\n+#!/bin/sh\n+\n+test_description='pack-object partial'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\tperl -e \"print \\\"a\\\" x 11;\" > a &&\n+\tperl -e \"print \\\"a\\\" x 1100;\" > b &&\n+\tperl -e \"print \\\"a\\\" x 1100000;\" > c &&\n+\techo \"ignored\" > .gitignore &&\n+\tgit add a b c .gitignore &&\n+\tgit commit -m test\n+\t'\n+\n+test_expect_success 'all blobs' '\n+\tgit pack-objects --revs --thin --stdout >all.pack <<-EOF &&\n+\tmaster\n+\t\n+\tEOF\n+\tgit index-pack all.pack &&\n+\ttest 4 = $(git verify-pack -v all.pack | grep blob | wc -l)\n+\t'\n+\n+test_expect_success 'no blobs' '\n+\tgit pack-objects --revs --thin --stdout --partial-by-size=0 >none.pack <<-EOF &&\n+\tmaster\n+\t\n+\tEOF\n+\tgit index-pack none.pack &&\n+\ttest 0 = $(git verify-pack -v none.pack | grep blob | wc -l)\n+\t'\n+\n+test_expect_success 'small blobs' '\n+\tgit pack-objects --revs --thin --stdout --partial-by-size=1M >small.pack <<-EOF &&\n+\tmaster\n+\t\n+\tEOF\n+\tgit index-pack small.pack &&\n+\ttest 3 = $(git verify-pack -v small.pack | grep blob | wc -l)\n+\t'\n+\n+test_expect_success 'tiny blobs' '\n+\tgit pack-objects --revs --thin --stdout --partial-by-size=100 >tiny.pack <<-EOF &&\n+\tmaster\n+\t\n+\tEOF\n+\tgit index-pack tiny.pack &&\n+\ttest 2 = $(git verify-pack -v tiny.pack | grep blob | wc -l)\n+\t'\n+\n+test_expect_success 'special' '\n+\tgit pack-objects --revs --thin --stdout --partial-special >spec.pack <<-EOF &&\n+\tmaster\n+\t\n+\tEOF\n+\tgit index-pack spec.pack &&\n+\ttest 1 = $(git verify-pack -v spec.pack | grep blob | wc -l)\n+\t'\n+\n+test_expect_success 'union' '\n+\tgit pack-objects --revs --thin --stdout --partial-by-size=0 --partial-special >union.pack <<-EOF &&\n+\tmaster\n+\t\n+\tEOF\n+\tgit index-pack union.pack &&\n+\ttest 1 = $(git verify-pack -v union.pack | grep blob | wc -l)\n+\t'\n+\n+test_done\n+\n+\n-- \n2.7.4\n\n"},{"id":"313519","messageId":"1488994685-37403-9-git-send-email-jeffhost@microsoft.com","threadId":"45317","inReplyTo":"1488994685-37403-1-git-send-email-jeffhost@microsoft.com","subject":"[PATCH 08/10] fetch: add partial-by-size and partial-special arguments","fromName":"Jeff Hostetler","fromEmail":"jeffhost@microsoft.com","sentAt":"2017-03-08T17:38:03Z","receivedAt":"2017-03-08T17:46:20Z","isPatch":true,"sender":{"key":"jeffhost@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/7493832?v=4"},"body":"From: Jeff Hostetler \n\nTeach fetch to accept --partial-by-size=n and --partial-special\narguments and pass them to fetch-patch to request that the\nserver certain blobs from the generated packfile.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fetch.c | 26 +++++++++++++++++++++++++-\n connected.c | 3 +++\n connected.h | 3 +++\n 3 files changed, 31 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex b5ad09d..3d47107 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -52,6 +52,8 @@ static const char *recurse_submodules_default;\n static int shown_url = 0;\n static int refmap_alloc, refmap_nr;\n static const char **refmap_array;\n+static const char *partial_by_size;\n+static int partial_special;\n \n static int option_parse_recurse_submodules(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n@@ -141,6 +143,11 @@ static struct option builtin_fetch_options[] = {\n \t\t\tTRANSPORT_FAMILY_IPV4),\n \tOPT_SET_INT('6', \"ipv6\", &family, N_(\"use IPv6 addresses only\"),\n \t\t\tTRANSPORT_FAMILY_IPV6),\n+\tOPT_STRING(0, \"partial-by-size\", &partial_by_size,\n+\t\t\t N_(\"size\"),\n+\t\t\t N_(\"only include blobs smaller than this\")),\n+\tOPT_BOOL(0, \"partial-special\", &partial_special,\n+\t\t\t N_(\"only include blobs for git special files\")),\n \tOPT_END()\n };\n \n@@ -731,6 +738,10 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n \tconst char *filename = dry_run ? \"/dev/null\" : git_path_fetch_head();\n \tint want_status;\n \tint summary_width = transport_summary_width(ref_map);\n+\tstruct check_connected_options opt = CHECK_CONNECTED_INIT;\n+\n+\tif (partial_by_size || partial_special)\n+\t\topt.allow_partial = 1;\n \n \tfp = fopen(filename, \"a\");\n \tif (!fp)\n@@ -742,7 +753,7 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n \t\turl = xstrdup(\"foreign\");\n \n \trm = ref_map;\n-\tif (check_connected(iterate_ref_map, &rm, NULL)) {\n+\tif (check_connected(iterate_ref_map, &rm, &opt)) {\n \t\trc = error(_(\"%s did not send all necessary objects\\n\"), url);\n \t\tgoto abort;\n \t}\n@@ -882,6 +893,9 @@ static int quickfetch(struct ref *ref_map)\n \tstruct ref *rm = ref_map;\n \tstruct check_connected_options opt = CHECK_CONNECTED_INIT;\n \n+\tif (partial_by_size || partial_special)\n+\t\topt.allow_partial = 1;\n+\n \t/*\n \t * If we are deepening a shallow clone we already have these\n \t * objects reachable. Running rev-list here will return with\n@@ -1020,6 +1034,10 @@ static struct transport *prepare_transport(struct remote *remote, int deepen)\n \t\tset_option(transport, TRANS_OPT_DEEPEN_RELATIVE, \"yes\");\n \tif (update_shallow)\n \t\tset_option(transport, TRANS_OPT_UPDATE_SHALLOW, \"yes\");\n+\tif (partial_by_size)\n+\t\tset_option(transport, TRANS_OPT_PARTIAL_BY_SIZE, partial_by_size);\n+\tif (partial_special)\n+\t\tset_option(transport, TRANS_OPT_PARTIAL_SPECIAL, \"yes\");\n \treturn transport;\n }\n \n@@ -1314,6 +1332,12 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_fetch_options, builtin_fetch_usage, 0);\n \n+\tif (partial_by_size) {\n+\t\tunsigned long s;\n+\t\tif (!git_parse_ulong(partial_by_size, &s))\n+\t\t\tdie(_(\"invalid partial-by-size value\"));\n+\t}\n+\n \tif (deepen_relative) {\n \t\tif (deepen_relative < 0)\n \t\t\tdie(_(\"Negative depth in --deepen is not supported\"));\ndiff --git a/connected.c b/connected.c\nindex 136c2ac..b07cbb5 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -62,6 +62,9 @@ int check_connected(sha1_iterate_fn fn, void *cb_data,\n \t\targv_array_pushf(&rev_list.args, \"--progress=%s\",\n \t\t\t\t _(\"Checking connectivity\"));\n \n+\tif (opt->allow_partial)\n+\t\targv_array_push(&rev_list.args, \"--allow-partial\");\n+\n \trev_list.git_cmd = 1;\n \trev_list.env = opt->env;\n \trev_list.in = -1;\ndiff --git a/connected.h b/connected.h\nindex 4ca325f..756259e 100644\n--- a/connected.h\n+++ b/connected.h\n@@ -34,6 +34,9 @@ struct check_connected_options {\n \t/* If non-zero, show progress as we traverse the objects. */\n \tint progress;\n \n+\t/* A previous partial clone/fetch may have omitted some blobs. */\n+\tint allow_partial;\n+\n \t/*\n \t * Insert these variables into the environment of the child process.\n \t */\n-- \n2.7.4\n\n"},{"id":"313520","messageId":"1488994685-37403-2-git-send-email-jeffhost@microsoft.com","threadId":"45317","inReplyTo":"1488994685-37403-1-git-send-email-jeffhost@microsoft.com","subject":"[PATCH 01/10] pack-objects: eat CR in addition to LF after fgets.","fromName":"Jeff Hostetler","fromEmail":"jeffhost@microsoft.com","sentAt":"2017-03-08T17:37:56Z","receivedAt":"2017-03-08T17:46:29Z","isPatch":true,"sender":{"key":"jeffhost@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/7493832?v=4"},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n builtin/pack-objects.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex f294dcf..7e052bb 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2764,6 +2764,8 @@ static void get_object_list(int ac, const char **av)\n \t\tint len = strlen(line);\n \t\tif (len && line[len - 1] == '\\n')\n \t\t\tline[--len] = 0;\n+\t\tif (len && line[len - 1] == '\\r')\n+\t\t\tline[--len] = 0;\n \t\tif (!len)\n \t\t\tbreak;\n \t\tif (*line == '-') {\n-- \n2.7.4\n\n"},{"id":"313521","messageId":"1488994685-37403-10-git-send-email-jeffhost@microsoft.com","threadId":"45317","inReplyTo":"1488994685-37403-1-git-send-email-jeffhost@microsoft.com","subject":"[PATCH 09/10] clone: add partial-by-size and partial-special arguments","fromName":"Jeff Hostetler","fromEmail":"jeffhost@microsoft.com","sentAt":"2017-03-08T17:38:04Z","receivedAt":"2017-03-08T17:46:31Z","isPatch":true,"sender":{"key":"jeffhost@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/7493832?v=4"},"body":"From: Jeff Hostetler \n\nTeach clone to accept --partial-by-size=n and --partial-special\narguments to request that the server omit certain blobs from\nthe generated packfile.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/clone.c | 26 ++++++++++++++++++++++++++\n 1 file changed, 26 insertions(+)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 3f63edb..e5a5904 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -56,6 +56,8 @@ static struct string_list option_required_reference = STRING_LIST_INIT_NODUP;\n static struct string_list option_optional_reference = STRING_LIST_INIT_NODUP;\n static int option_dissociate;\n static int max_jobs = -1;\n+static const char *partial_by_size;\n+static int partial_special;\n \n static struct option builtin_clone_options[] = {\n \tOPT__VERBOSITY(&option_verbosity),\n@@ -112,6 +114,11 @@ static struct option builtin_clone_options[] = {\n \t\t\tTRANSPORT_FAMILY_IPV4),\n \tOPT_SET_INT('6', \"ipv6\", &family, N_(\"use IPv6 addresses only\"),\n \t\t\tTRANSPORT_FAMILY_IPV6),\n+\tOPT_STRING(0, \"partial-by-size\", &partial_by_size,\n+\t\t\t N_(\"size\"),\n+\t\t\t N_(\"only include blobs smaller than this\")),\n+\tOPT_BOOL(0, \"partial-special\", &partial_special,\n+\t\t\t N_(\"only include blobs for git special files\")),\n \tOPT_END()\n };\n \n@@ -625,6 +632,9 @@ static void update_remote_refs(const struct ref *refs,\n \tif (check_connectivity) {\n \t\tstruct check_connected_options opt = CHECK_CONNECTED_INIT;\n \n+\t\tif (partial_by_size || partial_special)\n+\t\t\topt.allow_partial = 1;\n+\n \t\topt.transport = transport;\n \t\topt.progress = transport->progress;\n \n@@ -1021,6 +1031,10 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\t\twarning(_(\"--shallow-since is ignored in local clones; use file:// instead.\"));\n \t\tif (option_not.nr)\n \t\t\twarning(_(\"--shallow-exclude is ignored in local clones; use file:// instead.\"));\n+\t\tif (partial_by_size)\n+\t\t\twarning(_(\"--partial-by-size is ignored in local clones; use file:// instead.\"));\n+\t\tif (partial_special)\n+\t\t\twarning(_(\"--partial-special is ignored in local clones; use file:// instead.\"));\n \t\tif (!access(mkpath(\"%s/shallow\", path), F_OK)) {\n \t\t\tif (option_local > 0)\n \t\t\t\twarning(_(\"source repository is shallow, ignoring --local\"));\n@@ -1052,6 +1066,18 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK,\n \t\t\t\t option_upload_pack);\n \n+\tif (partial_by_size) {\n+\t\ttransport_set_option(transport, TRANS_OPT_PARTIAL_BY_SIZE,\n+\t\t\t\t partial_by_size);\n+\t\tif (transport->smart_options)\n+\t\t\ttransport->smart_options->partial_by_size = partial_by_size;\n+\t}\n+\tif (partial_special) {\n+\t\ttransport_set_option(transport, TRANS_OPT_PARTIAL_SPECIAL, \"yes\");\n+\t\tif (transport->smart_options)\n+\t\t\ttransport->smart_options->partial_special = 1;\n+\t}\n+\n \tif (transport->smart_options && !deepen)\n \t\ttransport->smart_options->check_self_contained_and_connected = 1;\n \n-- \n2.7.4\n\n"},{"id":"313522","messageId":"1488994685-37403-11-git-send-email-jeffhost@microsoft.com","threadId":"45317","inReplyTo":"1488994685-37403-1-git-send-email-jeffhost@microsoft.com","subject":"[PATCH 10/10] ls-partial: created command to list missing blobs","fromName":"Jeff Hostetler","fromEmail":"jeffhost@microsoft.com","sentAt":"2017-03-08T17:38:05Z","receivedAt":"2017-03-08T17:46:38Z","isPatch":true,"sender":{"key":"jeffhost@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/7493832?v=4"},"body":"From: Jeff Hostetler \n\nAdded a command to list the missing blobs for a commit.\nThis can be used after a partial clone or fetch to list\nthe omitted blobs that the client would need to checkout\nthe given commit/branch. Optionally respecting or ignoring\nthe current sparse-checkout definition.\n\nThis command prints a simple list of blob SHAs. It is\nexpected that this would be piped into another command\nwith knowledge of the transport and/or blob store.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 2 +\n builtin.h | 1 +\n builtin/ls-partial.c | 110 ++++++++++++++++++++\n git.c | 1 +\n partial-utils.c | 279 +++++++++++++++++++++++++++++++++++++++++++++++++++\n partial-utils.h | 93 +++++++++++++++++\n 6 files changed, 486 insertions(+)\n create mode 100644 builtin/ls-partial.c\n create mode 100644 partial-utils.c\n create mode 100644 partial-utils.h\n\ndiff --git a/Makefile b/Makefile\nindex 9ec6065..96e9e1e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -791,6 +791,7 @@ LIB_OBJS += pack-write.o\n LIB_OBJS += pager.o\n LIB_OBJS += parse-options.o\n LIB_OBJS += parse-options-cb.o\n+LIB_OBJS += partial-utils.o\n LIB_OBJS += patch-delta.o\n LIB_OBJS += patch-ids.o\n LIB_OBJS += path.o\n@@ -908,6 +909,7 @@ BUILTIN_OBJS += builtin/init-db.o\n BUILTIN_OBJS += builtin/interpret-trailers.o\n BUILTIN_OBJS += builtin/log.o\n BUILTIN_OBJS += builtin/ls-files.o\n+BUILTIN_OBJS += builtin/ls-partial.o\n BUILTIN_OBJS += builtin/ls-remote.o\n BUILTIN_OBJS += builtin/ls-tree.o\n BUILTIN_OBJS += builtin/mailinfo.o\ndiff --git a/builtin.h b/builtin.h\nindex 9e4a898..df00c4b 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -79,6 +79,7 @@ extern int cmd_interpret_trailers(int argc, const char **argv, const char *prefi\n extern int cmd_log(int argc, const char **argv, const char *prefix);\n extern int cmd_log_reflog(int argc, const char **argv, const char *prefix);\n extern int cmd_ls_files(int argc, const char **argv, const char *prefix);\n+extern int cmd_ls_partial(int argc, const char **argv, const char *prefix);\n extern int cmd_ls_tree(int argc, const char **argv, const char *prefix);\n extern int cmd_ls_remote(int argc, const char **argv, const char *prefix);\n extern int cmd_mailinfo(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/ls-partial.c b/builtin/ls-partial.c\nnew file mode 100644\nindex 0000000..8ebf045\n--- /dev/null\n+++ b/builtin/ls-partial.c\n@@ -0,0 +1,110 @@\n+#include \"cache.h\"\n+#include \"blob.h\"\n+#include \"tree.h\"\n+#include \"commit.h\"\n+#include \"quote.h\"\n+#include \"builtin.h\"\n+#include \"parse-options.h\"\n+#include \"pathspec.h\"\n+#include \"dir.h\"\n+#include \"partial-utils.h\"\n+\n+static struct trace_key trace_partial = TRACE_KEY_INIT(PARTIAL);\n+\n+static int verbose;\n+static int ignore_sparse;\n+struct exclude_list el;\n+\n+static const char * const ls_partial_usage[] = {\n+\tN_(\"git ls-partial [] \"),\n+\tNULL\n+};\n+\n+/*\n+ * map arg into SHA1 and get the root treenode.\n+ */\n+static struct tree *lookup_tree_from_treeish(const char *arg)\n+{\n+\tunsigned char sha1[20];\n+\tstruct tree *tree;\n+\n+\tif (get_sha1(arg, sha1))\n+\t\tdie(\"not a valid object name '%s'\", arg);\n+\n+\ttrace_printf_key(\n+\t\t&trace_partial,\n+\t\t\"ls-partial: treeish '%s' '%s'\\n\",\n+\t\targ, sha1_to_hex(sha1));\n+\n+\tif (verbose) {\n+\t\tprintf(\"commit\\t%s\\n\", sha1_to_hex(sha1));\n+\t\tprintf(\"branch\\t%s\\n\", arg);\n+\t}\n+\t\n+\ttree = parse_tree_indirect(sha1);\n+\tif (!tree)\n+\t\tdie(\"not a tree object '%s'\", arg);\n+\n+\treturn tree;\n+}\n+\n+static void print_results(const struct pu_vec *vec)\n+{\n+\tint k;\n+\n+\tfor (k = 0; k < vec->data_nr; k++)\n+\t\tprintf(\"%s\\n\", oid_to_hex(&vec->data[k]->oid));\n+}\n+\n+static void print_results_verbose(const struct pu_vec *vec)\n+{\n+\tint k;\n+\n+\t/* TODO Consider -z version */\n+\n+\tfor (k = 0; k < vec->data_nr; k++)\n+\t\tprintf(\"%s\\t%s\\n\", oid_to_hex(&vec->data[k]->oid), vec->data[k]->fullpath.buf);\n+}\n+\n+int cmd_ls_partial(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct exclude_list el;\n+\tstruct tree *tree;\n+\tstruct pu_vec *vec;\n+\tstruct pu_vec *vec_all = NULL;\n+\tstruct pu_vec *vec_sparse = NULL;\n+\tstruct pu_vec *vec_missing = NULL;\n+\t\n+\tconst struct option ls_partial_options[] = {\n+\t\tOPT__VERBOSE(&verbose, N_(\"show verbose blob details\")),\n+\t\tOPT_BOOL(0, \"ignore-sparse\", &ignore_sparse,\n+\t\t\t\t N_(\"ignore sparse-checkout settings (scan whole tree)\")),\n+\t\tOPT_END()\n+\t};\n+\n+\tgit_config(git_default_config, NULL);\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t\t\t\t ls_partial_options, ls_partial_usage, 0);\n+\tif (argc < 1)\n+\t\tusage_with_options(ls_partial_usage, ls_partial_options);\n+\n+\ttree = lookup_tree_from_treeish(argv[0]);\n+\n+\tvec_all = pu_vec_ls_tree(tree, prefix, argv + 1);\n+\tif (ignore_sparse || pu_load_sparse_definitions(\"info/sparse-checkout\", &el) < 0)\n+\t\tvec = vec_all;\n+\telse {\n+\t\tvec_sparse = pu_vec_filter_sparse(vec_all, &el);\n+\t\tvec = vec_sparse;\n+\t}\n+\n+\tvec_missing = pu_vec_filter_missing(vec);\n+\tvec = vec_missing;\n+\n+\tif (verbose)\n+\t\tprint_results_verbose(vec);\n+\telse\n+\t\tprint_results(vec);\n+\n+\treturn 0;\n+}\ndiff --git a/git.c b/git.c\nindex 33f52ac..ef1e019 100644\n--- a/git.c\n+++ b/git.c\n@@ -444,6 +444,7 @@ static struct cmd_struct commands[] = {\n \t{ \"interpret-trailers\", cmd_interpret_trailers, RUN_SETUP_GENTLY },\n \t{ \"log\", cmd_log, RUN_SETUP },\n \t{ \"ls-files\", cmd_ls_files, RUN_SETUP | SUPPORT_SUPER_PREFIX },\n+\t{ \"ls-partial\", cmd_ls_partial, RUN_SETUP },\n \t{ \"ls-remote\", cmd_ls_remote, RUN_SETUP_GENTLY },\n \t{ \"ls-tree\", cmd_ls_tree, RUN_SETUP },\n \t{ \"mailinfo\", cmd_mailinfo, RUN_SETUP_GENTLY },\ndiff --git a/partial-utils.c b/partial-utils.c\nnew file mode 100644\nindex 0000000..b75e91e\n--- /dev/null\n+++ b/partial-utils.c\n@@ -0,0 +1,279 @@\n+#include \"cache.h\"\n+#include \"blob.h\"\n+#include \"tree.h\"\n+#include \"commit.h\"\n+#include \"quote.h\"\n+#include \"builtin.h\"\n+#include \"parse-options.h\"\n+#include \"pathspec.h\"\n+#include \"dir.h\"\n+#include \"partial-utils.h\"\n+\n+static struct trace_key trace_partial_utils = TRACE_KEY_INIT(PARTIAL_UTILS);\n+\n+void pu_row_trace(\n+\tconst struct pu_row *row,\n+\tconst char *label)\n+{\n+\ttrace_printf_key(\n+\t\t&trace_partial_utils,\n+\t\t\"%s: %06o %s %.*s\\n\",\n+\t\tlabel,\n+\t\trow->mode,\n+\t\toid_to_hex(&row->oid),\n+\t\t(int)row->fullpath.len,\n+\t\trow->fullpath.buf);\n+}\n+\n+struct pu_row *pu_row_alloc(\n+\tconst unsigned char *sha1,\n+\tconst struct strbuf *base,\n+\tconst char *entryname,\n+\tunsigned mode)\n+{\n+\tstruct pu_row *row = xcalloc(1, sizeof(struct pu_row));\n+\n+\thashcpy(row->oid.hash, sha1);\n+\tstrbuf_init(&row->fullpath, base->len + strlen(entryname) + 1);\n+\tif (base->len)\n+\t\tstrbuf_addbuf(&row->fullpath, base);\n+\tstrbuf_addstr(&row->fullpath, entryname);\n+\trow->mode = mode;\n+\trow->entryname_offset = base->len;\n+\n+\tpu_row_trace(row, \"alloc\");\n+\n+\treturn row;\n+}\n+\n+struct pu_vec *pu_vec_alloc(\n+\tunsigned int nr_pre_alloc)\n+{\n+\tstruct pu_vec *vec = xcalloc(1, sizeof(struct pu_vec));\n+\n+\tvec->data = xcalloc(nr_pre_alloc, sizeof(struct pu_row *));\n+\tvec->data_alloc = nr_pre_alloc;\n+\n+\treturn vec;\n+}\n+\n+void pu_vec_append(\n+\tstruct pu_vec *vec,\n+\tstruct pu_row *row)\n+{\n+\tALLOC_GROW(vec->data, vec->data_nr + 1, vec->data_alloc);\n+\tvec->data[vec->data_nr++] = row;\n+}\n+\n+static int ls_tree_cb(\n+\tconst unsigned char *sha1,\n+\tstruct strbuf *base,\n+\tconst char *pathname,\n+\tunsigned mode,\n+\tint stage,\n+\tvoid *context)\n+{\n+\tstruct pu_vec *vec = (struct pu_vec *)context;\n+\n+\t/* omit submodules */\n+\tif (S_ISGITLINK(mode))\n+\t\treturn 0;\n+\n+\tpu_vec_append(vec, pu_row_alloc(sha1, base, pathname, mode));\n+\n+\tif (S_ISDIR(mode))\n+\t\treturn READ_TREE_RECURSIVE;\n+\n+\treturn 0;\n+}\n+\n+struct pu_vec *pu_vec_ls_tree(\n+\tstruct tree *tree,\n+\tconst char *prefix,\n+\tconst char **argv)\n+{\n+\tstruct pu_vec *vec;\n+\tstruct pathspec pathspec;\n+\tint k;\n+\n+\tvec = pu_vec_alloc(PU_VEC_DEFAULT_SIZE);\n+\n+\tparse_pathspec(\n+\t\t&pathspec, PATHSPEC_GLOB | PATHSPEC_ICASE | PATHSPEC_EXCLUDE,\n+\t\tPATHSPEC_PREFER_CWD, prefix, argv);\n+\tfor (k = 0; k < pathspec.nr; k++)\n+\t\tpathspec.items[k].nowildcard_len = pathspec.items[k].len;\n+\tpathspec.has_wildcard = 0;\n+\n+\tif (read_tree_recursive(tree, \"\", 0, 0, &pathspec, ls_tree_cb, vec) != 0)\n+\t\tdie(\"Could not read tree\");\n+\n+\treturn vec;\n+}\n+\n+int pu_load_sparse_definitions(\n+\tconst char *path,\n+\tstruct exclude_list *pel)\n+{\n+\tint result;\n+\tchar *sparse = git_pathdup(\"info/sparse-checkout\");\n+\tmemset(pel, 0, sizeof(*pel));\n+\tresult = add_excludes_from_file_to_list(sparse, \"\", 0, pel, 0);\n+\tfree(sparse);\n+\treturn result;\n+}\n+\n+static int mode_to_dtype(unsigned mode)\n+{\n+\tif (S_ISREG(mode))\n+\t\treturn DT_REG;\n+\tif (S_ISDIR(mode) || S_ISGITLINK(mode))\n+\t\treturn DT_DIR;\n+\tif (S_ISLNK(mode))\n+\t\treturn DT_LNK;\n+\treturn DT_UNKNOWN;\n+}\n+\n+static int apply_excludes_1(\n+\tstruct pu_row **subset,\n+\tunsigned int nr,\n+\tstruct strbuf *prefix,\n+\tstruct exclude_list *pel,\n+\tint defval,\n+\tstruct pu_vec *vec_out);\n+\n+/* apply directory rules. based on clear_ce_flags_dir() */\n+static int apply_excludes_dir(\n+\tstruct pu_row **subset,\n+\tunsigned int nr,\n+\tstruct strbuf *prefix,\n+\tchar *basename,\n+\tstruct exclude_list *pel,\n+\tint defval,\n+\tstruct pu_vec *vec_out)\n+{\n+\tstruct pu_row **subset_end;\n+\tint dtype = DT_DIR;\n+\tint ret = is_excluded_from_list(\n+\t\tprefix->buf, prefix->len, basename, &dtype, pel);\n+\tint rc;\n+\n+\tstrbuf_addch(prefix, '/');\n+\n+\tif (ret < 0)\n+\t\tret = defval;\n+\n+\tfor (subset_end = subset; subset_end != subset + nr; subset_end++) {\n+\t\tstruct pu_row *row = *subset_end;\n+\t\tif (strncmp(row->fullpath.buf, prefix->buf, prefix->len))\n+\t\t\tbreak;\n+\t}\n+\n+\trc = apply_excludes_1(\n+\t\tsubset, subset_end - subset,\n+\t\tprefix, pel, ret,\n+\t\tvec_out);\n+\tstrbuf_setlen(prefix, prefix->len - 1);\n+\treturn rc;\n+}\n+\n+/* apply sparse rules to subset[0..nr). based on clear_ce_flags_1() */\n+static int apply_excludes_1(\n+\tstruct pu_row **subset,\n+\tunsigned int nr,\n+\tstruct strbuf *prefix,\n+\tstruct exclude_list *pel,\n+\tint defval,\n+\tstruct pu_vec *vec_out)\n+{\n+\tstruct pu_row **subset_end = subset + nr;\n+\n+\twhile (subset != subset_end) {\n+\t\tstruct pu_row *row = *subset;\n+\t\tconst char *name, *slash;\n+\t\tint len, dtype, val;\n+\n+\t\tif (prefix->len && strncmp(row->fullpath.buf, prefix->buf, prefix->len))\n+\t\t\tbreak;\n+\n+\t\tname = row->fullpath.buf + prefix->len;\n+\t\tslash = strchr(name, '/');\n+\n+\t\tif (slash) {\n+\t\t\tint processed;\n+\n+\t\t\tlen = slash - name;\n+\t\t\tstrbuf_add(prefix, name, len);\n+\n+\t\t\tprocessed = apply_excludes_dir(\n+\t\t\t\tsubset, subset_end - subset,\n+\t\t\t\tprefix, prefix->buf + prefix->len - len,\n+\t\t\t\tpel, defval,\n+\t\t\t\tvec_out);\n+\n+\t\t\tif (processed) {\n+\t\t\t\tsubset += processed;\n+\t\t\t\tstrbuf_setlen(prefix, prefix->len - len);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tstrbuf_addch(prefix, '/');\n+\t\t\tsubset += apply_excludes_1(\n+\t\t\t\tsubset, subset_end - subset,\n+\t\t\t\tprefix, pel, defval,\n+\t\t\t\tvec_out);\n+\t\t\tstrbuf_setlen(prefix, prefix->len - len - 1);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tdtype = mode_to_dtype(row->mode);\n+\t\tval = is_excluded_from_list(\n+\t\t\trow->fullpath.buf, row->fullpath.len, name, &dtype, pel);\n+\t\tif (val < 0)\n+\t\t\tval = defval;\n+\t\tif (val > 0) {\n+\t\t\tpu_row_trace(row, \"sparse\");\n+\t\t\tpu_vec_append(vec_out, row);\n+\t\t}\n+\t\tsubset++;\n+\t}\n+\n+\treturn nr - (subset_end - subset);\n+}\n+\n+struct pu_vec *pu_vec_filter_sparse(\n+\tconst struct pu_vec *vec_in,\n+\tstruct exclude_list *pel)\n+{\n+\tstruct pu_vec *vec_out;\n+\tstruct strbuf prefix = STRBUF_INIT;\n+\tint defval = 0;\n+\n+\tvec_out = pu_vec_alloc(vec_in->data_nr);\n+\n+\tapply_excludes_1(\n+\t\tvec_in->data, vec_in->data_nr,\n+\t\t&prefix, pel, defval,\n+\t\tvec_out);\n+\n+\treturn vec_out;\n+}\n+\n+struct pu_vec *pu_vec_filter_missing(\n+\tconst struct pu_vec *vec_in)\n+{\n+\tstruct pu_vec *vec_out;\n+\tint k;\n+\n+\tvec_out = pu_vec_alloc(vec_in->data_nr);\n+\n+\tfor (k = 0; k < vec_in->data_nr; k++) {\n+\t\tstruct pu_row *row = vec_in->data[k];\n+\t\tif (!has_sha1_file(row->oid.hash)) {\n+\t\t\tpu_row_trace(row, \"missing\");\n+\t\t\tpu_vec_append(vec_out, row);\n+\t\t}\n+\t}\n+\n+\treturn vec_out;\n+}\ndiff --git a/partial-utils.h b/partial-utils.h\nnew file mode 100644\nindex 0000000..3bdf2e4\n--- /dev/null\n+++ b/partial-utils.h\n@@ -0,0 +1,93 @@\n+#ifndef PARTIAL_UTILS_H\n+#define PARTIAL_UTILS_H\n+\n+/*\n+ * A 'partial-utils row' represents a single item in the tree.\n+ * This is conceptually equivalent to a cache_entry, but does\n+ * not require an index_state and lets us operate on any commit\n+ * and not be tied to the current worktree.\n+ */\n+struct pu_row\n+{\n+\tstruct strbuf fullpath;\n+\tstruct object_id oid;\n+\tunsigned mode;\n+\tunsigned entryname_offset;\n+};\n+\n+/*\n+ * A 'partial-utils vec' represents a vector of 'pu row'\n+ * values using the normal vector machinery.\n+ */\n+struct pu_vec\n+{\n+\tstruct pu_row **data;\n+\tunsigned int data_nr;\n+\tunsigned int data_alloc;\n+};\n+\n+#define PU_VEC_DEFAULT_SIZE (1024*1024)\n+\n+\n+void pu_row_trace(\n+\tconst struct pu_row *row,\n+\tconst char *label);\n+\n+struct pu_row *pu_row_alloc(\n+\tconst unsigned char *sha1,\n+\tconst struct strbuf *base,\n+\tconst char *entryname,\n+\tunsigned mode);\n+\n+struct pu_vec *pu_vec_alloc(\n+\tunsigned int nr_pre_alloc);\n+\n+/*\n+ * Append the given row onto the vector WITHOUT\n+ * assuming ownership of the pointer.\n+ */\n+void pu_vec_append(\n+\tstruct pu_vec *vec,\n+\tstruct pu_row *row);\n+\n+/*\n+ * Enumerate the contents of the tree (recursively) into\n+ * a vector of rows. This is essentially \"ls-tree -r -t\"\n+ * into a vector.\n+ */ \n+struct pu_vec *pu_vec_ls_tree(\n+\tstruct tree *tree,\n+\tconst char *prefix,\n+\tconst char **argv);\n+\n+/*\n+ * Load a sparse-checkout file into (*pel).\n+ * Returns -1 if none or error.\n+ */\n+int pu_load_sparse_definitions(\n+\tconst char *path,\n+\tstruct exclude_list *pel);\n+\n+/*\n+ * Filter the given vector using the sparse-checkout\n+ * definitions and return new vector of just the paths\n+ * that WOULD BE populated.\n+ *\n+ * The returned vector BORROWS rows from the input vector.\n+ *\n+ * This is loosely based upon clear_ce_flags() in unpack-trees.c\n+ */\n+struct pu_vec *pu_vec_filter_sparse(\n+\tconst struct pu_vec *vec_in,\n+\tstruct exclude_list *pel);\n+\n+/*\n+ * Filter the given vector and return the list of blobs\n+ * missing from the local ODB.\n+ *\n+ * The returned vector BORROWS rows from the input vector.\n+ */\n+struct pu_vec *pu_vec_filter_missing(\n+\tconst struct pu_vec *vec_in);\n+\n+#endif /* PARTIAL_UTILS_H */\n-- \n2.7.4\n\n"},{"id":"313533","messageId":"xmqqh93338s2.fsf@gitster.mtv.corp.google.com","threadId":"45317","inReplyTo":"1488994685-37403-3-git-send-email-jeffhost@microsoft.com","subject":"Re: [PATCH 02/10] pack-objects: add --partial-by-size=n --partial-special","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-03-08T18:47:41Z","receivedAt":"2017-03-08T18:48:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff Hostetler writes:\n\n> From: Jeff Hostetler \n>\n> Teach pack-objects to omit blobs from the generated packfile.\n>\n> When the --partial-by-size=n[kmg] argument is used, only blobs\n> smaller than the requested size are included. When n is zero,\n> no blobs are included.\n\nDoes this interact with a more traditional way of feeding output of\nan external \"rev-list --objects\" to pack-objects via its standard\ninput, and if so, should it (and if not, shouldn't it)? \n\nIt is perfectly OK if the answer is \"this applies only to the case\nwhere we generate the list of objects with internal traversal.\" but\nthat needs to be documented and discussed in the proposed log\nmessage.\n\n> When the --partial-special argument is used, git special files,\n> such as \".gitattributes\" and \".gitignores\" are included.\n\nAnd not .\"gitmodules\"? \n\nWhat happens when we later add \".gitsomethingelse\"?\n\nDo we have to worry about the case where the set of git \"special\nfiles\" (can we have a better name for them please, by the way?)\nunderstood by the sending side and the receiving end is different?\n\nI have a feeling that a mode that makes anything whose name begins\nwith \".git\" excempt from the size based cutoff may generally be\neasier to handle.\n\nI am not sure how \"back-filling\" of a resulting narrow clone would\nsafely be done and how this impacts \"git fsck\" at this point, but if\nthey are solved within this effort, that would be a very welcome\nchange.\n\nThanks.\n"},{"id":"313537","messageId":"xmqqd1dr38f0.fsf@gitster.mtv.corp.google.com","threadId":"45317","inReplyTo":"1488994685-37403-7-git-send-email-jeffhost@microsoft.com","subject":"Re: [PATCH 06/10] rev-list: add --allow-partial option to relax connectivity checks","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-03-08T18:55:31Z","receivedAt":"2017-03-08T19:09:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff Hostetler writes:\n\n> From: Jeff Hostetler \n>\n> Teach rev-list to optionally not complain when there are missing\n> blobs. This is for use following a partial clone or fetch when\n> the server omitted certain blobs.\n\nThis makes it impossible to tell from objects missing by design\n(because we did an --partial-by-size clone earlier, expecting we can\nlater fetch from elsewhere when necessary) and objects inaccessible\nby accident (because you have a repository corruption), no?\n\nEven though I do very much like the basic \"high level\" premise to\nomit often useless large blobs that are buried deep in the history\nwe would not necessarily need from the initial cloning and\nsubsequent fetches, I find it somewhat disturbing that the code\n\"Assume\"s that any missing blob is due to an previous partial clone.\nAdding this option smells like telling the users that they are not\nsupposed to run \"git fsck\" because a partially cloned repository is\ninherently a corrupt repository.\n\nCan't we do a bit better? If we want to make the world safer again,\nwhat additional complexity is required to allow us to tell the\n\"missing by design\" and \"corrupt repository\" apart?\n\nThanks.\n"},{"id":"313540","messageId":"2cc503b4-f6cf-336a-bc3f-f44b1d187eaf@jeffhostetler.com","threadId":"45317","inReplyTo":"xmqqd1dr38f0.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 06/10] rev-list: add --allow-partial option to relax connectivity checks","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-03-08T20:10:54Z","receivedAt":"2017-03-08T20:12:26Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/8/2017 1:55 PM, Junio C Hamano wrote:\n> Jeff Hostetler writes:\n>\n>> From: Jeff Hostetler \n>>\n>> Teach rev-list to optionally not complain when there are missing\n>> blobs. This is for use following a partial clone or fetch when\n>> the server omitted certain blobs.\n>\n> This makes it impossible to tell from objects missing by design\n> (because we did an --partial-by-size clone earlier, expecting we can\n> later fetch from elsewhere when necessary) and objects inaccessible\n> by accident (because you have a repository corruption), no?\n\nRight. It will effectively neuter several commands like\nindex-pack, gc, and fsck WRT missing blobs.\n\n> Even though I do very much like the basic \"high level\" premise to\n> omit often useless large blobs that are buried deep in the history\n> we would not necessarily need from the initial cloning and\n> subsequent fetches, I find it somewhat disturbing that the code\n> \"Assume\"s that any missing blob is due to an previous partial clone.\n> Adding this option smells like telling the users that they are not\n> supposed to run \"git fsck\" because a partially cloned repository is\n> inherently a corrupt repository.\n>\n> Can't we do a bit better? If we want to make the world safer again,\n> what additional complexity is required to allow us to tell the\n> \"missing by design\" and \"corrupt repository\" apart?\n\nI'm open to suggestions here. It would be nice to extend the\nfetch-pack/upload-pack protocol to return a list of the SHAa\n(and maybe the sizes) of the omitted blobs, so that a partial\nclone or fetch would still be able to be integrity checked.\n\nJeff\n\n"},{"id":"313542","messageId":"1c38ffbc-e4a2-32e5-d324-506c2111043e@jeffhostetler.com","threadId":"45317","inReplyTo":"xmqqh93338s2.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 02/10] pack-objects: add --partial-by-size=n --partial-special","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-03-08T20:21:11Z","receivedAt":"2017-03-08T20:32:04Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/8/2017 1:47 PM, Junio C Hamano wrote:\n> Jeff Hostetler writes:\n>\n>> From: Jeff Hostetler \n>>\n>> Teach pack-objects to omit blobs from the generated packfile.\n>>\n>> When the --partial-by-size=n[kmg] argument is used, only blobs\n>> smaller than the requested size are included. When n is zero,\n>> no blobs are included.\n>\n> Does this interact with a more traditional way of feeding output of\n> an external \"rev-list --objects\" to pack-objects via its standard\n> input, and if so, should it (and if not, shouldn't it)?\n>\n> It is perfectly OK if the answer is \"this applies only to the case\n> where we generate the list of objects with internal traversal.\" but\n> that needs to be documented and discussed in the proposed log\n> message.\n>\n\nLet me study that and see. I'm still thinking thru ways and\noptions for doing the sparse-checkout like filtering.\n\n\n>> When the --partial-special argument is used, git special files,\n>> such as \".gitattributes\" and \".gitignores\" are included.\n>\n> And not .\"gitmodules\"?\n>\n> What happens when we later add \".gitsomethingelse\"?\n>\n> Do we have to worry about the case where the set of git \"special\n> files\" (can we have a better name for them please, by the way?)\n> understood by the sending side and the receiving end is different?\n>\n> I have a feeling that a mode that makes anything whose name begins\n> with \".git\" excempt from the size based cutoff may generally be\n> easier to handle.\n\nI forgot about \".gitmodules\". The more I think about it, maybe\nwe should always include them (or anything starting with \".git*\")\nand ignore the size, since they are important for correct behavior.\n\n\n> I am not sure how \"back-filling\" of a resulting narrow clone would\n> safely be done and how this impacts \"git fsck\" at this point, but if\n> they are solved within this effort, that would be a very welcome\n> change.\n>\n> Thanks.\n>\n"},{"id":"313614","messageId":"20170309070128.we6hbraea5am3ado@sigill.intra.peff.net","threadId":"45317","inReplyTo":"1488994685-37403-2-git-send-email-jeffhost@microsoft.com","subject":"Re: [PATCH 01/10] pack-objects: eat CR in addition to LF after fgets.","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-03-09T07:01:28Z","receivedAt":"2017-03-09T07:08:17Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Mar 08, 2017 at 05:37:56PM +0000, Jeff Hostetler wrote:\n\n> From: Jeff Hostetler \n> \n> Signed-off-by: Jeff Hostetler \n> ---\n> builtin/pack-objects.c | 2 ++\n> 1 file changed, 2 insertions(+)\n> \n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index f294dcf..7e052bb 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -2764,6 +2764,8 @@ static void get_object_list(int ac, const char **av)\n> \t\tint len = strlen(line);\n> \t\tif (len && line[len - 1] == '\\n')\n> \t\t\tline[--len] = 0;\n> +\t\tif (len && line[len - 1] == '\\r')\n> +\t\t\tline[--len] = 0;\n\nRather than add features to this bespoke line-reader, can we switch this\nto use strbuf_getline()? That handles line endings, and avoids the\nawkward corner case where fgets \"breaks\" a long line across two calls.\n\nSomething like the patch below. I suspect read_object_list_from_stdin()\nshould get the same treatment.\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 76b1919ca..6b9fffe9c 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2765,7 +2765,7 @@ static void record_recent_commit(struct commit *commit, void *data)\n static void get_object_list(int ac, const char **av)\n {\n \tstruct rev_info revs;\n-\tchar line[1000];\n+\tstruct strbuf buf = STRBUF_INIT;\n \tint flags = 0;\n \n \tinit_revisions(&revs, NULL);\n@@ -2775,12 +2775,12 @@ static void get_object_list(int ac, const char **av)\n \t/* make sure shallows are read */\n \tis_repository_shallow();\n \n-\twhile (fgets(line, sizeof(line), stdin) != NULL) {\n-\t\tint len = strlen(line);\n-\t\tif (len && line[len - 1] == '\\n')\n-\t\t\tline[--len] = 0;\n-\t\tif (!len)\n+\twhile (strbuf_getline(&buf, stdin) != EOF) {\n+\t\tconst char *line = buf.buf;\n+\n+\t\tif (!buf.len)\n \t\t\tbreak;\n+\n \t\tif (*line == '-') {\n \t\t\tif (!strcmp(line, \"--not\")) {\n \t\t\t\tflags ^= UNINTERESTING;\n@@ -2800,6 +2800,7 @@ static void get_object_list(int ac, const char **av)\n \t\tif (handle_revision_arg(line, &revs, flags, REVARG_CANNOT_BE_FILENAME))\n \t\t\tdie(\"bad revision '%s'\", line);\n \t}\n+\tstrbuf_release(&buf);\n \n \tif (use_bitmap_index && !get_object_list_from_bitmap(&revs))\n \t\treturn;\n\n"},{"id":"313615","messageId":"20170309070438.r5g2h5vlmgzds4kp@sigill.intra.peff.net","threadId":"45317","inReplyTo":"1c38ffbc-e4a2-32e5-d324-506c2111043e@jeffhostetler.com","subject":"Re: [PATCH 02/10] pack-objects: add --partial-by-size=n --partial-special","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-03-09T07:04:39Z","receivedAt":"2017-03-09T07:11:43Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Mar 08, 2017 at 03:21:11PM -0500, Jeff Hostetler wrote:\n\n> > And not .\"gitmodules\"?\n> > \n> > What happens when we later add \".gitsomethingelse\"?\n> > \n> > Do we have to worry about the case where the set of git \"special\n> > files\" (can we have a better name for them please, by the way?)\n> > understood by the sending side and the receiving end is different?\n> > \n> > I have a feeling that a mode that makes anything whose name begins\n> > with \".git\" excempt from the size based cutoff may generally be\n> > easier to handle.\n> \n> I forgot about \".gitmodules\". The more I think about it, maybe\n> we should always include them (or anything starting with \".git*\")\n> and ignore the size, since they are important for correct behavior.\n\nI'm also in favor of staking out \".git*\" as \"this is special and belongs\nto Git\".\n\nA while back when we discussed whether to allow symlinks for\n.gitattributes, etc, I think the consensus was to treat the whole\n\".git*\" namespace consistently. I haven't followed up with patches yet,\nbut my plan was to go that route.\n\n-Peff\n"},{"id":"313616","messageId":"20170309073516.rksys3jljadupxtc@sigill.intra.peff.net","threadId":"45317","inReplyTo":"1488994685-37403-4-git-send-email-jeffhost@microsoft.com","subject":"Re: [PATCH 03/10] pack-objects: test for --partial-by-size --partial-special","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-03-09T07:35:17Z","receivedAt":"2017-03-09T07:35:24Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Mar 08, 2017 at 05:37:58PM +0000, Jeff Hostetler wrote:\n\n> diff --git a/t/5316-pack-objects-partial.sh b/t/5316-pack-objects-partial.sh\n> [...]\n> +test_expect_success 'setup' '\n> +\tperl -e \"print \\\"a\\\" x 11;\" > a &&\n> +\tperl -e \"print \\\"a\\\" x 1100;\" > b &&\n> +\tperl -e \"print \\\"a\\\" x 1100000;\" > c &&\n> +\techo \"ignored\" > .gitignore &&\n> +\tgit add a b c .gitignore &&\n> +\tgit commit -m test\n> +\t'\n\nA few minor style nits. We usually prefer \">a\" with no space, and the\nclosing single-quote isn't indented.\n\n> +test_expect_success 'special' '\n> +\tgit pack-objects --revs --thin --stdout --partial-special >spec.pack <<-EOF &&\n> +\tmaster\n> +\t\n> +\tEOF\n> +\tgit index-pack spec.pack &&\n> +\ttest 1 = $(git verify-pack -v spec.pack | grep blob | wc -l)\n> +\t'\n\nAll of the tests make sense to me except this one. I see from the code\nin pack-objects why this returns only the .gitattributes file. I'm just\nnot clear on whether that would ever be useful. I guess it lets you ask\n\"give me only the special files\", but again, that seems kind of weird if\nyou are not otherwise limiting.\n\n-Peff\n"},{"id":"313618","messageId":"20170309074849.ktl5vqbzkiwwwbob@sigill.intra.peff.net","threadId":"45317","inReplyTo":"1488994685-37403-5-git-send-email-jeffhost@microsoft.com","subject":"Re: [PATCH 04/10] upload-pack: add partial (sparse) fetch","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-03-09T07:48:49Z","receivedAt":"2017-03-09T07:49:50Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Mar 08, 2017 at 05:37:59PM +0000, Jeff Hostetler wrote:\n\n> diff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt\n> index c59ac99..0032729 100644\n> --- a/Documentation/technical/pack-protocol.txt\n> +++ b/Documentation/technical/pack-protocol.txt\n> @@ -212,6 +212,7 @@ out of what the server said it could do with the first 'want' line.\n> upload-request = want-list\n> \t\t *shallow-line\n> \t\t *1depth-request\n> +\t\t *partial\n> \t\t flush-pkt\n> \n> want-list = first-want\n> @@ -223,10 +224,15 @@ out of what the server said it could do with the first 'want' line.\n> \t\t PKT-LINE(\"deepen-since\" SP timestamp) /\n> \t\t PKT-LINE(\"deepen-not\" SP ref)\n> \n> + partial = PKT-LINE(\"partial-by-size\" SP magnitude) /\n> +\t\t PKT-LINE(\"partial-special) \n> +\n\nI probably would have added this as a capability coming back from the\nclient, since it only makes sense to send once (the same way we ask for\nother features like include-tag or ofs-delta). I guess it's six of one,\nhalf a dozen of the other, though.\n\nI notice that you require the client to request the \"partial\" capability\n_and_ to send these commands. I'm not sure what the client capability\nresponse is helping. The server has said \"I can do this\" and the client\neither asks for it or not.\n\n> +\t\tif (skip_prefix(line, \"partial-by-size \", &arg)) {\n> +\t\t\tunsigned long s;\n> +\t\t\tif (!client_requested_partial_capability)\n> +\t\t\t\tdie(\"git upload-pack: 'partial-by-size' option requires 'partial' capability\");\n> +\t\t\tif (!git_parse_ulong(arg, &s))\n> +\t\t\t\tdie(\"git upload-pack: invalid partial-by-size value: %s\", line);\n> +\t\t\tstrbuf_addstr(&partial_by_size, \"--partial-by-size=\");\n> +\t\t\tstrbuf_addstr(&partial_by_size, arg);\n> +\t\t\thave_partial_by_size = 1;\n> +\t\t\tcontinue;\n\nSo we parse it here for validation, but then pass the original string on\nto be parsed again by pack-objects. I think I'd rather see us use the\nresult of our parse here, just to avoid any bugs where the parsing isn't\nidentical (and there is such a bug currently due to the signed/unsigned\nthing I mentioned).\n\nI also wonder whether the magnitude suffixes are worth exposing across\nthe wire. Anybody touching the list of units in git_parse_ulong() would\nprobably be surprised that the protocol is dependent on them (not that I\nexpect us to really take any away, but it just seems like an unnecessary\nprotocol complication).\n\n-Peff\n"},{"id":"313619","messageId":"20170309075642.jy5o353ann524k7f@sigill.intra.peff.net","threadId":"45317","inReplyTo":"2cc503b4-f6cf-336a-bc3f-f44b1d187eaf@jeffhostetler.com","subject":"Re: [PATCH 06/10] rev-list: add --allow-partial option to relax connectivity checks","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-03-09T07:56:42Z","receivedAt":"2017-03-09T07:56:51Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Mar 08, 2017 at 03:10:54PM -0500, Jeff Hostetler wrote:\n\n> > Even though I do very much like the basic \"high level\" premise to\n> > omit often useless large blobs that are buried deep in the history\n> > we would not necessarily need from the initial cloning and\n> > subsequent fetches, I find it somewhat disturbing that the code\n> > \"Assume\"s that any missing blob is due to an previous partial clone.\n> > Adding this option smells like telling the users that they are not\n> > supposed to run \"git fsck\" because a partially cloned repository is\n> > inherently a corrupt repository.\n> > \n> > Can't we do a bit better? If we want to make the world safer again,\n> > what additional complexity is required to allow us to tell the\n> > \"missing by design\" and \"corrupt repository\" apart?\n> \n> I'm open to suggestions here. It would be nice to extend the\n> fetch-pack/upload-pack protocol to return a list of the SHAa\n> (and maybe the sizes) of the omitted blobs, so that a partial\n> clone or fetch would still be able to be integrity checked.\n\nYeah, the early external-odb patches did this. It lets you do a more\naccurate fsck, and it also helps diff avoid faulting in large-object\ncases (because we can mark them as binary for \"free\" by comparing the\nsize to big_file_threshold).\n\nSo I think it makes a lot of sense in the large-blob case, where\ntransmitting a type/size/sha1 tuple is way more efficient than sending\nthe blob itself. But it's less clear for \"sparse\" cases where just\nenumerating the set of blobs may be prohibitively large.\n\nI have a feeling that the \"sparse\" thing needs to be handled separately\nfrom \"partial\". IOW, the client needs to tell the server \"I'm only\ninterested in the path foo/bar, so just send that\". Then you don't find\nout about the types and sizes outside of that path, but you don't need\nto; the sparse path is stored locally and fsck knows to avoid looking\ninto it.\n\n-Peff\n"},{"id":"313620","messageId":"20170309073117.g3br5btsfwntcdpe@sigill.intra.peff.net","threadId":"45317","inReplyTo":"1488994685-37403-3-git-send-email-jeffhost@microsoft.com","subject":"Re: [PATCH 02/10] pack-objects: add --partial-by-size=n --partial-special","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-03-09T07:31:18Z","receivedAt":"2017-03-09T08:01:08Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Mar 08, 2017 at 05:37:57PM +0000, Jeff Hostetler wrote:\n\n> From: Jeff Hostetler \n> \n> Teach pack-objects to omit blobs from the generated packfile.\n> \n> When the --partial-by-size=n[kmg] argument is used, only blobs\n> smaller than the requested size are included. When n is zero,\n> no blobs are included.\n> \n> When the --partial-special argument is used, git special files,\n> such as \".gitattributes\" and \".gitignores\" are included.\n> \n> When both are given, the union of two are included.\n\nI understand why one would want to do:\n\n --partial-by-size=100 --partial-special\n\nand get the union. The first one restricts, and the second one adds back\nin. But I don't understand why \"--partial-special\" by itself makes any\nsense. Wouldn't we already be including all blobs, and it would be a\nnoop?\n\n\nAlso, I was thinking a bit on Junio's comment elsewhere on whether\nread_object_list_from_stdin() should do the same limiting. I think the\nanswer is \"probably not\", because whoever is generating that object list\ncan cull the set. You could do it today with something like:\n\n git rev-list --objects HEAD |\n git cat-file --batch-check='%(objectsize) %(objecttype) %(objectname) %(rest)' |\n perl -lne 's/^(\\d+) (\\S+) //; print if $2 ne \"blob\" || $1 < 100' |\n git pack-objects\n\nBut if we are going to add this --partial-by-size for the pack-objects\ntraversal, shouldn't we just add it to rev-list? Then:\n\n git rev-list --objects --partial-by-size=100 --partial-special |\n git pack-objects\n\nworks, and you should get it in the pack-objects basically for free (I\nthink you'd have to allow through the \"--partial\" arguments on stdin,\nand make sure the rev-list implementation is done via\ntraverse_commit_list).\n\nAs a bonus, I suspect it would make the --partial-special path-handling\neasier, because you'd see each tree entry rather than the fully\nconstructed path (so no more monkeying around with \"/\").\n\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 7e052bb..2df2f49 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -77,6 +77,10 @@ static unsigned long cache_max_small_delta_size = 1000;\n> \n> static unsigned long window_memory_limit = 0;\n> \n> +static signed long partial_by_size = -1;\n\nI would have expected this to be an off_t, though I think\nOPT_MAGNITUDE() forces you into \"unsigned long\". I guess it is nothing\nnew for Git; we use \"unsigned long\" for single object sizes elsewhere,\nso systems with a 32-bit long are out of luck anyway until we fix that.\n\nThe signed \"long\" here is unfortunate, as it limits us to 2G on such\nsystems. Maybe it is not worth worrying too much about. The \"big object\"\nthreshold is usually around 500MB. I think the failure behavior is not\ngreat, though (asking for \"3G\" would go negative and effectively be\nignored).\n\nI think handling all cases would involve swapping out OPT_MAGNITUDE()\nfor a special callback that writes the \"yes, the user set this\" bit in a\nseparate variable.\n\n-Peff\n"},{"id":"313649","messageId":"52221b43-0c38-97a2-d96d-0dedf5f45796@jeffhostetler.com","threadId":"45317","inReplyTo":"20170309070128.we6hbraea5am3ado@sigill.intra.peff.net","subject":"Re: [PATCH 01/10] pack-objects: eat CR in addition to LF after fgets.","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-03-09T15:46:53Z","receivedAt":"2017-03-09T15:47:19Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/9/2017 2:01 AM, Jeff King wrote:\n> On Wed, Mar 08, 2017 at 05:37:56PM +0000, Jeff Hostetler wrote:\n>\n>> From: Jeff Hostetler \n>>\n>> Signed-off-by: Jeff Hostetler \n>> ---\n>> builtin/pack-objects.c | 2 ++\n>> 1 file changed, 2 insertions(+)\n>>\n>> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n>> index f294dcf..7e052bb 100644\n>> --- a/builtin/pack-objects.c\n>> +++ b/builtin/pack-objects.c\n>> @@ -2764,6 +2764,8 @@ static void get_object_list(int ac, const char **av)\n>> \t\tint len = strlen(line);\n>> \t\tif (len && line[len - 1] == '\\n')\n>> \t\t\tline[--len] = 0;\n>> +\t\tif (len && line[len - 1] == '\\r')\n>> +\t\t\tline[--len] = 0;\n>\n> Rather than add features to this bespoke line-reader, can we switch this\n> to use strbuf_getline()? That handles line endings, and avoids the\n> awkward corner case where fgets \"breaks\" a long line across two calls.\n>\n> Something like the patch below. I suspect read_object_list_from_stdin()\n> should get the same treatment.\n\nMuch nicer. Will do. Thanks!\n\n>\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 76b1919ca..6b9fffe9c 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -2765,7 +2765,7 @@ static void record_recent_commit(struct commit *commit, void *data)\n> static void get_object_list(int ac, const char **av)\n> {\n> \tstruct rev_info revs;\n> -\tchar line[1000];\n> +\tstruct strbuf buf = STRBUF_INIT;\n> \tint flags = 0;\n>\n> \tinit_revisions(&revs, NULL);\n> @@ -2775,12 +2775,12 @@ static void get_object_list(int ac, const char **av)\n> \t/* make sure shallows are read */\n> \tis_repository_shallow();\n>\n> -\twhile (fgets(line, sizeof(line), stdin) != NULL) {\n> -\t\tint len = strlen(line);\n> -\t\tif (len && line[len - 1] == '\\n')\n> -\t\t\tline[--len] = 0;\n> -\t\tif (!len)\n> +\twhile (strbuf_getline(&buf, stdin) != EOF) {\n> +\t\tconst char *line = buf.buf;\n> +\n> +\t\tif (!buf.len)\n> \t\t\tbreak;\n> +\n> \t\tif (*line == '-') {\n> \t\t\tif (!strcmp(line, \"--not\")) {\n> \t\t\t\tflags ^= UNINTERESTING;\n> @@ -2800,6 +2800,7 @@ static void get_object_list(int ac, const char **av)\n> \t\tif (handle_revision_arg(line, &revs, flags, REVARG_CANNOT_BE_FILENAME))\n> \t\t\tdie(\"bad revision '%s'\", line);\n> \t}\n> +\tstrbuf_release(&buf);\n>\n> \tif (use_bitmap_index && !get_object_list_from_bitmap(&revs))\n> \t\treturn;\n>\n"},{"id":"313656","messageId":"afd40132-ef85-1713-e3ab-f6bef6b30646@kdbg.org","threadId":"45317","inReplyTo":"1488994685-37403-4-git-send-email-jeffhost@microsoft.com","subject":"Re: [PATCH 03/10] pack-objects: test for --partial-by-size --partial-special","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2017-03-09T18:11:51Z","receivedAt":"2017-03-09T18:12:00Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 08.03.2017 um 18:37 schrieb Jeff Hostetler:\n> +test_expect_success 'setup' '\n> +\tperl -e \"print \\\"a\\\" x 11;\" > a &&\n> +\tperl -e \"print \\\"a\\\" x 1100;\" > b &&\n> +\tperl -e \"print \\\"a\\\" x 1100000;\" > c &&\n\nIf the file contents do not matter, you can have the same without perl \nlike this:\n\n\tprintf \"%011d\" 0 >a &&\n\tprintf \"%01100d\" 0 >b &&\n\tprintf \"%01100000d\" 0 >c &&\n\n-- Hannes\n\n"},{"id":"313662","messageId":"6157cad3-6bbd-7f2c-15bb-6487089272a7@jeffhostetler.com","threadId":"45317","inReplyTo":"20170309073117.g3br5btsfwntcdpe@sigill.intra.peff.net","subject":"Re: [PATCH 02/10] pack-objects: add --partial-by-size=n --partial-special","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-03-09T18:26:11Z","receivedAt":"2017-03-09T18:26:32Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/9/2017 2:31 AM, Jeff King wrote:\n> On Wed, Mar 08, 2017 at 05:37:57PM +0000, Jeff Hostetler wrote:\n>\n>> From: Jeff Hostetler \n>>\n>> Teach pack-objects to omit blobs from the generated packfile.\n>>\n>> When the --partial-by-size=n[kmg] argument is used, only blobs\n>> smaller than the requested size are included. When n is zero,\n>> no blobs are included.\n>>\n>> When the --partial-special argument is used, git special files,\n>> such as \".gitattributes\" and \".gitignores\" are included.\n>>\n>> When both are given, the union of two are included.\n>\n> I understand why one would want to do:\n>\n> --partial-by-size=100 --partial-special\n>\n> and get the union. The first one restricts, and the second one adds back\n> in. But I don't understand why \"--partial-special\" by itself makes any\n> sense. Wouldn't we already be including all blobs, and it would be a\n> noop?\n\nMy thought was that the \"--partial-special\" when used by itself\nwould *only* give you the .git* files (and if we had something\nlike a .gitsparse/ directory, everything under it). The client\ncould then do a \"special\" clone -- mainly to get the sparse checkout\ntemplates under .gitsparse/ and then come back for a sparse fetch\nusing one of them. Somewhat of a chicken-n-egg problem, unless the\nuser knows the template names in advance.\n\n>\n>\n> Also, I was thinking a bit on Junio's comment elsewhere on whether\n> read_object_list_from_stdin() should do the same limiting. I think the\n> answer is \"probably not\", because whoever is generating that object list\n> can cull the set. You could do it today with something like:\n>\n> git rev-list --objects HEAD |\n> git cat-file --batch-check='%(objectsize) %(objecttype) %(objectname) %(rest)' |\n> perl -lne 's/^(\\d+) (\\S+) //; print if $2 ne \"blob\" || $1 < 100' |\n> git pack-objects\n>\n> But if we are going to add this --partial-by-size for the pack-objects\n> traversal, shouldn't we just add it to rev-list? Then:\n>\n> git rev-list --objects --partial-by-size=100 --partial-special |\n> git pack-objects\n>\n> works, and you should get it in the pack-objects basically for free (I\n> think you'd have to allow through the \"--partial\" arguments on stdin,\n> and make sure the rev-list implementation is done via\n> traverse_commit_list).\n>\n> As a bonus, I suspect it would make the --partial-special path-handling\n> easier, because you'd see each tree entry rather than the fully\n> constructed path (so no more monkeying around with \"/\").\n\nInteresting. Let me give that a try and see what it looks like.\n\n>\n>> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n>> index 7e052bb..2df2f49 100644\n>> --- a/builtin/pack-objects.c\n>> +++ b/builtin/pack-objects.c\n>> @@ -77,6 +77,10 @@ static unsigned long cache_max_small_delta_size = 1000;\n>>\n>> static unsigned long window_memory_limit = 0;\n>>\n>> +static signed long partial_by_size = -1;\n>\n> I would have expected this to be an off_t, though I think\n> OPT_MAGNITUDE() forces you into \"unsigned long\". I guess it is nothing\n> new for Git; we use \"unsigned long\" for single object sizes elsewhere,\n> so systems with a 32-bit long are out of luck anyway until we fix that.\n>\n> The signed \"long\" here is unfortunate, as it limits us to 2G on such\n> systems. Maybe it is not worth worrying too much about. The \"big object\"\n> threshold is usually around 500MB. I think the failure behavior is not\n> great, though (asking for \"3G\" would go negative and effectively be\n> ignored).\n>\n> I think handling all cases would involve swapping out OPT_MAGNITUDE()\n> for a special callback that writes the \"yes, the user set this\" bit in a\n> separate variable.\n\nYeah, there is a bit of confusion there. I used OPT_MAGNITUDE in\none place (for the argument checking), but couldn't in another place.\nAnd I tried to pass the original string across the wire for sanity.\nAnd I had to fight with the types a little. It would probably be\nsimpler to replace that with a custom handler (or a uint64_t version\nof magnitude) that would do the right thing and then use that numeric\nvalue elsewhere.\n\nThanks,\nJeff\n\n"},{"id":"313663","messageId":"ce170fec-846b-fc51-221d-c005b4a42b97@jeffhostetler.com","threadId":"45317","inReplyTo":"20170309074849.ktl5vqbzkiwwwbob@sigill.intra.peff.net","subject":"Re: [PATCH 04/10] upload-pack: add partial (sparse) fetch","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-03-09T18:34:32Z","receivedAt":"2017-03-09T18:35:05Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/9/2017 2:48 AM, Jeff King wrote:\n> On Wed, Mar 08, 2017 at 05:37:59PM +0000, Jeff Hostetler wrote:\n>\n>> diff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt\n>> index c59ac99..0032729 100644\n>> --- a/Documentation/technical/pack-protocol.txt\n>> +++ b/Documentation/technical/pack-protocol.txt\n>> @@ -212,6 +212,7 @@ out of what the server said it could do with the first 'want' line.\n>> upload-request = want-list\n>> \t\t *shallow-line\n>> \t\t *1depth-request\n>> +\t\t *partial\n>> \t\t flush-pkt\n>>\n>> want-list = first-want\n>> @@ -223,10 +224,15 @@ out of what the server said it could do with the first 'want' line.\n>> \t\t PKT-LINE(\"deepen-since\" SP timestamp) /\n>> \t\t PKT-LINE(\"deepen-not\" SP ref)\n>>\n>> + partial = PKT-LINE(\"partial-by-size\" SP magnitude) /\n>> +\t\t PKT-LINE(\"partial-special)\n>> +\n>\n> I probably would have added this as a capability coming back from the\n> client, since it only makes sense to send once (the same way we ask for\n> other features like include-tag or ofs-delta). I guess it's six of one,\n> half a dozen of the other, though.\n\nTrue. I wanted the size argument. And later want to add\na sparse-file. It seemed like they better belonged in a\nPKT-LINE than in the capability header.\n\n>\n> I notice that you require the client to request the \"partial\" capability\n> _and_ to send these commands. I'm not sure what the client capability\n> response is helping. The server has said \"I can do this\" and the client\n> either asks for it or not.\n\nYeah, I wasn't sure if that was necessary or not.\nIt looked like there were other fields where the\nclient advised that it wanted to used a capability\nthat the server had previously advertised. If we\ndon't need it, that's fine.\n\n\n>\n>> +\t\tif (skip_prefix(line, \"partial-by-size \", &arg)) {\n>> +\t\t\tunsigned long s;\n>> +\t\t\tif (!client_requested_partial_capability)\n>> +\t\t\t\tdie(\"git upload-pack: 'partial-by-size' option requires 'partial' capability\");\n>> +\t\t\tif (!git_parse_ulong(arg, &s))\n>> +\t\t\t\tdie(\"git upload-pack: invalid partial-by-size value: %s\", line);\n>> +\t\t\tstrbuf_addstr(&partial_by_size, \"--partial-by-size=\");\n>> +\t\t\tstrbuf_addstr(&partial_by_size, arg);\n>> +\t\t\thave_partial_by_size = 1;\n>> +\t\t\tcontinue;\n>\n> So we parse it here for validation, but then pass the original string on\n> to be parsed again by pack-objects. I think I'd rather see us use the\n> result of our parse here, just to avoid any bugs where the parsing isn't\n> identical (and there is such a bug currently due to the signed/unsigned\n> thing I mentioned).\n>\n> I also wonder whether the magnitude suffixes are worth exposing across\n> the wire. Anybody touching the list of units in git_parse_ulong() would\n> probably be surprised that the protocol is dependent on them (not that I\n> expect us to really take any away, but it just seems like an unnecessary\n> protocol complication).\n\nYeah, I'll change this as I described in an earlier sub-thread.\n\nJeff\n\n"},{"id":"313664","messageId":"cb3418b8-8ecf-a38b-67d3-e73dc28d4f69@jeffhostetler.com","threadId":"45317","inReplyTo":"20170309075642.jy5o353ann524k7f@sigill.intra.peff.net","subject":"Re: [PATCH 06/10] rev-list: add --allow-partial option to relax connectivity checks","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-03-09T18:38:35Z","receivedAt":"2017-03-09T18:39:10Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/9/2017 2:56 AM, Jeff King wrote:\n> On Wed, Mar 08, 2017 at 03:10:54PM -0500, Jeff Hostetler wrote:\n>\n>>> Even though I do very much like the basic \"high level\" premise to\n>>> omit often useless large blobs that are buried deep in the history\n>>> we would not necessarily need from the initial cloning and\n>>> subsequent fetches, I find it somewhat disturbing that the code\n>>> \"Assume\"s that any missing blob is due to an previous partial clone.\n>>> Adding this option smells like telling the users that they are not\n>>> supposed to run \"git fsck\" because a partially cloned repository is\n>>> inherently a corrupt repository.\n>>>\n>>> Can't we do a bit better? If we want to make the world safer again,\n>>> what additional complexity is required to allow us to tell the\n>>> \"missing by design\" and \"corrupt repository\" apart?\n>>\n>> I'm open to suggestions here. It would be nice to extend the\n>> fetch-pack/upload-pack protocol to return a list of the SHAa\n>> (and maybe the sizes) of the omitted blobs, so that a partial\n>> clone or fetch would still be able to be integrity checked.\n>\n> Yeah, the early external-odb patches did this. It lets you do a more\n> accurate fsck, and it also helps diff avoid faulting in large-object\n> cases (because we can mark them as binary for \"free\" by comparing the\n> size to big_file_threshold).\n>\n> So I think it makes a lot of sense in the large-blob case, where\n> transmitting a type/size/sha1 tuple is way more efficient than sending\n> the blob itself. But it's less clear for \"sparse\" cases where just\n> enumerating the set of blobs may be prohibitively large.\n>\n> I have a feeling that the \"sparse\" thing needs to be handled separately\n> from \"partial\". IOW, the client needs to tell the server \"I'm only\n> interested in the path foo/bar, so just send that\". Then you don't find\n> out about the types and sizes outside of that path, but you don't need\n> to; the sparse path is stored locally and fsck knows to avoid looking\n> into it.\n>\n> -Peff\n>\n\nThat makes sense. I'd like to get both concepts (by-size/special vs\nsparse-file) in, but they don't really overlap that much (internally).\nSo I could see doing this in 2 separate efforts.\n\nThanks,\nJeff\n"},{"id":"313667","messageId":"20170309190950.om4wiyy3e4nawkgs@sigill.intra.peff.net","threadId":"45317","inReplyTo":"ce170fec-846b-fc51-221d-c005b4a42b97@jeffhostetler.com","subject":"Re: [PATCH 04/10] upload-pack: add partial (sparse) fetch","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-03-09T19:09:50Z","receivedAt":"2017-03-09T19:09:57Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 09, 2017 at 01:34:32PM -0500, Jeff Hostetler wrote:\n\n> > > + partial = PKT-LINE(\"partial-by-size\" SP magnitude) /\n> > > +\t\t PKT-LINE(\"partial-special)\n> > > +\n> > \n> > I probably would have added this as a capability coming back from the\n> > client, since it only makes sense to send once (the same way we ask for\n> > other features like include-tag or ofs-delta). I guess it's six of one,\n> > half a dozen of the other, though.\n> \n> True. I wanted the size argument. And later want to add\n> a sparse-file. It seemed like they better belonged in a\n> PKT-LINE than in the capability header.\n\nYeah, at some point we will run out of room in the capabilities\nresponse. :) If the sparse information might be arbitrarily long (e.g.,\na list of pathspecs) then it probably is better for it to get split into\nits own pktline (or even a series of pktlines).\n\n-Peff\n"},{"id":"313761","messageId":"20170310175823.GA53198@google.com","threadId":"45317","inReplyTo":"20170309070438.r5g2h5vlmgzds4kp@sigill.intra.peff.net","subject":"Re: [PATCH 02/10] pack-objects: add --partial-by-size=n --partial-special","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-03-10T17:58:23Z","receivedAt":"2017-03-10T17:58:31Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 03/09, Jeff King wrote:\n> On Wed, Mar 08, 2017 at 03:21:11PM -0500, Jeff Hostetler wrote:\n> \n> > > And not .\"gitmodules\"?\n> > > \n> > > What happens when we later add \".gitsomethingelse\"?\n> > > \n> > > Do we have to worry about the case where the set of git \"special\n> > > files\" (can we have a better name for them please, by the way?)\n> > > understood by the sending side and the receiving end is different?\n> > > \n> > > I have a feeling that a mode that makes anything whose name begins\n> > > with \".git\" excempt from the size based cutoff may generally be\n> > > easier to handle.\n> > \n> > I forgot about \".gitmodules\". The more I think about it, maybe\n> > we should always include them (or anything starting with \".git*\")\n> > and ignore the size, since they are important for correct behavior.\n> \n> I'm also in favor of staking out \".git*\" as \"this is special and belongs\n> to Git\".\n\nI agree, .git* files should probably be the bare minimum of files\nincluded. Especially since things like .gitattributes can effect things\nlike checkout.\n\n> \n> A while back when we discussed whether to allow symlinks for\n> .gitattributes, etc, I think the consensus was to treat the whole\n> \".git*\" namespace consistently. I haven't followed up with patches yet,\n> but my plan was to go that route.\n\nWell if I remember correctly you sent out some patches for\n.gitattributes but I got in the way with the refactoring work! :)\n\n-- \nBrandon Williams\n"},{"id":"313762","messageId":"20170310180341.bbzukqrj2zlkje2b@sigill.intra.peff.net","threadId":"45317","inReplyTo":"20170310175823.GA53198@google.com","subject":"Re: [PATCH 02/10] pack-objects: add --partial-by-size=n --partial-special","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-03-10T18:03:41Z","receivedAt":"2017-03-10T18:03:49Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 10, 2017 at 09:58:23AM -0800, Brandon Williams wrote:\n\n> > A while back when we discussed whether to allow symlinks for\n> > .gitattributes, etc, I think the consensus was to treat the whole\n> > \".git*\" namespace consistently. I haven't followed up with patches yet,\n> > but my plan was to go that route.\n> \n> Well if I remember correctly you sent out some patches for\n> .gitattributes but I got in the way with the refactoring work! :)\n\nTrue. :) But those were the old method that tries to treat\n.gitattributes specially, by using O_NOFOLLOW in the attribute code (but\nonly for in-tree files, naturally).\n\nI think we ended up deciding that it would be better to just disallow\nsymlink .gitattributes (and .git*) from entering the index, the way we\ndisallow \".git\".\n\n-Peff\n"},{"id":"313779","messageId":"xmqqzigsylb1.fsf@gitster.mtv.corp.google.com","threadId":"45317","inReplyTo":"20170310180341.bbzukqrj2zlkje2b@sigill.intra.peff.net","subject":"Re: [PATCH 02/10] pack-objects: add --partial-by-size=n --partial-special","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-03-10T19:38:10Z","receivedAt":"2017-03-10T19:38:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> I think we ended up deciding that it would be better to just disallow\n> symlink .gitattributes (and .git*) from entering the index, the way we\n> disallow \".git\".\n\nHmph, I thought we would need both, though. Or do we specifically\nwant to honor untracked .gitattributes that is left as a symlink\npointing to elsewhere in the filesystem or something like that?\n\n"},{"id":"313783","messageId":"20170310194725.36acsc2bdfhl4aih@sigill.intra.peff.net","threadId":"45317","inReplyTo":"xmqqzigsylb1.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 02/10] pack-objects: add --partial-by-size=n --partial-special","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-03-10T19:47:26Z","receivedAt":"2017-03-10T19:47:57Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 10, 2017 at 11:38:10AM -0800, Junio C Hamano wrote:\n\n> Jeff King writes:\n> \n> > I think we ended up deciding that it would be better to just disallow\n> > symlink .gitattributes (and .git*) from entering the index, the way we\n> > disallow \".git\".\n> \n> Hmph, I thought we would need both, though. Or do we specifically\n> want to honor untracked .gitattributes that is left as a symlink\n> pointing to elsewhere in the filesystem or something like that?\n\nI wasn't going to worry about an untracked .gitattributes. The reasons\nfor disallowing symlinked .gitattributes are:\n\n - it doesn't behave the same when accessed internally via the tree\n objects\n\n - malicious symlinks that try to leave the repository\n\nNeither of those issues is at play if your symlink .gitattributes file\nisn't tracked. So there's some inconsistency in the sense that it\n\"works\" until you try to \"git add\" it. But either you aren't going to\nadd it (in which case it's a feature that it works), or you are going to\nadd it, and you'll get notified then that it's disallowed.\n\n-Peff\n"}]}