{"thread":{"id":"46376","subject":"[PATCH v2 00/19] WIP object filtering for partial clone","startedAt":"2017-07-13T17:35:27Z","lastAt":"2017-07-13T17:38:00Z","messageCount":20,"participants":["Jeff Hostetler"],"isPatch":true,"patchVersion":2,"patchTotal":19},"messages":[{"id":"324379","messageId":"20170713173459.3559-1-git@jeffhostetler.com","threadId":"46376","inReplyTo":null,"subject":"[PATCH v2 00/19] WIP object filtering for partial clone","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-07-13T17:34:40Z","receivedAt":"2017-07-13T17:35:27Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nThis WIP is a follow up to my earlier patch series to teach\npack-objects to omit large blobs from packfiles. [1]\n\nLike the previous version, this version builds upon a suggestion from\nPeff [2] to use the traverse_commit_list() machinery to allow custom\nobject filtering using a filter callback. This hides the filtering\nlogic in list-objects.c and list-objects-filters.c and minimizes the\nchanges to actual commands, such as pack-objects.\n\nThis version adds that same filtering capability to rev-list allowing\nfiltering to be demonstrated without building a packfile. Filtered\nblobs are printed with a leading \"~\" (along with their sizes).\n\n $ ./git rev-list --objects HEAD~1..HEAD\n 74f806c70507317b8bdbcf3b08459c7c83906bee\n 818617707aac81ae4620239182b514f65638e37e \n d21329bffeb9801682d8d6d6acedc2958d17f4e0 builtin\n 306c16551e548ace12c709a332bfea22adcc395f builtin/fetch.c\n\n $ ./git rev-list --objects --filter-omit-all-blobs --filter-print-manifest HEAD~1..HEAD\n 74f806c70507317b8bdbcf3b08459c7c83906bee\n 818617707aac81ae4620239182b514f65638e37e \n d21329bffeb9801682d8d6d6acedc2958d17f4e0 builtin\n ~306c16551e548ace12c709a332bfea22adcc395f 40732\n\n $ ./git rev-list --objects --filter-omit-all-blobs --filter-print-manifest --quiet HEAD~1..HEAD\n ~306c16551e548ace12c709a332bfea22adcc395f 40732\n\nThis version contains 3 filters:\n1. filter-omit-all-blobs to exclude all blobs (trees and commits only).\n\n2. filter-omit-large-blobs=[kmg] to exclude blobs larger than \n (but always including \".git*\" special files).\n\n3. filter-use-sparse= to exclude blobs not needed by the\n corresponding sparse-checkout.\n\nSparse-checkout filtering is currently limited to filtering unneeded blobs.\nA later enhancement should be able to also filter unneeded tree objects.\n\nThis version updates clone, fetch, fetch-pack, and upload-pack commands\nto pass the additional object-filter parameters.\n\nAs a (possibly) temporary measure, some commands have been updated to\nrelax missing blob errors during consistency checks. Maintining info\non missing blobs is currently being discussed in [3].\n\nTODO\n1. Incorporate with a patch series like [4] to dynamically fetch a\n missing blob from the server in read_object on demand.\n2. Resolve missing blob consistency check issue.\n3. Store filter options from clone in config or .git/info and default\n to them in subsequent fetches.\n4. fsck, gc, and assorted commands.\n5. testing.\n\n\n[1] https://public-inbox.org/git/20170622203615.34135-1-git@jeffhostetler.com/\n[2] https://public-inbox.org/git/20170309073117.g3br5btsfwntcdpe@sigill.intra.peff.net/\n[3] https://public-inbox.org/git/cover.1499800530.git.jonathantanmy@google.com/\n[4] https://public-inbox.org/git/20170505152802.6724-1-benpeart@microsoft.com/\n\n\nJeff Hostetler (19):\n dir: refactor add_excludes()\n oidset2: create oidset subclass with object length and pathname\n list-objects: filter objects in traverse_commit_list\n list-objects-filters: add omit-all-blobs filter\n list-objects-filters: add omit-large-blobs filter\n list-objects-filters: add use-sparse-checkout filter\n object-filter: common declarations for object filtering\n rev-list: add object filtering support\n rev-list: add filtering help text\n t6112: rev-list object filtering test\n pack-objects: add object filtering support\n pack-objects: add filtering help text\n upload-pack: add filter-objects to protocol documentation\n upload-pack: add object filtering\n fetch-pack: add object filtering support\n connected: add filter_allow_omitted option to API\n clone: add filter arguments\n index-pack: relax consistency checks for omitted objects\n fetch: add object filtering to fetch\n\n Documentation/git-pack-objects.txt | 14 +\n Documentation/git-rev-list.txt | 7 +-\n Documentation/rev-list-options.txt | 26 ++\n Documentation/technical/pack-protocol.txt | 16 +\n Documentation/technical/protocol-capabilities.txt | 7 +\n Makefile | 3 +\n builtin/clone.c | 28 ++\n builtin/fetch-pack.c | 3 +\n builtin/fetch.c | 27 +-\n builtin/index-pack.c | 15 +\n builtin/pack-objects.c | 33 +-\n builtin/rev-list.c | 58 +++-\n connected.c | 3 +\n connected.h | 6 +\n dir.c | 53 +++-\n dir.h | 4 +\n fetch-pack.c | 28 ++\n fetch-pack.h | 2 +\n list-objects-filters.c | 361 ++++++++++++++++++++++\n list-objects-filters.h | 45 +++\n list-objects.c | 66 +++-\n list-objects.h | 30 ++\n object-filter.c | 201 ++++++++++++\n object-filter.h | 145 +++++++++\n oidset2.c | 101 ++++++\n oidset2.h | 56 ++++\n t/t6112-rev-list-filters-objects.sh | 37 +++\n transport.c | 27 ++\n transport.h | 8 +\n upload-pack.c | 39 ++-\n 30 files changed, 1425 insertions(+), 24 deletions(-)\n create mode 100644 list-objects-filters.c\n create mode 100644 list-objects-filters.h\n create mode 100644 object-filter.c\n create mode 100644 object-filter.h\n create mode 100644 oidset2.c\n create mode 100644 oidset2.h\n create mode 100644 t/t6112-rev-list-filters-objects.sh\n\n-- \n2.9.3\n\n"},{"id":"324380","messageId":"20170713173459.3559-2-git@jeffhostetler.com","threadId":"46376","inReplyTo":"20170713173459.3559-1-git@jeffhostetler.com","subject":"[PATCH v2 01/19] dir: refactor add_excludes()","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-07-13T17:34:41Z","receivedAt":"2017-07-13T17:35:36Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRefactor add_excludes() to separate the reading of the\nexclude file into a buffer and the parsing of the buffer\ninto exclude_list items.\n\nAdd add_excludes_from_blob_to_list() to allow an exclude\nfile be specified with an OID.\n\nSigned-off-by: Jeff Hostetler \n---\n dir.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++--\n dir.h | 4 ++++\n 2 files changed, 55 insertions(+), 2 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 31f9343..aeba965 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -725,6 +725,11 @@ static void invalidate_directory(struct untracked_cache *uc,\n \t\tdir->dirs[i]->recurse = 0;\n }\n \n+static int add_excludes_from_buffer(\n+\tchar *buf, size_t size,\n+\tconst char *base, int baselen,\n+\tstruct exclude_list *el);\n+\n /*\n * Given a file with name \"fname\", read it (either from disk, or from\n * the index if \"check_index\" is non-zero), parse it and store the\n@@ -739,9 +744,9 @@ static int add_excludes(const char *fname, const char *base, int baselen,\n \t\t\tstruct sha1_stat *sha1_stat)\n {\n \tstruct stat st;\n-\tint fd, i, lineno = 1;\n+\tint fd;\n \tsize_t size = 0;\n-\tchar *buf, *entry;\n+\tchar *buf;\n \n \tfd = open(fname, O_RDONLY);\n \tif (fd < 0 || fstat(fd, &st) < 0) {\n@@ -798,6 +803,18 @@ static int add_excludes(const char *fname, const char *base, int baselen,\n \t\t}\n \t}\n \n+\tadd_excludes_from_buffer(buf, size, base, baselen, el);\n+\treturn 0;\n+}\n+\n+static int add_excludes_from_buffer(\n+\tchar *buf, size_t size,\n+\tconst char *base, int baselen,\n+\tstruct exclude_list *el)\n+{\n+\tint i, lineno = 1;\n+\tchar *entry;\n+\n \tel->filebuf = buf;\n \n \tif (skip_utf8_bom(&buf, size))\n@@ -826,6 +843,38 @@ int add_excludes_from_file_to_list(const char *fname, const char *base,\n \treturn add_excludes(fname, base, baselen, el, check_index, NULL);\n }\n \n+int add_excludes_from_blob_to_list(\n+\tstruct object_id *oid,\n+\tconst char *base, int baselen,\n+\tstruct exclude_list *el)\n+{\n+\tchar *buf;\n+\tunsigned long size;\n+\tenum object_type type;\n+\n+\tbuf = read_sha1_file(oid->hash, &type, &size);\n+\tif (!buf)\n+\t\treturn -1;\n+\n+\tif (type != OBJ_BLOB) {\n+\t\tfree(buf);\n+\t\treturn -1;\n+\t}\n+\n+\tif (size == 0) {\n+\t\tfree(buf);\n+\t\treturn 0;\n+\t}\n+\n+\tif (buf[size - 1] != '\\n') {\n+\t\tbuf = xrealloc(buf, st_add(size, 1));\n+\t\tbuf[size++] = '\\n';\n+\t}\n+\n+\tadd_excludes_from_buffer(buf, size, base, baselen, el);\n+\treturn 0;\n+}\n+\n struct exclude_list *add_exclude_list(struct dir_struct *dir,\n \t\t\t\t int group_type, const char *src)\n {\ndiff --git a/dir.h b/dir.h\nindex edb5fda..8e754e5 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -242,6 +242,10 @@ extern struct exclude_list *add_exclude_list(struct dir_struct *dir,\n extern int add_excludes_from_file_to_list(const char *fname, const char *base, int baselen,\n \t\t\t\t\t struct exclude_list *el, int check_index);\n extern void add_excludes_from_file(struct dir_struct *, const char *fname);\n+extern int add_excludes_from_blob_to_list(\n+\tstruct object_id *oid,\n+\tconst char *base, int baselen,\n+\tstruct exclude_list *el);\n extern void parse_exclude_pattern(const char **string, int *patternlen, unsigned *flags, int *nowildcardlen);\n extern void add_exclude(const char *string, const char *base,\n \t\t\tint baselen, struct exclude_list *el, int srcpos);\n-- \n2.9.3\n\n"},{"id":"324381","messageId":"20170713173459.3559-4-git@jeffhostetler.com","threadId":"46376","inReplyTo":"20170713173459.3559-1-git@jeffhostetler.com","subject":"[PATCH v2 03/19] list-objects: filter objects in traverse_commit_list","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-07-13T17:34:43Z","receivedAt":"2017-07-13T17:35:38Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate traverse_commit_list_filtered() and add filtering\ninterface to allow certain objects to be omitted (not shown)\nduring a traversal.\n\nUpdate traverse_commit_list() to be a wrapper for the above.\n\nFiltering will be used in a future commit by rev-list and\npack-objects for narrow/partial clone/fetch to omit certain\nblobs from the output.\n\ntraverse_bitmap_commit_list() does not work with filtering.\nIf a packfile bitmap is present, it will not be used.\n\nSigned-off-by: Jeff Hostetler \n---\n list-objects.c | 66 ++++++++++++++++++++++++++++++++++++++++++++--------------\n list-objects.h | 30 ++++++++++++++++++++++++++\n 2 files changed, 80 insertions(+), 16 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex f3ca6aa..8dddeda 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -13,10 +13,13 @@ static void process_blob(struct rev_info *revs,\n \t\t\t show_object_fn show,\n \t\t\t struct strbuf *path,\n \t\t\t const char *name,\n-\t\t\t void *cb_data)\n+\t\t\t void *cb_data,\n+\t\t\t filter_object_fn filter,\n+\t\t\t void *filter_data)\n {\n \tstruct object *obj = &blob->object;\n \tsize_t pathlen;\n+\tlist_objects_filter_result r = LOFR_MARK_SEEN | LOFR_SHOW;\n \n \tif (!revs->blob_objects)\n \t\treturn;\n@@ -24,11 +27,15 @@ static void process_blob(struct rev_info *revs,\n \t\tdie(\"bad blob object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n-\tobj->flags |= SEEN;\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tshow(obj, path->buf, cb_data);\n+\tif (filter)\n+\t\tr = filter(LOFT_BLOB, obj, path->buf, &path->buf[pathlen], filter_data);\n+\tif (r & LOFR_MARK_SEEN)\n+\t\tobj->flags |= SEEN;\n+\tif (r & LOFR_SHOW)\n+\t\tshow(obj, path->buf, cb_data);\n \tstrbuf_setlen(path, pathlen);\n }\n \n@@ -69,7 +76,9 @@ static void process_tree(struct rev_info *revs,\n \t\t\t show_object_fn show,\n \t\t\t struct strbuf *base,\n \t\t\t const char *name,\n-\t\t\t void *cb_data)\n+\t\t\t void *cb_data,\n+\t\t\t filter_object_fn filter,\n+\t\t\t void *filter_data)\n {\n \tstruct object *obj = &tree->object;\n \tstruct tree_desc desc;\n@@ -77,6 +86,7 @@ static void process_tree(struct rev_info *revs,\n \tenum interesting match = revs->diffopt.pathspec.nr == 0 ?\n \t\tall_entries_interesting: entry_not_interesting;\n \tint baselen = base->len;\n+\tlist_objects_filter_result r = LOFR_MARK_SEEN | LOFR_SHOW;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n@@ -90,9 +100,13 @@ static void process_tree(struct rev_info *revs,\n \t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n \t}\n \n-\tobj->flags |= SEEN;\n \tstrbuf_addstr(base, name);\n-\tshow(obj, base->buf, cb_data);\n+\tif (filter)\n+\t\tr = filter(LOFT_BEGIN_TREE, obj, base->buf, &base->buf[baselen], filter_data);\n+\tif (r & LOFR_MARK_SEEN)\n+\t\tobj->flags |= SEEN;\n+\tif (r & LOFR_SHOW)\n+\t\tshow(obj, base->buf, cb_data);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n@@ -112,7 +126,7 @@ static void process_tree(struct rev_info *revs,\n \t\t\tprocess_tree(revs,\n \t\t\t\t lookup_tree(entry.oid->hash),\n \t\t\t\t show, base, entry.path,\n-\t\t\t\t cb_data);\n+\t\t\t\t cb_data, filter, filter_data);\n \t\telse if (S_ISGITLINK(entry.mode))\n \t\t\tprocess_gitlink(revs, entry.oid->hash,\n \t\t\t\t\tshow, base, entry.path,\n@@ -121,8 +135,17 @@ static void process_tree(struct rev_info *revs,\n \t\t\tprocess_blob(revs,\n \t\t\t\t lookup_blob(entry.oid->hash),\n \t\t\t\t show, base, entry.path,\n-\t\t\t\t cb_data);\n+\t\t\t\t cb_data, filter, filter_data);\n \t}\n+\n+\tif (filter) {\n+\t\tr = filter(LOFT_END_TREE, obj, base->buf, &base->buf[baselen], filter_data);\n+\t\tif (r & LOFR_MARK_SEEN)\n+\t\t\tobj->flags |= SEEN;\n+\t\tif (r & LOFR_SHOW)\n+\t\t\tshow(obj, base->buf, cb_data);\n+\t}\n+\n \tstrbuf_setlen(base, baselen);\n \tfree_tree_buffer(tree);\n }\n@@ -183,10 +206,10 @@ static void add_pending_tree(struct rev_info *revs, struct tree *tree)\n \tadd_pending_object(revs, &tree->object, \"\");\n }\n \n-void traverse_commit_list(struct rev_info *revs,\n-\t\t\t show_commit_fn show_commit,\n-\t\t\t show_object_fn show_object,\n-\t\t\t void *data)\n+void traverse_commit_list_filtered(\n+\tstruct rev_info *revs,\n+\tshow_commit_fn show_commit, show_object_fn show_object, void *show_data,\n+\tfilter_object_fn filter, void *filter_data)\n {\n \tint i;\n \tstruct commit *commit;\n@@ -200,7 +223,7 @@ void traverse_commit_list(struct rev_info *revs,\n \t\t */\n \t\tif (commit->tree)\n \t\t\tadd_pending_tree(revs, commit->tree);\n-\t\tshow_commit(commit, data);\n+\t\tshow_commit(commit, show_data);\n \t}\n \tfor (i = 0; i < revs->pending.nr; i++) {\n \t\tstruct object_array_entry *pending = revs->pending.objects + i;\n@@ -211,19 +234,19 @@ void traverse_commit_list(struct rev_info *revs,\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n \t\t\tobj->flags |= SEEN;\n-\t\t\tshow_object(obj, name, data);\n+\t\t\tshow_object(obj, name, show_data);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\n \t\t\tpath = \"\";\n \t\tif (obj->type == OBJ_TREE) {\n \t\t\tprocess_tree(revs, (struct tree *)obj, show_object,\n-\t\t\t\t &base, path, data);\n+\t\t\t\t &base, path, show_data, filter, filter_data);\n \t\t\tcontinue;\n \t\t}\n \t\tif (obj->type == OBJ_BLOB) {\n \t\t\tprocess_blob(revs, (struct blob *)obj, show_object,\n-\t\t\t\t &base, path, data);\n+\t\t\t\t &base, path, show_data, filter, filter_data);\n \t\t\tcontinue;\n \t\t}\n \t\tdie(\"unknown pending object %s (%s)\",\n@@ -232,3 +255,14 @@ void traverse_commit_list(struct rev_info *revs,\n \tobject_array_clear(&revs->pending);\n \tstrbuf_release(&base);\n }\n+\n+void traverse_commit_list(struct rev_info *revs,\n+\t\t\t show_commit_fn show_commit,\n+\t\t\t show_object_fn show_object,\n+\t\t\t void *show_data)\n+{\n+\ttraverse_commit_list_filtered(\n+\t\trevs,\n+\t\tshow_commit, show_object, show_data,\n+\t\tNULL, NULL);\n+}\ndiff --git a/list-objects.h b/list-objects.h\nindex 0cebf85..964e7d3 100644\n--- a/list-objects.h\n+++ b/list-objects.h\n@@ -8,4 +8,34 @@ void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, voi\n typedef void (*show_edge_fn)(struct commit *);\n void mark_edges_uninteresting(struct rev_info *, show_edge_fn);\n \n+enum list_objects_filter_result {\n+\tLOFR_ZERO = 0,\n+\tLOFR_MARK_SEEN = 1<<0,\n+\tLOFR_SHOW = 1<<1,\n+};\n+\n+/* See object.h and revision.h */\n+#define FILTER_REVISIT (1<<25)\n+\n+enum list_objects_filter_type {\n+\tLOFT_BEGIN_TREE,\n+\tLOFT_END_TREE,\n+\tLOFT_BLOB\n+};\n+\n+typedef enum list_objects_filter_result list_objects_filter_result;\n+typedef enum list_objects_filter_type list_objects_filter_type;\n+\n+typedef list_objects_filter_result (*filter_object_fn)(\n+\tlist_objects_filter_type filter_type,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tvoid *filter_data);\n+\n+void traverse_commit_list_filtered(\n+\tstruct rev_info *,\n+\tshow_commit_fn, show_object_fn, void *show_data,\n+\tfilter_object_fn filter, void *filter_data);\n+\n #endif\n-- \n2.9.3\n\n"},{"id":"324382","messageId":"20170713173459.3559-7-git@jeffhostetler.com","threadId":"46376","inReplyTo":"20170713173459.3559-1-git@jeffhostetler.com","subject":"[PATCH v2 06/19] list-objects-filters: add use-sparse-checkout filter","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-07-13T17:34:46Z","receivedAt":"2017-07-13T17:35:41Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a filter for traverse_commit_list_filtered() to omit the\nblobs that would not be needed by a sparse checkout using the\ngiven sparse-checkout spec.\n\nThis filter will be used in a future commit by rev-list and\npack-objects for partial/narrow clone/fetch.\n\nA future enhancement should be able to also omit tree objects\nnot needed by such a sparse checkout, but that is not currently\nsupported.\n\nSigned-off-by: Jeff Hostetler \n---\n list-objects-filters.c | 179 +++++++++++++++++++++++++++++++++++++++++++++++++\n list-objects-filters.h | 16 +++++\n 2 files changed, 195 insertions(+)\n\ndiff --git a/list-objects-filters.c b/list-objects-filters.c\nindex f04d70e..cacf645 100644\n--- a/list-objects-filters.c\n+++ b/list-objects-filters.c\n@@ -180,3 +180,182 @@ void traverse_commit_list_omit_large_blobs(\n \n \toidset2_clear(&d.omits);\n }\n+\n+/*\n+ * A filter driven by a sparse-checkout specification to only\n+ * include blobs that a sparse checkout would populate.\n+ *\n+ * The sparse-checkout spec is loaded from the blob with the\n+ * given OID (rather than .git/info/sparse-checkout) because\n+ * the repo may be bare.\n+ */\n+struct frame {\n+\tint defval;\n+\tint child_prov_omit : 1;\n+};\n+\n+struct filter_use_sparse_data {\n+\tstruct oidset2 omits;\n+\tstruct exclude_list el;\n+\n+\tsize_t nr, alloc;\n+\tstruct frame *array_frame;\n+};\n+\n+static list_objects_filter_result filter_use_sparse(\n+\tlist_objects_filter_type filter_type,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tvoid *filter_data_)\n+{\n+\tstruct filter_use_sparse_data *filter_data = filter_data_;\n+\tint64_t object_length = -1;\n+\tint val, dtype;\n+\tunsigned long s;\n+\tenum object_type t;\n+\tstruct frame *frame;\n+\n+\tswitch (filter_type) {\n+\tdefault:\n+\t\tdie(\"unkown filter_type\");\n+\t\treturn LOFR_ZERO;\n+\n+\tcase LOFT_BEGIN_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\tdtype = DT_DIR;\n+\t\tval = is_excluded_from_list(pathname, strlen(pathname),\n+\t\t\t\t\t filename, &dtype, &filter_data->el);\n+\t\tif (val < 0)\n+\t\t\tval = filter_data->array_frame[filter_data->nr].defval;\n+\n+\t\tALLOC_GROW(filter_data->array_frame, filter_data->nr + 1,\n+\t\t\t filter_data->alloc);\n+\t\tfilter_data->nr++;\n+\t\tfilter_data->array_frame[filter_data->nr].defval = val;\n+\t\tfilter_data->array_frame[filter_data->nr].child_prov_omit = 0;\n+\n+\t\t/*\n+\t\t * A directory with this tree OID may appear in multiple\n+\t\t * places in the tree. (Think of a directory move, with\n+\t\t * no other changes.) And with a different pathname, the\n+\t\t * is_excluded...() results for this directory and items\n+\t\t * contained within it may be different. So we cannot\n+\t\t * mark it SEEN (yet), since that will prevent process_tree()\n+\t\t * from revisiting this tree object with other pathnames.\n+\t\t *\n+\t\t * Only SHOW the tree object the first time we visit this\n+\t\t * tree object.\n+\t\t *\n+\t\t * We always show all tree objects. A future optimization\n+\t\t * may want to attempt to narrow this.\n+\t\t */\n+\t\tif (obj->flags & FILTER_REVISIT)\n+\t\t\treturn LOFR_ZERO;\n+\t\tobj->flags |= FILTER_REVISIT;\n+\t\treturn LOFR_SHOW;\n+\n+\tcase LOFT_END_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\tassert(filter_data->nr > 0);\n+\n+\t\tframe = &filter_data->array_frame[filter_data->nr];\n+\t\tfilter_data->nr--;\n+\n+\t\t/*\n+\t\t * Tell our parent directory if any of our children were\n+\t\t * provisionally omitted.\n+\t\t */\n+\t\tfilter_data->array_frame[filter_data->nr].child_prov_omit |=\n+\t\t\tframe->child_prov_omit;\n+\n+\t\t/*\n+\t\t * If there are NO provisionally omitted child objects (ALL child\n+\t\t * objects in this folder were INCLUDED), then we can mark the\n+\t\t * folder as SEEN (so we will not have to revisit it again).\n+\t\t */\n+\t\tif (!frame->child_prov_omit)\n+\t\t\treturn LOFR_MARK_SEEN;\n+\t\treturn LOFR_ZERO;\n+\n+\tcase LOFT_BLOB:\n+\t\tassert(obj->type == OBJ_BLOB);\n+\t\tassert((obj->flags & SEEN) == 0);\n+\n+\t\tframe = &filter_data->array_frame[filter_data->nr];\n+\n+\t\t/*\n+\t\t * If we previously provisionally omitted this blob because\n+\t\t * its pathname was not in the sparse-checkout AND this\n+\t\t * reference to the blob has the same pathname, we can avoid\n+\t\t * repeating the exclusion logic on this pathname and just\n+\t\t * continue to provisionally omit it.\n+\t\t */\n+\t\tif (obj->flags & FILTER_REVISIT) {\n+\t\t\tstruct oidset2_entry *entry_prev;\n+\t\t\tentry_prev = oidset2_get(&filter_data->omits, &obj->oid);\n+\t\t\tif (entry_prev && !strcmp(pathname, entry_prev->pathname)) {\n+\t\t\t\tframe->child_prov_omit = 1;\n+\t\t\t\treturn LOFR_ZERO;\n+\t\t\t}\n+\t\t}\n+\n+\t\tdtype = DT_REG;\n+\t\tval = is_excluded_from_list(pathname, strlen(pathname),\n+\t\t\t\t\t filename, &dtype, &filter_data->el);\n+\t\tif (val < 0)\n+\t\t\tval = frame->defval;\n+\t\tif (val > 0)\n+\t\t\treturn LOFR_MARK_SEEN | LOFR_SHOW;\n+\n+\t\tt = sha1_object_info(obj->oid.hash, &s);\n+\t\tassert(t == OBJ_BLOB);\n+\t\tobject_length = (int64_t)((uint64_t)(s));\n+\n+\t\t/*\n+\t\t * Provisionally omit it. We've already established that\n+\t\t * this pathname is not in the sparse-checkout specification,\n+\t\t * so we WANT to omit this blob. However, a pathname elsewhere\n+\t\t * in the tree may also reference this same blob, so we cannot\n+\t\t * reject it yet. Leave the LOFR_ bits unset so that if the\n+\t\t * blob appears again in the traversal, we will be asked again.\n+\t\t *\n+\t\t * The pathname we associate with this omit is just the first\n+\t\t * one we saw for this blob. Other instances of this blob may\n+\t\t * have other pathnames and that is fine. We just use it for\n+\t\t * perf because most of the time, the blob will be in the same\n+\t\t * place as we walk the commits.\n+\t\t */\n+\t\toidset2_insert(&filter_data->omits, &obj->oid, object_length,\n+\t\t\t pathname);\n+\t\tobj->flags |= FILTER_REVISIT;\n+\t\tframe->child_prov_omit = 1;\n+\t\treturn LOFR_ZERO;\n+\t}\n+}\n+\n+void traverse_commit_list_use_sparse(\n+\tstruct rev_info *revs,\n+\tshow_commit_fn show_commit,\n+\tshow_object_fn show_object,\n+\toidset2_foreach_cb print_omitted_object,\n+\tvoid *ctx_data,\n+\tstruct object_id *oid)\n+{\n+\tstruct filter_use_sparse_data d;\n+\n+\tmemset(&d, 0, sizeof(d));\n+\tif (add_excludes_from_blob_to_list(oid, NULL, 0, &d.el) < 0)\n+\t\tdie(\"filter_use_sparse could not load specification\");\n+\tALLOC_GROW(d.array_frame, d.nr + 1, d.alloc);\n+\td.array_frame[d.nr].defval = 0; /* default to include */\n+\td.array_frame[d.nr].child_prov_omit = 0;\n+\n+\ttraverse_commit_list_filtered(revs, show_commit, show_object, ctx_data,\n+\t\t\t\t filter_use_sparse, &d);\n+\n+\tif (print_omitted_object)\n+\t\toidset2_foreach(&d.omits, print_omitted_object, ctx_data);\n+\n+\toidset2_clear(&d.omits);\n+}\ndiff --git a/list-objects-filters.h b/list-objects-filters.h\nindex 32b2833..52e507b 100644\n--- a/list-objects-filters.h\n+++ b/list-objects-filters.h\n@@ -26,4 +26,20 @@ void traverse_commit_list_omit_large_blobs(\n \tvoid *ctx_data,\n \tint64_t large_byte_limit);\n \n+/*\n+ * A filter driven by a sparse-checkout specification to only\n+ * include blobs that a sparse checkout would populate.\n+ *\n+ * The sparse-checkout spec is loaded from the blob with the\n+ * given OID (rather than .git/info/sparse-checkout) because\n+ * the repo may be bare.\n+ */\n+void traverse_commit_list_use_sparse(\n+\tstruct rev_info *revs,\n+\tshow_commit_fn show_commit,\n+\tshow_object_fn show_object,\n+\toidset2_foreach_cb print_omitted_object,\n+\tvoid *ctx_data,\n+\tstruct object_id *oid);\n+\n #endif /* LIST_OBJECTS_FILTERS_H */\n-- \n2.9.3\n\n"},{"id":"324383","messageId":"20170713173459.3559-8-git@jeffhostetler.com","threadId":"46376","inReplyTo":"20170713173459.3559-1-git@jeffhostetler.com","subject":"[PATCH v2 07/19] object-filter: common declarations for object filtering","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-07-13T17:34:47Z","receivedAt":"2017-07-13T17:35:43Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate common routines and defines for parsing\nobject-filter-related command line arguments and\npack-protocol fields.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 1 +\n object-filter.c | 201 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n object-filter.h | 145 ++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 347 insertions(+)\n create mode 100644 object-filter.c\n create mode 100644 object-filter.h\n\ndiff --git a/Makefile b/Makefile\nindex 48fdcf2..daa9ea2 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -791,6 +791,7 @@ LIB_OBJS += notes-cache.o\n LIB_OBJS += notes-merge.o\n LIB_OBJS += notes-utils.o\n LIB_OBJS += object.o\n+LIB_OBJS += object-filter.o\n LIB_OBJS += oidset.o\n LIB_OBJS += oidset2.o\n LIB_OBJS += pack-bitmap.o\ndiff --git a/object-filter.c b/object-filter.c\nnew file mode 100644\nindex 0000000..5be6129\n--- /dev/null\n+++ b/object-filter.c\n@@ -0,0 +1,201 @@\n+#include \"cache.h\"\n+#include \"commit.h\"\n+#include \"revision.h\"\n+#include \"list-objects.h\"\n+#include \"oidset2.h\"\n+#include \"list-objects-filters.h\"\n+#include \"object-filter.h\"\n+\n+int parse_filter_omit_all_blobs(struct object_filter_options *filter_options)\n+{\n+\tif (object_filter_enabled(filter_options))\n+\t\tdie(_(\"multiple object filter types cannot be combined\"));\n+\n+\tfilter_options->omit_all_blobs = 1;\n+\treturn 0;\n+}\n+\n+int parse_filter_omit_large_blobs(struct object_filter_options *filter_options,\n+\t\t\t\t const char *arg)\n+{\n+\tif (object_filter_enabled(filter_options))\n+\t\tdie(_(\"multiple object filter types cannot be combined\"));\n+\n+\tfilter_options->omit_large_blobs = 1;\n+\n+\t/* we allow \"[kmg]\" */\n+\tif (!git_parse_ulong(arg, &filter_options->large_byte_limit))\n+\t\tdie(_(\"invalid size limit for large object filter\"));\n+\n+\tfilter_options->large_byte_limit_string = strdup(arg);\n+\treturn 0;\n+}\n+\n+int parse_filter_use_sparse(struct object_filter_options *filter_options,\n+\t\t\t const char *arg)\n+{\n+\tstruct object_context oc;\n+\n+\tif (object_filter_enabled(filter_options))\n+\t\tdie(_(\"multiple object filter types cannot be combined\"));\n+\n+\tfilter_options->use_sparse = 1;\n+\n+\t/*\n+\t * The command line argument needs to resolve to an known OID\n+\t * representing the content of the desired sparse-checkout file.\n+\t * We allow various syntax forms for the convenience of the user.\n+\t * See sha1_name.c:get_sha1_with_context_1().\n+\t *\n+\t * Try to evaluate the arg locally in case they use one of the\n+\t * convenience patterns. This must resolve to a blob.\n+\t */\n+\tif (get_sha1_with_context(arg, GET_SHA1_BLOB,\n+\t\t\t\t filter_options->sparse_oid.hash, &oc)) {\n+\t\t/*\n+\t\t * If that fails, keep the original string in case a client\n+\t\t * command wants to send it to the server. This allows the\n+\t\t * client to name an OID for a blob they don't have.\n+\t\t */\n+\t\tfilter_options->sparse_value = strdup(arg);\n+\t\toidcpy(&filter_options->sparse_oid, &null_oid);\n+\t} else {\n+\t\t/*\n+\t\t * Round-trip the found OID to normalize it.\n+\t\t */\n+\t\tfilter_options->sparse_value =\n+\t\t\tstrdup(oid_to_hex(&filter_options->sparse_oid));\n+\t}\n+\t\n+\treturn 0;\n+}\n+\n+int parse_filter_print_manifest(struct object_filter_options *filter_options)\n+{\n+\tfilter_options->print_manifest = 1;\n+\treturn 0;\n+}\n+\n+int parse_filter_relax(struct object_filter_options *filter_options)\n+{\n+\tfilter_options->relax = 1;\n+\treturn 0;\n+}\n+\n+int opt_parse_filter_omit_all_blobs(const struct option *opt,\n+\t\t\t\t const char *arg, int unset)\n+{\n+\tstruct object_filter_options *filter_options = opt->value;\n+\n+\tassert(!arg);\n+\tassert(!unset);\n+\n+\treturn parse_filter_omit_all_blobs(filter_options);\n+}\n+\n+int opt_parse_filter_omit_large_blobs(const struct option *opt,\n+\t\t\t\t const char *arg, int unset)\n+{\n+\tstruct object_filter_options *filter_options = opt->value;\n+\n+\tassert(arg);\n+\tassert(!unset);\n+\n+\treturn parse_filter_omit_large_blobs(filter_options, arg);\n+}\n+\n+int opt_parse_filter_use_sparse(const struct option *opt,\n+\t\t\t\tconst char *arg, int unset)\n+{\n+\tstruct object_filter_options *filter_options = opt->value;\n+\n+\tassert(arg);\n+\tassert(!unset);\n+\n+\treturn parse_filter_use_sparse(filter_options, arg);\n+}\n+\n+int opt_parse_filter_print_manifest(const struct option *opt,\n+\t\t\t\t const char *arg, int unset)\n+{\n+\tstruct object_filter_options *filter_options = opt->value;\n+\n+\tassert(!arg);\n+\tassert(!unset);\n+\n+\treturn parse_filter_print_manifest(filter_options);\n+}\n+\n+int opt_parse_filter_relax(const struct option *opt,\n+\t\t\t const char *arg, int unset)\n+{\n+\tstruct object_filter_options *filter_options = opt->value;\n+\n+\tassert(!arg);\n+\tassert(!unset);\n+\n+\treturn parse_filter_relax(filter_options);\n+}\n+\n+int object_filter_hand_parse_arg(struct object_filter_options *filter_options,\n+\t\t\t\t const char *arg,\n+\t\t\t\t int allow_print_manifest,\n+\t\t\t\t int allow_relax)\n+{\n+\tif (!strcmp(arg, (\"--\"CL_ARG_FILTER_OMIT_ALL_BLOBS))) {\n+\t\tparse_filter_omit_all_blobs(filter_options);\n+\t\treturn 1;\n+\t}\n+\tif (skip_prefix(arg, (\"--\"CL_ARG_FILTER_OMIT_LARGE_BLOBS\"=\"), &arg)) {\n+\t\tparse_filter_omit_large_blobs(filter_options, arg);\n+\t\treturn 1;\n+\t}\n+\tif (skip_prefix(arg, (\"--\"CL_ARG_FILTER_USE_SPARSE\"=\"), &arg)) {\n+\t\tparse_filter_use_sparse(filter_options, arg);\n+\t\treturn 1;\n+\t}\n+\n+\tif (allow_print_manifest &&\n+\t !strcmp(arg, (\"--\"CL_ARG_FILTER_PRINT_MANIFEST))) {\n+\t\tparse_filter_print_manifest(filter_options);\n+\t\treturn 1;\n+\t}\n+\n+\tif (allow_relax && !strcmp(arg, (\"--\"CL_ARG_FILTER_RELAX))) {\n+\t\tparse_filter_relax(filter_options);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+int object_filter_hand_parse_protocol(struct object_filter_options *filter_options,\n+\t\t\t\t const char *arg,\n+\t\t\t\t int allow_print_manifest,\n+\t\t\t\t int allow_relax)\n+{\n+\tif (!strcmp(arg, CL_ARG_FILTER_OMIT_ALL_BLOBS)) {\n+\t\tparse_filter_omit_all_blobs(filter_options);\n+\t\treturn 1;\n+\t}\n+\tif (skip_prefix(arg, (CL_ARG_FILTER_OMIT_LARGE_BLOBS\" \"), &arg)) {\n+\t\tparse_filter_omit_large_blobs(filter_options, arg);\n+\t\treturn 1;\n+\t}\n+\tif (skip_prefix(arg, (CL_ARG_FILTER_USE_SPARSE\" \"), &arg)) {\n+\t\tparse_filter_use_sparse(filter_options, arg);\n+\t\treturn 1;\n+\t}\n+\n+\tif (allow_print_manifest &&\n+\t !strcmp(arg, CL_ARG_FILTER_PRINT_MANIFEST)) {\n+\t\tparse_filter_print_manifest(filter_options);\n+\t\treturn 1;\n+\t}\n+\tif (allow_relax && !strcmp(arg, CL_ARG_FILTER_RELAX)) {\n+\t\tparse_filter_relax(filter_options);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/object-filter.h b/object-filter.h\nnew file mode 100644\nindex 0000000..f1ca5fb\n--- /dev/null\n+++ b/object-filter.h\n@@ -0,0 +1,145 @@\n+#ifndef OBJECT_FILTER_H\n+#define OBJECT_FILTER_H\n+\n+#include \"parse-options.h\"\n+\n+/*\n+ * Common declarations and utilities for filtering objects (such as omitting\n+ * large blobs) during fetch-pack, upload-pack, and the pack-protocol. These\n+ * are intended for partial/narrow clone/fetch.\n+ */\n+\n+struct object_filter_options {\n+\t/*\n+\t * blob-ish path or value that get_sha1_with_context() can turn into\n+\t * an OID to find the blob containing the sparse-checkout specification.\n+\t * only used when use_sparse is set.\n+\t */\n+\tconst char *sparse_value;\n+\tstruct object_id sparse_oid;\n+\n+\t/*\n+\t * blob size byte limit for filtering. only blobs smaller than this\n+\t * value will be included. a value of zero, omits all blobs.\n+\t * only used when omit_large_blobs is set. Integer and string versions\n+\t * of this are kept for convenience.\n+\t */\n+\tunsigned long large_byte_limit;\n+\tconst char *large_byte_limit_string;\n+\n+\t/* valid filter types (only one may be used at a time) */\n+\tunsigned omit_all_blobs : 1;\n+\tunsigned omit_large_blobs : 1;\n+\tunsigned use_sparse : 1;\n+\n+\t/* true if the filter should output a manifest of the omitted objects. */\n+\tunsigned print_manifest : 1;\n+\n+\t/* true to suppress missing object errors during consistency checks */\n+\tunsigned relax : 1;\n+};\n+\n+/*\n+ * Return true if a filter is enabled.\n+ */\n+inline int object_filter_enabled(const struct object_filter_options *p)\n+{\n+\treturn p->omit_all_blobs || p->omit_large_blobs || p->use_sparse;\n+}\n+\n+/* See Documentation/technical/protocol-capabilities.txt */\n+#define PROTOCOL_CAPABILITY_FILTER_OBJECTS \"filter-objects\"\n+\n+/* See Documentation/technical/pack-protocol.txt */\n+#define PROTOCOL_REQUEST_FILTER_OMIT_ALL_BLOBS \"filter-omit-all-blobs\"\n+#define PROTOCOL_REQUEST_FILTER_OMIT_LARGE_BLOBS \"filter-omit-large-blobs\"\n+#define PROTOCOL_REQUEST_FILTER_USE_SPARSE \"filter-use-sparse\"\n+\n+/* Normalized command line arguments */\n+#define CL_ARG_FILTER_OMIT_ALL_BLOBS \"filter-omit-all-blobs\"\n+#define CL_ARG_FILTER_OMIT_LARGE_BLOBS \"filter-omit-large-blobs\"\n+#define CL_ARG_FILTER_USE_SPARSE \"filter-use-sparse\"\n+#define CL_ARG_FILTER_PRINT_MANIFEST \"filter-print-manifest\"\n+#define CL_ARG_FILTER_RELAX \"filter-relax\"\n+\n+/*\n+ * Common command line argument parsing for object-filter-related\n+ * arguments (whether from a hand-parsed or parse-options style\n+ * parser.\n+ */\n+int parse_filter_omit_all_blobs(struct object_filter_options *filter_options);\n+int parse_filter_omit_large_blobs(struct object_filter_options *filter_options,\n+\t\t\t\t const char *arg);\n+int parse_filter_use_sparse(struct object_filter_options *filter_options,\n+\t\t\t const char *arg);\n+int parse_filter_print_manifest(struct object_filter_options *filter_options);\n+int parse_filter_relax(struct object_filter_options *filter_options);\n+\n+/*\n+ * Common command line argument parsers for object-filter-related\n+ * arguments comming from parse-options style parsers.\n+ */\n+\n+int opt_parse_filter_omit_all_blobs(const struct option *opt,\n+\t\t\t\t const char *arg, int unset);\n+int opt_parse_filter_omit_large_blobs(const struct option *opt,\n+\t\t\t\t const char *arg, int unset);\n+int opt_parse_filter_use_sparse(const struct option *opt,\n+\t\t\t\tconst char *arg, int unset);\n+int opt_parse_filter_print_manifest(const struct option *opt,\n+\t\t\t\t const char *arg, int unset);\n+int opt_parse_filter_relax(const struct option *opt,\n+\t\t\t const char *arg, int unset);\n+\n+#define OPT_PARSE_FILTER_OMIT_ALL_BLOBS(fo) \\\n+\t{ OPTION_CALLBACK, 0, CL_ARG_FILTER_OMIT_ALL_BLOBS, fo, NULL, \\\n+\t N_(\"omit all blobs from result\"), PARSE_OPT_NOARG | PARSE_OPT_NONEG, \\\n+\t opt_parse_filter_omit_all_blobs }\n+\n+#define OPT_PARSE_FILTER_OMIT_LARGE_BLOBS(fo) \\\n+\t{ OPTION_CALLBACK, 0, CL_ARG_FILTER_OMIT_LARGE_BLOBS, fo, N_(\"size\"), \\\n+\t N_(\"omit large blobs from result\"), PARSE_OPT_NONEG, \\\n+\t opt_parse_filter_omit_large_blobs }\n+\n+#define OPT_PARSE_FILTER_USE_SPARSE(fo) \\\n+\t{ OPTION_CALLBACK, 0, CL_ARG_FILTER_USE_SPARSE, fo, N_(\"object\"), \\\n+\t N_(\"filter results using sparse-checkout specification\"), PARSE_OPT_NONEG, \\\n+\t opt_parse_filter_use_sparse }\n+\n+#define OPT_PARSE_FILTER_PRINT_MANIFEST(fo) \\\n+\t{ OPTION_CALLBACK, 0, CL_ARG_FILTER_PRINT_MANIFEST, fo, NULL,\t\\\n+\t N_(\"print manifest of omitted objects\"), PARSE_OPT_NOARG | PARSE_OPT_NONEG, \\\n+\t opt_parse_filter_print_manifest }\n+\n+#define OPT_PARSE_FILTER_RELAX(fo) \\\n+\t{ OPTION_CALLBACK, 0, CL_ARG_FILTER_RELAX, fo, NULL, \\\n+\t N_(\"relax consistency checks for previously omitted objects\"), \\\n+\t PARSE_OPT_NOARG | PARSE_OPT_NONEG, opt_parse_filter_relax }\n+\n+/*\n+ * Hand parse known object-filter command line options.\n+ * Use this when the caller DOES NOT use the normal OPT_\n+ * routines.\n+ *\n+ * Here we assume args of the form \"--\" or \"--=\".\n+ * Note the literal dash-dash and equals.\n+ *\n+ * Returns 1 if we handled the argument.\n+ */\n+int object_filter_hand_parse_arg(struct object_filter_options *filter_options,\n+\t\t\t\t const char *arg,\n+\t\t\t\t int allow_print_manifest,\n+\t\t\t\t int allow_relax);\n+\n+/*\n+ * Hand parse known object-filter protocol lines.\n+ *\n+ * Here we assume args of the form \"\" or \" \".\n+ * Note the literal space before between the key and value.\n+ */ \n+int object_filter_hand_parse_protocol(struct object_filter_options *filter_options,\n+\t\t\t\t const char *arg,\n+\t\t\t\t int allow_print_manifest,\n+\t\t\t\t int allow_relax);\n+\n+#endif /* OBJECT_FILTER_H */\n-- \n2.9.3\n\n"},{"id":"324384","messageId":"20170713173459.3559-12-git@jeffhostetler.com","threadId":"46376","inReplyTo":"20170713173459.3559-1-git@jeffhostetler.com","subject":"[PATCH v2 11/19] pack-objects: add object filtering support","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-07-13T17:34:51Z","receivedAt":"2017-07-13T17:35:44Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach pack-objects to use filtering provided by the\ntraverse_commit_list_filtered() interface to omit\nunwanted objects from the result.\n\nThis feature is intended for narrow/partial clone/fetch.\n\nFiltering requires use of \"--stdout\" option.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/pack-objects.c | 33 ++++++++++++++++++++++++++++++++-\n 1 file changed, 32 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 50e01aa..614ad60 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -14,6 +14,8 @@\n #include \"diff.h\"\n #include \"revision.h\"\n #include \"list-objects.h\"\n+#include \"list-objects-filters.h\"\n+#include \"object-filter.h\"\n #include \"pack-objects.h\"\n #include \"progress.h\"\n #include \"refs.h\"\n@@ -77,6 +79,8 @@ static unsigned long cache_max_small_delta_size = 1000;\n \n static unsigned long window_memory_limit = 0;\n \n+static struct object_filter_options filter_options;\n+\n /*\n * stats\n */\n@@ -2800,7 +2804,20 @@ static void get_object_list(int ac, const char **av)\n \tif (prepare_revision_walk(&revs))\n \t\tdie(\"revision walk setup failed\");\n \tmark_edges_uninteresting(&revs, show_edge);\n-\ttraverse_commit_list(&revs, show_commit, show_object, NULL);\n+\n+\tif (filter_options.omit_all_blobs)\n+\t\ttraverse_commit_list_omit_all_blobs(\n+\t\t\t&revs, show_commit, show_object, NULL, NULL);\n+\telse if (filter_options.omit_large_blobs)\n+\t\ttraverse_commit_list_omit_large_blobs(\n+\t\t\t&revs, show_commit, show_object, NULL, NULL,\n+\t\t\t(int64_t)(uint64_t)filter_options.large_byte_limit);\n+\telse if (filter_options.use_sparse)\n+\t\ttraverse_commit_list_use_sparse(\n+\t\t\t&revs, show_commit, show_object, NULL, NULL,\n+\t\t\t&filter_options.sparse_oid);\n+\telse\n+\t\ttraverse_commit_list(&revs, show_commit, show_object, NULL);\n \n \tif (unpack_unreachable_expiration) {\n \t\trevs.ignore_missing_links = 1;\n@@ -2936,6 +2953,14 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"use a bitmap index if available to speed up counting objects\")),\n \t\tOPT_BOOL(0, \"write-bitmap-index\", &write_bitmap_index,\n \t\t\t N_(\"write a bitmap index together with the pack index\")),\n+\n+\t\tOPT_PARSE_FILTER_OMIT_ALL_BLOBS(&filter_options),\n+\t\tOPT_PARSE_FILTER_OMIT_LARGE_BLOBS(&filter_options),\n+\t\tOPT_PARSE_FILTER_USE_SPARSE(&filter_options),\n+\n+\t\t/* OPT_PARSE_FILTER_PRINT_MANIFEST(&filter_options), */\n+\t\t/* OPT_PARSE_FILTER_RELAX(&filter_options), */\n+\n \t\tOPT_END(),\n \t};\n \n@@ -3007,6 +3032,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!pack_to_stdout && thin)\n \t\tdie(\"--thin cannot be used to build an indexable pack.\");\n \n+\tif (!pack_to_stdout && object_filter_enabled(&filter_options))\n+\t\tdie(\"object filtering cannot be used when building an indexable pack.\");\n+\n \tif (keep_unreachable && unpack_unreachable)\n \t\tdie(\"--keep-unreachable and --unpack-unreachable are incompatible.\");\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n@@ -3031,6 +3059,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!use_internal_rev_list || (!pack_to_stdout && write_bitmap_index) || is_repository_shallow())\n \t\tuse_bitmap_index = 0;\n \n+\tif (object_filter_enabled(&filter_options))\n+\t\tuse_bitmap_index = 0;\n+\n \tif (pack_to_stdout || !rev_list_all)\n \t\twrite_bitmap_index = 0;\n \n-- \n2.9.3\n\n"},{"id":"324385","messageId":"20170713173459.3559-11-git@jeffhostetler.com","threadId":"46376","inReplyTo":"20170713173459.3559-1-git@jeffhostetler.com","subject":"[PATCH v2 10/19] t6112: rev-list object filtering test","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-07-13T17:34:50Z","receivedAt":"2017-07-13T17:35:46Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n t/t6112-rev-list-filters-objects.sh | 37 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 37 insertions(+)\n create mode 100644 t/t6112-rev-list-filters-objects.sh\n\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nnew file mode 100644\nindex 0000000..ded2b04\n--- /dev/null\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -0,0 +1,37 @@\n+#!/bin/sh\n+\n+test_description='git rev-list with object filtering'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\tfor n in 1 2 3 4 5 ; do \\\n+\t\techo $n > file.$n ; \\\n+\t\tgit add file.$n ; \\\n+\t\tgit commit -m \"$n\" ; \\\n+\tdone\n+'\n+\n+test_expect_success 'omit-all-blobs omitted 5 blobs' '\n+\tgit rev-list HEAD --objects --filter-print-manifest --filter-omit-all-blobs >omit_all &&\n+\tgrep \"^~\" omit_all >omitted &&\n+\ttest $(cat omitted | wc -l) = 5\n+'\n+\n+test_expect_success 'omit-all-blobs blob sha match' '\n+\tgit rev-list HEAD --objects >normal &&\n+\tawk \"/file/ {print \\$1;}\" normal_sha &&\n+\tsed \"s/~//\" omit_all_sha &&\n+\ttest_cmp normal_sha omit_all_sha\n+'\n+\n+test_expect_success 'omit-all-blobs nothing else changed' '\n+\tgrep -v \"file\" normal_other &&\n+\tgrep -v \"~\" omit_other &&\n+\ttest_cmp normal_other omit_other\n+'\n+\n+# TODO test filter-omit-large-blobs\n+# TODO test filter-use-sparse\n+\n+test_done\n-- \n2.9.3\n\n"},{"id":"324386","messageId":"20170713173459.3559-14-git@jeffhostetler.com","threadId":"46376","inReplyTo":"20170713173459.3559-1-git@jeffhostetler.com","subject":"[PATCH v2 13/19] upload-pack: add filter-objects to protocol documentation","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-07-13T17:34:53Z","receivedAt":"2017-07-13T17:35:47Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n Documentation/technical/pack-protocol.txt | 16 ++++++++++++++++\n Documentation/technical/protocol-capabilities.txt | 7 +++++++\n 2 files changed, 23 insertions(+)\n\ndiff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt\nindex a349171..dce6e04 100644\n--- a/Documentation/technical/pack-protocol.txt\n+++ b/Documentation/technical/pack-protocol.txt\n@@ -212,6 +212,7 @@ out of what the server said it could do with the first 'want' line.\n upload-request = want-list\n \t\t *shallow-line\n \t\t *1depth-request\n+\t\t [filter-request]\n \t\t flush-pkt\n \n want-list = first-want\n@@ -226,7 +227,13 @@ out of what the server said it could do with the first 'want' line.\n first-want = PKT-LINE(\"want\" SP obj-id SP capability-list)\n additional-want = PKT-LINE(\"want\" SP obj-id)\n \n+ filter-request = PKT-LINE(\"filter-omit-all-blobs\") /\n+\t\t PKT-LINE(\"filter-omit-large-blobs\" SP magnitude) /\n+\t\t PKT-LINE(\"filter-use-sparse\" SP obj-id)\n+\n depth = 1*DIGIT\n+\n+ magnitude = 1*DIGIT [ \"k\" | \"m\" | \"g\" ]\n ----\n \n Clients MUST send all the obj-ids it wants from the reference\n@@ -249,6 +256,15 @@ complete those commits. Commits whose parents are not received as a\n result are defined as shallow and marked as such in the server. This\n information is sent back to the client in the next step.\n \n+The client can optionally request that pack-objects omit various\n+objects from the packfile using one of several filtering techniques.\n+These are intended for use with partial/narrow clone/fetch operations.\n+\"filter-omit-all-blobs\" requests that all blobs be omitted from\n+the packfile. \"filter-omit-large-blobs\" requests that blobs larger\n+than the requested size be omitted, unless they have a \".git*\"\n+special filename. \"filter-use-sparse\" requests blob filtering based\n+upon a sparse-checkout specification in the given blob id.\n+\n Once all the 'want's and 'shallow's (and optional 'deepen') are\n transferred, clients MUST send a flush-pkt, to tell the server side\n that it is done sending the list.\ndiff --git a/Documentation/technical/protocol-capabilities.txt b/Documentation/technical/protocol-capabilities.txt\nindex 26dcc6f..7011eb3 100644\n--- a/Documentation/technical/protocol-capabilities.txt\n+++ b/Documentation/technical/protocol-capabilities.txt\n@@ -309,3 +309,10 @@ to accept a signed push certificate, and asks the to be\n included in the push certificate. A send-pack client MUST NOT\n send a push-cert packet unless the receive-pack server advertises\n this capability.\n+\n+filter-objects\n+--------------\n+\n+If the upload-pack server advertises the 'filter-objects' capability,\n+fetch-pack may send \"filter-*\" commands to request a partial/narrow\n+clone/fetch where the server omits various objects from the packfile.\n-- \n2.9.3\n\n"},{"id":"324387","messageId":"20170713173459.3559-18-git@jeffhostetler.com","threadId":"46376","inReplyTo":"20170713173459.3559-1-git@jeffhostetler.com","subject":"[PATCH v2 17/19] clone: add filter arguments","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-07-13T17:34:57Z","receivedAt":"2017-07-13T17:35:49Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n builtin/clone.c | 28 ++++++++++++++++++++++++++++\n 1 file changed, 28 insertions(+)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex a6ae7d6..1408396 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -24,6 +24,7 @@\n #include \"remote.h\"\n #include \"run-command.h\"\n #include \"connected.h\"\n+#include \"object-filter.h\"\n \n /*\n * Overall FIXMEs:\n@@ -57,6 +58,7 @@ static struct string_list option_optional_reference = STRING_LIST_INIT_NODUP;\n static int option_dissociate;\n static int max_jobs = -1;\n static struct string_list option_recurse_submodules = STRING_LIST_INIT_NODUP;\n+static struct object_filter_options filter_options;\n \n static int recurse_submodules_cb(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n@@ -130,6 +132,14 @@ static struct option builtin_clone_options[] = {\n \t\t\tTRANSPORT_FAMILY_IPV4),\n \tOPT_SET_INT('6', \"ipv6\", &family, N_(\"use IPv6 addresses only\"),\n \t\t\tTRANSPORT_FAMILY_IPV6),\n+\n+\tOPT_PARSE_FILTER_OMIT_ALL_BLOBS(&filter_options),\n+\tOPT_PARSE_FILTER_OMIT_LARGE_BLOBS(&filter_options),\n+\tOPT_PARSE_FILTER_USE_SPARSE(&filter_options),\n+\n+\t/* OPT_PARSE_FILTER_PRINT_MANIFEST(&filter_options), */\n+\t/* OPT_PARSE_FILTER_RELAX(&filter_options), */\n+\n \tOPT_END()\n };\n \n@@ -643,6 +653,13 @@ static void update_remote_refs(const struct ref *refs,\n \tif (check_connectivity) {\n \t\tstruct check_connected_options opt = CHECK_CONNECTED_INIT;\n \n+\t\t/*\n+\t\t * Relax consistency check to allow missing blobs (presumably\n+\t\t * because they are exactly the set that we requested be\n+\t\t * omitted.\n+\t\t */\n+\t\topt.filter_relax = object_filter_enabled(&filter_options);\n+\n \t\topt.transport = transport;\n \t\topt.progress = transport->progress;\n \n@@ -1059,6 +1076,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\t\twarning(_(\"--shallow-since is ignored in local clones; use file:// instead.\"));\n \t\tif (option_not.nr)\n \t\t\twarning(_(\"--shallow-exclude is ignored in local clones; use file:// instead.\"));\n+\t\tif (object_filter_enabled(&filter_options))\n+\t\t\twarning(_(\"--filter-* options are ignored in local clones; use file:// instead.\"));\n \t\tif (!access(mkpath(\"%s/shallow\", path), F_OK)) {\n \t\t\tif (option_local > 0)\n \t\t\t\twarning(_(\"source repository is shallow, ignoring --local\"));\n@@ -1090,6 +1109,15 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK,\n \t\t\t\t option_upload_pack);\n \n+\tif (filter_options.omit_all_blobs)\n+\t\ttransport_set_option(transport, TRANS_OPT_FILTER_OMIT_ALL_BLOBS, \"1\");\n+\tif (filter_options.omit_large_blobs)\n+\t\ttransport_set_option(transport, TRANS_OPT_FILTER_OMIT_LARGE_BLOBS,\n+\t\t\t\t filter_options.large_byte_limit_string);\n+\tif (filter_options.use_sparse)\n+\t\ttransport_set_option(transport, TRANS_OPT_FILTER_USE_SPARSE,\n+\t\t\t\t filter_options.sparse_value);\n+\n \tif (transport->smart_options && !deepen)\n \t\ttransport->smart_options->check_self_contained_and_connected = 1;\n \n-- \n2.9.3\n\n"},{"id":"324388","messageId":"20170713173459.3559-19-git@jeffhostetler.com","threadId":"46376","inReplyTo":"20170713173459.3559-1-git@jeffhostetler.com","subject":"[PATCH v2 18/19] index-pack: relax consistency checks for omitted objects","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-07-13T17:34:58Z","receivedAt":"2017-07-13T17:35:53Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n builtin/index-pack.c | 15 +++++++++++++++\n 1 file changed, 15 insertions(+)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 4ff567d..30ff409 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -11,6 +11,7 @@\n #include \"exec_cmd.h\"\n #include \"streaming.h\"\n #include \"thread-utils.h\"\n+#include \"object-filter.h\"\n \n static const char index_pack_usage[] =\n \"git index-pack [-v] [-o ] [--keep | --keep=] [--verify] [--strict] ( | --stdin [--fix-thin] [])\";\n@@ -80,6 +81,7 @@ static int verbose;\n static int show_resolving_progress;\n static int show_stat;\n static int check_self_contained_and_connected;\n+static int filter_relax;\n \n static struct progress *progress;\n \n@@ -220,6 +222,17 @@ static unsigned check_object(struct object *obj)\n \tif (!(obj->flags & FLAG_CHECKED)) {\n \t\tunsigned long size;\n \t\tint type = sha1_object_info(obj->oid.hash, &size);\n+\n+\t\tif (type <= 0 && filter_relax) {\n+\t\t\t/*\n+\t\t\t * Relax consistency checks to not complain about\n+\t\t\t * omitted objects (presumably caused by use of\n+\t\t\t * the 'filter-objects' feature).\n+\t\t\t */\n+\t\t\tobj->flags |= FLAG_CHECKED;\n+\t\t\treturn 0;\n+\t\t}\n+\n \t\tif (type <= 0)\n \t\t\tdie(_(\"did not receive expected object %s\"),\n \t\t\t oid_to_hex(&obj->oid));\n@@ -1721,6 +1734,8 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\t\tdie(_(\"bad %s\"), arg);\n \t\t\t} else if (skip_prefix(arg, \"--max-input-size=\", &arg)) {\n \t\t\t\tmax_input_size = strtoumax(arg, NULL, 10);\n+\t\t\t} else if (!strcmp(arg, (\"--\"CL_ARG_FILTER_RELAX))) {\n+\t\t\t\tfilter_relax = 1;\n \t\t\t} else\n \t\t\t\tusage(index_pack_usage);\n \t\t\tcontinue;\n-- \n2.9.3\n\n"},{"id":"324389","messageId":"20170713173459.3559-13-git@jeffhostetler.com","threadId":"46376","inReplyTo":"20170713173459.3559-1-git@jeffhostetler.com","subject":"[PATCH v2 12/19] pack-objects: add filtering help text","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-07-13T17:34:52Z","receivedAt":"2017-07-13T17:35:55Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nUpdate pack-objects help text to describe object filtering.\n\nSigned-off-by: Jeff Hostetler \n---\n Documentation/git-pack-objects.txt | 14 ++++++++++++++\n 1 file changed, 14 insertions(+)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 8973510..084641f 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -231,6 +231,20 @@ So does `git bundle` (see linkgit:git-bundle[1]) when it creates a bundle.\n \tWith this option, parents that are hidden by grafts are packed\n \tnevertheless.\n \n+--filter-omit-all-blobs::\n+\tOmits all blobs from the packfile. This option requires --stdout.\n+\n+--filter-omit-large-blobs=[kmg]::\n+\tOmits blobs larger than\tn bytes from packfile. May optionally be\n+\tfollowed by 'k', 'm', or 'g' units. Value may be zero. Special\n+\tfiles (matching \".git*\") are always included, regardless of size.\n+\tThis option requires --stdout.\n+\n+--filter-use-sparse=::\n+\tUses a sparse-checkout specification given by to filter\n+\tthe result by omitting blobs that would not be used by the\n+\tcorresponding sparse checkout. This option requires --stdout.\n+\n SEE ALSO\n --------\n linkgit:git-rev-list[1]\n-- \n2.9.3\n\n"},{"id":"324390","messageId":"20170713173459.3559-15-git@jeffhostetler.com","threadId":"46376","inReplyTo":"20170713173459.3559-1-git@jeffhostetler.com","subject":"[PATCH v2 14/19] upload-pack: add object filtering","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-07-13T17:34:54Z","receivedAt":"2017-07-13T17:35:56Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n upload-pack.c | 39 ++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 38 insertions(+), 1 deletion(-)\n\ndiff --git a/upload-pack.c b/upload-pack.c\nindex ffb028d..c709054 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -17,6 +17,7 @@\n #include \"parse-options.h\"\n #include \"argv-array.h\"\n #include \"prio-queue.h\"\n+#include \"object-filter.h\"\n \n static const char * const upload_pack_usage[] = {\n \tN_(\"git upload-pack [] \"),\n@@ -63,6 +64,9 @@ static int advertise_refs;\n static int stateless_rpc;\n static const char *pack_objects_hook;\n \n+static int capability_filter_objects_requested;\n+static struct object_filter_options filter_options;\n+\n static void reset_timeout(void)\n {\n \talarm(timeout);\n@@ -131,6 +135,30 @@ static void create_pack_file(void)\n \tif (use_include_tag)\n \t\targv_array_push(&pack_objects.args, \"--include-tag\");\n \n+\tif (filter_options.omit_all_blobs)\n+\t\targv_array_push(&pack_objects.args,\n+\t\t\t\t(\"--\" CL_ARG_FILTER_OMIT_ALL_BLOBS));\n+\telse if (filter_options.omit_large_blobs) {\n+\t\tif (filter_options.large_byte_limit_string)\n+\t\t\targv_array_pushf(&pack_objects.args, \"--%s=%s\",\n+\t\t\t\t\t CL_ARG_FILTER_OMIT_LARGE_BLOBS,\n+\t\t\t\t\t filter_options.large_byte_limit_string);\n+\t\telse\n+\t\t\targv_array_pushf(&pack_objects.args, \"--%s=%ld\",\n+\t\t\t\t\t CL_ARG_FILTER_OMIT_LARGE_BLOBS,\n+\t\t\t\t\t filter_options.large_byte_limit);\n+\t}\n+\telse if (filter_options.use_sparse) {\n+\t\tif (!oidcmp(&filter_options.sparse_oid, &null_oid))\n+\t\t\targv_array_pushf(&pack_objects.args, \"--%s=%s\",\n+\t\t\t\t\t CL_ARG_FILTER_USE_SPARSE,\n+\t\t\t\t\t oid_to_hex(&filter_options.sparse_oid));\n+\t\telse\n+\t\t\targv_array_pushf(&pack_objects.args, \"--%s=%s\",\n+\t\t\t\t\t CL_ARG_FILTER_USE_SPARSE,\n+\t\t\t\t\t filter_options.sparse_value);\n+\t}\n+\n \tpack_objects.in = -1;\n \tpack_objects.out = -1;\n \tpack_objects.err = -1;\n@@ -793,6 +821,12 @@ static void receive_needs(void)\n \t\t\tdeepen_rev_list = 1;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (object_filter_hand_parse_protocol(&filter_options, line, 0, 0)) {\n+\t\t\tif (!capability_filter_objects_requested)\n+\t\t\t\tdie(\"git upload-pack: object filtering requires '%s' capability\",\n+\t\t\t\t PROTOCOL_CAPABILITY_FILTER_OBJECTS);\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!skip_prefix(line, \"want \", &arg) ||\n \t\t get_sha1_hex(arg, sha1_buf))\n \t\t\tdie(\"git upload-pack: protocol error, \"\n@@ -820,6 +854,8 @@ static void receive_needs(void)\n \t\t\tno_progress = 1;\n \t\tif (parse_feature_request(features, \"include-tag\"))\n \t\t\tuse_include_tag = 1;\n+\t\tif (parse_feature_request(features, PROTOCOL_CAPABILITY_FILTER_OBJECTS))\n+\t\t\tcapability_filter_objects_requested = 1;\n \n \t\to = parse_object(sha1_buf);\n \t\tif (!o) {\n@@ -928,7 +964,8 @@ static int send_ref(const char *refname, const struct object_id *oid,\n {\n \tstatic const char *capabilities = \"multi_ack thin-pack side-band\"\n \t\t\" side-band-64k ofs-delta shallow deepen-since deepen-not\"\n-\t\t\" deepen-relative no-progress include-tag multi_ack_detailed\";\n+\t\t\" deepen-relative no-progress include-tag multi_ack_detailed\"\n+\t\t\" \" PROTOCOL_CAPABILITY_FILTER_OBJECTS;\n \tconst char *refname_nons = strip_namespace(refname);\n \tstruct object_id peeled;\n \n-- \n2.9.3\n\n"},{"id":"324391","messageId":"20170713173459.3559-16-git@jeffhostetler.com","threadId":"46376","inReplyTo":"20170713173459.3559-1-git@jeffhostetler.com","subject":"[PATCH v2 15/19] fetch-pack: add object filtering support","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-07-13T17:34:55Z","receivedAt":"2017-07-13T17:36:02Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fetch-pack.c | 3 +++\n fetch-pack.c | 28 ++++++++++++++++++++++++++++\n fetch-pack.h | 2 ++\n transport.c | 27 +++++++++++++++++++++++++++\n transport.h | 8 ++++++++\n 5 files changed, 68 insertions(+)\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex 366b9d1..72f9063 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -143,6 +143,9 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\targs.update_shallow = 1;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (object_filter_hand_parse_arg(&args.filter_options, arg, 0, 0)) {\n+\t\t\tcontinue;\n+\t\t}\n \t\tusage(fetch_pack_usage);\n \t}\n \tif (deepen_not.nr)\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex afb8b05..642077d 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -374,6 +374,8 @@ static int find_common(struct fetch_pack_args *args,\n \t\t\tif (prefer_ofs_delta) strbuf_addstr(&c, \" ofs-delta\");\n \t\t\tif (deepen_since_ok) strbuf_addstr(&c, \" deepen-since\");\n \t\t\tif (deepen_not_ok) strbuf_addstr(&c, \" deepen-not\");\n+\t\t\tif (object_filter_enabled(&args->filter_options))\n+\t\t\t\tstrbuf_addstr(&c, (\" \" PROTOCOL_CAPABILITY_FILTER_OBJECTS));\n \t\t\tif (agent_supported) strbuf_addf(&c, \" agent=%s\",\n \t\t\t\t\t\t\t git_user_agent_sanitized());\n \t\t\tpacket_buf_write(&req_buf, \"want %s%s\\n\", remote_hex, c.buf);\n@@ -404,6 +406,18 @@ static int find_common(struct fetch_pack_args *args,\n \t\t\tpacket_buf_write(&req_buf, \"deepen-not %s\", s->string);\n \t\t}\n \t}\n+\n+\tif (args->filter_options.omit_all_blobs)\n+\t\tpacket_buf_write(&req_buf, PROTOCOL_REQUEST_FILTER_OMIT_ALL_BLOBS);\n+\telse if (args->filter_options.omit_large_blobs)\n+\t\tpacket_buf_write(&req_buf,\n+\t\t\t\t PROTOCOL_REQUEST_FILTER_OMIT_ALL_BLOBS \" %ld\",\n+\t\t\t\t args->filter_options.large_byte_limit);\n+\telse if (args->filter_options.use_sparse)\n+\t\tpacket_buf_write(&req_buf,\n+\t\t\t\t PROTOCOL_REQUEST_FILTER_USE_SPARSE \" %s\",\n+\t\t\t\t args->filter_options.sparse_value);\n+\n \tpacket_buf_flush(&req_buf);\n \tstate_len = req_buf.len;\n \n@@ -811,6 +825,15 @@ static int get_pack(struct fetch_pack_args *args,\n \t\t\t\t\t\"--keep=fetch-pack %\"PRIuMAX \" on %s\",\n \t\t\t\t\t(uintmax_t)getpid(), hostname);\n \t\t}\n+\n+\t\t/*\n+\t\t * Relax consistency check to allow missing blobs (presumably\n+\t\t * because they are exactly the set that we requested be\n+\t\t * omitted.\n+\t\t */\n+\t\tif (object_filter_enabled(&args->filter_options))\n+\t\t\targv_array_push(&cmd.args, (\"--\" CL_ARG_FILTER_RELAX));\n+\n \t\tif (args->check_self_contained_and_connected)\n \t\t\targv_array_push(&cmd.args, \"--check-self-contained-and-connected\");\n \t}\n@@ -924,6 +947,11 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \telse\n \t\tprefer_ofs_delta = 0;\n \n+\tif (server_supports(PROTOCOL_CAPABILITY_FILTER_OBJECTS))\n+\t\tprint_verbose(args, _(\"Server supports \"PROTOCOL_CAPABILITY_FILTER_OBJECTS));\n+\telse if (object_filter_enabled(&args->filter_options))\n+\t\tdie(_(\"Server does not support \"PROTOCOL_CAPABILITY_FILTER_OBJECTS));\n+\n \tif ((agent_feature = server_feature_value(\"agent\", &agent_len))) {\n \t\tagent_supported = 1;\n \t\tif (agent_len)\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex b6aeb43..5e6bf3b 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -3,6 +3,7 @@\n \n #include \"string-list.h\"\n #include \"run-command.h\"\n+#include \"object-filter.h\"\n \n struct oid_array;\n \n@@ -12,6 +13,7 @@ struct fetch_pack_args {\n \tint depth;\n \tconst char *deepen_since;\n \tconst struct string_list *deepen_not;\n+\tstruct object_filter_options filter_options;\n \tunsigned deepen_relative:1;\n \tunsigned quiet:1;\n \tunsigned keep_pack:1;\ndiff --git a/transport.c b/transport.c\nindex 4d33138..7abf0b6 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -160,6 +160,32 @@ static int set_git_option(struct git_transport_options *opts,\n \t} else if (!strcmp(name, TRANS_OPT_DEEPEN_RELATIVE)) {\n \t\topts->deepen_relative = !!value;\n \t\treturn 0;\n+\t} else if (!strcmp(name, TRANS_OPT_FILTER_OMIT_ALL_BLOBS)) {\n+\t\topts->filter_options.omit_all_blobs = !!value;\n+\t\treturn 0;\n+\t} else if (!strcmp(name, TRANS_OPT_FILTER_OMIT_LARGE_BLOBS)) {\n+\t\topts->filter_options.omit_large_blobs = 1;\n+\t\topts->filter_options.large_byte_limit_string = value;\n+\t\tif (!value)\n+\t\t\topts->filter_options.large_byte_limit = 0;\n+\t\telse if (!git_parse_ulong(value,\n+\t\t\t\t\t &opts->filter_options.large_byte_limit))\n+\t\t\tdie(_(\"transport: invalid filter value '%s'\"), value);\n+\t\treturn 0;\n+\t} else if (!strcmp(name, TRANS_OPT_FILTER_USE_SPARSE)) {\n+\t\topts->filter_options.use_sparse = 1;\n+\t\topts->filter_options.sparse_value = value;\n+\t\t/*\n+\t\t * We're constrained by the API for this set_ operation and\n+\t\t * only take a single value. We don't want to do the get_sha1*()\n+\t\t * lookup (possibly for the second time), because the caller\n+\t\t * should already know and normalized the hex OID string\n+\t\t * (assuming that it used the normal parsing methods). So we\n+\t\t * assume that the above string value is sufficient here and\n+\t\t * can just NULL the binary OID field.\n+\t\t */\n+\t\toidcpy(&opts->filter_options.sparse_oid, &null_oid);\n+\t\treturn 0;\n \t}\n \treturn 1;\n }\n@@ -228,6 +254,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \t\tdata->options.check_self_contained_and_connected;\n \targs.cloning = transport->cloning;\n \targs.update_shallow = data->options.update_shallow;\n+\targs.filter_options = data->options.filter_options;\n \n \tif (!data->got_remote_heads) {\n \t\tconnect_setup(transport, 0);\ndiff --git a/transport.h b/transport.h\nindex bc55715..490f827 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -4,6 +4,8 @@\n #include \"cache.h\"\n #include \"run-command.h\"\n #include \"remote.h\"\n+#include \"fetch-pack.h\"\n+#include \"object-filter.h\"\n \n struct string_list;\n \n@@ -21,6 +23,7 @@ struct git_transport_options {\n \tconst char *uploadpack;\n \tconst char *receivepack;\n \tstruct push_cas_option *cas;\n+\tstruct object_filter_options filter_options;\n };\n \n enum transport_family {\n@@ -210,6 +213,11 @@ void transport_check_allowed(const char *type);\n /* Send push certificates */\n #define TRANS_OPT_PUSH_CERT \"pushcert\"\n \n+/* See Documentation/technical/pack-protocol.txt */\n+#define TRANS_OPT_FILTER_OMIT_ALL_BLOBS \"filter-omit-all-blobs\"\n+#define TRANS_OPT_FILTER_OMIT_LARGE_BLOBS \"filter-omit-large-blobs\"\n+#define TRANS_OPT_FILTER_USE_SPARSE \"filter-use-sparse\"\n+\n /**\n * Returns 0 if the option was used, non-zero otherwise. Prints a\n * message to stderr if the option is not used.\n-- \n2.9.3\n\n"},{"id":"324392","messageId":"20170713173459.3559-17-git@jeffhostetler.com","threadId":"46376","inReplyTo":"20170713173459.3559-1-git@jeffhostetler.com","subject":"[PATCH v2 16/19] connected: add filter_allow_omitted option to API","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-07-13T17:34:56Z","receivedAt":"2017-07-13T17:36:03Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n connected.c | 3 +++\n connected.h | 6 ++++++\n 2 files changed, 9 insertions(+)\n\ndiff --git a/connected.c b/connected.c\nindex 136c2ac..c25b816 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -62,6 +62,9 @@ int check_connected(sha1_iterate_fn fn, void *cb_data,\n \t\targv_array_pushf(&rev_list.args, \"--progress=%s\",\n \t\t\t\t _(\"Checking connectivity\"));\n \n+\tif (opt->filter_relax)\n+\t\targv_array_push(&rev_list.args, (\"--\" CL_ARG_FILTER_RELAX));\n+\n \trev_list.git_cmd = 1;\n \trev_list.env = opt->env;\n \trev_list.in = -1;\ndiff --git a/connected.h b/connected.h\nindex 4ca325f..370710e 100644\n--- a/connected.h\n+++ b/connected.h\n@@ -35,6 +35,12 @@ struct check_connected_options {\n \tint progress;\n \n \t/*\n+\t * Relax consistency checks for missing blobs (presumably\n+\t * due to the use 'filter-objects' feature).\n+\t */\n+\tint filter_relax;\n+\n+\t/*\n \t * Insert these variables into the environment of the child process.\n \t */\n \tconst char **env;\n-- \n2.9.3\n\n"},{"id":"324393","messageId":"20170713173459.3559-20-git@jeffhostetler.com","threadId":"46376","inReplyTo":"20170713173459.3559-1-git@jeffhostetler.com","subject":"[PATCH v2 19/19] fetch: add object filtering to fetch","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-07-13T17:34:59Z","receivedAt":"2017-07-13T17:36:05Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fetch.c | 27 ++++++++++++++++++++++++++-\n 1 file changed, 26 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 5f2c2ab..306c165 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -16,6 +16,7 @@\n #include \"connected.h\"\n #include \"argv-array.h\"\n #include \"utf8.h\"\n+#include \"object-filter.h\"\n \n static const char * const builtin_fetch_usage[] = {\n \tN_(\"git fetch [] [ [...]]\"),\n@@ -52,6 +53,7 @@ static const char *recurse_submodules_default;\n static int shown_url = 0;\n static int refmap_alloc, refmap_nr;\n static const char **refmap_array;\n+static struct object_filter_options filter_options;\n \n static int option_parse_recurse_submodules(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n@@ -141,6 +143,14 @@ static struct option builtin_fetch_options[] = {\n \t\t\tTRANSPORT_FAMILY_IPV4),\n \tOPT_SET_INT('6', \"ipv6\", &family, N_(\"use IPv6 addresses only\"),\n \t\t\tTRANSPORT_FAMILY_IPV6),\n+\n+\tOPT_PARSE_FILTER_OMIT_ALL_BLOBS(&filter_options),\n+\tOPT_PARSE_FILTER_OMIT_LARGE_BLOBS(&filter_options),\n+\tOPT_PARSE_FILTER_USE_SPARSE(&filter_options),\n+\n+\t/* OPT_PARSE_FILTER_PRINT_MANIFEST(&filter_options), */\n+\t/* OPT_PARSE_FILTER_RELAX(&filter_options), */\n+\n \tOPT_END()\n };\n \n@@ -733,6 +743,14 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n \tconst char *filename = dry_run ? \"/dev/null\" : git_path_fetch_head();\n \tint want_status;\n \tint summary_width = transport_summary_width(ref_map);\n+\tstruct check_connected_options opt = CHECK_CONNECTED_INIT;\n+\n+\t/*\n+\t * Relax consistency check to allow missing blobs (presumably\n+\t * because they are exactly the set that we requested be\n+\t * omitted.\n+\t */\n+\topt.filter_relax = object_filter_enabled(&filter_options);\n \n \tfp = fopen(filename, \"a\");\n \tif (!fp)\n@@ -744,7 +762,7 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n \t\turl = xstrdup(\"foreign\");\n \n \trm = ref_map;\n-\tif (check_connected(iterate_ref_map, &rm, NULL)) {\n+\tif (check_connected(iterate_ref_map, &rm, &opt)) {\n \t\trc = error(_(\"%s did not send all necessary objects\\n\"), url);\n \t\tgoto abort;\n \t}\n@@ -885,6 +903,13 @@ static int quickfetch(struct ref *ref_map)\n \tstruct check_connected_options opt = CHECK_CONNECTED_INIT;\n \n \t/*\n+\t * Relax consistency check to allow missing blobs (presumably\n+\t * because they are exactly the set that we requested be\n+\t * omitted.\n+\t */\n+\topt.filter_relax = object_filter_enabled(&filter_options);\n+\n+\t/*\n \t * If we are deepening a shallow clone we already have these\n \t * objects reachable. Running rev-list here will return with\n \t * a good (0) exit status and we'll bypass the fetch that we\n-- \n2.9.3\n\n"},{"id":"324394","messageId":"20170713173459.3559-9-git@jeffhostetler.com","threadId":"46376","inReplyTo":"20170713173459.3559-1-git@jeffhostetler.com","subject":"[PATCH v2 08/19] rev-list: add object filtering support","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-07-13T17:34:48Z","receivedAt":"2017-07-13T17:36:10Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach rev-list to use the filtering provided by the\ntraverse_commit_list_filtered() interface to omit\nunwanted objects from the result.\n\nThis feature is only enabled when one of the \"--objects*\"\noptions are used.\n\nWhen the \"--filter-print-manifest\" option is used, the\nomitted objects and their sizes are printed at the end.\nThese are marked with a \"~\". This can be combined with\n\"--quiet\" to get a list of just the omitted objects.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/rev-list.c | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 56 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex bcf77f0..fd9a7e5 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -3,6 +3,8 @@\n #include \"diff.h\"\n #include \"revision.h\"\n #include \"list-objects.h\"\n+#include \"list-objects-filters.h\"\n+#include \"object-filter.h\"\n #include \"pack.h\"\n #include \"pack-bitmap.h\"\n #include \"builtin.h\"\n@@ -52,6 +54,7 @@ static const char rev_list_usage[] =\n \n static struct progress *progress;\n static unsigned progress_counter;\n+static struct object_filter_options filter_options;\n \n static void finish_commit(struct commit *commit, void *data);\n static void show_commit(struct commit *commit, void *data)\n@@ -178,8 +181,20 @@ static void finish_commit(struct commit *commit, void *data)\n static void finish_object(struct object *obj, const char *name, void *cb_data)\n {\n \tstruct rev_list_info *info = cb_data;\n-\tif (obj->type == OBJ_BLOB && !has_object_file(&obj->oid))\n+\tif (obj->type == OBJ_BLOB && !has_object_file(&obj->oid)) {\n+\t\tif (filter_options.relax) {\n+\t\t\t/*\n+\t\t\t * Relax consistency checks to not complain about\n+\t\t\t * omitted objects (presumably caused by use of\n+\t\t\t * the previous use of the 'filter-objects' feature).\n+\t\t\t *\n+\t\t\t * Note that this is independent of any filtering that\n+\t\t\t * we are doing in this run.\n+\t\t\t */\n+\t\t\treturn;\n+\t\t}\n \t\tdie(\"missing blob object '%s'\", oid_to_hex(&obj->oid));\n+\t}\n \tif (info->revs->verify_objects && !obj->parsed && obj->type != OBJ_COMMIT)\n \t\tparse_object(obj->oid.hash);\n }\n@@ -199,6 +214,16 @@ static void show_edge(struct commit *commit)\n \tprintf(\"-%s\\n\", oid_to_hex(&commit->object.oid));\n }\n \n+static void print_omitted_object(int i, int i_limit, struct oidset2_entry *e, void *cb_data)\n+{\n+\t/* struct rev_list_info *info = cb_data; */\n+\n+\tif (e->object_length == -1)\n+\t\tprintf(\"~%s\\n\", oid_to_hex(&e->oid));\n+\telse\n+\t\tprintf(\"~%s %\"PRIuMAX\"\\n\", oid_to_hex(&e->oid), e->object_length);\n+}\n+\n static void print_var_str(const char *var, const char *val)\n {\n \tprintf(\"%s='%s'\\n\", var, val);\n@@ -276,6 +301,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \tint bisect_find_all = 0;\n \tint use_bitmap_index = 0;\n \tconst char *show_progress = NULL;\n+\toidset2_foreach_cb fn_filter_print = NULL;\n \n \tgit_config(git_default_config, NULL);\n \tinit_revisions(&revs, prefix);\n@@ -329,6 +355,14 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\tshow_progress = arg;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (object_filter_hand_parse_arg(&filter_options, arg, 1, 1)) {\n+\t\t\tif (!revs.blob_objects)\n+\t\t\t\tdie(_(\"object filtering requires --objects\"));\n+\t\t\tif (filter_options.use_sparse &&\n+\t\t\t !oidcmp(&filter_options.sparse_oid, &null_oid))\n+\t\t\t\tdie(_(\"invalid sparse value\"));\n+\t\t\tcontinue;\n+\t\t}\n \t\tusage(rev_list_usage);\n \n \t}\n@@ -353,6 +387,11 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \tif (revs.show_notes)\n \t\tdie(_(\"rev-list does not support display of notes\"));\n \n+\tif (object_filter_enabled(&filter_options)) {\n+\t\tif (use_bitmap_index)\n+\t\t\tdie(_(\"cannot combine --use-bitmap-index with object filtering\"));\n+\t}\n+\n \tsave_commit_buffer = (revs.verbose_header ||\n \t\t\t revs.grep_filter.pattern_list ||\n \t\t\t revs.grep_filter.header_list);\n@@ -397,7 +436,22 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\treturn show_bisect_vars(&info, reaches, all);\n \t}\n \n-\ttraverse_commit_list(&revs, show_commit, show_object, &info);\n+\tif (filter_options.print_manifest)\n+\t\tfn_filter_print = print_omitted_object;\n+\n+\tif (filter_options.omit_all_blobs)\n+\t\ttraverse_commit_list_omit_all_blobs(\n+\t\t\t&revs, show_commit, show_object, fn_filter_print, &info);\n+\telse if (filter_options.omit_large_blobs)\n+\t\ttraverse_commit_list_omit_large_blobs(\n+\t\t\t&revs, show_commit, show_object, fn_filter_print, &info,\n+\t\t\t(int64_t)(uint64_t)filter_options.large_byte_limit);\n+\telse if (filter_options.use_sparse)\n+\t\ttraverse_commit_list_use_sparse(\n+\t\t\t&revs, show_commit, show_object, fn_filter_print, &info,\n+\t\t\t&filter_options.sparse_oid);\n+\telse\n+\t\ttraverse_commit_list(&revs, show_commit, show_object, &info);\n \n \tstop_progress(&progress);\n \n-- \n2.9.3\n\n"},{"id":"324395","messageId":"20170713173459.3559-10-git@jeffhostetler.com","threadId":"46376","inReplyTo":"20170713173459.3559-1-git@jeffhostetler.com","subject":"[PATCH v2 09/19] rev-list: add filtering help text","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-07-13T17:34:49Z","receivedAt":"2017-07-13T17:36:12Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n Documentation/git-rev-list.txt | 7 ++++++-\n Documentation/rev-list-options.txt | 26 ++++++++++++++++++++++++++\n 2 files changed, 32 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-rev-list.txt b/Documentation/git-rev-list.txt\nindex ef22f17..d20c2ab 100644\n--- a/Documentation/git-rev-list.txt\n+++ b/Documentation/git-rev-list.txt\n@@ -47,7 +47,12 @@ SYNOPSIS\n \t [ --fixed-strings | -F ]\n \t [ --date=]\n \t [ [ --objects | --objects-edge | --objects-edge-aggressive ]\n-\t [ --unpacked ] ]\n+\t [ --unpacked ]\n+\t [ [ --filter-omit-all-blobs |\n+\t\t --filter-omit-large-blobs=[kmg] |\n+\t\t --filter-use-sparse= ]\n+\t\t [ --filter-print-manifest ] ] ]\n+\t [ --filter-relax ]\n \t [ --pretty | --header ]\n \t [ --bisect ]\n \t [ --bisect-vars ]\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex a02f732..e0112dd 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -693,6 +693,32 @@ ifdef::git-rev-list[]\n --unpacked::\n \tOnly useful with `--objects`; print the object IDs that are not\n \tin packs.\n+\n+--filter-omit-all-blobs::\n+\tOnly useful with one of the `--objects*`; omits all blobs from\n+\tthe printed list of objects.\n+\n+--filter-omit-large-blobs=[kmg]::\n+\tOnly useful with one of the `--objects*`; omits blobs larger than\n+\tn bytes from the printed list of objects. May optionally be\n+\tfollowed by 'k', 'm', or 'g' units. Value may be zero. Special\n+\tfiles (matching \".git*\") are always included, regardless of size.\n+\n+--filter-use-sparse=::\n+\tOnly useful with one of the `--objects*`; uses a sparse-checkout\n+\tspecification contained in the given object to filter the result\n+\tby omitting blobs that would not be used by the corresponding\n+\tsparse checkout.\n+\n+--filter-print-manifest::\n+\tOnly useful with one of the above `--filter*`; prints a manifest\n+\tof the omitted objects. Object IDs are prefixed with a ``~''\n+\tcharacter. The object size is printed after the ID.\n+\n+--filter-relax::\n+\tRelax consistency checking for missing blobs. Do not warn of\n+\tmissing blobs during normal (non-filtering) object traversal\n+\tfollowing an earlier partial/narrow clone or fetch.\n endif::git-rev-list[]\n \n --no-walk[=(sorted|unsorted)]::\n-- \n2.9.3\n\n"},{"id":"324396","messageId":"20170713173459.3559-3-git@jeffhostetler.com","threadId":"46376","inReplyTo":"20170713173459.3559-1-git@jeffhostetler.com","subject":"[PATCH v2 02/19] oidset2: create oidset subclass with object length and pathname","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-07-13T17:34:42Z","receivedAt":"2017-07-13T17:37:26Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate subclass of oidset where each entry has a\nfield to store the length of the object's content\nand an optional pathname.\n\nThis will be used in a future commit to build a\nmanifest of omitted objects in a partial/narrow\nclone/fetch.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 1 +\n oidset2.c | 101 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n oidset2.h | 56 ++++++++++++++++++++++++++++++++++\n 3 files changed, 158 insertions(+)\n create mode 100644 oidset2.c\n create mode 100644 oidset2.h\n\ndiff --git a/Makefile b/Makefile\nindex ffa6da7..d590508 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -791,6 +791,7 @@ LIB_OBJS += notes-merge.o\n LIB_OBJS += notes-utils.o\n LIB_OBJS += object.o\n LIB_OBJS += oidset.o\n+LIB_OBJS += oidset2.o\n LIB_OBJS += pack-bitmap.o\n LIB_OBJS += pack-bitmap-write.o\n LIB_OBJS += pack-check.o\ndiff --git a/oidset2.c b/oidset2.c\nnew file mode 100644\nindex 0000000..806d153\n--- /dev/null\n+++ b/oidset2.c\n@@ -0,0 +1,101 @@\n+#include \"cache.h\"\n+#include \"oidset2.h\"\n+\n+static int oidset2_hashcmp(const void *va, const void *vb,\n+\t\t\t const void *vkey)\n+{\n+\tconst struct oidset2_entry *a = va, *b = vb;\n+\tconst struct object_id *key = vkey;\n+\treturn oidcmp(&a->oid, key ? key : &b->oid);\n+}\n+\n+struct oidset2_entry *oidset2_get(const struct oidset2 *set, const struct object_id *oid)\n+{\n+\tstruct hashmap_entry key;\n+\tstruct oidset2_entry *value;\n+\n+\tif (!set->map.cmpfn)\n+\t\treturn NULL;\n+\n+\thashmap_entry_init(&key, sha1hash(oid->hash));\n+\tvalue = hashmap_get(&set->map, &key, oid);\n+\n+\treturn value;\n+}\n+\n+int oidset2_contains(const struct oidset2 *set, const struct object_id *oid)\n+{\n+\treturn !!oidset2_get(set, oid);\n+}\n+\n+int oidset2_insert(struct oidset2 *set, const struct object_id *oid,\n+\t\t int64_t object_length, const char *pathname)\n+{\n+\tstruct oidset2_entry *entry;\n+\n+\tif (!set->map.cmpfn)\n+\t\thashmap_init(&set->map, oidset2_hashcmp, 0);\n+\n+\tif (oidset2_contains(set, oid))\n+\t\treturn 1;\n+\n+\tentry = xcalloc(1, sizeof(*entry));\n+\thashmap_entry_init(&entry->hash, sha1hash(oid->hash));\n+\toidcpy(&entry->oid, oid);\n+\n+\tentry->object_length = object_length;\n+\tif (pathname)\n+\t entry->pathname = strdup(pathname);\n+\n+\thashmap_add(&set->map, entry);\n+\treturn 0;\n+}\n+\n+void oidset2_remove(struct oidset2 *set, const struct object_id *oid)\n+{\n+\tstruct hashmap_entry key;\n+\tstruct oidset2_entry *e;\n+\n+\thashmap_entry_init(&key, sha1hash(oid->hash));\n+\te = hashmap_remove(&set->map, &key, oid);\n+\n+\tfree(e->pathname);\n+\tfree(e);\n+}\n+\n+void oidset2_clear(struct oidset2 *set)\n+{\n+\thashmap_free(&set->map, 1);\n+}\n+\n+static int oidset2_cmp(const void *a, const void *b)\n+{\n+\tconst struct oidset2_entry *ae = *((const struct oidset2_entry **)a);\n+\tconst struct oidset2_entry *be = *((const struct oidset2_entry **)b);\n+\n+\treturn oidcmp(&ae->oid, &be->oid);\n+}\n+\n+void oidset2_foreach(struct oidset2 *set, oidset2_foreach_cb cb, void *cb_data)\n+{\n+\tstruct hashmap_iter iter;\n+\tstruct oidset2_entry **array;\n+\tstruct oidset2_entry *e;\n+\tint j, k;\n+\n+\tarray = xcalloc(set->map.size, sizeof(*e));\n+\n+\thashmap_iter_init(&set->map, &iter);\n+\tk = 0;\n+\twhile ((e = hashmap_iter_next(&iter)))\n+\t\tarray[k++] = e;\n+\n+\tQSORT(array, k, oidset2_cmp);\n+\n+\tfor (j = 0; j < k; j++) {\n+\t\te = array[j];\n+\t\tcb(j, k, e, cb_data);\n+\t}\n+\n+\tfree(array);\n+}\ndiff --git a/oidset2.h b/oidset2.h\nnew file mode 100644\nindex 0000000..c498eae\n--- /dev/null\n+++ b/oidset2.h\n@@ -0,0 +1,56 @@\n+#ifndef OIDSET2_H\n+#define OIDSET2_H\n+\n+/**\n+ * oidset2 is a variant of oidset, but allows additional fields for each object.\n+ */\n+\n+/**\n+ * A single oidset2; should be zero-initialized (or use OIDSET2_INIT).\n+ */\n+struct oidset2 {\n+\tstruct hashmap map;\n+};\n+\n+#define OIDSET2_INIT { { NULL } }\n+\n+struct oidset2_entry {\n+\tstruct hashmap_entry hash;\n+\tstruct object_id oid;\n+\n+\tint64_t object_length;\t/* This is SIGNED. Use -1 when unknown. */\n+\tchar *pathname;\n+};\n+\n+struct oidset2_entry *oidset2_get(const struct oidset2 *set, const struct object_id *oid);\n+\n+/**\n+ * Returns true iff `set` contains `oid`.\n+ */\n+int oidset2_contains(const struct oidset2 *set, const struct object_id *oid);\n+\n+/**\n+ * Insert the oid into the set; a copy is made, so \"oid\" does not need\n+ * to persist after this function is called.\n+ *\n+ * Returns 1 if the oid was already in the set, 0 otherwise. This can be used\n+ * to perform an efficient check-and-add.\n+ */\n+int oidset2_insert(struct oidset2 *set, const struct object_id *oid,\n+\t\t int64_t object_length, const char *pathname);\n+\n+void oidset2_remove(struct oidset2 *set, const struct object_id *oid);\n+\n+typedef void (*oidset2_foreach_cb)(\n+\tint i, int i_limit,\n+\tstruct oidset2_entry *e, void *cb_data);\n+\n+void oidset2_foreach(struct oidset2 *set, oidset2_foreach_cb cb, void *cb_data);\n+\n+/**\n+ * Remove all entries from the oidset2, freeing any resources associated with\n+ * it.\n+ */\n+void oidset2_clear(struct oidset2 *set);\n+\n+#endif /* OIDSET2_H */\n-- \n2.9.3\n\n"},{"id":"324397","messageId":"20170713173459.3559-6-git@jeffhostetler.com","threadId":"46376","inReplyTo":"20170713173459.3559-1-git@jeffhostetler.com","subject":"[PATCH v2 05/19] list-objects-filters: add omit-large-blobs filter","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-07-13T17:34:45Z","receivedAt":"2017-07-13T17:37:43Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a filter for traverse_commit_list_filtered() to omit\nblobs larger than a requested size from the result, but always\ninclude \".git*\" special files.\n\nThis filter will be used in a future commit by rev-list and\npack-objects for partial/narrow clone/fetch.\n\nSigned-off-by: Jeff Hostetler \n---\n list-objects-filters.c | 97 ++++++++++++++++++++++++++++++++++++++++++++++++++\n list-objects-filters.h | 12 +++++++\n 2 files changed, 109 insertions(+)\n\ndiff --git a/list-objects-filters.c b/list-objects-filters.c\nindex f29d8bc..f04d70e 100644\n--- a/list-objects-filters.c\n+++ b/list-objects-filters.c\n@@ -83,3 +83,100 @@ void traverse_commit_list_omit_all_blobs(\n \n \toidset2_clear(&d.omits);\n }\n+\n+/*\n+ * A filter for list-objects to omit large blobs,\n+ * but always include \".git*\" special files.\n+ */\n+struct filter_omit_large_blobs_data {\n+\tstruct oidset2 omits;\n+\tint64_t max_bytes;\n+};\n+\n+static list_objects_filter_result filter_omit_large_blobs(\n+\tlist_objects_filter_type filter_type,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tvoid *filter_data_)\n+{\n+\tstruct filter_omit_large_blobs_data *filter_data = filter_data_;\n+\tint64_t object_length = -1;\n+\tunsigned long s;\n+\tenum object_type t;\n+\n+\tswitch (filter_type) {\n+\tdefault:\n+\t\tdie(\"unkown filter_type\");\n+\t\treturn LOFR_ZERO;\n+\n+\tcase LOFT_BEGIN_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\t/* always include all tree objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_SHOW;\n+\n+\tcase LOFT_END_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\treturn LOFR_ZERO;\n+\n+\tcase LOFT_BLOB:\n+\t\tassert(obj->type == OBJ_BLOB);\n+\t\tassert((obj->flags & SEEN) == 0);\n+\n+\t\t/*\n+\t\t * If previously provisionally omitted (because of size), see if the\n+\t\t * current filename is special and force it to be included.\n+\t\t */\n+\t\tif (oidset2_contains(&filter_data->omits, &obj->oid)) {\n+\t\t\tif ((strncmp(filename, \".git\", 4) == 0) && filename[4]) {\n+\t\t\t\toidset2_remove(&filter_data->omits, &obj->oid);\n+\t\t\t\treturn LOFR_MARK_SEEN | LOFR_SHOW;\n+\t\t\t}\n+\t\t\treturn LOFR_ZERO; /* continue provisionally omitting it */\n+\t\t}\n+\n+\t\tt = sha1_object_info(obj->oid.hash, &s);\n+\t\tassert(t == OBJ_BLOB);\n+\t\tobject_length = (int64_t)((uint64_t)(s));\n+\n+\t\tif (object_length < filter_data->max_bytes)\n+\t\t\treturn LOFR_MARK_SEEN | LOFR_SHOW;\n+\n+\t\t/*\n+\t\t * Provisionally omit it. We've already established that this blob\n+\t\t * is too big and doesn't have a special filename, so we WANT to\n+\t\t * omit it. However, there may be a special file elsewhere in the\n+\t\t * tree that references this same blob, so we cannot reject it yet.\n+\t\t * Leave the LOFR_ bits unset so that if the blob appears again in\n+\t\t * the traversal, we will be asked again.\n+\t\t *\n+\t\t * No need for a pathname, since we only test for special filenames\n+\t\t * above.\n+\t\t */\n+\t\toidset2_insert(&filter_data->omits, &obj->oid, object_length,\n+\t\t\t NULL);\n+\t\treturn LOFR_ZERO;\n+\t}\n+}\n+\n+void traverse_commit_list_omit_large_blobs(\n+\tstruct rev_info *revs,\n+\tshow_commit_fn show_commit,\n+\tshow_object_fn show_object,\n+\toidset2_foreach_cb print_omitted_object,\n+\tvoid *ctx_data,\n+\tint64_t large_byte_limit)\n+{\n+\tstruct filter_omit_large_blobs_data d;\n+\n+\tmemset(&d, 0, sizeof(d));\n+\td.max_bytes = large_byte_limit;\n+\n+\ttraverse_commit_list_filtered(revs, show_commit, show_object, ctx_data,\n+\t\t\t\t filter_omit_large_blobs, &d);\n+\n+\tif (print_omitted_object)\n+\t\toidset2_foreach(&d.omits, print_omitted_object, ctx_data);\n+\n+\toidset2_clear(&d.omits);\n+}\ndiff --git a/list-objects-filters.h b/list-objects-filters.h\nindex b981020..32b2833 100644\n--- a/list-objects-filters.h\n+++ b/list-objects-filters.h\n@@ -14,4 +14,16 @@ void traverse_commit_list_omit_all_blobs(\n \toidset2_foreach_cb print_omitted_object,\n \tvoid *ctx_data);\n \n+/*\n+ * A filter for list-objects to omit large blobs,\n+ * but always include \".git*\" special files.\n+ */\n+void traverse_commit_list_omit_large_blobs(\n+\tstruct rev_info *revs,\n+\tshow_commit_fn show_commit,\n+\tshow_object_fn show_object,\n+\toidset2_foreach_cb print_omitted_object,\n+\tvoid *ctx_data,\n+\tint64_t large_byte_limit);\n+\n #endif /* LIST_OBJECTS_FILTERS_H */\n-- \n2.9.3\n\n"},{"id":"324398","messageId":"20170713173459.3559-5-git@jeffhostetler.com","threadId":"46376","inReplyTo":"20170713173459.3559-1-git@jeffhostetler.com","subject":"[PATCH v2 04/19] list-objects-filters: add omit-all-blobs filter","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-07-13T17:34:44Z","receivedAt":"2017-07-13T17:38:00Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a simple filter for traverse_commit_list_filtered() to\nomit all blobs from the result.\n\nThis filter will be used in a future commit by rev-list and\npack-objects to create a \"commits and trees\" result. This\nis intended for a narrow/partial clone/fetch.\n\nSigned-off-by: Jeff Hostetler \n---\n Makefile | 1 +\n list-objects-filters.c | 85 ++++++++++++++++++++++++++++++++++++++++++++++++++\n list-objects-filters.h | 17 ++++++++++\n 3 files changed, 103 insertions(+)\n create mode 100644 list-objects-filters.c\n create mode 100644 list-objects-filters.h\n\ndiff --git a/Makefile b/Makefile\nindex d590508..48fdcf2 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -773,6 +773,7 @@ LIB_OBJS += levenshtein.o\n LIB_OBJS += line-log.o\n LIB_OBJS += line-range.o\n LIB_OBJS += list-objects.o\n+LIB_OBJS += list-objects-filters.o\n LIB_OBJS += ll-merge.o\n LIB_OBJS += lockfile.o\n LIB_OBJS += log-tree.o\ndiff --git a/list-objects-filters.c b/list-objects-filters.c\nnew file mode 100644\nindex 0000000..f29d8bc\n--- /dev/null\n+++ b/list-objects-filters.c\n@@ -0,0 +1,85 @@\n+#include \"cache.h\"\n+#include \"dir.h\"\n+#include \"tag.h\"\n+#include \"commit.h\"\n+#include \"tree.h\"\n+#include \"blob.h\"\n+#include \"diff.h\"\n+#include \"tree-walk.h\"\n+#include \"revision.h\"\n+#include \"list-objects.h\"\n+#include \"list-objects-filters.h\"\n+\n+/*\n+ * A filter for list-objects to omit ALL blobs from the traversal.\n+ */\n+struct filter_omit_all_blobs_data {\n+\tstruct oidset2 omits;\n+};\n+\n+static list_objects_filter_result filter_omit_all_blobs(\n+\tlist_objects_filter_type filter_type,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tvoid *filter_data_)\n+{\n+\tstruct filter_omit_all_blobs_data *filter_data = filter_data_;\n+\tint64_t object_length = -1;\n+\tunsigned long s;\n+\tenum object_type t;\n+\n+\tswitch (filter_type) {\n+\tdefault:\n+\t\tdie(\"unkown filter_type\");\n+\t\treturn LOFR_ZERO;\n+\n+\tcase LOFT_BEGIN_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\t/* always include all tree objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_SHOW;\n+\n+\tcase LOFT_END_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\treturn LOFR_ZERO;\n+\n+\tcase LOFT_BLOB:\n+\t\tassert(obj->type == OBJ_BLOB);\n+\t\tassert((obj->flags & SEEN) == 0);\n+\n+\t\t/*\n+\t\t * Since we always omit all blobs (and never provisionally omit),\n+\t\t * we should never see a blob twice.\n+\t\t */\n+\t\tassert(!oidset2_contains(&filter_data->omits, &obj->oid));\n+\n+\t\tt = sha1_object_info(obj->oid.hash, &s);\n+\t\tassert(t == OBJ_BLOB);\n+\t\tobject_length = (int64_t)((uint64_t)(s));\n+\n+\t\t/* Insert OID into the omitted list. No need for a pathname. */\n+\t\toidset2_insert(&filter_data->omits, &obj->oid, object_length,\n+\t\t\t NULL);\n+\t\treturn LOFR_MARK_SEEN; /* but not LOFR_SHOW (hard omit) */\n+\t}\n+}\n+\n+void traverse_commit_list_omit_all_blobs(\n+\tstruct rev_info *revs,\n+\tshow_commit_fn show_commit,\n+\tshow_object_fn show_object,\n+\toidset2_foreach_cb print_omitted_object,\n+\tvoid *ctx_data)\n+{\n+\tstruct filter_omit_all_blobs_data d;\n+\n+\tmemset(&d, 0, sizeof(d));\n+\n+\ttraverse_commit_list_filtered(revs, show_commit, show_object, ctx_data,\n+\t\t\t\t filter_omit_all_blobs, &d);\n+\n+\tif (print_omitted_object)\n+\t\toidset2_foreach(&d.omits, print_omitted_object, ctx_data);\n+\n+\toidset2_clear(&d.omits);\n+}\ndiff --git a/list-objects-filters.h b/list-objects-filters.h\nnew file mode 100644\nindex 0000000..b981020\n--- /dev/null\n+++ b/list-objects-filters.h\n@@ -0,0 +1,17 @@\n+#ifndef LIST_OBJECTS_FILTERS_H\n+#define LIST_OBJECTS_FILTERS_H\n+\n+#include \"oidset2.h\"\n+\n+/*\n+ * A filter for list-objects to omit ALL blobs\n+ * from the traversal.\n+ */\n+void traverse_commit_list_omit_all_blobs(\n+\tstruct rev_info *revs,\n+\tshow_commit_fn show_commit,\n+\tshow_object_fn show_object,\n+\toidset2_foreach_cb print_omitted_object,\n+\tvoid *ctx_data);\n+\n+#endif /* LIST_OBJECTS_FILTERS_H */\n-- \n2.9.3\n\n"}]}