{"thread":{"id":"46810","subject":"[PATCH 05/13] list-objects-filter-large: add large blob filter to list-objects","startedAt":"2017-09-22T20:31:18Z","lastAt":"2017-09-29T19:47:10Z","messageCount":16,"participants":["Jeff Hostetler","Jonathan Tan"],"isPatch":true,"patchVersion":1,"patchTotal":13},"messages":[{"id":"328674","messageId":"20170922203017.53986-6-git@jeffhostetler.com","threadId":"46810","inReplyTo":null,"subject":"[PATCH 05/13] list-objects-filter-large: add large blob filter to list-objects","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-09-22T20:30:09Z","receivedAt":"2017-09-22T20:31:18Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nCreate a filter for traverse_commit_list_worker() to omit blobs\nlarger than a requested size from the result, but always include\n\".git*\" special files.\n\nSigned-off-by: Jeff Hostetler <jeffhost>@microsoft.com>\n---\n Makefile                    |   1 +\n list-objects-filter-large.c | 108 ++++++++++++++++++++++++++++++++++++++++++++\n list-objects-filter-large.h |  18 ++++++++\n 3 files changed, 127 insertions(+)\n create mode 100644 list-objects-filter-large.c\n create mode 100644 list-objects-filter-large.h\n\ndiff --git a/Makefile b/Makefile\nindex b98e3dc..f1f3979 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -799,6 +799,7 @@ LIB_OBJS += line-log.o\n LIB_OBJS += line-range.o\n LIB_OBJS += list-objects.o\n LIB_OBJS += list-objects-filter-all.o\n+LIB_OBJS += list-objects-filter-large.o\n LIB_OBJS += ll-merge.o\n LIB_OBJS += lockfile.o\n LIB_OBJS += log-tree.o\ndiff --git a/list-objects-filter-large.c b/list-objects-filter-large.c\nnew file mode 100644\nindex 0000000..1af39b6\n--- /dev/null\n+++ b/list-objects-filter-large.c\n@@ -0,0 +1,108 @@\n+#include \"cache.h\"\n+#include \"dir.h\"\n+#include \"tag.h\"\n+#include \"commit.h\"\n+#include \"tree.h\"\n+#include \"blob.h\"\n+#include \"diff.h\"\n+#include \"tree-walk.h\"\n+#include \"revision.h\"\n+#include \"list-objects.h\"\n+#include \"list-objects-filter-large.h\"\n+\n+/*\n+ * A filter for list-objects to omit large blobs,\n+ * but always include \".git*\" special files.\n+ */\n+struct filter_omit_large_blobs_data {\n+\tstruct oidset2 omits;\n+\tint64_t max_bytes;\n+};\n+\n+static list_objects_filter_result filter_omit_large_blobs(\n+\tlist_objects_filter_type filter_type,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tvoid *filter_data_)\n+{\n+\tstruct filter_omit_large_blobs_data *filter_data = filter_data_;\n+\tint64_t object_length = -1;\n+\tunsigned long s;\n+\tenum object_type t;\n+\n+\tswitch (filter_type) {\n+\tdefault:\n+\t\tdie(\"unkown filter_type\");\n+\t\treturn LOFR_ZERO;\n+\n+\tcase LOFT_BEGIN_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\t/* always include all tree objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_SHOW;\n+\n+\tcase LOFT_END_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\treturn LOFR_ZERO;\n+\n+\tcase LOFT_BLOB:\n+\t\tassert(obj->type == OBJ_BLOB);\n+\t\tassert((obj->flags & SEEN) == 0);\n+\n+\t\t/*\n+\t\t * If previously provisionally omitted (because of size), see if the\n+\t\t * current filename is special and force it to be included.\n+\t\t */\n+\t\tif (oidset2_contains(&filter_data->omits, &obj->oid)) {\n+\t\t\tif ((strncmp(filename, \".git\", 4) == 0) && filename[4]) {\n+\t\t\t\toidset2_remove(&filter_data->omits, &obj->oid);\n+\t\t\t\treturn LOFR_MARK_SEEN | LOFR_SHOW;\n+\t\t\t}\n+\t\t\treturn LOFR_ZERO; /* continue provisionally omitting it */\n+\t\t}\n+\n+\t\tt = sha1_object_info(obj->oid.hash, &s);\n+\t\tassert(t == OBJ_BLOB);\n+\t\tobject_length = (int64_t)((uint64_t)(s));\n+\n+\t\tif (object_length < filter_data->max_bytes)\n+\t\t\treturn LOFR_MARK_SEEN | LOFR_SHOW;\n+\n+\t\t/*\n+\t\t * Provisionally omit it.  We've already established that this blob\n+\t\t * is too big and doesn't have a special filename, so we WANT to\n+\t\t * omit it.  However, there may be a special file elsewhere in the\n+\t\t * tree that references this same blob, so we cannot reject it yet.\n+\t\t * Leave the LOFR_ bits unset so that if the blob appears again in\n+\t\t * the traversal, we will be asked again.\n+\t\t *\n+\t\t * No need for a pathname, since we only test for special filenames\n+\t\t * above.\n+\t\t */\n+\t\toidset2_insert(&filter_data->omits, &obj->oid, t, object_length,\n+\t\t\t       NULL);\n+\t\treturn LOFR_ZERO;\n+\t}\n+}\n+\n+void traverse_commit_list_omit_large_blobs(\n+\tstruct rev_info *revs,\n+\tshow_commit_fn show_commit,\n+\tshow_object_fn show_object,\n+\toidset2_foreach_cb print_omitted_object,\n+\tvoid *ctx_data,\n+\tint64_t large_byte_limit)\n+{\n+\tstruct filter_omit_large_blobs_data d;\n+\n+\tmemset(&d, 0, sizeof(d));\n+\td.max_bytes = large_byte_limit;\n+\n+\ttraverse_commit_list_worker(revs, show_commit, show_object, ctx_data,\n+\t\t\t\t    filter_omit_large_blobs, &d);\n+\n+\tif (print_omitted_object)\n+\t\toidset2_foreach(&d.omits, print_omitted_object, ctx_data);\n+\n+\toidset2_clear(&d.omits);\n+}\ndiff --git a/list-objects-filter-large.h b/list-objects-filter-large.h\nnew file mode 100644\nindex 0000000..4a5c772\n--- /dev/null\n+++ b/list-objects-filter-large.h\n@@ -0,0 +1,18 @@\n+#ifndef LIST_OBJECTS_FILTER_LARGE_H\n+#define LIST_OBJECTS_FILTER_LARGE_H\n+\n+#include \"oidset2.h\"\n+\n+/*\n+ * A filter for list-objects to omit large blobs,\n+ * but always include \".git*\" special files.\n+ */\n+void traverse_commit_list_omit_large_blobs(\n+\tstruct rev_info *revs,\n+\tshow_commit_fn show_commit,\n+\tshow_object_fn show_object,\n+\toidset2_foreach_cb print_omitted_object,\n+\tvoid *ctx_data,\n+\tint64_t large_byte_limit);\n+\n+#endif /* LIST_OBJECTS_FILTER_LARGE_H */\n-- \n2.9.3\n\n"},{"id":"328675","messageId":"20170922203017.53986-7-git@jeffhostetler.com","threadId":"46810","inReplyTo":"20170922203017.53986-6-git@jeffhostetler.com","subject":"[PATCH 06/13] list-objects-filter-sparse: add sparse-checkout based filter","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-09-22T20:30:10Z","receivedAt":"2017-09-22T20:31:23Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nCreate a filter for traverse_commit_list_worker() to only include\nthe blobs the would be referenced by a sparse-checkout using the\ngiven specification.\n\nA future enhancement should be able to also omit unneeded tree\nobjects, but that is not currently supported.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n Makefile                     |   1 +\n list-objects-filter-sparse.c | 221 +++++++++++++++++++++++++++++++++++++++++++\n list-objects-filter-sparse.h |  30 ++++++\n 3 files changed, 252 insertions(+)\n create mode 100644 list-objects-filter-sparse.c\n create mode 100644 list-objects-filter-sparse.h\n\ndiff --git a/Makefile b/Makefile\nindex f1f3979..6e0bd91 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -800,6 +800,7 @@ LIB_OBJS += line-range.o\n LIB_OBJS += list-objects.o\n LIB_OBJS += list-objects-filter-all.o\n LIB_OBJS += list-objects-filter-large.o\n+LIB_OBJS += list-objects-filter-sparse.o\n LIB_OBJS += ll-merge.o\n LIB_OBJS += lockfile.o\n LIB_OBJS += log-tree.o\ndiff --git a/list-objects-filter-sparse.c b/list-objects-filter-sparse.c\nnew file mode 100644\nindex 0000000..59155d5\n--- /dev/null\n+++ b/list-objects-filter-sparse.c\n@@ -0,0 +1,221 @@\n+#include \"cache.h\"\n+#include \"dir.h\"\n+#include \"tag.h\"\n+#include \"commit.h\"\n+#include \"tree.h\"\n+#include \"blob.h\"\n+#include \"diff.h\"\n+#include \"tree-walk.h\"\n+#include \"revision.h\"\n+#include \"list-objects.h\"\n+#include \"list-objects-filter-sparse.h\"\n+\n+/*\n+ * A filter driven by a sparse-checkout specification to only\n+ * include blobs that a sparse checkout would populate.\n+ *\n+ * The sparse-checkout spec can be loaded from a blob with the\n+ * given OID or from a local pathname.  We allow an OID because\n+ * the repo may be bare or we may be doing the filtering on the\n+ * server.\n+ */\n+struct frame {\n+\tint defval;\n+\tint child_prov_omit : 1;\n+};\n+\n+struct filter_use_sparse_data {\n+\tstruct oidset2 omits;\n+\tstruct exclude_list el;\n+\n+\tsize_t nr, alloc;\n+\tstruct frame *array_frame;\n+};\n+\n+static list_objects_filter_result filter_use_sparse(\n+\tlist_objects_filter_type filter_type,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tvoid *filter_data_)\n+{\n+\tstruct filter_use_sparse_data *filter_data = filter_data_;\n+\tint64_t object_length = -1;\n+\tint val, dtype;\n+\tunsigned long s;\n+\tenum object_type t;\n+\tstruct frame *frame;\n+\n+\tswitch (filter_type) {\n+\tdefault:\n+\t\tdie(\"unkown filter_type\");\n+\t\treturn LOFR_ZERO;\n+\n+\tcase LOFT_BEGIN_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\tdtype = DT_DIR;\n+\t\tval = is_excluded_from_list(pathname, strlen(pathname),\n+\t\t\t\t\t    filename, &dtype, &filter_data->el,\n+\t\t\t\t\t    &the_index);\n+\t\tif (val < 0)\n+\t\t\tval = filter_data->array_frame[filter_data->nr].defval;\n+\n+\t\tALLOC_GROW(filter_data->array_frame, filter_data->nr + 1,\n+\t\t\t   filter_data->alloc);\n+\t\tfilter_data->nr++;\n+\t\tfilter_data->array_frame[filter_data->nr].defval = val;\n+\t\tfilter_data->array_frame[filter_data->nr].child_prov_omit = 0;\n+\n+\t\t/*\n+\t\t * A directory with this tree OID may appear in multiple\n+\t\t * places in the tree. (Think of a directory move, with\n+\t\t * no other changes.)  And with a different pathname, the\n+\t\t * is_excluded...() results for this directory and items\n+\t\t * contained within it may be different.  So we cannot\n+\t\t * mark it SEEN (yet), since that will prevent process_tree()\n+\t\t * from revisiting this tree object with other pathnames.\n+\t\t *\n+\t\t * Only SHOW the tree object the first time we visit this\n+\t\t * tree object.\n+\t\t *\n+\t\t * We always show all tree objects.  A future optimization\n+\t\t * may want to attempt to narrow this.\n+\t\t */\n+\t\tif (obj->flags & FILTER_REVISIT)\n+\t\t\treturn LOFR_ZERO;\n+\t\tobj->flags |= FILTER_REVISIT;\n+\t\treturn LOFR_SHOW;\n+\n+\tcase LOFT_END_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\t\tassert(filter_data->nr > 0);\n+\n+\t\tframe = &filter_data->array_frame[filter_data->nr];\n+\t\tfilter_data->nr--;\n+\n+\t\t/*\n+\t\t * Tell our parent directory if any of our children were\n+\t\t * provisionally omitted.\n+\t\t */\n+\t\tfilter_data->array_frame[filter_data->nr].child_prov_omit |=\n+\t\t\tframe->child_prov_omit;\n+\n+\t\t/*\n+\t\t * If there are NO provisionally omitted child objects (ALL child\n+\t\t * objects in this folder were INCLUDED), then we can mark the\n+\t\t * folder as SEEN (so we will not have to revisit it again).\n+\t\t */\n+\t\tif (!frame->child_prov_omit)\n+\t\t\treturn LOFR_MARK_SEEN;\n+\t\treturn LOFR_ZERO;\n+\n+\tcase LOFT_BLOB:\n+\t\tassert(obj->type == OBJ_BLOB);\n+\t\tassert((obj->flags & SEEN) == 0);\n+\n+\t\tframe = &filter_data->array_frame[filter_data->nr];\n+\n+\t\t/*\n+\t\t * If we previously provisionally omitted this blob because\n+\t\t * its pathname was not in the sparse-checkout AND this\n+\t\t * reference to the blob has the same pathname, we can avoid\n+\t\t * repeating the exclusion logic on this pathname and just\n+\t\t * continue to provisionally omit it.\n+\t\t */\n+\t\tif (obj->flags & FILTER_REVISIT) {\n+\t\t\tstruct oidset2_entry *entry_prev;\n+\t\t\tentry_prev = oidset2_get(&filter_data->omits, &obj->oid);\n+\t\t\tif (entry_prev && !strcmp(pathname, entry_prev->pathname)) {\n+\t\t\t\tframe->child_prov_omit = 1;\n+\t\t\t\treturn LOFR_ZERO;\n+\t\t\t}\n+\t\t}\n+\n+\t\tdtype = DT_REG;\n+\t\tval = is_excluded_from_list(pathname, strlen(pathname),\n+\t\t\t\t\t    filename, &dtype, &filter_data->el,\n+\t\t\t\t\t    &the_index);\n+\t\tif (val < 0)\n+\t\t\tval = frame->defval;\n+\t\tif (val > 0)\n+\t\t\treturn LOFR_MARK_SEEN | LOFR_SHOW;\n+\n+\t\tt = sha1_object_info(obj->oid.hash, &s);\n+\t\tassert(t == OBJ_BLOB);\n+\t\tobject_length = (int64_t)((uint64_t)(s));\n+\n+\t\t/*\n+\t\t * Provisionally omit it.  We've already established that\n+\t\t * this pathname is not in the sparse-checkout specification,\n+\t\t * so we WANT to omit this blob.  However, a pathname elsewhere\n+\t\t * in the tree may also reference this same blob, so we cannot\n+\t\t * reject it yet.  Leave the LOFR_ bits unset so that if the\n+\t\t * blob appears again in the traversal, we will be asked again.\n+\t\t *\n+\t\t * The pathname we associate with this omit is just the first\n+\t\t * one we saw for this blob.  Other instances of this blob may\n+\t\t * have other pathnames and that is fine.  We just use it for\n+\t\t * perf because most of the time, the blob will be in the same\n+\t\t * place as we walk the commits.\n+\t\t */\n+\t\toidset2_insert(&filter_data->omits, &obj->oid, t, object_length,\n+\t\t\t       pathname);\n+\t\tobj->flags |= FILTER_REVISIT;\n+\t\tframe->child_prov_omit = 1;\n+\t\treturn LOFR_ZERO;\n+\t}\n+}\n+\n+void traverse_commit_list_use_blob(\n+\tstruct rev_info *revs,\n+\tshow_commit_fn show_commit,\n+\tshow_object_fn show_object,\n+\toidset2_foreach_cb print_omitted_object,\n+\tvoid *ctx_data,\n+\tstruct object_id *oid)\n+{\n+\tstruct filter_use_sparse_data d;\n+\n+\tmemset(&d, 0, sizeof(d));\n+\tif (add_excludes_from_blob_to_list(oid, NULL, 0, &d.el) < 0)\n+\t\tdie(\"could not load filter specification\");\n+\n+\tALLOC_GROW(d.array_frame, d.nr + 1, d.alloc);\n+\td.array_frame[d.nr].defval = 0; /* default to include */\n+\td.array_frame[d.nr].child_prov_omit = 0;\n+\n+\ttraverse_commit_list_worker(revs, show_commit, show_object, ctx_data,\n+\t\t\t\t    filter_use_sparse, &d);\n+\n+\tif (print_omitted_object)\n+\t\toidset2_foreach(&d.omits, print_omitted_object, ctx_data);\n+\n+\toidset2_clear(&d.omits);\n+}\n+\n+void traverse_commit_list_use_path(\n+\tstruct rev_info *revs,\n+\tshow_commit_fn show_commit,\n+\tshow_object_fn show_object,\n+\toidset2_foreach_cb print_omitted_object,\n+\tvoid *ctx_data,\n+\tconst char *path)\n+{\n+\tstruct filter_use_sparse_data d;\n+\n+\tmemset(&d, 0, sizeof(d));\n+\tif (add_excludes_from_file_to_list(path, NULL, 0, &d.el, NULL) < 0)\n+\t\tdie(\"could not load filter specification\");\n+\n+\tALLOC_GROW(d.array_frame, d.nr + 1, d.alloc);\n+\td.array_frame[d.nr].defval = 0; /* default to include */\n+\td.array_frame[d.nr].child_prov_omit = 0;\n+\n+\ttraverse_commit_list_worker(revs, show_commit, show_object, ctx_data,\n+\t\t\t\t    filter_use_sparse, &d);\n+\n+\tif (print_omitted_object)\n+\t\toidset2_foreach(&d.omits, print_omitted_object, ctx_data);\n+\n+\toidset2_clear(&d.omits);\n+}\ndiff --git a/list-objects-filter-sparse.h b/list-objects-filter-sparse.h\nnew file mode 100644\nindex 0000000..aa89390\n--- /dev/null\n+++ b/list-objects-filter-sparse.h\n@@ -0,0 +1,30 @@\n+#ifndef LIST_OBJECTS_FILTERS_SPARSE_H\n+#define LIST_OBJECTS_FILTERS_SPARSE_H\n+\n+#include \"oidset2.h\"\n+\n+/*\n+ * A filter driven by a sparse-checkout specification to only\n+ * include blobs that a sparse checkout would populate.\n+ *\n+ * The sparse-checkout spec can be loaded from a blob with the\n+ * given OID, a blob with a blob-ish path, or from a local pathname.\n+ * We allow an OID because the repo may be bare or we may be doing\n+ * the filtering on the server.\n+ */\n+void traverse_commit_list_use_blob(\n+\tstruct rev_info *revs,\n+\tshow_commit_fn show_commit,\n+\tshow_object_fn show_object,\n+\toidset2_foreach_cb print_omitted_object,\n+\tvoid *ctx_data,\n+\tstruct object_id *oid);\n+void traverse_commit_list_use_path(\n+\tstruct rev_info *revs,\n+\tshow_commit_fn show_commit,\n+\tshow_object_fn show_object,\n+\toidset2_foreach_cb print_omitted_object,\n+\tvoid *ctx_data,\n+\tconst char *path);\n+\n+#endif /* LIST_OBJECTS_FILTERS_SPARSE_H */\n-- \n2.9.3\n\n"},{"id":"328676","messageId":"20170922203017.53986-8-git@jeffhostetler.com","threadId":"46810","inReplyTo":"20170922203017.53986-6-git@jeffhostetler.com","subject":"[PATCH 07/13] object-filter: common declarations for object filtering","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-09-22T20:30:11Z","receivedAt":"2017-09-22T20:31:30Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nCreate common routines and defines for parsing\nobject-filter-related command line arguments and\npack-protocol fields.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n Makefile        |   1 +\n object-filter.c | 269 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n object-filter.h | 173 ++++++++++++++++++++++++++++++++++++\n 3 files changed, 443 insertions(+)\n create mode 100644 object-filter.c\n create mode 100644 object-filter.h\n\ndiff --git a/Makefile b/Makefile\nindex 6e0bd91..bddd6b5 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -818,6 +818,7 @@ LIB_OBJS += notes-cache.o\n LIB_OBJS += notes-merge.o\n LIB_OBJS += notes-utils.o\n LIB_OBJS += object.o\n+LIB_OBJS += object-filter.o\n LIB_OBJS += oidset.o\n LIB_OBJS += oidset2.o\n LIB_OBJS += pack-bitmap.o\ndiff --git a/object-filter.c b/object-filter.c\nnew file mode 100644\nindex 0000000..c88f79f\n--- /dev/null\n+++ b/object-filter.c\n@@ -0,0 +1,269 @@\n+#include \"cache.h\"\n+#include \"commit.h\"\n+#include \"config.h\"\n+#include \"revision.h\"\n+#include \"list-objects.h\"\n+#include \"oidset2.h\"\n+#include \"list-objects-filter-all.h\"\n+#include \"list-objects-filter-large.h\"\n+#include \"list-objects-filter-sparse.h\"\n+#include \"object-filter.h\"\n+\n+int parse_filter_omit_all_blobs(struct object_filter_options *filter_options)\n+{\n+\tif (object_filter_enabled(filter_options))\n+\t\tdie(_(\"multiple object filter types cannot be combined\"));\n+\n+\tfilter_options->omit_all_blobs = 1;\n+\treturn 0;\n+}\n+\n+int parse_filter_omit_large_blobs(struct object_filter_options *filter_options,\n+\t\t\t\t  const char *arg)\n+{\n+\tif (object_filter_enabled(filter_options))\n+\t\tdie(_(\"multiple object filter types cannot be combined\"));\n+\n+\tfilter_options->omit_large_blobs = 1;\n+\n+\t/* we allow \"<digits>[kmg]\" */\n+\tif (!git_parse_ulong(arg, &filter_options->large_byte_limit))\n+\t\tdie(_(\"invalid size limit for large object filter\"));\n+\n+\tfilter_options->large_byte_limit_string = strdup(arg);\n+\treturn 0;\n+}\n+\n+int parse_filter_use_blob(struct object_filter_options *filter_options,\n+\t\t\t  const char *arg)\n+{\n+\tstruct object_context oc;\n+\n+\tif (object_filter_enabled(filter_options))\n+\t\tdie(_(\"multiple object filter types cannot be combined\"));\n+\n+\tfilter_options->use_blob = 1;\n+\n+\t/*\n+\t * The command line argument needs to resolve to an known OID\n+\t * representing the content of the desired sparse-checkout file.\n+\t *\n+\t * We allow various syntax forms for the convenience of the user.\n+\t * See sha1_name.c:get_sha1_with_context_1().\n+\t *\n+\t * Try to evaluate the arg locally in case they use one of the\n+\t * convenience patterns.  This must resolve to a blob.\n+\t */\n+\tif (get_sha1_with_context(arg, GET_SHA1_BLOB,\n+\t\t\t\t  filter_options->sparse_oid.hash, &oc)) {\n+\t\t/*\n+\t\t * If that fails, keep the original string in case a client\n+\t\t * command wants to send it to the server.  This allows the\n+\t\t * client to name an OID for a blob they don't have.\n+\t\t */\n+\t\tfilter_options->sparse_value = strdup(arg);\n+\t\toidcpy(&filter_options->sparse_oid, &null_oid);\n+\t} else {\n+\t\t/*\n+\t\t * Round-trip the found OID to normalize it.\n+\t\t */\n+\t\tfilter_options->sparse_value =\n+\t\t\tstrdup(oid_to_hex(&filter_options->sparse_oid));\n+\t}\n+\treturn 0;\n+}\n+\n+int parse_filter_use_path(struct object_filter_options *filter_options,\n+\t\t\t  const char *arg)\n+{\n+\tif (object_filter_enabled(filter_options))\n+\t\tdie(_(\"multiple object filter types cannot be combined\"));\n+\n+\tfilter_options->use_path = 1;\n+\n+\t/*\n+\t * The command line argument needs to resolve to a local file\n+\t * containing the content of the desired sparse-checkout file.\n+\t */\n+\tfilter_options->sparse_value = strdup(arg);\n+\treturn 0;\n+}\n+\n+int parse_filter_print_omitted(struct object_filter_options *filter_options)\n+{\n+\tfilter_options->print_omitted = 1;\n+\treturn 0;\n+}\n+\n+int parse_filter_print_missing(struct object_filter_options *filter_options)\n+{\n+\tfilter_options->print_missing = 1;\n+\treturn 0;\n+}\n+\n+int parse_filter_relax(struct object_filter_options *filter_options)\n+{\n+\tfilter_options->relax = 1;\n+\treturn 0;\n+}\n+\n+int opt_parse_filter_omit_all_blobs(const struct option *opt,\n+\t\t\t\t    const char *arg, int unset)\n+{\n+\tstruct object_filter_options *filter_options = opt->value;\n+\n+\tassert(!arg);\n+\tassert(!unset);\n+\n+\treturn parse_filter_omit_all_blobs(filter_options);\n+}\n+\n+int opt_parse_filter_omit_large_blobs(const struct option *opt,\n+\t\t\t\t      const char *arg, int unset)\n+{\n+\tstruct object_filter_options *filter_options = opt->value;\n+\n+\tassert(arg);\n+\tassert(!unset);\n+\n+\treturn parse_filter_omit_large_blobs(filter_options, arg);\n+}\n+\n+int opt_parse_filter_use_blob(const struct option *opt,\n+\t\t\t      const char *arg, int unset)\n+{\n+\tstruct object_filter_options *filter_options = opt->value;\n+\n+\tassert(arg);\n+\tassert(!unset);\n+\n+\treturn parse_filter_use_blob(filter_options, arg);\n+}\n+\n+int opt_parse_filter_use_path(const struct option *opt,\n+\t\t\t      const char *arg, int unset)\n+{\n+\tstruct object_filter_options *filter_options = opt->value;\n+\n+\tassert(arg);\n+\tassert(!unset);\n+\n+\treturn parse_filter_use_path(filter_options, arg);\n+}\n+\n+int opt_parse_filter_print_omitted(const struct option *opt,\n+\t\t\t\t   const char *arg, int unset)\n+{\n+\tstruct object_filter_options *filter_options = opt->value;\n+\n+\tassert(!arg);\n+\tassert(!unset);\n+\n+\treturn parse_filter_print_omitted(filter_options);\n+}\n+\n+int opt_parse_filter_print_missing(const struct option *opt,\n+\t\t\t\t   const char *arg, int unset)\n+{\n+\tstruct object_filter_options *filter_options = opt->value;\n+\n+\tassert(!arg);\n+\tassert(!unset);\n+\n+\treturn parse_filter_print_missing(filter_options);\n+}\n+\n+int opt_parse_filter_relax(const struct option *opt,\n+\t\t\t   const char *arg, int unset)\n+{\n+\tstruct object_filter_options *filter_options = opt->value;\n+\n+\tassert(!arg);\n+\tassert(!unset);\n+\n+\treturn parse_filter_relax(filter_options);\n+}\n+\n+int object_filter_hand_parse_arg(struct object_filter_options *filter_options,\n+\t\t\t\t const char *arg,\n+\t\t\t\t int allow_print_omitted,\n+\t\t\t\t int allow_print_missing,\n+\t\t\t\t int allow_relax)\n+{\n+\tif (!strcmp(arg, (\"--\"CL_ARG_FILTER_OMIT_ALL_BLOBS))) {\n+\t\tparse_filter_omit_all_blobs(filter_options);\n+\t\treturn 1;\n+\t}\n+\tif (skip_prefix(arg, (\"--\"CL_ARG_FILTER_OMIT_LARGE_BLOBS\"=\"), &arg)) {\n+\t\tparse_filter_omit_large_blobs(filter_options, arg);\n+\t\treturn 1;\n+\t}\n+\tif (skip_prefix(arg, (\"--\"CL_ARG_FILTER_USE_BLOB\"=\"), &arg)) {\n+\t\tparse_filter_use_blob(filter_options, arg);\n+\t\treturn 1;\n+\t}\n+\tif (skip_prefix(arg, (\"--\"CL_ARG_FILTER_USE_PATH\"=\"), &arg)) {\n+\t\tparse_filter_use_path(filter_options, arg);\n+\t\treturn 1;\n+\t}\n+\n+\tif (allow_print_omitted &&\n+\t    !strcmp(arg, (\"--\"CL_ARG_FILTER_PRINT_OMITTED))) {\n+\t\tparse_filter_print_omitted(filter_options);\n+\t\treturn 1;\n+\t}\n+\n+\tif (allow_print_missing &&\n+\t    !strcmp(arg, (\"--\"CL_ARG_FILTER_PRINT_MISSING))) {\n+\t\tparse_filter_print_missing(filter_options);\n+\t\treturn 1;\n+\t}\n+\n+\tif (allow_relax && !strcmp(arg, (\"--\"CL_ARG_FILTER_RELAX))) {\n+\t\tparse_filter_relax(filter_options);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+int object_filter_hand_parse_protocol(struct object_filter_options *filter_options,\n+\t\t\t\t      const char *arg,\n+\t\t\t\t      int allow_print_omitted,\n+\t\t\t\t      int allow_print_missing,\n+\t\t\t\t      int allow_relax)\n+{\n+\tif (!strcmp(arg, CL_ARG_FILTER_OMIT_ALL_BLOBS)) {\n+\t\tparse_filter_omit_all_blobs(filter_options);\n+\t\treturn 1;\n+\t}\n+\tif (skip_prefix(arg, (CL_ARG_FILTER_OMIT_LARGE_BLOBS\" \"), &arg)) {\n+\t\tparse_filter_omit_large_blobs(filter_options, arg);\n+\t\treturn 1;\n+\t}\n+\tif (skip_prefix(arg, (CL_ARG_FILTER_USE_BLOB\" \"), &arg)) {\n+\t\tparse_filter_use_blob(filter_options, arg);\n+\t\treturn 1;\n+\t}\n+\tif (skip_prefix(arg, (CL_ARG_FILTER_USE_PATH\" \"), &arg)) {\n+\t\tparse_filter_use_path(filter_options, arg);\n+\t\treturn 1;\n+\t}\n+\n+\tif (allow_print_omitted &&\n+\t    !strcmp(arg, CL_ARG_FILTER_PRINT_OMITTED)) {\n+\t\tparse_filter_print_omitted(filter_options);\n+\t\treturn 1;\n+\t}\n+\tif (allow_print_missing &&\n+\t    !strcmp(arg, CL_ARG_FILTER_PRINT_MISSING)) {\n+\t\tparse_filter_print_missing(filter_options);\n+\t\treturn 1;\n+\t}\n+\tif (allow_relax && !strcmp(arg, CL_ARG_FILTER_RELAX)) {\n+\t\tparse_filter_relax(filter_options);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/object-filter.h b/object-filter.h\nnew file mode 100644\nindex 0000000..13a638b\n--- /dev/null\n+++ b/object-filter.h\n@@ -0,0 +1,173 @@\n+#ifndef OBJECT_FILTER_H\n+#define OBJECT_FILTER_H\n+\n+#include \"parse-options.h\"\n+\n+/*\n+ * Common declarations and utilities for filtering objects (such as omitting\n+ * large blobs) in list_objects:traverse_commit_list() and git-rev-list.\n+ */\n+\n+struct object_filter_options {\n+\t/*\n+\t * File pathname or blob-ish path/OID (that get_sha1_with_context() can\n+\t * use to find the blob containing the sparse-checkout specification.\n+\t * This is only used when use_blob or use_path is set.\n+\t */\n+\tconst char *sparse_value;\n+\tstruct object_id sparse_oid;\n+\n+\t/*\n+\t * Blob size byte limit for filtering.  Only blobs smaller than this\n+\t * value will be included.  A value of zero, omits all blobs.\n+\t * only used when omit_large_blobs is set.  Integer and string versions\n+\t * of this are kept for convenience.  The string version may contain\n+\t * a [kmg] suffix.\n+\t */\n+\tunsigned long large_byte_limit;\n+\tconst char *large_byte_limit_string;\n+\n+\t/* Valid filter types (only one may be used at a time) */\n+\tunsigned omit_all_blobs : 1;\n+\tunsigned omit_large_blobs : 1;\n+\tunsigned use_blob : 1;\n+\tunsigned use_path : 1;\n+\n+\t/*\n+\t * True if rev-list should print a list of the objects omitted\n+\t * by this invocation of a filter.\n+\t */\n+\tunsigned print_omitted : 1;\n+\n+\t/*\n+\t * True if rev-list should print a list of missing objects.\n+\t * Objects can be missing because of a previously filtered\n+\t * clone or fetch. The set reported here can also be filtered\n+\t * by the current filter in effect.\n+\t */\n+\tunsigned print_missing : 1;\n+\n+\t/* True to suppress missing object errors during consistency checks */\n+\tunsigned relax : 1;\n+};\n+\n+/*\n+ * Return true if a filter is enabled.\n+ */\n+inline int object_filter_enabled(const struct object_filter_options *p)\n+{\n+\treturn p->omit_all_blobs ||\n+\t\tp->omit_large_blobs ||\n+\t\tp->use_blob ||\n+\t\tp->use_path;\n+}\n+\n+/* Normalized command line arguments */\n+#define CL_ARG_FILTER_OMIT_ALL_BLOBS     \"filter-omit-all-blobs\"\n+#define CL_ARG_FILTER_OMIT_LARGE_BLOBS   \"filter-omit-large-blobs\"\n+#define CL_ARG_FILTER_USE_BLOB           \"filter-use-blob\"\n+#define CL_ARG_FILTER_USE_PATH           \"filter-use-path\"\n+#define CL_ARG_FILTER_PRINT_OMITTED      \"filter-print-omitted\"\n+#define CL_ARG_FILTER_PRINT_MISSING      \"filter-print-missing\"\n+#define CL_ARG_FILTER_RELAX              \"filter-relax\"\n+\n+/*\n+ * Common command line argument parsing for object-filter-related\n+ * arguments (whether from a hand-parsed or parse-options style\n+ * parser.\n+ */\n+int parse_filter_omit_all_blobs(struct object_filter_options *filter_options);\n+int parse_filter_omit_large_blobs(struct object_filter_options *filter_options,\n+\t\t\t\t  const char *arg);\n+int parse_filter_use_blob(struct object_filter_options *filter_options,\n+\t\t\t  const char *arg);\n+int parse_filter_use_path(struct object_filter_options *filter_options,\n+\t\t\t  const char *arg);\n+int parse_filter_print_omitted(struct object_filter_options *filter_options);\n+int parse_filter_print_missing(struct object_filter_options *filter_options);\n+int parse_filter_relax(struct object_filter_options *filter_options);\n+\n+/*\n+ * Common command line argument parsers for object-filter-related\n+ * arguments comming from parse-options style parsers.\n+ */\n+\n+int opt_parse_filter_omit_all_blobs(const struct option *opt,\n+\t\t\t\t    const char *arg, int unset);\n+int opt_parse_filter_omit_large_blobs(const struct option *opt,\n+\t\t\t\t      const char *arg, int unset);\n+int opt_parse_filter_use_blob(const struct option *opt,\n+\t\t\t      const char *arg, int unset);\n+int opt_parse_filter_use_path(const struct option *opt,\n+\t\t\t      const char *arg, int unset);\n+int opt_parse_filter_print_omitted(const struct option *opt,\n+\t\t\t\t   const char *arg, int unset);\n+int opt_parse_filter_print_missing(const struct option *opt,\n+\t\t\t\t   const char *arg, int unset);\n+int opt_parse_filter_relax(const struct option *opt,\n+\t\t\t   const char *arg, int unset);\n+\n+#define OPT_PARSE_FILTER_OMIT_ALL_BLOBS(fo) \\\n+\t{ OPTION_CALLBACK, 0, CL_ARG_FILTER_OMIT_ALL_BLOBS, fo, NULL, \\\n+\t  N_(\"omit all blobs from result\"), PARSE_OPT_NOARG | PARSE_OPT_NONEG, \\\n+\t  opt_parse_filter_omit_all_blobs }\n+\n+#define OPT_PARSE_FILTER_OMIT_LARGE_BLOBS(fo) \\\n+\t{ OPTION_CALLBACK, 0, CL_ARG_FILTER_OMIT_LARGE_BLOBS, fo, N_(\"size\"), \\\n+\t  N_(\"omit large blobs from result\"), PARSE_OPT_NONEG, \\\n+\t  opt_parse_filter_omit_large_blobs }\n+\n+#define OPT_PARSE_FILTER_USE_BLOB(fo) \\\n+\t{ OPTION_CALLBACK, 0, CL_ARG_FILTER_USE_BLOB, fo, N_(\"object\"), \\\n+\t  N_(\"filter results using sparse-checkout specification\"), PARSE_OPT_NONEG, \\\n+\t  opt_parse_filter_use_blob }\n+\n+#define OPT_PARSE_FILTER_USE_PATH(fo) \\\n+\t{ OPTION_CALLBACK, 0, CL_ARG_FILTER_USE_PATH, fo, N_(\"path\"), \\\n+\t  N_(\"filter results using sparse-checkout specification\"), PARSE_OPT_NONEG, \\\n+\t  opt_parse_filter_use_path }\n+\n+#define OPT_PARSE_FILTER_PRINT_OMITTED(fo) \\\n+\t{ OPTION_CALLBACK, 0, CL_ARG_FILTER_PRINT_OMITTED, fo, NULL,\t\\\n+\t  N_(\"print list of omitted objects\"), PARSE_OPT_NOARG | PARSE_OPT_NONEG, \\\n+\t  opt_parse_filter_print_omitted }\n+\n+#define OPT_PARSE_FILTER_PRINT_MISSING(fo) \\\n+\t{ OPTION_CALLBACK, 0, CL_ARG_FILTER_PRINT_MISSING, fo, NULL,\t\\\n+\t  N_(\"print list of missing objects\"), PARSE_OPT_NOARG | PARSE_OPT_NONEG, \\\n+\t  opt_parse_filter_print_missing }\n+\n+#define OPT_PARSE_FILTER_RELAX(fo) \\\n+\t{ OPTION_CALLBACK, 0, CL_ARG_FILTER_RELAX, fo, NULL, \\\n+\t  N_(\"relax consistency checks for previously omitted objects\"), \\\n+\t  PARSE_OPT_NOARG | PARSE_OPT_NONEG, opt_parse_filter_relax }\n+\n+/*\n+ * Hand parse known object-filter command line options.\n+ * Use this when the caller DOES NOT use the normal OPT_\n+ * routines.\n+ *\n+ * Here we assume args of the form \"--<key>\" or \"--<key>=<value>\".\n+ * Note the literal dash-dash and equals.\n+ *\n+ * Returns 1 if we handled the argument.\n+ */\n+int object_filter_hand_parse_arg(struct object_filter_options *filter_options,\n+\t\t\t\t const char *arg,\n+\t\t\t\t int allow_print_omitted,\n+\t\t\t\t int allow_print_missing,\n+\t\t\t\t int allow_relax);\n+\n+/*\n+ * Hand parse known object-filter protocol lines.\n+ *\n+ * Here we assume args of the form \"<key>\" or \"<key> <value>\".\n+ * Note the literal space before between the key and value.\n+ */ \n+int object_filter_hand_parse_protocol(struct object_filter_options *filter_options,\n+\t\t\t\t      const char *arg,\n+\t\t\t\t      int allow_print_omitted,\n+\t\t\t\t      int allow_print_missing,\n+\t\t\t\t      int allow_relax);\n+\n+#endif /* OBJECT_FILTER_H */\n-- \n2.9.3\n\n"},{"id":"328677","messageId":"20170922203017.53986-9-git@jeffhostetler.com","threadId":"46810","inReplyTo":"20170922203017.53986-6-git@jeffhostetler.com","subject":"[PATCH 08/13] list-objects: add traverse_commit_list_filtered method","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-09-22T20:30:12Z","receivedAt":"2017-09-22T20:31:32Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nAdd traverse_commit_list_filtered() wrapper around the various\nfilter methods using common data in object_filter_options.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n list-objects.c | 34 ++++++++++++++++++++++++++++++++++\n list-objects.h | 11 +++++++++++\n 2 files changed, 45 insertions(+)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex 3e86008..0f063d9 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -7,6 +7,9 @@\n #include \"tree-walk.h\"\n #include \"revision.h\"\n #include \"list-objects.h\"\n+#include \"list-objects-filter-all.h\"\n+#include \"list-objects-filter-large.h\"\n+#include \"list-objects-filter-sparse.h\"\n \n static void process_blob(struct rev_info *revs,\n \t\t\t struct blob *blob,\n@@ -266,3 +269,34 @@ void traverse_commit_list(struct rev_info *revs,\n \t\tshow_commit, show_object, show_data,\n \t\tNULL, NULL);\n }\n+\n+void traverse_commit_list_filtered(\n+\tstruct object_filter_options *filter_options,\n+\tstruct rev_info *revs,\n+\tshow_commit_fn show_commit,\n+\tshow_object_fn show_object,\n+\toidset2_foreach_cb print_omitted_object,\n+\tvoid *show_data)\n+{\n+\tif (filter_options->omit_all_blobs)\n+\t\ttraverse_commit_list_omit_all_blobs(\n+\t\t\trevs, show_commit, show_object, print_omitted_object, show_data);\n+\n+\telse if (filter_options->omit_large_blobs)\n+\t\ttraverse_commit_list_omit_large_blobs(\n+\t\t\trevs, show_commit, show_object, print_omitted_object, show_data,\n+\t\t\t(int64_t)(uint64_t)filter_options->large_byte_limit);\n+\n+\telse if (filter_options->use_blob)\n+\t\ttraverse_commit_list_use_blob(\n+\t\t\trevs, show_commit, show_object, print_omitted_object, show_data,\n+\t\t\t&filter_options->sparse_oid);\n+\n+\telse if (filter_options->use_path)\n+\t\ttraverse_commit_list_use_path(\n+\t\t\trevs, show_commit, show_object, print_omitted_object, show_data,\n+\t\t\tfilter_options->sparse_value);\n+\n+\telse\n+\t\tdie(\"unspecified list-objects filter\");\n+}\ndiff --git a/list-objects.h b/list-objects.h\nindex 39fcbb5..a8acedc 100644\n--- a/list-objects.h\n+++ b/list-objects.h\n@@ -1,6 +1,9 @@\n #ifndef LIST_OBJECTS_H\n #define LIST_OBJECTS_H\n \n+#include \"oidset2.h\"\n+#include \"object-filter.h\"\n+\n typedef void (*show_commit_fn)(struct commit *, void *);\n typedef void (*show_object_fn)(struct object *, const char *, void *);\n void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *);\n@@ -38,4 +41,12 @@ void traverse_commit_list_worker(\n \tshow_commit_fn, show_object_fn, void *show_data,\n \tfilter_object_fn filter, void *filter_data);\n \n+void traverse_commit_list_filtered(\n+\tstruct object_filter_options *filter_options,\n+\tstruct rev_info *revs,\n+\tshow_commit_fn show_commit,\n+\tshow_object_fn show_object,\n+\toidset2_foreach_cb print_omitted_object,\n+\tvoid *show_data);\n+\n #endif\n-- \n2.9.3\n\n"},{"id":"328678","messageId":"20170922203017.53986-10-git@jeffhostetler.com","threadId":"46810","inReplyTo":"20170922203017.53986-6-git@jeffhostetler.com","subject":"[PATCH 09/13] rev-list: add object filtering support","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-09-22T20:30:13Z","receivedAt":"2017-09-22T20:31:36Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nTeach rev-list to use the filtering provided by the\ntraverse_commit_list_filtered() interface to omit\nunwanted objects from the result.\n\nThis feature is only enabled when one of the \"--objects*\"\noptions are used.\n\nWhen the \"--filter-print-manifest\" option is used, the\nomitted objects and their sizes are printed at the end.\nThese are marked with a \"~\".  This can be combined with\n\"--quiet\" to get a list of just the omitted objects.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n builtin/rev-list.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 71 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 95d84d5..5f54495 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -53,6 +53,8 @@ static const char rev_list_usage[] =\n \n static struct progress *progress;\n static unsigned progress_counter;\n+static struct object_filter_options filter_options;\n+static struct oidset2 missing_objects;\n \n static void finish_commit(struct commit *commit, void *data);\n static void show_commit(struct commit *commit, void *data)\n@@ -179,8 +181,25 @@ static void finish_commit(struct commit *commit, void *data)\n static void finish_object(struct object *obj, const char *name, void *cb_data)\n {\n \tstruct rev_list_info *info = cb_data;\n-\tif (obj->type == OBJ_BLOB && !has_object_file(&obj->oid))\n+\tif (obj->type == OBJ_BLOB && !has_object_file(&obj->oid)) {\n+\t\tif (filter_options.print_missing) {\n+\t\t\toidset2_insert(&missing_objects, &obj->oid, obj->type,\n+\t\t\t\t       -1, name);\n+\t\t\treturn;\n+\t\t}\n+\t\tif (filter_options.relax) {\n+\t\t\t/*\n+\t\t\t * Relax consistency checks to not complain about\n+\t\t\t * omitted objects (presumably caused by use of\n+\t\t\t * the previous use of the 'filter-objects' feature).\n+\t\t\t *\n+\t\t\t * Note that this is independent of any filtering that\n+\t\t\t * we are doing in this run.\n+\t\t\t */\n+\t\t\treturn;\n+\t\t}\n \t\tdie(\"missing blob object '%s'\", oid_to_hex(&obj->oid));\n+\t}\n \tif (info->revs->verify_objects && !obj->parsed && obj->type != OBJ_COMMIT)\n \t\tparse_object(&obj->oid);\n }\n@@ -200,6 +219,25 @@ static void show_edge(struct commit *commit)\n \tprintf(\"-%s\\n\", oid_to_hex(&commit->object.oid));\n }\n \n+static void print_omitted_object(int i, int i_limit, struct oidset2_entry *e, void *cb_data)\n+{\n+\t/* struct rev_list_info *info = cb_data; */\n+\tconst char *tn = typename(e->type);\n+\n+\tif (e->object_length == -1)\n+\t\tprintf(\"~%s %s\\n\", oid_to_hex(&e->oid), tn);\n+\telse\n+\t\tprintf(\"~%s %s %\"PRIuMAX\"\\n\", oid_to_hex(&e->oid), tn, e->object_length);\n+}\n+\n+static void print_missing_object(int i, int i_limit, struct oidset2_entry *e, void *cb_data)\n+{\n+\t/* struct rev_list_info *info = cb_data; */\n+\tconst char *tn = typename(e->type);\n+\n+\tprintf(\"?%s %s\\n\", oid_to_hex(&e->oid), tn);\n+}\n+\n static void print_var_str(const char *var, const char *val)\n {\n \tprintf(\"%s='%s'\\n\", var, val);\n@@ -333,6 +371,15 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\tshow_progress = arg;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (object_filter_hand_parse_arg(\n+\t\t\t    &filter_options, arg, 1, 1, 1)) {\n+\t\t\tif (!revs.blob_objects)\n+\t\t\t\tdie(_(\"object filtering requires --objects\"));\n+\t\t\tif (filter_options.use_blob &&\n+\t\t\t    !oidcmp(&filter_options.sparse_oid, &null_oid))\n+\t\t\t\tdie(_(\"invalid sparse value\"));\n+\t\t\tcontinue;\n+\t\t}\n \t\tusage(rev_list_usage);\n \n \t}\n@@ -357,6 +404,11 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \tif (revs.show_notes)\n \t\tdie(_(\"rev-list does not support display of notes\"));\n \n+\tif (object_filter_enabled(&filter_options)) {\n+\t\tif (use_bitmap_index)\n+\t\t\tdie(_(\"cannot combine --use-bitmap-index with object filtering\"));\n+\t}\n+\n \tsave_commit_buffer = (revs.verbose_header ||\n \t\t\t      revs.grep_filter.pattern_list ||\n \t\t\t      revs.grep_filter.header_list);\n@@ -401,7 +453,24 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\treturn show_bisect_vars(&info, reaches, all);\n \t}\n \n-\ttraverse_commit_list(&revs, show_commit, show_object, &info);\n+\tif (filter_options.print_missing)\n+\t\tmemset(&missing_objects, 0, sizeof(missing_objects));\n+\n+\tif (object_filter_enabled(&filter_options))\n+\t\ttraverse_commit_list_filtered(\n+\t\t\t&filter_options, &revs,\n+\t\t\tshow_commit, show_object,\n+\t\t\t(filter_options.print_omitted\n+\t\t\t ? print_omitted_object\n+\t\t\t : NULL),\n+\t\t\t&info);\n+\telse\n+\t\ttraverse_commit_list(&revs, show_commit, show_object, &info);\n+\n+\tif (filter_options.print_missing) {\n+\t\toidset2_foreach(&missing_objects, print_missing_object, &info);\n+\t\toidset2_clear(&missing_objects);\n+\t}\n \n \tstop_progress(&progress);\n \n-- \n2.9.3\n\n"},{"id":"328679","messageId":"20170922203017.53986-11-git@jeffhostetler.com","threadId":"46810","inReplyTo":"20170922203017.53986-6-git@jeffhostetler.com","subject":"[PATCH 10/13] rev-list: add filtering help text","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-09-22T20:30:14Z","receivedAt":"2017-09-22T20:31:38Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n Documentation/git-rev-list.txt     |  9 ++++++++-\n Documentation/rev-list-options.txt | 32 ++++++++++++++++++++++++++++++++\n 2 files changed, 40 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-rev-list.txt b/Documentation/git-rev-list.txt\nindex ef22f17..b2e8255 100644\n--- a/Documentation/git-rev-list.txt\n+++ b/Documentation/git-rev-list.txt\n@@ -47,7 +47,14 @@ SYNOPSIS\n \t     [ --fixed-strings | -F ]\n \t     [ --date=<format>]\n \t     [ [ --objects | --objects-edge | --objects-edge-aggressive ]\n-\t       [ --unpacked ] ]\n+\t       [ --unpacked ]\n+\t       [ [ --filter-omit-all-blobs |\n+\t\t   --filter-omit-large-blobs=<n>[kmg] |\n+\t\t   --filter-use-blob=<blob-ish> |\n+\t\t   --filter-use-path=<path> ]\n+\t\t [ --filter-print-missing ]\n+\t\t [ --filter-print-omitted ] ] ]\n+\t     [ --filter-relax ]\n \t     [ --pretty | --header ]\n \t     [ --bisect ]\n \t     [ --bisect-vars ]\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex a6cf9eb..5c7d53c 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -698,6 +698,38 @@ ifdef::git-rev-list[]\n --unpacked::\n \tOnly useful with `--objects`; print the object IDs that are not\n \tin packs.\n+\n+--filter-omit-all-blobs::\n+\tOnly useful with one of the `--objects*`; omits all blobs from\n+\tthe printed list of objects.\n+\n+--filter-omit-large-blobs=<n>[kmg]::\n+\tOnly useful with one of the `--objects*`; omits blobs larger than\n+\tn bytes from the printed list of objects.  May optionally be\n+\tfollowed by 'k', 'm', or 'g' units.  Value may be zero.  Special\n+\tfiles (matching \".git*\") are always included, regardless of size.\n+\n+--filter-use-blob=<blob-ish>::\n+--filter-use-path=<path>::\n+\tOnly useful with one of the `--objects*`; uses a sparse-checkout\n+\tspecification contained in the given object or file to filter the\n+\tresult to only contain blobs referenced by such a sparse-checkout.\n+\n+--filter-print-missing::\n+\tPrints a list of the missing objects for the requested traversal.\n+\tObject IDs are prefixed with a ``?'' character.  The object type\n+\tis printed after the ID.  This may be used with or without any of\n+\tthe above filtering options.\n+\n+--filter-print-omitted::\n+\tOnly useful with one of the above `--filter*`; prints a list\n+\tof the omitted objects.  Object IDs are prefixed with a ``~''\n+\tcharacter.  The object size is printed after the ID.\n+\n+--filter-relax::\n+\tRelax consistency checking for missing blobs.  Do not warn of\n+\tmissing blobs during normal (non-filtering) object traversal\n+\tfollowing an earlier partial/narrow clone or fetch.\n endif::git-rev-list[]\n \n --no-walk[=(sorted|unsorted)]::\n-- \n2.9.3\n\n"},{"id":"328680","messageId":"20170922203017.53986-12-git@jeffhostetler.com","threadId":"46810","inReplyTo":"20170922203017.53986-6-git@jeffhostetler.com","subject":"[PATCH 11/13] t6112: rev-list object filtering test","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-09-22T20:30:15Z","receivedAt":"2017-09-22T20:31:43Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n t/t6112-rev-list-filters-objects.sh | 237 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 237 insertions(+)\n create mode 100755 t/t6112-rev-list-filters-objects.sh\n\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nnew file mode 100755\nindex 0000000..66ff022\n--- /dev/null\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -0,0 +1,237 @@\n+#!/bin/sh\n+\n+test_description='git rev-list with object filtering'\n+\n+. ./test-lib.sh\n+\n+# test the omit-all filter\n+\n+test_expect_success 'setup' '\n+\techo \"{print \\$1}\" >print_1.awk &&\n+\techo \"{print \\$2}\" >print_2.awk &&\n+\n+\tfor n in 1 2 3 4 5\n+\tdo\n+\t\techo $n > file.$n\n+\t\tgit add file.$n\n+\t\tgit commit -m \"$n\"\n+\tdone\n+'\n+\n+# Verify the omitted (\"~OID\") lines match the predicted list of OIDs.\n+test_expect_success 'omit-all-blobs omitted 5 blobs' '\n+\tgit ls-files -s file.1 file.2 file.3 file.4 file.5 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\tgit rev-list HEAD --quiet --objects --filter-print-omitted --filter-omit-all-blobs \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed \"s/~//\" >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+# Verify the complete OID list matches the unfiltered OIDs plus the omitted OIDs.\n+test_expect_success 'omit-all-blobs nothing else changed' '\n+\tgit rev-list HEAD --objects \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sort >expected &&\n+\tgit rev-list HEAD --objects --filter-print-omitted --filter-omit-all-blobs \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed \"s/~//\" \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+# test the size-based filtering.\n+\n+test_expect_success 'setup_large' '\n+\tfor n in 1000 10000\n+\tdo\n+\t\tprintf \"%\"$n\"s\" X > large.$n\n+\t\tgit add large.$n\n+\t\tgit commit -m \"$n\"\n+\tdone\n+'\n+\n+test_expect_success 'omit-large-blobs omit 2 blobs' '\n+\tgit ls-files -s large.1000 large.10000 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\tgit rev-list HEAD --quiet --objects --filter-print-omitted --filter-omit-large-blobs=500 \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed \"s/~//\" >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+test_expect_success 'omit-large-blobs nothing else changed' '\n+\tgit rev-list HEAD --objects \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sort >expected &&\n+\tgit rev-list HEAD --objects --filter-print-omitted --filter-omit-large-blobs=500 \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed \"s/~//\" \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+# boundary test around the size parameter.\n+# filter is strictly less than the value, so size 500 and 1000 should have the\n+# same results, but 1001 should filter more.\n+\n+test_expect_success 'omit-large-blobs omit 2 blobs' '\n+\tgit ls-files -s large.1000 large.10000 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\tgit rev-list HEAD --quiet --objects --filter-print-omitted --filter-omit-large-blobs=1000 \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed \"s/~//\" >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+test_expect_success 'omit-large-blobs omit 1 blob' '\n+\tgit ls-files -s large.10000 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\tgit rev-list HEAD --quiet --objects --filter-print-omitted --filter-omit-large-blobs=1001 \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed \"s/~//\" >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+test_expect_success 'omit-large-blobs omit 1 blob (1k)' '\n+\tgit ls-files -s large.10000 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\tgit rev-list HEAD --quiet --objects --filter-print-omitted --filter-omit-large-blobs=1k \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed \"s/~//\" >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+test_expect_success 'omit-large-blobs omit no blob (1m)' '\n+\tcat </dev/null >expected &&\n+\tgit rev-list HEAD --quiet --objects --filter-print-omitted --filter-omit-large-blobs=1m \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed \"s/~//\" >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+# Test sparse-pattern filtering (using explicit local patterns).\n+# We use the same disk format as sparse-checkout to specify the\n+# filtering, but do not require sparse-checkout to be enabled.\n+\n+test_expect_success 'setup using sparse file' '\n+\tmkdir dir1 &&\n+\tfor n in sparse1 sparse2\n+\tdo\n+\t\techo $n > $n\n+\t\tgit add $n\n+\t\techo dir1/$n > dir1/$n\n+\t\tgit add dir1/$n\n+\tdone &&\n+\tgit commit -m \"sparse\" &&\n+\techo dir1/ >pattern1 &&\n+\techo sparse1 >pattern2\n+'\n+\n+# pattern1 should only include the 2 dir1/* files.\n+# and omit the 5 file.*, 2 large.*, and 2 top-level sparse* files.\n+test_expect_success 'sparse using path pattern1' '\n+\tgit rev-list HEAD --objects --filter-print-omitted --filter-use-path=pattern1 >out &&\n+\n+\tgrep \"^~\" out >blobs.omitted &&\n+\ttest $(cat blobs.omitted | wc -l) = 9 &&\n+\n+\tgrep \"dir1/sparse\" out >blobs.included &&\n+\ttest $(cat blobs.included | wc -l) = 2\n+'\n+\n+# pattern2 should include the sparse1 and dir1/sparse1.\n+# and omit the 5 file.*, 2 large.*, and the 2 sparse2 files.\n+test_expect_success 'sparse using path pattern2' '\n+\tgit rev-list HEAD --objects --filter-print-omitted --filter-use-path=pattern2 >out &&\n+\n+\tgrep \"^~\" out >blobs.omitted &&\n+\ttest $(cat blobs.omitted | wc -l) = 9 &&\n+\n+\tgrep \"sparse1\" out >blobs.included &&\n+\ttest $(cat blobs.included | wc -l) = 2\n+'\n+\n+# Test sparse-pattern filtering (using a blob in the repo).\n+# This could be used to later let pack-objects do filtering.\n+\n+# pattern1 should only include the 2 dir1/* files.\n+# and omit the 5 file.*, 2 large.*, 2 top-level sparse*, and 1 pattern file.\n+test_expect_success 'sparse using OID for pattern1' '\n+\tgit add pattern1 &&\n+\tgit commit -m \"pattern1\" &&\n+\n+\tgit rev-list HEAD --objects >normal.output &&\n+\tgrep \"pattern1\" <normal.output | awk \"{print \\$1;}\" >pattern1.oid &&\n+\n+\tgit rev-list HEAD --objects --filter-print-omitted --filter-use-blob=`cat pattern1.oid` >out &&\n+\n+\tgrep \"^~\" out >blobs.omitted &&\n+\ttest $(cat blobs.omitted | wc -l) = 10 &&\n+\n+\tgrep \"dir1/sparse\" out >blobs.included &&\n+\ttest $(cat blobs.included | wc -l) = 2\n+'\n+\n+# repeat previous test but use blob-ish expression rather than OID.\n+test_expect_success 'sparse using blob-ish to get OID for pattern spec' '\n+\tgit rev-list HEAD --objects --filter-print-omitted --filter-use-blob=HEAD:pattern1 >out &&\n+\n+\tgrep \"^~\" out >blobs.omitted &&\n+\ttest $(cat blobs.omitted | wc -l) = 10 &&\n+\n+\tgrep \"dir1/sparse\" out >blobs.included &&\n+\ttest $(cat blobs.included | wc -l) = 2\n+'\n+\n+# pattern2 should include the sparse1 and dir1/sparse1.\n+# and omit the 5 file.*, 2 large.*, 2 top-level sparse*, and 2 pattern files.\n+test_expect_success 'sparse using OID for pattern2' '\n+\tgit add pattern2 &&\n+\tgit commit -m \"pattern2\" &&\n+\n+\tgit rev-list HEAD --objects >normal.output &&\n+\tgrep \"pattern2\" <normal.output | awk \"{print \\$1;}\" >pattern2.oid &&\n+\n+\tgit rev-list HEAD --objects --filter-print-omitted --filter-use-blob=`cat pattern2.oid` >out &&\n+\n+\tgrep \"^~\" out >blobs.omitted &&\n+\ttest $(cat blobs.omitted | wc -l) = 11 &&\n+\n+\tgrep \"sparse1\" out >blobs.included &&\n+\ttest $(cat blobs.included | wc -l) = 2\n+'\n+\n+# repeat previous test but use blob-ish expression rather than OID.\n+test_expect_success 'sparse using blob-ish rather than OID for pattern2' '\n+\tgit rev-list HEAD --objects --filter-print-omitted --filter-use-blob=HEAD:pattern2 >out &&\n+\n+\tgrep \"^~\" out >blobs.omitted &&\n+\ttest $(cat blobs.omitted | wc -l) = 11 &&\n+\n+\tgrep \"sparse1\" out >blobs.included &&\n+\ttest $(cat blobs.included | wc -l) = 2\n+'\n+\n+# delete some loose objects and test rev-list printing them as missing.\n+test_expect_success 'print missing objects' '\n+\tgit ls-files -s file.1 file.2 file.3 file.4 file.5 \\\n+\t\t| awk -f print_2.awk \\\n+\t\t| sort >expected &&\n+\tfor id in `cat expected | sed \"s|..|&/|\"`\n+\tdo\n+\t\trm .git/objects/$id\n+\tdone &&\n+\tgit rev-list --quiet HEAD --filter-print-missing --objects \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed \"s/?//\" \\\n+\t\t| sort >observed &&\n+\ttest_cmp observed expected\n+'\n+\n+test_done\n-- \n2.9.3\n\n"},{"id":"328681","messageId":"20170922203017.53986-14-git@jeffhostetler.com","threadId":"46810","inReplyTo":"20170922203017.53986-6-git@jeffhostetler.com","subject":"[PATCH 13/13] pack-objects: add filtering help text","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-09-22T20:30:17Z","receivedAt":"2017-09-22T20:31:45Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nAdd help text for new object filtering options to\npack-objects documentation.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n Documentation/git-pack-objects.txt | 17 +++++++++++++++++\n 1 file changed, 17 insertions(+)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 8973510..1ed7d4b 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -231,6 +231,23 @@ So does `git bundle` (see linkgit:git-bundle[1]) when it creates a bundle.\n \tWith this option, parents that are hidden by grafts are packed\n \tnevertheless.\n \n+--filter-omit-all-blobs::\n+\tRequires `--stdout`.  Omits all blobs from the packfile.\n+\n+--filter-omit-large-blobs=<n>[kmg]::\n+\tRequires `--stdout`.  Omits large blobs larger than n bytes from\n+\tthe packfile.  May optionally be followed by 'k', 'm', or 'g' units.\n+\tValue may be zero.  Special files (matching \".git*\") are always\n+\tincluded, regardless of size.\n+\n+--filter-use-blob=<blob-ish>::\n+--filter-use-path=<path>::\n+\tRequires `--stdout`.  Use a sparse-checkout specification to\n+\tfilter the resulting packfile to only contain the blobs that\n+\twould be referenced by such a sparse-checkout.  `<path>` specifies\n+\ta local pathname.  `<blob-ish>` specifies an expression that can\n+\tbe evaluated to a blob.\n+\n SEE ALSO\n --------\n linkgit:git-rev-list[1]\n-- \n2.9.3\n\n"},{"id":"328682","messageId":"20170922203017.53986-13-git@jeffhostetler.com","threadId":"46810","inReplyTo":"20170922203017.53986-6-git@jeffhostetler.com","subject":"[PATCH 12/13] pack-objects: add object filtering support","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-09-22T20:30:16Z","receivedAt":"2017-09-22T20:31:47Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nTeach pack-objects to use the filtering provided by the\ntraverse_commit_list_filtered() interface to omit unwanted\nobjects from the resulting packfile.\n\nThis feature is intended for partial clone/fetch.\n\nFiltering requires the use of the \"--stdout\" option.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n builtin/pack-objects.c | 24 +++++++++++++++++++++++-\n 1 file changed, 23 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex f4a8441..1712d1b 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -78,6 +78,8 @@ static unsigned long cache_max_small_delta_size = 1000;\n \n static unsigned long window_memory_limit = 0;\n \n+static struct object_filter_options filter_options;\n+\n /*\n  * stats\n  */\n@@ -2810,7 +2812,12 @@ static void get_object_list(int ac, const char **av)\n \tif (prepare_revision_walk(&revs))\n \t\tdie(\"revision walk setup failed\");\n \tmark_edges_uninteresting(&revs, show_edge);\n-\ttraverse_commit_list(&revs, show_commit, show_object, NULL);\n+\tif (object_filter_enabled(&filter_options))\n+\t\ttraverse_commit_list_filtered(&filter_options, &revs,\n+\t\t\t\t\t      show_commit, show_object,\n+\t\t\t\t\t      NULL, NULL);\n+\telse\n+\t\ttraverse_commit_list(&revs, show_commit, show_object, NULL);\n \n \tif (unpack_unreachable_expiration) {\n \t\trevs.ignore_missing_links = 1;\n@@ -2946,6 +2953,15 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"use a bitmap index if available to speed up counting objects\")),\n \t\tOPT_BOOL(0, \"write-bitmap-index\", &write_bitmap_index,\n \t\t\t N_(\"write a bitmap index together with the pack index\")),\n+\n+\t\tOPT_PARSE_FILTER_OMIT_ALL_BLOBS(&filter_options),\n+\t\tOPT_PARSE_FILTER_OMIT_LARGE_BLOBS(&filter_options),\n+\t\tOPT_PARSE_FILTER_USE_BLOB(&filter_options),\n+\t\tOPT_PARSE_FILTER_USE_PATH(&filter_options),\n+\t\t/* not needed: OPT_PARSE_FILTER_PRINT_MISSING */\n+\t\t/* not needed: OPT_PARSE_FILTER_PRINT_OMITTED */\n+\t\t/* not needed: OPT_PARSE_FILTER_RELAX */\n+\n \t\tOPT_END(),\n \t};\n \n@@ -3022,6 +3038,12 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n+\tif (object_filter_enabled(&filter_options)) {\n+\t\tif (!pack_to_stdout)\n+\t\t\tdie(\"cannot use filtering with an indexable pack.\");\n+\t\tuse_bitmap_index = 0;\n+\t}\n+\n \t/*\n \t * \"soft\" reasons not to use bitmaps - for on-disk repack by default we want\n \t *\n-- \n2.9.3\n\n"},{"id":"328970","messageId":"20170926153950.a147c48ce62f73702eff7ebb@google.com","threadId":"46810","inReplyTo":"20170922203017.53986-8-git@jeffhostetler.com","subject":"Re: [PATCH 07/13] object-filter: common declarations for object filtering","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-09-26T22:39:50Z","receivedAt":"2017-09-26T22:40:02Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Fri, 22 Sep 2017 20:30:11 +0000\nJeff Hostetler <git@jeffhostetler.com> wrote:\n\n>  Makefile        |   1 +\n>  object-filter.c | 269 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n>  object-filter.h | 173 ++++++++++++++++++++++++++++++++++++\n>  3 files changed, 443 insertions(+)\n>  create mode 100644 object-filter.c\n>  create mode 100644 object-filter.h\n\nI think these and list-objects-filter-* are multiple levels of\nindirection too many. Would a single file with a few implementations of\nfilter_object_fn be sufficient?\n"},{"id":"328971","messageId":"20170926154457.51f4da732122f00381d646e1@google.com","threadId":"46810","inReplyTo":"20170922203017.53986-10-git@jeffhostetler.com","subject":"Re: [PATCH 09/13] rev-list: add object filtering support","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-09-26T22:44:57Z","receivedAt":"2017-09-26T22:45:04Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Fri, 22 Sep 2017 20:30:13 +0000\nJeff Hostetler <git@jeffhostetler.com> wrote:\n\n> +\t\tif (filter_options.relax) {\n\nAdd some documentation about how this differs from ignore_missing_links\nin struct rev_info.\n"},{"id":"329044","messageId":"7774ff8d-3a53-860d-9343-292938d59d12@jeffhostetler.com","threadId":"46810","inReplyTo":"20170926153950.a147c48ce62f73702eff7ebb@google.com","subject":"Re: [PATCH 07/13] object-filter: common declarations for object filtering","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-09-27T17:09:42Z","receivedAt":"2017-09-27T17:09:47Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 9/26/2017 6:39 PM, Jonathan Tan wrote:\n> On Fri, 22 Sep 2017 20:30:11 +0000\n> Jeff Hostetler <git@jeffhostetler.com> wrote:\n> \n>>   Makefile        |   1 +\n>>   object-filter.c | 269 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n>>   object-filter.h | 173 ++++++++++++++++++++++++++++++++++++\n>>   3 files changed, 443 insertions(+)\n>>   create mode 100644 object-filter.c\n>>   create mode 100644 object-filter.h\n> \n> I think these and list-objects-filter-* are multiple levels of\n> indirection too many. Would a single file with a few implementations of\n> filter_object_fn be sufficient?\n\nI did that in my first draft and I found it confusing.\n\nEach filter has 3 parts (some filter-specific data structures,\na filter callback routine, a driver to call the traverse code).\nI found it easier to reason about each filter in isolation.\nAnd it makes it easier to work on each independently and keep\ntheir inclusion in separate commits.\n\n\n  \n"},{"id":"329045","messageId":"0373bf21-af3c-0ab6-9fba-8559ba05ebb7@jeffhostetler.com","threadId":"46810","inReplyTo":"20170926154457.51f4da732122f00381d646e1@google.com","subject":"Re: [PATCH 09/13] rev-list: add object filtering support","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-09-27T17:26:16Z","receivedAt":"2017-09-27T17:26:23Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 9/26/2017 6:44 PM, Jonathan Tan wrote:\n> On Fri, 22 Sep 2017 20:30:13 +0000\n> Jeff Hostetler <git@jeffhostetler.com> wrote:\n> \n>> +\t\tif (filter_options.relax) {\n> \n> Add some documentation about how this differs from ignore_missing_links\n> in struct rev_info.\n> \n\nIt's unclear what that field means.  (It's not documented that I saw.)\nAnd it is explicitly turned on and off in pack-bitmap.c, so again I'm\nnot sure how that agrees with what I'm doing.\n\nMy relax field is probably an interim thing (to allow rev-list, index-pack,\nand friends not complain about missing objects during my development).\n\nI suspect that you and I will need to merge our efforts here.  You have\na similar variable \"revs->exclude_promisor_objects\".  But I need to study\nhow our usages differ.\n\nThanks,\nJeff\n"},{"id":"329080","messageId":"20170927170533.65498396e008fa148a3fda90@google.com","threadId":"46810","inReplyTo":"7774ff8d-3a53-860d-9343-292938d59d12@jeffhostetler.com","subject":"Re: [PATCH 07/13] object-filter: common declarations for object filtering","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-09-28T00:05:33Z","receivedAt":"2017-09-28T00:05:41Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Wed, 27 Sep 2017 13:09:42 -0400\nJeff Hostetler <git@jeffhostetler.com> wrote:\n\n> On 9/26/2017 6:39 PM, Jonathan Tan wrote:\n> > On Fri, 22 Sep 2017 20:30:11 +0000\n> > Jeff Hostetler <git@jeffhostetler.com> wrote:\n> > \n> >>   Makefile        |   1 +\n> >>   object-filter.c | 269 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n> >>   object-filter.h | 173 ++++++++++++++++++++++++++++++++++++\n> >>   3 files changed, 443 insertions(+)\n> >>   create mode 100644 object-filter.c\n> >>   create mode 100644 object-filter.h\n> > \n> > I think these and list-objects-filter-* are multiple levels of\n> > indirection too many. Would a single file with a few implementations of\n> > filter_object_fn be sufficient?\n> \n> I did that in my first draft and I found it confusing.\n> \n> Each filter has 3 parts (some filter-specific data structures,\n> a filter callback routine, a driver to call the traverse code).\n> I found it easier to reason about each filter in isolation.\n> And it makes it easier to work on each independently and keep\n> their inclusion in separate commits.\n\nI looked at object-filter.{c,h} a bit more. It seems that these files:\n 1) define a struct that contains all the options that we would want\n 2) supplies a way to populate this struct from code that uses parse-options\n 3) supplies a way to populate this struct from code that calculates\n    options by hand\n 4) supplies a way to populate this struct from \"protocol\" (\"<key>\" or\n    \"<key> <value>\" strings)\n\nAnd the next commit takes the struct that object-filter.{c,h} produces\nand actually performs the traversal.\n\nI think this can be significantly simplified, though. Would this work:\n a) Define the object_filter_options struct, but make all fields\n    correspond to a single parameter each. Define\n    OBJECT_FILTER_OPTIONS_INIT to initialize everything to 0 except for\n    large_byte_limit to ULONG_MAX (so that we can detect if something\n    else is set to it).\n b) Define one single OPT_PARSE_FILTER macro containing all the\n    parameters. We can use the non-callback macros here. That solves 2)\n    above.\n c) Define a function that takes in (int *argc, char ***argv) that can\n    \"massage\" it to remove all filter-related arguments, storing them in\n    a object_filter_options struct. That solves 3) above. As discussed\n    in the API documentation, this means that argument lists of the form\n    \"--unknown --known\" (where \"--unknown\" takes an argument) are\n    processed differently, but then again, rev-list never supported them\n    anyway (it required \"--unknown=<arg>\").\n d) Define a function that converts \"<key>\" into \"--<key>\" and \"<key>\n    <value>\" into \"--<key>=<value>\", and use the existing mechanism.\n    That solves 4) above.\n\nThis removes the need to maintain the lists of one-per-argument\nfunctions, including the parse_filter_* and opt_parse_filter_* functions\ndeclared in the header file. If we were to add a feature, we wouldn't\nneed to change anything in the caller, and wouldn't need to hand-edit\nobject_filter_hand_parse_arg() and object_filter_hand_parse_protocol().\n"},{"id":"329108","messageId":"3cab0259-f20e-5913-d886-e37c4da9fd04@jeffhostetler.com","threadId":"46810","inReplyTo":"20170927170533.65498396e008fa148a3fda90@google.com","subject":"Re: [PATCH 07/13] object-filter: common declarations for object filtering","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-09-28T14:33:39Z","receivedAt":"2017-09-28T14:33:47Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 9/27/2017 8:05 PM, Jonathan Tan wrote:\n> On Wed, 27 Sep 2017 13:09:42 -0400\n> Jeff Hostetler <git@jeffhostetler.com> wrote:\n> \n>> On 9/26/2017 6:39 PM, Jonathan Tan wrote:\n>>> On Fri, 22 Sep 2017 20:30:11 +0000\n>>> Jeff Hostetler <git@jeffhostetler.com> wrote:\n>>>\n>>>>    Makefile        |   1 +\n>>>>    object-filter.c | 269 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n>>>>    object-filter.h | 173 ++++++++++++++++++++++++++++++++++++\n>>>>    3 files changed, 443 insertions(+)\n>>>>    create mode 100644 object-filter.c\n>>>>    create mode 100644 object-filter.h\n>>>\n>>> I think these and list-objects-filter-* are multiple levels of\n>>> indirection too many. Would a single file with a few implementations of\n>>> filter_object_fn be sufficient?\n>>\n>> I did that in my first draft and I found it confusing.\n>>\n>> Each filter has 3 parts (some filter-specific data structures,\n>> a filter callback routine, a driver to call the traverse code).\n>> I found it easier to reason about each filter in isolation.\n>> And it makes it easier to work on each independently and keep\n>> their inclusion in separate commits.\n> \n> I looked at object-filter.{c,h} a bit more. It seems that these files:\n>   1) define a struct that contains all the options that we would want\n>   2) supplies a way to populate this struct from code that uses parse-options\n>   3) supplies a way to populate this struct from code that calculates\n>      options by hand\n>   4) supplies a way to populate this struct from \"protocol\" (\"<key>\" or\n>      \"<key> <value>\" strings)\n> \n> And the next commit takes the struct that object-filter.{c,h} produces\n> and actually performs the traversal.\n> \n> I think this can be significantly simplified, though. Would this work:\n>   a) Define the object_filter_options struct, but make all fields\n>      correspond to a single parameter each. Define\n>      OBJECT_FILTER_OPTIONS_INIT to initialize everything to 0 except for\n>      large_byte_limit to ULONG_MAX (so that we can detect if something\n>      else is set to it).\n>   b) Define one single OPT_PARSE_FILTER macro containing all the\n>      parameters. We can use the non-callback macros here. That solves 2)\n>      above.\n>   c) Define a function that takes in (int *argc, char ***argv) that can\n>      \"massage\" it to remove all filter-related arguments, storing them in\n>      a object_filter_options struct. That solves 3) above. As discussed\n>      in the API documentation, this means that argument lists of the form\n>      \"--unknown --known\" (where \"--unknown\" takes an argument) are\n>      processed differently, but then again, rev-list never supported them\n>      anyway (it required \"--unknown=<arg>\").\n>   d) Define a function that converts \"<key>\" into \"--<key>\" and \"<key>\n>      <value>\" into \"--<key>=<value>\", and use the existing mechanism.\n>      That solves 4) above.\n> \n> This removes the need to maintain the lists of one-per-argument\n> functions, including the parse_filter_* and opt_parse_filter_* functions\n> declared in the header file. If we were to add a feature, we wouldn't\n> need to change anything in the caller, and wouldn't need to hand-edit\n> object_filter_hand_parse_arg() and object_filter_hand_parse_protocol().\n\nMaybe.  What I have here now is the result of adding these arguments to\nrev-list and pack-objects (in the current patch series), and also to\nfetch-pack, fetch, clone, upload-pack, index-pack, and the transport and\nprotocol code (in a follow-on patch series that I've omitted for the moment).\nAnd there will probably be a few more, such as fsck, gc, and etc.  I hesitate\nto refine the macros too much further until we've agreement on the overall\napproach and terms.\n\nThanks\nJeff\n\n\n"},{"id":"329190","messageId":"20170929124703.29539de1a7a72c34e7d8ebe5@google.com","threadId":"46810","inReplyTo":"3cab0259-f20e-5913-d886-e37c4da9fd04@jeffhostetler.com","subject":"Re: [PATCH 07/13] object-filter: common declarations for object filtering","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-09-29T19:47:03Z","receivedAt":"2017-09-29T19:47:10Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Thu, 28 Sep 2017 10:33:39 -0400\nJeff Hostetler <git@jeffhostetler.com> wrote:\n\n> Maybe.  What I have here now is the result of adding these arguments to\n> rev-list and pack-objects (in the current patch series), and also to\n> fetch-pack, fetch, clone, upload-pack, index-pack, and the transport and\n> protocol code (in a follow-on patch series that I've omitted for the moment).\n> And there will probably be a few more, such as fsck, gc, and etc.  I hesitate\n> to refine the macros too much further until we've agreement on the overall\n> approach and terms.\n\nFair enough. My current opinion on the overall approach (others might\ndiffer, of course):\n - Filtering based on a sparse checkout specification in rev-list and\n   pack-objects sounds useful to me, and is worth the filtering\n   mechanism.\n - Filtering based on size (or based on type) still doesn't seem useful\n   to me in rev-list, but if we're going to implement the filtering\n   mechanism anyway, we might as well use the mechanism.\n - Besides my comments in [1], I think the API could still be slightly\n   better organized. For example, object-filter probably should be the\n   one to define the traverse_ function that takes in struct\n   object_filter_options, and optionally a set of excluded objects to\n   populate.\n\n[1] https://public-inbox.org/git/20170927170533.65498396e008fa148a3fda90@google.com/\n"}]}