{"thread":{"id":"51217","subject":"[PATCH v2 0/9] Filter combination","startedAt":"2019-06-01T00:36:15Z","lastAt":"2019-06-28T17:16:38Z","messageCount":74,"participants":["Matthew DeVore","Jeff Hostetler","Jacob Keller","Junio C Hamano","Johannes Schindelin","Jonathan Tan"],"isPatch":true,"patchVersion":2,"patchTotal":9},"messages":[{"id":"376516","messageId":"20190601003603.90794-1-matvore@google.com","threadId":"51217","inReplyTo":null,"subject":"[PATCH v2 0/9] Filter combination","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-01T00:35:54Z","receivedAt":"2019-06-01T00:36:15Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Here is a roll-up with hopefully all comments applied or responded to. Notable\nchanges since the last one include:\n\n - Added an ALLOC_GROW_BY which is used twice by this patchset to make growing\n arrays safer and cleaner\n - Cleaned up the URL-encoding by (1) using hex_to_bytes rather than rolling my\n own helpers and (2) making error-string-generation non-conditional\n - Switched to an array-based data structure rather than a linked list for both\n LOFC_COMBINE filter spec objects and the filter object itself\n - Changed the list_objects_filter API to be cleaner to use\n - Changed test cases to use sparse:oid= rather than sparse:path= since the\n latter is being disabled.\n\nThank you,\n\nMatthew DeVore (9):\n list-objects-filter: make API easier to use\n list-objects-filter: put omits set in filter struct\n list-objects-filter-options: always supply *errbuf\n list-objects-filter: implement composite filters\n list-objects-filter-options: move error check up\n list-objects-filter-options: make filter_spec a strbuf\n list-objects-filter-options: allow mult. --filter\n list-objects-filter-options: clean up use of ALLOC_GROW\n list-objects-filter-options: make parser void\n\n Documentation/rev-list-options.txt | 16 ++\n builtin/rev-list.c | 2 +-\n cache.h | 22 ++\n list-objects-filter-options.c | 264 ++++++++++++++++++---\n list-objects-filter-options.h | 32 ++-\n list-objects-filter.c | 345 +++++++++++++++++++++-------\n list-objects-filter.h | 35 ++-\n list-objects.c | 55 ++---\n t/t5616-partial-clone.sh | 19 ++\n t/t6112-rev-list-filters-objects.sh | 197 +++++++++++++++-\n transport.c | 1 +\n upload-pack.c | 4 +-\n 12 files changed, 816 insertions(+), 176 deletions(-)\n\n-- \n2.17.1\n\n"},{"id":"376517","messageId":"20190601003603.90794-2-matvore@google.com","threadId":"51217","inReplyTo":"20190601003603.90794-1-matvore@google.com","subject":"[PATCH v2 1/9] list-objects-filter: make API easier to use","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-01T00:35:55Z","receivedAt":"2019-06-01T00:36:19Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Make the list-objects-filter.h API more opaque and easier to use. This\nprepares for combined filter support, where filters will be created and\nused in a new context.\n\nHelped-by: Jeff Hostetler \nHelped-by: Junio C Hamano \nSigned-off-by: Matthew DeVore \n---\n list-objects-filter.c | 122 +++++++++++++++++++++++++++---------------\n list-objects-filter.h | 35 ++++++------\n list-objects.c | 55 ++++++++-----------\n 3 files changed, 117 insertions(+), 95 deletions(-)\n\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex ee449de3f7..35e0bbe123 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -19,20 +19,34 @@\n * FILTER_SHOWN_BUT_REVISIT -- we set this bit on tree objects\n * that have been shown, but should be revisited if they appear\n * in the traversal (until we mark it SEEN). This is a way to\n * let us silently de-dup calls to show() in the caller. This\n * is subtly different from the \"revision.h:SHOWN\" and the\n * \"sha1-name.c:ONELINE_SEEN\" bits. And also different from\n * the non-de-dup usage in pack-bitmap.c\n */\n #define FILTER_SHOWN_BUT_REVISIT (1<<21)\n \n+struct filter {\n+\tenum list_objects_filter_result (*filter_object_fn)(\n+\t\tstruct repository *r,\n+\t\tenum list_objects_filter_situation filter_situation,\n+\t\tstruct object *obj,\n+\t\tconst char *pathname,\n+\t\tconst char *filename,\n+\t\tvoid *filter_data);\n+\n+\tvoid (*free_fn)(void *filter_data);\n+\n+\tvoid *filter_data;\n+};\n+\n /*\n * A filter for list-objects to omit ALL blobs from the traversal.\n * And to OPTIONALLY collect a list of the omitted OIDs.\n */\n struct filter_blobs_none_data {\n \tstruct oidset *omits;\n };\n \n static enum list_objects_filter_result filter_blobs_none(\n \tstruct repository *r,\n@@ -60,32 +74,31 @@ static enum list_objects_filter_result filter_blobs_none(\n \tcase LOFS_BLOB:\n \t\tassert(obj->type == OBJ_BLOB);\n \t\tassert((obj->flags & SEEN) == 0);\n \n \t\tif (filter_data->omits)\n \t\t\toidset_insert(filter_data->omits, &obj->oid);\n \t\treturn LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n \t}\n }\n \n-static void *filter_blobs_none__init(\n+static void filter_blobs_none__init(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn)\n+\tstruct filter *filter)\n {\n \tstruct filter_blobs_none_data *d = xcalloc(1, sizeof(*d));\n \td->omits = omitted;\n \n-\t*filter_fn = filter_blobs_none;\n-\t*filter_free_fn = free;\n-\treturn d;\n+\tfilter->filter_data = d;\n+\tfilter->filter_object_fn = filter_blobs_none;\n+\tfilter->free_fn = free;\n }\n \n /*\n * A filter for list-objects to omit ALL trees and blobs from the traversal.\n * Can OPTIONALLY collect a list of the omitted OIDs.\n */\n struct filter_trees_depth_data {\n \tstruct oidset *omits;\n \n \t/*\n@@ -194,35 +207,34 @@ static enum list_objects_filter_result filter_trees_depth(\n }\n \n static void filter_trees_free(void *filter_data) {\n \tstruct filter_trees_depth_data *d = filter_data;\n \tif (!d)\n \t\treturn;\n \toidmap_free(&d->seen_at_depth, 1);\n \tfree(d);\n }\n \n-static void *filter_trees_depth__init(\n+static void filter_trees_depth__init(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn)\n+\tstruct filter *filter)\n {\n \tstruct filter_trees_depth_data *d = xcalloc(1, sizeof(*d));\n \td->omits = omitted;\n \toidmap_init(&d->seen_at_depth, 0);\n \td->exclude_depth = filter_options->tree_exclude_depth;\n \td->current_depth = 0;\n \n-\t*filter_fn = filter_trees_depth;\n-\t*filter_free_fn = filter_trees_free;\n-\treturn d;\n+\tfilter->filter_data = d;\n+\tfilter->filter_object_fn = filter_trees_depth;\n+\tfilter->free_fn = filter_trees_free;\n }\n \n /*\n * A filter for list-objects to omit large blobs.\n * And to OPTIONALLY collect a list of the omitted OIDs.\n */\n struct filter_blobs_limit_data {\n \tstruct oidset *omits;\n \tunsigned long max_bytes;\n };\n@@ -274,33 +286,32 @@ static enum list_objects_filter_result filter_blobs_limit(\n \t\t\toidset_insert(filter_data->omits, &obj->oid);\n \t\treturn LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n \t}\n \n include_it:\n \tif (filter_data->omits)\n \t\toidset_remove(filter_data->omits, &obj->oid);\n \treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n }\n \n-static void *filter_blobs_limit__init(\n+static void filter_blobs_limit__init(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn)\n+\tstruct filter *filter)\n {\n \tstruct filter_blobs_limit_data *d = xcalloc(1, sizeof(*d));\n \td->omits = omitted;\n \td->max_bytes = filter_options->blob_limit_value;\n \n-\t*filter_fn = filter_blobs_limit;\n-\t*filter_free_fn = free;\n-\treturn d;\n+\tfilter->filter_data = d;\n+\tfilter->filter_object_fn = filter_blobs_limit;\n+\tfilter->free_fn = free;\n }\n \n /*\n * A filter driven by a sparse-checkout specification to only\n * include blobs that a sparse checkout would populate.\n *\n * The sparse-checkout spec can be loaded from a blob with the\n * given OID or from a local pathname. We allow an OID because\n * the repo may be bare or we may be doing the filtering on the\n * server.\n@@ -450,92 +461,117 @@ static enum list_objects_filter_result filter_sparse(\n }\n \n \n static void filter_sparse_free(void *filter_data)\n {\n \tstruct filter_sparse_data *d = filter_data;\n \t/* TODO free contents of 'd' */\n \tfree(d);\n }\n \n-static void *filter_sparse_oid__init(\n+static void filter_sparse_oid__init(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn)\n+\tstruct filter *filter)\n {\n \tstruct filter_sparse_data *d = xcalloc(1, sizeof(*d));\n \td->omits = omitted;\n \tif (add_excludes_from_blob_to_list(filter_options->sparse_oid_value,\n \t\t\t\t\t NULL, 0, &d->el) < 0)\n \t\tdie(\"could not load filter specification\");\n \n \tALLOC_GROW(d->array_frame, d->nr + 1, d->alloc);\n \td->array_frame[d->nr].defval = 0; /* default to include */\n \td->array_frame[d->nr].child_prov_omit = 0;\n \n-\t*filter_fn = filter_sparse;\n-\t*filter_free_fn = filter_sparse_free;\n-\treturn d;\n+\tfilter->filter_data = d;\n+\tfilter->filter_object_fn = filter_sparse;\n+\tfilter->free_fn = filter_sparse_free;\n }\n \n-static void *filter_sparse_path__init(\n+static void filter_sparse_path__init(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn)\n+\tstruct filter *filter)\n {\n \tstruct filter_sparse_data *d = xcalloc(1, sizeof(*d));\n \td->omits = omitted;\n \tif (add_excludes_from_file_to_list(filter_options->sparse_path_value,\n \t\t\t\t\t NULL, 0, &d->el, NULL) < 0)\n \t\tdie(\"could not load filter specification\");\n \n \tALLOC_GROW(d->array_frame, d->nr + 1, d->alloc);\n \td->array_frame[d->nr].defval = 0; /* default to include */\n \td->array_frame[d->nr].child_prov_omit = 0;\n \n-\t*filter_fn = filter_sparse;\n-\t*filter_free_fn = filter_sparse_free;\n-\treturn d;\n+\tfilter->filter_data = d;\n+\tfilter->filter_object_fn = filter_sparse;\n+\tfilter->free_fn = filter_sparse_free;\n }\n \n-typedef void *(*filter_init_fn)(\n+typedef void (*filter_init_fn)(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn);\n+\tstruct filter *filter);\n \n /*\n * Must match \"enum list_objects_filter_choice\".\n */\n static filter_init_fn s_filters[] = {\n \tNULL,\n \tfilter_blobs_none__init,\n \tfilter_blobs_limit__init,\n \tfilter_trees_depth__init,\n \tfilter_sparse_oid__init,\n \tfilter_sparse_path__init,\n };\n \n-void *list_objects_filter__init(\n+struct filter *list_objects_filter__init(\n \tstruct oidset *omitted,\n-\tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn)\n+\tstruct list_objects_filter_options *filter_options)\n {\n+\tstruct filter *filter;\n \tfilter_init_fn init_fn;\n \n \tassert((sizeof(s_filters) / sizeof(s_filters[0])) == LOFC__COUNT);\n \n \tif (filter_options->choice >= LOFC__COUNT)\n \t\tBUG(\"invalid list-objects filter choice: %d\",\n \t\t filter_options->choice);\n \n \tinit_fn = s_filters[filter_options->choice];\n-\tif (init_fn)\n-\t\treturn init_fn(omitted, filter_options,\n-\t\t\t filter_fn, filter_free_fn);\n-\t*filter_fn = NULL;\n-\t*filter_free_fn = NULL;\n-\treturn NULL;\n+\tif (!init_fn)\n+\t\treturn NULL;\n+\n+\tfilter = xcalloc(1, sizeof(*filter));\n+\tinit_fn(omitted, filter_options, filter);\n+\treturn filter;\n+}\n+\n+enum list_objects_filter_result list_objects_filter__filter_object(\n+\tstruct repository *r,\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tstruct filter *filter)\n+{\n+\tif (filter && (obj->flags & NOT_USER_GIVEN))\n+\t\treturn filter->filter_object_fn(r, filter_situation, obj,\n+\t\t\t\t\t\tpathname, filename,\n+\t\t\t\t\t\tfilter->filter_data);\n+\t/*\n+\t * No filter is active or user gave object explicitly. Choose default\n+\t * behavior based on filter situation.\n+\t */\n+\tif (filter_situation == LOFS_END_TREE)\n+\t\treturn 0;\n+\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+}\n+\n+void list_objects_filter__free(struct filter *filter)\n+{\n+\tif (!filter)\n+\t\treturn;\n+\tfilter->free_fn(filter->filter_data);\n+\tfree(filter);\n }\ndiff --git a/list-objects-filter.h b/list-objects-filter.h\nindex 1d45a4ad57..6908954266 100644\n--- a/list-objects-filter.h\n+++ b/list-objects-filter.h\n@@ -53,37 +53,34 @@ enum list_objects_filter_result {\n \tLOFR_DO_SHOW = 1<<1,\n \tLOFR_SKIP_TREE = 1<<2,\n };\n \n enum list_objects_filter_situation {\n \tLOFS_BEGIN_TREE,\n \tLOFS_END_TREE,\n \tLOFS_BLOB\n };\n \n-typedef enum list_objects_filter_result (*filter_object_fn)(\n+struct filter;\n+\n+/* Constructor for the set of defined list-objects filters. */\n+struct filter *list_objects_filter__init(\n+\tstruct oidset *omitted,\n+\tstruct list_objects_filter_options *filter_options);\n+\n+/*\n+ * Lets `filter` decide how to handle the `obj`. If `filter` is NULL, this\n+ * function behaves as expected if no filter is configured: all objects are\n+ * included.\n+ */\n+enum list_objects_filter_result list_objects_filter__filter_object(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n-\tvoid *filter_data);\n-\n-typedef void (*filter_free_fn)(void *filter_data);\n+\tstruct filter *filter);\n \n-/*\n- * Constructor for the set of defined list-objects filters.\n- * Returns a generic \"void *filter_data\".\n- *\n- * The returned \"filter_fn\" will be used by traverse_commit_list()\n- * to filter the results.\n- *\n- * The returned \"filter_free_fn\" is a destructor for the\n- * filter_data.\n- */\n-void *list_objects_filter__init(\n-\tstruct oidset *omitted,\n-\tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn);\n+/* Destroys `filter`. Does nothing if `filter` is null. */\n+void list_objects_filter__free(struct filter *filter);\n \n #endif /* LIST_OBJECTS_FILTER_H */\ndiff --git a/list-objects.c b/list-objects.c\nindex b5651ddd5b..9307d91fb3 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -11,32 +11,31 @@\n #include \"list-objects-filter-options.h\"\n #include \"packfile.h\"\n #include \"object-store.h\"\n #include \"trace.h\"\n \n struct traversal_context {\n \tstruct rev_info *revs;\n \tshow_object_fn show_object;\n \tshow_commit_fn show_commit;\n \tvoid *show_data;\n-\tfilter_object_fn filter_fn;\n-\tvoid *filter_data;\n+\tstruct filter *filter;\n };\n \n static void process_blob(struct traversal_context *ctx,\n \t\t\t struct blob *blob,\n \t\t\t struct strbuf *path,\n \t\t\t const char *name)\n {\n \tstruct object *obj = &blob->object;\n \tsize_t pathlen;\n-\tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\tenum list_objects_filter_result r;\n \n \tif (!ctx->revs->blob_objects)\n \t\treturn;\n \tif (!obj)\n \t\tdie(\"bad blob object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n \n \t/*\n \t * Pre-filter known-missing objects when explicitly requested.\n@@ -47,25 +46,24 @@ static void process_blob(struct traversal_context *ctx,\n \t * may cause the actual filter to report an incomplete list\n \t * of missing objects.\n \t */\n \tif (ctx->revs->exclude_promisor_objects &&\n \t !has_object_file(&obj->oid) &&\n \t is_promisor_object(&obj->oid))\n \t\treturn;\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n-\t\tr = ctx->filter_fn(ctx->revs->repo,\n-\t\t\t\t LOFS_BLOB, obj,\n-\t\t\t\t path->buf, &path->buf[pathlen],\n-\t\t\t\t ctx->filter_data);\n+\tr = list_objects_filter__filter_object(ctx->revs->repo,\n+\t\t\t\t\t LOFS_BLOB, obj,\n+\t\t\t\t\t path->buf, &path->buf[pathlen],\n+\t\t\t\t\t ctx->filter);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n \t\tctx->show_object(obj, path->buf, ctx->show_data);\n \tstrbuf_setlen(path, pathlen);\n }\n \n /*\n * Processing a gitlink entry currently does nothing, since\n * we do not recurse into the subproject.\n@@ -150,21 +148,21 @@ static void process_tree_contents(struct traversal_context *ctx,\n }\n \n static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n \t\t\t const char *name)\n {\n \tstruct object *obj = &tree->object;\n \tstruct rev_info *revs = ctx->revs;\n \tint baselen = base->len;\n-\tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\tenum list_objects_filter_result r;\n \tint failed_parse;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n \tif (!obj)\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n \n \tfailed_parse = parse_tree_gently(tree, 1);\n@@ -179,47 +177,44 @@ static void process_tree(struct traversal_context *ctx,\n \t\t */\n \t\tif (revs->exclude_promisor_objects &&\n \t\t is_promisor_object(&obj->oid))\n \t\t\treturn;\n \n \t\tif (!revs->do_not_die_on_missing_tree)\n \t\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n-\t\tr = ctx->filter_fn(ctx->revs->repo,\n-\t\t\t\t LOFS_BEGIN_TREE, obj,\n-\t\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t\t ctx->filter_data);\n+\tr = list_objects_filter__filter_object(ctx->revs->repo,\n+\t\t\t\t\t LOFS_BEGIN_TREE, obj,\n+\t\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t\t ctx->filter);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n \t\tctx->show_object(obj, base->buf, ctx->show_data);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n \tif (r & LOFR_SKIP_TREE)\n \t\ttrace_printf(\"Skipping contents of tree %s...\\n\", base->buf);\n \telse if (!failed_parse)\n \t\tprocess_tree_contents(ctx, tree, base);\n \n-\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) {\n-\t\tr = ctx->filter_fn(ctx->revs->repo,\n-\t\t\t\t LOFS_END_TREE, obj,\n-\t\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t\t ctx->filter_data);\n-\t\tif (r & LOFR_MARK_SEEN)\n-\t\t\tobj->flags |= SEEN;\n-\t\tif (r & LOFR_DO_SHOW)\n-\t\t\tctx->show_object(obj, base->buf, ctx->show_data);\n-\t}\n+\tr = list_objects_filter__filter_object(ctx->revs->repo,\n+\t\t\t\t\t LOFS_END_TREE, obj,\n+\t\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t\t ctx->filter);\n+\tif (r & LOFR_MARK_SEEN)\n+\t\tobj->flags |= SEEN;\n+\tif (r & LOFR_DO_SHOW)\n+\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \n \tstrbuf_setlen(base, baselen);\n \tfree_tree_buffer(tree);\n }\n \n static void mark_edge_parents_uninteresting(struct commit *commit,\n \t\t\t\t\t struct rev_info *revs,\n \t\t\t\t\t show_edge_fn show_edge)\n {\n \tstruct commit_list *parents;\n@@ -395,38 +390,32 @@ static void do_traverse(struct traversal_context *ctx)\n void traverse_commit_list(struct rev_info *revs,\n \t\t\t show_commit_fn show_commit,\n \t\t\t show_object_fn show_object,\n \t\t\t void *show_data)\n {\n \tstruct traversal_context ctx;\n \tctx.revs = revs;\n \tctx.show_commit = show_commit;\n \tctx.show_object = show_object;\n \tctx.show_data = show_data;\n-\tctx.filter_fn = NULL;\n-\tctx.filter_data = NULL;\n+\tctx.filter = NULL;\n \tdo_traverse(&ctx);\n }\n \n void traverse_commit_list_filtered(\n \tstruct list_objects_filter_options *filter_options,\n \tstruct rev_info *revs,\n \tshow_commit_fn show_commit,\n \tshow_object_fn show_object,\n \tvoid *show_data,\n \tstruct oidset *omitted)\n {\n \tstruct traversal_context ctx;\n-\tfilter_free_fn filter_free_fn = NULL;\n \n \tctx.revs = revs;\n \tctx.show_object = show_object;\n \tctx.show_commit = show_commit;\n \tctx.show_data = show_data;\n-\tctx.filter_fn = NULL;\n-\n-\tctx.filter_data = list_objects_filter__init(omitted, filter_options,\n-\t\t\t\t\t\t &ctx.filter_fn, &filter_free_fn);\n+\tctx.filter = list_objects_filter__init(omitted, filter_options);\n \tdo_traverse(&ctx);\n-\tif (ctx.filter_data && filter_free_fn)\n-\t\tfilter_free_fn(ctx.filter_data);\n+\tlist_objects_filter__free(ctx.filter);\n }\n-- \n2.17.1\n\n"},{"id":"376518","messageId":"20190601003603.90794-3-matvore@google.com","threadId":"51217","inReplyTo":"20190601003603.90794-1-matvore@google.com","subject":"[PATCH v2 2/9] list-objects-filter: put omits set in filter struct","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-01T00:35:56Z","receivedAt":"2019-06-01T00:36:21Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"The oidset *omits pointer must be accessed by the combine filter in a\ntype-agnostic way once the graph traversal is over. Store that pointer\nin the general `filter` struct. This will be used in a follow-up patch\nto implement the combine filter.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects-filter.c | 70 ++++++++++++++++---------------------------\n 1 file changed, 26 insertions(+), 44 deletions(-)\n\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex 35e0bbe123..57bbf6ec1c 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -26,88 +26,76 @@\n */\n #define FILTER_SHOWN_BUT_REVISIT (1<<21)\n \n struct filter {\n \tenum list_objects_filter_result (*filter_object_fn)(\n \t\tstruct repository *r,\n \t\tenum list_objects_filter_situation filter_situation,\n \t\tstruct object *obj,\n \t\tconst char *pathname,\n \t\tconst char *filename,\n+\t\tstruct oidset *omits,\n \t\tvoid *filter_data);\n \n \tvoid (*free_fn)(void *filter_data);\n \n \tvoid *filter_data;\n-};\n \n-/*\n- * A filter for list-objects to omit ALL blobs from the traversal.\n- * And to OPTIONALLY collect a list of the omitted OIDs.\n- */\n-struct filter_blobs_none_data {\n+\t/* If non-NULL, the filter collects a list of the omitted OIDs here. */\n \tstruct oidset *omits;\n };\n \n static enum list_objects_filter_result filter_blobs_none(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n+\tstruct oidset *omits,\n \tvoid *filter_data_)\n {\n-\tstruct filter_blobs_none_data *filter_data = filter_data_;\n-\n \tswitch (filter_situation) {\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\t/* always include all tree objects */\n \t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \n \tcase LOFS_END_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\treturn LOFR_ZERO;\n \n \tcase LOFS_BLOB:\n \t\tassert(obj->type == OBJ_BLOB);\n \t\tassert((obj->flags & SEEN) == 0);\n \n-\t\tif (filter_data->omits)\n-\t\t\toidset_insert(filter_data->omits, &obj->oid);\n+\t\tif (omits)\n+\t\t\toidset_insert(omits, &obj->oid);\n \t\treturn LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n \t}\n }\n \n static void filter_blobs_none__init(\n-\tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter)\n {\n-\tstruct filter_blobs_none_data *d = xcalloc(1, sizeof(*d));\n-\td->omits = omitted;\n-\n-\tfilter->filter_data = d;\n \tfilter->filter_object_fn = filter_blobs_none;\n \tfilter->free_fn = free;\n }\n \n /*\n * A filter for list-objects to omit ALL trees and blobs from the traversal.\n * Can OPTIONALLY collect a list of the omitted OIDs.\n */\n struct filter_trees_depth_data {\n-\tstruct oidset *omits;\n-\n \t/*\n \t * Maps trees to the minimum depth at which they were seen. It is not\n \t * necessary to re-traverse a tree at deeper or equal depths than it has\n \t * already been traversed.\n \t *\n \t * We can't use LOFR_MARK_SEEN for tree objects since this will prevent\n \t * it from being traversed at shallower depths.\n \t */\n \tstruct oidmap seen_at_depth;\n \n@@ -116,38 +104,39 @@ struct filter_trees_depth_data {\n };\n \n struct seen_map_entry {\n \tstruct oidmap_entry base;\n \tsize_t depth;\n };\n \n /* Returns 1 if the oid was in the omits set before it was invoked. */\n static int filter_trees_update_omits(\n \tstruct object *obj,\n-\tstruct filter_trees_depth_data *filter_data,\n+\tstruct oidset *omits,\n \tint include_it)\n {\n-\tif (!filter_data->omits)\n+\tif (!omits)\n \t\treturn 0;\n \n \tif (include_it)\n-\t\treturn oidset_remove(filter_data->omits, &obj->oid);\n+\t\treturn oidset_remove(omits, &obj->oid);\n \telse\n-\t\treturn oidset_insert(filter_data->omits, &obj->oid);\n+\t\treturn oidset_insert(omits, &obj->oid);\n }\n \n static enum list_objects_filter_result filter_trees_depth(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n+\tstruct oidset *omits,\n \tvoid *filter_data_)\n {\n \tstruct filter_trees_depth_data *filter_data = filter_data_;\n \tstruct seen_map_entry *seen_info;\n \tint include_it = filter_data->current_depth <\n \t\tfilter_data->exclude_depth;\n \tint filter_res;\n \tint already_seen;\n \n \t/*\n@@ -158,47 +147,47 @@ static enum list_objects_filter_result filter_trees_depth(\n \tswitch (filter_situation) {\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n \tcase LOFS_END_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\tfilter_data->current_depth--;\n \t\treturn LOFR_ZERO;\n \n \tcase LOFS_BLOB:\n-\t\tfilter_trees_update_omits(obj, filter_data, include_it);\n+\t\tfilter_trees_update_omits(obj, omits, include_it);\n \t\treturn include_it ? LOFR_MARK_SEEN | LOFR_DO_SHOW : LOFR_ZERO;\n \n \tcase LOFS_BEGIN_TREE:\n \t\tseen_info = oidmap_get(\n \t\t\t&filter_data->seen_at_depth, &obj->oid);\n \t\tif (!seen_info) {\n \t\t\tseen_info = xcalloc(1, sizeof(*seen_info));\n \t\t\toidcpy(&seen_info->base.oid, &obj->oid);\n \t\t\tseen_info->depth = filter_data->current_depth;\n \t\t\toidmap_put(&filter_data->seen_at_depth, seen_info);\n \t\t\talready_seen = 0;\n \t\t} else {\n \t\t\talready_seen =\n \t\t\t\tfilter_data->current_depth >= seen_info->depth;\n \t\t}\n \n \t\tif (already_seen) {\n \t\t\tfilter_res = LOFR_SKIP_TREE;\n \t\t} else {\n \t\t\tint been_omitted = filter_trees_update_omits(\n-\t\t\t\tobj, filter_data, include_it);\n+\t\t\t\tobj, omits, include_it);\n \t\t\tseen_info->depth = filter_data->current_depth;\n \n \t\t\tif (include_it)\n \t\t\t\tfilter_res = LOFR_DO_SHOW;\n-\t\t\telse if (filter_data->omits && !been_omitted)\n+\t\t\telse if (omits && !been_omitted)\n \t\t\t\t/*\n \t\t\t\t * Must update omit information of children\n \t\t\t\t * recursively; they have not been omitted yet.\n \t\t\t\t */\n \t\t\t\tfilter_res = LOFR_ZERO;\n \t\t\telse\n \t\t\t\tfilter_res = LOFR_SKIP_TREE;\n \t\t}\n \n \t\tfilter_data->current_depth++;\n@@ -208,50 +197,48 @@ static enum list_objects_filter_result filter_trees_depth(\n \n static void filter_trees_free(void *filter_data) {\n \tstruct filter_trees_depth_data *d = filter_data;\n \tif (!d)\n \t\treturn;\n \toidmap_free(&d->seen_at_depth, 1);\n \tfree(d);\n }\n \n static void filter_trees_depth__init(\n-\tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter)\n {\n \tstruct filter_trees_depth_data *d = xcalloc(1, sizeof(*d));\n-\td->omits = omitted;\n \toidmap_init(&d->seen_at_depth, 0);\n \td->exclude_depth = filter_options->tree_exclude_depth;\n \td->current_depth = 0;\n \n \tfilter->filter_data = d;\n \tfilter->filter_object_fn = filter_trees_depth;\n \tfilter->free_fn = filter_trees_free;\n }\n \n /*\n * A filter for list-objects to omit large blobs.\n * And to OPTIONALLY collect a list of the omitted OIDs.\n */\n struct filter_blobs_limit_data {\n-\tstruct oidset *omits;\n \tunsigned long max_bytes;\n };\n \n static enum list_objects_filter_result filter_blobs_limit(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n+\tstruct oidset *omits,\n \tvoid *filter_data_)\n {\n \tstruct filter_blobs_limit_data *filter_data = filter_data_;\n \tunsigned long object_length;\n \tenum object_type t;\n \n \tswitch (filter_situation) {\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n@@ -275,38 +262,36 @@ static enum list_objects_filter_result filter_blobs_limit(\n \t\t\t * apply the size filter criteria. Be conservative\n \t\t\t * and force show it (and let the caller deal with\n \t\t\t * the ambiguity).\n \t\t\t */\n \t\t\tgoto include_it;\n \t\t}\n \n \t\tif (object_length < filter_data->max_bytes)\n \t\t\tgoto include_it;\n \n-\t\tif (filter_data->omits)\n-\t\t\toidset_insert(filter_data->omits, &obj->oid);\n+\t\tif (omits)\n+\t\t\toidset_insert(omits, &obj->oid);\n \t\treturn LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n \t}\n \n include_it:\n-\tif (filter_data->omits)\n-\t\toidset_remove(filter_data->omits, &obj->oid);\n+\tif (omits)\n+\t\toidset_remove(omits, &obj->oid);\n \treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n }\n \n static void filter_blobs_limit__init(\n-\tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter)\n {\n \tstruct filter_blobs_limit_data *d = xcalloc(1, sizeof(*d));\n-\td->omits = omitted;\n \td->max_bytes = filter_options->blob_limit_value;\n \n \tfilter->filter_data = d;\n \tfilter->filter_object_fn = filter_blobs_limit;\n \tfilter->free_fn = free;\n }\n \n /*\n * A filter driven by a sparse-checkout specification to only\n * include blobs that a sparse checkout would populate.\n@@ -330,33 +315,33 @@ struct frame {\n \t * omitted objects.\n \t *\n \t * 0 if everything (recursively) contained in this directory\n \t * has been explicitly included (SHOWN) in the result and\n \t * the directory may be short-cut later in the traversal.\n \t */\n \tunsigned child_prov_omit : 1;\n };\n \n struct filter_sparse_data {\n-\tstruct oidset *omits;\n \tstruct exclude_list el;\n \n \tsize_t nr, alloc;\n \tstruct frame *array_frame;\n };\n \n static enum list_objects_filter_result filter_sparse(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n+\tstruct oidset *omits,\n \tvoid *filter_data_)\n {\n \tstruct filter_sparse_data *filter_data = filter_data_;\n \tint val, dtype;\n \tstruct frame *frame;\n \n \tswitch (filter_situation) {\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n@@ -425,98 +410,93 @@ static enum list_objects_filter_result filter_sparse(\n \n \t\tframe = &filter_data->array_frame[filter_data->nr];\n \n \t\tdtype = DT_REG;\n \t\tval = is_excluded_from_list(pathname, strlen(pathname),\n \t\t\t\t\t filename, &dtype, &filter_data->el,\n \t\t\t\t\t r->index);\n \t\tif (val < 0)\n \t\t\tval = frame->defval;\n \t\tif (val > 0) {\n-\t\t\tif (filter_data->omits)\n-\t\t\t\toidset_remove(filter_data->omits, &obj->oid);\n+\t\t\tif (omits)\n+\t\t\t\toidset_remove(omits, &obj->oid);\n \t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \t\t}\n \n \t\t/*\n \t\t * Provisionally omit it. We've already established that\n \t\t * this pathname is not in the sparse-checkout specification\n \t\t * with the CURRENT pathname, so we *WANT* to omit this blob.\n \t\t *\n \t\t * However, a pathname elsewhere in the tree may also\n \t\t * reference this same blob, so we cannot reject it yet.\n \t\t * Leave the LOFR_ bits unset so that if the blob appears\n \t\t * again in the traversal, we will be asked again.\n \t\t */\n-\t\tif (filter_data->omits)\n-\t\t\toidset_insert(filter_data->omits, &obj->oid);\n+\t\tif (omits)\n+\t\t\toidset_insert(omits, &obj->oid);\n \n \t\t/*\n \t\t * Remember that at least 1 blob in this tree was\n \t\t * provisionally omitted. This prevents us from short\n \t\t * cutting the tree in future iterations.\n \t\t */\n \t\tframe->child_prov_omit = 1;\n \t\treturn LOFR_ZERO;\n \t}\n }\n \n \n static void filter_sparse_free(void *filter_data)\n {\n \tstruct filter_sparse_data *d = filter_data;\n \t/* TODO free contents of 'd' */\n \tfree(d);\n }\n \n static void filter_sparse_oid__init(\n-\tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter)\n {\n \tstruct filter_sparse_data *d = xcalloc(1, sizeof(*d));\n-\td->omits = omitted;\n \tif (add_excludes_from_blob_to_list(filter_options->sparse_oid_value,\n \t\t\t\t\t NULL, 0, &d->el) < 0)\n \t\tdie(\"could not load filter specification\");\n \n \tALLOC_GROW(d->array_frame, d->nr + 1, d->alloc);\n \td->array_frame[d->nr].defval = 0; /* default to include */\n \td->array_frame[d->nr].child_prov_omit = 0;\n \n \tfilter->filter_data = d;\n \tfilter->filter_object_fn = filter_sparse;\n \tfilter->free_fn = filter_sparse_free;\n }\n \n static void filter_sparse_path__init(\n-\tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter)\n {\n \tstruct filter_sparse_data *d = xcalloc(1, sizeof(*d));\n-\td->omits = omitted;\n \tif (add_excludes_from_file_to_list(filter_options->sparse_path_value,\n \t\t\t\t\t NULL, 0, &d->el, NULL) < 0)\n \t\tdie(\"could not load filter specification\");\n \n \tALLOC_GROW(d->array_frame, d->nr + 1, d->alloc);\n \td->array_frame[d->nr].defval = 0; /* default to include */\n \td->array_frame[d->nr].child_prov_omit = 0;\n \n \tfilter->filter_data = d;\n \tfilter->filter_object_fn = filter_sparse;\n \tfilter->free_fn = filter_sparse_free;\n }\n \n typedef void (*filter_init_fn)(\n-\tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter);\n \n /*\n * Must match \"enum list_objects_filter_choice\".\n */\n static filter_init_fn s_filters[] = {\n \tNULL,\n \tfilter_blobs_none__init,\n \tfilter_blobs_limit__init,\n@@ -536,35 +516,37 @@ struct filter *list_objects_filter__init(\n \n \tif (filter_options->choice >= LOFC__COUNT)\n \t\tBUG(\"invalid list-objects filter choice: %d\",\n \t\t filter_options->choice);\n \n \tinit_fn = s_filters[filter_options->choice];\n \tif (!init_fn)\n \t\treturn NULL;\n \n \tfilter = xcalloc(1, sizeof(*filter));\n-\tinit_fn(omitted, filter_options, filter);\n+\tfilter->omits = omitted;\n+\tinit_fn(filter_options, filter);\n \treturn filter;\n }\n \n enum list_objects_filter_result list_objects_filter__filter_object(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n \tstruct filter *filter)\n {\n \tif (filter && (obj->flags & NOT_USER_GIVEN))\n \t\treturn filter->filter_object_fn(r, filter_situation, obj,\n \t\t\t\t\t\tpathname, filename,\n+\t\t\t\t\t\tfilter->omits,\n \t\t\t\t\t\tfilter->filter_data);\n \t/*\n \t * No filter is active or user gave object explicitly. Choose default\n \t * behavior based on filter situation.\n \t */\n \tif (filter_situation == LOFS_END_TREE)\n \t\treturn 0;\n \treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n }\n \n-- \n2.17.1\n\n"},{"id":"376519","messageId":"20190601003603.90794-4-matvore@google.com","threadId":"51217","inReplyTo":"20190601003603.90794-1-matvore@google.com","subject":"[PATCH v2 3/9] list-objects-filter-options: always supply *errbuf","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-01T00:35:57Z","receivedAt":"2019-06-01T00:36:24Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Making errbuf an optional argument complicates error reporting. Fix this\nby making all callers supply an errbuf, even if they may ignore it. This\nwill be important in follow-up patches where the filter-spec parsing has\nmore pitfalls and possible errors.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects-filter-options.c | 21 ++++++++-------------\n 1 file changed, 8 insertions(+), 13 deletions(-)\n\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex c0036f7378..aef24ddae3 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -23,47 +23,40 @@\n * convenience of the current command.\n */\n static int gently_parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg,\n \tstruct strbuf *errbuf)\n {\n \tconst char *v0;\n \n \tif (filter_options->choice) {\n-\t\tif (errbuf) {\n-\t\t\tstrbuf_addstr(\n-\t\t\t\terrbuf,\n-\t\t\t\t_(\"multiple filter-specs cannot be combined\"));\n-\t\t}\n+\t\tstrbuf_addstr(\n+\t\t\terrbuf, _(\"multiple filter-specs cannot be combined\"));\n \t\treturn 1;\n \t}\n \n \tfilter_options->filter_spec = strdup(arg);\n \n \tif (!strcmp(arg, \"blob:none\")) {\n \t\tfilter_options->choice = LOFC_BLOB_NONE;\n \t\treturn 0;\n \n \t} else if (skip_prefix(arg, \"blob:limit=\", &v0)) {\n \t\tif (git_parse_ulong(v0, &filter_options->blob_limit_value)) {\n \t\t\tfilter_options->choice = LOFC_BLOB_LIMIT;\n \t\t\treturn 0;\n \t\t}\n \n \t} else if (skip_prefix(arg, \"tree:\", &v0)) {\n \t\tif (!git_parse_ulong(v0, &filter_options->tree_exclude_depth)) {\n-\t\t\tif (errbuf) {\n-\t\t\t\tstrbuf_addstr(\n-\t\t\t\t\terrbuf,\n-\t\t\t\t\t_(\"expected 'tree:'\"));\n-\t\t\t}\n+\t\t\tstrbuf_addstr(errbuf, _(\"expected 'tree:'\"));\n \t\t\treturn 1;\n \t\t}\n \t\tfilter_options->choice = LOFC_TREE_DEPTH;\n \t\treturn 0;\n \n \t} else if (skip_prefix(arg, \"sparse:oid=\", &v0)) {\n \t\tstruct object_context oc;\n \t\tstruct object_id sparse_oid;\n \n \t\t/*\n@@ -80,22 +73,21 @@ static int gently_parse_list_objects_filter(\n \t} else if (skip_prefix(arg, \"sparse:path=\", &v0)) {\n \t\tfilter_options->choice = LOFC_SPARSE_PATH;\n \t\tfilter_options->sparse_path_value = strdup(v0);\n \t\treturn 0;\n \t}\n \t/*\n \t * Please update _git_fetch() in git-completion.bash when you\n \t * add new filters\n \t */\n \n-\tif (errbuf)\n-\t\tstrbuf_addf(errbuf, \"invalid filter-spec '%s'\", arg);\n+\tstrbuf_addf(errbuf, \"invalid filter-spec '%s'\", arg);\n \n \tmemset(filter_options, 0, sizeof(*filter_options));\n \treturn 1;\n }\n \n int parse_list_objects_filter(struct list_objects_filter_options *filter_options,\n \t\t\t const char *arg)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tif (gently_parse_list_objects_filter(filter_options, arg, &buf))\n@@ -166,19 +158,22 @@ void partial_clone_register(\n \t */\n \tcore_partial_clone_filter_default =\n \t\txstrdup(filter_options->filter_spec);\n \tgit_config_set(\"core.partialclonefilter\",\n \t\t core_partial_clone_filter_default);\n }\n \n void partial_clone_get_default_filter_spec(\n \tstruct list_objects_filter_options *filter_options)\n {\n+\tstruct strbuf errbuf = STRBUF_INIT;\n+\n \t/*\n \t * Parse default value, but silently ignore it if it is invalid.\n \t */\n \tif (!core_partial_clone_filter_default)\n \t\treturn;\n \tgently_parse_list_objects_filter(filter_options,\n \t\t\t\t\t core_partial_clone_filter_default,\n-\t\t\t\t\t NULL);\n+\t\t\t\t\t &errbuf);\n+\tstrbuf_release(&errbuf);\n }\n-- \n2.17.1\n\n"},{"id":"376520","messageId":"20190601003603.90794-5-matvore@google.com","threadId":"51217","inReplyTo":"20190601003603.90794-1-matvore@google.com","subject":"[PATCH v2 4/9] list-objects-filter: implement composite filters","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-01T00:35:58Z","receivedAt":"2019-06-01T00:36:27Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Allow combining filters such that only objects accepted by all filters\nare shown. The motivation for this is to allow getting directory\nlistings without also fetching blobs. This can be done by combining\nblob:none with tree:. There are massive repositories that have\nlarger-than-expected trees - even if you include only a single commit.\n\nThe current usage requires passing the filter to rev-list in the\nfollowing form:\n\n\t--filter= --filter= ...\n\nSuch usage is currently an error, so giving it a meaning is backwards-\ncompatible.\n\nThe URL-encoding method is being implemented before the repeated flag\nlogic, and the user-facing documentation for URL-encoding is being\nwithheld until the repeated flag feature is implemented. The\nURL-encoding is in general not meant to be used directly by the user,\nand it is better to describe the URL-encoding feature in terms of the\nrepeated flag.\n\nHelped-by: Emily Shaffer \nHelped-by: Jeff Hostetler \nHelped-by: Junio C Hamano \nSigned-off-by: Matthew DeVore \n---\n list-objects-filter-options.c | 135 ++++++++++++++++++++++-\n list-objects-filter-options.h | 17 ++-\n list-objects-filter.c | 159 +++++++++++++++++++++++++++\n t/t6112-rev-list-filters-objects.sh | 163 +++++++++++++++++++++++++++-\n 4 files changed, 468 insertions(+), 6 deletions(-)\n\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex aef24ddae3..0f1d4181cb 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -1,19 +1,24 @@\n #include \"cache.h\"\n #include \"commit.h\"\n #include \"config.h\"\n #include \"revision.h\"\n #include \"argv-array.h\"\n #include \"list-objects.h\"\n #include \"list-objects-filter.h\"\n #include \"list-objects-filter-options.h\"\n \n+static int parse_combine_filter(\n+\tstruct list_objects_filter_options *filter_options,\n+\tconst char *arg,\n+\tstruct strbuf *errbuf);\n+\n /*\n * Parse value of the argument to the \"filter\" keyword.\n * On the command line this looks like:\n * --filter=\n * and in the pack protocol as:\n * \"filter\" SP \n *\n * The filter keyword will be used by many commands.\n * See Documentation/rev-list-options.txt for allowed values for .\n *\n@@ -28,22 +33,20 @@ static int gently_parse_list_objects_filter(\n \tstruct strbuf *errbuf)\n {\n \tconst char *v0;\n \n \tif (filter_options->choice) {\n \t\tstrbuf_addstr(\n \t\t\terrbuf, _(\"multiple filter-specs cannot be combined\"));\n \t\treturn 1;\n \t}\n \n-\tfilter_options->filter_spec = strdup(arg);\n-\n \tif (!strcmp(arg, \"blob:none\")) {\n \t\tfilter_options->choice = LOFC_BLOB_NONE;\n \t\treturn 0;\n \n \t} else if (skip_prefix(arg, \"blob:limit=\", &v0)) {\n \t\tif (git_parse_ulong(v0, &filter_options->blob_limit_value)) {\n \t\t\tfilter_options->choice = LOFC_BLOB_LIMIT;\n \t\t\treturn 0;\n \t\t}\n \n@@ -67,36 +70,155 @@ static int gently_parse_list_objects_filter(\n \t\tif (!get_oid_with_context(the_repository, v0, GET_OID_BLOB,\n \t\t\t\t\t &sparse_oid, &oc))\n \t\t\tfilter_options->sparse_oid_value = oiddup(&sparse_oid);\n \t\tfilter_options->choice = LOFC_SPARSE_OID;\n \t\treturn 0;\n \n \t} else if (skip_prefix(arg, \"sparse:path=\", &v0)) {\n \t\tfilter_options->choice = LOFC_SPARSE_PATH;\n \t\tfilter_options->sparse_path_value = strdup(v0);\n \t\treturn 0;\n+\n+\t} else if (skip_prefix(arg, \"combine:\", &v0)) {\n+\t\treturn parse_combine_filter(filter_options, v0, errbuf);\n+\n \t}\n \t/*\n \t * Please update _git_fetch() in git-completion.bash when you\n \t * add new filters\n \t */\n \n \tstrbuf_addf(errbuf, \"invalid filter-spec '%s'\", arg);\n \n \tmemset(filter_options, 0, sizeof(*filter_options));\n \treturn 1;\n }\n \n+static int url_decode(struct strbuf *s, struct strbuf *errbuf)\n+{\n+\tchar *dest = s->buf;\n+\tchar *src = s->buf;\n+\tsize_t new_len;\n+\n+\twhile (*src) {\n+\t\tif (src[0] != '%') {\n+\t\t\t*dest++ = *src++;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (hex_to_bytes((unsigned char *)dest, src + 1, 1)) {\n+\t\t\tstrbuf_addstr(errbuf,\n+\t\t\t\t \"error in filter-spec - \"\n+\t\t\t\t \"invalid hex sequence after %\");\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\tif (!*dest) {\n+\t\t\tstrbuf_addstr(errbuf,\n+\t\t\t\t \"error in filter-spec - unexpected %00\");\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\tsrc += 3;\n+\t\tdest++;\n+\t}\n+\tnew_len = dest - s->buf;\n+\tstrbuf_remove(s, new_len, s->len - new_len);\n+\n+\treturn 0;\n+}\n+\n+static const char *RESERVED_NON_WS = \"~`!@#$^&*()[]{}\\\\;'\\\",<>?\";\n+\n+static int has_reserved_character(\n+\tstruct strbuf *sub_spec, struct strbuf *errbuf)\n+{\n+\tconst char *c = sub_spec->buf;\n+\twhile (*c) {\n+\t\tif (*c <= ' ' || strchr(RESERVED_NON_WS, *c)) {\n+\t\t\tstrbuf_addf(errbuf,\n+\t\t\t\t \"must escape char in sub-filter-spec: '%c'\",\n+\t\t\t\t *c);\n+\t\t\treturn 1;\n+\t\t}\n+\t\tc++;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int parse_combine_subfilter(\n+\tstruct list_objects_filter_options *filter_options,\n+\tstruct strbuf *subspec,\n+\tstruct strbuf *errbuf)\n+{\n+\tsize_t new_index = filter_options->sub_nr++;\n+\n+\tALLOC_GROW(filter_options->sub, filter_options->sub_nr,\n+\t\t filter_options->sub_alloc);\n+\tmemset(&filter_options->sub[new_index], 0,\n+\t sizeof(*filter_options->sub));\n+\n+\treturn has_reserved_character(subspec, errbuf) ||\n+\t\turl_decode(subspec, errbuf) ||\n+\t\tgently_parse_list_objects_filter(\n+\t\t\t&filter_options->sub[new_index], subspec->buf, errbuf);\n+}\n+\n+static int parse_combine_filter(\n+\tstruct list_objects_filter_options *filter_options,\n+\tconst char *arg,\n+\tstruct strbuf *errbuf)\n+{\n+\tstruct strbuf **subspecs = strbuf_split_str(arg, '+', 0);\n+\tsize_t sub;\n+\tint result;\n+\n+\tif (!subspecs[0]) {\n+\t\tstrbuf_addf(errbuf,\n+\t\t\t _(\"expected something after combine:\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tfor (sub = 0; subspecs[sub]; sub++) {\n+\t\tif (subspecs[sub + 1]) {\n+\t\t\t/*\n+\t\t\t * This is not the last subspec. Remove trailing \"+\" so\n+\t\t\t * we can parse it.\n+\t\t\t */\n+\t\t\tsize_t last = subspecs[sub]->len - 1;\n+\t\t\tassert(subspecs[sub]->buf[last] == '+');\n+\t\t\tstrbuf_remove(subspecs[sub], last, 1);\n+\t\t}\n+\t\tresult = parse_combine_subfilter(\n+\t\t\tfilter_options, subspecs[sub], errbuf);\n+\t\tif (result)\n+\t\t\tgoto cleanup;\n+\t}\n+\n+\tfilter_options->choice = LOFC_COMBINE;\n+\n+cleanup:\n+\tstrbuf_list_free(subspecs);\n+\tif (result) {\n+\t\tlist_objects_filter_release(filter_options);\n+\t\tmemset(filter_options, 0, sizeof(*filter_options));\n+\t}\n+\treturn result;\n+}\n+\n int parse_list_objects_filter(struct list_objects_filter_options *filter_options,\n \t\t\t const char *arg)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n+\tfilter_options->filter_spec = strdup(arg);\n \tif (gently_parse_list_objects_filter(filter_options, arg, &buf))\n \t\tdie(\"%s\", buf.buf);\n \treturn 0;\n }\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct list_objects_filter_options *filter_options = opt->value;\n \n@@ -119,23 +241,30 @@ void expand_list_objects_filter_spec(\n \telse if (filter->choice == LOFC_TREE_DEPTH)\n \t\tstrbuf_addf(expanded_spec, \"tree:%lu\",\n \t\t\t filter->tree_exclude_depth);\n \telse\n \t\tstrbuf_addstr(expanded_spec, filter->filter_spec);\n }\n \n void list_objects_filter_release(\n \tstruct list_objects_filter_options *filter_options)\n {\n+\tsize_t sub;\n+\n+\tif (!filter_options)\n+\t\treturn;\n \tfree(filter_options->filter_spec);\n \tfree(filter_options->sparse_oid_value);\n \tfree(filter_options->sparse_path_value);\n+\tfor (sub = 0; sub < filter_options->sub_nr; sub++)\n+\t\tlist_objects_filter_release(&filter_options->sub[sub]);\n+\tfree(filter_options->sub);\n \tmemset(filter_options, 0, sizeof(*filter_options));\n }\n \n void partial_clone_register(\n \tconst char *remote,\n \tconst struct list_objects_filter_options *filter_options)\n {\n \t/*\n \t * Record the name of the partial clone remote in the\n \t * config and in the global variable -- the latter is\n@@ -165,15 +294,17 @@ void partial_clone_register(\n void partial_clone_get_default_filter_spec(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tstruct strbuf errbuf = STRBUF_INIT;\n \n \t/*\n \t * Parse default value, but silently ignore it if it is invalid.\n \t */\n \tif (!core_partial_clone_filter_default)\n \t\treturn;\n+\n+\tfilter_options->filter_spec = strdup(core_partial_clone_filter_default);\n \tgently_parse_list_objects_filter(filter_options,\n \t\t\t\t\t core_partial_clone_filter_default,\n \t\t\t\t\t &errbuf);\n \tstrbuf_release(&errbuf);\n }\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex e3adc78ebf..8f08ed74a1 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -7,20 +7,21 @@\n /*\n * The list of defined filters for list-objects.\n */\n enum list_objects_filter_choice {\n \tLOFC_DISABLED = 0,\n \tLOFC_BLOB_NONE,\n \tLOFC_BLOB_LIMIT,\n \tLOFC_TREE_DEPTH,\n \tLOFC_SPARSE_OID,\n \tLOFC_SPARSE_PATH,\n+\tLOFC_COMBINE,\n \tLOFC__COUNT /* must be last */\n };\n \n struct list_objects_filter_options {\n \t/*\n \t * 'filter_spec' is the raw argument value given on the command line\n \t * or protocol request. (The part after the \"--keyword=\".) For\n \t * commands that launch filtering sub-processes, or for communication\n \t * over the network, don't use this value; use the result of\n \t * expand_list_objects_filter_spec() instead.\n@@ -32,28 +33,38 @@ struct list_objects_filter_options {\n \t * the filtering algorithm to use.\n \t */\n \tenum list_objects_filter_choice choice;\n \n \t/*\n \t * Choice is LOFC_DISABLED because \"--no-filter\" was requested.\n \t */\n \tunsigned int no_filter : 1;\n \n \t/*\n-\t * Parsed values (fields) from within the filter-spec. These are\n-\t * choice-specific; not all values will be defined for any given\n-\t * choice.\n+\t * BEGIN choice-specific parsed values from within the filter-spec. Only\n+\t * some values will be defined for any given choice.\n \t */\n+\n \tstruct object_id *sparse_oid_value;\n \tchar *sparse_path_value;\n \tunsigned long blob_limit_value;\n \tunsigned long tree_exclude_depth;\n+\n+\t/* LOFC_COMBINE values */\n+\n+\t/* This array contains all the subfilters which this filter combines. */\n+\tsize_t sub_nr, sub_alloc;\n+\tstruct list_objects_filter_options *sub;\n+\n+\t/*\n+\t * END choice-specific parsed values.\n+\t */\n };\n \n /* Normalized command line arguments */\n #define CL_ARG__FILTER \"filter\"\n \n int parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg);\n \n int opt_parse_list_objects_filter(const struct option *opt,\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex 57bbf6ec1c..c8a006edf9 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -19,30 +19,45 @@\n * FILTER_SHOWN_BUT_REVISIT -- we set this bit on tree objects\n * that have been shown, but should be revisited if they appear\n * in the traversal (until we mark it SEEN). This is a way to\n * let us silently de-dup calls to show() in the caller. This\n * is subtly different from the \"revision.h:SHOWN\" and the\n * \"sha1-name.c:ONELINE_SEEN\" bits. And also different from\n * the non-de-dup usage in pack-bitmap.c\n */\n #define FILTER_SHOWN_BUT_REVISIT (1<<21)\n \n+struct subfilter {\n+\tstruct filter *filter;\n+\tstruct oidset seen;\n+\tstruct oidset omits;\n+\tstruct object_id skip_tree;\n+\tunsigned is_skipping_tree : 1;\n+};\n+\n struct filter {\n \tenum list_objects_filter_result (*filter_object_fn)(\n \t\tstruct repository *r,\n \t\tenum list_objects_filter_situation filter_situation,\n \t\tstruct object *obj,\n \t\tconst char *pathname,\n \t\tconst char *filename,\n \t\tstruct oidset *omits,\n \t\tvoid *filter_data);\n \n+\t/*\n+\t * Optional. If this function is supplied and the filter needs to\n+\t * collect omits, then this function is called once before free_fn is\n+\t * called.\n+\t */\n+\tvoid (*finalize_omits_fn)(struct oidset *omits, void *filter_data);\n+\n \tvoid (*free_fn)(void *filter_data);\n \n \tvoid *filter_data;\n \n \t/* If non-NULL, the filter collects a list of the omitted OIDs here. */\n \tstruct oidset *omits;\n };\n \n static enum list_objects_filter_result filter_blobs_none(\n \tstruct repository *r,\n@@ -482,34 +497,176 @@ static void filter_sparse_path__init(\n \n \tALLOC_GROW(d->array_frame, d->nr + 1, d->alloc);\n \td->array_frame[d->nr].defval = 0; /* default to include */\n \td->array_frame[d->nr].child_prov_omit = 0;\n \n \tfilter->filter_data = d;\n \tfilter->filter_object_fn = filter_sparse;\n \tfilter->free_fn = filter_sparse_free;\n }\n \n+/* A filter which only shows objects shown by all sub-filters. */\n+struct combine_filter_data {\n+\tstruct subfilter *sub;\n+\tsize_t nr;\n+};\n+\n+static int should_delegate(enum list_objects_filter_situation filter_situation,\n+\t\t\t struct object *obj,\n+\t\t\t struct subfilter *sub)\n+{\n+\tif (!sub->is_skipping_tree)\n+\t\treturn 1;\n+\tif (filter_situation == LOFS_END_TREE &&\n+\t\toideq(&obj->oid, &sub->skip_tree)) {\n+\t\tsub->is_skipping_tree = 0;\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+static enum list_objects_filter_result process_subfilter(\n+\tstruct repository *r,\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tstruct subfilter *sub)\n+{\n+\tenum list_objects_filter_result result;\n+\n+\t/*\n+\t * Check should_delegate before oidset_contains so that\n+\t * is_skipping_tree gets unset even when the object is marked as seen.\n+\t * As of this writing, no filter uses LOFR_MARK_SEEN on trees that also\n+\t * uses LOFR_SKIP_TREE, so the ordering is only theoretically\n+\t * important. Be cautious if you change the order of the below checks\n+\t * and more filters have been added!\n+\t */\n+\tif (!should_delegate(filter_situation, obj, sub))\n+\t\treturn LOFR_ZERO;\n+\tif (oidset_contains(&sub->seen, &obj->oid))\n+\t\treturn LOFR_ZERO;\n+\n+\tresult = list_objects_filter__filter_object(\n+\t\tr, filter_situation, obj, pathname, filename, sub->filter);\n+\n+\tif (result & LOFR_MARK_SEEN)\n+\t\toidset_insert(&sub->seen, &obj->oid);\n+\n+\tif (result & LOFR_SKIP_TREE) {\n+\t\tsub->is_skipping_tree = 1;\n+\t\tsub->skip_tree = obj->oid;\n+\t}\n+\n+\treturn result;\n+}\n+\n+static enum list_objects_filter_result filter_combine(\n+\tstruct repository *r,\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tstruct oidset *omits,\n+\tvoid *filter_data)\n+{\n+\tstruct combine_filter_data *d = filter_data;\n+\tenum list_objects_filter_result combined_result =\n+\t\tLOFR_DO_SHOW | LOFR_MARK_SEEN | LOFR_SKIP_TREE;\n+\tsize_t sub;\n+\n+\tfor (sub = 0; sub < d->nr; sub++) {\n+\t\tenum list_objects_filter_result sub_result = process_subfilter(\n+\t\t\tr, filter_situation, obj, pathname, filename,\n+\t\t\t&d->sub[sub]);\n+\t\tif (!(sub_result & LOFR_DO_SHOW))\n+\t\t\tcombined_result &= ~LOFR_DO_SHOW;\n+\t\tif (!(sub_result & LOFR_MARK_SEEN))\n+\t\t\tcombined_result &= ~LOFR_MARK_SEEN;\n+\t\tif (!d->sub[sub].is_skipping_tree)\n+\t\t\tcombined_result &= ~LOFR_SKIP_TREE;\n+\t}\n+\n+\treturn combined_result;\n+}\n+\n+static void filter_combine__free(void *filter_data)\n+{\n+\tstruct combine_filter_data *d = filter_data;\n+\tsize_t sub;\n+\tfor (sub = 0; sub < d->nr; sub++) {\n+\t\tlist_objects_filter__free(d->sub[sub].filter);\n+\t\toidset_clear(&d->sub[sub].seen);\n+\t\tif (d->sub[sub].omits.set.size)\n+\t\t\tBUG(\"expected oidset to be cleared already\");\n+\t}\n+\tfree(d->sub);\n+}\n+\n+static void add_all(struct oidset *dest, struct oidset *src) {\n+\tstruct oidset_iter iter;\n+\tstruct object_id *src_oid;\n+\n+\toidset_iter_init(src, &iter);\n+\twhile ((src_oid = oidset_iter_next(&iter)) != NULL)\n+\t\toidset_insert(dest, src_oid);\n+}\n+\n+static void filter_combine__finalize_omits(\n+\tstruct oidset *omits,\n+\tvoid *filter_data)\n+{\n+\tstruct combine_filter_data *d = filter_data;\n+\tsize_t sub;\n+\n+\tfor (sub = 0; sub < d->nr; sub++) {\n+\t\tadd_all(omits, &d->sub[sub].omits);\n+\t\toidset_clear(&d->sub[sub].omits);\n+\t}\n+}\n+\n+static void filter_combine__init(\n+\tstruct list_objects_filter_options *filter_options,\n+\tstruct filter* filter)\n+{\n+\tstruct combine_filter_data *d = xcalloc(1, sizeof(*d));\n+\tsize_t sub;\n+\n+\td->nr = filter_options->sub_nr;\n+\td->sub = xcalloc(d->nr, sizeof(*d->sub));\n+\tfor (sub = 0; sub < d->nr; sub++)\n+\t\td->sub[sub].filter = list_objects_filter__init(\n+\t\t\tfilter->omits ? &d->sub[sub].omits : NULL,\n+\t\t\t&filter_options->sub[sub]);\n+\n+\tfilter->filter_data = d;\n+\tfilter->filter_object_fn = filter_combine;\n+\tfilter->free_fn = filter_combine__free;\n+\tfilter->finalize_omits_fn = filter_combine__finalize_omits;\n+}\n+\n typedef void (*filter_init_fn)(\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter);\n \n /*\n * Must match \"enum list_objects_filter_choice\".\n */\n static filter_init_fn s_filters[] = {\n \tNULL,\n \tfilter_blobs_none__init,\n \tfilter_blobs_limit__init,\n \tfilter_trees_depth__init,\n \tfilter_sparse_oid__init,\n \tfilter_sparse_path__init,\n+\tfilter_combine__init,\n };\n \n struct filter *list_objects_filter__init(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options)\n {\n \tstruct filter *filter;\n \tfilter_init_fn init_fn;\n \n \tassert((sizeof(s_filters) / sizeof(s_filters[0])) == LOFC__COUNT);\n@@ -547,13 +704,15 @@ enum list_objects_filter_result list_objects_filter__filter_object(\n \t */\n \tif (filter_situation == LOFS_END_TREE)\n \t\treturn 0;\n \treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n }\n \n void list_objects_filter__free(struct filter *filter)\n {\n \tif (!filter)\n \t\treturn;\n+\tif (filter->finalize_omits_fn && filter->omits)\n+\t\tfilter->finalize_omits_fn(filter->omits, filter->filter_data);\n \tfilter->free_fn(filter->filter_data);\n \tfree(filter);\n }\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex 9c11427719..c36199457d 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -284,21 +284,33 @@ test_expect_success 'verify tree:0 includes trees in \"filtered\" output' '\n # Make sure tree:0 does not iterate through any trees.\n \n test_expect_success 'verify skipping tree iteration when not collecting omits' '\n \tGIT_TRACE=1 git -C r3 rev-list \\\n \t\t--objects --filter=tree:0 HEAD 2>filter_trace &&\n \tgrep \"Skipping contents of tree [.][.][.]\" filter_trace >actual &&\n \t# One line for each commit traversed.\n \ttest_line_count = 2 actual &&\n \n \t# Make sure no other trees were considered besides the root.\n-\t! grep \"Skipping contents of tree [^.]\" filter_trace\n+\t! grep \"Skipping contents of tree [^.]\" filter_trace &&\n+\n+\t# Try this again with \"combine:\". If both sub-filters are skipping\n+\t# trees, the composite filter should also skip trees. This is not\n+\t# important unless the user does combine:tree:X+tree:Y or another filter\n+\t# besides \"tree:\" is implemented in the future which can skip trees.\n+\tGIT_TRACE=1 git -C r3 rev-list \\\n+\t\t--objects --filter=combine:tree:1+tree:3 HEAD 2>filter_trace &&\n+\n+\t# Only skip the dir1/ tree, which is shared between the two commits.\n+\tgrep \"Skipping contents of tree \" filter_trace >actual &&\n+\ttest_write_lines \"Skipping contents of tree dir1/...\" >expected &&\n+\ttest_cmp expected actual\n '\n \n # Test tree:# filters.\n \n expect_has () {\n \tcommit=$1 &&\n \tname=$2 &&\n \n \thash=$(git -C r3 rev-parse $commit:$name) &&\n \tgrep \"^$hash $name$\" actual\n@@ -336,20 +348,138 @@ test_expect_success 'verify tree:3 includes everything expected' '\n \texpect_has HEAD dir1/sparse1 &&\n \texpect_has HEAD dir1/sparse2 &&\n \texpect_has HEAD pattern &&\n \texpect_has HEAD sparse1 &&\n \texpect_has HEAD sparse2 &&\n \n \t# There are also 2 commit objects\n \ttest_line_count = 10 actual\n '\n \n+test_expect_success 'combine:... for a simple combination' '\n+\tgit -C r3 rev-list --objects --filter=combine:tree:2+blob:none HEAD \\\n+\t\t>actual &&\n+\n+\texpect_has HEAD \"\" &&\n+\texpect_has HEAD~1 \"\" &&\n+\texpect_has HEAD dir1 &&\n+\n+\t# There are also 2 commit objects\n+\ttest_line_count = 5 actual\n+'\n+\n+test_expect_success 'combine:... with URL encoding' '\n+\tgit -C r3 rev-list --objects \\\n+\t\t--filter=combine:tree%3a2+blob:%6Eon%65 HEAD >actual &&\n+\n+\texpect_has HEAD \"\" &&\n+\texpect_has HEAD~1 \"\" &&\n+\texpect_has HEAD dir1 &&\n+\n+\t# There are also 2 commit objects\n+\ttest_line_count = 5 actual\n+'\n+\n+expect_invalid_filter_spec () {\n+\tspec=\"$1\" &&\n+\terr=\"$2\" &&\n+\n+\ttest_must_fail git -C r3 rev-list --objects --filter=\"$spec\" HEAD \\\n+\t\t>actual 2>actual_stderr &&\n+\ttest_must_be_empty actual &&\n+\ttest_i18ngrep \"$err\" actual_stderr\n+}\n+\n+test_expect_success 'combine:... while URL-encoding things that should not be' '\n+\texpect_invalid_filter_spec combine%3Atree:2+blob:none \\\n+\t\t\"invalid filter-spec\"\n+'\n+\n+test_expect_success 'combine: with nothing after the :' '\n+\texpect_invalid_filter_spec combine: \"expected something after combine:\"\n+'\n+\n+test_expect_success 'parse error in first sub-filter in combine:' '\n+\texpect_invalid_filter_spec combine:tree:asdf+blob:none \\\n+\t\t\"expected .tree:.\"\n+'\n+\n+test_expect_success 'combine:... with invalid URL-encoded sequences' '\n+\t# Not enough hex chars\n+\texpect_invalid_filter_spec combine:tree:2+blob:non%a \\\n+\t\t\"error in filter-spec - invalid hex sequence after %\" &&\n+\t# Non-hex digit after %\n+\texpect_invalid_filter_spec combine:tree:2+blob%G5none \\\n+\t\t\"error in filter-spec - invalid hex sequence after %\" &&\n+\t# Null byte encoded by %\n+\texpect_invalid_filter_spec combine:tree:2+blob%00none \\\n+\t\t\"error in filter-spec - unexpected %00\"\n+'\n+\n+test_expect_success 'combine:... with non-encoded reserved chars' '\n+\texpect_invalid_filter_spec combine:tree:2+sparse:@xyz \\\n+\t\t\"must escape char in sub-filter-spec: .@.\" &&\n+\texpect_invalid_filter_spec combine:tree:2+sparse:\\` \\\n+\t\t\"must escape char in sub-filter-spec: .\\`.\" &&\n+\texpect_invalid_filter_spec combine:tree:2+sparse:~abc \\\n+\t\t\"must escape char in sub-filter-spec: .\\~.\"\n+'\n+\n+test_expect_success 'validate err msg for \"combine:+\"' '\n+\texpect_invalid_filter_spec combine:tree:2+ \"expected .tree:.\"\n+'\n+\n+test_expect_success 'combine:... with edge-case hex digits: Ff Aa 0 9' '\n+\tgit -C r3 rev-list --objects --filter=\"combine:tree:2+bl%6Fb:n%6fne\" \\\n+\t\tHEAD >actual &&\n+\ttest_line_count = 5 actual &&\n+\tgit -C r3 rev-list --objects --filter=\"combine:tree%3A2+blob%3anone\" \\\n+\t\tHEAD >actual &&\n+\ttest_line_count = 5 actual &&\n+\tgit -C r3 rev-list --objects --filter=\"combine:tree:%30\" HEAD >actual &&\n+\ttest_line_count = 2 actual &&\n+\tgit -C r3 rev-list --objects --filter=\"combine:tree:%39+blob:none\" \\\n+\t\tHEAD >actual &&\n+\ttest_line_count = 5 actual\n+'\n+\n+test_expect_success 'add a sparse pattern blob whose path has reserved chars' '\n+\tcp r3/pattern r3/pattern1+renamed% &&\n+\tgit -C r3 add pattern1+renamed% &&\n+\tgit -C r3 commit -m \"add sparse pattern file with reserved chars\"\n+'\n+\n+test_expect_success 'combine:... with more than two sub-filters' '\n+\tgit -C r3 rev-list --objects \\\n+\t\t--filter=combine:tree:3+blob:limit=40+sparse:oid=master:pattern \\\n+\t\tHEAD >actual &&\n+\n+\texpect_has HEAD \"\" &&\n+\texpect_has HEAD~1 \"\" &&\n+\texpect_has HEAD~2 \"\" &&\n+\texpect_has HEAD dir1 &&\n+\texpect_has HEAD dir1/sparse1 &&\n+\texpect_has HEAD dir1/sparse2 &&\n+\n+\t# Should also have 3 commits\n+\ttest_line_count = 9 actual &&\n+\n+\t# Try again, this time making sure the last sub-filter is only\n+\t# URL-decoded once.\n+\tcp actual expect &&\n+\n+\tgit -C r3 rev-list --objects \\\n+\t\t--filter=combine:tree:3+blob:limit=40+sparse:oid=master:pattern1%2brenamed%25 \\\n+\t\tHEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n # Test provisional omit collection logic with a repo that has objects appearing\n # at multiple depths - first deeper than the filter's threshold, then shallow.\n \n test_expect_success 'setup r4' '\n \tgit init r4 &&\n \n \techo foo > r4/foo &&\n \tmkdir r4/subdir &&\n \techo bar > r4/subdir/bar &&\n \n@@ -379,20 +509,51 @@ test_expect_success 'test tree:# filter provisional omit for blob and tree' '\n \n test_expect_success 'verify skipping tree iteration when collecting omits' '\n \tGIT_TRACE=1 git -C r4 rev-list --filter-print-omitted \\\n \t\t--objects --filter=tree:0 HEAD 2>filter_trace &&\n \tgrep \"^Skipping contents of tree \" filter_trace >actual &&\n \n \techo \"Skipping contents of tree subdir/...\" >expect &&\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'setup r5' '\n+\tgit init r5 &&\n+\tmkdir -p r5/subdir &&\n+\n+\techo 1 >r5/short-root &&\n+\techo 12345 >r5/long-root &&\n+\techo a >r5/subdir/short-subdir &&\n+\techo abcde >r5/subdir/long-subdir &&\n+\n+\tgit -C r5 add short-root long-root subdir &&\n+\tgit -C r5 commit -m \"commit msg\"\n+'\n+\n+test_expect_success 'verify collecting omits in combined: filter' '\n+\t# Note that this test guards against the naive implementation of simply\n+\t# giving both filters the same \"omits\" set and expecting it to\n+\t# automatically merge them.\n+\tgit -C r5 rev-list --objects --quiet --filter-print-omitted \\\n+\t\t--filter=combine:tree:2+blob:limit=3 HEAD >actual &&\n+\n+\t# Expect 0 trees/commits, 3 blobs omitted (all blobs except short-root)\n+\tomitted_1=$(echo 12345 | git hash-object --stdin) &&\n+\tomitted_2=$(echo a | git hash-object --stdin) &&\n+\tomitted_3=$(echo abcde | git hash-object --stdin) &&\n+\n+\tgrep ~$omitted_1 actual &&\n+\tgrep ~$omitted_2 actual &&\n+\tgrep ~$omitted_3 actual &&\n+\ttest_line_count = 3 actual\n+'\n+\n # Test tree: where a tree is iterated to twice - once where a subentry is\n # too deep to be included, and again where the blob inside it is shallow enough\n # to be included. This makes sure we don't use LOFR_MARK_SEEN incorrectly (we\n # can't use it because a tree can be iterated over again at a lower depth).\n \n test_expect_success 'tree: where we iterate over tree at two levels' '\n \tgit init r5 &&\n \n \tmkdir -p r5/a/subdir/b &&\n \techo foo > r5/a/subdir/b/foo &&\n-- \n2.17.1\n\n"},{"id":"376521","messageId":"20190601003603.90794-6-matvore@google.com","threadId":"51217","inReplyTo":"20190601003603.90794-1-matvore@google.com","subject":"[PATCH v2 5/9] list-objects-filter-options: move error check up","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-01T00:35:59Z","receivedAt":"2019-06-01T00:36:29Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Move the check that filter_options->choice is set to higher in the call\nstack. This can only be set when the gentle parse function is called\nfrom one of the two call sites.\n\nThis is important because in an upcoming patch this may or may not be an\nerror, and whether it is an error is only known to the\nparse_list_objects_filter function.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects-filter-options.c | 9 ++++-----\n 1 file changed, 4 insertions(+), 5 deletions(-)\n\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex 0f1d4181cb..e8132b811e 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -27,25 +27,22 @@ static int parse_combine_filter(\n * expand_list_objects_filter_spec() first). We also \"intern\" the arg for the\n * convenience of the current command.\n */\n static int gently_parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg,\n \tstruct strbuf *errbuf)\n {\n \tconst char *v0;\n \n-\tif (filter_options->choice) {\n-\t\tstrbuf_addstr(\n-\t\t\terrbuf, _(\"multiple filter-specs cannot be combined\"));\n-\t\treturn 1;\n-\t}\n+\tif (filter_options->choice)\n+\t\tBUG(\"filter_options already populated\");\n \n \tif (!strcmp(arg, \"blob:none\")) {\n \t\tfilter_options->choice = LOFC_BLOB_NONE;\n \t\treturn 0;\n \n \t} else if (skip_prefix(arg, \"blob:limit=\", &v0)) {\n \t\tif (git_parse_ulong(v0, &filter_options->blob_limit_value)) {\n \t\t\tfilter_options->choice = LOFC_BLOB_LIMIT;\n \t\t\treturn 0;\n \t\t}\n@@ -204,20 +201,22 @@ cleanup:\n \t\tlist_objects_filter_release(filter_options);\n \t\tmemset(filter_options, 0, sizeof(*filter_options));\n \t}\n \treturn result;\n }\n \n int parse_list_objects_filter(struct list_objects_filter_options *filter_options,\n \t\t\t const char *arg)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n+\tif (filter_options->choice)\n+\t\tdie(_(\"multiple filter-specs cannot be combined\"));\n \tfilter_options->filter_spec = strdup(arg);\n \tif (gently_parse_list_objects_filter(filter_options, arg, &buf))\n \t\tdie(\"%s\", buf.buf);\n \treturn 0;\n }\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct list_objects_filter_options *filter_options = opt->value;\n-- \n2.17.1\n\n"},{"id":"376522","messageId":"20190601003603.90794-7-matvore@google.com","threadId":"51217","inReplyTo":"20190601003603.90794-1-matvore@google.com","subject":"[PATCH v2 6/9] list-objects-filter-options: make filter_spec a strbuf","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-01T00:36:00Z","receivedAt":"2019-06-01T00:36:32Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Make the filter_spec string a strbuf rather than a raw C string. A\nfuture patch will need to grow this string dynamically.\n\nSigned-off-by: Matthew DeVore \n---\n builtin/rev-list.c | 2 +-\n list-objects-filter-options.c | 16 ++++++++++------\n list-objects-filter-options.h | 2 +-\n upload-pack.c | 2 +-\n 4 files changed, 13 insertions(+), 9 deletions(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 9f31837d30..7137f13a74 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -460,21 +460,21 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\tcontinue;\n \t\t}\n \n \t\tif (skip_prefix(arg, (\"--\" CL_ARG__FILTER \"=\"), &arg)) {\n \t\t\tparse_list_objects_filter(&filter_options, arg);\n \t\t\tif (filter_options.choice && !revs.blob_objects)\n \t\t\t\tdie(_(\"object filtering requires --objects\"));\n \t\t\tif (filter_options.choice == LOFC_SPARSE_OID &&\n \t\t\t !filter_options.sparse_oid_value)\n \t\t\t\tdie(_(\"invalid sparse value '%s'\"),\n-\t\t\t\t filter_options.filter_spec);\n+\t\t\t\t filter_options.filter_spec.buf);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!strcmp(arg, (\"--no-\" CL_ARG__FILTER))) {\n \t\t\tlist_objects_filter_set_no_filter(&filter_options);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!strcmp(arg, \"--filter-print-omitted\")) {\n \t\t\targ_print_omitted = 1;\n \t\t\tcontinue;\n \t\t}\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex e8132b811e..5687425847 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -203,21 +203,22 @@ cleanup:\n \t}\n \treturn result;\n }\n \n int parse_list_objects_filter(struct list_objects_filter_options *filter_options,\n \t\t\t const char *arg)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tif (filter_options->choice)\n \t\tdie(_(\"multiple filter-specs cannot be combined\"));\n-\tfilter_options->filter_spec = strdup(arg);\n+\tstrbuf_init(&filter_options->filter_spec, 0);\n+\tstrbuf_addstr(&filter_options->filter_spec, arg);\n \tif (gently_parse_list_objects_filter(filter_options, arg, &buf))\n \t\tdie(\"%s\", buf.buf);\n \treturn 0;\n }\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct list_objects_filter_options *filter_options = opt->value;\n \n@@ -226,39 +227,39 @@ int opt_parse_list_objects_filter(const struct option *opt,\n \t\treturn 0;\n \t}\n \n \treturn parse_list_objects_filter(filter_options, arg);\n }\n \n void expand_list_objects_filter_spec(\n \tconst struct list_objects_filter_options *filter,\n \tstruct strbuf *expanded_spec)\n {\n-\tstrbuf_init(expanded_spec, strlen(filter->filter_spec));\n+\tstrbuf_init(expanded_spec, 0);\n \tif (filter->choice == LOFC_BLOB_LIMIT)\n \t\tstrbuf_addf(expanded_spec, \"blob:limit=%lu\",\n \t\t\t filter->blob_limit_value);\n \telse if (filter->choice == LOFC_TREE_DEPTH)\n \t\tstrbuf_addf(expanded_spec, \"tree:%lu\",\n \t\t\t filter->tree_exclude_depth);\n \telse\n-\t\tstrbuf_addstr(expanded_spec, filter->filter_spec);\n+\t\tstrbuf_addstr(expanded_spec, filter->filter_spec.buf);\n }\n \n void list_objects_filter_release(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tsize_t sub;\n \n \tif (!filter_options)\n \t\treturn;\n-\tfree(filter_options->filter_spec);\n+\tstrbuf_release(&filter_options->filter_spec);\n \tfree(filter_options->sparse_oid_value);\n \tfree(filter_options->sparse_path_value);\n \tfor (sub = 0; sub < filter_options->sub_nr; sub++)\n \t\tlist_objects_filter_release(&filter_options->sub[sub]);\n \tfree(filter_options->sub);\n \tmemset(filter_options, 0, sizeof(*filter_options));\n }\n \n void partial_clone_register(\n \tconst char *remote,\n@@ -278,32 +279,35 @@ void partial_clone_register(\n \tgit_config_set(\"core.repositoryformatversion\", \"1\");\n \tgit_config_set(\"extensions.partialclone\", remote);\n \n \trepository_format_partial_clone = xstrdup(remote);\n \n \t/*\n \t * Record the initial filter-spec in the config as\n \t * the default for subsequent fetches from this remote.\n \t */\n \tcore_partial_clone_filter_default =\n-\t\txstrdup(filter_options->filter_spec);\n+\t\txstrdup(filter_options->filter_spec.buf);\n \tgit_config_set(\"core.partialclonefilter\",\n \t\t core_partial_clone_filter_default);\n }\n \n void partial_clone_get_default_filter_spec(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tstruct strbuf errbuf = STRBUF_INIT;\n \n \t/*\n \t * Parse default value, but silently ignore it if it is invalid.\n \t */\n \tif (!core_partial_clone_filter_default)\n \t\treturn;\n \n-\tfilter_options->filter_spec = strdup(core_partial_clone_filter_default);\n+\tif (!filter_options->filter_spec.buf)\n+\t\tstrbuf_init(&filter_options->filter_spec, 0);\n+\tstrbuf_addstr(&filter_options->filter_spec,\n+\t\t core_partial_clone_filter_default);\n \tgently_parse_list_objects_filter(filter_options,\n \t\t\t\t\t core_partial_clone_filter_default,\n \t\t\t\t\t &errbuf);\n \tstrbuf_release(&errbuf);\n }\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 8f08ed74a1..e1e23fd191 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -19,21 +19,21 @@ enum list_objects_filter_choice {\n };\n \n struct list_objects_filter_options {\n \t/*\n \t * 'filter_spec' is the raw argument value given on the command line\n \t * or protocol request. (The part after the \"--keyword=\".) For\n \t * commands that launch filtering sub-processes, or for communication\n \t * over the network, don't use this value; use the result of\n \t * expand_list_objects_filter_spec() instead.\n \t */\n-\tchar *filter_spec;\n+\tstruct strbuf filter_spec;\n \n \t/*\n \t * 'choice' is determined by parsing the filter-spec. This indicates\n \t * the filtering algorithm to use.\n \t */\n \tenum list_objects_filter_choice choice;\n \n \t/*\n \t * Choice is LOFC_DISABLED because \"--no-filter\" was requested.\n \t */\ndiff --git a/upload-pack.c b/upload-pack.c\nindex d2ea5eb20d..2cdd499f28 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -133,21 +133,21 @@ static void create_pack_file(const struct object_array *have_obj,\n \n \targv_array_push(&pack_objects.args, \"--stdout\");\n \tif (shallow_nr)\n \t\targv_array_push(&pack_objects.args, \"--shallow\");\n \tif (!no_progress)\n \t\targv_array_push(&pack_objects.args, \"--progress\");\n \tif (use_ofs_delta)\n \t\targv_array_push(&pack_objects.args, \"--delta-base-offset\");\n \tif (use_include_tag)\n \t\targv_array_push(&pack_objects.args, \"--include-tag\");\n-\tif (filter_options.filter_spec) {\n+\tif (filter_options.filter_spec.len) {\n \t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n \t\texpand_list_objects_filter_spec(&filter_options,\n \t\t\t\t\t\t&expanded_filter_spec);\n \t\tif (pack_objects.use_shell) {\n \t\t\tstruct strbuf buf = STRBUF_INIT;\n \t\t\tsq_quote_buf(&buf, expanded_filter_spec.buf);\n \t\t\targv_array_pushf(&pack_objects.args, \"--filter=%s\", buf.buf);\n \t\t\tstrbuf_release(&buf);\n \t\t} else {\n \t\t\targv_array_pushf(&pack_objects.args, \"--filter=%s\",\n-- \n2.17.1\n\n"},{"id":"376523","messageId":"20190601003603.90794-8-matvore@google.com","threadId":"51217","inReplyTo":"20190601003603.90794-1-matvore@google.com","subject":"[PATCH v2 7/9] list-objects-filter-options: allow mult. --filter","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-01T00:36:01Z","receivedAt":"2019-06-01T00:36:35Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Allow combining of multiple filters by simply repeating the --filter\nflag. Before this patch, the user had to combine them in a single flag\nsomewhat awkwardly (e.g. --filter=combine:FOO+BAR), including\nURL-encoding the individual filters.\n\nTo make this work, in the --filter flag parsing callback, rather than\nerror out when we detect that the filter_options struct is already\npopulated, we modify it in-place to contain the added sub-filter. The\nexisting sub-filter becomes the lhs of the combined filter, and the\nnext sub-filter becomes the rhs. We also have to URL-encode the LHS and\nRHS sub-filters.\n\nWe can simplify the operation if the LHS is already a combine: filter.\nIn that case, we just append the URL-encoded RHS sub-filter to the LHS\nspec to get the new spec.\n\nHelped-by: Emily Shaffer \nHelped-by: Jeff Hostetler \nHelped-by: Junio C Hamano \nSigned-off-by: Matthew DeVore \n---\n Documentation/rev-list-options.txt | 16 +++++\n list-objects-filter-options.c | 90 ++++++++++++++++++++++++++---\n list-objects-filter-options.h | 11 ++++\n t/t5616-partial-clone.sh | 19 ++++++\n t/t6112-rev-list-filters-objects.sh | 44 ++++++++++++--\n transport.c | 1 +\n upload-pack.c | 2 +\n 7 files changed, 171 insertions(+), 12 deletions(-)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex ddbc1de43f..7b4116f279 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -730,20 +730,36 @@ specification contained in .\n +\n The form '--filter=tree:' omits all blobs and trees whose depth\n from the root tree is >= (minimum depth if an object is located\n at multiple depths in the commits traversed). =0 will not include\n any trees or blobs unless included explicitly in the command-line (or\n standard input when --stdin is used). =1 will include only the\n tree and blobs which are referenced directly by a commit reachable from\n or an explicitly-given object. =2 is like =1\n while also including trees and blobs one more level removed from an\n explicitly-given commit or tree.\n++\n+Multiple '--filter=' flags can be specified to combine filters. Only\n+objects which are accepted by every filter are included.\n++\n+The form '--filter=combine:++...' can also be\n+used to combined several filters, but this is harder than just repeating\n+the '--filter' flag and is usually not necessary. Filters are joined by\n+'{plus}' and individual filters are %-encoded (i.e. URL-encoded).\n+Besides the '{plus}' and '%' characters, the following characters are\n+reserved and also must be encoded: `~!@#$^&*()[]{}\\;\",<>?`+'`+\n+as well as all characters with ASCII code <= `0x20`, which includes\n+space and newline.\n++\n+Other arbitrary characters can also be encoded. For instance,\n+'combine:tree:3+blob:none' and 'combine:tree%3A3+blob%3Anone' are\n+equivalent.\n \n --no-filter::\n \tTurn off any previous `--filter=` argument.\n \n --filter-print-omitted::\n \tOnly useful with `--filter=`; prints a list of the objects omitted\n \tby the filter. Object IDs are prefixed with a ``~'' character.\n \n --missing=::\n \tA debug option to help with future \"partial clone\" development.\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex 5687425847..5e98e4a309 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -1,18 +1,19 @@\n #include \"cache.h\"\n #include \"commit.h\"\n #include \"config.h\"\n #include \"revision.h\"\n #include \"argv-array.h\"\n #include \"list-objects.h\"\n #include \"list-objects-filter.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"trace.h\"\n \n static int parse_combine_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg,\n \tstruct strbuf *errbuf);\n \n /*\n * Parse value of the argument to the \"filter\" keyword.\n * On the command line this looks like:\n * --filter=\n@@ -197,30 +198,105 @@ static int parse_combine_filter(\n \n cleanup:\n \tstrbuf_list_free(subspecs);\n \tif (result) {\n \t\tlist_objects_filter_release(filter_options);\n \t\tmemset(filter_options, 0, sizeof(*filter_options));\n \t}\n \treturn result;\n }\n \n-int parse_list_objects_filter(struct list_objects_filter_options *filter_options,\n-\t\t\t const char *arg)\n+static void add_url_encoded(struct strbuf *dest, const char *s)\n+{\n+\twhile (*s) {\n+\t\tif (*s <= ' ' || strchr(RESERVED_NON_WS, *s) ||\n+\t\t\t*s == '%' || *s == '+')\n+\t\t\tstrbuf_addf(dest, \"%%%02X\", (int)*s);\n+\t\telse\n+\t\t\tstrbuf_addf(dest, \"%c\", *s);\n+\t\ts++;\n+\t}\n+}\n+\n+/*\n+ * Changes filter_options into an equivalent LOFC_COMBINE filter options\n+ * instance. Does not do anything if filter_options is already LOFC_COMBINE.\n+ */\n+static void transform_to_combine_type(\n+\tstruct list_objects_filter_options *filter_options)\n+{\n+\tassert(filter_options->choice);\n+\tif (filter_options->choice == LOFC_COMBINE)\n+\t\treturn;\n+\t{\n+\t\tconst int initial_sub_alloc = 2;\n+\t\tstruct list_objects_filter_options *sub_array =\n+\t\t\txcalloc(initial_sub_alloc, sizeof(*sub_array));\n+\t\tsub_array[0] = *filter_options;\n+\t\tmemset(filter_options, 0, sizeof(*filter_options));\n+\t\tfilter_options->sub = sub_array;\n+\t\tfilter_options->sub_alloc = initial_sub_alloc;\n+\t}\n+\tfilter_options->sub_nr = 1;\n+\tfilter_options->choice = LOFC_COMBINE;\n+\tstrbuf_init(&filter_options->filter_spec, 0);\n+\tstrbuf_addstr(&filter_options->filter_spec, \"combine:\");\n+\tadd_url_encoded(&filter_options->filter_spec,\n+\t\t\tfilter_options->sub[0].filter_spec.buf);\n+\t/*\n+\t * We don't need the filter_spec strings for subfilter specs, only the\n+\t * top level.\n+\t */\n+\tstrbuf_release(&filter_options->sub[0].filter_spec);\n+}\n+\n+void list_objects_filter_die_if_populated(\n+\tstruct list_objects_filter_options *filter_options)\n {\n-\tstruct strbuf buf = STRBUF_INIT;\n \tif (filter_options->choice)\n \t\tdie(_(\"multiple filter-specs cannot be combined\"));\n-\tstrbuf_init(&filter_options->filter_spec, 0);\n-\tstrbuf_addstr(&filter_options->filter_spec, arg);\n-\tif (gently_parse_list_objects_filter(filter_options, arg, &buf))\n-\t\tdie(\"%s\", buf.buf);\n+}\n+\n+int parse_list_objects_filter(\n+\tstruct list_objects_filter_options *filter_options,\n+\tconst char *arg)\n+{\n+\tstruct strbuf errbuf = STRBUF_INIT;\n+\tint parse_error;\n+\n+\tif (!filter_options->choice) {\n+\t\tstrbuf_init(&filter_options->filter_spec, 0);\n+\t\tstrbuf_addstr(&filter_options->filter_spec, arg);\n+\n+\t\tparse_error = gently_parse_list_objects_filter(\n+\t\t\tfilter_options, arg, &errbuf);\n+\t} else {\n+\t\t/*\n+\t\t * Make filter_options an LOFC_COMBINE spec so we can trivially\n+\t\t * add subspecs to it.\n+\t\t */\n+\t\ttransform_to_combine_type(filter_options);\n+\n+\t\tstrbuf_addstr(&filter_options->filter_spec, \"+\");\n+\t\tadd_url_encoded(&filter_options->filter_spec, arg);\n+\t\ttrace_printf(\"Generated composite filter-spec: %s\\n\",\n+\t\t\t filter_options->filter_spec.buf);\n+\t\tALLOC_GROW(filter_options->sub, filter_options->sub_nr + 1,\n+\t\t\t filter_options->sub_alloc);\n+\t\tfilter_options = &filter_options->sub[filter_options->sub_nr++];\n+\t\tmemset(filter_options, 0, sizeof(*filter_options));\n+\n+\t\tparse_error = gently_parse_list_objects_filter(\n+\t\t\tfilter_options, arg, &errbuf);\n+\t}\n+\tif (parse_error)\n+\t\tdie(\"%s\", errbuf.buf);\n \treturn 0;\n }\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct list_objects_filter_options *filter_options = opt->value;\n \n \tif (unset || !arg) {\n \t\tlist_objects_filter_set_no_filter(filter_options);\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex e1e23fd191..f8c8a624e4 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -56,20 +56,31 @@ struct list_objects_filter_options {\n \tstruct list_objects_filter_options *sub;\n \n \t/*\n \t * END choice-specific parsed values.\n \t */\n };\n \n /* Normalized command line arguments */\n #define CL_ARG__FILTER \"filter\"\n \n+void list_objects_filter_die_if_populated(\n+\tstruct list_objects_filter_options *filter_options);\n+\n+/*\n+ * Parses the filter spec string given by arg and either (1) simply places the\n+ * result in filter_options if it is not yet populated or (2) combines it with\n+ * the filter already in filter_options if it is already populated. In the case\n+ * of (2), the filter specs are combined as if specified with 'combine:'.\n+ *\n+ * Dies and prints a user-facing message if an error occurs.\n+ */\n int parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg);\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset);\n \n #define OPT_PARSE_LIST_OBJECTS_FILTER(fo) \\\n \t{ OPTION_CALLBACK, 0, CL_ARG__FILTER, fo, N_(\"args\"), \\\n \t N_(\"object filtering\"), 0, \\\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex 9a8f9886b3..11536f4028 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -201,20 +201,39 @@ test_expect_success 'use fsck before and after manually fetching a missing subtr\n \ttest_line_count = 70 fetched_objects &&\n \n \tawk -f print_1.awk fetched_objects |\n \txargs -n1 git -C dst cat-file -t >fetched_types &&\n \n \tsort -u fetched_types >unique_types.observed &&\n \ttest_write_lines blob commit tree >unique_types.expected &&\n \ttest_cmp unique_types.expected unique_types.observed\n '\n \n+test_expect_success 'implicitly construct combine: filter with repeated flags' '\n+\tGIT_TRACE=$(pwd)/trace git clone --bare \\\n+\t\t--filter=blob:none --filter=tree:1 \\\n+\t\t\"file://$(pwd)/srv.bare\" pc2 &&\n+\tgrep \"trace:.* git pack-objects .*--filter=combine:blob:none+tree:1\" \\\n+\t\ttrace &&\n+\tgit -C pc2 rev-list --objects --missing=allow-any HEAD >objects &&\n+\n+\t# We should have gotten some root trees.\n+\tgrep \" $\" objects &&\n+\t# Should not have gotten any non-root trees or blobs.\n+\t! grep \" .\" objects &&\n+\n+\txargs -n 1 git -C pc2 cat-file -t types &&\n+\tsort -u types >unique_types.actual &&\n+\ttest_write_lines commit tree >unique_types.expected &&\n+\ttest_cmp unique_types.expected unique_types.actual\n+'\n+\n test_expect_success 'partial clone fetches blobs pointed to by refs even if normally filtered out' '\n \trm -rf src dst &&\n \tgit init src &&\n \ttest_commit -C src x &&\n \ttest_config -C src uploadpack.allowfilter 1 &&\n \ttest_config -C src uploadpack.allowanysha1inwant 1 &&\n \n \t# Create a tag pointing to a blob.\n \tBLOB=$(echo blob-contents | git -C src hash-object --stdin -w) &&\n \tgit -C src tag myblob \"$BLOB\" &&\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex c36199457d..7fb5e50cde 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -357,21 +357,30 @@ test_expect_success 'verify tree:3 includes everything expected' '\n \n test_expect_success 'combine:... for a simple combination' '\n \tgit -C r3 rev-list --objects --filter=combine:tree:2+blob:none HEAD \\\n \t\t>actual &&\n \n \texpect_has HEAD \"\" &&\n \texpect_has HEAD~1 \"\" &&\n \texpect_has HEAD dir1 &&\n \n \t# There are also 2 commit objects\n-\ttest_line_count = 5 actual\n+\ttest_line_count = 5 actual &&\n+\n+\tcp actual expected &&\n+\n+\t# Try again using repeated --filter - this is equivalent to a manual\n+\t# combine with \"combine:...+...\"\n+\tgit -C r3 rev-list --objects --filter=combine:tree:2 \\\n+\t\t--filter=blob:none HEAD >actual &&\n+\n+\ttest_cmp expected actual\n '\n \n test_expect_success 'combine:... with URL encoding' '\n \tgit -C r3 rev-list --objects \\\n \t\t--filter=combine:tree%3a2+blob:%6Eon%65 HEAD >actual &&\n \n \texpect_has HEAD \"\" &&\n \texpect_has HEAD~1 \"\" &&\n \texpect_has HEAD dir1 &&\n \n@@ -435,24 +444,26 @@ test_expect_success 'combine:... with edge-case hex digits: Ff Aa 0 9' '\n \tgit -C r3 rev-list --objects --filter=\"combine:tree%3A2+blob%3anone\" \\\n \t\tHEAD >actual &&\n \ttest_line_count = 5 actual &&\n \tgit -C r3 rev-list --objects --filter=\"combine:tree:%30\" HEAD >actual &&\n \ttest_line_count = 2 actual &&\n \tgit -C r3 rev-list --objects --filter=\"combine:tree:%39+blob:none\" \\\n \t\tHEAD >actual &&\n \ttest_line_count = 5 actual\n '\n \n-test_expect_success 'add a sparse pattern blob whose path has reserved chars' '\n+test_expect_success 'add sparse pattern blobs whose paths have reserved chars' '\n \tcp r3/pattern r3/pattern1+renamed% &&\n-\tgit -C r3 add pattern1+renamed% &&\n-\tgit -C r3 commit -m \"add sparse pattern file with reserved chars\"\n+\tcp r3/pattern \"r3/p;at%ter+n\" &&\n+\tcp r3/pattern r3/^~pattern &&\n+\tgit -C r3 add pattern1+renamed% \"p;at%ter+n\" ^~pattern &&\n+\tgit -C r3 commit -m \"add sparse pattern files with reserved chars\"\n '\n \n test_expect_success 'combine:... with more than two sub-filters' '\n \tgit -C r3 rev-list --objects \\\n \t\t--filter=combine:tree:3+blob:limit=40+sparse:oid=master:pattern \\\n \t\tHEAD >actual &&\n \n \texpect_has HEAD \"\" &&\n \texpect_has HEAD~1 \"\" &&\n \texpect_has HEAD~2 \"\" &&\n@@ -463,21 +474,44 @@ test_expect_success 'combine:... with more than two sub-filters' '\n \t# Should also have 3 commits\n \ttest_line_count = 9 actual &&\n \n \t# Try again, this time making sure the last sub-filter is only\n \t# URL-decoded once.\n \tcp actual expect &&\n \n \tgit -C r3 rev-list --objects \\\n \t\t--filter=combine:tree:3+blob:limit=40+sparse:oid=master:pattern1%2brenamed%25 \\\n \t\tHEAD >actual &&\n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\n+\t# Use the same composite filter again, but with a pattern file name that\n+\t# requires encoding multiple characters, and use implicit filter\n+\t# combining.\n+\tGIT_TRACE=$(pwd)/trace git -C r3 rev-list --objects \\\n+\t\t--filter=tree:3 --filter=blob:limit=40 \\\n+\t\t--filter=sparse:oid=\"master:p;at%ter+n\" \\\n+\t\tHEAD >actual &&\n+\n+\ttest_cmp expect actual &&\n+\tgrep \"Generated composite filter-spec: combine:tree:3+blob:limit=40+sparse:oid=master:p%3Bat%25ter%2B\" \\\n+\t\ttrace &&\n+\n+\t# Repeat the above test, but this time, the characters to encode are in\n+\t# the LHS of the combined filter.\n+\tGIT_TRACE=$(pwd)/trace git -C r3 rev-list --objects \\\n+\t\t--filter=sparse:oid=master:^~pattern \\\n+\t\t--filter=tree:3 --filter=blob:limit=40 \\\n+\t\tHEAD >actual &&\n+\n+\ttest_cmp expect actual &&\n+\tgrep \"Generated composite filter-spec: combine:sparse:oid=master:%5E%7Epattern+tree:3+blob:limit=40\" \\\n+\t\ttrace\n '\n \n # Test provisional omit collection logic with a repo that has objects appearing\n # at multiple depths - first deeper than the filter's threshold, then shallow.\n \n test_expect_success 'setup r4' '\n \tgit init r4 &&\n \n \techo foo > r4/foo &&\n \tmkdir r4/subdir &&\ndiff --git a/transport.c b/transport.c\nindex f1fcd2c4b0..ee7dd1c062 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -217,20 +217,21 @@ static int set_git_option(struct git_transport_options *opts,\n \t} else if (!strcmp(name, TRANS_OPT_DEEPEN_RELATIVE)) {\n \t\topts->deepen_relative = !!value;\n \t\treturn 0;\n \t} else if (!strcmp(name, TRANS_OPT_FROM_PROMISOR)) {\n \t\topts->from_promisor = !!value;\n \t\treturn 0;\n \t} else if (!strcmp(name, TRANS_OPT_NO_DEPENDENTS)) {\n \t\topts->no_dependents = !!value;\n \t\treturn 0;\n \t} else if (!strcmp(name, TRANS_OPT_LIST_OBJECTS_FILTER)) {\n+\t\tlist_objects_filter_die_if_populated(&opts->filter_options);\n \t\tparse_list_objects_filter(&opts->filter_options, value);\n \t\treturn 0;\n \t}\n \treturn 1;\n }\n \n static int connect_setup(struct transport *transport, int for_push)\n {\n \tstruct git_transport_data *data = transport->data;\n \tint flags = transport->verbose > 0 ? CONNECT_VERBOSE : 0;\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 2cdd499f28..16e748ba58 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -877,20 +877,21 @@ static void receive_needs(struct packet_reader *reader, struct object_array *wan\n \t\tif (process_deepen(reader->line, &depth))\n \t\t\tcontinue;\n \t\tif (process_deepen_since(reader->line, &deepen_since, &deepen_rev_list))\n \t\t\tcontinue;\n \t\tif (process_deepen_not(reader->line, &deepen_not, &deepen_rev_list))\n \t\t\tcontinue;\n \n \t\tif (skip_prefix(reader->line, \"filter \", &arg)) {\n \t\t\tif (!filter_capability_requested)\n \t\t\t\tdie(\"git upload-pack: filtering capability not negotiated\");\n+\t\t\tlist_objects_filter_die_if_populated(&filter_options);\n \t\t\tparse_list_objects_filter(&filter_options, arg);\n \t\t\tcontinue;\n \t\t}\n \n \t\tif (!skip_prefix(reader->line, \"want \", &arg) ||\n \t\t parse_oid_hex(arg, &oid_buf, &features))\n \t\t\tdie(\"git upload-pack: protocol error, \"\n \t\t\t \"expected to get object ID, not '%s'\", reader->line);\n \n \t\tif (parse_feature_request(features, \"deepen-relative\"))\n@@ -1296,20 +1297,21 @@ static void process_args(struct packet_reader *request,\n \t\t\tcontinue;\n \t\tif (process_deepen_not(arg, &data->deepen_not,\n \t\t\t\t &data->deepen_rev_list))\n \t\t\tcontinue;\n \t\tif (!strcmp(arg, \"deepen-relative\")) {\n \t\t\tdata->deepen_relative = 1;\n \t\t\tcontinue;\n \t\t}\n \n \t\tif (allow_filter && skip_prefix(arg, \"filter \", &p)) {\n+\t\t\tlist_objects_filter_die_if_populated(&filter_options);\n \t\t\tparse_list_objects_filter(&filter_options, p);\n \t\t\tcontinue;\n \t\t}\n \n \t\tif ((git_env_bool(\"GIT_TEST_SIDEBAND_ALL\", 0) ||\n \t\t allow_sideband_all) &&\n \t\t !strcmp(arg, \"sideband-all\")) {\n \t\t\tdata->writer.use_sideband = 1;\n \t\t\tcontinue;\n \t\t}\n-- \n2.17.1\n\n"},{"id":"376524","messageId":"20190601003603.90794-9-matvore@google.com","threadId":"51217","inReplyTo":"20190601003603.90794-1-matvore@google.com","subject":"[PATCH v2 8/9] list-objects-filter-options: clean up use of ALLOC_GROW","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-01T00:36:02Z","receivedAt":"2019-06-01T00:36:38Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Introduce a new macro ALLOC_GROW_BY which automatically zeros the added\narray elements and takes care of updating the nr value. Use the macro in\ncode introduced earlier in this patchset.\n\nSigned-off-by: Matthew DeVore \n---\n cache.h | 22 ++++++++++++++++++++++\n list-objects-filter-options.c | 17 +++++++----------\n 2 files changed, 29 insertions(+), 10 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex fa8ede9a2d..847fbdeff0 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -652,33 +652,55 @@ int init_db(const char *git_dir, const char *real_git_dir,\n void sanitize_stdfds(void);\n int daemonize(void);\n \n #define alloc_nr(x) (((x)+16)*3/2)\n \n /*\n * Realloc the buffer pointed at by variable 'x' so that it can hold\n * at least 'nr' entries; the number of entries currently allocated\n * is 'alloc', using the standard growing factor alloc_nr() macro.\n *\n+ * Consider using ALLOC_GROW_BY instead of ALLOC_GROW as it has some\n+ * added niceties.\n+ *\n * DO NOT USE any expression with side-effect for 'x', 'nr', or 'alloc'.\n */\n #define ALLOC_GROW(x, nr, alloc) \\\n \tdo { \\\n \t\tif ((nr) > alloc) { \\\n \t\t\tif (alloc_nr(alloc) < (nr)) \\\n \t\t\t\talloc = (nr); \\\n \t\t\telse \\\n \t\t\t\talloc = alloc_nr(alloc); \\\n \t\t\tREALLOC_ARRAY(x, alloc); \\\n \t\t} \\\n \t} while (0)\n \n+/*\n+ * Similar to ALLOC_GROW but handles updating of the nr value and\n+ * zeroing the bytes of the newly-grown array elements.\n+ *\n+ * DO NOT USE any expression with side-effect for any of the\n+ * arguments.\n+ */\n+#define ALLOC_GROW_BY(x, nr, increase, alloc) \\\n+\tdo { \\\n+\t\tif (increase) { \\\n+\t\t\tsize_t new_nr = nr + (increase); \\\n+\t\t\tif (new_nr < nr) \\\n+\t\t\t\tBUG(\"negative growth in ALLOC_GROW_BY\"); \\\n+\t\t\tALLOC_GROW(x, new_nr, alloc); \\\n+\t\t\tmemset((x) + nr, 0, sizeof(*(x)) * (increase)); \\\n+\t\t\tnr = new_nr; \\\n+\t\t} \\\n+\t} while (0)\n+\n /* Initialize and use the cache information */\n struct lock_file;\n void preload_index(struct index_state *index,\n \t\t const struct pathspec *pathspec,\n \t\t unsigned int refresh_flags);\n int do_read_index(struct index_state *istate, const char *path,\n \t\t int must_exist); /* for testting only! */\n int read_index_from(struct index_state *, const char *path,\n \t\t const char *gitdir);\n int is_index_unborn(struct index_state *);\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex 5e98e4a309..d8abe6cfcf 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -142,26 +142,24 @@ static int has_reserved_character(\n \t}\n \n \treturn 0;\n }\n \n static int parse_combine_subfilter(\n \tstruct list_objects_filter_options *filter_options,\n \tstruct strbuf *subspec,\n \tstruct strbuf *errbuf)\n {\n-\tsize_t new_index = filter_options->sub_nr++;\n+\tsize_t new_index = filter_options->sub_nr;\n \n-\tALLOC_GROW(filter_options->sub, filter_options->sub_nr,\n-\t\t filter_options->sub_alloc);\n-\tmemset(&filter_options->sub[new_index], 0,\n-\t sizeof(*filter_options->sub));\n+\tALLOC_GROW_BY(filter_options->sub, filter_options->sub_nr, 1,\n+\t\t filter_options->sub_alloc);\n \n \treturn has_reserved_character(subspec, errbuf) ||\n \t\turl_decode(subspec, errbuf) ||\n \t\tgently_parse_list_objects_filter(\n \t\t\t&filter_options->sub[new_index], subspec->buf, errbuf);\n }\n \n static int parse_combine_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg,\n@@ -273,27 +271,26 @@ int parse_list_objects_filter(\n \t\t/*\n \t\t * Make filter_options an LOFC_COMBINE spec so we can trivially\n \t\t * add subspecs to it.\n \t\t */\n \t\ttransform_to_combine_type(filter_options);\n \n \t\tstrbuf_addstr(&filter_options->filter_spec, \"+\");\n \t\tadd_url_encoded(&filter_options->filter_spec, arg);\n \t\ttrace_printf(\"Generated composite filter-spec: %s\\n\",\n \t\t\t filter_options->filter_spec.buf);\n-\t\tALLOC_GROW(filter_options->sub, filter_options->sub_nr + 1,\n-\t\t\t filter_options->sub_alloc);\n-\t\tfilter_options = &filter_options->sub[filter_options->sub_nr++];\n-\t\tmemset(filter_options, 0, sizeof(*filter_options));\n+\t\tALLOC_GROW_BY(filter_options->sub, filter_options->sub_nr, 1,\n+\t\t\t filter_options->sub_alloc);\n \n \t\tparse_error = gently_parse_list_objects_filter(\n-\t\t\tfilter_options, arg, &errbuf);\n+\t\t\t&filter_options->sub[filter_options->sub_nr - 1], arg,\n+\t\t\t&errbuf);\n \t}\n \tif (parse_error)\n \t\tdie(\"%s\", errbuf.buf);\n \treturn 0;\n }\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct list_objects_filter_options *filter_options = opt->value;\n-- \n2.17.1\n\n"},{"id":"376525","messageId":"20190601003603.90794-10-matvore@google.com","threadId":"51217","inReplyTo":"20190601003603.90794-1-matvore@google.com","subject":"[PATCH v2 9/9] list-objects-filter-options: make parser void","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-01T00:36:03Z","receivedAt":"2019-06-01T00:36:40Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This function always returns 0, so make it return void instead.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects-filter-options.c | 12 +++++-------\n list-objects-filter-options.h | 2 +-\n 2 files changed, 6 insertions(+), 8 deletions(-)\n\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex d8abe6cfcf..ed02c88eb6 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -247,21 +247,21 @@ static void transform_to_combine_type(\n \tstrbuf_release(&filter_options->sub[0].filter_spec);\n }\n \n void list_objects_filter_die_if_populated(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tif (filter_options->choice)\n \t\tdie(_(\"multiple filter-specs cannot be combined\"));\n }\n \n-int parse_list_objects_filter(\n+void parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg)\n {\n \tstruct strbuf errbuf = STRBUF_INIT;\n \tint parse_error;\n \n \tif (!filter_options->choice) {\n \t\tstrbuf_init(&filter_options->filter_spec, 0);\n \t\tstrbuf_addstr(&filter_options->filter_spec, arg);\n \n@@ -280,34 +280,32 @@ int parse_list_objects_filter(\n \t\t\t filter_options->filter_spec.buf);\n \t\tALLOC_GROW_BY(filter_options->sub, filter_options->sub_nr, 1,\n \t\t\t filter_options->sub_alloc);\n \n \t\tparse_error = gently_parse_list_objects_filter(\n \t\t\t&filter_options->sub[filter_options->sub_nr - 1], arg,\n \t\t\t&errbuf);\n \t}\n \tif (parse_error)\n \t\tdie(\"%s\", errbuf.buf);\n-\treturn 0;\n }\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct list_objects_filter_options *filter_options = opt->value;\n \n-\tif (unset || !arg) {\n+\tif (unset || !arg)\n \t\tlist_objects_filter_set_no_filter(filter_options);\n-\t\treturn 0;\n-\t}\n-\n-\treturn parse_list_objects_filter(filter_options, arg);\n+\telse\n+\t\tparse_list_objects_filter(filter_options, arg);\n+\treturn 0;\n }\n \n void expand_list_objects_filter_spec(\n \tconst struct list_objects_filter_options *filter,\n \tstruct strbuf *expanded_spec)\n {\n \tstrbuf_init(expanded_spec, 0);\n \tif (filter->choice == LOFC_BLOB_LIMIT)\n \t\tstrbuf_addf(expanded_spec, \"blob:limit=%lu\",\n \t\t\t filter->blob_limit_value);\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex f8c8a624e4..2c0ce6383a 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -67,21 +67,21 @@ void list_objects_filter_die_if_populated(\n \tstruct list_objects_filter_options *filter_options);\n \n /*\n * Parses the filter spec string given by arg and either (1) simply places the\n * result in filter_options if it is not yet populated or (2) combines it with\n * the filter already in filter_options if it is already populated. In the case\n * of (2), the filter specs are combined as if specified with 'combine:'.\n *\n * Dies and prints a user-facing message if an error occurs.\n */\n-int parse_list_objects_filter(\n+void parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg);\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset);\n \n #define OPT_PARSE_LIST_OBJECTS_FILTER(fo) \\\n \t{ OPTION_CALLBACK, 0, CL_ARG__FILTER, fo, N_(\"args\"), \\\n \t N_(\"object filtering\"), 0, \\\n \t opt_parse_list_objects_filter }\n-- \n2.17.1\n\n"},{"id":"376616","messageId":"4ec3a011-0f08-542e-f131-ee6de7749bce@jeffhostetler.com","threadId":"51217","inReplyTo":"20190601003603.90794-1-matvore@google.com","subject":"Re: [PATCH v2 0/9] Filter combination","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2019-06-03T21:35:39Z","receivedAt":"2019-06-03T21:43:52Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 5/31/2019 8:35 PM, Matthew DeVore wrote:\n> Here is a roll-up with hopefully all comments applied or responded to. Notable\n> changes since the last one include:\n> \n> - Added an ALLOC_GROW_BY which is used twice by this patchset to make growing\n> arrays safer and cleaner\n> - Cleaned up the URL-encoding by (1) using hex_to_bytes rather than rolling my\n> own helpers and (2) making error-string-generation non-conditional\n> - Switched to an array-based data structure rather than a linked list for both\n> LOFC_COMBINE filter spec objects and the filter object itself\n> - Changed the list_objects_filter API to be cleaner to use\n> - Changed test cases to use sparse:oid= rather than sparse:path= since the\n> latter is being disabled.\n> \n> Thank you,\n> \n> Matthew DeVore (9):\n> list-objects-filter: make API easier to use\n> list-objects-filter: put omits set in filter struct\n> list-objects-filter-options: always supply *errbuf\n> list-objects-filter: implement composite filters\n> list-objects-filter-options: move error check up\n> list-objects-filter-options: make filter_spec a strbuf\n> list-objects-filter-options: allow mult. --filter\n> list-objects-filter-options: clean up use of ALLOC_GROW\n> list-objects-filter-options: make parser void\n> \n> Documentation/rev-list-options.txt | 16 ++\n> builtin/rev-list.c | 2 +-\n> cache.h | 22 ++\n> list-objects-filter-options.c | 264 ++++++++++++++++++---\n> list-objects-filter-options.h | 32 ++-\n> list-objects-filter.c | 345 +++++++++++++++++++++-------\n> list-objects-filter.h | 35 ++-\n> list-objects.c | 55 ++---\n> t/t5616-partial-clone.sh | 19 ++\n> t/t6112-rev-list-filters-objects.sh | 197 +++++++++++++++-\n> transport.c | 1 +\n> upload-pack.c | 4 +-\n> 12 files changed, 816 insertions(+), 176 deletions(-)\n> \n\nThis looks much nicer.\nThanks\nJeff\n"},{"id":"376619","messageId":"0005347e-ceed-ac9e-ad0d-b7b11bc55d38@jeffhostetler.com","threadId":"51217","inReplyTo":"20190601003603.90794-5-matvore@google.com","subject":"Re: [PATCH v2 4/9] list-objects-filter: implement composite filters","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2019-06-03T21:51:28Z","receivedAt":"2019-06-03T21:51:32Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 5/31/2019 8:35 PM, Matthew DeVore wrote:\n> Allow combining filters such that only objects accepted by all filters\n> are shown. The motivation for this is to allow getting directory\n> listings without also fetching blobs. This can be done by combining\n> blob:none with tree:. There are massive repositories that have\n> larger-than-expected trees - even if you include only a single commit.\n> \n> The current usage requires passing the filter to rev-list in the\n> following form:\n> \n> \t--filter= --filter= ...\n> \n> Such usage is currently an error, so giving it a meaning is backwards-\n> compatible.\n> \n> The URL-encoding method is being implemented before the repeated flag\n> logic, and the user-facing documentation for URL-encoding is being\n> withheld until the repeated flag feature is implemented. The\n> URL-encoding is in general not meant to be used directly by the user,\n> and it is better to describe the URL-encoding feature in terms of the\n> repeated flag.\n> \n> Helped-by: Emily Shaffer \n> Helped-by: Jeff Hostetler \n> Helped-by: Junio C Hamano \n> Signed-off-by: Matthew DeVore \n> ---\n> list-objects-filter-options.c | 135 ++++++++++++++++++++++-\n> list-objects-filter-options.h | 17 ++-\n> list-objects-filter.c | 159 +++++++++++++++++++++++++++\n> t/t6112-rev-list-filters-objects.sh | 163 +++++++++++++++++++++++++++-\n> 4 files changed, 468 insertions(+), 6 deletions(-)\n> \n\n[...]\n\n> +static enum list_objects_filter_result filter_combine(\n> +\tstruct repository *r,\n> +\tenum list_objects_filter_situation filter_situation,\n> +\tstruct object *obj,\n> +\tconst char *pathname,\n> +\tconst char *filename,\n> +\tstruct oidset *omits,\n> +\tvoid *filter_data)\n> +{\n> +\tstruct combine_filter_data *d = filter_data;\n> +\tenum list_objects_filter_result combined_result =\n> +\t\tLOFR_DO_SHOW | LOFR_MARK_SEEN | LOFR_SKIP_TREE;\n> +\tsize_t sub;\n> +\n> +\tfor (sub = 0; sub < d->nr; sub++) {\n> +\t\tenum list_objects_filter_result sub_result = process_subfilter(\n> +\t\t\tr, filter_situation, obj, pathname, filename,\n> +\t\t\t&d->sub[sub]);\n> +\t\tif (!(sub_result & LOFR_DO_SHOW))\n> +\t\t\tcombined_result &= ~LOFR_DO_SHOW;\n> +\t\tif (!(sub_result & LOFR_MARK_SEEN))\n> +\t\t\tcombined_result &= ~LOFR_MARK_SEEN;\n> +\t\tif (!d->sub[sub].is_skipping_tree)\n> +\t\t\tcombined_result &= ~LOFR_SKIP_TREE;\n> +\t}\n> +\n> +\treturn combined_result;\n> +}\n\nThis may be too subtle a point for this phase, so feel free to ignore\nthis.\n\nSince we are assuming 'compose' is an AND operation, there may be an\nopportunity to short-cut some of this loop for blobs. That is, if the\nobject is a blob and any filter rejects it, it is omitted, so we don't\nneed to keep looping for that object. (Tree objects cannot be short-cut\nthis way because a tree may appear at different depths or in different\nsparse \"cones\" and may have to be reconsidered.)\n\nSo you could add an \"affects blobs only\" bit to the per-filter data\nand try this out. For example a \"compose:blob:none+sparse:foo\" should\nperform better than \"compose:sparse:foo+blob:none\" but give the same\nresults.\n\nAgain, this might be premature, so feel free to disregard.\nJeff\n\n"},{"id":"376623","messageId":"CA+P7+xqqS8wMeNw1E8yXzStNHgrCU5ME1wpWckbPA7pBD3OBHg@mail.gmail.com","threadId":"51217","inReplyTo":"20190601003603.90794-9-matvore@google.com","subject":"Re: [PATCH v2 8/9] list-objects-filter-options: clean up use of ALLOC_GROW","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2019-06-03T22:07:40Z","receivedAt":"2019-06-03T22:07:55Z","isPatch":true,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Fri, May 31, 2019 at 5:40 PM Matthew DeVore wrote:\n>\n> Introduce a new macro ALLOC_GROW_BY which automatically zeros the added\n> array elements and takes care of updating the nr value. Use the macro in\n> code introduced earlier in this patchset.\n>\n> Signed-off-by: Matthew DeVore \n> ---\n> cache.h | 22 ++++++++++++++++++++++\n> list-objects-filter-options.c | 17 +++++++----------\n> 2 files changed, 29 insertions(+), 10 deletions(-)\n>\n> diff --git a/cache.h b/cache.h\n> index fa8ede9a2d..847fbdeff0 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -652,33 +652,55 @@ int init_db(const char *git_dir, const char *real_git_dir,\n> void sanitize_stdfds(void);\n> int daemonize(void);\n>\n> #define alloc_nr(x) (((x)+16)*3/2)\n>\n> /*\n> * Realloc the buffer pointed at by variable 'x' so that it can hold\n> * at least 'nr' entries; the number of entries currently allocated\n> * is 'alloc', using the standard growing factor alloc_nr() macro.\n> *\n> + * Consider using ALLOC_GROW_BY instead of ALLOC_GROW as it has some\n> + * added niceties.\n> + *\n> * DO NOT USE any expression with side-effect for 'x', 'nr', or 'alloc'.\n> */\n> #define ALLOC_GROW(x, nr, alloc) \\\n> do { \\\n> if ((nr) > alloc) { \\\n> if (alloc_nr(alloc) < (nr)) \\\n> alloc = (nr); \\\n> else \\\n> alloc = alloc_nr(alloc); \\\n> REALLOC_ARRAY(x, alloc); \\\n> } \\\n> } while (0)\n>\n> +/*\n> + * Similar to ALLOC_GROW but handles updating of the nr value and\n> + * zeroing the bytes of the newly-grown array elements.\n> + *\n> + * DO NOT USE any expression with side-effect for any of the\n> + * arguments.\n> + */\n\nSince ALLOC_GROW already doesn't handle this safely, there isn't\nnecessarily a reason to fix it, but you could read the macro values\ninto temporary variables inside the do { } while(0) loop in order to\navoid the multiple-expansion side effect issues...\n\nThanks,\nJake\n\n> +#define ALLOC_GROW_BY(x, nr, increase, alloc) \\\n> + do { \\\n> + if (increase) { \\\n> + size_t new_nr = nr + (increase); \\\n> + if (new_nr < nr) \\\n> + BUG(\"negative growth in ALLOC_GROW_BY\"); \\\n> + ALLOC_GROW(x, new_nr, alloc); \\\n> + memset((x) + nr, 0, sizeof(*(x)) * (increase)); \\\n> + nr = new_nr; \\\n> + } \\\n> + } while (0)\n> +\n> /* Initialize and use the cache information */\n> struct lock_file;\n> void preload_index(struct index_state *index,\n> const struct pathspec *pathspec,\n> unsigned int refresh_flags);\n> int do_read_index(struct index_state *istate, const char *path,\n> int must_exist); /* for testting only! */\n> int read_index_from(struct index_state *, const char *path,\n> const char *gitdir);\n> int is_index_unborn(struct index_state *);\n> diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\n> index 5e98e4a309..d8abe6cfcf 100644\n> --- a/list-objects-filter-options.c\n> +++ b/list-objects-filter-options.c\n> @@ -142,26 +142,24 @@ static int has_reserved_character(\n> }\n>\n> return 0;\n> }\n>\n> static int parse_combine_subfilter(\n> struct list_objects_filter_options *filter_options,\n> struct strbuf *subspec,\n> struct strbuf *errbuf)\n> {\n> - size_t new_index = filter_options->sub_nr++;\n> + size_t new_index = filter_options->sub_nr;\n>\n> - ALLOC_GROW(filter_options->sub, filter_options->sub_nr,\n> - filter_options->sub_alloc);\n> - memset(&filter_options->sub[new_index], 0,\n> - sizeof(*filter_options->sub));\n> + ALLOC_GROW_BY(filter_options->sub, filter_options->sub_nr, 1,\n> + filter_options->sub_alloc);\n>\n> return has_reserved_character(subspec, errbuf) ||\n> url_decode(subspec, errbuf) ||\n> gently_parse_list_objects_filter(\n> &filter_options->sub[new_index], subspec->buf, errbuf);\n> }\n>\n> static int parse_combine_filter(\n> struct list_objects_filter_options *filter_options,\n> const char *arg,\n> @@ -273,27 +271,26 @@ int parse_list_objects_filter(\n> /*\n> * Make filter_options an LOFC_COMBINE spec so we can trivially\n> * add subspecs to it.\n> */\n> transform_to_combine_type(filter_options);\n>\n> strbuf_addstr(&filter_options->filter_spec, \"+\");\n> add_url_encoded(&filter_options->filter_spec, arg);\n> trace_printf(\"Generated composite filter-spec: %s\\n\",\n> filter_options->filter_spec.buf);\n> - ALLOC_GROW(filter_options->sub, filter_options->sub_nr + 1,\n> - filter_options->sub_alloc);\n> - filter_options = &filter_options->sub[filter_options->sub_nr++];\n> - memset(filter_options, 0, sizeof(*filter_options));\n> + ALLOC_GROW_BY(filter_options->sub, filter_options->sub_nr, 1,\n> + filter_options->sub_alloc);\n>\n> parse_error = gently_parse_list_objects_filter(\n> - filter_options, arg, &errbuf);\n> + &filter_options->sub[filter_options->sub_nr - 1], arg,\n> + &errbuf);\n> }\n> if (parse_error)\n> die(\"%s\", errbuf.buf);\n> return 0;\n> }\n>\n> int opt_parse_list_objects_filter(const struct option *opt,\n> const char *arg, int unset)\n> {\n> struct list_objects_filter_options *filter_options = opt->value;\n> --\n> 2.17.1\n>\n"},{"id":"376627","messageId":"20190603223925.GH4641@comcast.net","threadId":"51217","inReplyTo":"CA+P7+xqqS8wMeNw1E8yXzStNHgrCU5ME1wpWckbPA7pBD3OBHg@mail.gmail.com","subject":"Re: [PATCH v2 8/9] list-objects-filter-options: clean up use of ALLOC_GROW","fromName":"Matthew DeVore","fromEmail":"matvore@comcast.net","sentAt":"2019-06-03T22:39:25Z","receivedAt":"2019-06-03T22:39:29Z","isPatch":true,"sender":{"key":"matvore@comcast.net","avatar":"https://gravatar.com/avatar/550c64ce544f82818ad931e244dfb08bbb1febfa6d1ce3cfd65e76215ca0ac8a?d=mp&s=160"},"body":"On Mon, Jun 03, 2019 at 03:07:40PM -0700, Jacob Keller wrote:\n> > +/*\n> > + * Similar to ALLOC_GROW but handles updating of the nr value and\n> > + * zeroing the bytes of the newly-grown array elements.\n> > + *\n> > + * DO NOT USE any expression with side-effect for any of the\n> > + * arguments.\n> > + */\n> \n> Since ALLOC_GROW already doesn't handle this safely, there isn't\n> necessarily a reason to fix it, but you could read the macro values\n> into temporary variables inside the do { } while(0) loop in order to\n> avoid the multiple-expansion side effect issues...\n\nFor x I don't think that's possible since we don't know the pointer type. For\nnr and alloc it doesn't make sense since they're being assigned to. For\n`increase` I could try this:\n\n\tsize_t ALLOC_GROW_BY__increase = (increase);\n\nbut I'm not sure how well this works when `increase` is a signed type. This\nseemed sufficiently pitfall-y that I didn't attempt it. Relatedly, I was\nthinking something like this would be nice, if anyone has time for such a\nrefactor:\n\nstruct growth_info {\n\tsize_t nr, alloc;\n}\n\nAnd use that to replace individual \"size_t foo_nr, foo_alloc\"\n\nAnd make ALLOC_GROW_BY use it. I think a bulk, maybe even most, ALLOC_GROW\ninvocations can be changed to ALLOC_GROW_BY.\n"},{"id":"376641","messageId":"CA+P7+xp6FYWZA8yXcksw6OiMqiM3Ja5EpVSTbcgeaGD-s+c6=w@mail.gmail.com","threadId":"51217","inReplyTo":"20190603223925.GH4641@comcast.net","subject":"Re: [PATCH v2 8/9] list-objects-filter-options: clean up use of ALLOC_GROW","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2019-06-04T03:16:34Z","receivedAt":"2019-06-04T03:16:46Z","isPatch":true,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Mon, Jun 3, 2019 at 3:39 PM Matthew DeVore wrote:\n>\n> On Mon, Jun 03, 2019 at 03:07:40PM -0700, Jacob Keller wrote:\n> > > +/*\n> > > + * Similar to ALLOC_GROW but handles updating of the nr value and\n> > > + * zeroing the bytes of the newly-grown array elements.\n> > > + *\n> > > + * DO NOT USE any expression with side-effect for any of the\n> > > + * arguments.\n> > > + */\n> >\n> > Since ALLOC_GROW already doesn't handle this safely, there isn't\n> > necessarily a reason to fix it, but you could read the macro values\n> > into temporary variables inside the do { } while(0) loop in order to\n> > avoid the multiple-expansion side effect issues...\n>\n> For x I don't think that's possible since we don't know the pointer type. For\n> nr and alloc it doesn't make sense since they're being assigned to. For\n> `increase` I could try this:\n>\n\nAh.. you could do the compiler typeof extensions, but I guess we\nprobably don't wanna rely on that.\n\n> size_t ALLOC_GROW_BY__increase = (increase);\n>\n> but I'm not sure how well this works when `increase` is a signed type. This\n> seemed sufficiently pitfall-y that I didn't attempt it.\n\nOk that makes sense.\n\nRegards,\nJake\n"},{"id":"376787","messageId":"20190606223251.GA7246@comcast.net","threadId":"51217","inReplyTo":"0005347e-ceed-ac9e-ad0d-b7b11bc55d38@jeffhostetler.com","subject":"Re: [PATCH v2 4/9] list-objects-filter: implement composite filters","fromName":"Matthew DeVore","fromEmail":"matvore@comcast.net","sentAt":"2019-06-06T22:32:51Z","receivedAt":"2019-06-06T22:33:47Z","isPatch":true,"sender":{"key":"matvore@comcast.net","avatar":"https://gravatar.com/avatar/550c64ce544f82818ad931e244dfb08bbb1febfa6d1ce3cfd65e76215ca0ac8a?d=mp&s=160"},"body":"On Mon, Jun 03, 2019 at 05:51:28PM -0400, Jeff Hostetler wrote:\n> Since we are assuming 'compose' is an AND operation, there may be an\n> opportunity to short-cut some of this loop for blobs. That is, if the\n> object is a blob and any filter rejects it, it is omitted, so we don't\n> need to keep looping for that object. (Tree objects cannot be short-cut\n> this way because a tree may appear at different depths or in different\n> sparse \"cones\" and may have to be reconsidered.)\n\nBlobs are also treated almost the same way as tree objects in tree:\nfilters - they can be included by tree: - so they also need to be\nreconsidered when found at different depths.\n\nBut I agree it's always true that if some prior filter has excluded a blob, the\nlater filters don't even need to be *called at all* for that blob, unless\nperhaps it's found under a different tree later. I also think it may be too\nearly to implement this optimization, since filter in a later release may just\nwant to \"know\" about a blob even if it must be excluded in the final result.\n\nDoes the optimization apply to trees as well? Does a tree: filter still\nwant to consider children of tree X if tree X has already been excluded by\nanother filter? If it doesn't want to consider, we can short-circuit the checks\nvery aggressively. If it does want to consider, we want the short-circuiting to\nbe customizable at least for trees.\n\nA minor point - I don't think that short-circuiting the for loop (breaking out\nearly) is important, since it will be very rare that a combine: filter has more\nthan 4 or so sub-filters anyway. Calling the filter_fn implementation and\nletting that do internal short-circuiting (informed by the previous filters'\nresults) can, however, skip a lot of computation.\n\n> So you could add an \"affects blobs only\" bit to the per-filter data\n> and try this out. For example a \"compose:blob:none+sparse:foo\" should\n> perform better than \"compose:sparse:foo+blob:none\" but give the same\n> results.\n\nDoes \"affects blobs only\" mean the filter includes all non-blob objects?\n"},{"id":"376827","messageId":"5dc58484-b206-dfe0-981a-437d0a162444@jeffhostetler.com","threadId":"51217","inReplyTo":"20190606223251.GA7246@comcast.net","subject":"Re: [PATCH v2 4/9] list-objects-filter: implement composite filters","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2019-06-07T17:58:01Z","receivedAt":"2019-06-07T17:58:11Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 6/6/2019 6:32 PM, Matthew DeVore wrote:\n> On Mon, Jun 03, 2019 at 05:51:28PM -0400, Jeff Hostetler wrote:\n>> Since we are assuming 'compose' is an AND operation, there may be an\n>> opportunity to short-cut some of this loop for blobs. That is, if the\n>> object is a blob and any filter rejects it, it is omitted, so we don't\n>> need to keep looping for that object. (Tree objects cannot be short-cut\n>> this way because a tree may appear at different depths or in different\n>> sparse \"cones\" and may have to be reconsidered.)\n> \n> Blobs are also treated almost the same way as tree objects in tree:\n> filters - they can be included by tree: - so they also need to be\n> reconsidered when found at different depths.\n> \n> But I agree it's always true that if some prior filter has excluded a blob, the\n> later filters don't even need to be *called at all* for that blob, unless\n> perhaps it's found under a different tree later. I also think it may be too\n> early to implement this optimization, since filter in a later release may just\n> want to \"know\" about a blob even if it must be excluded in the final result.\n> \n> Does the optimization apply to trees as well? Does a tree: filter still\n> want to consider children of tree X if tree X has already been excluded by\n> another filter? If it doesn't want to consider, we can short-circuit the checks\n> very aggressively. If it does want to consider, we want the short-circuiting to\n> be customizable at least for trees.\n> \n> A minor point - I don't think that short-circuiting the for loop (breaking out\n> early) is important, since it will be very rare that a combine: filter has more\n> than 4 or so sub-filters anyway. Calling the filter_fn implementation and\n> letting that do internal short-circuiting (informed by the previous filters'\n> results) can, however, skip a lot of computation.\n> \n>> So you could add an \"affects blobs only\" bit to the per-filter data\n>> and try this out. For example a \"compose:blob:none+sparse:foo\" should\n>> perform better than \"compose:sparse:foo+blob:none\" but give the same\n>> results.\n> \n> Does \"affects blobs only\" mean the filter includes all non-blob objects?\n> \n\nI just meant that the blobs:none and blobs:limit filters give you a hard\nomit. Other filters later in the chain cannot change or override that\nanswer (because of the AND assumption); it doesn't matter how deep or\nshallow the blob is the tree.\n\nIn the case of the tree:depth filter, a blob deep in the tree should\nbe provisionally omitted in case it appears later in a shallow tree\nand should be included. The tree filter can't do a hard omit on a blob\n(just like it can't do a hard omit on a tree node).\n\nWRT your question about a later filter \"just wanting to know\" about\na blob, I'm not sure.\n\nSo yeah, let's wait on this. We can always add it later as an\noptimization if/when it becomes a perf problem (and we have more\nexperience using them in practice).\n\nJeff\n\n\n"},{"id":"376967","messageId":"xmqqimtdmc59.fsf@gitster-ct.c.googlers.com","threadId":"51217","inReplyTo":"20190601003603.90794-7-matvore@google.com","subject":"Re: [PATCH v2 6/9] list-objects-filter-options: make filter_spec a strbuf","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-10T20:13:54Z","receivedAt":"2019-06-10T20:13:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthew DeVore writes:\n\n> -\tfilter_options->filter_spec = strdup(core_partial_clone_filter_default);\n> +\tif (!filter_options->filter_spec.buf)\n> +\t\tstrbuf_init(&filter_options->filter_spec, 0);\n\nThis part made me go \"Huh?\" a bit.\n\nDo we document that .buf==NULL means an uninitialized strbuf that is\nsafe to run strbuf_init() on? I do not mind that as a general\nconvention, and it may even be a useful one (i.e. it allows you to\ncalloc() a structure with an embedded strbuf in it and the \"if\n.buf==NULL, call strbuf_init() lazily\" can become an established\npattern), but at the same time it feels a bit brittle. \n\nSuch a convention forces everybody who might want to use such an\nembedded strbuf to first check .buf==NULL and lazily initialize\nit---and at some point when the embedded strbuf to be used by enough\ncodepaths, it would make the code more robust by giving up on the\nlazy initialization (iow, when *filter_options is initialized, run\nstrbuf_init() on its .filter_spec field).\n"},{"id":"376990","messageId":"20190611003456.GB10396@comcast.net","threadId":"51217","inReplyTo":"xmqqimtdmc59.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 6/9] list-objects-filter-options: make filter_spec a strbuf","fromName":"Matthew DeVore","fromEmail":"matvore@comcast.net","sentAt":"2019-06-11T00:34:56Z","receivedAt":"2019-06-11T00:35:29Z","isPatch":true,"sender":{"key":"matvore@comcast.net","avatar":"https://gravatar.com/avatar/550c64ce544f82818ad931e244dfb08bbb1febfa6d1ce3cfd65e76215ca0ac8a?d=mp&s=160"},"body":"On Mon, Jun 10, 2019 at 01:13:54PM -0700, Junio C Hamano wrote:\n> Matthew DeVore writes:\n> \n> > -\tfilter_options->filter_spec = strdup(core_partial_clone_filter_default);\n> > +\tif (!filter_options->filter_spec.buf)\n> > +\t\tstrbuf_init(&filter_options->filter_spec, 0);\n> \n> This part made me go \"Huh?\" a bit.\n> \n> Do we document that .buf==NULL means an uninitialized strbuf that is\n> safe to run strbuf_init() on? I do not mind that as a general\n\nKind of. The first bullet point in strbuf.h says:\n\n * - The `buf` member is never NULL, so it can be used in any usual C\n * string operations safely. strbuf's _have_ to be initialized either by\n * `strbuf_init()` or by `= STRBUF_INIT` before the invariants, though.\n\nSo I extrapolated that if buf is NULL, it must be because it was just xcalloc'd\nand not initialized. One possible improvement to the API would be to refactor\nit such that there is no STRBUF_INIT, but a zero-initialized strbuf is valid.\nIf you expect to get a non-NULL buf, even for a zero-initialized strbuf, you\nshould call a function like strbuf_nonnull_buf(&buf), and that will return the\nslop buf if buf is null, or the actual buf if it is non-null.\n\nI don't understand why the API designer was so strict about requiring the\nbuffer to be set to non-null, since it's quite a burden for API users. If I\neagerly set all filter_options's strbuf's to STRBUF_INIT, it involves changing\na couple of global variables which currently do not need an initializer, and it\nwould make the code a bit messy. The structs which have a strbuf somewhere in\ntheir nested fields would need to know that, and set up an initialization macro\nto avoid the null buf.\n\nI kind of suspect the right short-term fix is to avoid strbuf's and use a\nstring_list, which I join later to a full string when needed.\n\n> convention, and it may even be a useful one (i.e. it allows you to\n> calloc() a structure with an embedded strbuf in it and the \"if\n> .buf==NULL, call strbuf_init() lazily\" can become an established\n> pattern), but at the same time it feels a bit brittle. \n\nIs it brittle because a strbuf may be initialized to non-zero memory, and so\nthe \"if (buf.buf == NULL)\" may evaluate to false, and then go on treating\ngarbage like a valid buffer? I would think that's almost impossible because of\nthe use of xcalloc.\n\nThe only reason I realized the strbuf_init was necessary was not because I read\nthe documentation, but because I mistakenly called strbuf_reset, which calls\nstrbuf_setlen, which doesn't handle a null buf. Many other functions seem to\nhandle it well semi-accidentially. After I ran into the crash, I finally read\nthe documentation I cited above.\n\n> \n> Such a convention forces everybody who might want to use such an\n> embedded strbuf to first check .buf==NULL and lazily initialize\n> it---and at some point when the embedded strbuf to be used by enough\n> codepaths, it would make the code more robust by giving up on the\n> lazy initialization (iow, when *filter_options is initialized, run\n> strbuf_init() on its .filter_spec field).\n"},{"id":"377015","messageId":"xmqqtvcwkowx.fsf@gitster-ct.c.googlers.com","threadId":"51217","inReplyTo":"20190611003456.GB10396@comcast.net","subject":"Re: [PATCH v2 6/9] list-objects-filter-options: make filter_spec a strbuf","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-11T17:33:18Z","receivedAt":"2019-06-11T17:33:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthew DeVore writes:\n\n>> convention, and it may even be a useful one (i.e. it allows you to\n>> calloc() a structure with an embedded strbuf in it and the \"if\n>> .buf==NULL, call strbuf_init() lazily\" can become an established\n>> pattern), but at the same time it feels a bit brittle. \n>\n> Is it brittle because a strbuf may be initialized to non-zero memory, and so\n> the \"if (buf.buf == NULL)\" may evaluate to false, and then go on treating\n> garbage like a valid buffer?\n\nIt is brittle because callers are bound to forget doing \"if\n(!x->buf.buf) lazy_init(&x->buf)\" at some point, and blindly use an\nuninitialized x->buf. Making sure x->buf is always initialized\nbefore any caller touches is the only way to solve it, and as you\nsaid, there are two possible ways to make that happen. One way that\ndoes not violate the current API contract is to make sure whoever\nallocates and/or initializes the surrounding struct that embeds a\nstrbuf does strbuf_init(&x->buf) before any user sees the struct.\n\nAnother would be to update strbuf API so that strbuf_init() does not\neven have to use slopbuf. But that is a much larger change that\npotentially breaks existing users of strbuf API. When you have a\nstrbuf that has been prepared to be usable, the current API contract\nallows its users to expect buf.buf is never NULL, so they assume\nthat they can safely write \"if (!buf.buf)\", so auditing strbuf.c and\nmaking sure a strbuf with buf==NULL gets lazily initialized is not\nenough.\n"},{"id":"377018","messageId":"20190611184426.GB58112@comcast.net","threadId":"51217","inReplyTo":"xmqqtvcwkowx.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 6/9] list-objects-filter-options: make filter_spec a strbuf","fromName":"Matthew DeVore","fromEmail":"matvore@comcast.net","sentAt":"2019-06-11T18:44:27Z","receivedAt":"2019-06-11T18:45:02Z","isPatch":true,"sender":{"key":"matvore@comcast.net","avatar":"https://gravatar.com/avatar/550c64ce544f82818ad931e244dfb08bbb1febfa6d1ce3cfd65e76215ca0ac8a?d=mp&s=160"},"body":"On Tue, Jun 11, 2019 at 10:33:18AM -0700, Junio C Hamano wrote:\n> Matthew DeVore writes:\n> \n> >> convention, and it may even be a useful one (i.e. it allows you to\n> >> calloc() a structure with an embedded strbuf in it and the \"if\n> >> .buf==NULL, call strbuf_init() lazily\" can become an established\n> >> pattern), but at the same time it feels a bit brittle. \n> >\n> > Is it brittle because a strbuf may be initialized to non-zero memory, and so\n> > the \"if (buf.buf == NULL)\" may evaluate to false, and then go on treating\n> > garbage like a valid buffer?\n> \n> It is brittle because callers are bound to forget doing \"if\n> (!x->buf.buf) lazy_init(&x->buf)\" at some point, and blindly use an\n> uninitialized x->buf. Making sure x->buf is always initialized\n\nA corallary proposition would be to make this particular strbuf a \"struct\nstrbuf *\" rather than an inline strbuf. It should then be rather clear to users\nthat it may be null. Then whoever allocates the memory can also do the\nstrbuf_init one-liner. The free'ing logic of list_objects_filter_options then\nonly becomes trivially more complicated than it was before. Does that sound\nlike a good compromise to you?\n\n> before any caller touches is the only way to solve it, and as you\n> said, there are two possible ways to make that happen. One way that\n> does not violate the current API contract is to make sure whoever\n> allocates and/or initializes the surrounding struct that embeds a\n> strbuf does strbuf_init(&x->buf) before any user sees the struct.\n\nThe thing I don't like about that is that the non-zeroness of its\ninitialization percolates upward to whatever the top-level struct is, which\nmeans implementation details leak a lot. This seems quite brittle as well,\nsince anyone can forget to initialize some struct in the nested line.\n\n> \n> Another would be to update strbuf API so that strbuf_init() does not\n> even have to use slopbuf. But that is a much larger change that\n> potentially breaks existing users of strbuf API. When you have a\n> strbuf that has been prepared to be usable, the current API contract\n> allows its users to expect buf.buf is never NULL, so they assume\n> that they can safely write \"if (!buf.buf)\", so auditing strbuf.c and\n> making sure a strbuf with buf==NULL gets lazily initialized is not\n> enough.\n\nThat's true. I didn't think it matters in the case of filter_spec in\nparticular, since users of list_objects_filter_options are supposed to use an\naccessor and not touch the strbuf directly, but looking at it like a more\ngeneral API change, it seems risky.\n"},{"id":"377035","messageId":"20190611213426.GC58112@comcast.net","threadId":"51217","inReplyTo":"20190611184426.GB58112@comcast.net","subject":"Re: [PATCH v2 6/9] list-objects-filter-options: make filter_spec a strbuf","fromName":"Matthew DeVore","fromEmail":"matvore@comcast.net","sentAt":"2019-06-11T21:34:26Z","receivedAt":"2019-06-11T21:34:59Z","isPatch":true,"sender":{"key":"matvore@comcast.net","avatar":"https://gravatar.com/avatar/550c64ce544f82818ad931e244dfb08bbb1febfa6d1ce3cfd65e76215ca0ac8a?d=mp&s=160"},"body":"On Tue, Jun 11, 2019 at 11:44:27AM -0700, Matthew DeVore wrote:\n> A corallary proposition would be to make this particular strbuf a \"struct\n> strbuf *\" rather than an inline strbuf. It should then be rather clear to users\n> that it may be null. Then whoever allocates the memory can also do the\n> strbuf_init one-liner. The free'ing logic of list_objects_filter_options then\n> only becomes trivially more complicated than it was before. Does that sound\n> like a good compromise to you?\n> \n\nThis interdiff illustrates what I'm talking about. I don't think I like the\nfact there are two strbuf's now, but I think you get the idea. This also fixes\na memory leak in upload-pack.c, and makes the API cleaner to use:\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 85b0d3155d..81e6010779 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1135,27 +1135,25 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\ttransport_set_option(transport, TRANS_OPT_FOLLOWTAGS, \"1\");\n \n \tif (option_upload_pack)\n \t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK,\n \t\t\t\t option_upload_pack);\n \n \tif (server_options.nr)\n \t\ttransport->server_options = &server_options;\n \n \tif (filter_options.choice) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n-\t\texpand_list_objects_filter_spec(&filter_options,\n-\t\t\t\t\t\t&expanded_filter_spec);\n+\t\tconst char *spec =\n+\t\t\texpand_list_objects_filter_spec(&filter_options);\n \t\ttransport_set_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER,\n-\t\t\t\t expanded_filter_spec.buf);\n+\t\t\t\t spec);\n \t\ttransport_set_option(transport, TRANS_OPT_FROM_PROMISOR, \"1\");\n-\t\tstrbuf_release(&expanded_filter_spec);\n \t}\n \n \tif (transport->smart_options && !deepen && !filter_options.choice)\n \t\ttransport->smart_options->check_self_contained_and_connected = 1;\n \n \n \targv_array_push(&ref_prefixes, \"HEAD\");\n \trefspec_ref_prefixes(&remote->fetch, &ref_prefixes);\n \tif (option_branch)\n \t\texpand_ref_prefix(&ref_prefixes, option_branch);\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 4ba63d5ac6..dee89e1a19 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1181,27 +1181,24 @@ static struct transport *prepare_transport(struct remote *remote, int deepen)\n \tif (deepen && deepen_since)\n \t\tset_option(transport, TRANS_OPT_DEEPEN_SINCE, deepen_since);\n \tif (deepen && deepen_not.nr)\n \t\tset_option(transport, TRANS_OPT_DEEPEN_NOT,\n \t\t\t (const char *)&deepen_not);\n \tif (deepen_relative)\n \t\tset_option(transport, TRANS_OPT_DEEPEN_RELATIVE, \"yes\");\n \tif (update_shallow)\n \t\tset_option(transport, TRANS_OPT_UPDATE_SHALLOW, \"yes\");\n \tif (filter_options.choice) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n-\t\texpand_list_objects_filter_spec(&filter_options,\n-\t\t\t\t\t\t&expanded_filter_spec);\n-\t\tset_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER,\n-\t\t\t expanded_filter_spec.buf);\n+\t\tconst char *spec =\n+\t\t\texpand_list_objects_filter_spec(&filter_options);\n+\t\tset_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER, spec);\n \t\tset_option(transport, TRANS_OPT_FROM_PROMISOR, \"1\");\n-\t\tstrbuf_release(&expanded_filter_spec);\n \t}\n \tif (negotiation_tip.nr) {\n \t\tif (transport->smart_options)\n \t\t\tadd_negotiation_tips(transport->smart_options);\n \t\telse\n \t\t\twarning(\"Ignoring --negotiation-tip because the protocol does not support it.\");\n \t}\n \treturn transport;\n }\n \ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 7137f13a74..b194430217 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -458,23 +458,26 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\tif (skip_prefix(arg, \"--progress=\", &arg)) {\n \t\t\tshow_progress = arg;\n \t\t\tcontinue;\n \t\t}\n \n \t\tif (skip_prefix(arg, (\"--\" CL_ARG__FILTER \"=\"), &arg)) {\n \t\t\tparse_list_objects_filter(&filter_options, arg);\n \t\t\tif (filter_options.choice && !revs.blob_objects)\n \t\t\t\tdie(_(\"object filtering requires --objects\"));\n \t\t\tif (filter_options.choice == LOFC_SPARSE_OID &&\n-\t\t\t !filter_options.sparse_oid_value)\n-\t\t\t\tdie(_(\"invalid sparse value '%s'\"),\n-\t\t\t\t filter_options.filter_spec.buf);\n+\t\t\t !filter_options.sparse_oid_value) {\n+\t\t\t\tconst char *spec =\n+\t\t\t\t\texpand_list_objects_filter_spec(\n+\t\t\t\t\t\t&filter_options);\n+\t\t\t\tdie(_(\"invalid sparse value '%s'\"), spec);\n+\t\t\t}\n \t\t\tcontinue;\n \t\t}\n \t\tif (!strcmp(arg, (\"--no-\" CL_ARG__FILTER))) {\n \t\t\tlist_objects_filter_set_no_filter(&filter_options);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!strcmp(arg, \"--filter-print-omitted\")) {\n \t\t\targ_print_omitted = 1;\n \t\t\tcontinue;\n \t\t}\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 1c10f54e78..72e13b0a1d 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -332,26 +332,23 @@ static int find_common(struct fetch_negotiator *negotiator,\n \t\tpacket_buf_write(&req_buf, \"deepen-since %\"PRItime, max_age);\n \t}\n \tif (args->deepen_not) {\n \t\tint i;\n \t\tfor (i = 0; i < args->deepen_not->nr; i++) {\n \t\t\tstruct string_list_item *s = args->deepen_not->items + i;\n \t\t\tpacket_buf_write(&req_buf, \"deepen-not %s\", s->string);\n \t\t}\n \t}\n \tif (server_supports_filtering && args->filter_options.choice) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n-\t\texpand_list_objects_filter_spec(&args->filter_options,\n-\t\t\t\t\t\t&expanded_filter_spec);\n-\t\tpacket_buf_write(&req_buf, \"filter %s\",\n-\t\t\t\t expanded_filter_spec.buf);\n-\t\tstrbuf_release(&expanded_filter_spec);\n+\t\tconst char *spec =\n+\t\t\texpand_list_objects_filter_spec(&args->filter_options);\n+\t\tpacket_buf_write(&req_buf, \"filter %s\", spec);\n \t}\n \tpacket_buf_flush(&req_buf);\n \tstate_len = req_buf.len;\n \n \tif (args->deepen) {\n \t\tconst char *arg;\n \t\tstruct object_id oid;\n \n \t\tsend_request(args, fd[1], &req_buf);\n \t\twhile (packet_reader_read(&reader) == PACKET_READ_NORMAL) {\n@@ -1092,21 +1089,21 @@ static int add_haves(struct fetch_negotiator *negotiator,\n \t\tret = 1;\n \t}\n \n \t/* Increase haves to send on next round */\n \t*haves_to_send = next_flush(1, *haves_to_send);\n \n \treturn ret;\n }\n \n static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n-\t\t\t const struct fetch_pack_args *args,\n+\t\t\t struct fetch_pack_args *args,\n \t\t\t const struct ref *wants, struct oidset *common,\n \t\t\t int *haves_to_send, int *in_vain,\n \t\t\t int sideband_all)\n {\n \tint ret = 0;\n \tstruct strbuf req_buf = STRBUF_INIT;\n \n \tif (server_supports_v2(\"fetch\", 1))\n \t\tpacket_buf_write(&req_buf, \"command=fetch\");\n \tif (server_supports_v2(\"agent\", 0))\n@@ -1133,27 +1130,24 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n \n \t/* Add shallow-info and deepen request */\n \tif (server_supports_feature(\"fetch\", \"shallow\", 0))\n \t\tadd_shallow_requests(&req_buf, args);\n \telse if (is_repository_shallow(the_repository) || args->deepen)\n \t\tdie(_(\"Server does not support shallow requests\"));\n \n \t/* Add filter */\n \tif (server_supports_feature(\"fetch\", \"filter\", 0) &&\n \t args->filter_options.choice) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n+\t\tconst char *spec =\n+\t\t\texpand_list_objects_filter_spec(&args->filter_options);\n \t\tprint_verbose(args, _(\"Server supports filter\"));\n-\t\texpand_list_objects_filter_spec(&args->filter_options,\n-\t\t\t\t\t\t&expanded_filter_spec);\n-\t\tpacket_buf_write(&req_buf, \"filter %s\",\n-\t\t\t\t expanded_filter_spec.buf);\n-\t\tstrbuf_release(&expanded_filter_spec);\n+\t\tpacket_buf_write(&req_buf, \"filter %s\", spec);\n \t} else if (args->filter_options.choice) {\n \t\twarning(\"filtering not recognized by server, ignoring\");\n \t}\n \n \t/* add wants */\n \tadd_wants(args->no_dependents, wants, &req_buf);\n \n \tif (args->no_dependents) {\n \t\tpacket_buf_write(&req_buf, \"done\");\n \t\tret = 1;\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex 9a5677c2c8..2523f96223 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -7,20 +7,35 @@\n #include \"list-objects-filter.h\"\n #include \"list-objects-filter-options.h\"\n #include \"trace.h\"\n #include \"url.h\"\n \n static int parse_combine_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg,\n \tstruct strbuf *errbuf);\n \n+struct filter_spec {\n+\tstruct strbuf raw;\n+\tstruct strbuf expanded;\n+};\n+\n+static void maybe_init_filter_spec(struct list_objects_filter_options *o)\n+{\n+\tif (o->filter_spec)\n+\t\treturn;\n+\n+\to->filter_spec = xcalloc(1, sizeof(*o->filter_spec));\n+\tstrbuf_init(&o->filter_spec->raw, 0);\n+\tstrbuf_init(&o->filter_spec->expanded, 0);\n+}\n+\n /*\n * Parse value of the argument to the \"filter\" keyword.\n * On the command line this looks like:\n * --filter=\n * and in the pack protocol as:\n * \"filter\" SP \n *\n * The filter keyword will be used by many commands.\n * See Documentation/rev-list-options.txt for allowed values for .\n *\n@@ -182,77 +197,78 @@ static int allow_unencoded(char ch)\n }\n \n /*\n * Changes filter_options into an equivalent LOFC_COMBINE filter options\n * instance. Does not do anything if filter_options is already LOFC_COMBINE.\n */\n static void transform_to_combine_type(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tassert(filter_options->choice);\n+\tassert(filter_options->filter_spec);\n \tif (filter_options->choice == LOFC_COMBINE)\n \t\treturn;\n \t{\n \t\tconst int initial_sub_alloc = 2;\n \t\tstruct list_objects_filter_options *sub_array =\n \t\t\txcalloc(initial_sub_alloc, sizeof(*sub_array));\n \t\tsub_array[0] = *filter_options;\n \t\tmemset(filter_options, 0, sizeof(*filter_options));\n \t\tfilter_options->sub = sub_array;\n \t\tfilter_options->sub_alloc = initial_sub_alloc;\n \t}\n \tfilter_options->sub_nr = 1;\n \tfilter_options->choice = LOFC_COMBINE;\n-\tstrbuf_init(&filter_options->filter_spec, 0);\n-\tstrbuf_addstr(&filter_options->filter_spec, \"combine:\");\n-\tstrbuf_addstr_urlencode(&filter_options->filter_spec,\n-\t\t\t\tfilter_options->sub[0].filter_spec.buf,\n+\tstrbuf_addstr(&filter_options->filter_spec->raw, \"combine:\");\n+\tstrbuf_addstr_urlencode(&filter_options->filter_spec->raw,\n+\t\t\t\tfilter_options->sub[0].filter_spec->raw.buf,\n \t\t\t\tallow_unencoded);\n \t/*\n \t * We don't need the filter_spec strings for subfilter specs, only the\n \t * top level.\n \t */\n-\tstrbuf_release(&filter_options->sub[0].filter_spec);\n+\tstrbuf_release(&filter_options->sub[0].filter_spec->raw);\n }\n \n void list_objects_filter_die_if_populated(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tif (filter_options->choice)\n \t\tdie(_(\"multiple filter-specs cannot be combined\"));\n }\n \n void parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg)\n {\n \tstruct strbuf errbuf = STRBUF_INIT;\n \tint parse_error;\n \n+\tmaybe_init_filter_spec(filter_options);\n+\n \tif (!filter_options->choice) {\n-\t\tstrbuf_init(&filter_options->filter_spec, 0);\n-\t\tstrbuf_addstr(&filter_options->filter_spec, arg);\n+\t\tstrbuf_addstr(&filter_options->filter_spec->raw, arg);\n \n \t\tparse_error = gently_parse_list_objects_filter(\n \t\t\tfilter_options, arg, &errbuf);\n \t} else {\n \t\t/*\n \t\t * Make filter_options an LOFC_COMBINE spec so we can trivially\n \t\t * add subspecs to it.\n \t\t */\n \t\ttransform_to_combine_type(filter_options);\n \n-\t\tstrbuf_addstr(&filter_options->filter_spec, \"+\");\n-\t\tstrbuf_addstr_urlencode(&filter_options->filter_spec, arg,\n+\t\tstrbuf_addstr(&filter_options->filter_spec->raw, \"+\");\n+\t\tstrbuf_addstr_urlencode(&filter_options->filter_spec->raw, arg,\n \t\t\t\t\tallow_unencoded);\n \t\ttrace_printf(\"Generated composite filter-spec: %s\\n\",\n-\t\t\t filter_options->filter_spec.buf);\n+\t\t\t filter_options->filter_spec->raw.buf);\n \t\tALLOC_GROW_BY(filter_options->sub, filter_options->sub_nr, 1,\n \t\t\t filter_options->sub_alloc);\n \n \t\tparse_error = gently_parse_list_objects_filter(\n \t\t\t&filter_options->sub[filter_options->sub_nr - 1], arg,\n \t\t\t&errbuf);\n \t}\n \tif (parse_error)\n \t\tdie(\"%s\", errbuf.buf);\n }\n@@ -262,54 +278,62 @@ int opt_parse_list_objects_filter(const struct option *opt,\n {\n \tstruct list_objects_filter_options *filter_options = opt->value;\n \n \tif (unset || !arg)\n \t\tlist_objects_filter_set_no_filter(filter_options);\n \telse\n \t\tparse_list_objects_filter(filter_options, arg);\n \treturn 0;\n }\n \n-void expand_list_objects_filter_spec(\n-\tconst struct list_objects_filter_options *filter,\n-\tstruct strbuf *expanded_spec)\n+const char *expand_list_objects_filter_spec(\n+\tstruct list_objects_filter_options *filter)\n {\n-\tstrbuf_init(expanded_spec, 0);\n+\tstruct strbuf *expanded_spec = &filter->filter_spec->expanded;\n+\tif (expanded_spec->len)\n+\t\treturn expanded_spec->buf;\n+\n \tif (filter->choice == LOFC_BLOB_LIMIT)\n \t\tstrbuf_addf(expanded_spec, \"blob:limit=%lu\",\n \t\t\t filter->blob_limit_value);\n \telse if (filter->choice == LOFC_TREE_DEPTH)\n \t\tstrbuf_addf(expanded_spec, \"tree:%lu\",\n \t\t\t filter->tree_exclude_depth);\n \telse\n-\t\tstrbuf_addstr(expanded_spec, filter->filter_spec.buf);\n+\t\tstrbuf_addstr(expanded_spec, filter->filter_spec->raw.buf);\n+\n+\treturn expanded_spec->buf;\n }\n \n void list_objects_filter_release(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tsize_t sub;\n \n \tif (!filter_options)\n \t\treturn;\n-\tstrbuf_release(&filter_options->filter_spec);\n+\tif (filter_options->filter_spec) {\n+\t\tstrbuf_release(&filter_options->filter_spec->raw);\n+\t\tstrbuf_release(&filter_options->filter_spec->expanded);\n+\t\tFREE_AND_NULL(filter_options->filter_spec);\n+\t}\n \tfree(filter_options->sparse_oid_value);\n \tfree(filter_options->sparse_path_value);\n \tfor (sub = 0; sub < filter_options->sub_nr; sub++)\n \t\tlist_objects_filter_release(&filter_options->sub[sub]);\n \tfree(filter_options->sub);\n \tmemset(filter_options, 0, sizeof(*filter_options));\n }\n \n void partial_clone_register(\n \tconst char *remote,\n-\tconst struct list_objects_filter_options *filter_options)\n+\tstruct list_objects_filter_options *filter_options)\n {\n \t/*\n \t * Record the name of the partial clone remote in the\n \t * config and in the global variable -- the latter is\n \t * used throughout to indicate that partial clone is\n \t * enabled and to expect missing objects.\n \t */\n \tif (repository_format_partial_clone &&\n \t *repository_format_partial_clone &&\n \t strcmp(remote, repository_format_partial_clone))\n@@ -318,35 +342,34 @@ void partial_clone_register(\n \tgit_config_set(\"core.repositoryformatversion\", \"1\");\n \tgit_config_set(\"extensions.partialclone\", remote);\n \n \trepository_format_partial_clone = xstrdup(remote);\n \n \t/*\n \t * Record the initial filter-spec in the config as\n \t * the default for subsequent fetches from this remote.\n \t */\n \tcore_partial_clone_filter_default =\n-\t\txstrdup(filter_options->filter_spec.buf);\n+\t\txstrdup(expand_list_objects_filter_spec(filter_options));\n \tgit_config_set(\"core.partialclonefilter\",\n \t\t core_partial_clone_filter_default);\n }\n \n void partial_clone_get_default_filter_spec(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tstruct strbuf errbuf = STRBUF_INIT;\n \n \t/*\n \t * Parse default value, but silently ignore it if it is invalid.\n \t */\n \tif (!core_partial_clone_filter_default)\n \t\treturn;\n \n-\tif (!filter_options->filter_spec.buf)\n-\t\tstrbuf_init(&filter_options->filter_spec, 0);\n-\tstrbuf_addstr(&filter_options->filter_spec,\n+\tmaybe_init_filter_spec(filter_options);\n+\tstrbuf_addstr(&filter_options->filter_spec->raw,\n \t\t core_partial_clone_filter_default);\n \tgently_parse_list_objects_filter(filter_options,\n \t\t\t\t\t core_partial_clone_filter_default,\n \t\t\t\t\t &errbuf);\n \tstrbuf_release(&errbuf);\n }\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 2c0ce6383a..07995449f1 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -11,29 +11,31 @@ enum list_objects_filter_choice {\n \tLOFC_DISABLED = 0,\n \tLOFC_BLOB_NONE,\n \tLOFC_BLOB_LIMIT,\n \tLOFC_TREE_DEPTH,\n \tLOFC_SPARSE_OID,\n \tLOFC_SPARSE_PATH,\n \tLOFC_COMBINE,\n \tLOFC__COUNT /* must be last */\n };\n \n+struct filter_spec;\n+\n struct list_objects_filter_options {\n \t/*\n-\t * 'filter_spec' is the raw argument value given on the command line\n-\t * or protocol request. (The part after the \"--keyword=\".) For\n+\t * 'filter_spec' contains the raw argument value given on the command\n+\t * line or protocol request. (The part after the \"--keyword=\".) For\n \t * commands that launch filtering sub-processes, or for communication\n \t * over the network, don't use this value; use the result of\n \t * expand_list_objects_filter_spec() instead.\n \t */\n-\tstruct strbuf filter_spec;\n+\tstruct filter_spec *filter_spec;\n \n \t/*\n \t * 'choice' is determined by parsing the filter-spec. This indicates\n \t * the filtering algorithm to use.\n \t */\n \tenum list_objects_filter_choice choice;\n \n \t/*\n \t * Choice is LOFC_DISABLED because \"--no-filter\" was requested.\n \t */\n@@ -86,31 +88,30 @@ int opt_parse_list_objects_filter(const struct option *opt,\n \t N_(\"object filtering\"), 0, \\\n \t opt_parse_list_objects_filter }\n \n /*\n * Translates abbreviated numbers in the filter's filter_spec into their\n * fully-expanded forms (e.g., \"limit:blob=1k\" becomes \"limit:blob=1024\").\n *\n * This form should be used instead of the raw filter_spec field when\n * communicating with a remote process or subprocess.\n */\n-void expand_list_objects_filter_spec(\n-\tconst struct list_objects_filter_options *filter,\n-\tstruct strbuf *expanded_spec);\n+const char *expand_list_objects_filter_spec(\n+\tstruct list_objects_filter_options *filter);\n \n void list_objects_filter_release(\n \tstruct list_objects_filter_options *filter_options);\n \n static inline void list_objects_filter_set_no_filter(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tlist_objects_filter_release(filter_options);\n \tfilter_options->no_filter = 1;\n }\n \n void partial_clone_register(\n \tconst char *remote,\n-\tconst struct list_objects_filter_options *filter_options);\n+\tstruct list_objects_filter_options *filter_options);\n void partial_clone_get_default_filter_spec(\n \tstruct list_objects_filter_options *filter_options);\n \n #endif /* LIST_OBJECTS_FILTER_OPTIONS_H */\ndiff --git a/transport-helper.c b/transport-helper.c\nindex cec83bd663..d6313ef9f5 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -675,27 +675,23 @@ static int fetch(struct transport *transport,\n \t data->transport_options.check_self_contained_and_connected)\n \t\tset_helper_option(transport, \"check-connectivity\", \"true\");\n \n \tif (transport->cloning)\n \t\tset_helper_option(transport, \"cloning\", \"true\");\n \n \tif (data->transport_options.update_shallow)\n \t\tset_helper_option(transport, \"update-shallow\", \"true\");\n \n \tif (data->transport_options.filter_options.choice) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n-\t\texpand_list_objects_filter_spec(\n-\t\t\t&data->transport_options.filter_options,\n-\t\t\t&expanded_filter_spec);\n-\t\tset_helper_option(transport, \"filter\",\n-\t\t\t\t expanded_filter_spec.buf);\n-\t\tstrbuf_release(&expanded_filter_spec);\n+\t\tconst char *spec = expand_list_objects_filter_spec(\n+\t\t\t&data->transport_options.filter_options);\n+\t\tset_helper_option(transport, \"filter\", spec);\n \t}\n \n \tif (data->transport_options.negotiation_tips)\n \t\twarning(\"Ignoring --negotiation-tip because the protocol does not support it.\");\n \n \tif (data->fetch)\n \t\treturn fetch_with_fetch(transport, nr_heads, to_fetch);\n \n \tif (data->import)\n \t\treturn fetch_with_import(transport, nr_heads, to_fetch);\ndiff --git a/upload-pack.c b/upload-pack.c\nindex ba8c3a1f8e..dda2ac6f44 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -133,32 +133,31 @@ static void create_pack_file(const struct object_array *have_obj,\n \n \targv_array_push(&pack_objects.args, \"--stdout\");\n \tif (shallow_nr)\n \t\targv_array_push(&pack_objects.args, \"--shallow\");\n \tif (!no_progress)\n \t\targv_array_push(&pack_objects.args, \"--progress\");\n \tif (use_ofs_delta)\n \t\targv_array_push(&pack_objects.args, \"--delta-base-offset\");\n \tif (use_include_tag)\n \t\targv_array_push(&pack_objects.args, \"--include-tag\");\n-\tif (filter_options.filter_spec.len) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n-\t\texpand_list_objects_filter_spec(&filter_options,\n-\t\t\t\t\t\t&expanded_filter_spec);\n+\tif (filter_options.choice) {\n+\t\tconst char *spec =\n+\t\t\texpand_list_objects_filter_spec(&filter_options);\n \t\tif (pack_objects.use_shell) {\n \t\t\tstruct strbuf buf = STRBUF_INIT;\n-\t\t\tsq_quote_buf(&buf, expanded_filter_spec.buf);\n+\t\t\tsq_quote_buf(&buf, spec);\n \t\t\targv_array_pushf(&pack_objects.args, \"--filter=%s\", buf.buf);\n \t\t\tstrbuf_release(&buf);\n \t\t} else {\n \t\t\targv_array_pushf(&pack_objects.args, \"--filter=%s\",\n-\t\t\t\t\t expanded_filter_spec.buf);\n+\t\t\t\t\t spec);\n \t\t}\n \t}\n \n \tpack_objects.in = -1;\n \tpack_objects.out = -1;\n \tpack_objects.err = -1;\n \n \tif (start_command(&pack_objects))\n \t\tdie(\"git upload-pack: unable to fork git-pack-objects\");\n \n"},{"id":"377036","messageId":"xmqqmuinkd30.fsf@gitster-ct.c.googlers.com","threadId":"51217","inReplyTo":"20190611184426.GB58112@comcast.net","subject":"Re: [PATCH v2 6/9] list-objects-filter-options: make filter_spec a strbuf","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-11T21:48:51Z","receivedAt":"2019-06-11T21:48:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthew DeVore writes:\n\n>> It is brittle because callers are bound to forget doing \"if\n>> (!x->buf.buf) lazy_init(&x->buf)\" at some point, and blindly use an\n>> uninitialized x->buf. Making sure x->buf is always initialized\n>\n> A corallary proposition would be to make this particular strbuf a \"struct\n> strbuf *\" rather than an inline strbuf. It should then be rather clear to users\n> that it may be null.\n\nWould make it less likely for uses of an uninitialized strbuf to be\nleft undetected as errors? I guess so, and if that is the case it\nwould definitely be an improvement.\n\nBut initializing the strbuf at the point where the enclosing\nstructure is initialized (or calloc()'ed) is also a vaiable option,\nand between the two, I think that would be even more robust.\n\nThere may be reasons why it is cumbersome to arrange it that way,\nthough (e.g. if the code does not introduce a \"new_stuff()\"\nallocator that also initializes, and instead uses xcalloc() from\nmany places, initializing the enclosing structure properly might\ntake a preliminary clean-up step before the main part of the patch\nseries can begin).\n"},{"id":"377047","messageId":"20190612003716.GD58112@comcast.net","threadId":"51217","inReplyTo":"xmqqmuinkd30.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 6/9] list-objects-filter-options: make filter_spec a strbuf","fromName":"Matthew DeVore","fromEmail":"matvore@comcast.net","sentAt":"2019-06-12T00:37:16Z","receivedAt":"2019-06-12T00:37:47Z","isPatch":true,"sender":{"key":"matvore@comcast.net","avatar":"https://gravatar.com/avatar/550c64ce544f82818ad931e244dfb08bbb1febfa6d1ce3cfd65e76215ca0ac8a?d=mp&s=160"},"body":"On Tue, Jun 11, 2019 at 02:48:51PM -0700, Junio C Hamano wrote:\n> Matthew DeVore writes:\n> \n> >> It is brittle because callers are bound to forget doing \"if\n> >> (!x->buf.buf) lazy_init(&x->buf)\" at some point, and blindly use an\n> >> uninitialized x->buf. Making sure x->buf is always initialized\n> >\n> > A corallary proposition would be to make this particular strbuf a \"struct\n> > strbuf *\" rather than an inline strbuf. It should then be rather clear to users\n> > that it may be null.\n> \n> Would make it less likely for uses of an uninitialized strbuf to be\n> left undetected as errors? I guess so, and if that is the case it\n> would definitely be an improvement.\n> \n> But initializing the strbuf at the point where the enclosing\n> structure is initialized (or calloc()'ed) is also a vaiable option,\n> and between the two, I think that would be even more robust.\n> \n> There may be reasons why it is cumbersome to arrange it that way,\n> though (e.g. if the code does not introduce a \"new_stuff()\"\n> allocator that also initializes, and instead uses xcalloc() from\n> many places, initializing the enclosing structure properly might\n> take a preliminary clean-up step before the main part of the patch\n> series can begin).\n\nThese are all the locations where a struct which ultimately contains a\nlist_objects_filter_options is instantiated:\n\nGLOBAL VARIABLES:\n\nbuiltin/clone.c:68:static struct list_objects_filter_options filter_options;\nbuiltin/fetch.c:66:static struct list_objects_filter_options filter_options;\nbuiltin/pack-objects.c:112:static struct list_objects_filter_options filter_options;\nbuiltin/rev-list.c:65:static struct list_objects_filter_options filter_options;\n\nLOCAL VARIABLES:\n\nbuiltin/fetch-pack.c:54: struct fetch_pack_args args;\ntransport.c:327: struct fetch_pack_args args;\n\nHEAP ALLOCATIONS:\n\ntransport-helper.c:1123:\tstruct helper_data *data = xcalloc(1, sizeof(*data));\ntransport.c:964: struct git_transport_data *data = xcalloc(1, sizeof(*data));\n\ngit_transport_options is also not directly instantiated as a local or static\nvariable, but it would need to have a git_transport_options_init function\ndefined.\n\nI didn't count exactly the number of _INIT macros and _init functions that\nwould need to be defined. It seems like a lot of work. It is hard to believe\nthat our ability to exhaustively pinpoint all these instantiations, and to\ncatch ALL future instantiations, is all that reliable. I think our ability to\nfind the places we need to lazily instantiate the strbuf-containing-struct\n(struct filter_spec in the interdiff) is more reliable.\n"},{"id":"377079","messageId":"20190612145524.GE58112@comcast.net","threadId":"51217","inReplyTo":"20190612003716.GD58112@comcast.net","subject":"Re: [PATCH v2 6/9] list-objects-filter-options: make filter_spec a strbuf","fromName":"Matthew DeVore","fromEmail":"matvore@comcast.net","sentAt":"2019-06-12T14:55:24Z","receivedAt":"2019-06-12T14:55:58Z","isPatch":true,"sender":{"key":"matvore@comcast.net","avatar":"https://gravatar.com/avatar/550c64ce544f82818ad931e244dfb08bbb1febfa6d1ce3cfd65e76215ca0ac8a?d=mp&s=160"},"body":"On Tue, Jun 11, 2019 at 05:37:16PM -0700, Matthew DeVore wrote:\n> On Tue, Jun 11, 2019 at 02:48:51PM -0700, Junio C Hamano wrote:\n> > Matthew DeVore writes:\n> > \n> > >> It is brittle because callers are bound to forget doing \"if\n> > >> (!x->buf.buf) lazy_init(&x->buf)\" at some point, and blindly use an\n> > >> uninitialized x->buf. Making sure x->buf is always initialized\n> > >\n> > > A corallary proposition would be to make this particular strbuf a \"struct\n> > > strbuf *\" rather than an inline strbuf. It should then be rather clear to users\n> > > that it may be null.\n> > \n> > Would make it less likely for uses of an uninitialized strbuf to be\n> > left undetected as errors? I guess so, and if that is the case it\n> > would definitely be an improvement.\n> > \n> > But initializing the strbuf at the point where the enclosing\n> > structure is initialized (or calloc()'ed) is also a vaiable option,\n> > and between the two, I think that would be even more robust.\n> > \n> > There may be reasons why it is cumbersome to arrange it that way,\n> > though (e.g. if the code does not introduce a \"new_stuff()\"\n> > allocator that also initializes, and instead uses xcalloc() from\n> > many places, initializing the enclosing structure properly might\n> > take a preliminary clean-up step before the main part of the patch\n> > series can begin).\n\nHere is an alternate interdiff where I use a string_list rather than a strbuf\nfor the filter_spec. This is actually slightly shorter code than the earlier\ninterdiff using a struct filter_spec type. (not by much, maybe half-dozen lines)\nI think this is my favorite approach so far.\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 85b0d3155d..81e6010779 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1135,27 +1135,25 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\ttransport_set_option(transport, TRANS_OPT_FOLLOWTAGS, \"1\");\n \n \tif (option_upload_pack)\n \t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK,\n \t\t\t\t option_upload_pack);\n \n \tif (server_options.nr)\n \t\ttransport->server_options = &server_options;\n \n \tif (filter_options.choice) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n-\t\texpand_list_objects_filter_spec(&filter_options,\n-\t\t\t\t\t\t&expanded_filter_spec);\n+\t\tconst char *spec =\n+\t\t\texpand_list_objects_filter_spec(&filter_options);\n \t\ttransport_set_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER,\n-\t\t\t\t expanded_filter_spec.buf);\n+\t\t\t\t spec);\n \t\ttransport_set_option(transport, TRANS_OPT_FROM_PROMISOR, \"1\");\n-\t\tstrbuf_release(&expanded_filter_spec);\n \t}\n \n \tif (transport->smart_options && !deepen && !filter_options.choice)\n \t\ttransport->smart_options->check_self_contained_and_connected = 1;\n \n \n \targv_array_push(&ref_prefixes, \"HEAD\");\n \trefspec_ref_prefixes(&remote->fetch, &ref_prefixes);\n \tif (option_branch)\n \t\texpand_ref_prefix(&ref_prefixes, option_branch);\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 4ba63d5ac6..dee89e1a19 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1181,27 +1181,24 @@ static struct transport *prepare_transport(struct remote *remote, int deepen)\n \tif (deepen && deepen_since)\n \t\tset_option(transport, TRANS_OPT_DEEPEN_SINCE, deepen_since);\n \tif (deepen && deepen_not.nr)\n \t\tset_option(transport, TRANS_OPT_DEEPEN_NOT,\n \t\t\t (const char *)&deepen_not);\n \tif (deepen_relative)\n \t\tset_option(transport, TRANS_OPT_DEEPEN_RELATIVE, \"yes\");\n \tif (update_shallow)\n \t\tset_option(transport, TRANS_OPT_UPDATE_SHALLOW, \"yes\");\n \tif (filter_options.choice) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n-\t\texpand_list_objects_filter_spec(&filter_options,\n-\t\t\t\t\t\t&expanded_filter_spec);\n-\t\tset_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER,\n-\t\t\t expanded_filter_spec.buf);\n+\t\tconst char *spec =\n+\t\t\texpand_list_objects_filter_spec(&filter_options);\n+\t\tset_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER, spec);\n \t\tset_option(transport, TRANS_OPT_FROM_PROMISOR, \"1\");\n-\t\tstrbuf_release(&expanded_filter_spec);\n \t}\n \tif (negotiation_tip.nr) {\n \t\tif (transport->smart_options)\n \t\t\tadd_negotiation_tips(transport->smart_options);\n \t\telse\n \t\t\twarning(\"Ignoring --negotiation-tip because the protocol does not support it.\");\n \t}\n \treturn transport;\n }\n \ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 7137f13a74..823e87c1c9 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -459,22 +459,24 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\tshow_progress = arg;\n \t\t\tcontinue;\n \t\t}\n \n \t\tif (skip_prefix(arg, (\"--\" CL_ARG__FILTER \"=\"), &arg)) {\n \t\t\tparse_list_objects_filter(&filter_options, arg);\n \t\t\tif (filter_options.choice && !revs.blob_objects)\n \t\t\t\tdie(_(\"object filtering requires --objects\"));\n \t\t\tif (filter_options.choice == LOFC_SPARSE_OID &&\n \t\t\t !filter_options.sparse_oid_value)\n-\t\t\t\tdie(_(\"invalid sparse value '%s'\"),\n-\t\t\t\t filter_options.filter_spec.buf);\n+\t\t\t\tdie(\n+\t\t\t\t\t_(\"invalid sparse value '%s'\"),\n+\t\t\t\t\tlist_objects_filter_spec(\n+\t\t\t\t\t\t&filter_options));\n \t\t\tcontinue;\n \t\t}\n \t\tif (!strcmp(arg, (\"--no-\" CL_ARG__FILTER))) {\n \t\t\tlist_objects_filter_set_no_filter(&filter_options);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!strcmp(arg, \"--filter-print-omitted\")) {\n \t\t\targ_print_omitted = 1;\n \t\t\tcontinue;\n \t\t}\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 1c10f54e78..72e13b0a1d 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -332,26 +332,23 @@ static int find_common(struct fetch_negotiator *negotiator,\n \t\tpacket_buf_write(&req_buf, \"deepen-since %\"PRItime, max_age);\n \t}\n \tif (args->deepen_not) {\n \t\tint i;\n \t\tfor (i = 0; i < args->deepen_not->nr; i++) {\n \t\t\tstruct string_list_item *s = args->deepen_not->items + i;\n \t\t\tpacket_buf_write(&req_buf, \"deepen-not %s\", s->string);\n \t\t}\n \t}\n \tif (server_supports_filtering && args->filter_options.choice) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n-\t\texpand_list_objects_filter_spec(&args->filter_options,\n-\t\t\t\t\t\t&expanded_filter_spec);\n-\t\tpacket_buf_write(&req_buf, \"filter %s\",\n-\t\t\t\t expanded_filter_spec.buf);\n-\t\tstrbuf_release(&expanded_filter_spec);\n+\t\tconst char *spec =\n+\t\t\texpand_list_objects_filter_spec(&args->filter_options);\n+\t\tpacket_buf_write(&req_buf, \"filter %s\", spec);\n \t}\n \tpacket_buf_flush(&req_buf);\n \tstate_len = req_buf.len;\n \n \tif (args->deepen) {\n \t\tconst char *arg;\n \t\tstruct object_id oid;\n \n \t\tsend_request(args, fd[1], &req_buf);\n \t\twhile (packet_reader_read(&reader) == PACKET_READ_NORMAL) {\n@@ -1092,21 +1089,21 @@ static int add_haves(struct fetch_negotiator *negotiator,\n \t\tret = 1;\n \t}\n \n \t/* Increase haves to send on next round */\n \t*haves_to_send = next_flush(1, *haves_to_send);\n \n \treturn ret;\n }\n \n static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n-\t\t\t const struct fetch_pack_args *args,\n+\t\t\t struct fetch_pack_args *args,\n \t\t\t const struct ref *wants, struct oidset *common,\n \t\t\t int *haves_to_send, int *in_vain,\n \t\t\t int sideband_all)\n {\n \tint ret = 0;\n \tstruct strbuf req_buf = STRBUF_INIT;\n \n \tif (server_supports_v2(\"fetch\", 1))\n \t\tpacket_buf_write(&req_buf, \"command=fetch\");\n \tif (server_supports_v2(\"agent\", 0))\n@@ -1133,27 +1130,24 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n \n \t/* Add shallow-info and deepen request */\n \tif (server_supports_feature(\"fetch\", \"shallow\", 0))\n \t\tadd_shallow_requests(&req_buf, args);\n \telse if (is_repository_shallow(the_repository) || args->deepen)\n \t\tdie(_(\"Server does not support shallow requests\"));\n \n \t/* Add filter */\n \tif (server_supports_feature(\"fetch\", \"filter\", 0) &&\n \t args->filter_options.choice) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n+\t\tconst char *spec =\n+\t\t\texpand_list_objects_filter_spec(&args->filter_options);\n \t\tprint_verbose(args, _(\"Server supports filter\"));\n-\t\texpand_list_objects_filter_spec(&args->filter_options,\n-\t\t\t\t\t\t&expanded_filter_spec);\n-\t\tpacket_buf_write(&req_buf, \"filter %s\",\n-\t\t\t\t expanded_filter_spec.buf);\n-\t\tstrbuf_release(&expanded_filter_spec);\n+\t\tpacket_buf_write(&req_buf, \"filter %s\", spec);\n \t} else if (args->filter_options.choice) {\n \t\twarning(\"filtering not recognized by server, ignoring\");\n \t}\n \n \t/* add wants */\n \tadd_wants(args->no_dependents, wants, &req_buf);\n \n \tif (args->no_dependents) {\n \t\tpacket_buf_write(&req_buf, \"done\");\n \t\tret = 1;\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex 9a5677c2c8..38729a7238 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -174,20 +174,29 @@ static int parse_combine_filter(\n \treturn result;\n }\n \n static int allow_unencoded(char ch)\n {\n \tif (ch <= ' ' || ch == '%' || ch == '+')\n \t\treturn 0;\n \treturn !strchr(RESERVED_NON_WS, ch);\n }\n \n+static void filter_spec_append_urlencode(\n+\tstruct list_objects_filter_options *filter, const char *raw)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstrbuf_addstr_urlencode(&buf, raw, allow_unencoded);\n+\ttrace_printf(\"Added to composite filter-spec: %s\\n\", buf.buf);\n+\tstring_list_append(&filter->filter_spec, strbuf_detach(&buf, NULL));\n+}\n+\n /*\n * Changes filter_options into an equivalent LOFC_COMBINE filter options\n * instance. Does not do anything if filter_options is already LOFC_COMBINE.\n */\n static void transform_to_combine_type(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tassert(filter_options->choice);\n \tif (filter_options->choice == LOFC_COMBINE)\n \t\treturn;\n@@ -195,64 +204,59 @@ static void transform_to_combine_type(\n \t\tconst int initial_sub_alloc = 2;\n \t\tstruct list_objects_filter_options *sub_array =\n \t\t\txcalloc(initial_sub_alloc, sizeof(*sub_array));\n \t\tsub_array[0] = *filter_options;\n \t\tmemset(filter_options, 0, sizeof(*filter_options));\n \t\tfilter_options->sub = sub_array;\n \t\tfilter_options->sub_alloc = initial_sub_alloc;\n \t}\n \tfilter_options->sub_nr = 1;\n \tfilter_options->choice = LOFC_COMBINE;\n-\tstrbuf_init(&filter_options->filter_spec, 0);\n-\tstrbuf_addstr(&filter_options->filter_spec, \"combine:\");\n-\tstrbuf_addstr_urlencode(&filter_options->filter_spec,\n-\t\t\t\tfilter_options->sub[0].filter_spec.buf,\n-\t\t\t\tallow_unencoded);\n+\tstring_list_append(&filter_options->filter_spec, xstrdup(\"combine:\"));\n+\tfilter_spec_append_urlencode(\n+\t\tfilter_options,\n+\t\tlist_objects_filter_spec(&filter_options->sub[0]));\n \t/*\n \t * We don't need the filter_spec strings for subfilter specs, only the\n \t * top level.\n \t */\n-\tstrbuf_release(&filter_options->sub[0].filter_spec);\n+\tstring_list_clear(&filter_options->sub[0].filter_spec, /*free_util=*/0);\n }\n \n void list_objects_filter_die_if_populated(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tif (filter_options->choice)\n \t\tdie(_(\"multiple filter-specs cannot be combined\"));\n }\n \n void parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg)\n {\n \tstruct strbuf errbuf = STRBUF_INIT;\n \tint parse_error;\n \n \tif (!filter_options->choice) {\n-\t\tstrbuf_init(&filter_options->filter_spec, 0);\n-\t\tstrbuf_addstr(&filter_options->filter_spec, arg);\n+\t\tstring_list_append(&filter_options->filter_spec, xstrdup(arg));\n \n \t\tparse_error = gently_parse_list_objects_filter(\n \t\t\tfilter_options, arg, &errbuf);\n \t} else {\n \t\t/*\n \t\t * Make filter_options an LOFC_COMBINE spec so we can trivially\n \t\t * add subspecs to it.\n \t\t */\n \t\ttransform_to_combine_type(filter_options);\n \n-\t\tstrbuf_addstr(&filter_options->filter_spec, \"+\");\n-\t\tstrbuf_addstr_urlencode(&filter_options->filter_spec, arg,\n-\t\t\t\t\tallow_unencoded);\n-\t\ttrace_printf(\"Generated composite filter-spec: %s\\n\",\n-\t\t\t filter_options->filter_spec.buf);\n+\t\tstring_list_append(&filter_options->filter_spec, xstrdup(\"+\"));\n+\t\tfilter_spec_append_urlencode(filter_options, arg);\n \t\tALLOC_GROW_BY(filter_options->sub, filter_options->sub_nr, 1,\n \t\t\t filter_options->sub_alloc);\n \n \t\tparse_error = gently_parse_list_objects_filter(\n \t\t\t&filter_options->sub[filter_options->sub_nr - 1], arg,\n \t\t\t&errbuf);\n \t}\n \tif (parse_error)\n \t\tdie(\"%s\", errbuf.buf);\n }\n@@ -262,54 +266,71 @@ int opt_parse_list_objects_filter(const struct option *opt,\n {\n \tstruct list_objects_filter_options *filter_options = opt->value;\n \n \tif (unset || !arg)\n \t\tlist_objects_filter_set_no_filter(filter_options);\n \telse\n \t\tparse_list_objects_filter(filter_options, arg);\n \treturn 0;\n }\n \n-void expand_list_objects_filter_spec(\n-\tconst struct list_objects_filter_options *filter,\n-\tstruct strbuf *expanded_spec)\n+const char *list_objects_filter_spec(struct list_objects_filter_options *filter)\n+{\n+\tif (!filter->filter_spec.nr)\n+\t\tBUG(\"no filter_spec available for this filter\");\n+\tif (filter->filter_spec.nr != 1) {\n+\t\tstruct strbuf concatted = STRBUF_INIT;\n+\t\tstrbuf_add_separated_string_list(\n+\t\t\t&concatted, \"\", &filter->filter_spec);\n+\t\tstring_list_clear(&filter->filter_spec, /*free_util=*/0);\n+\t\tstring_list_append(\n+\t\t\t&filter->filter_spec, strbuf_detach(&concatted, NULL));\n+\t}\n+\n+\treturn filter->filter_spec.items[0].string;\n+}\n+\n+const char *expand_list_objects_filter_spec(\n+\tstruct list_objects_filter_options *filter)\n {\n-\tstrbuf_init(expanded_spec, 0);\n-\tif (filter->choice == LOFC_BLOB_LIMIT)\n-\t\tstrbuf_addf(expanded_spec, \"blob:limit=%lu\",\n+\tif (filter->choice == LOFC_BLOB_LIMIT) {\n+\t\tstruct strbuf expanded_spec;\n+\t\tstrbuf_addf(&expanded_spec, \"blob:limit=%lu\",\n \t\t\t filter->blob_limit_value);\n-\telse if (filter->choice == LOFC_TREE_DEPTH)\n-\t\tstrbuf_addf(expanded_spec, \"tree:%lu\",\n-\t\t\t filter->tree_exclude_depth);\n-\telse\n-\t\tstrbuf_addstr(expanded_spec, filter->filter_spec.buf);\n+\t\tstring_list_clear(&filter->filter_spec, /*free_util=*/0);\n+\t\tstring_list_append(\n+\t\t\t&filter->filter_spec,\n+\t\t\tstrbuf_detach(&expanded_spec, NULL));\n+\t}\n+\n+\treturn list_objects_filter_spec(filter);\n }\n \n void list_objects_filter_release(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tsize_t sub;\n \n \tif (!filter_options)\n \t\treturn;\n-\tstrbuf_release(&filter_options->filter_spec);\n+\tstring_list_clear(&filter_options->filter_spec, /*free_util=*/0);\n \tfree(filter_options->sparse_oid_value);\n \tfree(filter_options->sparse_path_value);\n \tfor (sub = 0; sub < filter_options->sub_nr; sub++)\n \t\tlist_objects_filter_release(&filter_options->sub[sub]);\n \tfree(filter_options->sub);\n \tmemset(filter_options, 0, sizeof(*filter_options));\n }\n \n void partial_clone_register(\n \tconst char *remote,\n-\tconst struct list_objects_filter_options *filter_options)\n+\tstruct list_objects_filter_options *filter_options)\n {\n \t/*\n \t * Record the name of the partial clone remote in the\n \t * config and in the global variable -- the latter is\n \t * used throughout to indicate that partial clone is\n \t * enabled and to expect missing objects.\n \t */\n \tif (repository_format_partial_clone &&\n \t *repository_format_partial_clone &&\n \t strcmp(remote, repository_format_partial_clone))\n@@ -318,35 +339,33 @@ void partial_clone_register(\n \tgit_config_set(\"core.repositoryformatversion\", \"1\");\n \tgit_config_set(\"extensions.partialclone\", remote);\n \n \trepository_format_partial_clone = xstrdup(remote);\n \n \t/*\n \t * Record the initial filter-spec in the config as\n \t * the default for subsequent fetches from this remote.\n \t */\n \tcore_partial_clone_filter_default =\n-\t\txstrdup(filter_options->filter_spec.buf);\n+\t\txstrdup(expand_list_objects_filter_spec(filter_options));\n \tgit_config_set(\"core.partialclonefilter\",\n \t\t core_partial_clone_filter_default);\n }\n \n void partial_clone_get_default_filter_spec(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tstruct strbuf errbuf = STRBUF_INIT;\n \n \t/*\n \t * Parse default value, but silently ignore it if it is invalid.\n \t */\n \tif (!core_partial_clone_filter_default)\n \t\treturn;\n \n-\tif (!filter_options->filter_spec.buf)\n-\t\tstrbuf_init(&filter_options->filter_spec, 0);\n-\tstrbuf_addstr(&filter_options->filter_spec,\n-\t\t core_partial_clone_filter_default);\n+\tstring_list_append(&filter_options->filter_spec,\n+\t\t\t core_partial_clone_filter_default);\n \tgently_parse_list_objects_filter(filter_options,\n \t\t\t\t\t core_partial_clone_filter_default,\n \t\t\t\t\t &errbuf);\n \tstrbuf_release(&errbuf);\n }\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 2c0ce6383a..9b31048ada 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -1,15 +1,15 @@\n #ifndef LIST_OBJECTS_FILTER_OPTIONS_H\n #define LIST_OBJECTS_FILTER_OPTIONS_H\n \n #include \"parse-options.h\"\n-#include \"strbuf.h\"\n+#include \"string-list.h\"\n \n /*\n * The list of defined filters for list-objects.\n */\n enum list_objects_filter_choice {\n \tLOFC_DISABLED = 0,\n \tLOFC_BLOB_NONE,\n \tLOFC_BLOB_LIMIT,\n \tLOFC_TREE_DEPTH,\n \tLOFC_SPARSE_OID,\n@@ -18,22 +18,24 @@ enum list_objects_filter_choice {\n \tLOFC__COUNT /* must be last */\n };\n \n struct list_objects_filter_options {\n \t/*\n \t * 'filter_spec' is the raw argument value given on the command line\n \t * or protocol request. (The part after the \"--keyword=\".) For\n \t * commands that launch filtering sub-processes, or for communication\n \t * over the network, don't use this value; use the result of\n \t * expand_list_objects_filter_spec() instead.\n+\t * To get the raw filter spec given by the user, use the result of\n+\t * list_objects_filter_spec().\n \t */\n-\tstruct strbuf filter_spec;\n+\tstruct string_list filter_spec;\n \n \t/*\n \t * 'choice' is determined by parsing the filter-spec. This indicates\n \t * the filtering algorithm to use.\n \t */\n \tenum list_objects_filter_choice choice;\n \n \t/*\n \t * Choice is LOFC_DISABLED because \"--no-filter\" was requested.\n \t */\n@@ -86,31 +88,33 @@ int opt_parse_list_objects_filter(const struct option *opt,\n \t N_(\"object filtering\"), 0, \\\n \t opt_parse_list_objects_filter }\n \n /*\n * Translates abbreviated numbers in the filter's filter_spec into their\n * fully-expanded forms (e.g., \"limit:blob=1k\" becomes \"limit:blob=1024\").\n *\n * This form should be used instead of the raw filter_spec field when\n * communicating with a remote process or subprocess.\n */\n-void expand_list_objects_filter_spec(\n-\tconst struct list_objects_filter_options *filter,\n-\tstruct strbuf *expanded_spec);\n+const char *expand_list_objects_filter_spec(\n+\tstruct list_objects_filter_options *filter);\n+\n+const char *list_objects_filter_spec(\n+\tstruct list_objects_filter_options *filter);\n \n void list_objects_filter_release(\n \tstruct list_objects_filter_options *filter_options);\n \n static inline void list_objects_filter_set_no_filter(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tlist_objects_filter_release(filter_options);\n \tfilter_options->no_filter = 1;\n }\n \n void partial_clone_register(\n \tconst char *remote,\n-\tconst struct list_objects_filter_options *filter_options);\n+\tstruct list_objects_filter_options *filter_options);\n void partial_clone_get_default_filter_spec(\n \tstruct list_objects_filter_options *filter_options);\n \n #endif /* LIST_OBJECTS_FILTER_OPTIONS_H */\ndiff --git a/transport-helper.c b/transport-helper.c\nindex cec83bd663..d6313ef9f5 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -675,27 +675,23 @@ static int fetch(struct transport *transport,\n \t data->transport_options.check_self_contained_and_connected)\n \t\tset_helper_option(transport, \"check-connectivity\", \"true\");\n \n \tif (transport->cloning)\n \t\tset_helper_option(transport, \"cloning\", \"true\");\n \n \tif (data->transport_options.update_shallow)\n \t\tset_helper_option(transport, \"update-shallow\", \"true\");\n \n \tif (data->transport_options.filter_options.choice) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n-\t\texpand_list_objects_filter_spec(\n-\t\t\t&data->transport_options.filter_options,\n-\t\t\t&expanded_filter_spec);\n-\t\tset_helper_option(transport, \"filter\",\n-\t\t\t\t expanded_filter_spec.buf);\n-\t\tstrbuf_release(&expanded_filter_spec);\n+\t\tconst char *spec = expand_list_objects_filter_spec(\n+\t\t\t&data->transport_options.filter_options);\n+\t\tset_helper_option(transport, \"filter\", spec);\n \t}\n \n \tif (data->transport_options.negotiation_tips)\n \t\twarning(\"Ignoring --negotiation-tip because the protocol does not support it.\");\n \n \tif (data->fetch)\n \t\treturn fetch_with_fetch(transport, nr_heads, to_fetch);\n \n \tif (data->import)\n \t\treturn fetch_with_import(transport, nr_heads, to_fetch);\ndiff --git a/upload-pack.c b/upload-pack.c\nindex ba8c3a1f8e..dda2ac6f44 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -133,32 +133,31 @@ static void create_pack_file(const struct object_array *have_obj,\n \n \targv_array_push(&pack_objects.args, \"--stdout\");\n \tif (shallow_nr)\n \t\targv_array_push(&pack_objects.args, \"--shallow\");\n \tif (!no_progress)\n \t\targv_array_push(&pack_objects.args, \"--progress\");\n \tif (use_ofs_delta)\n \t\targv_array_push(&pack_objects.args, \"--delta-base-offset\");\n \tif (use_include_tag)\n \t\targv_array_push(&pack_objects.args, \"--include-tag\");\n-\tif (filter_options.filter_spec.len) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n-\t\texpand_list_objects_filter_spec(&filter_options,\n-\t\t\t\t\t\t&expanded_filter_spec);\n+\tif (filter_options.choice) {\n+\t\tconst char *spec =\n+\t\t\texpand_list_objects_filter_spec(&filter_options);\n \t\tif (pack_objects.use_shell) {\n \t\t\tstruct strbuf buf = STRBUF_INIT;\n-\t\t\tsq_quote_buf(&buf, expanded_filter_spec.buf);\n+\t\t\tsq_quote_buf(&buf, spec);\n \t\t\targv_array_pushf(&pack_objects.args, \"--filter=%s\", buf.buf);\n \t\t\tstrbuf_release(&buf);\n \t\t} else {\n \t\t\targv_array_pushf(&pack_objects.args, \"--filter=%s\",\n-\t\t\t\t\t expanded_filter_spec.buf);\n+\t\t\t\t\t spec);\n \t\t}\n \t}\n \n \tpack_objects.in = -1;\n \tpack_objects.out = -1;\n \tpack_objects.err = -1;\n \n \tif (start_command(&pack_objects))\n \t\tdie(\"git upload-pack: unable to fork git-pack-objects\");\n \n"},{"id":"377191","messageId":"cover.1560462201.git.matvore@google.com","threadId":"51217","inReplyTo":"20190601003603.90794-1-matvore@google.com","subject":"[PATCH v3 00/10] Filter combination","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-13T21:51:23Z","receivedAt":"2019-06-13T21:51:39Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"It has been a while since a sent a roll-up. Here are the changes since v2:\n\n - Re-use more URL-encoding logic in strbuf.c\n * This was partially achieved by changing the helper function to accept a\n function that will indicate whether some character must be escaped.\n - Re-use more URL-decoding logic in url.c\n - changed the filter_spec strbuf to a string_list to avoid explicit\n initialization\n - Remove logic to \"expand\" tree:#k and tree:#m filter specs since there is no\n server that supports tree:# but does not support tree:#k, as they were\n implemented at the same time.\n\nThanks,\n\nMatthew DeVore (10):\n list-objects-filter: make API easier to use\n list-objects-filter: put omits set in filter struct\n list-objects-filter-options: always supply *errbuf\n list-objects-filter: implement composite filters\n list-objects-filter-options: move error check up\n list-objects-filter-options: make filter_spec a string_list\n strbuf: give URL-encoding API a char predicate fn\n list-objects-filter-options: allow mult. --filter\n list-objects-filter-options: clean up use of ALLOC_GROW\n list-objects-filter-options: make parser void\n\n Documentation/rev-list-options.txt | 16 ++\n builtin/clone.c | 8 +-\n builtin/fetch.c | 9 +-\n builtin/rev-list.c | 6 +-\n cache.h | 22 ++\n credential-store.c | 9 +-\n fetch-pack.c | 20 +-\n http.c | 6 +-\n list-objects-filter-options.c | 267 +++++++++++++++++----\n list-objects-filter-options.h | 57 ++++-\n list-objects-filter.c | 345 +++++++++++++++++++++-------\n list-objects-filter.h | 35 ++-\n list-objects.c | 55 ++---\n strbuf.c | 15 +-\n strbuf.h | 7 +-\n t/t5616-partial-clone.sh | 19 ++\n t/t6112-rev-list-filters-objects.sh | 194 +++++++++++++++-\n transport-helper.c | 10 +-\n transport.c | 1 +\n upload-pack.c | 13 +-\n url.c | 6 +\n url.h | 8 +\n 22 files changed, 879 insertions(+), 249 deletions(-)\n\n-- \n2.21.0\n\n"},{"id":"377192","messageId":"0ab5685d4fa6532afa7d9bfc0a2a6f5441ffc045.1560462201.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1560462201.git.matvore@google.com","subject":"[PATCH v3 01/10] list-objects-filter: make API easier to use","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-13T21:51:24Z","receivedAt":"2019-06-13T21:51:43Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Make the list-objects-filter.h API more opaque and easier to use. This\nprepares for combined filter support, where filters will be created and\nused in a new context.\n\nHelped-by: Jeff Hostetler \nHelped-by: Junio C Hamano \nSigned-off-by: Matthew DeVore \n---\n list-objects-filter.c | 122 +++++++++++++++++++++++++++---------------\n list-objects-filter.h | 35 ++++++------\n list-objects.c | 55 ++++++++-----------\n 3 files changed, 117 insertions(+), 95 deletions(-)\n\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex ee449de3f7..35e0bbe123 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -19,20 +19,34 @@\n * FILTER_SHOWN_BUT_REVISIT -- we set this bit on tree objects\n * that have been shown, but should be revisited if they appear\n * in the traversal (until we mark it SEEN). This is a way to\n * let us silently de-dup calls to show() in the caller. This\n * is subtly different from the \"revision.h:SHOWN\" and the\n * \"sha1-name.c:ONELINE_SEEN\" bits. And also different from\n * the non-de-dup usage in pack-bitmap.c\n */\n #define FILTER_SHOWN_BUT_REVISIT (1<<21)\n \n+struct filter {\n+\tenum list_objects_filter_result (*filter_object_fn)(\n+\t\tstruct repository *r,\n+\t\tenum list_objects_filter_situation filter_situation,\n+\t\tstruct object *obj,\n+\t\tconst char *pathname,\n+\t\tconst char *filename,\n+\t\tvoid *filter_data);\n+\n+\tvoid (*free_fn)(void *filter_data);\n+\n+\tvoid *filter_data;\n+};\n+\n /*\n * A filter for list-objects to omit ALL blobs from the traversal.\n * And to OPTIONALLY collect a list of the omitted OIDs.\n */\n struct filter_blobs_none_data {\n \tstruct oidset *omits;\n };\n \n static enum list_objects_filter_result filter_blobs_none(\n \tstruct repository *r,\n@@ -60,32 +74,31 @@ static enum list_objects_filter_result filter_blobs_none(\n \tcase LOFS_BLOB:\n \t\tassert(obj->type == OBJ_BLOB);\n \t\tassert((obj->flags & SEEN) == 0);\n \n \t\tif (filter_data->omits)\n \t\t\toidset_insert(filter_data->omits, &obj->oid);\n \t\treturn LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n \t}\n }\n \n-static void *filter_blobs_none__init(\n+static void filter_blobs_none__init(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn)\n+\tstruct filter *filter)\n {\n \tstruct filter_blobs_none_data *d = xcalloc(1, sizeof(*d));\n \td->omits = omitted;\n \n-\t*filter_fn = filter_blobs_none;\n-\t*filter_free_fn = free;\n-\treturn d;\n+\tfilter->filter_data = d;\n+\tfilter->filter_object_fn = filter_blobs_none;\n+\tfilter->free_fn = free;\n }\n \n /*\n * A filter for list-objects to omit ALL trees and blobs from the traversal.\n * Can OPTIONALLY collect a list of the omitted OIDs.\n */\n struct filter_trees_depth_data {\n \tstruct oidset *omits;\n \n \t/*\n@@ -194,35 +207,34 @@ static enum list_objects_filter_result filter_trees_depth(\n }\n \n static void filter_trees_free(void *filter_data) {\n \tstruct filter_trees_depth_data *d = filter_data;\n \tif (!d)\n \t\treturn;\n \toidmap_free(&d->seen_at_depth, 1);\n \tfree(d);\n }\n \n-static void *filter_trees_depth__init(\n+static void filter_trees_depth__init(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn)\n+\tstruct filter *filter)\n {\n \tstruct filter_trees_depth_data *d = xcalloc(1, sizeof(*d));\n \td->omits = omitted;\n \toidmap_init(&d->seen_at_depth, 0);\n \td->exclude_depth = filter_options->tree_exclude_depth;\n \td->current_depth = 0;\n \n-\t*filter_fn = filter_trees_depth;\n-\t*filter_free_fn = filter_trees_free;\n-\treturn d;\n+\tfilter->filter_data = d;\n+\tfilter->filter_object_fn = filter_trees_depth;\n+\tfilter->free_fn = filter_trees_free;\n }\n \n /*\n * A filter for list-objects to omit large blobs.\n * And to OPTIONALLY collect a list of the omitted OIDs.\n */\n struct filter_blobs_limit_data {\n \tstruct oidset *omits;\n \tunsigned long max_bytes;\n };\n@@ -274,33 +286,32 @@ static enum list_objects_filter_result filter_blobs_limit(\n \t\t\toidset_insert(filter_data->omits, &obj->oid);\n \t\treturn LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n \t}\n \n include_it:\n \tif (filter_data->omits)\n \t\toidset_remove(filter_data->omits, &obj->oid);\n \treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n }\n \n-static void *filter_blobs_limit__init(\n+static void filter_blobs_limit__init(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn)\n+\tstruct filter *filter)\n {\n \tstruct filter_blobs_limit_data *d = xcalloc(1, sizeof(*d));\n \td->omits = omitted;\n \td->max_bytes = filter_options->blob_limit_value;\n \n-\t*filter_fn = filter_blobs_limit;\n-\t*filter_free_fn = free;\n-\treturn d;\n+\tfilter->filter_data = d;\n+\tfilter->filter_object_fn = filter_blobs_limit;\n+\tfilter->free_fn = free;\n }\n \n /*\n * A filter driven by a sparse-checkout specification to only\n * include blobs that a sparse checkout would populate.\n *\n * The sparse-checkout spec can be loaded from a blob with the\n * given OID or from a local pathname. We allow an OID because\n * the repo may be bare or we may be doing the filtering on the\n * server.\n@@ -450,92 +461,117 @@ static enum list_objects_filter_result filter_sparse(\n }\n \n \n static void filter_sparse_free(void *filter_data)\n {\n \tstruct filter_sparse_data *d = filter_data;\n \t/* TODO free contents of 'd' */\n \tfree(d);\n }\n \n-static void *filter_sparse_oid__init(\n+static void filter_sparse_oid__init(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn)\n+\tstruct filter *filter)\n {\n \tstruct filter_sparse_data *d = xcalloc(1, sizeof(*d));\n \td->omits = omitted;\n \tif (add_excludes_from_blob_to_list(filter_options->sparse_oid_value,\n \t\t\t\t\t NULL, 0, &d->el) < 0)\n \t\tdie(\"could not load filter specification\");\n \n \tALLOC_GROW(d->array_frame, d->nr + 1, d->alloc);\n \td->array_frame[d->nr].defval = 0; /* default to include */\n \td->array_frame[d->nr].child_prov_omit = 0;\n \n-\t*filter_fn = filter_sparse;\n-\t*filter_free_fn = filter_sparse_free;\n-\treturn d;\n+\tfilter->filter_data = d;\n+\tfilter->filter_object_fn = filter_sparse;\n+\tfilter->free_fn = filter_sparse_free;\n }\n \n-static void *filter_sparse_path__init(\n+static void filter_sparse_path__init(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn)\n+\tstruct filter *filter)\n {\n \tstruct filter_sparse_data *d = xcalloc(1, sizeof(*d));\n \td->omits = omitted;\n \tif (add_excludes_from_file_to_list(filter_options->sparse_path_value,\n \t\t\t\t\t NULL, 0, &d->el, NULL) < 0)\n \t\tdie(\"could not load filter specification\");\n \n \tALLOC_GROW(d->array_frame, d->nr + 1, d->alloc);\n \td->array_frame[d->nr].defval = 0; /* default to include */\n \td->array_frame[d->nr].child_prov_omit = 0;\n \n-\t*filter_fn = filter_sparse;\n-\t*filter_free_fn = filter_sparse_free;\n-\treturn d;\n+\tfilter->filter_data = d;\n+\tfilter->filter_object_fn = filter_sparse;\n+\tfilter->free_fn = filter_sparse_free;\n }\n \n-typedef void *(*filter_init_fn)(\n+typedef void (*filter_init_fn)(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn);\n+\tstruct filter *filter);\n \n /*\n * Must match \"enum list_objects_filter_choice\".\n */\n static filter_init_fn s_filters[] = {\n \tNULL,\n \tfilter_blobs_none__init,\n \tfilter_blobs_limit__init,\n \tfilter_trees_depth__init,\n \tfilter_sparse_oid__init,\n \tfilter_sparse_path__init,\n };\n \n-void *list_objects_filter__init(\n+struct filter *list_objects_filter__init(\n \tstruct oidset *omitted,\n-\tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn)\n+\tstruct list_objects_filter_options *filter_options)\n {\n+\tstruct filter *filter;\n \tfilter_init_fn init_fn;\n \n \tassert((sizeof(s_filters) / sizeof(s_filters[0])) == LOFC__COUNT);\n \n \tif (filter_options->choice >= LOFC__COUNT)\n \t\tBUG(\"invalid list-objects filter choice: %d\",\n \t\t filter_options->choice);\n \n \tinit_fn = s_filters[filter_options->choice];\n-\tif (init_fn)\n-\t\treturn init_fn(omitted, filter_options,\n-\t\t\t filter_fn, filter_free_fn);\n-\t*filter_fn = NULL;\n-\t*filter_free_fn = NULL;\n-\treturn NULL;\n+\tif (!init_fn)\n+\t\treturn NULL;\n+\n+\tfilter = xcalloc(1, sizeof(*filter));\n+\tinit_fn(omitted, filter_options, filter);\n+\treturn filter;\n+}\n+\n+enum list_objects_filter_result list_objects_filter__filter_object(\n+\tstruct repository *r,\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tstruct filter *filter)\n+{\n+\tif (filter && (obj->flags & NOT_USER_GIVEN))\n+\t\treturn filter->filter_object_fn(r, filter_situation, obj,\n+\t\t\t\t\t\tpathname, filename,\n+\t\t\t\t\t\tfilter->filter_data);\n+\t/*\n+\t * No filter is active or user gave object explicitly. Choose default\n+\t * behavior based on filter situation.\n+\t */\n+\tif (filter_situation == LOFS_END_TREE)\n+\t\treturn 0;\n+\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+}\n+\n+void list_objects_filter__free(struct filter *filter)\n+{\n+\tif (!filter)\n+\t\treturn;\n+\tfilter->free_fn(filter->filter_data);\n+\tfree(filter);\n }\ndiff --git a/list-objects-filter.h b/list-objects-filter.h\nindex 1d45a4ad57..6908954266 100644\n--- a/list-objects-filter.h\n+++ b/list-objects-filter.h\n@@ -53,37 +53,34 @@ enum list_objects_filter_result {\n \tLOFR_DO_SHOW = 1<<1,\n \tLOFR_SKIP_TREE = 1<<2,\n };\n \n enum list_objects_filter_situation {\n \tLOFS_BEGIN_TREE,\n \tLOFS_END_TREE,\n \tLOFS_BLOB\n };\n \n-typedef enum list_objects_filter_result (*filter_object_fn)(\n+struct filter;\n+\n+/* Constructor for the set of defined list-objects filters. */\n+struct filter *list_objects_filter__init(\n+\tstruct oidset *omitted,\n+\tstruct list_objects_filter_options *filter_options);\n+\n+/*\n+ * Lets `filter` decide how to handle the `obj`. If `filter` is NULL, this\n+ * function behaves as expected if no filter is configured: all objects are\n+ * included.\n+ */\n+enum list_objects_filter_result list_objects_filter__filter_object(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n-\tvoid *filter_data);\n-\n-typedef void (*filter_free_fn)(void *filter_data);\n+\tstruct filter *filter);\n \n-/*\n- * Constructor for the set of defined list-objects filters.\n- * Returns a generic \"void *filter_data\".\n- *\n- * The returned \"filter_fn\" will be used by traverse_commit_list()\n- * to filter the results.\n- *\n- * The returned \"filter_free_fn\" is a destructor for the\n- * filter_data.\n- */\n-void *list_objects_filter__init(\n-\tstruct oidset *omitted,\n-\tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn);\n+/* Destroys `filter`. Does nothing if `filter` is null. */\n+void list_objects_filter__free(struct filter *filter);\n \n #endif /* LIST_OBJECTS_FILTER_H */\ndiff --git a/list-objects.c b/list-objects.c\nindex b5651ddd5b..9307d91fb3 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -11,32 +11,31 @@\n #include \"list-objects-filter-options.h\"\n #include \"packfile.h\"\n #include \"object-store.h\"\n #include \"trace.h\"\n \n struct traversal_context {\n \tstruct rev_info *revs;\n \tshow_object_fn show_object;\n \tshow_commit_fn show_commit;\n \tvoid *show_data;\n-\tfilter_object_fn filter_fn;\n-\tvoid *filter_data;\n+\tstruct filter *filter;\n };\n \n static void process_blob(struct traversal_context *ctx,\n \t\t\t struct blob *blob,\n \t\t\t struct strbuf *path,\n \t\t\t const char *name)\n {\n \tstruct object *obj = &blob->object;\n \tsize_t pathlen;\n-\tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\tenum list_objects_filter_result r;\n \n \tif (!ctx->revs->blob_objects)\n \t\treturn;\n \tif (!obj)\n \t\tdie(\"bad blob object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n \n \t/*\n \t * Pre-filter known-missing objects when explicitly requested.\n@@ -47,25 +46,24 @@ static void process_blob(struct traversal_context *ctx,\n \t * may cause the actual filter to report an incomplete list\n \t * of missing objects.\n \t */\n \tif (ctx->revs->exclude_promisor_objects &&\n \t !has_object_file(&obj->oid) &&\n \t is_promisor_object(&obj->oid))\n \t\treturn;\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n-\t\tr = ctx->filter_fn(ctx->revs->repo,\n-\t\t\t\t LOFS_BLOB, obj,\n-\t\t\t\t path->buf, &path->buf[pathlen],\n-\t\t\t\t ctx->filter_data);\n+\tr = list_objects_filter__filter_object(ctx->revs->repo,\n+\t\t\t\t\t LOFS_BLOB, obj,\n+\t\t\t\t\t path->buf, &path->buf[pathlen],\n+\t\t\t\t\t ctx->filter);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n \t\tctx->show_object(obj, path->buf, ctx->show_data);\n \tstrbuf_setlen(path, pathlen);\n }\n \n /*\n * Processing a gitlink entry currently does nothing, since\n * we do not recurse into the subproject.\n@@ -150,21 +148,21 @@ static void process_tree_contents(struct traversal_context *ctx,\n }\n \n static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n \t\t\t const char *name)\n {\n \tstruct object *obj = &tree->object;\n \tstruct rev_info *revs = ctx->revs;\n \tint baselen = base->len;\n-\tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\tenum list_objects_filter_result r;\n \tint failed_parse;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n \tif (!obj)\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n \n \tfailed_parse = parse_tree_gently(tree, 1);\n@@ -179,47 +177,44 @@ static void process_tree(struct traversal_context *ctx,\n \t\t */\n \t\tif (revs->exclude_promisor_objects &&\n \t\t is_promisor_object(&obj->oid))\n \t\t\treturn;\n \n \t\tif (!revs->do_not_die_on_missing_tree)\n \t\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n-\t\tr = ctx->filter_fn(ctx->revs->repo,\n-\t\t\t\t LOFS_BEGIN_TREE, obj,\n-\t\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t\t ctx->filter_data);\n+\tr = list_objects_filter__filter_object(ctx->revs->repo,\n+\t\t\t\t\t LOFS_BEGIN_TREE, obj,\n+\t\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t\t ctx->filter);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n \t\tctx->show_object(obj, base->buf, ctx->show_data);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n \tif (r & LOFR_SKIP_TREE)\n \t\ttrace_printf(\"Skipping contents of tree %s...\\n\", base->buf);\n \telse if (!failed_parse)\n \t\tprocess_tree_contents(ctx, tree, base);\n \n-\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) {\n-\t\tr = ctx->filter_fn(ctx->revs->repo,\n-\t\t\t\t LOFS_END_TREE, obj,\n-\t\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t\t ctx->filter_data);\n-\t\tif (r & LOFR_MARK_SEEN)\n-\t\t\tobj->flags |= SEEN;\n-\t\tif (r & LOFR_DO_SHOW)\n-\t\t\tctx->show_object(obj, base->buf, ctx->show_data);\n-\t}\n+\tr = list_objects_filter__filter_object(ctx->revs->repo,\n+\t\t\t\t\t LOFS_END_TREE, obj,\n+\t\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t\t ctx->filter);\n+\tif (r & LOFR_MARK_SEEN)\n+\t\tobj->flags |= SEEN;\n+\tif (r & LOFR_DO_SHOW)\n+\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \n \tstrbuf_setlen(base, baselen);\n \tfree_tree_buffer(tree);\n }\n \n static void mark_edge_parents_uninteresting(struct commit *commit,\n \t\t\t\t\t struct rev_info *revs,\n \t\t\t\t\t show_edge_fn show_edge)\n {\n \tstruct commit_list *parents;\n@@ -395,38 +390,32 @@ static void do_traverse(struct traversal_context *ctx)\n void traverse_commit_list(struct rev_info *revs,\n \t\t\t show_commit_fn show_commit,\n \t\t\t show_object_fn show_object,\n \t\t\t void *show_data)\n {\n \tstruct traversal_context ctx;\n \tctx.revs = revs;\n \tctx.show_commit = show_commit;\n \tctx.show_object = show_object;\n \tctx.show_data = show_data;\n-\tctx.filter_fn = NULL;\n-\tctx.filter_data = NULL;\n+\tctx.filter = NULL;\n \tdo_traverse(&ctx);\n }\n \n void traverse_commit_list_filtered(\n \tstruct list_objects_filter_options *filter_options,\n \tstruct rev_info *revs,\n \tshow_commit_fn show_commit,\n \tshow_object_fn show_object,\n \tvoid *show_data,\n \tstruct oidset *omitted)\n {\n \tstruct traversal_context ctx;\n-\tfilter_free_fn filter_free_fn = NULL;\n \n \tctx.revs = revs;\n \tctx.show_object = show_object;\n \tctx.show_commit = show_commit;\n \tctx.show_data = show_data;\n-\tctx.filter_fn = NULL;\n-\n-\tctx.filter_data = list_objects_filter__init(omitted, filter_options,\n-\t\t\t\t\t\t &ctx.filter_fn, &filter_free_fn);\n+\tctx.filter = list_objects_filter__init(omitted, filter_options);\n \tdo_traverse(&ctx);\n-\tif (ctx.filter_data && filter_free_fn)\n-\t\tfilter_free_fn(ctx.filter_data);\n+\tlist_objects_filter__free(ctx.filter);\n }\n-- \n2.21.0\n\n"},{"id":"377193","messageId":"5e1792a67ab00b3373ea689e35db4704e387fbe2.1560462201.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1560462201.git.matvore@google.com","subject":"[PATCH v3 02/10] list-objects-filter: put omits set in filter struct","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-13T21:51:25Z","receivedAt":"2019-06-13T21:51:46Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"The oidset *omits pointer must be accessed by the combine filter in a\ntype-agnostic way once the graph traversal is over. Store that pointer\nin the general `filter` struct. This will be used in a follow-up patch\nto implement the combine filter.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects-filter.c | 70 ++++++++++++++++---------------------------\n 1 file changed, 26 insertions(+), 44 deletions(-)\n\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex 35e0bbe123..57bbf6ec1c 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -26,88 +26,76 @@\n */\n #define FILTER_SHOWN_BUT_REVISIT (1<<21)\n \n struct filter {\n \tenum list_objects_filter_result (*filter_object_fn)(\n \t\tstruct repository *r,\n \t\tenum list_objects_filter_situation filter_situation,\n \t\tstruct object *obj,\n \t\tconst char *pathname,\n \t\tconst char *filename,\n+\t\tstruct oidset *omits,\n \t\tvoid *filter_data);\n \n \tvoid (*free_fn)(void *filter_data);\n \n \tvoid *filter_data;\n-};\n \n-/*\n- * A filter for list-objects to omit ALL blobs from the traversal.\n- * And to OPTIONALLY collect a list of the omitted OIDs.\n- */\n-struct filter_blobs_none_data {\n+\t/* If non-NULL, the filter collects a list of the omitted OIDs here. */\n \tstruct oidset *omits;\n };\n \n static enum list_objects_filter_result filter_blobs_none(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n+\tstruct oidset *omits,\n \tvoid *filter_data_)\n {\n-\tstruct filter_blobs_none_data *filter_data = filter_data_;\n-\n \tswitch (filter_situation) {\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\t/* always include all tree objects */\n \t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \n \tcase LOFS_END_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\treturn LOFR_ZERO;\n \n \tcase LOFS_BLOB:\n \t\tassert(obj->type == OBJ_BLOB);\n \t\tassert((obj->flags & SEEN) == 0);\n \n-\t\tif (filter_data->omits)\n-\t\t\toidset_insert(filter_data->omits, &obj->oid);\n+\t\tif (omits)\n+\t\t\toidset_insert(omits, &obj->oid);\n \t\treturn LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n \t}\n }\n \n static void filter_blobs_none__init(\n-\tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter)\n {\n-\tstruct filter_blobs_none_data *d = xcalloc(1, sizeof(*d));\n-\td->omits = omitted;\n-\n-\tfilter->filter_data = d;\n \tfilter->filter_object_fn = filter_blobs_none;\n \tfilter->free_fn = free;\n }\n \n /*\n * A filter for list-objects to omit ALL trees and blobs from the traversal.\n * Can OPTIONALLY collect a list of the omitted OIDs.\n */\n struct filter_trees_depth_data {\n-\tstruct oidset *omits;\n-\n \t/*\n \t * Maps trees to the minimum depth at which they were seen. It is not\n \t * necessary to re-traverse a tree at deeper or equal depths than it has\n \t * already been traversed.\n \t *\n \t * We can't use LOFR_MARK_SEEN for tree objects since this will prevent\n \t * it from being traversed at shallower depths.\n \t */\n \tstruct oidmap seen_at_depth;\n \n@@ -116,38 +104,39 @@ struct filter_trees_depth_data {\n };\n \n struct seen_map_entry {\n \tstruct oidmap_entry base;\n \tsize_t depth;\n };\n \n /* Returns 1 if the oid was in the omits set before it was invoked. */\n static int filter_trees_update_omits(\n \tstruct object *obj,\n-\tstruct filter_trees_depth_data *filter_data,\n+\tstruct oidset *omits,\n \tint include_it)\n {\n-\tif (!filter_data->omits)\n+\tif (!omits)\n \t\treturn 0;\n \n \tif (include_it)\n-\t\treturn oidset_remove(filter_data->omits, &obj->oid);\n+\t\treturn oidset_remove(omits, &obj->oid);\n \telse\n-\t\treturn oidset_insert(filter_data->omits, &obj->oid);\n+\t\treturn oidset_insert(omits, &obj->oid);\n }\n \n static enum list_objects_filter_result filter_trees_depth(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n+\tstruct oidset *omits,\n \tvoid *filter_data_)\n {\n \tstruct filter_trees_depth_data *filter_data = filter_data_;\n \tstruct seen_map_entry *seen_info;\n \tint include_it = filter_data->current_depth <\n \t\tfilter_data->exclude_depth;\n \tint filter_res;\n \tint already_seen;\n \n \t/*\n@@ -158,47 +147,47 @@ static enum list_objects_filter_result filter_trees_depth(\n \tswitch (filter_situation) {\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n \tcase LOFS_END_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\tfilter_data->current_depth--;\n \t\treturn LOFR_ZERO;\n \n \tcase LOFS_BLOB:\n-\t\tfilter_trees_update_omits(obj, filter_data, include_it);\n+\t\tfilter_trees_update_omits(obj, omits, include_it);\n \t\treturn include_it ? LOFR_MARK_SEEN | LOFR_DO_SHOW : LOFR_ZERO;\n \n \tcase LOFS_BEGIN_TREE:\n \t\tseen_info = oidmap_get(\n \t\t\t&filter_data->seen_at_depth, &obj->oid);\n \t\tif (!seen_info) {\n \t\t\tseen_info = xcalloc(1, sizeof(*seen_info));\n \t\t\toidcpy(&seen_info->base.oid, &obj->oid);\n \t\t\tseen_info->depth = filter_data->current_depth;\n \t\t\toidmap_put(&filter_data->seen_at_depth, seen_info);\n \t\t\talready_seen = 0;\n \t\t} else {\n \t\t\talready_seen =\n \t\t\t\tfilter_data->current_depth >= seen_info->depth;\n \t\t}\n \n \t\tif (already_seen) {\n \t\t\tfilter_res = LOFR_SKIP_TREE;\n \t\t} else {\n \t\t\tint been_omitted = filter_trees_update_omits(\n-\t\t\t\tobj, filter_data, include_it);\n+\t\t\t\tobj, omits, include_it);\n \t\t\tseen_info->depth = filter_data->current_depth;\n \n \t\t\tif (include_it)\n \t\t\t\tfilter_res = LOFR_DO_SHOW;\n-\t\t\telse if (filter_data->omits && !been_omitted)\n+\t\t\telse if (omits && !been_omitted)\n \t\t\t\t/*\n \t\t\t\t * Must update omit information of children\n \t\t\t\t * recursively; they have not been omitted yet.\n \t\t\t\t */\n \t\t\t\tfilter_res = LOFR_ZERO;\n \t\t\telse\n \t\t\t\tfilter_res = LOFR_SKIP_TREE;\n \t\t}\n \n \t\tfilter_data->current_depth++;\n@@ -208,50 +197,48 @@ static enum list_objects_filter_result filter_trees_depth(\n \n static void filter_trees_free(void *filter_data) {\n \tstruct filter_trees_depth_data *d = filter_data;\n \tif (!d)\n \t\treturn;\n \toidmap_free(&d->seen_at_depth, 1);\n \tfree(d);\n }\n \n static void filter_trees_depth__init(\n-\tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter)\n {\n \tstruct filter_trees_depth_data *d = xcalloc(1, sizeof(*d));\n-\td->omits = omitted;\n \toidmap_init(&d->seen_at_depth, 0);\n \td->exclude_depth = filter_options->tree_exclude_depth;\n \td->current_depth = 0;\n \n \tfilter->filter_data = d;\n \tfilter->filter_object_fn = filter_trees_depth;\n \tfilter->free_fn = filter_trees_free;\n }\n \n /*\n * A filter for list-objects to omit large blobs.\n * And to OPTIONALLY collect a list of the omitted OIDs.\n */\n struct filter_blobs_limit_data {\n-\tstruct oidset *omits;\n \tunsigned long max_bytes;\n };\n \n static enum list_objects_filter_result filter_blobs_limit(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n+\tstruct oidset *omits,\n \tvoid *filter_data_)\n {\n \tstruct filter_blobs_limit_data *filter_data = filter_data_;\n \tunsigned long object_length;\n \tenum object_type t;\n \n \tswitch (filter_situation) {\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n@@ -275,38 +262,36 @@ static enum list_objects_filter_result filter_blobs_limit(\n \t\t\t * apply the size filter criteria. Be conservative\n \t\t\t * and force show it (and let the caller deal with\n \t\t\t * the ambiguity).\n \t\t\t */\n \t\t\tgoto include_it;\n \t\t}\n \n \t\tif (object_length < filter_data->max_bytes)\n \t\t\tgoto include_it;\n \n-\t\tif (filter_data->omits)\n-\t\t\toidset_insert(filter_data->omits, &obj->oid);\n+\t\tif (omits)\n+\t\t\toidset_insert(omits, &obj->oid);\n \t\treturn LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n \t}\n \n include_it:\n-\tif (filter_data->omits)\n-\t\toidset_remove(filter_data->omits, &obj->oid);\n+\tif (omits)\n+\t\toidset_remove(omits, &obj->oid);\n \treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n }\n \n static void filter_blobs_limit__init(\n-\tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter)\n {\n \tstruct filter_blobs_limit_data *d = xcalloc(1, sizeof(*d));\n-\td->omits = omitted;\n \td->max_bytes = filter_options->blob_limit_value;\n \n \tfilter->filter_data = d;\n \tfilter->filter_object_fn = filter_blobs_limit;\n \tfilter->free_fn = free;\n }\n \n /*\n * A filter driven by a sparse-checkout specification to only\n * include blobs that a sparse checkout would populate.\n@@ -330,33 +315,33 @@ struct frame {\n \t * omitted objects.\n \t *\n \t * 0 if everything (recursively) contained in this directory\n \t * has been explicitly included (SHOWN) in the result and\n \t * the directory may be short-cut later in the traversal.\n \t */\n \tunsigned child_prov_omit : 1;\n };\n \n struct filter_sparse_data {\n-\tstruct oidset *omits;\n \tstruct exclude_list el;\n \n \tsize_t nr, alloc;\n \tstruct frame *array_frame;\n };\n \n static enum list_objects_filter_result filter_sparse(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n+\tstruct oidset *omits,\n \tvoid *filter_data_)\n {\n \tstruct filter_sparse_data *filter_data = filter_data_;\n \tint val, dtype;\n \tstruct frame *frame;\n \n \tswitch (filter_situation) {\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n@@ -425,98 +410,93 @@ static enum list_objects_filter_result filter_sparse(\n \n \t\tframe = &filter_data->array_frame[filter_data->nr];\n \n \t\tdtype = DT_REG;\n \t\tval = is_excluded_from_list(pathname, strlen(pathname),\n \t\t\t\t\t filename, &dtype, &filter_data->el,\n \t\t\t\t\t r->index);\n \t\tif (val < 0)\n \t\t\tval = frame->defval;\n \t\tif (val > 0) {\n-\t\t\tif (filter_data->omits)\n-\t\t\t\toidset_remove(filter_data->omits, &obj->oid);\n+\t\t\tif (omits)\n+\t\t\t\toidset_remove(omits, &obj->oid);\n \t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \t\t}\n \n \t\t/*\n \t\t * Provisionally omit it. We've already established that\n \t\t * this pathname is not in the sparse-checkout specification\n \t\t * with the CURRENT pathname, so we *WANT* to omit this blob.\n \t\t *\n \t\t * However, a pathname elsewhere in the tree may also\n \t\t * reference this same blob, so we cannot reject it yet.\n \t\t * Leave the LOFR_ bits unset so that if the blob appears\n \t\t * again in the traversal, we will be asked again.\n \t\t */\n-\t\tif (filter_data->omits)\n-\t\t\toidset_insert(filter_data->omits, &obj->oid);\n+\t\tif (omits)\n+\t\t\toidset_insert(omits, &obj->oid);\n \n \t\t/*\n \t\t * Remember that at least 1 blob in this tree was\n \t\t * provisionally omitted. This prevents us from short\n \t\t * cutting the tree in future iterations.\n \t\t */\n \t\tframe->child_prov_omit = 1;\n \t\treturn LOFR_ZERO;\n \t}\n }\n \n \n static void filter_sparse_free(void *filter_data)\n {\n \tstruct filter_sparse_data *d = filter_data;\n \t/* TODO free contents of 'd' */\n \tfree(d);\n }\n \n static void filter_sparse_oid__init(\n-\tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter)\n {\n \tstruct filter_sparse_data *d = xcalloc(1, sizeof(*d));\n-\td->omits = omitted;\n \tif (add_excludes_from_blob_to_list(filter_options->sparse_oid_value,\n \t\t\t\t\t NULL, 0, &d->el) < 0)\n \t\tdie(\"could not load filter specification\");\n \n \tALLOC_GROW(d->array_frame, d->nr + 1, d->alloc);\n \td->array_frame[d->nr].defval = 0; /* default to include */\n \td->array_frame[d->nr].child_prov_omit = 0;\n \n \tfilter->filter_data = d;\n \tfilter->filter_object_fn = filter_sparse;\n \tfilter->free_fn = filter_sparse_free;\n }\n \n static void filter_sparse_path__init(\n-\tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter)\n {\n \tstruct filter_sparse_data *d = xcalloc(1, sizeof(*d));\n-\td->omits = omitted;\n \tif (add_excludes_from_file_to_list(filter_options->sparse_path_value,\n \t\t\t\t\t NULL, 0, &d->el, NULL) < 0)\n \t\tdie(\"could not load filter specification\");\n \n \tALLOC_GROW(d->array_frame, d->nr + 1, d->alloc);\n \td->array_frame[d->nr].defval = 0; /* default to include */\n \td->array_frame[d->nr].child_prov_omit = 0;\n \n \tfilter->filter_data = d;\n \tfilter->filter_object_fn = filter_sparse;\n \tfilter->free_fn = filter_sparse_free;\n }\n \n typedef void (*filter_init_fn)(\n-\tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter);\n \n /*\n * Must match \"enum list_objects_filter_choice\".\n */\n static filter_init_fn s_filters[] = {\n \tNULL,\n \tfilter_blobs_none__init,\n \tfilter_blobs_limit__init,\n@@ -536,35 +516,37 @@ struct filter *list_objects_filter__init(\n \n \tif (filter_options->choice >= LOFC__COUNT)\n \t\tBUG(\"invalid list-objects filter choice: %d\",\n \t\t filter_options->choice);\n \n \tinit_fn = s_filters[filter_options->choice];\n \tif (!init_fn)\n \t\treturn NULL;\n \n \tfilter = xcalloc(1, sizeof(*filter));\n-\tinit_fn(omitted, filter_options, filter);\n+\tfilter->omits = omitted;\n+\tinit_fn(filter_options, filter);\n \treturn filter;\n }\n \n enum list_objects_filter_result list_objects_filter__filter_object(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n \tstruct filter *filter)\n {\n \tif (filter && (obj->flags & NOT_USER_GIVEN))\n \t\treturn filter->filter_object_fn(r, filter_situation, obj,\n \t\t\t\t\t\tpathname, filename,\n+\t\t\t\t\t\tfilter->omits,\n \t\t\t\t\t\tfilter->filter_data);\n \t/*\n \t * No filter is active or user gave object explicitly. Choose default\n \t * behavior based on filter situation.\n \t */\n \tif (filter_situation == LOFS_END_TREE)\n \t\treturn 0;\n \treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n }\n \n-- \n2.21.0\n\n"},{"id":"377194","messageId":"a0f5671d4865f5325b7a75de57b30a5ac4e611a8.1560462201.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1560462201.git.matvore@google.com","subject":"[PATCH v3 03/10] list-objects-filter-options: always supply *errbuf","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-13T21:51:26Z","receivedAt":"2019-06-13T21:51:48Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Making errbuf an optional argument complicates error reporting. Fix this\nby making all callers supply an errbuf, even if they may ignore it. This\nwill be important in follow-up patches where the filter-spec parsing has\nmore pitfalls and possible errors.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects-filter-options.c | 21 ++++++++-------------\n 1 file changed, 8 insertions(+), 13 deletions(-)\n\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex c0036f7378..aef24ddae3 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -23,47 +23,40 @@\n * convenience of the current command.\n */\n static int gently_parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg,\n \tstruct strbuf *errbuf)\n {\n \tconst char *v0;\n \n \tif (filter_options->choice) {\n-\t\tif (errbuf) {\n-\t\t\tstrbuf_addstr(\n-\t\t\t\terrbuf,\n-\t\t\t\t_(\"multiple filter-specs cannot be combined\"));\n-\t\t}\n+\t\tstrbuf_addstr(\n+\t\t\terrbuf, _(\"multiple filter-specs cannot be combined\"));\n \t\treturn 1;\n \t}\n \n \tfilter_options->filter_spec = strdup(arg);\n \n \tif (!strcmp(arg, \"blob:none\")) {\n \t\tfilter_options->choice = LOFC_BLOB_NONE;\n \t\treturn 0;\n \n \t} else if (skip_prefix(arg, \"blob:limit=\", &v0)) {\n \t\tif (git_parse_ulong(v0, &filter_options->blob_limit_value)) {\n \t\t\tfilter_options->choice = LOFC_BLOB_LIMIT;\n \t\t\treturn 0;\n \t\t}\n \n \t} else if (skip_prefix(arg, \"tree:\", &v0)) {\n \t\tif (!git_parse_ulong(v0, &filter_options->tree_exclude_depth)) {\n-\t\t\tif (errbuf) {\n-\t\t\t\tstrbuf_addstr(\n-\t\t\t\t\terrbuf,\n-\t\t\t\t\t_(\"expected 'tree:'\"));\n-\t\t\t}\n+\t\t\tstrbuf_addstr(errbuf, _(\"expected 'tree:'\"));\n \t\t\treturn 1;\n \t\t}\n \t\tfilter_options->choice = LOFC_TREE_DEPTH;\n \t\treturn 0;\n \n \t} else if (skip_prefix(arg, \"sparse:oid=\", &v0)) {\n \t\tstruct object_context oc;\n \t\tstruct object_id sparse_oid;\n \n \t\t/*\n@@ -80,22 +73,21 @@ static int gently_parse_list_objects_filter(\n \t} else if (skip_prefix(arg, \"sparse:path=\", &v0)) {\n \t\tfilter_options->choice = LOFC_SPARSE_PATH;\n \t\tfilter_options->sparse_path_value = strdup(v0);\n \t\treturn 0;\n \t}\n \t/*\n \t * Please update _git_fetch() in git-completion.bash when you\n \t * add new filters\n \t */\n \n-\tif (errbuf)\n-\t\tstrbuf_addf(errbuf, \"invalid filter-spec '%s'\", arg);\n+\tstrbuf_addf(errbuf, \"invalid filter-spec '%s'\", arg);\n \n \tmemset(filter_options, 0, sizeof(*filter_options));\n \treturn 1;\n }\n \n int parse_list_objects_filter(struct list_objects_filter_options *filter_options,\n \t\t\t const char *arg)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tif (gently_parse_list_objects_filter(filter_options, arg, &buf))\n@@ -166,19 +158,22 @@ void partial_clone_register(\n \t */\n \tcore_partial_clone_filter_default =\n \t\txstrdup(filter_options->filter_spec);\n \tgit_config_set(\"core.partialclonefilter\",\n \t\t core_partial_clone_filter_default);\n }\n \n void partial_clone_get_default_filter_spec(\n \tstruct list_objects_filter_options *filter_options)\n {\n+\tstruct strbuf errbuf = STRBUF_INIT;\n+\n \t/*\n \t * Parse default value, but silently ignore it if it is invalid.\n \t */\n \tif (!core_partial_clone_filter_default)\n \t\treturn;\n \tgently_parse_list_objects_filter(filter_options,\n \t\t\t\t\t core_partial_clone_filter_default,\n-\t\t\t\t\t NULL);\n+\t\t\t\t\t &errbuf);\n+\tstrbuf_release(&errbuf);\n }\n-- \n2.21.0\n\n"},{"id":"377195","messageId":"8f65643a0385c0e30947d5a3761798a79ad32939.1560462201.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1560462201.git.matvore@google.com","subject":"[PATCH v3 04/10] list-objects-filter: implement composite filters","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-13T21:51:27Z","receivedAt":"2019-06-13T21:51:52Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Allow combining filters such that only objects accepted by all filters\nare shown. The motivation for this is to allow getting directory\nlistings without also fetching blobs. This can be done by combining\nblob:none with tree:. There are massive repositories that have\nlarger-than-expected trees - even if you include only a single commit.\n\nThe current usage requires passing the filter to rev-list in the\nfollowing form:\n\n\t--filter= --filter= ...\n\nSuch usage is currently an error, so giving it a meaning is backwards-\ncompatible.\n\nThe URL-encoding scheme is being introduced before the repeated flag\nlogic, and the user-facing documentation for URL-encoding is being\nwithheld until the repeated flag feature is implemented. The\nURL-encoding is in general not meant to be used directly by the user,\nand it is better to describe the URL-encoding feature in terms of the\nrepeated flag.\n\nHelped-by: Emily Shaffer \nHelped-by: Jeff Hostetler \nHelped-by: Junio C Hamano \nSigned-off-by: Matthew DeVore \n---\n list-objects-filter-options.c | 106 ++++++++++++++++++-\n list-objects-filter-options.h | 17 ++-\n list-objects-filter.c | 159 ++++++++++++++++++++++++++++\n t/t6112-rev-list-filters-objects.sh | 151 +++++++++++++++++++++++++-\n url.c | 6 ++\n url.h | 8 ++\n 6 files changed, 441 insertions(+), 6 deletions(-)\n\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex aef24ddae3..ffbadf337b 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -1,18 +1,24 @@\n #include \"cache.h\"\n #include \"commit.h\"\n #include \"config.h\"\n #include \"revision.h\"\n #include \"argv-array.h\"\n #include \"list-objects.h\"\n #include \"list-objects-filter.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"url.h\"\n+\n+static int parse_combine_filter(\n+\tstruct list_objects_filter_options *filter_options,\n+\tconst char *arg,\n+\tstruct strbuf *errbuf);\n \n /*\n * Parse value of the argument to the \"filter\" keyword.\n * On the command line this looks like:\n * --filter=\n * and in the pack protocol as:\n * \"filter\" SP \n *\n * The filter keyword will be used by many commands.\n * See Documentation/rev-list-options.txt for allowed values for .\n@@ -28,22 +34,20 @@ static int gently_parse_list_objects_filter(\n \tstruct strbuf *errbuf)\n {\n \tconst char *v0;\n \n \tif (filter_options->choice) {\n \t\tstrbuf_addstr(\n \t\t\terrbuf, _(\"multiple filter-specs cannot be combined\"));\n \t\treturn 1;\n \t}\n \n-\tfilter_options->filter_spec = strdup(arg);\n-\n \tif (!strcmp(arg, \"blob:none\")) {\n \t\tfilter_options->choice = LOFC_BLOB_NONE;\n \t\treturn 0;\n \n \t} else if (skip_prefix(arg, \"blob:limit=\", &v0)) {\n \t\tif (git_parse_ulong(v0, &filter_options->blob_limit_value)) {\n \t\t\tfilter_options->choice = LOFC_BLOB_LIMIT;\n \t\t\treturn 0;\n \t\t}\n \n@@ -67,36 +71,125 @@ static int gently_parse_list_objects_filter(\n \t\tif (!get_oid_with_context(the_repository, v0, GET_OID_BLOB,\n \t\t\t\t\t &sparse_oid, &oc))\n \t\t\tfilter_options->sparse_oid_value = oiddup(&sparse_oid);\n \t\tfilter_options->choice = LOFC_SPARSE_OID;\n \t\treturn 0;\n \n \t} else if (skip_prefix(arg, \"sparse:path=\", &v0)) {\n \t\tfilter_options->choice = LOFC_SPARSE_PATH;\n \t\tfilter_options->sparse_path_value = strdup(v0);\n \t\treturn 0;\n+\n+\t} else if (skip_prefix(arg, \"combine:\", &v0)) {\n+\t\treturn parse_combine_filter(filter_options, v0, errbuf);\n+\n \t}\n \t/*\n \t * Please update _git_fetch() in git-completion.bash when you\n \t * add new filters\n \t */\n \n \tstrbuf_addf(errbuf, \"invalid filter-spec '%s'\", arg);\n \n \tmemset(filter_options, 0, sizeof(*filter_options));\n \treturn 1;\n }\n \n+static const char *RESERVED_NON_WS = \"~`!@#$^&*()[]{}\\\\;'\\\",<>?\";\n+\n+static int has_reserved_character(\n+\tstruct strbuf *sub_spec, struct strbuf *errbuf)\n+{\n+\tconst char *c = sub_spec->buf;\n+\twhile (*c) {\n+\t\tif (*c <= ' ' || strchr(RESERVED_NON_WS, *c)) {\n+\t\t\tstrbuf_addf(errbuf,\n+\t\t\t\t \"must escape char in sub-filter-spec: '%c'\",\n+\t\t\t\t *c);\n+\t\t\treturn 1;\n+\t\t}\n+\t\tc++;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int parse_combine_subfilter(\n+\tstruct list_objects_filter_options *filter_options,\n+\tstruct strbuf *subspec,\n+\tstruct strbuf *errbuf)\n+{\n+\tsize_t new_index = filter_options->sub_nr++;\n+\tchar *decoded;\n+\tint result;\n+\n+\tALLOC_GROW(filter_options->sub, filter_options->sub_nr,\n+\t\t filter_options->sub_alloc);\n+\tmemset(&filter_options->sub[new_index], 0,\n+\t sizeof(*filter_options->sub));\n+\n+\tdecoded = url_percent_decode(subspec->buf);\n+\n+\tresult = has_reserved_character(subspec, errbuf) ||\n+\t\tgently_parse_list_objects_filter(\n+\t\t\t&filter_options->sub[new_index], decoded, errbuf);\n+\n+\tfree(decoded);\n+\treturn result;\n+}\n+\n+static int parse_combine_filter(\n+\tstruct list_objects_filter_options *filter_options,\n+\tconst char *arg,\n+\tstruct strbuf *errbuf)\n+{\n+\tstruct strbuf **subspecs = strbuf_split_str(arg, '+', 0);\n+\tsize_t sub;\n+\tint result = 0;\n+\n+\tif (!subspecs[0]) {\n+\t\tstrbuf_addf(errbuf,\n+\t\t\t _(\"expected something after combine:\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tfor (sub = 0; subspecs[sub] && !result; sub++) {\n+\t\tif (subspecs[sub + 1]) {\n+\t\t\t/*\n+\t\t\t * This is not the last subspec. Remove trailing \"+\" so\n+\t\t\t * we can parse it.\n+\t\t\t */\n+\t\t\tsize_t last = subspecs[sub]->len - 1;\n+\t\t\tassert(subspecs[sub]->buf[last] == '+');\n+\t\t\tstrbuf_remove(subspecs[sub], last, 1);\n+\t\t}\n+\t\tresult = parse_combine_subfilter(\n+\t\t\tfilter_options, subspecs[sub], errbuf);\n+\t}\n+\n+\tfilter_options->choice = LOFC_COMBINE;\n+\n+cleanup:\n+\tstrbuf_list_free(subspecs);\n+\tif (result) {\n+\t\tlist_objects_filter_release(filter_options);\n+\t\tmemset(filter_options, 0, sizeof(*filter_options));\n+\t}\n+\treturn result;\n+}\n+\n int parse_list_objects_filter(struct list_objects_filter_options *filter_options,\n \t\t\t const char *arg)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n+\tfilter_options->filter_spec = strdup(arg);\n \tif (gently_parse_list_objects_filter(filter_options, arg, &buf))\n \t\tdie(\"%s\", buf.buf);\n \treturn 0;\n }\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct list_objects_filter_options *filter_options = opt->value;\n \n@@ -119,23 +212,30 @@ void expand_list_objects_filter_spec(\n \telse if (filter->choice == LOFC_TREE_DEPTH)\n \t\tstrbuf_addf(expanded_spec, \"tree:%lu\",\n \t\t\t filter->tree_exclude_depth);\n \telse\n \t\tstrbuf_addstr(expanded_spec, filter->filter_spec);\n }\n \n void list_objects_filter_release(\n \tstruct list_objects_filter_options *filter_options)\n {\n+\tsize_t sub;\n+\n+\tif (!filter_options)\n+\t\treturn;\n \tfree(filter_options->filter_spec);\n \tfree(filter_options->sparse_oid_value);\n \tfree(filter_options->sparse_path_value);\n+\tfor (sub = 0; sub < filter_options->sub_nr; sub++)\n+\t\tlist_objects_filter_release(&filter_options->sub[sub]);\n+\tfree(filter_options->sub);\n \tmemset(filter_options, 0, sizeof(*filter_options));\n }\n \n void partial_clone_register(\n \tconst char *remote,\n \tconst struct list_objects_filter_options *filter_options)\n {\n \t/*\n \t * Record the name of the partial clone remote in the\n \t * config and in the global variable -- the latter is\n@@ -165,15 +265,17 @@ void partial_clone_register(\n void partial_clone_get_default_filter_spec(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tstruct strbuf errbuf = STRBUF_INIT;\n \n \t/*\n \t * Parse default value, but silently ignore it if it is invalid.\n \t */\n \tif (!core_partial_clone_filter_default)\n \t\treturn;\n+\n+\tfilter_options->filter_spec = strdup(core_partial_clone_filter_default);\n \tgently_parse_list_objects_filter(filter_options,\n \t\t\t\t\t core_partial_clone_filter_default,\n \t\t\t\t\t &errbuf);\n \tstrbuf_release(&errbuf);\n }\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex e3adc78ebf..8f08ed74a1 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -7,20 +7,21 @@\n /*\n * The list of defined filters for list-objects.\n */\n enum list_objects_filter_choice {\n \tLOFC_DISABLED = 0,\n \tLOFC_BLOB_NONE,\n \tLOFC_BLOB_LIMIT,\n \tLOFC_TREE_DEPTH,\n \tLOFC_SPARSE_OID,\n \tLOFC_SPARSE_PATH,\n+\tLOFC_COMBINE,\n \tLOFC__COUNT /* must be last */\n };\n \n struct list_objects_filter_options {\n \t/*\n \t * 'filter_spec' is the raw argument value given on the command line\n \t * or protocol request. (The part after the \"--keyword=\".) For\n \t * commands that launch filtering sub-processes, or for communication\n \t * over the network, don't use this value; use the result of\n \t * expand_list_objects_filter_spec() instead.\n@@ -32,28 +33,38 @@ struct list_objects_filter_options {\n \t * the filtering algorithm to use.\n \t */\n \tenum list_objects_filter_choice choice;\n \n \t/*\n \t * Choice is LOFC_DISABLED because \"--no-filter\" was requested.\n \t */\n \tunsigned int no_filter : 1;\n \n \t/*\n-\t * Parsed values (fields) from within the filter-spec. These are\n-\t * choice-specific; not all values will be defined for any given\n-\t * choice.\n+\t * BEGIN choice-specific parsed values from within the filter-spec. Only\n+\t * some values will be defined for any given choice.\n \t */\n+\n \tstruct object_id *sparse_oid_value;\n \tchar *sparse_path_value;\n \tunsigned long blob_limit_value;\n \tunsigned long tree_exclude_depth;\n+\n+\t/* LOFC_COMBINE values */\n+\n+\t/* This array contains all the subfilters which this filter combines. */\n+\tsize_t sub_nr, sub_alloc;\n+\tstruct list_objects_filter_options *sub;\n+\n+\t/*\n+\t * END choice-specific parsed values.\n+\t */\n };\n \n /* Normalized command line arguments */\n #define CL_ARG__FILTER \"filter\"\n \n int parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg);\n \n int opt_parse_list_objects_filter(const struct option *opt,\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex 57bbf6ec1c..c8a006edf9 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -19,30 +19,45 @@\n * FILTER_SHOWN_BUT_REVISIT -- we set this bit on tree objects\n * that have been shown, but should be revisited if they appear\n * in the traversal (until we mark it SEEN). This is a way to\n * let us silently de-dup calls to show() in the caller. This\n * is subtly different from the \"revision.h:SHOWN\" and the\n * \"sha1-name.c:ONELINE_SEEN\" bits. And also different from\n * the non-de-dup usage in pack-bitmap.c\n */\n #define FILTER_SHOWN_BUT_REVISIT (1<<21)\n \n+struct subfilter {\n+\tstruct filter *filter;\n+\tstruct oidset seen;\n+\tstruct oidset omits;\n+\tstruct object_id skip_tree;\n+\tunsigned is_skipping_tree : 1;\n+};\n+\n struct filter {\n \tenum list_objects_filter_result (*filter_object_fn)(\n \t\tstruct repository *r,\n \t\tenum list_objects_filter_situation filter_situation,\n \t\tstruct object *obj,\n \t\tconst char *pathname,\n \t\tconst char *filename,\n \t\tstruct oidset *omits,\n \t\tvoid *filter_data);\n \n+\t/*\n+\t * Optional. If this function is supplied and the filter needs to\n+\t * collect omits, then this function is called once before free_fn is\n+\t * called.\n+\t */\n+\tvoid (*finalize_omits_fn)(struct oidset *omits, void *filter_data);\n+\n \tvoid (*free_fn)(void *filter_data);\n \n \tvoid *filter_data;\n \n \t/* If non-NULL, the filter collects a list of the omitted OIDs here. */\n \tstruct oidset *omits;\n };\n \n static enum list_objects_filter_result filter_blobs_none(\n \tstruct repository *r,\n@@ -482,34 +497,176 @@ static void filter_sparse_path__init(\n \n \tALLOC_GROW(d->array_frame, d->nr + 1, d->alloc);\n \td->array_frame[d->nr].defval = 0; /* default to include */\n \td->array_frame[d->nr].child_prov_omit = 0;\n \n \tfilter->filter_data = d;\n \tfilter->filter_object_fn = filter_sparse;\n \tfilter->free_fn = filter_sparse_free;\n }\n \n+/* A filter which only shows objects shown by all sub-filters. */\n+struct combine_filter_data {\n+\tstruct subfilter *sub;\n+\tsize_t nr;\n+};\n+\n+static int should_delegate(enum list_objects_filter_situation filter_situation,\n+\t\t\t struct object *obj,\n+\t\t\t struct subfilter *sub)\n+{\n+\tif (!sub->is_skipping_tree)\n+\t\treturn 1;\n+\tif (filter_situation == LOFS_END_TREE &&\n+\t\toideq(&obj->oid, &sub->skip_tree)) {\n+\t\tsub->is_skipping_tree = 0;\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+static enum list_objects_filter_result process_subfilter(\n+\tstruct repository *r,\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tstruct subfilter *sub)\n+{\n+\tenum list_objects_filter_result result;\n+\n+\t/*\n+\t * Check should_delegate before oidset_contains so that\n+\t * is_skipping_tree gets unset even when the object is marked as seen.\n+\t * As of this writing, no filter uses LOFR_MARK_SEEN on trees that also\n+\t * uses LOFR_SKIP_TREE, so the ordering is only theoretically\n+\t * important. Be cautious if you change the order of the below checks\n+\t * and more filters have been added!\n+\t */\n+\tif (!should_delegate(filter_situation, obj, sub))\n+\t\treturn LOFR_ZERO;\n+\tif (oidset_contains(&sub->seen, &obj->oid))\n+\t\treturn LOFR_ZERO;\n+\n+\tresult = list_objects_filter__filter_object(\n+\t\tr, filter_situation, obj, pathname, filename, sub->filter);\n+\n+\tif (result & LOFR_MARK_SEEN)\n+\t\toidset_insert(&sub->seen, &obj->oid);\n+\n+\tif (result & LOFR_SKIP_TREE) {\n+\t\tsub->is_skipping_tree = 1;\n+\t\tsub->skip_tree = obj->oid;\n+\t}\n+\n+\treturn result;\n+}\n+\n+static enum list_objects_filter_result filter_combine(\n+\tstruct repository *r,\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tstruct oidset *omits,\n+\tvoid *filter_data)\n+{\n+\tstruct combine_filter_data *d = filter_data;\n+\tenum list_objects_filter_result combined_result =\n+\t\tLOFR_DO_SHOW | LOFR_MARK_SEEN | LOFR_SKIP_TREE;\n+\tsize_t sub;\n+\n+\tfor (sub = 0; sub < d->nr; sub++) {\n+\t\tenum list_objects_filter_result sub_result = process_subfilter(\n+\t\t\tr, filter_situation, obj, pathname, filename,\n+\t\t\t&d->sub[sub]);\n+\t\tif (!(sub_result & LOFR_DO_SHOW))\n+\t\t\tcombined_result &= ~LOFR_DO_SHOW;\n+\t\tif (!(sub_result & LOFR_MARK_SEEN))\n+\t\t\tcombined_result &= ~LOFR_MARK_SEEN;\n+\t\tif (!d->sub[sub].is_skipping_tree)\n+\t\t\tcombined_result &= ~LOFR_SKIP_TREE;\n+\t}\n+\n+\treturn combined_result;\n+}\n+\n+static void filter_combine__free(void *filter_data)\n+{\n+\tstruct combine_filter_data *d = filter_data;\n+\tsize_t sub;\n+\tfor (sub = 0; sub < d->nr; sub++) {\n+\t\tlist_objects_filter__free(d->sub[sub].filter);\n+\t\toidset_clear(&d->sub[sub].seen);\n+\t\tif (d->sub[sub].omits.set.size)\n+\t\t\tBUG(\"expected oidset to be cleared already\");\n+\t}\n+\tfree(d->sub);\n+}\n+\n+static void add_all(struct oidset *dest, struct oidset *src) {\n+\tstruct oidset_iter iter;\n+\tstruct object_id *src_oid;\n+\n+\toidset_iter_init(src, &iter);\n+\twhile ((src_oid = oidset_iter_next(&iter)) != NULL)\n+\t\toidset_insert(dest, src_oid);\n+}\n+\n+static void filter_combine__finalize_omits(\n+\tstruct oidset *omits,\n+\tvoid *filter_data)\n+{\n+\tstruct combine_filter_data *d = filter_data;\n+\tsize_t sub;\n+\n+\tfor (sub = 0; sub < d->nr; sub++) {\n+\t\tadd_all(omits, &d->sub[sub].omits);\n+\t\toidset_clear(&d->sub[sub].omits);\n+\t}\n+}\n+\n+static void filter_combine__init(\n+\tstruct list_objects_filter_options *filter_options,\n+\tstruct filter* filter)\n+{\n+\tstruct combine_filter_data *d = xcalloc(1, sizeof(*d));\n+\tsize_t sub;\n+\n+\td->nr = filter_options->sub_nr;\n+\td->sub = xcalloc(d->nr, sizeof(*d->sub));\n+\tfor (sub = 0; sub < d->nr; sub++)\n+\t\td->sub[sub].filter = list_objects_filter__init(\n+\t\t\tfilter->omits ? &d->sub[sub].omits : NULL,\n+\t\t\t&filter_options->sub[sub]);\n+\n+\tfilter->filter_data = d;\n+\tfilter->filter_object_fn = filter_combine;\n+\tfilter->free_fn = filter_combine__free;\n+\tfilter->finalize_omits_fn = filter_combine__finalize_omits;\n+}\n+\n typedef void (*filter_init_fn)(\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter);\n \n /*\n * Must match \"enum list_objects_filter_choice\".\n */\n static filter_init_fn s_filters[] = {\n \tNULL,\n \tfilter_blobs_none__init,\n \tfilter_blobs_limit__init,\n \tfilter_trees_depth__init,\n \tfilter_sparse_oid__init,\n \tfilter_sparse_path__init,\n+\tfilter_combine__init,\n };\n \n struct filter *list_objects_filter__init(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options)\n {\n \tstruct filter *filter;\n \tfilter_init_fn init_fn;\n \n \tassert((sizeof(s_filters) / sizeof(s_filters[0])) == LOFC__COUNT);\n@@ -547,13 +704,15 @@ enum list_objects_filter_result list_objects_filter__filter_object(\n \t */\n \tif (filter_situation == LOFS_END_TREE)\n \t\treturn 0;\n \treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n }\n \n void list_objects_filter__free(struct filter *filter)\n {\n \tif (!filter)\n \t\treturn;\n+\tif (filter->finalize_omits_fn && filter->omits)\n+\t\tfilter->finalize_omits_fn(filter->omits, filter->filter_data);\n \tfilter->free_fn(filter->filter_data);\n \tfree(filter);\n }\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex 9c11427719..a87341e051 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -284,21 +284,33 @@ test_expect_success 'verify tree:0 includes trees in \"filtered\" output' '\n # Make sure tree:0 does not iterate through any trees.\n \n test_expect_success 'verify skipping tree iteration when not collecting omits' '\n \tGIT_TRACE=1 git -C r3 rev-list \\\n \t\t--objects --filter=tree:0 HEAD 2>filter_trace &&\n \tgrep \"Skipping contents of tree [.][.][.]\" filter_trace >actual &&\n \t# One line for each commit traversed.\n \ttest_line_count = 2 actual &&\n \n \t# Make sure no other trees were considered besides the root.\n-\t! grep \"Skipping contents of tree [^.]\" filter_trace\n+\t! grep \"Skipping contents of tree [^.]\" filter_trace &&\n+\n+\t# Try this again with \"combine:\". If both sub-filters are skipping\n+\t# trees, the composite filter should also skip trees. This is not\n+\t# important unless the user does combine:tree:X+tree:Y or another filter\n+\t# besides \"tree:\" is implemented in the future which can skip trees.\n+\tGIT_TRACE=1 git -C r3 rev-list \\\n+\t\t--objects --filter=combine:tree:1+tree:3 HEAD 2>filter_trace &&\n+\n+\t# Only skip the dir1/ tree, which is shared between the two commits.\n+\tgrep \"Skipping contents of tree \" filter_trace >actual &&\n+\ttest_write_lines \"Skipping contents of tree dir1/...\" >expected &&\n+\ttest_cmp expected actual\n '\n \n # Test tree:# filters.\n \n expect_has () {\n \tcommit=$1 &&\n \tname=$2 &&\n \n \thash=$(git -C r3 rev-parse $commit:$name) &&\n \tgrep \"^$hash $name$\" actual\n@@ -336,20 +348,126 @@ test_expect_success 'verify tree:3 includes everything expected' '\n \texpect_has HEAD dir1/sparse1 &&\n \texpect_has HEAD dir1/sparse2 &&\n \texpect_has HEAD pattern &&\n \texpect_has HEAD sparse1 &&\n \texpect_has HEAD sparse2 &&\n \n \t# There are also 2 commit objects\n \ttest_line_count = 10 actual\n '\n \n+test_expect_success 'combine:... for a simple combination' '\n+\tgit -C r3 rev-list --objects --filter=combine:tree:2+blob:none HEAD \\\n+\t\t>actual &&\n+\n+\texpect_has HEAD \"\" &&\n+\texpect_has HEAD~1 \"\" &&\n+\texpect_has HEAD dir1 &&\n+\n+\t# There are also 2 commit objects\n+\ttest_line_count = 5 actual\n+'\n+\n+test_expect_success 'combine:... with URL encoding' '\n+\tgit -C r3 rev-list --objects \\\n+\t\t--filter=combine:tree%3a2+blob:%6Eon%65 HEAD >actual &&\n+\n+\texpect_has HEAD \"\" &&\n+\texpect_has HEAD~1 \"\" &&\n+\texpect_has HEAD dir1 &&\n+\n+\t# There are also 2 commit objects\n+\ttest_line_count = 5 actual\n+'\n+\n+expect_invalid_filter_spec () {\n+\tspec=\"$1\" &&\n+\terr=\"$2\" &&\n+\n+\ttest_must_fail git -C r3 rev-list --objects --filter=\"$spec\" HEAD \\\n+\t\t>actual 2>actual_stderr &&\n+\ttest_must_be_empty actual &&\n+\ttest_i18ngrep \"$err\" actual_stderr\n+}\n+\n+test_expect_success 'combine:... while URL-encoding things that should not be' '\n+\texpect_invalid_filter_spec combine%3Atree:2+blob:none \\\n+\t\t\"invalid filter-spec\"\n+'\n+\n+test_expect_success 'combine: with nothing after the :' '\n+\texpect_invalid_filter_spec combine: \"expected something after combine:\"\n+'\n+\n+test_expect_success 'parse error in first sub-filter in combine:' '\n+\texpect_invalid_filter_spec combine:tree:asdf+blob:none \\\n+\t\t\"expected .tree:.\"\n+'\n+\n+test_expect_success 'combine:... with non-encoded reserved chars' '\n+\texpect_invalid_filter_spec combine:tree:2+sparse:@xyz \\\n+\t\t\"must escape char in sub-filter-spec: .@.\" &&\n+\texpect_invalid_filter_spec combine:tree:2+sparse:\\` \\\n+\t\t\"must escape char in sub-filter-spec: .\\`.\" &&\n+\texpect_invalid_filter_spec combine:tree:2+sparse:~abc \\\n+\t\t\"must escape char in sub-filter-spec: .\\~.\"\n+'\n+\n+test_expect_success 'validate err msg for \"combine:+\"' '\n+\texpect_invalid_filter_spec combine:tree:2+ \"expected .tree:.\"\n+'\n+\n+test_expect_success 'combine:... with edge-case hex digits: Ff Aa 0 9' '\n+\tgit -C r3 rev-list --objects --filter=\"combine:tree:2+bl%6Fb:n%6fne\" \\\n+\t\tHEAD >actual &&\n+\ttest_line_count = 5 actual &&\n+\tgit -C r3 rev-list --objects --filter=\"combine:tree%3A2+blob%3anone\" \\\n+\t\tHEAD >actual &&\n+\ttest_line_count = 5 actual &&\n+\tgit -C r3 rev-list --objects --filter=\"combine:tree:%30\" HEAD >actual &&\n+\ttest_line_count = 2 actual &&\n+\tgit -C r3 rev-list --objects --filter=\"combine:tree:%39+blob:none\" \\\n+\t\tHEAD >actual &&\n+\ttest_line_count = 5 actual\n+'\n+\n+test_expect_success 'add a sparse pattern blob whose path has reserved chars' '\n+\tcp r3/pattern r3/pattern1+renamed% &&\n+\tgit -C r3 add pattern1+renamed% &&\n+\tgit -C r3 commit -m \"add sparse pattern file with reserved chars\"\n+'\n+\n+test_expect_success 'combine:... with more than two sub-filters' '\n+\tgit -C r3 rev-list --objects \\\n+\t\t--filter=combine:tree:3+blob:limit=40+sparse:oid=master:pattern \\\n+\t\tHEAD >actual &&\n+\n+\texpect_has HEAD \"\" &&\n+\texpect_has HEAD~1 \"\" &&\n+\texpect_has HEAD~2 \"\" &&\n+\texpect_has HEAD dir1 &&\n+\texpect_has HEAD dir1/sparse1 &&\n+\texpect_has HEAD dir1/sparse2 &&\n+\n+\t# Should also have 3 commits\n+\ttest_line_count = 9 actual &&\n+\n+\t# Try again, this time making sure the last sub-filter is only\n+\t# URL-decoded once.\n+\tcp actual expect &&\n+\n+\tgit -C r3 rev-list --objects \\\n+\t\t--filter=combine:tree:3+blob:limit=40+sparse:oid=master:pattern1%2brenamed%25 \\\n+\t\tHEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n # Test provisional omit collection logic with a repo that has objects appearing\n # at multiple depths - first deeper than the filter's threshold, then shallow.\n \n test_expect_success 'setup r4' '\n \tgit init r4 &&\n \n \techo foo > r4/foo &&\n \tmkdir r4/subdir &&\n \techo bar > r4/subdir/bar &&\n \n@@ -379,20 +497,51 @@ test_expect_success 'test tree:# filter provisional omit for blob and tree' '\n \n test_expect_success 'verify skipping tree iteration when collecting omits' '\n \tGIT_TRACE=1 git -C r4 rev-list --filter-print-omitted \\\n \t\t--objects --filter=tree:0 HEAD 2>filter_trace &&\n \tgrep \"^Skipping contents of tree \" filter_trace >actual &&\n \n \techo \"Skipping contents of tree subdir/...\" >expect &&\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'setup r5' '\n+\tgit init r5 &&\n+\tmkdir -p r5/subdir &&\n+\n+\techo 1 >r5/short-root &&\n+\techo 12345 >r5/long-root &&\n+\techo a >r5/subdir/short-subdir &&\n+\techo abcde >r5/subdir/long-subdir &&\n+\n+\tgit -C r5 add short-root long-root subdir &&\n+\tgit -C r5 commit -m \"commit msg\"\n+'\n+\n+test_expect_success 'verify collecting omits in combined: filter' '\n+\t# Note that this test guards against the naive implementation of simply\n+\t# giving both filters the same \"omits\" set and expecting it to\n+\t# automatically merge them.\n+\tgit -C r5 rev-list --objects --quiet --filter-print-omitted \\\n+\t\t--filter=combine:tree:2+blob:limit=3 HEAD >actual &&\n+\n+\t# Expect 0 trees/commits, 3 blobs omitted (all blobs except short-root)\n+\tomitted_1=$(echo 12345 | git hash-object --stdin) &&\n+\tomitted_2=$(echo a | git hash-object --stdin) &&\n+\tomitted_3=$(echo abcde | git hash-object --stdin) &&\n+\n+\tgrep ~$omitted_1 actual &&\n+\tgrep ~$omitted_2 actual &&\n+\tgrep ~$omitted_3 actual &&\n+\ttest_line_count = 3 actual\n+'\n+\n # Test tree: where a tree is iterated to twice - once where a subentry is\n # too deep to be included, and again where the blob inside it is shallow enough\n # to be included. This makes sure we don't use LOFR_MARK_SEEN incorrectly (we\n # can't use it because a tree can be iterated over again at a lower depth).\n \n test_expect_success 'tree: where we iterate over tree at two levels' '\n \tgit init r5 &&\n \n \tmkdir -p r5/a/subdir/b &&\n \techo foo > r5/a/subdir/b/foo &&\ndiff --git a/url.c b/url.c\nindex 25576c390b..bdede647bc 100644\n--- a/url.c\n+++ b/url.c\n@@ -79,20 +79,26 @@ char *url_decode_mem(const char *url, int len)\n \n \t/* Skip protocol part if present */\n \tif (colon && url < colon) {\n \t\tstrbuf_add(&out, url, colon - url);\n \t\tlen -= colon - url;\n \t\turl = colon;\n \t}\n \treturn url_decode_internal(&url, len, NULL, &out, 0);\n }\n \n+char *url_percent_decode(const char *encoded)\n+{\n+\tstruct strbuf out = STRBUF_INIT;\n+\treturn url_decode_internal(&encoded, strlen(encoded), NULL, &out, 0);\n+}\n+\n char *url_decode_parameter_name(const char **query)\n {\n \tstruct strbuf out = STRBUF_INIT;\n \treturn url_decode_internal(query, -1, \"&=\", &out, 1);\n }\n \n char *url_decode_parameter_value(const char **query)\n {\n \tstruct strbuf out = STRBUF_INIT;\n \treturn url_decode_internal(query, -1, \"&\", &out, 1);\ndiff --git a/url.h b/url.h\nindex 00b7d58c33..2a27c34277 100644\n--- a/url.h\n+++ b/url.h\n@@ -1,16 +1,24 @@\n #ifndef URL_H\n #define URL_H\n \n struct strbuf;\n \n int is_url(const char *url);\n int is_urlschemechar(int first_flag, int ch);\n char *url_decode(const char *url);\n char *url_decode_mem(const char *url, int len);\n+\n+/*\n+ * Similar to the url_decode_{,mem} methods above, but doesn't assume there\n+ * is a scheme followed by a : at the start of the string. Instead, %-sequences\n+ * before any : are also parsed.\n+ */\n+char *url_percent_decode(const char *encoded);\n+\n char *url_decode_parameter_name(const char **query);\n char *url_decode_parameter_value(const char **query);\n \n void end_url_with_slash(struct strbuf *buf, const char *url);\n void str_end_url_with_slash(const char *url, char **dest);\n \n #endif /* URL_H */\n-- \n2.21.0\n\n"},{"id":"377196","messageId":"737cd0dd5b8a60389f862180f8c1d5ee107fe172.1560462201.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1560462201.git.matvore@google.com","subject":"[PATCH v3 05/10] list-objects-filter-options: move error check up","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-13T21:51:28Z","receivedAt":"2019-06-13T21:51:53Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Move the check that filter_options->choice is set to higher in the call\nstack. This can only be set when the gentle parse function is called\nfrom one of the two call sites.\n\nThis is important because in an upcoming patch this may or may not be an\nerror, and whether it is an error is only known to the\nparse_list_objects_filter function.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects-filter-options.c | 9 ++++-----\n 1 file changed, 4 insertions(+), 5 deletions(-)\n\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex ffbadf337b..5ff5135a91 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -28,25 +28,22 @@ static int parse_combine_filter(\n * expand_list_objects_filter_spec() first). We also \"intern\" the arg for the\n * convenience of the current command.\n */\n static int gently_parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg,\n \tstruct strbuf *errbuf)\n {\n \tconst char *v0;\n \n-\tif (filter_options->choice) {\n-\t\tstrbuf_addstr(\n-\t\t\terrbuf, _(\"multiple filter-specs cannot be combined\"));\n-\t\treturn 1;\n-\t}\n+\tif (filter_options->choice)\n+\t\tBUG(\"filter_options already populated\");\n \n \tif (!strcmp(arg, \"blob:none\")) {\n \t\tfilter_options->choice = LOFC_BLOB_NONE;\n \t\treturn 0;\n \n \t} else if (skip_prefix(arg, \"blob:limit=\", &v0)) {\n \t\tif (git_parse_ulong(v0, &filter_options->blob_limit_value)) {\n \t\t\tfilter_options->choice = LOFC_BLOB_LIMIT;\n \t\t\treturn 0;\n \t\t}\n@@ -175,20 +172,22 @@ static int parse_combine_filter(\n \t\tlist_objects_filter_release(filter_options);\n \t\tmemset(filter_options, 0, sizeof(*filter_options));\n \t}\n \treturn result;\n }\n \n int parse_list_objects_filter(struct list_objects_filter_options *filter_options,\n \t\t\t const char *arg)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n+\tif (filter_options->choice)\n+\t\tdie(_(\"multiple filter-specs cannot be combined\"));\n \tfilter_options->filter_spec = strdup(arg);\n \tif (gently_parse_list_objects_filter(filter_options, arg, &buf))\n \t\tdie(\"%s\", buf.buf);\n \treturn 0;\n }\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct list_objects_filter_options *filter_options = opt->value;\n-- \n2.21.0\n\n"},{"id":"377197","messageId":"23706c83ef114f0d9c89904e06e223bed3dc007a.1560462201.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1560462201.git.matvore@google.com","subject":"[PATCH v3 06/10] list-objects-filter-options: make filter_spec a string_list","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-13T21:51:29Z","receivedAt":"2019-06-13T21:51:57Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Make the filter_spec string a string_list rather than a raw C string.\nThe list of strings must be concatted together to make a complete\nfilter_spec. A future patch will use this capability to build \"combine:\"\nfilter specs gradually.\n\nA strbuf would seem to be a more natural choice for this object, but it\nunfortunately requires initialization besides just zero'ing out the\nmemory. This results in all container structs, and all containers of\nthose structs, etc., to also require initialization. Initializing them\nall would be more cumbersome that simply using a string_list, which\nbehaves properly when its contents are zero'd.\n\nFor the purposes of code simplification, change behavior in how filter\nspecs are conveyed over the protocol: do not normalize the tree:\nfilter specs since there should be no server in existence that supports\ntree:# but not tree:#k etc.\n\nHelped-by: Junio C Hamano \nSigned-off-by: Matthew DeVore \n---\n builtin/clone.c | 8 ++---\n builtin/fetch.c | 9 ++----\n builtin/rev-list.c | 6 ++--\n fetch-pack.c | 20 ++++--------\n list-objects-filter-options.c | 50 ++++++++++++++++++++---------\n list-objects-filter-options.h | 27 +++++++++++-----\n t/t6112-rev-list-filters-objects.sh | 7 ----\n transport-helper.c | 10 ++----\n upload-pack.c | 11 +++----\n 9 files changed, 78 insertions(+), 70 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 85b0d3155d..81e6010779 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1135,27 +1135,25 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\ttransport_set_option(transport, TRANS_OPT_FOLLOWTAGS, \"1\");\n \n \tif (option_upload_pack)\n \t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK,\n \t\t\t\t option_upload_pack);\n \n \tif (server_options.nr)\n \t\ttransport->server_options = &server_options;\n \n \tif (filter_options.choice) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n-\t\texpand_list_objects_filter_spec(&filter_options,\n-\t\t\t\t\t\t&expanded_filter_spec);\n+\t\tconst char *spec =\n+\t\t\texpand_list_objects_filter_spec(&filter_options);\n \t\ttransport_set_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER,\n-\t\t\t\t expanded_filter_spec.buf);\n+\t\t\t\t spec);\n \t\ttransport_set_option(transport, TRANS_OPT_FROM_PROMISOR, \"1\");\n-\t\tstrbuf_release(&expanded_filter_spec);\n \t}\n \n \tif (transport->smart_options && !deepen && !filter_options.choice)\n \t\ttransport->smart_options->check_self_contained_and_connected = 1;\n \n \n \targv_array_push(&ref_prefixes, \"HEAD\");\n \trefspec_ref_prefixes(&remote->fetch, &ref_prefixes);\n \tif (option_branch)\n \t\texpand_ref_prefix(&ref_prefixes, option_branch);\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 4ba63d5ac6..dee89e1a19 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1181,27 +1181,24 @@ static struct transport *prepare_transport(struct remote *remote, int deepen)\n \tif (deepen && deepen_since)\n \t\tset_option(transport, TRANS_OPT_DEEPEN_SINCE, deepen_since);\n \tif (deepen && deepen_not.nr)\n \t\tset_option(transport, TRANS_OPT_DEEPEN_NOT,\n \t\t\t (const char *)&deepen_not);\n \tif (deepen_relative)\n \t\tset_option(transport, TRANS_OPT_DEEPEN_RELATIVE, \"yes\");\n \tif (update_shallow)\n \t\tset_option(transport, TRANS_OPT_UPDATE_SHALLOW, \"yes\");\n \tif (filter_options.choice) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n-\t\texpand_list_objects_filter_spec(&filter_options,\n-\t\t\t\t\t\t&expanded_filter_spec);\n-\t\tset_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER,\n-\t\t\t expanded_filter_spec.buf);\n+\t\tconst char *spec =\n+\t\t\texpand_list_objects_filter_spec(&filter_options);\n+\t\tset_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER, spec);\n \t\tset_option(transport, TRANS_OPT_FROM_PROMISOR, \"1\");\n-\t\tstrbuf_release(&expanded_filter_spec);\n \t}\n \tif (negotiation_tip.nr) {\n \t\tif (transport->smart_options)\n \t\t\tadd_negotiation_tips(transport->smart_options);\n \t\telse\n \t\t\twarning(\"Ignoring --negotiation-tip because the protocol does not support it.\");\n \t}\n \treturn transport;\n }\n \ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 9f31837d30..823e87c1c9 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -459,22 +459,24 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\tshow_progress = arg;\n \t\t\tcontinue;\n \t\t}\n \n \t\tif (skip_prefix(arg, (\"--\" CL_ARG__FILTER \"=\"), &arg)) {\n \t\t\tparse_list_objects_filter(&filter_options, arg);\n \t\t\tif (filter_options.choice && !revs.blob_objects)\n \t\t\t\tdie(_(\"object filtering requires --objects\"));\n \t\t\tif (filter_options.choice == LOFC_SPARSE_OID &&\n \t\t\t !filter_options.sparse_oid_value)\n-\t\t\t\tdie(_(\"invalid sparse value '%s'\"),\n-\t\t\t\t filter_options.filter_spec);\n+\t\t\t\tdie(\n+\t\t\t\t\t_(\"invalid sparse value '%s'\"),\n+\t\t\t\t\tlist_objects_filter_spec(\n+\t\t\t\t\t\t&filter_options));\n \t\t\tcontinue;\n \t\t}\n \t\tif (!strcmp(arg, (\"--no-\" CL_ARG__FILTER))) {\n \t\t\tlist_objects_filter_set_no_filter(&filter_options);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!strcmp(arg, \"--filter-print-omitted\")) {\n \t\t\targ_print_omitted = 1;\n \t\t\tcontinue;\n \t\t}\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 1c10f54e78..72e13b0a1d 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -332,26 +332,23 @@ static int find_common(struct fetch_negotiator *negotiator,\n \t\tpacket_buf_write(&req_buf, \"deepen-since %\"PRItime, max_age);\n \t}\n \tif (args->deepen_not) {\n \t\tint i;\n \t\tfor (i = 0; i < args->deepen_not->nr; i++) {\n \t\t\tstruct string_list_item *s = args->deepen_not->items + i;\n \t\t\tpacket_buf_write(&req_buf, \"deepen-not %s\", s->string);\n \t\t}\n \t}\n \tif (server_supports_filtering && args->filter_options.choice) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n-\t\texpand_list_objects_filter_spec(&args->filter_options,\n-\t\t\t\t\t\t&expanded_filter_spec);\n-\t\tpacket_buf_write(&req_buf, \"filter %s\",\n-\t\t\t\t expanded_filter_spec.buf);\n-\t\tstrbuf_release(&expanded_filter_spec);\n+\t\tconst char *spec =\n+\t\t\texpand_list_objects_filter_spec(&args->filter_options);\n+\t\tpacket_buf_write(&req_buf, \"filter %s\", spec);\n \t}\n \tpacket_buf_flush(&req_buf);\n \tstate_len = req_buf.len;\n \n \tif (args->deepen) {\n \t\tconst char *arg;\n \t\tstruct object_id oid;\n \n \t\tsend_request(args, fd[1], &req_buf);\n \t\twhile (packet_reader_read(&reader) == PACKET_READ_NORMAL) {\n@@ -1092,21 +1089,21 @@ static int add_haves(struct fetch_negotiator *negotiator,\n \t\tret = 1;\n \t}\n \n \t/* Increase haves to send on next round */\n \t*haves_to_send = next_flush(1, *haves_to_send);\n \n \treturn ret;\n }\n \n static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n-\t\t\t const struct fetch_pack_args *args,\n+\t\t\t struct fetch_pack_args *args,\n \t\t\t const struct ref *wants, struct oidset *common,\n \t\t\t int *haves_to_send, int *in_vain,\n \t\t\t int sideband_all)\n {\n \tint ret = 0;\n \tstruct strbuf req_buf = STRBUF_INIT;\n \n \tif (server_supports_v2(\"fetch\", 1))\n \t\tpacket_buf_write(&req_buf, \"command=fetch\");\n \tif (server_supports_v2(\"agent\", 0))\n@@ -1133,27 +1130,24 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n \n \t/* Add shallow-info and deepen request */\n \tif (server_supports_feature(\"fetch\", \"shallow\", 0))\n \t\tadd_shallow_requests(&req_buf, args);\n \telse if (is_repository_shallow(the_repository) || args->deepen)\n \t\tdie(_(\"Server does not support shallow requests\"));\n \n \t/* Add filter */\n \tif (server_supports_feature(\"fetch\", \"filter\", 0) &&\n \t args->filter_options.choice) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n+\t\tconst char *spec =\n+\t\t\texpand_list_objects_filter_spec(&args->filter_options);\n \t\tprint_verbose(args, _(\"Server supports filter\"));\n-\t\texpand_list_objects_filter_spec(&args->filter_options,\n-\t\t\t\t\t\t&expanded_filter_spec);\n-\t\tpacket_buf_write(&req_buf, \"filter %s\",\n-\t\t\t\t expanded_filter_spec.buf);\n-\t\tstrbuf_release(&expanded_filter_spec);\n+\t\tpacket_buf_write(&req_buf, \"filter %s\", spec);\n \t} else if (args->filter_options.choice) {\n \t\twarning(\"filtering not recognized by server, ignoring\");\n \t}\n \n \t/* add wants */\n \tadd_wants(args->no_dependents, wants, &req_buf);\n \n \tif (args->no_dependents) {\n \t\tpacket_buf_write(&req_buf, \"done\");\n \t\tret = 1;\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex 5ff5135a91..c9dd41cd06 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -174,73 +174,90 @@ static int parse_combine_filter(\n \t}\n \treturn result;\n }\n \n int parse_list_objects_filter(struct list_objects_filter_options *filter_options,\n \t\t\t const char *arg)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tif (filter_options->choice)\n \t\tdie(_(\"multiple filter-specs cannot be combined\"));\n-\tfilter_options->filter_spec = strdup(arg);\n+\tstring_list_append(&filter_options->filter_spec, xstrdup(arg));\n \tif (gently_parse_list_objects_filter(filter_options, arg, &buf))\n \t\tdie(\"%s\", buf.buf);\n \treturn 0;\n }\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct list_objects_filter_options *filter_options = opt->value;\n \n \tif (unset || !arg) {\n \t\tlist_objects_filter_set_no_filter(filter_options);\n \t\treturn 0;\n \t}\n \n \treturn parse_list_objects_filter(filter_options, arg);\n }\n \n-void expand_list_objects_filter_spec(\n-\tconst struct list_objects_filter_options *filter,\n-\tstruct strbuf *expanded_spec)\n+const char *list_objects_filter_spec(struct list_objects_filter_options *filter)\n {\n-\tstrbuf_init(expanded_spec, strlen(filter->filter_spec));\n-\tif (filter->choice == LOFC_BLOB_LIMIT)\n-\t\tstrbuf_addf(expanded_spec, \"blob:limit=%lu\",\n+\tif (!filter->filter_spec.nr)\n+\t\tBUG(\"no filter_spec available for this filter\");\n+\tif (filter->filter_spec.nr != 1) {\n+\t\tstruct strbuf concatted = STRBUF_INIT;\n+\t\tstrbuf_add_separated_string_list(\n+\t\t\t&concatted, \"\", &filter->filter_spec);\n+\t\tstring_list_clear(&filter->filter_spec, /*free_util=*/0);\n+\t\tstring_list_append(\n+\t\t\t&filter->filter_spec, strbuf_detach(&concatted, NULL));\n+\t}\n+\n+\treturn filter->filter_spec.items[0].string;\n+}\n+\n+const char *expand_list_objects_filter_spec(\n+\tstruct list_objects_filter_options *filter)\n+{\n+\tif (filter->choice == LOFC_BLOB_LIMIT) {\n+\t\tstruct strbuf expanded_spec = STRBUF_INIT;\n+\t\tstrbuf_addf(&expanded_spec, \"blob:limit=%lu\",\n \t\t\t filter->blob_limit_value);\n-\telse if (filter->choice == LOFC_TREE_DEPTH)\n-\t\tstrbuf_addf(expanded_spec, \"tree:%lu\",\n-\t\t\t filter->tree_exclude_depth);\n-\telse\n-\t\tstrbuf_addstr(expanded_spec, filter->filter_spec);\n+\t\tstring_list_clear(&filter->filter_spec, /*free_util=*/0);\n+\t\tstring_list_append(\n+\t\t\t&filter->filter_spec,\n+\t\t\tstrbuf_detach(&expanded_spec, NULL));\n+\t}\n+\n+\treturn list_objects_filter_spec(filter);\n }\n \n void list_objects_filter_release(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tsize_t sub;\n \n \tif (!filter_options)\n \t\treturn;\n-\tfree(filter_options->filter_spec);\n+\tstring_list_clear(&filter_options->filter_spec, /*free_util=*/0);\n \tfree(filter_options->sparse_oid_value);\n \tfree(filter_options->sparse_path_value);\n \tfor (sub = 0; sub < filter_options->sub_nr; sub++)\n \t\tlist_objects_filter_release(&filter_options->sub[sub]);\n \tfree(filter_options->sub);\n \tmemset(filter_options, 0, sizeof(*filter_options));\n }\n \n void partial_clone_register(\n \tconst char *remote,\n-\tconst struct list_objects_filter_options *filter_options)\n+\tstruct list_objects_filter_options *filter_options)\n {\n \t/*\n \t * Record the name of the partial clone remote in the\n \t * config and in the global variable -- the latter is\n \t * used throughout to indicate that partial clone is\n \t * enabled and to expect missing objects.\n \t */\n \tif (repository_format_partial_clone &&\n \t *repository_format_partial_clone &&\n \t strcmp(remote, repository_format_partial_clone))\n@@ -249,32 +266,33 @@ void partial_clone_register(\n \tgit_config_set(\"core.repositoryformatversion\", \"1\");\n \tgit_config_set(\"extensions.partialclone\", remote);\n \n \trepository_format_partial_clone = xstrdup(remote);\n \n \t/*\n \t * Record the initial filter-spec in the config as\n \t * the default for subsequent fetches from this remote.\n \t */\n \tcore_partial_clone_filter_default =\n-\t\txstrdup(filter_options->filter_spec);\n+\t\txstrdup(expand_list_objects_filter_spec(filter_options));\n \tgit_config_set(\"core.partialclonefilter\",\n \t\t core_partial_clone_filter_default);\n }\n \n void partial_clone_get_default_filter_spec(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tstruct strbuf errbuf = STRBUF_INIT;\n \n \t/*\n \t * Parse default value, but silently ignore it if it is invalid.\n \t */\n \tif (!core_partial_clone_filter_default)\n \t\treturn;\n \n-\tfilter_options->filter_spec = strdup(core_partial_clone_filter_default);\n+\tstring_list_append(&filter_options->filter_spec,\n+\t\t\t core_partial_clone_filter_default);\n \tgently_parse_list_objects_filter(filter_options,\n \t\t\t\t\t core_partial_clone_filter_default,\n \t\t\t\t\t &errbuf);\n \tstrbuf_release(&errbuf);\n }\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 8f08ed74a1..1786c80eb4 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -1,15 +1,15 @@\n #ifndef LIST_OBJECTS_FILTER_OPTIONS_H\n #define LIST_OBJECTS_FILTER_OPTIONS_H\n \n #include \"parse-options.h\"\n-#include \"strbuf.h\"\n+#include \"string-list.h\"\n \n /*\n * The list of defined filters for list-objects.\n */\n enum list_objects_filter_choice {\n \tLOFC_DISABLED = 0,\n \tLOFC_BLOB_NONE,\n \tLOFC_BLOB_LIMIT,\n \tLOFC_TREE_DEPTH,\n \tLOFC_SPARSE_OID,\n@@ -18,22 +18,24 @@ enum list_objects_filter_choice {\n \tLOFC__COUNT /* must be last */\n };\n \n struct list_objects_filter_options {\n \t/*\n \t * 'filter_spec' is the raw argument value given on the command line\n \t * or protocol request. (The part after the \"--keyword=\".) For\n \t * commands that launch filtering sub-processes, or for communication\n \t * over the network, don't use this value; use the result of\n \t * expand_list_objects_filter_spec() instead.\n+\t * To get the raw filter spec given by the user, use the result of\n+\t * list_objects_filter_spec().\n \t */\n-\tchar *filter_spec;\n+\tstruct string_list filter_spec;\n \n \t/*\n \t * 'choice' is determined by parsing the filter-spec. This indicates\n \t * the filtering algorithm to use.\n \t */\n \tenum list_objects_filter_choice choice;\n \n \t/*\n \t * Choice is LOFC_DISABLED because \"--no-filter\" was requested.\n \t */\n@@ -71,35 +73,44 @@ int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset);\n \n #define OPT_PARSE_LIST_OBJECTS_FILTER(fo) \\\n \t{ OPTION_CALLBACK, 0, CL_ARG__FILTER, fo, N_(\"args\"), \\\n \t N_(\"object filtering\"), 0, \\\n \t opt_parse_list_objects_filter }\n \n /*\n * Translates abbreviated numbers in the filter's filter_spec into their\n * fully-expanded forms (e.g., \"limit:blob=1k\" becomes \"limit:blob=1024\").\n+ * Returns a string owned by the list_objects_filter_options object.\n *\n- * This form should be used instead of the raw filter_spec field when\n- * communicating with a remote process or subprocess.\n+ * This form should be used instead of the raw list_objects_filter_spec()\n+ * value when communicating with a remote process or subprocess.\n */\n-void expand_list_objects_filter_spec(\n-\tconst struct list_objects_filter_options *filter,\n-\tstruct strbuf *expanded_spec);\n+const char *expand_list_objects_filter_spec(\n+\tstruct list_objects_filter_options *filter);\n+\n+/*\n+ * Returns the filter spec string more or less in the form as the user\n+ * entered it. This form of the filter_spec can be used in user-facing\n+ * messages. Returns a string owned by the list_objects_filter_options\n+ * object.\n+ */\n+const char *list_objects_filter_spec(\n+\tstruct list_objects_filter_options *filter);\n \n void list_objects_filter_release(\n \tstruct list_objects_filter_options *filter_options);\n \n static inline void list_objects_filter_set_no_filter(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tlist_objects_filter_release(filter_options);\n \tfilter_options->no_filter = 1;\n }\n \n void partial_clone_register(\n \tconst char *remote,\n-\tconst struct list_objects_filter_options *filter_options);\n+\tstruct list_objects_filter_options *filter_options);\n void partial_clone_get_default_filter_spec(\n \tstruct list_objects_filter_options *filter_options);\n \n #endif /* LIST_OBJECTS_FILTER_OPTIONS_H */\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex a87341e051..4523c8f066 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -596,18 +596,11 @@ test_expect_success 'rev-list W/ missing=allow-any' '\n # Test expansion of filter specs.\n \n test_expect_success 'expand blob limit in protocol' '\n \tgit -C r2 config --local uploadpack.allowfilter 1 &&\n \tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -c protocol.version=2 clone \\\n \t\t--filter=blob:limit=1k \"file://$(pwd)/r2\" limit &&\n \t! grep \"blob:limit=1k\" trace &&\n \tgrep \"blob:limit=1024\" trace\n '\n \n-test_expect_success 'expand tree depth limit in protocol' '\n-\tGIT_TRACE_PACKET=\"$(pwd)/tree_trace\" git -c protocol.version=2 clone \\\n-\t\t--filter=tree:0k \"file://$(pwd)/r2\" tree &&\n-\t! grep \"tree:0k\" tree_trace &&\n-\tgrep \"tree:0\" tree_trace\n-'\n-\n test_done\ndiff --git a/transport-helper.c b/transport-helper.c\nindex cec83bd663..d6313ef9f5 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -675,27 +675,23 @@ static int fetch(struct transport *transport,\n \t data->transport_options.check_self_contained_and_connected)\n \t\tset_helper_option(transport, \"check-connectivity\", \"true\");\n \n \tif (transport->cloning)\n \t\tset_helper_option(transport, \"cloning\", \"true\");\n \n \tif (data->transport_options.update_shallow)\n \t\tset_helper_option(transport, \"update-shallow\", \"true\");\n \n \tif (data->transport_options.filter_options.choice) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n-\t\texpand_list_objects_filter_spec(\n-\t\t\t&data->transport_options.filter_options,\n-\t\t\t&expanded_filter_spec);\n-\t\tset_helper_option(transport, \"filter\",\n-\t\t\t\t expanded_filter_spec.buf);\n-\t\tstrbuf_release(&expanded_filter_spec);\n+\t\tconst char *spec = expand_list_objects_filter_spec(\n+\t\t\t&data->transport_options.filter_options);\n+\t\tset_helper_option(transport, \"filter\", spec);\n \t}\n \n \tif (data->transport_options.negotiation_tips)\n \t\twarning(\"Ignoring --negotiation-tip because the protocol does not support it.\");\n \n \tif (data->fetch)\n \t\treturn fetch_with_fetch(transport, nr_heads, to_fetch);\n \n \tif (data->import)\n \t\treturn fetch_with_import(transport, nr_heads, to_fetch);\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 24298913c0..a74d293fef 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -133,32 +133,31 @@ static void create_pack_file(const struct object_array *have_obj,\n \n \targv_array_push(&pack_objects.args, \"--stdout\");\n \tif (shallow_nr)\n \t\targv_array_push(&pack_objects.args, \"--shallow\");\n \tif (!no_progress)\n \t\targv_array_push(&pack_objects.args, \"--progress\");\n \tif (use_ofs_delta)\n \t\targv_array_push(&pack_objects.args, \"--delta-base-offset\");\n \tif (use_include_tag)\n \t\targv_array_push(&pack_objects.args, \"--include-tag\");\n-\tif (filter_options.filter_spec) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n-\t\texpand_list_objects_filter_spec(&filter_options,\n-\t\t\t\t\t\t&expanded_filter_spec);\n+\tif (filter_options.choice) {\n+\t\tconst char *spec =\n+\t\t\texpand_list_objects_filter_spec(&filter_options);\n \t\tif (pack_objects.use_shell) {\n \t\t\tstruct strbuf buf = STRBUF_INIT;\n-\t\t\tsq_quote_buf(&buf, expanded_filter_spec.buf);\n+\t\t\tsq_quote_buf(&buf, spec);\n \t\t\targv_array_pushf(&pack_objects.args, \"--filter=%s\", buf.buf);\n \t\t\tstrbuf_release(&buf);\n \t\t} else {\n \t\t\targv_array_pushf(&pack_objects.args, \"--filter=%s\",\n-\t\t\t\t\t expanded_filter_spec.buf);\n+\t\t\t\t\t spec);\n \t\t}\n \t}\n \n \tpack_objects.in = -1;\n \tpack_objects.out = -1;\n \tpack_objects.err = -1;\n \n \tif (start_command(&pack_objects))\n \t\tdie(\"git upload-pack: unable to fork git-pack-objects\");\n \n-- \n2.21.0\n\n"},{"id":"377198","messageId":"e23b7c5ca92bfb23eba3c065b5cb20e9035b3a78.1560462201.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1560462201.git.matvore@google.com","subject":"[PATCH v3 07/10] strbuf: give URL-encoding API a char predicate fn","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-13T21:51:30Z","receivedAt":"2019-06-13T21:51:59Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Allow callers to specify exactly what characters need to be URL-encoded\nand which do not. This new API will be taken advantage of in a patch\nlater in this set.\n\nHelped-by: Jeff King \nSigned-off-by: Matthew DeVore \n---\n credential-store.c | 9 +++++----\n http.c | 6 ++++--\n strbuf.c | 15 ++++++++-------\n strbuf.h | 7 ++++++-\n 4 files changed, 23 insertions(+), 14 deletions(-)\n\ndiff --git a/credential-store.c b/credential-store.c\nindex ac295420dd..c010497cb2 100644\n--- a/credential-store.c\n+++ b/credential-store.c\n@@ -65,29 +65,30 @@ static void rewrite_credential_file(const char *fn, struct credential *c,\n \tparse_credential_file(fn, c, NULL, print_line);\n \tif (commit_lock_file(&credential_lock) < 0)\n \t\tdie_errno(\"unable to write credential store\");\n }\n \n static void store_credential_file(const char *fn, struct credential *c)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \n \tstrbuf_addf(&buf, \"%s://\", c->protocol);\n-\tstrbuf_addstr_urlencode(&buf, c->username, 1);\n+\tstrbuf_addstr_urlencode(&buf, c->username, is_rfc3986_unreserved);\n \tstrbuf_addch(&buf, ':');\n-\tstrbuf_addstr_urlencode(&buf, c->password, 1);\n+\tstrbuf_addstr_urlencode(&buf, c->password, is_rfc3986_unreserved);\n \tstrbuf_addch(&buf, '@');\n \tif (c->host)\n-\t\tstrbuf_addstr_urlencode(&buf, c->host, 1);\n+\t\tstrbuf_addstr_urlencode(&buf, c->host, is_rfc3986_unreserved);\n \tif (c->path) {\n \t\tstrbuf_addch(&buf, '/');\n-\t\tstrbuf_addstr_urlencode(&buf, c->path, 0);\n+\t\tstrbuf_addstr_urlencode(&buf, c->path,\n+\t\t\t\t\tis_rfc3986_reserved_or_unreserved);\n \t}\n \n \trewrite_credential_file(fn, c, &buf);\n \tstrbuf_release(&buf);\n }\n \n static void store_credential(const struct string_list *fns, struct credential *c)\n {\n \tstruct string_list_item *fn;\n \ndiff --git a/http.c b/http.c\nindex 27aa0a3192..938b9e55af 100644\n--- a/http.c\n+++ b/http.c\n@@ -506,23 +506,25 @@ static void var_override(const char **var, char *value)\n static void set_proxyauth_name_password(CURL *result)\n {\n #if LIBCURL_VERSION_NUM >= 0x071301\n \t\tcurl_easy_setopt(result, CURLOPT_PROXYUSERNAME,\n \t\t\tproxy_auth.username);\n \t\tcurl_easy_setopt(result, CURLOPT_PROXYPASSWORD,\n \t\t\tproxy_auth.password);\n #else\n \t\tstruct strbuf s = STRBUF_INIT;\n \n-\t\tstrbuf_addstr_urlencode(&s, proxy_auth.username, 1);\n+\t\tstrbuf_addstr_urlencode(&s, proxy_auth.username,\n+\t\t\t\t\tis_rfc3986_unreserved);\n \t\tstrbuf_addch(&s, ':');\n-\t\tstrbuf_addstr_urlencode(&s, proxy_auth.password, 1);\n+\t\tstrbuf_addstr_urlencode(&s, proxy_auth.password,\n+\t\t\t\t\tis_rfc3986_unreserved);\n \t\tcurl_proxyuserpwd = strbuf_detach(&s, NULL);\n \t\tcurl_easy_setopt(result, CURLOPT_PROXYUSERPWD, curl_proxyuserpwd);\n #endif\n }\n \n static void init_curl_proxy_auth(CURL *result)\n {\n \tif (proxy_auth.username) {\n \t\tif (!proxy_auth.password)\n \t\t\tcredential_fill(&proxy_auth);\ndiff --git a/strbuf.c b/strbuf.c\nindex 0e18b259ce..60ab5144f2 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -767,55 +767,56 @@ void strbuf_addstr_xml_quoted(struct strbuf *buf, const char *s)\n \t\tcase '&':\n \t\t\tstrbuf_addstr(buf, \"&\");\n \t\t\tbreak;\n \t\tcase 0:\n \t\t\treturn;\n \t\t}\n \t\ts++;\n \t}\n }\n \n-static int is_rfc3986_reserved(char ch)\n+int is_rfc3986_reserved_or_unreserved(char ch)\n {\n+\tif (is_rfc3986_unreserved(ch))\n+\t\treturn 1;\n \tswitch (ch) {\n \t\tcase '!': case '*': case '\\'': case '(': case ')': case ';':\n \t\tcase ':': case '@': case '&': case '=': case '+': case '$':\n \t\tcase ',': case '/': case '?': case '#': case '[': case ']':\n \t\t\treturn 1;\n \t}\n \treturn 0;\n }\n \n-static int is_rfc3986_unreserved(char ch)\n+int is_rfc3986_unreserved(char ch)\n {\n \treturn isalnum(ch) ||\n \t\tch == '-' || ch == '_' || ch == '.' || ch == '~';\n }\n \n static void strbuf_add_urlencode(struct strbuf *sb, const char *s, size_t len,\n-\t\t\t\t int reserved)\n+\t\t\t\t char_predicate allow_unencoded_fn)\n {\n \tstrbuf_grow(sb, len);\n \twhile (len--) {\n \t\tchar ch = *s++;\n-\t\tif (is_rfc3986_unreserved(ch) ||\n-\t\t (!reserved && is_rfc3986_reserved(ch)))\n+\t\tif (allow_unencoded_fn(ch))\n \t\t\tstrbuf_addch(sb, ch);\n \t\telse\n \t\t\tstrbuf_addf(sb, \"%%%02x\", (unsigned char)ch);\n \t}\n }\n \n void strbuf_addstr_urlencode(struct strbuf *sb, const char *s,\n-\t\t\t int reserved)\n+\t\t\t char_predicate allow_unencoded_fn)\n {\n-\tstrbuf_add_urlencode(sb, s, strlen(s), reserved);\n+\tstrbuf_add_urlencode(sb, s, strlen(s), allow_unencoded_fn);\n }\n \n void strbuf_humanise_bytes(struct strbuf *buf, off_t bytes)\n {\n \tif (bytes > 1 << 30) {\n \t\tstrbuf_addf(buf, \"%u.%2.2u GiB\",\n \t\t\t (unsigned)(bytes >> 30),\n \t\t\t (unsigned)(bytes & ((1 << 30) - 1)) / 10737419);\n \t} else if (bytes > 1 << 20) {\n \t\tunsigned x = bytes + 5243; /* for rounding */\ndiff --git a/strbuf.h b/strbuf.h\nindex c8d98dfb95..346d722492 100644\n--- a/strbuf.h\n+++ b/strbuf.h\n@@ -659,22 +659,27 @@ void strbuf_branchname(struct strbuf *sb, const char *name,\n \t\t unsigned allowed);\n \n /*\n * Like strbuf_branchname() above, but confirm that the result is\n * syntactically valid to be used as a local branch name in refs/heads/.\n *\n * The return value is \"0\" if the result is valid, and \"-1\" otherwise.\n */\n int strbuf_check_branch_ref(struct strbuf *sb, const char *name);\n \n+typedef int (*char_predicate)(char ch);\n+\n+int is_rfc3986_unreserved(char ch);\n+int is_rfc3986_reserved_or_unreserved(char ch);\n+\n void strbuf_addstr_urlencode(struct strbuf *sb, const char *name,\n-\t\t\t int reserved);\n+\t\t\t char_predicate allow_unencoded_fn);\n \n __attribute__((format (printf,1,2)))\n int printf_ln(const char *fmt, ...);\n __attribute__((format (printf,2,3)))\n int fprintf_ln(FILE *fp, const char *fmt, ...);\n \n char *xstrdup_tolower(const char *);\n char *xstrdup_toupper(const char *);\n \n /**\n-- \n2.21.0\n\n"},{"id":"377199","messageId":"cc59be1cefae7a90a3a093105d09a3f253b5459e.1560462201.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1560462201.git.matvore@google.com","subject":"[PATCH v3 08/10] list-objects-filter-options: allow mult. --filter","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-13T21:51:31Z","receivedAt":"2019-06-13T21:52:02Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Allow combining of multiple filters by simply repeating the --filter\nflag. Before this patch, the user had to combine them in a single flag\nsomewhat awkwardly (e.g. --filter=combine:FOO+BAR), including\nURL-encoding the individual filters.\n\nTo make this work, in the --filter flag parsing callback, rather than\nerror out when we detect that the filter_options struct is already\npopulated, we modify it in-place to contain the added sub-filter. The\nexisting sub-filter becomes the lhs of the combined filter, and the\nnext sub-filter becomes the rhs. We also have to URL-encode the LHS and\nRHS sub-filters.\n\nWe can simplify the operation if the LHS is already a combine: filter.\nIn that case, we just append the URL-encoded RHS sub-filter to the LHS\nspec to get the new spec.\n\nHelped-by: Emily Shaffer \nHelped-by: Jeff Hostetler \nHelped-by: Jeff King \nHelped-by: Junio C Hamano \nSigned-off-by: Matthew DeVore \n---\n Documentation/rev-list-options.txt | 16 ++++++\n list-objects-filter-options.c | 88 +++++++++++++++++++++++++++--\n list-objects-filter-options.h | 11 ++++\n t/t5616-partial-clone.sh | 19 +++++++\n t/t6112-rev-list-filters-objects.sh | 46 +++++++++++++--\n transport.c | 1 +\n upload-pack.c | 2 +\n 7 files changed, 173 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex ddbc1de43f..7b4116f279 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -730,20 +730,36 @@ specification contained in .\n +\n The form '--filter=tree:' omits all blobs and trees whose depth\n from the root tree is >= (minimum depth if an object is located\n at multiple depths in the commits traversed). =0 will not include\n any trees or blobs unless included explicitly in the command-line (or\n standard input when --stdin is used). =1 will include only the\n tree and blobs which are referenced directly by a commit reachable from\n or an explicitly-given object. =2 is like =1\n while also including trees and blobs one more level removed from an\n explicitly-given commit or tree.\n++\n+Multiple '--filter=' flags can be specified to combine filters. Only\n+objects which are accepted by every filter are included.\n++\n+The form '--filter=combine:++...' can also be\n+used to combined several filters, but this is harder than just repeating\n+the '--filter' flag and is usually not necessary. Filters are joined by\n+'{plus}' and individual filters are %-encoded (i.e. URL-encoded).\n+Besides the '{plus}' and '%' characters, the following characters are\n+reserved and also must be encoded: `~!@#$^&*()[]{}\\;\",<>?`+'`+\n+as well as all characters with ASCII code <= `0x20`, which includes\n+space and newline.\n++\n+Other arbitrary characters can also be encoded. For instance,\n+'combine:tree:3+blob:none' and 'combine:tree%3A3+blob%3Anone' are\n+equivalent.\n \n --no-filter::\n \tTurn off any previous `--filter=` argument.\n \n --filter-print-omitted::\n \tOnly useful with `--filter=`; prints a list of the objects omitted\n \tby the filter. Object IDs are prefixed with a ``~'' character.\n \n --missing=::\n \tA debug option to help with future \"partial clone\" development.\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex c9dd41cd06..ce274b1f35 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -1,18 +1,19 @@\n #include \"cache.h\"\n #include \"commit.h\"\n #include \"config.h\"\n #include \"revision.h\"\n #include \"argv-array.h\"\n #include \"list-objects.h\"\n #include \"list-objects-filter.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"trace.h\"\n #include \"url.h\"\n \n static int parse_combine_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg,\n \tstruct strbuf *errbuf);\n \n /*\n * Parse value of the argument to the \"filter\" keyword.\n * On the command line this looks like:\n@@ -168,29 +169,106 @@ static int parse_combine_filter(\n \n cleanup:\n \tstrbuf_list_free(subspecs);\n \tif (result) {\n \t\tlist_objects_filter_release(filter_options);\n \t\tmemset(filter_options, 0, sizeof(*filter_options));\n \t}\n \treturn result;\n }\n \n-int parse_list_objects_filter(struct list_objects_filter_options *filter_options,\n-\t\t\t const char *arg)\n+static int allow_unencoded(char ch)\n+{\n+\tif (ch <= ' ' || ch == '%' || ch == '+')\n+\t\treturn 0;\n+\treturn !strchr(RESERVED_NON_WS, ch);\n+}\n+\n+static void filter_spec_append_urlencode(\n+\tstruct list_objects_filter_options *filter, const char *raw)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n+\tstrbuf_addstr_urlencode(&buf, raw, allow_unencoded);\n+\ttrace_printf(\"Add to combine filter-spec: %s\\n\", buf.buf);\n+\tstring_list_append(&filter->filter_spec, strbuf_detach(&buf, NULL));\n+}\n+\n+/*\n+ * Changes filter_options into an equivalent LOFC_COMBINE filter options\n+ * instance. Does not do anything if filter_options is already LOFC_COMBINE.\n+ */\n+static void transform_to_combine_type(\n+\tstruct list_objects_filter_options *filter_options)\n+{\n+\tassert(filter_options->choice);\n+\tif (filter_options->choice == LOFC_COMBINE)\n+\t\treturn;\n+\t{\n+\t\tconst int initial_sub_alloc = 2;\n+\t\tstruct list_objects_filter_options *sub_array =\n+\t\t\txcalloc(initial_sub_alloc, sizeof(*sub_array));\n+\t\tsub_array[0] = *filter_options;\n+\t\tmemset(filter_options, 0, sizeof(*filter_options));\n+\t\tfilter_options->sub = sub_array;\n+\t\tfilter_options->sub_alloc = initial_sub_alloc;\n+\t}\n+\tfilter_options->sub_nr = 1;\n+\tfilter_options->choice = LOFC_COMBINE;\n+\tstring_list_append(&filter_options->filter_spec, xstrdup(\"combine:\"));\n+\tfilter_spec_append_urlencode(\n+\t\tfilter_options,\n+\t\tlist_objects_filter_spec(&filter_options->sub[0]));\n+\t/*\n+\t * We don't need the filter_spec strings for subfilter specs, only the\n+\t * top level.\n+\t */\n+\tstring_list_clear(&filter_options->sub[0].filter_spec, /*free_util=*/0);\n+}\n+\n+void list_objects_filter_die_if_populated(\n+\tstruct list_objects_filter_options *filter_options)\n+{\n \tif (filter_options->choice)\n \t\tdie(_(\"multiple filter-specs cannot be combined\"));\n-\tstring_list_append(&filter_options->filter_spec, xstrdup(arg));\n-\tif (gently_parse_list_objects_filter(filter_options, arg, &buf))\n-\t\tdie(\"%s\", buf.buf);\n+}\n+\n+int parse_list_objects_filter(\n+\tstruct list_objects_filter_options *filter_options,\n+\tconst char *arg)\n+{\n+\tstruct strbuf errbuf = STRBUF_INIT;\n+\tint parse_error;\n+\n+\tif (!filter_options->choice) {\n+\t\tstring_list_append(&filter_options->filter_spec, xstrdup(arg));\n+\n+\t\tparse_error = gently_parse_list_objects_filter(\n+\t\t\tfilter_options, arg, &errbuf);\n+\t} else {\n+\t\t/*\n+\t\t * Make filter_options an LOFC_COMBINE spec so we can trivially\n+\t\t * add subspecs to it.\n+\t\t */\n+\t\ttransform_to_combine_type(filter_options);\n+\n+\t\tstring_list_append(&filter_options->filter_spec, xstrdup(\"+\"));\n+\t\tfilter_spec_append_urlencode(filter_options, arg);\n+\t\tALLOC_GROW(filter_options->sub, filter_options->sub_nr + 1,\n+\t\t\t filter_options->sub_alloc);\n+\t\tfilter_options = &filter_options->sub[filter_options->sub_nr++];\n+\t\tmemset(filter_options, 0, sizeof(*filter_options));\n+\n+\t\tparse_error = gently_parse_list_objects_filter(\n+\t\t\tfilter_options, arg, &errbuf);\n+\t}\n+\tif (parse_error)\n+\t\tdie(\"%s\", errbuf.buf);\n \treturn 0;\n }\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct list_objects_filter_options *filter_options = opt->value;\n \n \tif (unset || !arg) {\n \t\tlist_objects_filter_set_no_filter(filter_options);\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 1786c80eb4..fe2e4d5649 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -58,20 +58,31 @@ struct list_objects_filter_options {\n \tstruct list_objects_filter_options *sub;\n \n \t/*\n \t * END choice-specific parsed values.\n \t */\n };\n \n /* Normalized command line arguments */\n #define CL_ARG__FILTER \"filter\"\n \n+void list_objects_filter_die_if_populated(\n+\tstruct list_objects_filter_options *filter_options);\n+\n+/*\n+ * Parses the filter spec string given by arg and either (1) simply places the\n+ * result in filter_options if it is not yet populated or (2) combines it with\n+ * the filter already in filter_options if it is already populated. In the case\n+ * of (2), the filter specs are combined as if specified with 'combine:'.\n+ *\n+ * Dies and prints a user-facing message if an error occurs.\n+ */\n int parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg);\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset);\n \n #define OPT_PARSE_LIST_OBJECTS_FILTER(fo) \\\n \t{ OPTION_CALLBACK, 0, CL_ARG__FILTER, fo, N_(\"args\"), \\\n \t N_(\"object filtering\"), 0, \\\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex 9a8f9886b3..11536f4028 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -201,20 +201,39 @@ test_expect_success 'use fsck before and after manually fetching a missing subtr\n \ttest_line_count = 70 fetched_objects &&\n \n \tawk -f print_1.awk fetched_objects |\n \txargs -n1 git -C dst cat-file -t >fetched_types &&\n \n \tsort -u fetched_types >unique_types.observed &&\n \ttest_write_lines blob commit tree >unique_types.expected &&\n \ttest_cmp unique_types.expected unique_types.observed\n '\n \n+test_expect_success 'implicitly construct combine: filter with repeated flags' '\n+\tGIT_TRACE=$(pwd)/trace git clone --bare \\\n+\t\t--filter=blob:none --filter=tree:1 \\\n+\t\t\"file://$(pwd)/srv.bare\" pc2 &&\n+\tgrep \"trace:.* git pack-objects .*--filter=combine:blob:none+tree:1\" \\\n+\t\ttrace &&\n+\tgit -C pc2 rev-list --objects --missing=allow-any HEAD >objects &&\n+\n+\t# We should have gotten some root trees.\n+\tgrep \" $\" objects &&\n+\t# Should not have gotten any non-root trees or blobs.\n+\t! grep \" .\" objects &&\n+\n+\txargs -n 1 git -C pc2 cat-file -t types &&\n+\tsort -u types >unique_types.actual &&\n+\ttest_write_lines commit tree >unique_types.expected &&\n+\ttest_cmp unique_types.expected unique_types.actual\n+'\n+\n test_expect_success 'partial clone fetches blobs pointed to by refs even if normally filtered out' '\n \trm -rf src dst &&\n \tgit init src &&\n \ttest_commit -C src x &&\n \ttest_config -C src uploadpack.allowfilter 1 &&\n \ttest_config -C src uploadpack.allowanysha1inwant 1 &&\n \n \t# Create a tag pointing to a blob.\n \tBLOB=$(echo blob-contents | git -C src hash-object --stdin -w) &&\n \tgit -C src tag myblob \"$BLOB\" &&\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex 4523c8f066..fd8aec4b4f 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -357,21 +357,30 @@ test_expect_success 'verify tree:3 includes everything expected' '\n \n test_expect_success 'combine:... for a simple combination' '\n \tgit -C r3 rev-list --objects --filter=combine:tree:2+blob:none HEAD \\\n \t\t>actual &&\n \n \texpect_has HEAD \"\" &&\n \texpect_has HEAD~1 \"\" &&\n \texpect_has HEAD dir1 &&\n \n \t# There are also 2 commit objects\n-\ttest_line_count = 5 actual\n+\ttest_line_count = 5 actual &&\n+\n+\tcp actual expected &&\n+\n+\t# Try again using repeated --filter - this is equivalent to a manual\n+\t# combine with \"combine:...+...\"\n+\tgit -C r3 rev-list --objects --filter=combine:tree:2 \\\n+\t\t--filter=blob:none HEAD >actual &&\n+\n+\ttest_cmp expected actual\n '\n \n test_expect_success 'combine:... with URL encoding' '\n \tgit -C r3 rev-list --objects \\\n \t\t--filter=combine:tree%3a2+blob:%6Eon%65 HEAD >actual &&\n \n \texpect_has HEAD \"\" &&\n \texpect_has HEAD~1 \"\" &&\n \texpect_has HEAD dir1 &&\n \n@@ -423,24 +432,26 @@ test_expect_success 'combine:... with edge-case hex digits: Ff Aa 0 9' '\n \tgit -C r3 rev-list --objects --filter=\"combine:tree%3A2+blob%3anone\" \\\n \t\tHEAD >actual &&\n \ttest_line_count = 5 actual &&\n \tgit -C r3 rev-list --objects --filter=\"combine:tree:%30\" HEAD >actual &&\n \ttest_line_count = 2 actual &&\n \tgit -C r3 rev-list --objects --filter=\"combine:tree:%39+blob:none\" \\\n \t\tHEAD >actual &&\n \ttest_line_count = 5 actual\n '\n \n-test_expect_success 'add a sparse pattern blob whose path has reserved chars' '\n+test_expect_success 'add sparse pattern blobs whose paths have reserved chars' '\n \tcp r3/pattern r3/pattern1+renamed% &&\n-\tgit -C r3 add pattern1+renamed% &&\n-\tgit -C r3 commit -m \"add sparse pattern file with reserved chars\"\n+\tcp r3/pattern \"r3/p;at%ter+n\" &&\n+\tcp r3/pattern r3/^~pattern &&\n+\tgit -C r3 add pattern1+renamed% \"p;at%ter+n\" ^~pattern &&\n+\tgit -C r3 commit -m \"add sparse pattern files with reserved chars\"\n '\n \n test_expect_success 'combine:... with more than two sub-filters' '\n \tgit -C r3 rev-list --objects \\\n \t\t--filter=combine:tree:3+blob:limit=40+sparse:oid=master:pattern \\\n \t\tHEAD >actual &&\n \n \texpect_has HEAD \"\" &&\n \texpect_has HEAD~1 \"\" &&\n \texpect_has HEAD~2 \"\" &&\n@@ -451,21 +462,46 @@ test_expect_success 'combine:... with more than two sub-filters' '\n \t# Should also have 3 commits\n \ttest_line_count = 9 actual &&\n \n \t# Try again, this time making sure the last sub-filter is only\n \t# URL-decoded once.\n \tcp actual expect &&\n \n \tgit -C r3 rev-list --objects \\\n \t\t--filter=combine:tree:3+blob:limit=40+sparse:oid=master:pattern1%2brenamed%25 \\\n \t\tHEAD >actual &&\n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\n+\t# Use the same composite filter again, but with a pattern file name that\n+\t# requires encoding multiple characters, and use implicit filter\n+\t# combining.\n+\ttest_when_finished \"rm -f trace1\" &&\n+\tGIT_TRACE=$(pwd)/trace1 git -C r3 rev-list --objects \\\n+\t\t--filter=tree:3 --filter=blob:limit=40 \\\n+\t\t--filter=sparse:oid=\"master:p;at%ter+n\" \\\n+\t\tHEAD >actual &&\n+\n+\ttest_cmp expect actual &&\n+\tgrep \"Add to combine filter-spec: sparse:oid=master:p%3bat%25ter%2bn\" \\\n+\t\ttrace1 &&\n+\n+\t# Repeat the above test, but this time, the characters to encode are in\n+\t# the LHS of the combined filter.\n+\ttest_when_finished \"rm -f trace2\" &&\n+\tGIT_TRACE=$(pwd)/trace2 git -C r3 rev-list --objects \\\n+\t\t--filter=sparse:oid=master:^~pattern \\\n+\t\t--filter=tree:3 --filter=blob:limit=40 \\\n+\t\tHEAD >actual &&\n+\n+\ttest_cmp expect actual &&\n+\tgrep \"Add to combine filter-spec: sparse:oid=master:%5e%7epattern\" \\\n+\t\ttrace2\n '\n \n # Test provisional omit collection logic with a repo that has objects appearing\n # at multiple depths - first deeper than the filter's threshold, then shallow.\n \n test_expect_success 'setup r4' '\n \tgit init r4 &&\n \n \techo foo > r4/foo &&\n \tmkdir r4/subdir &&\ndiff --git a/transport.c b/transport.c\nindex f1fcd2c4b0..ee7dd1c062 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -217,20 +217,21 @@ static int set_git_option(struct git_transport_options *opts,\n \t} else if (!strcmp(name, TRANS_OPT_DEEPEN_RELATIVE)) {\n \t\topts->deepen_relative = !!value;\n \t\treturn 0;\n \t} else if (!strcmp(name, TRANS_OPT_FROM_PROMISOR)) {\n \t\topts->from_promisor = !!value;\n \t\treturn 0;\n \t} else if (!strcmp(name, TRANS_OPT_NO_DEPENDENTS)) {\n \t\topts->no_dependents = !!value;\n \t\treturn 0;\n \t} else if (!strcmp(name, TRANS_OPT_LIST_OBJECTS_FILTER)) {\n+\t\tlist_objects_filter_die_if_populated(&opts->filter_options);\n \t\tparse_list_objects_filter(&opts->filter_options, value);\n \t\treturn 0;\n \t}\n \treturn 1;\n }\n \n static int connect_setup(struct transport *transport, int for_push)\n {\n \tstruct git_transport_data *data = transport->data;\n \tint flags = transport->verbose > 0 ? CONNECT_VERBOSE : 0;\ndiff --git a/upload-pack.c b/upload-pack.c\nindex a74d293fef..dda2ac6f44 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -876,20 +876,21 @@ static void receive_needs(struct packet_reader *reader, struct object_array *wan\n \t\tif (process_deepen(reader->line, &depth))\n \t\t\tcontinue;\n \t\tif (process_deepen_since(reader->line, &deepen_since, &deepen_rev_list))\n \t\t\tcontinue;\n \t\tif (process_deepen_not(reader->line, &deepen_not, &deepen_rev_list))\n \t\t\tcontinue;\n \n \t\tif (skip_prefix(reader->line, \"filter \", &arg)) {\n \t\t\tif (!filter_capability_requested)\n \t\t\t\tdie(\"git upload-pack: filtering capability not negotiated\");\n+\t\t\tlist_objects_filter_die_if_populated(&filter_options);\n \t\t\tparse_list_objects_filter(&filter_options, arg);\n \t\t\tcontinue;\n \t\t}\n \n \t\tif (!skip_prefix(reader->line, \"want \", &arg) ||\n \t\t parse_oid_hex(arg, &oid_buf, &features))\n \t\t\tdie(\"git upload-pack: protocol error, \"\n \t\t\t \"expected to get object ID, not '%s'\", reader->line);\n \n \t\tif (parse_feature_request(features, \"deepen-relative\"))\n@@ -1297,20 +1298,21 @@ static void process_args(struct packet_reader *request,\n \t\t\tcontinue;\n \t\tif (process_deepen_not(arg, &data->deepen_not,\n \t\t\t\t &data->deepen_rev_list))\n \t\t\tcontinue;\n \t\tif (!strcmp(arg, \"deepen-relative\")) {\n \t\t\tdata->deepen_relative = 1;\n \t\t\tcontinue;\n \t\t}\n \n \t\tif (allow_filter && skip_prefix(arg, \"filter \", &p)) {\n+\t\t\tlist_objects_filter_die_if_populated(&filter_options);\n \t\t\tparse_list_objects_filter(&filter_options, p);\n \t\t\tcontinue;\n \t\t}\n \n \t\tif ((git_env_bool(\"GIT_TEST_SIDEBAND_ALL\", 0) ||\n \t\t allow_sideband_all) &&\n \t\t !strcmp(arg, \"sideband-all\")) {\n \t\t\tdata->writer.use_sideband = 1;\n \t\t\tcontinue;\n \t\t}\n-- \n2.21.0\n\n"},{"id":"377200","messageId":"148f0936a7a9156b95a8aa71e0a96aeddcccef96.1560462201.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1560462201.git.matvore@google.com","subject":"[PATCH v3 09/10] list-objects-filter-options: clean up use of ALLOC_GROW","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-13T21:51:32Z","receivedAt":"2019-06-13T21:52:05Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Introduce a new macro ALLOC_GROW_BY which automatically zeros the added\narray elements and takes care of updating the nr value. Use the macro in\ncode introduced earlier in this patchset.\n\nSigned-off-by: Matthew DeVore \n---\n cache.h | 22 ++++++++++++++++++++++\n list-objects-filter-options.c | 17 +++++++----------\n 2 files changed, 29 insertions(+), 10 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex b4bb2e2c11..48fb0f63c2 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -653,33 +653,55 @@ int init_db(const char *git_dir, const char *real_git_dir,\n void sanitize_stdfds(void);\n int daemonize(void);\n \n #define alloc_nr(x) (((x)+16)*3/2)\n \n /*\n * Realloc the buffer pointed at by variable 'x' so that it can hold\n * at least 'nr' entries; the number of entries currently allocated\n * is 'alloc', using the standard growing factor alloc_nr() macro.\n *\n+ * Consider using ALLOC_GROW_BY instead of ALLOC_GROW as it has some\n+ * added niceties.\n+ *\n * DO NOT USE any expression with side-effect for 'x', 'nr', or 'alloc'.\n */\n #define ALLOC_GROW(x, nr, alloc) \\\n \tdo { \\\n \t\tif ((nr) > alloc) { \\\n \t\t\tif (alloc_nr(alloc) < (nr)) \\\n \t\t\t\talloc = (nr); \\\n \t\t\telse \\\n \t\t\t\talloc = alloc_nr(alloc); \\\n \t\t\tREALLOC_ARRAY(x, alloc); \\\n \t\t} \\\n \t} while (0)\n \n+/*\n+ * Similar to ALLOC_GROW but handles updating of the nr value and\n+ * zeroing the bytes of the newly-grown array elements.\n+ *\n+ * DO NOT USE any expression with side-effect for any of the\n+ * arguments.\n+ */\n+#define ALLOC_GROW_BY(x, nr, increase, alloc) \\\n+\tdo { \\\n+\t\tif (increase) { \\\n+\t\t\tsize_t new_nr = nr + (increase); \\\n+\t\t\tif (new_nr < nr) \\\n+\t\t\t\tBUG(\"negative growth in ALLOC_GROW_BY\"); \\\n+\t\t\tALLOC_GROW(x, new_nr, alloc); \\\n+\t\t\tmemset((x) + nr, 0, sizeof(*(x)) * (increase)); \\\n+\t\t\tnr = new_nr; \\\n+\t\t} \\\n+\t} while (0)\n+\n /* Initialize and use the cache information */\n struct lock_file;\n void preload_index(struct index_state *index,\n \t\t const struct pathspec *pathspec,\n \t\t unsigned int refresh_flags);\n int do_read_index(struct index_state *istate, const char *path,\n \t\t int must_exist); /* for testting only! */\n int read_index_from(struct index_state *, const char *path,\n \t\t const char *gitdir);\n int is_index_unborn(struct index_state *);\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex ce274b1f35..9f08390628 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -109,28 +109,26 @@ static int has_reserved_character(\n \t}\n \n \treturn 0;\n }\n \n static int parse_combine_subfilter(\n \tstruct list_objects_filter_options *filter_options,\n \tstruct strbuf *subspec,\n \tstruct strbuf *errbuf)\n {\n-\tsize_t new_index = filter_options->sub_nr++;\n+\tsize_t new_index = filter_options->sub_nr;\n \tchar *decoded;\n \tint result;\n \n-\tALLOC_GROW(filter_options->sub, filter_options->sub_nr,\n-\t\t filter_options->sub_alloc);\n-\tmemset(&filter_options->sub[new_index], 0,\n-\t sizeof(*filter_options->sub));\n+\tALLOC_GROW_BY(filter_options->sub, filter_options->sub_nr, 1,\n+\t\t filter_options->sub_alloc);\n \n \tdecoded = url_percent_decode(subspec->buf);\n \n \tresult = has_reserved_character(subspec, errbuf) ||\n \t\tgently_parse_list_objects_filter(\n \t\t\t&filter_options->sub[new_index], decoded, errbuf);\n \n \tfree(decoded);\n \treturn result;\n }\n@@ -245,27 +243,26 @@ int parse_list_objects_filter(\n \t\t\tfilter_options, arg, &errbuf);\n \t} else {\n \t\t/*\n \t\t * Make filter_options an LOFC_COMBINE spec so we can trivially\n \t\t * add subspecs to it.\n \t\t */\n \t\ttransform_to_combine_type(filter_options);\n \n \t\tstring_list_append(&filter_options->filter_spec, xstrdup(\"+\"));\n \t\tfilter_spec_append_urlencode(filter_options, arg);\n-\t\tALLOC_GROW(filter_options->sub, filter_options->sub_nr + 1,\n-\t\t\t filter_options->sub_alloc);\n-\t\tfilter_options = &filter_options->sub[filter_options->sub_nr++];\n-\t\tmemset(filter_options, 0, sizeof(*filter_options));\n+\t\tALLOC_GROW_BY(filter_options->sub, filter_options->sub_nr, 1,\n+\t\t\t filter_options->sub_alloc);\n \n \t\tparse_error = gently_parse_list_objects_filter(\n-\t\t\tfilter_options, arg, &errbuf);\n+\t\t\t&filter_options->sub[filter_options->sub_nr - 1], arg,\n+\t\t\t&errbuf);\n \t}\n \tif (parse_error)\n \t\tdie(\"%s\", errbuf.buf);\n \treturn 0;\n }\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct list_objects_filter_options *filter_options = opt->value;\n-- \n2.21.0\n\n"},{"id":"377201","messageId":"e2d87f8cb6a0079811386e701c7b9f90da163618.1560462201.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1560462201.git.matvore@google.com","subject":"[PATCH v3 10/10] list-objects-filter-options: make parser void","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-13T21:51:33Z","receivedAt":"2019-06-13T21:52:07Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This function always returns 0, so make it return void instead.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects-filter-options.c | 12 +++++-------\n list-objects-filter-options.h | 2 +-\n 2 files changed, 6 insertions(+), 8 deletions(-)\n\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex 9f08390628..a4ebf21a5b 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -222,21 +222,21 @@ static void transform_to_combine_type(\n \tstring_list_clear(&filter_options->sub[0].filter_spec, /*free_util=*/0);\n }\n \n void list_objects_filter_die_if_populated(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tif (filter_options->choice)\n \t\tdie(_(\"multiple filter-specs cannot be combined\"));\n }\n \n-int parse_list_objects_filter(\n+void parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg)\n {\n \tstruct strbuf errbuf = STRBUF_INIT;\n \tint parse_error;\n \n \tif (!filter_options->choice) {\n \t\tstring_list_append(&filter_options->filter_spec, xstrdup(arg));\n \n \t\tparse_error = gently_parse_list_objects_filter(\n@@ -252,34 +252,32 @@ int parse_list_objects_filter(\n \t\tfilter_spec_append_urlencode(filter_options, arg);\n \t\tALLOC_GROW_BY(filter_options->sub, filter_options->sub_nr, 1,\n \t\t\t filter_options->sub_alloc);\n \n \t\tparse_error = gently_parse_list_objects_filter(\n \t\t\t&filter_options->sub[filter_options->sub_nr - 1], arg,\n \t\t\t&errbuf);\n \t}\n \tif (parse_error)\n \t\tdie(\"%s\", errbuf.buf);\n-\treturn 0;\n }\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct list_objects_filter_options *filter_options = opt->value;\n \n-\tif (unset || !arg) {\n+\tif (unset || !arg)\n \t\tlist_objects_filter_set_no_filter(filter_options);\n-\t\treturn 0;\n-\t}\n-\n-\treturn parse_list_objects_filter(filter_options, arg);\n+\telse\n+\t\tparse_list_objects_filter(filter_options, arg);\n+\treturn 0;\n }\n \n const char *list_objects_filter_spec(struct list_objects_filter_options *filter)\n {\n \tif (!filter->filter_spec.nr)\n \t\tBUG(\"no filter_spec available for this filter\");\n \tif (filter->filter_spec.nr != 1) {\n \t\tstruct strbuf concatted = STRBUF_INIT;\n \t\tstrbuf_add_separated_string_list(\n \t\t\t&concatted, \"\", &filter->filter_spec);\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex fe2e4d5649..0a48f541d2 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -69,21 +69,21 @@ void list_objects_filter_die_if_populated(\n \tstruct list_objects_filter_options *filter_options);\n \n /*\n * Parses the filter spec string given by arg and either (1) simply places the\n * result in filter_options if it is not yet populated or (2) combines it with\n * the filter already in filter_options if it is already populated. In the case\n * of (2), the filter specs are combined as if specified with 'combine:'.\n *\n * Dies and prints a user-facing message if an error occurs.\n */\n-int parse_list_objects_filter(\n+void parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg);\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset);\n \n #define OPT_PARSE_LIST_OBJECTS_FILTER(fo) \\\n \t{ OPTION_CALLBACK, 0, CL_ARG__FILTER, fo, N_(\"args\"), \\\n \t N_(\"object filtering\"), 0, \\\n \t opt_parse_list_objects_filter }\n-- \n2.21.0\n\n"},{"id":"377253","messageId":"xmqqpnngt089.fsf@gitster-ct.c.googlers.com","threadId":"51217","inReplyTo":"cover.1560462201.git.matvore@google.com","subject":"Re: [PATCH v3 00/10] Filter combination","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-14T19:50:46Z","receivedAt":"2019-06-14T19:50:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthew DeVore writes:\n\n> It has been a while since a sent a roll-up. Here are the changes since v2:\n>\n> - Re-use more URL-encoding logic in strbuf.c\n> * This was partially achieved by changing the helper function to accept a\n> function that will indicate whether some character must be escaped.\n> - Re-use more URL-decoding logic in url.c\n> - changed the filter_spec strbuf to a string_list to avoid explicit\n> initialization\n> - Remove logic to \"expand\" tree:#k and tree:#m filter specs since there is no\n> server that supports tree:# but does not support tree:#k, as they were\n> implemented at the same time.\n\nSince the v2 of this topic, cc/list-objects-filter-wo-sparse-path\nwas merged to the mainline before Git 2.22 was tagged. As we won't\nbe merging this topic to any maintenance track anyway, it is\nprobably a good time to rebase it on v2.22.0, to avoid unnecessary\nconflicts.\n\nThanks.\n"},{"id":"377278","messageId":"cover.1560558910.git.matvore@google.com","threadId":"51217","inReplyTo":"20190601003603.90794-1-matvore@google.com","subject":"[PATCH v4 00/10] Filter combination","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-15T00:40:06Z","receivedAt":"2019-06-15T00:42:03Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"I had to rebase this onto the latest master rev. master now has the patch which\ndisables the sparse:path filter, and v3 of this patch set has conflicts with it.\nThis version does not so it can be patched in and tried out by others.\n\nI have re-run the test suite on each commit. Sorry for the spamminess.\n\nThanks,\n\nMatthew DeVore (10):\n list-objects-filter: make API easier to use\n list-objects-filter: put omits set in filter struct\n list-objects-filter-options: always supply *errbuf\n list-objects-filter: implement composite filters\n list-objects-filter-options: move error check up\n list-objects-filter-options: make filter_spec a string_list\n strbuf: give URL-encoding API a char predicate fn\n list-objects-filter-options: allow mult. --filter\n list-objects-filter-options: clean up use of ALLOC_GROW\n list-objects-filter-options: make parser void\n\n Documentation/rev-list-options.txt | 16 ++\n builtin/clone.c | 8 +-\n builtin/fetch.c | 9 +-\n builtin/rev-list.c | 6 +-\n cache.h | 22 ++\n credential-store.c | 9 +-\n fetch-pack.c | 20 +-\n http.c | 6 +-\n list-objects-filter-options.c | 267 ++++++++++++++++++----\n list-objects-filter-options.h | 57 ++++-\n list-objects-filter.c | 332 +++++++++++++++++++++-------\n list-objects-filter.h | 35 ++-\n list-objects.c | 55 ++---\n strbuf.c | 15 +-\n strbuf.h | 7 +-\n t/t5616-partial-clone.sh | 19 ++\n t/t6112-rev-list-filters-objects.sh | 194 +++++++++++++++-\n transport-helper.c | 10 +-\n transport.c | 1 +\n upload-pack.c | 13 +-\n url.c | 6 +\n url.h | 8 +\n 22 files changed, 874 insertions(+), 241 deletions(-)\n\n-- \n2.21.0\n\n"},{"id":"377279","messageId":"70568c42ae6d59dacbb36ffb8e4a8828b6595158.1560558910.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1560558910.git.matvore@google.com","subject":"[PATCH v4 01/10] list-objects-filter: make API easier to use","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-15T00:40:07Z","receivedAt":"2019-06-15T00:42:09Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Make the list-objects-filter.h API more opaque and easier to use. This\nprepares for combined filter support, where filters will be created and\nused in a new context.\n\nHelped-by: Jeff Hostetler \nHelped-by: Junio C Hamano \nSigned-off-by: Matthew DeVore \n---\n list-objects-filter.c | 111 ++++++++++++++++++++++++++++--------------\n list-objects-filter.h | 35 ++++++-------\n list-objects.c | 55 +++++++++------------\n 3 files changed, 112 insertions(+), 89 deletions(-)\n\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex 53f90442c5..a8c9d8dfe0 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -19,20 +19,34 @@\n * FILTER_SHOWN_BUT_REVISIT -- we set this bit on tree objects\n * that have been shown, but should be revisited if they appear\n * in the traversal (until we mark it SEEN). This is a way to\n * let us silently de-dup calls to show() in the caller. This\n * is subtly different from the \"revision.h:SHOWN\" and the\n * \"sha1-name.c:ONELINE_SEEN\" bits. And also different from\n * the non-de-dup usage in pack-bitmap.c\n */\n #define FILTER_SHOWN_BUT_REVISIT (1<<21)\n \n+struct filter {\n+\tenum list_objects_filter_result (*filter_object_fn)(\n+\t\tstruct repository *r,\n+\t\tenum list_objects_filter_situation filter_situation,\n+\t\tstruct object *obj,\n+\t\tconst char *pathname,\n+\t\tconst char *filename,\n+\t\tvoid *filter_data);\n+\n+\tvoid (*free_fn)(void *filter_data);\n+\n+\tvoid *filter_data;\n+};\n+\n /*\n * A filter for list-objects to omit ALL blobs from the traversal.\n * And to OPTIONALLY collect a list of the omitted OIDs.\n */\n struct filter_blobs_none_data {\n \tstruct oidset *omits;\n };\n \n static enum list_objects_filter_result filter_blobs_none(\n \tstruct repository *r,\n@@ -60,32 +74,31 @@ static enum list_objects_filter_result filter_blobs_none(\n \tcase LOFS_BLOB:\n \t\tassert(obj->type == OBJ_BLOB);\n \t\tassert((obj->flags & SEEN) == 0);\n \n \t\tif (filter_data->omits)\n \t\t\toidset_insert(filter_data->omits, &obj->oid);\n \t\treturn LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n \t}\n }\n \n-static void *filter_blobs_none__init(\n+static void filter_blobs_none__init(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn)\n+\tstruct filter *filter)\n {\n \tstruct filter_blobs_none_data *d = xcalloc(1, sizeof(*d));\n \td->omits = omitted;\n \n-\t*filter_fn = filter_blobs_none;\n-\t*filter_free_fn = free;\n-\treturn d;\n+\tfilter->filter_data = d;\n+\tfilter->filter_object_fn = filter_blobs_none;\n+\tfilter->free_fn = free;\n }\n \n /*\n * A filter for list-objects to omit ALL trees and blobs from the traversal.\n * Can OPTIONALLY collect a list of the omitted OIDs.\n */\n struct filter_trees_depth_data {\n \tstruct oidset *omits;\n \n \t/*\n@@ -194,35 +207,34 @@ static enum list_objects_filter_result filter_trees_depth(\n }\n \n static void filter_trees_free(void *filter_data) {\n \tstruct filter_trees_depth_data *d = filter_data;\n \tif (!d)\n \t\treturn;\n \toidmap_free(&d->seen_at_depth, 1);\n \tfree(d);\n }\n \n-static void *filter_trees_depth__init(\n+static void filter_trees_depth__init(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn)\n+\tstruct filter *filter)\n {\n \tstruct filter_trees_depth_data *d = xcalloc(1, sizeof(*d));\n \td->omits = omitted;\n \toidmap_init(&d->seen_at_depth, 0);\n \td->exclude_depth = filter_options->tree_exclude_depth;\n \td->current_depth = 0;\n \n-\t*filter_fn = filter_trees_depth;\n-\t*filter_free_fn = filter_trees_free;\n-\treturn d;\n+\tfilter->filter_data = d;\n+\tfilter->filter_object_fn = filter_trees_depth;\n+\tfilter->free_fn = filter_trees_free;\n }\n \n /*\n * A filter for list-objects to omit large blobs.\n * And to OPTIONALLY collect a list of the omitted OIDs.\n */\n struct filter_blobs_limit_data {\n \tstruct oidset *omits;\n \tunsigned long max_bytes;\n };\n@@ -274,33 +286,32 @@ static enum list_objects_filter_result filter_blobs_limit(\n \t\t\toidset_insert(filter_data->omits, &obj->oid);\n \t\treturn LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n \t}\n \n include_it:\n \tif (filter_data->omits)\n \t\toidset_remove(filter_data->omits, &obj->oid);\n \treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n }\n \n-static void *filter_blobs_limit__init(\n+static void filter_blobs_limit__init(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn)\n+\tstruct filter *filter)\n {\n \tstruct filter_blobs_limit_data *d = xcalloc(1, sizeof(*d));\n \td->omits = omitted;\n \td->max_bytes = filter_options->blob_limit_value;\n \n-\t*filter_fn = filter_blobs_limit;\n-\t*filter_free_fn = free;\n-\treturn d;\n+\tfilter->filter_data = d;\n+\tfilter->filter_object_fn = filter_blobs_limit;\n+\tfilter->free_fn = free;\n }\n \n /*\n * A filter driven by a sparse-checkout specification to only\n * include blobs that a sparse checkout would populate.\n *\n * The sparse-checkout spec can be loaded from a blob with the\n * given OID or from a local pathname. We allow an OID because\n * the repo may be bare or we may be doing the filtering on the\n * server.\n@@ -450,70 +461,96 @@ static enum list_objects_filter_result filter_sparse(\n }\n \n \n static void filter_sparse_free(void *filter_data)\n {\n \tstruct filter_sparse_data *d = filter_data;\n \t/* TODO free contents of 'd' */\n \tfree(d);\n }\n \n-static void *filter_sparse_oid__init(\n+static void filter_sparse_oid__init(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn)\n+\tstruct filter *filter)\n {\n \tstruct filter_sparse_data *d = xcalloc(1, sizeof(*d));\n \td->omits = omitted;\n \tif (add_excludes_from_blob_to_list(filter_options->sparse_oid_value,\n \t\t\t\t\t NULL, 0, &d->el) < 0)\n \t\tdie(\"could not load filter specification\");\n \n \tALLOC_GROW(d->array_frame, d->nr + 1, d->alloc);\n \td->array_frame[d->nr].defval = 0; /* default to include */\n \td->array_frame[d->nr].child_prov_omit = 0;\n \n-\t*filter_fn = filter_sparse;\n-\t*filter_free_fn = filter_sparse_free;\n-\treturn d;\n+\tfilter->filter_data = d;\n+\tfilter->filter_object_fn = filter_sparse;\n+\tfilter->free_fn = filter_sparse_free;\n }\n \n-typedef void *(*filter_init_fn)(\n+typedef void (*filter_init_fn)(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn);\n+\tstruct filter *filter);\n \n /*\n * Must match \"enum list_objects_filter_choice\".\n */\n static filter_init_fn s_filters[] = {\n \tNULL,\n \tfilter_blobs_none__init,\n \tfilter_blobs_limit__init,\n \tfilter_trees_depth__init,\n \tfilter_sparse_oid__init,\n };\n \n-void *list_objects_filter__init(\n+struct filter *list_objects_filter__init(\n \tstruct oidset *omitted,\n-\tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn)\n+\tstruct list_objects_filter_options *filter_options)\n {\n+\tstruct filter *filter;\n \tfilter_init_fn init_fn;\n \n \tassert((sizeof(s_filters) / sizeof(s_filters[0])) == LOFC__COUNT);\n \n \tif (filter_options->choice >= LOFC__COUNT)\n \t\tBUG(\"invalid list-objects filter choice: %d\",\n \t\t filter_options->choice);\n \n \tinit_fn = s_filters[filter_options->choice];\n-\tif (init_fn)\n-\t\treturn init_fn(omitted, filter_options,\n-\t\t\t filter_fn, filter_free_fn);\n-\t*filter_fn = NULL;\n-\t*filter_free_fn = NULL;\n-\treturn NULL;\n+\tif (!init_fn)\n+\t\treturn NULL;\n+\n+\tfilter = xcalloc(1, sizeof(*filter));\n+\tinit_fn(omitted, filter_options, filter);\n+\treturn filter;\n+}\n+\n+enum list_objects_filter_result list_objects_filter__filter_object(\n+\tstruct repository *r,\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tstruct filter *filter)\n+{\n+\tif (filter && (obj->flags & NOT_USER_GIVEN))\n+\t\treturn filter->filter_object_fn(r, filter_situation, obj,\n+\t\t\t\t\t\tpathname, filename,\n+\t\t\t\t\t\tfilter->filter_data);\n+\t/*\n+\t * No filter is active or user gave object explicitly. Choose default\n+\t * behavior based on filter situation.\n+\t */\n+\tif (filter_situation == LOFS_END_TREE)\n+\t\treturn 0;\n+\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+}\n+\n+void list_objects_filter__free(struct filter *filter)\n+{\n+\tif (!filter)\n+\t\treturn;\n+\tfilter->free_fn(filter->filter_data);\n+\tfree(filter);\n }\ndiff --git a/list-objects-filter.h b/list-objects-filter.h\nindex 1d45a4ad57..6908954266 100644\n--- a/list-objects-filter.h\n+++ b/list-objects-filter.h\n@@ -53,37 +53,34 @@ enum list_objects_filter_result {\n \tLOFR_DO_SHOW = 1<<1,\n \tLOFR_SKIP_TREE = 1<<2,\n };\n \n enum list_objects_filter_situation {\n \tLOFS_BEGIN_TREE,\n \tLOFS_END_TREE,\n \tLOFS_BLOB\n };\n \n-typedef enum list_objects_filter_result (*filter_object_fn)(\n+struct filter;\n+\n+/* Constructor for the set of defined list-objects filters. */\n+struct filter *list_objects_filter__init(\n+\tstruct oidset *omitted,\n+\tstruct list_objects_filter_options *filter_options);\n+\n+/*\n+ * Lets `filter` decide how to handle the `obj`. If `filter` is NULL, this\n+ * function behaves as expected if no filter is configured: all objects are\n+ * included.\n+ */\n+enum list_objects_filter_result list_objects_filter__filter_object(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n-\tvoid *filter_data);\n-\n-typedef void (*filter_free_fn)(void *filter_data);\n+\tstruct filter *filter);\n \n-/*\n- * Constructor for the set of defined list-objects filters.\n- * Returns a generic \"void *filter_data\".\n- *\n- * The returned \"filter_fn\" will be used by traverse_commit_list()\n- * to filter the results.\n- *\n- * The returned \"filter_free_fn\" is a destructor for the\n- * filter_data.\n- */\n-void *list_objects_filter__init(\n-\tstruct oidset *omitted,\n-\tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn);\n+/* Destroys `filter`. Does nothing if `filter` is null. */\n+void list_objects_filter__free(struct filter *filter);\n \n #endif /* LIST_OBJECTS_FILTER_H */\ndiff --git a/list-objects.c b/list-objects.c\nindex b5651ddd5b..9307d91fb3 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -11,32 +11,31 @@\n #include \"list-objects-filter-options.h\"\n #include \"packfile.h\"\n #include \"object-store.h\"\n #include \"trace.h\"\n \n struct traversal_context {\n \tstruct rev_info *revs;\n \tshow_object_fn show_object;\n \tshow_commit_fn show_commit;\n \tvoid *show_data;\n-\tfilter_object_fn filter_fn;\n-\tvoid *filter_data;\n+\tstruct filter *filter;\n };\n \n static void process_blob(struct traversal_context *ctx,\n \t\t\t struct blob *blob,\n \t\t\t struct strbuf *path,\n \t\t\t const char *name)\n {\n \tstruct object *obj = &blob->object;\n \tsize_t pathlen;\n-\tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\tenum list_objects_filter_result r;\n \n \tif (!ctx->revs->blob_objects)\n \t\treturn;\n \tif (!obj)\n \t\tdie(\"bad blob object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n \n \t/*\n \t * Pre-filter known-missing objects when explicitly requested.\n@@ -47,25 +46,24 @@ static void process_blob(struct traversal_context *ctx,\n \t * may cause the actual filter to report an incomplete list\n \t * of missing objects.\n \t */\n \tif (ctx->revs->exclude_promisor_objects &&\n \t !has_object_file(&obj->oid) &&\n \t is_promisor_object(&obj->oid))\n \t\treturn;\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n-\t\tr = ctx->filter_fn(ctx->revs->repo,\n-\t\t\t\t LOFS_BLOB, obj,\n-\t\t\t\t path->buf, &path->buf[pathlen],\n-\t\t\t\t ctx->filter_data);\n+\tr = list_objects_filter__filter_object(ctx->revs->repo,\n+\t\t\t\t\t LOFS_BLOB, obj,\n+\t\t\t\t\t path->buf, &path->buf[pathlen],\n+\t\t\t\t\t ctx->filter);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n \t\tctx->show_object(obj, path->buf, ctx->show_data);\n \tstrbuf_setlen(path, pathlen);\n }\n \n /*\n * Processing a gitlink entry currently does nothing, since\n * we do not recurse into the subproject.\n@@ -150,21 +148,21 @@ static void process_tree_contents(struct traversal_context *ctx,\n }\n \n static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n \t\t\t const char *name)\n {\n \tstruct object *obj = &tree->object;\n \tstruct rev_info *revs = ctx->revs;\n \tint baselen = base->len;\n-\tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\tenum list_objects_filter_result r;\n \tint failed_parse;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n \tif (!obj)\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n \n \tfailed_parse = parse_tree_gently(tree, 1);\n@@ -179,47 +177,44 @@ static void process_tree(struct traversal_context *ctx,\n \t\t */\n \t\tif (revs->exclude_promisor_objects &&\n \t\t is_promisor_object(&obj->oid))\n \t\t\treturn;\n \n \t\tif (!revs->do_not_die_on_missing_tree)\n \t\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n-\t\tr = ctx->filter_fn(ctx->revs->repo,\n-\t\t\t\t LOFS_BEGIN_TREE, obj,\n-\t\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t\t ctx->filter_data);\n+\tr = list_objects_filter__filter_object(ctx->revs->repo,\n+\t\t\t\t\t LOFS_BEGIN_TREE, obj,\n+\t\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t\t ctx->filter);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n \t\tctx->show_object(obj, base->buf, ctx->show_data);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n \tif (r & LOFR_SKIP_TREE)\n \t\ttrace_printf(\"Skipping contents of tree %s...\\n\", base->buf);\n \telse if (!failed_parse)\n \t\tprocess_tree_contents(ctx, tree, base);\n \n-\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) {\n-\t\tr = ctx->filter_fn(ctx->revs->repo,\n-\t\t\t\t LOFS_END_TREE, obj,\n-\t\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t\t ctx->filter_data);\n-\t\tif (r & LOFR_MARK_SEEN)\n-\t\t\tobj->flags |= SEEN;\n-\t\tif (r & LOFR_DO_SHOW)\n-\t\t\tctx->show_object(obj, base->buf, ctx->show_data);\n-\t}\n+\tr = list_objects_filter__filter_object(ctx->revs->repo,\n+\t\t\t\t\t LOFS_END_TREE, obj,\n+\t\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t\t ctx->filter);\n+\tif (r & LOFR_MARK_SEEN)\n+\t\tobj->flags |= SEEN;\n+\tif (r & LOFR_DO_SHOW)\n+\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \n \tstrbuf_setlen(base, baselen);\n \tfree_tree_buffer(tree);\n }\n \n static void mark_edge_parents_uninteresting(struct commit *commit,\n \t\t\t\t\t struct rev_info *revs,\n \t\t\t\t\t show_edge_fn show_edge)\n {\n \tstruct commit_list *parents;\n@@ -395,38 +390,32 @@ static void do_traverse(struct traversal_context *ctx)\n void traverse_commit_list(struct rev_info *revs,\n \t\t\t show_commit_fn show_commit,\n \t\t\t show_object_fn show_object,\n \t\t\t void *show_data)\n {\n \tstruct traversal_context ctx;\n \tctx.revs = revs;\n \tctx.show_commit = show_commit;\n \tctx.show_object = show_object;\n \tctx.show_data = show_data;\n-\tctx.filter_fn = NULL;\n-\tctx.filter_data = NULL;\n+\tctx.filter = NULL;\n \tdo_traverse(&ctx);\n }\n \n void traverse_commit_list_filtered(\n \tstruct list_objects_filter_options *filter_options,\n \tstruct rev_info *revs,\n \tshow_commit_fn show_commit,\n \tshow_object_fn show_object,\n \tvoid *show_data,\n \tstruct oidset *omitted)\n {\n \tstruct traversal_context ctx;\n-\tfilter_free_fn filter_free_fn = NULL;\n \n \tctx.revs = revs;\n \tctx.show_object = show_object;\n \tctx.show_commit = show_commit;\n \tctx.show_data = show_data;\n-\tctx.filter_fn = NULL;\n-\n-\tctx.filter_data = list_objects_filter__init(omitted, filter_options,\n-\t\t\t\t\t\t &ctx.filter_fn, &filter_free_fn);\n+\tctx.filter = list_objects_filter__init(omitted, filter_options);\n \tdo_traverse(&ctx);\n-\tif (ctx.filter_data && filter_free_fn)\n-\t\tfilter_free_fn(ctx.filter_data);\n+\tlist_objects_filter__free(ctx.filter);\n }\n-- \n2.21.0\n\n"},{"id":"377280","messageId":"d4508639949f08e8beb949d3ccaede0f272f136f.1560558910.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1560558910.git.matvore@google.com","subject":"[PATCH v4 02/10] list-objects-filter: put omits set in filter struct","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-15T00:40:08Z","receivedAt":"2019-06-15T00:42:09Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"The oidset *omits pointer must be accessed by the combine filter in a\ntype-agnostic way once the graph traversal is over. Store that pointer\nin the general `filter` struct. This will be used in a follow-up patch\nto implement the combine filter.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects-filter.c | 68 +++++++++++++++++--------------------------\n 1 file changed, 26 insertions(+), 42 deletions(-)\n\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex a8c9d8dfe0..b259039bd0 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -26,88 +26,76 @@\n */\n #define FILTER_SHOWN_BUT_REVISIT (1<<21)\n \n struct filter {\n \tenum list_objects_filter_result (*filter_object_fn)(\n \t\tstruct repository *r,\n \t\tenum list_objects_filter_situation filter_situation,\n \t\tstruct object *obj,\n \t\tconst char *pathname,\n \t\tconst char *filename,\n+\t\tstruct oidset *omits,\n \t\tvoid *filter_data);\n \n \tvoid (*free_fn)(void *filter_data);\n \n \tvoid *filter_data;\n-};\n \n-/*\n- * A filter for list-objects to omit ALL blobs from the traversal.\n- * And to OPTIONALLY collect a list of the omitted OIDs.\n- */\n-struct filter_blobs_none_data {\n+\t/* If non-NULL, the filter collects a list of the omitted OIDs here. */\n \tstruct oidset *omits;\n };\n \n static enum list_objects_filter_result filter_blobs_none(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n+\tstruct oidset *omits,\n \tvoid *filter_data_)\n {\n-\tstruct filter_blobs_none_data *filter_data = filter_data_;\n-\n \tswitch (filter_situation) {\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\t/* always include all tree objects */\n \t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \n \tcase LOFS_END_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\treturn LOFR_ZERO;\n \n \tcase LOFS_BLOB:\n \t\tassert(obj->type == OBJ_BLOB);\n \t\tassert((obj->flags & SEEN) == 0);\n \n-\t\tif (filter_data->omits)\n-\t\t\toidset_insert(filter_data->omits, &obj->oid);\n+\t\tif (omits)\n+\t\t\toidset_insert(omits, &obj->oid);\n \t\treturn LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n \t}\n }\n \n static void filter_blobs_none__init(\n-\tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter)\n {\n-\tstruct filter_blobs_none_data *d = xcalloc(1, sizeof(*d));\n-\td->omits = omitted;\n-\n-\tfilter->filter_data = d;\n \tfilter->filter_object_fn = filter_blobs_none;\n \tfilter->free_fn = free;\n }\n \n /*\n * A filter for list-objects to omit ALL trees and blobs from the traversal.\n * Can OPTIONALLY collect a list of the omitted OIDs.\n */\n struct filter_trees_depth_data {\n-\tstruct oidset *omits;\n-\n \t/*\n \t * Maps trees to the minimum depth at which they were seen. It is not\n \t * necessary to re-traverse a tree at deeper or equal depths than it has\n \t * already been traversed.\n \t *\n \t * We can't use LOFR_MARK_SEEN for tree objects since this will prevent\n \t * it from being traversed at shallower depths.\n \t */\n \tstruct oidmap seen_at_depth;\n \n@@ -116,38 +104,39 @@ struct filter_trees_depth_data {\n };\n \n struct seen_map_entry {\n \tstruct oidmap_entry base;\n \tsize_t depth;\n };\n \n /* Returns 1 if the oid was in the omits set before it was invoked. */\n static int filter_trees_update_omits(\n \tstruct object *obj,\n-\tstruct filter_trees_depth_data *filter_data,\n+\tstruct oidset *omits,\n \tint include_it)\n {\n-\tif (!filter_data->omits)\n+\tif (!omits)\n \t\treturn 0;\n \n \tif (include_it)\n-\t\treturn oidset_remove(filter_data->omits, &obj->oid);\n+\t\treturn oidset_remove(omits, &obj->oid);\n \telse\n-\t\treturn oidset_insert(filter_data->omits, &obj->oid);\n+\t\treturn oidset_insert(omits, &obj->oid);\n }\n \n static enum list_objects_filter_result filter_trees_depth(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n+\tstruct oidset *omits,\n \tvoid *filter_data_)\n {\n \tstruct filter_trees_depth_data *filter_data = filter_data_;\n \tstruct seen_map_entry *seen_info;\n \tint include_it = filter_data->current_depth <\n \t\tfilter_data->exclude_depth;\n \tint filter_res;\n \tint already_seen;\n \n \t/*\n@@ -158,47 +147,47 @@ static enum list_objects_filter_result filter_trees_depth(\n \tswitch (filter_situation) {\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n \tcase LOFS_END_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\tfilter_data->current_depth--;\n \t\treturn LOFR_ZERO;\n \n \tcase LOFS_BLOB:\n-\t\tfilter_trees_update_omits(obj, filter_data, include_it);\n+\t\tfilter_trees_update_omits(obj, omits, include_it);\n \t\treturn include_it ? LOFR_MARK_SEEN | LOFR_DO_SHOW : LOFR_ZERO;\n \n \tcase LOFS_BEGIN_TREE:\n \t\tseen_info = oidmap_get(\n \t\t\t&filter_data->seen_at_depth, &obj->oid);\n \t\tif (!seen_info) {\n \t\t\tseen_info = xcalloc(1, sizeof(*seen_info));\n \t\t\toidcpy(&seen_info->base.oid, &obj->oid);\n \t\t\tseen_info->depth = filter_data->current_depth;\n \t\t\toidmap_put(&filter_data->seen_at_depth, seen_info);\n \t\t\talready_seen = 0;\n \t\t} else {\n \t\t\talready_seen =\n \t\t\t\tfilter_data->current_depth >= seen_info->depth;\n \t\t}\n \n \t\tif (already_seen) {\n \t\t\tfilter_res = LOFR_SKIP_TREE;\n \t\t} else {\n \t\t\tint been_omitted = filter_trees_update_omits(\n-\t\t\t\tobj, filter_data, include_it);\n+\t\t\t\tobj, omits, include_it);\n \t\t\tseen_info->depth = filter_data->current_depth;\n \n \t\t\tif (include_it)\n \t\t\t\tfilter_res = LOFR_DO_SHOW;\n-\t\t\telse if (filter_data->omits && !been_omitted)\n+\t\t\telse if (omits && !been_omitted)\n \t\t\t\t/*\n \t\t\t\t * Must update omit information of children\n \t\t\t\t * recursively; they have not been omitted yet.\n \t\t\t\t */\n \t\t\t\tfilter_res = LOFR_ZERO;\n \t\t\telse\n \t\t\t\tfilter_res = LOFR_SKIP_TREE;\n \t\t}\n \n \t\tfilter_data->current_depth++;\n@@ -208,50 +197,48 @@ static enum list_objects_filter_result filter_trees_depth(\n \n static void filter_trees_free(void *filter_data) {\n \tstruct filter_trees_depth_data *d = filter_data;\n \tif (!d)\n \t\treturn;\n \toidmap_free(&d->seen_at_depth, 1);\n \tfree(d);\n }\n \n static void filter_trees_depth__init(\n-\tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter)\n {\n \tstruct filter_trees_depth_data *d = xcalloc(1, sizeof(*d));\n-\td->omits = omitted;\n \toidmap_init(&d->seen_at_depth, 0);\n \td->exclude_depth = filter_options->tree_exclude_depth;\n \td->current_depth = 0;\n \n \tfilter->filter_data = d;\n \tfilter->filter_object_fn = filter_trees_depth;\n \tfilter->free_fn = filter_trees_free;\n }\n \n /*\n * A filter for list-objects to omit large blobs.\n * And to OPTIONALLY collect a list of the omitted OIDs.\n */\n struct filter_blobs_limit_data {\n-\tstruct oidset *omits;\n \tunsigned long max_bytes;\n };\n \n static enum list_objects_filter_result filter_blobs_limit(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n+\tstruct oidset *omits,\n \tvoid *filter_data_)\n {\n \tstruct filter_blobs_limit_data *filter_data = filter_data_;\n \tunsigned long object_length;\n \tenum object_type t;\n \n \tswitch (filter_situation) {\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n@@ -275,38 +262,36 @@ static enum list_objects_filter_result filter_blobs_limit(\n \t\t\t * apply the size filter criteria. Be conservative\n \t\t\t * and force show it (and let the caller deal with\n \t\t\t * the ambiguity).\n \t\t\t */\n \t\t\tgoto include_it;\n \t\t}\n \n \t\tif (object_length < filter_data->max_bytes)\n \t\t\tgoto include_it;\n \n-\t\tif (filter_data->omits)\n-\t\t\toidset_insert(filter_data->omits, &obj->oid);\n+\t\tif (omits)\n+\t\t\toidset_insert(omits, &obj->oid);\n \t\treturn LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n \t}\n \n include_it:\n-\tif (filter_data->omits)\n-\t\toidset_remove(filter_data->omits, &obj->oid);\n+\tif (omits)\n+\t\toidset_remove(omits, &obj->oid);\n \treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n }\n \n static void filter_blobs_limit__init(\n-\tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter)\n {\n \tstruct filter_blobs_limit_data *d = xcalloc(1, sizeof(*d));\n-\td->omits = omitted;\n \td->max_bytes = filter_options->blob_limit_value;\n \n \tfilter->filter_data = d;\n \tfilter->filter_object_fn = filter_blobs_limit;\n \tfilter->free_fn = free;\n }\n \n /*\n * A filter driven by a sparse-checkout specification to only\n * include blobs that a sparse checkout would populate.\n@@ -330,33 +315,33 @@ struct frame {\n \t * omitted objects.\n \t *\n \t * 0 if everything (recursively) contained in this directory\n \t * has been explicitly included (SHOWN) in the result and\n \t * the directory may be short-cut later in the traversal.\n \t */\n \tunsigned child_prov_omit : 1;\n };\n \n struct filter_sparse_data {\n-\tstruct oidset *omits;\n \tstruct exclude_list el;\n \n \tsize_t nr, alloc;\n \tstruct frame *array_frame;\n };\n \n static enum list_objects_filter_result filter_sparse(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n+\tstruct oidset *omits,\n \tvoid *filter_data_)\n {\n \tstruct filter_sparse_data *filter_data = filter_data_;\n \tint val, dtype;\n \tstruct frame *frame;\n \n \tswitch (filter_situation) {\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n@@ -425,78 +410,75 @@ static enum list_objects_filter_result filter_sparse(\n \n \t\tframe = &filter_data->array_frame[filter_data->nr];\n \n \t\tdtype = DT_REG;\n \t\tval = is_excluded_from_list(pathname, strlen(pathname),\n \t\t\t\t\t filename, &dtype, &filter_data->el,\n \t\t\t\t\t r->index);\n \t\tif (val < 0)\n \t\t\tval = frame->defval;\n \t\tif (val > 0) {\n-\t\t\tif (filter_data->omits)\n-\t\t\t\toidset_remove(filter_data->omits, &obj->oid);\n+\t\t\tif (omits)\n+\t\t\t\toidset_remove(omits, &obj->oid);\n \t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \t\t}\n \n \t\t/*\n \t\t * Provisionally omit it. We've already established that\n \t\t * this pathname is not in the sparse-checkout specification\n \t\t * with the CURRENT pathname, so we *WANT* to omit this blob.\n \t\t *\n \t\t * However, a pathname elsewhere in the tree may also\n \t\t * reference this same blob, so we cannot reject it yet.\n \t\t * Leave the LOFR_ bits unset so that if the blob appears\n \t\t * again in the traversal, we will be asked again.\n \t\t */\n-\t\tif (filter_data->omits)\n-\t\t\toidset_insert(filter_data->omits, &obj->oid);\n+\t\tif (omits)\n+\t\t\toidset_insert(omits, &obj->oid);\n \n \t\t/*\n \t\t * Remember that at least 1 blob in this tree was\n \t\t * provisionally omitted. This prevents us from short\n \t\t * cutting the tree in future iterations.\n \t\t */\n \t\tframe->child_prov_omit = 1;\n \t\treturn LOFR_ZERO;\n \t}\n }\n \n \n static void filter_sparse_free(void *filter_data)\n {\n \tstruct filter_sparse_data *d = filter_data;\n \t/* TODO free contents of 'd' */\n \tfree(d);\n }\n \n static void filter_sparse_oid__init(\n-\tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter)\n {\n \tstruct filter_sparse_data *d = xcalloc(1, sizeof(*d));\n-\td->omits = omitted;\n \tif (add_excludes_from_blob_to_list(filter_options->sparse_oid_value,\n \t\t\t\t\t NULL, 0, &d->el) < 0)\n \t\tdie(\"could not load filter specification\");\n \n \tALLOC_GROW(d->array_frame, d->nr + 1, d->alloc);\n \td->array_frame[d->nr].defval = 0; /* default to include */\n \td->array_frame[d->nr].child_prov_omit = 0;\n \n \tfilter->filter_data = d;\n \tfilter->filter_object_fn = filter_sparse;\n \tfilter->free_fn = filter_sparse_free;\n }\n \n typedef void (*filter_init_fn)(\n-\tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter);\n \n /*\n * Must match \"enum list_objects_filter_choice\".\n */\n static filter_init_fn s_filters[] = {\n \tNULL,\n \tfilter_blobs_none__init,\n \tfilter_blobs_limit__init,\n@@ -515,35 +497,37 @@ struct filter *list_objects_filter__init(\n \n \tif (filter_options->choice >= LOFC__COUNT)\n \t\tBUG(\"invalid list-objects filter choice: %d\",\n \t\t filter_options->choice);\n \n \tinit_fn = s_filters[filter_options->choice];\n \tif (!init_fn)\n \t\treturn NULL;\n \n \tfilter = xcalloc(1, sizeof(*filter));\n-\tinit_fn(omitted, filter_options, filter);\n+\tfilter->omits = omitted;\n+\tinit_fn(filter_options, filter);\n \treturn filter;\n }\n \n enum list_objects_filter_result list_objects_filter__filter_object(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n \tstruct filter *filter)\n {\n \tif (filter && (obj->flags & NOT_USER_GIVEN))\n \t\treturn filter->filter_object_fn(r, filter_situation, obj,\n \t\t\t\t\t\tpathname, filename,\n+\t\t\t\t\t\tfilter->omits,\n \t\t\t\t\t\tfilter->filter_data);\n \t/*\n \t * No filter is active or user gave object explicitly. Choose default\n \t * behavior based on filter situation.\n \t */\n \tif (filter_situation == LOFS_END_TREE)\n \t\treturn 0;\n \treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n }\n \n-- \n2.21.0\n\n"},{"id":"377281","messageId":"1e2ee8a15f32acbd0aa02e51641be4b209e9505b.1560558910.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1560558910.git.matvore@google.com","subject":"[PATCH v4 03/10] list-objects-filter-options: always supply *errbuf","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-15T00:40:09Z","receivedAt":"2019-06-15T00:42:12Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Making errbuf an optional argument complicates error reporting. Fix this\nby making all callers supply an errbuf, even if they may ignore it. This\nwill be important in follow-up patches where the filter-spec parsing has\nmore pitfalls and possible errors.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects-filter-options.c | 21 ++++++++-------------\n 1 file changed, 8 insertions(+), 13 deletions(-)\n\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex a15d0f7829..8e7b4f96fa 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -23,47 +23,40 @@\n * convenience of the current command.\n */\n static int gently_parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg,\n \tstruct strbuf *errbuf)\n {\n \tconst char *v0;\n \n \tif (filter_options->choice) {\n-\t\tif (errbuf) {\n-\t\t\tstrbuf_addstr(\n-\t\t\t\terrbuf,\n-\t\t\t\t_(\"multiple filter-specs cannot be combined\"));\n-\t\t}\n+\t\tstrbuf_addstr(\n+\t\t\terrbuf, _(\"multiple filter-specs cannot be combined\"));\n \t\treturn 1;\n \t}\n \n \tfilter_options->filter_spec = strdup(arg);\n \n \tif (!strcmp(arg, \"blob:none\")) {\n \t\tfilter_options->choice = LOFC_BLOB_NONE;\n \t\treturn 0;\n \n \t} else if (skip_prefix(arg, \"blob:limit=\", &v0)) {\n \t\tif (git_parse_ulong(v0, &filter_options->blob_limit_value)) {\n \t\t\tfilter_options->choice = LOFC_BLOB_LIMIT;\n \t\t\treturn 0;\n \t\t}\n \n \t} else if (skip_prefix(arg, \"tree:\", &v0)) {\n \t\tif (!git_parse_ulong(v0, &filter_options->tree_exclude_depth)) {\n-\t\t\tif (errbuf) {\n-\t\t\t\tstrbuf_addstr(\n-\t\t\t\t\terrbuf,\n-\t\t\t\t\t_(\"expected 'tree:'\"));\n-\t\t\t}\n+\t\t\tstrbuf_addstr(errbuf, _(\"expected 'tree:'\"));\n \t\t\treturn 1;\n \t\t}\n \t\tfilter_options->choice = LOFC_TREE_DEPTH;\n \t\treturn 0;\n \n \t} else if (skip_prefix(arg, \"sparse:oid=\", &v0)) {\n \t\tstruct object_context oc;\n \t\tstruct object_id sparse_oid;\n \n \t\t/*\n@@ -83,22 +76,21 @@ static int gently_parse_list_objects_filter(\n \t\t\t\terrbuf,\n \t\t\t\t_(\"sparse:path filters support has been dropped\"));\n \t\t}\n \t\treturn 1;\n \t}\n \t/*\n \t * Please update _git_fetch() in git-completion.bash when you\n \t * add new filters\n \t */\n \n-\tif (errbuf)\n-\t\tstrbuf_addf(errbuf, \"invalid filter-spec '%s'\", arg);\n+\tstrbuf_addf(errbuf, \"invalid filter-spec '%s'\", arg);\n \n \tmemset(filter_options, 0, sizeof(*filter_options));\n \treturn 1;\n }\n \n int parse_list_objects_filter(struct list_objects_filter_options *filter_options,\n \t\t\t const char *arg)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tif (gently_parse_list_objects_filter(filter_options, arg, &buf))\n@@ -168,19 +160,22 @@ void partial_clone_register(\n \t */\n \tcore_partial_clone_filter_default =\n \t\txstrdup(filter_options->filter_spec);\n \tgit_config_set(\"core.partialclonefilter\",\n \t\t core_partial_clone_filter_default);\n }\n \n void partial_clone_get_default_filter_spec(\n \tstruct list_objects_filter_options *filter_options)\n {\n+\tstruct strbuf errbuf = STRBUF_INIT;\n+\n \t/*\n \t * Parse default value, but silently ignore it if it is invalid.\n \t */\n \tif (!core_partial_clone_filter_default)\n \t\treturn;\n \tgently_parse_list_objects_filter(filter_options,\n \t\t\t\t\t core_partial_clone_filter_default,\n-\t\t\t\t\t NULL);\n+\t\t\t\t\t &errbuf);\n+\tstrbuf_release(&errbuf);\n }\n-- \n2.21.0\n\n"},{"id":"377282","messageId":"47a2680875e6f68fbf1f2e5a5a2630d263cdf426.1560558910.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1560558910.git.matvore@google.com","subject":"[PATCH v4 04/10] list-objects-filter: implement composite filters","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-15T00:40:10Z","receivedAt":"2019-06-15T00:42:15Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Allow combining filters such that only objects accepted by all filters\nare shown. The motivation for this is to allow getting directory\nlistings without also fetching blobs. This can be done by combining\nblob:none with tree:. There are massive repositories that have\nlarger-than-expected trees - even if you include only a single commit.\n\nThe current usage requires passing the filter to rev-list in the\nfollowing form:\n\n\t--filter= --filter= ...\n\nSuch usage is currently an error, so giving it a meaning is backwards-\ncompatible.\n\nThe URL-encoding scheme is being introduced before the repeated flag\nlogic, and the user-facing documentation for URL-encoding is being\nwithheld until the repeated flag feature is implemented. The\nURL-encoding is in general not meant to be used directly by the user,\nand it is better to describe the URL-encoding feature in terms of the\nrepeated flag.\n\nHelped-by: Emily Shaffer \nHelped-by: Jeff Hostetler \nHelped-by: Junio C Hamano \nSigned-off-by: Matthew DeVore \n---\n list-objects-filter-options.c | 106 ++++++++++++++++++-\n list-objects-filter-options.h | 17 ++-\n list-objects-filter.c | 159 ++++++++++++++++++++++++++++\n t/t6112-rev-list-filters-objects.sh | 151 +++++++++++++++++++++++++-\n url.c | 6 ++\n url.h | 8 ++\n 6 files changed, 441 insertions(+), 6 deletions(-)\n\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex 8e7b4f96fa..1c402c6059 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -1,18 +1,24 @@\n #include \"cache.h\"\n #include \"commit.h\"\n #include \"config.h\"\n #include \"revision.h\"\n #include \"argv-array.h\"\n #include \"list-objects.h\"\n #include \"list-objects-filter.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"url.h\"\n+\n+static int parse_combine_filter(\n+\tstruct list_objects_filter_options *filter_options,\n+\tconst char *arg,\n+\tstruct strbuf *errbuf);\n \n /*\n * Parse value of the argument to the \"filter\" keyword.\n * On the command line this looks like:\n * --filter=\n * and in the pack protocol as:\n * \"filter\" SP \n *\n * The filter keyword will be used by many commands.\n * See Documentation/rev-list-options.txt for allowed values for .\n@@ -28,22 +34,20 @@ static int gently_parse_list_objects_filter(\n \tstruct strbuf *errbuf)\n {\n \tconst char *v0;\n \n \tif (filter_options->choice) {\n \t\tstrbuf_addstr(\n \t\t\terrbuf, _(\"multiple filter-specs cannot be combined\"));\n \t\treturn 1;\n \t}\n \n-\tfilter_options->filter_spec = strdup(arg);\n-\n \tif (!strcmp(arg, \"blob:none\")) {\n \t\tfilter_options->choice = LOFC_BLOB_NONE;\n \t\treturn 0;\n \n \t} else if (skip_prefix(arg, \"blob:limit=\", &v0)) {\n \t\tif (git_parse_ulong(v0, &filter_options->blob_limit_value)) {\n \t\t\tfilter_options->choice = LOFC_BLOB_LIMIT;\n \t\t\treturn 0;\n \t\t}\n \n@@ -70,36 +74,125 @@ static int gently_parse_list_objects_filter(\n \t\tfilter_options->choice = LOFC_SPARSE_OID;\n \t\treturn 0;\n \n \t} else if (skip_prefix(arg, \"sparse:path=\", &v0)) {\n \t\tif (errbuf) {\n \t\t\tstrbuf_addstr(\n \t\t\t\terrbuf,\n \t\t\t\t_(\"sparse:path filters support has been dropped\"));\n \t\t}\n \t\treturn 1;\n+\n+\t} else if (skip_prefix(arg, \"combine:\", &v0)) {\n+\t\treturn parse_combine_filter(filter_options, v0, errbuf);\n+\n \t}\n \t/*\n \t * Please update _git_fetch() in git-completion.bash when you\n \t * add new filters\n \t */\n \n \tstrbuf_addf(errbuf, \"invalid filter-spec '%s'\", arg);\n \n \tmemset(filter_options, 0, sizeof(*filter_options));\n \treturn 1;\n }\n \n+static const char *RESERVED_NON_WS = \"~`!@#$^&*()[]{}\\\\;'\\\",<>?\";\n+\n+static int has_reserved_character(\n+\tstruct strbuf *sub_spec, struct strbuf *errbuf)\n+{\n+\tconst char *c = sub_spec->buf;\n+\twhile (*c) {\n+\t\tif (*c <= ' ' || strchr(RESERVED_NON_WS, *c)) {\n+\t\t\tstrbuf_addf(errbuf,\n+\t\t\t\t \"must escape char in sub-filter-spec: '%c'\",\n+\t\t\t\t *c);\n+\t\t\treturn 1;\n+\t\t}\n+\t\tc++;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int parse_combine_subfilter(\n+\tstruct list_objects_filter_options *filter_options,\n+\tstruct strbuf *subspec,\n+\tstruct strbuf *errbuf)\n+{\n+\tsize_t new_index = filter_options->sub_nr++;\n+\tchar *decoded;\n+\tint result;\n+\n+\tALLOC_GROW(filter_options->sub, filter_options->sub_nr,\n+\t\t filter_options->sub_alloc);\n+\tmemset(&filter_options->sub[new_index], 0,\n+\t sizeof(*filter_options->sub));\n+\n+\tdecoded = url_percent_decode(subspec->buf);\n+\n+\tresult = has_reserved_character(subspec, errbuf) ||\n+\t\tgently_parse_list_objects_filter(\n+\t\t\t&filter_options->sub[new_index], decoded, errbuf);\n+\n+\tfree(decoded);\n+\treturn result;\n+}\n+\n+static int parse_combine_filter(\n+\tstruct list_objects_filter_options *filter_options,\n+\tconst char *arg,\n+\tstruct strbuf *errbuf)\n+{\n+\tstruct strbuf **subspecs = strbuf_split_str(arg, '+', 0);\n+\tsize_t sub;\n+\tint result = 0;\n+\n+\tif (!subspecs[0]) {\n+\t\tstrbuf_addf(errbuf,\n+\t\t\t _(\"expected something after combine:\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tfor (sub = 0; subspecs[sub] && !result; sub++) {\n+\t\tif (subspecs[sub + 1]) {\n+\t\t\t/*\n+\t\t\t * This is not the last subspec. Remove trailing \"+\" so\n+\t\t\t * we can parse it.\n+\t\t\t */\n+\t\t\tsize_t last = subspecs[sub]->len - 1;\n+\t\t\tassert(subspecs[sub]->buf[last] == '+');\n+\t\t\tstrbuf_remove(subspecs[sub], last, 1);\n+\t\t}\n+\t\tresult = parse_combine_subfilter(\n+\t\t\tfilter_options, subspecs[sub], errbuf);\n+\t}\n+\n+\tfilter_options->choice = LOFC_COMBINE;\n+\n+cleanup:\n+\tstrbuf_list_free(subspecs);\n+\tif (result) {\n+\t\tlist_objects_filter_release(filter_options);\n+\t\tmemset(filter_options, 0, sizeof(*filter_options));\n+\t}\n+\treturn result;\n+}\n+\n int parse_list_objects_filter(struct list_objects_filter_options *filter_options,\n \t\t\t const char *arg)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n+\tfilter_options->filter_spec = strdup(arg);\n \tif (gently_parse_list_objects_filter(filter_options, arg, &buf))\n \t\tdie(\"%s\", buf.buf);\n \treturn 0;\n }\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct list_objects_filter_options *filter_options = opt->value;\n \n@@ -122,22 +215,29 @@ void expand_list_objects_filter_spec(\n \telse if (filter->choice == LOFC_TREE_DEPTH)\n \t\tstrbuf_addf(expanded_spec, \"tree:%lu\",\n \t\t\t filter->tree_exclude_depth);\n \telse\n \t\tstrbuf_addstr(expanded_spec, filter->filter_spec);\n }\n \n void list_objects_filter_release(\n \tstruct list_objects_filter_options *filter_options)\n {\n+\tsize_t sub;\n+\n+\tif (!filter_options)\n+\t\treturn;\n \tfree(filter_options->filter_spec);\n \tfree(filter_options->sparse_oid_value);\n+\tfor (sub = 0; sub < filter_options->sub_nr; sub++)\n+\t\tlist_objects_filter_release(&filter_options->sub[sub]);\n+\tfree(filter_options->sub);\n \tmemset(filter_options, 0, sizeof(*filter_options));\n }\n \n void partial_clone_register(\n \tconst char *remote,\n \tconst struct list_objects_filter_options *filter_options)\n {\n \t/*\n \t * Record the name of the partial clone remote in the\n \t * config and in the global variable -- the latter is\n@@ -167,15 +267,17 @@ void partial_clone_register(\n void partial_clone_get_default_filter_spec(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tstruct strbuf errbuf = STRBUF_INIT;\n \n \t/*\n \t * Parse default value, but silently ignore it if it is invalid.\n \t */\n \tif (!core_partial_clone_filter_default)\n \t\treturn;\n+\n+\tfilter_options->filter_spec = strdup(core_partial_clone_filter_default);\n \tgently_parse_list_objects_filter(filter_options,\n \t\t\t\t\t core_partial_clone_filter_default,\n \t\t\t\t\t &errbuf);\n \tstrbuf_release(&errbuf);\n }\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex c54f0000fb..789faef1e5 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -6,20 +6,21 @@\n \n /*\n * The list of defined filters for list-objects.\n */\n enum list_objects_filter_choice {\n \tLOFC_DISABLED = 0,\n \tLOFC_BLOB_NONE,\n \tLOFC_BLOB_LIMIT,\n \tLOFC_TREE_DEPTH,\n \tLOFC_SPARSE_OID,\n+\tLOFC_COMBINE,\n \tLOFC__COUNT /* must be last */\n };\n \n struct list_objects_filter_options {\n \t/*\n \t * 'filter_spec' is the raw argument value given on the command line\n \t * or protocol request. (The part after the \"--keyword=\".) For\n \t * commands that launch filtering sub-processes, or for communication\n \t * over the network, don't use this value; use the result of\n \t * expand_list_objects_filter_spec() instead.\n@@ -31,27 +32,37 @@ struct list_objects_filter_options {\n \t * the filtering algorithm to use.\n \t */\n \tenum list_objects_filter_choice choice;\n \n \t/*\n \t * Choice is LOFC_DISABLED because \"--no-filter\" was requested.\n \t */\n \tunsigned int no_filter : 1;\n \n \t/*\n-\t * Parsed values (fields) from within the filter-spec. These are\n-\t * choice-specific; not all values will be defined for any given\n-\t * choice.\n+\t * BEGIN choice-specific parsed values from within the filter-spec. Only\n+\t * some values will be defined for any given choice.\n \t */\n+\n \tstruct object_id *sparse_oid_value;\n \tunsigned long blob_limit_value;\n \tunsigned long tree_exclude_depth;\n+\n+\t/* LOFC_COMBINE values */\n+\n+\t/* This array contains all the subfilters which this filter combines. */\n+\tsize_t sub_nr, sub_alloc;\n+\tstruct list_objects_filter_options *sub;\n+\n+\t/*\n+\t * END choice-specific parsed values.\n+\t */\n };\n \n /* Normalized command line arguments */\n #define CL_ARG__FILTER \"filter\"\n \n int parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg);\n \n int opt_parse_list_objects_filter(const struct option *opt,\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex b259039bd0..8d015bf164 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -19,30 +19,45 @@\n * FILTER_SHOWN_BUT_REVISIT -- we set this bit on tree objects\n * that have been shown, but should be revisited if they appear\n * in the traversal (until we mark it SEEN). This is a way to\n * let us silently de-dup calls to show() in the caller. This\n * is subtly different from the \"revision.h:SHOWN\" and the\n * \"sha1-name.c:ONELINE_SEEN\" bits. And also different from\n * the non-de-dup usage in pack-bitmap.c\n */\n #define FILTER_SHOWN_BUT_REVISIT (1<<21)\n \n+struct subfilter {\n+\tstruct filter *filter;\n+\tstruct oidset seen;\n+\tstruct oidset omits;\n+\tstruct object_id skip_tree;\n+\tunsigned is_skipping_tree : 1;\n+};\n+\n struct filter {\n \tenum list_objects_filter_result (*filter_object_fn)(\n \t\tstruct repository *r,\n \t\tenum list_objects_filter_situation filter_situation,\n \t\tstruct object *obj,\n \t\tconst char *pathname,\n \t\tconst char *filename,\n \t\tstruct oidset *omits,\n \t\tvoid *filter_data);\n \n+\t/*\n+\t * Optional. If this function is supplied and the filter needs to\n+\t * collect omits, then this function is called once before free_fn is\n+\t * called.\n+\t */\n+\tvoid (*finalize_omits_fn)(struct oidset *omits, void *filter_data);\n+\n \tvoid (*free_fn)(void *filter_data);\n \n \tvoid *filter_data;\n \n \t/* If non-NULL, the filter collects a list of the omitted OIDs here. */\n \tstruct oidset *omits;\n };\n \n static enum list_objects_filter_result filter_blobs_none(\n \tstruct repository *r,\n@@ -464,33 +479,175 @@ static void filter_sparse_oid__init(\n \n \tALLOC_GROW(d->array_frame, d->nr + 1, d->alloc);\n \td->array_frame[d->nr].defval = 0; /* default to include */\n \td->array_frame[d->nr].child_prov_omit = 0;\n \n \tfilter->filter_data = d;\n \tfilter->filter_object_fn = filter_sparse;\n \tfilter->free_fn = filter_sparse_free;\n }\n \n+/* A filter which only shows objects shown by all sub-filters. */\n+struct combine_filter_data {\n+\tstruct subfilter *sub;\n+\tsize_t nr;\n+};\n+\n+static int should_delegate(enum list_objects_filter_situation filter_situation,\n+\t\t\t struct object *obj,\n+\t\t\t struct subfilter *sub)\n+{\n+\tif (!sub->is_skipping_tree)\n+\t\treturn 1;\n+\tif (filter_situation == LOFS_END_TREE &&\n+\t\toideq(&obj->oid, &sub->skip_tree)) {\n+\t\tsub->is_skipping_tree = 0;\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+static enum list_objects_filter_result process_subfilter(\n+\tstruct repository *r,\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tstruct subfilter *sub)\n+{\n+\tenum list_objects_filter_result result;\n+\n+\t/*\n+\t * Check should_delegate before oidset_contains so that\n+\t * is_skipping_tree gets unset even when the object is marked as seen.\n+\t * As of this writing, no filter uses LOFR_MARK_SEEN on trees that also\n+\t * uses LOFR_SKIP_TREE, so the ordering is only theoretically\n+\t * important. Be cautious if you change the order of the below checks\n+\t * and more filters have been added!\n+\t */\n+\tif (!should_delegate(filter_situation, obj, sub))\n+\t\treturn LOFR_ZERO;\n+\tif (oidset_contains(&sub->seen, &obj->oid))\n+\t\treturn LOFR_ZERO;\n+\n+\tresult = list_objects_filter__filter_object(\n+\t\tr, filter_situation, obj, pathname, filename, sub->filter);\n+\n+\tif (result & LOFR_MARK_SEEN)\n+\t\toidset_insert(&sub->seen, &obj->oid);\n+\n+\tif (result & LOFR_SKIP_TREE) {\n+\t\tsub->is_skipping_tree = 1;\n+\t\tsub->skip_tree = obj->oid;\n+\t}\n+\n+\treturn result;\n+}\n+\n+static enum list_objects_filter_result filter_combine(\n+\tstruct repository *r,\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tstruct oidset *omits,\n+\tvoid *filter_data)\n+{\n+\tstruct combine_filter_data *d = filter_data;\n+\tenum list_objects_filter_result combined_result =\n+\t\tLOFR_DO_SHOW | LOFR_MARK_SEEN | LOFR_SKIP_TREE;\n+\tsize_t sub;\n+\n+\tfor (sub = 0; sub < d->nr; sub++) {\n+\t\tenum list_objects_filter_result sub_result = process_subfilter(\n+\t\t\tr, filter_situation, obj, pathname, filename,\n+\t\t\t&d->sub[sub]);\n+\t\tif (!(sub_result & LOFR_DO_SHOW))\n+\t\t\tcombined_result &= ~LOFR_DO_SHOW;\n+\t\tif (!(sub_result & LOFR_MARK_SEEN))\n+\t\t\tcombined_result &= ~LOFR_MARK_SEEN;\n+\t\tif (!d->sub[sub].is_skipping_tree)\n+\t\t\tcombined_result &= ~LOFR_SKIP_TREE;\n+\t}\n+\n+\treturn combined_result;\n+}\n+\n+static void filter_combine__free(void *filter_data)\n+{\n+\tstruct combine_filter_data *d = filter_data;\n+\tsize_t sub;\n+\tfor (sub = 0; sub < d->nr; sub++) {\n+\t\tlist_objects_filter__free(d->sub[sub].filter);\n+\t\toidset_clear(&d->sub[sub].seen);\n+\t\tif (d->sub[sub].omits.set.size)\n+\t\t\tBUG(\"expected oidset to be cleared already\");\n+\t}\n+\tfree(d->sub);\n+}\n+\n+static void add_all(struct oidset *dest, struct oidset *src) {\n+\tstruct oidset_iter iter;\n+\tstruct object_id *src_oid;\n+\n+\toidset_iter_init(src, &iter);\n+\twhile ((src_oid = oidset_iter_next(&iter)) != NULL)\n+\t\toidset_insert(dest, src_oid);\n+}\n+\n+static void filter_combine__finalize_omits(\n+\tstruct oidset *omits,\n+\tvoid *filter_data)\n+{\n+\tstruct combine_filter_data *d = filter_data;\n+\tsize_t sub;\n+\n+\tfor (sub = 0; sub < d->nr; sub++) {\n+\t\tadd_all(omits, &d->sub[sub].omits);\n+\t\toidset_clear(&d->sub[sub].omits);\n+\t}\n+}\n+\n+static void filter_combine__init(\n+\tstruct list_objects_filter_options *filter_options,\n+\tstruct filter* filter)\n+{\n+\tstruct combine_filter_data *d = xcalloc(1, sizeof(*d));\n+\tsize_t sub;\n+\n+\td->nr = filter_options->sub_nr;\n+\td->sub = xcalloc(d->nr, sizeof(*d->sub));\n+\tfor (sub = 0; sub < d->nr; sub++)\n+\t\td->sub[sub].filter = list_objects_filter__init(\n+\t\t\tfilter->omits ? &d->sub[sub].omits : NULL,\n+\t\t\t&filter_options->sub[sub]);\n+\n+\tfilter->filter_data = d;\n+\tfilter->filter_object_fn = filter_combine;\n+\tfilter->free_fn = filter_combine__free;\n+\tfilter->finalize_omits_fn = filter_combine__finalize_omits;\n+}\n+\n typedef void (*filter_init_fn)(\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter);\n \n /*\n * Must match \"enum list_objects_filter_choice\".\n */\n static filter_init_fn s_filters[] = {\n \tNULL,\n \tfilter_blobs_none__init,\n \tfilter_blobs_limit__init,\n \tfilter_trees_depth__init,\n \tfilter_sparse_oid__init,\n+\tfilter_combine__init,\n };\n \n struct filter *list_objects_filter__init(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options)\n {\n \tstruct filter *filter;\n \tfilter_init_fn init_fn;\n \n \tassert((sizeof(s_filters) / sizeof(s_filters[0])) == LOFC__COUNT);\n@@ -528,13 +685,15 @@ enum list_objects_filter_result list_objects_filter__filter_object(\n \t */\n \tif (filter_situation == LOFS_END_TREE)\n \t\treturn 0;\n \treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n }\n \n void list_objects_filter__free(struct filter *filter)\n {\n \tif (!filter)\n \t\treturn;\n+\tif (filter->finalize_omits_fn && filter->omits)\n+\t\tfilter->finalize_omits_fn(filter->omits, filter->filter_data);\n \tfilter->free_fn(filter->filter_data);\n \tfree(filter);\n }\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex acd7f5ab80..05d4f2e9c2 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -271,21 +271,33 @@ test_expect_success 'verify tree:0 includes trees in \"filtered\" output' '\n # Make sure tree:0 does not iterate through any trees.\n \n test_expect_success 'verify skipping tree iteration when not collecting omits' '\n \tGIT_TRACE=1 git -C r3 rev-list \\\n \t\t--objects --filter=tree:0 HEAD 2>filter_trace &&\n \tgrep \"Skipping contents of tree [.][.][.]\" filter_trace >actual &&\n \t# One line for each commit traversed.\n \ttest_line_count = 2 actual &&\n \n \t# Make sure no other trees were considered besides the root.\n-\t! grep \"Skipping contents of tree [^.]\" filter_trace\n+\t! grep \"Skipping contents of tree [^.]\" filter_trace &&\n+\n+\t# Try this again with \"combine:\". If both sub-filters are skipping\n+\t# trees, the composite filter should also skip trees. This is not\n+\t# important unless the user does combine:tree:X+tree:Y or another filter\n+\t# besides \"tree:\" is implemented in the future which can skip trees.\n+\tGIT_TRACE=1 git -C r3 rev-list \\\n+\t\t--objects --filter=combine:tree:1+tree:3 HEAD 2>filter_trace &&\n+\n+\t# Only skip the dir1/ tree, which is shared between the two commits.\n+\tgrep \"Skipping contents of tree \" filter_trace >actual &&\n+\ttest_write_lines \"Skipping contents of tree dir1/...\" >expected &&\n+\ttest_cmp expected actual\n '\n \n # Test tree:# filters.\n \n expect_has () {\n \tcommit=$1 &&\n \tname=$2 &&\n \n \thash=$(git -C r3 rev-parse $commit:$name) &&\n \tgrep \"^$hash $name$\" actual\n@@ -323,20 +335,126 @@ test_expect_success 'verify tree:3 includes everything expected' '\n \texpect_has HEAD dir1/sparse1 &&\n \texpect_has HEAD dir1/sparse2 &&\n \texpect_has HEAD pattern &&\n \texpect_has HEAD sparse1 &&\n \texpect_has HEAD sparse2 &&\n \n \t# There are also 2 commit objects\n \ttest_line_count = 10 actual\n '\n \n+test_expect_success 'combine:... for a simple combination' '\n+\tgit -C r3 rev-list --objects --filter=combine:tree:2+blob:none HEAD \\\n+\t\t>actual &&\n+\n+\texpect_has HEAD \"\" &&\n+\texpect_has HEAD~1 \"\" &&\n+\texpect_has HEAD dir1 &&\n+\n+\t# There are also 2 commit objects\n+\ttest_line_count = 5 actual\n+'\n+\n+test_expect_success 'combine:... with URL encoding' '\n+\tgit -C r3 rev-list --objects \\\n+\t\t--filter=combine:tree%3a2+blob:%6Eon%65 HEAD >actual &&\n+\n+\texpect_has HEAD \"\" &&\n+\texpect_has HEAD~1 \"\" &&\n+\texpect_has HEAD dir1 &&\n+\n+\t# There are also 2 commit objects\n+\ttest_line_count = 5 actual\n+'\n+\n+expect_invalid_filter_spec () {\n+\tspec=\"$1\" &&\n+\terr=\"$2\" &&\n+\n+\ttest_must_fail git -C r3 rev-list --objects --filter=\"$spec\" HEAD \\\n+\t\t>actual 2>actual_stderr &&\n+\ttest_must_be_empty actual &&\n+\ttest_i18ngrep \"$err\" actual_stderr\n+}\n+\n+test_expect_success 'combine:... while URL-encoding things that should not be' '\n+\texpect_invalid_filter_spec combine%3Atree:2+blob:none \\\n+\t\t\"invalid filter-spec\"\n+'\n+\n+test_expect_success 'combine: with nothing after the :' '\n+\texpect_invalid_filter_spec combine: \"expected something after combine:\"\n+'\n+\n+test_expect_success 'parse error in first sub-filter in combine:' '\n+\texpect_invalid_filter_spec combine:tree:asdf+blob:none \\\n+\t\t\"expected .tree:.\"\n+'\n+\n+test_expect_success 'combine:... with non-encoded reserved chars' '\n+\texpect_invalid_filter_spec combine:tree:2+sparse:@xyz \\\n+\t\t\"must escape char in sub-filter-spec: .@.\" &&\n+\texpect_invalid_filter_spec combine:tree:2+sparse:\\` \\\n+\t\t\"must escape char in sub-filter-spec: .\\`.\" &&\n+\texpect_invalid_filter_spec combine:tree:2+sparse:~abc \\\n+\t\t\"must escape char in sub-filter-spec: .\\~.\"\n+'\n+\n+test_expect_success 'validate err msg for \"combine:+\"' '\n+\texpect_invalid_filter_spec combine:tree:2+ \"expected .tree:.\"\n+'\n+\n+test_expect_success 'combine:... with edge-case hex digits: Ff Aa 0 9' '\n+\tgit -C r3 rev-list --objects --filter=\"combine:tree:2+bl%6Fb:n%6fne\" \\\n+\t\tHEAD >actual &&\n+\ttest_line_count = 5 actual &&\n+\tgit -C r3 rev-list --objects --filter=\"combine:tree%3A2+blob%3anone\" \\\n+\t\tHEAD >actual &&\n+\ttest_line_count = 5 actual &&\n+\tgit -C r3 rev-list --objects --filter=\"combine:tree:%30\" HEAD >actual &&\n+\ttest_line_count = 2 actual &&\n+\tgit -C r3 rev-list --objects --filter=\"combine:tree:%39+blob:none\" \\\n+\t\tHEAD >actual &&\n+\ttest_line_count = 5 actual\n+'\n+\n+test_expect_success 'add a sparse pattern blob whose path has reserved chars' '\n+\tcp r3/pattern r3/pattern1+renamed% &&\n+\tgit -C r3 add pattern1+renamed% &&\n+\tgit -C r3 commit -m \"add sparse pattern file with reserved chars\"\n+'\n+\n+test_expect_success 'combine:... with more than two sub-filters' '\n+\tgit -C r3 rev-list --objects \\\n+\t\t--filter=combine:tree:3+blob:limit=40+sparse:oid=master:pattern \\\n+\t\tHEAD >actual &&\n+\n+\texpect_has HEAD \"\" &&\n+\texpect_has HEAD~1 \"\" &&\n+\texpect_has HEAD~2 \"\" &&\n+\texpect_has HEAD dir1 &&\n+\texpect_has HEAD dir1/sparse1 &&\n+\texpect_has HEAD dir1/sparse2 &&\n+\n+\t# Should also have 3 commits\n+\ttest_line_count = 9 actual &&\n+\n+\t# Try again, this time making sure the last sub-filter is only\n+\t# URL-decoded once.\n+\tcp actual expect &&\n+\n+\tgit -C r3 rev-list --objects \\\n+\t\t--filter=combine:tree:3+blob:limit=40+sparse:oid=master:pattern1%2brenamed%25 \\\n+\t\tHEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n # Test provisional omit collection logic with a repo that has objects appearing\n # at multiple depths - first deeper than the filter's threshold, then shallow.\n \n test_expect_success 'setup r4' '\n \tgit init r4 &&\n \n \techo foo > r4/foo &&\n \tmkdir r4/subdir &&\n \techo bar > r4/subdir/bar &&\n \n@@ -366,20 +484,51 @@ test_expect_success 'test tree:# filter provisional omit for blob and tree' '\n \n test_expect_success 'verify skipping tree iteration when collecting omits' '\n \tGIT_TRACE=1 git -C r4 rev-list --filter-print-omitted \\\n \t\t--objects --filter=tree:0 HEAD 2>filter_trace &&\n \tgrep \"^Skipping contents of tree \" filter_trace >actual &&\n \n \techo \"Skipping contents of tree subdir/...\" >expect &&\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'setup r5' '\n+\tgit init r5 &&\n+\tmkdir -p r5/subdir &&\n+\n+\techo 1 >r5/short-root &&\n+\techo 12345 >r5/long-root &&\n+\techo a >r5/subdir/short-subdir &&\n+\techo abcde >r5/subdir/long-subdir &&\n+\n+\tgit -C r5 add short-root long-root subdir &&\n+\tgit -C r5 commit -m \"commit msg\"\n+'\n+\n+test_expect_success 'verify collecting omits in combined: filter' '\n+\t# Note that this test guards against the naive implementation of simply\n+\t# giving both filters the same \"omits\" set and expecting it to\n+\t# automatically merge them.\n+\tgit -C r5 rev-list --objects --quiet --filter-print-omitted \\\n+\t\t--filter=combine:tree:2+blob:limit=3 HEAD >actual &&\n+\n+\t# Expect 0 trees/commits, 3 blobs omitted (all blobs except short-root)\n+\tomitted_1=$(echo 12345 | git hash-object --stdin) &&\n+\tomitted_2=$(echo a | git hash-object --stdin) &&\n+\tomitted_3=$(echo abcde | git hash-object --stdin) &&\n+\n+\tgrep ~$omitted_1 actual &&\n+\tgrep ~$omitted_2 actual &&\n+\tgrep ~$omitted_3 actual &&\n+\ttest_line_count = 3 actual\n+'\n+\n # Test tree: where a tree is iterated to twice - once where a subentry is\n # too deep to be included, and again where the blob inside it is shallow enough\n # to be included. This makes sure we don't use LOFR_MARK_SEEN incorrectly (we\n # can't use it because a tree can be iterated over again at a lower depth).\n \n test_expect_success 'tree: where we iterate over tree at two levels' '\n \tgit init r5 &&\n \n \tmkdir -p r5/a/subdir/b &&\n \techo foo > r5/a/subdir/b/foo &&\ndiff --git a/url.c b/url.c\nindex 25576c390b..bdede647bc 100644\n--- a/url.c\n+++ b/url.c\n@@ -79,20 +79,26 @@ char *url_decode_mem(const char *url, int len)\n \n \t/* Skip protocol part if present */\n \tif (colon && url < colon) {\n \t\tstrbuf_add(&out, url, colon - url);\n \t\tlen -= colon - url;\n \t\turl = colon;\n \t}\n \treturn url_decode_internal(&url, len, NULL, &out, 0);\n }\n \n+char *url_percent_decode(const char *encoded)\n+{\n+\tstruct strbuf out = STRBUF_INIT;\n+\treturn url_decode_internal(&encoded, strlen(encoded), NULL, &out, 0);\n+}\n+\n char *url_decode_parameter_name(const char **query)\n {\n \tstruct strbuf out = STRBUF_INIT;\n \treturn url_decode_internal(query, -1, \"&=\", &out, 1);\n }\n \n char *url_decode_parameter_value(const char **query)\n {\n \tstruct strbuf out = STRBUF_INIT;\n \treturn url_decode_internal(query, -1, \"&\", &out, 1);\ndiff --git a/url.h b/url.h\nindex 00b7d58c33..2a27c34277 100644\n--- a/url.h\n+++ b/url.h\n@@ -1,16 +1,24 @@\n #ifndef URL_H\n #define URL_H\n \n struct strbuf;\n \n int is_url(const char *url);\n int is_urlschemechar(int first_flag, int ch);\n char *url_decode(const char *url);\n char *url_decode_mem(const char *url, int len);\n+\n+/*\n+ * Similar to the url_decode_{,mem} methods above, but doesn't assume there\n+ * is a scheme followed by a : at the start of the string. Instead, %-sequences\n+ * before any : are also parsed.\n+ */\n+char *url_percent_decode(const char *encoded);\n+\n char *url_decode_parameter_name(const char **query);\n char *url_decode_parameter_value(const char **query);\n \n void end_url_with_slash(struct strbuf *buf, const char *url);\n void str_end_url_with_slash(const char *url, char **dest);\n \n #endif /* URL_H */\n-- \n2.21.0\n\n"},{"id":"377283","messageId":"2ca58159914d0712916915b92027fd56b0b93fef.1560558910.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1560558910.git.matvore@google.com","subject":"[PATCH v4 05/10] list-objects-filter-options: move error check up","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-15T00:40:11Z","receivedAt":"2019-06-15T00:42:17Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Move the check that filter_options->choice is set to higher in the call\nstack. This can only be set when the gentle parse function is called\nfrom one of the two call sites.\n\nThis is important because in an upcoming patch this may or may not be an\nerror, and whether it is an error is only known to the\nparse_list_objects_filter function.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects-filter-options.c | 9 ++++-----\n 1 file changed, 4 insertions(+), 5 deletions(-)\n\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex 1c402c6059..ab2c983031 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -28,25 +28,22 @@ static int parse_combine_filter(\n * expand_list_objects_filter_spec() first). We also \"intern\" the arg for the\n * convenience of the current command.\n */\n static int gently_parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg,\n \tstruct strbuf *errbuf)\n {\n \tconst char *v0;\n \n-\tif (filter_options->choice) {\n-\t\tstrbuf_addstr(\n-\t\t\terrbuf, _(\"multiple filter-specs cannot be combined\"));\n-\t\treturn 1;\n-\t}\n+\tif (filter_options->choice)\n+\t\tBUG(\"filter_options already populated\");\n \n \tif (!strcmp(arg, \"blob:none\")) {\n \t\tfilter_options->choice = LOFC_BLOB_NONE;\n \t\treturn 0;\n \n \t} else if (skip_prefix(arg, \"blob:limit=\", &v0)) {\n \t\tif (git_parse_ulong(v0, &filter_options->blob_limit_value)) {\n \t\t\tfilter_options->choice = LOFC_BLOB_LIMIT;\n \t\t\treturn 0;\n \t\t}\n@@ -178,20 +175,22 @@ static int parse_combine_filter(\n \t\tlist_objects_filter_release(filter_options);\n \t\tmemset(filter_options, 0, sizeof(*filter_options));\n \t}\n \treturn result;\n }\n \n int parse_list_objects_filter(struct list_objects_filter_options *filter_options,\n \t\t\t const char *arg)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n+\tif (filter_options->choice)\n+\t\tdie(_(\"multiple filter-specs cannot be combined\"));\n \tfilter_options->filter_spec = strdup(arg);\n \tif (gently_parse_list_objects_filter(filter_options, arg, &buf))\n \t\tdie(\"%s\", buf.buf);\n \treturn 0;\n }\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct list_objects_filter_options *filter_options = opt->value;\n-- \n2.21.0\n\n"},{"id":"377284","messageId":"1a95dd91927973038c3d59bc3215556e448f0e63.1560558910.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1560558910.git.matvore@google.com","subject":"[PATCH v4 06/10] list-objects-filter-options: make filter_spec a string_list","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-15T00:40:12Z","receivedAt":"2019-06-15T00:42:20Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Make the filter_spec string a string_list rather than a raw C string.\nThe list of strings must be concatted together to make a complete\nfilter_spec. A future patch will use this capability to build \"combine:\"\nfilter specs gradually.\n\nA strbuf would seem to be a more natural choice for this object, but it\nunfortunately requires initialization besides just zero'ing out the\nmemory. This results in all container structs, and all containers of\nthose structs, etc., to also require initialization. Initializing them\nall would be more cumbersome that simply using a string_list, which\nbehaves properly when its contents are zero'd.\n\nFor the purposes of code simplification, change behavior in how filter\nspecs are conveyed over the protocol: do not normalize the tree:\nfilter specs since there should be no server in existence that supports\ntree:# but not tree:#k etc.\n\nHelped-by: Junio C Hamano \nSigned-off-by: Matthew DeVore \n---\n builtin/clone.c | 8 ++---\n builtin/fetch.c | 9 ++----\n builtin/rev-list.c | 6 ++--\n fetch-pack.c | 20 ++++--------\n list-objects-filter-options.c | 50 ++++++++++++++++++++---------\n list-objects-filter-options.h | 27 +++++++++++-----\n t/t6112-rev-list-filters-objects.sh | 7 ----\n transport-helper.c | 10 ++----\n upload-pack.c | 11 +++----\n 9 files changed, 78 insertions(+), 70 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex e3231864ca..921df72d84 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1134,27 +1134,25 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\ttransport_set_option(transport, TRANS_OPT_FOLLOWTAGS, \"1\");\n \n \tif (option_upload_pack)\n \t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK,\n \t\t\t\t option_upload_pack);\n \n \tif (server_options.nr)\n \t\ttransport->server_options = &server_options;\n \n \tif (filter_options.choice) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n-\t\texpand_list_objects_filter_spec(&filter_options,\n-\t\t\t\t\t\t&expanded_filter_spec);\n+\t\tconst char *spec =\n+\t\t\texpand_list_objects_filter_spec(&filter_options);\n \t\ttransport_set_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER,\n-\t\t\t\t expanded_filter_spec.buf);\n+\t\t\t\t spec);\n \t\ttransport_set_option(transport, TRANS_OPT_FROM_PROMISOR, \"1\");\n-\t\tstrbuf_release(&expanded_filter_spec);\n \t}\n \n \tif (transport->smart_options && !deepen && !filter_options.choice)\n \t\ttransport->smart_options->check_self_contained_and_connected = 1;\n \n \n \targv_array_push(&ref_prefixes, \"HEAD\");\n \trefspec_ref_prefixes(&remote->fetch, &ref_prefixes);\n \tif (option_branch)\n \t\texpand_ref_prefix(&ref_prefixes, option_branch);\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 4ba63d5ac6..dee89e1a19 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1181,27 +1181,24 @@ static struct transport *prepare_transport(struct remote *remote, int deepen)\n \tif (deepen && deepen_since)\n \t\tset_option(transport, TRANS_OPT_DEEPEN_SINCE, deepen_since);\n \tif (deepen && deepen_not.nr)\n \t\tset_option(transport, TRANS_OPT_DEEPEN_NOT,\n \t\t\t (const char *)&deepen_not);\n \tif (deepen_relative)\n \t\tset_option(transport, TRANS_OPT_DEEPEN_RELATIVE, \"yes\");\n \tif (update_shallow)\n \t\tset_option(transport, TRANS_OPT_UPDATE_SHALLOW, \"yes\");\n \tif (filter_options.choice) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n-\t\texpand_list_objects_filter_spec(&filter_options,\n-\t\t\t\t\t\t&expanded_filter_spec);\n-\t\tset_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER,\n-\t\t\t expanded_filter_spec.buf);\n+\t\tconst char *spec =\n+\t\t\texpand_list_objects_filter_spec(&filter_options);\n+\t\tset_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER, spec);\n \t\tset_option(transport, TRANS_OPT_FROM_PROMISOR, \"1\");\n-\t\tstrbuf_release(&expanded_filter_spec);\n \t}\n \tif (negotiation_tip.nr) {\n \t\tif (transport->smart_options)\n \t\t\tadd_negotiation_tips(transport->smart_options);\n \t\telse\n \t\t\twarning(\"Ignoring --negotiation-tip because the protocol does not support it.\");\n \t}\n \treturn transport;\n }\n \ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 660172b014..68acbe8fd2 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -459,22 +459,24 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\tshow_progress = arg;\n \t\t\tcontinue;\n \t\t}\n \n \t\tif (skip_prefix(arg, (\"--\" CL_ARG__FILTER \"=\"), &arg)) {\n \t\t\tparse_list_objects_filter(&filter_options, arg);\n \t\t\tif (filter_options.choice && !revs.blob_objects)\n \t\t\t\tdie(_(\"object filtering requires --objects\"));\n \t\t\tif (filter_options.choice == LOFC_SPARSE_OID &&\n \t\t\t !filter_options.sparse_oid_value)\n-\t\t\t\tdie(_(\"invalid sparse value '%s'\"),\n-\t\t\t\t filter_options.filter_spec);\n+\t\t\t\tdie(\n+\t\t\t\t\t_(\"invalid sparse value '%s'\"),\n+\t\t\t\t\tlist_objects_filter_spec(\n+\t\t\t\t\t\t&filter_options));\n \t\t\tcontinue;\n \t\t}\n \t\tif (!strcmp(arg, (\"--no-\" CL_ARG__FILTER))) {\n \t\t\tlist_objects_filter_set_no_filter(&filter_options);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!strcmp(arg, \"--filter-print-omitted\")) {\n \t\t\targ_print_omitted = 1;\n \t\t\tcontinue;\n \t\t}\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 1c10f54e78..72e13b0a1d 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -332,26 +332,23 @@ static int find_common(struct fetch_negotiator *negotiator,\n \t\tpacket_buf_write(&req_buf, \"deepen-since %\"PRItime, max_age);\n \t}\n \tif (args->deepen_not) {\n \t\tint i;\n \t\tfor (i = 0; i < args->deepen_not->nr; i++) {\n \t\t\tstruct string_list_item *s = args->deepen_not->items + i;\n \t\t\tpacket_buf_write(&req_buf, \"deepen-not %s\", s->string);\n \t\t}\n \t}\n \tif (server_supports_filtering && args->filter_options.choice) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n-\t\texpand_list_objects_filter_spec(&args->filter_options,\n-\t\t\t\t\t\t&expanded_filter_spec);\n-\t\tpacket_buf_write(&req_buf, \"filter %s\",\n-\t\t\t\t expanded_filter_spec.buf);\n-\t\tstrbuf_release(&expanded_filter_spec);\n+\t\tconst char *spec =\n+\t\t\texpand_list_objects_filter_spec(&args->filter_options);\n+\t\tpacket_buf_write(&req_buf, \"filter %s\", spec);\n \t}\n \tpacket_buf_flush(&req_buf);\n \tstate_len = req_buf.len;\n \n \tif (args->deepen) {\n \t\tconst char *arg;\n \t\tstruct object_id oid;\n \n \t\tsend_request(args, fd[1], &req_buf);\n \t\twhile (packet_reader_read(&reader) == PACKET_READ_NORMAL) {\n@@ -1092,21 +1089,21 @@ static int add_haves(struct fetch_negotiator *negotiator,\n \t\tret = 1;\n \t}\n \n \t/* Increase haves to send on next round */\n \t*haves_to_send = next_flush(1, *haves_to_send);\n \n \treturn ret;\n }\n \n static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n-\t\t\t const struct fetch_pack_args *args,\n+\t\t\t struct fetch_pack_args *args,\n \t\t\t const struct ref *wants, struct oidset *common,\n \t\t\t int *haves_to_send, int *in_vain,\n \t\t\t int sideband_all)\n {\n \tint ret = 0;\n \tstruct strbuf req_buf = STRBUF_INIT;\n \n \tif (server_supports_v2(\"fetch\", 1))\n \t\tpacket_buf_write(&req_buf, \"command=fetch\");\n \tif (server_supports_v2(\"agent\", 0))\n@@ -1133,27 +1130,24 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n \n \t/* Add shallow-info and deepen request */\n \tif (server_supports_feature(\"fetch\", \"shallow\", 0))\n \t\tadd_shallow_requests(&req_buf, args);\n \telse if (is_repository_shallow(the_repository) || args->deepen)\n \t\tdie(_(\"Server does not support shallow requests\"));\n \n \t/* Add filter */\n \tif (server_supports_feature(\"fetch\", \"filter\", 0) &&\n \t args->filter_options.choice) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n+\t\tconst char *spec =\n+\t\t\texpand_list_objects_filter_spec(&args->filter_options);\n \t\tprint_verbose(args, _(\"Server supports filter\"));\n-\t\texpand_list_objects_filter_spec(&args->filter_options,\n-\t\t\t\t\t\t&expanded_filter_spec);\n-\t\tpacket_buf_write(&req_buf, \"filter %s\",\n-\t\t\t\t expanded_filter_spec.buf);\n-\t\tstrbuf_release(&expanded_filter_spec);\n+\t\tpacket_buf_write(&req_buf, \"filter %s\", spec);\n \t} else if (args->filter_options.choice) {\n \t\twarning(\"filtering not recognized by server, ignoring\");\n \t}\n \n \t/* add wants */\n \tadd_wants(args->no_dependents, wants, &req_buf);\n \n \tif (args->no_dependents) {\n \t\tpacket_buf_write(&req_buf, \"done\");\n \t\tret = 1;\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex ab2c983031..411d23004c 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -177,72 +177,89 @@ static int parse_combine_filter(\n \t}\n \treturn result;\n }\n \n int parse_list_objects_filter(struct list_objects_filter_options *filter_options,\n \t\t\t const char *arg)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tif (filter_options->choice)\n \t\tdie(_(\"multiple filter-specs cannot be combined\"));\n-\tfilter_options->filter_spec = strdup(arg);\n+\tstring_list_append(&filter_options->filter_spec, xstrdup(arg));\n \tif (gently_parse_list_objects_filter(filter_options, arg, &buf))\n \t\tdie(\"%s\", buf.buf);\n \treturn 0;\n }\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct list_objects_filter_options *filter_options = opt->value;\n \n \tif (unset || !arg) {\n \t\tlist_objects_filter_set_no_filter(filter_options);\n \t\treturn 0;\n \t}\n \n \treturn parse_list_objects_filter(filter_options, arg);\n }\n \n-void expand_list_objects_filter_spec(\n-\tconst struct list_objects_filter_options *filter,\n-\tstruct strbuf *expanded_spec)\n+const char *list_objects_filter_spec(struct list_objects_filter_options *filter)\n {\n-\tstrbuf_init(expanded_spec, strlen(filter->filter_spec));\n-\tif (filter->choice == LOFC_BLOB_LIMIT)\n-\t\tstrbuf_addf(expanded_spec, \"blob:limit=%lu\",\n+\tif (!filter->filter_spec.nr)\n+\t\tBUG(\"no filter_spec available for this filter\");\n+\tif (filter->filter_spec.nr != 1) {\n+\t\tstruct strbuf concatted = STRBUF_INIT;\n+\t\tstrbuf_add_separated_string_list(\n+\t\t\t&concatted, \"\", &filter->filter_spec);\n+\t\tstring_list_clear(&filter->filter_spec, /*free_util=*/0);\n+\t\tstring_list_append(\n+\t\t\t&filter->filter_spec, strbuf_detach(&concatted, NULL));\n+\t}\n+\n+\treturn filter->filter_spec.items[0].string;\n+}\n+\n+const char *expand_list_objects_filter_spec(\n+\tstruct list_objects_filter_options *filter)\n+{\n+\tif (filter->choice == LOFC_BLOB_LIMIT) {\n+\t\tstruct strbuf expanded_spec = STRBUF_INIT;\n+\t\tstrbuf_addf(&expanded_spec, \"blob:limit=%lu\",\n \t\t\t filter->blob_limit_value);\n-\telse if (filter->choice == LOFC_TREE_DEPTH)\n-\t\tstrbuf_addf(expanded_spec, \"tree:%lu\",\n-\t\t\t filter->tree_exclude_depth);\n-\telse\n-\t\tstrbuf_addstr(expanded_spec, filter->filter_spec);\n+\t\tstring_list_clear(&filter->filter_spec, /*free_util=*/0);\n+\t\tstring_list_append(\n+\t\t\t&filter->filter_spec,\n+\t\t\tstrbuf_detach(&expanded_spec, NULL));\n+\t}\n+\n+\treturn list_objects_filter_spec(filter);\n }\n \n void list_objects_filter_release(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tsize_t sub;\n \n \tif (!filter_options)\n \t\treturn;\n-\tfree(filter_options->filter_spec);\n+\tstring_list_clear(&filter_options->filter_spec, /*free_util=*/0);\n \tfree(filter_options->sparse_oid_value);\n \tfor (sub = 0; sub < filter_options->sub_nr; sub++)\n \t\tlist_objects_filter_release(&filter_options->sub[sub]);\n \tfree(filter_options->sub);\n \tmemset(filter_options, 0, sizeof(*filter_options));\n }\n \n void partial_clone_register(\n \tconst char *remote,\n-\tconst struct list_objects_filter_options *filter_options)\n+\tstruct list_objects_filter_options *filter_options)\n {\n \t/*\n \t * Record the name of the partial clone remote in the\n \t * config and in the global variable -- the latter is\n \t * used throughout to indicate that partial clone is\n \t * enabled and to expect missing objects.\n \t */\n \tif (repository_format_partial_clone &&\n \t *repository_format_partial_clone &&\n \t strcmp(remote, repository_format_partial_clone))\n@@ -251,32 +268,33 @@ void partial_clone_register(\n \tgit_config_set(\"core.repositoryformatversion\", \"1\");\n \tgit_config_set(\"extensions.partialclone\", remote);\n \n \trepository_format_partial_clone = xstrdup(remote);\n \n \t/*\n \t * Record the initial filter-spec in the config as\n \t * the default for subsequent fetches from this remote.\n \t */\n \tcore_partial_clone_filter_default =\n-\t\txstrdup(filter_options->filter_spec);\n+\t\txstrdup(expand_list_objects_filter_spec(filter_options));\n \tgit_config_set(\"core.partialclonefilter\",\n \t\t core_partial_clone_filter_default);\n }\n \n void partial_clone_get_default_filter_spec(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tstruct strbuf errbuf = STRBUF_INIT;\n \n \t/*\n \t * Parse default value, but silently ignore it if it is invalid.\n \t */\n \tif (!core_partial_clone_filter_default)\n \t\treturn;\n \n-\tfilter_options->filter_spec = strdup(core_partial_clone_filter_default);\n+\tstring_list_append(&filter_options->filter_spec,\n+\t\t\t core_partial_clone_filter_default);\n \tgently_parse_list_objects_filter(filter_options,\n \t\t\t\t\t core_partial_clone_filter_default,\n \t\t\t\t\t &errbuf);\n \tstrbuf_release(&errbuf);\n }\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 789faef1e5..bb33303f9b 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -1,15 +1,15 @@\n #ifndef LIST_OBJECTS_FILTER_OPTIONS_H\n #define LIST_OBJECTS_FILTER_OPTIONS_H\n \n #include \"parse-options.h\"\n-#include \"strbuf.h\"\n+#include \"string-list.h\"\n \n /*\n * The list of defined filters for list-objects.\n */\n enum list_objects_filter_choice {\n \tLOFC_DISABLED = 0,\n \tLOFC_BLOB_NONE,\n \tLOFC_BLOB_LIMIT,\n \tLOFC_TREE_DEPTH,\n \tLOFC_SPARSE_OID,\n@@ -17,22 +17,24 @@ enum list_objects_filter_choice {\n \tLOFC__COUNT /* must be last */\n };\n \n struct list_objects_filter_options {\n \t/*\n \t * 'filter_spec' is the raw argument value given on the command line\n \t * or protocol request. (The part after the \"--keyword=\".) For\n \t * commands that launch filtering sub-processes, or for communication\n \t * over the network, don't use this value; use the result of\n \t * expand_list_objects_filter_spec() instead.\n+\t * To get the raw filter spec given by the user, use the result of\n+\t * list_objects_filter_spec().\n \t */\n-\tchar *filter_spec;\n+\tstruct string_list filter_spec;\n \n \t/*\n \t * 'choice' is determined by parsing the filter-spec. This indicates\n \t * the filtering algorithm to use.\n \t */\n \tenum list_objects_filter_choice choice;\n \n \t/*\n \t * Choice is LOFC_DISABLED because \"--no-filter\" was requested.\n \t */\n@@ -69,35 +71,44 @@ int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset);\n \n #define OPT_PARSE_LIST_OBJECTS_FILTER(fo) \\\n \t{ OPTION_CALLBACK, 0, CL_ARG__FILTER, fo, N_(\"args\"), \\\n \t N_(\"object filtering\"), 0, \\\n \t opt_parse_list_objects_filter }\n \n /*\n * Translates abbreviated numbers in the filter's filter_spec into their\n * fully-expanded forms (e.g., \"limit:blob=1k\" becomes \"limit:blob=1024\").\n+ * Returns a string owned by the list_objects_filter_options object.\n *\n- * This form should be used instead of the raw filter_spec field when\n- * communicating with a remote process or subprocess.\n+ * This form should be used instead of the raw list_objects_filter_spec()\n+ * value when communicating with a remote process or subprocess.\n */\n-void expand_list_objects_filter_spec(\n-\tconst struct list_objects_filter_options *filter,\n-\tstruct strbuf *expanded_spec);\n+const char *expand_list_objects_filter_spec(\n+\tstruct list_objects_filter_options *filter);\n+\n+/*\n+ * Returns the filter spec string more or less in the form as the user\n+ * entered it. This form of the filter_spec can be used in user-facing\n+ * messages. Returns a string owned by the list_objects_filter_options\n+ * object.\n+ */\n+const char *list_objects_filter_spec(\n+\tstruct list_objects_filter_options *filter);\n \n void list_objects_filter_release(\n \tstruct list_objects_filter_options *filter_options);\n \n static inline void list_objects_filter_set_no_filter(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tlist_objects_filter_release(filter_options);\n \tfilter_options->no_filter = 1;\n }\n \n void partial_clone_register(\n \tconst char *remote,\n-\tconst struct list_objects_filter_options *filter_options);\n+\tstruct list_objects_filter_options *filter_options);\n void partial_clone_get_default_filter_spec(\n \tstruct list_objects_filter_options *filter_options);\n \n #endif /* LIST_OBJECTS_FILTER_OPTIONS_H */\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex 05d4f2e9c2..27ba15719a 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -583,18 +583,11 @@ test_expect_success 'rev-list W/ missing=allow-any' '\n # Test expansion of filter specs.\n \n test_expect_success 'expand blob limit in protocol' '\n \tgit -C r2 config --local uploadpack.allowfilter 1 &&\n \tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -c protocol.version=2 clone \\\n \t\t--filter=blob:limit=1k \"file://$(pwd)/r2\" limit &&\n \t! grep \"blob:limit=1k\" trace &&\n \tgrep \"blob:limit=1024\" trace\n '\n \n-test_expect_success 'expand tree depth limit in protocol' '\n-\tGIT_TRACE_PACKET=\"$(pwd)/tree_trace\" git -c protocol.version=2 clone \\\n-\t\t--filter=tree:0k \"file://$(pwd)/r2\" tree &&\n-\t! grep \"tree:0k\" tree_trace &&\n-\tgrep \"tree:0\" tree_trace\n-'\n-\n test_done\ndiff --git a/transport-helper.c b/transport-helper.c\nindex c7e17ec9cb..0a34544df0 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -675,27 +675,23 @@ static int fetch(struct transport *transport,\n \t data->transport_options.check_self_contained_and_connected)\n \t\tset_helper_option(transport, \"check-connectivity\", \"true\");\n \n \tif (transport->cloning)\n \t\tset_helper_option(transport, \"cloning\", \"true\");\n \n \tif (data->transport_options.update_shallow)\n \t\tset_helper_option(transport, \"update-shallow\", \"true\");\n \n \tif (data->transport_options.filter_options.choice) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n-\t\texpand_list_objects_filter_spec(\n-\t\t\t&data->transport_options.filter_options,\n-\t\t\t&expanded_filter_spec);\n-\t\tset_helper_option(transport, \"filter\",\n-\t\t\t\t expanded_filter_spec.buf);\n-\t\tstrbuf_release(&expanded_filter_spec);\n+\t\tconst char *spec = expand_list_objects_filter_spec(\n+\t\t\t&data->transport_options.filter_options);\n+\t\tset_helper_option(transport, \"filter\", spec);\n \t}\n \n \tif (data->transport_options.negotiation_tips)\n \t\twarning(\"Ignoring --negotiation-tip because the protocol does not support it.\");\n \n \tif (data->fetch)\n \t\treturn fetch_with_fetch(transport, nr_heads, to_fetch);\n \n \tif (data->import)\n \t\treturn fetch_with_import(transport, nr_heads, to_fetch);\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 24298913c0..a74d293fef 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -133,32 +133,31 @@ static void create_pack_file(const struct object_array *have_obj,\n \n \targv_array_push(&pack_objects.args, \"--stdout\");\n \tif (shallow_nr)\n \t\targv_array_push(&pack_objects.args, \"--shallow\");\n \tif (!no_progress)\n \t\targv_array_push(&pack_objects.args, \"--progress\");\n \tif (use_ofs_delta)\n \t\targv_array_push(&pack_objects.args, \"--delta-base-offset\");\n \tif (use_include_tag)\n \t\targv_array_push(&pack_objects.args, \"--include-tag\");\n-\tif (filter_options.filter_spec) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n-\t\texpand_list_objects_filter_spec(&filter_options,\n-\t\t\t\t\t\t&expanded_filter_spec);\n+\tif (filter_options.choice) {\n+\t\tconst char *spec =\n+\t\t\texpand_list_objects_filter_spec(&filter_options);\n \t\tif (pack_objects.use_shell) {\n \t\t\tstruct strbuf buf = STRBUF_INIT;\n-\t\t\tsq_quote_buf(&buf, expanded_filter_spec.buf);\n+\t\t\tsq_quote_buf(&buf, spec);\n \t\t\targv_array_pushf(&pack_objects.args, \"--filter=%s\", buf.buf);\n \t\t\tstrbuf_release(&buf);\n \t\t} else {\n \t\t\targv_array_pushf(&pack_objects.args, \"--filter=%s\",\n-\t\t\t\t\t expanded_filter_spec.buf);\n+\t\t\t\t\t spec);\n \t\t}\n \t}\n \n \tpack_objects.in = -1;\n \tpack_objects.out = -1;\n \tpack_objects.err = -1;\n \n \tif (start_command(&pack_objects))\n \t\tdie(\"git upload-pack: unable to fork git-pack-objects\");\n \n-- \n2.21.0\n\n"},{"id":"377285","messageId":"880570027ec7c4405f3342d00087c7ad4efd05f4.1560558910.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1560558910.git.matvore@google.com","subject":"[PATCH v4 07/10] strbuf: give URL-encoding API a char predicate fn","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-15T00:40:13Z","receivedAt":"2019-06-15T00:42:22Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Allow callers to specify exactly what characters need to be URL-encoded\nand which do not. This new API will be taken advantage of in a patch\nlater in this set.\n\nHelped-by: Jeff King \nSigned-off-by: Matthew DeVore \n---\n credential-store.c | 9 +++++----\n http.c | 6 ++++--\n strbuf.c | 15 ++++++++-------\n strbuf.h | 7 ++++++-\n 4 files changed, 23 insertions(+), 14 deletions(-)\n\ndiff --git a/credential-store.c b/credential-store.c\nindex ac295420dd..c010497cb2 100644\n--- a/credential-store.c\n+++ b/credential-store.c\n@@ -65,29 +65,30 @@ static void rewrite_credential_file(const char *fn, struct credential *c,\n \tparse_credential_file(fn, c, NULL, print_line);\n \tif (commit_lock_file(&credential_lock) < 0)\n \t\tdie_errno(\"unable to write credential store\");\n }\n \n static void store_credential_file(const char *fn, struct credential *c)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \n \tstrbuf_addf(&buf, \"%s://\", c->protocol);\n-\tstrbuf_addstr_urlencode(&buf, c->username, 1);\n+\tstrbuf_addstr_urlencode(&buf, c->username, is_rfc3986_unreserved);\n \tstrbuf_addch(&buf, ':');\n-\tstrbuf_addstr_urlencode(&buf, c->password, 1);\n+\tstrbuf_addstr_urlencode(&buf, c->password, is_rfc3986_unreserved);\n \tstrbuf_addch(&buf, '@');\n \tif (c->host)\n-\t\tstrbuf_addstr_urlencode(&buf, c->host, 1);\n+\t\tstrbuf_addstr_urlencode(&buf, c->host, is_rfc3986_unreserved);\n \tif (c->path) {\n \t\tstrbuf_addch(&buf, '/');\n-\t\tstrbuf_addstr_urlencode(&buf, c->path, 0);\n+\t\tstrbuf_addstr_urlencode(&buf, c->path,\n+\t\t\t\t\tis_rfc3986_reserved_or_unreserved);\n \t}\n \n \trewrite_credential_file(fn, c, &buf);\n \tstrbuf_release(&buf);\n }\n \n static void store_credential(const struct string_list *fns, struct credential *c)\n {\n \tstruct string_list_item *fn;\n \ndiff --git a/http.c b/http.c\nindex 27aa0a3192..938b9e55af 100644\n--- a/http.c\n+++ b/http.c\n@@ -506,23 +506,25 @@ static void var_override(const char **var, char *value)\n static void set_proxyauth_name_password(CURL *result)\n {\n #if LIBCURL_VERSION_NUM >= 0x071301\n \t\tcurl_easy_setopt(result, CURLOPT_PROXYUSERNAME,\n \t\t\tproxy_auth.username);\n \t\tcurl_easy_setopt(result, CURLOPT_PROXYPASSWORD,\n \t\t\tproxy_auth.password);\n #else\n \t\tstruct strbuf s = STRBUF_INIT;\n \n-\t\tstrbuf_addstr_urlencode(&s, proxy_auth.username, 1);\n+\t\tstrbuf_addstr_urlencode(&s, proxy_auth.username,\n+\t\t\t\t\tis_rfc3986_unreserved);\n \t\tstrbuf_addch(&s, ':');\n-\t\tstrbuf_addstr_urlencode(&s, proxy_auth.password, 1);\n+\t\tstrbuf_addstr_urlencode(&s, proxy_auth.password,\n+\t\t\t\t\tis_rfc3986_unreserved);\n \t\tcurl_proxyuserpwd = strbuf_detach(&s, NULL);\n \t\tcurl_easy_setopt(result, CURLOPT_PROXYUSERPWD, curl_proxyuserpwd);\n #endif\n }\n \n static void init_curl_proxy_auth(CURL *result)\n {\n \tif (proxy_auth.username) {\n \t\tif (!proxy_auth.password)\n \t\t\tcredential_fill(&proxy_auth);\ndiff --git a/strbuf.c b/strbuf.c\nindex 0e18b259ce..60ab5144f2 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -767,55 +767,56 @@ void strbuf_addstr_xml_quoted(struct strbuf *buf, const char *s)\n \t\tcase '&':\n \t\t\tstrbuf_addstr(buf, \"&\");\n \t\t\tbreak;\n \t\tcase 0:\n \t\t\treturn;\n \t\t}\n \t\ts++;\n \t}\n }\n \n-static int is_rfc3986_reserved(char ch)\n+int is_rfc3986_reserved_or_unreserved(char ch)\n {\n+\tif (is_rfc3986_unreserved(ch))\n+\t\treturn 1;\n \tswitch (ch) {\n \t\tcase '!': case '*': case '\\'': case '(': case ')': case ';':\n \t\tcase ':': case '@': case '&': case '=': case '+': case '$':\n \t\tcase ',': case '/': case '?': case '#': case '[': case ']':\n \t\t\treturn 1;\n \t}\n \treturn 0;\n }\n \n-static int is_rfc3986_unreserved(char ch)\n+int is_rfc3986_unreserved(char ch)\n {\n \treturn isalnum(ch) ||\n \t\tch == '-' || ch == '_' || ch == '.' || ch == '~';\n }\n \n static void strbuf_add_urlencode(struct strbuf *sb, const char *s, size_t len,\n-\t\t\t\t int reserved)\n+\t\t\t\t char_predicate allow_unencoded_fn)\n {\n \tstrbuf_grow(sb, len);\n \twhile (len--) {\n \t\tchar ch = *s++;\n-\t\tif (is_rfc3986_unreserved(ch) ||\n-\t\t (!reserved && is_rfc3986_reserved(ch)))\n+\t\tif (allow_unencoded_fn(ch))\n \t\t\tstrbuf_addch(sb, ch);\n \t\telse\n \t\t\tstrbuf_addf(sb, \"%%%02x\", (unsigned char)ch);\n \t}\n }\n \n void strbuf_addstr_urlencode(struct strbuf *sb, const char *s,\n-\t\t\t int reserved)\n+\t\t\t char_predicate allow_unencoded_fn)\n {\n-\tstrbuf_add_urlencode(sb, s, strlen(s), reserved);\n+\tstrbuf_add_urlencode(sb, s, strlen(s), allow_unencoded_fn);\n }\n \n void strbuf_humanise_bytes(struct strbuf *buf, off_t bytes)\n {\n \tif (bytes > 1 << 30) {\n \t\tstrbuf_addf(buf, \"%u.%2.2u GiB\",\n \t\t\t (unsigned)(bytes >> 30),\n \t\t\t (unsigned)(bytes & ((1 << 30) - 1)) / 10737419);\n \t} else if (bytes > 1 << 20) {\n \t\tunsigned x = bytes + 5243; /* for rounding */\ndiff --git a/strbuf.h b/strbuf.h\nindex c8d98dfb95..346d722492 100644\n--- a/strbuf.h\n+++ b/strbuf.h\n@@ -659,22 +659,27 @@ void strbuf_branchname(struct strbuf *sb, const char *name,\n \t\t unsigned allowed);\n \n /*\n * Like strbuf_branchname() above, but confirm that the result is\n * syntactically valid to be used as a local branch name in refs/heads/.\n *\n * The return value is \"0\" if the result is valid, and \"-1\" otherwise.\n */\n int strbuf_check_branch_ref(struct strbuf *sb, const char *name);\n \n+typedef int (*char_predicate)(char ch);\n+\n+int is_rfc3986_unreserved(char ch);\n+int is_rfc3986_reserved_or_unreserved(char ch);\n+\n void strbuf_addstr_urlencode(struct strbuf *sb, const char *name,\n-\t\t\t int reserved);\n+\t\t\t char_predicate allow_unencoded_fn);\n \n __attribute__((format (printf,1,2)))\n int printf_ln(const char *fmt, ...);\n __attribute__((format (printf,2,3)))\n int fprintf_ln(FILE *fp, const char *fmt, ...);\n \n char *xstrdup_tolower(const char *);\n char *xstrdup_toupper(const char *);\n \n /**\n-- \n2.21.0\n\n"},{"id":"377286","messageId":"89c4171c11a74bde6bb361051b661b2fbed333b9.1560558910.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1560558910.git.matvore@google.com","subject":"[PATCH v4 08/10] list-objects-filter-options: allow mult. --filter","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-15T00:40:14Z","receivedAt":"2019-06-15T00:42:25Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Allow combining of multiple filters by simply repeating the --filter\nflag. Before this patch, the user had to combine them in a single flag\nsomewhat awkwardly (e.g. --filter=combine:FOO+BAR), including\nURL-encoding the individual filters.\n\nTo make this work, in the --filter flag parsing callback, rather than\nerror out when we detect that the filter_options struct is already\npopulated, we modify it in-place to contain the added sub-filter. The\nexisting sub-filter becomes the lhs of the combined filter, and the\nnext sub-filter becomes the rhs. We also have to URL-encode the LHS and\nRHS sub-filters.\n\nWe can simplify the operation if the LHS is already a combine: filter.\nIn that case, we just append the URL-encoded RHS sub-filter to the LHS\nspec to get the new spec.\n\nHelped-by: Emily Shaffer \nHelped-by: Jeff Hostetler \nHelped-by: Jeff King \nHelped-by: Junio C Hamano \nSigned-off-by: Matthew DeVore \n---\n Documentation/rev-list-options.txt | 16 ++++++\n list-objects-filter-options.c | 88 +++++++++++++++++++++++++++--\n list-objects-filter-options.h | 11 ++++\n t/t5616-partial-clone.sh | 19 +++++++\n t/t6112-rev-list-filters-objects.sh | 46 +++++++++++++--\n transport.c | 1 +\n upload-pack.c | 2 +\n 7 files changed, 173 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 71a1fcc093..d1f080bf6d 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -731,20 +731,36 @@ at multiple depths in the commits traversed). =0 will not include\n any trees or blobs unless included explicitly in the command-line (or\n standard input when --stdin is used). =1 will include only the\n tree and blobs which are referenced directly by a commit reachable from\n or an explicitly-given object. =2 is like =1\n while also including trees and blobs one more level removed from an\n explicitly-given commit or tree.\n +\n Note that the form '--filter=sparse:path=' that wants to read\n from an arbitrary path on the filesystem has been dropped for security\n reasons.\n++\n+Multiple '--filter=' flags can be specified to combine filters. Only\n+objects which are accepted by every filter are included.\n++\n+The form '--filter=combine:++...' can also be\n+used to combined several filters, but this is harder than just repeating\n+the '--filter' flag and is usually not necessary. Filters are joined by\n+'{plus}' and individual filters are %-encoded (i.e. URL-encoded).\n+Besides the '{plus}' and '%' characters, the following characters are\n+reserved and also must be encoded: `~!@#$^&*()[]{}\\;\",<>?`+'`+\n+as well as all characters with ASCII code <= `0x20`, which includes\n+space and newline.\n++\n+Other arbitrary characters can also be encoded. For instance,\n+'combine:tree:3+blob:none' and 'combine:tree%3A3+blob%3Anone' are\n+equivalent.\n \n --no-filter::\n \tTurn off any previous `--filter=` argument.\n \n --filter-print-omitted::\n \tOnly useful with `--filter=`; prints a list of the objects omitted\n \tby the filter. Object IDs are prefixed with a ``~'' character.\n \n --missing=::\n \tA debug option to help with future \"partial clone\" development.\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex 411d23004c..ab9c455fbd 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -1,18 +1,19 @@\n #include \"cache.h\"\n #include \"commit.h\"\n #include \"config.h\"\n #include \"revision.h\"\n #include \"argv-array.h\"\n #include \"list-objects.h\"\n #include \"list-objects-filter.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"trace.h\"\n #include \"url.h\"\n \n static int parse_combine_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg,\n \tstruct strbuf *errbuf);\n \n /*\n * Parse value of the argument to the \"filter\" keyword.\n * On the command line this looks like:\n@@ -171,29 +172,106 @@ static int parse_combine_filter(\n \n cleanup:\n \tstrbuf_list_free(subspecs);\n \tif (result) {\n \t\tlist_objects_filter_release(filter_options);\n \t\tmemset(filter_options, 0, sizeof(*filter_options));\n \t}\n \treturn result;\n }\n \n-int parse_list_objects_filter(struct list_objects_filter_options *filter_options,\n-\t\t\t const char *arg)\n+static int allow_unencoded(char ch)\n+{\n+\tif (ch <= ' ' || ch == '%' || ch == '+')\n+\t\treturn 0;\n+\treturn !strchr(RESERVED_NON_WS, ch);\n+}\n+\n+static void filter_spec_append_urlencode(\n+\tstruct list_objects_filter_options *filter, const char *raw)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n+\tstrbuf_addstr_urlencode(&buf, raw, allow_unencoded);\n+\ttrace_printf(\"Add to combine filter-spec: %s\\n\", buf.buf);\n+\tstring_list_append(&filter->filter_spec, strbuf_detach(&buf, NULL));\n+}\n+\n+/*\n+ * Changes filter_options into an equivalent LOFC_COMBINE filter options\n+ * instance. Does not do anything if filter_options is already LOFC_COMBINE.\n+ */\n+static void transform_to_combine_type(\n+\tstruct list_objects_filter_options *filter_options)\n+{\n+\tassert(filter_options->choice);\n+\tif (filter_options->choice == LOFC_COMBINE)\n+\t\treturn;\n+\t{\n+\t\tconst int initial_sub_alloc = 2;\n+\t\tstruct list_objects_filter_options *sub_array =\n+\t\t\txcalloc(initial_sub_alloc, sizeof(*sub_array));\n+\t\tsub_array[0] = *filter_options;\n+\t\tmemset(filter_options, 0, sizeof(*filter_options));\n+\t\tfilter_options->sub = sub_array;\n+\t\tfilter_options->sub_alloc = initial_sub_alloc;\n+\t}\n+\tfilter_options->sub_nr = 1;\n+\tfilter_options->choice = LOFC_COMBINE;\n+\tstring_list_append(&filter_options->filter_spec, xstrdup(\"combine:\"));\n+\tfilter_spec_append_urlencode(\n+\t\tfilter_options,\n+\t\tlist_objects_filter_spec(&filter_options->sub[0]));\n+\t/*\n+\t * We don't need the filter_spec strings for subfilter specs, only the\n+\t * top level.\n+\t */\n+\tstring_list_clear(&filter_options->sub[0].filter_spec, /*free_util=*/0);\n+}\n+\n+void list_objects_filter_die_if_populated(\n+\tstruct list_objects_filter_options *filter_options)\n+{\n \tif (filter_options->choice)\n \t\tdie(_(\"multiple filter-specs cannot be combined\"));\n-\tstring_list_append(&filter_options->filter_spec, xstrdup(arg));\n-\tif (gently_parse_list_objects_filter(filter_options, arg, &buf))\n-\t\tdie(\"%s\", buf.buf);\n+}\n+\n+int parse_list_objects_filter(\n+\tstruct list_objects_filter_options *filter_options,\n+\tconst char *arg)\n+{\n+\tstruct strbuf errbuf = STRBUF_INIT;\n+\tint parse_error;\n+\n+\tif (!filter_options->choice) {\n+\t\tstring_list_append(&filter_options->filter_spec, xstrdup(arg));\n+\n+\t\tparse_error = gently_parse_list_objects_filter(\n+\t\t\tfilter_options, arg, &errbuf);\n+\t} else {\n+\t\t/*\n+\t\t * Make filter_options an LOFC_COMBINE spec so we can trivially\n+\t\t * add subspecs to it.\n+\t\t */\n+\t\ttransform_to_combine_type(filter_options);\n+\n+\t\tstring_list_append(&filter_options->filter_spec, xstrdup(\"+\"));\n+\t\tfilter_spec_append_urlencode(filter_options, arg);\n+\t\tALLOC_GROW(filter_options->sub, filter_options->sub_nr + 1,\n+\t\t\t filter_options->sub_alloc);\n+\t\tfilter_options = &filter_options->sub[filter_options->sub_nr++];\n+\t\tmemset(filter_options, 0, sizeof(*filter_options));\n+\n+\t\tparse_error = gently_parse_list_objects_filter(\n+\t\t\tfilter_options, arg, &errbuf);\n+\t}\n+\tif (parse_error)\n+\t\tdie(\"%s\", errbuf.buf);\n \treturn 0;\n }\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct list_objects_filter_options *filter_options = opt->value;\n \n \tif (unset || !arg) {\n \t\tlist_objects_filter_set_no_filter(filter_options);\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex bb33303f9b..d8bc7e946e 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -56,20 +56,31 @@ struct list_objects_filter_options {\n \tstruct list_objects_filter_options *sub;\n \n \t/*\n \t * END choice-specific parsed values.\n \t */\n };\n \n /* Normalized command line arguments */\n #define CL_ARG__FILTER \"filter\"\n \n+void list_objects_filter_die_if_populated(\n+\tstruct list_objects_filter_options *filter_options);\n+\n+/*\n+ * Parses the filter spec string given by arg and either (1) simply places the\n+ * result in filter_options if it is not yet populated or (2) combines it with\n+ * the filter already in filter_options if it is already populated. In the case\n+ * of (2), the filter specs are combined as if specified with 'combine:'.\n+ *\n+ * Dies and prints a user-facing message if an error occurs.\n+ */\n int parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg);\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset);\n \n #define OPT_PARSE_LIST_OBJECTS_FILTER(fo) \\\n \t{ OPTION_CALLBACK, 0, CL_ARG__FILTER, fo, N_(\"args\"), \\\n \t N_(\"object filtering\"), 0, \\\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex 9a8f9886b3..11536f4028 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -201,20 +201,39 @@ test_expect_success 'use fsck before and after manually fetching a missing subtr\n \ttest_line_count = 70 fetched_objects &&\n \n \tawk -f print_1.awk fetched_objects |\n \txargs -n1 git -C dst cat-file -t >fetched_types &&\n \n \tsort -u fetched_types >unique_types.observed &&\n \ttest_write_lines blob commit tree >unique_types.expected &&\n \ttest_cmp unique_types.expected unique_types.observed\n '\n \n+test_expect_success 'implicitly construct combine: filter with repeated flags' '\n+\tGIT_TRACE=$(pwd)/trace git clone --bare \\\n+\t\t--filter=blob:none --filter=tree:1 \\\n+\t\t\"file://$(pwd)/srv.bare\" pc2 &&\n+\tgrep \"trace:.* git pack-objects .*--filter=combine:blob:none+tree:1\" \\\n+\t\ttrace &&\n+\tgit -C pc2 rev-list --objects --missing=allow-any HEAD >objects &&\n+\n+\t# We should have gotten some root trees.\n+\tgrep \" $\" objects &&\n+\t# Should not have gotten any non-root trees or blobs.\n+\t! grep \" .\" objects &&\n+\n+\txargs -n 1 git -C pc2 cat-file -t types &&\n+\tsort -u types >unique_types.actual &&\n+\ttest_write_lines commit tree >unique_types.expected &&\n+\ttest_cmp unique_types.expected unique_types.actual\n+'\n+\n test_expect_success 'partial clone fetches blobs pointed to by refs even if normally filtered out' '\n \trm -rf src dst &&\n \tgit init src &&\n \ttest_commit -C src x &&\n \ttest_config -C src uploadpack.allowfilter 1 &&\n \ttest_config -C src uploadpack.allowanysha1inwant 1 &&\n \n \t# Create a tag pointing to a blob.\n \tBLOB=$(echo blob-contents | git -C src hash-object --stdin -w) &&\n \tgit -C src tag myblob \"$BLOB\" &&\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex 27ba15719a..de0e5a5d36 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -344,21 +344,30 @@ test_expect_success 'verify tree:3 includes everything expected' '\n \n test_expect_success 'combine:... for a simple combination' '\n \tgit -C r3 rev-list --objects --filter=combine:tree:2+blob:none HEAD \\\n \t\t>actual &&\n \n \texpect_has HEAD \"\" &&\n \texpect_has HEAD~1 \"\" &&\n \texpect_has HEAD dir1 &&\n \n \t# There are also 2 commit objects\n-\ttest_line_count = 5 actual\n+\ttest_line_count = 5 actual &&\n+\n+\tcp actual expected &&\n+\n+\t# Try again using repeated --filter - this is equivalent to a manual\n+\t# combine with \"combine:...+...\"\n+\tgit -C r3 rev-list --objects --filter=combine:tree:2 \\\n+\t\t--filter=blob:none HEAD >actual &&\n+\n+\ttest_cmp expected actual\n '\n \n test_expect_success 'combine:... with URL encoding' '\n \tgit -C r3 rev-list --objects \\\n \t\t--filter=combine:tree%3a2+blob:%6Eon%65 HEAD >actual &&\n \n \texpect_has HEAD \"\" &&\n \texpect_has HEAD~1 \"\" &&\n \texpect_has HEAD dir1 &&\n \n@@ -410,24 +419,26 @@ test_expect_success 'combine:... with edge-case hex digits: Ff Aa 0 9' '\n \tgit -C r3 rev-list --objects --filter=\"combine:tree%3A2+blob%3anone\" \\\n \t\tHEAD >actual &&\n \ttest_line_count = 5 actual &&\n \tgit -C r3 rev-list --objects --filter=\"combine:tree:%30\" HEAD >actual &&\n \ttest_line_count = 2 actual &&\n \tgit -C r3 rev-list --objects --filter=\"combine:tree:%39+blob:none\" \\\n \t\tHEAD >actual &&\n \ttest_line_count = 5 actual\n '\n \n-test_expect_success 'add a sparse pattern blob whose path has reserved chars' '\n+test_expect_success 'add sparse pattern blobs whose paths have reserved chars' '\n \tcp r3/pattern r3/pattern1+renamed% &&\n-\tgit -C r3 add pattern1+renamed% &&\n-\tgit -C r3 commit -m \"add sparse pattern file with reserved chars\"\n+\tcp r3/pattern \"r3/p;at%ter+n\" &&\n+\tcp r3/pattern r3/^~pattern &&\n+\tgit -C r3 add pattern1+renamed% \"p;at%ter+n\" ^~pattern &&\n+\tgit -C r3 commit -m \"add sparse pattern files with reserved chars\"\n '\n \n test_expect_success 'combine:... with more than two sub-filters' '\n \tgit -C r3 rev-list --objects \\\n \t\t--filter=combine:tree:3+blob:limit=40+sparse:oid=master:pattern \\\n \t\tHEAD >actual &&\n \n \texpect_has HEAD \"\" &&\n \texpect_has HEAD~1 \"\" &&\n \texpect_has HEAD~2 \"\" &&\n@@ -438,21 +449,46 @@ test_expect_success 'combine:... with more than two sub-filters' '\n \t# Should also have 3 commits\n \ttest_line_count = 9 actual &&\n \n \t# Try again, this time making sure the last sub-filter is only\n \t# URL-decoded once.\n \tcp actual expect &&\n \n \tgit -C r3 rev-list --objects \\\n \t\t--filter=combine:tree:3+blob:limit=40+sparse:oid=master:pattern1%2brenamed%25 \\\n \t\tHEAD >actual &&\n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\n+\t# Use the same composite filter again, but with a pattern file name that\n+\t# requires encoding multiple characters, and use implicit filter\n+\t# combining.\n+\ttest_when_finished \"rm -f trace1\" &&\n+\tGIT_TRACE=$(pwd)/trace1 git -C r3 rev-list --objects \\\n+\t\t--filter=tree:3 --filter=blob:limit=40 \\\n+\t\t--filter=sparse:oid=\"master:p;at%ter+n\" \\\n+\t\tHEAD >actual &&\n+\n+\ttest_cmp expect actual &&\n+\tgrep \"Add to combine filter-spec: sparse:oid=master:p%3bat%25ter%2bn\" \\\n+\t\ttrace1 &&\n+\n+\t# Repeat the above test, but this time, the characters to encode are in\n+\t# the LHS of the combined filter.\n+\ttest_when_finished \"rm -f trace2\" &&\n+\tGIT_TRACE=$(pwd)/trace2 git -C r3 rev-list --objects \\\n+\t\t--filter=sparse:oid=master:^~pattern \\\n+\t\t--filter=tree:3 --filter=blob:limit=40 \\\n+\t\tHEAD >actual &&\n+\n+\ttest_cmp expect actual &&\n+\tgrep \"Add to combine filter-spec: sparse:oid=master:%5e%7epattern\" \\\n+\t\ttrace2\n '\n \n # Test provisional omit collection logic with a repo that has objects appearing\n # at multiple depths - first deeper than the filter's threshold, then shallow.\n \n test_expect_success 'setup r4' '\n \tgit init r4 &&\n \n \techo foo > r4/foo &&\n \tmkdir r4/subdir &&\ndiff --git a/transport.c b/transport.c\nindex f1fcd2c4b0..ee7dd1c062 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -217,20 +217,21 @@ static int set_git_option(struct git_transport_options *opts,\n \t} else if (!strcmp(name, TRANS_OPT_DEEPEN_RELATIVE)) {\n \t\topts->deepen_relative = !!value;\n \t\treturn 0;\n \t} else if (!strcmp(name, TRANS_OPT_FROM_PROMISOR)) {\n \t\topts->from_promisor = !!value;\n \t\treturn 0;\n \t} else if (!strcmp(name, TRANS_OPT_NO_DEPENDENTS)) {\n \t\topts->no_dependents = !!value;\n \t\treturn 0;\n \t} else if (!strcmp(name, TRANS_OPT_LIST_OBJECTS_FILTER)) {\n+\t\tlist_objects_filter_die_if_populated(&opts->filter_options);\n \t\tparse_list_objects_filter(&opts->filter_options, value);\n \t\treturn 0;\n \t}\n \treturn 1;\n }\n \n static int connect_setup(struct transport *transport, int for_push)\n {\n \tstruct git_transport_data *data = transport->data;\n \tint flags = transport->verbose > 0 ? CONNECT_VERBOSE : 0;\ndiff --git a/upload-pack.c b/upload-pack.c\nindex a74d293fef..dda2ac6f44 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -876,20 +876,21 @@ static void receive_needs(struct packet_reader *reader, struct object_array *wan\n \t\tif (process_deepen(reader->line, &depth))\n \t\t\tcontinue;\n \t\tif (process_deepen_since(reader->line, &deepen_since, &deepen_rev_list))\n \t\t\tcontinue;\n \t\tif (process_deepen_not(reader->line, &deepen_not, &deepen_rev_list))\n \t\t\tcontinue;\n \n \t\tif (skip_prefix(reader->line, \"filter \", &arg)) {\n \t\t\tif (!filter_capability_requested)\n \t\t\t\tdie(\"git upload-pack: filtering capability not negotiated\");\n+\t\t\tlist_objects_filter_die_if_populated(&filter_options);\n \t\t\tparse_list_objects_filter(&filter_options, arg);\n \t\t\tcontinue;\n \t\t}\n \n \t\tif (!skip_prefix(reader->line, \"want \", &arg) ||\n \t\t parse_oid_hex(arg, &oid_buf, &features))\n \t\t\tdie(\"git upload-pack: protocol error, \"\n \t\t\t \"expected to get object ID, not '%s'\", reader->line);\n \n \t\tif (parse_feature_request(features, \"deepen-relative\"))\n@@ -1297,20 +1298,21 @@ static void process_args(struct packet_reader *request,\n \t\t\tcontinue;\n \t\tif (process_deepen_not(arg, &data->deepen_not,\n \t\t\t\t &data->deepen_rev_list))\n \t\t\tcontinue;\n \t\tif (!strcmp(arg, \"deepen-relative\")) {\n \t\t\tdata->deepen_relative = 1;\n \t\t\tcontinue;\n \t\t}\n \n \t\tif (allow_filter && skip_prefix(arg, \"filter \", &p)) {\n+\t\t\tlist_objects_filter_die_if_populated(&filter_options);\n \t\t\tparse_list_objects_filter(&filter_options, p);\n \t\t\tcontinue;\n \t\t}\n \n \t\tif ((git_env_bool(\"GIT_TEST_SIDEBAND_ALL\", 0) ||\n \t\t allow_sideband_all) &&\n \t\t !strcmp(arg, \"sideband-all\")) {\n \t\t\tdata->writer.use_sideband = 1;\n \t\t\tcontinue;\n \t\t}\n-- \n2.21.0\n\n"},{"id":"377287","messageId":"4359a1db402ed42063b04f0f364055c4d4fe5b63.1560558910.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1560558910.git.matvore@google.com","subject":"[PATCH v4 09/10] list-objects-filter-options: clean up use of ALLOC_GROW","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-15T00:40:15Z","receivedAt":"2019-06-15T00:42:28Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Introduce a new macro ALLOC_GROW_BY which automatically zeros the added\narray elements and takes care of updating the nr value. Use the macro in\ncode introduced earlier in this patchset.\n\nSigned-off-by: Matthew DeVore \n---\n cache.h | 22 ++++++++++++++++++++++\n list-objects-filter-options.c | 17 +++++++----------\n 2 files changed, 29 insertions(+), 10 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex b4bb2e2c11..48fb0f63c2 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -653,33 +653,55 @@ int init_db(const char *git_dir, const char *real_git_dir,\n void sanitize_stdfds(void);\n int daemonize(void);\n \n #define alloc_nr(x) (((x)+16)*3/2)\n \n /*\n * Realloc the buffer pointed at by variable 'x' so that it can hold\n * at least 'nr' entries; the number of entries currently allocated\n * is 'alloc', using the standard growing factor alloc_nr() macro.\n *\n+ * Consider using ALLOC_GROW_BY instead of ALLOC_GROW as it has some\n+ * added niceties.\n+ *\n * DO NOT USE any expression with side-effect for 'x', 'nr', or 'alloc'.\n */\n #define ALLOC_GROW(x, nr, alloc) \\\n \tdo { \\\n \t\tif ((nr) > alloc) { \\\n \t\t\tif (alloc_nr(alloc) < (nr)) \\\n \t\t\t\talloc = (nr); \\\n \t\t\telse \\\n \t\t\t\talloc = alloc_nr(alloc); \\\n \t\t\tREALLOC_ARRAY(x, alloc); \\\n \t\t} \\\n \t} while (0)\n \n+/*\n+ * Similar to ALLOC_GROW but handles updating of the nr value and\n+ * zeroing the bytes of the newly-grown array elements.\n+ *\n+ * DO NOT USE any expression with side-effect for any of the\n+ * arguments.\n+ */\n+#define ALLOC_GROW_BY(x, nr, increase, alloc) \\\n+\tdo { \\\n+\t\tif (increase) { \\\n+\t\t\tsize_t new_nr = nr + (increase); \\\n+\t\t\tif (new_nr < nr) \\\n+\t\t\t\tBUG(\"negative growth in ALLOC_GROW_BY\"); \\\n+\t\t\tALLOC_GROW(x, new_nr, alloc); \\\n+\t\t\tmemset((x) + nr, 0, sizeof(*(x)) * (increase)); \\\n+\t\t\tnr = new_nr; \\\n+\t\t} \\\n+\t} while (0)\n+\n /* Initialize and use the cache information */\n struct lock_file;\n void preload_index(struct index_state *index,\n \t\t const struct pathspec *pathspec,\n \t\t unsigned int refresh_flags);\n int do_read_index(struct index_state *istate, const char *path,\n \t\t int must_exist); /* for testting only! */\n int read_index_from(struct index_state *, const char *path,\n \t\t const char *gitdir);\n int is_index_unborn(struct index_state *);\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex ab9c455fbd..f07928ea21 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -112,28 +112,26 @@ static int has_reserved_character(\n \t}\n \n \treturn 0;\n }\n \n static int parse_combine_subfilter(\n \tstruct list_objects_filter_options *filter_options,\n \tstruct strbuf *subspec,\n \tstruct strbuf *errbuf)\n {\n-\tsize_t new_index = filter_options->sub_nr++;\n+\tsize_t new_index = filter_options->sub_nr;\n \tchar *decoded;\n \tint result;\n \n-\tALLOC_GROW(filter_options->sub, filter_options->sub_nr,\n-\t\t filter_options->sub_alloc);\n-\tmemset(&filter_options->sub[new_index], 0,\n-\t sizeof(*filter_options->sub));\n+\tALLOC_GROW_BY(filter_options->sub, filter_options->sub_nr, 1,\n+\t\t filter_options->sub_alloc);\n \n \tdecoded = url_percent_decode(subspec->buf);\n \n \tresult = has_reserved_character(subspec, errbuf) ||\n \t\tgently_parse_list_objects_filter(\n \t\t\t&filter_options->sub[new_index], decoded, errbuf);\n \n \tfree(decoded);\n \treturn result;\n }\n@@ -248,27 +246,26 @@ int parse_list_objects_filter(\n \t\t\tfilter_options, arg, &errbuf);\n \t} else {\n \t\t/*\n \t\t * Make filter_options an LOFC_COMBINE spec so we can trivially\n \t\t * add subspecs to it.\n \t\t */\n \t\ttransform_to_combine_type(filter_options);\n \n \t\tstring_list_append(&filter_options->filter_spec, xstrdup(\"+\"));\n \t\tfilter_spec_append_urlencode(filter_options, arg);\n-\t\tALLOC_GROW(filter_options->sub, filter_options->sub_nr + 1,\n-\t\t\t filter_options->sub_alloc);\n-\t\tfilter_options = &filter_options->sub[filter_options->sub_nr++];\n-\t\tmemset(filter_options, 0, sizeof(*filter_options));\n+\t\tALLOC_GROW_BY(filter_options->sub, filter_options->sub_nr, 1,\n+\t\t\t filter_options->sub_alloc);\n \n \t\tparse_error = gently_parse_list_objects_filter(\n-\t\t\tfilter_options, arg, &errbuf);\n+\t\t\t&filter_options->sub[filter_options->sub_nr - 1], arg,\n+\t\t\t&errbuf);\n \t}\n \tif (parse_error)\n \t\tdie(\"%s\", errbuf.buf);\n \treturn 0;\n }\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct list_objects_filter_options *filter_options = opt->value;\n-- \n2.21.0\n\n"},{"id":"377288","messageId":"2f7566f697be759614a04c1277194f974bdcd662.1560558910.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1560558910.git.matvore@google.com","subject":"[PATCH v4 10/10] list-objects-filter-options: make parser void","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-15T00:40:16Z","receivedAt":"2019-06-15T00:42:30Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This function always returns 0, so make it return void instead.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects-filter-options.c | 12 +++++-------\n list-objects-filter-options.h | 2 +-\n 2 files changed, 6 insertions(+), 8 deletions(-)\n\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex f07928ea21..e19ecdcafa 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -225,21 +225,21 @@ static void transform_to_combine_type(\n \tstring_list_clear(&filter_options->sub[0].filter_spec, /*free_util=*/0);\n }\n \n void list_objects_filter_die_if_populated(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tif (filter_options->choice)\n \t\tdie(_(\"multiple filter-specs cannot be combined\"));\n }\n \n-int parse_list_objects_filter(\n+void parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg)\n {\n \tstruct strbuf errbuf = STRBUF_INIT;\n \tint parse_error;\n \n \tif (!filter_options->choice) {\n \t\tstring_list_append(&filter_options->filter_spec, xstrdup(arg));\n \n \t\tparse_error = gently_parse_list_objects_filter(\n@@ -255,34 +255,32 @@ int parse_list_objects_filter(\n \t\tfilter_spec_append_urlencode(filter_options, arg);\n \t\tALLOC_GROW_BY(filter_options->sub, filter_options->sub_nr, 1,\n \t\t\t filter_options->sub_alloc);\n \n \t\tparse_error = gently_parse_list_objects_filter(\n \t\t\t&filter_options->sub[filter_options->sub_nr - 1], arg,\n \t\t\t&errbuf);\n \t}\n \tif (parse_error)\n \t\tdie(\"%s\", errbuf.buf);\n-\treturn 0;\n }\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct list_objects_filter_options *filter_options = opt->value;\n \n-\tif (unset || !arg) {\n+\tif (unset || !arg)\n \t\tlist_objects_filter_set_no_filter(filter_options);\n-\t\treturn 0;\n-\t}\n-\n-\treturn parse_list_objects_filter(filter_options, arg);\n+\telse\n+\t\tparse_list_objects_filter(filter_options, arg);\n+\treturn 0;\n }\n \n const char *list_objects_filter_spec(struct list_objects_filter_options *filter)\n {\n \tif (!filter->filter_spec.nr)\n \t\tBUG(\"no filter_spec available for this filter\");\n \tif (filter->filter_spec.nr != 1) {\n \t\tstruct strbuf concatted = STRBUF_INIT;\n \t\tstrbuf_add_separated_string_list(\n \t\t\t&concatted, \"\", &filter->filter_spec);\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex d8bc7e946e..db37dfb34a 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -67,21 +67,21 @@ void list_objects_filter_die_if_populated(\n \tstruct list_objects_filter_options *filter_options);\n \n /*\n * Parses the filter spec string given by arg and either (1) simply places the\n * result in filter_options if it is not yet populated or (2) combines it with\n * the filter already in filter_options if it is already populated. In the case\n * of (2), the filter specs are combined as if specified with 'combine:'.\n *\n * Dies and prints a user-facing message if an error occurs.\n */\n-int parse_list_objects_filter(\n+void parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg);\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset);\n \n #define OPT_PARSE_LIST_OBJECTS_FILTER(fo) \\\n \t{ OPTION_CALLBACK, 0, CL_ARG__FILTER, fo, N_(\"args\"), \\\n \t N_(\"object filtering\"), 0, \\\n \t opt_parse_list_objects_filter }\n-- \n2.21.0\n\n"},{"id":"377380","messageId":"xmqqblyvsn0d.fsf@gitster-ct.c.googlers.com","threadId":"51217","inReplyTo":"cover.1560558910.git.matvore@google.com","subject":"Re: [PATCH v4 00/10] Filter combination","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-18T01:25:22Z","receivedAt":"2019-06-18T01:25:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthew DeVore writes:\n\n> I had to rebase this onto the latest master rev. master now has the patch which\n> disables the sparse:path filter, and v3 of this patch set has conflicts with it.\n> This version does not so it can be patched in and tried out by others.\n>\n> I have re-run the test suite on each commit. Sorry for the spamminess.\n\nThanks. Will queue.\n"},{"id":"377387","messageId":"nycvar.QRO.7.76.6.1906181040490.44@tvgsbejvaqbjf.bet","threadId":"51217","inReplyTo":"47a2680875e6f68fbf1f2e5a5a2630d263cdf426.1560558910.git.matvore@google.com","subject":"Re: [PATCH v4 04/10] list-objects-filter: implement composite filters","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-06-18T08:42:10Z","receivedAt":"2019-06-18T08:42:23Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Matthew,\n\nOn Fri, 14 Jun 2019, Matthew DeVore wrote:\n\n> diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\n> index 8e7b4f96fa..1c402c6059 100644\n> --- a/list-objects-filter-options.c\n> +++ b/list-objects-filter-options.c\n> [...]\n> +\n> +static int parse_combine_filter(\n> +\tstruct list_objects_filter_options *filter_options,\n> +\tconst char *arg,\n> +\tstruct strbuf *errbuf)\n> +{\n> +\tstruct strbuf **subspecs = strbuf_split_str(arg, '+', 0);\n> +\tsize_t sub;\n> +\tint result = 0;\n> +\n> +\tif (!subspecs[0]) {\n> +\t\tstrbuf_addf(errbuf,\n> +\t\t\t _(\"expected something after combine:\"));\n\nPlease squash this in, to pacify Coccinelle:\n\n-- snipsnap --\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex 5e5e30bc6a17..483ab512e24c 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -150,7 +150,7 @@ static int parse_combine_filter(\n \tint result = 0;\n\n \tif (!subspecs[0]) {\n-\t\tstrbuf_addf(errbuf,\n+\t\tstrbuf_addstr(errbuf,\n \t\t\t _(\"expected something after combine:\"));\n \t\tresult = 1;\n \t\tgoto cleanup;\n\n"},{"id":"377453","messageId":"20190618202216.GA35347@comcast.net","threadId":"51217","inReplyTo":"nycvar.QRO.7.76.6.1906181040490.44@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v4 04/10] list-objects-filter: implement composite filters","fromName":"Matthew DeVore","fromEmail":"matvore@comcast.net","sentAt":"2019-06-18T20:22:16Z","receivedAt":"2019-06-18T20:22:57Z","isPatch":true,"sender":{"key":"matvore@comcast.net","avatar":"https://gravatar.com/avatar/550c64ce544f82818ad931e244dfb08bbb1febfa6d1ce3cfd65e76215ca0ac8a?d=mp&s=160"},"body":"On Tue, Jun 18, 2019 at 10:42:10AM +0200, Johannes Schindelin wrote:\n> > +\tif (!subspecs[0]) {\n> > +\t\tstrbuf_addf(errbuf,\n> > +\t\t\t _(\"expected something after combine:\"));\n> \n> Please squash this in, to pacify Coccinelle:\n> \n> -- snipsnap --\n> diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\n> index 5e5e30bc6a17..483ab512e24c 100644\n> --- a/list-objects-filter-options.c\n> +++ b/list-objects-filter-options.c\n> @@ -150,7 +150,7 @@ static int parse_combine_filter(\n> \tint result = 0;\n> \n> \tif (!subspecs[0]) {\n> -\t\tstrbuf_addf(errbuf,\n> +\t\tstrbuf_addstr(errbuf,\n\nThank you - fixed locally for the next re-roll.\n"},{"id":"377759","messageId":"nycvar.QRO.7.76.6.1906212017150.44@tvgsbejvaqbjf.bet","threadId":"51217","inReplyTo":"20190618202216.GA35347@comcast.net","subject":"Re: [PATCH v4 04/10] list-objects-filter: implement composite filters","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-06-21T18:17:50Z","receivedAt":"2019-06-21T18:18:00Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 18 Jun 2019, Matthew DeVore wrote:\n\n> On Tue, Jun 18, 2019 at 10:42:10AM +0200, Johannes Schindelin wrote:\n> > > +\tif (!subspecs[0]) {\n> > > +\t\tstrbuf_addf(errbuf,\n> > > +\t\t\t _(\"expected something after combine:\"));\n> >\n> > Please squash this in, to pacify Coccinelle:\n\nJunio, maybe you can apply a SQUASH??? on top? This is the reason why `pu`\nis failing in the Azure Pipeline.\n\nThanks,\nDscho\n\n> >\n> > -- snipsnap --\n> > diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\n> > index 5e5e30bc6a17..483ab512e24c 100644\n> > --- a/list-objects-filter-options.c\n> > +++ b/list-objects-filter-options.c\n> > @@ -150,7 +150,7 @@ static int parse_combine_filter(\n> > \tint result = 0;\n> >\n> > \tif (!subspecs[0]) {\n> > -\t\tstrbuf_addf(errbuf,\n> > +\t\tstrbuf_addstr(errbuf,\n>\n> Thank you - fixed locally for the next re-roll.\n>\n"},{"id":"377794","messageId":"20190621225838.226321-1-jonathantanmy@google.com","threadId":"51217","inReplyTo":"70568c42ae6d59dacbb36ffb8e4a8828b6595158.1560558910.git.matvore@google.com","subject":"Re: [PATCH v4 01/10] list-objects-filter: make API easier to use","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2019-06-21T22:58:38Z","receivedAt":"2019-06-21T22:58:46Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Make the list-objects-filter.h API more opaque and easier to use. This\n> prepares for combined filter support, where filters will be created and\n> used in a new context.\n> \n> Helped-by: Jeff Hostetler \n> Helped-by: Junio C Hamano \n> Signed-off-by: Matthew DeVore \n\nSo what happens is that filter_fn, filter_free_fn, and filter_data are\nencapsulated into one opaque object, and users will now use filter_fn\nand filter_free_fn through other functions that we expose, allowing us\nto add some conveniences that currently have to be repeated at each call\nsite.\n\nI would prefer the following commit message:\n\n list-objects-filter: encapsulate filter components\n\n Encapsulate filter_fn, filter_free_fn, and filter_data into its own\n opaque struct.\n\n Due to opaqueness, filter_fn and filter_free_fn can no longer be\n accessed directly by users. Currently, all usages of filter_fn are\n guarded by a necessary check:\n\n (obj->flags & NOT_USER_GIVEN) && filter_fn\n\n Take the opportunity to include this check into the new function\n list_objects_filter__filter_object(), so that we no longer need to\n write this check at every caller of the filter function.\n\n Also, the init functions in list-objects-filter.c no longer need to\n confusingly return the filter constituents in various places\n (filter_fn and filter_free_fn as out parameters, and filter_data as\n the function's return value); they can just initialize the \"struct\n filter\" passed in.\n\n> +enum list_objects_filter_result list_objects_filter__filter_object(\n> +\tstruct repository *r,\n> +\tenum list_objects_filter_situation filter_situation,\n> +\tstruct object *obj,\n> +\tconst char *pathname,\n> +\tconst char *filename,\n> +\tstruct filter *filter)\n> +{\n> +\tif (filter && (obj->flags & NOT_USER_GIVEN))\n> +\t\treturn filter->filter_object_fn(r, filter_situation, obj,\n> +\t\t\t\t\t\tpathname, filename,\n> +\t\t\t\t\t\tfilter->filter_data);\n> +\t/*\n> +\t * No filter is active or user gave object explicitly. Choose default\n> +\t * behavior based on filter situation.\n> +\t */\n\nThis part is when we do not need to apply the filter (or none exists). I\nthink the comment will be better if stated more explicitly:\n\n No filter is active or user gave object explicitly. In this case,\n always show the object (except when LOFS_END_TREE, since this tree had\n already been shown when LOFS_BEGIN_TREE).\n\n> +\tif (filter_situation == LOFS_END_TREE)\n> +\t\treturn 0;\n> +\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n> +}\n"},{"id":"377796","messageId":"20190622002626.245441-1-jonathantanmy@google.com","threadId":"51217","inReplyTo":"47a2680875e6f68fbf1f2e5a5a2630d263cdf426.1560558910.git.matvore@google.com","subject":"Re: [PATCH v4 04/10] list-objects-filter: implement composite filters","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2019-06-22T00:26:26Z","receivedAt":"2019-06-22T00:26:32Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Allow combining filters such that only objects accepted by all filters\n> are shown. The motivation for this is to allow getting directory\n> listings without also fetching blobs. This can be done by combining\n> blob:none with tree:. There are massive repositories that have\n> larger-than-expected trees - even if you include only a single commit.\n\nFirst of all, patches 2 and 3 are straightforward and LGTM. On to patch\n4...\n\n[snip]\n\n> The current usage requires passing the filter to rev-list in the\n> following form:\n> \n> \t--filter= --filter= ...\n> \n> Such usage is currently an error, so giving it a meaning is backwards-\n> compatible.\n> \n> The URL-encoding scheme is being introduced before the repeated flag\n> logic, and the user-facing documentation for URL-encoding is being\n> withheld until the repeated flag feature is implemented. The\n> URL-encoding is in general not meant to be used directly by the user,\n> and it is better to describe the URL-encoding feature in terms of the\n> repeated flag.\n\nAs of this commit, we don't support such arguments passed to rev-list in\nthis way, so I would write these paragraphs as:\n\n A combined filter supports any number of subfilters, and is written in\n the following form:\n\n combine:++\n\n Certain non-alphanumeric characters in each filter must be\n URL-encoded.\n\n For now, combined filters must be specified in this form. In a\n subsequent commit, rev-list will support multiple --filter arguments\n which will have the same effect as specifying one filter argument\n starting with \"combine:\".\n\n> Helped-by: Emily Shaffer \n> Helped-by: Jeff Hostetler \n> Helped-by: Junio C Hamano \n> Signed-off-by: Matthew DeVore \n> ---\n> list-objects-filter-options.c | 106 ++++++++++++++++++-\n> list-objects-filter-options.h | 17 ++-\n> list-objects-filter.c | 159 ++++++++++++++++++++++++++++\n> t/t6112-rev-list-filters-objects.sh | 151 +++++++++++++++++++++++++-\n> url.c | 6 ++\n> url.h | 8 ++\n> 6 files changed, 441 insertions(+), 6 deletions(-)\n> \n> @@ -28,22 +34,20 @@ static int gently_parse_list_objects_filter(\n> \tstruct strbuf *errbuf)\n> {\n> \tconst char *v0;\n> \n> \tif (filter_options->choice) {\n> \t\tstrbuf_addstr(\n> \t\t\terrbuf, _(\"multiple filter-specs cannot be combined\"));\n> \t\treturn 1;\n> \t}\n> \n> -\tfilter_options->filter_spec = strdup(arg);\n> -\n\nThis line has been removed from gently_parse_list_objects_filter()\nbecause this function gains another caller that does not need it.\nTo compensate, this line has been added to both its existing callers.\n\n> @@ -31,27 +32,37 @@ struct list_objects_filter_options {\n> \t * the filtering algorithm to use.\n> \t */\n> \tenum list_objects_filter_choice choice;\n> \n> \t/*\n> \t * Choice is LOFC_DISABLED because \"--no-filter\" was requested.\n> \t */\n> \tunsigned int no_filter : 1;\n> \n> \t/*\n> -\t * Parsed values (fields) from within the filter-spec. These are\n> -\t * choice-specific; not all values will be defined for any given\n> -\t * choice.\n> +\t * BEGIN choice-specific parsed values from within the filter-spec. Only\n> +\t * some values will be defined for any given choice.\n> \t */\n> +\n> \tstruct object_id *sparse_oid_value;\n> \tunsigned long blob_limit_value;\n> \tunsigned long tree_exclude_depth;\n> +\n> +\t/* LOFC_COMBINE values */\n> +\n> +\t/* This array contains all the subfilters which this filter combines. */\n> +\tsize_t sub_nr, sub_alloc;\n> +\tstruct list_objects_filter_options *sub;\n> +\n> +\t/*\n> +\t * END choice-specific parsed values.\n> +\t */\n> };\n\nI still think it's cleaner to just have a \"left subfilter\" and \"right\nsubfilter\", but I don't feel strongly about it. In any case, this is an\ninternal detail and can always be changed in the future.\n\n> +\t/*\n> +\t * Optional. If this function is supplied and the filter needs to\n> +\t * collect omits, then this function is called once before free_fn is\n> +\t * called.\n> +\t */\n> +\tvoid (*finalize_omits_fn)(struct oidset *omits, void *filter_data);\n\nThis is needed because a combined filter's omits actually lie in the\nsubfilters. Resolving it this way means that callers must call\nlist_objects_filter__free() before using the omits set. Can you add\ndocumentation to __init() (which is the first function to take in the\nomits set) and __free() describing this?\n\n(As stated in the test below, we cannot just share one omits set amongst\nall the subfilters - see filter_trees_update_omits and the call site\nthat relies on its return value.)\n\nHere comes the tricky part...\n\n> +static int should_delegate(enum list_objects_filter_situation filter_situation,\n> +\t\t\t struct object *obj,\n> +\t\t\t struct subfilter *sub)\n> +{\n> +\tif (!sub->is_skipping_tree)\n> +\t\treturn 1;\n> +\tif (filter_situation == LOFS_END_TREE &&\n> +\t\toideq(&obj->oid, &sub->skip_tree)) {\n> +\t\tsub->is_skipping_tree = 0;\n> +\t\treturn 1;\n> +\t}\n> +\treturn 0;\n> +}\n\nOptional: I think this should be called \"test_and_set_skip_tree\" or\nsomething like that, made to return the inverse of its current return\nvalue, and documented:\n\n Returns the value of sub->is_skipping_tree at the moment of\n invocation. If iteration is at the LOFS_END_TREE of the tree currently\n being skipped, first clears sub->is_skipping_tree before returning.\n\n> +static enum list_objects_filter_result process_subfilter(\n> +\tstruct repository *r,\n> +\tenum list_objects_filter_situation filter_situation,\n> +\tstruct object *obj,\n> +\tconst char *pathname,\n> +\tconst char *filename,\n> +\tstruct subfilter *sub)\n> +{\n> +\tenum list_objects_filter_result result;\n> +\n> +\t/*\n> +\t * Check should_delegate before oidset_contains so that\n> +\t * is_skipping_tree gets unset even when the object is marked as seen.\n> +\t * As of this writing, no filter uses LOFR_MARK_SEEN on trees that also\n> +\t * uses LOFR_SKIP_TREE, so the ordering is only theoretically\n> +\t * important. Be cautious if you change the order of the below checks\n> +\t * and more filters have been added!\n> +\t */\n> +\tif (!should_delegate(filter_situation, obj, sub))\n> +\t\treturn LOFR_ZERO;\n> +\tif (oidset_contains(&sub->seen, &obj->oid))\n> +\t\treturn LOFR_ZERO;\n> +\n> +\tresult = list_objects_filter__filter_object(\n> +\t\tr, filter_situation, obj, pathname, filename, sub->filter);\n> +\n> +\tif (result & LOFR_MARK_SEEN)\n> +\t\toidset_insert(&sub->seen, &obj->oid);\n> +\n> +\tif (result & LOFR_SKIP_TREE) {\n> +\t\tsub->is_skipping_tree = 1;\n> +\t\tsub->skip_tree = obj->oid;\n> +\t}\n> +\n> +\treturn result;\n> +}\n\nLooks good.\n\n> +static enum list_objects_filter_result filter_combine(\n> +\tstruct repository *r,\n> +\tenum list_objects_filter_situation filter_situation,\n> +\tstruct object *obj,\n> +\tconst char *pathname,\n> +\tconst char *filename,\n> +\tstruct oidset *omits,\n> +\tvoid *filter_data)\n> +{\n> +\tstruct combine_filter_data *d = filter_data;\n> +\tenum list_objects_filter_result combined_result =\n> +\t\tLOFR_DO_SHOW | LOFR_MARK_SEEN | LOFR_SKIP_TREE;\n> +\tsize_t sub;\n> +\n> +\tfor (sub = 0; sub < d->nr; sub++) {\n> +\t\tenum list_objects_filter_result sub_result = process_subfilter(\n> +\t\t\tr, filter_situation, obj, pathname, filename,\n> +\t\t\t&d->sub[sub]);\n> +\t\tif (!(sub_result & LOFR_DO_SHOW))\n> +\t\t\tcombined_result &= ~LOFR_DO_SHOW;\n> +\t\tif (!(sub_result & LOFR_MARK_SEEN))\n> +\t\t\tcombined_result &= ~LOFR_MARK_SEEN;\n> +\t\tif (!d->sub[sub].is_skipping_tree)\n> +\t\t\tcombined_result &= ~LOFR_SKIP_TREE;\n> +\t}\n> +\n> +\treturn combined_result;\n> +}\n\nAnd also looks good. Might be confusing for tree skipping to be\ncommunicated through is_skipping_tree instead of the return value, but\nis_skipping_tree needs to be set anyway for other reasons, so that's\nconvenient.\n"},{"id":"377797","messageId":"20190622003713.248581-1-jonathantanmy@google.com","threadId":"51217","inReplyTo":"1a95dd91927973038c3d59bc3215556e448f0e63.1560558910.git.matvore@google.com","subject":"Re: [PATCH v4 06/10] list-objects-filter-options: make filter_spec a string_list","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2019-06-22T00:37:13Z","receivedAt":"2019-06-22T00:37:20Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Patch 5 and this patch look good to me.\n\n> @@ -1134,27 +1134,25 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n> \t\ttransport_set_option(transport, TRANS_OPT_FOLLOWTAGS, \"1\");\n> \n> \tif (option_upload_pack)\n> \t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK,\n> \t\t\t\t option_upload_pack);\n> \n> \tif (server_options.nr)\n> \t\ttransport->server_options = &server_options;\n> \n> \tif (filter_options.choice) {\n> -\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n> -\t\texpand_list_objects_filter_spec(&filter_options,\n> -\t\t\t\t\t\t&expanded_filter_spec);\n> +\t\tconst char *spec =\n> +\t\t\texpand_list_objects_filter_spec(&filter_options);\n> \t\ttransport_set_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER,\n> -\t\t\t\t expanded_filter_spec.buf);\n> +\t\t\t\t spec);\n> \t\ttransport_set_option(transport, TRANS_OPT_FROM_PROMISOR, \"1\");\n> -\t\tstrbuf_release(&expanded_filter_spec);\n\nSo expand_list_objects_filter_spec() now returns a filter_options-owned\nstring (instead of previously writing to a strbuf), which is why we no\nlonger need to do any freeing or releasing. That makes sense. (Same for\nthe other call sites.)\n\n> @@ -177,72 +177,89 @@ static int parse_combine_filter(\n> \t}\n> \treturn result;\n> }\n> \n> int parse_list_objects_filter(struct list_objects_filter_options *filter_options,\n> \t\t\t const char *arg)\n> {\n> \tstruct strbuf buf = STRBUF_INIT;\n> \tif (filter_options->choice)\n> \t\tdie(_(\"multiple filter-specs cannot be combined\"));\n> -\tfilter_options->filter_spec = strdup(arg);\n> +\tstring_list_append(&filter_options->filter_spec, xstrdup(arg));\n\nThis append needs to be called with xstrdup, because a zero-initialized\nstring list is NODUP. OK.\n"},{"id":"377798","messageId":"20190622004631.251573-1-jonathantanmy@google.com","threadId":"51217","inReplyTo":"2f7566f697be759614a04c1277194f974bdcd662.1560558910.git.matvore@google.com","subject":"Re: [PATCH v4 10/10] list-objects-filter-options: make parser void","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2019-06-22T00:46:31Z","receivedAt":"2019-06-22T00:46:37Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> This function always returns 0, so make it return void instead.\n\nAnd...patches 7-10 look straightforward and good to me.\n\nIn summary, I don't think any changes need to be made to all 10 patches\nother than textual ones (commit messages, documentation, and function\nnames).\n"},{"id":"378107","messageId":"20190627004635.GA24081@comcast.net","threadId":"51217","inReplyTo":"20190621225838.226321-1-jonathantanmy@google.com","subject":"Re: [PATCH v4 01/10] list-objects-filter: make API easier to use","fromName":"Matthew DeVore","fromEmail":"matvore@comcast.net","sentAt":"2019-06-27T00:46:35Z","receivedAt":"2019-06-27T00:47:06Z","isPatch":true,"sender":{"key":"matvore@comcast.net","avatar":"https://gravatar.com/avatar/550c64ce544f82818ad931e244dfb08bbb1febfa6d1ce3cfd65e76215ca0ac8a?d=mp&s=160"},"body":"On Fri, Jun 21, 2019 at 03:58:38PM -0700, Jonathan Tan wrote:\n> So what happens is that filter_fn, filter_free_fn, and filter_data are\n> encapsulated into one opaque object, and users will now use filter_fn\n> and filter_free_fn through other functions that we expose, allowing us\n> to add some conveniences that currently have to be repeated at each call\n> site.\n> \n> I would prefer the following commit message:\n> \n> list-objects-filter: encapsulate filter components\n> \n> Encapsulate filter_fn, filter_free_fn, and filter_data into its own\n> opaque struct.\n> \n> Due to opaqueness, filter_fn and filter_free_fn can no longer be\n> accessed directly by users. Currently, all usages of filter_fn are\n> guarded by a necessary check:\n> \n> (obj->flags & NOT_USER_GIVEN) && filter_fn\n> \n> Take the opportunity to include this check into the new function\n> list_objects_filter__filter_object(), so that we no longer need to\n> write this check at every caller of the filter function.\n> \n> Also, the init functions in list-objects-filter.c no longer need to\n> confusingly return the filter constituents in various places\n> (filter_fn and filter_free_fn as out parameters, and filter_data as\n> the function's return value); they can just initialize the \"struct\n> filter\" passed in.\n> \n\nVery nice, applied. I think your commit message is much more helpful than mine\nand doesn't use the filter combination feature as an excuse for the change.\n\n> > +\t/*\n> > +\t * No filter is active or user gave object explicitly. Choose default\n> > +\t * behavior based on filter situation.\n> > +\t */\n> \n> This part is when we do not need to apply the filter (or none exists). I\n> think the comment will be better if stated more explicitly:\n> \n> No filter is active or user gave object explicitly. In this case,\n> always show the object (except when LOFS_END_TREE, since this tree had\n> already been shown when LOFS_BEGIN_TREE).\n> \n\nAgreed, this is a little better. Applied.\n"},{"id":"378200","messageId":"20190627211247.GB54617@comcast.net","threadId":"51217","inReplyTo":"20190622002626.245441-1-jonathantanmy@google.com","subject":"Re: [PATCH v4 04/10] list-objects-filter: implement composite filters","fromName":"Matthew DeVore","fromEmail":"matvore@comcast.net","sentAt":"2019-06-27T21:12:47Z","receivedAt":"2019-06-27T21:13:09Z","isPatch":true,"sender":{"key":"matvore@comcast.net","avatar":"https://gravatar.com/avatar/550c64ce544f82818ad931e244dfb08bbb1febfa6d1ce3cfd65e76215ca0ac8a?d=mp&s=160"},"body":"On Fri, Jun 21, 2019 at 05:26:26PM -0700, Jonathan Tan wrote:\n> > Allow combining filters such that only objects accepted by all filters\n> > are shown. The motivation for this is to allow getting directory\n> > listings without also fetching blobs. This can be done by combining\n> > blob:none with tree:. There are massive repositories that have\n> > larger-than-expected trees - even if you include only a single commit.\n> \n> First of all, patches 2 and 3 are straightforward and LGTM. On to patch\n> 4...\n> \n> [snip]\n> \n> > The current usage requires passing the filter to rev-list in the\n> > following form:\n> > \n> > \t--filter= --filter= ...\n> > \n> > Such usage is currently an error, so giving it a meaning is backwards-\n> > compatible.\n> > \n> > The URL-encoding scheme is being introduced before the repeated flag\n> > logic, and the user-facing documentation for URL-encoding is being\n> > withheld until the repeated flag feature is implemented. The\n> > URL-encoding is in general not meant to be used directly by the user,\n> > and it is better to describe the URL-encoding feature in terms of the\n> > repeated flag.\n> \n> As of this commit, we don't support such arguments passed to rev-list in\n> this way, so I would write these paragraphs as:\n> \n> A combined filter supports any number of subfilters, and is written in\n> the following form:\n> \n> combine:++\n> \n> Certain non-alphanumeric characters in each filter must be\n> URL-encoded.\n> \n> For now, combined filters must be specified in this form. In a\n> subsequent commit, rev-list will support multiple --filter arguments\n> which will have the same effect as specifying one filter argument\n> starting with \"combine:\".\n\nDone, but I've amended your last paragraph to include the excuse about the\nmissing documentation:\n\n For now, combined filters must be specified in this form. In a\n subsequent commit, rev-list will support multiple --filter arguments\n which will have the same effect as specifying one filter argument\n starting with \"combine:\". The documentation will be updated in that\n commit, as the URL-encoding scheme is in general not meant to be used\n directly by the user, and it is better to describe the URL-encoding\n feature in terms of the repeated flag.\n\n> > +\n> > +\t/* LOFC_COMBINE values */\n> > +\n> > +\t/* This array contains all the subfilters which this filter combines. */\n> > +\tsize_t sub_nr, sub_alloc;\n> > +\tstruct list_objects_filter_options *sub;\n> > +\n> > +\t/*\n> > +\t * END choice-specific parsed values.\n> > +\t */\n> > };\n> \n> I still think it's cleaner to just have a \"left subfilter\" and \"right\n> subfilter\", but I don't feel strongly about it. In any case, this is an\n> internal detail and can always be changed in the future.\n> \n\nInteresting. I think there are two reasonable ways of thinking about it:\n\na. we are parsing a nested left-associative binary expression, and it's\n helpful if the internal representation matches that semantic. (Maybe\n this is what you're thinking?)\n\nb. linked-lists are somewhat aberrational and should only be used in a\n very narrow class of code. Arrays are more universal and therefore more\n readable.\n\nI'll keep it as-is for now since I don't see a great reason to revert it to the\nprevious style. Thank you for pointing this out - I *did* suspect there was a\nbig subjective factor to people's preference for the array form.\n\n> > +\t/*\n> > +\t * Optional. If this function is supplied and the filter needs to\n> > +\t * collect omits, then this function is called once before free_fn is\n> > +\t * called.\n> > +\t */\n> > +\tvoid (*finalize_omits_fn)(struct oidset *omits, void *filter_data);\n> \n> This is needed because a combined filter's omits actually lie in the\n> subfilters. Resolving it this way means that callers must call\n> list_objects_filter__free() before using the omits set. Can you add\n> documentation to __init() (which is the first function to take in the\n> omits set) and __free() describing this?\n> \n> (As stated in the test below, we cannot just share one omits set amongst\n> all the subfilters - see filter_trees_update_omits and the call site\n> that relies on its return value.)\n> \n\nI documented __init as follows:\n\n/*\n * Constructor for the set of defined list-objects filters.\n * The `omitted` set is optional. It is populated with objects that the\n * filter excludes. This set should not be considered finalized until\n * after list_objects_filter__free is called on the returned `struct\n * filter *`.\n */\n\nAnd __free:\n\n/*\n * Destroys `filter` and finalizes the `omitted` set, if present. Does\n * nothing if `filter` is null.\n */\n\nAnd finalize_omits_fn (which has the internal specifics):\n\n\t/*\n\t * Optional. If this function is supplied and the filter needs\n\t * to collect omits, then this function is called once before\n\t * free_fn is called.\n\t *\n\t * This is required because the following two conditions hold:\n\t *\n\t * a. A tree filter can add and remove objects as an object\n\t * graph is traversed.\n\t * b. A combine filter's omit set is the union of all its\n\t * subfilters, which may include tree: filters.\n\t *\n\t * As such, the omits sets must be separate sets, and can only\n\t * be unioned after the traversal is completed.\n\t */\n\n> Here comes the tricky part...\n> \n> > +static int should_delegate(enum list_objects_filter_situation filter_situation,\n> > +\t\t\t struct object *obj,\n> > +\t\t\t struct subfilter *sub)\n> > +{\n> > +\tif (!sub->is_skipping_tree)\n> > +\t\treturn 1;\n> > +\tif (filter_situation == LOFS_END_TREE &&\n> > +\t\toideq(&obj->oid, &sub->skip_tree)) {\n> > +\t\tsub->is_skipping_tree = 0;\n> > +\t\treturn 1;\n> > +\t}\n> > +\treturn 0;\n> > +}\n> \n> Optional: I think this should be called \"test_and_set_skip_tree\" or\n> something like that, made to return the inverse of its current return\n> value, and documented:\n> \n> Returns the value of sub->is_skipping_tree at the moment of\n> invocation. If iteration is at the LOFS_END_TREE of the tree currently\n> being skipped, first clears sub->is_skipping_tree before returning.\n> \n\nThat is not totally accurate, since in the second if block\n(LOFS_END_TREE) we would return 0 even though is_skipping_tree starts\nout as 1.\n\nSince it's not equal to is_skipping_tree before invocation, I don't\nthink there is a better name to give it ATM.\n\nSo in the interest of removing \"trickiness\" I've inlined the function\n(since there is no real way to express what it's doing in a function\nname anyway) and I think it's more readable. This is the new callsite:\n\n\t/*\n\t * Check and update is_skipping_tree before oidset_contains so\n\t * that is_skipping_tree gets unset even when the object is\n\t * marked as seen. As of this writing, no filter uses\n\t * LOFR_MARK_SEEN on trees that also uses LOFR_SKIP_TREE, so the\n\t * ordering is only theoretically important. Be cautious if you\n\t * change the order of the below checks and more filters have\n\t * been added!\n\t */\n\tif (sub->is_skipping_tree) {\n\t\tif (filter_situation == LOFS_END_TREE &&\n\t\t oideq(&obj->oid, &sub->skip_tree))\n\t\t\tsub->is_skipping_tree = 0;\n\t\telse\n\t\t\treturn LOFR_ZERO;\n\t}\n\nHopefully that's agreeable for everyone.\n"},{"id":"378201","messageId":"20190627211704.GC54617@comcast.net","threadId":"51217","inReplyTo":"20190622003713.248581-1-jonathantanmy@google.com","subject":"Re: [PATCH v4 06/10] list-objects-filter-options: make filter_spec a string_list","fromName":"Matthew DeVore","fromEmail":"matvore@comcast.net","sentAt":"2019-06-27T21:17:04Z","receivedAt":"2019-06-27T21:17:21Z","isPatch":true,"sender":{"key":"matvore@comcast.net","avatar":"https://gravatar.com/avatar/550c64ce544f82818ad931e244dfb08bbb1febfa6d1ce3cfd65e76215ca0ac8a?d=mp&s=160"},"body":"On Fri, Jun 21, 2019 at 05:37:13PM -0700, Jonathan Tan wrote:\n> Patch 5 and this patch look good to me.\n> \n> \n> \n> So expand_list_objects_filter_spec() now returns a filter_options-owned\n> string (instead of previously writing to a strbuf), which is why we no\n> longer need to do any freeing or releasing. That makes sense. (Same for\n> the other call sites.)\n> \n> \n> \n> This append needs to be called with xstrdup, because a zero-initialized\n> string list is NODUP. OK.\n\nThank you for taking a look.\n"},{"id":"378203","messageId":"20190627212457.GD54617@comcast.net","threadId":"51217","inReplyTo":"20190622004631.251573-1-jonathantanmy@google.com","subject":"Re: [PATCH v4 10/10] list-objects-filter-options: make parser void","fromName":"Matthew DeVore","fromEmail":"matvore@comcast.net","sentAt":"2019-06-27T21:24:57Z","receivedAt":"2019-06-27T21:25:14Z","isPatch":true,"sender":{"key":"matvore@comcast.net","avatar":"https://gravatar.com/avatar/550c64ce544f82818ad931e244dfb08bbb1febfa6d1ce3cfd65e76215ca0ac8a?d=mp&s=160"},"body":"On Fri, Jun 21, 2019 at 05:46:31PM -0700, Jonathan Tan wrote:\n> > This function always returns 0, so make it return void instead.\n> \n> And...patches 7-10 look straightforward and good to me.\n> \n> In summary, I don't think any changes need to be made to all 10 patches\n> other than textual ones (commit messages, documentation, and function\n> names).\n\nGreat. I feel much better about the comments and commit messages now. I\nam about to send a roll-up (v5). Here is the interdiff which catches\nyour comments and Dscho's comment about strbuf_addstr:\n\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex d9c8da5b70..9e64832a5e 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -139,22 +139,21 @@ static int parse_combine_subfilter(\n static int parse_combine_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg,\n \tstruct strbuf *errbuf)\n {\n \tstruct strbuf **subspecs = strbuf_split_str(arg, '+', 0);\n \tsize_t sub;\n \tint result = 0;\n \n \tif (!subspecs[0]) {\n-\t\tstrbuf_addf(errbuf,\n-\t\t\t _(\"expected something after combine:\"));\n+\t\tstrbuf_addstr(errbuf, _(\"expected something after combine:\"));\n \t\tresult = 1;\n \t\tgoto cleanup;\n \t}\n \n \tfor (sub = 0; subspecs[sub] && !result; sub++) {\n \t\tif (subspecs[sub + 1]) {\n \t\t\t/*\n \t\t\t * This is not the last subspec. Remove trailing \"+\" so\n \t\t\t * we can parse it.\n \t\t\t */\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex 6b99f707e4..d664264d65 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -38,23 +38,33 @@ struct filter {\n \tenum list_objects_filter_result (*filter_object_fn)(\n \t\tstruct repository *r,\n \t\tenum list_objects_filter_situation filter_situation,\n \t\tstruct object *obj,\n \t\tconst char *pathname,\n \t\tconst char *filename,\n \t\tstruct oidset *omits,\n \t\tvoid *filter_data);\n \n \t/*\n-\t * Optional. If this function is supplied and the filter needs to\n-\t * collect omits, then this function is called once before free_fn is\n-\t * called.\n+\t * Optional. If this function is supplied and the filter needs\n+\t * to collect omits, then this function is called once before\n+\t * free_fn is called.\n+\t *\n+\t * This is required because the following two conditions hold:\n+\t *\n+\t * a. A tree filter can add and remove objects as an object\n+\t * graph is traversed.\n+\t * b. A combine filter's omit set is the union of all its\n+\t * subfilters, which may include tree: filters.\n+\t *\n+\t * As such, the omits sets must be separate sets, and can only\n+\t * be unioned after the traversal is completed.\n \t */\n \tvoid (*finalize_omits_fn)(struct oidset *omits, void *filter_data);\n \n \tvoid (*free_fn)(void *filter_data);\n \n \tvoid *filter_data;\n \n \t/* If non-NULL, the filter collects a list of the omitted OIDs here. */\n \tstruct oidset *omits;\n };\n@@ -485,54 +495,46 @@ static void filter_sparse_oid__init(\n \tfilter->filter_object_fn = filter_sparse;\n \tfilter->free_fn = filter_sparse_free;\n }\n \n /* A filter which only shows objects shown by all sub-filters. */\n struct combine_filter_data {\n \tstruct subfilter *sub;\n \tsize_t nr;\n };\n \n-static int should_delegate(enum list_objects_filter_situation filter_situation,\n-\t\t\t struct object *obj,\n-\t\t\t struct subfilter *sub)\n-{\n-\tif (!sub->is_skipping_tree)\n-\t\treturn 1;\n-\tif (filter_situation == LOFS_END_TREE &&\n-\t\toideq(&obj->oid, &sub->skip_tree)) {\n-\t\tsub->is_skipping_tree = 0;\n-\t\treturn 1;\n-\t}\n-\treturn 0;\n-}\n-\n static enum list_objects_filter_result process_subfilter(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n \tstruct subfilter *sub)\n {\n \tenum list_objects_filter_result result;\n \n \t/*\n-\t * Check should_delegate before oidset_contains so that\n-\t * is_skipping_tree gets unset even when the object is marked as seen.\n-\t * As of this writing, no filter uses LOFR_MARK_SEEN on trees that also\n-\t * uses LOFR_SKIP_TREE, so the ordering is only theoretically\n-\t * important. Be cautious if you change the order of the below checks\n-\t * and more filters have been added!\n+\t * Check and update is_skipping_tree before oidset_contains so\n+\t * that is_skipping_tree gets unset even when the object is\n+\t * marked as seen. As of this writing, no filter uses\n+\t * LOFR_MARK_SEEN on trees that also uses LOFR_SKIP_TREE, so the\n+\t * ordering is only theoretically important. Be cautious if you\n+\t * change the order of the below checks and more filters have\n+\t * been added!\n \t */\n-\tif (!should_delegate(filter_situation, obj, sub))\n-\t\treturn LOFR_ZERO;\n+\tif (sub->is_skipping_tree) {\n+\t\tif (filter_situation == LOFS_END_TREE &&\n+\t\t oideq(&obj->oid, &sub->skip_tree))\n+\t\t\tsub->is_skipping_tree = 0;\n+\t\telse\n+\t\t\treturn LOFR_ZERO;\n+\t}\n \tif (oidset_contains(&sub->seen, &obj->oid))\n \t\treturn LOFR_ZERO;\n \n \tresult = list_objects_filter__filter_object(\n \t\tr, filter_situation, obj, pathname, filename, sub->filter);\n \n \tif (result & LOFR_MARK_SEEN)\n \t\toidset_insert(&sub->seen, &obj->oid);\n \n \tif (result & LOFR_SKIP_TREE) {\n@@ -673,22 +675,23 @@ enum list_objects_filter_result list_objects_filter__filter_object(\n \tconst char *pathname,\n \tconst char *filename,\n \tstruct filter *filter)\n {\n \tif (filter && (obj->flags & NOT_USER_GIVEN))\n \t\treturn filter->filter_object_fn(r, filter_situation, obj,\n \t\t\t\t\t\tpathname, filename,\n \t\t\t\t\t\tfilter->omits,\n \t\t\t\t\t\tfilter->filter_data);\n \t/*\n-\t * No filter is active or user gave object explicitly. Choose default\n-\t * behavior based on filter situation.\n+\t * No filter is active or user gave object explicitly. In this case,\n+\t * always show the object (except when LOFS_END_TREE, since this tree\n+\t * had already been shown when LOFS_BEGIN_TREE).\n \t */\n \tif (filter_situation == LOFS_END_TREE)\n \t\treturn 0;\n \treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n }\n \n void list_objects_filter__free(struct filter *filter)\n {\n \tif (!filter)\n \t\treturn;\ndiff --git a/list-objects-filter.h b/list-objects-filter.h\nindex 6908954266..cfd784e203 100644\n--- a/list-objects-filter.h\n+++ b/list-objects-filter.h\n@@ -55,32 +55,41 @@ enum list_objects_filter_result {\n };\n \n enum list_objects_filter_situation {\n \tLOFS_BEGIN_TREE,\n \tLOFS_END_TREE,\n \tLOFS_BLOB\n };\n \n struct filter;\n \n-/* Constructor for the set of defined list-objects filters. */\n+/*\n+ * Constructor for the set of defined list-objects filters.\n+ * The `omitted` set is optional. It is populated with objects that the\n+ * filter excludes. This set should not be considered finalized until\n+ * after list_objects_filter__free is called on the returned `struct\n+ * filter *`.\n+ */\n struct filter *list_objects_filter__init(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options);\n \n /*\n * Lets `filter` decide how to handle the `obj`. If `filter` is NULL, this\n * function behaves as expected if no filter is configured: all objects are\n * included.\n */\n enum list_objects_filter_result list_objects_filter__filter_object(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n \tstruct filter *filter);\n \n-/* Destroys `filter`. Does nothing if `filter` is null. */\n+/*\n+ * Destroys `filter` and finalizes the `omitted` set, if present. Does\n+ * nothing if `filter` is null.\n+ */\n void list_objects_filter__free(struct filter *filter);\n \n #endif /* LIST_OBJECTS_FILTER_H */\n"},{"id":"378208","messageId":"20190627222717.GE54617@comcast.net","threadId":"51217","inReplyTo":"20190627212457.GD54617@comcast.net","subject":"Re: [PATCH v4 10/10] list-objects-filter-options: make parser void","fromName":"Matthew DeVore","fromEmail":"matvore@comcast.net","sentAt":"2019-06-27T22:27:18Z","receivedAt":"2019-06-27T22:27:35Z","isPatch":true,"sender":{"key":"matvore@comcast.net","avatar":"https://gravatar.com/avatar/550c64ce544f82818ad931e244dfb08bbb1febfa6d1ce3cfd65e76215ca0ac8a?d=mp&s=160"},"body":"On Thu, Jun 27, 2019 at 02:24:57PM -0700, Matthew DeVore wrote:\n> Great. I feel much better about the comments and commit messages now. I\n> am about to send a roll-up (v5). Here is the interdiff which catches\n> your comments and Dscho's comment about strbuf_addstr:\n> \n> \n\nForgot to make a string localizable. Add this to the prior interdiff:\n\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex 9e64832a5e..ba1425cb4a 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -96,23 +96,24 @@ static int gently_parse_list_objects_filter(\n }\n \n static const char *RESERVED_NON_WS = \"~`!@#$^&*()[]{}\\\\;'\\\",<>?\";\n \n static int has_reserved_character(\n \tstruct strbuf *sub_spec, struct strbuf *errbuf)\n {\n \tconst char *c = sub_spec->buf;\n \twhile (*c) {\n \t\tif (*c <= ' ' || strchr(RESERVED_NON_WS, *c)) {\n-\t\t\tstrbuf_addf(errbuf,\n-\t\t\t\t \"must escape char in sub-filter-spec: '%c'\",\n-\t\t\t\t *c);\n+\t\t\tstrbuf_addf(\n+\t\t\t\terrbuf,\n+\t\t\t\t_(\"must escape char in sub-filter-spec: '%c'\"),\n+\t\t\t\t*c);\n \t\t\treturn 1;\n \t\t}\n \t\tc++;\n \t}\n \n \treturn 0;\n }\n \n static int parse_combine_subfilter(\n \tstruct list_objects_filter_options *filter_options,\n"},{"id":"378211","messageId":"cover.1561675151.git.matvore@google.com","threadId":"51217","inReplyTo":"20190601003603.90794-1-matvore@google.com","subject":"[PATCH v5 00/10] Filter combination","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-27T22:54:04Z","receivedAt":"2019-06-27T22:54:24Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This applies suggestions made by Jonathan Tan, as well as fixes a\nCoccinelle-breaking error in strbuf usage, and makes an additional string\nlocalizable.\n\nThanks,\n\nMatthew DeVore (10):\n list-objects-filter: encapsulate filter components\n list-objects-filter: put omits set in filter struct\n list-objects-filter-options: always supply *errbuf\n list-objects-filter: implement composite filters\n list-objects-filter-options: move error check up\n list-objects-filter-options: make filter_spec a string_list\n strbuf: give URL-encoding API a char predicate fn\n list-objects-filter-options: allow mult. --filter\n list-objects-filter-options: clean up use of ALLOC_GROW\n list-objects-filter-options: make parser void\n\n Documentation/rev-list-options.txt | 16 ++\n builtin/clone.c | 8 +-\n builtin/fetch.c | 9 +-\n builtin/rev-list.c | 6 +-\n cache.h | 22 ++\n credential-store.c | 9 +-\n fetch-pack.c | 20 +-\n http.c | 6 +-\n list-objects-filter-options.c | 267 ++++++++++++++++++----\n list-objects-filter-options.h | 57 ++++-\n list-objects-filter.c | 335 +++++++++++++++++++++-------\n list-objects-filter.h | 40 ++--\n list-objects.c | 55 ++---\n strbuf.c | 15 +-\n strbuf.h | 7 +-\n t/t5616-partial-clone.sh | 19 ++\n t/t6112-rev-list-filters-objects.sh | 194 +++++++++++++++-\n transport-helper.c | 10 +-\n transport.c | 1 +\n upload-pack.c | 13 +-\n url.c | 6 +\n url.h | 8 +\n 22 files changed, 884 insertions(+), 239 deletions(-)\n\n-- \n2.21.0\n\n"},{"id":"378212","messageId":"4fe4baf34506a956a53688b7a2d2d6665878cdf6.1561675151.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1561675151.git.matvore@google.com","subject":"[PATCH v5 01/10] list-objects-filter: encapsulate filter components","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-27T22:54:05Z","receivedAt":"2019-06-27T22:54:27Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Encapsulate filter_fn, filter_free_fn, and filter_data into their own\nopaque struct.\n\nDue to opaqueness, filter_fn and filter_free_fn can no longer be\naccessed directly by users. Currently, all usages of filter_fn are\nguarded by a necessary check:\n\n\t(obj->flags & NOT_USER_GIVEN) && filter_fn\n\nTake the opportunity to include this check into the new function\nlist_objects_filter__filter_object(), so that we no longer need to write\nthis check at every caller of the filter function.\n\nAlso, the init functions in list-objects-filter.c no longer need to\nconfusingly return the filter constituents in various places (filter_fn\nand filter_free_fn as out parameters, and filter_data as the function's\nreturn value); they can just initialize the \"struct filter\" passed in.\n\nHelped-by: Jeff Hostetler \nHelped-by: Jonathan Tan \nHelped-by: Junio C Hamano \nSigned-off-by: Matthew DeVore \n---\n list-objects-filter.c | 112 ++++++++++++++++++++++++++++--------------\n list-objects-filter.h | 35 ++++++-------\n list-objects.c | 55 +++++++++------------\n 3 files changed, 113 insertions(+), 89 deletions(-)\n\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex 36e1f774bc..e06b82def0 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -19,20 +19,34 @@\n * FILTER_SHOWN_BUT_REVISIT -- we set this bit on tree objects\n * that have been shown, but should be revisited if they appear\n * in the traversal (until we mark it SEEN). This is a way to\n * let us silently de-dup calls to show() in the caller. This\n * is subtly different from the \"revision.h:SHOWN\" and the\n * \"sha1-name.c:ONELINE_SEEN\" bits. And also different from\n * the non-de-dup usage in pack-bitmap.c\n */\n #define FILTER_SHOWN_BUT_REVISIT (1<<21)\n \n+struct filter {\n+\tenum list_objects_filter_result (*filter_object_fn)(\n+\t\tstruct repository *r,\n+\t\tenum list_objects_filter_situation filter_situation,\n+\t\tstruct object *obj,\n+\t\tconst char *pathname,\n+\t\tconst char *filename,\n+\t\tvoid *filter_data);\n+\n+\tvoid (*free_fn)(void *filter_data);\n+\n+\tvoid *filter_data;\n+};\n+\n /*\n * A filter for list-objects to omit ALL blobs from the traversal.\n * And to OPTIONALLY collect a list of the omitted OIDs.\n */\n struct filter_blobs_none_data {\n \tstruct oidset *omits;\n };\n \n static enum list_objects_filter_result filter_blobs_none(\n \tstruct repository *r,\n@@ -60,32 +74,31 @@ static enum list_objects_filter_result filter_blobs_none(\n \tcase LOFS_BLOB:\n \t\tassert(obj->type == OBJ_BLOB);\n \t\tassert((obj->flags & SEEN) == 0);\n \n \t\tif (filter_data->omits)\n \t\t\toidset_insert(filter_data->omits, &obj->oid);\n \t\treturn LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n \t}\n }\n \n-static void *filter_blobs_none__init(\n+static void filter_blobs_none__init(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn)\n+\tstruct filter *filter)\n {\n \tstruct filter_blobs_none_data *d = xcalloc(1, sizeof(*d));\n \td->omits = omitted;\n \n-\t*filter_fn = filter_blobs_none;\n-\t*filter_free_fn = free;\n-\treturn d;\n+\tfilter->filter_data = d;\n+\tfilter->filter_object_fn = filter_blobs_none;\n+\tfilter->free_fn = free;\n }\n \n /*\n * A filter for list-objects to omit ALL trees and blobs from the traversal.\n * Can OPTIONALLY collect a list of the omitted OIDs.\n */\n struct filter_trees_depth_data {\n \tstruct oidset *omits;\n \n \t/*\n@@ -194,35 +207,34 @@ static enum list_objects_filter_result filter_trees_depth(\n }\n \n static void filter_trees_free(void *filter_data) {\n \tstruct filter_trees_depth_data *d = filter_data;\n \tif (!d)\n \t\treturn;\n \toidmap_free(&d->seen_at_depth, 1);\n \tfree(d);\n }\n \n-static void *filter_trees_depth__init(\n+static void filter_trees_depth__init(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn)\n+\tstruct filter *filter)\n {\n \tstruct filter_trees_depth_data *d = xcalloc(1, sizeof(*d));\n \td->omits = omitted;\n \toidmap_init(&d->seen_at_depth, 0);\n \td->exclude_depth = filter_options->tree_exclude_depth;\n \td->current_depth = 0;\n \n-\t*filter_fn = filter_trees_depth;\n-\t*filter_free_fn = filter_trees_free;\n-\treturn d;\n+\tfilter->filter_data = d;\n+\tfilter->filter_object_fn = filter_trees_depth;\n+\tfilter->free_fn = filter_trees_free;\n }\n \n /*\n * A filter for list-objects to omit large blobs.\n * And to OPTIONALLY collect a list of the omitted OIDs.\n */\n struct filter_blobs_limit_data {\n \tstruct oidset *omits;\n \tunsigned long max_bytes;\n };\n@@ -274,33 +286,32 @@ static enum list_objects_filter_result filter_blobs_limit(\n \t\t\toidset_insert(filter_data->omits, &obj->oid);\n \t\treturn LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n \t}\n \n include_it:\n \tif (filter_data->omits)\n \t\toidset_remove(filter_data->omits, &obj->oid);\n \treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n }\n \n-static void *filter_blobs_limit__init(\n+static void filter_blobs_limit__init(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn)\n+\tstruct filter *filter)\n {\n \tstruct filter_blobs_limit_data *d = xcalloc(1, sizeof(*d));\n \td->omits = omitted;\n \td->max_bytes = filter_options->blob_limit_value;\n \n-\t*filter_fn = filter_blobs_limit;\n-\t*filter_free_fn = free;\n-\treturn d;\n+\tfilter->filter_data = d;\n+\tfilter->filter_object_fn = filter_blobs_limit;\n+\tfilter->free_fn = free;\n }\n \n /*\n * A filter driven by a sparse-checkout specification to only\n * include blobs that a sparse checkout would populate.\n *\n * The sparse-checkout spec can be loaded from a blob with the\n * given OID or from a local pathname. We allow an OID because\n * the repo may be bare or we may be doing the filtering on the\n * server.\n@@ -449,71 +460,98 @@ static enum list_objects_filter_result filter_sparse(\n }\n \n \n static void filter_sparse_free(void *filter_data)\n {\n \tstruct filter_sparse_data *d = filter_data;\n \tfree(d->array_frame);\n \tfree(d);\n }\n \n-static void *filter_sparse_oid__init(\n+static void filter_sparse_oid__init(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn)\n+\tstruct filter *filter)\n {\n \tstruct filter_sparse_data *d = xcalloc(1, sizeof(*d));\n \td->omits = omitted;\n \tif (add_excludes_from_blob_to_list(filter_options->sparse_oid_value,\n \t\t\t\t\t NULL, 0, &d->el) < 0)\n \t\tdie(\"could not load filter specification\");\n \n \tALLOC_GROW(d->array_frame, d->nr + 1, d->alloc);\n \td->array_frame[d->nr].defval = 0; /* default to include */\n \td->array_frame[d->nr].child_prov_omit = 0;\n \td->nr++;\n \n-\t*filter_fn = filter_sparse;\n-\t*filter_free_fn = filter_sparse_free;\n-\treturn d;\n+\tfilter->filter_data = d;\n+\tfilter->filter_object_fn = filter_sparse;\n+\tfilter->free_fn = filter_sparse_free;\n }\n \n-typedef void *(*filter_init_fn)(\n+typedef void (*filter_init_fn)(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn);\n+\tstruct filter *filter);\n \n /*\n * Must match \"enum list_objects_filter_choice\".\n */\n static filter_init_fn s_filters[] = {\n \tNULL,\n \tfilter_blobs_none__init,\n \tfilter_blobs_limit__init,\n \tfilter_trees_depth__init,\n \tfilter_sparse_oid__init,\n };\n \n-void *list_objects_filter__init(\n+struct filter *list_objects_filter__init(\n \tstruct oidset *omitted,\n-\tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn)\n+\tstruct list_objects_filter_options *filter_options)\n {\n+\tstruct filter *filter;\n \tfilter_init_fn init_fn;\n \n \tassert((sizeof(s_filters) / sizeof(s_filters[0])) == LOFC__COUNT);\n \n \tif (filter_options->choice >= LOFC__COUNT)\n \t\tBUG(\"invalid list-objects filter choice: %d\",\n \t\t filter_options->choice);\n \n \tinit_fn = s_filters[filter_options->choice];\n-\tif (init_fn)\n-\t\treturn init_fn(omitted, filter_options,\n-\t\t\t filter_fn, filter_free_fn);\n-\t*filter_fn = NULL;\n-\t*filter_free_fn = NULL;\n-\treturn NULL;\n+\tif (!init_fn)\n+\t\treturn NULL;\n+\n+\tfilter = xcalloc(1, sizeof(*filter));\n+\tinit_fn(omitted, filter_options, filter);\n+\treturn filter;\n+}\n+\n+enum list_objects_filter_result list_objects_filter__filter_object(\n+\tstruct repository *r,\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tstruct filter *filter)\n+{\n+\tif (filter && (obj->flags & NOT_USER_GIVEN))\n+\t\treturn filter->filter_object_fn(r, filter_situation, obj,\n+\t\t\t\t\t\tpathname, filename,\n+\t\t\t\t\t\tfilter->filter_data);\n+\t/*\n+\t * No filter is active or user gave object explicitly. In this case,\n+\t * always show the object (except when LOFS_END_TREE, since this tree\n+\t * had already been shown when LOFS_BEGIN_TREE).\n+\t */\n+\tif (filter_situation == LOFS_END_TREE)\n+\t\treturn 0;\n+\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+}\n+\n+void list_objects_filter__free(struct filter *filter)\n+{\n+\tif (!filter)\n+\t\treturn;\n+\tfilter->free_fn(filter->filter_data);\n+\tfree(filter);\n }\ndiff --git a/list-objects-filter.h b/list-objects-filter.h\nindex 1d45a4ad57..6908954266 100644\n--- a/list-objects-filter.h\n+++ b/list-objects-filter.h\n@@ -53,37 +53,34 @@ enum list_objects_filter_result {\n \tLOFR_DO_SHOW = 1<<1,\n \tLOFR_SKIP_TREE = 1<<2,\n };\n \n enum list_objects_filter_situation {\n \tLOFS_BEGIN_TREE,\n \tLOFS_END_TREE,\n \tLOFS_BLOB\n };\n \n-typedef enum list_objects_filter_result (*filter_object_fn)(\n+struct filter;\n+\n+/* Constructor for the set of defined list-objects filters. */\n+struct filter *list_objects_filter__init(\n+\tstruct oidset *omitted,\n+\tstruct list_objects_filter_options *filter_options);\n+\n+/*\n+ * Lets `filter` decide how to handle the `obj`. If `filter` is NULL, this\n+ * function behaves as expected if no filter is configured: all objects are\n+ * included.\n+ */\n+enum list_objects_filter_result list_objects_filter__filter_object(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n-\tvoid *filter_data);\n-\n-typedef void (*filter_free_fn)(void *filter_data);\n+\tstruct filter *filter);\n \n-/*\n- * Constructor for the set of defined list-objects filters.\n- * Returns a generic \"void *filter_data\".\n- *\n- * The returned \"filter_fn\" will be used by traverse_commit_list()\n- * to filter the results.\n- *\n- * The returned \"filter_free_fn\" is a destructor for the\n- * filter_data.\n- */\n-void *list_objects_filter__init(\n-\tstruct oidset *omitted,\n-\tstruct list_objects_filter_options *filter_options,\n-\tfilter_object_fn *filter_fn,\n-\tfilter_free_fn *filter_free_fn);\n+/* Destroys `filter`. Does nothing if `filter` is null. */\n+void list_objects_filter__free(struct filter *filter);\n \n #endif /* LIST_OBJECTS_FILTER_H */\ndiff --git a/list-objects.c b/list-objects.c\nindex b5651ddd5b..9307d91fb3 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -11,32 +11,31 @@\n #include \"list-objects-filter-options.h\"\n #include \"packfile.h\"\n #include \"object-store.h\"\n #include \"trace.h\"\n \n struct traversal_context {\n \tstruct rev_info *revs;\n \tshow_object_fn show_object;\n \tshow_commit_fn show_commit;\n \tvoid *show_data;\n-\tfilter_object_fn filter_fn;\n-\tvoid *filter_data;\n+\tstruct filter *filter;\n };\n \n static void process_blob(struct traversal_context *ctx,\n \t\t\t struct blob *blob,\n \t\t\t struct strbuf *path,\n \t\t\t const char *name)\n {\n \tstruct object *obj = &blob->object;\n \tsize_t pathlen;\n-\tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\tenum list_objects_filter_result r;\n \n \tif (!ctx->revs->blob_objects)\n \t\treturn;\n \tif (!obj)\n \t\tdie(\"bad blob object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n \n \t/*\n \t * Pre-filter known-missing objects when explicitly requested.\n@@ -47,25 +46,24 @@ static void process_blob(struct traversal_context *ctx,\n \t * may cause the actual filter to report an incomplete list\n \t * of missing objects.\n \t */\n \tif (ctx->revs->exclude_promisor_objects &&\n \t !has_object_file(&obj->oid) &&\n \t is_promisor_object(&obj->oid))\n \t\treturn;\n \n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n-\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n-\t\tr = ctx->filter_fn(ctx->revs->repo,\n-\t\t\t\t LOFS_BLOB, obj,\n-\t\t\t\t path->buf, &path->buf[pathlen],\n-\t\t\t\t ctx->filter_data);\n+\tr = list_objects_filter__filter_object(ctx->revs->repo,\n+\t\t\t\t\t LOFS_BLOB, obj,\n+\t\t\t\t\t path->buf, &path->buf[pathlen],\n+\t\t\t\t\t ctx->filter);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n \t\tctx->show_object(obj, path->buf, ctx->show_data);\n \tstrbuf_setlen(path, pathlen);\n }\n \n /*\n * Processing a gitlink entry currently does nothing, since\n * we do not recurse into the subproject.\n@@ -150,21 +148,21 @@ static void process_tree_contents(struct traversal_context *ctx,\n }\n \n static void process_tree(struct traversal_context *ctx,\n \t\t\t struct tree *tree,\n \t\t\t struct strbuf *base,\n \t\t\t const char *name)\n {\n \tstruct object *obj = &tree->object;\n \tstruct rev_info *revs = ctx->revs;\n \tint baselen = base->len;\n-\tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\tenum list_objects_filter_result r;\n \tint failed_parse;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n \tif (!obj)\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n \n \tfailed_parse = parse_tree_gently(tree, 1);\n@@ -179,47 +177,44 @@ static void process_tree(struct traversal_context *ctx,\n \t\t */\n \t\tif (revs->exclude_promisor_objects &&\n \t\t is_promisor_object(&obj->oid))\n \t\t\treturn;\n \n \t\tif (!revs->do_not_die_on_missing_tree)\n \t\t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n \t}\n \n \tstrbuf_addstr(base, name);\n-\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn)\n-\t\tr = ctx->filter_fn(ctx->revs->repo,\n-\t\t\t\t LOFS_BEGIN_TREE, obj,\n-\t\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t\t ctx->filter_data);\n+\tr = list_objects_filter__filter_object(ctx->revs->repo,\n+\t\t\t\t\t LOFS_BEGIN_TREE, obj,\n+\t\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t\t ctx->filter);\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n \t\tctx->show_object(obj, base->buf, ctx->show_data);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n \tif (r & LOFR_SKIP_TREE)\n \t\ttrace_printf(\"Skipping contents of tree %s...\\n\", base->buf);\n \telse if (!failed_parse)\n \t\tprocess_tree_contents(ctx, tree, base);\n \n-\tif ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) {\n-\t\tr = ctx->filter_fn(ctx->revs->repo,\n-\t\t\t\t LOFS_END_TREE, obj,\n-\t\t\t\t base->buf, &base->buf[baselen],\n-\t\t\t\t ctx->filter_data);\n-\t\tif (r & LOFR_MARK_SEEN)\n-\t\t\tobj->flags |= SEEN;\n-\t\tif (r & LOFR_DO_SHOW)\n-\t\t\tctx->show_object(obj, base->buf, ctx->show_data);\n-\t}\n+\tr = list_objects_filter__filter_object(ctx->revs->repo,\n+\t\t\t\t\t LOFS_END_TREE, obj,\n+\t\t\t\t\t base->buf, &base->buf[baselen],\n+\t\t\t\t\t ctx->filter);\n+\tif (r & LOFR_MARK_SEEN)\n+\t\tobj->flags |= SEEN;\n+\tif (r & LOFR_DO_SHOW)\n+\t\tctx->show_object(obj, base->buf, ctx->show_data);\n \n \tstrbuf_setlen(base, baselen);\n \tfree_tree_buffer(tree);\n }\n \n static void mark_edge_parents_uninteresting(struct commit *commit,\n \t\t\t\t\t struct rev_info *revs,\n \t\t\t\t\t show_edge_fn show_edge)\n {\n \tstruct commit_list *parents;\n@@ -395,38 +390,32 @@ static void do_traverse(struct traversal_context *ctx)\n void traverse_commit_list(struct rev_info *revs,\n \t\t\t show_commit_fn show_commit,\n \t\t\t show_object_fn show_object,\n \t\t\t void *show_data)\n {\n \tstruct traversal_context ctx;\n \tctx.revs = revs;\n \tctx.show_commit = show_commit;\n \tctx.show_object = show_object;\n \tctx.show_data = show_data;\n-\tctx.filter_fn = NULL;\n-\tctx.filter_data = NULL;\n+\tctx.filter = NULL;\n \tdo_traverse(&ctx);\n }\n \n void traverse_commit_list_filtered(\n \tstruct list_objects_filter_options *filter_options,\n \tstruct rev_info *revs,\n \tshow_commit_fn show_commit,\n \tshow_object_fn show_object,\n \tvoid *show_data,\n \tstruct oidset *omitted)\n {\n \tstruct traversal_context ctx;\n-\tfilter_free_fn filter_free_fn = NULL;\n \n \tctx.revs = revs;\n \tctx.show_object = show_object;\n \tctx.show_commit = show_commit;\n \tctx.show_data = show_data;\n-\tctx.filter_fn = NULL;\n-\n-\tctx.filter_data = list_objects_filter__init(omitted, filter_options,\n-\t\t\t\t\t\t &ctx.filter_fn, &filter_free_fn);\n+\tctx.filter = list_objects_filter__init(omitted, filter_options);\n \tdo_traverse(&ctx);\n-\tif (ctx.filter_data && filter_free_fn)\n-\t\tfilter_free_fn(ctx.filter_data);\n+\tlist_objects_filter__free(ctx.filter);\n }\n-- \n2.21.0\n\n"},{"id":"378213","messageId":"14400b99b888d6181ef95220de0b372216e6ef1c.1561675151.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1561675151.git.matvore@google.com","subject":"[PATCH v5 03/10] list-objects-filter-options: always supply *errbuf","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-27T22:54:07Z","receivedAt":"2019-06-27T22:54:32Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Making errbuf an optional argument complicates error reporting. Fix this\nby making all callers supply an errbuf, even if they may ignore it. This\nwill be important in follow-up patches where the filter-spec parsing has\nmore pitfalls and possible errors.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects-filter-options.c | 21 ++++++++-------------\n 1 file changed, 8 insertions(+), 13 deletions(-)\n\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex 1cb20c659c..7c3e397d29 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -23,47 +23,40 @@\n * convenience of the current command.\n */\n static int gently_parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg,\n \tstruct strbuf *errbuf)\n {\n \tconst char *v0;\n \n \tif (filter_options->choice) {\n-\t\tif (errbuf) {\n-\t\t\tstrbuf_addstr(\n-\t\t\t\terrbuf,\n-\t\t\t\t_(\"multiple filter-specs cannot be combined\"));\n-\t\t}\n+\t\tstrbuf_addstr(\n+\t\t\terrbuf, _(\"multiple filter-specs cannot be combined\"));\n \t\treturn 1;\n \t}\n \n \tfilter_options->filter_spec = strdup(arg);\n \n \tif (!strcmp(arg, \"blob:none\")) {\n \t\tfilter_options->choice = LOFC_BLOB_NONE;\n \t\treturn 0;\n \n \t} else if (skip_prefix(arg, \"blob:limit=\", &v0)) {\n \t\tif (git_parse_ulong(v0, &filter_options->blob_limit_value)) {\n \t\t\tfilter_options->choice = LOFC_BLOB_LIMIT;\n \t\t\treturn 0;\n \t\t}\n \n \t} else if (skip_prefix(arg, \"tree:\", &v0)) {\n \t\tif (!git_parse_ulong(v0, &filter_options->tree_exclude_depth)) {\n-\t\t\tif (errbuf) {\n-\t\t\t\tstrbuf_addstr(\n-\t\t\t\t\terrbuf,\n-\t\t\t\t\t_(\"expected 'tree:'\"));\n-\t\t\t}\n+\t\t\tstrbuf_addstr(errbuf, _(\"expected 'tree:'\"));\n \t\t\treturn 1;\n \t\t}\n \t\tfilter_options->choice = LOFC_TREE_DEPTH;\n \t\treturn 0;\n \n \t} else if (skip_prefix(arg, \"sparse:oid=\", &v0)) {\n \t\tstruct object_context oc;\n \t\tstruct object_id sparse_oid;\n \n \t\t/*\n@@ -83,22 +76,21 @@ static int gently_parse_list_objects_filter(\n \t\t\t\terrbuf,\n \t\t\t\t_(\"sparse:path filters support has been dropped\"));\n \t\t}\n \t\treturn 1;\n \t}\n \t/*\n \t * Please update _git_fetch() in git-completion.bash when you\n \t * add new filters\n \t */\n \n-\tif (errbuf)\n-\t\tstrbuf_addf(errbuf, _(\"invalid filter-spec '%s'\"), arg);\n+\tstrbuf_addf(errbuf, _(\"invalid filter-spec '%s'\"), arg);\n \n \tmemset(filter_options, 0, sizeof(*filter_options));\n \treturn 1;\n }\n \n int parse_list_objects_filter(struct list_objects_filter_options *filter_options,\n \t\t\t const char *arg)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tif (gently_parse_list_objects_filter(filter_options, arg, &buf))\n@@ -168,19 +160,22 @@ void partial_clone_register(\n \t */\n \tcore_partial_clone_filter_default =\n \t\txstrdup(filter_options->filter_spec);\n \tgit_config_set(\"core.partialclonefilter\",\n \t\t core_partial_clone_filter_default);\n }\n \n void partial_clone_get_default_filter_spec(\n \tstruct list_objects_filter_options *filter_options)\n {\n+\tstruct strbuf errbuf = STRBUF_INIT;\n+\n \t/*\n \t * Parse default value, but silently ignore it if it is invalid.\n \t */\n \tif (!core_partial_clone_filter_default)\n \t\treturn;\n \tgently_parse_list_objects_filter(filter_options,\n \t\t\t\t\t core_partial_clone_filter_default,\n-\t\t\t\t\t NULL);\n+\t\t\t\t\t &errbuf);\n+\tstrbuf_release(&errbuf);\n }\n-- \n2.21.0\n\n"},{"id":"378214","messageId":"62441a2214528b114341796673c6f8fda14d24b8.1561675151.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1561675151.git.matvore@google.com","subject":"[PATCH v5 02/10] list-objects-filter: put omits set in filter struct","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-27T22:54:06Z","receivedAt":"2019-06-27T22:54:33Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"The oidset *omits pointer must be accessed by the combine filter in a\ntype-agnostic way once the graph traversal is over. Store that pointer\nin the general `filter` struct. This will be used in a follow-up patch\nto implement the combine filter.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects-filter.c | 68 +++++++++++++++++--------------------------\n 1 file changed, 26 insertions(+), 42 deletions(-)\n\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex e06b82def0..3b4b6764ca 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -26,88 +26,76 @@\n */\n #define FILTER_SHOWN_BUT_REVISIT (1<<21)\n \n struct filter {\n \tenum list_objects_filter_result (*filter_object_fn)(\n \t\tstruct repository *r,\n \t\tenum list_objects_filter_situation filter_situation,\n \t\tstruct object *obj,\n \t\tconst char *pathname,\n \t\tconst char *filename,\n+\t\tstruct oidset *omits,\n \t\tvoid *filter_data);\n \n \tvoid (*free_fn)(void *filter_data);\n \n \tvoid *filter_data;\n-};\n \n-/*\n- * A filter for list-objects to omit ALL blobs from the traversal.\n- * And to OPTIONALLY collect a list of the omitted OIDs.\n- */\n-struct filter_blobs_none_data {\n+\t/* If non-NULL, the filter collects a list of the omitted OIDs here. */\n \tstruct oidset *omits;\n };\n \n static enum list_objects_filter_result filter_blobs_none(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n+\tstruct oidset *omits,\n \tvoid *filter_data_)\n {\n-\tstruct filter_blobs_none_data *filter_data = filter_data_;\n-\n \tswitch (filter_situation) {\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\t/* always include all tree objects */\n \t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \n \tcase LOFS_END_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\treturn LOFR_ZERO;\n \n \tcase LOFS_BLOB:\n \t\tassert(obj->type == OBJ_BLOB);\n \t\tassert((obj->flags & SEEN) == 0);\n \n-\t\tif (filter_data->omits)\n-\t\t\toidset_insert(filter_data->omits, &obj->oid);\n+\t\tif (omits)\n+\t\t\toidset_insert(omits, &obj->oid);\n \t\treturn LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n \t}\n }\n \n static void filter_blobs_none__init(\n-\tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter)\n {\n-\tstruct filter_blobs_none_data *d = xcalloc(1, sizeof(*d));\n-\td->omits = omitted;\n-\n-\tfilter->filter_data = d;\n \tfilter->filter_object_fn = filter_blobs_none;\n \tfilter->free_fn = free;\n }\n \n /*\n * A filter for list-objects to omit ALL trees and blobs from the traversal.\n * Can OPTIONALLY collect a list of the omitted OIDs.\n */\n struct filter_trees_depth_data {\n-\tstruct oidset *omits;\n-\n \t/*\n \t * Maps trees to the minimum depth at which they were seen. It is not\n \t * necessary to re-traverse a tree at deeper or equal depths than it has\n \t * already been traversed.\n \t *\n \t * We can't use LOFR_MARK_SEEN for tree objects since this will prevent\n \t * it from being traversed at shallower depths.\n \t */\n \tstruct oidmap seen_at_depth;\n \n@@ -116,38 +104,39 @@ struct filter_trees_depth_data {\n };\n \n struct seen_map_entry {\n \tstruct oidmap_entry base;\n \tsize_t depth;\n };\n \n /* Returns 1 if the oid was in the omits set before it was invoked. */\n static int filter_trees_update_omits(\n \tstruct object *obj,\n-\tstruct filter_trees_depth_data *filter_data,\n+\tstruct oidset *omits,\n \tint include_it)\n {\n-\tif (!filter_data->omits)\n+\tif (!omits)\n \t\treturn 0;\n \n \tif (include_it)\n-\t\treturn oidset_remove(filter_data->omits, &obj->oid);\n+\t\treturn oidset_remove(omits, &obj->oid);\n \telse\n-\t\treturn oidset_insert(filter_data->omits, &obj->oid);\n+\t\treturn oidset_insert(omits, &obj->oid);\n }\n \n static enum list_objects_filter_result filter_trees_depth(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n+\tstruct oidset *omits,\n \tvoid *filter_data_)\n {\n \tstruct filter_trees_depth_data *filter_data = filter_data_;\n \tstruct seen_map_entry *seen_info;\n \tint include_it = filter_data->current_depth <\n \t\tfilter_data->exclude_depth;\n \tint filter_res;\n \tint already_seen;\n \n \t/*\n@@ -158,47 +147,47 @@ static enum list_objects_filter_result filter_trees_depth(\n \tswitch (filter_situation) {\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n \tcase LOFS_END_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\tfilter_data->current_depth--;\n \t\treturn LOFR_ZERO;\n \n \tcase LOFS_BLOB:\n-\t\tfilter_trees_update_omits(obj, filter_data, include_it);\n+\t\tfilter_trees_update_omits(obj, omits, include_it);\n \t\treturn include_it ? LOFR_MARK_SEEN | LOFR_DO_SHOW : LOFR_ZERO;\n \n \tcase LOFS_BEGIN_TREE:\n \t\tseen_info = oidmap_get(\n \t\t\t&filter_data->seen_at_depth, &obj->oid);\n \t\tif (!seen_info) {\n \t\t\tseen_info = xcalloc(1, sizeof(*seen_info));\n \t\t\toidcpy(&seen_info->base.oid, &obj->oid);\n \t\t\tseen_info->depth = filter_data->current_depth;\n \t\t\toidmap_put(&filter_data->seen_at_depth, seen_info);\n \t\t\talready_seen = 0;\n \t\t} else {\n \t\t\talready_seen =\n \t\t\t\tfilter_data->current_depth >= seen_info->depth;\n \t\t}\n \n \t\tif (already_seen) {\n \t\t\tfilter_res = LOFR_SKIP_TREE;\n \t\t} else {\n \t\t\tint been_omitted = filter_trees_update_omits(\n-\t\t\t\tobj, filter_data, include_it);\n+\t\t\t\tobj, omits, include_it);\n \t\t\tseen_info->depth = filter_data->current_depth;\n \n \t\t\tif (include_it)\n \t\t\t\tfilter_res = LOFR_DO_SHOW;\n-\t\t\telse if (filter_data->omits && !been_omitted)\n+\t\t\telse if (omits && !been_omitted)\n \t\t\t\t/*\n \t\t\t\t * Must update omit information of children\n \t\t\t\t * recursively; they have not been omitted yet.\n \t\t\t\t */\n \t\t\t\tfilter_res = LOFR_ZERO;\n \t\t\telse\n \t\t\t\tfilter_res = LOFR_SKIP_TREE;\n \t\t}\n \n \t\tfilter_data->current_depth++;\n@@ -208,50 +197,48 @@ static enum list_objects_filter_result filter_trees_depth(\n \n static void filter_trees_free(void *filter_data) {\n \tstruct filter_trees_depth_data *d = filter_data;\n \tif (!d)\n \t\treturn;\n \toidmap_free(&d->seen_at_depth, 1);\n \tfree(d);\n }\n \n static void filter_trees_depth__init(\n-\tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter)\n {\n \tstruct filter_trees_depth_data *d = xcalloc(1, sizeof(*d));\n-\td->omits = omitted;\n \toidmap_init(&d->seen_at_depth, 0);\n \td->exclude_depth = filter_options->tree_exclude_depth;\n \td->current_depth = 0;\n \n \tfilter->filter_data = d;\n \tfilter->filter_object_fn = filter_trees_depth;\n \tfilter->free_fn = filter_trees_free;\n }\n \n /*\n * A filter for list-objects to omit large blobs.\n * And to OPTIONALLY collect a list of the omitted OIDs.\n */\n struct filter_blobs_limit_data {\n-\tstruct oidset *omits;\n \tunsigned long max_bytes;\n };\n \n static enum list_objects_filter_result filter_blobs_limit(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n+\tstruct oidset *omits,\n \tvoid *filter_data_)\n {\n \tstruct filter_blobs_limit_data *filter_data = filter_data_;\n \tunsigned long object_length;\n \tenum object_type t;\n \n \tswitch (filter_situation) {\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n@@ -275,38 +262,36 @@ static enum list_objects_filter_result filter_blobs_limit(\n \t\t\t * apply the size filter criteria. Be conservative\n \t\t\t * and force show it (and let the caller deal with\n \t\t\t * the ambiguity).\n \t\t\t */\n \t\t\tgoto include_it;\n \t\t}\n \n \t\tif (object_length < filter_data->max_bytes)\n \t\t\tgoto include_it;\n \n-\t\tif (filter_data->omits)\n-\t\t\toidset_insert(filter_data->omits, &obj->oid);\n+\t\tif (omits)\n+\t\t\toidset_insert(omits, &obj->oid);\n \t\treturn LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */\n \t}\n \n include_it:\n-\tif (filter_data->omits)\n-\t\toidset_remove(filter_data->omits, &obj->oid);\n+\tif (omits)\n+\t\toidset_remove(omits, &obj->oid);\n \treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n }\n \n static void filter_blobs_limit__init(\n-\tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter)\n {\n \tstruct filter_blobs_limit_data *d = xcalloc(1, sizeof(*d));\n-\td->omits = omitted;\n \td->max_bytes = filter_options->blob_limit_value;\n \n \tfilter->filter_data = d;\n \tfilter->filter_object_fn = filter_blobs_limit;\n \tfilter->free_fn = free;\n }\n \n /*\n * A filter driven by a sparse-checkout specification to only\n * include blobs that a sparse checkout would populate.\n@@ -330,33 +315,33 @@ struct frame {\n \t * omitted objects.\n \t *\n \t * 0 if everything (recursively) contained in this directory\n \t * has been explicitly included (SHOWN) in the result and\n \t * the directory may be short-cut later in the traversal.\n \t */\n \tunsigned child_prov_omit : 1;\n };\n \n struct filter_sparse_data {\n-\tstruct oidset *omits;\n \tstruct exclude_list el;\n \n \tsize_t nr, alloc;\n \tstruct frame *array_frame;\n };\n \n static enum list_objects_filter_result filter_sparse(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n+\tstruct oidset *omits,\n \tvoid *filter_data_)\n {\n \tstruct filter_sparse_data *filter_data = filter_data_;\n \tint val, dtype;\n \tstruct frame *frame;\n \n \tswitch (filter_situation) {\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n@@ -424,79 +409,76 @@ static enum list_objects_filter_result filter_sparse(\n \n \t\tframe = &filter_data->array_frame[filter_data->nr - 1];\n \n \t\tdtype = DT_REG;\n \t\tval = is_excluded_from_list(pathname, strlen(pathname),\n \t\t\t\t\t filename, &dtype, &filter_data->el,\n \t\t\t\t\t r->index);\n \t\tif (val < 0)\n \t\t\tval = frame->defval;\n \t\tif (val > 0) {\n-\t\t\tif (filter_data->omits)\n-\t\t\t\toidset_remove(filter_data->omits, &obj->oid);\n+\t\t\tif (omits)\n+\t\t\t\toidset_remove(omits, &obj->oid);\n \t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n \t\t}\n \n \t\t/*\n \t\t * Provisionally omit it. We've already established that\n \t\t * this pathname is not in the sparse-checkout specification\n \t\t * with the CURRENT pathname, so we *WANT* to omit this blob.\n \t\t *\n \t\t * However, a pathname elsewhere in the tree may also\n \t\t * reference this same blob, so we cannot reject it yet.\n \t\t * Leave the LOFR_ bits unset so that if the blob appears\n \t\t * again in the traversal, we will be asked again.\n \t\t */\n-\t\tif (filter_data->omits)\n-\t\t\toidset_insert(filter_data->omits, &obj->oid);\n+\t\tif (omits)\n+\t\t\toidset_insert(omits, &obj->oid);\n \n \t\t/*\n \t\t * Remember that at least 1 blob in this tree was\n \t\t * provisionally omitted. This prevents us from short\n \t\t * cutting the tree in future iterations.\n \t\t */\n \t\tframe->child_prov_omit = 1;\n \t\treturn LOFR_ZERO;\n \t}\n }\n \n \n static void filter_sparse_free(void *filter_data)\n {\n \tstruct filter_sparse_data *d = filter_data;\n \tfree(d->array_frame);\n \tfree(d);\n }\n \n static void filter_sparse_oid__init(\n-\tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter)\n {\n \tstruct filter_sparse_data *d = xcalloc(1, sizeof(*d));\n-\td->omits = omitted;\n \tif (add_excludes_from_blob_to_list(filter_options->sparse_oid_value,\n \t\t\t\t\t NULL, 0, &d->el) < 0)\n \t\tdie(\"could not load filter specification\");\n \n \tALLOC_GROW(d->array_frame, d->nr + 1, d->alloc);\n \td->array_frame[d->nr].defval = 0; /* default to include */\n \td->array_frame[d->nr].child_prov_omit = 0;\n \td->nr++;\n \n \tfilter->filter_data = d;\n \tfilter->filter_object_fn = filter_sparse;\n \tfilter->free_fn = filter_sparse_free;\n }\n \n typedef void (*filter_init_fn)(\n-\tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter);\n \n /*\n * Must match \"enum list_objects_filter_choice\".\n */\n static filter_init_fn s_filters[] = {\n \tNULL,\n \tfilter_blobs_none__init,\n \tfilter_blobs_limit__init,\n@@ -515,35 +497,37 @@ struct filter *list_objects_filter__init(\n \n \tif (filter_options->choice >= LOFC__COUNT)\n \t\tBUG(\"invalid list-objects filter choice: %d\",\n \t\t filter_options->choice);\n \n \tinit_fn = s_filters[filter_options->choice];\n \tif (!init_fn)\n \t\treturn NULL;\n \n \tfilter = xcalloc(1, sizeof(*filter));\n-\tinit_fn(omitted, filter_options, filter);\n+\tfilter->omits = omitted;\n+\tinit_fn(filter_options, filter);\n \treturn filter;\n }\n \n enum list_objects_filter_result list_objects_filter__filter_object(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n \tstruct filter *filter)\n {\n \tif (filter && (obj->flags & NOT_USER_GIVEN))\n \t\treturn filter->filter_object_fn(r, filter_situation, obj,\n \t\t\t\t\t\tpathname, filename,\n+\t\t\t\t\t\tfilter->omits,\n \t\t\t\t\t\tfilter->filter_data);\n \t/*\n \t * No filter is active or user gave object explicitly. In this case,\n \t * always show the object (except when LOFS_END_TREE, since this tree\n \t * had already been shown when LOFS_BEGIN_TREE).\n \t */\n \tif (filter_situation == LOFS_END_TREE)\n \t\treturn 0;\n \treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n }\n-- \n2.21.0\n\n"},{"id":"378215","messageId":"17b525b6104147658504f4e7a9044e3dceee5a99.1561675151.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1561675151.git.matvore@google.com","subject":"[PATCH v5 04/10] list-objects-filter: implement composite filters","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-27T22:54:08Z","receivedAt":"2019-06-27T22:54:35Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Allow combining filters such that only objects accepted by all filters\nare shown. The motivation for this is to allow getting directory\nlistings without also fetching blobs. This can be done by combining\nblob:none with tree:. There are massive repositories that have\nlarger-than-expected trees - even if you include only a single commit.\n\nA combined filter supports any number of subfilters, and is written in\nthe following form:\n\n\tcombine:++\n\nCertain non-alphanumeric characters in each filter must be\nURL-encoded.\n\nFor now, combined filters must be specified in this form. In a\nsubsequent commit, rev-list will support multiple --filter arguments\nwhich will have the same effect as specifying one filter argument\nstarting with \"combine:\". The documentation will be updated in that\ncommit, as the URL-encoding scheme is in general not meant to be used\ndirectly by the user, and it is better to describe the URL-encoding\nfeature in terms of the repeated flag.\n\nHelped-by: Emily Shaffer \nHelped-by: Jeff Hostetler \nHelped-by: Johannes Schindelin \nHelped-by: Jonathan Tan \nHelped-by: Junio C Hamano \nSigned-off-by: Matthew DeVore \n---\n list-objects-filter-options.c | 106 +++++++++++++++++-\n list-objects-filter-options.h | 17 ++-\n list-objects-filter.c | 161 ++++++++++++++++++++++++++++\n list-objects-filter.h | 13 ++-\n t/t6112-rev-list-filters-objects.sh | 151 +++++++++++++++++++++++++-\n url.c | 6 ++\n url.h | 8 ++\n 7 files changed, 454 insertions(+), 8 deletions(-)\n\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex 7c3e397d29..75d0236ee2 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -1,18 +1,24 @@\n #include \"cache.h\"\n #include \"commit.h\"\n #include \"config.h\"\n #include \"revision.h\"\n #include \"argv-array.h\"\n #include \"list-objects.h\"\n #include \"list-objects-filter.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"url.h\"\n+\n+static int parse_combine_filter(\n+\tstruct list_objects_filter_options *filter_options,\n+\tconst char *arg,\n+\tstruct strbuf *errbuf);\n \n /*\n * Parse value of the argument to the \"filter\" keyword.\n * On the command line this looks like:\n * --filter=\n * and in the pack protocol as:\n * \"filter\" SP \n *\n * The filter keyword will be used by many commands.\n * See Documentation/rev-list-options.txt for allowed values for .\n@@ -28,22 +34,20 @@ static int gently_parse_list_objects_filter(\n \tstruct strbuf *errbuf)\n {\n \tconst char *v0;\n \n \tif (filter_options->choice) {\n \t\tstrbuf_addstr(\n \t\t\terrbuf, _(\"multiple filter-specs cannot be combined\"));\n \t\treturn 1;\n \t}\n \n-\tfilter_options->filter_spec = strdup(arg);\n-\n \tif (!strcmp(arg, \"blob:none\")) {\n \t\tfilter_options->choice = LOFC_BLOB_NONE;\n \t\treturn 0;\n \n \t} else if (skip_prefix(arg, \"blob:limit=\", &v0)) {\n \t\tif (git_parse_ulong(v0, &filter_options->blob_limit_value)) {\n \t\t\tfilter_options->choice = LOFC_BLOB_LIMIT;\n \t\t\treturn 0;\n \t\t}\n \n@@ -70,36 +74,125 @@ static int gently_parse_list_objects_filter(\n \t\tfilter_options->choice = LOFC_SPARSE_OID;\n \t\treturn 0;\n \n \t} else if (skip_prefix(arg, \"sparse:path=\", &v0)) {\n \t\tif (errbuf) {\n \t\t\tstrbuf_addstr(\n \t\t\t\terrbuf,\n \t\t\t\t_(\"sparse:path filters support has been dropped\"));\n \t\t}\n \t\treturn 1;\n+\n+\t} else if (skip_prefix(arg, \"combine:\", &v0)) {\n+\t\treturn parse_combine_filter(filter_options, v0, errbuf);\n+\n \t}\n \t/*\n \t * Please update _git_fetch() in git-completion.bash when you\n \t * add new filters\n \t */\n \n \tstrbuf_addf(errbuf, _(\"invalid filter-spec '%s'\"), arg);\n \n \tmemset(filter_options, 0, sizeof(*filter_options));\n \treturn 1;\n }\n \n+static const char *RESERVED_NON_WS = \"~`!@#$^&*()[]{}\\\\;'\\\",<>?\";\n+\n+static int has_reserved_character(\n+\tstruct strbuf *sub_spec, struct strbuf *errbuf)\n+{\n+\tconst char *c = sub_spec->buf;\n+\twhile (*c) {\n+\t\tif (*c <= ' ' || strchr(RESERVED_NON_WS, *c)) {\n+\t\t\tstrbuf_addf(\n+\t\t\t\terrbuf,\n+\t\t\t\t_(\"must escape char in sub-filter-spec: '%c'\"),\n+\t\t\t\t*c);\n+\t\t\treturn 1;\n+\t\t}\n+\t\tc++;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int parse_combine_subfilter(\n+\tstruct list_objects_filter_options *filter_options,\n+\tstruct strbuf *subspec,\n+\tstruct strbuf *errbuf)\n+{\n+\tsize_t new_index = filter_options->sub_nr++;\n+\tchar *decoded;\n+\tint result;\n+\n+\tALLOC_GROW(filter_options->sub, filter_options->sub_nr,\n+\t\t filter_options->sub_alloc);\n+\tmemset(&filter_options->sub[new_index], 0,\n+\t sizeof(*filter_options->sub));\n+\n+\tdecoded = url_percent_decode(subspec->buf);\n+\n+\tresult = has_reserved_character(subspec, errbuf) ||\n+\t\tgently_parse_list_objects_filter(\n+\t\t\t&filter_options->sub[new_index], decoded, errbuf);\n+\n+\tfree(decoded);\n+\treturn result;\n+}\n+\n+static int parse_combine_filter(\n+\tstruct list_objects_filter_options *filter_options,\n+\tconst char *arg,\n+\tstruct strbuf *errbuf)\n+{\n+\tstruct strbuf **subspecs = strbuf_split_str(arg, '+', 0);\n+\tsize_t sub;\n+\tint result = 0;\n+\n+\tif (!subspecs[0]) {\n+\t\tstrbuf_addstr(errbuf, _(\"expected something after combine:\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tfor (sub = 0; subspecs[sub] && !result; sub++) {\n+\t\tif (subspecs[sub + 1]) {\n+\t\t\t/*\n+\t\t\t * This is not the last subspec. Remove trailing \"+\" so\n+\t\t\t * we can parse it.\n+\t\t\t */\n+\t\t\tsize_t last = subspecs[sub]->len - 1;\n+\t\t\tassert(subspecs[sub]->buf[last] == '+');\n+\t\t\tstrbuf_remove(subspecs[sub], last, 1);\n+\t\t}\n+\t\tresult = parse_combine_subfilter(\n+\t\t\tfilter_options, subspecs[sub], errbuf);\n+\t}\n+\n+\tfilter_options->choice = LOFC_COMBINE;\n+\n+cleanup:\n+\tstrbuf_list_free(subspecs);\n+\tif (result) {\n+\t\tlist_objects_filter_release(filter_options);\n+\t\tmemset(filter_options, 0, sizeof(*filter_options));\n+\t}\n+\treturn result;\n+}\n+\n int parse_list_objects_filter(struct list_objects_filter_options *filter_options,\n \t\t\t const char *arg)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n+\tfilter_options->filter_spec = strdup(arg);\n \tif (gently_parse_list_objects_filter(filter_options, arg, &buf))\n \t\tdie(\"%s\", buf.buf);\n \treturn 0;\n }\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct list_objects_filter_options *filter_options = opt->value;\n \n@@ -122,22 +215,29 @@ void expand_list_objects_filter_spec(\n \telse if (filter->choice == LOFC_TREE_DEPTH)\n \t\tstrbuf_addf(expanded_spec, \"tree:%lu\",\n \t\t\t filter->tree_exclude_depth);\n \telse\n \t\tstrbuf_addstr(expanded_spec, filter->filter_spec);\n }\n \n void list_objects_filter_release(\n \tstruct list_objects_filter_options *filter_options)\n {\n+\tsize_t sub;\n+\n+\tif (!filter_options)\n+\t\treturn;\n \tfree(filter_options->filter_spec);\n \tfree(filter_options->sparse_oid_value);\n+\tfor (sub = 0; sub < filter_options->sub_nr; sub++)\n+\t\tlist_objects_filter_release(&filter_options->sub[sub]);\n+\tfree(filter_options->sub);\n \tmemset(filter_options, 0, sizeof(*filter_options));\n }\n \n void partial_clone_register(\n \tconst char *remote,\n \tconst struct list_objects_filter_options *filter_options)\n {\n \t/*\n \t * Record the name of the partial clone remote in the\n \t * config and in the global variable -- the latter is\n@@ -167,15 +267,17 @@ void partial_clone_register(\n void partial_clone_get_default_filter_spec(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tstruct strbuf errbuf = STRBUF_INIT;\n \n \t/*\n \t * Parse default value, but silently ignore it if it is invalid.\n \t */\n \tif (!core_partial_clone_filter_default)\n \t\treturn;\n+\n+\tfilter_options->filter_spec = strdup(core_partial_clone_filter_default);\n \tgently_parse_list_objects_filter(filter_options,\n \t\t\t\t\t core_partial_clone_filter_default,\n \t\t\t\t\t &errbuf);\n \tstrbuf_release(&errbuf);\n }\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex c54f0000fb..789faef1e5 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -6,20 +6,21 @@\n \n /*\n * The list of defined filters for list-objects.\n */\n enum list_objects_filter_choice {\n \tLOFC_DISABLED = 0,\n \tLOFC_BLOB_NONE,\n \tLOFC_BLOB_LIMIT,\n \tLOFC_TREE_DEPTH,\n \tLOFC_SPARSE_OID,\n+\tLOFC_COMBINE,\n \tLOFC__COUNT /* must be last */\n };\n \n struct list_objects_filter_options {\n \t/*\n \t * 'filter_spec' is the raw argument value given on the command line\n \t * or protocol request. (The part after the \"--keyword=\".) For\n \t * commands that launch filtering sub-processes, or for communication\n \t * over the network, don't use this value; use the result of\n \t * expand_list_objects_filter_spec() instead.\n@@ -31,27 +32,37 @@ struct list_objects_filter_options {\n \t * the filtering algorithm to use.\n \t */\n \tenum list_objects_filter_choice choice;\n \n \t/*\n \t * Choice is LOFC_DISABLED because \"--no-filter\" was requested.\n \t */\n \tunsigned int no_filter : 1;\n \n \t/*\n-\t * Parsed values (fields) from within the filter-spec. These are\n-\t * choice-specific; not all values will be defined for any given\n-\t * choice.\n+\t * BEGIN choice-specific parsed values from within the filter-spec. Only\n+\t * some values will be defined for any given choice.\n \t */\n+\n \tstruct object_id *sparse_oid_value;\n \tunsigned long blob_limit_value;\n \tunsigned long tree_exclude_depth;\n+\n+\t/* LOFC_COMBINE values */\n+\n+\t/* This array contains all the subfilters which this filter combines. */\n+\tsize_t sub_nr, sub_alloc;\n+\tstruct list_objects_filter_options *sub;\n+\n+\t/*\n+\t * END choice-specific parsed values.\n+\t */\n };\n \n /* Normalized command line arguments */\n #define CL_ARG__FILTER \"filter\"\n \n int parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg);\n \n int opt_parse_list_objects_filter(const struct option *opt,\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex 3b4b6764ca..d664264d65 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -19,30 +19,55 @@\n * FILTER_SHOWN_BUT_REVISIT -- we set this bit on tree objects\n * that have been shown, but should be revisited if they appear\n * in the traversal (until we mark it SEEN). This is a way to\n * let us silently de-dup calls to show() in the caller. This\n * is subtly different from the \"revision.h:SHOWN\" and the\n * \"sha1-name.c:ONELINE_SEEN\" bits. And also different from\n * the non-de-dup usage in pack-bitmap.c\n */\n #define FILTER_SHOWN_BUT_REVISIT (1<<21)\n \n+struct subfilter {\n+\tstruct filter *filter;\n+\tstruct oidset seen;\n+\tstruct oidset omits;\n+\tstruct object_id skip_tree;\n+\tunsigned is_skipping_tree : 1;\n+};\n+\n struct filter {\n \tenum list_objects_filter_result (*filter_object_fn)(\n \t\tstruct repository *r,\n \t\tenum list_objects_filter_situation filter_situation,\n \t\tstruct object *obj,\n \t\tconst char *pathname,\n \t\tconst char *filename,\n \t\tstruct oidset *omits,\n \t\tvoid *filter_data);\n \n+\t/*\n+\t * Optional. If this function is supplied and the filter needs\n+\t * to collect omits, then this function is called once before\n+\t * free_fn is called.\n+\t *\n+\t * This is required because the following two conditions hold:\n+\t *\n+\t * a. A tree filter can add and remove objects as an object\n+\t * graph is traversed.\n+\t * b. A combine filter's omit set is the union of all its\n+\t * subfilters, which may include tree: filters.\n+\t *\n+\t * As such, the omits sets must be separate sets, and can only\n+\t * be unioned after the traversal is completed.\n+\t */\n+\tvoid (*finalize_omits_fn)(struct oidset *omits, void *filter_data);\n+\n \tvoid (*free_fn)(void *filter_data);\n \n \tvoid *filter_data;\n \n \t/* If non-NULL, the filter collects a list of the omitted OIDs here. */\n \tstruct oidset *omits;\n };\n \n static enum list_objects_filter_result filter_blobs_none(\n \tstruct repository *r,\n@@ -464,33 +489,167 @@ static void filter_sparse_oid__init(\n \tALLOC_GROW(d->array_frame, d->nr + 1, d->alloc);\n \td->array_frame[d->nr].defval = 0; /* default to include */\n \td->array_frame[d->nr].child_prov_omit = 0;\n \td->nr++;\n \n \tfilter->filter_data = d;\n \tfilter->filter_object_fn = filter_sparse;\n \tfilter->free_fn = filter_sparse_free;\n }\n \n+/* A filter which only shows objects shown by all sub-filters. */\n+struct combine_filter_data {\n+\tstruct subfilter *sub;\n+\tsize_t nr;\n+};\n+\n+static enum list_objects_filter_result process_subfilter(\n+\tstruct repository *r,\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tstruct subfilter *sub)\n+{\n+\tenum list_objects_filter_result result;\n+\n+\t/*\n+\t * Check and update is_skipping_tree before oidset_contains so\n+\t * that is_skipping_tree gets unset even when the object is\n+\t * marked as seen. As of this writing, no filter uses\n+\t * LOFR_MARK_SEEN on trees that also uses LOFR_SKIP_TREE, so the\n+\t * ordering is only theoretically important. Be cautious if you\n+\t * change the order of the below checks and more filters have\n+\t * been added!\n+\t */\n+\tif (sub->is_skipping_tree) {\n+\t\tif (filter_situation == LOFS_END_TREE &&\n+\t\t oideq(&obj->oid, &sub->skip_tree))\n+\t\t\tsub->is_skipping_tree = 0;\n+\t\telse\n+\t\t\treturn LOFR_ZERO;\n+\t}\n+\tif (oidset_contains(&sub->seen, &obj->oid))\n+\t\treturn LOFR_ZERO;\n+\n+\tresult = list_objects_filter__filter_object(\n+\t\tr, filter_situation, obj, pathname, filename, sub->filter);\n+\n+\tif (result & LOFR_MARK_SEEN)\n+\t\toidset_insert(&sub->seen, &obj->oid);\n+\n+\tif (result & LOFR_SKIP_TREE) {\n+\t\tsub->is_skipping_tree = 1;\n+\t\tsub->skip_tree = obj->oid;\n+\t}\n+\n+\treturn result;\n+}\n+\n+static enum list_objects_filter_result filter_combine(\n+\tstruct repository *r,\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tstruct oidset *omits,\n+\tvoid *filter_data)\n+{\n+\tstruct combine_filter_data *d = filter_data;\n+\tenum list_objects_filter_result combined_result =\n+\t\tLOFR_DO_SHOW | LOFR_MARK_SEEN | LOFR_SKIP_TREE;\n+\tsize_t sub;\n+\n+\tfor (sub = 0; sub < d->nr; sub++) {\n+\t\tenum list_objects_filter_result sub_result = process_subfilter(\n+\t\t\tr, filter_situation, obj, pathname, filename,\n+\t\t\t&d->sub[sub]);\n+\t\tif (!(sub_result & LOFR_DO_SHOW))\n+\t\t\tcombined_result &= ~LOFR_DO_SHOW;\n+\t\tif (!(sub_result & LOFR_MARK_SEEN))\n+\t\t\tcombined_result &= ~LOFR_MARK_SEEN;\n+\t\tif (!d->sub[sub].is_skipping_tree)\n+\t\t\tcombined_result &= ~LOFR_SKIP_TREE;\n+\t}\n+\n+\treturn combined_result;\n+}\n+\n+static void filter_combine__free(void *filter_data)\n+{\n+\tstruct combine_filter_data *d = filter_data;\n+\tsize_t sub;\n+\tfor (sub = 0; sub < d->nr; sub++) {\n+\t\tlist_objects_filter__free(d->sub[sub].filter);\n+\t\toidset_clear(&d->sub[sub].seen);\n+\t\tif (d->sub[sub].omits.set.size)\n+\t\t\tBUG(\"expected oidset to be cleared already\");\n+\t}\n+\tfree(d->sub);\n+}\n+\n+static void add_all(struct oidset *dest, struct oidset *src) {\n+\tstruct oidset_iter iter;\n+\tstruct object_id *src_oid;\n+\n+\toidset_iter_init(src, &iter);\n+\twhile ((src_oid = oidset_iter_next(&iter)) != NULL)\n+\t\toidset_insert(dest, src_oid);\n+}\n+\n+static void filter_combine__finalize_omits(\n+\tstruct oidset *omits,\n+\tvoid *filter_data)\n+{\n+\tstruct combine_filter_data *d = filter_data;\n+\tsize_t sub;\n+\n+\tfor (sub = 0; sub < d->nr; sub++) {\n+\t\tadd_all(omits, &d->sub[sub].omits);\n+\t\toidset_clear(&d->sub[sub].omits);\n+\t}\n+}\n+\n+static void filter_combine__init(\n+\tstruct list_objects_filter_options *filter_options,\n+\tstruct filter* filter)\n+{\n+\tstruct combine_filter_data *d = xcalloc(1, sizeof(*d));\n+\tsize_t sub;\n+\n+\td->nr = filter_options->sub_nr;\n+\td->sub = xcalloc(d->nr, sizeof(*d->sub));\n+\tfor (sub = 0; sub < d->nr; sub++)\n+\t\td->sub[sub].filter = list_objects_filter__init(\n+\t\t\tfilter->omits ? &d->sub[sub].omits : NULL,\n+\t\t\t&filter_options->sub[sub]);\n+\n+\tfilter->filter_data = d;\n+\tfilter->filter_object_fn = filter_combine;\n+\tfilter->free_fn = filter_combine__free;\n+\tfilter->finalize_omits_fn = filter_combine__finalize_omits;\n+}\n+\n typedef void (*filter_init_fn)(\n \tstruct list_objects_filter_options *filter_options,\n \tstruct filter *filter);\n \n /*\n * Must match \"enum list_objects_filter_choice\".\n */\n static filter_init_fn s_filters[] = {\n \tNULL,\n \tfilter_blobs_none__init,\n \tfilter_blobs_limit__init,\n \tfilter_trees_depth__init,\n \tfilter_sparse_oid__init,\n+\tfilter_combine__init,\n };\n \n struct filter *list_objects_filter__init(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options)\n {\n \tstruct filter *filter;\n \tfilter_init_fn init_fn;\n \n \tassert((sizeof(s_filters) / sizeof(s_filters[0])) == LOFC__COUNT);\n@@ -529,13 +688,15 @@ enum list_objects_filter_result list_objects_filter__filter_object(\n \t */\n \tif (filter_situation == LOFS_END_TREE)\n \t\treturn 0;\n \treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n }\n \n void list_objects_filter__free(struct filter *filter)\n {\n \tif (!filter)\n \t\treturn;\n+\tif (filter->finalize_omits_fn && filter->omits)\n+\t\tfilter->finalize_omits_fn(filter->omits, filter->filter_data);\n \tfilter->free_fn(filter->filter_data);\n \tfree(filter);\n }\ndiff --git a/list-objects-filter.h b/list-objects-filter.h\nindex 6908954266..cfd784e203 100644\n--- a/list-objects-filter.h\n+++ b/list-objects-filter.h\n@@ -55,32 +55,41 @@ enum list_objects_filter_result {\n };\n \n enum list_objects_filter_situation {\n \tLOFS_BEGIN_TREE,\n \tLOFS_END_TREE,\n \tLOFS_BLOB\n };\n \n struct filter;\n \n-/* Constructor for the set of defined list-objects filters. */\n+/*\n+ * Constructor for the set of defined list-objects filters.\n+ * The `omitted` set is optional. It is populated with objects that the\n+ * filter excludes. This set should not be considered finalized until\n+ * after list_objects_filter__free is called on the returned `struct\n+ * filter *`.\n+ */\n struct filter *list_objects_filter__init(\n \tstruct oidset *omitted,\n \tstruct list_objects_filter_options *filter_options);\n \n /*\n * Lets `filter` decide how to handle the `obj`. If `filter` is NULL, this\n * function behaves as expected if no filter is configured: all objects are\n * included.\n */\n enum list_objects_filter_result list_objects_filter__filter_object(\n \tstruct repository *r,\n \tenum list_objects_filter_situation filter_situation,\n \tstruct object *obj,\n \tconst char *pathname,\n \tconst char *filename,\n \tstruct filter *filter);\n \n-/* Destroys `filter`. Does nothing if `filter` is null. */\n+/*\n+ * Destroys `filter` and finalizes the `omitted` set, if present. Does\n+ * nothing if `filter` is null.\n+ */\n void list_objects_filter__free(struct filter *filter);\n \n #endif /* LIST_OBJECTS_FILTER_H */\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex acd7f5ab80..05d4f2e9c2 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -271,21 +271,33 @@ test_expect_success 'verify tree:0 includes trees in \"filtered\" output' '\n # Make sure tree:0 does not iterate through any trees.\n \n test_expect_success 'verify skipping tree iteration when not collecting omits' '\n \tGIT_TRACE=1 git -C r3 rev-list \\\n \t\t--objects --filter=tree:0 HEAD 2>filter_trace &&\n \tgrep \"Skipping contents of tree [.][.][.]\" filter_trace >actual &&\n \t# One line for each commit traversed.\n \ttest_line_count = 2 actual &&\n \n \t# Make sure no other trees were considered besides the root.\n-\t! grep \"Skipping contents of tree [^.]\" filter_trace\n+\t! grep \"Skipping contents of tree [^.]\" filter_trace &&\n+\n+\t# Try this again with \"combine:\". If both sub-filters are skipping\n+\t# trees, the composite filter should also skip trees. This is not\n+\t# important unless the user does combine:tree:X+tree:Y or another filter\n+\t# besides \"tree:\" is implemented in the future which can skip trees.\n+\tGIT_TRACE=1 git -C r3 rev-list \\\n+\t\t--objects --filter=combine:tree:1+tree:3 HEAD 2>filter_trace &&\n+\n+\t# Only skip the dir1/ tree, which is shared between the two commits.\n+\tgrep \"Skipping contents of tree \" filter_trace >actual &&\n+\ttest_write_lines \"Skipping contents of tree dir1/...\" >expected &&\n+\ttest_cmp expected actual\n '\n \n # Test tree:# filters.\n \n expect_has () {\n \tcommit=$1 &&\n \tname=$2 &&\n \n \thash=$(git -C r3 rev-parse $commit:$name) &&\n \tgrep \"^$hash $name$\" actual\n@@ -323,20 +335,126 @@ test_expect_success 'verify tree:3 includes everything expected' '\n \texpect_has HEAD dir1/sparse1 &&\n \texpect_has HEAD dir1/sparse2 &&\n \texpect_has HEAD pattern &&\n \texpect_has HEAD sparse1 &&\n \texpect_has HEAD sparse2 &&\n \n \t# There are also 2 commit objects\n \ttest_line_count = 10 actual\n '\n \n+test_expect_success 'combine:... for a simple combination' '\n+\tgit -C r3 rev-list --objects --filter=combine:tree:2+blob:none HEAD \\\n+\t\t>actual &&\n+\n+\texpect_has HEAD \"\" &&\n+\texpect_has HEAD~1 \"\" &&\n+\texpect_has HEAD dir1 &&\n+\n+\t# There are also 2 commit objects\n+\ttest_line_count = 5 actual\n+'\n+\n+test_expect_success 'combine:... with URL encoding' '\n+\tgit -C r3 rev-list --objects \\\n+\t\t--filter=combine:tree%3a2+blob:%6Eon%65 HEAD >actual &&\n+\n+\texpect_has HEAD \"\" &&\n+\texpect_has HEAD~1 \"\" &&\n+\texpect_has HEAD dir1 &&\n+\n+\t# There are also 2 commit objects\n+\ttest_line_count = 5 actual\n+'\n+\n+expect_invalid_filter_spec () {\n+\tspec=\"$1\" &&\n+\terr=\"$2\" &&\n+\n+\ttest_must_fail git -C r3 rev-list --objects --filter=\"$spec\" HEAD \\\n+\t\t>actual 2>actual_stderr &&\n+\ttest_must_be_empty actual &&\n+\ttest_i18ngrep \"$err\" actual_stderr\n+}\n+\n+test_expect_success 'combine:... while URL-encoding things that should not be' '\n+\texpect_invalid_filter_spec combine%3Atree:2+blob:none \\\n+\t\t\"invalid filter-spec\"\n+'\n+\n+test_expect_success 'combine: with nothing after the :' '\n+\texpect_invalid_filter_spec combine: \"expected something after combine:\"\n+'\n+\n+test_expect_success 'parse error in first sub-filter in combine:' '\n+\texpect_invalid_filter_spec combine:tree:asdf+blob:none \\\n+\t\t\"expected .tree:.\"\n+'\n+\n+test_expect_success 'combine:... with non-encoded reserved chars' '\n+\texpect_invalid_filter_spec combine:tree:2+sparse:@xyz \\\n+\t\t\"must escape char in sub-filter-spec: .@.\" &&\n+\texpect_invalid_filter_spec combine:tree:2+sparse:\\` \\\n+\t\t\"must escape char in sub-filter-spec: .\\`.\" &&\n+\texpect_invalid_filter_spec combine:tree:2+sparse:~abc \\\n+\t\t\"must escape char in sub-filter-spec: .\\~.\"\n+'\n+\n+test_expect_success 'validate err msg for \"combine:+\"' '\n+\texpect_invalid_filter_spec combine:tree:2+ \"expected .tree:.\"\n+'\n+\n+test_expect_success 'combine:... with edge-case hex digits: Ff Aa 0 9' '\n+\tgit -C r3 rev-list --objects --filter=\"combine:tree:2+bl%6Fb:n%6fne\" \\\n+\t\tHEAD >actual &&\n+\ttest_line_count = 5 actual &&\n+\tgit -C r3 rev-list --objects --filter=\"combine:tree%3A2+blob%3anone\" \\\n+\t\tHEAD >actual &&\n+\ttest_line_count = 5 actual &&\n+\tgit -C r3 rev-list --objects --filter=\"combine:tree:%30\" HEAD >actual &&\n+\ttest_line_count = 2 actual &&\n+\tgit -C r3 rev-list --objects --filter=\"combine:tree:%39+blob:none\" \\\n+\t\tHEAD >actual &&\n+\ttest_line_count = 5 actual\n+'\n+\n+test_expect_success 'add a sparse pattern blob whose path has reserved chars' '\n+\tcp r3/pattern r3/pattern1+renamed% &&\n+\tgit -C r3 add pattern1+renamed% &&\n+\tgit -C r3 commit -m \"add sparse pattern file with reserved chars\"\n+'\n+\n+test_expect_success 'combine:... with more than two sub-filters' '\n+\tgit -C r3 rev-list --objects \\\n+\t\t--filter=combine:tree:3+blob:limit=40+sparse:oid=master:pattern \\\n+\t\tHEAD >actual &&\n+\n+\texpect_has HEAD \"\" &&\n+\texpect_has HEAD~1 \"\" &&\n+\texpect_has HEAD~2 \"\" &&\n+\texpect_has HEAD dir1 &&\n+\texpect_has HEAD dir1/sparse1 &&\n+\texpect_has HEAD dir1/sparse2 &&\n+\n+\t# Should also have 3 commits\n+\ttest_line_count = 9 actual &&\n+\n+\t# Try again, this time making sure the last sub-filter is only\n+\t# URL-decoded once.\n+\tcp actual expect &&\n+\n+\tgit -C r3 rev-list --objects \\\n+\t\t--filter=combine:tree:3+blob:limit=40+sparse:oid=master:pattern1%2brenamed%25 \\\n+\t\tHEAD >actual &&\n+\ttest_cmp expect actual\n+'\n+\n # Test provisional omit collection logic with a repo that has objects appearing\n # at multiple depths - first deeper than the filter's threshold, then shallow.\n \n test_expect_success 'setup r4' '\n \tgit init r4 &&\n \n \techo foo > r4/foo &&\n \tmkdir r4/subdir &&\n \techo bar > r4/subdir/bar &&\n \n@@ -366,20 +484,51 @@ test_expect_success 'test tree:# filter provisional omit for blob and tree' '\n \n test_expect_success 'verify skipping tree iteration when collecting omits' '\n \tGIT_TRACE=1 git -C r4 rev-list --filter-print-omitted \\\n \t\t--objects --filter=tree:0 HEAD 2>filter_trace &&\n \tgrep \"^Skipping contents of tree \" filter_trace >actual &&\n \n \techo \"Skipping contents of tree subdir/...\" >expect &&\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'setup r5' '\n+\tgit init r5 &&\n+\tmkdir -p r5/subdir &&\n+\n+\techo 1 >r5/short-root &&\n+\techo 12345 >r5/long-root &&\n+\techo a >r5/subdir/short-subdir &&\n+\techo abcde >r5/subdir/long-subdir &&\n+\n+\tgit -C r5 add short-root long-root subdir &&\n+\tgit -C r5 commit -m \"commit msg\"\n+'\n+\n+test_expect_success 'verify collecting omits in combined: filter' '\n+\t# Note that this test guards against the naive implementation of simply\n+\t# giving both filters the same \"omits\" set and expecting it to\n+\t# automatically merge them.\n+\tgit -C r5 rev-list --objects --quiet --filter-print-omitted \\\n+\t\t--filter=combine:tree:2+blob:limit=3 HEAD >actual &&\n+\n+\t# Expect 0 trees/commits, 3 blobs omitted (all blobs except short-root)\n+\tomitted_1=$(echo 12345 | git hash-object --stdin) &&\n+\tomitted_2=$(echo a | git hash-object --stdin) &&\n+\tomitted_3=$(echo abcde | git hash-object --stdin) &&\n+\n+\tgrep ~$omitted_1 actual &&\n+\tgrep ~$omitted_2 actual &&\n+\tgrep ~$omitted_3 actual &&\n+\ttest_line_count = 3 actual\n+'\n+\n # Test tree: where a tree is iterated to twice - once where a subentry is\n # too deep to be included, and again where the blob inside it is shallow enough\n # to be included. This makes sure we don't use LOFR_MARK_SEEN incorrectly (we\n # can't use it because a tree can be iterated over again at a lower depth).\n \n test_expect_success 'tree: where we iterate over tree at two levels' '\n \tgit init r5 &&\n \n \tmkdir -p r5/a/subdir/b &&\n \techo foo > r5/a/subdir/b/foo &&\ndiff --git a/url.c b/url.c\nindex 1b8ef78cea..e34e5e7517 100644\n--- a/url.c\n+++ b/url.c\n@@ -79,20 +79,26 @@ char *url_decode_mem(const char *url, int len)\n \n \t/* Skip protocol part if present */\n \tif (colon && url < colon) {\n \t\tstrbuf_add(&out, url, colon - url);\n \t\tlen -= colon - url;\n \t\turl = colon;\n \t}\n \treturn url_decode_internal(&url, len, NULL, &out, 0);\n }\n \n+char *url_percent_decode(const char *encoded)\n+{\n+\tstruct strbuf out = STRBUF_INIT;\n+\treturn url_decode_internal(&encoded, strlen(encoded), NULL, &out, 0);\n+}\n+\n char *url_decode_parameter_name(const char **query)\n {\n \tstruct strbuf out = STRBUF_INIT;\n \treturn url_decode_internal(query, -1, \"&=\", &out, 1);\n }\n \n char *url_decode_parameter_value(const char **query)\n {\n \tstruct strbuf out = STRBUF_INIT;\n \treturn url_decode_internal(query, -1, \"&\", &out, 1);\ndiff --git a/url.h b/url.h\nindex 00b7d58c33..2a27c34277 100644\n--- a/url.h\n+++ b/url.h\n@@ -1,16 +1,24 @@\n #ifndef URL_H\n #define URL_H\n \n struct strbuf;\n \n int is_url(const char *url);\n int is_urlschemechar(int first_flag, int ch);\n char *url_decode(const char *url);\n char *url_decode_mem(const char *url, int len);\n+\n+/*\n+ * Similar to the url_decode_{,mem} methods above, but doesn't assume there\n+ * is a scheme followed by a : at the start of the string. Instead, %-sequences\n+ * before any : are also parsed.\n+ */\n+char *url_percent_decode(const char *encoded);\n+\n char *url_decode_parameter_name(const char **query);\n char *url_decode_parameter_value(const char **query);\n \n void end_url_with_slash(struct strbuf *buf, const char *url);\n void str_end_url_with_slash(const char *url, char **dest);\n \n #endif /* URL_H */\n-- \n2.21.0\n\n"},{"id":"378216","messageId":"32d1d81e5dc3b6ccbaa94266823e9c6c4657c191.1561675151.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1561675151.git.matvore@google.com","subject":"[PATCH v5 05/10] list-objects-filter-options: move error check up","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-27T22:54:09Z","receivedAt":"2019-06-27T22:54:38Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Move the check that filter_options->choice is set to higher in the call\nstack. This can only be set when the gentle parse function is called\nfrom one of the two call sites.\n\nThis is important because in an upcoming patch this may or may not be an\nerror, and whether it is an error is only known to the\nparse_list_objects_filter function.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects-filter-options.c | 9 ++++-----\n 1 file changed, 4 insertions(+), 5 deletions(-)\n\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex 75d0236ee2..5fe2814841 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -28,25 +28,22 @@ static int parse_combine_filter(\n * expand_list_objects_filter_spec() first). We also \"intern\" the arg for the\n * convenience of the current command.\n */\n static int gently_parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg,\n \tstruct strbuf *errbuf)\n {\n \tconst char *v0;\n \n-\tif (filter_options->choice) {\n-\t\tstrbuf_addstr(\n-\t\t\terrbuf, _(\"multiple filter-specs cannot be combined\"));\n-\t\treturn 1;\n-\t}\n+\tif (filter_options->choice)\n+\t\tBUG(\"filter_options already populated\");\n \n \tif (!strcmp(arg, \"blob:none\")) {\n \t\tfilter_options->choice = LOFC_BLOB_NONE;\n \t\treturn 0;\n \n \t} else if (skip_prefix(arg, \"blob:limit=\", &v0)) {\n \t\tif (git_parse_ulong(v0, &filter_options->blob_limit_value)) {\n \t\t\tfilter_options->choice = LOFC_BLOB_LIMIT;\n \t\t\treturn 0;\n \t\t}\n@@ -178,20 +175,22 @@ static int parse_combine_filter(\n \t\tlist_objects_filter_release(filter_options);\n \t\tmemset(filter_options, 0, sizeof(*filter_options));\n \t}\n \treturn result;\n }\n \n int parse_list_objects_filter(struct list_objects_filter_options *filter_options,\n \t\t\t const char *arg)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n+\tif (filter_options->choice)\n+\t\tdie(_(\"multiple filter-specs cannot be combined\"));\n \tfilter_options->filter_spec = strdup(arg);\n \tif (gently_parse_list_objects_filter(filter_options, arg, &buf))\n \t\tdie(\"%s\", buf.buf);\n \treturn 0;\n }\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct list_objects_filter_options *filter_options = opt->value;\n-- \n2.21.0\n\n"},{"id":"378217","messageId":"ec22c61189a93ce3c1ade42f654d72cf34bfb7a9.1561675151.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1561675151.git.matvore@google.com","subject":"[PATCH v5 06/10] list-objects-filter-options: make filter_spec a string_list","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-27T22:54:10Z","receivedAt":"2019-06-27T22:54:40Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Make the filter_spec string a string_list rather than a raw C string.\nThe list of strings must be concatted together to make a complete\nfilter_spec. A future patch will use this capability to build \"combine:\"\nfilter specs gradually.\n\nA strbuf would seem to be a more natural choice for this object, but it\nunfortunately requires initialization besides just zero'ing out the\nmemory. This results in all container structs, and all containers of\nthose structs, etc., to also require initialization. Initializing them\nall would be more cumbersome that simply using a string_list, which\nbehaves properly when its contents are zero'd.\n\nFor the purposes of code simplification, change behavior in how filter\nspecs are conveyed over the protocol: do not normalize the tree:\nfilter specs since there should be no server in existence that supports\ntree:# but not tree:#k etc.\n\nHelped-by: Junio C Hamano \nSigned-off-by: Matthew DeVore \n---\n builtin/clone.c | 8 ++---\n builtin/fetch.c | 9 ++----\n builtin/rev-list.c | 6 ++--\n fetch-pack.c | 20 ++++--------\n list-objects-filter-options.c | 50 ++++++++++++++++++++---------\n list-objects-filter-options.h | 27 +++++++++++-----\n t/t6112-rev-list-filters-objects.sh | 7 ----\n transport-helper.c | 10 ++----\n upload-pack.c | 11 +++----\n 9 files changed, 78 insertions(+), 70 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 5b9ebe9947..a693e6ca44 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1142,27 +1142,25 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\ttransport_set_option(transport, TRANS_OPT_FOLLOWTAGS, \"1\");\n \n \tif (option_upload_pack)\n \t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK,\n \t\t\t\t option_upload_pack);\n \n \tif (server_options.nr)\n \t\ttransport->server_options = &server_options;\n \n \tif (filter_options.choice) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n-\t\texpand_list_objects_filter_spec(&filter_options,\n-\t\t\t\t\t\t&expanded_filter_spec);\n+\t\tconst char *spec =\n+\t\t\texpand_list_objects_filter_spec(&filter_options);\n \t\ttransport_set_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER,\n-\t\t\t\t expanded_filter_spec.buf);\n+\t\t\t\t spec);\n \t\ttransport_set_option(transport, TRANS_OPT_FROM_PROMISOR, \"1\");\n-\t\tstrbuf_release(&expanded_filter_spec);\n \t}\n \n \tif (transport->smart_options && !deepen && !filter_options.choice)\n \t\ttransport->smart_options->check_self_contained_and_connected = 1;\n \n \n \targv_array_push(&ref_prefixes, \"HEAD\");\n \trefspec_ref_prefixes(&remote->fetch, &ref_prefixes);\n \tif (option_branch)\n \t\texpand_ref_prefix(&ref_prefixes, option_branch);\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 4ba63d5ac6..dee89e1a19 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1181,27 +1181,24 @@ static struct transport *prepare_transport(struct remote *remote, int deepen)\n \tif (deepen && deepen_since)\n \t\tset_option(transport, TRANS_OPT_DEEPEN_SINCE, deepen_since);\n \tif (deepen && deepen_not.nr)\n \t\tset_option(transport, TRANS_OPT_DEEPEN_NOT,\n \t\t\t (const char *)&deepen_not);\n \tif (deepen_relative)\n \t\tset_option(transport, TRANS_OPT_DEEPEN_RELATIVE, \"yes\");\n \tif (update_shallow)\n \t\tset_option(transport, TRANS_OPT_UPDATE_SHALLOW, \"yes\");\n \tif (filter_options.choice) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n-\t\texpand_list_objects_filter_spec(&filter_options,\n-\t\t\t\t\t\t&expanded_filter_spec);\n-\t\tset_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER,\n-\t\t\t expanded_filter_spec.buf);\n+\t\tconst char *spec =\n+\t\t\texpand_list_objects_filter_spec(&filter_options);\n+\t\tset_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER, spec);\n \t\tset_option(transport, TRANS_OPT_FROM_PROMISOR, \"1\");\n-\t\tstrbuf_release(&expanded_filter_spec);\n \t}\n \tif (negotiation_tip.nr) {\n \t\tif (transport->smart_options)\n \t\t\tadd_negotiation_tips(transport->smart_options);\n \t\telse\n \t\t\twarning(\"Ignoring --negotiation-tip because the protocol does not support it.\");\n \t}\n \treturn transport;\n }\n \ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 660172b014..68acbe8fd2 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -459,22 +459,24 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\tshow_progress = arg;\n \t\t\tcontinue;\n \t\t}\n \n \t\tif (skip_prefix(arg, (\"--\" CL_ARG__FILTER \"=\"), &arg)) {\n \t\t\tparse_list_objects_filter(&filter_options, arg);\n \t\t\tif (filter_options.choice && !revs.blob_objects)\n \t\t\t\tdie(_(\"object filtering requires --objects\"));\n \t\t\tif (filter_options.choice == LOFC_SPARSE_OID &&\n \t\t\t !filter_options.sparse_oid_value)\n-\t\t\t\tdie(_(\"invalid sparse value '%s'\"),\n-\t\t\t\t filter_options.filter_spec);\n+\t\t\t\tdie(\n+\t\t\t\t\t_(\"invalid sparse value '%s'\"),\n+\t\t\t\t\tlist_objects_filter_spec(\n+\t\t\t\t\t\t&filter_options));\n \t\t\tcontinue;\n \t\t}\n \t\tif (!strcmp(arg, (\"--no-\" CL_ARG__FILTER))) {\n \t\t\tlist_objects_filter_set_no_filter(&filter_options);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!strcmp(arg, \"--filter-print-omitted\")) {\n \t\t\targ_print_omitted = 1;\n \t\t\tcontinue;\n \t\t}\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 1c10f54e78..72e13b0a1d 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -332,26 +332,23 @@ static int find_common(struct fetch_negotiator *negotiator,\n \t\tpacket_buf_write(&req_buf, \"deepen-since %\"PRItime, max_age);\n \t}\n \tif (args->deepen_not) {\n \t\tint i;\n \t\tfor (i = 0; i < args->deepen_not->nr; i++) {\n \t\t\tstruct string_list_item *s = args->deepen_not->items + i;\n \t\t\tpacket_buf_write(&req_buf, \"deepen-not %s\", s->string);\n \t\t}\n \t}\n \tif (server_supports_filtering && args->filter_options.choice) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n-\t\texpand_list_objects_filter_spec(&args->filter_options,\n-\t\t\t\t\t\t&expanded_filter_spec);\n-\t\tpacket_buf_write(&req_buf, \"filter %s\",\n-\t\t\t\t expanded_filter_spec.buf);\n-\t\tstrbuf_release(&expanded_filter_spec);\n+\t\tconst char *spec =\n+\t\t\texpand_list_objects_filter_spec(&args->filter_options);\n+\t\tpacket_buf_write(&req_buf, \"filter %s\", spec);\n \t}\n \tpacket_buf_flush(&req_buf);\n \tstate_len = req_buf.len;\n \n \tif (args->deepen) {\n \t\tconst char *arg;\n \t\tstruct object_id oid;\n \n \t\tsend_request(args, fd[1], &req_buf);\n \t\twhile (packet_reader_read(&reader) == PACKET_READ_NORMAL) {\n@@ -1092,21 +1089,21 @@ static int add_haves(struct fetch_negotiator *negotiator,\n \t\tret = 1;\n \t}\n \n \t/* Increase haves to send on next round */\n \t*haves_to_send = next_flush(1, *haves_to_send);\n \n \treturn ret;\n }\n \n static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n-\t\t\t const struct fetch_pack_args *args,\n+\t\t\t struct fetch_pack_args *args,\n \t\t\t const struct ref *wants, struct oidset *common,\n \t\t\t int *haves_to_send, int *in_vain,\n \t\t\t int sideband_all)\n {\n \tint ret = 0;\n \tstruct strbuf req_buf = STRBUF_INIT;\n \n \tif (server_supports_v2(\"fetch\", 1))\n \t\tpacket_buf_write(&req_buf, \"command=fetch\");\n \tif (server_supports_v2(\"agent\", 0))\n@@ -1133,27 +1130,24 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n \n \t/* Add shallow-info and deepen request */\n \tif (server_supports_feature(\"fetch\", \"shallow\", 0))\n \t\tadd_shallow_requests(&req_buf, args);\n \telse if (is_repository_shallow(the_repository) || args->deepen)\n \t\tdie(_(\"Server does not support shallow requests\"));\n \n \t/* Add filter */\n \tif (server_supports_feature(\"fetch\", \"filter\", 0) &&\n \t args->filter_options.choice) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n+\t\tconst char *spec =\n+\t\t\texpand_list_objects_filter_spec(&args->filter_options);\n \t\tprint_verbose(args, _(\"Server supports filter\"));\n-\t\texpand_list_objects_filter_spec(&args->filter_options,\n-\t\t\t\t\t\t&expanded_filter_spec);\n-\t\tpacket_buf_write(&req_buf, \"filter %s\",\n-\t\t\t\t expanded_filter_spec.buf);\n-\t\tstrbuf_release(&expanded_filter_spec);\n+\t\tpacket_buf_write(&req_buf, \"filter %s\", spec);\n \t} else if (args->filter_options.choice) {\n \t\twarning(\"filtering not recognized by server, ignoring\");\n \t}\n \n \t/* add wants */\n \tadd_wants(args->no_dependents, wants, &req_buf);\n \n \tif (args->no_dependents) {\n \t\tpacket_buf_write(&req_buf, \"done\");\n \t\tret = 1;\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex 5fe2814841..01c0f13346 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -177,72 +177,89 @@ static int parse_combine_filter(\n \t}\n \treturn result;\n }\n \n int parse_list_objects_filter(struct list_objects_filter_options *filter_options,\n \t\t\t const char *arg)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tif (filter_options->choice)\n \t\tdie(_(\"multiple filter-specs cannot be combined\"));\n-\tfilter_options->filter_spec = strdup(arg);\n+\tstring_list_append(&filter_options->filter_spec, xstrdup(arg));\n \tif (gently_parse_list_objects_filter(filter_options, arg, &buf))\n \t\tdie(\"%s\", buf.buf);\n \treturn 0;\n }\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct list_objects_filter_options *filter_options = opt->value;\n \n \tif (unset || !arg) {\n \t\tlist_objects_filter_set_no_filter(filter_options);\n \t\treturn 0;\n \t}\n \n \treturn parse_list_objects_filter(filter_options, arg);\n }\n \n-void expand_list_objects_filter_spec(\n-\tconst struct list_objects_filter_options *filter,\n-\tstruct strbuf *expanded_spec)\n+const char *list_objects_filter_spec(struct list_objects_filter_options *filter)\n {\n-\tstrbuf_init(expanded_spec, strlen(filter->filter_spec));\n-\tif (filter->choice == LOFC_BLOB_LIMIT)\n-\t\tstrbuf_addf(expanded_spec, \"blob:limit=%lu\",\n+\tif (!filter->filter_spec.nr)\n+\t\tBUG(\"no filter_spec available for this filter\");\n+\tif (filter->filter_spec.nr != 1) {\n+\t\tstruct strbuf concatted = STRBUF_INIT;\n+\t\tstrbuf_add_separated_string_list(\n+\t\t\t&concatted, \"\", &filter->filter_spec);\n+\t\tstring_list_clear(&filter->filter_spec, /*free_util=*/0);\n+\t\tstring_list_append(\n+\t\t\t&filter->filter_spec, strbuf_detach(&concatted, NULL));\n+\t}\n+\n+\treturn filter->filter_spec.items[0].string;\n+}\n+\n+const char *expand_list_objects_filter_spec(\n+\tstruct list_objects_filter_options *filter)\n+{\n+\tif (filter->choice == LOFC_BLOB_LIMIT) {\n+\t\tstruct strbuf expanded_spec = STRBUF_INIT;\n+\t\tstrbuf_addf(&expanded_spec, \"blob:limit=%lu\",\n \t\t\t filter->blob_limit_value);\n-\telse if (filter->choice == LOFC_TREE_DEPTH)\n-\t\tstrbuf_addf(expanded_spec, \"tree:%lu\",\n-\t\t\t filter->tree_exclude_depth);\n-\telse\n-\t\tstrbuf_addstr(expanded_spec, filter->filter_spec);\n+\t\tstring_list_clear(&filter->filter_spec, /*free_util=*/0);\n+\t\tstring_list_append(\n+\t\t\t&filter->filter_spec,\n+\t\t\tstrbuf_detach(&expanded_spec, NULL));\n+\t}\n+\n+\treturn list_objects_filter_spec(filter);\n }\n \n void list_objects_filter_release(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tsize_t sub;\n \n \tif (!filter_options)\n \t\treturn;\n-\tfree(filter_options->filter_spec);\n+\tstring_list_clear(&filter_options->filter_spec, /*free_util=*/0);\n \tfree(filter_options->sparse_oid_value);\n \tfor (sub = 0; sub < filter_options->sub_nr; sub++)\n \t\tlist_objects_filter_release(&filter_options->sub[sub]);\n \tfree(filter_options->sub);\n \tmemset(filter_options, 0, sizeof(*filter_options));\n }\n \n void partial_clone_register(\n \tconst char *remote,\n-\tconst struct list_objects_filter_options *filter_options)\n+\tstruct list_objects_filter_options *filter_options)\n {\n \t/*\n \t * Record the name of the partial clone remote in the\n \t * config and in the global variable -- the latter is\n \t * used throughout to indicate that partial clone is\n \t * enabled and to expect missing objects.\n \t */\n \tif (repository_format_partial_clone &&\n \t *repository_format_partial_clone &&\n \t strcmp(remote, repository_format_partial_clone))\n@@ -251,32 +268,33 @@ void partial_clone_register(\n \tgit_config_set(\"core.repositoryformatversion\", \"1\");\n \tgit_config_set(\"extensions.partialclone\", remote);\n \n \trepository_format_partial_clone = xstrdup(remote);\n \n \t/*\n \t * Record the initial filter-spec in the config as\n \t * the default for subsequent fetches from this remote.\n \t */\n \tcore_partial_clone_filter_default =\n-\t\txstrdup(filter_options->filter_spec);\n+\t\txstrdup(expand_list_objects_filter_spec(filter_options));\n \tgit_config_set(\"core.partialclonefilter\",\n \t\t core_partial_clone_filter_default);\n }\n \n void partial_clone_get_default_filter_spec(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tstruct strbuf errbuf = STRBUF_INIT;\n \n \t/*\n \t * Parse default value, but silently ignore it if it is invalid.\n \t */\n \tif (!core_partial_clone_filter_default)\n \t\treturn;\n \n-\tfilter_options->filter_spec = strdup(core_partial_clone_filter_default);\n+\tstring_list_append(&filter_options->filter_spec,\n+\t\t\t core_partial_clone_filter_default);\n \tgently_parse_list_objects_filter(filter_options,\n \t\t\t\t\t core_partial_clone_filter_default,\n \t\t\t\t\t &errbuf);\n \tstrbuf_release(&errbuf);\n }\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 789faef1e5..bb33303f9b 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -1,15 +1,15 @@\n #ifndef LIST_OBJECTS_FILTER_OPTIONS_H\n #define LIST_OBJECTS_FILTER_OPTIONS_H\n \n #include \"parse-options.h\"\n-#include \"strbuf.h\"\n+#include \"string-list.h\"\n \n /*\n * The list of defined filters for list-objects.\n */\n enum list_objects_filter_choice {\n \tLOFC_DISABLED = 0,\n \tLOFC_BLOB_NONE,\n \tLOFC_BLOB_LIMIT,\n \tLOFC_TREE_DEPTH,\n \tLOFC_SPARSE_OID,\n@@ -17,22 +17,24 @@ enum list_objects_filter_choice {\n \tLOFC__COUNT /* must be last */\n };\n \n struct list_objects_filter_options {\n \t/*\n \t * 'filter_spec' is the raw argument value given on the command line\n \t * or protocol request. (The part after the \"--keyword=\".) For\n \t * commands that launch filtering sub-processes, or for communication\n \t * over the network, don't use this value; use the result of\n \t * expand_list_objects_filter_spec() instead.\n+\t * To get the raw filter spec given by the user, use the result of\n+\t * list_objects_filter_spec().\n \t */\n-\tchar *filter_spec;\n+\tstruct string_list filter_spec;\n \n \t/*\n \t * 'choice' is determined by parsing the filter-spec. This indicates\n \t * the filtering algorithm to use.\n \t */\n \tenum list_objects_filter_choice choice;\n \n \t/*\n \t * Choice is LOFC_DISABLED because \"--no-filter\" was requested.\n \t */\n@@ -69,35 +71,44 @@ int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset);\n \n #define OPT_PARSE_LIST_OBJECTS_FILTER(fo) \\\n \t{ OPTION_CALLBACK, 0, CL_ARG__FILTER, fo, N_(\"args\"), \\\n \t N_(\"object filtering\"), 0, \\\n \t opt_parse_list_objects_filter }\n \n /*\n * Translates abbreviated numbers in the filter's filter_spec into their\n * fully-expanded forms (e.g., \"limit:blob=1k\" becomes \"limit:blob=1024\").\n+ * Returns a string owned by the list_objects_filter_options object.\n *\n- * This form should be used instead of the raw filter_spec field when\n- * communicating with a remote process or subprocess.\n+ * This form should be used instead of the raw list_objects_filter_spec()\n+ * value when communicating with a remote process or subprocess.\n */\n-void expand_list_objects_filter_spec(\n-\tconst struct list_objects_filter_options *filter,\n-\tstruct strbuf *expanded_spec);\n+const char *expand_list_objects_filter_spec(\n+\tstruct list_objects_filter_options *filter);\n+\n+/*\n+ * Returns the filter spec string more or less in the form as the user\n+ * entered it. This form of the filter_spec can be used in user-facing\n+ * messages. Returns a string owned by the list_objects_filter_options\n+ * object.\n+ */\n+const char *list_objects_filter_spec(\n+\tstruct list_objects_filter_options *filter);\n \n void list_objects_filter_release(\n \tstruct list_objects_filter_options *filter_options);\n \n static inline void list_objects_filter_set_no_filter(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tlist_objects_filter_release(filter_options);\n \tfilter_options->no_filter = 1;\n }\n \n void partial_clone_register(\n \tconst char *remote,\n-\tconst struct list_objects_filter_options *filter_options);\n+\tstruct list_objects_filter_options *filter_options);\n void partial_clone_get_default_filter_spec(\n \tstruct list_objects_filter_options *filter_options);\n \n #endif /* LIST_OBJECTS_FILTER_OPTIONS_H */\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex 05d4f2e9c2..27ba15719a 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -583,18 +583,11 @@ test_expect_success 'rev-list W/ missing=allow-any' '\n # Test expansion of filter specs.\n \n test_expect_success 'expand blob limit in protocol' '\n \tgit -C r2 config --local uploadpack.allowfilter 1 &&\n \tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -c protocol.version=2 clone \\\n \t\t--filter=blob:limit=1k \"file://$(pwd)/r2\" limit &&\n \t! grep \"blob:limit=1k\" trace &&\n \tgrep \"blob:limit=1024\" trace\n '\n \n-test_expect_success 'expand tree depth limit in protocol' '\n-\tGIT_TRACE_PACKET=\"$(pwd)/tree_trace\" git -c protocol.version=2 clone \\\n-\t\t--filter=tree:0k \"file://$(pwd)/r2\" tree &&\n-\t! grep \"tree:0k\" tree_trace &&\n-\tgrep \"tree:0\" tree_trace\n-'\n-\n test_done\ndiff --git a/transport-helper.c b/transport-helper.c\nindex c7e17ec9cb..0a34544df0 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -675,27 +675,23 @@ static int fetch(struct transport *transport,\n \t data->transport_options.check_self_contained_and_connected)\n \t\tset_helper_option(transport, \"check-connectivity\", \"true\");\n \n \tif (transport->cloning)\n \t\tset_helper_option(transport, \"cloning\", \"true\");\n \n \tif (data->transport_options.update_shallow)\n \t\tset_helper_option(transport, \"update-shallow\", \"true\");\n \n \tif (data->transport_options.filter_options.choice) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n-\t\texpand_list_objects_filter_spec(\n-\t\t\t&data->transport_options.filter_options,\n-\t\t\t&expanded_filter_spec);\n-\t\tset_helper_option(transport, \"filter\",\n-\t\t\t\t expanded_filter_spec.buf);\n-\t\tstrbuf_release(&expanded_filter_spec);\n+\t\tconst char *spec = expand_list_objects_filter_spec(\n+\t\t\t&data->transport_options.filter_options);\n+\t\tset_helper_option(transport, \"filter\", spec);\n \t}\n \n \tif (data->transport_options.negotiation_tips)\n \t\twarning(\"Ignoring --negotiation-tip because the protocol does not support it.\");\n \n \tif (data->fetch)\n \t\treturn fetch_with_fetch(transport, nr_heads, to_fetch);\n \n \tif (data->import)\n \t\treturn fetch_with_import(transport, nr_heads, to_fetch);\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 4d2129e7fc..d404d88941 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -133,32 +133,31 @@ static void create_pack_file(const struct object_array *have_obj,\n \n \targv_array_push(&pack_objects.args, \"--stdout\");\n \tif (shallow_nr)\n \t\targv_array_push(&pack_objects.args, \"--shallow\");\n \tif (!no_progress)\n \t\targv_array_push(&pack_objects.args, \"--progress\");\n \tif (use_ofs_delta)\n \t\targv_array_push(&pack_objects.args, \"--delta-base-offset\");\n \tif (use_include_tag)\n \t\targv_array_push(&pack_objects.args, \"--include-tag\");\n-\tif (filter_options.filter_spec) {\n-\t\tstruct strbuf expanded_filter_spec = STRBUF_INIT;\n-\t\texpand_list_objects_filter_spec(&filter_options,\n-\t\t\t\t\t\t&expanded_filter_spec);\n+\tif (filter_options.choice) {\n+\t\tconst char *spec =\n+\t\t\texpand_list_objects_filter_spec(&filter_options);\n \t\tif (pack_objects.use_shell) {\n \t\t\tstruct strbuf buf = STRBUF_INIT;\n-\t\t\tsq_quote_buf(&buf, expanded_filter_spec.buf);\n+\t\t\tsq_quote_buf(&buf, spec);\n \t\t\targv_array_pushf(&pack_objects.args, \"--filter=%s\", buf.buf);\n \t\t\tstrbuf_release(&buf);\n \t\t} else {\n \t\t\targv_array_pushf(&pack_objects.args, \"--filter=%s\",\n-\t\t\t\t\t expanded_filter_spec.buf);\n+\t\t\t\t\t spec);\n \t\t}\n \t}\n \n \tpack_objects.in = -1;\n \tpack_objects.out = -1;\n \tpack_objects.err = -1;\n \n \tif (start_command(&pack_objects))\n \t\tdie(\"git upload-pack: unable to fork git-pack-objects\");\n \n-- \n2.21.0\n\n"},{"id":"378218","messageId":"dfa8f5bc8d7d18b7305b97e41cdb25737aaf0baf.1561675151.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1561675151.git.matvore@google.com","subject":"[PATCH v5 07/10] strbuf: give URL-encoding API a char predicate fn","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-27T22:54:11Z","receivedAt":"2019-06-27T22:54:42Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Allow callers to specify exactly what characters need to be URL-encoded\nand which do not. This new API will be taken advantage of in a patch\nlater in this set.\n\nHelped-by: Jeff King \nSigned-off-by: Matthew DeVore \n---\n credential-store.c | 9 +++++----\n http.c | 6 ++++--\n strbuf.c | 15 ++++++++-------\n strbuf.h | 7 ++++++-\n 4 files changed, 23 insertions(+), 14 deletions(-)\n\ndiff --git a/credential-store.c b/credential-store.c\nindex ac295420dd..c010497cb2 100644\n--- a/credential-store.c\n+++ b/credential-store.c\n@@ -65,29 +65,30 @@ static void rewrite_credential_file(const char *fn, struct credential *c,\n \tparse_credential_file(fn, c, NULL, print_line);\n \tif (commit_lock_file(&credential_lock) < 0)\n \t\tdie_errno(\"unable to write credential store\");\n }\n \n static void store_credential_file(const char *fn, struct credential *c)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \n \tstrbuf_addf(&buf, \"%s://\", c->protocol);\n-\tstrbuf_addstr_urlencode(&buf, c->username, 1);\n+\tstrbuf_addstr_urlencode(&buf, c->username, is_rfc3986_unreserved);\n \tstrbuf_addch(&buf, ':');\n-\tstrbuf_addstr_urlencode(&buf, c->password, 1);\n+\tstrbuf_addstr_urlencode(&buf, c->password, is_rfc3986_unreserved);\n \tstrbuf_addch(&buf, '@');\n \tif (c->host)\n-\t\tstrbuf_addstr_urlencode(&buf, c->host, 1);\n+\t\tstrbuf_addstr_urlencode(&buf, c->host, is_rfc3986_unreserved);\n \tif (c->path) {\n \t\tstrbuf_addch(&buf, '/');\n-\t\tstrbuf_addstr_urlencode(&buf, c->path, 0);\n+\t\tstrbuf_addstr_urlencode(&buf, c->path,\n+\t\t\t\t\tis_rfc3986_reserved_or_unreserved);\n \t}\n \n \trewrite_credential_file(fn, c, &buf);\n \tstrbuf_release(&buf);\n }\n \n static void store_credential(const struct string_list *fns, struct credential *c)\n {\n \tstruct string_list_item *fn;\n \ndiff --git a/http.c b/http.c\nindex 27aa0a3192..938b9e55af 100644\n--- a/http.c\n+++ b/http.c\n@@ -506,23 +506,25 @@ static void var_override(const char **var, char *value)\n static void set_proxyauth_name_password(CURL *result)\n {\n #if LIBCURL_VERSION_NUM >= 0x071301\n \t\tcurl_easy_setopt(result, CURLOPT_PROXYUSERNAME,\n \t\t\tproxy_auth.username);\n \t\tcurl_easy_setopt(result, CURLOPT_PROXYPASSWORD,\n \t\t\tproxy_auth.password);\n #else\n \t\tstruct strbuf s = STRBUF_INIT;\n \n-\t\tstrbuf_addstr_urlencode(&s, proxy_auth.username, 1);\n+\t\tstrbuf_addstr_urlencode(&s, proxy_auth.username,\n+\t\t\t\t\tis_rfc3986_unreserved);\n \t\tstrbuf_addch(&s, ':');\n-\t\tstrbuf_addstr_urlencode(&s, proxy_auth.password, 1);\n+\t\tstrbuf_addstr_urlencode(&s, proxy_auth.password,\n+\t\t\t\t\tis_rfc3986_unreserved);\n \t\tcurl_proxyuserpwd = strbuf_detach(&s, NULL);\n \t\tcurl_easy_setopt(result, CURLOPT_PROXYUSERPWD, curl_proxyuserpwd);\n #endif\n }\n \n static void init_curl_proxy_auth(CURL *result)\n {\n \tif (proxy_auth.username) {\n \t\tif (!proxy_auth.password)\n \t\t\tcredential_fill(&proxy_auth);\ndiff --git a/strbuf.c b/strbuf.c\nindex 0e18b259ce..60ab5144f2 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -767,55 +767,56 @@ void strbuf_addstr_xml_quoted(struct strbuf *buf, const char *s)\n \t\tcase '&':\n \t\t\tstrbuf_addstr(buf, \"&\");\n \t\t\tbreak;\n \t\tcase 0:\n \t\t\treturn;\n \t\t}\n \t\ts++;\n \t}\n }\n \n-static int is_rfc3986_reserved(char ch)\n+int is_rfc3986_reserved_or_unreserved(char ch)\n {\n+\tif (is_rfc3986_unreserved(ch))\n+\t\treturn 1;\n \tswitch (ch) {\n \t\tcase '!': case '*': case '\\'': case '(': case ')': case ';':\n \t\tcase ':': case '@': case '&': case '=': case '+': case '$':\n \t\tcase ',': case '/': case '?': case '#': case '[': case ']':\n \t\t\treturn 1;\n \t}\n \treturn 0;\n }\n \n-static int is_rfc3986_unreserved(char ch)\n+int is_rfc3986_unreserved(char ch)\n {\n \treturn isalnum(ch) ||\n \t\tch == '-' || ch == '_' || ch == '.' || ch == '~';\n }\n \n static void strbuf_add_urlencode(struct strbuf *sb, const char *s, size_t len,\n-\t\t\t\t int reserved)\n+\t\t\t\t char_predicate allow_unencoded_fn)\n {\n \tstrbuf_grow(sb, len);\n \twhile (len--) {\n \t\tchar ch = *s++;\n-\t\tif (is_rfc3986_unreserved(ch) ||\n-\t\t (!reserved && is_rfc3986_reserved(ch)))\n+\t\tif (allow_unencoded_fn(ch))\n \t\t\tstrbuf_addch(sb, ch);\n \t\telse\n \t\t\tstrbuf_addf(sb, \"%%%02x\", (unsigned char)ch);\n \t}\n }\n \n void strbuf_addstr_urlencode(struct strbuf *sb, const char *s,\n-\t\t\t int reserved)\n+\t\t\t char_predicate allow_unencoded_fn)\n {\n-\tstrbuf_add_urlencode(sb, s, strlen(s), reserved);\n+\tstrbuf_add_urlencode(sb, s, strlen(s), allow_unencoded_fn);\n }\n \n void strbuf_humanise_bytes(struct strbuf *buf, off_t bytes)\n {\n \tif (bytes > 1 << 30) {\n \t\tstrbuf_addf(buf, \"%u.%2.2u GiB\",\n \t\t\t (unsigned)(bytes >> 30),\n \t\t\t (unsigned)(bytes & ((1 << 30) - 1)) / 10737419);\n \t} else if (bytes > 1 << 20) {\n \t\tunsigned x = bytes + 5243; /* for rounding */\ndiff --git a/strbuf.h b/strbuf.h\nindex c8d98dfb95..346d722492 100644\n--- a/strbuf.h\n+++ b/strbuf.h\n@@ -659,22 +659,27 @@ void strbuf_branchname(struct strbuf *sb, const char *name,\n \t\t unsigned allowed);\n \n /*\n * Like strbuf_branchname() above, but confirm that the result is\n * syntactically valid to be used as a local branch name in refs/heads/.\n *\n * The return value is \"0\" if the result is valid, and \"-1\" otherwise.\n */\n int strbuf_check_branch_ref(struct strbuf *sb, const char *name);\n \n+typedef int (*char_predicate)(char ch);\n+\n+int is_rfc3986_unreserved(char ch);\n+int is_rfc3986_reserved_or_unreserved(char ch);\n+\n void strbuf_addstr_urlencode(struct strbuf *sb, const char *name,\n-\t\t\t int reserved);\n+\t\t\t char_predicate allow_unencoded_fn);\n \n __attribute__((format (printf,1,2)))\n int printf_ln(const char *fmt, ...);\n __attribute__((format (printf,2,3)))\n int fprintf_ln(FILE *fp, const char *fmt, ...);\n \n char *xstrdup_tolower(const char *);\n char *xstrdup_toupper(const char *);\n \n /**\n-- \n2.21.0\n\n"},{"id":"378219","messageId":"a8c2e4c461066c0750ce4652daf7fe6fce1711a3.1561675151.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1561675151.git.matvore@google.com","subject":"[PATCH v5 08/10] list-objects-filter-options: allow mult. --filter","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-27T22:54:12Z","receivedAt":"2019-06-27T22:54:45Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Allow combining of multiple filters by simply repeating the --filter\nflag. Before this patch, the user had to combine them in a single flag\nsomewhat awkwardly (e.g. --filter=combine:FOO+BAR), including\nURL-encoding the individual filters.\n\nTo make this work, in the --filter flag parsing callback, rather than\nerror out when we detect that the filter_options struct is already\npopulated, we modify it in-place to contain the added sub-filter. The\nexisting sub-filter becomes the lhs of the combined filter, and the\nnext sub-filter becomes the rhs. We also have to URL-encode the LHS and\nRHS sub-filters.\n\nWe can simplify the operation if the LHS is already a combine: filter.\nIn that case, we just append the URL-encoded RHS sub-filter to the LHS\nspec to get the new spec.\n\nHelped-by: Emily Shaffer \nHelped-by: Jeff Hostetler \nHelped-by: Jeff King \nHelped-by: Junio C Hamano \nSigned-off-by: Matthew DeVore \n---\n Documentation/rev-list-options.txt | 16 ++++++\n list-objects-filter-options.c | 88 +++++++++++++++++++++++++++--\n list-objects-filter-options.h | 11 ++++\n t/t5616-partial-clone.sh | 19 +++++++\n t/t6112-rev-list-filters-objects.sh | 46 +++++++++++++--\n transport.c | 1 +\n upload-pack.c | 2 +\n 7 files changed, 173 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 71a1fcc093..d1f080bf6d 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -731,20 +731,36 @@ at multiple depths in the commits traversed). =0 will not include\n any trees or blobs unless included explicitly in the command-line (or\n standard input when --stdin is used). =1 will include only the\n tree and blobs which are referenced directly by a commit reachable from\n or an explicitly-given object. =2 is like =1\n while also including trees and blobs one more level removed from an\n explicitly-given commit or tree.\n +\n Note that the form '--filter=sparse:path=' that wants to read\n from an arbitrary path on the filesystem has been dropped for security\n reasons.\n++\n+Multiple '--filter=' flags can be specified to combine filters. Only\n+objects which are accepted by every filter are included.\n++\n+The form '--filter=combine:++...' can also be\n+used to combined several filters, but this is harder than just repeating\n+the '--filter' flag and is usually not necessary. Filters are joined by\n+'{plus}' and individual filters are %-encoded (i.e. URL-encoded).\n+Besides the '{plus}' and '%' characters, the following characters are\n+reserved and also must be encoded: `~!@#$^&*()[]{}\\;\",<>?`+'`+\n+as well as all characters with ASCII code <= `0x20`, which includes\n+space and newline.\n++\n+Other arbitrary characters can also be encoded. For instance,\n+'combine:tree:3+blob:none' and 'combine:tree%3A3+blob%3Anone' are\n+equivalent.\n \n --no-filter::\n \tTurn off any previous `--filter=` argument.\n \n --filter-print-omitted::\n \tOnly useful with `--filter=`; prints a list of the objects omitted\n \tby the filter. Object IDs are prefixed with a ``~'' character.\n \n --missing=::\n \tA debug option to help with future \"partial clone\" development.\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex 01c0f13346..2506dc8327 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -1,18 +1,19 @@\n #include \"cache.h\"\n #include \"commit.h\"\n #include \"config.h\"\n #include \"revision.h\"\n #include \"argv-array.h\"\n #include \"list-objects.h\"\n #include \"list-objects-filter.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"trace.h\"\n #include \"url.h\"\n \n static int parse_combine_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg,\n \tstruct strbuf *errbuf);\n \n /*\n * Parse value of the argument to the \"filter\" keyword.\n * On the command line this looks like:\n@@ -171,29 +172,106 @@ static int parse_combine_filter(\n \n cleanup:\n \tstrbuf_list_free(subspecs);\n \tif (result) {\n \t\tlist_objects_filter_release(filter_options);\n \t\tmemset(filter_options, 0, sizeof(*filter_options));\n \t}\n \treturn result;\n }\n \n-int parse_list_objects_filter(struct list_objects_filter_options *filter_options,\n-\t\t\t const char *arg)\n+static int allow_unencoded(char ch)\n+{\n+\tif (ch <= ' ' || ch == '%' || ch == '+')\n+\t\treturn 0;\n+\treturn !strchr(RESERVED_NON_WS, ch);\n+}\n+\n+static void filter_spec_append_urlencode(\n+\tstruct list_objects_filter_options *filter, const char *raw)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n+\tstrbuf_addstr_urlencode(&buf, raw, allow_unencoded);\n+\ttrace_printf(\"Add to combine filter-spec: %s\\n\", buf.buf);\n+\tstring_list_append(&filter->filter_spec, strbuf_detach(&buf, NULL));\n+}\n+\n+/*\n+ * Changes filter_options into an equivalent LOFC_COMBINE filter options\n+ * instance. Does not do anything if filter_options is already LOFC_COMBINE.\n+ */\n+static void transform_to_combine_type(\n+\tstruct list_objects_filter_options *filter_options)\n+{\n+\tassert(filter_options->choice);\n+\tif (filter_options->choice == LOFC_COMBINE)\n+\t\treturn;\n+\t{\n+\t\tconst int initial_sub_alloc = 2;\n+\t\tstruct list_objects_filter_options *sub_array =\n+\t\t\txcalloc(initial_sub_alloc, sizeof(*sub_array));\n+\t\tsub_array[0] = *filter_options;\n+\t\tmemset(filter_options, 0, sizeof(*filter_options));\n+\t\tfilter_options->sub = sub_array;\n+\t\tfilter_options->sub_alloc = initial_sub_alloc;\n+\t}\n+\tfilter_options->sub_nr = 1;\n+\tfilter_options->choice = LOFC_COMBINE;\n+\tstring_list_append(&filter_options->filter_spec, xstrdup(\"combine:\"));\n+\tfilter_spec_append_urlencode(\n+\t\tfilter_options,\n+\t\tlist_objects_filter_spec(&filter_options->sub[0]));\n+\t/*\n+\t * We don't need the filter_spec strings for subfilter specs, only the\n+\t * top level.\n+\t */\n+\tstring_list_clear(&filter_options->sub[0].filter_spec, /*free_util=*/0);\n+}\n+\n+void list_objects_filter_die_if_populated(\n+\tstruct list_objects_filter_options *filter_options)\n+{\n \tif (filter_options->choice)\n \t\tdie(_(\"multiple filter-specs cannot be combined\"));\n-\tstring_list_append(&filter_options->filter_spec, xstrdup(arg));\n-\tif (gently_parse_list_objects_filter(filter_options, arg, &buf))\n-\t\tdie(\"%s\", buf.buf);\n+}\n+\n+int parse_list_objects_filter(\n+\tstruct list_objects_filter_options *filter_options,\n+\tconst char *arg)\n+{\n+\tstruct strbuf errbuf = STRBUF_INIT;\n+\tint parse_error;\n+\n+\tif (!filter_options->choice) {\n+\t\tstring_list_append(&filter_options->filter_spec, xstrdup(arg));\n+\n+\t\tparse_error = gently_parse_list_objects_filter(\n+\t\t\tfilter_options, arg, &errbuf);\n+\t} else {\n+\t\t/*\n+\t\t * Make filter_options an LOFC_COMBINE spec so we can trivially\n+\t\t * add subspecs to it.\n+\t\t */\n+\t\ttransform_to_combine_type(filter_options);\n+\n+\t\tstring_list_append(&filter_options->filter_spec, xstrdup(\"+\"));\n+\t\tfilter_spec_append_urlencode(filter_options, arg);\n+\t\tALLOC_GROW(filter_options->sub, filter_options->sub_nr + 1,\n+\t\t\t filter_options->sub_alloc);\n+\t\tfilter_options = &filter_options->sub[filter_options->sub_nr++];\n+\t\tmemset(filter_options, 0, sizeof(*filter_options));\n+\n+\t\tparse_error = gently_parse_list_objects_filter(\n+\t\t\tfilter_options, arg, &errbuf);\n+\t}\n+\tif (parse_error)\n+\t\tdie(\"%s\", errbuf.buf);\n \treturn 0;\n }\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct list_objects_filter_options *filter_options = opt->value;\n \n \tif (unset || !arg) {\n \t\tlist_objects_filter_set_no_filter(filter_options);\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex bb33303f9b..d8bc7e946e 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -56,20 +56,31 @@ struct list_objects_filter_options {\n \tstruct list_objects_filter_options *sub;\n \n \t/*\n \t * END choice-specific parsed values.\n \t */\n };\n \n /* Normalized command line arguments */\n #define CL_ARG__FILTER \"filter\"\n \n+void list_objects_filter_die_if_populated(\n+\tstruct list_objects_filter_options *filter_options);\n+\n+/*\n+ * Parses the filter spec string given by arg and either (1) simply places the\n+ * result in filter_options if it is not yet populated or (2) combines it with\n+ * the filter already in filter_options if it is already populated. In the case\n+ * of (2), the filter specs are combined as if specified with 'combine:'.\n+ *\n+ * Dies and prints a user-facing message if an error occurs.\n+ */\n int parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg);\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset);\n \n #define OPT_PARSE_LIST_OBJECTS_FILTER(fo) \\\n \t{ OPTION_CALLBACK, 0, CL_ARG__FILTER, fo, N_(\"args\"), \\\n \t N_(\"object filtering\"), 0, \\\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex b91ef548f8..32b7d72f3c 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -201,20 +201,39 @@ test_expect_success 'use fsck before and after manually fetching a missing subtr\n \ttest_line_count = 70 fetched_objects &&\n \n \tawk -f print_1.awk fetched_objects |\n \txargs -n1 git -C dst cat-file -t >fetched_types &&\n \n \tsort -u fetched_types >unique_types.observed &&\n \ttest_write_lines blob commit tree >unique_types.expected &&\n \ttest_cmp unique_types.expected unique_types.observed\n '\n \n+test_expect_success 'implicitly construct combine: filter with repeated flags' '\n+\tGIT_TRACE=$(pwd)/trace git clone --bare \\\n+\t\t--filter=blob:none --filter=tree:1 \\\n+\t\t\"file://$(pwd)/srv.bare\" pc2 &&\n+\tgrep \"trace:.* git pack-objects .*--filter=combine:blob:none+tree:1\" \\\n+\t\ttrace &&\n+\tgit -C pc2 rev-list --objects --missing=allow-any HEAD >objects &&\n+\n+\t# We should have gotten some root trees.\n+\tgrep \" $\" objects &&\n+\t# Should not have gotten any non-root trees or blobs.\n+\t! grep \" .\" objects &&\n+\n+\txargs -n 1 git -C pc2 cat-file -t types &&\n+\tsort -u types >unique_types.actual &&\n+\ttest_write_lines commit tree >unique_types.expected &&\n+\ttest_cmp unique_types.expected unique_types.actual\n+'\n+\n test_expect_success 'partial clone fetches blobs pointed to by refs even if normally filtered out' '\n \trm -rf src dst &&\n \tgit init src &&\n \ttest_commit -C src x &&\n \ttest_config -C src uploadpack.allowfilter 1 &&\n \ttest_config -C src uploadpack.allowanysha1inwant 1 &&\n \n \t# Create a tag pointing to a blob.\n \tBLOB=$(echo blob-contents | git -C src hash-object --stdin -w) &&\n \tgit -C src tag myblob \"$BLOB\" &&\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex 27ba15719a..de0e5a5d36 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -344,21 +344,30 @@ test_expect_success 'verify tree:3 includes everything expected' '\n \n test_expect_success 'combine:... for a simple combination' '\n \tgit -C r3 rev-list --objects --filter=combine:tree:2+blob:none HEAD \\\n \t\t>actual &&\n \n \texpect_has HEAD \"\" &&\n \texpect_has HEAD~1 \"\" &&\n \texpect_has HEAD dir1 &&\n \n \t# There are also 2 commit objects\n-\ttest_line_count = 5 actual\n+\ttest_line_count = 5 actual &&\n+\n+\tcp actual expected &&\n+\n+\t# Try again using repeated --filter - this is equivalent to a manual\n+\t# combine with \"combine:...+...\"\n+\tgit -C r3 rev-list --objects --filter=combine:tree:2 \\\n+\t\t--filter=blob:none HEAD >actual &&\n+\n+\ttest_cmp expected actual\n '\n \n test_expect_success 'combine:... with URL encoding' '\n \tgit -C r3 rev-list --objects \\\n \t\t--filter=combine:tree%3a2+blob:%6Eon%65 HEAD >actual &&\n \n \texpect_has HEAD \"\" &&\n \texpect_has HEAD~1 \"\" &&\n \texpect_has HEAD dir1 &&\n \n@@ -410,24 +419,26 @@ test_expect_success 'combine:... with edge-case hex digits: Ff Aa 0 9' '\n \tgit -C r3 rev-list --objects --filter=\"combine:tree%3A2+blob%3anone\" \\\n \t\tHEAD >actual &&\n \ttest_line_count = 5 actual &&\n \tgit -C r3 rev-list --objects --filter=\"combine:tree:%30\" HEAD >actual &&\n \ttest_line_count = 2 actual &&\n \tgit -C r3 rev-list --objects --filter=\"combine:tree:%39+blob:none\" \\\n \t\tHEAD >actual &&\n \ttest_line_count = 5 actual\n '\n \n-test_expect_success 'add a sparse pattern blob whose path has reserved chars' '\n+test_expect_success 'add sparse pattern blobs whose paths have reserved chars' '\n \tcp r3/pattern r3/pattern1+renamed% &&\n-\tgit -C r3 add pattern1+renamed% &&\n-\tgit -C r3 commit -m \"add sparse pattern file with reserved chars\"\n+\tcp r3/pattern \"r3/p;at%ter+n\" &&\n+\tcp r3/pattern r3/^~pattern &&\n+\tgit -C r3 add pattern1+renamed% \"p;at%ter+n\" ^~pattern &&\n+\tgit -C r3 commit -m \"add sparse pattern files with reserved chars\"\n '\n \n test_expect_success 'combine:... with more than two sub-filters' '\n \tgit -C r3 rev-list --objects \\\n \t\t--filter=combine:tree:3+blob:limit=40+sparse:oid=master:pattern \\\n \t\tHEAD >actual &&\n \n \texpect_has HEAD \"\" &&\n \texpect_has HEAD~1 \"\" &&\n \texpect_has HEAD~2 \"\" &&\n@@ -438,21 +449,46 @@ test_expect_success 'combine:... with more than two sub-filters' '\n \t# Should also have 3 commits\n \ttest_line_count = 9 actual &&\n \n \t# Try again, this time making sure the last sub-filter is only\n \t# URL-decoded once.\n \tcp actual expect &&\n \n \tgit -C r3 rev-list --objects \\\n \t\t--filter=combine:tree:3+blob:limit=40+sparse:oid=master:pattern1%2brenamed%25 \\\n \t\tHEAD >actual &&\n-\ttest_cmp expect actual\n+\ttest_cmp expect actual &&\n+\n+\t# Use the same composite filter again, but with a pattern file name that\n+\t# requires encoding multiple characters, and use implicit filter\n+\t# combining.\n+\ttest_when_finished \"rm -f trace1\" &&\n+\tGIT_TRACE=$(pwd)/trace1 git -C r3 rev-list --objects \\\n+\t\t--filter=tree:3 --filter=blob:limit=40 \\\n+\t\t--filter=sparse:oid=\"master:p;at%ter+n\" \\\n+\t\tHEAD >actual &&\n+\n+\ttest_cmp expect actual &&\n+\tgrep \"Add to combine filter-spec: sparse:oid=master:p%3bat%25ter%2bn\" \\\n+\t\ttrace1 &&\n+\n+\t# Repeat the above test, but this time, the characters to encode are in\n+\t# the LHS of the combined filter.\n+\ttest_when_finished \"rm -f trace2\" &&\n+\tGIT_TRACE=$(pwd)/trace2 git -C r3 rev-list --objects \\\n+\t\t--filter=sparse:oid=master:^~pattern \\\n+\t\t--filter=tree:3 --filter=blob:limit=40 \\\n+\t\tHEAD >actual &&\n+\n+\ttest_cmp expect actual &&\n+\tgrep \"Add to combine filter-spec: sparse:oid=master:%5e%7epattern\" \\\n+\t\ttrace2\n '\n \n # Test provisional omit collection logic with a repo that has objects appearing\n # at multiple depths - first deeper than the filter's threshold, then shallow.\n \n test_expect_success 'setup r4' '\n \tgit init r4 &&\n \n \techo foo > r4/foo &&\n \tmkdir r4/subdir &&\ndiff --git a/transport.c b/transport.c\nindex f1fcd2c4b0..ee7dd1c062 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -217,20 +217,21 @@ static int set_git_option(struct git_transport_options *opts,\n \t} else if (!strcmp(name, TRANS_OPT_DEEPEN_RELATIVE)) {\n \t\topts->deepen_relative = !!value;\n \t\treturn 0;\n \t} else if (!strcmp(name, TRANS_OPT_FROM_PROMISOR)) {\n \t\topts->from_promisor = !!value;\n \t\treturn 0;\n \t} else if (!strcmp(name, TRANS_OPT_NO_DEPENDENTS)) {\n \t\topts->no_dependents = !!value;\n \t\treturn 0;\n \t} else if (!strcmp(name, TRANS_OPT_LIST_OBJECTS_FILTER)) {\n+\t\tlist_objects_filter_die_if_populated(&opts->filter_options);\n \t\tparse_list_objects_filter(&opts->filter_options, value);\n \t\treturn 0;\n \t}\n \treturn 1;\n }\n \n static int connect_setup(struct transport *transport, int for_push)\n {\n \tstruct git_transport_data *data = transport->data;\n \tint flags = transport->verbose > 0 ? CONNECT_VERBOSE : 0;\ndiff --git a/upload-pack.c b/upload-pack.c\nindex d404d88941..f8a76ebda3 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -876,20 +876,21 @@ static void receive_needs(struct packet_reader *reader, struct object_array *wan\n \t\tif (process_deepen(reader->line, &depth))\n \t\t\tcontinue;\n \t\tif (process_deepen_since(reader->line, &deepen_since, &deepen_rev_list))\n \t\t\tcontinue;\n \t\tif (process_deepen_not(reader->line, &deepen_not, &deepen_rev_list))\n \t\t\tcontinue;\n \n \t\tif (skip_prefix(reader->line, \"filter \", &arg)) {\n \t\t\tif (!filter_capability_requested)\n \t\t\t\tdie(\"git upload-pack: filtering capability not negotiated\");\n+\t\t\tlist_objects_filter_die_if_populated(&filter_options);\n \t\t\tparse_list_objects_filter(&filter_options, arg);\n \t\t\tcontinue;\n \t\t}\n \n \t\tif (!skip_prefix(reader->line, \"want \", &arg) ||\n \t\t parse_oid_hex(arg, &oid_buf, &features))\n \t\t\tdie(\"git upload-pack: protocol error, \"\n \t\t\t \"expected to get object ID, not '%s'\", reader->line);\n \n \t\tif (parse_feature_request(features, \"deepen-relative\"))\n@@ -1297,20 +1298,21 @@ static void process_args(struct packet_reader *request,\n \t\t\tcontinue;\n \t\tif (process_deepen_not(arg, &data->deepen_not,\n \t\t\t\t &data->deepen_rev_list))\n \t\t\tcontinue;\n \t\tif (!strcmp(arg, \"deepen-relative\")) {\n \t\t\tdata->deepen_relative = 1;\n \t\t\tcontinue;\n \t\t}\n \n \t\tif (allow_filter && skip_prefix(arg, \"filter \", &p)) {\n+\t\t\tlist_objects_filter_die_if_populated(&filter_options);\n \t\t\tparse_list_objects_filter(&filter_options, p);\n \t\t\tcontinue;\n \t\t}\n \n \t\tif ((git_env_bool(\"GIT_TEST_SIDEBAND_ALL\", 0) ||\n \t\t allow_sideband_all) &&\n \t\t !strcmp(arg, \"sideband-all\")) {\n \t\t\tdata->writer.use_sideband = 1;\n \t\t\tcontinue;\n \t\t}\n-- \n2.21.0\n\n"},{"id":"378220","messageId":"0af31656ccceefbf9eff82506b89c613613cef94.1561675151.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1561675151.git.matvore@google.com","subject":"[PATCH v5 09/10] list-objects-filter-options: clean up use of ALLOC_GROW","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-27T22:54:13Z","receivedAt":"2019-06-27T22:54:48Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"Introduce a new macro ALLOC_GROW_BY which automatically zeros the added\narray elements and takes care of updating the nr value. Use the macro in\ncode introduced earlier in this patchset.\n\nSigned-off-by: Matthew DeVore \n---\n cache.h | 22 ++++++++++++++++++++++\n list-objects-filter-options.c | 17 +++++++----------\n 2 files changed, 29 insertions(+), 10 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex bf20337ef4..cf5d70c196 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -653,33 +653,55 @@ int init_db(const char *git_dir, const char *real_git_dir,\n void sanitize_stdfds(void);\n int daemonize(void);\n \n #define alloc_nr(x) (((x)+16)*3/2)\n \n /*\n * Realloc the buffer pointed at by variable 'x' so that it can hold\n * at least 'nr' entries; the number of entries currently allocated\n * is 'alloc', using the standard growing factor alloc_nr() macro.\n *\n+ * Consider using ALLOC_GROW_BY instead of ALLOC_GROW as it has some\n+ * added niceties.\n+ *\n * DO NOT USE any expression with side-effect for 'x', 'nr', or 'alloc'.\n */\n #define ALLOC_GROW(x, nr, alloc) \\\n \tdo { \\\n \t\tif ((nr) > alloc) { \\\n \t\t\tif (alloc_nr(alloc) < (nr)) \\\n \t\t\t\talloc = (nr); \\\n \t\t\telse \\\n \t\t\t\talloc = alloc_nr(alloc); \\\n \t\t\tREALLOC_ARRAY(x, alloc); \\\n \t\t} \\\n \t} while (0)\n \n+/*\n+ * Similar to ALLOC_GROW but handles updating of the nr value and\n+ * zeroing the bytes of the newly-grown array elements.\n+ *\n+ * DO NOT USE any expression with side-effect for any of the\n+ * arguments.\n+ */\n+#define ALLOC_GROW_BY(x, nr, increase, alloc) \\\n+\tdo { \\\n+\t\tif (increase) { \\\n+\t\t\tsize_t new_nr = nr + (increase); \\\n+\t\t\tif (new_nr < nr) \\\n+\t\t\t\tBUG(\"negative growth in ALLOC_GROW_BY\"); \\\n+\t\t\tALLOC_GROW(x, new_nr, alloc); \\\n+\t\t\tmemset((x) + nr, 0, sizeof(*(x)) * (increase)); \\\n+\t\t\tnr = new_nr; \\\n+\t\t} \\\n+\t} while (0)\n+\n /* Initialize and use the cache information */\n struct lock_file;\n void preload_index(struct index_state *index,\n \t\t const struct pathspec *pathspec,\n \t\t unsigned int refresh_flags);\n int do_read_index(struct index_state *istate, const char *path,\n \t\t int must_exist); /* for testting only! */\n int read_index_from(struct index_state *, const char *path,\n \t\t const char *gitdir);\n int is_index_unborn(struct index_state *);\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex 2506dc8327..44bc1153d1 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -113,28 +113,26 @@ static int has_reserved_character(\n \t}\n \n \treturn 0;\n }\n \n static int parse_combine_subfilter(\n \tstruct list_objects_filter_options *filter_options,\n \tstruct strbuf *subspec,\n \tstruct strbuf *errbuf)\n {\n-\tsize_t new_index = filter_options->sub_nr++;\n+\tsize_t new_index = filter_options->sub_nr;\n \tchar *decoded;\n \tint result;\n \n-\tALLOC_GROW(filter_options->sub, filter_options->sub_nr,\n-\t\t filter_options->sub_alloc);\n-\tmemset(&filter_options->sub[new_index], 0,\n-\t sizeof(*filter_options->sub));\n+\tALLOC_GROW_BY(filter_options->sub, filter_options->sub_nr, 1,\n+\t\t filter_options->sub_alloc);\n \n \tdecoded = url_percent_decode(subspec->buf);\n \n \tresult = has_reserved_character(subspec, errbuf) ||\n \t\tgently_parse_list_objects_filter(\n \t\t\t&filter_options->sub[new_index], decoded, errbuf);\n \n \tfree(decoded);\n \treturn result;\n }\n@@ -248,27 +246,26 @@ int parse_list_objects_filter(\n \t\t\tfilter_options, arg, &errbuf);\n \t} else {\n \t\t/*\n \t\t * Make filter_options an LOFC_COMBINE spec so we can trivially\n \t\t * add subspecs to it.\n \t\t */\n \t\ttransform_to_combine_type(filter_options);\n \n \t\tstring_list_append(&filter_options->filter_spec, xstrdup(\"+\"));\n \t\tfilter_spec_append_urlencode(filter_options, arg);\n-\t\tALLOC_GROW(filter_options->sub, filter_options->sub_nr + 1,\n-\t\t\t filter_options->sub_alloc);\n-\t\tfilter_options = &filter_options->sub[filter_options->sub_nr++];\n-\t\tmemset(filter_options, 0, sizeof(*filter_options));\n+\t\tALLOC_GROW_BY(filter_options->sub, filter_options->sub_nr, 1,\n+\t\t\t filter_options->sub_alloc);\n \n \t\tparse_error = gently_parse_list_objects_filter(\n-\t\t\tfilter_options, arg, &errbuf);\n+\t\t\t&filter_options->sub[filter_options->sub_nr - 1], arg,\n+\t\t\t&errbuf);\n \t}\n \tif (parse_error)\n \t\tdie(\"%s\", errbuf.buf);\n \treturn 0;\n }\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct list_objects_filter_options *filter_options = opt->value;\n-- \n2.21.0\n\n"},{"id":"378221","messageId":"c0b9243ab488633245153782da9820027790c533.1561675151.git.matvore@google.com","threadId":"51217","inReplyTo":"cover.1561675151.git.matvore@google.com","subject":"[PATCH v5 10/10] list-objects-filter-options: make parser void","fromName":"Matthew DeVore","fromEmail":"matvore@google.com","sentAt":"2019-06-27T22:54:14Z","receivedAt":"2019-06-27T22:54:50Z","isPatch":true,"sender":{"key":"matvore@google.com","avatar":"https://avatars.githubusercontent.com/u/946637?v=4"},"body":"This function always returns 0, so make it return void instead.\n\nSigned-off-by: Matthew DeVore \n---\n list-objects-filter-options.c | 12 +++++-------\n list-objects-filter-options.h | 2 +-\n 2 files changed, 6 insertions(+), 8 deletions(-)\n\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex 44bc1153d1..ba1425cb4a 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -225,21 +225,21 @@ static void transform_to_combine_type(\n \tstring_list_clear(&filter_options->sub[0].filter_spec, /*free_util=*/0);\n }\n \n void list_objects_filter_die_if_populated(\n \tstruct list_objects_filter_options *filter_options)\n {\n \tif (filter_options->choice)\n \t\tdie(_(\"multiple filter-specs cannot be combined\"));\n }\n \n-int parse_list_objects_filter(\n+void parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg)\n {\n \tstruct strbuf errbuf = STRBUF_INIT;\n \tint parse_error;\n \n \tif (!filter_options->choice) {\n \t\tstring_list_append(&filter_options->filter_spec, xstrdup(arg));\n \n \t\tparse_error = gently_parse_list_objects_filter(\n@@ -255,34 +255,32 @@ int parse_list_objects_filter(\n \t\tfilter_spec_append_urlencode(filter_options, arg);\n \t\tALLOC_GROW_BY(filter_options->sub, filter_options->sub_nr, 1,\n \t\t\t filter_options->sub_alloc);\n \n \t\tparse_error = gently_parse_list_objects_filter(\n \t\t\t&filter_options->sub[filter_options->sub_nr - 1], arg,\n \t\t\t&errbuf);\n \t}\n \tif (parse_error)\n \t\tdie(\"%s\", errbuf.buf);\n-\treturn 0;\n }\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n {\n \tstruct list_objects_filter_options *filter_options = opt->value;\n \n-\tif (unset || !arg) {\n+\tif (unset || !arg)\n \t\tlist_objects_filter_set_no_filter(filter_options);\n-\t\treturn 0;\n-\t}\n-\n-\treturn parse_list_objects_filter(filter_options, arg);\n+\telse\n+\t\tparse_list_objects_filter(filter_options, arg);\n+\treturn 0;\n }\n \n const char *list_objects_filter_spec(struct list_objects_filter_options *filter)\n {\n \tif (!filter->filter_spec.nr)\n \t\tBUG(\"no filter_spec available for this filter\");\n \tif (filter->filter_spec.nr != 1) {\n \t\tstruct strbuf concatted = STRBUF_INIT;\n \t\tstrbuf_add_separated_string_list(\n \t\t\t&concatted, \"\", &filter->filter_spec);\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex d8bc7e946e..db37dfb34a 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -67,21 +67,21 @@ void list_objects_filter_die_if_populated(\n \tstruct list_objects_filter_options *filter_options);\n \n /*\n * Parses the filter spec string given by arg and either (1) simply places the\n * result in filter_options if it is not yet populated or (2) combines it with\n * the filter already in filter_options if it is already populated. In the case\n * of (2), the filter specs are combined as if specified with 'combine:'.\n *\n * Dies and prints a user-facing message if an error occurs.\n */\n-int parse_list_objects_filter(\n+void parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg);\n \n int opt_parse_list_objects_filter(const struct option *opt,\n \t\t\t\t const char *arg, int unset);\n \n #define OPT_PARSE_LIST_OBJECTS_FILTER(fo) \\\n \t{ OPTION_CALLBACK, 0, CL_ARG__FILTER, fo, N_(\"args\"), \\\n \t N_(\"object filtering\"), 0, \\\n \t opt_parse_list_objects_filter }\n-- \n2.21.0\n\n"},{"id":"378272","messageId":"xmqq36jtae5w.fsf@gitster-ct.c.googlers.com","threadId":"51217","inReplyTo":"cover.1561675151.git.matvore@google.com","subject":"Re: [PATCH v5 00/10] Filter combination","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-28T16:05:31Z","receivedAt":"2019-06-28T16:05:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthew DeVore writes:\n\n> This applies suggestions made by Jonathan Tan, as well as fixes a\n> Coccinelle-breaking error in strbuf usage, and makes an additional string\n> localizable.\n\nOK, so the convention is that errbuf has already translated string,\nso the call to die() made in a fairly high point in the callchain do\nnot use _(). I looked at the new _() in deeper level of the\ncallchain this round adds relative to the previous one, and they all\nlooked sensible.\n\nThe helper function should_delegate is now gone---it had only one\ncallsite in the previous round, so I suspect the resulting code\nwould be the same either way with moderately decent compilers.\n\nWill queue; thanks.\n"},{"id":"378277","messageId":"20190628171632.114453-1-jonathantanmy@google.com","threadId":"51217","inReplyTo":"cover.1561675151.git.matvore@google.com","subject":"Re: [PATCH v5 00/10] Filter combination","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2019-06-28T17:16:32Z","receivedAt":"2019-06-28T17:16:38Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> This applies suggestions made by Jonathan Tan, as well as fixes a\n> Coccinelle-breaking error in strbuf usage, and makes an additional string\n> localizable.\n> \n> Thanks,\n> \n> Matthew DeVore (10):\n> list-objects-filter: encapsulate filter components\n> list-objects-filter: put omits set in filter struct\n> list-objects-filter-options: always supply *errbuf\n> list-objects-filter: implement composite filters\n> list-objects-filter-options: move error check up\n> list-objects-filter-options: make filter_spec a string_list\n> strbuf: give URL-encoding API a char predicate fn\n> list-objects-filter-options: allow mult. --filter\n> list-objects-filter-options: clean up use of ALLOC_GROW\n> list-objects-filter-options: make parser void\n\nThanks - the range-diff looks good to me. (I generated the range diff using a\nbase of a6a95cd1b4 for v4 and 8dca754b1 for v5.)\n"}]}