{"thread":{"id":"55236","subject":"[PATCH 3/7] list-objects: support filtering by tag and commit","startedAt":"2021-03-01T12:21:35Z","lastAt":"2021-04-28T02:18:51Z","messageCount":97,"participants":["Patrick Steinhardt","Jeff King","Taylor Blau","Junio C Hamano","Philip Oakley","Ævar Arnfjörð Bjarmason","Andreas Schwab","Ramsay Jones"],"isPatch":true,"patchVersion":1,"patchTotal":7},"messages":[{"id":"418026","messageId":"4aa13ee83fa81ef5cfe6da5810202400493a3008.1614600555.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1614600555.git.ps@pks.im","subject":"[PATCH 3/7] list-objects: support filtering by tag and commit","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-03-01T12:20:42Z","receivedAt":"2021-03-01T12:21:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Object filters currently only support filtering blobs or trees based on\nsome criteria. This commit lays the foundation to also allow filtering\nof tags and commits.\n\nNo change in behaviour is expected from this commit given that there are\nno filters yet for those object types.\n\nSigned-off-by: Patrick Steinhardt \n---\n list-objects-filter.c | 40 ++++++++++++++++++++++++++++++++++++++++\n list-objects-filter.h | 2 ++\n list-objects.c | 24 +++++++++++++++++++++---\n 3 files changed, 63 insertions(+), 3 deletions(-)\n\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex 4ec0041cfb..7def039435 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -82,6 +82,16 @@ static enum list_objects_filter_result filter_blobs_none(\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n+\tcase LOFS_TAG:\n+\t\tassert(obj->type == OBJ_TAG);\n+\t\t/* always include all tag objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\tcase LOFS_COMMIT:\n+\t\tassert(obj->type == OBJ_COMMIT);\n+\t\t/* always include all commit objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\t/* always include all tree objects */\n@@ -173,6 +183,16 @@ static enum list_objects_filter_result filter_trees_depth(\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n+\tcase LOFS_TAG:\n+\t\tassert(obj->type == OBJ_TAG);\n+\t\t/* always include all tag objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\tcase LOFS_COMMIT:\n+\t\tassert(obj->type == OBJ_COMMIT);\n+\t\t/* always include all commit objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n \tcase LOFS_END_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\tfilter_data->current_depth--;\n@@ -267,6 +287,16 @@ static enum list_objects_filter_result filter_blobs_limit(\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n+\tcase LOFS_TAG:\n+\t\tassert(obj->type == OBJ_TAG);\n+\t\t/* always include all tag objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\tcase LOFS_COMMIT:\n+\t\tassert(obj->type == OBJ_COMMIT);\n+\t\t/* always include all commit objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\t/* always include all tree objects */\n@@ -371,6 +401,16 @@ static enum list_objects_filter_result filter_sparse(\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n+\tcase LOFS_TAG:\n+\t\tassert(obj->type == OBJ_TAG);\n+\t\t/* always include all tag objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\tcase LOFS_COMMIT:\n+\t\tassert(obj->type == OBJ_COMMIT);\n+\t\t/* always include all commit objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\tdtype = DT_DIR;\ndiff --git a/list-objects-filter.h b/list-objects-filter.h\nindex cfd784e203..9e98814111 100644\n--- a/list-objects-filter.h\n+++ b/list-objects-filter.h\n@@ -55,6 +55,8 @@ enum list_objects_filter_result {\n };\n \n enum list_objects_filter_situation {\n+\tLOFS_COMMIT,\n+\tLOFS_TAG,\n \tLOFS_BEGIN_TREE,\n \tLOFS_END_TREE,\n \tLOFS_BLOB\ndiff --git a/list-objects.c b/list-objects.c\nindex 093adf85b1..3b63dfd4f2 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -218,8 +218,16 @@ static void process_tag(struct traversal_context *ctx,\n \t\t\tstruct strbuf *base,\n \t\t\tconst char *name)\n {\n-\ttag->object.flags |= SEEN;\n-\tctx->show_object(&tag->object, name, ctx->show_data);\n+\tenum list_objects_filter_result r;\n+\n+\tr = list_objects_filter__filter_object(ctx->revs->repo, LOFS_TAG,\n+\t\t\t\t\t &tag->object, base->buf,\n+\t\t\t\t\t &base->buf[base->len],\n+\t\t\t\t\t ctx->filter);\n+\tif (r & LOFR_MARK_SEEN)\n+\t\ttag->object.flags |= SEEN;\n+\tif (r & LOFR_DO_SHOW)\n+\t\tctx->show_object(&tag->object, name, ctx->show_data);\n }\n \n static void mark_edge_parents_uninteresting(struct commit *commit,\n@@ -369,6 +377,12 @@ static void do_traverse(struct traversal_context *ctx)\n \tstrbuf_init(&csp, PATH_MAX);\n \n \twhile ((commit = get_revision(ctx->revs)) != NULL) {\n+\t\tenum list_objects_filter_result r;\n+\n+\t\tr = list_objects_filter__filter_object(ctx->revs->repo,\n+\t\t\t\tLOFS_COMMIT, &commit->object,\n+\t\t\t\tNULL, NULL, ctx->filter);\n+\n \t\t/*\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n@@ -383,7 +397,11 @@ static void do_traverse(struct traversal_context *ctx)\n \t\t\tdie(_(\"unable to load root tree for commit %s\"),\n \t\t\t oid_to_hex(&commit->object.oid));\n \t\t}\n-\t\tctx->show_commit(commit, ctx->show_data);\n+\n+\t\tif (r & LOFR_MARK_SEEN)\n+\t\t\tcommit->object.flags |= SEEN;\n+\t\tif (r & LOFR_DO_SHOW)\n+\t\t\tctx->show_commit(commit, ctx->show_data);\n \n \t\tif (ctx->revs->tree_blobs_in_commit_order)\n \t\t\t/*\n-- \n2.30.1\n\n"},{"id":"418027","messageId":"01b9fdbb9c0ca8d70281422a13f7f4df85ffed90.1614600555.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1614600555.git.ps@pks.im","subject":"[PATCH 4/7] list-objects: implement object type filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-03-01T12:20:47Z","receivedAt":"2021-03-01T12:21:40Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"While it already is possible to filter objects by some criteria in\ngit-rev-list(1), it is not yet possible to filter out only a specific\ntype of objects. This makes some filters less useful. The `blob:limit`\nfilter for example filters blobs such that only those which are smaller\nthan the given limit are returned. But it is unfit to ask only for these\nsmallish blobs, given that git-rev-list(1) will continue to print tags,\ncommits and trees.\n\nNow that we have the infrastructure in place to also filter tags and\ncommits, we can improve this situation by implementing a new filter\nwhich selects objects based on their type. Above query can thus\ntrivially be implemented with the following command:\n\n $ git rev-list --objects --filter=object:type=blob \\\n --filter=blob:limit=200\n\nFurthermore, this filter allows to optimize for certain other cases: if\nfor example only tags or commits have been selected, there is no need to\nwalk down trees.\n\nThe new filter is not yet supported in bitmaps. This is going to be\nimplemented in a subsequent commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/rev-list-options.txt | 3 ++\n list-objects-filter-options.c | 14 ++++++\n list-objects-filter-options.h | 2 +\n list-objects-filter.c | 76 +++++++++++++++++++++++++++++\n t/t6112-rev-list-filters-objects.sh | 48 ++++++++++++++++++\n 5 files changed, 143 insertions(+)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex b1c8f86c6e..3afa8fffbd 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -892,6 +892,9 @@ or units. n may be zero. The suffixes k, m, and g can be used to name\n units in KiB, MiB, or GiB. For example, 'blob:limit=1k' is the same\n as 'blob:limit=1024'.\n +\n+The form '--filter=object:type=(tag|commit|tree|blob)' omits all objects\n+which are not of the requested type.\n++\n The form '--filter=sparse:oid=' uses a sparse-checkout\n specification contained in the blob (or blob-expression) ''\n to omit blobs that would not be not required for a sparse checkout on\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex d2d1c81caf..bb6f6577d5 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -29,6 +29,8 @@ const char *list_object_filter_config_name(enum list_objects_filter_choice c)\n \t\treturn \"tree\";\n \tcase LOFC_SPARSE_OID:\n \t\treturn \"sparse:oid\";\n+\tcase LOFC_OBJECT_TYPE:\n+\t\treturn \"object:type\";\n \tcase LOFC_COMBINE:\n \t\treturn \"combine\";\n \tcase LOFC__COUNT:\n@@ -97,6 +99,18 @@ static int gently_parse_list_objects_filter(\n \t\t}\n \t\treturn 1;\n \n+\t} else if (skip_prefix(arg, \"object:type=\", &v0)) {\n+\t\tint type = type_from_string_gently(v0, -1, 1);\n+\t\tif (type < 0) {\n+\t\t\tstrbuf_addstr(errbuf, _(\"expected 'object:type='\"));\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\tfilter_options->object_type = type;\n+\t\tfilter_options->choice = LOFC_OBJECT_TYPE;\n+\n+\t\treturn 0;\n+\n \t} else if (skip_prefix(arg, \"combine:\", &v0)) {\n \t\treturn parse_combine_filter(filter_options, v0, errbuf);\n \ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 01767c3c96..4d0d0588cc 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -13,6 +13,7 @@ enum list_objects_filter_choice {\n \tLOFC_BLOB_LIMIT,\n \tLOFC_TREE_DEPTH,\n \tLOFC_SPARSE_OID,\n+\tLOFC_OBJECT_TYPE,\n \tLOFC_COMBINE,\n \tLOFC__COUNT /* must be last */\n };\n@@ -54,6 +55,7 @@ struct list_objects_filter_options {\n \tchar *sparse_oid_name;\n \tunsigned long blob_limit_value;\n \tunsigned long tree_exclude_depth;\n+\tenum object_type object_type;\n \n \t/* LOFC_COMBINE values */\n \ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex 7def039435..650a7c2c80 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -545,6 +545,81 @@ static void filter_sparse_oid__init(\n \tfilter->free_fn = filter_sparse_free;\n }\n \n+/*\n+ * A filter for list-objects to omit large blobs.\n+ * And to OPTIONALLY collect a list of the omitted OIDs.\n+ */\n+struct filter_object_type_data {\n+\tenum object_type object_type;\n+};\n+\n+static enum list_objects_filter_result filter_object_type(\n+\tstruct repository *r,\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tstruct oidset *omits,\n+\tvoid *filter_data_)\n+{\n+\tstruct filter_object_type_data *filter_data = filter_data_;\n+\n+\tswitch (filter_situation) {\n+\tdefault:\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n+\n+\tcase LOFS_TAG:\n+\t\tassert(obj->type == OBJ_TAG);\n+\t\tif (filter_data->object_type == OBJ_TAG)\n+\t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\t\treturn LOFR_MARK_SEEN;\n+\n+\tcase LOFS_COMMIT:\n+\t\tassert(obj->type == OBJ_COMMIT);\n+\t\tif (filter_data->object_type == OBJ_COMMIT)\n+\t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\t\treturn LOFR_MARK_SEEN;\n+\n+\tcase LOFS_BEGIN_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\n+\t\t/*\n+\t\t * If we only want to show commits or tags, then there is no\n+\t\t * need to walk down trees.\n+\t\t */\n+\t\tif (filter_data->object_type == OBJ_COMMIT ||\n+\t\t filter_data->object_type == OBJ_TAG)\n+\t\t\treturn LOFR_SKIP_TREE;\n+\n+\t\tif (filter_data->object_type == OBJ_TREE)\n+\t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\t\treturn LOFR_MARK_SEEN;\n+\n+\tcase LOFS_BLOB:\n+\t\tassert(obj->type == OBJ_BLOB);\n+\n+\t\tif (filter_data->object_type == OBJ_BLOB)\n+\t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\t\treturn LOFR_MARK_SEEN;\n+\n+\tcase LOFS_END_TREE:\n+\t\treturn LOFR_ZERO;\n+\t}\n+}\n+\n+static void filter_object_type__init(\n+\tstruct list_objects_filter_options *filter_options,\n+\tstruct filter *filter)\n+{\n+\tstruct filter_object_type_data *d = xcalloc(1, sizeof(*d));\n+\td->object_type = filter_options->object_type;\n+\n+\tfilter->filter_data = d;\n+\tfilter->filter_object_fn = filter_object_type;\n+\tfilter->free_fn = free;\n+}\n+\n /* A filter which only shows objects shown by all sub-filters. */\n struct combine_filter_data {\n \tstruct subfilter *sub;\n@@ -691,6 +766,7 @@ static filter_init_fn s_filters[] = {\n \tfilter_blobs_limit__init,\n \tfilter_trees_depth__init,\n \tfilter_sparse_oid__init,\n+\tfilter_object_type__init,\n \tfilter_combine__init,\n };\n \ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex 31457d13b9..c79ec04060 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -159,6 +159,54 @@ test_expect_success 'verify blob:limit=1m' '\n \ttest_must_be_empty observed\n '\n \n+# Test object:type= filter.\n+\n+test_expect_success 'setup object-type' '\n+\tgit init object-type &&\n+\techo contents >object-type/blob &&\n+\tgit -C object-type add blob &&\n+\tgit -C object-type commit -m commit-message &&\n+\tgit -C object-type tag tag -m tag-message\n+'\n+\n+test_expect_success 'verify object:type= fails with invalid type' '\n+\ttest_must_fail git -C object-type rev-list --objects --filter=object:type= HEAD &&\n+\ttest_must_fail git -C object-type rev-list --objects --filter=object:type=invalid HEAD\n+'\n+\n+test_expect_success 'verify object:type=blob prints blob and commit' '\n+\t(\n+\t\tgit -C object-type rev-parse HEAD &&\n+\t\tprintf \"%s blob\\n\" $(git -C object-type rev-parse HEAD:blob)\n+\t) >expected &&\n+\tgit -C object-type rev-list --objects --filter=object:type=blob HEAD >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=tree prints tree and commit' '\n+\t(\n+\t\tgit -C object-type rev-parse HEAD &&\n+\t\tprintf \"%s \\n\" $(git -C object-type rev-parse HEAD^{tree})\n+\t) >expected &&\n+\tgit -C object-type rev-list --objects --filter=object:type=tree HEAD >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=commit prints commit' '\n+\tgit -C object-type rev-parse HEAD >expected &&\n+\tgit -C object-type rev-list --objects --filter=object:type=commit HEAD >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=tag prints tag' '\n+\t(\n+\t\tgit -C object-type rev-parse HEAD &&\n+\t\tprintf \"%s tag\\n\" $(git -C object-type rev-parse tag)\n+\t) >expected &&\n+\tgit -C object-type rev-list --objects --filter=object:type=tag tag >actual &&\n+\ttest_cmp expected actual\n+'\n+\n # Test sparse:path= filter.\n # !!!!\n # NOTE: sparse:path filter support has been dropped for security reasons,\n-- \n2.30.1\n\n"},{"id":"418028","messageId":"cover.1614600555.git.ps@pks.im","threadId":"55236","inReplyTo":null,"subject":"[PATCH 0/7] rev-parse: implement object type filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-03-01T12:20:26Z","receivedAt":"2021-03-01T12:21:42Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nI've recently had the usecase to retrieve all blobs introduces between\ntwo versions which have a limit smaller than 200 bytes in order to find\nall potential candidates for LFS pointers. This is currently done with\n`git rev-list --objects --filter=blob:limit=200 ^`, but\nthis is kind of inefficient: the resulting list is way too long as it\nalso potentially includes tags, commits and trees.\n\nTo be able to more efficiently answer this query, I've implemented\nmultiple things:\n\n- A new object type filter `--filter=object:type=` for\n git-rev-list(1), which is implemented both for normal graph walks and\n for the packfile bitmap index.\n\n- Given that above usecase requires two filters (the object type\n and blob size filters), bitmap filters were extended to support\n combined filters.\n\n- git-rev-list(1) doesn't filter user-provided objects and always prints\n them. I don't want the listed commits though and only their referenced\n potential LFS blobs. So I've added a new flag `--filter-provided`\n which marks all provided objects as not-user-provided such that they\n get filtered the same as all the other objects.\n\nAltogether, this ends up with the following queries, both of which have\nbeen executed in a well-packed linux.git repository:\n\n # Previous query which uses object names as a heuristic to filter\n # non-blob objects, which bars us from using bitmap indices because\n # they cannot print paths.\n $ time git rev-list --objects --filter=blob:limit=200 \\\n --object-names --all | sed -r '/^.{,41}$/d' | wc -l\n 4502300\n\n real 1m23.872s\n user 1m30.076s\n sys 0m6.002s\n\n # New query.\n $ time git rev-list --objects --filter-provided \\\n --filter=object:type=blob --filter=blob:limit=200 \\\n --use-bitmap-index --all | wc -l\n 22585\n\n real 0m19.216s\n user 0m16.768s\n sys 0m2.450s\n\nSo with the new optimized query, we can both significantly reduce the\nlist of candidate LFS pointers and execution time.\n\nPatrick\n\nPatrick Steinhardt (7):\n revision: mark commit parents as NOT_USER_GIVEN\n list-objects: move tag processing into its own function\n list-objects: support filtering by tag and commit\n list-objects: implement object type filter\n pack-bitmap: implement object type filter\n pack-bitmap: implement combined filter\n rev-list: allow filtering of provided items\n\n Documentation/rev-list-options.txt | 3 +\n builtin/rev-list.c | 14 ++++\n list-objects-filter-options.c | 14 ++++\n list-objects-filter-options.h | 8 ++\n list-objects-filter.c | 116 ++++++++++++++++++++++++++++\n list-objects-filter.h | 2 +\n list-objects.c | 32 +++++++-\n pack-bitmap.c | 71 +++++++++++++++--\n revision.c | 4 +-\n revision.h | 3 -\n t/t6112-rev-list-filters-objects.sh | 76 ++++++++++++++++++\n t/t6113-rev-list-bitmap-filters.sh | 54 ++++++++++++-\n 12 files changed, 380 insertions(+), 17 deletions(-)\n\n-- \n2.30.1\n\n"},{"id":"418029","messageId":"f2ce5dac8965ab3ccb13d7600ea29310cbb5b863.1614600555.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1614600555.git.ps@pks.im","subject":"[PATCH 1/7] revision: mark commit parents as NOT_USER_GIVEN","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-03-01T12:20:33Z","receivedAt":"2021-03-01T12:22:01Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The NOT_USER_GIVEN flag of an object marks whether a flag was explicitly\nprovided by the user or not. The most important use case for this is\nwhen filtering objects: only objects that were not explicitly requested\nwill get filtered.\n\nThe flag is currently only set for blobs and trees, which has been fine\ngiven that there are no filters for tags or commits currently. We're\nabout to extend filtering capabilities to add object type filter though,\nwhich requires us to set up the NOT_USER_GIVEN flag correctly -- if it's\nnot set, the object wouldn't get filtered at all.\n\nMark unseen commit parents as NOT_USER_GIVEN when processing parents.\nLike this, explicitly provided parents stay user-given and thus\nunfiltered, while parents which get loaded as part of the graph walk\ncan be filtered.\n\nThis commit shouldn't have any user-visible impact yet as there is no\nlogic to filter commits yet.\n\nSigned-off-by: Patrick Steinhardt \n---\n revision.c | 4 ++--\n revision.h | 3 ---\n 2 files changed, 2 insertions(+), 5 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex b78733f508..26f422f50d 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1123,7 +1123,7 @@ static int process_parents(struct rev_info *revs, struct commit *commit,\n \t\t\t\tmark_parents_uninteresting(p);\n \t\t\tif (p->object.flags & SEEN)\n \t\t\t\tcontinue;\n-\t\t\tp->object.flags |= SEEN;\n+\t\t\tp->object.flags |= (SEEN | NOT_USER_GIVEN);\n \t\t\tif (list)\n \t\t\t\tcommit_list_insert_by_date(p, list);\n \t\t\tif (queue)\n@@ -1165,7 +1165,7 @@ static int process_parents(struct rev_info *revs, struct commit *commit,\n \t\t}\n \t\tp->object.flags |= left_flag;\n \t\tif (!(p->object.flags & SEEN)) {\n-\t\t\tp->object.flags |= SEEN;\n+\t\t\tp->object.flags |= (SEEN | NOT_USER_GIVEN);\n \t\t\tif (list)\n \t\t\t\tcommit_list_insert_by_date(p, list);\n \t\t\tif (queue)\ndiff --git a/revision.h b/revision.h\nindex e6be3c845e..f1f324a19b 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -44,9 +44,6 @@\n /*\n * Indicates object was reached by traversal. i.e. not given by user on\n * command-line or stdin.\n- * NEEDSWORK: NOT_USER_GIVEN doesn't apply to commits because we only support\n- * filtering trees and blobs, but it may be useful to support filtering commits\n- * in the future.\n */\n #define NOT_USER_GIVEN\t(1u<<25)\n #define TRACK_LINEAR\t(1u<<26)\n-- \n2.30.1\n\n"},{"id":"418030","messageId":"9feadba12420842437047fe6ea1351a8a330714a.1614600555.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1614600555.git.ps@pks.im","subject":"[PATCH 2/7] list-objects: move tag processing into its own function","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-03-01T12:20:37Z","receivedAt":"2021-03-01T12:22:02Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Move processing of tags into its own function to make the logic easier\nto extend when we're going to implement filtering for tags. No change in\nbehaviour is expected from this commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n list-objects.c | 12 ++++++++++--\n 1 file changed, 10 insertions(+), 2 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex e19589baa0..093adf85b1 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -213,6 +213,15 @@ static void process_tree(struct traversal_context *ctx,\n \tfree_tree_buffer(tree);\n }\n \n+static void process_tag(struct traversal_context *ctx,\n+\t\t\tstruct tag *tag,\n+\t\t\tstruct strbuf *base,\n+\t\t\tconst char *name)\n+{\n+\ttag->object.flags |= SEEN;\n+\tctx->show_object(&tag->object, name, ctx->show_data);\n+}\n+\n static void mark_edge_parents_uninteresting(struct commit *commit,\n \t\t\t\t\t struct rev_info *revs,\n \t\t\t\t\t show_edge_fn show_edge)\n@@ -334,8 +343,7 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx,\n \t\tif (obj->flags & (UNINTERESTING | SEEN))\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n-\t\t\tobj->flags |= SEEN;\n-\t\t\tctx->show_object(obj, name, ctx->show_data);\n+\t\t\tprocess_tag(ctx, (struct tag *)obj, base, name);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\n-- \n2.30.1\n\n"},{"id":"418031","messageId":"fe2b7a1e552546c7048b42610047847f5948adb0.1614600555.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1614600555.git.ps@pks.im","subject":"[PATCH 6/7] pack-bitmap: implement combined filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-03-01T12:20:56Z","receivedAt":"2021-03-01T12:22:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When the user has multiple objects filters specified, then this is\ninternally represented by having a \"combined\" filter. These combined\nfilters aren't yet supported by bitmap indices and can thus not be\naccelerated.\n\nFix this by implementing support for these combined filters. The\nimplementation is quite trivial: when there's a combined filter, we\nsimply recurse into `filter_bitmap()` for all of the sub-filters.\n\nSigned-off-by: Patrick Steinhardt \n---\n pack-bitmap.c | 40 +++++++++++++++++++++++++++---\n t/t6113-rev-list-bitmap-filters.sh | 7 ++++++\n 2 files changed, 43 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 196d38c91d..e33805e076 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -925,6 +925,29 @@ static void filter_bitmap_object_type(struct bitmap_index *bitmap_git,\n \t}\n }\n \n+static int filter_supported(struct list_objects_filter_options *filter)\n+{\n+\tint i;\n+\n+\tswitch (filter->choice) {\n+\tcase LOFC_BLOB_NONE:\n+\tcase LOFC_BLOB_LIMIT:\n+\tcase LOFC_OBJECT_TYPE:\n+\t\treturn 1;\n+\tcase LOFC_TREE_DEPTH:\n+\t\tif (filter->tree_exclude_depth == 0)\n+\t\t\treturn 1;\n+\t\treturn 0;\n+\tcase LOFC_COMBINE:\n+\t\tfor (i = 0; i < filter->sub_nr; i++)\n+\t\t\tif (!filter_supported(&filter->sub[i]))\n+\t\t\t\treturn 0;\n+\t\treturn 1;\n+\tdefault:\n+\t\treturn 0;\n+\t}\n+}\n+\n static int filter_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t struct object_list *tip_objects,\n \t\t\t struct bitmap *to_filter,\n@@ -932,6 +955,8 @@ static int filter_bitmap(struct bitmap_index *bitmap_git,\n {\n \tif (!filter || filter->choice == LOFC_DISABLED)\n \t\treturn 0;\n+\tif (!filter_supported(filter))\n+\t\treturn -1;\n \n \tif (filter->choice == LOFC_BLOB_NONE) {\n \t\tif (bitmap_git)\n@@ -948,8 +973,7 @@ static int filter_bitmap(struct bitmap_index *bitmap_git,\n \t\treturn 0;\n \t}\n \n-\tif (filter->choice == LOFC_TREE_DEPTH &&\n-\t filter->tree_exclude_depth == 0) {\n+\tif (filter->choice == LOFC_TREE_DEPTH) {\n \t\tif (bitmap_git)\n \t\t\tfilter_bitmap_tree_depth(bitmap_git, tip_objects,\n \t\t\t\t\t\t to_filter,\n@@ -965,8 +989,16 @@ static int filter_bitmap(struct bitmap_index *bitmap_git,\n \t\treturn 0;\n \t}\n \n-\t/* filter choice not handled */\n-\treturn -1;\n+\tif (filter->choice == LOFC_COMBINE) {\n+\t\tint i;\n+\t\tfor (i = 0; i < filter->sub_nr; i++) {\n+\t\t\tfilter_bitmap(bitmap_git, tip_objects, to_filter,\n+\t\t\t\t &filter->sub[i]);\n+\t\t}\n+\t\treturn 0;\n+\t}\n+\n+\tBUG(\"unsupported filter choice\");\n }\n \n static int can_filter_bitmap(struct list_objects_filter_options *filter)\ndiff --git a/t/t6113-rev-list-bitmap-filters.sh b/t/t6113-rev-list-bitmap-filters.sh\nindex fb66735ac8..cb9db7df6f 100755\n--- a/t/t6113-rev-list-bitmap-filters.sh\n+++ b/t/t6113-rev-list-bitmap-filters.sh\n@@ -98,4 +98,11 @@ test_expect_success 'object:type filter' '\n \ttest_bitmap_traversal expect actual\n '\n \n+test_expect_success 'combine filter' '\n+\tgit rev-list --objects --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=blob:limit=1000 --filter=object:type=blob tag >actual &&\n+\ttest_bitmap_traversal expect actual\n+'\n+\n test_done\n-- \n2.30.1\n\n"},{"id":"418032","messageId":"c97fd28d8ff3ef4f5a841e5d86a64e332579939a.1614600555.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1614600555.git.ps@pks.im","subject":"[PATCH 5/7] pack-bitmap: implement object type filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-03-01T12:20:51Z","receivedAt":"2021-03-01T12:22:34Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The preceding commit has added a new object filter for git-rev-list(1)\nwhich allows to filter objects by type. Implement the equivalent filter\nfor packfile bitmaps so that we can answer these queries fast.\n\nSigned-off-by: Patrick Steinhardt \n---\n pack-bitmap.c | 28 +++++++++++++++++++++++++---\n t/t6113-rev-list-bitmap-filters.sh | 25 ++++++++++++++++++++++++-\n 2 files changed, 49 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 1f69b5fa85..196d38c91d 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -779,9 +779,6 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n \teword_t mask;\n \tuint32_t i;\n \n-\tif (type != OBJ_BLOB && type != OBJ_TREE)\n-\t\tBUG(\"filter_bitmap_exclude_type: unsupported type '%d'\", type);\n-\n \t/*\n \t * The non-bitmap version of this filter never removes\n \t * objects which the other side specifically asked for,\n@@ -911,6 +908,23 @@ static void filter_bitmap_tree_depth(struct bitmap_index *bitmap_git,\n \t\t\t\t OBJ_BLOB);\n }\n \n+static void filter_bitmap_object_type(struct bitmap_index *bitmap_git,\n+\t\t\t\t struct object_list *tip_objects,\n+\t\t\t\t struct bitmap *to_filter,\n+\t\t\t\t enum object_type object_type)\n+{\n+\tenum object_type t;\n+\n+\tif (object_type < OBJ_COMMIT || object_type > OBJ_TAG)\n+\t\tBUG(\"filter_bitmap_object_type given invalid object\");\n+\n+\tfor (t = OBJ_COMMIT; t <= OBJ_TAG; t++) {\n+\t\tif (t == object_type)\n+\t\t\tcontinue;\n+\t\tfilter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, t);\n+\t}\n+}\n+\n static int filter_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t struct object_list *tip_objects,\n \t\t\t struct bitmap *to_filter,\n@@ -943,6 +957,14 @@ static int filter_bitmap(struct bitmap_index *bitmap_git,\n \t\treturn 0;\n \t}\n \n+\tif (filter->choice == LOFC_OBJECT_TYPE) {\n+\t\tif (bitmap_git)\n+\t\t\tfilter_bitmap_object_type(bitmap_git, tip_objects,\n+\t\t\t\t\t\t to_filter,\n+\t\t\t\t\t\t filter->object_type);\n+\t\treturn 0;\n+\t}\n+\n \t/* filter choice not handled */\n \treturn -1;\n }\ndiff --git a/t/t6113-rev-list-bitmap-filters.sh b/t/t6113-rev-list-bitmap-filters.sh\nindex 3f889949ca..fb66735ac8 100755\n--- a/t/t6113-rev-list-bitmap-filters.sh\n+++ b/t/t6113-rev-list-bitmap-filters.sh\n@@ -10,7 +10,8 @@ test_expect_success 'set up bitmapped repo' '\n \ttest_commit much-larger-blob-one &&\n \tgit repack -adb &&\n \ttest_commit two &&\n-\ttest_commit much-larger-blob-two\n+\ttest_commit much-larger-blob-two &&\n+\tgit tag tag\n '\n \n test_expect_success 'filters fallback to non-bitmap traversal' '\n@@ -75,4 +76,26 @@ test_expect_success 'tree:1 filter' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'object:type filter' '\n+\tgit rev-list --objects --filter=object:type=tag tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=object:type=tag tag >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tgit rev-list --objects --filter=object:type=commit tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=object:type=commit tag >actual &&\n+\ttest_bitmap_traversal expect actual &&\n+\n+\tgit rev-list --objects --filter=object:type=tree tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=object:type=tree tag >actual &&\n+\ttest_bitmap_traversal expect actual &&\n+\n+\tgit rev-list --objects --filter=object:type=blob tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=object:type=blob tag >actual &&\n+\ttest_bitmap_traversal expect actual\n+'\n+\n test_done\n-- \n2.30.1\n\n"},{"id":"418033","messageId":"b43bf401dfda53092a0d6f586c71747be966714a.1614600555.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1614600555.git.ps@pks.im","subject":"[PATCH 7/7] rev-list: allow filtering of provided items","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-03-01T12:21:01Z","receivedAt":"2021-03-01T12:22:50Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When providing an object filter, it is currently impossible to also\nfilter provided items. E.g. when executing `git rev-list HEAD` , the\ncommit this reference points to will be treated as user-provided and is\nthus excluded from the filtering mechanism. This makes it harder than\nnecessary to properly use the new `--filter=object:type` filter given\nthat even if the user wants to only see blobs, he'll still see commits\nof provided references.\n\nImprove this by introducing a new `--filter-provided` option to the\ngit-rev-parse(1) command. If given, then all user-provided references\nwill be subject to filtering.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/rev-list.c | 14 ++++++++++++++\n list-objects-filter-options.h | 6 ++++++\n pack-bitmap.c | 3 ++-\n t/t6112-rev-list-filters-objects.sh | 28 ++++++++++++++++++++++++++++\n t/t6113-rev-list-bitmap-filters.sh | 22 ++++++++++++++++++++++\n 5 files changed, 72 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex b4d8ea0a35..0f959b266d 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -599,6 +599,10 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\tlist_objects_filter_set_no_filter(&filter_options);\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!strcmp(arg, \"--filter-provided\")) {\n+\t\t\tfilter_options.filter_wants = 1;\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!strcmp(arg, \"--filter-print-omitted\")) {\n \t\t\targ_print_omitted = 1;\n \t\t\tcontinue;\n@@ -694,6 +698,16 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\treturn show_bisect_vars(&info, reaches, all);\n \t}\n \n+\tif (filter_options.filter_wants) {\n+\t\tstruct commit_list *c;\n+\t\tfor (i = 0; i < revs.pending.nr; i++) {\n+\t\t\tstruct object_array_entry *pending = revs.pending.objects + i;\n+\t\t\tpending->item->flags |= NOT_USER_GIVEN;\n+\t\t}\n+\t\tfor (c = revs.commits; c; c = c->next)\n+\t\t\tc->item->object.flags |= NOT_USER_GIVEN;\n+\t}\n+\n \tif (arg_print_omitted)\n \t\toidset_init(&omitted_objects, DEFAULT_OIDSET_SIZE);\n \tif (arg_missing_action == MA_PRINT)\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 4d0d0588cc..5e609e307a 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -42,6 +42,12 @@ struct list_objects_filter_options {\n \t */\n \tenum list_objects_filter_choice choice;\n \n+\t/*\n+\t * \"--filter-provided\" was given by the user, instructing us to also\n+\t * filter all explicitly provided objects.\n+\t */\n+\tunsigned int filter_wants : 1;\n+\n \t/*\n \t * Choice is LOFC_DISABLED because \"--no-filter\" was requested.\n \t */\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex e33805e076..5ff800316b 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1101,7 +1101,8 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \tif (haves_bitmap)\n \t\tbitmap_and_not(wants_bitmap, haves_bitmap);\n \n-\tfilter_bitmap(bitmap_git, wants, wants_bitmap, filter);\n+\tfilter_bitmap(bitmap_git, (filter && filter->filter_wants) ? NULL : wants,\n+\t\t wants_bitmap, filter);\n \n \tbitmap_git->result = wants_bitmap;\n \tbitmap_git->haves = haves_bitmap;\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex c79ec04060..47c558ab0e 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -207,6 +207,34 @@ test_expect_success 'verify object:type=tag prints tag' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'verify object:type=blob prints only blob with --filter-provided' '\n+\tprintf \"%s blob\\n\" $(git -C object-type rev-parse HEAD:blob) >expected &&\n+\tgit -C object-type rev-list --objects \\\n+\t\t--filter=object:type=blob --filter-provided HEAD >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=tree prints only tree with --filter-provided' '\n+\tprintf \"%s \\n\" $(git -C object-type rev-parse HEAD^{tree}) >expected &&\n+\tgit -C object-type rev-list --objects \\\n+\t\t--filter=object:type=tree HEAD --filter-provided >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=commit prints only commit with --filter-provided' '\n+\tgit -C object-type rev-parse HEAD >expected &&\n+\tgit -C object-type rev-list --objects \\\n+\t\t--filter=object:type=commit --filter-provided HEAD >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=tag prints only tag with --filter-provided' '\n+\tprintf \"%s tag\\n\" $(git -C object-type rev-parse tag) >expected &&\n+\tgit -C object-type rev-list --objects \\\n+\t\t--filter=object:type=tag --filter-provided tag >actual &&\n+\ttest_cmp expected actual\n+'\n+\n # Test sparse:path= filter.\n # !!!!\n # NOTE: sparse:path filter support has been dropped for security reasons,\ndiff --git a/t/t6113-rev-list-bitmap-filters.sh b/t/t6113-rev-list-bitmap-filters.sh\nindex cb9db7df6f..fe3df0ee14 100755\n--- a/t/t6113-rev-list-bitmap-filters.sh\n+++ b/t/t6113-rev-list-bitmap-filters.sh\n@@ -98,6 +98,28 @@ test_expect_success 'object:type filter' '\n \ttest_bitmap_traversal expect actual\n '\n \n+test_expect_success 'object:type filter with --filter-provided' '\n+\tgit rev-list --objects --filter=object:type=tag --filter-provided tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=object:type=tag --filter-provided tag >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tgit rev-list --objects --filter=object:type=commit --filter-provided tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=object:type=commit --filter-provided tag >actual &&\n+\ttest_bitmap_traversal expect actual &&\n+\n+\tgit rev-list --objects --filter=object:type=tree --filter-provided tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=object:type=tree --filter-provided tag >actual &&\n+\ttest_bitmap_traversal expect actual &&\n+\n+\tgit rev-list --objects --filter=object:type=blob --filter-provided tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=object:type=blob --filter-provided tag >actual &&\n+\ttest_bitmap_traversal expect actual\n+'\n+\n test_expect_success 'combine filter' '\n \tgit rev-list --objects --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&\n \tgit rev-list --use-bitmap-index \\\n-- \n2.30.1\n\n"},{"id":"418739","messageId":"YEk8iiDf/FMxzhIF@coredump.intra.peff.net","threadId":"55236","inReplyTo":"cover.1614600555.git.ps@pks.im","subject":"Re: [PATCH 0/7] rev-parse: implement object type filter","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-03-10T21:39:22Z","receivedAt":"2021-03-10T21:40:41Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 01, 2021 at 01:20:26PM +0100, Patrick Steinhardt wrote:\n\n> Altogether, this ends up with the following queries, both of which have\n> been executed in a well-packed linux.git repository:\n> \n> # Previous query which uses object names as a heuristic to filter\n> # non-blob objects, which bars us from using bitmap indices because\n> # they cannot print paths.\n> $ time git rev-list --objects --filter=blob:limit=200 \\\n> --object-names --all | sed -r '/^.{,41}$/d' | wc -l\n> 4502300\n> \n> real 1m23.872s\n> user 1m30.076s\n> sys 0m6.002s\n> \n> # New query.\n> $ time git rev-list --objects --filter-provided \\\n> --filter=object:type=blob --filter=blob:limit=200 \\\n> --use-bitmap-index --all | wc -l\n> 22585\n> \n> real 0m19.216s\n> user 0m16.768s\n> sys 0m2.450s\n\nThose produce very different answers. I guess because in the first one,\nyou still have a bunch of tree objects, too. You'd do much better to get\nthe actual types from cat-file, and filter on that. That also lets you\nuse bitmaps for the traversal portion. E.g.:\n\n $ time git rev-list --use-bitmap-index --objects --filter=blob:limit=200 --all |\n git cat-file --buffer --batch-check='%(objecttype) %(objectname)' |\n\t perl -lne 'print $1 if /^blob (.*)/' | wc -l\n 14966\n \n real\t0m6.248s\n user\t0m7.810s\n sys\t0m0.440s\n\nwhich is faster than what you showed above (this is on linux.git, but my\nresult is different; maybe you have more refs than me?). But we should\nbe able to do better purely internally, so I suspect my computer is just\nfaster (or maybe your extra refs just aren't well-covered by bitmaps).\nRunning with your patches I get:\n\n $ time git rev-list --objects --use-bitmap-index --all \\\n --filter-provided --filter=object:type=blob \\\n\t --filter=blob:limit=200 | wc -l\n 16339\n\n real\t0m1.309s\n user\t0m1.234s\n sys\t0m0.079s\n\nwhich is indeed faster. It's quite curious that the answer is not the\nsame, though! I think yours has some bugs. If I sort and diff the\nresults, I see some commits mentioned in the output. Perhaps this is\n--filter-provided not working, as they all seem to be ref tips.\n\n> To be able to more efficiently answer this query, I've implemented\n> multiple things:\n> \n> - A new object type filter `--filter=object:type=` for\n> git-rev-list(1), which is implemented both for normal graph walks and\n> for the packfile bitmap index.\n> \n> - Given that above usecase requires two filters (the object type\n> and blob size filters), bitmap filters were extended to support\n> combined filters.\n\nThat's probably reasonable, especially because it lets us use bitmaps. I\ndo have a dream that we'll eventually be able to support more extensive\nformatting via log/rev-list, which would allow:\n\n git rev-list --use-bitmap-index --objects --all \\\n --format=%(objecttype) %(objectname) |\n perl -ne 'print $1 if /^blob (.*)/'\n\nThat should be faster than the separate cat-file (which has to re-lookup\neach object, in addition to the extra pipe overhead), but I expect the\n--filter solution should always be faster still, as it can very quickly\neliminate the majority of the objects at the bitmap level.\n\n> - git-rev-list(1) doesn't filter user-provided objects and always prints\n> them. I don't want the listed commits though and only their referenced\n> potential LFS blobs. So I've added a new flag `--filter-provided`\n> which marks all provided objects as not-user-provided such that they\n> get filtered the same as all the other objects.\n\nYeah, this \"user-provided\" behavior was quite a surprise to me when I\nstarted implementing the bitmap versions of the existing filters. It's\nnice to have the option to specify which you want.\n\n-Peff\n"},{"id":"418741","messageId":"YElA+GI3oAZiueOs@nand.local","threadId":"55236","inReplyTo":"cover.1614600555.git.ps@pks.im","subject":"Re: [PATCH 0/7] rev-parse: implement object type filter","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-03-10T21:58:16Z","receivedAt":"2021-03-10T21:58:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Mar 01, 2021 at 01:20:26PM +0100, Patrick Steinhardt wrote:\n> - A new object type filter `--filter=object:type=` for\n> git-rev-list(1), which is implemented both for normal graph walks and\n> for the packfile bitmap index.\n\nI understand what you're looking for here, but I worry that '--filter'\nmight be too leaky of an abstraction.\n\nI was a little surprised to learn that you can clone a repository with\n--filter=object:type=tree (excluding commits), but it does work. I'm\nfine reusing a lot of the object filtering code if it makes this an\neasier task, but I think it may be worthwhile to hide this new kind of\nfilter from upload-pack.\n\n> - Given that above usecase requires two filters (the object type\n> and blob size filters), bitmap filters were extended to support\n> combined filters.\n\nNice. We didn't do this since the only previously supported filters were\nblob:none and tree:0 (the latter implying the former), so there was no\nneed.\n\nThanks,\nTaylor\n"},{"id":"418743","messageId":"YElGANcek5DPxYKo@coredump.intra.peff.net","threadId":"55236","inReplyTo":"YElA+GI3oAZiueOs@nand.local","subject":"Re: [PATCH 0/7] rev-parse: implement object type filter","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-03-10T22:19:44Z","receivedAt":"2021-03-10T22:20:24Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Mar 10, 2021 at 04:58:16PM -0500, Taylor Blau wrote:\n\n> On Mon, Mar 01, 2021 at 01:20:26PM +0100, Patrick Steinhardt wrote:\n> > - A new object type filter `--filter=object:type=` for\n> > git-rev-list(1), which is implemented both for normal graph walks and\n> > for the packfile bitmap index.\n> \n> I understand what you're looking for here, but I worry that '--filter'\n> might be too leaky of an abstraction.\n> \n> I was a little surprised to learn that you can clone a repository with\n> --filter=object:type=tree (excluding commits), but it does work. I'm\n> fine reusing a lot of the object filtering code if it makes this an\n> easier task, but I think it may be worthwhile to hide this new kind of\n> filter from upload-pack.\n\nI had a similar thought, but wouldn't the existing uploadpackfilter\nconfig take care of this?\n\nI guess the catch-all \"allow\" option defaults to \"true\", so we'd support\nany new filters that are added. Which seems like a poor choice in\ngeneral, but flipping it would mean that servers have to update their\nconfig.\n\nI do wonder if it's that bad for clients to be able to specify something\nlike this, though. Even though there's not that much use for it with a\nregular partial clone, it could conceivably used for some special cases.\nI do think it would be more useful if you could OR together multiple\ntypes. Asking for \"commits|tags|trees\" is really the same as the already\nuseful \"blob:none\". And \"commits|tags\" is the same as tree:depth=0.\n\n-Peff\n"},{"id":"418848","messageId":"YEorU+rkX4sgMtKi@ncase","threadId":"55236","inReplyTo":"YEk8iiDf/FMxzhIF@coredump.intra.peff.net","subject":"Re: [PATCH 0/7] rev-parse: implement object type filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-03-11T14:38:11Z","receivedAt":"2021-03-11T14:38:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Mar 10, 2021 at 04:39:22PM -0500, Jeff King wrote:\n> On Mon, Mar 01, 2021 at 01:20:26PM +0100, Patrick Steinhardt wrote:\n> \n> > Altogether, this ends up with the following queries, both of which have\n> > been executed in a well-packed linux.git repository:\n> > \n> > # Previous query which uses object names as a heuristic to filter\n> > # non-blob objects, which bars us from using bitmap indices because\n> > # they cannot print paths.\n> > $ time git rev-list --objects --filter=blob:limit=200 \\\n> > --object-names --all | sed -r '/^.{,41}$/d' | wc -l\n> > 4502300\n> > \n> > real 1m23.872s\n> > user 1m30.076s\n> > sys 0m6.002s\n> > \n> > # New query.\n> > $ time git rev-list --objects --filter-provided \\\n> > --filter=object:type=blob --filter=blob:limit=200 \\\n> > --use-bitmap-index --all | wc -l\n> > 22585\n> > \n> > real 0m19.216s\n> > user 0m16.768s\n> > sys 0m2.450s\n> \n> Those produce very different answers. I guess because in the first one,\n> you still have a bunch of tree objects, too. You'd do much better to get\n> the actual types from cat-file, and filter on that. That also lets you\n> use bitmaps for the traversal portion. E.g.:\n\nThey do provide different answers, and you're right that `--batch-check`\nwould have helped to filter by type. Your idea doesn't really work in my\nusecase though to identify LFS pointers, at least not without additional\ntooling on top of what you've provided. There'd at least need to be two\ngit-cat-file(1) processes: one to do the `--batch-check` thing to\nactually filter by object type, and one to then read the actual LFS\npointer candidates from disk in order to see whether they are LFS\npointers or not.\n\nActually, we currently are doing something similar to that at GitLab: we\nlist all potential candidates via git-rev-list(1), write the output into\n`git-cat-file --batch-check`, and anything that is a blob then gets\nforwarded into `git-cat-file --batch`.\n\n> $ time git rev-list --use-bitmap-index --objects --filter=blob:limit=200 --all |\n> git cat-file --buffer --batch-check='%(objecttype) %(objectname)' |\n> \t perl -lne 'print $1 if /^blob (.*)/' | wc -l\n> 14966\n> \n> real\t0m6.248s\n> user\t0m7.810s\n> sys\t0m0.440s\n> \n> which is faster than what you showed above (this is on linux.git, but my\n> result is different; maybe you have more refs than me?). But we should\n> be able to do better purely internally, so I suspect my computer is just\n> faster (or maybe your extra refs just aren't well-covered by bitmaps).\n> Running with your patches I get:\n\nI've got quite a beefy machine with a Ryzen 3 5800X, and I did do a `git\nrepack -Adfb` right before doig benchmarks. I do have the stable kernel\nrepository added though, which accounts for quite a lot of additional\nreferences (3938) and objects (9.3M).\n\n> $ time git rev-list --objects --use-bitmap-index --all \\\n> --filter-provided --filter=object:type=blob \\\n> \t --filter=blob:limit=200 | wc -l\n> 16339\n> \n> real\t0m1.309s\n> user\t0m1.234s\n> sys\t0m0.079s\n> \n> which is indeed faster. It's quite curious that the answer is not the\n> same, though! I think yours has some bugs. If I sort and diff the\n> results, I see some commits mentioned in the output. Perhaps this is\n> --filter-provided not working, as they all seem to be ref tips.\n\nI noticed it, too, and couldn't yet find an answer why that is.\nHonestly, I found the NOT_USER_GIVEN flag quite confusing and I'm not at\nall sure whether I've got all cases covered correctly. The previous was\nhow this was handled (`USER_GIVEN` instead of `NOT_USER_GIVEN`) would've\nbeen easier to figure out for this specific usecase. But I guess it was\nconverted due to specific reasons.\n\nI'll invest some more time to figure out what's happening here.\n\n> > To be able to more efficiently answer this query, I've implemented\n> > multiple things:\n> > \n> > - A new object type filter `--filter=object:type=` for\n> > git-rev-list(1), which is implemented both for normal graph walks and\n> > for the packfile bitmap index.\n> > \n> > - Given that above usecase requires two filters (the object type\n> > and blob size filters), bitmap filters were extended to support\n> > combined filters.\n> \n> That's probably reasonable, especially because it lets us use bitmaps. I\n> do have a dream that we'll eventually be able to support more extensive\n> formatting via log/rev-list, which would allow:\n> \n> git rev-list --use-bitmap-index --objects --all \\\n> --format=%(objecttype) %(objectname) |\n> perl -ne 'print $1 if /^blob (.*)/'\n> \n> That should be faster than the separate cat-file (which has to re-lookup\n> each object, in addition to the extra pipe overhead), but I expect the\n> --filter solution should always be faster still, as it can very quickly\n> eliminate the majority of the objects at the bitmap level.\n\nThat'd be nice, even though it wouldn't help in my particular usecase: I\nneed to read each candidate blob to see whether it's an LFS pointer or\nnot anyway.\n\n> > - git-rev-list(1) doesn't filter user-provided objects and always prints\n> > them. I don't want the listed commits though and only their referenced\n> > potential LFS blobs. So I've added a new flag `--filter-provided`\n> > which marks all provided objects as not-user-provided such that they\n> > get filtered the same as all the other objects.\n> \n> Yeah, this \"user-provided\" behavior was quite a surprise to me when I\n> started implementing the bitmap versions of the existing filters. It's\n> nice to have the option to specify which you want.\n> \n> -Peff\n\nPatrick\n"},{"id":"418849","messageId":"YEosm+RjD/YRIcLo@ncase","threadId":"55236","inReplyTo":"YElGANcek5DPxYKo@coredump.intra.peff.net","subject":"Re: [PATCH 0/7] rev-parse: implement object type filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-03-11T14:43:39Z","receivedAt":"2021-03-11T14:44:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Mar 10, 2021 at 05:19:44PM -0500, Jeff King wrote:\n> On Wed, Mar 10, 2021 at 04:58:16PM -0500, Taylor Blau wrote:\n> \n> > On Mon, Mar 01, 2021 at 01:20:26PM +0100, Patrick Steinhardt wrote:\n> > > - A new object type filter `--filter=object:type=` for\n> > > git-rev-list(1), which is implemented both for normal graph walks and\n> > > for the packfile bitmap index.\n> > \n> > I understand what you're looking for here, but I worry that '--filter'\n> > might be too leaky of an abstraction.\n> > \n> > I was a little surprised to learn that you can clone a repository with\n> > --filter=object:type=tree (excluding commits), but it does work. I'm\n> > fine reusing a lot of the object filtering code if it makes this an\n> > easier task, but I think it may be worthwhile to hide this new kind of\n> > filter from upload-pack.\n> \n> I had a similar thought, but wouldn't the existing uploadpackfilter\n> config take care of this?\n> \n> I guess the catch-all \"allow\" option defaults to \"true\", so we'd support\n> any new filters that are added. Which seems like a poor choice in\n> general, but flipping it would mean that servers have to update their\n> config.\n> \n> I do wonder if it's that bad for clients to be able to specify something\n> like this, though. Even though there's not that much use for it with a\n> regular partial clone, it could conceivably used for some special cases.\n> I do think it would be more useful if you could OR together multiple\n> types. Asking for \"commits|tags|trees\" is really the same as the already\n> useful \"blob:none\". And \"commits|tags\" is the same as tree:depth=0.\n\nI did waste a few thoughts on how this should be handled. I see two ways\nof doing it:\n\n - We could just implement the new `object:type` filter such that it\n directly supports OR'ing. That's the easy way to do it, but it's\n inflexible.\n\n - We could extend combined filters to support OR-semantics in\n addition to the current AND-semantics. In the end, that'd be a\n much more flexible approach and potentially allow additional\n usecases.\n\nI lean more towards the latter as it feels like the better design. But\nit's more involved, and I'm not sure I want to do it as part of this\npatch series.\n\nPatrick\n"},{"id":"418887","messageId":"YEpZXjPxT3npyhhv@coredump.intra.peff.net","threadId":"55236","inReplyTo":"YEorU+rkX4sgMtKi@ncase","subject":"Re: [PATCH 0/7] rev-parse: implement object type filter","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-03-11T17:54:38Z","receivedAt":"2021-03-11T17:55:29Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 11, 2021 at 03:38:11PM +0100, Patrick Steinhardt wrote:\n\n> > Those produce very different answers. I guess because in the first one,\n> > you still have a bunch of tree objects, too. You'd do much better to get\n> > the actual types from cat-file, and filter on that. That also lets you\n> > use bitmaps for the traversal portion. E.g.:\n> \n> They do provide different answers, and you're right that `--batch-check`\n> would have helped to filter by type. Your idea doesn't really work in my\n> usecase though to identify LFS pointers, at least not without additional\n> tooling on top of what you've provided. There'd at least need to be two\n> git-cat-file(1) processes: one to do the `--batch-check` thing to\n> actually filter by object type, and one to then read the actual LFS\n> pointer candidates from disk in order to see whether they are LFS\n> pointers or not.\n> \n> Actually, we currently are doing something similar to that at GitLab: we\n> list all potential candidates via git-rev-list(1), write the output into\n> `git-cat-file --batch-check`, and anything that is a blob then gets\n> forwarded into `git-cat-file --batch`.\n\nYou'd need that final cat-file with your patch, too, though. So I think\nit makes sense to think about \"generate the list of blobs\" as the\nprimary action.\n\nYou can of course do the type and content dump as a single cat-file, but\nin my experience that is much slower (because we waste time dumping\nobject content that the caller ultimately won't care about).\n\nThinking in the opposite direction, if we are filtering by type via\ncat-file, we could do the size filter there, too. So:\n\n git rev-list --use-bitmap-index --objects --all |\n git cat-file --batch-check='%(objecttype) %(objectsize) %(objectname)' |\n perl -lne 'print $2 if /^blob (\\d+) (.*)/ && $1 < 200'\n\nwhich produces the same answer as my earlier:\n\n> > $ time git rev-list --use-bitmap-index --objects --filter=blob:limit=200 --all |\n> > git cat-file --buffer --batch-check='%(objecttype) %(objectname)' |\n> > \t perl -lne 'print $1 if /^blob (.*)/' | wc -l\n\nbut takes about twice as long. Which is really just a roundabout way of\nsaying that yes, shoving things into \"rev-list\" can provide substantial\nspeedups. :)\n\n> > which is faster than what you showed above (this is on linux.git, but my\n> > result is different; maybe you have more refs than me?). But we should\n> > be able to do better purely internally, so I suspect my computer is just\n> > faster (or maybe your extra refs just aren't well-covered by bitmaps).\n> > Running with your patches I get:\n> \n> I've got quite a beefy machine with a Ryzen 3 5800X, and I did do a `git\n> repack -Adfb` right before doig benchmarks. I do have the stable kernel\n> repository added though, which accounts for quite a lot of additional\n> references (3938) and objects (9.3M).\n\nYeah, I wondered if it was something like that. Mine is just\ntorvalds/linux.git. Fetching stable/linux.git from kernel.org, running\n\"git repack -adb\" on the result, and then repeating my timings gets me\nnumbers close to yours.\n\n> > which is indeed faster. It's quite curious that the answer is not the\n> > same, though! I think yours has some bugs. If I sort and diff the\n> > results, I see some commits mentioned in the output. Perhaps this is\n> > --filter-provided not working, as they all seem to be ref tips.\n> \n> I noticed it, too, and couldn't yet find an answer why that is.\n> Honestly, I found the NOT_USER_GIVEN flag quite confusing and I'm not at\n> all sure whether I've got all cases covered correctly. The previous was\n> how this was handled (`USER_GIVEN` instead of `NOT_USER_GIVEN`) would've\n> been easier to figure out for this specific usecase. But I guess it was\n> converted due to specific reasons.\n> \n> I'll invest some more time to figure out what's happening here.\n\nThanks. I also scratched my head at NOT_USER_GIVEN. I haven't looked at\nthis part of the filter code very much, but it seems like that is a\nrecipe for accidentally marking a commit as NOT_USER_GIVEN if we\ntraverse to it (even if it was originally _also_ given by the user).\n\n-Peff\n\n> > That's probably reasonable, especially because it lets us use bitmaps. I\n> > do have a dream that we'll eventually be able to support more extensive\n> > formatting via log/rev-list, which would allow:\n> > \n> > git rev-list --use-bitmap-index --objects --all \\\n> > --format=%(objecttype) %(objectname) |\n> > perl -ne 'print $1 if /^blob (.*)/'\n> > \n> > That should be faster than the separate cat-file (which has to re-lookup\n> > each object, in addition to the extra pipe overhead), but I expect the\n> > --filter solution should always be faster still, as it can very quickly\n> > eliminate the majority of the objects at the bitmap level.\n> \n> That'd be nice, even though it wouldn't help in my particular usecase: I\n> need to read each candidate blob to see whether it's an LFS pointer or\n> not anyway.\n\nI think it works out roughly the same as the --filter solution, in the\nsense that both generate a list of candidate blobs that you'd read with\n\"cat-file --batch\" (but of course it's still slower).\n\n-Peff\n"},{"id":"418888","messageId":"YEpZ30YOZ/akZ7pm@coredump.intra.peff.net","threadId":"55236","inReplyTo":"YEosm+RjD/YRIcLo@ncase","subject":"Re: [PATCH 0/7] rev-parse: implement object type filter","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-03-11T17:56:47Z","receivedAt":"2021-03-11T17:57:38Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 11, 2021 at 03:43:39PM +0100, Patrick Steinhardt wrote:\n\n> > I do wonder if it's that bad for clients to be able to specify something\n> > like this, though. Even though there's not that much use for it with a\n> > regular partial clone, it could conceivably used for some special cases.\n> > I do think it would be more useful if you could OR together multiple\n> > types. Asking for \"commits|tags|trees\" is really the same as the already\n> > useful \"blob:none\". And \"commits|tags\" is the same as tree:depth=0.\n> \n> I did waste a few thoughts on how this should be handled. I see two ways\n> of doing it:\n> \n> - We could just implement the new `object:type` filter such that it\n> directly supports OR'ing. That's the easy way to do it, but it's\n> inflexible.\n> \n> - We could extend combined filters to support OR-semantics in\n> addition to the current AND-semantics. In the end, that'd be a\n> much more flexible approach and potentially allow additional\n> usecases.\n> \n> I lean more towards the latter as it feels like the better design. But\n> it's more involved, and I'm not sure I want to do it as part of this\n> patch series.\n\nYeah, I don't think that needs to be part of this series. The only thing\nto consider for this series is whether it's a problem for clients to be\nable to ask for type=blob from a server which has blindly turned on\nuploadpack.allowFilter without restricting the types.\n\nMy gut is to say yes. Even if we don't have a particular use, I don't\nthink it hurts (and in general, I think people running public servers\nwith bitmaps really ought to set uploadpackfilter.allow=false anyway,\nbecause stuff like non-zero tree-depth filters are expensive).\n\n-Peff\n"},{"id":"419144","messageId":"YE9ENGKkKRbzUL7I@ncase","threadId":"55236","inReplyTo":"YEk8iiDf/FMxzhIF@coredump.intra.peff.net","subject":"Re: [PATCH 0/7] rev-parse: implement object type filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-03-15T11:25:40Z","receivedAt":"2021-03-15T11:26:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Mar 10, 2021 at 04:39:22PM -0500, Jeff King wrote:\n> On Mon, Mar 01, 2021 at 01:20:26PM +0100, Patrick Steinhardt wrote:\n> \n> > Altogether, this ends up with the following queries, both of which have\n> > been executed in a well-packed linux.git repository:\n> > \n> > # Previous query which uses object names as a heuristic to filter\n> > # non-blob objects, which bars us from using bitmap indices because\n> > # they cannot print paths.\n> > $ time git rev-list --objects --filter=blob:limit=200 \\\n> > --object-names --all | sed -r '/^.{,41}$/d' | wc -l\n> > 4502300\n> > \n> > real 1m23.872s\n> > user 1m30.076s\n> > sys 0m6.002s\n> > \n> > # New query.\n> > $ time git rev-list --objects --filter-provided \\\n> > --filter=object:type=blob --filter=blob:limit=200 \\\n> > --use-bitmap-index --all | wc -l\n> > 22585\n> > \n> > real 0m19.216s\n> > user 0m16.768s\n> > sys 0m2.450s\n> \n> Those produce very different answers. I guess because in the first one,\n> you still have a bunch of tree objects, too. You'd do much better to get\n> the actual types from cat-file, and filter on that. That also lets you\n> use bitmaps for the traversal portion. E.g.:\n> \n> $ time git rev-list --use-bitmap-index --objects --filter=blob:limit=200 --all |\n> git cat-file --buffer --batch-check='%(objecttype) %(objectname)' |\n> \t perl -lne 'print $1 if /^blob (.*)/' | wc -l\n> 14966\n> \n> real\t0m6.248s\n> user\t0m7.810s\n> sys\t0m0.440s\n> \n> which is faster than what you showed above (this is on linux.git, but my\n> result is different; maybe you have more refs than me?). But we should\n> be able to do better purely internally, so I suspect my computer is just\n> faster (or maybe your extra refs just aren't well-covered by bitmaps).\n> Running with your patches I get:\n> \n> $ time git rev-list --objects --use-bitmap-index --all \\\n> --filter-provided --filter=object:type=blob \\\n> \t --filter=blob:limit=200 | wc -l\n> 16339\n> \n> real\t0m1.309s\n> user\t0m1.234s\n> sys\t0m0.079s\n> \n> which is indeed faster. It's quite curious that the answer is not the\n> same, though! I think yours has some bugs. If I sort and diff the\n> results, I see some commits mentioned in the output. Perhaps this is\n> --filter-provided not working, as they all seem to be ref tips.\n[snip]\n\nI've found the issue: when converting filters to a combined filter via\n`transform_to_combine_type()`, we reset the top-level filter via a call\nto `memset()`. So for combined filters, the option wouldn't have taken\nany effect because it got reset iff the `--filter-provided` option comes\nbefore the second filter.\n\nPatrick\n"},{"id":"419151","messageId":"cover.1615813673.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1614600555.git.ps@pks.im","subject":"[PATCH v2 0/8] rev-parse: implement object type filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-03-15T13:14:26Z","receivedAt":"2021-03-15T13:15:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the second version of my patch series which implements a new\n`object:type` filter for git-rev-parse(1) and git-upload-pack(1) and\nextends support for bitmap indices to work with combined filters.\n\nChanges compared to v1:\n\n - I've added a patch up front which changes the uploadpack\n documentation to explicitly document that setting\n `uploadpackfilter.allow=true` will enable all future filters. I'm\n not yet saying that this is the correct thing to do, but rather\n added this patch such that we have a proper place to discuss this\n topic. In the context of object-type filters, I do think though\n that it's not an issue to default-enable type filters: they're not\n expensive to compute anyway.\n\n - `uploadpackfilter..allow` documentation was updated to\n mention the new filter.\n\n - A bug was fixed which caused us to reset `--filter-allowed` in\n case a normal filter was converted to a combined filter. I've\n added tests to more thoroughly verify that filters work as\n expected and also filter provided objects.\n\nPlease see the attached range-diff for more details.\n\nPatrick\n\nPatrick Steinhardt (8):\n uploadpack.txt: document implication of `uploadpackfilter.allow`\n revision: mark commit parents as NOT_USER_GIVEN\n list-objects: move tag processing into its own function\n list-objects: support filtering by tag and commit\n list-objects: implement object type filter\n pack-bitmap: implement object type filter\n pack-bitmap: implement combined filter\n rev-list: allow filtering of provided items\n\n Documentation/config/uploadpack.txt | 9 ++-\n Documentation/rev-list-options.txt | 3 +\n builtin/rev-list.c | 14 ++++\n list-objects-filter-options.c | 18 +++++\n list-objects-filter-options.h | 8 ++\n list-objects-filter.c | 116 ++++++++++++++++++++++++++++\n list-objects-filter.h | 2 +\n list-objects.c | 32 +++++++-\n pack-bitmap.c | 71 +++++++++++++++--\n revision.c | 4 +-\n revision.h | 3 -\n t/t6112-rev-list-filters-objects.sh | 76 ++++++++++++++++++\n t/t6113-rev-list-bitmap-filters.sh | 68 +++++++++++++++-\n 13 files changed, 403 insertions(+), 21 deletions(-)\n\nRange-diff against v1:\n-: ---------- > 1: 270ff80dac uploadpack.txt: document implication of `uploadpackfilter.allow`\n1: f2ce5dac89 = 2: ddbec75986 revision: mark commit parents as NOT_USER_GIVEN\n2: 9feadba124 = 3: d8da0b24f4 list-objects: move tag processing into its own function\n3: 4aa13ee83f = 4: 5545c189c5 list-objects: support filtering by tag and commit\n4: 01b9fdbb9c ! 5: acf01472af list-objects: implement object type filter\n @@ Commit message\n \n Signed-off-by: Patrick Steinhardt \n \n + ## Documentation/config/uploadpack.txt ##\n +@@ Documentation/config/uploadpack.txt: uploadpackfilter.allow::\n + uploadpackfilter..allow::\n + \tExplicitly allow or ban the object filter corresponding to\n + \t``, where `` may be one of: `blob:none`,\n +-\t`blob:limit`, `tree`, `sparse:oid`, or `combine`. If using\n +-\tcombined filters, both `combine` and all of the nested filter\n +-\tkinds must be allowed. Defaults to `uploadpackfilter.allow`.\n ++\t`blob:limit`, `object:type`, `tree`, `sparse:oid`, or `combine`.\n ++\tIf using combined filters, both `combine` and all of the nested\n ++\tfilter kinds must be allowed. Defaults to `uploadpackfilter.allow`.\n + \n + uploadpackfilter.tree.maxDepth::\n + \tOnly allow `--filter=tree:` when `` is no more than the value of\n +\n ## Documentation/rev-list-options.txt ##\n @@ Documentation/rev-list-options.txt: or units. n may be zero. The suffixes k, m, and g can be used to name\n units in KiB, MiB, or GiB. For example, 'blob:limit=1k' is the same\n5: c97fd28d8f = 6: 8073ab665b pack-bitmap: implement object type filter\n6: fe2b7a1e55 = 7: fac3477d97 pack-bitmap: implement combined filter\n7: b43bf401df ! 8: 0e26fee8b3 rev-list: allow filtering of provided items\n @@ builtin/rev-list.c: int cmd_rev_list(int argc, const char **argv, const char *pr\n \t\toidset_init(&omitted_objects, DEFAULT_OIDSET_SIZE);\n \tif (arg_missing_action == MA_PRINT)\n \n + ## list-objects-filter-options.c ##\n +@@ list-objects-filter-options.c: static void transform_to_combine_type(\n + \t\tmemset(filter_options, 0, sizeof(*filter_options));\n + \t\tfilter_options->sub = sub_array;\n + \t\tfilter_options->sub_alloc = initial_sub_alloc;\n ++\t\tfilter_options->filter_wants = sub_array[0].filter_wants;\n + \t}\n + \tfilter_options->sub_nr = 1;\n + \tfilter_options->choice = LOFC_COMBINE;\n +@@ list-objects-filter-options.c: void parse_list_objects_filter(\n + \t\tparse_error = gently_parse_list_objects_filter(\n + \t\t\t&filter_options->sub[filter_options->sub_nr - 1], arg,\n + \t\t\t&errbuf);\n ++\t\tif (!parse_error)\n ++\t\t\tfilter_options->sub[filter_options->sub_nr - 1].filter_wants =\n ++\t\t\t\tfilter_options->filter_wants;\n + \t}\n + \tif (parse_error)\n + \t\tdie(\"%s\", errbuf.buf);\n +\n ## list-objects-filter-options.h ##\n @@ list-objects-filter-options.h: struct list_objects_filter_options {\n \t */\n @@ t/t6113-rev-list-bitmap-filters.sh: test_expect_success 'object:type filter' '\n '\n \n +test_expect_success 'object:type filter with --filter-provided' '\n -+\tgit rev-list --objects --filter=object:type=tag --filter-provided tag >expect &&\n ++\tgit rev-list --objects --filter-provided --filter=object:type=tag tag >expect &&\n +\tgit rev-list --use-bitmap-index \\\n -+\t\t --objects --filter=object:type=tag --filter-provided tag >actual &&\n ++\t\t --objects --filter-provided --filter=object:type=tag tag >actual &&\n +\ttest_cmp expect actual &&\n +\n -+\tgit rev-list --objects --filter=object:type=commit --filter-provided tag >expect &&\n ++\tgit rev-list --objects --filter-provided --filter=object:type=commit tag >expect &&\n +\tgit rev-list --use-bitmap-index \\\n -+\t\t --objects --filter=object:type=commit --filter-provided tag >actual &&\n ++\t\t --objects --filter-provided --filter=object:type=commit tag >actual &&\n +\ttest_bitmap_traversal expect actual &&\n +\n -+\tgit rev-list --objects --filter=object:type=tree --filter-provided tag >expect &&\n ++\tgit rev-list --objects --filter-provided --filter=object:type=tree tag >expect &&\n +\tgit rev-list --use-bitmap-index \\\n -+\t\t --objects --filter=object:type=tree --filter-provided tag >actual &&\n ++\t\t --objects --filter-provided --filter=object:type=tree tag >actual &&\n +\ttest_bitmap_traversal expect actual &&\n +\n -+\tgit rev-list --objects --filter=object:type=blob --filter-provided tag >expect &&\n ++\tgit rev-list --objects --filter-provided --filter=object:type=blob tag >expect &&\n +\tgit rev-list --use-bitmap-index \\\n -+\t\t --objects --filter=object:type=blob --filter-provided tag >actual &&\n ++\t\t --objects --filter-provided --filter=object:type=blob tag >actual &&\n +\ttest_bitmap_traversal expect actual\n +'\n +\n test_expect_success 'combine filter' '\n \tgit rev-list --objects --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&\n \tgit rev-list --use-bitmap-index \\\n +@@ t/t6113-rev-list-bitmap-filters.sh: test_expect_success 'combine filter' '\n + \ttest_bitmap_traversal expect actual\n + '\n + \n ++test_expect_success 'combine filter with --filter-provided' '\n ++\tgit rev-list --objects --filter-provided --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&\n ++\tgit rev-list --use-bitmap-index \\\n ++\t\t --objects --filter-provided --filter=blob:limit=1000 --filter=object:type=blob tag >actual &&\n ++\ttest_bitmap_traversal expect actual &&\n ++\n ++\tgit cat-file --batch-check=\"%(objecttype) %(objectsize)\" objects &&\n ++\twhile read objecttype objectsize\n ++\tdo\n ++\t\ttest \"$objecttype\" = blob || return 1\n ++\t\ttest \"$objectsize\" -le 1000 || return 1\n ++\tdone .allow`. This option will not only\nenable the currently supported set of filters, but also any filters\nwhich get added in the future. As such, an admin which wants to have\ntight control over which filters are allowed and which aren't probably\nshouldn't ever set `uploadpackfilter.allow=true`.\n\nAmend the documentation to make the ramifications more explicit so that\nadmins are aware of this.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/config/uploadpack.txt | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config/uploadpack.txt b/Documentation/config/uploadpack.txt\nindex b0d761282c..6729a072ea 100644\n--- a/Documentation/config/uploadpack.txt\n+++ b/Documentation/config/uploadpack.txt\n@@ -59,7 +59,8 @@ uploadpack.allowFilter::\n \n uploadpackfilter.allow::\n \tProvides a default value for unspecified object filters (see: the\n-\tbelow configuration variable).\n+\tbelow configuration variable). If set to `true`, this will also\n+\tenable all filters which get added in the future.\n \tDefaults to `true`.\n \n uploadpackfilter..allow::\n-- \n2.30.2\n\n"},{"id":"419153","messageId":"ddbec7598664bceee50213a41fefa248d249435e.1615813673.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1615813673.git.ps@pks.im","subject":"[PATCH v2 2/8] revision: mark commit parents as NOT_USER_GIVEN","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-03-15T13:14:36Z","receivedAt":"2021-03-15T13:15:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The NOT_USER_GIVEN flag of an object marks whether a flag was explicitly\nprovided by the user or not. The most important use case for this is\nwhen filtering objects: only objects that were not explicitly requested\nwill get filtered.\n\nThe flag is currently only set for blobs and trees, which has been fine\ngiven that there are no filters for tags or commits currently. We're\nabout to extend filtering capabilities to add object type filter though,\nwhich requires us to set up the NOT_USER_GIVEN flag correctly -- if it's\nnot set, the object wouldn't get filtered at all.\n\nMark unseen commit parents as NOT_USER_GIVEN when processing parents.\nLike this, explicitly provided parents stay user-given and thus\nunfiltered, while parents which get loaded as part of the graph walk\ncan be filtered.\n\nThis commit shouldn't have any user-visible impact yet as there is no\nlogic to filter commits yet.\n\nSigned-off-by: Patrick Steinhardt \n---\n revision.c | 4 ++--\n revision.h | 3 ---\n 2 files changed, 2 insertions(+), 5 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex b78733f508..26f422f50d 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1123,7 +1123,7 @@ static int process_parents(struct rev_info *revs, struct commit *commit,\n \t\t\t\tmark_parents_uninteresting(p);\n \t\t\tif (p->object.flags & SEEN)\n \t\t\t\tcontinue;\n-\t\t\tp->object.flags |= SEEN;\n+\t\t\tp->object.flags |= (SEEN | NOT_USER_GIVEN);\n \t\t\tif (list)\n \t\t\t\tcommit_list_insert_by_date(p, list);\n \t\t\tif (queue)\n@@ -1165,7 +1165,7 @@ static int process_parents(struct rev_info *revs, struct commit *commit,\n \t\t}\n \t\tp->object.flags |= left_flag;\n \t\tif (!(p->object.flags & SEEN)) {\n-\t\t\tp->object.flags |= SEEN;\n+\t\t\tp->object.flags |= (SEEN | NOT_USER_GIVEN);\n \t\t\tif (list)\n \t\t\t\tcommit_list_insert_by_date(p, list);\n \t\t\tif (queue)\ndiff --git a/revision.h b/revision.h\nindex e6be3c845e..f1f324a19b 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -44,9 +44,6 @@\n /*\n * Indicates object was reached by traversal. i.e. not given by user on\n * command-line or stdin.\n- * NEEDSWORK: NOT_USER_GIVEN doesn't apply to commits because we only support\n- * filtering trees and blobs, but it may be useful to support filtering commits\n- * in the future.\n */\n #define NOT_USER_GIVEN\t(1u<<25)\n #define TRACK_LINEAR\t(1u<<26)\n-- \n2.30.2\n\n"},{"id":"419154","messageId":"5545c189c56cb44d0621fb24a531420c29de55d0.1615813673.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1615813673.git.ps@pks.im","subject":"[PATCH v2 4/8] list-objects: support filtering by tag and commit","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-03-15T13:14:45Z","receivedAt":"2021-03-15T13:15:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Object filters currently only support filtering blobs or trees based on\nsome criteria. This commit lays the foundation to also allow filtering\nof tags and commits.\n\nNo change in behaviour is expected from this commit given that there are\nno filters yet for those object types.\n\nSigned-off-by: Patrick Steinhardt \n---\n list-objects-filter.c | 40 ++++++++++++++++++++++++++++++++++++++++\n list-objects-filter.h | 2 ++\n list-objects.c | 24 +++++++++++++++++++++---\n 3 files changed, 63 insertions(+), 3 deletions(-)\n\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex 4ec0041cfb..7def039435 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -82,6 +82,16 @@ static enum list_objects_filter_result filter_blobs_none(\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n+\tcase LOFS_TAG:\n+\t\tassert(obj->type == OBJ_TAG);\n+\t\t/* always include all tag objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\tcase LOFS_COMMIT:\n+\t\tassert(obj->type == OBJ_COMMIT);\n+\t\t/* always include all commit objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\t/* always include all tree objects */\n@@ -173,6 +183,16 @@ static enum list_objects_filter_result filter_trees_depth(\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n+\tcase LOFS_TAG:\n+\t\tassert(obj->type == OBJ_TAG);\n+\t\t/* always include all tag objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\tcase LOFS_COMMIT:\n+\t\tassert(obj->type == OBJ_COMMIT);\n+\t\t/* always include all commit objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n \tcase LOFS_END_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\tfilter_data->current_depth--;\n@@ -267,6 +287,16 @@ static enum list_objects_filter_result filter_blobs_limit(\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n+\tcase LOFS_TAG:\n+\t\tassert(obj->type == OBJ_TAG);\n+\t\t/* always include all tag objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\tcase LOFS_COMMIT:\n+\t\tassert(obj->type == OBJ_COMMIT);\n+\t\t/* always include all commit objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\t/* always include all tree objects */\n@@ -371,6 +401,16 @@ static enum list_objects_filter_result filter_sparse(\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n+\tcase LOFS_TAG:\n+\t\tassert(obj->type == OBJ_TAG);\n+\t\t/* always include all tag objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\tcase LOFS_COMMIT:\n+\t\tassert(obj->type == OBJ_COMMIT);\n+\t\t/* always include all commit objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\tdtype = DT_DIR;\ndiff --git a/list-objects-filter.h b/list-objects-filter.h\nindex cfd784e203..9e98814111 100644\n--- a/list-objects-filter.h\n+++ b/list-objects-filter.h\n@@ -55,6 +55,8 @@ enum list_objects_filter_result {\n };\n \n enum list_objects_filter_situation {\n+\tLOFS_COMMIT,\n+\tLOFS_TAG,\n \tLOFS_BEGIN_TREE,\n \tLOFS_END_TREE,\n \tLOFS_BLOB\ndiff --git a/list-objects.c b/list-objects.c\nindex 093adf85b1..3b63dfd4f2 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -218,8 +218,16 @@ static void process_tag(struct traversal_context *ctx,\n \t\t\tstruct strbuf *base,\n \t\t\tconst char *name)\n {\n-\ttag->object.flags |= SEEN;\n-\tctx->show_object(&tag->object, name, ctx->show_data);\n+\tenum list_objects_filter_result r;\n+\n+\tr = list_objects_filter__filter_object(ctx->revs->repo, LOFS_TAG,\n+\t\t\t\t\t &tag->object, base->buf,\n+\t\t\t\t\t &base->buf[base->len],\n+\t\t\t\t\t ctx->filter);\n+\tif (r & LOFR_MARK_SEEN)\n+\t\ttag->object.flags |= SEEN;\n+\tif (r & LOFR_DO_SHOW)\n+\t\tctx->show_object(&tag->object, name, ctx->show_data);\n }\n \n static void mark_edge_parents_uninteresting(struct commit *commit,\n@@ -369,6 +377,12 @@ static void do_traverse(struct traversal_context *ctx)\n \tstrbuf_init(&csp, PATH_MAX);\n \n \twhile ((commit = get_revision(ctx->revs)) != NULL) {\n+\t\tenum list_objects_filter_result r;\n+\n+\t\tr = list_objects_filter__filter_object(ctx->revs->repo,\n+\t\t\t\tLOFS_COMMIT, &commit->object,\n+\t\t\t\tNULL, NULL, ctx->filter);\n+\n \t\t/*\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n@@ -383,7 +397,11 @@ static void do_traverse(struct traversal_context *ctx)\n \t\t\tdie(_(\"unable to load root tree for commit %s\"),\n \t\t\t oid_to_hex(&commit->object.oid));\n \t\t}\n-\t\tctx->show_commit(commit, ctx->show_data);\n+\n+\t\tif (r & LOFR_MARK_SEEN)\n+\t\t\tcommit->object.flags |= SEEN;\n+\t\tif (r & LOFR_DO_SHOW)\n+\t\t\tctx->show_commit(commit, ctx->show_data);\n \n \t\tif (ctx->revs->tree_blobs_in_commit_order)\n \t\t\t/*\n-- \n2.30.2\n\n"},{"id":"419155","messageId":"d8da0b24f46cd305cb1be304251745b6d9da641b.1615813673.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1615813673.git.ps@pks.im","subject":"[PATCH v2 3/8] list-objects: move tag processing into its own function","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-03-15T13:14:40Z","receivedAt":"2021-03-15T13:15:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Move processing of tags into its own function to make the logic easier\nto extend when we're going to implement filtering for tags. No change in\nbehaviour is expected from this commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n list-objects.c | 12 ++++++++++--\n 1 file changed, 10 insertions(+), 2 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex e19589baa0..093adf85b1 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -213,6 +213,15 @@ static void process_tree(struct traversal_context *ctx,\n \tfree_tree_buffer(tree);\n }\n \n+static void process_tag(struct traversal_context *ctx,\n+\t\t\tstruct tag *tag,\n+\t\t\tstruct strbuf *base,\n+\t\t\tconst char *name)\n+{\n+\ttag->object.flags |= SEEN;\n+\tctx->show_object(&tag->object, name, ctx->show_data);\n+}\n+\n static void mark_edge_parents_uninteresting(struct commit *commit,\n \t\t\t\t\t struct rev_info *revs,\n \t\t\t\t\t show_edge_fn show_edge)\n@@ -334,8 +343,7 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx,\n \t\tif (obj->flags & (UNINTERESTING | SEEN))\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n-\t\t\tobj->flags |= SEEN;\n-\t\t\tctx->show_object(obj, name, ctx->show_data);\n+\t\t\tprocess_tag(ctx, (struct tag *)obj, base, name);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\n-- \n2.30.2\n\n"},{"id":"419156","messageId":"acf01472af9a8e370f549d57593a4f9924c24f7c.1615813673.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1615813673.git.ps@pks.im","subject":"[PATCH v2 5/8] list-objects: implement object type filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-03-15T13:14:50Z","receivedAt":"2021-03-15T13:15:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"While it already is possible to filter objects by some criteria in\ngit-rev-list(1), it is not yet possible to filter out only a specific\ntype of objects. This makes some filters less useful. The `blob:limit`\nfilter for example filters blobs such that only those which are smaller\nthan the given limit are returned. But it is unfit to ask only for these\nsmallish blobs, given that git-rev-list(1) will continue to print tags,\ncommits and trees.\n\nNow that we have the infrastructure in place to also filter tags and\ncommits, we can improve this situation by implementing a new filter\nwhich selects objects based on their type. Above query can thus\ntrivially be implemented with the following command:\n\n $ git rev-list --objects --filter=object:type=blob \\\n --filter=blob:limit=200\n\nFurthermore, this filter allows to optimize for certain other cases: if\nfor example only tags or commits have been selected, there is no need to\nwalk down trees.\n\nThe new filter is not yet supported in bitmaps. This is going to be\nimplemented in a subsequent commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/config/uploadpack.txt | 6 +--\n Documentation/rev-list-options.txt | 3 ++\n list-objects-filter-options.c | 14 ++++++\n list-objects-filter-options.h | 2 +\n list-objects-filter.c | 76 +++++++++++++++++++++++++++++\n t/t6112-rev-list-filters-objects.sh | 48 ++++++++++++++++++\n 6 files changed, 146 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/config/uploadpack.txt b/Documentation/config/uploadpack.txt\nindex 6729a072ea..32fad5bbe8 100644\n--- a/Documentation/config/uploadpack.txt\n+++ b/Documentation/config/uploadpack.txt\n@@ -66,9 +66,9 @@ uploadpackfilter.allow::\n uploadpackfilter..allow::\n \tExplicitly allow or ban the object filter corresponding to\n \t``, where `` may be one of: `blob:none`,\n-\t`blob:limit`, `tree`, `sparse:oid`, or `combine`. If using\n-\tcombined filters, both `combine` and all of the nested filter\n-\tkinds must be allowed. Defaults to `uploadpackfilter.allow`.\n+\t`blob:limit`, `object:type`, `tree`, `sparse:oid`, or `combine`.\n+\tIf using combined filters, both `combine` and all of the nested\n+\tfilter kinds must be allowed. Defaults to `uploadpackfilter.allow`.\n \n uploadpackfilter.tree.maxDepth::\n \tOnly allow `--filter=tree:` when `` is no more than the value of\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex b1c8f86c6e..3afa8fffbd 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -892,6 +892,9 @@ or units. n may be zero. The suffixes k, m, and g can be used to name\n units in KiB, MiB, or GiB. For example, 'blob:limit=1k' is the same\n as 'blob:limit=1024'.\n +\n+The form '--filter=object:type=(tag|commit|tree|blob)' omits all objects\n+which are not of the requested type.\n++\n The form '--filter=sparse:oid=' uses a sparse-checkout\n specification contained in the blob (or blob-expression) ''\n to omit blobs that would not be not required for a sparse checkout on\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex d2d1c81caf..bb6f6577d5 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -29,6 +29,8 @@ const char *list_object_filter_config_name(enum list_objects_filter_choice c)\n \t\treturn \"tree\";\n \tcase LOFC_SPARSE_OID:\n \t\treturn \"sparse:oid\";\n+\tcase LOFC_OBJECT_TYPE:\n+\t\treturn \"object:type\";\n \tcase LOFC_COMBINE:\n \t\treturn \"combine\";\n \tcase LOFC__COUNT:\n@@ -97,6 +99,18 @@ static int gently_parse_list_objects_filter(\n \t\t}\n \t\treturn 1;\n \n+\t} else if (skip_prefix(arg, \"object:type=\", &v0)) {\n+\t\tint type = type_from_string_gently(v0, -1, 1);\n+\t\tif (type < 0) {\n+\t\t\tstrbuf_addstr(errbuf, _(\"expected 'object:type='\"));\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\tfilter_options->object_type = type;\n+\t\tfilter_options->choice = LOFC_OBJECT_TYPE;\n+\n+\t\treturn 0;\n+\n \t} else if (skip_prefix(arg, \"combine:\", &v0)) {\n \t\treturn parse_combine_filter(filter_options, v0, errbuf);\n \ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 01767c3c96..4d0d0588cc 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -13,6 +13,7 @@ enum list_objects_filter_choice {\n \tLOFC_BLOB_LIMIT,\n \tLOFC_TREE_DEPTH,\n \tLOFC_SPARSE_OID,\n+\tLOFC_OBJECT_TYPE,\n \tLOFC_COMBINE,\n \tLOFC__COUNT /* must be last */\n };\n@@ -54,6 +55,7 @@ struct list_objects_filter_options {\n \tchar *sparse_oid_name;\n \tunsigned long blob_limit_value;\n \tunsigned long tree_exclude_depth;\n+\tenum object_type object_type;\n \n \t/* LOFC_COMBINE values */\n \ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex 7def039435..650a7c2c80 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -545,6 +545,81 @@ static void filter_sparse_oid__init(\n \tfilter->free_fn = filter_sparse_free;\n }\n \n+/*\n+ * A filter for list-objects to omit large blobs.\n+ * And to OPTIONALLY collect a list of the omitted OIDs.\n+ */\n+struct filter_object_type_data {\n+\tenum object_type object_type;\n+};\n+\n+static enum list_objects_filter_result filter_object_type(\n+\tstruct repository *r,\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tstruct oidset *omits,\n+\tvoid *filter_data_)\n+{\n+\tstruct filter_object_type_data *filter_data = filter_data_;\n+\n+\tswitch (filter_situation) {\n+\tdefault:\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n+\n+\tcase LOFS_TAG:\n+\t\tassert(obj->type == OBJ_TAG);\n+\t\tif (filter_data->object_type == OBJ_TAG)\n+\t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\t\treturn LOFR_MARK_SEEN;\n+\n+\tcase LOFS_COMMIT:\n+\t\tassert(obj->type == OBJ_COMMIT);\n+\t\tif (filter_data->object_type == OBJ_COMMIT)\n+\t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\t\treturn LOFR_MARK_SEEN;\n+\n+\tcase LOFS_BEGIN_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\n+\t\t/*\n+\t\t * If we only want to show commits or tags, then there is no\n+\t\t * need to walk down trees.\n+\t\t */\n+\t\tif (filter_data->object_type == OBJ_COMMIT ||\n+\t\t filter_data->object_type == OBJ_TAG)\n+\t\t\treturn LOFR_SKIP_TREE;\n+\n+\t\tif (filter_data->object_type == OBJ_TREE)\n+\t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\t\treturn LOFR_MARK_SEEN;\n+\n+\tcase LOFS_BLOB:\n+\t\tassert(obj->type == OBJ_BLOB);\n+\n+\t\tif (filter_data->object_type == OBJ_BLOB)\n+\t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\t\treturn LOFR_MARK_SEEN;\n+\n+\tcase LOFS_END_TREE:\n+\t\treturn LOFR_ZERO;\n+\t}\n+}\n+\n+static void filter_object_type__init(\n+\tstruct list_objects_filter_options *filter_options,\n+\tstruct filter *filter)\n+{\n+\tstruct filter_object_type_data *d = xcalloc(1, sizeof(*d));\n+\td->object_type = filter_options->object_type;\n+\n+\tfilter->filter_data = d;\n+\tfilter->filter_object_fn = filter_object_type;\n+\tfilter->free_fn = free;\n+}\n+\n /* A filter which only shows objects shown by all sub-filters. */\n struct combine_filter_data {\n \tstruct subfilter *sub;\n@@ -691,6 +766,7 @@ static filter_init_fn s_filters[] = {\n \tfilter_blobs_limit__init,\n \tfilter_trees_depth__init,\n \tfilter_sparse_oid__init,\n+\tfilter_object_type__init,\n \tfilter_combine__init,\n };\n \ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex 31457d13b9..c79ec04060 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -159,6 +159,54 @@ test_expect_success 'verify blob:limit=1m' '\n \ttest_must_be_empty observed\n '\n \n+# Test object:type= filter.\n+\n+test_expect_success 'setup object-type' '\n+\tgit init object-type &&\n+\techo contents >object-type/blob &&\n+\tgit -C object-type add blob &&\n+\tgit -C object-type commit -m commit-message &&\n+\tgit -C object-type tag tag -m tag-message\n+'\n+\n+test_expect_success 'verify object:type= fails with invalid type' '\n+\ttest_must_fail git -C object-type rev-list --objects --filter=object:type= HEAD &&\n+\ttest_must_fail git -C object-type rev-list --objects --filter=object:type=invalid HEAD\n+'\n+\n+test_expect_success 'verify object:type=blob prints blob and commit' '\n+\t(\n+\t\tgit -C object-type rev-parse HEAD &&\n+\t\tprintf \"%s blob\\n\" $(git -C object-type rev-parse HEAD:blob)\n+\t) >expected &&\n+\tgit -C object-type rev-list --objects --filter=object:type=blob HEAD >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=tree prints tree and commit' '\n+\t(\n+\t\tgit -C object-type rev-parse HEAD &&\n+\t\tprintf \"%s \\n\" $(git -C object-type rev-parse HEAD^{tree})\n+\t) >expected &&\n+\tgit -C object-type rev-list --objects --filter=object:type=tree HEAD >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=commit prints commit' '\n+\tgit -C object-type rev-parse HEAD >expected &&\n+\tgit -C object-type rev-list --objects --filter=object:type=commit HEAD >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=tag prints tag' '\n+\t(\n+\t\tgit -C object-type rev-parse HEAD &&\n+\t\tprintf \"%s tag\\n\" $(git -C object-type rev-parse tag)\n+\t) >expected &&\n+\tgit -C object-type rev-list --objects --filter=object:type=tag tag >actual &&\n+\ttest_cmp expected actual\n+'\n+\n # Test sparse:path= filter.\n # !!!!\n # NOTE: sparse:path filter support has been dropped for security reasons,\n-- \n2.30.2\n\n"},{"id":"419157","messageId":"fac3477d979058da0430b974a34f7c7f866bf456.1615813673.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1615813673.git.ps@pks.im","subject":"[PATCH v2 7/8] pack-bitmap: implement combined filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-03-15T13:14:59Z","receivedAt":"2021-03-15T13:15:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When the user has multiple objects filters specified, then this is\ninternally represented by having a \"combined\" filter. These combined\nfilters aren't yet supported by bitmap indices and can thus not be\naccelerated.\n\nFix this by implementing support for these combined filters. The\nimplementation is quite trivial: when there's a combined filter, we\nsimply recurse into `filter_bitmap()` for all of the sub-filters.\n\nSigned-off-by: Patrick Steinhardt \n---\n pack-bitmap.c | 40 +++++++++++++++++++++++++++---\n t/t6113-rev-list-bitmap-filters.sh | 7 ++++++\n 2 files changed, 43 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 196d38c91d..e33805e076 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -925,6 +925,29 @@ static void filter_bitmap_object_type(struct bitmap_index *bitmap_git,\n \t}\n }\n \n+static int filter_supported(struct list_objects_filter_options *filter)\n+{\n+\tint i;\n+\n+\tswitch (filter->choice) {\n+\tcase LOFC_BLOB_NONE:\n+\tcase LOFC_BLOB_LIMIT:\n+\tcase LOFC_OBJECT_TYPE:\n+\t\treturn 1;\n+\tcase LOFC_TREE_DEPTH:\n+\t\tif (filter->tree_exclude_depth == 0)\n+\t\t\treturn 1;\n+\t\treturn 0;\n+\tcase LOFC_COMBINE:\n+\t\tfor (i = 0; i < filter->sub_nr; i++)\n+\t\t\tif (!filter_supported(&filter->sub[i]))\n+\t\t\t\treturn 0;\n+\t\treturn 1;\n+\tdefault:\n+\t\treturn 0;\n+\t}\n+}\n+\n static int filter_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t struct object_list *tip_objects,\n \t\t\t struct bitmap *to_filter,\n@@ -932,6 +955,8 @@ static int filter_bitmap(struct bitmap_index *bitmap_git,\n {\n \tif (!filter || filter->choice == LOFC_DISABLED)\n \t\treturn 0;\n+\tif (!filter_supported(filter))\n+\t\treturn -1;\n \n \tif (filter->choice == LOFC_BLOB_NONE) {\n \t\tif (bitmap_git)\n@@ -948,8 +973,7 @@ static int filter_bitmap(struct bitmap_index *bitmap_git,\n \t\treturn 0;\n \t}\n \n-\tif (filter->choice == LOFC_TREE_DEPTH &&\n-\t filter->tree_exclude_depth == 0) {\n+\tif (filter->choice == LOFC_TREE_DEPTH) {\n \t\tif (bitmap_git)\n \t\t\tfilter_bitmap_tree_depth(bitmap_git, tip_objects,\n \t\t\t\t\t\t to_filter,\n@@ -965,8 +989,16 @@ static int filter_bitmap(struct bitmap_index *bitmap_git,\n \t\treturn 0;\n \t}\n \n-\t/* filter choice not handled */\n-\treturn -1;\n+\tif (filter->choice == LOFC_COMBINE) {\n+\t\tint i;\n+\t\tfor (i = 0; i < filter->sub_nr; i++) {\n+\t\t\tfilter_bitmap(bitmap_git, tip_objects, to_filter,\n+\t\t\t\t &filter->sub[i]);\n+\t\t}\n+\t\treturn 0;\n+\t}\n+\n+\tBUG(\"unsupported filter choice\");\n }\n \n static int can_filter_bitmap(struct list_objects_filter_options *filter)\ndiff --git a/t/t6113-rev-list-bitmap-filters.sh b/t/t6113-rev-list-bitmap-filters.sh\nindex fb66735ac8..cb9db7df6f 100755\n--- a/t/t6113-rev-list-bitmap-filters.sh\n+++ b/t/t6113-rev-list-bitmap-filters.sh\n@@ -98,4 +98,11 @@ test_expect_success 'object:type filter' '\n \ttest_bitmap_traversal expect actual\n '\n \n+test_expect_success 'combine filter' '\n+\tgit rev-list --objects --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=blob:limit=1000 --filter=object:type=blob tag >actual &&\n+\ttest_bitmap_traversal expect actual\n+'\n+\n test_done\n-- \n2.30.2\n\n"},{"id":"419158","messageId":"0e26fee8b31e46e87fb9fa1ac599506502a9d622.1615813673.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1615813673.git.ps@pks.im","subject":"[PATCH v2 8/8] rev-list: allow filtering of provided items","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-03-15T13:15:05Z","receivedAt":"2021-03-15T13:16:01Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When providing an object filter, it is currently impossible to also\nfilter provided items. E.g. when executing `git rev-list HEAD` , the\ncommit this reference points to will be treated as user-provided and is\nthus excluded from the filtering mechanism. This makes it harder than\nnecessary to properly use the new `--filter=object:type` filter given\nthat even if the user wants to only see blobs, he'll still see commits\nof provided references.\n\nImprove this by introducing a new `--filter-provided` option to the\ngit-rev-parse(1) command. If given, then all user-provided references\nwill be subject to filtering.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/rev-list.c | 14 +++++++++++\n list-objects-filter-options.c | 4 ++++\n list-objects-filter-options.h | 6 +++++\n pack-bitmap.c | 3 ++-\n t/t6112-rev-list-filters-objects.sh | 28 ++++++++++++++++++++++\n t/t6113-rev-list-bitmap-filters.sh | 36 +++++++++++++++++++++++++++++\n 6 files changed, 90 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex b4d8ea0a35..0f959b266d 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -599,6 +599,10 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\tlist_objects_filter_set_no_filter(&filter_options);\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!strcmp(arg, \"--filter-provided\")) {\n+\t\t\tfilter_options.filter_wants = 1;\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!strcmp(arg, \"--filter-print-omitted\")) {\n \t\t\targ_print_omitted = 1;\n \t\t\tcontinue;\n@@ -694,6 +698,16 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\treturn show_bisect_vars(&info, reaches, all);\n \t}\n \n+\tif (filter_options.filter_wants) {\n+\t\tstruct commit_list *c;\n+\t\tfor (i = 0; i < revs.pending.nr; i++) {\n+\t\t\tstruct object_array_entry *pending = revs.pending.objects + i;\n+\t\t\tpending->item->flags |= NOT_USER_GIVEN;\n+\t\t}\n+\t\tfor (c = revs.commits; c; c = c->next)\n+\t\t\tc->item->object.flags |= NOT_USER_GIVEN;\n+\t}\n+\n \tif (arg_print_omitted)\n \t\toidset_init(&omitted_objects, DEFAULT_OIDSET_SIZE);\n \tif (arg_missing_action == MA_PRINT)\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex bb6f6577d5..2877aa9e96 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -242,6 +242,7 @@ static void transform_to_combine_type(\n \t\tmemset(filter_options, 0, sizeof(*filter_options));\n \t\tfilter_options->sub = sub_array;\n \t\tfilter_options->sub_alloc = initial_sub_alloc;\n+\t\tfilter_options->filter_wants = sub_array[0].filter_wants;\n \t}\n \tfilter_options->sub_nr = 1;\n \tfilter_options->choice = LOFC_COMBINE;\n@@ -290,6 +291,9 @@ void parse_list_objects_filter(\n \t\tparse_error = gently_parse_list_objects_filter(\n \t\t\t&filter_options->sub[filter_options->sub_nr - 1], arg,\n \t\t\t&errbuf);\n+\t\tif (!parse_error)\n+\t\t\tfilter_options->sub[filter_options->sub_nr - 1].filter_wants =\n+\t\t\t\tfilter_options->filter_wants;\n \t}\n \tif (parse_error)\n \t\tdie(\"%s\", errbuf.buf);\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 4d0d0588cc..5e609e307a 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -42,6 +42,12 @@ struct list_objects_filter_options {\n \t */\n \tenum list_objects_filter_choice choice;\n \n+\t/*\n+\t * \"--filter-provided\" was given by the user, instructing us to also\n+\t * filter all explicitly provided objects.\n+\t */\n+\tunsigned int filter_wants : 1;\n+\n \t/*\n \t * Choice is LOFC_DISABLED because \"--no-filter\" was requested.\n \t */\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex e33805e076..5ff800316b 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1101,7 +1101,8 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \tif (haves_bitmap)\n \t\tbitmap_and_not(wants_bitmap, haves_bitmap);\n \n-\tfilter_bitmap(bitmap_git, wants, wants_bitmap, filter);\n+\tfilter_bitmap(bitmap_git, (filter && filter->filter_wants) ? NULL : wants,\n+\t\t wants_bitmap, filter);\n \n \tbitmap_git->result = wants_bitmap;\n \tbitmap_git->haves = haves_bitmap;\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex c79ec04060..47c558ab0e 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -207,6 +207,34 @@ test_expect_success 'verify object:type=tag prints tag' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'verify object:type=blob prints only blob with --filter-provided' '\n+\tprintf \"%s blob\\n\" $(git -C object-type rev-parse HEAD:blob) >expected &&\n+\tgit -C object-type rev-list --objects \\\n+\t\t--filter=object:type=blob --filter-provided HEAD >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=tree prints only tree with --filter-provided' '\n+\tprintf \"%s \\n\" $(git -C object-type rev-parse HEAD^{tree}) >expected &&\n+\tgit -C object-type rev-list --objects \\\n+\t\t--filter=object:type=tree HEAD --filter-provided >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=commit prints only commit with --filter-provided' '\n+\tgit -C object-type rev-parse HEAD >expected &&\n+\tgit -C object-type rev-list --objects \\\n+\t\t--filter=object:type=commit --filter-provided HEAD >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=tag prints only tag with --filter-provided' '\n+\tprintf \"%s tag\\n\" $(git -C object-type rev-parse tag) >expected &&\n+\tgit -C object-type rev-list --objects \\\n+\t\t--filter=object:type=tag --filter-provided tag >actual &&\n+\ttest_cmp expected actual\n+'\n+\n # Test sparse:path= filter.\n # !!!!\n # NOTE: sparse:path filter support has been dropped for security reasons,\ndiff --git a/t/t6113-rev-list-bitmap-filters.sh b/t/t6113-rev-list-bitmap-filters.sh\nindex cb9db7df6f..9053ac5059 100755\n--- a/t/t6113-rev-list-bitmap-filters.sh\n+++ b/t/t6113-rev-list-bitmap-filters.sh\n@@ -98,6 +98,28 @@ test_expect_success 'object:type filter' '\n \ttest_bitmap_traversal expect actual\n '\n \n+test_expect_success 'object:type filter with --filter-provided' '\n+\tgit rev-list --objects --filter-provided --filter=object:type=tag tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter-provided --filter=object:type=tag tag >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tgit rev-list --objects --filter-provided --filter=object:type=commit tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter-provided --filter=object:type=commit tag >actual &&\n+\ttest_bitmap_traversal expect actual &&\n+\n+\tgit rev-list --objects --filter-provided --filter=object:type=tree tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter-provided --filter=object:type=tree tag >actual &&\n+\ttest_bitmap_traversal expect actual &&\n+\n+\tgit rev-list --objects --filter-provided --filter=object:type=blob tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter-provided --filter=object:type=blob tag >actual &&\n+\ttest_bitmap_traversal expect actual\n+'\n+\n test_expect_success 'combine filter' '\n \tgit rev-list --objects --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&\n \tgit rev-list --use-bitmap-index \\\n@@ -105,4 +127,18 @@ test_expect_success 'combine filter' '\n \ttest_bitmap_traversal expect actual\n '\n \n+test_expect_success 'combine filter with --filter-provided' '\n+\tgit rev-list --objects --filter-provided --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter-provided --filter=blob:limit=1000 --filter=object:type=blob tag >actual &&\n+\ttest_bitmap_traversal expect actual &&\n+\n+\tgit cat-file --batch-check=\"%(objecttype) %(objectsize)\" objects &&\n+\twhile read objecttype objectsize\n+\tdo\n+\t\ttest \"$objecttype\" = blob || return 1\n+\t\ttest \"$objectsize\" -le 1000 || return 1\n+\tdone \n---\n pack-bitmap.c | 28 +++++++++++++++++++++++++---\n t/t6113-rev-list-bitmap-filters.sh | 25 ++++++++++++++++++++++++-\n 2 files changed, 49 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 1f69b5fa85..196d38c91d 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -779,9 +779,6 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n \teword_t mask;\n \tuint32_t i;\n \n-\tif (type != OBJ_BLOB && type != OBJ_TREE)\n-\t\tBUG(\"filter_bitmap_exclude_type: unsupported type '%d'\", type);\n-\n \t/*\n \t * The non-bitmap version of this filter never removes\n \t * objects which the other side specifically asked for,\n@@ -911,6 +908,23 @@ static void filter_bitmap_tree_depth(struct bitmap_index *bitmap_git,\n \t\t\t\t OBJ_BLOB);\n }\n \n+static void filter_bitmap_object_type(struct bitmap_index *bitmap_git,\n+\t\t\t\t struct object_list *tip_objects,\n+\t\t\t\t struct bitmap *to_filter,\n+\t\t\t\t enum object_type object_type)\n+{\n+\tenum object_type t;\n+\n+\tif (object_type < OBJ_COMMIT || object_type > OBJ_TAG)\n+\t\tBUG(\"filter_bitmap_object_type given invalid object\");\n+\n+\tfor (t = OBJ_COMMIT; t <= OBJ_TAG; t++) {\n+\t\tif (t == object_type)\n+\t\t\tcontinue;\n+\t\tfilter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, t);\n+\t}\n+}\n+\n static int filter_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t struct object_list *tip_objects,\n \t\t\t struct bitmap *to_filter,\n@@ -943,6 +957,14 @@ static int filter_bitmap(struct bitmap_index *bitmap_git,\n \t\treturn 0;\n \t}\n \n+\tif (filter->choice == LOFC_OBJECT_TYPE) {\n+\t\tif (bitmap_git)\n+\t\t\tfilter_bitmap_object_type(bitmap_git, tip_objects,\n+\t\t\t\t\t\t to_filter,\n+\t\t\t\t\t\t filter->object_type);\n+\t\treturn 0;\n+\t}\n+\n \t/* filter choice not handled */\n \treturn -1;\n }\ndiff --git a/t/t6113-rev-list-bitmap-filters.sh b/t/t6113-rev-list-bitmap-filters.sh\nindex 3f889949ca..fb66735ac8 100755\n--- a/t/t6113-rev-list-bitmap-filters.sh\n+++ b/t/t6113-rev-list-bitmap-filters.sh\n@@ -10,7 +10,8 @@ test_expect_success 'set up bitmapped repo' '\n \ttest_commit much-larger-blob-one &&\n \tgit repack -adb &&\n \ttest_commit two &&\n-\ttest_commit much-larger-blob-two\n+\ttest_commit much-larger-blob-two &&\n+\tgit tag tag\n '\n \n test_expect_success 'filters fallback to non-bitmap traversal' '\n@@ -75,4 +76,26 @@ test_expect_success 'tree:1 filter' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'object:type filter' '\n+\tgit rev-list --objects --filter=object:type=tag tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=object:type=tag tag >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tgit rev-list --objects --filter=object:type=commit tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=object:type=commit tag >actual &&\n+\ttest_bitmap_traversal expect actual &&\n+\n+\tgit rev-list --objects --filter=object:type=tree tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=object:type=tree tag >actual &&\n+\ttest_bitmap_traversal expect actual &&\n+\n+\tgit rev-list --objects --filter=object:type=blob tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=object:type=blob tag >actual &&\n+\ttest_bitmap_traversal expect actual\n+'\n+\n test_done\n-- \n2.30.2\n\n"},{"id":"419781","messageId":"xmqqblbdjzu6.fsf@gitster.g","threadId":"55236","inReplyTo":"cover.1615813673.git.ps@pks.im","subject":"Re: [PATCH v2 0/8] rev-parse: implement object type filter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-03-20T21:10:41Z","receivedAt":"2021-03-20T21:11:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> this is the second version of my patch series which implements a new\n> `object:type` filter for git-rev-parse(1) and git-upload-pack(1) and\n> extends support for bitmap indices to work with combined filters.\n> ...\n> Please see the attached range-diff for more details.\n\nAny comment from stakeholders?\n\nThanks.\n"},{"id":"421072","messageId":"YGyXrVElnVSALuDU@coredump.intra.peff.net","threadId":"55236","inReplyTo":"270ff80dacff96f441e12954b059a68300157f2d.1615813673.git.ps@pks.im","subject":"Re: [PATCH v2 1/8] uploadpack.txt: document implication of `uploadpackfilter.allow`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-06T17:17:33Z","receivedAt":"2021-04-06T17:17:36Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 15, 2021 at 02:14:31PM +0100, Patrick Steinhardt wrote:\n\n> When `uploadpackfilter.allow` is set to `true`, it means that filters\n> are enabled by default except in the case where a filter is explicitly\n> disabled via `uploadpackilter..allow`. This option will not only\n> enable the currently supported set of filters, but also any filters\n> which get added in the future. As such, an admin which wants to have\n> tight control over which filters are allowed and which aren't probably\n> shouldn't ever set `uploadpackfilter.allow=true`.\n> \n> Amend the documentation to make the ramifications more explicit so that\n> admins are aware of this.\n\nIt might help to guide the admin a bit more here. What are we really\nworried about? Probably that an expensive filter would be added that\nwould make an admin with a public-facing server unhappy.\n\nMaybe we should be more explicit about our recommendations, like:\n\n This defaults to `true` for historical reasons, but that includes\n expensive-to-compute filters (both existing ones like `sparse`, but\n also future ones). A safer value is to set this to `false` and\n mark individual filters as allowed.\n\nBut then of course somebody wonders which set are expensive and which\nones are not. And really, \"expensive\" here is not that expensive. It is\n\"do not support bitmaps\".\n\nSo I wonder if this concern is overblown in the first place. People who\ncare about using only bitmap-supported filters probably already set this\nto \"false\". And vaguely calling things \"expensive\" is probably being\noverly scary. But in that case, I'm not sure we even need to add a\nreminder that future ones will also be enabled (OTOH, I do not mind it\nso much; it is encouraging people to set this to false and mark\nindividual ones as allowed).\n\n-Peff\n"},{"id":"421073","messageId":"YGya0XY969HlyPWl@coredump.intra.peff.net","threadId":"55236","inReplyTo":"ddbec7598664bceee50213a41fefa248d249435e.1615813673.git.ps@pks.im","subject":"Re: [PATCH v2 2/8] revision: mark commit parents as NOT_USER_GIVEN","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-06T17:30:57Z","receivedAt":"2021-04-06T17:31:01Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 15, 2021 at 02:14:36PM +0100, Patrick Steinhardt wrote:\n\n> The NOT_USER_GIVEN flag of an object marks whether a flag was explicitly\n> provided by the user or not. The most important use case for this is\n> when filtering objects: only objects that were not explicitly requested\n> will get filtered.\n> \n> The flag is currently only set for blobs and trees, which has been fine\n> given that there are no filters for tags or commits currently. We're\n> about to extend filtering capabilities to add object type filter though,\n> which requires us to set up the NOT_USER_GIVEN flag correctly -- if it's\n> not set, the object wouldn't get filtered at all.\n> \n> Mark unseen commit parents as NOT_USER_GIVEN when processing parents.\n> Like this, explicitly provided parents stay user-given and thus\n> unfiltered, while parents which get loaded as part of the graph walk\n> can be filtered.\n> \n> This commit shouldn't have any user-visible impact yet as there is no\n> logic to filter commits yet.\n\nI'm still scratching my head a bit to understand how NOT_USER_GIVEN can\npossibly be correct (as opposed to USER_GIVEN). If we visit the commit\nin a not-user-given context and add the flag, how do we know it wasn't\n_also_ visited in a user-given context?\n\nJust guessing, but perhaps the SEEN flag is saving us here? If we visit\nthe user-given commit itself first, then we give it the SEEN flag. Then\nif we try to visit it again via parent traversal, we've already\nprocessed it and don't add the NOT_USER_GIVEN flag here.\n\nThat seems the opposite of the order we'd usually traverse, but I think\nwe set SEEN on each commit in prepare_revision_walk(), before we do any\ntraversing.\n\nSo I _think_ it all works even with your changes here, but I have to say\nthis NOT_USER_GIVEN thing seems really fragile to me. Not new in your\nseries, of course, but something we may want to look at.\n\nJust grepping around, \"rev-list -g\" will happily remove SEEN flags, so I\nsuspect it interacts badly with --filter. Just trying \"rev-list -g\n--objects --filter=object:type=blob HEAD\" shows that it produces quite a\nlot of commits (which I think is a more fundamental problem: it is not\nwalking the parent chain at all to assign these NOT_USER_GIVEN flags).\n\n-Peff\n"},{"id":"421074","messageId":"YGycyLTZflh3olRE@coredump.intra.peff.net","threadId":"55236","inReplyTo":"d8da0b24f46cd305cb1be304251745b6d9da641b.1615813673.git.ps@pks.im","subject":"Re: [PATCH v2 3/8] list-objects: move tag processing into its own function","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-06T17:39:20Z","receivedAt":"2021-04-06T17:39:22Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 15, 2021 at 02:14:40PM +0100, Patrick Steinhardt wrote:\n\n> Move processing of tags into its own function to make the logic easier\n> to extend when we're going to implement filtering for tags. No change in\n> behaviour is expected from this commit.\n\nMakes sense. Even without extending the logic, it is nice to see the\nsymmetric with the tree/blob paths.\n\nAlthough I think it's not quite symmetric in practice...\n\n> +static void process_tag(struct traversal_context *ctx,\n> +\t\t\tstruct tag *tag,\n> +\t\t\tstruct strbuf *base,\n> +\t\t\tconst char *name)\n> +{\n> +\ttag->object.flags |= SEEN;\n> +\tctx->show_object(&tag->object, name, ctx->show_data);\n> +}\n\nI'm skeptical that \"base\" will ever be meaningful here (as it would be\nfor trees and blobs), because we are never recursing a tree to hit a\ntag. We do later pass it to filter_object(), but I think it will always\nbe the empty string (we even assert(base->len == 0) in the caller).\n\nSo I am tempted to say it should not take a base parameter at all, and\nlater the call to filter_object() added to process_tag() should just\npass an empty string as the base. That would make it clear we do not\nexpect any kind of \"base\". That's mostly academic, but I think it also\nmakes clear that the \"name\" field is not something that should be\nappended to the base (unlike trees and blobs, it is the name we got from\nthe top-level parsing, not a pathname).\n\n-Peff\n"},{"id":"421075","messageId":"YGydoFoIROcqFGHC@coredump.intra.peff.net","threadId":"55236","inReplyTo":"acf01472af9a8e370f549d57593a4f9924c24f7c.1615813673.git.ps@pks.im","subject":"Re: [PATCH v2 5/8] list-objects: implement object type filter","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-06T17:42:56Z","receivedAt":"2021-04-06T17:42:59Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 15, 2021 at 02:14:50PM +0100, Patrick Steinhardt wrote:\n\n> While it already is possible to filter objects by some criteria in\n> git-rev-list(1), it is not yet possible to filter out only a specific\n> type of objects. This makes some filters less useful. The `blob:limit`\n> filter for example filters blobs such that only those which are smaller\n> than the given limit are returned. But it is unfit to ask only for these\n> smallish blobs, given that git-rev-list(1) will continue to print tags,\n> commits and trees.\n> \n> Now that we have the infrastructure in place to also filter tags and\n> commits, we can improve this situation by implementing a new filter\n> which selects objects based on their type. Above query can thus\n> trivially be implemented with the following command:\n> \n> $ git rev-list --objects --filter=object:type=blob \\\n> --filter=blob:limit=200\n> \n> Furthermore, this filter allows to optimize for certain other cases: if\n> for example only tags or commits have been selected, there is no need to\n> walk down trees.\n\nMakes sense, and the implementation looks reasonable to me.\n\n-Peff\n"},{"id":"421076","messageId":"YGyfBc9I6YtyDWSr@coredump.intra.peff.net","threadId":"55236","inReplyTo":"8073ab665b07cf653478482f801a06e072233230.1615813673.git.ps@pks.im","subject":"Re: [PATCH v2 6/8] pack-bitmap: implement object type filter","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-06T17:48:53Z","receivedAt":"2021-04-06T17:48:56Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 15, 2021 at 02:14:55PM +0100, Patrick Steinhardt wrote:\n\n> The preceding commit has added a new object filter for git-rev-list(1)\n> which allows to filter objects by type. Implement the equivalent filter\n> for packfile bitmaps so that we can answer these queries fast.\n\nMakes sense. The implementation looks pretty sensible. One observation:\n\n> +static void filter_bitmap_object_type(struct bitmap_index *bitmap_git,\n> +\t\t\t\t struct object_list *tip_objects,\n> +\t\t\t\t struct bitmap *to_filter,\n> +\t\t\t\t enum object_type object_type)\n> +{\n> +\tenum object_type t;\n> +\n> +\tif (object_type < OBJ_COMMIT || object_type > OBJ_TAG)\n> +\t\tBUG(\"filter_bitmap_object_type given invalid object\");\n> +\n> +\tfor (t = OBJ_COMMIT; t <= OBJ_TAG; t++) {\n> +\t\tif (t == object_type)\n> +\t\t\tcontinue;\n> +\t\tfilter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, t);\n> +\t}\n> +}\n\nThe OBJ_* constants are a contiguous set between COMMIT and TAG, and it\nhas to remain this way (because we use them to decipher the type fields\nin pack files). But I don't think we've generally baked that assumption\ninto the code in this way.\n\nWriting it out long-hand would be something like:\n\n if (t != OBJ_COMMIT)\n\tfilter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, OBJ_COMMIT);\n if (t != OBJ_TREE)\n\tfilter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, OBJ_TREE);\n\nand so on, which isn't too bad. I dunno. That may be overly picky.\n\n-Peff\n"},{"id":"421077","messageId":"YGygV1mesbJzdPr5@coredump.intra.peff.net","threadId":"55236","inReplyTo":"fac3477d979058da0430b974a34f7c7f866bf456.1615813673.git.ps@pks.im","subject":"Re: [PATCH v2 7/8] pack-bitmap: implement combined filter","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-06T17:54:31Z","receivedAt":"2021-04-06T17:54:34Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 15, 2021 at 02:14:59PM +0100, Patrick Steinhardt wrote:\n\n> When the user has multiple objects filters specified, then this is\n> internally represented by having a \"combined\" filter. These combined\n> filters aren't yet supported by bitmap indices and can thus not be\n> accelerated.\n> \n> Fix this by implementing support for these combined filters. The\n> implementation is quite trivial: when there's a combined filter, we\n> simply recurse into `filter_bitmap()` for all of the sub-filters.\n\nThe goal makes sense.\n\nBefore this patch, I think your test:\n\n> +test_expect_success 'combine filter' '\n> +\tgit rev-list --objects --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&\n> +\tgit rev-list --use-bitmap-index \\\n> +\t\t --objects --filter=blob:limit=1000 --filter=object:type=blob tag >actual &&\n> +\ttest_bitmap_traversal expect actual\n> +'\n\nwould pass anyway, because we'd just skip using bitmaps. Is there a way\nwe can tell that the bitmap code actually kicked in? Maybe a perf test\nwould make it clear (those aren't always run, but hopefully we'd\neventually notice a regression there).\n\n> +static int filter_supported(struct list_objects_filter_options *filter)\n> +{\n> +\tint i;\n> +\n> +\tswitch (filter->choice) {\n> +\tcase LOFC_BLOB_NONE:\n> +\tcase LOFC_BLOB_LIMIT:\n> +\tcase LOFC_OBJECT_TYPE:\n> +\t\treturn 1;\n> +\tcase LOFC_TREE_DEPTH:\n> +\t\tif (filter->tree_exclude_depth == 0)\n> +\t\t\treturn 1;\n> +\t\treturn 0;\n> +\tcase LOFC_COMBINE:\n> +\t\tfor (i = 0; i < filter->sub_nr; i++)\n> +\t\t\tif (!filter_supported(&filter->sub[i]))\n> +\t\t\t\treturn 0;\n> +\t\treturn 1;\n> +\tdefault:\n> +\t\treturn 0;\n> +\t}\n> +}\n\nHmm. This is essentially reproducing the list in filter_bitmap() of\nwhat's OK for bitmaps. So when adding a new filter, it would have to be\nadded in both places.\n\nCan we preserve that property of the original code? I'd think that just\nadding LOFC_COMBINE to filter_bitmap() would be sufficient. I.e., this\nhunk:\n\n> +\tif (filter->choice == LOFC_COMBINE) {\n> +\t\tint i;\n> +\t\tfor (i = 0; i < filter->sub_nr; i++) {\n> +\t\t\tfilter_bitmap(bitmap_git, tip_objects, to_filter,\n> +\t\t\t\t &filter->sub[i]);\n> +\t\t}\n> +\t\treturn 0;\n> +\t}\n\n...except that we need to see if filter_bitmap() returns \"-1\" for any of\nthe recursive calls. Which we probably should be doing anyway to\npropagate any errors (though I think the only \"errors\" we'd return are\n\"not supported\", at least for now).\n\n-Peff\n"},{"id":"421078","messageId":"YGyinyL2UU421hoX@coredump.intra.peff.net","threadId":"55236","inReplyTo":"0e26fee8b31e46e87fb9fa1ac599506502a9d622.1615813673.git.ps@pks.im","subject":"Re: [PATCH v2 8/8] rev-list: allow filtering of provided items","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-06T18:04:15Z","receivedAt":"2021-04-06T18:04:19Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 15, 2021 at 02:15:05PM +0100, Patrick Steinhardt wrote:\n\n> When providing an object filter, it is currently impossible to also\n> filter provided items. E.g. when executing `git rev-list HEAD` , the\n> commit this reference points to will be treated as user-provided and is\n> thus excluded from the filtering mechanism. This makes it harder than\n> necessary to properly use the new `--filter=object:type` filter given\n> that even if the user wants to only see blobs, he'll still see commits\n> of provided references.\n> \n> Improve this by introducing a new `--filter-provided` option to the\n> git-rev-parse(1) command. If given, then all user-provided references\n> will be subject to filtering.\n\nI think this option is a good thing to have.\n\nThe name seems a little confusing to me, as I can read is as both\n\"please filter the provided objects\" and \"a filter has been provided\".\nI guess \"--filter-print-provided\" would be more clear. And also the\ndefault, so you'd want \"--no-filter-print-provided\". That's kind of\nclunky, though. Maybe \"--filter-omit-provided\"?\n\n> @@ -694,6 +698,16 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n> \t\t\treturn show_bisect_vars(&info, reaches, all);\n> \t}\n> \n> +\tif (filter_options.filter_wants) {\n> +\t\tstruct commit_list *c;\n> +\t\tfor (i = 0; i < revs.pending.nr; i++) {\n> +\t\t\tstruct object_array_entry *pending = revs.pending.objects + i;\n> +\t\t\tpending->item->flags |= NOT_USER_GIVEN;\n> +\t\t}\n> +\t\tfor (c = revs.commits; c; c = c->next)\n> +\t\t\tc->item->object.flags |= NOT_USER_GIVEN;\n> +\t}\n\nYou store the flag inside the filter_options struct, which implies to me\nthat it's something that could be applied per-filter (at least in\ntheory; the command line option doesn't allow us to distinguish).\n\nBut here you treat it as a global flag that munges the NOT_USER_GIVEN\nflags. Given that it's inside the filter_options struct, and that you\npropagate it via transform_to_combine_type(), I'd have expected the LOFC\ncode to look at the flag and decide to ignore the whole user-given\nconcept completely.\n\nTo be clear, I don't mind at all having it as a global that applies to\nall filters. I don't think the flexibility buys us anything. But since\nit only applies to rev-list, why not just make it a global option within\nrev-list?\n\nAnd then these hunks:\n\n> diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\n> index bb6f6577d5..2877aa9e96 100644\n> --- a/list-objects-filter-options.c\n> +++ b/list-objects-filter-options.c\n> @@ -242,6 +242,7 @@ static void transform_to_combine_type(\n> \t\tmemset(filter_options, 0, sizeof(*filter_options));\n> \t\tfilter_options->sub = sub_array;\n> \t\tfilter_options->sub_alloc = initial_sub_alloc;\n> +\t\tfilter_options->filter_wants = sub_array[0].filter_wants;\n> \t}\n> \tfilter_options->sub_nr = 1;\n> \tfilter_options->choice = LOFC_COMBINE;\n> @@ -290,6 +291,9 @@ void parse_list_objects_filter(\n> \t\tparse_error = gently_parse_list_objects_filter(\n> \t\t\t&filter_options->sub[filter_options->sub_nr - 1], arg,\n> \t\t\t&errbuf);\n> +\t\tif (!parse_error)\n> +\t\t\tfilter_options->sub[filter_options->sub_nr - 1].filter_wants =\n> +\t\t\t\tfilter_options->filter_wants;\n> \t}\n> \tif (parse_error)\n> \t\tdie(\"%s\", errbuf.buf);\n> diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\n> index 4d0d0588cc..5e609e307a 100644\n> --- a/list-objects-filter-options.h\n> +++ b/list-objects-filter-options.h\n> @@ -42,6 +42,12 @@ struct list_objects_filter_options {\n> \t */\n> \tenum list_objects_filter_choice choice;\n> \n> +\t/*\n> +\t * \"--filter-provided\" was given by the user, instructing us to also\n> +\t * filter all explicitly provided objects.\n> +\t */\n> +\tunsigned int filter_wants : 1;\n> +\n> \t/*\n> \t * Choice is LOFC_DISABLED because \"--no-filter\" was requested.\n> \t */\n\nwould not be needed at all.\n\n> diff --git a/pack-bitmap.c b/pack-bitmap.c\n> index e33805e076..5ff800316b 100644\n> --- a/pack-bitmap.c\n> +++ b/pack-bitmap.c\n> @@ -1101,7 +1101,8 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n> \tif (haves_bitmap)\n> \t\tbitmap_and_not(wants_bitmap, haves_bitmap);\n> \n> -\tfilter_bitmap(bitmap_git, wants, wants_bitmap, filter);\n> +\tfilter_bitmap(bitmap_git, (filter && filter->filter_wants) ? NULL : wants,\n> +\t\t wants_bitmap, filter);\n> \n> \tbitmap_git->result = wants_bitmap;\n> \tbitmap_git->haves = haves_bitmap;\n\nI guess we'd need to pass that flag into prepare_bitmap_walk() here so\nit knows not to bother with the wants-filtering. But that seems less bad\nthat stuffing it into the filter struct.\n\n-Peff\n"},{"id":"421079","messageId":"YGyjtCWqdeCj3S3U@coredump.intra.peff.net","threadId":"55236","inReplyTo":"xmqqblbdjzu6.fsf@gitster.g","subject":"Re: [PATCH v2 0/8] rev-parse: implement object type filter","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-06T18:08:52Z","receivedAt":"2021-04-06T18:09:24Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 20, 2021 at 02:10:41PM -0700, Junio C Hamano wrote:\n\n> Patrick Steinhardt writes:\n> \n> > this is the second version of my patch series which implements a new\n> > `object:type` filter for git-rev-parse(1) and git-upload-pack(1) and\n> > extends support for bitmap indices to work with combined filters.\n> > ...\n> > Please see the attached range-diff for more details.\n> \n> Any comment from stakeholders?\n\nSorry, this languished on my to-review list for a while.\n\nI took a careful look. I found a few small nits, but the code overall\nlooks pretty good.\n\nI do still find the use of the filter code here a _little_ bit\noff-putting. It makes perfect sense in some ways: we are asking rev-list\nto filter the output, and it keeps our implementation nice and simple.\nIt took me a while to figure out what I think makes it weird, but I\nthink it's:\n\n - the partial-clone feature exposes the filter mechanism in a very\n transparent way. So while it's not _wrong_ to be able to ask for a\n partial clone of only trees, it's an odd thing that nobody would\n really use in practice. And so it's a bit funny that it gets\n documented alongside blob:limit, etc.\n\n - for the same reason, it's very rigid. We have no way to say \"this\n filter OR that filter\", and are unlikely to grow them (because this\n is all part of the network protocol). Whereas it's perfectly\n reasonable for somebody to ask for \"trees and blobs\" via rev-list.\n\nI dunno. Those aren't objections exactly. Just trying to put my finger\non why my initial reaction was \"huh, why --filter?\".\n\n-Peff\n"},{"id":"421368","messageId":"YHAqOj+vboM+9f7d@ncase","threadId":"55236","inReplyTo":"YGya0XY969HlyPWl@coredump.intra.peff.net","subject":"Re: [PATCH v2 2/8] revision: mark commit parents as NOT_USER_GIVEN","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-09T10:19:38Z","receivedAt":"2021-04-09T10:21:04Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Apr 06, 2021 at 01:30:57PM -0400, Jeff King wrote:\n> On Mon, Mar 15, 2021 at 02:14:36PM +0100, Patrick Steinhardt wrote:\n> \n> > The NOT_USER_GIVEN flag of an object marks whether a flag was explicitly\n> > provided by the user or not. The most important use case for this is\n> > when filtering objects: only objects that were not explicitly requested\n> > will get filtered.\n> > \n> > The flag is currently only set for blobs and trees, which has been fine\n> > given that there are no filters for tags or commits currently. We're\n> > about to extend filtering capabilities to add object type filter though,\n> > which requires us to set up the NOT_USER_GIVEN flag correctly -- if it's\n> > not set, the object wouldn't get filtered at all.\n> > \n> > Mark unseen commit parents as NOT_USER_GIVEN when processing parents.\n> > Like this, explicitly provided parents stay user-given and thus\n> > unfiltered, while parents which get loaded as part of the graph walk\n> > can be filtered.\n> > \n> > This commit shouldn't have any user-visible impact yet as there is no\n> > logic to filter commits yet.\n> \n> I'm still scratching my head a bit to understand how NOT_USER_GIVEN can\n> possibly be correct (as opposed to USER_GIVEN). If we visit the commit\n> in a not-user-given context and add the flag, how do we know it wasn't\n> _also_ visited in a user-given context?\n> \n> Just guessing, but perhaps the SEEN flag is saving us here? If we visit\n> the user-given commit itself first, then we give it the SEEN flag. Then\n> if we try to visit it again via parent traversal, we've already\n> processed it and don't add the NOT_USER_GIVEN flag here.\n\nYes, I think that's mostly it.\n\n> That seems the opposite of the order we'd usually traverse, but I think\n> we set SEEN on each commit in prepare_revision_walk(), before we do any\n> traversing.\n> \n> So I _think_ it all works even with your changes here, but I have to say\n> this NOT_USER_GIVEN thing seems really fragile to me. Not new in your\n> series, of course, but something we may want to look at.\n> \n> Just grepping around, \"rev-list -g\" will happily remove SEEN flags, so I\n> suspect it interacts badly with --filter. Just trying \"rev-list -g\n> --objects --filter=object:type=blob HEAD\" shows that it produces quite a\n> lot of commits (which I think is a more fundamental problem: it is not\n> walking the parent chain at all to assign these NOT_USER_GIVEN flags).\n\nI totally agree that this feels fragile, and developing this series with\nNOT_USER_GIVEN wasn't the most enjoyable experience either. I wouldn't\nlove to be doing the conversion back to USER_GIVEN as part of this\nseries, but I wouldn't oppose doing that job either. Right now I don't\nfeel like I'm sufficiently sure that it's working for all cases, and\nindeed your example with \"rev-list -g\" already shows one case where it's\nbreaking.\n\nSo let me know whether I should add the conversion as preparatory step.\n\nPatrick\n"},{"id":"421369","messageId":"YHAtDp11xFzdUSQl@ncase","threadId":"55236","inReplyTo":"YGygV1mesbJzdPr5@coredump.intra.peff.net","subject":"Re: [PATCH v2 7/8] pack-bitmap: implement combined filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-09T10:31:42Z","receivedAt":"2021-04-09T10:31:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Apr 06, 2021 at 01:54:31PM -0400, Jeff King wrote:\n> On Mon, Mar 15, 2021 at 02:14:59PM +0100, Patrick Steinhardt wrote:\n> \n> > When the user has multiple objects filters specified, then this is\n> > internally represented by having a \"combined\" filter. These combined\n> > filters aren't yet supported by bitmap indices and can thus not be\n> > accelerated.\n> > \n> > Fix this by implementing support for these combined filters. The\n> > implementation is quite trivial: when there's a combined filter, we\n> > simply recurse into `filter_bitmap()` for all of the sub-filters.\n> \n> The goal makes sense.\n> \n> Before this patch, I think your test:\n> \n> > +test_expect_success 'combine filter' '\n> > +\tgit rev-list --objects --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&\n> > +\tgit rev-list --use-bitmap-index \\\n> > +\t\t --objects --filter=blob:limit=1000 --filter=object:type=blob tag >actual &&\n> > +\ttest_bitmap_traversal expect actual\n> > +'\n> \n> would pass anyway, because we'd just skip using bitmaps. Is there a way\n> we can tell that the bitmap code actually kicked in? Maybe a perf test\n> would make it clear (those aren't always run, but hopefully we'd\n> eventually notice a regression there).\n> \n> > +static int filter_supported(struct list_objects_filter_options *filter)\n> > +{\n> > +\tint i;\n> > +\n> > +\tswitch (filter->choice) {\n> > +\tcase LOFC_BLOB_NONE:\n> > +\tcase LOFC_BLOB_LIMIT:\n> > +\tcase LOFC_OBJECT_TYPE:\n> > +\t\treturn 1;\n> > +\tcase LOFC_TREE_DEPTH:\n> > +\t\tif (filter->tree_exclude_depth == 0)\n> > +\t\t\treturn 1;\n> > +\t\treturn 0;\n> > +\tcase LOFC_COMBINE:\n> > +\t\tfor (i = 0; i < filter->sub_nr; i++)\n> > +\t\t\tif (!filter_supported(&filter->sub[i]))\n> > +\t\t\t\treturn 0;\n> > +\t\treturn 1;\n> > +\tdefault:\n> > +\t\treturn 0;\n> > +\t}\n> > +}\n> \n> Hmm. This is essentially reproducing the list in filter_bitmap() of\n> what's OK for bitmaps. So when adding a new filter, it would have to be\n> added in both places.\n> \n> Can we preserve that property of the original code? I'd think that just\n> adding LOFC_COMBINE to filter_bitmap() would be sufficient. I.e., this\n> hunk:\n> \n> > +\tif (filter->choice == LOFC_COMBINE) {\n> > +\t\tint i;\n> > +\t\tfor (i = 0; i < filter->sub_nr; i++) {\n> > +\t\t\tfilter_bitmap(bitmap_git, tip_objects, to_filter,\n> > +\t\t\t\t &filter->sub[i]);\n> > +\t\t}\n> > +\t\treturn 0;\n> > +\t}\n> \n> ...except that we need to see if filter_bitmap() returns \"-1\" for any of\n> the recursive calls. Which we probably should be doing anyway to\n> propagate any errors (though I think the only \"errors\" we'd return are\n> \"not supported\", at least for now).\n> \n> -Peff\n\nBut wouldn't that mean that we're now needlessly filtering via bitmaps\nall the way down the combined filters only to realize at the end that it\ncannot work because we've got a tree filter with non-zero tree depth?\nGranted, this will not be the common case. But it still feels like we're\ndoing needless work for cases where we know that bitmaps cannot answer\nthe query.\n\nPatrick\n"},{"id":"421370","messageId":"YHAznTddwG/ej4aS@ncase","threadId":"55236","inReplyTo":"YGyinyL2UU421hoX@coredump.intra.peff.net","subject":"Re: [PATCH v2 8/8] rev-list: allow filtering of provided items","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-09T10:59:41Z","receivedAt":"2021-04-09T10:59:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Apr 06, 2021 at 02:04:15PM -0400, Jeff King wrote:\n> On Mon, Mar 15, 2021 at 02:15:05PM +0100, Patrick Steinhardt wrote:\n> \n> > When providing an object filter, it is currently impossible to also\n> > filter provided items. E.g. when executing `git rev-list HEAD` , the\n> > commit this reference points to will be treated as user-provided and is\n> > thus excluded from the filtering mechanism. This makes it harder than\n> > necessary to properly use the new `--filter=object:type` filter given\n> > that even if the user wants to only see blobs, he'll still see commits\n> > of provided references.\n> > \n> > Improve this by introducing a new `--filter-provided` option to the\n> > git-rev-parse(1) command. If given, then all user-provided references\n> > will be subject to filtering.\n> \n> I think this option is a good thing to have.\n> \n> The name seems a little confusing to me, as I can read is as both\n> \"please filter the provided objects\" and \"a filter has been provided\".\n> I guess \"--filter-print-provided\" would be more clear. And also the\n> default, so you'd want \"--no-filter-print-provided\". That's kind of\n> clunky, though. Maybe \"--filter-omit-provided\"?\n\nHum, \"--filter-omit-provided\" doesn't sound good to me, either. Omit to\nme sounds like it'd omit filtering provided items, but we're doing\nthe reverse thing.\n\nHow about \"--filter-provided-revisions\"? Verbose, but at least it cannot\nbe confused with a filter being provided.\n\n> > @@ -694,6 +698,16 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n> > \t\t\treturn show_bisect_vars(&info, reaches, all);\n> > \t}\n> > \n> > +\tif (filter_options.filter_wants) {\n> > +\t\tstruct commit_list *c;\n> > +\t\tfor (i = 0; i < revs.pending.nr; i++) {\n> > +\t\t\tstruct object_array_entry *pending = revs.pending.objects + i;\n> > +\t\t\tpending->item->flags |= NOT_USER_GIVEN;\n> > +\t\t}\n> > +\t\tfor (c = revs.commits; c; c = c->next)\n> > +\t\t\tc->item->object.flags |= NOT_USER_GIVEN;\n> > +\t}\n> \n> You store the flag inside the filter_options struct, which implies to me\n> that it's something that could be applied per-filter (at least in\n> theory; the command line option doesn't allow us to distinguish).\n> \n> But here you treat it as a global flag that munges the NOT_USER_GIVEN\n> flags. Given that it's inside the filter_options struct, and that you\n> propagate it via transform_to_combine_type(), I'd have expected the LOFC\n> code to look at the flag and decide to ignore the whole user-given\n> concept completely.\n> \n> To be clear, I don't mind at all having it as a global that applies to\n> all filters. I don't think the flexibility buys us anything. But since\n> it only applies to rev-list, why not just make it a global option within\n> rev-list?\n[snip]\n\nFair point. This probably stems from the confusion where I initially\ndidn't realize that the filter_options is not a \"global\" options\nstructure, but in fact the filter itself already. That's also why there\nhad been the initial bug where converting filter options into a combined\nfilter led to `filter_wants` being dropped.\n\nIn any case, the resulting code with it being global to rev-list.c\ninstead of part of the options is a lot cleaner.\n\nPatrick\n"},{"id":"421372","messageId":"YHA3EkH3HDe45EKA@ncase","threadId":"55236","inReplyTo":"YGyjtCWqdeCj3S3U@coredump.intra.peff.net","subject":"Re: [PATCH v2 0/8] rev-parse: implement object type filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-09T11:14:26Z","receivedAt":"2021-04-09T11:14:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Apr 06, 2021 at 02:08:52PM -0400, Jeff King wrote:\n> On Sat, Mar 20, 2021 at 02:10:41PM -0700, Junio C Hamano wrote:\n> \n> > Patrick Steinhardt writes:\n> > \n> > > this is the second version of my patch series which implements a new\n> > > `object:type` filter for git-rev-parse(1) and git-upload-pack(1) and\n> > > extends support for bitmap indices to work with combined filters.\n> > > ...\n> > > Please see the attached range-diff for more details.\n> > \n> > Any comment from stakeholders?\n> \n> Sorry, this languished on my to-review list for a while.\n> \n> I took a careful look. I found a few small nits, but the code overall\n> looks pretty good.\n> \n> I do still find the use of the filter code here a _little_ bit\n> off-putting. It makes perfect sense in some ways: we are asking rev-list\n> to filter the output, and it keeps our implementation nice and simple.\n> It took me a while to figure out what I think makes it weird, but I\n> think it's:\n> \n> - the partial-clone feature exposes the filter mechanism in a very\n> transparent way. So while it's not _wrong_ to be able to ask for a\n> partial clone of only trees, it's an odd thing that nobody would\n> really use in practice. And so it's a bit funny that it gets\n> documented alongside blob:limit, etc.\n> \n> - for the same reason, it's very rigid. We have no way to say \"this\n> filter OR that filter\", and are unlikely to grow them (because this\n> is all part of the network protocol). Whereas it's perfectly\n> reasonable for somebody to ask for \"trees and blobs\" via rev-list.\n> \n> I dunno. Those aren't objections exactly. Just trying to put my finger\n> on why my initial reaction was \"huh, why --filter?\".\n\nYeah, I do kind of share these concerns. Ideally, we'd provide a nicer\nonly-user-facing interface to query the repository for various objects.\ngit-cat-file(1) would be the obvious thing that first gets into my mind,\nwhere it would be nice to have it filter stuff. But then on the other\nhand, it's really rather a simple \"Give me what I tell you to\" binary,\nwhich is probably a good thing. Other than that I don't think there's\nany executable that'd be a good fit -- we could do this via a new\ngit-list-objects(1), but then again git-rev-list(1) already does most of\nwhat git-list-objects(1) would do, so why bother.\n\nIt kind of feels like git-checkout(1) to me: it does many things, and if\nyou know how to wield it it works perfectly fine. But the user interface\nis lacking, which is why it was split up into git-switch(1) and\ngit-restore(1). It's telling already that the summary of git-rev-list(1)\nis \"Lists commit objects in reverse chronological order\". I mean yes,\nthat's what it does in many cases. But there's just as many cases where\nit doesn't.\n\nPatrick\n"},{"id":"421373","messageId":"YHA359zqnw7MV8Bp@ncase","threadId":"55236","inReplyTo":"YGygV1mesbJzdPr5@coredump.intra.peff.net","subject":"Re: [PATCH v2 7/8] pack-bitmap: implement combined filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-09T11:17:59Z","receivedAt":"2021-04-09T11:18:15Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Apr 06, 2021 at 01:54:31PM -0400, Jeff King wrote:\n> On Mon, Mar 15, 2021 at 02:14:59PM +0100, Patrick Steinhardt wrote:\n> \n> > When the user has multiple objects filters specified, then this is\n> > internally represented by having a \"combined\" filter. These combined\n> > filters aren't yet supported by bitmap indices and can thus not be\n> > accelerated.\n> > \n> > Fix this by implementing support for these combined filters. The\n> > implementation is quite trivial: when there's a combined filter, we\n> > simply recurse into `filter_bitmap()` for all of the sub-filters.\n> \n> The goal makes sense.\n> \n> Before this patch, I think your test:\n> \n> > +test_expect_success 'combine filter' '\n> > +\tgit rev-list --objects --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&\n> > +\tgit rev-list --use-bitmap-index \\\n> > +\t\t --objects --filter=blob:limit=1000 --filter=object:type=blob tag >actual &&\n> > +\ttest_bitmap_traversal expect actual\n> > +'\n> \n> would pass anyway, because we'd just skip using bitmaps. Is there a way\n> we can tell that the bitmap code actually kicked in? Maybe a perf test\n> would make it clear (those aren't always run, but hopefully we'd\n> eventually notice a regression there).\n\nI think that's not actually true. Note that we're using\n`test_bitmap_traversal`:\n\n test_bitmap_traversal () {\n if test \"$1\" = \"--no-confirm-bitmaps\"\n then\n shift\n elif cmp \"$1\" \"$2\"\n then\n echo >&2 \"identical raw outputs; are you sure bitmaps were used?\"\n return 1\n fi &&\n cut -d' ' -f1 \"$1\" | sort >\"$1.normalized\" &&\n sort \"$2\" >\"$2.normalized\" &&\n test_cmp \"$1.normalized\" \"$2.normalized\" &&\n rm -f \"$1.normalized\" \"$2.normalized\"\n }\n\nThe output is different when using bitmap indices, which is why the\nfunction knows to fail in case output is the same in both cases. So we\nknow that it cannot be the same here and thus we also know that the\nbitmap case kicked in.\n\nPatrick\n"},{"id":"421374","messageId":"cover.1617967252.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1615813673.git.ps@pks.im","subject":"[PATCH v3 0/8] rev-parse: implement object type filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-09T11:27:47Z","receivedAt":"2021-04-09T11:27:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the third version of my patch series which implements a new\n`object:type` filter for git-rev-parse(1) and git-upload-pack(1) and\nextends support for bitmap indices to work with combined filters.\n\nThis mostly addresses Peff's comments. Thanks for your feedback!\n\n - Removed the `base` parameter from `process_tag()`.\n\n - The object type filter doesn't assume ordering for the object type\n enum anymore.\n\n - Combined filters in the bitmap path now verify that\n `filter_bitmap` does not return any errors.\n\n - Renamed \"--filter-provided\" to \"--filter-provided-revisions\" and\n added documentation for it.\n\n - Refactored the code to not munge the `filter_provided` field in\n the filter options struct, but instead carry it in rev-list.c.\n\nPlease see the attached range-diff for more details.\n\nPatrick\n\nPatrick Steinhardt (8):\n uploadpack.txt: document implication of `uploadpackfilter.allow`\n revision: mark commit parents as NOT_USER_GIVEN\n list-objects: move tag processing into its own function\n list-objects: support filtering by tag and commit\n list-objects: implement object type filter\n pack-bitmap: implement object type filter\n pack-bitmap: implement combined filter\n rev-list: allow filtering of provided items\n\n Documentation/config/uploadpack.txt | 9 ++-\n Documentation/rev-list-options.txt | 8 ++\n builtin/pack-objects.c | 2 +-\n builtin/rev-list.c | 36 ++++++---\n list-objects-filter-options.c | 14 ++++\n list-objects-filter-options.h | 2 +\n list-objects-filter.c | 116 ++++++++++++++++++++++++++++\n list-objects-filter.h | 2 +\n list-objects.c | 29 ++++++-\n pack-bitmap.c | 76 +++++++++++++++---\n pack-bitmap.h | 3 +-\n reachable.c | 2 +-\n revision.c | 4 +-\n revision.h | 3 -\n t/t6112-rev-list-filters-objects.sh | 76 ++++++++++++++++++\n t/t6113-rev-list-bitmap-filters.sh | 68 +++++++++++++++-\n 16 files changed, 416 insertions(+), 34 deletions(-)\n\nRange-diff against v2:\n1: 270ff80dac = 1: f80b9570d4 uploadpack.txt: document implication of `uploadpackfilter.allow`\n2: ddbec75986 = 2: 46c1952405 revision: mark commit parents as NOT_USER_GIVEN\n3: d8da0b24f4 ! 3: 3d792f6339 list-objects: move tag processing into its own function\n @@ list-objects.c: static void process_tree(struct traversal_context *ctx,\n \n +static void process_tag(struct traversal_context *ctx,\n +\t\t\tstruct tag *tag,\n -+\t\t\tstruct strbuf *base,\n +\t\t\tconst char *name)\n +{\n +\ttag->object.flags |= SEEN;\n @@ list-objects.c: static void traverse_trees_and_blobs(struct traversal_context *c\n \t\tif (obj->type == OBJ_TAG) {\n -\t\t\tobj->flags |= SEEN;\n -\t\t\tctx->show_object(obj, name, ctx->show_data);\n -+\t\t\tprocess_tag(ctx, (struct tag *)obj, base, name);\n ++\t\t\tprocess_tag(ctx, (struct tag *)obj, name);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\n4: 5545c189c5 ! 4: 80193d6ba3 list-objects: support filtering by tag and commit\n @@ list-objects-filter.h: enum list_objects_filter_result {\n \n ## list-objects.c ##\n @@ list-objects.c: static void process_tag(struct traversal_context *ctx,\n - \t\t\tstruct strbuf *base,\n + \t\t\tstruct tag *tag,\n \t\t\tconst char *name)\n {\n -\ttag->object.flags |= SEEN;\n @@ list-objects.c: static void process_tag(struct traversal_context *ctx,\n +\tenum list_objects_filter_result r;\n +\n +\tr = list_objects_filter__filter_object(ctx->revs->repo, LOFS_TAG,\n -+\t\t\t\t\t &tag->object, base->buf,\n -+\t\t\t\t\t &base->buf[base->len],\n -+\t\t\t\t\t ctx->filter);\n ++\t\t\t\t\t &tag->object, \"\", 0, ctx->filter);\n +\tif (r & LOFR_MARK_SEEN)\n +\t\ttag->object.flags |= SEEN;\n +\tif (r & LOFR_DO_SHOW)\n5: acf01472af = 5: e2a14abf92 list-objects: implement object type filter\n6: 8073ab665b ! 6: 46d4450d38 pack-bitmap: implement object type filter\n @@ pack-bitmap.c: static void filter_bitmap_tree_depth(struct bitmap_index *bitmap_\n +\t\t\t\t struct bitmap *to_filter,\n +\t\t\t\t enum object_type object_type)\n +{\n -+\tenum object_type t;\n -+\n +\tif (object_type < OBJ_COMMIT || object_type > OBJ_TAG)\n +\t\tBUG(\"filter_bitmap_object_type given invalid object\");\n +\n -+\tfor (t = OBJ_COMMIT; t <= OBJ_TAG; t++) {\n -+\t\tif (t == object_type)\n -+\t\t\tcontinue;\n -+\t\tfilter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, t);\n -+\t}\n ++\tif (object_type != OBJ_TAG)\n ++\t\tfilter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, OBJ_TAG);\n ++\tif (object_type != OBJ_COMMIT)\n ++\t\tfilter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, OBJ_COMMIT);\n ++\tif (object_type != OBJ_TREE)\n ++\t\tfilter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, OBJ_TREE);\n ++\tif (object_type != OBJ_BLOB)\n ++\t\tfilter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, OBJ_BLOB);\n +}\n +\n static int filter_bitmap(struct bitmap_index *bitmap_git,\n7: fac3477d97 ! 7: 06a376399b pack-bitmap: implement combined filter\n @@ Commit message\n \n ## pack-bitmap.c ##\n @@ pack-bitmap.c: static void filter_bitmap_object_type(struct bitmap_index *bitmap_git,\n - \t}\n + \t\tfilter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, OBJ_BLOB);\n }\n \n +static int filter_supported(struct list_objects_filter_options *filter)\n @@ pack-bitmap.c: static int filter_bitmap(struct bitmap_index *bitmap_git,\n +\tif (filter->choice == LOFC_COMBINE) {\n +\t\tint i;\n +\t\tfor (i = 0; i < filter->sub_nr; i++) {\n -+\t\t\tfilter_bitmap(bitmap_git, tip_objects, to_filter,\n -+\t\t\t\t &filter->sub[i]);\n ++\t\t\tif (filter_bitmap(bitmap_git, tip_objects, to_filter,\n ++\t\t\t\t\t &filter->sub[i]) < 0)\n ++\t\t\t\treturn -1;\n +\t\t}\n +\t\treturn 0;\n +\t}\n8: 0e26fee8b3 ! 8: 796606f32b rev-list: allow filtering of provided items\n @@ Commit message\n \n Signed-off-by: Patrick Steinhardt \n \n + ## Documentation/rev-list-options.txt ##\n +@@ Documentation/rev-list-options.txt: equivalent.\n + --no-filter::\n + \tTurn off any previous `--filter=` argument.\n + \n ++--filter-provided-revisions::\n ++\tFilter the list of explicitly provided revisions, which would otherwise\n ++\talways be printed even if they did not match any of the filters. Only\n ++\tuseful with `--filter=`.\n ++\n + --filter-print-omitted::\n + \tOnly useful with `--filter=`; prints a list of the objects omitted\n + \tby the filter. Object IDs are prefixed with a ``~'' character.\n +\n + ## builtin/pack-objects.c ##\n +@@ builtin/pack-objects.c: static int pack_options_allow_reuse(void)\n + \n + static int get_object_list_from_bitmap(struct rev_info *revs)\n + {\n +-\tif (!(bitmap_git = prepare_bitmap_walk(revs, &filter_options)))\n ++\tif (!(bitmap_git = prepare_bitmap_walk(revs, &filter_options, 0)))\n + \t\treturn -1;\n + \n + \tif (pack_options_allow_reuse() &&\n +\n ## builtin/rev-list.c ##\n +@@ builtin/rev-list.c: static inline int parse_missing_action_value(const char *value)\n + }\n + \n + static int try_bitmap_count(struct rev_info *revs,\n +-\t\t\t struct list_objects_filter_options *filter)\n ++\t\t\t struct list_objects_filter_options *filter,\n ++\t\t\t int filter_provided_revs)\n + {\n + \tuint32_t commit_count = 0,\n + \t\t tag_count = 0,\n +@@ builtin/rev-list.c: static int try_bitmap_count(struct rev_info *revs,\n + \t */\n + \tmax_count = revs->max_count;\n + \n +-\tbitmap_git = prepare_bitmap_walk(revs, filter);\n ++\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_revs);\n + \tif (!bitmap_git)\n + \t\treturn -1;\n + \n +@@ builtin/rev-list.c: static int try_bitmap_count(struct rev_info *revs,\n + }\n + \n + static int try_bitmap_traversal(struct rev_info *revs,\n +-\t\t\t\tstruct list_objects_filter_options *filter)\n ++\t\t\t\tstruct list_objects_filter_options *filter,\n ++\t\t\t\tint filter_provided_revs)\n + {\n + \tstruct bitmap_index *bitmap_git;\n + \n +@@ builtin/rev-list.c: static int try_bitmap_traversal(struct rev_info *revs,\n + \tif (revs->max_count >= 0)\n + \t\treturn -1;\n + \n +-\tbitmap_git = prepare_bitmap_walk(revs, filter);\n ++\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_revs);\n + \tif (!bitmap_git)\n + \t\treturn -1;\n + \n +@@ builtin/rev-list.c: static int try_bitmap_traversal(struct rev_info *revs,\n + }\n + \n + static int try_bitmap_disk_usage(struct rev_info *revs,\n +-\t\t\t\t struct list_objects_filter_options *filter)\n ++\t\t\t\t struct list_objects_filter_options *filter,\n ++\t\t\t\t int filter_provided_revs)\n + {\n + \tstruct bitmap_index *bitmap_git;\n + \n + \tif (!show_disk_usage)\n + \t\treturn -1;\n + \n +-\tbitmap_git = prepare_bitmap_walk(revs, filter);\n ++\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_revs);\n + \tif (!bitmap_git)\n + \t\treturn -1;\n + \n +@@ builtin/rev-list.c: int cmd_rev_list(int argc, const char **argv, const char *prefix)\n + \tint bisect_show_vars = 0;\n + \tint bisect_find_all = 0;\n + \tint use_bitmap_index = 0;\n ++\tint filter_provided_revs = 0;\n + \tconst char *show_progress = NULL;\n + \n + \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n @@ builtin/rev-list.c: int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\tlist_objects_filter_set_no_filter(&filter_options);\n \t\t\tcontinue;\n \t\t}\n -+\t\tif (!strcmp(arg, \"--filter-provided\")) {\n -+\t\t\tfilter_options.filter_wants = 1;\n ++\t\tif (!strcmp(arg, \"--filter-provided-revisions\")) {\n ++\t\t\tfilter_provided_revs = 1;\n +\t\t\tcontinue;\n +\t\t}\n \t\tif (!strcmp(arg, \"--filter-print-omitted\")) {\n \t\t\targ_print_omitted = 1;\n \t\t\tcontinue;\n +@@ builtin/rev-list.c: int cmd_rev_list(int argc, const char **argv, const char *prefix)\n + \t\tprogress = start_delayed_progress(show_progress, 0);\n + \n + \tif (use_bitmap_index) {\n +-\t\tif (!try_bitmap_count(&revs, &filter_options))\n ++\t\tif (!try_bitmap_count(&revs, &filter_options, filter_provided_revs))\n + \t\t\treturn 0;\n +-\t\tif (!try_bitmap_disk_usage(&revs, &filter_options))\n ++\t\tif (!try_bitmap_disk_usage(&revs, &filter_options, filter_provided_revs))\n + \t\t\treturn 0;\n +-\t\tif (!try_bitmap_traversal(&revs, &filter_options))\n ++\t\tif (!try_bitmap_traversal(&revs, &filter_options, filter_provided_revs))\n + \t\t\treturn 0;\n + \t}\n + \n @@ builtin/rev-list.c: int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\treturn show_bisect_vars(&info, reaches, all);\n \t}\n \n -+\tif (filter_options.filter_wants) {\n ++\tif (filter_provided_revs) {\n +\t\tstruct commit_list *c;\n +\t\tfor (i = 0; i < revs.pending.nr; i++) {\n +\t\t\tstruct object_array_entry *pending = revs.pending.objects + i;\n @@ builtin/rev-list.c: int cmd_rev_list(int argc, const char **argv, const char *pr\n \t\toidset_init(&omitted_objects, DEFAULT_OIDSET_SIZE);\n \tif (arg_missing_action == MA_PRINT)\n \n - ## list-objects-filter-options.c ##\n -@@ list-objects-filter-options.c: static void transform_to_combine_type(\n - \t\tmemset(filter_options, 0, sizeof(*filter_options));\n - \t\tfilter_options->sub = sub_array;\n - \t\tfilter_options->sub_alloc = initial_sub_alloc;\n -+\t\tfilter_options->filter_wants = sub_array[0].filter_wants;\n - \t}\n - \tfilter_options->sub_nr = 1;\n - \tfilter_options->choice = LOFC_COMBINE;\n -@@ list-objects-filter-options.c: void parse_list_objects_filter(\n - \t\tparse_error = gently_parse_list_objects_filter(\n - \t\t\t&filter_options->sub[filter_options->sub_nr - 1], arg,\n - \t\t\t&errbuf);\n -+\t\tif (!parse_error)\n -+\t\t\tfilter_options->sub[filter_options->sub_nr - 1].filter_wants =\n -+\t\t\t\tfilter_options->filter_wants;\n - \t}\n - \tif (parse_error)\n - \t\tdie(\"%s\", errbuf.buf);\n -\n - ## list-objects-filter-options.h ##\n -@@ list-objects-filter-options.h: struct list_objects_filter_options {\n - \t */\n - \tenum list_objects_filter_choice choice;\n - \n -+\t/*\n -+\t * \"--filter-provided\" was given by the user, instructing us to also\n -+\t * filter all explicitly provided objects.\n -+\t */\n -+\tunsigned int filter_wants : 1;\n -+\n - \t/*\n - \t * Choice is LOFC_DISABLED because \"--no-filter\" was requested.\n - \t */\n -\n ## pack-bitmap.c ##\n +@@ pack-bitmap.c: static int can_filter_bitmap(struct list_objects_filter_options *filter)\n + }\n + \n + struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n +-\t\t\t\t\t struct list_objects_filter_options *filter)\n ++\t\t\t\t\t struct list_objects_filter_options *filter,\n ++\t\t\t\t\t int filter_provided_revs)\n + {\n + \tunsigned int i;\n + \n @@ pack-bitmap.c: struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \tif (haves_bitmap)\n \t\tbitmap_and_not(wants_bitmap, haves_bitmap);\n \n -\tfilter_bitmap(bitmap_git, wants, wants_bitmap, filter);\n -+\tfilter_bitmap(bitmap_git, (filter && filter->filter_wants) ? NULL : wants,\n ++\tfilter_bitmap(bitmap_git, (filter && filter_provided_revs) ? NULL : wants,\n +\t\t wants_bitmap, filter);\n \n \tbitmap_git->result = wants_bitmap;\n \tbitmap_git->haves = haves_bitmap;\n \n + ## pack-bitmap.h ##\n +@@ pack-bitmap.h: void traverse_bitmap_commit_list(struct bitmap_index *,\n + \t\t\t\t show_reachable_fn show_reachable);\n + void test_bitmap_walk(struct rev_info *revs);\n + struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n +-\t\t\t\t\t struct list_objects_filter_options *filter);\n ++\t\t\t\t\t struct list_objects_filter_options *filter,\n ++\t\t\t\t\t int filter_provided_revs);\n + int reuse_partial_packfile_from_bitmap(struct bitmap_index *,\n + \t\t\t\t struct packed_git **packfile,\n + \t\t\t\t uint32_t *entries,\n +\n + ## reachable.c ##\n +@@ reachable.c: void mark_reachable_objects(struct rev_info *revs, int mark_reflog,\n + \tcp.progress = progress;\n + \tcp.count = 0;\n + \n +-\tbitmap_git = prepare_bitmap_walk(revs, NULL);\n ++\tbitmap_git = prepare_bitmap_walk(revs, NULL, 0);\n + \tif (bitmap_git) {\n + \t\ttraverse_bitmap_commit_list(bitmap_git, revs, mark_object_seen);\n + \t\tfree_bitmap_index(bitmap_git);\n +\n ## t/t6112-rev-list-filters-objects.sh ##\n @@ t/t6112-rev-list-filters-objects.sh: test_expect_success 'verify object:type=tag prints tag' '\n \ttest_cmp expected actual\n-- \n2.31.1\n\n"},{"id":"421375","messageId":"f80b9570d49665b936e706063e42b9b00c2f7f45.1617967252.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1617967252.git.ps@pks.im","subject":"[PATCH v3 1/8] uploadpack.txt: document implication of `uploadpackfilter.allow`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-09T11:27:52Z","receivedAt":"2021-04-09T11:27:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When `uploadpackfilter.allow` is set to `true`, it means that filters\nare enabled by default except in the case where a filter is explicitly\ndisabled via `uploadpackilter..allow`. This option will not only\nenable the currently supported set of filters, but also any filters\nwhich get added in the future. As such, an admin which wants to have\ntight control over which filters are allowed and which aren't probably\nshouldn't ever set `uploadpackfilter.allow=true`.\n\nAmend the documentation to make the ramifications more explicit so that\nadmins are aware of this.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/config/uploadpack.txt | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config/uploadpack.txt b/Documentation/config/uploadpack.txt\nindex b0d761282c..6729a072ea 100644\n--- a/Documentation/config/uploadpack.txt\n+++ b/Documentation/config/uploadpack.txt\n@@ -59,7 +59,8 @@ uploadpack.allowFilter::\n \n uploadpackfilter.allow::\n \tProvides a default value for unspecified object filters (see: the\n-\tbelow configuration variable).\n+\tbelow configuration variable). If set to `true`, this will also\n+\tenable all filters which get added in the future.\n \tDefaults to `true`.\n \n uploadpackfilter..allow::\n-- \n2.31.1\n\n"},{"id":"421376","messageId":"46c1952405173eca40706ed7be3b939b210ad2d5.1617967252.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1617967252.git.ps@pks.im","subject":"[PATCH v3 2/8] revision: mark commit parents as NOT_USER_GIVEN","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-09T11:27:57Z","receivedAt":"2021-04-09T11:28:03Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The NOT_USER_GIVEN flag of an object marks whether a flag was explicitly\nprovided by the user or not. The most important use case for this is\nwhen filtering objects: only objects that were not explicitly requested\nwill get filtered.\n\nThe flag is currently only set for blobs and trees, which has been fine\ngiven that there are no filters for tags or commits currently. We're\nabout to extend filtering capabilities to add object type filter though,\nwhich requires us to set up the NOT_USER_GIVEN flag correctly -- if it's\nnot set, the object wouldn't get filtered at all.\n\nMark unseen commit parents as NOT_USER_GIVEN when processing parents.\nLike this, explicitly provided parents stay user-given and thus\nunfiltered, while parents which get loaded as part of the graph walk\ncan be filtered.\n\nThis commit shouldn't have any user-visible impact yet as there is no\nlogic to filter commits yet.\n\nSigned-off-by: Patrick Steinhardt \n---\n revision.c | 4 ++--\n revision.h | 3 ---\n 2 files changed, 2 insertions(+), 5 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex 553c0faa9b..fd34c75e23 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1123,7 +1123,7 @@ static int process_parents(struct rev_info *revs, struct commit *commit,\n \t\t\t\tmark_parents_uninteresting(p);\n \t\t\tif (p->object.flags & SEEN)\n \t\t\t\tcontinue;\n-\t\t\tp->object.flags |= SEEN;\n+\t\t\tp->object.flags |= (SEEN | NOT_USER_GIVEN);\n \t\t\tif (list)\n \t\t\t\tcommit_list_insert_by_date(p, list);\n \t\t\tif (queue)\n@@ -1165,7 +1165,7 @@ static int process_parents(struct rev_info *revs, struct commit *commit,\n \t\t}\n \t\tp->object.flags |= left_flag;\n \t\tif (!(p->object.flags & SEEN)) {\n-\t\t\tp->object.flags |= SEEN;\n+\t\t\tp->object.flags |= (SEEN | NOT_USER_GIVEN);\n \t\t\tif (list)\n \t\t\t\tcommit_list_insert_by_date(p, list);\n \t\t\tif (queue)\ndiff --git a/revision.h b/revision.h\nindex a24f72dcd1..93aa012f51 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -44,9 +44,6 @@\n /*\n * Indicates object was reached by traversal. i.e. not given by user on\n * command-line or stdin.\n- * NEEDSWORK: NOT_USER_GIVEN doesn't apply to commits because we only support\n- * filtering trees and blobs, but it may be useful to support filtering commits\n- * in the future.\n */\n #define NOT_USER_GIVEN\t(1u<<25)\n #define TRACK_LINEAR\t(1u<<26)\n-- \n2.31.1\n\n"},{"id":"421377","messageId":"3d792f63390da79a6d225de51bdfc651234cf893.1617967252.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1617967252.git.ps@pks.im","subject":"[PATCH v3 3/8] list-objects: move tag processing into its own function","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-09T11:28:02Z","receivedAt":"2021-04-09T11:28:07Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Move processing of tags into its own function to make the logic easier\nto extend when we're going to implement filtering for tags. No change in\nbehaviour is expected from this commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n list-objects.c | 11 +++++++++--\n 1 file changed, 9 insertions(+), 2 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex e19589baa0..a5a60301cb 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -213,6 +213,14 @@ static void process_tree(struct traversal_context *ctx,\n \tfree_tree_buffer(tree);\n }\n \n+static void process_tag(struct traversal_context *ctx,\n+\t\t\tstruct tag *tag,\n+\t\t\tconst char *name)\n+{\n+\ttag->object.flags |= SEEN;\n+\tctx->show_object(&tag->object, name, ctx->show_data);\n+}\n+\n static void mark_edge_parents_uninteresting(struct commit *commit,\n \t\t\t\t\t struct rev_info *revs,\n \t\t\t\t\t show_edge_fn show_edge)\n@@ -334,8 +342,7 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx,\n \t\tif (obj->flags & (UNINTERESTING | SEEN))\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n-\t\t\tobj->flags |= SEEN;\n-\t\t\tctx->show_object(obj, name, ctx->show_data);\n+\t\t\tprocess_tag(ctx, (struct tag *)obj, name);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\n-- \n2.31.1\n\n"},{"id":"421378","messageId":"80193d6ba38254aab3dd2cda39cabf12986fd939.1617967252.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1617967252.git.ps@pks.im","subject":"[PATCH v3 4/8] list-objects: support filtering by tag and commit","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-09T11:28:07Z","receivedAt":"2021-04-09T11:28:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Object filters currently only support filtering blobs or trees based on\nsome criteria. This commit lays the foundation to also allow filtering\nof tags and commits.\n\nNo change in behaviour is expected from this commit given that there are\nno filters yet for those object types.\n\nSigned-off-by: Patrick Steinhardt \n---\n list-objects-filter.c | 40 ++++++++++++++++++++++++++++++++++++++++\n list-objects-filter.h | 2 ++\n list-objects.c | 22 +++++++++++++++++++---\n 3 files changed, 61 insertions(+), 3 deletions(-)\n\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex 39e2f15333..0ebfa52966 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -82,6 +82,16 @@ static enum list_objects_filter_result filter_blobs_none(\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n+\tcase LOFS_TAG:\n+\t\tassert(obj->type == OBJ_TAG);\n+\t\t/* always include all tag objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\tcase LOFS_COMMIT:\n+\t\tassert(obj->type == OBJ_COMMIT);\n+\t\t/* always include all commit objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\t/* always include all tree objects */\n@@ -173,6 +183,16 @@ static enum list_objects_filter_result filter_trees_depth(\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n+\tcase LOFS_TAG:\n+\t\tassert(obj->type == OBJ_TAG);\n+\t\t/* always include all tag objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\tcase LOFS_COMMIT:\n+\t\tassert(obj->type == OBJ_COMMIT);\n+\t\t/* always include all commit objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n \tcase LOFS_END_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\tfilter_data->current_depth--;\n@@ -267,6 +287,16 @@ static enum list_objects_filter_result filter_blobs_limit(\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n+\tcase LOFS_TAG:\n+\t\tassert(obj->type == OBJ_TAG);\n+\t\t/* always include all tag objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\tcase LOFS_COMMIT:\n+\t\tassert(obj->type == OBJ_COMMIT);\n+\t\t/* always include all commit objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\t/* always include all tree objects */\n@@ -371,6 +401,16 @@ static enum list_objects_filter_result filter_sparse(\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n+\tcase LOFS_TAG:\n+\t\tassert(obj->type == OBJ_TAG);\n+\t\t/* always include all tag objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\tcase LOFS_COMMIT:\n+\t\tassert(obj->type == OBJ_COMMIT);\n+\t\t/* always include all commit objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\tdtype = DT_DIR;\ndiff --git a/list-objects-filter.h b/list-objects-filter.h\nindex cfd784e203..9e98814111 100644\n--- a/list-objects-filter.h\n+++ b/list-objects-filter.h\n@@ -55,6 +55,8 @@ enum list_objects_filter_result {\n };\n \n enum list_objects_filter_situation {\n+\tLOFS_COMMIT,\n+\tLOFS_TAG,\n \tLOFS_BEGIN_TREE,\n \tLOFS_END_TREE,\n \tLOFS_BLOB\ndiff --git a/list-objects.c b/list-objects.c\nindex a5a60301cb..0c524a81ac 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -217,8 +217,14 @@ static void process_tag(struct traversal_context *ctx,\n \t\t\tstruct tag *tag,\n \t\t\tconst char *name)\n {\n-\ttag->object.flags |= SEEN;\n-\tctx->show_object(&tag->object, name, ctx->show_data);\n+\tenum list_objects_filter_result r;\n+\n+\tr = list_objects_filter__filter_object(ctx->revs->repo, LOFS_TAG,\n+\t\t\t\t\t &tag->object, \"\", 0, ctx->filter);\n+\tif (r & LOFR_MARK_SEEN)\n+\t\ttag->object.flags |= SEEN;\n+\tif (r & LOFR_DO_SHOW)\n+\t\tctx->show_object(&tag->object, name, ctx->show_data);\n }\n \n static void mark_edge_parents_uninteresting(struct commit *commit,\n@@ -368,6 +374,12 @@ static void do_traverse(struct traversal_context *ctx)\n \tstrbuf_init(&csp, PATH_MAX);\n \n \twhile ((commit = get_revision(ctx->revs)) != NULL) {\n+\t\tenum list_objects_filter_result r;\n+\n+\t\tr = list_objects_filter__filter_object(ctx->revs->repo,\n+\t\t\t\tLOFS_COMMIT, &commit->object,\n+\t\t\t\tNULL, NULL, ctx->filter);\n+\n \t\t/*\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n@@ -382,7 +394,11 @@ static void do_traverse(struct traversal_context *ctx)\n \t\t\tdie(_(\"unable to load root tree for commit %s\"),\n \t\t\t oid_to_hex(&commit->object.oid));\n \t\t}\n-\t\tctx->show_commit(commit, ctx->show_data);\n+\n+\t\tif (r & LOFR_MARK_SEEN)\n+\t\t\tcommit->object.flags |= SEEN;\n+\t\tif (r & LOFR_DO_SHOW)\n+\t\t\tctx->show_commit(commit, ctx->show_data);\n \n \t\tif (ctx->revs->tree_blobs_in_commit_order)\n \t\t\t/*\n-- \n2.31.1\n\n"},{"id":"421379","messageId":"e2a14abf924667c51e6a5680b6a5da65e561497d.1617967252.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1617967252.git.ps@pks.im","subject":"[PATCH v3 5/8] list-objects: implement object type filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-09T11:28:11Z","receivedAt":"2021-04-09T11:28:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"While it already is possible to filter objects by some criteria in\ngit-rev-list(1), it is not yet possible to filter out only a specific\ntype of objects. This makes some filters less useful. The `blob:limit`\nfilter for example filters blobs such that only those which are smaller\nthan the given limit are returned. But it is unfit to ask only for these\nsmallish blobs, given that git-rev-list(1) will continue to print tags,\ncommits and trees.\n\nNow that we have the infrastructure in place to also filter tags and\ncommits, we can improve this situation by implementing a new filter\nwhich selects objects based on their type. Above query can thus\ntrivially be implemented with the following command:\n\n $ git rev-list --objects --filter=object:type=blob \\\n --filter=blob:limit=200\n\nFurthermore, this filter allows to optimize for certain other cases: if\nfor example only tags or commits have been selected, there is no need to\nwalk down trees.\n\nThe new filter is not yet supported in bitmaps. This is going to be\nimplemented in a subsequent commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/config/uploadpack.txt | 6 +--\n Documentation/rev-list-options.txt | 3 ++\n list-objects-filter-options.c | 14 ++++++\n list-objects-filter-options.h | 2 +\n list-objects-filter.c | 76 +++++++++++++++++++++++++++++\n t/t6112-rev-list-filters-objects.sh | 48 ++++++++++++++++++\n 6 files changed, 146 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/config/uploadpack.txt b/Documentation/config/uploadpack.txt\nindex 6729a072ea..32fad5bbe8 100644\n--- a/Documentation/config/uploadpack.txt\n+++ b/Documentation/config/uploadpack.txt\n@@ -66,9 +66,9 @@ uploadpackfilter.allow::\n uploadpackfilter..allow::\n \tExplicitly allow or ban the object filter corresponding to\n \t``, where `` may be one of: `blob:none`,\n-\t`blob:limit`, `tree`, `sparse:oid`, or `combine`. If using\n-\tcombined filters, both `combine` and all of the nested filter\n-\tkinds must be allowed. Defaults to `uploadpackfilter.allow`.\n+\t`blob:limit`, `object:type`, `tree`, `sparse:oid`, or `combine`.\n+\tIf using combined filters, both `combine` and all of the nested\n+\tfilter kinds must be allowed. Defaults to `uploadpackfilter.allow`.\n \n uploadpackfilter.tree.maxDepth::\n \tOnly allow `--filter=tree:` when `` is no more than the value of\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex b1c8f86c6e..3afa8fffbd 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -892,6 +892,9 @@ or units. n may be zero. The suffixes k, m, and g can be used to name\n units in KiB, MiB, or GiB. For example, 'blob:limit=1k' is the same\n as 'blob:limit=1024'.\n +\n+The form '--filter=object:type=(tag|commit|tree|blob)' omits all objects\n+which are not of the requested type.\n++\n The form '--filter=sparse:oid=' uses a sparse-checkout\n specification contained in the blob (or blob-expression) ''\n to omit blobs that would not be not required for a sparse checkout on\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex d2d1c81caf..bb6f6577d5 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -29,6 +29,8 @@ const char *list_object_filter_config_name(enum list_objects_filter_choice c)\n \t\treturn \"tree\";\n \tcase LOFC_SPARSE_OID:\n \t\treturn \"sparse:oid\";\n+\tcase LOFC_OBJECT_TYPE:\n+\t\treturn \"object:type\";\n \tcase LOFC_COMBINE:\n \t\treturn \"combine\";\n \tcase LOFC__COUNT:\n@@ -97,6 +99,18 @@ static int gently_parse_list_objects_filter(\n \t\t}\n \t\treturn 1;\n \n+\t} else if (skip_prefix(arg, \"object:type=\", &v0)) {\n+\t\tint type = type_from_string_gently(v0, -1, 1);\n+\t\tif (type < 0) {\n+\t\t\tstrbuf_addstr(errbuf, _(\"expected 'object:type='\"));\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\tfilter_options->object_type = type;\n+\t\tfilter_options->choice = LOFC_OBJECT_TYPE;\n+\n+\t\treturn 0;\n+\n \t} else if (skip_prefix(arg, \"combine:\", &v0)) {\n \t\treturn parse_combine_filter(filter_options, v0, errbuf);\n \ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 01767c3c96..4d0d0588cc 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -13,6 +13,7 @@ enum list_objects_filter_choice {\n \tLOFC_BLOB_LIMIT,\n \tLOFC_TREE_DEPTH,\n \tLOFC_SPARSE_OID,\n+\tLOFC_OBJECT_TYPE,\n \tLOFC_COMBINE,\n \tLOFC__COUNT /* must be last */\n };\n@@ -54,6 +55,7 @@ struct list_objects_filter_options {\n \tchar *sparse_oid_name;\n \tunsigned long blob_limit_value;\n \tunsigned long tree_exclude_depth;\n+\tenum object_type object_type;\n \n \t/* LOFC_COMBINE values */\n \ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex 0ebfa52966..1c1ee3d1bb 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -545,6 +545,81 @@ static void filter_sparse_oid__init(\n \tfilter->free_fn = filter_sparse_free;\n }\n \n+/*\n+ * A filter for list-objects to omit large blobs.\n+ * And to OPTIONALLY collect a list of the omitted OIDs.\n+ */\n+struct filter_object_type_data {\n+\tenum object_type object_type;\n+};\n+\n+static enum list_objects_filter_result filter_object_type(\n+\tstruct repository *r,\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tstruct oidset *omits,\n+\tvoid *filter_data_)\n+{\n+\tstruct filter_object_type_data *filter_data = filter_data_;\n+\n+\tswitch (filter_situation) {\n+\tdefault:\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n+\n+\tcase LOFS_TAG:\n+\t\tassert(obj->type == OBJ_TAG);\n+\t\tif (filter_data->object_type == OBJ_TAG)\n+\t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\t\treturn LOFR_MARK_SEEN;\n+\n+\tcase LOFS_COMMIT:\n+\t\tassert(obj->type == OBJ_COMMIT);\n+\t\tif (filter_data->object_type == OBJ_COMMIT)\n+\t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\t\treturn LOFR_MARK_SEEN;\n+\n+\tcase LOFS_BEGIN_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\n+\t\t/*\n+\t\t * If we only want to show commits or tags, then there is no\n+\t\t * need to walk down trees.\n+\t\t */\n+\t\tif (filter_data->object_type == OBJ_COMMIT ||\n+\t\t filter_data->object_type == OBJ_TAG)\n+\t\t\treturn LOFR_SKIP_TREE;\n+\n+\t\tif (filter_data->object_type == OBJ_TREE)\n+\t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\t\treturn LOFR_MARK_SEEN;\n+\n+\tcase LOFS_BLOB:\n+\t\tassert(obj->type == OBJ_BLOB);\n+\n+\t\tif (filter_data->object_type == OBJ_BLOB)\n+\t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\t\treturn LOFR_MARK_SEEN;\n+\n+\tcase LOFS_END_TREE:\n+\t\treturn LOFR_ZERO;\n+\t}\n+}\n+\n+static void filter_object_type__init(\n+\tstruct list_objects_filter_options *filter_options,\n+\tstruct filter *filter)\n+{\n+\tstruct filter_object_type_data *d = xcalloc(1, sizeof(*d));\n+\td->object_type = filter_options->object_type;\n+\n+\tfilter->filter_data = d;\n+\tfilter->filter_object_fn = filter_object_type;\n+\tfilter->free_fn = free;\n+}\n+\n /* A filter which only shows objects shown by all sub-filters. */\n struct combine_filter_data {\n \tstruct subfilter *sub;\n@@ -691,6 +766,7 @@ static filter_init_fn s_filters[] = {\n \tfilter_blobs_limit__init,\n \tfilter_trees_depth__init,\n \tfilter_sparse_oid__init,\n+\tfilter_object_type__init,\n \tfilter_combine__init,\n };\n \ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex 31457d13b9..c79ec04060 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -159,6 +159,54 @@ test_expect_success 'verify blob:limit=1m' '\n \ttest_must_be_empty observed\n '\n \n+# Test object:type= filter.\n+\n+test_expect_success 'setup object-type' '\n+\tgit init object-type &&\n+\techo contents >object-type/blob &&\n+\tgit -C object-type add blob &&\n+\tgit -C object-type commit -m commit-message &&\n+\tgit -C object-type tag tag -m tag-message\n+'\n+\n+test_expect_success 'verify object:type= fails with invalid type' '\n+\ttest_must_fail git -C object-type rev-list --objects --filter=object:type= HEAD &&\n+\ttest_must_fail git -C object-type rev-list --objects --filter=object:type=invalid HEAD\n+'\n+\n+test_expect_success 'verify object:type=blob prints blob and commit' '\n+\t(\n+\t\tgit -C object-type rev-parse HEAD &&\n+\t\tprintf \"%s blob\\n\" $(git -C object-type rev-parse HEAD:blob)\n+\t) >expected &&\n+\tgit -C object-type rev-list --objects --filter=object:type=blob HEAD >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=tree prints tree and commit' '\n+\t(\n+\t\tgit -C object-type rev-parse HEAD &&\n+\t\tprintf \"%s \\n\" $(git -C object-type rev-parse HEAD^{tree})\n+\t) >expected &&\n+\tgit -C object-type rev-list --objects --filter=object:type=tree HEAD >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=commit prints commit' '\n+\tgit -C object-type rev-parse HEAD >expected &&\n+\tgit -C object-type rev-list --objects --filter=object:type=commit HEAD >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=tag prints tag' '\n+\t(\n+\t\tgit -C object-type rev-parse HEAD &&\n+\t\tprintf \"%s tag\\n\" $(git -C object-type rev-parse tag)\n+\t) >expected &&\n+\tgit -C object-type rev-list --objects --filter=object:type=tag tag >actual &&\n+\ttest_cmp expected actual\n+'\n+\n # Test sparse:path= filter.\n # !!!!\n # NOTE: sparse:path filter support has been dropped for security reasons,\n-- \n2.31.1\n\n"},{"id":"421380","messageId":"46d4450d38231d1af0b17c4855b9671ad3446aef.1617967252.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1617967252.git.ps@pks.im","subject":"[PATCH v3 6/8] pack-bitmap: implement object type filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-09T11:28:16Z","receivedAt":"2021-04-09T11:28:21Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The preceding commit has added a new object filter for git-rev-list(1)\nwhich allows to filter objects by type. Implement the equivalent filter\nfor packfile bitmaps so that we can answer these queries fast.\n\nSigned-off-by: Patrick Steinhardt \n---\n pack-bitmap.c | 29 ++++++++++++++++++++++++++---\n t/t6113-rev-list-bitmap-filters.sh | 25 ++++++++++++++++++++++++-\n 2 files changed, 50 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex b4513f8672..cd3f5c433e 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -779,9 +779,6 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n \teword_t mask;\n \tuint32_t i;\n \n-\tif (type != OBJ_BLOB && type != OBJ_TREE)\n-\t\tBUG(\"filter_bitmap_exclude_type: unsupported type '%d'\", type);\n-\n \t/*\n \t * The non-bitmap version of this filter never removes\n \t * objects which the other side specifically asked for,\n@@ -911,6 +908,24 @@ static void filter_bitmap_tree_depth(struct bitmap_index *bitmap_git,\n \t\t\t\t OBJ_BLOB);\n }\n \n+static void filter_bitmap_object_type(struct bitmap_index *bitmap_git,\n+\t\t\t\t struct object_list *tip_objects,\n+\t\t\t\t struct bitmap *to_filter,\n+\t\t\t\t enum object_type object_type)\n+{\n+\tif (object_type < OBJ_COMMIT || object_type > OBJ_TAG)\n+\t\tBUG(\"filter_bitmap_object_type given invalid object\");\n+\n+\tif (object_type != OBJ_TAG)\n+\t\tfilter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, OBJ_TAG);\n+\tif (object_type != OBJ_COMMIT)\n+\t\tfilter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, OBJ_COMMIT);\n+\tif (object_type != OBJ_TREE)\n+\t\tfilter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, OBJ_TREE);\n+\tif (object_type != OBJ_BLOB)\n+\t\tfilter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, OBJ_BLOB);\n+}\n+\n static int filter_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t struct object_list *tip_objects,\n \t\t\t struct bitmap *to_filter,\n@@ -943,6 +958,14 @@ static int filter_bitmap(struct bitmap_index *bitmap_git,\n \t\treturn 0;\n \t}\n \n+\tif (filter->choice == LOFC_OBJECT_TYPE) {\n+\t\tif (bitmap_git)\n+\t\t\tfilter_bitmap_object_type(bitmap_git, tip_objects,\n+\t\t\t\t\t\t to_filter,\n+\t\t\t\t\t\t filter->object_type);\n+\t\treturn 0;\n+\t}\n+\n \t/* filter choice not handled */\n \treturn -1;\n }\ndiff --git a/t/t6113-rev-list-bitmap-filters.sh b/t/t6113-rev-list-bitmap-filters.sh\nindex 3f889949ca..fb66735ac8 100755\n--- a/t/t6113-rev-list-bitmap-filters.sh\n+++ b/t/t6113-rev-list-bitmap-filters.sh\n@@ -10,7 +10,8 @@ test_expect_success 'set up bitmapped repo' '\n \ttest_commit much-larger-blob-one &&\n \tgit repack -adb &&\n \ttest_commit two &&\n-\ttest_commit much-larger-blob-two\n+\ttest_commit much-larger-blob-two &&\n+\tgit tag tag\n '\n \n test_expect_success 'filters fallback to non-bitmap traversal' '\n@@ -75,4 +76,26 @@ test_expect_success 'tree:1 filter' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'object:type filter' '\n+\tgit rev-list --objects --filter=object:type=tag tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=object:type=tag tag >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tgit rev-list --objects --filter=object:type=commit tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=object:type=commit tag >actual &&\n+\ttest_bitmap_traversal expect actual &&\n+\n+\tgit rev-list --objects --filter=object:type=tree tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=object:type=tree tag >actual &&\n+\ttest_bitmap_traversal expect actual &&\n+\n+\tgit rev-list --objects --filter=object:type=blob tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=object:type=blob tag >actual &&\n+\ttest_bitmap_traversal expect actual\n+'\n+\n test_done\n-- \n2.31.1\n\n"},{"id":"421381","messageId":"06a376399bcce0caed993fd2e1a9cff5a57ce502.1617967252.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1617967252.git.ps@pks.im","subject":"[PATCH v3 7/8] pack-bitmap: implement combined filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-09T11:28:21Z","receivedAt":"2021-04-09T11:28:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When the user has multiple objects filters specified, then this is\ninternally represented by having a \"combined\" filter. These combined\nfilters aren't yet supported by bitmap indices and can thus not be\naccelerated.\n\nFix this by implementing support for these combined filters. The\nimplementation is quite trivial: when there's a combined filter, we\nsimply recurse into `filter_bitmap()` for all of the sub-filters.\n\nSigned-off-by: Patrick Steinhardt \n---\n pack-bitmap.c | 41 +++++++++++++++++++++++++++---\n t/t6113-rev-list-bitmap-filters.sh | 7 +++++\n 2 files changed, 44 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex cd3f5c433e..4385f15828 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -926,6 +926,29 @@ static void filter_bitmap_object_type(struct bitmap_index *bitmap_git,\n \t\tfilter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, OBJ_BLOB);\n }\n \n+static int filter_supported(struct list_objects_filter_options *filter)\n+{\n+\tint i;\n+\n+\tswitch (filter->choice) {\n+\tcase LOFC_BLOB_NONE:\n+\tcase LOFC_BLOB_LIMIT:\n+\tcase LOFC_OBJECT_TYPE:\n+\t\treturn 1;\n+\tcase LOFC_TREE_DEPTH:\n+\t\tif (filter->tree_exclude_depth == 0)\n+\t\t\treturn 1;\n+\t\treturn 0;\n+\tcase LOFC_COMBINE:\n+\t\tfor (i = 0; i < filter->sub_nr; i++)\n+\t\t\tif (!filter_supported(&filter->sub[i]))\n+\t\t\t\treturn 0;\n+\t\treturn 1;\n+\tdefault:\n+\t\treturn 0;\n+\t}\n+}\n+\n static int filter_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t struct object_list *tip_objects,\n \t\t\t struct bitmap *to_filter,\n@@ -933,6 +956,8 @@ static int filter_bitmap(struct bitmap_index *bitmap_git,\n {\n \tif (!filter || filter->choice == LOFC_DISABLED)\n \t\treturn 0;\n+\tif (!filter_supported(filter))\n+\t\treturn -1;\n \n \tif (filter->choice == LOFC_BLOB_NONE) {\n \t\tif (bitmap_git)\n@@ -949,8 +974,7 @@ static int filter_bitmap(struct bitmap_index *bitmap_git,\n \t\treturn 0;\n \t}\n \n-\tif (filter->choice == LOFC_TREE_DEPTH &&\n-\t filter->tree_exclude_depth == 0) {\n+\tif (filter->choice == LOFC_TREE_DEPTH) {\n \t\tif (bitmap_git)\n \t\t\tfilter_bitmap_tree_depth(bitmap_git, tip_objects,\n \t\t\t\t\t\t to_filter,\n@@ -966,8 +990,17 @@ static int filter_bitmap(struct bitmap_index *bitmap_git,\n \t\treturn 0;\n \t}\n \n-\t/* filter choice not handled */\n-\treturn -1;\n+\tif (filter->choice == LOFC_COMBINE) {\n+\t\tint i;\n+\t\tfor (i = 0; i < filter->sub_nr; i++) {\n+\t\t\tif (filter_bitmap(bitmap_git, tip_objects, to_filter,\n+\t\t\t\t\t &filter->sub[i]) < 0)\n+\t\t\t\treturn -1;\n+\t\t}\n+\t\treturn 0;\n+\t}\n+\n+\tBUG(\"unsupported filter choice\");\n }\n \n static int can_filter_bitmap(struct list_objects_filter_options *filter)\ndiff --git a/t/t6113-rev-list-bitmap-filters.sh b/t/t6113-rev-list-bitmap-filters.sh\nindex fb66735ac8..cb9db7df6f 100755\n--- a/t/t6113-rev-list-bitmap-filters.sh\n+++ b/t/t6113-rev-list-bitmap-filters.sh\n@@ -98,4 +98,11 @@ test_expect_success 'object:type filter' '\n \ttest_bitmap_traversal expect actual\n '\n \n+test_expect_success 'combine filter' '\n+\tgit rev-list --objects --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=blob:limit=1000 --filter=object:type=blob tag >actual &&\n+\ttest_bitmap_traversal expect actual\n+'\n+\n test_done\n-- \n2.31.1\n\n"},{"id":"421382","messageId":"796606f32b3ffc286a3157312d00ee0ee3e5600c.1617967252.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1617967252.git.ps@pks.im","subject":"[PATCH v3 8/8] rev-list: allow filtering of provided items","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-09T11:28:25Z","receivedAt":"2021-04-09T11:28:32Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When providing an object filter, it is currently impossible to also\nfilter provided items. E.g. when executing `git rev-list HEAD` , the\ncommit this reference points to will be treated as user-provided and is\nthus excluded from the filtering mechanism. This makes it harder than\nnecessary to properly use the new `--filter=object:type` filter given\nthat even if the user wants to only see blobs, he'll still see commits\nof provided references.\n\nImprove this by introducing a new `--filter-provided` option to the\ngit-rev-parse(1) command. If given, then all user-provided references\nwill be subject to filtering.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/rev-list-options.txt | 5 ++++\n builtin/pack-objects.c | 2 +-\n builtin/rev-list.c | 36 +++++++++++++++++++++--------\n pack-bitmap.c | 6 +++--\n pack-bitmap.h | 3 ++-\n reachable.c | 2 +-\n t/t6112-rev-list-filters-objects.sh | 28 ++++++++++++++++++++++\n t/t6113-rev-list-bitmap-filters.sh | 36 +++++++++++++++++++++++++++++\n 8 files changed, 104 insertions(+), 14 deletions(-)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 3afa8fffbd..7fa18fc6e6 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -933,6 +933,11 @@ equivalent.\n --no-filter::\n \tTurn off any previous `--filter=` argument.\n \n+--filter-provided-revisions::\n+\tFilter the list of explicitly provided revisions, which would otherwise\n+\talways be printed even if they did not match any of the filters. Only\n+\tuseful with `--filter=`.\n+\n --filter-print-omitted::\n \tOnly useful with `--filter=`; prints a list of the objects omitted\n \tby the filter. Object IDs are prefixed with a ``~'' character.\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 525c2d8552..2f2026dc87 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3516,7 +3516,7 @@ static int pack_options_allow_reuse(void)\n \n static int get_object_list_from_bitmap(struct rev_info *revs)\n {\n-\tif (!(bitmap_git = prepare_bitmap_walk(revs, &filter_options)))\n+\tif (!(bitmap_git = prepare_bitmap_walk(revs, &filter_options, 0)))\n \t\treturn -1;\n \n \tif (pack_options_allow_reuse() &&\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex b4d8ea0a35..13f0ff3f8d 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -398,7 +398,8 @@ static inline int parse_missing_action_value(const char *value)\n }\n \n static int try_bitmap_count(struct rev_info *revs,\n-\t\t\t struct list_objects_filter_options *filter)\n+\t\t\t struct list_objects_filter_options *filter,\n+\t\t\t int filter_provided_revs)\n {\n \tuint32_t commit_count = 0,\n \t\t tag_count = 0,\n@@ -433,7 +434,7 @@ static int try_bitmap_count(struct rev_info *revs,\n \t */\n \tmax_count = revs->max_count;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, filter);\n+\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_revs);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -450,7 +451,8 @@ static int try_bitmap_count(struct rev_info *revs,\n }\n \n static int try_bitmap_traversal(struct rev_info *revs,\n-\t\t\t\tstruct list_objects_filter_options *filter)\n+\t\t\t\tstruct list_objects_filter_options *filter,\n+\t\t\t\tint filter_provided_revs)\n {\n \tstruct bitmap_index *bitmap_git;\n \n@@ -461,7 +463,7 @@ static int try_bitmap_traversal(struct rev_info *revs,\n \tif (revs->max_count >= 0)\n \t\treturn -1;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, filter);\n+\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_revs);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -471,14 +473,15 @@ static int try_bitmap_traversal(struct rev_info *revs,\n }\n \n static int try_bitmap_disk_usage(struct rev_info *revs,\n-\t\t\t\t struct list_objects_filter_options *filter)\n+\t\t\t\t struct list_objects_filter_options *filter,\n+\t\t\t\t int filter_provided_revs)\n {\n \tstruct bitmap_index *bitmap_git;\n \n \tif (!show_disk_usage)\n \t\treturn -1;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, filter);\n+\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_revs);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -499,6 +502,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \tint bisect_show_vars = 0;\n \tint bisect_find_all = 0;\n \tint use_bitmap_index = 0;\n+\tint filter_provided_revs = 0;\n \tconst char *show_progress = NULL;\n \n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n@@ -599,6 +603,10 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\tlist_objects_filter_set_no_filter(&filter_options);\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!strcmp(arg, \"--filter-provided-revisions\")) {\n+\t\t\tfilter_provided_revs = 1;\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!strcmp(arg, \"--filter-print-omitted\")) {\n \t\t\targ_print_omitted = 1;\n \t\t\tcontinue;\n@@ -665,11 +673,11 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\tprogress = start_delayed_progress(show_progress, 0);\n \n \tif (use_bitmap_index) {\n-\t\tif (!try_bitmap_count(&revs, &filter_options))\n+\t\tif (!try_bitmap_count(&revs, &filter_options, filter_provided_revs))\n \t\t\treturn 0;\n-\t\tif (!try_bitmap_disk_usage(&revs, &filter_options))\n+\t\tif (!try_bitmap_disk_usage(&revs, &filter_options, filter_provided_revs))\n \t\t\treturn 0;\n-\t\tif (!try_bitmap_traversal(&revs, &filter_options))\n+\t\tif (!try_bitmap_traversal(&revs, &filter_options, filter_provided_revs))\n \t\t\treturn 0;\n \t}\n \n@@ -694,6 +702,16 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\treturn show_bisect_vars(&info, reaches, all);\n \t}\n \n+\tif (filter_provided_revs) {\n+\t\tstruct commit_list *c;\n+\t\tfor (i = 0; i < revs.pending.nr; i++) {\n+\t\t\tstruct object_array_entry *pending = revs.pending.objects + i;\n+\t\t\tpending->item->flags |= NOT_USER_GIVEN;\n+\t\t}\n+\t\tfor (c = revs.commits; c; c = c->next)\n+\t\t\tc->item->object.flags |= NOT_USER_GIVEN;\n+\t}\n+\n \tif (arg_print_omitted)\n \t\toidset_init(&omitted_objects, DEFAULT_OIDSET_SIZE);\n \tif (arg_missing_action == MA_PRINT)\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 4385f15828..0576a19a28 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1009,7 +1009,8 @@ static int can_filter_bitmap(struct list_objects_filter_options *filter)\n }\n \n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n-\t\t\t\t\t struct list_objects_filter_options *filter)\n+\t\t\t\t\t struct list_objects_filter_options *filter,\n+\t\t\t\t\t int filter_provided_revs)\n {\n \tunsigned int i;\n \n@@ -1104,7 +1105,8 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \tif (haves_bitmap)\n \t\tbitmap_and_not(wants_bitmap, haves_bitmap);\n \n-\tfilter_bitmap(bitmap_git, wants, wants_bitmap, filter);\n+\tfilter_bitmap(bitmap_git, (filter && filter_provided_revs) ? NULL : wants,\n+\t\t wants_bitmap, filter);\n \n \tbitmap_git->result = wants_bitmap;\n \tbitmap_git->haves = haves_bitmap;\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 36d99930d8..5d8ae3b590 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -50,7 +50,8 @@ void traverse_bitmap_commit_list(struct bitmap_index *,\n \t\t\t\t show_reachable_fn show_reachable);\n void test_bitmap_walk(struct rev_info *revs);\n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n-\t\t\t\t\t struct list_objects_filter_options *filter);\n+\t\t\t\t\t struct list_objects_filter_options *filter,\n+\t\t\t\t\t int filter_provided_revs);\n int reuse_partial_packfile_from_bitmap(struct bitmap_index *,\n \t\t\t\t struct packed_git **packfile,\n \t\t\t\t uint32_t *entries,\ndiff --git a/reachable.c b/reachable.c\nindex 77a60c70a5..fc833cae43 100644\n--- a/reachable.c\n+++ b/reachable.c\n@@ -223,7 +223,7 @@ void mark_reachable_objects(struct rev_info *revs, int mark_reflog,\n \tcp.progress = progress;\n \tcp.count = 0;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, NULL);\n+\tbitmap_git = prepare_bitmap_walk(revs, NULL, 0);\n \tif (bitmap_git) {\n \t\ttraverse_bitmap_commit_list(bitmap_git, revs, mark_object_seen);\n \t\tfree_bitmap_index(bitmap_git);\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex c79ec04060..47c558ab0e 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -207,6 +207,34 @@ test_expect_success 'verify object:type=tag prints tag' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'verify object:type=blob prints only blob with --filter-provided' '\n+\tprintf \"%s blob\\n\" $(git -C object-type rev-parse HEAD:blob) >expected &&\n+\tgit -C object-type rev-list --objects \\\n+\t\t--filter=object:type=blob --filter-provided HEAD >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=tree prints only tree with --filter-provided' '\n+\tprintf \"%s \\n\" $(git -C object-type rev-parse HEAD^{tree}) >expected &&\n+\tgit -C object-type rev-list --objects \\\n+\t\t--filter=object:type=tree HEAD --filter-provided >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=commit prints only commit with --filter-provided' '\n+\tgit -C object-type rev-parse HEAD >expected &&\n+\tgit -C object-type rev-list --objects \\\n+\t\t--filter=object:type=commit --filter-provided HEAD >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=tag prints only tag with --filter-provided' '\n+\tprintf \"%s tag\\n\" $(git -C object-type rev-parse tag) >expected &&\n+\tgit -C object-type rev-list --objects \\\n+\t\t--filter=object:type=tag --filter-provided tag >actual &&\n+\ttest_cmp expected actual\n+'\n+\n # Test sparse:path= filter.\n # !!!!\n # NOTE: sparse:path filter support has been dropped for security reasons,\ndiff --git a/t/t6113-rev-list-bitmap-filters.sh b/t/t6113-rev-list-bitmap-filters.sh\nindex cb9db7df6f..9053ac5059 100755\n--- a/t/t6113-rev-list-bitmap-filters.sh\n+++ b/t/t6113-rev-list-bitmap-filters.sh\n@@ -98,6 +98,28 @@ test_expect_success 'object:type filter' '\n \ttest_bitmap_traversal expect actual\n '\n \n+test_expect_success 'object:type filter with --filter-provided' '\n+\tgit rev-list --objects --filter-provided --filter=object:type=tag tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter-provided --filter=object:type=tag tag >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tgit rev-list --objects --filter-provided --filter=object:type=commit tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter-provided --filter=object:type=commit tag >actual &&\n+\ttest_bitmap_traversal expect actual &&\n+\n+\tgit rev-list --objects --filter-provided --filter=object:type=tree tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter-provided --filter=object:type=tree tag >actual &&\n+\ttest_bitmap_traversal expect actual &&\n+\n+\tgit rev-list --objects --filter-provided --filter=object:type=blob tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter-provided --filter=object:type=blob tag >actual &&\n+\ttest_bitmap_traversal expect actual\n+'\n+\n test_expect_success 'combine filter' '\n \tgit rev-list --objects --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&\n \tgit rev-list --use-bitmap-index \\\n@@ -105,4 +127,18 @@ test_expect_success 'combine filter' '\n \ttest_bitmap_traversal expect actual\n '\n \n+test_expect_success 'combine filter with --filter-provided' '\n+\tgit rev-list --objects --filter-provided --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter-provided --filter=blob:limit=1000 --filter=object:type=blob tag >actual &&\n+\ttest_bitmap_traversal expect actual &&\n+\n+\tgit cat-file --batch-check=\"%(objecttype) %(objectsize)\" objects &&\n+\twhile read objecttype objectsize\n+\tdo\n+\t\ttest \"$objecttype\" = blob || return 1\n+\t\ttest \"$objectsize\" -le 1000 || return 1\n+\tdone \n---\n\nForgot to stage and add test changes to adjust for the changed flag\nname.\n\n Documentation/rev-list-options.txt | 5 ++++\n builtin/pack-objects.c | 2 +-\n builtin/rev-list.c | 36 +++++++++++++++++++++--------\n pack-bitmap.c | 6 +++--\n pack-bitmap.h | 3 ++-\n reachable.c | 2 +-\n t/t6112-rev-list-filters-objects.sh | 28 ++++++++++++++++++++++\n t/t6113-rev-list-bitmap-filters.sh | 36 +++++++++++++++++++++++++++++\n 8 files changed, 104 insertions(+), 14 deletions(-)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 3afa8fffbd..7fa18fc6e6 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -933,6 +933,11 @@ equivalent.\n --no-filter::\n \tTurn off any previous `--filter=` argument.\n \n+--filter-provided-revisions::\n+\tFilter the list of explicitly provided revisions, which would otherwise\n+\talways be printed even if they did not match any of the filters. Only\n+\tuseful with `--filter=`.\n+\n --filter-print-omitted::\n \tOnly useful with `--filter=`; prints a list of the objects omitted\n \tby the filter. Object IDs are prefixed with a ``~'' character.\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 525c2d8552..2f2026dc87 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3516,7 +3516,7 @@ static int pack_options_allow_reuse(void)\n \n static int get_object_list_from_bitmap(struct rev_info *revs)\n {\n-\tif (!(bitmap_git = prepare_bitmap_walk(revs, &filter_options)))\n+\tif (!(bitmap_git = prepare_bitmap_walk(revs, &filter_options, 0)))\n \t\treturn -1;\n \n \tif (pack_options_allow_reuse() &&\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex b4d8ea0a35..13f0ff3f8d 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -398,7 +398,8 @@ static inline int parse_missing_action_value(const char *value)\n }\n \n static int try_bitmap_count(struct rev_info *revs,\n-\t\t\t struct list_objects_filter_options *filter)\n+\t\t\t struct list_objects_filter_options *filter,\n+\t\t\t int filter_provided_revs)\n {\n \tuint32_t commit_count = 0,\n \t\t tag_count = 0,\n@@ -433,7 +434,7 @@ static int try_bitmap_count(struct rev_info *revs,\n \t */\n \tmax_count = revs->max_count;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, filter);\n+\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_revs);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -450,7 +451,8 @@ static int try_bitmap_count(struct rev_info *revs,\n }\n \n static int try_bitmap_traversal(struct rev_info *revs,\n-\t\t\t\tstruct list_objects_filter_options *filter)\n+\t\t\t\tstruct list_objects_filter_options *filter,\n+\t\t\t\tint filter_provided_revs)\n {\n \tstruct bitmap_index *bitmap_git;\n \n@@ -461,7 +463,7 @@ static int try_bitmap_traversal(struct rev_info *revs,\n \tif (revs->max_count >= 0)\n \t\treturn -1;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, filter);\n+\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_revs);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -471,14 +473,15 @@ static int try_bitmap_traversal(struct rev_info *revs,\n }\n \n static int try_bitmap_disk_usage(struct rev_info *revs,\n-\t\t\t\t struct list_objects_filter_options *filter)\n+\t\t\t\t struct list_objects_filter_options *filter,\n+\t\t\t\t int filter_provided_revs)\n {\n \tstruct bitmap_index *bitmap_git;\n \n \tif (!show_disk_usage)\n \t\treturn -1;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, filter);\n+\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_revs);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -499,6 +502,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \tint bisect_show_vars = 0;\n \tint bisect_find_all = 0;\n \tint use_bitmap_index = 0;\n+\tint filter_provided_revs = 0;\n \tconst char *show_progress = NULL;\n \n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n@@ -599,6 +603,10 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\tlist_objects_filter_set_no_filter(&filter_options);\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!strcmp(arg, \"--filter-provided-revisions\")) {\n+\t\t\tfilter_provided_revs = 1;\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!strcmp(arg, \"--filter-print-omitted\")) {\n \t\t\targ_print_omitted = 1;\n \t\t\tcontinue;\n@@ -665,11 +673,11 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\tprogress = start_delayed_progress(show_progress, 0);\n \n \tif (use_bitmap_index) {\n-\t\tif (!try_bitmap_count(&revs, &filter_options))\n+\t\tif (!try_bitmap_count(&revs, &filter_options, filter_provided_revs))\n \t\t\treturn 0;\n-\t\tif (!try_bitmap_disk_usage(&revs, &filter_options))\n+\t\tif (!try_bitmap_disk_usage(&revs, &filter_options, filter_provided_revs))\n \t\t\treturn 0;\n-\t\tif (!try_bitmap_traversal(&revs, &filter_options))\n+\t\tif (!try_bitmap_traversal(&revs, &filter_options, filter_provided_revs))\n \t\t\treturn 0;\n \t}\n \n@@ -694,6 +702,16 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\treturn show_bisect_vars(&info, reaches, all);\n \t}\n \n+\tif (filter_provided_revs) {\n+\t\tstruct commit_list *c;\n+\t\tfor (i = 0; i < revs.pending.nr; i++) {\n+\t\t\tstruct object_array_entry *pending = revs.pending.objects + i;\n+\t\t\tpending->item->flags |= NOT_USER_GIVEN;\n+\t\t}\n+\t\tfor (c = revs.commits; c; c = c->next)\n+\t\t\tc->item->object.flags |= NOT_USER_GIVEN;\n+\t}\n+\n \tif (arg_print_omitted)\n \t\toidset_init(&omitted_objects, DEFAULT_OIDSET_SIZE);\n \tif (arg_missing_action == MA_PRINT)\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 4385f15828..0576a19a28 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1009,7 +1009,8 @@ static int can_filter_bitmap(struct list_objects_filter_options *filter)\n }\n \n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n-\t\t\t\t\t struct list_objects_filter_options *filter)\n+\t\t\t\t\t struct list_objects_filter_options *filter,\n+\t\t\t\t\t int filter_provided_revs)\n {\n \tunsigned int i;\n \n@@ -1104,7 +1105,8 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \tif (haves_bitmap)\n \t\tbitmap_and_not(wants_bitmap, haves_bitmap);\n \n-\tfilter_bitmap(bitmap_git, wants, wants_bitmap, filter);\n+\tfilter_bitmap(bitmap_git, (filter && filter_provided_revs) ? NULL : wants,\n+\t\t wants_bitmap, filter);\n \n \tbitmap_git->result = wants_bitmap;\n \tbitmap_git->haves = haves_bitmap;\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 36d99930d8..5d8ae3b590 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -50,7 +50,8 @@ void traverse_bitmap_commit_list(struct bitmap_index *,\n \t\t\t\t show_reachable_fn show_reachable);\n void test_bitmap_walk(struct rev_info *revs);\n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n-\t\t\t\t\t struct list_objects_filter_options *filter);\n+\t\t\t\t\t struct list_objects_filter_options *filter,\n+\t\t\t\t\t int filter_provided_revs);\n int reuse_partial_packfile_from_bitmap(struct bitmap_index *,\n \t\t\t\t struct packed_git **packfile,\n \t\t\t\t uint32_t *entries,\ndiff --git a/reachable.c b/reachable.c\nindex 77a60c70a5..fc833cae43 100644\n--- a/reachable.c\n+++ b/reachable.c\n@@ -223,7 +223,7 @@ void mark_reachable_objects(struct rev_info *revs, int mark_reflog,\n \tcp.progress = progress;\n \tcp.count = 0;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, NULL);\n+\tbitmap_git = prepare_bitmap_walk(revs, NULL, 0);\n \tif (bitmap_git) {\n \t\ttraverse_bitmap_commit_list(bitmap_git, revs, mark_object_seen);\n \t\tfree_bitmap_index(bitmap_git);\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex c79ec04060..0a305c9c49 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -207,6 +207,34 @@ test_expect_success 'verify object:type=tag prints tag' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'verify object:type=blob prints only blob with --filter-provided-revisions' '\n+\tprintf \"%s blob\\n\" $(git -C object-type rev-parse HEAD:blob) >expected &&\n+\tgit -C object-type rev-list --objects \\\n+\t\t--filter=object:type=blob --filter-provided-revisions HEAD >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=tree prints only tree with --filter-provided-revisions' '\n+\tprintf \"%s \\n\" $(git -C object-type rev-parse HEAD^{tree}) >expected &&\n+\tgit -C object-type rev-list --objects \\\n+\t\t--filter=object:type=tree HEAD --filter-provided-revisions >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=commit prints only commit with --filter-provided-revisions' '\n+\tgit -C object-type rev-parse HEAD >expected &&\n+\tgit -C object-type rev-list --objects \\\n+\t\t--filter=object:type=commit --filter-provided-revisions HEAD >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=tag prints only tag with --filter-provided-revisions' '\n+\tprintf \"%s tag\\n\" $(git -C object-type rev-parse tag) >expected &&\n+\tgit -C object-type rev-list --objects \\\n+\t\t--filter=object:type=tag --filter-provided-revisions tag >actual &&\n+\ttest_cmp expected actual\n+'\n+\n # Test sparse:path= filter.\n # !!!!\n # NOTE: sparse:path filter support has been dropped for security reasons,\ndiff --git a/t/t6113-rev-list-bitmap-filters.sh b/t/t6113-rev-list-bitmap-filters.sh\nindex cb9db7df6f..bfc9bdafa0 100755\n--- a/t/t6113-rev-list-bitmap-filters.sh\n+++ b/t/t6113-rev-list-bitmap-filters.sh\n@@ -98,6 +98,28 @@ test_expect_success 'object:type filter' '\n \ttest_bitmap_traversal expect actual\n '\n \n+test_expect_success 'object:type filter with --filter-provided-revisions' '\n+\tgit rev-list --objects --filter-provided-revisions --filter=object:type=tag tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter-provided-revisions --filter=object:type=tag tag >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tgit rev-list --objects --filter-provided-revisions --filter=object:type=commit tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter-provided-revisions --filter=object:type=commit tag >actual &&\n+\ttest_bitmap_traversal expect actual &&\n+\n+\tgit rev-list --objects --filter-provided-revisions --filter=object:type=tree tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter-provided-revisions --filter=object:type=tree tag >actual &&\n+\ttest_bitmap_traversal expect actual &&\n+\n+\tgit rev-list --objects --filter-provided-revisions --filter=object:type=blob tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter-provided-revisions --filter=object:type=blob tag >actual &&\n+\ttest_bitmap_traversal expect actual\n+'\n+\n test_expect_success 'combine filter' '\n \tgit rev-list --objects --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&\n \tgit rev-list --use-bitmap-index \\\n@@ -105,4 +127,18 @@ test_expect_success 'combine filter' '\n \ttest_bitmap_traversal expect actual\n '\n \n+test_expect_success 'combine filter with --filter-provided-revisions' '\n+\tgit rev-list --objects --filter-provided-revisions --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter-provided-revisions --filter=blob:limit=1000 --filter=object:type=blob tag >actual &&\n+\ttest_bitmap_traversal expect actual &&\n+\n+\tgit cat-file --batch-check=\"%(objecttype) %(objectsize)\" objects &&\n+\twhile read objecttype objectsize\n+\tdo\n+\t\ttest \"$objecttype\" = blob || return 1\n+\t\ttest \"$objectsize\" -le 1000 || return 1\n+\tdone When providing an object filter, it is currently impossible to also\n> filter provided items. E.g. when executing `git rev-list HEAD` , the\n> commit this reference points to will be treated as user-provided and is\n> thus excluded from the filtering mechanism. This makes it harder than\n> necessary to properly use the new `--filter=object:type` filter given\n> that even if the user wants to only see blobs, he'll still see commits\n> of provided references.\n>\n> Improve this by introducing a new `--filter-provided` option to the\ns/--filter-provided/--filter-provided-revisions/\n\nAlso in some tests - I presume the option should be spelled out in full.\n\n> git-rev-parse(1) command. If given, then all user-provided references\n> will be subject to filtering.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> Documentation/rev-list-options.txt | 5 ++++\n> builtin/pack-objects.c | 2 +-\n> builtin/rev-list.c | 36 +++++++++++++++++++++--------\n> pack-bitmap.c | 6 +++--\n> pack-bitmap.h | 3 ++-\n> reachable.c | 2 +-\n> t/t6112-rev-list-filters-objects.sh | 28 ++++++++++++++++++++++\n> t/t6113-rev-list-bitmap-filters.sh | 36 +++++++++++++++++++++++++++++\n> 8 files changed, 104 insertions(+), 14 deletions(-)\n>\n> diff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\n> index 3afa8fffbd..7fa18fc6e6 100644\n> --- a/Documentation/rev-list-options.txt\n> +++ b/Documentation/rev-list-options.txt\n> @@ -933,6 +933,11 @@ equivalent.\n> --no-filter::\n> \tTurn off any previous `--filter=` argument.\n> \n> +--filter-provided-revisions::\n> +\tFilter the list of explicitly provided revisions, which would otherwise\n> +\talways be printed even if they did not match any of the filters. Only\n> +\tuseful with `--filter=`.\n> +\n> --filter-print-omitted::\n> \tOnly useful with `--filter=`; prints a list of the objects omitted\n> \tby the filter. Object IDs are prefixed with a ``~'' character.\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 525c2d8552..2f2026dc87 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -3516,7 +3516,7 @@ static int pack_options_allow_reuse(void)\n> \n> static int get_object_list_from_bitmap(struct rev_info *revs)\n> {\n> -\tif (!(bitmap_git = prepare_bitmap_walk(revs, &filter_options)))\n> +\tif (!(bitmap_git = prepare_bitmap_walk(revs, &filter_options, 0)))\n> \t\treturn -1;\n> \n> \tif (pack_options_allow_reuse() &&\n> diff --git a/builtin/rev-list.c b/builtin/rev-list.c\n> index b4d8ea0a35..13f0ff3f8d 100644\n> --- a/builtin/rev-list.c\n> +++ b/builtin/rev-list.c\n> @@ -398,7 +398,8 @@ static inline int parse_missing_action_value(const char *value)\n> }\n> \n> static int try_bitmap_count(struct rev_info *revs,\n> -\t\t\t struct list_objects_filter_options *filter)\n> +\t\t\t struct list_objects_filter_options *filter,\n> +\t\t\t int filter_provided_revs)\n> {\n> \tuint32_t commit_count = 0,\n> \t\t tag_count = 0,\n> @@ -433,7 +434,7 @@ static int try_bitmap_count(struct rev_info *revs,\n> \t */\n> \tmax_count = revs->max_count;\n> \n> -\tbitmap_git = prepare_bitmap_walk(revs, filter);\n> +\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_revs);\n> \tif (!bitmap_git)\n> \t\treturn -1;\n> \n> @@ -450,7 +451,8 @@ static int try_bitmap_count(struct rev_info *revs,\n> }\n> \n> static int try_bitmap_traversal(struct rev_info *revs,\n> -\t\t\t\tstruct list_objects_filter_options *filter)\n> +\t\t\t\tstruct list_objects_filter_options *filter,\n> +\t\t\t\tint filter_provided_revs)\n> {\n> \tstruct bitmap_index *bitmap_git;\n> \n> @@ -461,7 +463,7 @@ static int try_bitmap_traversal(struct rev_info *revs,\n> \tif (revs->max_count >= 0)\n> \t\treturn -1;\n> \n> -\tbitmap_git = prepare_bitmap_walk(revs, filter);\n> +\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_revs);\n> \tif (!bitmap_git)\n> \t\treturn -1;\n> \n> @@ -471,14 +473,15 @@ static int try_bitmap_traversal(struct rev_info *revs,\n> }\n> \n> static int try_bitmap_disk_usage(struct rev_info *revs,\n> -\t\t\t\t struct list_objects_filter_options *filter)\n> +\t\t\t\t struct list_objects_filter_options *filter,\n> +\t\t\t\t int filter_provided_revs)\n> {\n> \tstruct bitmap_index *bitmap_git;\n> \n> \tif (!show_disk_usage)\n> \t\treturn -1;\n> \n> -\tbitmap_git = prepare_bitmap_walk(revs, filter);\n> +\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_revs);\n> \tif (!bitmap_git)\n> \t\treturn -1;\n> \n> @@ -499,6 +502,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n> \tint bisect_show_vars = 0;\n> \tint bisect_find_all = 0;\n> \tint use_bitmap_index = 0;\n> +\tint filter_provided_revs = 0;\n> \tconst char *show_progress = NULL;\n> \n> \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n> @@ -599,6 +603,10 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n> \t\t\tlist_objects_filter_set_no_filter(&filter_options);\n> \t\t\tcontinue;\n> \t\t}\n> +\t\tif (!strcmp(arg, \"--filter-provided-revisions\")) {\n> +\t\t\tfilter_provided_revs = 1;\n> +\t\t\tcontinue;\n> +\t\t}\n> \t\tif (!strcmp(arg, \"--filter-print-omitted\")) {\n> \t\t\targ_print_omitted = 1;\n> \t\t\tcontinue;\n> @@ -665,11 +673,11 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n> \t\tprogress = start_delayed_progress(show_progress, 0);\n> \n> \tif (use_bitmap_index) {\n> -\t\tif (!try_bitmap_count(&revs, &filter_options))\n> +\t\tif (!try_bitmap_count(&revs, &filter_options, filter_provided_revs))\n> \t\t\treturn 0;\n> -\t\tif (!try_bitmap_disk_usage(&revs, &filter_options))\n> +\t\tif (!try_bitmap_disk_usage(&revs, &filter_options, filter_provided_revs))\n> \t\t\treturn 0;\n> -\t\tif (!try_bitmap_traversal(&revs, &filter_options))\n> +\t\tif (!try_bitmap_traversal(&revs, &filter_options, filter_provided_revs))\n> \t\t\treturn 0;\n> \t}\n> \n> @@ -694,6 +702,16 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n> \t\t\treturn show_bisect_vars(&info, reaches, all);\n> \t}\n> \n> +\tif (filter_provided_revs) {\n> +\t\tstruct commit_list *c;\n> +\t\tfor (i = 0; i < revs.pending.nr; i++) {\n> +\t\t\tstruct object_array_entry *pending = revs.pending.objects + i;\n> +\t\t\tpending->item->flags |= NOT_USER_GIVEN;\n> +\t\t}\n> +\t\tfor (c = revs.commits; c; c = c->next)\n> +\t\t\tc->item->object.flags |= NOT_USER_GIVEN;\n> +\t}\n> +\n> \tif (arg_print_omitted)\n> \t\toidset_init(&omitted_objects, DEFAULT_OIDSET_SIZE);\n> \tif (arg_missing_action == MA_PRINT)\n> diff --git a/pack-bitmap.c b/pack-bitmap.c\n> index 4385f15828..0576a19a28 100644\n> --- a/pack-bitmap.c\n> +++ b/pack-bitmap.c\n> @@ -1009,7 +1009,8 @@ static int can_filter_bitmap(struct list_objects_filter_options *filter)\n> }\n> \n> struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n> -\t\t\t\t\t struct list_objects_filter_options *filter)\n> +\t\t\t\t\t struct list_objects_filter_options *filter,\n> +\t\t\t\t\t int filter_provided_revs)\n> {\n> \tunsigned int i;\n> \n> @@ -1104,7 +1105,8 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n> \tif (haves_bitmap)\n> \t\tbitmap_and_not(wants_bitmap, haves_bitmap);\n> \n> -\tfilter_bitmap(bitmap_git, wants, wants_bitmap, filter);\n> +\tfilter_bitmap(bitmap_git, (filter && filter_provided_revs) ? NULL : wants,\n> +\t\t wants_bitmap, filter);\n> \n> \tbitmap_git->result = wants_bitmap;\n> \tbitmap_git->haves = haves_bitmap;\n> diff --git a/pack-bitmap.h b/pack-bitmap.h\n> index 36d99930d8..5d8ae3b590 100644\n> --- a/pack-bitmap.h\n> +++ b/pack-bitmap.h\n> @@ -50,7 +50,8 @@ void traverse_bitmap_commit_list(struct bitmap_index *,\n> \t\t\t\t show_reachable_fn show_reachable);\n> void test_bitmap_walk(struct rev_info *revs);\n> struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n> -\t\t\t\t\t struct list_objects_filter_options *filter);\n> +\t\t\t\t\t struct list_objects_filter_options *filter,\n> +\t\t\t\t\t int filter_provided_revs);\n> int reuse_partial_packfile_from_bitmap(struct bitmap_index *,\n> \t\t\t\t struct packed_git **packfile,\n> \t\t\t\t uint32_t *entries,\n> diff --git a/reachable.c b/reachable.c\n> index 77a60c70a5..fc833cae43 100644\n> --- a/reachable.c\n> +++ b/reachable.c\n> @@ -223,7 +223,7 @@ void mark_reachable_objects(struct rev_info *revs, int mark_reflog,\n> \tcp.progress = progress;\n> \tcp.count = 0;\n> \n> -\tbitmap_git = prepare_bitmap_walk(revs, NULL);\n> +\tbitmap_git = prepare_bitmap_walk(revs, NULL, 0);\n> \tif (bitmap_git) {\n> \t\ttraverse_bitmap_commit_list(bitmap_git, revs, mark_object_seen);\n> \t\tfree_bitmap_index(bitmap_git);\n> diff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\n> index c79ec04060..47c558ab0e 100755\n> --- a/t/t6112-rev-list-filters-objects.sh\n> +++ b/t/t6112-rev-list-filters-objects.sh\n> @@ -207,6 +207,34 @@ test_expect_success 'verify object:type=tag prints tag' '\n> \ttest_cmp expected actual\n> '\n> \n> +test_expect_success 'verify object:type=blob prints only blob with --filter-provided' '\n> +\tprintf \"%s blob\\n\" $(git -C object-type rev-parse HEAD:blob) >expected &&\n> +\tgit -C object-type rev-list --objects \\\n> +\t\t--filter=object:type=blob --filter-provided HEAD >actual &&\n> +\ttest_cmp expected actual\n> +'\n> +\n> +test_expect_success 'verify object:type=tree prints only tree with --filter-provided' '\n> +\tprintf \"%s \\n\" $(git -C object-type rev-parse HEAD^{tree}) >expected &&\n> +\tgit -C object-type rev-list --objects \\\n> +\t\t--filter=object:type=tree HEAD --filter-provided >actual &&\n> +\ttest_cmp expected actual\n> +'\n> +\n> +test_expect_success 'verify object:type=commit prints only commit with --filter-provided' '\n> +\tgit -C object-type rev-parse HEAD >expected &&\n> +\tgit -C object-type rev-list --objects \\\n> +\t\t--filter=object:type=commit --filter-provided HEAD >actual &&\n> +\ttest_cmp expected actual\n> +'\n> +\n> +test_expect_success 'verify object:type=tag prints only tag with --filter-provided' '\n> +\tprintf \"%s tag\\n\" $(git -C object-type rev-parse tag) >expected &&\n> +\tgit -C object-type rev-list --objects \\\n> +\t\t--filter=object:type=tag --filter-provided tag >actual &&\n> +\ttest_cmp expected actual\n> +'\n> +\n> # Test sparse:path= filter.\n> # !!!!\n> # NOTE: sparse:path filter support has been dropped for security reasons,\n> diff --git a/t/t6113-rev-list-bitmap-filters.sh b/t/t6113-rev-list-bitmap-filters.sh\n> index cb9db7df6f..9053ac5059 100755\n> --- a/t/t6113-rev-list-bitmap-filters.sh\n> +++ b/t/t6113-rev-list-bitmap-filters.sh\n> @@ -98,6 +98,28 @@ test_expect_success 'object:type filter' '\n> \ttest_bitmap_traversal expect actual\n> '\n> \n> +test_expect_success 'object:type filter with --filter-provided' '\n> +\tgit rev-list --objects --filter-provided --filter=object:type=tag tag >expect &&\n> +\tgit rev-list --use-bitmap-index \\\n> +\t\t --objects --filter-provided --filter=object:type=tag tag >actual &&\n> +\ttest_cmp expect actual &&\n> +\n> +\tgit rev-list --objects --filter-provided --filter=object:type=commit tag >expect &&\n> +\tgit rev-list --use-bitmap-index \\\n> +\t\t --objects --filter-provided --filter=object:type=commit tag >actual &&\n> +\ttest_bitmap_traversal expect actual &&\n> +\n> +\tgit rev-list --objects --filter-provided --filter=object:type=tree tag >expect &&\n> +\tgit rev-list --use-bitmap-index \\\n> +\t\t --objects --filter-provided --filter=object:type=tree tag >actual &&\n> +\ttest_bitmap_traversal expect actual &&\n> +\n> +\tgit rev-list --objects --filter-provided --filter=object:type=blob tag >expect &&\n> +\tgit rev-list --use-bitmap-index \\\n> +\t\t --objects --filter-provided --filter=object:type=blob tag >actual &&\n> +\ttest_bitmap_traversal expect actual\n> +'\n> +\n> test_expect_success 'combine filter' '\n> \tgit rev-list --objects --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&\n> \tgit rev-list --use-bitmap-index \\\n> @@ -105,4 +127,18 @@ test_expect_success 'combine filter' '\n> \ttest_bitmap_traversal expect actual\n> '\n> \n> +test_expect_success 'combine filter with --filter-provided' '\n> +\tgit rev-list --objects --filter-provided --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&\n> +\tgit rev-list --use-bitmap-index \\\n> +\t\t --objects --filter-provided --filter=blob:limit=1000 --filter=object:type=blob tag >actual &&\n> +\ttest_bitmap_traversal expect actual &&\n> +\n> +\tgit cat-file --batch-check=\"%(objecttype) %(objectsize)\" objects &&\n> +\twhile read objecttype objectsize\n> +\tdo\n> +\t\ttest \"$objecttype\" = blob || return 1\n> +\t\ttest \"$objectsize\" -le 1000 || return 1\n> +\tdone +'\n> +\n> test_done\n\n"},{"id":"421413","messageId":"YHB4ZytlI8XI+4N1@coredump.intra.peff.net","threadId":"55236","inReplyTo":"YHAtDp11xFzdUSQl@ncase","subject":"Re: [PATCH v2 7/8] pack-bitmap: implement combined filter","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-09T15:53:11Z","receivedAt":"2021-04-09T15:53:13Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Apr 09, 2021 at 12:31:42PM +0200, Patrick Steinhardt wrote:\n\n> > Hmm. This is essentially reproducing the list in filter_bitmap() of\n> > what's OK for bitmaps. So when adding a new filter, it would have to be\n> > added in both places.\n> > \n> > Can we preserve that property of the original code? I'd think that just\n> > adding LOFC_COMBINE to filter_bitmap() would be sufficient. I.e., this\n> > hunk:\n> > \n> > > +\tif (filter->choice == LOFC_COMBINE) {\n> > > +\t\tint i;\n> > > +\t\tfor (i = 0; i < filter->sub_nr; i++) {\n> > > +\t\t\tfilter_bitmap(bitmap_git, tip_objects, to_filter,\n> > > +\t\t\t\t &filter->sub[i]);\n> > > +\t\t}\n> > > +\t\treturn 0;\n> > > +\t}\n> > \n> > ...except that we need to see if filter_bitmap() returns \"-1\" for any of\n> > the recursive calls. Which we probably should be doing anyway to\n> > propagate any errors (though I think the only \"errors\" we'd return are\n> > \"not supported\", at least for now).\n> > \n> > -Peff\n> \n> But wouldn't that mean that we're now needlessly filtering via bitmaps\n> all the way down the combined filters only to realize at the end that it\n> cannot work because we've got a tree filter with non-zero tree depth?\n> Granted, this will not be the common case. But it still feels like we're\n> doing needless work for cases where we know that bitmaps cannot answer\n> the query.\n\nI don't think so. We first call can_filter_bitmap(filter), which passes\nNULL for bitmap_git. And then in filter_bitmap(), we only do actual work\nif bitmap_git is non-NULL.\n\nThis is the same thing that saves us from even loading the bitmaps\n(which is itself a non-trivial amount of work) if the filter cannot be\nsatisfied by bitmaps.\n\n-Peff\n"},{"id":"421414","messageId":"YHB49723aQrh13yF@coredump.intra.peff.net","threadId":"55236","inReplyTo":"YHA359zqnw7MV8Bp@ncase","subject":"Re: [PATCH v2 7/8] pack-bitmap: implement combined filter","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-09T15:55:35Z","receivedAt":"2021-04-09T15:55:37Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Apr 09, 2021 at 01:17:59PM +0200, Patrick Steinhardt wrote:\n\n> > Before this patch, I think your test:\n> > \n> > > +test_expect_success 'combine filter' '\n> > > +\tgit rev-list --objects --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&\n> > > +\tgit rev-list --use-bitmap-index \\\n> > > +\t\t --objects --filter=blob:limit=1000 --filter=object:type=blob tag >actual &&\n> > > +\ttest_bitmap_traversal expect actual\n> > > +'\n> > \n> > would pass anyway, because we'd just skip using bitmaps. Is there a way\n> > we can tell that the bitmap code actually kicked in? Maybe a perf test\n> > would make it clear (those aren't always run, but hopefully we'd\n> > eventually notice a regression there).\n> \n> I think that's not actually true. Note that we're using\n> `test_bitmap_traversal`:\n\nAh, right. I forgot about the hackery in test_bitmap_traversal() to let\nus tell the difference (even though I was the one who wrote it, I still\nconsider it hackery ;) ).\n\nSo yes, this is a good test that we are allowing the combine filter.\n\n-Peff\n"},{"id":"421415","messageId":"YHB5j1rnC1mRjlqn@coredump.intra.peff.net","threadId":"55236","inReplyTo":"YHAznTddwG/ej4aS@ncase","subject":"Re: [PATCH v2 8/8] rev-list: allow filtering of provided items","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-09T15:58:07Z","receivedAt":"2021-04-09T15:58:10Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Apr 09, 2021 at 12:59:41PM +0200, Patrick Steinhardt wrote:\n\n> > The name seems a little confusing to me, as I can read is as both\n> > \"please filter the provided objects\" and \"a filter has been provided\".\n> > I guess \"--filter-print-provided\" would be more clear. And also the\n> > default, so you'd want \"--no-filter-print-provided\". That's kind of\n> > clunky, though. Maybe \"--filter-omit-provided\"?\n> \n> Hum, \"--filter-omit-provided\" doesn't sound good to me, either. Omit to\n> me sounds like it'd omit filtering provided items, but we're doing\n> the reverse thing.\n\nYeah, I can see that.\n\n> How about \"--filter-provided-revisions\"? Verbose, but at least it cannot\n> be confused with a filter being provided.\n\nYes, that works for me. Maybe \"--filter-provided-objects\", since you\ncould also provide a non-revision on the command line (though I think\nother parts of the docs are happy to refer to \"revisions\" or \"commits\"\non the command line, even though you can clearly provide non-commits\nwhen used with --objects).\n\n-Peff\n"},{"id":"421416","messageId":"YHB7R8hVRt+V+i2W@coredump.intra.peff.net","threadId":"55236","inReplyTo":"YHA3EkH3HDe45EKA@ncase","subject":"Re: [PATCH v2 0/8] rev-parse: implement object type filter","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-09T16:05:27Z","receivedAt":"2021-04-09T16:05:32Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Apr 09, 2021 at 01:14:26PM +0200, Patrick Steinhardt wrote:\n\n> > I dunno. Those aren't objections exactly. Just trying to put my finger\n> > on why my initial reaction was \"huh, why --filter?\".\n> \n> Yeah, I do kind of share these concerns. Ideally, we'd provide a nicer\n> only-user-facing interface to query the repository for various objects.\n> git-cat-file(1) would be the obvious thing that first gets into my mind,\n> where it would be nice to have it filter stuff. But then on the other\n> hand, it's really rather a simple \"Give me what I tell you to\" binary,\n> which is probably a good thing. Other than that I don't think there's\n> any executable that'd be a good fit -- we could do this via a new\n> git-list-objects(1), but then again git-rev-list(1) already does most of\n> what git-list-objects(1) would do, so why bother.\n\nI don't think cat-file does quite the same thing. An important part of\nrev-list is that it is traversing. So it is determining both\nreachability, but also eliminating excluded objects. For example, there\nis no cat-file equivalent (and can never be) of:\n\n git rev-list --objects --filter=object:type=blob $old..$new\n\nLikewise for list-objects (which cat-file really _does_ cover, with\n--batch-all-objects). Obviously you can pair rev-list with cat-file to\ntraverse and then filter, but the whole point of this series is to do so\nmore efficiently.\n\nSo I think putting this into rev-list is the only sensible option. The\nquestion is just whether to use --filter, or if it should be:\n\n git rev-list --show-blobs --show-trees $old..$new\n\nwith rules like:\n\n - if no --show-X is given, show only commits\n\n - if one or more --show-X is given, show all of them (but nothing else)\n\n - --objects is equivalent to providing each of --show-commits\n --show-blobs --show-trees --show-tags\n\n-Peff\n"},{"id":"421566","messageId":"xmqqpmz1icj4.fsf@gitster.g","threadId":"55236","inReplyTo":"cover.1617967252.git.ps@pks.im","subject":"Re: [PATCH v3 0/8] rev-parse: implement object type filter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-11T06:02:55Z","receivedAt":"2021-04-11T06:03:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> Subject: Re: [PATCH v3 0/8] rev-parse: implement object type filter\n>\n> this is the third version of my patch series which implements a new\n> `object:type` filter for git-rev-parse(1) and git-upload-pack(1) and\n> extends support for bitmap indices to work with combined filters.\n\nDo you truly mean rev-parse, or is it just a typo for rev-list?\n\n> This mostly addresses Peff's comments. Thanks for your feedback!\n>\n> - Removed the `base` parameter from `process_tag()`.\n>\n> - The object type filter doesn't assume ordering for the object type\n> enum anymore.\n>\n> - Combined filters in the bitmap path now verify that\n> `filter_bitmap` does not return any errors.\n>\n> - Renamed \"--filter-provided\" to \"--filter-provided-revisions\" and\n> added documentation for it.\n>\n> - Refactored the code to not munge the `filter_provided` field in\n> the filter options struct, but instead carry it in rev-list.c.\n>\n> Please see the attached range-diff for more details.\n>\n> Patrick\n"},{"id":"421568","messageId":"xmqqfszxiadn.fsf@gitster.g","threadId":"55236","inReplyTo":"80193d6ba38254aab3dd2cda39cabf12986fd939.1617967252.git.ps@pks.im","subject":"Re: [PATCH v3 4/8] list-objects: support filtering by tag and commit","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-11T06:49:24Z","receivedAt":"2021-04-11T06:53:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> Object filters currently only support filtering blobs or trees based on\n> some criteria. This commit lays the foundation to also allow filtering\n> of tags and commits.\n>\n> No change in behaviour is expected from this commit given that there are\n> no filters yet for those object types.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> list-objects-filter.c | 40 ++++++++++++++++++++++++++++++++++++++++\n> list-objects-filter.h | 2 ++\n> list-objects.c | 22 +++++++++++++++++++---\n> 3 files changed, 61 insertions(+), 3 deletions(-)\n>\n> diff --git a/list-objects-filter.c b/list-objects-filter.c\n> index 39e2f15333..0ebfa52966 100644\n> --- a/list-objects-filter.c\n> +++ b/list-objects-filter.c\n> @@ -82,6 +82,16 @@ static enum list_objects_filter_result filter_blobs_none(\n> \tdefault:\n> \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n> \n> +\tcase LOFS_TAG:\n> +\t\tassert(obj->type == OBJ_TAG);\n> +\t\t/* always include all tag objects */\n> +\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n> +\n> +\tcase LOFS_COMMIT:\n> +\t\tassert(obj->type == OBJ_COMMIT);\n> +\t\t/* always include all commit objects */\n> +\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n> +\n> \tcase LOFS_BEGIN_TREE:\n> \t\tassert(obj->type == OBJ_TREE);\n> \t\t/* always include all tree objects */\n> @@ -173,6 +183,16 @@ static enum list_objects_filter_result filter_trees_depth(\n> \tdefault:\n> \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n> \n> +\tcase LOFS_TAG:\n> +\t\tassert(obj->type == OBJ_TAG);\n> +\t\t/* always include all tag objects */\n> +\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n> +\n> +\tcase LOFS_COMMIT:\n> +\t\tassert(obj->type == OBJ_COMMIT);\n> +\t\t/* always include all commit objects */\n> +\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n> +\n> \tcase LOFS_END_TREE:\n> \t\tassert(obj->type == OBJ_TREE);\n> \t\tfilter_data->current_depth--;\n> @@ -267,6 +287,16 @@ static enum list_objects_filter_result filter_blobs_limit(\n> \tdefault:\n> \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n> \n> +\tcase LOFS_TAG:\n> +\t\tassert(obj->type == OBJ_TAG);\n> +\t\t/* always include all tag objects */\n> +\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n> +\n> +\tcase LOFS_COMMIT:\n> +\t\tassert(obj->type == OBJ_COMMIT);\n> +\t\t/* always include all commit objects */\n> +\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n> +\n> \tcase LOFS_BEGIN_TREE:\n> \t\tassert(obj->type == OBJ_TREE);\n> \t\t/* always include all tree objects */\n> @@ -371,6 +401,16 @@ static enum list_objects_filter_result filter_sparse(\n> \tdefault:\n> \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n> \n> +\tcase LOFS_TAG:\n> +\t\tassert(obj->type == OBJ_TAG);\n> +\t\t/* always include all tag objects */\n> +\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n> +\n> +\tcase LOFS_COMMIT:\n> +\t\tassert(obj->type == OBJ_COMMIT);\n> +\t\t/* always include all commit objects */\n> +\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n> +\n> \tcase LOFS_BEGIN_TREE:\n> \t\tassert(obj->type == OBJ_TREE);\n> \t\tdtype = DT_DIR;\n> diff --git a/list-objects-filter.h b/list-objects-filter.h\n> index cfd784e203..9e98814111 100644\n> --- a/list-objects-filter.h\n> +++ b/list-objects-filter.h\n> @@ -55,6 +55,8 @@ enum list_objects_filter_result {\n> };\n> \n> enum list_objects_filter_situation {\n> +\tLOFS_COMMIT,\n> +\tLOFS_TAG,\n> \tLOFS_BEGIN_TREE,\n> \tLOFS_END_TREE,\n> \tLOFS_BLOB\n> diff --git a/list-objects.c b/list-objects.c\n> index a5a60301cb..0c524a81ac 100644\n> --- a/list-objects.c\n> +++ b/list-objects.c\n> @@ -217,8 +217,14 @@ static void process_tag(struct traversal_context *ctx,\n> \t\t\tstruct tag *tag,\n> \t\t\tconst char *name)\n> {\n> -\ttag->object.flags |= SEEN;\n> -\tctx->show_object(&tag->object, name, ctx->show_data);\n> +\tenum list_objects_filter_result r;\n> +\n> +\tr = list_objects_filter__filter_object(ctx->revs->repo, LOFS_TAG,\n> +\t\t\t\t\t &tag->object, \"\", 0, ctx->filter);\n\ns/0/NULL/\n\n> +\tif (r & LOFR_MARK_SEEN)\n> +\t\ttag->object.flags |= SEEN;\n> +\tif (r & LOFR_DO_SHOW)\n> +\t\tctx->show_object(&tag->object, name, ctx->show_data);\n> }\n> \n> static void mark_edge_parents_uninteresting(struct commit *commit,\n> @@ -368,6 +374,12 @@ static void do_traverse(struct traversal_context *ctx)\n> \tstrbuf_init(&csp, PATH_MAX);\n> \n> \twhile ((commit = get_revision(ctx->revs)) != NULL) {\n> +\t\tenum list_objects_filter_result r;\n> +\n> +\t\tr = list_objects_filter__filter_object(ctx->revs->repo,\n> +\t\t\t\tLOFS_COMMIT, &commit->object,\n> +\t\t\t\tNULL, NULL, ctx->filter);\n> +\n> \t\t/*\n> \t\t * an uninteresting boundary commit may not have its tree\n> \t\t * parsed yet, but we are not going to show them anyway\n> @@ -382,7 +394,11 @@ static void do_traverse(struct traversal_context *ctx)\n> \t\t\tdie(_(\"unable to load root tree for commit %s\"),\n> \t\t\t oid_to_hex(&commit->object.oid));\n> \t\t}\n> -\t\tctx->show_commit(commit, ctx->show_data);\n> +\n> +\t\tif (r & LOFR_MARK_SEEN)\n> +\t\t\tcommit->object.flags |= SEEN;\n> +\t\tif (r & LOFR_DO_SHOW)\n> +\t\t\tctx->show_commit(commit, ctx->show_data);\n> \n> \t\tif (ctx->revs->tree_blobs_in_commit_order)\n> \t\t\t/*\n"},{"id":"421678","messageId":"YHRHTCHnV9dypO9s@ncase","threadId":"55236","inReplyTo":"xmqqpmz1icj4.fsf@gitster.g","subject":"Re: [PATCH v3 0/8] rev-parse: implement object type filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-12T13:12:44Z","receivedAt":"2021-04-12T13:13:02Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sat, Apr 10, 2021 at 11:02:55PM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > Subject: Re: [PATCH v3 0/8] rev-parse: implement object type filter\n> >\n> > this is the third version of my patch series which implements a new\n> > `object:type` filter for git-rev-parse(1) and git-upload-pack(1) and\n> > extends support for bitmap indices to work with combined filters.\n> \n> Do you truly mean rev-parse, or is it just a typo for rev-list?\n\nIt's a typo both in the series' title and here in the text.\n\nPatrick\n"},{"id":"421679","messageId":"YHRH5so2onO5cfwy@ncase","threadId":"55236","inReplyTo":"8788dbca-3caf-e9d6-7995-b09e276d1609@iee.email","subject":"Re: [PATCH v3 8/8] rev-list: allow filtering of provided items","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-12T13:15:18Z","receivedAt":"2021-04-12T13:15:29Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Apr 09, 2021 at 04:00:26PM +0100, Philip Oakley wrote:\n> typo nit.\n> On 09/04/2021 12:28, Patrick Steinhardt wrote:\n> > When providing an object filter, it is currently impossible to also\n> > filter provided items. E.g. when executing `git rev-list HEAD` , the\n> > commit this reference points to will be treated as user-provided and is\n> > thus excluded from the filtering mechanism. This makes it harder than\n> > necessary to properly use the new `--filter=object:type` filter given\n> > that even if the user wants to only see blobs, he'll still see commits\n> > of provided references.\n> >\n> > Improve this by introducing a new `--filter-provided` option to the\n> s/--filter-provided/--filter-provided-revisions/\n> \n> Also in some tests - I presume the option should be spelled out in full.\n\nRight. I did fix these in the resend because I forgot to stage changes,\nbut still had it in the commit message.\n\nFixed now, thanks!\n\nPatrick\n"},{"id":"421683","messageId":"f80b9570d49665b936e706063e42b9b00c2f7f45.1618234575.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1618234575.git.ps@pks.im","subject":"[PATCH v4 1/8] uploadpack.txt: document implication of `uploadpackfilter.allow`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-12T13:37:20Z","receivedAt":"2021-04-12T13:37:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When `uploadpackfilter.allow` is set to `true`, it means that filters\nare enabled by default except in the case where a filter is explicitly\ndisabled via `uploadpackilter..allow`. This option will not only\nenable the currently supported set of filters, but also any filters\nwhich get added in the future. As such, an admin which wants to have\ntight control over which filters are allowed and which aren't probably\nshouldn't ever set `uploadpackfilter.allow=true`.\n\nAmend the documentation to make the ramifications more explicit so that\nadmins are aware of this.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/config/uploadpack.txt | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config/uploadpack.txt b/Documentation/config/uploadpack.txt\nindex b0d761282c..6729a072ea 100644\n--- a/Documentation/config/uploadpack.txt\n+++ b/Documentation/config/uploadpack.txt\n@@ -59,7 +59,8 @@ uploadpack.allowFilter::\n \n uploadpackfilter.allow::\n \tProvides a default value for unspecified object filters (see: the\n-\tbelow configuration variable).\n+\tbelow configuration variable). If set to `true`, this will also\n+\tenable all filters which get added in the future.\n \tDefaults to `true`.\n \n uploadpackfilter..allow::\n-- \n2.31.1\n\n"},{"id":"421684","messageId":"cover.1618234575.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1617967252.git.ps@pks.im","subject":"[PATCH v4 0/8] rev-list: implement object type filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-12T13:37:14Z","receivedAt":"2021-04-12T13:37:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the fourth version of my patch series which implements a new\n`object:type` filter for git-rev-list(1) and git-upload-pack(1) and\nextends support for bitmap indices to work with combined filters.\n\nChanges compared to v3:\n\n - Small style fix to not pass an empty string and `0`, but instead\n simply pass two `NULL` pointers to\n `list_objects_filter__filter_object, pointed out by Junio.\n\n - I've changed patch 7/8 as proposed by Peff: support of combined\n filters is now determined directly in `filter_bitmap()`, without\n having to mirror all filter types in the new `filter_supported()`\n function.\n\n - Renamed `--filter-provided-revisions` to\n `--filter-provided-objects` as proposed by Peff and addressed both\n commit message and tests as pointed out by Philip.\n\nThanks for all your feedback! As alawys, the range-diff is attached\nbelow.\n\nPatrick\n\nPatrick Steinhardt (8):\n uploadpack.txt: document implication of `uploadpackfilter.allow`\n revision: mark commit parents as NOT_USER_GIVEN\n list-objects: move tag processing into its own function\n list-objects: support filtering by tag and commit\n list-objects: implement object type filter\n pack-bitmap: implement object type filter\n pack-bitmap: implement combined filter\n rev-list: allow filtering of provided items\n\n Documentation/config/uploadpack.txt | 9 ++-\n Documentation/rev-list-options.txt | 8 ++\n builtin/pack-objects.c | 2 +-\n builtin/rev-list.c | 36 ++++++---\n list-objects-filter-options.c | 14 ++++\n list-objects-filter-options.h | 2 +\n list-objects-filter.c | 116 ++++++++++++++++++++++++++++\n list-objects-filter.h | 2 +\n list-objects.c | 30 ++++++-\n pack-bitmap.c | 45 +++++++++--\n pack-bitmap.h | 3 +-\n reachable.c | 2 +-\n revision.c | 4 +-\n revision.h | 3 -\n t/t6112-rev-list-filters-objects.sh | 76 ++++++++++++++++++\n t/t6113-rev-list-bitmap-filters.sh | 68 +++++++++++++++-\n 16 files changed, 390 insertions(+), 30 deletions(-)\n\nRange-diff against v3:\n1: f80b9570d4 = 1: f80b9570d4 uploadpack.txt: document implication of `uploadpackfilter.allow`\n2: 46c1952405 = 2: 46c1952405 revision: mark commit parents as NOT_USER_GIVEN\n3: 3d792f6339 = 3: 3d792f6339 list-objects: move tag processing into its own function\n4: 80193d6ba3 ! 4: 674da0f9ac list-objects: support filtering by tag and commit\n @@ list-objects.c: static void process_tag(struct traversal_context *ctx,\n +\tenum list_objects_filter_result r;\n +\n +\tr = list_objects_filter__filter_object(ctx->revs->repo, LOFS_TAG,\n -+\t\t\t\t\t &tag->object, \"\", 0, ctx->filter);\n ++\t\t\t\t\t &tag->object, NULL, NULL,\n ++\t\t\t\t\t ctx->filter);\n +\tif (r & LOFR_MARK_SEEN)\n +\t\ttag->object.flags |= SEEN;\n +\tif (r & LOFR_DO_SHOW)\n5: e2a14abf92 = 5: d22a5fd37d list-objects: implement object type filter\n6: 46d4450d38 = 6: 17c9f66bbc pack-bitmap: implement object type filter\n7: 06a376399b ! 7: 759ac54bb2 pack-bitmap: implement combined filter\n @@ Commit message\n Signed-off-by: Patrick Steinhardt \n \n ## pack-bitmap.c ##\n -@@ pack-bitmap.c: static void filter_bitmap_object_type(struct bitmap_index *bitmap_git,\n - \t\tfilter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, OBJ_BLOB);\n - }\n - \n -+static int filter_supported(struct list_objects_filter_options *filter)\n -+{\n -+\tint i;\n -+\n -+\tswitch (filter->choice) {\n -+\tcase LOFC_BLOB_NONE:\n -+\tcase LOFC_BLOB_LIMIT:\n -+\tcase LOFC_OBJECT_TYPE:\n -+\t\treturn 1;\n -+\tcase LOFC_TREE_DEPTH:\n -+\t\tif (filter->tree_exclude_depth == 0)\n -+\t\t\treturn 1;\n -+\t\treturn 0;\n -+\tcase LOFC_COMBINE:\n -+\t\tfor (i = 0; i < filter->sub_nr; i++)\n -+\t\t\tif (!filter_supported(&filter->sub[i]))\n -+\t\t\t\treturn 0;\n -+\t\treturn 1;\n -+\tdefault:\n -+\t\treturn 0;\n -+\t}\n -+}\n -+\n - static int filter_bitmap(struct bitmap_index *bitmap_git,\n - \t\t\t struct object_list *tip_objects,\n - \t\t\t struct bitmap *to_filter,\n -@@ pack-bitmap.c: static int filter_bitmap(struct bitmap_index *bitmap_git,\n - {\n - \tif (!filter || filter->choice == LOFC_DISABLED)\n - \t\treturn 0;\n -+\tif (!filter_supported(filter))\n -+\t\treturn -1;\n - \n - \tif (filter->choice == LOFC_BLOB_NONE) {\n - \t\tif (bitmap_git)\n @@ pack-bitmap.c: static int filter_bitmap(struct bitmap_index *bitmap_git,\n \t\treturn 0;\n \t}\n \n --\tif (filter->choice == LOFC_TREE_DEPTH &&\n --\t filter->tree_exclude_depth == 0) {\n -+\tif (filter->choice == LOFC_TREE_DEPTH) {\n - \t\tif (bitmap_git)\n - \t\t\tfilter_bitmap_tree_depth(bitmap_git, tip_objects,\n - \t\t\t\t\t\t to_filter,\n -@@ pack-bitmap.c: static int filter_bitmap(struct bitmap_index *bitmap_git,\n - \t\treturn 0;\n - \t}\n - \n --\t/* filter choice not handled */\n --\treturn -1;\n +\tif (filter->choice == LOFC_COMBINE) {\n +\t\tint i;\n +\t\tfor (i = 0; i < filter->sub_nr; i++) {\n @@ pack-bitmap.c: static int filter_bitmap(struct bitmap_index *bitmap_git,\n +\t\treturn 0;\n +\t}\n +\n -+\tBUG(\"unsupported filter choice\");\n + \t/* filter choice not handled */\n + \treturn -1;\n }\n - \n - static int can_filter_bitmap(struct list_objects_filter_options *filter)\n \n ## t/t6113-rev-list-bitmap-filters.sh ##\n @@ t/t6113-rev-list-bitmap-filters.sh: test_expect_success 'object:type filter' '\n8: cf2297b413 ! 8: c779d222cf rev-list: allow filtering of provided items\n @@ Commit message\n that even if the user wants to only see blobs, he'll still see commits\n of provided references.\n \n - Improve this by introducing a new `--filter-provided` option to the\n - git-rev-parse(1) command. If given, then all user-provided references\n - will be subject to filtering.\n + Improve this by introducing a new `--filter-provided-objects` option\n + to the git-rev-parse(1) command. If given, then all user-provided\n + references will be subject to filtering.\n \n Signed-off-by: Patrick Steinhardt \n \n @@ Documentation/rev-list-options.txt: equivalent.\n --no-filter::\n \tTurn off any previous `--filter=` argument.\n \n -+--filter-provided-revisions::\n -+\tFilter the list of explicitly provided revisions, which would otherwise\n ++--filter-provided-objects::\n ++\tFilter the list of explicitly provided objects, which would otherwise\n +\talways be printed even if they did not match any of the filters. Only\n +\tuseful with `--filter=`.\n +\n @@ builtin/rev-list.c: static inline int parse_missing_action_value(const char *val\n static int try_bitmap_count(struct rev_info *revs,\n -\t\t\t struct list_objects_filter_options *filter)\n +\t\t\t struct list_objects_filter_options *filter,\n -+\t\t\t int filter_provided_revs)\n ++\t\t\t int filter_provided_objects)\n {\n \tuint32_t commit_count = 0,\n \t\t tag_count = 0,\n @@ builtin/rev-list.c: static int try_bitmap_count(struct rev_info *revs,\n \tmax_count = revs->max_count;\n \n -\tbitmap_git = prepare_bitmap_walk(revs, filter);\n -+\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_revs);\n ++\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n @@ builtin/rev-list.c: static int try_bitmap_count(struct rev_info *revs,\n static int try_bitmap_traversal(struct rev_info *revs,\n -\t\t\t\tstruct list_objects_filter_options *filter)\n +\t\t\t\tstruct list_objects_filter_options *filter,\n -+\t\t\t\tint filter_provided_revs)\n ++\t\t\t\tint filter_provided_objects)\n {\n \tstruct bitmap_index *bitmap_git;\n \n @@ builtin/rev-list.c: static int try_bitmap_traversal(struct rev_info *revs,\n \t\treturn -1;\n \n -\tbitmap_git = prepare_bitmap_walk(revs, filter);\n -+\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_revs);\n ++\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n @@ builtin/rev-list.c: static int try_bitmap_traversal(struct rev_info *revs,\n static int try_bitmap_disk_usage(struct rev_info *revs,\n -\t\t\t\t struct list_objects_filter_options *filter)\n +\t\t\t\t struct list_objects_filter_options *filter,\n -+\t\t\t\t int filter_provided_revs)\n ++\t\t\t\t int filter_provided_objects)\n {\n \tstruct bitmap_index *bitmap_git;\n \n @@ builtin/rev-list.c: static int try_bitmap_traversal(struct rev_info *revs,\n \t\treturn -1;\n \n -\tbitmap_git = prepare_bitmap_walk(revs, filter);\n -+\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_revs);\n ++\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n @@ builtin/rev-list.c: int cmd_rev_list(int argc, const char **argv, const char *pr\n \tint bisect_show_vars = 0;\n \tint bisect_find_all = 0;\n \tint use_bitmap_index = 0;\n -+\tint filter_provided_revs = 0;\n ++\tint filter_provided_objects = 0;\n \tconst char *show_progress = NULL;\n \n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n @@ builtin/rev-list.c: int cmd_rev_list(int argc, const char **argv, const char *pr\n \t\t\tlist_objects_filter_set_no_filter(&filter_options);\n \t\t\tcontinue;\n \t\t}\n -+\t\tif (!strcmp(arg, \"--filter-provided-revisions\")) {\n -+\t\t\tfilter_provided_revs = 1;\n ++\t\tif (!strcmp(arg, \"--filter-provided-objects\")) {\n ++\t\t\tfilter_provided_objects = 1;\n +\t\t\tcontinue;\n +\t\t}\n \t\tif (!strcmp(arg, \"--filter-print-omitted\")) {\n @@ builtin/rev-list.c: int cmd_rev_list(int argc, const char **argv, const char *pr\n \n \tif (use_bitmap_index) {\n -\t\tif (!try_bitmap_count(&revs, &filter_options))\n -+\t\tif (!try_bitmap_count(&revs, &filter_options, filter_provided_revs))\n ++\t\tif (!try_bitmap_count(&revs, &filter_options, filter_provided_objects))\n \t\t\treturn 0;\n -\t\tif (!try_bitmap_disk_usage(&revs, &filter_options))\n -+\t\tif (!try_bitmap_disk_usage(&revs, &filter_options, filter_provided_revs))\n ++\t\tif (!try_bitmap_disk_usage(&revs, &filter_options, filter_provided_objects))\n \t\t\treturn 0;\n -\t\tif (!try_bitmap_traversal(&revs, &filter_options))\n -+\t\tif (!try_bitmap_traversal(&revs, &filter_options, filter_provided_revs))\n ++\t\tif (!try_bitmap_traversal(&revs, &filter_options, filter_provided_objects))\n \t\t\treturn 0;\n \t}\n \n @@ builtin/rev-list.c: int cmd_rev_list(int argc, const char **argv, const char *pr\n \t\t\treturn show_bisect_vars(&info, reaches, all);\n \t}\n \n -+\tif (filter_provided_revs) {\n ++\tif (filter_provided_objects) {\n +\t\tstruct commit_list *c;\n +\t\tfor (i = 0; i < revs.pending.nr; i++) {\n +\t\t\tstruct object_array_entry *pending = revs.pending.objects + i;\n @@ pack-bitmap.c: static int can_filter_bitmap(struct list_objects_filter_options *\n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n -\t\t\t\t\t struct list_objects_filter_options *filter)\n +\t\t\t\t\t struct list_objects_filter_options *filter,\n -+\t\t\t\t\t int filter_provided_revs)\n ++\t\t\t\t\t int filter_provided_objects)\n {\n \tunsigned int i;\n \n @@ pack-bitmap.c: struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t\tbitmap_and_not(wants_bitmap, haves_bitmap);\n \n -\tfilter_bitmap(bitmap_git, wants, wants_bitmap, filter);\n -+\tfilter_bitmap(bitmap_git, (filter && filter_provided_revs) ? NULL : wants,\n ++\tfilter_bitmap(bitmap_git, (filter && filter_provided_objects) ? NULL : wants,\n +\t\t wants_bitmap, filter);\n \n \tbitmap_git->result = wants_bitmap;\n @@ pack-bitmap.h: void traverse_bitmap_commit_list(struct bitmap_index *,\n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n -\t\t\t\t\t struct list_objects_filter_options *filter);\n +\t\t\t\t\t struct list_objects_filter_options *filter,\n -+\t\t\t\t\t int filter_provided_revs);\n ++\t\t\t\t\t int filter_provided_objects);\n int reuse_partial_packfile_from_bitmap(struct bitmap_index *,\n \t\t\t\t struct packed_git **packfile,\n \t\t\t\t uint32_t *entries,\n @@ t/t6112-rev-list-filters-objects.sh: test_expect_success 'verify object:type=tag\n \ttest_cmp expected actual\n '\n \n -+test_expect_success 'verify object:type=blob prints only blob with --filter-provided-revisions' '\n ++test_expect_success 'verify object:type=blob prints only blob with --filter-provided-objects' '\n +\tprintf \"%s blob\\n\" $(git -C object-type rev-parse HEAD:blob) >expected &&\n +\tgit -C object-type rev-list --objects \\\n -+\t\t--filter=object:type=blob --filter-provided-revisions HEAD >actual &&\n ++\t\t--filter=object:type=blob --filter-provided-objects HEAD >actual &&\n +\ttest_cmp expected actual\n +'\n +\n -+test_expect_success 'verify object:type=tree prints only tree with --filter-provided-revisions' '\n ++test_expect_success 'verify object:type=tree prints only tree with --filter-provided-objects' '\n +\tprintf \"%s \\n\" $(git -C object-type rev-parse HEAD^{tree}) >expected &&\n +\tgit -C object-type rev-list --objects \\\n -+\t\t--filter=object:type=tree HEAD --filter-provided-revisions >actual &&\n ++\t\t--filter=object:type=tree HEAD --filter-provided-objects >actual &&\n +\ttest_cmp expected actual\n +'\n +\n -+test_expect_success 'verify object:type=commit prints only commit with --filter-provided-revisions' '\n ++test_expect_success 'verify object:type=commit prints only commit with --filter-provided-objects' '\n +\tgit -C object-type rev-parse HEAD >expected &&\n +\tgit -C object-type rev-list --objects \\\n -+\t\t--filter=object:type=commit --filter-provided-revisions HEAD >actual &&\n ++\t\t--filter=object:type=commit --filter-provided-objects HEAD >actual &&\n +\ttest_cmp expected actual\n +'\n +\n -+test_expect_success 'verify object:type=tag prints only tag with --filter-provided-revisions' '\n ++test_expect_success 'verify object:type=tag prints only tag with --filter-provided-objects' '\n +\tprintf \"%s tag\\n\" $(git -C object-type rev-parse tag) >expected &&\n +\tgit -C object-type rev-list --objects \\\n -+\t\t--filter=object:type=tag --filter-provided-revisions tag >actual &&\n ++\t\t--filter=object:type=tag --filter-provided-objects tag >actual &&\n +\ttest_cmp expected actual\n +'\n +\n @@ t/t6113-rev-list-bitmap-filters.sh: test_expect_success 'object:type filter' '\n \ttest_bitmap_traversal expect actual\n '\n \n -+test_expect_success 'object:type filter with --filter-provided-revisions' '\n -+\tgit rev-list --objects --filter-provided-revisions --filter=object:type=tag tag >expect &&\n ++test_expect_success 'object:type filter with --filter-provided-objects' '\n ++\tgit rev-list --objects --filter-provided-objects --filter=object:type=tag tag >expect &&\n +\tgit rev-list --use-bitmap-index \\\n -+\t\t --objects --filter-provided-revisions --filter=object:type=tag tag >actual &&\n ++\t\t --objects --filter-provided-objects --filter=object:type=tag tag >actual &&\n +\ttest_cmp expect actual &&\n +\n -+\tgit rev-list --objects --filter-provided-revisions --filter=object:type=commit tag >expect &&\n ++\tgit rev-list --objects --filter-provided-objects --filter=object:type=commit tag >expect &&\n +\tgit rev-list --use-bitmap-index \\\n -+\t\t --objects --filter-provided-revisions --filter=object:type=commit tag >actual &&\n ++\t\t --objects --filter-provided-objects --filter=object:type=commit tag >actual &&\n +\ttest_bitmap_traversal expect actual &&\n +\n -+\tgit rev-list --objects --filter-provided-revisions --filter=object:type=tree tag >expect &&\n ++\tgit rev-list --objects --filter-provided-objects --filter=object:type=tree tag >expect &&\n +\tgit rev-list --use-bitmap-index \\\n -+\t\t --objects --filter-provided-revisions --filter=object:type=tree tag >actual &&\n ++\t\t --objects --filter-provided-objects --filter=object:type=tree tag >actual &&\n +\ttest_bitmap_traversal expect actual &&\n +\n -+\tgit rev-list --objects --filter-provided-revisions --filter=object:type=blob tag >expect &&\n ++\tgit rev-list --objects --filter-provided-objects --filter=object:type=blob tag >expect &&\n +\tgit rev-list --use-bitmap-index \\\n -+\t\t --objects --filter-provided-revisions --filter=object:type=blob tag >actual &&\n ++\t\t --objects --filter-provided-objects --filter=object:type=blob tag >actual &&\n +\ttest_bitmap_traversal expect actual\n +'\n +\n @@ t/t6113-rev-list-bitmap-filters.sh: test_expect_success 'combine filter' '\n \ttest_bitmap_traversal expect actual\n '\n \n -+test_expect_success 'combine filter with --filter-provided-revisions' '\n -+\tgit rev-list --objects --filter-provided-revisions --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&\n ++test_expect_success 'combine filter with --filter-provided-objects' '\n ++\tgit rev-list --objects --filter-provided-objects --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&\n +\tgit rev-list --use-bitmap-index \\\n -+\t\t --objects --filter-provided-revisions --filter=blob:limit=1000 --filter=object:type=blob tag >actual &&\n ++\t\t --objects --filter-provided-objects --filter=blob:limit=1000 --filter=object:type=blob tag >actual &&\n +\ttest_bitmap_traversal expect actual &&\n +\n +\tgit cat-file --batch-check=\"%(objecttype) %(objectsize)\" objects &&\n-- \n2.31.1\n\n"},{"id":"421685","messageId":"46c1952405173eca40706ed7be3b939b210ad2d5.1618234575.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1618234575.git.ps@pks.im","subject":"[PATCH v4 2/8] revision: mark commit parents as NOT_USER_GIVEN","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-12T13:37:26Z","receivedAt":"2021-04-12T13:37:30Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The NOT_USER_GIVEN flag of an object marks whether a flag was explicitly\nprovided by the user or not. The most important use case for this is\nwhen filtering objects: only objects that were not explicitly requested\nwill get filtered.\n\nThe flag is currently only set for blobs and trees, which has been fine\ngiven that there are no filters for tags or commits currently. We're\nabout to extend filtering capabilities to add object type filter though,\nwhich requires us to set up the NOT_USER_GIVEN flag correctly -- if it's\nnot set, the object wouldn't get filtered at all.\n\nMark unseen commit parents as NOT_USER_GIVEN when processing parents.\nLike this, explicitly provided parents stay user-given and thus\nunfiltered, while parents which get loaded as part of the graph walk\ncan be filtered.\n\nThis commit shouldn't have any user-visible impact yet as there is no\nlogic to filter commits yet.\n\nSigned-off-by: Patrick Steinhardt \n---\n revision.c | 4 ++--\n revision.h | 3 ---\n 2 files changed, 2 insertions(+), 5 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex 553c0faa9b..fd34c75e23 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1123,7 +1123,7 @@ static int process_parents(struct rev_info *revs, struct commit *commit,\n \t\t\t\tmark_parents_uninteresting(p);\n \t\t\tif (p->object.flags & SEEN)\n \t\t\t\tcontinue;\n-\t\t\tp->object.flags |= SEEN;\n+\t\t\tp->object.flags |= (SEEN | NOT_USER_GIVEN);\n \t\t\tif (list)\n \t\t\t\tcommit_list_insert_by_date(p, list);\n \t\t\tif (queue)\n@@ -1165,7 +1165,7 @@ static int process_parents(struct rev_info *revs, struct commit *commit,\n \t\t}\n \t\tp->object.flags |= left_flag;\n \t\tif (!(p->object.flags & SEEN)) {\n-\t\t\tp->object.flags |= SEEN;\n+\t\t\tp->object.flags |= (SEEN | NOT_USER_GIVEN);\n \t\t\tif (list)\n \t\t\t\tcommit_list_insert_by_date(p, list);\n \t\t\tif (queue)\ndiff --git a/revision.h b/revision.h\nindex a24f72dcd1..93aa012f51 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -44,9 +44,6 @@\n /*\n * Indicates object was reached by traversal. i.e. not given by user on\n * command-line or stdin.\n- * NEEDSWORK: NOT_USER_GIVEN doesn't apply to commits because we only support\n- * filtering trees and blobs, but it may be useful to support filtering commits\n- * in the future.\n */\n #define NOT_USER_GIVEN\t(1u<<25)\n #define TRACK_LINEAR\t(1u<<26)\n-- \n2.31.1\n\n"},{"id":"421687","messageId":"3d792f63390da79a6d225de51bdfc651234cf893.1618234575.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1618234575.git.ps@pks.im","subject":"[PATCH v4 3/8] list-objects: move tag processing into its own function","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-12T13:37:30Z","receivedAt":"2021-04-12T13:37:38Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Move processing of tags into its own function to make the logic easier\nto extend when we're going to implement filtering for tags. No change in\nbehaviour is expected from this commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n list-objects.c | 11 +++++++++--\n 1 file changed, 9 insertions(+), 2 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex e19589baa0..a5a60301cb 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -213,6 +213,14 @@ static void process_tree(struct traversal_context *ctx,\n \tfree_tree_buffer(tree);\n }\n \n+static void process_tag(struct traversal_context *ctx,\n+\t\t\tstruct tag *tag,\n+\t\t\tconst char *name)\n+{\n+\ttag->object.flags |= SEEN;\n+\tctx->show_object(&tag->object, name, ctx->show_data);\n+}\n+\n static void mark_edge_parents_uninteresting(struct commit *commit,\n \t\t\t\t\t struct rev_info *revs,\n \t\t\t\t\t show_edge_fn show_edge)\n@@ -334,8 +342,7 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx,\n \t\tif (obj->flags & (UNINTERESTING | SEEN))\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n-\t\t\tobj->flags |= SEEN;\n-\t\t\tctx->show_object(obj, name, ctx->show_data);\n+\t\t\tprocess_tag(ctx, (struct tag *)obj, name);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\n-- \n2.31.1\n\n"},{"id":"421688","messageId":"674da0f9accf64c1d0506013cd216ba17849abc2.1618234575.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1618234575.git.ps@pks.im","subject":"[PATCH v4 4/8] list-objects: support filtering by tag and commit","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-12T13:37:35Z","receivedAt":"2021-04-12T13:37:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Object filters currently only support filtering blobs or trees based on\nsome criteria. This commit lays the foundation to also allow filtering\nof tags and commits.\n\nNo change in behaviour is expected from this commit given that there are\nno filters yet for those object types.\n\nSigned-off-by: Patrick Steinhardt \n---\n list-objects-filter.c | 40 ++++++++++++++++++++++++++++++++++++++++\n list-objects-filter.h | 2 ++\n list-objects.c | 23 ++++++++++++++++++++---\n 3 files changed, 62 insertions(+), 3 deletions(-)\n\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex 39e2f15333..0ebfa52966 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -82,6 +82,16 @@ static enum list_objects_filter_result filter_blobs_none(\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n+\tcase LOFS_TAG:\n+\t\tassert(obj->type == OBJ_TAG);\n+\t\t/* always include all tag objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\tcase LOFS_COMMIT:\n+\t\tassert(obj->type == OBJ_COMMIT);\n+\t\t/* always include all commit objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\t/* always include all tree objects */\n@@ -173,6 +183,16 @@ static enum list_objects_filter_result filter_trees_depth(\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n+\tcase LOFS_TAG:\n+\t\tassert(obj->type == OBJ_TAG);\n+\t\t/* always include all tag objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\tcase LOFS_COMMIT:\n+\t\tassert(obj->type == OBJ_COMMIT);\n+\t\t/* always include all commit objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n \tcase LOFS_END_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\tfilter_data->current_depth--;\n@@ -267,6 +287,16 @@ static enum list_objects_filter_result filter_blobs_limit(\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n+\tcase LOFS_TAG:\n+\t\tassert(obj->type == OBJ_TAG);\n+\t\t/* always include all tag objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\tcase LOFS_COMMIT:\n+\t\tassert(obj->type == OBJ_COMMIT);\n+\t\t/* always include all commit objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\t/* always include all tree objects */\n@@ -371,6 +401,16 @@ static enum list_objects_filter_result filter_sparse(\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n+\tcase LOFS_TAG:\n+\t\tassert(obj->type == OBJ_TAG);\n+\t\t/* always include all tag objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\tcase LOFS_COMMIT:\n+\t\tassert(obj->type == OBJ_COMMIT);\n+\t\t/* always include all commit objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\tdtype = DT_DIR;\ndiff --git a/list-objects-filter.h b/list-objects-filter.h\nindex cfd784e203..9e98814111 100644\n--- a/list-objects-filter.h\n+++ b/list-objects-filter.h\n@@ -55,6 +55,8 @@ enum list_objects_filter_result {\n };\n \n enum list_objects_filter_situation {\n+\tLOFS_COMMIT,\n+\tLOFS_TAG,\n \tLOFS_BEGIN_TREE,\n \tLOFS_END_TREE,\n \tLOFS_BLOB\ndiff --git a/list-objects.c b/list-objects.c\nindex a5a60301cb..7f404677d5 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -217,8 +217,15 @@ static void process_tag(struct traversal_context *ctx,\n \t\t\tstruct tag *tag,\n \t\t\tconst char *name)\n {\n-\ttag->object.flags |= SEEN;\n-\tctx->show_object(&tag->object, name, ctx->show_data);\n+\tenum list_objects_filter_result r;\n+\n+\tr = list_objects_filter__filter_object(ctx->revs->repo, LOFS_TAG,\n+\t\t\t\t\t &tag->object, NULL, NULL,\n+\t\t\t\t\t ctx->filter);\n+\tif (r & LOFR_MARK_SEEN)\n+\t\ttag->object.flags |= SEEN;\n+\tif (r & LOFR_DO_SHOW)\n+\t\tctx->show_object(&tag->object, name, ctx->show_data);\n }\n \n static void mark_edge_parents_uninteresting(struct commit *commit,\n@@ -368,6 +375,12 @@ static void do_traverse(struct traversal_context *ctx)\n \tstrbuf_init(&csp, PATH_MAX);\n \n \twhile ((commit = get_revision(ctx->revs)) != NULL) {\n+\t\tenum list_objects_filter_result r;\n+\n+\t\tr = list_objects_filter__filter_object(ctx->revs->repo,\n+\t\t\t\tLOFS_COMMIT, &commit->object,\n+\t\t\t\tNULL, NULL, ctx->filter);\n+\n \t\t/*\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n@@ -382,7 +395,11 @@ static void do_traverse(struct traversal_context *ctx)\n \t\t\tdie(_(\"unable to load root tree for commit %s\"),\n \t\t\t oid_to_hex(&commit->object.oid));\n \t\t}\n-\t\tctx->show_commit(commit, ctx->show_data);\n+\n+\t\tif (r & LOFR_MARK_SEEN)\n+\t\t\tcommit->object.flags |= SEEN;\n+\t\tif (r & LOFR_DO_SHOW)\n+\t\t\tctx->show_commit(commit, ctx->show_data);\n \n \t\tif (ctx->revs->tree_blobs_in_commit_order)\n \t\t\t/*\n-- \n2.31.1\n\n"},{"id":"421689","messageId":"d22a5fd37dd793c78b1ac17244601cadd1f7c0c9.1618234575.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1618234575.git.ps@pks.im","subject":"[PATCH v4 5/8] list-objects: implement object type filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-12T13:37:39Z","receivedAt":"2021-04-12T13:37:43Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"While it already is possible to filter objects by some criteria in\ngit-rev-list(1), it is not yet possible to filter out only a specific\ntype of objects. This makes some filters less useful. The `blob:limit`\nfilter for example filters blobs such that only those which are smaller\nthan the given limit are returned. But it is unfit to ask only for these\nsmallish blobs, given that git-rev-list(1) will continue to print tags,\ncommits and trees.\n\nNow that we have the infrastructure in place to also filter tags and\ncommits, we can improve this situation by implementing a new filter\nwhich selects objects based on their type. Above query can thus\ntrivially be implemented with the following command:\n\n $ git rev-list --objects --filter=object:type=blob \\\n --filter=blob:limit=200\n\nFurthermore, this filter allows to optimize for certain other cases: if\nfor example only tags or commits have been selected, there is no need to\nwalk down trees.\n\nThe new filter is not yet supported in bitmaps. This is going to be\nimplemented in a subsequent commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/config/uploadpack.txt | 6 +--\n Documentation/rev-list-options.txt | 3 ++\n list-objects-filter-options.c | 14 ++++++\n list-objects-filter-options.h | 2 +\n list-objects-filter.c | 76 +++++++++++++++++++++++++++++\n t/t6112-rev-list-filters-objects.sh | 48 ++++++++++++++++++\n 6 files changed, 146 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/config/uploadpack.txt b/Documentation/config/uploadpack.txt\nindex 6729a072ea..32fad5bbe8 100644\n--- a/Documentation/config/uploadpack.txt\n+++ b/Documentation/config/uploadpack.txt\n@@ -66,9 +66,9 @@ uploadpackfilter.allow::\n uploadpackfilter..allow::\n \tExplicitly allow or ban the object filter corresponding to\n \t``, where `` may be one of: `blob:none`,\n-\t`blob:limit`, `tree`, `sparse:oid`, or `combine`. If using\n-\tcombined filters, both `combine` and all of the nested filter\n-\tkinds must be allowed. Defaults to `uploadpackfilter.allow`.\n+\t`blob:limit`, `object:type`, `tree`, `sparse:oid`, or `combine`.\n+\tIf using combined filters, both `combine` and all of the nested\n+\tfilter kinds must be allowed. Defaults to `uploadpackfilter.allow`.\n \n uploadpackfilter.tree.maxDepth::\n \tOnly allow `--filter=tree:` when `` is no more than the value of\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex b1c8f86c6e..3afa8fffbd 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -892,6 +892,9 @@ or units. n may be zero. The suffixes k, m, and g can be used to name\n units in KiB, MiB, or GiB. For example, 'blob:limit=1k' is the same\n as 'blob:limit=1024'.\n +\n+The form '--filter=object:type=(tag|commit|tree|blob)' omits all objects\n+which are not of the requested type.\n++\n The form '--filter=sparse:oid=' uses a sparse-checkout\n specification contained in the blob (or blob-expression) ''\n to omit blobs that would not be not required for a sparse checkout on\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex d2d1c81caf..bb6f6577d5 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -29,6 +29,8 @@ const char *list_object_filter_config_name(enum list_objects_filter_choice c)\n \t\treturn \"tree\";\n \tcase LOFC_SPARSE_OID:\n \t\treturn \"sparse:oid\";\n+\tcase LOFC_OBJECT_TYPE:\n+\t\treturn \"object:type\";\n \tcase LOFC_COMBINE:\n \t\treturn \"combine\";\n \tcase LOFC__COUNT:\n@@ -97,6 +99,18 @@ static int gently_parse_list_objects_filter(\n \t\t}\n \t\treturn 1;\n \n+\t} else if (skip_prefix(arg, \"object:type=\", &v0)) {\n+\t\tint type = type_from_string_gently(v0, -1, 1);\n+\t\tif (type < 0) {\n+\t\t\tstrbuf_addstr(errbuf, _(\"expected 'object:type='\"));\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\tfilter_options->object_type = type;\n+\t\tfilter_options->choice = LOFC_OBJECT_TYPE;\n+\n+\t\treturn 0;\n+\n \t} else if (skip_prefix(arg, \"combine:\", &v0)) {\n \t\treturn parse_combine_filter(filter_options, v0, errbuf);\n \ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 01767c3c96..4d0d0588cc 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -13,6 +13,7 @@ enum list_objects_filter_choice {\n \tLOFC_BLOB_LIMIT,\n \tLOFC_TREE_DEPTH,\n \tLOFC_SPARSE_OID,\n+\tLOFC_OBJECT_TYPE,\n \tLOFC_COMBINE,\n \tLOFC__COUNT /* must be last */\n };\n@@ -54,6 +55,7 @@ struct list_objects_filter_options {\n \tchar *sparse_oid_name;\n \tunsigned long blob_limit_value;\n \tunsigned long tree_exclude_depth;\n+\tenum object_type object_type;\n \n \t/* LOFC_COMBINE values */\n \ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex 0ebfa52966..1c1ee3d1bb 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -545,6 +545,81 @@ static void filter_sparse_oid__init(\n \tfilter->free_fn = filter_sparse_free;\n }\n \n+/*\n+ * A filter for list-objects to omit large blobs.\n+ * And to OPTIONALLY collect a list of the omitted OIDs.\n+ */\n+struct filter_object_type_data {\n+\tenum object_type object_type;\n+};\n+\n+static enum list_objects_filter_result filter_object_type(\n+\tstruct repository *r,\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tstruct oidset *omits,\n+\tvoid *filter_data_)\n+{\n+\tstruct filter_object_type_data *filter_data = filter_data_;\n+\n+\tswitch (filter_situation) {\n+\tdefault:\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n+\n+\tcase LOFS_TAG:\n+\t\tassert(obj->type == OBJ_TAG);\n+\t\tif (filter_data->object_type == OBJ_TAG)\n+\t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\t\treturn LOFR_MARK_SEEN;\n+\n+\tcase LOFS_COMMIT:\n+\t\tassert(obj->type == OBJ_COMMIT);\n+\t\tif (filter_data->object_type == OBJ_COMMIT)\n+\t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\t\treturn LOFR_MARK_SEEN;\n+\n+\tcase LOFS_BEGIN_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\n+\t\t/*\n+\t\t * If we only want to show commits or tags, then there is no\n+\t\t * need to walk down trees.\n+\t\t */\n+\t\tif (filter_data->object_type == OBJ_COMMIT ||\n+\t\t filter_data->object_type == OBJ_TAG)\n+\t\t\treturn LOFR_SKIP_TREE;\n+\n+\t\tif (filter_data->object_type == OBJ_TREE)\n+\t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\t\treturn LOFR_MARK_SEEN;\n+\n+\tcase LOFS_BLOB:\n+\t\tassert(obj->type == OBJ_BLOB);\n+\n+\t\tif (filter_data->object_type == OBJ_BLOB)\n+\t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\t\treturn LOFR_MARK_SEEN;\n+\n+\tcase LOFS_END_TREE:\n+\t\treturn LOFR_ZERO;\n+\t}\n+}\n+\n+static void filter_object_type__init(\n+\tstruct list_objects_filter_options *filter_options,\n+\tstruct filter *filter)\n+{\n+\tstruct filter_object_type_data *d = xcalloc(1, sizeof(*d));\n+\td->object_type = filter_options->object_type;\n+\n+\tfilter->filter_data = d;\n+\tfilter->filter_object_fn = filter_object_type;\n+\tfilter->free_fn = free;\n+}\n+\n /* A filter which only shows objects shown by all sub-filters. */\n struct combine_filter_data {\n \tstruct subfilter *sub;\n@@ -691,6 +766,7 @@ static filter_init_fn s_filters[] = {\n \tfilter_blobs_limit__init,\n \tfilter_trees_depth__init,\n \tfilter_sparse_oid__init,\n+\tfilter_object_type__init,\n \tfilter_combine__init,\n };\n \ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex 31457d13b9..c79ec04060 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -159,6 +159,54 @@ test_expect_success 'verify blob:limit=1m' '\n \ttest_must_be_empty observed\n '\n \n+# Test object:type= filter.\n+\n+test_expect_success 'setup object-type' '\n+\tgit init object-type &&\n+\techo contents >object-type/blob &&\n+\tgit -C object-type add blob &&\n+\tgit -C object-type commit -m commit-message &&\n+\tgit -C object-type tag tag -m tag-message\n+'\n+\n+test_expect_success 'verify object:type= fails with invalid type' '\n+\ttest_must_fail git -C object-type rev-list --objects --filter=object:type= HEAD &&\n+\ttest_must_fail git -C object-type rev-list --objects --filter=object:type=invalid HEAD\n+'\n+\n+test_expect_success 'verify object:type=blob prints blob and commit' '\n+\t(\n+\t\tgit -C object-type rev-parse HEAD &&\n+\t\tprintf \"%s blob\\n\" $(git -C object-type rev-parse HEAD:blob)\n+\t) >expected &&\n+\tgit -C object-type rev-list --objects --filter=object:type=blob HEAD >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=tree prints tree and commit' '\n+\t(\n+\t\tgit -C object-type rev-parse HEAD &&\n+\t\tprintf \"%s \\n\" $(git -C object-type rev-parse HEAD^{tree})\n+\t) >expected &&\n+\tgit -C object-type rev-list --objects --filter=object:type=tree HEAD >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=commit prints commit' '\n+\tgit -C object-type rev-parse HEAD >expected &&\n+\tgit -C object-type rev-list --objects --filter=object:type=commit HEAD >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=tag prints tag' '\n+\t(\n+\t\tgit -C object-type rev-parse HEAD &&\n+\t\tprintf \"%s tag\\n\" $(git -C object-type rev-parse tag)\n+\t) >expected &&\n+\tgit -C object-type rev-list --objects --filter=object:type=tag tag >actual &&\n+\ttest_cmp expected actual\n+'\n+\n # Test sparse:path= filter.\n # !!!!\n # NOTE: sparse:path filter support has been dropped for security reasons,\n-- \n2.31.1\n\n"},{"id":"421690","messageId":"17c9f66bbcff1f949caaf928102fbab7b96685d6.1618234575.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1618234575.git.ps@pks.im","subject":"[PATCH v4 6/8] pack-bitmap: implement object type filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-12T13:37:43Z","receivedAt":"2021-04-12T13:37:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The preceding commit has added a new object filter for git-rev-list(1)\nwhich allows to filter objects by type. Implement the equivalent filter\nfor packfile bitmaps so that we can answer these queries fast.\n\nSigned-off-by: Patrick Steinhardt \n---\n pack-bitmap.c | 29 ++++++++++++++++++++++++++---\n t/t6113-rev-list-bitmap-filters.sh | 25 ++++++++++++++++++++++++-\n 2 files changed, 50 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex b4513f8672..cd3f5c433e 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -779,9 +779,6 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n \teword_t mask;\n \tuint32_t i;\n \n-\tif (type != OBJ_BLOB && type != OBJ_TREE)\n-\t\tBUG(\"filter_bitmap_exclude_type: unsupported type '%d'\", type);\n-\n \t/*\n \t * The non-bitmap version of this filter never removes\n \t * objects which the other side specifically asked for,\n@@ -911,6 +908,24 @@ static void filter_bitmap_tree_depth(struct bitmap_index *bitmap_git,\n \t\t\t\t OBJ_BLOB);\n }\n \n+static void filter_bitmap_object_type(struct bitmap_index *bitmap_git,\n+\t\t\t\t struct object_list *tip_objects,\n+\t\t\t\t struct bitmap *to_filter,\n+\t\t\t\t enum object_type object_type)\n+{\n+\tif (object_type < OBJ_COMMIT || object_type > OBJ_TAG)\n+\t\tBUG(\"filter_bitmap_object_type given invalid object\");\n+\n+\tif (object_type != OBJ_TAG)\n+\t\tfilter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, OBJ_TAG);\n+\tif (object_type != OBJ_COMMIT)\n+\t\tfilter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, OBJ_COMMIT);\n+\tif (object_type != OBJ_TREE)\n+\t\tfilter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, OBJ_TREE);\n+\tif (object_type != OBJ_BLOB)\n+\t\tfilter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, OBJ_BLOB);\n+}\n+\n static int filter_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t struct object_list *tip_objects,\n \t\t\t struct bitmap *to_filter,\n@@ -943,6 +958,14 @@ static int filter_bitmap(struct bitmap_index *bitmap_git,\n \t\treturn 0;\n \t}\n \n+\tif (filter->choice == LOFC_OBJECT_TYPE) {\n+\t\tif (bitmap_git)\n+\t\t\tfilter_bitmap_object_type(bitmap_git, tip_objects,\n+\t\t\t\t\t\t to_filter,\n+\t\t\t\t\t\t filter->object_type);\n+\t\treturn 0;\n+\t}\n+\n \t/* filter choice not handled */\n \treturn -1;\n }\ndiff --git a/t/t6113-rev-list-bitmap-filters.sh b/t/t6113-rev-list-bitmap-filters.sh\nindex 3f889949ca..fb66735ac8 100755\n--- a/t/t6113-rev-list-bitmap-filters.sh\n+++ b/t/t6113-rev-list-bitmap-filters.sh\n@@ -10,7 +10,8 @@ test_expect_success 'set up bitmapped repo' '\n \ttest_commit much-larger-blob-one &&\n \tgit repack -adb &&\n \ttest_commit two &&\n-\ttest_commit much-larger-blob-two\n+\ttest_commit much-larger-blob-two &&\n+\tgit tag tag\n '\n \n test_expect_success 'filters fallback to non-bitmap traversal' '\n@@ -75,4 +76,26 @@ test_expect_success 'tree:1 filter' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'object:type filter' '\n+\tgit rev-list --objects --filter=object:type=tag tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=object:type=tag tag >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tgit rev-list --objects --filter=object:type=commit tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=object:type=commit tag >actual &&\n+\ttest_bitmap_traversal expect actual &&\n+\n+\tgit rev-list --objects --filter=object:type=tree tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=object:type=tree tag >actual &&\n+\ttest_bitmap_traversal expect actual &&\n+\n+\tgit rev-list --objects --filter=object:type=blob tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=object:type=blob tag >actual &&\n+\ttest_bitmap_traversal expect actual\n+'\n+\n test_done\n-- \n2.31.1\n\n"},{"id":"421691","messageId":"759ac54bb2551097d62ffeac77e6e698eeae96ec.1618234575.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1618234575.git.ps@pks.im","subject":"[PATCH v4 7/8] pack-bitmap: implement combined filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-12T13:37:48Z","receivedAt":"2021-04-12T13:37:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When the user has multiple objects filters specified, then this is\ninternally represented by having a \"combined\" filter. These combined\nfilters aren't yet supported by bitmap indices and can thus not be\naccelerated.\n\nFix this by implementing support for these combined filters. The\nimplementation is quite trivial: when there's a combined filter, we\nsimply recurse into `filter_bitmap()` for all of the sub-filters.\n\nSigned-off-by: Patrick Steinhardt \n---\n pack-bitmap.c | 10 ++++++++++\n t/t6113-rev-list-bitmap-filters.sh | 7 +++++++\n 2 files changed, 17 insertions(+)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex cd3f5c433e..7ce3ede7e4 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -966,6 +966,16 @@ static int filter_bitmap(struct bitmap_index *bitmap_git,\n \t\treturn 0;\n \t}\n \n+\tif (filter->choice == LOFC_COMBINE) {\n+\t\tint i;\n+\t\tfor (i = 0; i < filter->sub_nr; i++) {\n+\t\t\tif (filter_bitmap(bitmap_git, tip_objects, to_filter,\n+\t\t\t\t\t &filter->sub[i]) < 0)\n+\t\t\t\treturn -1;\n+\t\t}\n+\t\treturn 0;\n+\t}\n+\n \t/* filter choice not handled */\n \treturn -1;\n }\ndiff --git a/t/t6113-rev-list-bitmap-filters.sh b/t/t6113-rev-list-bitmap-filters.sh\nindex fb66735ac8..cb9db7df6f 100755\n--- a/t/t6113-rev-list-bitmap-filters.sh\n+++ b/t/t6113-rev-list-bitmap-filters.sh\n@@ -98,4 +98,11 @@ test_expect_success 'object:type filter' '\n \ttest_bitmap_traversal expect actual\n '\n \n+test_expect_success 'combine filter' '\n+\tgit rev-list --objects --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=blob:limit=1000 --filter=object:type=blob tag >actual &&\n+\ttest_bitmap_traversal expect actual\n+'\n+\n test_done\n-- \n2.31.1\n\n"},{"id":"421692","messageId":"c779d222cf68bf1d6cd7973c8cbe75db3dbc9d26.1618234575.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1618234575.git.ps@pks.im","subject":"[PATCH v4 8/8] rev-list: allow filtering of provided items","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-12T13:37:53Z","receivedAt":"2021-04-12T13:37:58Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When providing an object filter, it is currently impossible to also\nfilter provided items. E.g. when executing `git rev-list HEAD` , the\ncommit this reference points to will be treated as user-provided and is\nthus excluded from the filtering mechanism. This makes it harder than\nnecessary to properly use the new `--filter=object:type` filter given\nthat even if the user wants to only see blobs, he'll still see commits\nof provided references.\n\nImprove this by introducing a new `--filter-provided-objects` option\nto the git-rev-parse(1) command. If given, then all user-provided\nreferences will be subject to filtering.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/rev-list-options.txt | 5 ++++\n builtin/pack-objects.c | 2 +-\n builtin/rev-list.c | 36 +++++++++++++++++++++--------\n pack-bitmap.c | 6 +++--\n pack-bitmap.h | 3 ++-\n reachable.c | 2 +-\n t/t6112-rev-list-filters-objects.sh | 28 ++++++++++++++++++++++\n t/t6113-rev-list-bitmap-filters.sh | 36 +++++++++++++++++++++++++++++\n 8 files changed, 104 insertions(+), 14 deletions(-)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 3afa8fffbd..5bf2a85f69 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -933,6 +933,11 @@ equivalent.\n --no-filter::\n \tTurn off any previous `--filter=` argument.\n \n+--filter-provided-objects::\n+\tFilter the list of explicitly provided objects, which would otherwise\n+\talways be printed even if they did not match any of the filters. Only\n+\tuseful with `--filter=`.\n+\n --filter-print-omitted::\n \tOnly useful with `--filter=`; prints a list of the objects omitted\n \tby the filter. Object IDs are prefixed with a ``~'' character.\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 525c2d8552..2f2026dc87 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3516,7 +3516,7 @@ static int pack_options_allow_reuse(void)\n \n static int get_object_list_from_bitmap(struct rev_info *revs)\n {\n-\tif (!(bitmap_git = prepare_bitmap_walk(revs, &filter_options)))\n+\tif (!(bitmap_git = prepare_bitmap_walk(revs, &filter_options, 0)))\n \t\treturn -1;\n \n \tif (pack_options_allow_reuse() &&\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex b4d8ea0a35..7677b1af5a 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -398,7 +398,8 @@ static inline int parse_missing_action_value(const char *value)\n }\n \n static int try_bitmap_count(struct rev_info *revs,\n-\t\t\t struct list_objects_filter_options *filter)\n+\t\t\t struct list_objects_filter_options *filter,\n+\t\t\t int filter_provided_objects)\n {\n \tuint32_t commit_count = 0,\n \t\t tag_count = 0,\n@@ -433,7 +434,7 @@ static int try_bitmap_count(struct rev_info *revs,\n \t */\n \tmax_count = revs->max_count;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, filter);\n+\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -450,7 +451,8 @@ static int try_bitmap_count(struct rev_info *revs,\n }\n \n static int try_bitmap_traversal(struct rev_info *revs,\n-\t\t\t\tstruct list_objects_filter_options *filter)\n+\t\t\t\tstruct list_objects_filter_options *filter,\n+\t\t\t\tint filter_provided_objects)\n {\n \tstruct bitmap_index *bitmap_git;\n \n@@ -461,7 +463,7 @@ static int try_bitmap_traversal(struct rev_info *revs,\n \tif (revs->max_count >= 0)\n \t\treturn -1;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, filter);\n+\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -471,14 +473,15 @@ static int try_bitmap_traversal(struct rev_info *revs,\n }\n \n static int try_bitmap_disk_usage(struct rev_info *revs,\n-\t\t\t\t struct list_objects_filter_options *filter)\n+\t\t\t\t struct list_objects_filter_options *filter,\n+\t\t\t\t int filter_provided_objects)\n {\n \tstruct bitmap_index *bitmap_git;\n \n \tif (!show_disk_usage)\n \t\treturn -1;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, filter);\n+\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -499,6 +502,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \tint bisect_show_vars = 0;\n \tint bisect_find_all = 0;\n \tint use_bitmap_index = 0;\n+\tint filter_provided_objects = 0;\n \tconst char *show_progress = NULL;\n \n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n@@ -599,6 +603,10 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\tlist_objects_filter_set_no_filter(&filter_options);\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!strcmp(arg, \"--filter-provided-objects\")) {\n+\t\t\tfilter_provided_objects = 1;\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!strcmp(arg, \"--filter-print-omitted\")) {\n \t\t\targ_print_omitted = 1;\n \t\t\tcontinue;\n@@ -665,11 +673,11 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\tprogress = start_delayed_progress(show_progress, 0);\n \n \tif (use_bitmap_index) {\n-\t\tif (!try_bitmap_count(&revs, &filter_options))\n+\t\tif (!try_bitmap_count(&revs, &filter_options, filter_provided_objects))\n \t\t\treturn 0;\n-\t\tif (!try_bitmap_disk_usage(&revs, &filter_options))\n+\t\tif (!try_bitmap_disk_usage(&revs, &filter_options, filter_provided_objects))\n \t\t\treturn 0;\n-\t\tif (!try_bitmap_traversal(&revs, &filter_options))\n+\t\tif (!try_bitmap_traversal(&revs, &filter_options, filter_provided_objects))\n \t\t\treturn 0;\n \t}\n \n@@ -694,6 +702,16 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\treturn show_bisect_vars(&info, reaches, all);\n \t}\n \n+\tif (filter_provided_objects) {\n+\t\tstruct commit_list *c;\n+\t\tfor (i = 0; i < revs.pending.nr; i++) {\n+\t\t\tstruct object_array_entry *pending = revs.pending.objects + i;\n+\t\t\tpending->item->flags |= NOT_USER_GIVEN;\n+\t\t}\n+\t\tfor (c = revs.commits; c; c = c->next)\n+\t\t\tc->item->object.flags |= NOT_USER_GIVEN;\n+\t}\n+\n \tif (arg_print_omitted)\n \t\toidset_init(&omitted_objects, DEFAULT_OIDSET_SIZE);\n \tif (arg_missing_action == MA_PRINT)\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 7ce3ede7e4..6b790a834b 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -986,7 +986,8 @@ static int can_filter_bitmap(struct list_objects_filter_options *filter)\n }\n \n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n-\t\t\t\t\t struct list_objects_filter_options *filter)\n+\t\t\t\t\t struct list_objects_filter_options *filter,\n+\t\t\t\t\t int filter_provided_objects)\n {\n \tunsigned int i;\n \n@@ -1081,7 +1082,8 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \tif (haves_bitmap)\n \t\tbitmap_and_not(wants_bitmap, haves_bitmap);\n \n-\tfilter_bitmap(bitmap_git, wants, wants_bitmap, filter);\n+\tfilter_bitmap(bitmap_git, (filter && filter_provided_objects) ? NULL : wants,\n+\t\t wants_bitmap, filter);\n \n \tbitmap_git->result = wants_bitmap;\n \tbitmap_git->haves = haves_bitmap;\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 36d99930d8..bb45217d3b 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -50,7 +50,8 @@ void traverse_bitmap_commit_list(struct bitmap_index *,\n \t\t\t\t show_reachable_fn show_reachable);\n void test_bitmap_walk(struct rev_info *revs);\n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n-\t\t\t\t\t struct list_objects_filter_options *filter);\n+\t\t\t\t\t struct list_objects_filter_options *filter,\n+\t\t\t\t\t int filter_provided_objects);\n int reuse_partial_packfile_from_bitmap(struct bitmap_index *,\n \t\t\t\t struct packed_git **packfile,\n \t\t\t\t uint32_t *entries,\ndiff --git a/reachable.c b/reachable.c\nindex 77a60c70a5..fc833cae43 100644\n--- a/reachable.c\n+++ b/reachable.c\n@@ -223,7 +223,7 @@ void mark_reachable_objects(struct rev_info *revs, int mark_reflog,\n \tcp.progress = progress;\n \tcp.count = 0;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, NULL);\n+\tbitmap_git = prepare_bitmap_walk(revs, NULL, 0);\n \tif (bitmap_git) {\n \t\ttraverse_bitmap_commit_list(bitmap_git, revs, mark_object_seen);\n \t\tfree_bitmap_index(bitmap_git);\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex c79ec04060..de751b65b4 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -207,6 +207,34 @@ test_expect_success 'verify object:type=tag prints tag' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'verify object:type=blob prints only blob with --filter-provided-objects' '\n+\tprintf \"%s blob\\n\" $(git -C object-type rev-parse HEAD:blob) >expected &&\n+\tgit -C object-type rev-list --objects \\\n+\t\t--filter=object:type=blob --filter-provided-objects HEAD >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=tree prints only tree with --filter-provided-objects' '\n+\tprintf \"%s \\n\" $(git -C object-type rev-parse HEAD^{tree}) >expected &&\n+\tgit -C object-type rev-list --objects \\\n+\t\t--filter=object:type=tree HEAD --filter-provided-objects >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=commit prints only commit with --filter-provided-objects' '\n+\tgit -C object-type rev-parse HEAD >expected &&\n+\tgit -C object-type rev-list --objects \\\n+\t\t--filter=object:type=commit --filter-provided-objects HEAD >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=tag prints only tag with --filter-provided-objects' '\n+\tprintf \"%s tag\\n\" $(git -C object-type rev-parse tag) >expected &&\n+\tgit -C object-type rev-list --objects \\\n+\t\t--filter=object:type=tag --filter-provided-objects tag >actual &&\n+\ttest_cmp expected actual\n+'\n+\n # Test sparse:path= filter.\n # !!!!\n # NOTE: sparse:path filter support has been dropped for security reasons,\ndiff --git a/t/t6113-rev-list-bitmap-filters.sh b/t/t6113-rev-list-bitmap-filters.sh\nindex cb9db7df6f..4d8e09167e 100755\n--- a/t/t6113-rev-list-bitmap-filters.sh\n+++ b/t/t6113-rev-list-bitmap-filters.sh\n@@ -98,6 +98,28 @@ test_expect_success 'object:type filter' '\n \ttest_bitmap_traversal expect actual\n '\n \n+test_expect_success 'object:type filter with --filter-provided-objects' '\n+\tgit rev-list --objects --filter-provided-objects --filter=object:type=tag tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter-provided-objects --filter=object:type=tag tag >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tgit rev-list --objects --filter-provided-objects --filter=object:type=commit tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter-provided-objects --filter=object:type=commit tag >actual &&\n+\ttest_bitmap_traversal expect actual &&\n+\n+\tgit rev-list --objects --filter-provided-objects --filter=object:type=tree tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter-provided-objects --filter=object:type=tree tag >actual &&\n+\ttest_bitmap_traversal expect actual &&\n+\n+\tgit rev-list --objects --filter-provided-objects --filter=object:type=blob tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter-provided-objects --filter=object:type=blob tag >actual &&\n+\ttest_bitmap_traversal expect actual\n+'\n+\n test_expect_success 'combine filter' '\n \tgit rev-list --objects --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&\n \tgit rev-list --use-bitmap-index \\\n@@ -105,4 +127,18 @@ test_expect_success 'combine filter' '\n \ttest_bitmap_traversal expect actual\n '\n \n+test_expect_success 'combine filter with --filter-provided-objects' '\n+\tgit rev-list --objects --filter-provided-objects --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter-provided-objects --filter=blob:limit=1000 --filter=object:type=blob tag >actual &&\n+\ttest_bitmap_traversal expect actual &&\n+\n+\tgit cat-file --batch-check=\"%(objecttype) %(objectsize)\" objects &&\n+\twhile read objecttype objectsize\n+\tdo\n+\t\ttest \"$objecttype\" = blob || return 1\n+\t\ttest \"$objectsize\" -le 1000 || return 1\n+\tdone this is the fourth version of my patch series which implements a new\n> `object:type` filter for git-rev-list(1) and git-upload-pack(1) and\n> extends support for bitmap indices to work with combined filters.\n> \n> Changes compared to v3:\n> \n> - Small style fix to not pass an empty string and `0`, but instead\n> simply pass two `NULL` pointers to\n> `list_objects_filter__filter_object, pointed out by Junio.\n> \n> - I've changed patch 7/8 as proposed by Peff: support of combined\n> filters is now determined directly in `filter_bitmap()`, without\n> having to mirror all filter types in the new `filter_supported()`\n> function.\n> \n> - Renamed `--filter-provided-revisions` to\n> `--filter-provided-objects` as proposed by Peff and addressed both\n> commit message and tests as pointed out by Philip.\n\nThanks. I have no more nits to pick. The only question left for me is\nthe big one of \"do we like this with --filter, or should it be some kind\nof rev-list option\", as discussed in:\n\n https://lore.kernel.org/git/YHB7R8hVRt+V+i2W@coredump.intra.peff.net/\n\nAs I said elsewhere, I'm OK going with this route. I just wanted to call\nattention to that earlier response in case you had any final thoughts\n(I'm guessing your final thoughts are \"jeez, I already wrote it with\n--filter, so let's just go with that\" which is perfectly reasonable to\nme. ;) ).\n\n-Peff\n"},{"id":"421850","messageId":"YHVQ9RPLk8/r+8mS@ncase","threadId":"55236","inReplyTo":"YHVMEdeDv9ZeXRAU@coredump.intra.peff.net","subject":"Re: [PATCH v4 0/8] rev-list: implement object type filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-13T08:06:13Z","receivedAt":"2021-04-13T08:06:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Apr 13, 2021 at 03:45:21AM -0400, Jeff King wrote:\n> On Mon, Apr 12, 2021 at 03:37:14PM +0200, Patrick Steinhardt wrote:\n> \n> > this is the fourth version of my patch series which implements a new\n> > `object:type` filter for git-rev-list(1) and git-upload-pack(1) and\n> > extends support for bitmap indices to work with combined filters.\n> > \n> > Changes compared to v3:\n> > \n> > - Small style fix to not pass an empty string and `0`, but instead\n> > simply pass two `NULL` pointers to\n> > `list_objects_filter__filter_object, pointed out by Junio.\n> > \n> > - I've changed patch 7/8 as proposed by Peff: support of combined\n> > filters is now determined directly in `filter_bitmap()`, without\n> > having to mirror all filter types in the new `filter_supported()`\n> > function.\n> > \n> > - Renamed `--filter-provided-revisions` to\n> > `--filter-provided-objects` as proposed by Peff and addressed both\n> > commit message and tests as pointed out by Philip.\n> \n> Thanks. I have no more nits to pick. The only question left for me is\n> the big one of \"do we like this with --filter, or should it be some kind\n> of rev-list option\", as discussed in:\n> \n> https://lore.kernel.org/git/YHB7R8hVRt+V+i2W@coredump.intra.peff.net/\n> \n> As I said elsewhere, I'm OK going with this route. I just wanted to call\n> attention to that earlier response in case you had any final thoughts\n> (I'm guessing your final thoughts are \"jeez, I already wrote it with\n> --filter, so let's just go with that\" which is perfectly reasonable to\n> me. ;) ).\n\nI don't think it would help usability to add new `--show-blobs` and\n`--show-trees` options. The user interface to show this kind of\ninformation exists already with `--objects`, and by adding another way\nof asking a similar query would raise the question of how these two ways\ninteract with each other:\n\n - Does `--show-blobs` have effect if `--objects` is not set?\n\n - Is `--objects` redundant if we have `--show-blobs`, or would\n `--objects --show-blobs` list all objects regardless of whether\n they're blobs or not?\n\n - What would happen if the user says `--show-blobs --no-objects`?\n\n - Are these options mutually exclusive?\n\nWe avoid all these questions by just adding it as a filter.\n\nFurthermore, the filter also allows future iterations which build on top\nof this. If we had a combined OR filter in addition to the existing\ncombined AND filter, the user could say \"Give me all blobs which aren't\nbigger than a specific size PLUS all trees with a depth smaller than 5\nPLUS all commits and tags\". It's not like I'd know of a specific usecase\nfor this right now, but I think the potential of having such filters in\nthe future is a plus.\n\nPatrick\n"},{"id":"421865","messageId":"877dl6eccv.fsf@evledraar.gmail.com","threadId":"55236","inReplyTo":"d22a5fd37dd793c78b1ac17244601cadd1f7c0c9.1618234575.git.ps@pks.im","subject":"Re: [PATCH v4 5/8] list-objects: implement object type filter","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-04-13T09:57:04Z","receivedAt":"2021-04-13T09:57:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Apr 12 2021, Patrick Steinhardt wrote:\n\n> +\t} else if (skip_prefix(arg, \"object:type=\", &v0)) {\n> +\t\tint type = type_from_string_gently(v0, -1, 1);\n> +\t\tif (type < 0) {\n> +\t\t\tstrbuf_addstr(errbuf, _(\"expected 'object:type='\"));\n\nMaybe:\n\n\tstrbuf_addstr(errbuf, _(\"'%s' for 'object:type=' is not a valid object type\"), v0);\n\n> +# Test object:type= filter.\n> +\n> +test_expect_success 'setup object-type' '\n> +\tgit init object-type &&\n> +\techo contents >object-type/blob &&\n> +\tgit -C object-type add blob &&\n> +\tgit -C object-type commit -m commit-message &&\n> +\tgit -C object-type tag tag -m tag-message\n> +'\n\nDoes this really need to not be the shorter:\n\n test_create_repo object-type &&\n test_commit -C object-type blob\n\nOr if it really needs the annotated tag maybe that + --no-tag + create\nthe tag after, and it can eventually use my\nhttps://lore.kernel.org/git/patch-06.16-8d43fdd5865-20210412T110456Z-avarab@gmail.com/\n\n> +test_expect_success 'verify object:type=blob prints blob and commit' '\n> +\t(\n> +\t\tgit -C object-type rev-parse HEAD &&\n> +\t\tprintf \"%s blob\\n\" $(git -C object-type rev-parse HEAD:blob)\n> +\t) >expected &&\n\nJust use > and >> or a single printf with two arguments instead of a\nsubshell?\n"},{"id":"421867","messageId":"87im4qh3cs.fsf@igel.home","threadId":"55236","inReplyTo":"877dl6eccv.fsf@evledraar.gmail.com","subject":"Re: [PATCH v4 5/8] list-objects: implement object type filter","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2021-04-13T10:43:15Z","receivedAt":"2021-04-13T10:43:25Z","isPatch":true,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"On Apr 13 2021, Ævar Arnfjörð Bjarmason wrote:\n\n> On Mon, Apr 12 2021, Patrick Steinhardt wrote:\n>> +test_expect_success 'verify object:type=blob prints blob and commit' '\n>> +\t(\n>> +\t\tgit -C object-type rev-parse HEAD &&\n>> +\t\tprintf \"%s blob\\n\" $(git -C object-type rev-parse HEAD:blob)\n>> +\t) >expected &&\n>\n> Just use > and >> or a single printf with two arguments instead of a\n> subshell?\n\nYou can also use just { } for grouping.\n\nAndreas.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1\n\"And now for something completely different.\"\n"},{"id":"421931","messageId":"xmqqfsztlwxb.fsf@gitster.g","threadId":"55236","inReplyTo":"YHVMEdeDv9ZeXRAU@coredump.intra.peff.net","subject":"Re: [PATCH v4 0/8] rev-list: implement object type filter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-13T21:03:12Z","receivedAt":"2021-04-13T21:03:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> Thanks. I have no more nits to pick. The only question left for me is\n> the big one of \"do we like this with --filter, or should it be some kind\n> of rev-list option\", as discussed in:\n>\n> https://lore.kernel.org/git/YHB7R8hVRt+V+i2W@coredump.intra.peff.net/\n\nI do agree that adding \"--blobs\", \"--trees\", and \"--tags\" options to\nthe \"--objects\" (and implicit --commits) to rev-list parameters is a\nlot more in line with the original design by Linus when we added\n\"--objects\" (vs not giving it). We even internally have had the \"do\nwe show trees?\" \"do we show blobs?\" separate bits from the very\nbeginning of the \"--objects\" feature at 9de48752 (git-rev-list: add\noption to list all objects (not just commits), 2005-06-24), which\nwas extended to cover tags at 3c90f03d (Prepare git-rev-list for\ntracking tag objects too, 2005-06-29). The basic design principle\nhasn't changed when the code was reorganized to be closer to the\ncurrent shape at ae563542 (First cut at libifying revlist\ngeneration, 2006-02-25).\n\nBut back then, we didn't have mechanisms to filter rev-list output\nusing other criteria, which brought us the umbrella notation to use\n\"--filter=...\", so as a notation, it might be OK, provided if\n\n\tgit rev-list \\\n\t --filter=object:type=tag \\\n\t --filter=object:type=commit \\\n\t --filter=object:type=tree \\\n\t --filter=object:type=blob \"$@ther args\"\n\nis an equivalent to existing\n\n\tgit rev-list --objects \"$@ther args\"\n\nI however have to wonder why this need so much code (in other words,\nwhy do we need more than just adding something similar to this block\nin the revision.c machinery:\n\n\t} else if (!strcmp(arg, \"--objects\")) {\n\t\trevs->tag_objects = 1;\n\t\trevs->tree_objects = 1;\n\t\trevs->blob_objects = 1;\n\nthat flips _objects member bits?) though.\n"},{"id":"421964","messageId":"YHbSxl3cVsJUSKGp@ncase","threadId":"55236","inReplyTo":"877dl6eccv.fsf@evledraar.gmail.com","subject":"Re: [PATCH v4 5/8] list-objects: implement object type filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-14T11:32:22Z","receivedAt":"2021-04-14T11:32:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Apr 13, 2021 at 11:57:04AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Mon, Apr 12 2021, Patrick Steinhardt wrote:\n> \n> > +\t} else if (skip_prefix(arg, \"object:type=\", &v0)) {\n> > +\t\tint type = type_from_string_gently(v0, -1, 1);\n> > +\t\tif (type < 0) {\n> > +\t\t\tstrbuf_addstr(errbuf, _(\"expected 'object:type='\"));\n> \n> Maybe:\n> \n> \tstrbuf_addstr(errbuf, _(\"'%s' for 'object:type=' is not a valid object type\"), v0);\n\n`strbuf_addf`, but otherwise I agree that it does make for a more\nactionable error message.\n\n> > +# Test object:type= filter.\n> > +\n> > +test_expect_success 'setup object-type' '\n> > +\tgit init object-type &&\n> > +\techo contents >object-type/blob &&\n> > +\tgit -C object-type add blob &&\n> > +\tgit -C object-type commit -m commit-message &&\n> > +\tgit -C object-type tag tag -m tag-message\n> > +'\n> \n> Does this really need to not be the shorter:\n> \n> test_create_repo object-type &&\n> test_commit -C object-type blob\n> \n> Or if it really needs the annotated tag maybe that + --no-tag + create\n> the tag after, and it can eventually use my\n> https://lore.kernel.org/git/patch-06.16-8d43fdd5865-20210412T110456Z-avarab@gmail.com/\n\nIt must be an annotated tag because that's what the `--object` flag of\ngit-rev-list(1) is working on: objects. But anyway, as you say I can\nstill use these two helpers and then manually create the tag.\n\n> > +test_expect_success 'verify object:type=blob prints blob and commit' '\n> > +\t(\n> > +\t\tgit -C object-type rev-parse HEAD &&\n> > +\t\tprintf \"%s blob\\n\" $(git -C object-type rev-parse HEAD:blob)\n> > +\t) >expected &&\n> \n> Just use > and >> or a single printf with two arguments instead of a\n> subshell?\n\nI don't really mind, happy to take this proposal.\n\nThanks!\n\nPatrick\n"},{"id":"421966","messageId":"YHbZNL0gzw0+1k5h@ncase","threadId":"55236","inReplyTo":"xmqqfsztlwxb.fsf@gitster.g","subject":"Re: [PATCH v4 0/8] rev-list: implement object type filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-14T11:59:48Z","receivedAt":"2021-04-14T12:00:01Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Apr 13, 2021 at 02:03:12PM -0700, Junio C Hamano wrote:\n> Jeff King writes:\n> \n> > Thanks. I have no more nits to pick. The only question left for me is\n> > the big one of \"do we like this with --filter, or should it be some kind\n> > of rev-list option\", as discussed in:\n> >\n> > https://lore.kernel.org/git/YHB7R8hVRt+V+i2W@coredump.intra.peff.net/\n> \n> I do agree that adding \"--blobs\", \"--trees\", and \"--tags\" options to\n> the \"--objects\" (and implicit --commits) to rev-list parameters is a\n> lot more in line with the original design by Linus when we added\n> \"--objects\" (vs not giving it). We even internally have had the \"do\n> we show trees?\" \"do we show blobs?\" separate bits from the very\n> beginning of the \"--objects\" feature at 9de48752 (git-rev-list: add\n> option to list all objects (not just commits), 2005-06-24), which\n> was extended to cover tags at 3c90f03d (Prepare git-rev-list for\n> tracking tag objects too, 2005-06-29). The basic design principle\n> hasn't changed when the code was reorganized to be closer to the\n> current shape at ae563542 (First cut at libifying revlist\n> generation, 2006-02-25).\n> \n> But back then, we didn't have mechanisms to filter rev-list output\n> using other criteria, which brought us the umbrella notation to use\n> \"--filter=...\", so as a notation, it might be OK, provided if\n> \n> \tgit rev-list \\\n> \t --filter=object:type=tag \\\n> \t --filter=object:type=commit \\\n> \t --filter=object:type=tree \\\n> \t --filter=object:type=blob \"$@ther args\"\n> \n> is an equivalent to existing\n> \n> \tgit rev-list --objects \"$@ther args\"\n\nFilters are currently AND filters, so specifying them multiple times\nwould cause us to print nothing. And I don't think we should treat\nobject:type filters any different compared to the other filters, because\nthat could limit our ability to iterate in the future where we may want\nto add OR filters.\n\nI initially said that I didn't want to add `object:type=tag,commit`\nfilters as a way to say that all types which are either a tag or commit\nshould be printed because I thought that having the ability to combine\nfilters with an OR instead of an AND would be the better design here.\nBut on the other hand, there is no reason we cannot have both, and it\nwould implement your above usecase, even though syntax is different.\n\n> I however have to wonder why this need so much code (in other words,\n> why do we need more than just adding something similar to this block\n> in the revision.c machinery:\n> \n> \t} else if (!strcmp(arg, \"--objects\")) {\n> \t\trevs->tag_objects = 1;\n> \t\trevs->tree_objects = 1;\n> \t\trevs->blob_objects = 1;\n> \n> that flips _objects member bits?) though.\n\nSo if we make this part of the rev machinery, I guess your idea is that\nwe can just put the logic into `show_object()`? That would indeed be a\nlot simpler to implement with a lot less code. But on the other hand,\nit's also less efficient: we cannot stop walking down the graph like we\ncan with the current design when e.g. saying \"Only give me\n{tags,commits,trees}\". And with bitmap indices, we can even skip all\nobjects of a certain type at once, without having to load the object,\nright inside the filtering logic. If this was part of `show_object()`,\nwe wouldn't be able to do so (easily).\n\nAnyway, this is assuming that I'm not misinterpreting what you mean by\nyour above comment. So please let me know if I misunderstood.\n\nPatrick\n"},{"id":"421984","messageId":"xmqqmtu0y3r8.fsf@gitster.g","threadId":"55236","inReplyTo":"YHbZNL0gzw0+1k5h@ncase","subject":"Re: [PATCH v4 0/8] rev-list: implement object type filter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-14T21:07:07Z","receivedAt":"2021-04-14T21:07:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> I however have to wonder why this need so much code (in other words,\n>> why do we need more than just adding something similar to this block\n>> in the revision.c machinery:\n>> \n>> \t} else if (!strcmp(arg, \"--objects\")) {\n>> \t\trevs->tag_objects = 1;\n>> \t\trevs->tree_objects = 1;\n>> \t\trevs->blob_objects = 1;\n>> \n>> that flips _objects member bits?) though.\n>\n> So if we make this part of the rev machinery, I guess your idea is that\n> we can just put the logic into `show_object()`?\n\nNot necessarily. I was making a lot more naïve observations.\n\n * Even though we have 3 independent (tag|tree|blob)_objects bits,\n they are all set to true with \"--objects\" or they are cleared\n without. There is no codepath that flips these bits to set some\n but not all of them to be set.\n\n * But if we look at the the hits from\n\n $ git grep -C2 -E -e '([.]|->)(tag|tree|blob)_objects' \\*.[ch]\n\n it is clear that the code is (at least trying to be) prepared for\n them to be set independently. The .tree_objects member is often\n checked without checking others to mark the tree objects on the\n edge of the range uninteresting, for example.\n\n It of course is unknown how well the code is actually prepared\n for these three bits to be set independently, as nobody can set\n these bits independently.\n\n * Which makes a naïve reader to wonder if it would be sufficient\n to have a silly option, like this:\n\n \t} else if (!strcmp(arg, \"--filter=object:type=tree\")) {\n \t\trevs->tag_objects = 0;\n \t\trevs->tree_objects = 1;\n \t\trevs->blob_objects = 0;\n\n in the same if/else if/... cascade as the above (and other types\n as well), in order to do the same thing as this series.\n\nAnd, the above led to the question---the patches in your series\napparently do a lot more (even if we discount the option parsing\npart), and I was wondering if that is because the independence\nbetween these three bits the existing code aspires to maintain is\nbroken.\n\n> Anyway, this is assuming that I'm not misinterpreting what you mean by\n> your above comment. So please let me know if I misunderstood.\n"},{"id":"422018","messageId":"YHgKiXRMq/1Fpg2x@coredump.intra.peff.net","threadId":"55236","inReplyTo":"YHVQ9RPLk8/r+8mS@ncase","subject":"Re: [PATCH v4 0/8] rev-list: implement object type filter","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-15T09:42:33Z","receivedAt":"2021-04-15T09:42:35Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Apr 13, 2021 at 10:06:13AM +0200, Patrick Steinhardt wrote:\n\n> I don't think it would help usability to add new `--show-blobs` and\n> `--show-trees` options. The user interface to show this kind of\n> information exists already with `--objects`, and by adding another way\n> of asking a similar query would raise the question of how these two ways\n> interact with each other:\n> \n> - Does `--show-blobs` have effect if `--objects` is not set?\n> \n> - Is `--objects` redundant if we have `--show-blobs`, or would\n> `--objects --show-blobs` list all objects regardless of whether\n> they're blobs or not?\n> \n> - What would happen if the user says `--show-blobs --no-objects`?\n> \n> - Are these options mutually exclusive?\n> \n> We avoid all these questions by just adding it as a filter.\n\nI'm not too worried about those. I'd imagine that \"--objects\" becomes a\ndocumented synonym for \"--show-trees --show-blobs --show-commits\n--show-tags\", and then the usual interactions take over.\n\nBut...\n\n> Furthermore, the filter also allows future iterations which build on top\n> of this. If we had a combined OR filter in addition to the existing\n> combined AND filter, the user could say \"Give me all blobs which aren't\n> bigger than a specific size PLUS all trees with a depth smaller than 5\n> PLUS all commits and tags\". It's not like I'd know of a specific usecase\n> for this right now, but I think the potential of having such filters in\n> the future is a plus.\n\nYeah, that's true. My biggest complaint is lack of an OR filter, but we\ncould add that later. And then we would be _more_ flexible, as you note,\nsince we could and/or more filters.\n\nSo I'm OK proceeding with this direction.\n\n-Peff\n"},{"id":"422019","messageId":"YHgOCUI++AY/7zwX@coredump.intra.peff.net","threadId":"55236","inReplyTo":"xmqqmtu0y3r8.fsf@gitster.g","subject":"Re: [PATCH v4 0/8] rev-list: implement object type filter","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-15T09:57:29Z","receivedAt":"2021-04-15T09:57:32Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 14, 2021 at 02:07:07PM -0700, Junio C Hamano wrote:\n\n> * But if we look at the the hits from\n> \n> $ git grep -C2 -E -e '([.]|->)(tag|tree|blob)_objects' \\*.[ch]\n> \n> it is clear that the code is (at least trying to be) prepared for\n> them to be set independently. The .tree_objects member is often\n> checked without checking others to mark the tree objects on the\n> edge of the range uninteresting, for example.\n> \n> It of course is unknown how well the code is actually prepared\n> for these three bits to be set independently, as nobody can set\n> these bits independently.\n\nYeah, as somebody who has added or touched a lot of those paths, I've\noften wondered this myself: what would break if you asked for blobs but\nnot trees? I think it does not work like you'd expect, because\nlist-objects.c:process_tree() will not recurse the trees at all (and\nhence you'd never see any blob).\n\n> * Which makes a naïve reader to wonder if it would be sufficient\n> to have a silly option, like this:\n> \n> \t} else if (!strcmp(arg, \"--filter=object:type=tree\")) {\n> \t\trevs->tag_objects = 0;\n> \t\trevs->tree_objects = 1;\n> \t\trevs->blob_objects = 0;\n> \n> in the same if/else if/... cascade as the above (and other types\n> as well), in order to do the same thing as this series.\n> \n> And, the above led to the question---the patches in your series\n> apparently do a lot more (even if we discount the option parsing\n> part), and I was wondering if that is because the independence\n> between these three bits the existing code aspires to maintain is\n> broken.\n\nI don't think the code Patrick added is that much more complex. Most if\nit was cleaning up rough edges in the filtering system, and making sure\nthat bitmaps supported this style of filtering. I _think_ the existing\nbitmap code would Just Work with the example you showed above. It uses\nlist-objects.c to do any fill-in traversal, and then\ntraverse_bitmap_commit_list() uses the type-bitmaps to filter the\nresults.\n\nBut it would be likewise broken for the case of \"no trees, just\nblobs\", because of the problem in list-objects.c that I mentioned (but\nworse, it would only be _half_ broken; we might even produce the right\nanswer if we don't have to do any fill-in traversal!).\n\nAnyway. I do think this all could be done using the existing bits in\nrev_info. But there is value in making it all part of the modular\nfiltering system.\n\n-Peff\n"},{"id":"422040","messageId":"xmqqczuvsac6.fsf@gitster.g","threadId":"55236","inReplyTo":"YHgOCUI++AY/7zwX@coredump.intra.peff.net","subject":"Re: [PATCH v4 0/8] rev-list: implement object type filter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-15T17:53:45Z","receivedAt":"2021-04-15T17:53:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> Yeah, as somebody who has added or touched a lot of those paths, I've\n> often wondered this myself: what would break if you asked for blobs but\n> not trees? I think it does not work like you'd expect, because\n> list-objects.c:process_tree() will not recurse the trees at all (and\n> hence you'd never see any blob).\n\nAh, OK, that much I figured. I was wondering if it is worth\n\"fixing\", IOW, if we see blobs being asked without trees being\nasked, perhaps process_tree() should be taught to descend---even if\nit won't show the trees themselves, blobs contained within would.\n\n>> And, the above led to the question---the patches in your series\n>> apparently do a lot more (even if we discount the option parsing\n>> part), and I was wondering if that is because the independence\n>> between these three bits the existing code aspires to maintain is\n>> broken.\n> ...\n> Anyway. I do think this all could be done using the existing bits in\n> rev_info. But there is value in making it all part of the modular\n> filtering system.\n\nYes, I do not have a problem with the approach to add new features\nas part of the \"modular filtering system\". But that leads me to\nwonder into a different direction---coalesce (tag|tree|blob)_objects\nmembers into a single bit, say all_objects, have \"--objects\" and\nfriends set that single bit, and update places like these to check\nthat single bit.\n\nThe patch is for illustration purposes; I didn't even aim to cover\nthe entirety of these two files.\n\n builtin/rev-list.c | 9 ++++-----\n pack-bitmap.c | 4 +---\n 2 files changed, 5 insertions(+), 8 deletions(-)\n\ndiff --git c/builtin/rev-list.c w/builtin/rev-list.c\nindex 7677b1af5a..b6f0d7e9a3 100644\n--- c/builtin/rev-list.c\n+++ w/builtin/rev-list.c\n@@ -424,8 +424,7 @@ static int try_bitmap_count(struct rev_info *revs,\n \t * commits to traverse, since we don't know which objects go with which\n \t * commit.\n \t */\n-\tif (revs->max_count >= 0 &&\n-\t (revs->tag_objects || revs->tree_objects || revs->blob_objects))\n+\tif (revs->max_count >= 0 && revs->all_objects)\n \t\treturn -1;\n \n \t/*\n@@ -439,9 +438,9 @@ static int try_bitmap_count(struct rev_info *revs,\n \t\treturn -1;\n \n \tcount_bitmap_commit_list(bitmap_git, &commit_count,\n-\t\t\t\t revs->tree_objects ? &tree_count : NULL,\n-\t\t\t\t revs->blob_objects ? &blob_count : NULL,\n-\t\t\t\t revs->tag_objects ? &tag_count : NULL);\n+\t\t\t\t revs->all_objects ? &tree_count : NULL,\n+\t\t\t\t revs->all_objects ? &blob_count : NULL,\n+\t\t\t\t revs->all_objects ? &tag_count : NULL);\n \tif (max_count >= 0 && max_count < commit_count)\n \t\tcommit_count = max_count;\n \ndiff --git c/pack-bitmap.c w/pack-bitmap.c\nindex d7e9f14f92..918c80a391 100644\n--- c/pack-bitmap.c\n+++ w/pack-bitmap.c\n@@ -801,9 +801,7 @@ static void show_extended_objects(struct bitmap_index *bitmap_git,\n \t\t\tcontinue;\n \n \t\tobj = eindex->objects[i];\n-\t\tif ((obj->type == OBJ_BLOB && !revs->blob_objects) ||\n-\t\t (obj->type == OBJ_TREE && !revs->tree_objects) ||\n-\t\t (obj->type == OBJ_TAG && !revs->tag_objects))\n+\t\tif (!revs->all_objects && obj->type != OBJ_COMMIT)\n \t\t\tcontinue;\n \n \t\tshow_reach(&obj->oid, obj->type, 0, eindex->hashes[i], NULL, 0);\n"},{"id":"422041","messageId":"xmqq8s5jsa5c.fsf@gitster.g","threadId":"55236","inReplyTo":"xmqqczuvsac6.fsf@gitster.g","subject":"Re: [PATCH v4 0/8] rev-list: implement object type filter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-15T17:57:51Z","receivedAt":"2021-04-15T17:57:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> ... But that leads me to\n> wonder into a different direction---coalesce (tag|tree|blob)_objects\n> members into a single bit, say all_objects, have \"--objects\" and\n> friends set that single bit, and update places like these to check\n> that single bit.\n\nJust to avoid misunderstanding, I am not saying this topic needs to\naddress any of this unifying of three bits.\n\nIt is just an idea for those interested to think about, when they\nhave nothing better to do, when the codebase is quiescent.\n"},{"id":"422129","messageId":"xmqqlf9hkhpp.fsf@gitster.g","threadId":"55236","inReplyTo":"YHgKiXRMq/1Fpg2x@coredump.intra.peff.net","subject":"Re: [PATCH v4 0/8] rev-list: implement object type filter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-16T22:06:10Z","receivedAt":"2021-04-16T22:06:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n>> Furthermore, the filter also allows future iterations which build on top\n>> of this. If we had a combined OR filter in addition to the existing\n>> combined AND filter, the user could say \"Give me all blobs which aren't\n>> bigger than a specific size PLUS all trees with a depth smaller than 5\n>> PLUS all commits and tags\". It's not like I'd know of a specific usecase\n>> for this right now, but I think the potential of having such filters in\n>> the future is a plus.\n>\n> Yeah, that's true. My biggest complaint is lack of an OR filter, but we\n> could add that later. And then we would be _more_ flexible, as you note,\n> since we could and/or more filters.\n>\n> So I'm OK proceeding with this direction.\n\nI think the only remaining issues are the comments on 5/8 on tests,\nthen? Hopefully we can have one more iteration to finalize the\ntopic and merge it down to 'next'?\n\nThanks.\n"},{"id":"422132","messageId":"xmqq4kg5kehw.fsf@gitster.g","threadId":"55236","inReplyTo":"xmqqlf9hkhpp.fsf@gitster.g","subject":"Re: [PATCH v4 0/8] rev-list: implement object type filter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-16T23:15:39Z","receivedAt":"2021-04-16T23:15:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> I think the only remaining issues are the comments on 5/8 on tests,\n> then? Hopefully we can have one more iteration to finalize the\n> topic and merge it down to 'next'?\n>\n> Thanks.\n\nI guess not. I am guessing this topic is responsible for\n\n https://github.com/git/git/runs/2366364023?check_suite_focus=true#step:4:115\n"},{"id":"422144","messageId":"YHo3LO2RKfe0W/z7@satellite","threadId":"55236","inReplyTo":"xmqq4kg5kehw.fsf@gitster.g","subject":"Re: [PATCH v4 0/8] rev-list: implement object type filter","fromName":"Ramsay Jones","fromEmail":"ramsay@ramsayjones.plus.com","sentAt":"2021-04-17T01:17:32Z","receivedAt":"2021-04-17T01:17:39Z","isPatch":true,"sender":{"key":"ramsay@ramsayjones.plus.com","avatar":"https://avatars.githubusercontent.com/u/33702710?v=4"},"body":"On Fri, Apr 16, 2021 at 04:15:39PM -0700, Junio C Hamano wrote:\n> Junio C Hamano writes:\n> \n> > I think the only remaining issues are the comments on 5/8 on tests,\n> > then? Hopefully we can have one more iteration to finalize the\n> > topic and merge it down to 'next'?\n> >\n> > Thanks.\n> \n> I guess not. I am guessing this topic is responsible for\n> \n> https://github.com/git/git/runs/2366364023?check_suite_focus=true#step:4:115\n\nYes, I noticed this a few days ago, and tried the obvious fix (ie to\n#include \"cache.h\" at the start of the list-objects-filter-options.h\nheader file) which does indeed work fine. However, I then thought that\nmoving the definition of 'enum object_type' (along with the TYPE_BITS\n#define) to the 'object.h' header file (and #include \"object.h\" into\ncache.h) would be a better idea...\n\nHaving done that, I wondered how many '#include \"cache.h\"' could be\nreplaced by \"object.h\", and ... well, that was a few days ago and\nsomething came up...\n\nATB,\nRamsay Jones\n\n"},{"id":"422158","messageId":"YHqjHA6ZUYvT6w1T@coredump.intra.peff.net","threadId":"55236","inReplyTo":"xmqq8s5jsa5c.fsf@gitster.g","subject":"Re: [PATCH v4 0/8] rev-list: implement object type filter","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-17T08:58:04Z","receivedAt":"2021-04-17T08:58:06Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Apr 15, 2021 at 10:57:51AM -0700, Junio C Hamano wrote:\n\n> Junio C Hamano writes:\n> \n> > ... But that leads me to\n> > wonder into a different direction---coalesce (tag|tree|blob)_objects\n> > members into a single bit, say all_objects, have \"--objects\" and\n> > friends set that single bit, and update places like these to check\n> > that single bit.\n> \n> Just to avoid misunderstanding, I am not saying this topic needs to\n> address any of this unifying of three bits.\n> \n> It is just an idea for those interested to think about, when they\n> have nothing better to do, when the codebase is quiescent.\n\nIt does feel like going \"backwards\" in a sense. We have the three flags\nmostly split, and we'd lose that distinction. On the other hand, if the\ncurrent split is imperfect, it may be leading people down a confusing\npath (I _think_ this \"trees must be set in order to see blobs\" thing is\nthe only real gotcha, but there could be others).\n\nThere's some other discussion in this old thread:\n\n https://lore.kernel.org/git/06a84f8c77924b275606384ead8bb2fd7d75f7b6.1487984670.git.jonathantanmy@google.com/\n\n(I didn't remember it, but my spider sense tingling caused me to dig in\nthe archive a bit).\n\n-Peff\n"},{"id":"422159","messageId":"YHqkBFp+CGzWW4Xs@coredump.intra.peff.net","threadId":"55236","inReplyTo":"YHo3LO2RKfe0W/z7@satellite","subject":"Re: [PATCH v4 0/8] rev-list: implement object type filter","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-17T09:01:56Z","receivedAt":"2021-04-17T09:01:59Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Apr 17, 2021 at 02:17:32AM +0100, Ramsay Jones wrote:\n\n> On Fri, Apr 16, 2021 at 04:15:39PM -0700, Junio C Hamano wrote:\n> > Junio C Hamano writes:\n> > \n> > > I think the only remaining issues are the comments on 5/8 on tests,\n> > > then? Hopefully we can have one more iteration to finalize the\n> > > topic and merge it down to 'next'?\n> > >\n> > > Thanks.\n> > \n> > I guess not. I am guessing this topic is responsible for\n> > \n> > https://github.com/git/git/runs/2366364023?check_suite_focus=true#step:4:115\n> \n> Yes, I noticed this a few days ago, and tried the obvious fix (ie to\n> #include \"cache.h\" at the start of the list-objects-filter-options.h\n> header file) which does indeed work fine. However, I then thought that\n> moving the definition of 'enum object_type' (along with the TYPE_BITS\n> #define) to the 'object.h' header file (and #include \"object.h\" into\n> cache.h) would be a better idea...\n> \n> Having done that, I wondered how many '#include \"cache.h\"' could be\n> replaced by \"object.h\", and ... well, that was a few days ago and\n> something came up...\n\nI agree that would probably be nice, but let's not hold up the topic. It\ncan include \"cache.h\" like a bunch of other headers in the meantime.\n\n(Too bad we can't just forward-declare the enum like we do for some\nother types, but it's impossible in C).\n\n-Peff\n"},{"id":"422191","messageId":"xmqq35vod1q9.fsf@gitster.g","threadId":"55236","inReplyTo":"YHqkBFp+CGzWW4Xs@coredump.intra.peff.net","subject":"Re: [PATCH v4 0/8] rev-list: implement object type filter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-17T21:45:34Z","receivedAt":"2021-04-17T21:45:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> I agree that would probably be nice, but let's not hold up the topic. It\n> can include \"cache.h\" like a bunch of other headers in the meantime.\n>\n> (Too bad we can't just forward-declare the enum like we do for some\n> other types, but it's impossible in C).\n\nFor now...\n\nFrom f08346ce05e5512e5914e760aba769d0ef8035bc Mon Sep 17 00:00:00 2001\nFrom: Junio C Hamano \nDate: Sat, 17 Apr 2021 14:44:37 -0700\nSubject: [PATCH] fixup! list-objects: implement object type filter\n\n---\n list-objects-filter-options.h | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 4d0d0588cc..da5b6737e2 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -1,6 +1,7 @@\n #ifndef LIST_OBJECTS_FILTER_OPTIONS_H\n #define LIST_OBJECTS_FILTER_OPTIONS_H\n \n+#include \"cache.h\"\n #include \"parse-options.h\"\n #include \"string-list.h\"\n \n-- \n2.31.1-450-gca40e2789b\n\n"},{"id":"422296","messageId":"cover.1618832437.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1618234575.git.ps@pks.im","subject":"[PATCH v5 0/8] rev-list: implement object type filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-19T11:46:32Z","receivedAt":"2021-04-19T11:46:43Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the fifth version of my patch series which implements a new\n`object:type` filter for git-rev-list(1) and git-upload-pack(1) and\nextends support for bitmap indices to work with combined filters.\n\nChanges compared to v4:\n\n - I'm now explicitly passing `strlen(v0)` to\n `type_from_string_gently()` to be prepared for the future\n semantics change here.\n\n - The error message printed in case the user passes in invalid\n object type to `--filter=object:type=` now prints the wrong\n value passed by the user.\n\n - Fixed missing header in list-objects-filter-options.h.\n\n - Tests now use test_create_repo and test_commit.\n\n - Removed a needless subshell in the tests.\n\nI hope that this catches all feedback and that I didn't misunderstand or\nmiss something. If I did, please give me a shout!\n\nPatrick\n\nPatrick Steinhardt (8):\n uploadpack.txt: document implication of `uploadpackfilter.allow`\n revision: mark commit parents as NOT_USER_GIVEN\n list-objects: move tag processing into its own function\n list-objects: support filtering by tag and commit\n list-objects: implement object type filter\n pack-bitmap: implement object type filter\n pack-bitmap: implement combined filter\n rev-list: allow filtering of provided items\n\n Documentation/config/uploadpack.txt | 9 ++-\n Documentation/rev-list-options.txt | 8 ++\n builtin/pack-objects.c | 2 +-\n builtin/rev-list.c | 36 ++++++---\n list-objects-filter-options.c | 15 ++++\n list-objects-filter-options.h | 3 +\n list-objects-filter.c | 116 ++++++++++++++++++++++++++++\n list-objects-filter.h | 2 +\n list-objects.c | 30 ++++++-\n pack-bitmap.c | 45 +++++++++--\n pack-bitmap.h | 3 +-\n reachable.c | 2 +-\n revision.c | 4 +-\n revision.h | 3 -\n t/t6112-rev-list-filters-objects.sh | 72 +++++++++++++++++\n t/t6113-rev-list-bitmap-filters.sh | 68 +++++++++++++++-\n 16 files changed, 388 insertions(+), 30 deletions(-)\n\nRange-diff against v4:\n1: f80b9570d4 = 1: bcc81336b1 uploadpack.txt: document implication of `uploadpackfilter.allow`\n2: 46c1952405 = 2: ffe8e2bf73 revision: mark commit parents as NOT_USER_GIVEN\n3: 3d792f6339 = 3: 21d7f06d0a list-objects: move tag processing into its own function\n4: 674da0f9ac = 4: b5d6ab6b4a list-objects: support filtering by tag and commit\n5: d22a5fd37d ! 5: f462745290 list-objects: implement object type filter\n @@ list-objects-filter-options.c: static int gently_parse_list_objects_filter(\n \t\treturn 1;\n \n +\t} else if (skip_prefix(arg, \"object:type=\", &v0)) {\n -+\t\tint type = type_from_string_gently(v0, -1, 1);\n ++\t\tint type = type_from_string_gently(v0, strlen(v0), 1);\n +\t\tif (type < 0) {\n -+\t\t\tstrbuf_addstr(errbuf, _(\"expected 'object:type='\"));\n ++\t\t\tstrbuf_addf(errbuf, _(\"'%s' for 'object:type=' is\"\n ++\t\t\t\t\t \"not a valid object type\"), v0);\n +\t\t\treturn 1;\n +\t\t}\n +\n @@ list-objects-filter-options.c: static int gently_parse_list_objects_filter(\n \n \n ## list-objects-filter-options.h ##\n +@@\n + #ifndef LIST_OBJECTS_FILTER_OPTIONS_H\n + #define LIST_OBJECTS_FILTER_OPTIONS_H\n + \n ++#include \"cache.h\"\n + #include \"parse-options.h\"\n + #include \"string-list.h\"\n + \n @@ list-objects-filter-options.h: enum list_objects_filter_choice {\n \tLOFC_BLOB_LIMIT,\n \tLOFC_TREE_DEPTH,\n @@ t/t6112-rev-list-filters-objects.sh: test_expect_success 'verify blob:limit=1m'\n +# Test object:type= filter.\n +\n +test_expect_success 'setup object-type' '\n -+\tgit init object-type &&\n -+\techo contents >object-type/blob &&\n -+\tgit -C object-type add blob &&\n -+\tgit -C object-type commit -m commit-message &&\n ++\ttest_create_repo object-type &&\n ++\ttest_commit --no-tag -C object-type message blob &&\n +\tgit -C object-type tag tag -m tag-message\n +'\n +\n @@ t/t6112-rev-list-filters-objects.sh: test_expect_success 'verify blob:limit=1m'\n +'\n +\n +test_expect_success 'verify object:type=blob prints blob and commit' '\n -+\t(\n -+\t\tgit -C object-type rev-parse HEAD &&\n -+\t\tprintf \"%s blob\\n\" $(git -C object-type rev-parse HEAD:blob)\n -+\t) >expected &&\n ++\tgit -C object-type rev-parse HEAD >expected &&\n ++\tprintf \"%s blob\\n\" $(git -C object-type rev-parse HEAD:blob) >>expected &&\n +\tgit -C object-type rev-list --objects --filter=object:type=blob HEAD >actual &&\n +\ttest_cmp expected actual\n +'\n6: 17c9f66bbc = 6: 90c80c1efa pack-bitmap: implement object type filter\n7: 759ac54bb2 = 7: 9726e69861 pack-bitmap: implement combined filter\n8: c779d222cf ! 8: fce3551e48 rev-list: allow filtering of provided items\n @@ pack-bitmap.c: struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \n ## pack-bitmap.h ##\n @@ pack-bitmap.h: void traverse_bitmap_commit_list(struct bitmap_index *,\n - \t\t\t\t show_reachable_fn show_reachable);\n void test_bitmap_walk(struct rev_info *revs);\n + int test_bitmap_commits(struct repository *r);\n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n -\t\t\t\t\t struct list_objects_filter_options *filter);\n +\t\t\t\t\t struct list_objects_filter_options *filter,\n-- \n2.31.1\n\n"},{"id":"422297","messageId":"bcc81336b10ff22f0844fdd7f7fe7e2c3807cec6.1618832437.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1618832437.git.ps@pks.im","subject":"[PATCH v5 1/8] uploadpack.txt: document implication of `uploadpackfilter.allow`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-19T11:46:36Z","receivedAt":"2021-04-19T11:46:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When `uploadpackfilter.allow` is set to `true`, it means that filters\nare enabled by default except in the case where a filter is explicitly\ndisabled via `uploadpackilter..allow`. This option will not only\nenable the currently supported set of filters, but also any filters\nwhich get added in the future. As such, an admin which wants to have\ntight control over which filters are allowed and which aren't probably\nshouldn't ever set `uploadpackfilter.allow=true`.\n\nAmend the documentation to make the ramifications more explicit so that\nadmins are aware of this.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/config/uploadpack.txt | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config/uploadpack.txt b/Documentation/config/uploadpack.txt\nindex b0d761282c..6729a072ea 100644\n--- a/Documentation/config/uploadpack.txt\n+++ b/Documentation/config/uploadpack.txt\n@@ -59,7 +59,8 @@ uploadpack.allowFilter::\n \n uploadpackfilter.allow::\n \tProvides a default value for unspecified object filters (see: the\n-\tbelow configuration variable).\n+\tbelow configuration variable). If set to `true`, this will also\n+\tenable all filters which get added in the future.\n \tDefaults to `true`.\n \n uploadpackfilter..allow::\n-- \n2.31.1\n\n"},{"id":"422298","messageId":"ffe8e2bf733dcfbe3847e172f6a397a205531cc9.1618832437.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1618832437.git.ps@pks.im","subject":"[PATCH v5 2/8] revision: mark commit parents as NOT_USER_GIVEN","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-19T11:46:41Z","receivedAt":"2021-04-19T11:46:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The NOT_USER_GIVEN flag of an object marks whether a flag was explicitly\nprovided by the user or not. The most important use case for this is\nwhen filtering objects: only objects that were not explicitly requested\nwill get filtered.\n\nThe flag is currently only set for blobs and trees, which has been fine\ngiven that there are no filters for tags or commits currently. We're\nabout to extend filtering capabilities to add object type filter though,\nwhich requires us to set up the NOT_USER_GIVEN flag correctly -- if it's\nnot set, the object wouldn't get filtered at all.\n\nMark unseen commit parents as NOT_USER_GIVEN when processing parents.\nLike this, explicitly provided parents stay user-given and thus\nunfiltered, while parents which get loaded as part of the graph walk\ncan be filtered.\n\nThis commit shouldn't have any user-visible impact yet as there is no\nlogic to filter commits yet.\n\nSigned-off-by: Patrick Steinhardt \n---\n revision.c | 4 ++--\n revision.h | 3 ---\n 2 files changed, 2 insertions(+), 5 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex 553c0faa9b..fd34c75e23 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1123,7 +1123,7 @@ static int process_parents(struct rev_info *revs, struct commit *commit,\n \t\t\t\tmark_parents_uninteresting(p);\n \t\t\tif (p->object.flags & SEEN)\n \t\t\t\tcontinue;\n-\t\t\tp->object.flags |= SEEN;\n+\t\t\tp->object.flags |= (SEEN | NOT_USER_GIVEN);\n \t\t\tif (list)\n \t\t\t\tcommit_list_insert_by_date(p, list);\n \t\t\tif (queue)\n@@ -1165,7 +1165,7 @@ static int process_parents(struct rev_info *revs, struct commit *commit,\n \t\t}\n \t\tp->object.flags |= left_flag;\n \t\tif (!(p->object.flags & SEEN)) {\n-\t\t\tp->object.flags |= SEEN;\n+\t\t\tp->object.flags |= (SEEN | NOT_USER_GIVEN);\n \t\t\tif (list)\n \t\t\t\tcommit_list_insert_by_date(p, list);\n \t\t\tif (queue)\ndiff --git a/revision.h b/revision.h\nindex a24f72dcd1..93aa012f51 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -44,9 +44,6 @@\n /*\n * Indicates object was reached by traversal. i.e. not given by user on\n * command-line or stdin.\n- * NEEDSWORK: NOT_USER_GIVEN doesn't apply to commits because we only support\n- * filtering trees and blobs, but it may be useful to support filtering commits\n- * in the future.\n */\n #define NOT_USER_GIVEN\t(1u<<25)\n #define TRACK_LINEAR\t(1u<<26)\n-- \n2.31.1\n\n"},{"id":"422299","messageId":"21d7f06d0a94fc8b4f48f90007adb30fae18aa28.1618832437.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1618832437.git.ps@pks.im","subject":"[PATCH v5 3/8] list-objects: move tag processing into its own function","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-19T11:46:45Z","receivedAt":"2021-04-19T11:46:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Move processing of tags into its own function to make the logic easier\nto extend when we're going to implement filtering for tags. No change in\nbehaviour is expected from this commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n list-objects.c | 11 +++++++++--\n 1 file changed, 9 insertions(+), 2 deletions(-)\n\ndiff --git a/list-objects.c b/list-objects.c\nindex e19589baa0..a5a60301cb 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -213,6 +213,14 @@ static void process_tree(struct traversal_context *ctx,\n \tfree_tree_buffer(tree);\n }\n \n+static void process_tag(struct traversal_context *ctx,\n+\t\t\tstruct tag *tag,\n+\t\t\tconst char *name)\n+{\n+\ttag->object.flags |= SEEN;\n+\tctx->show_object(&tag->object, name, ctx->show_data);\n+}\n+\n static void mark_edge_parents_uninteresting(struct commit *commit,\n \t\t\t\t\t struct rev_info *revs,\n \t\t\t\t\t show_edge_fn show_edge)\n@@ -334,8 +342,7 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx,\n \t\tif (obj->flags & (UNINTERESTING | SEEN))\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n-\t\t\tobj->flags |= SEEN;\n-\t\t\tctx->show_object(obj, name, ctx->show_data);\n+\t\t\tprocess_tag(ctx, (struct tag *)obj, name);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!path)\n-- \n2.31.1\n\n"},{"id":"422300","messageId":"b5d6ab6b4ad9bf5bab4440977af5acb9f3e05a73.1618832437.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1618832437.git.ps@pks.im","subject":"[PATCH v5 4/8] list-objects: support filtering by tag and commit","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-19T11:46:49Z","receivedAt":"2021-04-19T11:46:58Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Object filters currently only support filtering blobs or trees based on\nsome criteria. This commit lays the foundation to also allow filtering\nof tags and commits.\n\nNo change in behaviour is expected from this commit given that there are\nno filters yet for those object types.\n\nSigned-off-by: Patrick Steinhardt \n---\n list-objects-filter.c | 40 ++++++++++++++++++++++++++++++++++++++++\n list-objects-filter.h | 2 ++\n list-objects.c | 23 ++++++++++++++++++++---\n 3 files changed, 62 insertions(+), 3 deletions(-)\n\ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex 39e2f15333..0ebfa52966 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -82,6 +82,16 @@ static enum list_objects_filter_result filter_blobs_none(\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n+\tcase LOFS_TAG:\n+\t\tassert(obj->type == OBJ_TAG);\n+\t\t/* always include all tag objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\tcase LOFS_COMMIT:\n+\t\tassert(obj->type == OBJ_COMMIT);\n+\t\t/* always include all commit objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\t/* always include all tree objects */\n@@ -173,6 +183,16 @@ static enum list_objects_filter_result filter_trees_depth(\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n+\tcase LOFS_TAG:\n+\t\tassert(obj->type == OBJ_TAG);\n+\t\t/* always include all tag objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\tcase LOFS_COMMIT:\n+\t\tassert(obj->type == OBJ_COMMIT);\n+\t\t/* always include all commit objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n \tcase LOFS_END_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\tfilter_data->current_depth--;\n@@ -267,6 +287,16 @@ static enum list_objects_filter_result filter_blobs_limit(\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n+\tcase LOFS_TAG:\n+\t\tassert(obj->type == OBJ_TAG);\n+\t\t/* always include all tag objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\tcase LOFS_COMMIT:\n+\t\tassert(obj->type == OBJ_COMMIT);\n+\t\t/* always include all commit objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\t/* always include all tree objects */\n@@ -371,6 +401,16 @@ static enum list_objects_filter_result filter_sparse(\n \tdefault:\n \t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n \n+\tcase LOFS_TAG:\n+\t\tassert(obj->type == OBJ_TAG);\n+\t\t/* always include all tag objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\tcase LOFS_COMMIT:\n+\t\tassert(obj->type == OBJ_COMMIT);\n+\t\t/* always include all commit objects */\n+\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n \tcase LOFS_BEGIN_TREE:\n \t\tassert(obj->type == OBJ_TREE);\n \t\tdtype = DT_DIR;\ndiff --git a/list-objects-filter.h b/list-objects-filter.h\nindex cfd784e203..9e98814111 100644\n--- a/list-objects-filter.h\n+++ b/list-objects-filter.h\n@@ -55,6 +55,8 @@ enum list_objects_filter_result {\n };\n \n enum list_objects_filter_situation {\n+\tLOFS_COMMIT,\n+\tLOFS_TAG,\n \tLOFS_BEGIN_TREE,\n \tLOFS_END_TREE,\n \tLOFS_BLOB\ndiff --git a/list-objects.c b/list-objects.c\nindex a5a60301cb..7f404677d5 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -217,8 +217,15 @@ static void process_tag(struct traversal_context *ctx,\n \t\t\tstruct tag *tag,\n \t\t\tconst char *name)\n {\n-\ttag->object.flags |= SEEN;\n-\tctx->show_object(&tag->object, name, ctx->show_data);\n+\tenum list_objects_filter_result r;\n+\n+\tr = list_objects_filter__filter_object(ctx->revs->repo, LOFS_TAG,\n+\t\t\t\t\t &tag->object, NULL, NULL,\n+\t\t\t\t\t ctx->filter);\n+\tif (r & LOFR_MARK_SEEN)\n+\t\ttag->object.flags |= SEEN;\n+\tif (r & LOFR_DO_SHOW)\n+\t\tctx->show_object(&tag->object, name, ctx->show_data);\n }\n \n static void mark_edge_parents_uninteresting(struct commit *commit,\n@@ -368,6 +375,12 @@ static void do_traverse(struct traversal_context *ctx)\n \tstrbuf_init(&csp, PATH_MAX);\n \n \twhile ((commit = get_revision(ctx->revs)) != NULL) {\n+\t\tenum list_objects_filter_result r;\n+\n+\t\tr = list_objects_filter__filter_object(ctx->revs->repo,\n+\t\t\t\tLOFS_COMMIT, &commit->object,\n+\t\t\t\tNULL, NULL, ctx->filter);\n+\n \t\t/*\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n@@ -382,7 +395,11 @@ static void do_traverse(struct traversal_context *ctx)\n \t\t\tdie(_(\"unable to load root tree for commit %s\"),\n \t\t\t oid_to_hex(&commit->object.oid));\n \t\t}\n-\t\tctx->show_commit(commit, ctx->show_data);\n+\n+\t\tif (r & LOFR_MARK_SEEN)\n+\t\t\tcommit->object.flags |= SEEN;\n+\t\tif (r & LOFR_DO_SHOW)\n+\t\t\tctx->show_commit(commit, ctx->show_data);\n \n \t\tif (ctx->revs->tree_blobs_in_commit_order)\n \t\t\t/*\n-- \n2.31.1\n\n"},{"id":"422301","messageId":"f462745290b4c8afe34a199aea07965092d156be.1618832437.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1618832437.git.ps@pks.im","subject":"[PATCH v5 5/8] list-objects: implement object type filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-19T11:46:53Z","receivedAt":"2021-04-19T11:47:19Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"While it already is possible to filter objects by some criteria in\ngit-rev-list(1), it is not yet possible to filter out only a specific\ntype of objects. This makes some filters less useful. The `blob:limit`\nfilter for example filters blobs such that only those which are smaller\nthan the given limit are returned. But it is unfit to ask only for these\nsmallish blobs, given that git-rev-list(1) will continue to print tags,\ncommits and trees.\n\nNow that we have the infrastructure in place to also filter tags and\ncommits, we can improve this situation by implementing a new filter\nwhich selects objects based on their type. Above query can thus\ntrivially be implemented with the following command:\n\n $ git rev-list --objects --filter=object:type=blob \\\n --filter=blob:limit=200\n\nFurthermore, this filter allows to optimize for certain other cases: if\nfor example only tags or commits have been selected, there is no need to\nwalk down trees.\n\nThe new filter is not yet supported in bitmaps. This is going to be\nimplemented in a subsequent commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/config/uploadpack.txt | 6 +--\n Documentation/rev-list-options.txt | 3 ++\n list-objects-filter-options.c | 15 ++++++\n list-objects-filter-options.h | 3 ++\n list-objects-filter.c | 76 +++++++++++++++++++++++++++++\n t/t6112-rev-list-filters-objects.sh | 44 +++++++++++++++++\n 6 files changed, 144 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/config/uploadpack.txt b/Documentation/config/uploadpack.txt\nindex 6729a072ea..32fad5bbe8 100644\n--- a/Documentation/config/uploadpack.txt\n+++ b/Documentation/config/uploadpack.txt\n@@ -66,9 +66,9 @@ uploadpackfilter.allow::\n uploadpackfilter..allow::\n \tExplicitly allow or ban the object filter corresponding to\n \t``, where `` may be one of: `blob:none`,\n-\t`blob:limit`, `tree`, `sparse:oid`, or `combine`. If using\n-\tcombined filters, both `combine` and all of the nested filter\n-\tkinds must be allowed. Defaults to `uploadpackfilter.allow`.\n+\t`blob:limit`, `object:type`, `tree`, `sparse:oid`, or `combine`.\n+\tIf using combined filters, both `combine` and all of the nested\n+\tfilter kinds must be allowed. Defaults to `uploadpackfilter.allow`.\n \n uploadpackfilter.tree.maxDepth::\n \tOnly allow `--filter=tree:` when `` is no more than the value of\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex b1c8f86c6e..3afa8fffbd 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -892,6 +892,9 @@ or units. n may be zero. The suffixes k, m, and g can be used to name\n units in KiB, MiB, or GiB. For example, 'blob:limit=1k' is the same\n as 'blob:limit=1024'.\n +\n+The form '--filter=object:type=(tag|commit|tree|blob)' omits all objects\n+which are not of the requested type.\n++\n The form '--filter=sparse:oid=' uses a sparse-checkout\n specification contained in the blob (or blob-expression) ''\n to omit blobs that would not be not required for a sparse checkout on\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex d2d1c81caf..96a605c8ad 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -29,6 +29,8 @@ const char *list_object_filter_config_name(enum list_objects_filter_choice c)\n \t\treturn \"tree\";\n \tcase LOFC_SPARSE_OID:\n \t\treturn \"sparse:oid\";\n+\tcase LOFC_OBJECT_TYPE:\n+\t\treturn \"object:type\";\n \tcase LOFC_COMBINE:\n \t\treturn \"combine\";\n \tcase LOFC__COUNT:\n@@ -97,6 +99,19 @@ static int gently_parse_list_objects_filter(\n \t\t}\n \t\treturn 1;\n \n+\t} else if (skip_prefix(arg, \"object:type=\", &v0)) {\n+\t\tint type = type_from_string_gently(v0, strlen(v0), 1);\n+\t\tif (type < 0) {\n+\t\t\tstrbuf_addf(errbuf, _(\"'%s' for 'object:type=' is\"\n+\t\t\t\t\t \"not a valid object type\"), v0);\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\tfilter_options->object_type = type;\n+\t\tfilter_options->choice = LOFC_OBJECT_TYPE;\n+\n+\t\treturn 0;\n+\n \t} else if (skip_prefix(arg, \"combine:\", &v0)) {\n \t\treturn parse_combine_filter(filter_options, v0, errbuf);\n \ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 01767c3c96..da5b6737e2 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -1,6 +1,7 @@\n #ifndef LIST_OBJECTS_FILTER_OPTIONS_H\n #define LIST_OBJECTS_FILTER_OPTIONS_H\n \n+#include \"cache.h\"\n #include \"parse-options.h\"\n #include \"string-list.h\"\n \n@@ -13,6 +14,7 @@ enum list_objects_filter_choice {\n \tLOFC_BLOB_LIMIT,\n \tLOFC_TREE_DEPTH,\n \tLOFC_SPARSE_OID,\n+\tLOFC_OBJECT_TYPE,\n \tLOFC_COMBINE,\n \tLOFC__COUNT /* must be last */\n };\n@@ -54,6 +56,7 @@ struct list_objects_filter_options {\n \tchar *sparse_oid_name;\n \tunsigned long blob_limit_value;\n \tunsigned long tree_exclude_depth;\n+\tenum object_type object_type;\n \n \t/* LOFC_COMBINE values */\n \ndiff --git a/list-objects-filter.c b/list-objects-filter.c\nindex 0ebfa52966..1c1ee3d1bb 100644\n--- a/list-objects-filter.c\n+++ b/list-objects-filter.c\n@@ -545,6 +545,81 @@ static void filter_sparse_oid__init(\n \tfilter->free_fn = filter_sparse_free;\n }\n \n+/*\n+ * A filter for list-objects to omit large blobs.\n+ * And to OPTIONALLY collect a list of the omitted OIDs.\n+ */\n+struct filter_object_type_data {\n+\tenum object_type object_type;\n+};\n+\n+static enum list_objects_filter_result filter_object_type(\n+\tstruct repository *r,\n+\tenum list_objects_filter_situation filter_situation,\n+\tstruct object *obj,\n+\tconst char *pathname,\n+\tconst char *filename,\n+\tstruct oidset *omits,\n+\tvoid *filter_data_)\n+{\n+\tstruct filter_object_type_data *filter_data = filter_data_;\n+\n+\tswitch (filter_situation) {\n+\tdefault:\n+\t\tBUG(\"unknown filter_situation: %d\", filter_situation);\n+\n+\tcase LOFS_TAG:\n+\t\tassert(obj->type == OBJ_TAG);\n+\t\tif (filter_data->object_type == OBJ_TAG)\n+\t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\t\treturn LOFR_MARK_SEEN;\n+\n+\tcase LOFS_COMMIT:\n+\t\tassert(obj->type == OBJ_COMMIT);\n+\t\tif (filter_data->object_type == OBJ_COMMIT)\n+\t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\t\treturn LOFR_MARK_SEEN;\n+\n+\tcase LOFS_BEGIN_TREE:\n+\t\tassert(obj->type == OBJ_TREE);\n+\n+\t\t/*\n+\t\t * If we only want to show commits or tags, then there is no\n+\t\t * need to walk down trees.\n+\t\t */\n+\t\tif (filter_data->object_type == OBJ_COMMIT ||\n+\t\t filter_data->object_type == OBJ_TAG)\n+\t\t\treturn LOFR_SKIP_TREE;\n+\n+\t\tif (filter_data->object_type == OBJ_TREE)\n+\t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\n+\t\treturn LOFR_MARK_SEEN;\n+\n+\tcase LOFS_BLOB:\n+\t\tassert(obj->type == OBJ_BLOB);\n+\n+\t\tif (filter_data->object_type == OBJ_BLOB)\n+\t\t\treturn LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\t\treturn LOFR_MARK_SEEN;\n+\n+\tcase LOFS_END_TREE:\n+\t\treturn LOFR_ZERO;\n+\t}\n+}\n+\n+static void filter_object_type__init(\n+\tstruct list_objects_filter_options *filter_options,\n+\tstruct filter *filter)\n+{\n+\tstruct filter_object_type_data *d = xcalloc(1, sizeof(*d));\n+\td->object_type = filter_options->object_type;\n+\n+\tfilter->filter_data = d;\n+\tfilter->filter_object_fn = filter_object_type;\n+\tfilter->free_fn = free;\n+}\n+\n /* A filter which only shows objects shown by all sub-filters. */\n struct combine_filter_data {\n \tstruct subfilter *sub;\n@@ -691,6 +766,7 @@ static filter_init_fn s_filters[] = {\n \tfilter_blobs_limit__init,\n \tfilter_trees_depth__init,\n \tfilter_sparse_oid__init,\n+\tfilter_object_type__init,\n \tfilter_combine__init,\n };\n \ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex 31457d13b9..c810a62807 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -159,6 +159,50 @@ test_expect_success 'verify blob:limit=1m' '\n \ttest_must_be_empty observed\n '\n \n+# Test object:type= filter.\n+\n+test_expect_success 'setup object-type' '\n+\ttest_create_repo object-type &&\n+\ttest_commit --no-tag -C object-type message blob &&\n+\tgit -C object-type tag tag -m tag-message\n+'\n+\n+test_expect_success 'verify object:type= fails with invalid type' '\n+\ttest_must_fail git -C object-type rev-list --objects --filter=object:type= HEAD &&\n+\ttest_must_fail git -C object-type rev-list --objects --filter=object:type=invalid HEAD\n+'\n+\n+test_expect_success 'verify object:type=blob prints blob and commit' '\n+\tgit -C object-type rev-parse HEAD >expected &&\n+\tprintf \"%s blob\\n\" $(git -C object-type rev-parse HEAD:blob) >>expected &&\n+\tgit -C object-type rev-list --objects --filter=object:type=blob HEAD >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=tree prints tree and commit' '\n+\t(\n+\t\tgit -C object-type rev-parse HEAD &&\n+\t\tprintf \"%s \\n\" $(git -C object-type rev-parse HEAD^{tree})\n+\t) >expected &&\n+\tgit -C object-type rev-list --objects --filter=object:type=tree HEAD >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=commit prints commit' '\n+\tgit -C object-type rev-parse HEAD >expected &&\n+\tgit -C object-type rev-list --objects --filter=object:type=commit HEAD >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=tag prints tag' '\n+\t(\n+\t\tgit -C object-type rev-parse HEAD &&\n+\t\tprintf \"%s tag\\n\" $(git -C object-type rev-parse tag)\n+\t) >expected &&\n+\tgit -C object-type rev-list --objects --filter=object:type=tag tag >actual &&\n+\ttest_cmp expected actual\n+'\n+\n # Test sparse:path= filter.\n # !!!!\n # NOTE: sparse:path filter support has been dropped for security reasons,\n-- \n2.31.1\n\n"},{"id":"422302","messageId":"9726e698617fa6a5a4d41283400e6a72e08baefb.1618832437.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1618832437.git.ps@pks.im","subject":"[PATCH v5 7/8] pack-bitmap: implement combined filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-19T11:47:02Z","receivedAt":"2021-04-19T11:47:21Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When the user has multiple objects filters specified, then this is\ninternally represented by having a \"combined\" filter. These combined\nfilters aren't yet supported by bitmap indices and can thus not be\naccelerated.\n\nFix this by implementing support for these combined filters. The\nimplementation is quite trivial: when there's a combined filter, we\nsimply recurse into `filter_bitmap()` for all of the sub-filters.\n\nSigned-off-by: Patrick Steinhardt \n---\n pack-bitmap.c | 10 ++++++++++\n t/t6113-rev-list-bitmap-filters.sh | 7 +++++++\n 2 files changed, 17 insertions(+)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex e419f4206b..53f35d41c6 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -967,6 +967,16 @@ static int filter_bitmap(struct bitmap_index *bitmap_git,\n \t\treturn 0;\n \t}\n \n+\tif (filter->choice == LOFC_COMBINE) {\n+\t\tint i;\n+\t\tfor (i = 0; i < filter->sub_nr; i++) {\n+\t\t\tif (filter_bitmap(bitmap_git, tip_objects, to_filter,\n+\t\t\t\t\t &filter->sub[i]) < 0)\n+\t\t\t\treturn -1;\n+\t\t}\n+\t\treturn 0;\n+\t}\n+\n \t/* filter choice not handled */\n \treturn -1;\n }\ndiff --git a/t/t6113-rev-list-bitmap-filters.sh b/t/t6113-rev-list-bitmap-filters.sh\nindex fb66735ac8..cb9db7df6f 100755\n--- a/t/t6113-rev-list-bitmap-filters.sh\n+++ b/t/t6113-rev-list-bitmap-filters.sh\n@@ -98,4 +98,11 @@ test_expect_success 'object:type filter' '\n \ttest_bitmap_traversal expect actual\n '\n \n+test_expect_success 'combine filter' '\n+\tgit rev-list --objects --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=blob:limit=1000 --filter=object:type=blob tag >actual &&\n+\ttest_bitmap_traversal expect actual\n+'\n+\n test_done\n-- \n2.31.1\n\n"},{"id":"422303","messageId":"90c80c1efad7974d5f1d87970c0c7918c06808b5.1618832437.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1618832437.git.ps@pks.im","subject":"[PATCH v5 6/8] pack-bitmap: implement object type filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-19T11:46:58Z","receivedAt":"2021-04-19T11:47:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The preceding commit has added a new object filter for git-rev-list(1)\nwhich allows to filter objects by type. Implement the equivalent filter\nfor packfile bitmaps so that we can answer these queries fast.\n\nSigned-off-by: Patrick Steinhardt \n---\n pack-bitmap.c | 29 ++++++++++++++++++++++++++---\n t/t6113-rev-list-bitmap-filters.sh | 25 ++++++++++++++++++++++++-\n 2 files changed, 50 insertions(+), 4 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 3ed15431cd..e419f4206b 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -780,9 +780,6 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git,\n \teword_t mask;\n \tuint32_t i;\n \n-\tif (type != OBJ_BLOB && type != OBJ_TREE)\n-\t\tBUG(\"filter_bitmap_exclude_type: unsupported type '%d'\", type);\n-\n \t/*\n \t * The non-bitmap version of this filter never removes\n \t * objects which the other side specifically asked for,\n@@ -912,6 +909,24 @@ static void filter_bitmap_tree_depth(struct bitmap_index *bitmap_git,\n \t\t\t\t OBJ_BLOB);\n }\n \n+static void filter_bitmap_object_type(struct bitmap_index *bitmap_git,\n+\t\t\t\t struct object_list *tip_objects,\n+\t\t\t\t struct bitmap *to_filter,\n+\t\t\t\t enum object_type object_type)\n+{\n+\tif (object_type < OBJ_COMMIT || object_type > OBJ_TAG)\n+\t\tBUG(\"filter_bitmap_object_type given invalid object\");\n+\n+\tif (object_type != OBJ_TAG)\n+\t\tfilter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, OBJ_TAG);\n+\tif (object_type != OBJ_COMMIT)\n+\t\tfilter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, OBJ_COMMIT);\n+\tif (object_type != OBJ_TREE)\n+\t\tfilter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, OBJ_TREE);\n+\tif (object_type != OBJ_BLOB)\n+\t\tfilter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, OBJ_BLOB);\n+}\n+\n static int filter_bitmap(struct bitmap_index *bitmap_git,\n \t\t\t struct object_list *tip_objects,\n \t\t\t struct bitmap *to_filter,\n@@ -944,6 +959,14 @@ static int filter_bitmap(struct bitmap_index *bitmap_git,\n \t\treturn 0;\n \t}\n \n+\tif (filter->choice == LOFC_OBJECT_TYPE) {\n+\t\tif (bitmap_git)\n+\t\t\tfilter_bitmap_object_type(bitmap_git, tip_objects,\n+\t\t\t\t\t\t to_filter,\n+\t\t\t\t\t\t filter->object_type);\n+\t\treturn 0;\n+\t}\n+\n \t/* filter choice not handled */\n \treturn -1;\n }\ndiff --git a/t/t6113-rev-list-bitmap-filters.sh b/t/t6113-rev-list-bitmap-filters.sh\nindex 3f889949ca..fb66735ac8 100755\n--- a/t/t6113-rev-list-bitmap-filters.sh\n+++ b/t/t6113-rev-list-bitmap-filters.sh\n@@ -10,7 +10,8 @@ test_expect_success 'set up bitmapped repo' '\n \ttest_commit much-larger-blob-one &&\n \tgit repack -adb &&\n \ttest_commit two &&\n-\ttest_commit much-larger-blob-two\n+\ttest_commit much-larger-blob-two &&\n+\tgit tag tag\n '\n \n test_expect_success 'filters fallback to non-bitmap traversal' '\n@@ -75,4 +76,26 @@ test_expect_success 'tree:1 filter' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'object:type filter' '\n+\tgit rev-list --objects --filter=object:type=tag tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=object:type=tag tag >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tgit rev-list --objects --filter=object:type=commit tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=object:type=commit tag >actual &&\n+\ttest_bitmap_traversal expect actual &&\n+\n+\tgit rev-list --objects --filter=object:type=tree tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=object:type=tree tag >actual &&\n+\ttest_bitmap_traversal expect actual &&\n+\n+\tgit rev-list --objects --filter=object:type=blob tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter=object:type=blob tag >actual &&\n+\ttest_bitmap_traversal expect actual\n+'\n+\n test_done\n-- \n2.31.1\n\n"},{"id":"422304","messageId":"fce3551e48f2d66518ee9d3d878d393988393b35.1618832437.git.ps@pks.im","threadId":"55236","inReplyTo":"cover.1618832437.git.ps@pks.im","subject":"[PATCH v5 8/8] rev-list: allow filtering of provided items","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-04-19T11:47:06Z","receivedAt":"2021-04-19T11:48:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When providing an object filter, it is currently impossible to also\nfilter provided items. E.g. when executing `git rev-list HEAD` , the\ncommit this reference points to will be treated as user-provided and is\nthus excluded from the filtering mechanism. This makes it harder than\nnecessary to properly use the new `--filter=object:type` filter given\nthat even if the user wants to only see blobs, he'll still see commits\nof provided references.\n\nImprove this by introducing a new `--filter-provided-objects` option\nto the git-rev-parse(1) command. If given, then all user-provided\nreferences will be subject to filtering.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/rev-list-options.txt | 5 ++++\n builtin/pack-objects.c | 2 +-\n builtin/rev-list.c | 36 +++++++++++++++++++++--------\n pack-bitmap.c | 6 +++--\n pack-bitmap.h | 3 ++-\n reachable.c | 2 +-\n t/t6112-rev-list-filters-objects.sh | 28 ++++++++++++++++++++++\n t/t6113-rev-list-bitmap-filters.sh | 36 +++++++++++++++++++++++++++++\n 8 files changed, 104 insertions(+), 14 deletions(-)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 3afa8fffbd..5bf2a85f69 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -933,6 +933,11 @@ equivalent.\n --no-filter::\n \tTurn off any previous `--filter=` argument.\n \n+--filter-provided-objects::\n+\tFilter the list of explicitly provided objects, which would otherwise\n+\talways be printed even if they did not match any of the filters. Only\n+\tuseful with `--filter=`.\n+\n --filter-print-omitted::\n \tOnly useful with `--filter=`; prints a list of the objects omitted\n \tby the filter. Object IDs are prefixed with a ``~'' character.\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex a1e33d7507..c89e996f0d 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3516,7 +3516,7 @@ static int pack_options_allow_reuse(void)\n \n static int get_object_list_from_bitmap(struct rev_info *revs)\n {\n-\tif (!(bitmap_git = prepare_bitmap_walk(revs, &filter_options)))\n+\tif (!(bitmap_git = prepare_bitmap_walk(revs, &filter_options, 0)))\n \t\treturn -1;\n \n \tif (pack_options_allow_reuse() &&\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex b4d8ea0a35..7677b1af5a 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -398,7 +398,8 @@ static inline int parse_missing_action_value(const char *value)\n }\n \n static int try_bitmap_count(struct rev_info *revs,\n-\t\t\t struct list_objects_filter_options *filter)\n+\t\t\t struct list_objects_filter_options *filter,\n+\t\t\t int filter_provided_objects)\n {\n \tuint32_t commit_count = 0,\n \t\t tag_count = 0,\n@@ -433,7 +434,7 @@ static int try_bitmap_count(struct rev_info *revs,\n \t */\n \tmax_count = revs->max_count;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, filter);\n+\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -450,7 +451,8 @@ static int try_bitmap_count(struct rev_info *revs,\n }\n \n static int try_bitmap_traversal(struct rev_info *revs,\n-\t\t\t\tstruct list_objects_filter_options *filter)\n+\t\t\t\tstruct list_objects_filter_options *filter,\n+\t\t\t\tint filter_provided_objects)\n {\n \tstruct bitmap_index *bitmap_git;\n \n@@ -461,7 +463,7 @@ static int try_bitmap_traversal(struct rev_info *revs,\n \tif (revs->max_count >= 0)\n \t\treturn -1;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, filter);\n+\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -471,14 +473,15 @@ static int try_bitmap_traversal(struct rev_info *revs,\n }\n \n static int try_bitmap_disk_usage(struct rev_info *revs,\n-\t\t\t\t struct list_objects_filter_options *filter)\n+\t\t\t\t struct list_objects_filter_options *filter,\n+\t\t\t\t int filter_provided_objects)\n {\n \tstruct bitmap_index *bitmap_git;\n \n \tif (!show_disk_usage)\n \t\treturn -1;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, filter);\n+\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -499,6 +502,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \tint bisect_show_vars = 0;\n \tint bisect_find_all = 0;\n \tint use_bitmap_index = 0;\n+\tint filter_provided_objects = 0;\n \tconst char *show_progress = NULL;\n \n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n@@ -599,6 +603,10 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\tlist_objects_filter_set_no_filter(&filter_options);\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!strcmp(arg, \"--filter-provided-objects\")) {\n+\t\t\tfilter_provided_objects = 1;\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!strcmp(arg, \"--filter-print-omitted\")) {\n \t\t\targ_print_omitted = 1;\n \t\t\tcontinue;\n@@ -665,11 +673,11 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\tprogress = start_delayed_progress(show_progress, 0);\n \n \tif (use_bitmap_index) {\n-\t\tif (!try_bitmap_count(&revs, &filter_options))\n+\t\tif (!try_bitmap_count(&revs, &filter_options, filter_provided_objects))\n \t\t\treturn 0;\n-\t\tif (!try_bitmap_disk_usage(&revs, &filter_options))\n+\t\tif (!try_bitmap_disk_usage(&revs, &filter_options, filter_provided_objects))\n \t\t\treturn 0;\n-\t\tif (!try_bitmap_traversal(&revs, &filter_options))\n+\t\tif (!try_bitmap_traversal(&revs, &filter_options, filter_provided_objects))\n \t\t\treturn 0;\n \t}\n \n@@ -694,6 +702,16 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\treturn show_bisect_vars(&info, reaches, all);\n \t}\n \n+\tif (filter_provided_objects) {\n+\t\tstruct commit_list *c;\n+\t\tfor (i = 0; i < revs.pending.nr; i++) {\n+\t\t\tstruct object_array_entry *pending = revs.pending.objects + i;\n+\t\t\tpending->item->flags |= NOT_USER_GIVEN;\n+\t\t}\n+\t\tfor (c = revs.commits; c; c = c->next)\n+\t\t\tc->item->object.flags |= NOT_USER_GIVEN;\n+\t}\n+\n \tif (arg_print_omitted)\n \t\toidset_init(&omitted_objects, DEFAULT_OIDSET_SIZE);\n \tif (arg_missing_action == MA_PRINT)\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 53f35d41c6..6efea70ae7 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -987,7 +987,8 @@ static int can_filter_bitmap(struct list_objects_filter_options *filter)\n }\n \n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n-\t\t\t\t\t struct list_objects_filter_options *filter)\n+\t\t\t\t\t struct list_objects_filter_options *filter,\n+\t\t\t\t\t int filter_provided_objects)\n {\n \tunsigned int i;\n \n@@ -1082,7 +1083,8 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \tif (haves_bitmap)\n \t\tbitmap_and_not(wants_bitmap, haves_bitmap);\n \n-\tfilter_bitmap(bitmap_git, wants, wants_bitmap, filter);\n+\tfilter_bitmap(bitmap_git, (filter && filter_provided_objects) ? NULL : wants,\n+\t\t wants_bitmap, filter);\n \n \tbitmap_git->result = wants_bitmap;\n \tbitmap_git->haves = haves_bitmap;\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 78f2b3ff79..99d733eb26 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -52,7 +52,8 @@ void traverse_bitmap_commit_list(struct bitmap_index *,\n void test_bitmap_walk(struct rev_info *revs);\n int test_bitmap_commits(struct repository *r);\n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n-\t\t\t\t\t struct list_objects_filter_options *filter);\n+\t\t\t\t\t struct list_objects_filter_options *filter,\n+\t\t\t\t\t int filter_provided_objects);\n int reuse_partial_packfile_from_bitmap(struct bitmap_index *,\n \t\t\t\t struct packed_git **packfile,\n \t\t\t\t uint32_t *entries,\ndiff --git a/reachable.c b/reachable.c\nindex 77a60c70a5..fc833cae43 100644\n--- a/reachable.c\n+++ b/reachable.c\n@@ -223,7 +223,7 @@ void mark_reachable_objects(struct rev_info *revs, int mark_reflog,\n \tcp.progress = progress;\n \tcp.count = 0;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, NULL);\n+\tbitmap_git = prepare_bitmap_walk(revs, NULL, 0);\n \tif (bitmap_git) {\n \t\ttraverse_bitmap_commit_list(bitmap_git, revs, mark_object_seen);\n \t\tfree_bitmap_index(bitmap_git);\ndiff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh\nindex c810a62807..4ade105db3 100755\n--- a/t/t6112-rev-list-filters-objects.sh\n+++ b/t/t6112-rev-list-filters-objects.sh\n@@ -203,6 +203,34 @@ test_expect_success 'verify object:type=tag prints tag' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'verify object:type=blob prints only blob with --filter-provided-objects' '\n+\tprintf \"%s blob\\n\" $(git -C object-type rev-parse HEAD:blob) >expected &&\n+\tgit -C object-type rev-list --objects \\\n+\t\t--filter=object:type=blob --filter-provided-objects HEAD >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=tree prints only tree with --filter-provided-objects' '\n+\tprintf \"%s \\n\" $(git -C object-type rev-parse HEAD^{tree}) >expected &&\n+\tgit -C object-type rev-list --objects \\\n+\t\t--filter=object:type=tree HEAD --filter-provided-objects >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=commit prints only commit with --filter-provided-objects' '\n+\tgit -C object-type rev-parse HEAD >expected &&\n+\tgit -C object-type rev-list --objects \\\n+\t\t--filter=object:type=commit --filter-provided-objects HEAD >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify object:type=tag prints only tag with --filter-provided-objects' '\n+\tprintf \"%s tag\\n\" $(git -C object-type rev-parse tag) >expected &&\n+\tgit -C object-type rev-list --objects \\\n+\t\t--filter=object:type=tag --filter-provided-objects tag >actual &&\n+\ttest_cmp expected actual\n+'\n+\n # Test sparse:path= filter.\n # !!!!\n # NOTE: sparse:path filter support has been dropped for security reasons,\ndiff --git a/t/t6113-rev-list-bitmap-filters.sh b/t/t6113-rev-list-bitmap-filters.sh\nindex cb9db7df6f..4d8e09167e 100755\n--- a/t/t6113-rev-list-bitmap-filters.sh\n+++ b/t/t6113-rev-list-bitmap-filters.sh\n@@ -98,6 +98,28 @@ test_expect_success 'object:type filter' '\n \ttest_bitmap_traversal expect actual\n '\n \n+test_expect_success 'object:type filter with --filter-provided-objects' '\n+\tgit rev-list --objects --filter-provided-objects --filter=object:type=tag tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter-provided-objects --filter=object:type=tag tag >actual &&\n+\ttest_cmp expect actual &&\n+\n+\tgit rev-list --objects --filter-provided-objects --filter=object:type=commit tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter-provided-objects --filter=object:type=commit tag >actual &&\n+\ttest_bitmap_traversal expect actual &&\n+\n+\tgit rev-list --objects --filter-provided-objects --filter=object:type=tree tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter-provided-objects --filter=object:type=tree tag >actual &&\n+\ttest_bitmap_traversal expect actual &&\n+\n+\tgit rev-list --objects --filter-provided-objects --filter=object:type=blob tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter-provided-objects --filter=object:type=blob tag >actual &&\n+\ttest_bitmap_traversal expect actual\n+'\n+\n test_expect_success 'combine filter' '\n \tgit rev-list --objects --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&\n \tgit rev-list --use-bitmap-index \\\n@@ -105,4 +127,18 @@ test_expect_success 'combine filter' '\n \ttest_bitmap_traversal expect actual\n '\n \n+test_expect_success 'combine filter with --filter-provided-objects' '\n+\tgit rev-list --objects --filter-provided-objects --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&\n+\tgit rev-list --use-bitmap-index \\\n+\t\t --objects --filter-provided-objects --filter=blob:limit=1000 --filter=object:type=blob tag >actual &&\n+\ttest_bitmap_traversal expect actual &&\n+\n+\tgit cat-file --batch-check=\"%(objecttype) %(objectsize)\" objects &&\n+\twhile read objecttype objectsize\n+\tdo\n+\t\ttest \"$objecttype\" = blob || return 1\n+\t\ttest \"$objectsize\" -le 1000 || return 1\n+\tdone writes:\n\n> Hi,\n>\n> this is the fifth version of my patch series which implements a new\n> `object:type` filter for git-rev-list(1) and git-upload-pack(1) and\n> extends support for bitmap indices to work with combined filters.\n>\n> Changes compared to v4:\n>\n> - I'm now explicitly passing `strlen(v0)` to\n> `type_from_string_gently()` to be prepared for the future\n> semantics change here.\n>\n> - The error message printed in case the user passes in invalid\n> object type to `--filter=object:type=` now prints the wrong\n> value passed by the user.\n>\n> - Fixed missing header in list-objects-filter-options.h.\n>\n> - Tests now use test_create_repo and test_commit.\n>\n> - Removed a needless subshell in the tests.\n>\n> I hope that this catches all feedback and that I didn't misunderstand or\n> miss something. If I did, please give me a shout!\n\nThanks. Will queue and wait.\n\nI am expecting to go offline for about a week starting 21st or so;\nhopefully the topyc is ready to merge it to 'next' by the time I\ncome back online.\n\n"},{"id":"422735","messageId":"YIKPwRFbgzKJ1EBg@coredump.intra.peff.net","threadId":"55236","inReplyTo":"xmqq1rb5985x.fsf@gitster.g","subject":"Re: [PATCH v5 0/8] rev-list: implement object type filter","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-23T09:13:37Z","receivedAt":"2021-04-23T09:13:40Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Apr 19, 2021 at 04:16:58PM -0700, Junio C Hamano wrote:\n\n> Patrick Steinhardt writes:\n> \n> > Hi,\n> >\n> > this is the fifth version of my patch series which implements a new\n> > `object:type` filter for git-rev-list(1) and git-upload-pack(1) and\n> > extends support for bitmap indices to work with combined filters.\n> >\n> > Changes compared to v4:\n> >\n> > - I'm now explicitly passing `strlen(v0)` to\n> > `type_from_string_gently()` to be prepared for the future\n> > semantics change here.\n> >\n> > - The error message printed in case the user passes in invalid\n> > object type to `--filter=object:type=` now prints the wrong\n> > value passed by the user.\n> >\n> > - Fixed missing header in list-objects-filter-options.h.\n> >\n> > - Tests now use test_create_repo and test_commit.\n> >\n> > - Removed a needless subshell in the tests.\n> >\n> > I hope that this catches all feedback and that I didn't misunderstand or\n> > miss something. If I did, please give me a shout!\n> \n> Thanks. Will queue and wait.\n> \n> I am expecting to go offline for about a week starting 21st or so;\n> hopefully the topyc is ready to merge it to 'next' by the time I\n> come back online.\n\nThis iteration looks good to me.\n\n-Peff\n"},{"id":"423165","messageId":"xmqqwnsncfsr.fsf@gitster.g","threadId":"55236","inReplyTo":"YIKPwRFbgzKJ1EBg@coredump.intra.peff.net","subject":"Re: [PATCH v5 0/8] rev-list: implement object type filter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-28T02:18:44Z","receivedAt":"2021-04-28T02:18:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n>> Thanks. Will queue and wait.\n>> \n>> I am expecting to go offline for about a week starting 21st or so;\n>> hopefully the topyc is ready to merge it to 'next' by the time I\n>> come back online.\n>\n> This iteration looks good to me.\n\nThanks, both.\n"}]}