{"thread":{"id":"62984","subject":"[PATCH 3/9] builtin/cat-file: support \"blob:none\" objects filter","startedAt":"2025-02-21T07:47:37Z","lastAt":"2025-04-08T00:32:15Z","messageCount":44,"participants":["Patrick Steinhardt","Junio C Hamano","Toon Claes","Karthik Nayak","Taylor Blau"],"isPatch":true,"patchVersion":1,"patchTotal":9},"messages":[{"id":"512794","messageId":"20250221-pks-cat-file-object-type-filter-v1-1-0852530888e2@pks.im","threadId":"62984","inReplyTo":"20250221-pks-cat-file-object-type-filter-v1-0-0852530888e2@pks.im","subject":"[PATCH 1/9] builtin/cat-file: rename variable that tracks usage","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-21T07:47:26Z","receivedAt":"2025-02-21T07:47:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The usage strings for git-cat-file(1) that we pass to `parse_options()`\nand `usage_msg_optf()` are stored in a variable called `usage`. This\nvariable shadows the declaration of `usage()`, which we'll want to use\nin a subsequent commit.\n\nRename the variable to `builtin_catfile_usage`, which is in line with\nhow the variable is typically called in other builtins.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/cat-file.c | 32 ++++++++++++++++----------------\n 1 file changed, 16 insertions(+), 16 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex b13561cf73b..8e40016dd24 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -941,7 +941,7 @@ int cmd_cat_file(int argc,\n \tint input_nul_terminated = 0;\n \tint nul_terminated = 0;\n \n-\tconst char * const usage[] = {\n+\tconst char * const builtin_catfile_usage[] = {\n \t\tN_(\"git cat-file <type> <object>\"),\n \t\tN_(\"git cat-file (-e | -p) <object>\"),\n \t\tN_(\"git cat-file (-t | -s) [--allow-unknown-type] <object>\"),\n@@ -1007,7 +1007,7 @@ int cmd_cat_file(int argc,\n \n \tbatch.buffer_output = -1;\n \n-\targc = parse_options(argc, argv, prefix, options, usage, 0);\n+\targc = parse_options(argc, argv, prefix, options, builtin_catfile_usage, 0);\n \topt_cw = (opt == 'c' || opt == 'w');\n \topt_epts = (opt == 'e' || opt == 'p' || opt == 't' || opt == 's');\n \n@@ -1021,7 +1021,7 @@ int cmd_cat_file(int argc,\n \t/* Option compatibility */\n \tif (force_path && !opt_cw)\n \t\tusage_msg_optf(_(\"'%s=<%s>' needs '%s' or '%s'\"),\n-\t\t\t       usage, options,\n+\t\t\t       builtin_catfile_usage, options,\n \t\t\t       \"--path\", _(\"path|tree-ish\"), \"--filters\",\n \t\t\t       \"--textconv\");\n \n@@ -1029,19 +1029,19 @@ int cmd_cat_file(int argc,\n \tif (batch.enabled)\n \t\t;\n \telse if (batch.follow_symlinks)\n-\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), usage, options,\n+\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage, options,\n \t\t\t       \"--follow-symlinks\");\n \telse if (batch.buffer_output >= 0)\n-\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), usage, options,\n+\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage, options,\n \t\t\t       \"--buffer\");\n \telse if (batch.all_objects)\n-\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), usage, options,\n+\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage, options,\n \t\t\t       \"--batch-all-objects\");\n \telse if (input_nul_terminated)\n-\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), usage, options,\n+\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage, options,\n \t\t\t       \"-z\");\n \telse if (nul_terminated)\n-\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), usage, options,\n+\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage, options,\n \t\t\t       \"-Z\");\n \n \tbatch.input_delim = batch.output_delim = '\\n';\n@@ -1063,9 +1063,9 @@ int cmd_cat_file(int argc,\n \t\t\tbatch.transform_mode = opt;\n \t\telse if (opt && opt != 'b')\n \t\t\tusage_msg_optf(_(\"'-%c' is incompatible with batch mode\"),\n-\t\t\t\t       usage, options, opt);\n+\t\t\t\t       builtin_catfile_usage, options, opt);\n \t\telse if (argc)\n-\t\t\tusage_msg_opt(_(\"batch modes take no arguments\"), usage,\n+\t\t\tusage_msg_opt(_(\"batch modes take no arguments\"), builtin_catfile_usage,\n \t\t\t\t      options);\n \n \t\treturn batch_objects(&batch);\n@@ -1074,22 +1074,22 @@ int cmd_cat_file(int argc,\n \tif (opt) {\n \t\tif (!argc && opt == 'c')\n \t\t\tusage_msg_optf(_(\"<rev> required with '%s'\"),\n-\t\t\t\t       usage, options, \"--textconv\");\n+\t\t\t\t       builtin_catfile_usage, options, \"--textconv\");\n \t\telse if (!argc && opt == 'w')\n \t\t\tusage_msg_optf(_(\"<rev> required with '%s'\"),\n-\t\t\t\t       usage, options, \"--filters\");\n+\t\t\t\t       builtin_catfile_usage, options, \"--filters\");\n \t\telse if (!argc && opt_epts)\n \t\t\tusage_msg_optf(_(\"<object> required with '-%c'\"),\n-\t\t\t\t       usage, options, opt);\n+\t\t\t\t       builtin_catfile_usage, options, opt);\n \t\telse if (argc == 1)\n \t\t\tobj_name = argv[0];\n \t\telse\n-\t\t\tusage_msg_opt(_(\"too many arguments\"), usage, options);\n+\t\t\tusage_msg_opt(_(\"too many arguments\"), builtin_catfile_usage, options);\n \t} else if (!argc) {\n-\t\tusage_with_options(usage, options);\n+\t\tusage_with_options(builtin_catfile_usage, options);\n \t} else if (argc != 2) {\n \t\tusage_msg_optf(_(\"only two arguments allowed in <type> <object> mode, not %d\"),\n-\t\t\t      usage, options, argc);\n+\t\t\t      builtin_catfile_usage, options, argc);\n \t} else if (argc) {\n \t\texp_type = argv[0];\n \t\tobj_name = argv[1];\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512797","messageId":"20250221-pks-cat-file-object-type-filter-v1-0-0852530888e2@pks.im","threadId":"62984","inReplyTo":null,"subject":"[PATCH 0/9] builtin/cat-file: allow filtering objects in batch mode","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-21T07:47:25Z","receivedAt":"2025-02-21T07:47:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nat GitLab, we sometimes have the need to list all objects regardless of\ntheir reachability. We use git-cat-file(1) with `--batch-all-objects` to\ndo this, and typically this is quite a good fit. In some cases though,\nwe only want to list objects of a specific type, where we then basically\nhave the following pipeline:\n\n    git cat-file --batch-all-objects --batch-check='%(objecttype) %(objectname)' |\n    grep '^commit ' |\n    cut -d' ' -f2 |\n    git cat-file --batch\n\nThis works okayish in medium-sized repositories, but once you reach a\ncertain size this isn't really an option anymore. In the Chromium\nrepository for example [1] simply listing all objects in the first\ninvocation of git-cat-file(1) takes around 80 to 100 seconds. The\nworkload is completely I/O-bottlenecked: my machine reads at ~500MB/s,\nand the packfile is 50GB in size, which matches the 100 seconds that I\nobserve.\n\nThis series addresses the issue by introducing object filters into\ngit-cat-file(1). These object filters use the exact same syntax as the\nfilters we have in git-rev-list(1), but only a subset of them is\nsupported because not all filters can be computed by git-cat-file(1).\nSupported are \"blob:none\", \"blob:limit=\" as well as \"object:type=\".\n\nThe filters alone don't really help though: we still have to scan\nthrough the whole packfile in order to compute the packfiles. While we\nare able to shed a bit of CPU time because we can stop emitting some of\nthe objects, we're still I/O-bottlenecked.\n\nThe second part of the series thus expands the filters so that they can\nmake use of bitmap indices for some of the filters, if available. This\nallows us to efficiently answer the question where to find all objects\nof a specific type, and thus we can avoid scanning through the packfile\nand instead directly look up relevant objects, leading to a significant\nspeedup:\n\n    Benchmark 1: git cat-file --batch-check --batch-all-objects --unordered --buffer --no-objects-filter\n      Time (mean ± σ):     82.806 s ±  6.363 s    [User: 30.956 s, System: 8.264 s]\n      Range (min … max):   73.936 s … 89.690 s    10 runs\n\n    Benchmark 2: git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=object:type=tag\n      Time (mean ± σ):      20.8 ms ±   1.3 ms    [User: 6.1 ms, System: 14.5 ms]\n      Range (min … max):    18.2 ms …  23.6 ms    127 runs\n\n    Benchmark 3: git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=object:type=commit\n      Time (mean ± σ):      1.551 s ±  0.008 s    [User: 1.401 s, System: 0.147 s]\n      Range (min … max):    1.541 s …  1.566 s    10 runs\n\n    Benchmark 4: git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=object:type=tree\n      Time (mean ± σ):     11.169 s ±  0.046 s    [User: 10.076 s, System: 1.063 s]\n      Range (min … max):   11.114 s … 11.245 s    10 runs\n\n    Benchmark 5: git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=object:type=blob\n      Time (mean ± σ):     67.342 s ±  3.368 s    [User: 20.318 s, System: 7.787 s]\n      Range (min … max):   62.836 s … 73.618 s    10 runs\n\n    Benchmark 6: git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=blob:none\n      Time (mean ± σ):     13.032 s ±  0.072 s    [User: 11.638 s, System: 1.368 s]\n      Range (min … max):   12.960 s … 13.199 s    10 runs\n\n    Summary\n      git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=object:type=tag\n       74.75 ± 4.61 times faster than git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=object:type=commit\n      538.17 ± 33.17 times faster than git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=object:type=tree\n      627.98 ± 38.77 times faster than git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=blob:none\n     3244.93 ± 257.23 times faster than git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=object:type=blob\n     3990.07 ± 392.72 times faster than git cat-file --batch-check --batch-all-objects --unordered --buffer --no-objects-filter\n\nWe now directly scale with the number of objects of a specific type\ncontained in the packfile instead of scaling with the overall number of\nobjects. It's quite fun to see how the math plays out: if you sum up the\ntimes for each of the types you arrive at the time for the unfiltered\ncase.\n\nThanks!\n\nPatrick\n\n[1]: https://github.com/chromium/chromium.git\n\n---\nPatrick Steinhardt (9):\n      builtin/cat-file: rename variable that tracks usage\n      builtin/cat-file: wire up an option to filter objects\n      builtin/cat-file: support \"blob:none\" objects filter\n      builtin/cat-file: support \"blob:limit=\" objects filter\n      builtin/cat-file: support \"object:type=\" objects filter\n      pack-bitmap: expose function to iterate over bitmapped objects\n      pack-bitmap: introduce function to check whether a pack is bitmapped\n      builtin/cat-file: deduplicate logic to iterate over all objects\n      builtin/cat-file: use bitmaps to efficiently filter by object type\n\n Documentation/git-cat-file.adoc |  16 +++\n builtin/cat-file.c              | 225 +++++++++++++++++++++++++++++-----------\n builtin/pack-objects.c          |   3 +-\n builtin/rev-list.c              |   3 +-\n pack-bitmap.c                   |  80 +++++++++-----\n pack-bitmap.h                   |  19 +++-\n reachable.c                     |   3 +-\n t/t1006-cat-file.sh             |  77 ++++++++++++++\n 8 files changed, 339 insertions(+), 87 deletions(-)\n\n\n---\nbase-commit: a554262210b4a2ee6fa2d594e1f09f5830888c56\nchange-id: 20250220-pks-cat-file-object-type-filter-9140c0ed5ee1\n\n"},{"id":"512793","messageId":"20250221-pks-cat-file-object-type-filter-v1-3-0852530888e2@pks.im","threadId":"62984","inReplyTo":"20250221-pks-cat-file-object-type-filter-v1-0-0852530888e2@pks.im","subject":"[PATCH 3/9] builtin/cat-file: support \"blob:none\" objects filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-21T07:47:28Z","receivedAt":"2025-02-21T07:47:38Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement support for the \"blob:none\" filter in git-cat-file(1), which\ncauses us to omit all blobs.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n Documentation/git-cat-file.adoc |  2 ++\n builtin/cat-file.c              | 11 ++++++++++-\n t/t1006-cat-file.sh             | 33 +++++++++++++++++++++++++++++++--\n 3 files changed, 43 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-cat-file.adoc b/Documentation/git-cat-file.adoc\nindex 7c1c888079a..c11952d9eca 100644\n--- a/Documentation/git-cat-file.adoc\n+++ b/Documentation/git-cat-file.adoc\n@@ -86,6 +86,8 @@ OPTIONS\n \tOmit objects from the list of printed objects. This can only be used in\n \tcombination with one of the batched modes. The '<filter-spec>' may be\n \tone of the following:\n++\n+The form '--filter=blob:none' omits all blobs.\n \n --path=<path>::\n \tFor use with `--textconv` or `--filters`, to allow specifying an object\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 723644fbba8..8e5572ba43e 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -472,7 +472,8 @@ static void batch_object_write(const char *obj_name,\n \tif (!data->skip_object_info) {\n \t\tint ret;\n \n-\t\tif (use_mailmap)\n+\t\tif (use_mailmap ||\n+\t\t    opt->objects_filter.choice == LOFC_BLOB_NONE)\n \t\t\tdata->info.typep = &data->type;\n \n \t\tif (pack)\n@@ -492,6 +493,10 @@ static void batch_object_write(const char *obj_name,\n \t\tswitch (opt->objects_filter.choice) {\n \t\tcase LOFC_DISABLED:\n \t\t\tbreak;\n+\t\tcase LOFC_BLOB_NONE:\n+\t\t\tif (data->type == OBJ_BLOB)\n+\t\t\t\treturn;\n+\t\t\tbreak;\n \t\tdefault:\n \t\t\tBUG(\"unsupported objects filter\");\n \t\t}\n@@ -1032,6 +1037,10 @@ int cmd_cat_file(int argc,\n \tswitch (batch.objects_filter.choice) {\n \tcase LOFC_DISABLED:\n \t\tbreak;\n+\tcase LOFC_BLOB_NONE:\n+\t\tif (!batch.enabled)\n+\t\t\tusage(_(\"objects filter only supported in batch mode\"));\n+\t\tbreak;\n \tdefault:\n \t\tusagef(_(\"objects filter not supported: '%s'\"),\n \t\t       list_object_filter_config_name(batch.objects_filter.choice));\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 48840a13561..97533225982 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -1354,7 +1354,22 @@ test_expect_success PERL '--batch-command info is unbuffered by default' '\n '\n \n test_expect_success 'setup for objects filter' '\n-\tgit init repo\n+\tgit init repo &&\n+\t(\n+\t\t# Seed the repository with three different sets of objects:\n+\t\t#\n+\t\t#   - The first set is fully packed and has a bitmap.\n+\t\t#   - The second set is packed, but has no bitmap.\n+\t\t#   - The third set is loose.\n+\t\t#\n+\t\t# This ensures that we cover all these types as expected.\n+\t\tcd repo &&\n+\t\ttest_commit first &&\n+\t\tgit repack -Adb &&\n+\t\ttest_commit second &&\n+\t\tgit repack -d &&\n+\t\ttest_commit third\n+\t)\n '\n \n test_expect_success 'objects filter with unknown option' '\n@@ -1365,7 +1380,7 @@ test_expect_success 'objects filter with unknown option' '\n \ttest_cmp expect err\n '\n \n-for option in blob:none blob:limit=1 object:type=tag sparse:oid=1234 tree:1 sparse:path=x\n+for option in blob:limit=1 object:type=tag sparse:oid=1234 tree:1 sparse:path=x\n do\n \ttest_expect_success \"objects filter with unsupported option $option\" '\n \t\tcase \"$option\" in\n@@ -1385,4 +1400,18 @@ do\n \t'\n done\n \n+test_objects_filter () {\n+\tfilter=\"$1\"\n+\n+\ttest_expect_success \"objects filter: $filter\" '\n+\t\tgit -C repo cat-file --batch-check=\"%(objectname)\" --batch-all-objects --objects-filter=\"$filter\" >actual &&\n+\t\tsort actual >actual.sorted &&\n+\t\tgit -C repo rev-list --objects --no-object-names --all --filter=\"$filter\" --filter-provided-objects >expect &&\n+\t\tsort expect >expect.sorted &&\n+\t\ttest_cmp expect.sorted actual.sorted\n+\t'\n+}\n+\n+test_objects_filter \"blob:none\"\n+\n test_done\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512795","messageId":"20250221-pks-cat-file-object-type-filter-v1-2-0852530888e2@pks.im","threadId":"62984","inReplyTo":"20250221-pks-cat-file-object-type-filter-v1-0-0852530888e2@pks.im","subject":"[PATCH 2/9] builtin/cat-file: wire up an option to filter objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-21T07:47:27Z","receivedAt":"2025-02-21T07:47:38Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In batch mode, git-cat-file(1) enumerates all objects and prints them\nby iterating through both loose and packed objects. This works without\nconsidering their reachability at all, and consequently most options to\nfilter objects as they exist in e.g. git-rev-list(1) are not applicable.\nIn some situations it may still be useful though to filter objects based\non properties that are inherent to them. This includes the object size\nas well as its type.\n\nSuch a filter already exists in git-rev-list(1) with the `--filter=`\ncommand line option. While this option supports a couple of filters that\nare not applicable to our usecase, some of them are quite a neat fit.\n\nWire up the filter as an option for git-cat-file(1). This allows us to\nreuse the same syntax as in git-rev-list(1) so that we don't have to\nreinvent the wheel. For now, we die when any of the filter options has\nbeen passed by the user, but they will be wired up in subsequent\ncommits.\n\nNote that we don't use the same `--filter=` name fo the option as we use\nin git-rev-list(1). We already have `--filters`, and having both\n`--filter=` and `--filters` would be quite confusing. Instead, the new\noption is called `--objects-filter`.\n\nFurther note that the filters that we are about to introduce don't\nsignificantly speed up the runtime of git-cat-file(1). While we can skip\nemitting a lot of objects in case they are uninteresting to us, the\nmajority of time is spent reading the packfile, which is bottlenecked by\nI/O and not the processor. This will change though once we start to make\nuse of bitmaps, which will allow us to skip reading the whole packfile.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n Documentation/git-cat-file.adoc |  6 ++++++\n builtin/cat-file.c              | 37 +++++++++++++++++++++++++++++++++----\n t/t1006-cat-file.sh             | 32 ++++++++++++++++++++++++++++++++\n 3 files changed, 71 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-cat-file.adoc b/Documentation/git-cat-file.adoc\nindex d5890ae3686..7c1c888079a 100644\n--- a/Documentation/git-cat-file.adoc\n+++ b/Documentation/git-cat-file.adoc\n@@ -81,6 +81,12 @@ OPTIONS\n \tend-of-line conversion, etc). In this case, `<object>` has to be of\n \tthe form `<tree-ish>:<path>`, or `:<path>`.\n \n+--objects-filter=<filter-spec>::\n+--no-objects-filter::\n+\tOmit objects from the list of printed objects. This can only be used in\n+\tcombination with one of the batched modes. The '<filter-spec>' may be\n+\tone of the following:\n+\n --path=<path>::\n \tFor use with `--textconv` or `--filters`, to allow specifying an object\n \tname and a path separately, e.g. when it is difficult to figure out\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 8e40016dd24..723644fbba8 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -15,6 +15,7 @@\n #include \"gettext.h\"\n #include \"hex.h\"\n #include \"ident.h\"\n+#include \"list-objects-filter-options.h\"\n #include \"parse-options.h\"\n #include \"userdiff.h\"\n #include \"streaming.h\"\n@@ -35,6 +36,7 @@ enum batch_mode {\n };\n \n struct batch_options {\n+\tstruct list_objects_filter_options objects_filter;\n \tint enabled;\n \tint follow_symlinks;\n \tenum batch_mode batch_mode;\n@@ -487,6 +489,13 @@ static void batch_object_write(const char *obj_name,\n \t\t\treturn;\n \t\t}\n \n+\t\tswitch (opt->objects_filter.choice) {\n+\t\tcase LOFC_DISABLED:\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tBUG(\"unsupported objects filter\");\n+\t\t}\n+\n \t\tif (use_mailmap && (data->type == OBJ_COMMIT || data->type == OBJ_TAG)) {\n \t\t\tsize_t s = data->size;\n \t\t\tchar *buf = NULL;\n@@ -812,7 +821,8 @@ static int batch_objects(struct batch_options *opt)\n \t\tstruct object_cb_data cb;\n \t\tstruct object_info empty = OBJECT_INFO_INIT;\n \n-\t\tif (!memcmp(&data.info, &empty, sizeof(empty)))\n+\t\tif (!memcmp(&data.info, &empty, sizeof(empty)) &&\n+\t\t    opt->objects_filter.choice == LOFC_DISABLED)\n \t\t\tdata.skip_object_info = 1;\n \n \t\tif (repo_has_promisor_remote(the_repository))\n@@ -936,10 +946,13 @@ int cmd_cat_file(int argc,\n \tint opt_cw = 0;\n \tint opt_epts = 0;\n \tconst char *exp_type = NULL, *obj_name = NULL;\n-\tstruct batch_options batch = {0};\n+\tstruct batch_options batch = {\n+\t\t.objects_filter = LIST_OBJECTS_FILTER_INIT,\n+\t};\n \tint unknown_type = 0;\n \tint input_nul_terminated = 0;\n \tint nul_terminated = 0;\n+\tint ret;\n \n \tconst char * const builtin_catfile_usage[] = {\n \t\tN_(\"git cat-file <type> <object>\"),\n@@ -1000,6 +1013,8 @@ int cmd_cat_file(int argc,\n \t\t\t    N_(\"run filters on object's content\"), 'w'),\n \t\tOPT_STRING(0, \"path\", &force_path, N_(\"blob|tree\"),\n \t\t\t   N_(\"use a <path> for (--textconv | --filters); Not with 'batch'\")),\n+\t\tOPT_CALLBACK(0, \"objects-filter\", &batch.objects_filter, N_(\"args\"),\n+\t\t\t     N_(\"object filtering\"), opt_parse_list_objects_filter),\n \t\tOPT_END()\n \t};\n \n@@ -1014,6 +1029,14 @@ int cmd_cat_file(int argc,\n \tif (use_mailmap)\n \t\tread_mailmap(&mailmap);\n \n+\tswitch (batch.objects_filter.choice) {\n+\tcase LOFC_DISABLED:\n+\t\tbreak;\n+\tdefault:\n+\t\tusagef(_(\"objects filter not supported: '%s'\"),\n+\t\t       list_object_filter_config_name(batch.objects_filter.choice));\n+\t}\n+\n \t/* --batch-all-objects? */\n \tif (opt == 'b')\n \t\tbatch.all_objects = 1;\n@@ -1068,7 +1091,8 @@ int cmd_cat_file(int argc,\n \t\t\tusage_msg_opt(_(\"batch modes take no arguments\"), builtin_catfile_usage,\n \t\t\t\t      options);\n \n-\t\treturn batch_objects(&batch);\n+\t\tret = batch_objects(&batch);\n+\t\tgoto out;\n \t}\n \n \tif (opt) {\n@@ -1097,5 +1121,10 @@ int cmd_cat_file(int argc,\n \n \tif (unknown_type && opt != 't' && opt != 's')\n \t\tdie(\"git cat-file --allow-unknown-type: use with -s or -t\");\n-\treturn cat_one_file(opt, exp_type, obj_name, unknown_type);\n+\n+\tret = cat_one_file(opt, exp_type, obj_name, unknown_type);\n+\n+out:\n+\tlist_objects_filter_release(&batch.objects_filter);\n+\treturn ret;\n }\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 398865d6ebe..48840a13561 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -1353,4 +1353,36 @@ test_expect_success PERL '--batch-command info is unbuffered by default' '\n \tperl -e \"$script\" -- --batch-command $hello_oid \"$expect\" \"info \"\n '\n \n+test_expect_success 'setup for objects filter' '\n+\tgit init repo\n+'\n+\n+test_expect_success 'objects filter with unknown option' '\n+\tcat >expect <<-EOF &&\n+\tfatal: invalid filter-spec ${SQ}unknown${SQ}\n+\tEOF\n+\ttest_must_fail git -C repo cat-file --objects-filter=unknown 2>err &&\n+\ttest_cmp expect err\n+'\n+\n+for option in blob:none blob:limit=1 object:type=tag sparse:oid=1234 tree:1 sparse:path=x\n+do\n+\ttest_expect_success \"objects filter with unsupported option $option\" '\n+\t\tcase \"$option\" in\n+\t\ttree:1)\n+\t\t\techo \"usage: objects filter not supported: ${SQ}tree${SQ}\" >expect\n+\t\t\t;;\n+\t\tsparse:path=x)\n+\t\t\techo \"fatal: sparse:path filters support has been dropped\" >expect\n+\t\t\t;;\n+\t\t*)\n+\t\t\toption_name=$(echo \"$option\" | cut -d= -f1) &&\n+\t\t\tprintf \"usage: objects filter not supported: ${SQ}%s${SQ}\\n\" \"$option_name\" >expect\n+\t\t\t;;\n+\t\tesac &&\n+\t\ttest_must_fail git -C repo cat-file --objects-filter=$option 2>err &&\n+\t\ttest_cmp expect err\n+\t'\n+done\n+\n test_done\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512796","messageId":"20250221-pks-cat-file-object-type-filter-v1-4-0852530888e2@pks.im","threadId":"62984","inReplyTo":"20250221-pks-cat-file-object-type-filter-v1-0-0852530888e2@pks.im","subject":"[PATCH 4/9] builtin/cat-file: support \"blob:limit=\" objects filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-21T07:47:29Z","receivedAt":"2025-02-21T07:47:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement support for the \"blob:limit=\" filter in git-cat-file(1), which\ncauses us to omit all blobs that are bigger than a certain size.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n Documentation/git-cat-file.adoc |  5 +++++\n builtin/cat-file.c              | 11 ++++++++++-\n t/t1006-cat-file.sh             | 18 +++++++++++++++---\n 3 files changed, 30 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-cat-file.adoc b/Documentation/git-cat-file.adoc\nindex c11952d9eca..8c474418b52 100644\n--- a/Documentation/git-cat-file.adoc\n+++ b/Documentation/git-cat-file.adoc\n@@ -88,6 +88,11 @@ OPTIONS\n \tone of the following:\n +\n The form '--filter=blob:none' omits all blobs.\n++\n+The form '--filter=blob:limit=<n>[kmg]' omits blobs of size at least n\n+bytes or units.  n may be zero.  The suffixes k, m, and g can be used\n+to name units in KiB, MiB, or GiB.  For example, 'blob:limit=1k'\n+is the same as 'blob:limit=1024'.\n \n --path=<path>::\n \tFor use with `--textconv` or `--filters`, to allow specifying an object\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 8e5572ba43e..f57bf65cb03 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -473,8 +473,11 @@ static void batch_object_write(const char *obj_name,\n \t\tint ret;\n \n \t\tif (use_mailmap ||\n-\t\t    opt->objects_filter.choice == LOFC_BLOB_NONE)\n+\t\t    opt->objects_filter.choice == LOFC_BLOB_NONE ||\n+\t\t    opt->objects_filter.choice == LOFC_BLOB_LIMIT)\n \t\t\tdata->info.typep = &data->type;\n+\t\tif (opt->objects_filter.choice == LOFC_BLOB_LIMIT)\n+\t\t\tdata->info.sizep = &data->size;\n \n \t\tif (pack)\n \t\t\tret = packed_object_info(the_repository, pack, offset,\n@@ -497,6 +500,11 @@ static void batch_object_write(const char *obj_name,\n \t\t\tif (data->type == OBJ_BLOB)\n \t\t\t\treturn;\n \t\t\tbreak;\n+\t\tcase LOFC_BLOB_LIMIT:\n+\t\t\tif (data->type == OBJ_BLOB &&\n+\t\t\t    data->size >= opt->objects_filter.blob_limit_value)\n+\t\t\t\treturn;\n+\t\t\tbreak;\n \t\tdefault:\n \t\t\tBUG(\"unsupported objects filter\");\n \t\t}\n@@ -1038,6 +1046,7 @@ int cmd_cat_file(int argc,\n \tcase LOFC_DISABLED:\n \t\tbreak;\n \tcase LOFC_BLOB_NONE:\n+\tcase LOFC_BLOB_LIMIT:\n \t\tif (!batch.enabled)\n \t\t\tusage(_(\"objects filter only supported in batch mode\"));\n \t\tbreak;\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 97533225982..86c53e01b2f 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -1356,11 +1356,12 @@ test_expect_success PERL '--batch-command info is unbuffered by default' '\n test_expect_success 'setup for objects filter' '\n \tgit init repo &&\n \t(\n-\t\t# Seed the repository with three different sets of objects:\n+\t\t# Seed the repository with four different sets of objects:\n \t\t#\n \t\t#   - The first set is fully packed and has a bitmap.\n \t\t#   - The second set is packed, but has no bitmap.\n \t\t#   - The third set is loose.\n+\t\t#   - The fourth set is loose and contains big objects.\n \t\t#\n \t\t# This ensures that we cover all these types as expected.\n \t\tcd repo &&\n@@ -1368,7 +1369,14 @@ test_expect_success 'setup for objects filter' '\n \t\tgit repack -Adb &&\n \t\ttest_commit second &&\n \t\tgit repack -d &&\n-\t\ttest_commit third\n+\t\ttest_commit third &&\n+\n+\t\tfor n in 1000 10000\n+\t\tdo\n+\t\t\tprintf \"%\"$n\"s\" X >large.$n || return 1\n+\t\tdone &&\n+\t\tgit add large.* &&\n+\t\tgit commit -m fourth\n \t)\n '\n \n@@ -1380,7 +1388,7 @@ test_expect_success 'objects filter with unknown option' '\n \ttest_cmp expect err\n '\n \n-for option in blob:limit=1 object:type=tag sparse:oid=1234 tree:1 sparse:path=x\n+for option in object:type=tag sparse:oid=1234 tree:1 sparse:path=x\n do\n \ttest_expect_success \"objects filter with unsupported option $option\" '\n \t\tcase \"$option\" in\n@@ -1413,5 +1421,9 @@ test_objects_filter () {\n }\n \n test_objects_filter \"blob:none\"\n+test_objects_filter \"blob:limit=1\"\n+test_objects_filter \"blob:limit=500\"\n+test_objects_filter \"blob:limit=1000\"\n+test_objects_filter \"blob:limit=1g\"\n \n test_done\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512798","messageId":"20250221-pks-cat-file-object-type-filter-v1-5-0852530888e2@pks.im","threadId":"62984","inReplyTo":"20250221-pks-cat-file-object-type-filter-v1-0-0852530888e2@pks.im","subject":"[PATCH 5/9] builtin/cat-file: support \"object:type=\" objects filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-21T07:47:30Z","receivedAt":"2025-02-21T07:47:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement support for the \"object:type=\" filter in git-cat-file(1),\nwhich causes us to omit all objects that don't match the provided object\ntype.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n Documentation/git-cat-file.adoc | 3 +++\n builtin/cat-file.c              | 8 +++++++-\n t/t1006-cat-file.sh             | 6 +++++-\n 3 files changed, 15 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-cat-file.adoc b/Documentation/git-cat-file.adoc\nindex 8c474418b52..540d9dffdf9 100644\n--- a/Documentation/git-cat-file.adoc\n+++ b/Documentation/git-cat-file.adoc\n@@ -93,6 +93,9 @@ The form '--filter=blob:limit=<n>[kmg]' omits blobs of size at least n\n bytes or units.  n may be zero.  The suffixes k, m, and g can be used\n to name units in KiB, MiB, or GiB.  For example, 'blob:limit=1k'\n is the same as 'blob:limit=1024'.\n++\n+The form '--filter=object:type=(tag|commit|tree|blob)' omits all objects\n+which are not of the requested type.\n \n --path=<path>::\n \tFor use with `--textconv` or `--filters`, to allow specifying an object\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex f57bf65cb03..b374c2bb104 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -474,7 +474,8 @@ static void batch_object_write(const char *obj_name,\n \n \t\tif (use_mailmap ||\n \t\t    opt->objects_filter.choice == LOFC_BLOB_NONE ||\n-\t\t    opt->objects_filter.choice == LOFC_BLOB_LIMIT)\n+\t\t    opt->objects_filter.choice == LOFC_BLOB_LIMIT ||\n+\t\t    opt->objects_filter.choice == LOFC_OBJECT_TYPE)\n \t\t\tdata->info.typep = &data->type;\n \t\tif (opt->objects_filter.choice == LOFC_BLOB_LIMIT)\n \t\t\tdata->info.sizep = &data->size;\n@@ -505,6 +506,10 @@ static void batch_object_write(const char *obj_name,\n \t\t\t    data->size >= opt->objects_filter.blob_limit_value)\n \t\t\t\treturn;\n \t\t\tbreak;\n+\t\tcase LOFC_OBJECT_TYPE:\n+\t\t\tif (data->type != opt->objects_filter.object_type)\n+\t\t\t\treturn;\n+\t\t\tbreak;\n \t\tdefault:\n \t\t\tBUG(\"unsupported objects filter\");\n \t\t}\n@@ -1047,6 +1052,7 @@ int cmd_cat_file(int argc,\n \t\tbreak;\n \tcase LOFC_BLOB_NONE:\n \tcase LOFC_BLOB_LIMIT:\n+\tcase LOFC_OBJECT_TYPE:\n \t\tif (!batch.enabled)\n \t\t\tusage(_(\"objects filter only supported in batch mode\"));\n \t\tbreak;\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 86c53e01b2f..b908bbf60e1 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -1388,7 +1388,7 @@ test_expect_success 'objects filter with unknown option' '\n \ttest_cmp expect err\n '\n \n-for option in object:type=tag sparse:oid=1234 tree:1 sparse:path=x\n+for option in sparse:oid=1234 tree:1 sparse:path=x\n do\n \ttest_expect_success \"objects filter with unsupported option $option\" '\n \t\tcase \"$option\" in\n@@ -1425,5 +1425,9 @@ test_objects_filter \"blob:limit=1\"\n test_objects_filter \"blob:limit=500\"\n test_objects_filter \"blob:limit=1000\"\n test_objects_filter \"blob:limit=1g\"\n+test_objects_filter \"object:type=blob\"\n+test_objects_filter \"object:type=commit\"\n+test_objects_filter \"object:type=tag\"\n+test_objects_filter \"object:type=tree\"\n \n test_done\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512799","messageId":"20250221-pks-cat-file-object-type-filter-v1-6-0852530888e2@pks.im","threadId":"62984","inReplyTo":"20250221-pks-cat-file-object-type-filter-v1-0-0852530888e2@pks.im","subject":"[PATCH 6/9] pack-bitmap: expose function to iterate over bitmapped objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-21T07:47:31Z","receivedAt":"2025-02-21T07:47:40Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Expose a function that allows the caller to iterate over all bitmapped\nobjects of a specific type. This mechanism allows us to use the object\ntype-specific bitmaps to enumerate all objects of that type without\nhaving to scan through a complete packfile.\n\nThis functionality will be used in a subsequent commit.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/pack-objects.c |  3 ++-\n builtin/rev-list.c     |  3 ++-\n pack-bitmap.c          | 65 +++++++++++++++++++++++++++++++-------------------\n pack-bitmap.h          | 12 +++++++++-\n reachable.c            |  3 ++-\n 5 files changed, 57 insertions(+), 29 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 58a9b161262..8f99e2b4fa8 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1735,7 +1735,8 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n static int add_object_entry_from_bitmap(const struct object_id *oid,\n \t\t\t\t\tenum object_type type,\n \t\t\t\t\tint flags UNUSED, uint32_t name_hash,\n-\t\t\t\t\tstruct packed_git *pack, off_t offset)\n+\t\t\t\t\tstruct packed_git *pack, off_t offset,\n+\t\t\t\t\tvoid *payload UNUSED)\n {\n \tdisplay_progress(progress_state, ++nr_seen);\n \ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex bb26bee0d45..1100dd2abe7 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -429,7 +429,8 @@ static int show_object_fast(\n \tint exclude UNUSED,\n \tuint32_t name_hash UNUSED,\n \tstruct packed_git *found_pack UNUSED,\n-\toff_t found_offset UNUSED)\n+\toff_t found_offset UNUSED,\n+\tvoid *payload UNUSED)\n {\n \tfprintf(stdout, \"%s\\n\", oid_to_hex(oid));\n \treturn 1;\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 6406953d322..fc92e0aae65 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1509,50 +1509,45 @@ static void show_extended_objects(struct bitmap_index *bitmap_git,\n \t\t    (obj->type == OBJ_TAG && !revs->tag_objects))\n \t\t\tcontinue;\n \n-\t\tshow_reach(&obj->oid, obj->type, 0, eindex->hashes[i], NULL, 0);\n+\t\tshow_reach(&obj->oid, obj->type, 0, eindex->hashes[i], NULL, 0, NULL);\n \t}\n }\n \n-static void init_type_iterator(struct ewah_iterator *it,\n-\t\t\t       struct bitmap_index *bitmap_git,\n-\t\t\t       enum object_type type)\n+static struct ewah_bitmap *ewah_for_type(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t enum object_type type)\n {\n \tswitch (type) {\n \tcase OBJ_COMMIT:\n-\t\tewah_iterator_init(it, bitmap_git->commits);\n-\t\tbreak;\n-\n+\t\treturn bitmap_git->commits;\n \tcase OBJ_TREE:\n-\t\tewah_iterator_init(it, bitmap_git->trees);\n-\t\tbreak;\n-\n+\t\treturn bitmap_git->trees;\n \tcase OBJ_BLOB:\n-\t\tewah_iterator_init(it, bitmap_git->blobs);\n-\t\tbreak;\n-\n+\t\treturn bitmap_git->blobs;\n \tcase OBJ_TAG:\n-\t\tewah_iterator_init(it, bitmap_git->tags);\n-\t\tbreak;\n-\n+\t\treturn bitmap_git->tags;\n \tdefault:\n \t\tBUG(\"object type %d not stored by bitmap type index\", type);\n-\t\tbreak;\n \t}\n }\n \n-static void show_objects_for_type(\n-\tstruct bitmap_index *bitmap_git,\n-\tenum object_type object_type,\n-\tshow_reachable_fn show_reach)\n+static void init_type_iterator(struct ewah_iterator *it,\n+\t\t\t       struct bitmap_index *bitmap_git,\n+\t\t\t       enum object_type type)\n+{\n+\tewah_iterator_init(it, ewah_for_type(bitmap_git, type));\n+}\n+\n+static void for_each_bitmapped_object_internal(struct bitmap_index *bitmap_git,\n+\t\t\t\t\t       struct bitmap *objects,\n+\t\t\t\t\t       enum object_type object_type,\n+\t\t\t\t\t       show_reachable_fn show_reach,\n+\t\t\t\t\t       void *payload)\n {\n \tsize_t i = 0;\n \tuint32_t offset;\n-\n \tstruct ewah_iterator it;\n \teword_t filter;\n \n-\tstruct bitmap *objects = bitmap_git->result;\n-\n \tinit_type_iterator(&it, bitmap_git, object_type);\n \n \tfor (i = 0; i < objects->word_alloc &&\n@@ -1595,11 +1590,31 @@ static void show_objects_for_type(\n \t\t\tif (bitmap_git->hashes)\n \t\t\t\thash = get_be32(bitmap_git->hashes + index_pos);\n \n-\t\t\tshow_reach(&oid, object_type, 0, hash, pack, ofs);\n+\t\t\tshow_reach(&oid, object_type, 0, hash, pack, ofs, payload);\n \t\t}\n \t}\n }\n \n+static void show_objects_for_type(\n+\tstruct bitmap_index *bitmap_git,\n+\tenum object_type object_type,\n+\tshow_reachable_fn show_reach)\n+{\n+\tfor_each_bitmapped_object_internal(bitmap_git, bitmap_git->result,\n+\t\t\t\t\t   object_type, show_reach, NULL);\n+}\n+\n+void for_each_bitmapped_object(struct bitmap_index *bitmap_git,\n+\t\t\t       enum object_type object_type,\n+\t\t\t       show_reachable_fn show_reach,\n+\t\t\t       void *payload)\n+{\n+\tstruct bitmap *bitmap = ewah_to_bitmap(ewah_for_type(bitmap_git, object_type));\n+\tfor_each_bitmapped_object_internal(bitmap_git, bitmap,\n+\t\t\t\t\t   object_type, show_reach, payload);\n+\tbitmap_free(bitmap);\n+}\n+\n static int in_bitmapped_pack(struct bitmap_index *bitmap_git,\n \t\t\t     struct object_list *roots)\n {\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex d7f4b8b8e95..3368e79ed5a 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -50,7 +50,8 @@ typedef int (*show_reachable_fn)(\n \tint flags,\n \tuint32_t hash,\n \tstruct packed_git *found_pack,\n-\toff_t found_offset);\n+\toff_t found_offset,\n+\tvoid *payload);\n \n struct bitmap_index;\n \n@@ -78,6 +79,15 @@ int test_bitmap_pseudo_merges(struct repository *r);\n int test_bitmap_pseudo_merge_commits(struct repository *r, uint32_t n);\n int test_bitmap_pseudo_merge_objects(struct repository *r, uint32_t n);\n \n+/*\n+ * Iterate through all bitmapped objects of the given type and execute the\n+ * `show_reach` for each of them.\n+ */\n+ void for_each_bitmapped_object(struct bitmap_index *bitmap_git,\n+\t\t\t       enum object_type object_type,\n+\t\t\t       show_reachable_fn show_reach,\n+\t\t\t       void *payload);\n+\n #define GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL \\\n \t\"GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL\"\n \ndiff --git a/reachable.c b/reachable.c\nindex ecf7ccf5041..dd33c7f07dd 100644\n--- a/reachable.c\n+++ b/reachable.c\n@@ -337,7 +337,8 @@ static int mark_object_seen(const struct object_id *oid,\n \t\t\t     int exclude UNUSED,\n \t\t\t     uint32_t name_hash UNUSED,\n \t\t\t     struct packed_git *found_pack UNUSED,\n-\t\t\t     off_t found_offset UNUSED)\n+\t\t\t     off_t found_offset UNUSED,\n+\t\t\t     void *payload UNUSED)\n {\n \tstruct object *obj = lookup_object_by_type(the_repository, oid, type);\n \tif (!obj)\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512800","messageId":"20250221-pks-cat-file-object-type-filter-v1-8-0852530888e2@pks.im","threadId":"62984","inReplyTo":"20250221-pks-cat-file-object-type-filter-v1-0-0852530888e2@pks.im","subject":"[PATCH 8/9] builtin/cat-file: deduplicate logic to iterate over all objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-21T07:47:33Z","receivedAt":"2025-02-21T07:47:40Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Pull out a common function that allows us to iterate over all objects in\na repository. Right now the logic is trivial and would only require two\nfunction calls, making this refactoring a bit pointless. But in the next\ncommit we will iterate on this logic to make use of bitmaps, so this is\nabout to become a bit more complex.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/cat-file.c | 85 ++++++++++++++++++++++++++++++------------------------\n 1 file changed, 48 insertions(+), 37 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex b374c2bb104..25d5429e391 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -622,25 +622,18 @@ static int batch_object_cb(const struct object_id *oid, void *vdata)\n \treturn 0;\n }\n \n-static int collect_loose_object(const struct object_id *oid,\n-\t\t\t\tconst char *path UNUSED,\n-\t\t\t\tvoid *data)\n-{\n-\toid_array_append(data, oid);\n-\treturn 0;\n-}\n-\n-static int collect_packed_object(const struct object_id *oid,\n-\t\t\t\t struct packed_git *pack UNUSED,\n-\t\t\t\t uint32_t pos UNUSED,\n-\t\t\t\t void *data)\n+static int collect_object(const struct object_id *oid,\n+\t\t\t  struct packed_git *pack UNUSED,\n+\t\t\t  off_t offset UNUSED,\n+\t\t\t  void *data)\n {\n \toid_array_append(data, oid);\n \treturn 0;\n }\n \n static int batch_unordered_object(const struct object_id *oid,\n-\t\t\t\t  struct packed_git *pack, off_t offset,\n+\t\t\t\t  struct packed_git *pack,\n+\t\t\t\t  off_t offset,\n \t\t\t\t  void *vdata)\n {\n \tstruct object_cb_data *data = vdata;\n@@ -654,23 +647,6 @@ static int batch_unordered_object(const struct object_id *oid,\n \treturn 0;\n }\n \n-static int batch_unordered_loose(const struct object_id *oid,\n-\t\t\t\t const char *path UNUSED,\n-\t\t\t\t void *data)\n-{\n-\treturn batch_unordered_object(oid, NULL, 0, data);\n-}\n-\n-static int batch_unordered_packed(const struct object_id *oid,\n-\t\t\t\t  struct packed_git *pack,\n-\t\t\t\t  uint32_t pos,\n-\t\t\t\t  void *data)\n-{\n-\treturn batch_unordered_object(oid, pack,\n-\t\t\t\t      nth_packed_object_offset(pack, pos),\n-\t\t\t\t      data);\n-}\n-\n typedef void (*parse_cmd_fn_t)(struct batch_options *, const char *,\n \t\t\t       struct strbuf *, struct expand_data *);\n \n@@ -803,6 +779,45 @@ static void batch_objects_command(struct batch_options *opt,\n \n #define DEFAULT_FORMAT \"%(objectname) %(objecttype) %(objectsize)\"\n \n+typedef int (*for_each_object_fn)(const struct object_id *oid, struct packed_git *pack,\n+\t\t\t\t  off_t offset, void *data);\n+\n+struct for_each_object_payload {\n+\tfor_each_object_fn callback;\n+\tvoid *payload;\n+};\n+\n+static int batch_one_object_loose(const struct object_id *oid,\n+\t\t\t\t  const char *path UNUSED,\n+\t\t\t\t  void *_payload)\n+{\n+\tstruct for_each_object_payload *payload = _payload;\n+\treturn payload->callback(oid, NULL, 0, payload->payload);\n+}\n+\n+static int batch_one_object_packed(const struct object_id *oid,\n+\t\t\t\t   struct packed_git *pack,\n+\t\t\t\t   uint32_t pos,\n+\t\t\t\t   void *_payload)\n+{\n+\tstruct for_each_object_payload *payload = _payload;\n+\treturn payload->callback(oid, pack, nth_packed_object_offset(pack, pos),\n+\t\t\t\t payload->payload);\n+}\n+\n+static void batch_each_object(for_each_object_fn callback,\n+\t\t\t      unsigned flags,\n+\t\t\t      void *_payload)\n+{\n+\tstruct for_each_object_payload payload = {\n+\t\t.callback = callback,\n+\t\t.payload = _payload,\n+\t};\n+\tfor_each_loose_object(batch_one_object_loose, &payload, 0);\n+\tfor_each_packed_object(the_repository, batch_one_object_packed,\n+\t\t\t       &payload, flags);\n+}\n+\n static int batch_objects(struct batch_options *opt)\n {\n \tstruct strbuf input = STRBUF_INIT;\n@@ -857,18 +872,14 @@ static int batch_objects(struct batch_options *opt)\n \n \t\t\tcb.seen = &seen;\n \n-\t\t\tfor_each_loose_object(batch_unordered_loose, &cb, 0);\n-\t\t\tfor_each_packed_object(the_repository, batch_unordered_packed,\n-\t\t\t\t\t       &cb, FOR_EACH_OBJECT_PACK_ORDER);\n+\t\t\tbatch_each_object(batch_unordered_object,\n+\t\t\t\t\t  FOR_EACH_OBJECT_PACK_ORDER, &cb);\n \n \t\t\toidset_clear(&seen);\n \t\t} else {\n \t\t\tstruct oid_array sa = OID_ARRAY_INIT;\n \n-\t\t\tfor_each_loose_object(collect_loose_object, &sa, 0);\n-\t\t\tfor_each_packed_object(the_repository, collect_packed_object,\n-\t\t\t\t\t       &sa, 0);\n-\n+\t\t\tbatch_each_object(collect_object, 0, &sa);\n \t\t\toid_array_for_each_unique(&sa, batch_object_cb, &cb);\n \n \t\t\toid_array_clear(&sa);\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512801","messageId":"20250221-pks-cat-file-object-type-filter-v1-7-0852530888e2@pks.im","threadId":"62984","inReplyTo":"20250221-pks-cat-file-object-type-filter-v1-0-0852530888e2@pks.im","subject":"[PATCH 7/9] pack-bitmap: introduce function to check whether a pack is bitmapped","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-21T07:47:32Z","receivedAt":"2025-02-21T07:47:41Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a function that allows us to verify whether a pack is\nbitmapped or not. This functionality will be used in a subsequent\ncommit.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n pack-bitmap.c | 15 +++++++++++++++\n pack-bitmap.h |  7 +++++++\n 2 files changed, 22 insertions(+)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex fc92e0aae65..3cbe5bfe909 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -658,6 +658,21 @@ struct bitmap_index *prepare_midx_bitmap_git(struct multi_pack_index *midx)\n \treturn NULL;\n }\n \n+int bitmap_index_contains_pack(struct bitmap_index *bitmap, struct packed_git *pack)\n+{\n+\tif (bitmap->pack)\n+\t\treturn bitmap->pack == pack;\n+\n+\tif (!bitmap->midx->chunk_bitmapped_packs)\n+\t\treturn 0;\n+\n+\tfor (size_t i = 0; i < bitmap->midx->num_packs; i++)\n+\t\tif (bitmap->midx->packs[i] == pack)\n+\t\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n struct include_data {\n \tstruct bitmap_index *bitmap_git;\n \tstruct bitmap *base;\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 3368e79ed5a..45e96b213e2 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -67,6 +67,13 @@ struct bitmapped_pack {\n \n struct bitmap_index *prepare_bitmap_git(struct repository *r);\n struct bitmap_index *prepare_midx_bitmap_git(struct multi_pack_index *midx);\n+\n+/*\n+ * Given a bitmap index, determine whether it contains the pack either directly\n+ * or via the multi-pack-index.\n+ */\n+int bitmap_index_contains_pack(struct bitmap_index *bitmap, struct packed_git *pack);\n+\n void count_bitmap_commit_list(struct bitmap_index *, uint32_t *commits,\n \t\t\t      uint32_t *trees, uint32_t *blobs, uint32_t *tags);\n void traverse_bitmap_commit_list(struct bitmap_index *,\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512802","messageId":"20250221-pks-cat-file-object-type-filter-v1-9-0852530888e2@pks.im","threadId":"62984","inReplyTo":"20250221-pks-cat-file-object-type-filter-v1-0-0852530888e2@pks.im","subject":"[PATCH 9/9] builtin/cat-file: use bitmaps to efficiently filter by object type","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-21T07:47:34Z","receivedAt":"2025-02-21T07:47:41Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"While it is now possible to filter objects by type, this mechanism is\nfor now mostly a convenience. Most importantly, we still have to iterate\nthrough the whole packfile to find all objects of a specific type. This\ncan be prohibitively expensive depending on the size of the packfiles.\n\nIt isn't really possible to do better than this when only considering a\npackfile itself, as the order of objects is not fixed. But when we have\na packfile with a corresponding bitmap, either because the packfile\nitself has one or because the multi-pack index has a bitmap for it, then\nwe can use these bitmaps to improve the runtime.\n\nWhile bitmaps are typically used to compute reachability of objects,\nthey also contain one bitmap per object type encodes which object has\nwhat type. So instead of reading through the whole packfile(s), we can\nuse the bitmaps and iterate through the type-specific bitmap. Typically,\nonly a subset of packfiles will have a bitmap. But this isn't really\nmuch of a problem: we can use bitmaps when available, and then use the\nnon-bitmap walk for every packfile that isn't covered by one.\n\nOverall, this leads to quite a significant speedup depending on how many\nobjects of a certain type exist. The following benchmarks have been\nexecuted in the Chromium repository, which has a 50GB packfile with\nalmost 25 million objects:\n\n    Benchmark 1: git cat-file --batch-check --batch-all-objects --unordered --buffer --no-objects-filter\n      Time (mean ± σ):     82.806 s ±  6.363 s    [User: 30.956 s, System: 8.264 s]\n      Range (min … max):   73.936 s … 89.690 s    10 runs\n\n    Benchmark 2: git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=object:type=tag\n      Time (mean ± σ):      20.8 ms ±   1.3 ms    [User: 6.1 ms, System: 14.5 ms]\n      Range (min … max):    18.2 ms …  23.6 ms    127 runs\n\n    Benchmark 3: git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=object:type=commit\n      Time (mean ± σ):      1.551 s ±  0.008 s    [User: 1.401 s, System: 0.147 s]\n      Range (min … max):    1.541 s …  1.566 s    10 runs\n\n    Benchmark 4: git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=object:type=tree\n      Time (mean ± σ):     11.169 s ±  0.046 s    [User: 10.076 s, System: 1.063 s]\n      Range (min … max):   11.114 s … 11.245 s    10 runs\n\n    Benchmark 5: git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=object:type=blob\n      Time (mean ± σ):     67.342 s ±  3.368 s    [User: 20.318 s, System: 7.787 s]\n      Range (min … max):   62.836 s … 73.618 s    10 runs\n\n    Benchmark 6: git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=blob:none\n      Time (mean ± σ):     13.032 s ±  0.072 s    [User: 11.638 s, System: 1.368 s]\n      Range (min … max):   12.960 s … 13.199 s    10 runs\n\n    Summary\n      git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=object:type=tag\n       74.75 ± 4.61 times faster than git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=object:type=commit\n      538.17 ± 33.17 times faster than git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=object:type=tree\n      627.98 ± 38.77 times faster than git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=blob:none\n     3244.93 ± 257.23 times faster than git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=object:type=blob\n     3990.07 ± 392.72 times faster than git cat-file --batch-check --batch-all-objects --unordered --buffer --no-objects-filter\n\nThe first benchmark is mostly equivalent in runtime compared to all the\nothers without the bitmap-optimization introduced in this commit. What\nis noticeable in the benchmarks is that we're I/O-bound, not CPU-bound,\nas can be seen from the user/system runtimes, which is often way lower\nthan the overall benchmarked runtime.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/cat-file.c | 55 +++++++++++++++++++++++++++++++++++++++++++++++++-----\n 1 file changed, 50 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 25d5429e391..9021fd52f30 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -21,6 +21,7 @@\n #include \"streaming.h\"\n #include \"oid-array.h\"\n #include \"packfile.h\"\n+#include \"pack-bitmap.h\"\n #include \"object-file.h\"\n #include \"object-name.h\"\n #include \"object-store-ll.h\"\n@@ -805,7 +806,20 @@ static int batch_one_object_packed(const struct object_id *oid,\n \t\t\t\t payload->payload);\n }\n \n-static void batch_each_object(for_each_object_fn callback,\n+static int batch_one_object_bitmapped(const struct object_id *oid,\n+\t\t\t\t      enum object_type type UNUSED,\n+\t\t\t\t      int flags UNUSED,\n+\t\t\t\t      uint32_t hash UNUSED,\n+\t\t\t\t      struct packed_git *pack,\n+\t\t\t\t      off_t offset,\n+\t\t\t\t      void *_payload)\n+{\n+\tstruct for_each_object_payload *payload = _payload;\n+\treturn payload->callback(oid, pack, offset, payload->payload);\n+}\n+\n+static void batch_each_object(struct batch_options *opt,\n+\t\t\t      for_each_object_fn callback,\n \t\t\t      unsigned flags,\n \t\t\t      void *_payload)\n {\n@@ -813,9 +827,40 @@ static void batch_each_object(for_each_object_fn callback,\n \t\t.callback = callback,\n \t\t.payload = _payload,\n \t};\n+\tstruct bitmap_index *bitmap = prepare_bitmap_git(the_repository);\n+\n \tfor_each_loose_object(batch_one_object_loose, &payload, 0);\n-\tfor_each_packed_object(the_repository, batch_one_object_packed,\n-\t\t\t       &payload, flags);\n+\n+\tif (bitmap &&\n+\t    (opt->objects_filter.choice == LOFC_OBJECT_TYPE ||\n+\t     opt->objects_filter.choice == LOFC_BLOB_NONE)) {\n+\t\tstruct packed_git *pack;\n+\n+\t\tif (opt->objects_filter.choice == LOFC_OBJECT_TYPE) {\n+\t\t\tfor_each_bitmapped_object(bitmap, opt->objects_filter.object_type,\n+\t\t\t\t\t\t  batch_one_object_bitmapped, &payload);\n+\t\t} else {\n+\t\t\tfor_each_bitmapped_object(bitmap, OBJ_COMMIT,\n+\t\t\t\t\t\t  batch_one_object_bitmapped, &payload);\n+\t\t\tfor_each_bitmapped_object(bitmap, OBJ_TAG,\n+\t\t\t\t\t\t  batch_one_object_bitmapped, &payload);\n+\t\t\tfor_each_bitmapped_object(bitmap, OBJ_TREE,\n+\t\t\t\t\t\t  batch_one_object_bitmapped, &payload);\n+\t\t}\n+\n+\t\tfor (pack = get_all_packs(the_repository); pack; pack = pack->next) {\n+\t\t\tif (bitmap_index_contains_pack(bitmap, pack) ||\n+\t\t\t    open_pack_index(pack))\n+\t\t\t\tcontinue;\n+\t\t\tfor_each_object_in_pack(pack, batch_one_object_packed,\n+\t\t\t\t\t\t&payload, flags);\n+\t\t}\n+\t} else {\n+\t\tfor_each_packed_object(the_repository, batch_one_object_packed,\n+\t\t\t\t       &payload, flags);\n+\t}\n+\n+\tfree_bitmap_index(bitmap);\n }\n \n static int batch_objects(struct batch_options *opt)\n@@ -872,14 +917,14 @@ static int batch_objects(struct batch_options *opt)\n \n \t\t\tcb.seen = &seen;\n \n-\t\t\tbatch_each_object(batch_unordered_object,\n+\t\t\tbatch_each_object(opt, batch_unordered_object,\n \t\t\t\t\t  FOR_EACH_OBJECT_PACK_ORDER, &cb);\n \n \t\t\toidset_clear(&seen);\n \t\t} else {\n \t\t\tstruct oid_array sa = OID_ARRAY_INIT;\n \n-\t\t\tbatch_each_object(collect_object, 0, &sa);\n+\t\t\tbatch_each_object(opt, collect_object, 0, &sa);\n \t\t\toid_array_for_each_unique(&sa, batch_object_cb, &cb);\n \n \t\t\toid_array_clear(&sa);\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512926","messageId":"xmqqseo35ic8.fsf@gitster.g","threadId":"62984","inReplyTo":"20250221-pks-cat-file-object-type-filter-v1-6-0852530888e2@pks.im","subject":"Re: [PATCH 6/9] pack-bitmap: expose function to iterate over bitmapped objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-02-24T18:05:27Z","receivedAt":"2025-02-24T18:05:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> Expose a function that allows the caller to iterate over all bitmapped\n> objects of a specific type. This mechanism allows us to use the object\n> type-specific bitmaps to enumerate all objects of that type without\n> having to scan through a complete packfile.\n>\n> This functionality will be used in a subsequent commit.\n>\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  builtin/pack-objects.c |  3 ++-\n>  builtin/rev-list.c     |  3 ++-\n>  pack-bitmap.c          | 65 +++++++++++++++++++++++++++++++-------------------\n>  pack-bitmap.h          | 12 +++++++++-\n>  reachable.c            |  3 ++-\n>  5 files changed, 57 insertions(+), 29 deletions(-)\n\nAfter 2189649b (pack-bitmap.c: keep track of each layer's type\nbitmaps, 2024-11-19) added <type>_all bitmaps to the bitmap_index\nstruct, this step would need some adjustment, I am afraid.\n\nTaylor Cc'ed.\n\nThanks.\n\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 58a9b161262..8f99e2b4fa8 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -1735,7 +1735,8 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n>  static int add_object_entry_from_bitmap(const struct object_id *oid,\n>  \t\t\t\t\tenum object_type type,\n>  \t\t\t\t\tint flags UNUSED, uint32_t name_hash,\n> -\t\t\t\t\tstruct packed_git *pack, off_t offset)\n> +\t\t\t\t\tstruct packed_git *pack, off_t offset,\n> +\t\t\t\t\tvoid *payload UNUSED)\n>  {\n>  \tdisplay_progress(progress_state, ++nr_seen);\n>  \n> diff --git a/builtin/rev-list.c b/builtin/rev-list.c\n> index bb26bee0d45..1100dd2abe7 100644\n> --- a/builtin/rev-list.c\n> +++ b/builtin/rev-list.c\n> @@ -429,7 +429,8 @@ static int show_object_fast(\n>  \tint exclude UNUSED,\n>  \tuint32_t name_hash UNUSED,\n>  \tstruct packed_git *found_pack UNUSED,\n> -\toff_t found_offset UNUSED)\n> +\toff_t found_offset UNUSED,\n> +\tvoid *payload UNUSED)\n>  {\n>  \tfprintf(stdout, \"%s\\n\", oid_to_hex(oid));\n>  \treturn 1;\n> diff --git a/pack-bitmap.c b/pack-bitmap.c\n> index 6406953d322..fc92e0aae65 100644\n> --- a/pack-bitmap.c\n> +++ b/pack-bitmap.c\n> @@ -1509,50 +1509,45 @@ static void show_extended_objects(struct bitmap_index *bitmap_git,\n>  \t\t    (obj->type == OBJ_TAG && !revs->tag_objects))\n>  \t\t\tcontinue;\n>  \n> -\t\tshow_reach(&obj->oid, obj->type, 0, eindex->hashes[i], NULL, 0);\n> +\t\tshow_reach(&obj->oid, obj->type, 0, eindex->hashes[i], NULL, 0, NULL);\n>  \t}\n>  }\n>  \n> -static void init_type_iterator(struct ewah_iterator *it,\n> -\t\t\t       struct bitmap_index *bitmap_git,\n> -\t\t\t       enum object_type type)\n> +static struct ewah_bitmap *ewah_for_type(struct bitmap_index *bitmap_git,\n> +\t\t\t\t\t enum object_type type)\n>  {\n>  \tswitch (type) {\n>  \tcase OBJ_COMMIT:\n> -\t\tewah_iterator_init(it, bitmap_git->commits);\n> -\t\tbreak;\n> -\n> +\t\treturn bitmap_git->commits;\n>  \tcase OBJ_TREE:\n> -\t\tewah_iterator_init(it, bitmap_git->trees);\n> -\t\tbreak;\n> -\n> +\t\treturn bitmap_git->trees;\n>  \tcase OBJ_BLOB:\n> -\t\tewah_iterator_init(it, bitmap_git->blobs);\n> -\t\tbreak;\n> -\n> +\t\treturn bitmap_git->blobs;\n>  \tcase OBJ_TAG:\n> -\t\tewah_iterator_init(it, bitmap_git->tags);\n> -\t\tbreak;\n> -\n> +\t\treturn bitmap_git->tags;\n>  \tdefault:\n>  \t\tBUG(\"object type %d not stored by bitmap type index\", type);\n> -\t\tbreak;\n>  \t}\n>  }\n>  \n> -static void show_objects_for_type(\n> -\tstruct bitmap_index *bitmap_git,\n> -\tenum object_type object_type,\n> -\tshow_reachable_fn show_reach)\n> +static void init_type_iterator(struct ewah_iterator *it,\n> +\t\t\t       struct bitmap_index *bitmap_git,\n> +\t\t\t       enum object_type type)\n> +{\n> +\tewah_iterator_init(it, ewah_for_type(bitmap_git, type));\n> +}\n> +\n> +static void for_each_bitmapped_object_internal(struct bitmap_index *bitmap_git,\n> +\t\t\t\t\t       struct bitmap *objects,\n> +\t\t\t\t\t       enum object_type object_type,\n> +\t\t\t\t\t       show_reachable_fn show_reach,\n> +\t\t\t\t\t       void *payload)\n>  {\n>  \tsize_t i = 0;\n>  \tuint32_t offset;\n> -\n>  \tstruct ewah_iterator it;\n>  \teword_t filter;\n>  \n> -\tstruct bitmap *objects = bitmap_git->result;\n> -\n>  \tinit_type_iterator(&it, bitmap_git, object_type);\n>  \n>  \tfor (i = 0; i < objects->word_alloc &&\n> @@ -1595,11 +1590,31 @@ static void show_objects_for_type(\n>  \t\t\tif (bitmap_git->hashes)\n>  \t\t\t\thash = get_be32(bitmap_git->hashes + index_pos);\n>  \n> -\t\t\tshow_reach(&oid, object_type, 0, hash, pack, ofs);\n> +\t\t\tshow_reach(&oid, object_type, 0, hash, pack, ofs, payload);\n>  \t\t}\n>  \t}\n>  }\n>  \n> +static void show_objects_for_type(\n> +\tstruct bitmap_index *bitmap_git,\n> +\tenum object_type object_type,\n> +\tshow_reachable_fn show_reach)\n> +{\n> +\tfor_each_bitmapped_object_internal(bitmap_git, bitmap_git->result,\n> +\t\t\t\t\t   object_type, show_reach, NULL);\n> +}\n> +\n> +void for_each_bitmapped_object(struct bitmap_index *bitmap_git,\n> +\t\t\t       enum object_type object_type,\n> +\t\t\t       show_reachable_fn show_reach,\n> +\t\t\t       void *payload)\n> +{\n> +\tstruct bitmap *bitmap = ewah_to_bitmap(ewah_for_type(bitmap_git, object_type));\n> +\tfor_each_bitmapped_object_internal(bitmap_git, bitmap,\n> +\t\t\t\t\t   object_type, show_reach, payload);\n> +\tbitmap_free(bitmap);\n> +}\n> +\n>  static int in_bitmapped_pack(struct bitmap_index *bitmap_git,\n>  \t\t\t     struct object_list *roots)\n>  {\n> diff --git a/pack-bitmap.h b/pack-bitmap.h\n> index d7f4b8b8e95..3368e79ed5a 100644\n> --- a/pack-bitmap.h\n> +++ b/pack-bitmap.h\n> @@ -50,7 +50,8 @@ typedef int (*show_reachable_fn)(\n>  \tint flags,\n>  \tuint32_t hash,\n>  \tstruct packed_git *found_pack,\n> -\toff_t found_offset);\n> +\toff_t found_offset,\n> +\tvoid *payload);\n>  \n>  struct bitmap_index;\n>  \n> @@ -78,6 +79,15 @@ int test_bitmap_pseudo_merges(struct repository *r);\n>  int test_bitmap_pseudo_merge_commits(struct repository *r, uint32_t n);\n>  int test_bitmap_pseudo_merge_objects(struct repository *r, uint32_t n);\n>  \n> +/*\n> + * Iterate through all bitmapped objects of the given type and execute the\n> + * `show_reach` for each of them.\n> + */\n> + void for_each_bitmapped_object(struct bitmap_index *bitmap_git,\n> +\t\t\t       enum object_type object_type,\n> +\t\t\t       show_reachable_fn show_reach,\n> +\t\t\t       void *payload);\n> +\n>  #define GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL \\\n>  \t\"GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL\"\n>  \n> diff --git a/reachable.c b/reachable.c\n> index ecf7ccf5041..dd33c7f07dd 100644\n> --- a/reachable.c\n> +++ b/reachable.c\n> @@ -337,7 +337,8 @@ static int mark_object_seen(const struct object_id *oid,\n>  \t\t\t     int exclude UNUSED,\n>  \t\t\t     uint32_t name_hash UNUSED,\n>  \t\t\t     struct packed_git *found_pack UNUSED,\n> -\t\t\t     off_t found_offset UNUSED)\n> +\t\t\t     off_t found_offset UNUSED,\n> +\t\t\t     void *payload UNUSED)\n>  {\n>  \tstruct object *obj = lookup_object_by_type(the_repository, oid, type);\n>  \tif (!obj)\n"},{"id":"512955","messageId":"Z71qQvlqtoOwDabu@pks.im","threadId":"62984","inReplyTo":"xmqqseo35ic8.fsf@gitster.g","subject":"Re: [PATCH 6/9] pack-bitmap: expose function to iterate over bitmapped objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-25T06:59:14Z","receivedAt":"2025-02-25T06:59:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Feb 24, 2025 at 10:05:27AM -0800, Junio C Hamano wrote:\n> Patrick Steinhardt <ps@pks.im> writes:\n> \n> > Expose a function that allows the caller to iterate over all bitmapped\n> > objects of a specific type. This mechanism allows us to use the object\n> > type-specific bitmaps to enumerate all objects of that type without\n> > having to scan through a complete packfile.\n> >\n> > This functionality will be used in a subsequent commit.\n> >\n> > Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> > ---\n> >  builtin/pack-objects.c |  3 ++-\n> >  builtin/rev-list.c     |  3 ++-\n> >  pack-bitmap.c          | 65 +++++++++++++++++++++++++++++++-------------------\n> >  pack-bitmap.h          | 12 +++++++++-\n> >  reachable.c            |  3 ++-\n> >  5 files changed, 57 insertions(+), 29 deletions(-)\n> \n> After 2189649b (pack-bitmap.c: keep track of each layer's type\n> bitmaps, 2024-11-19) added <type>_all bitmaps to the bitmap_index\n> struct, this step would need some adjustment, I am afraid.\n\nHm, does it? I understand that this commit only makes the bitmaps\naccessible individually per bitmapped packfile, but the bitmap indices\npart of `struct bitmap_index` would continue to be the union of all of\nthose bitmaps. Oh, but that changes in the subsequent commits indeed,\nwhere we start to use an `ewah_or_iterator`.\n\nI see that Taylor's series has been sitting in an unreviewed state for a\ncouple months already. I can review it with the hope of moving it\nforward and can then pull it in as a dependency of this series. But I'll\nwait for him to chime in first to see whether anything changed about its\ncurrent state.\n\nPatrick\n"},{"id":"513029","messageId":"xmqqeczmkljo.fsf@gitster.g","threadId":"62984","inReplyTo":"Z71qQvlqtoOwDabu@pks.im","subject":"Re: [PATCH 6/9] pack-bitmap: expose function to iterate over bitmapped objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-02-25T16:59:23Z","receivedAt":"2025-02-25T16:59:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> I see that Taylor's series has been sitting in an unreviewed state for a\n> couple months already. I can review it with the hope of moving it\n> forward and can then pull it in as a dependency of this series. But I'll\n> wait for him to chime in first to see whether anything changed about its\n> current state.\n\nThanks.  Making sure that one hand knows what other hand's doing\nwould be a good idea.\n"},{"id":"513094","messageId":"874j0graug.fsf@iotcl.com","threadId":"62984","inReplyTo":"20250221-pks-cat-file-object-type-filter-v1-2-0852530888e2@pks.im","subject":"Re: [PATCH 2/9] builtin/cat-file: wire up an option to filter objects","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2025-02-26T15:20:55Z","receivedAt":"2025-02-26T15:21:24Z","isPatch":true,"sender":{"key":"toon@iotcl.com","avatar":"https://avatars.githubusercontent.com/u/121621?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> In batch mode, git-cat-file(1) enumerates all objects and prints them\n> by iterating through both loose and packed objects. This works without\n> considering their reachability at all, and consequently most options to\n> filter objects as they exist in e.g. git-rev-list(1) are not applicable.\n> In some situations it may still be useful though to filter objects based\n> on properties that are inherent to them. This includes the object size\n> as well as its type.\n>\n> Such a filter already exists in git-rev-list(1) with the `--filter=`\n> command line option. While this option supports a couple of filters that\n> are not applicable to our usecase, some of them are quite a neat fit.\n>\n> Wire up the filter as an option for git-cat-file(1). This allows us to\n> reuse the same syntax as in git-rev-list(1) so that we don't have to\n> reinvent the wheel. For now, we die when any of the filter options has\n> been passed by the user, but they will be wired up in subsequent\n> commits.\n>\n> Note that we don't use the same `--filter=` name fo the option as we use\n> in git-rev-list(1). We already have `--filters`, and having both\n> `--filter=` and `--filters` would be quite confusing. Instead, the new\n> option is called `--objects-filter`.\n\nI'm not sure I agree. I would rather have consistency in various\ncommands. Because `--filters` doesn't accept an argument, so I would say\nhaving both `--filters` and `--filter=` is fine. I see in various places\nwe already use `OPT_PARSE_LIST_OBJECTS_FILTER` which defines the option\nas `--filter=`, so it's pretty standard for several commands. I'd\nprefer git-cat-file(1) to follow that as well. But that's my 2 cents.\n\n-- \nToon\n"},{"id":"513095","messageId":"8734g0rary.fsf@iotcl.com","threadId":"62984","inReplyTo":"20250221-pks-cat-file-object-type-filter-v1-3-0852530888e2@pks.im","subject":"Re: [PATCH 3/9] builtin/cat-file: support \"blob:none\" objects filter","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2025-02-26T15:22:25Z","receivedAt":"2025-02-26T15:22:38Z","isPatch":true,"sender":{"key":"toon@iotcl.com","avatar":"https://avatars.githubusercontent.com/u/121621?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> Implement support for the \"blob:none\" filter in git-cat-file(1), which\n> causes us to omit all blobs.\n>\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  Documentation/git-cat-file.adoc |  2 ++\n>  builtin/cat-file.c              | 11 ++++++++++-\n>  t/t1006-cat-file.sh             | 33 +++++++++++++++++++++++++++++++--\n>  3 files changed, 43 insertions(+), 3 deletions(-)\n>\n> diff --git a/Documentation/git-cat-file.adoc b/Documentation/git-cat-file.adoc\n> index 7c1c888079a..c11952d9eca 100644\n> --- a/Documentation/git-cat-file.adoc\n> +++ b/Documentation/git-cat-file.adoc\n> @@ -86,6 +86,8 @@ OPTIONS\n>  \tOmit objects from the list of printed objects. This can only be used in\n>  \tcombination with one of the batched modes. The '<filter-spec>' may be\n>  \tone of the following:\n> ++\n> +The form '--filter=blob:none' omits all blobs.\n\nIf we chose to use `--object-filter`, we need to use it here as well.\nAnd same for the following commits.\n\n\n-- \nToon\n"},{"id":"513096","messageId":"871pvkraqn.fsf@iotcl.com","threadId":"62984","inReplyTo":"20250221-pks-cat-file-object-type-filter-v1-5-0852530888e2@pks.im","subject":"Re: [PATCH 5/9] builtin/cat-file: support \"object:type=\" objects filter","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2025-02-26T15:23:12Z","receivedAt":"2025-02-26T15:23:22Z","isPatch":true,"sender":{"key":"toon@iotcl.com","avatar":"https://avatars.githubusercontent.com/u/121621?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> Implement support for the \"object:type=\" filter in git-cat-file(1),\n> which causes us to omit all objects that don't match the provided object\n> type.\n>\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  Documentation/git-cat-file.adoc | 3 +++\n>  builtin/cat-file.c              | 8 +++++++-\n>  t/t1006-cat-file.sh             | 6 +++++-\n>  3 files changed, 15 insertions(+), 2 deletions(-)\n>\n> diff --git a/Documentation/git-cat-file.adoc b/Documentation/git-cat-file.adoc\n> index 8c474418b52..540d9dffdf9 100644\n> --- a/Documentation/git-cat-file.adoc\n> +++ b/Documentation/git-cat-file.adoc\n> @@ -93,6 +93,9 @@ The form '--filter=blob:limit=<n>[kmg]' omits blobs of size at least n\n>  bytes or units.  n may be zero.  The suffixes k, m, and g can be used\n>  to name units in KiB, MiB, or GiB.  For example, 'blob:limit=1k'\n>  is the same as 'blob:limit=1024'.\n> ++\n> +The form '--filter=object:type=(tag|commit|tree|blob)' omits all objects\n> +which are not of the requested type.\n>  \n>  --path=<path>::\n>  \tFor use with `--textconv` or `--filters`, to allow specifying an object\n> diff --git a/builtin/cat-file.c b/builtin/cat-file.c\n> index f57bf65cb03..b374c2bb104 100644\n> --- a/builtin/cat-file.c\n> +++ b/builtin/cat-file.c\n> @@ -474,7 +474,8 @@ static void batch_object_write(const char *obj_name,\n>  \n>  \t\tif (use_mailmap ||\n>  \t\t    opt->objects_filter.choice == LOFC_BLOB_NONE ||\n> -\t\t    opt->objects_filter.choice == LOFC_BLOB_LIMIT)\n> +\t\t    opt->objects_filter.choice == LOFC_BLOB_LIMIT ||\n> +\t\t    opt->objects_filter.choice == LOFC_OBJECT_TYPE)\n>  \t\t\tdata->info.typep = &data->type;\n>  \t\tif (opt->objects_filter.choice == LOFC_BLOB_LIMIT)\n>  \t\t\tdata->info.sizep = &data->size;\n> @@ -505,6 +506,10 @@ static void batch_object_write(const char *obj_name,\n>  \t\t\t    data->size >= opt->objects_filter.blob_limit_value)\n>  \t\t\t\treturn;\n>  \t\t\tbreak;\n> +\t\tcase LOFC_OBJECT_TYPE:\n> +\t\t\tif (data->type != opt->objects_filter.object_type)\n> +\t\t\t\treturn;\n> +\t\t\tbreak;\n>  \t\tdefault:\n>  \t\t\tBUG(\"unsupported objects filter\");\n\nI see we don't support LOFC_COMBINE, so we won't be supporting repeating\nthe --filter= option, is this intentional? Should we support that too? I\nfeel it would make sense from the start, unless there are good reasons\nnot to?\n\n-- \nToon\n"},{"id":"513140","messageId":"CAOLa=ZRHWSpefDOZzPTZZY1k-Y-w_Vv9rri4WkGDy4NRVCW4HQ@mail.gmail.com","threadId":"62984","inReplyTo":"20250221-pks-cat-file-object-type-filter-v1-2-0852530888e2@pks.im","subject":"Re: [PATCH 2/9] builtin/cat-file: wire up an option to filter objects","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-02-27T11:20:40Z","receivedAt":"2025-02-27T11:20:42Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> In batch mode, git-cat-file(1) enumerates all objects and prints them\n> by iterating through both loose and packed objects. This works without\n> considering their reachability at all, and consequently most options to\n> filter objects as they exist in e.g. git-rev-list(1) are not applicable.\n> In some situations it may still be useful though to filter objects based\n> on properties that are inherent to them. This includes the object size\n> as well as its type.\n>\n> Such a filter already exists in git-rev-list(1) with the `--filter=`\n> command line option. While this option supports a couple of filters that\n> are not applicable to our usecase, some of them are quite a neat fit.\n>\n> Wire up the filter as an option for git-cat-file(1). This allows us to\n> reuse the same syntax as in git-rev-list(1) so that we don't have to\n> reinvent the wheel. For now, we die when any of the filter options has\n> been passed by the user, but they will be wired up in subsequent\n> commits.\n>\n> Note that we don't use the same `--filter=` name fo the option as we use\n\ns/fo/for\n\n> in git-rev-list(1). We already have `--filters`, and having both\n> `--filter=` and `--filters` would be quite confusing. Instead, the new\n> option is called `--objects-filter`.\n>\n> Further note that the filters that we are about to introduce don't\n> significantly speed up the runtime of git-cat-file(1). While we can skip\n> emitting a lot of objects in case they are uninteresting to us, the\n> majority of time is spent reading the packfile, which is bottlenecked by\n> I/O and not the processor. This will change though once we start to make\n> use of bitmaps, which will allow us to skip reading the whole packfile.\n>\n\n[snip]\n"},{"id":"513141","messageId":"CAOLa=ZR9yJqvDwnJ2t9y0MRNQfZFpWPssgsE1QrpbhEzDBB3Og@mail.gmail.com","threadId":"62984","inReplyTo":"20250221-pks-cat-file-object-type-filter-v1-3-0852530888e2@pks.im","subject":"Re: [PATCH 3/9] builtin/cat-file: support \"blob:none\" objects filter","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-02-27T11:26:45Z","receivedAt":"2025-02-27T11:26:47Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> Implement support for the \"blob:none\" filter in git-cat-file(1), which\n> causes us to omit all blobs.\n>\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  Documentation/git-cat-file.adoc |  2 ++\n>  builtin/cat-file.c              | 11 ++++++++++-\n>  t/t1006-cat-file.sh             | 33 +++++++++++++++++++++++++++++++--\n>  3 files changed, 43 insertions(+), 3 deletions(-)\n>\n> diff --git a/Documentation/git-cat-file.adoc b/Documentation/git-cat-file.adoc\n> index 7c1c888079a..c11952d9eca 100644\n> --- a/Documentation/git-cat-file.adoc\n> +++ b/Documentation/git-cat-file.adoc\n> @@ -86,6 +86,8 @@ OPTIONS\n>  \tOmit objects from the list of printed objects. This can only be used in\n>  \tcombination with one of the batched modes. The '<filter-spec>' may be\n>  \tone of the following:\n> ++\n> +The form '--filter=blob:none' omits all blobs.\n>\n\nShouldn't this be '--objects-filter' ?\n\n>  --path=<path>::\n>  \tFor use with `--textconv` or `--filters`, to allow specifying an object\n> diff --git a/builtin/cat-file.c b/builtin/cat-file.c\n> index 723644fbba8..8e5572ba43e 100644\n> --- a/builtin/cat-file.c\n> +++ b/builtin/cat-file.c\n> @@ -472,7 +472,8 @@ static void batch_object_write(const char *obj_name,\n>  \tif (!data->skip_object_info) {\n>  \t\tint ret;\n>\n> -\t\tif (use_mailmap)\n> +\t\tif (use_mailmap ||\n> +\t\t    opt->objects_filter.choice == LOFC_BLOB_NONE)\n\nSo since we support selective filters, we'd have to add this type only\nfor those filters. In other words, there is no generic way to do this.\n\n>  \t\t\tdata->info.typep = &data->type;\n>\n>  \t\tif (pack)\n> @@ -492,6 +493,10 @@ static void batch_object_write(const char *obj_name,\n>  \t\tswitch (opt->objects_filter.choice) {\n>  \t\tcase LOFC_DISABLED:\n>  \t\t\tbreak;\n> +\t\tcase LOFC_BLOB_NONE:\n> +\t\t\tif (data->type == OBJ_BLOB)\n> +\t\t\t\treturn;\n> +\t\t\tbreak;\n>  \t\tdefault:\n>  \t\t\tBUG(\"unsupported objects filter\");\n>  \t\t}\n> @@ -1032,6 +1037,10 @@ int cmd_cat_file(int argc,\n>  \tswitch (batch.objects_filter.choice) {\n>  \tcase LOFC_DISABLED:\n>  \t\tbreak;\n> +\tcase LOFC_BLOB_NONE:\n> +\t\tif (!batch.enabled)\n> +\t\t\tusage(_(\"objects filter only supported in batch mode\"));\n> +\t\tbreak;\n>  \tdefault:\n>  \t\tusagef(_(\"objects filter not supported: '%s'\"),\n>  \t\t       list_object_filter_config_name(batch.objects_filter.choice));\n> diff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\n> index 48840a13561..97533225982 100755\n> --- a/t/t1006-cat-file.sh\n> +++ b/t/t1006-cat-file.sh\n> @@ -1354,7 +1354,22 @@ test_expect_success PERL '--batch-command info is unbuffered by default' '\n>  '\n>\n>  test_expect_success 'setup for objects filter' '\n> -\tgit init repo\n> +\tgit init repo &&\n> +\t(\n> +\t\t# Seed the repository with three different sets of objects:\n> +\t\t#\n> +\t\t#   - The first set is fully packed and has a bitmap.\n> +\t\t#   - The second set is packed, but has no bitmap.\n> +\t\t#   - The third set is loose.\n> +\t\t#\n> +\t\t# This ensures that we cover all these types as expected.\n> +\t\tcd repo &&\n> +\t\ttest_commit first &&\n> +\t\tgit repack -Adb &&\n> +\t\ttest_commit second &&\n> +\t\tgit repack -d &&\n> +\t\ttest_commit third\n> +\t)\n>  '\n>\n>  test_expect_success 'objects filter with unknown option' '\n> @@ -1365,7 +1380,7 @@ test_expect_success 'objects filter with unknown option' '\n>  \ttest_cmp expect err\n>  '\n>\n> -for option in blob:none blob:limit=1 object:type=tag sparse:oid=1234 tree:1 sparse:path=x\n> +for option in blob:limit=1 object:type=tag sparse:oid=1234 tree:1 sparse:path=x\n>  do\n>  \ttest_expect_success \"objects filter with unsupported option $option\" '\n>  \t\tcase \"$option\" in\n> @@ -1385,4 +1400,18 @@ do\n>  \t'\n>  done\n>\n> +test_objects_filter () {\n> +\tfilter=\"$1\"\n> +\n> +\ttest_expect_success \"objects filter: $filter\" '\n> +\t\tgit -C repo cat-file --batch-check=\"%(objectname)\" --batch-all-objects --objects-filter=\"$filter\" >actual &&\n> +\t\tsort actual >actual.sorted &&\n> +\t\tgit -C repo rev-list --objects --no-object-names --all --filter=\"$filter\" --filter-provided-objects >expect &&\n> +\t\tsort expect >expect.sorted &&\n> +\t\ttest_cmp expect.sorted actual.sorted\n> +\t'\n> +}\n> +\n> +test_objects_filter \"blob:none\"\n> +\n\nNice, this builds up for the upcoming commits too.\n\n>  test_done\n>\n> --\n> 2.48.1.683.gf705b3209c.dirty\n"},{"id":"513142","messageId":"CAOLa=ZTbhar6Qyf9Q1r1cOHpVFCWmUySg57wNgoKgSCwgEtAKQ@mail.gmail.com","threadId":"62984","inReplyTo":"20250221-pks-cat-file-object-type-filter-v1-9-0852530888e2@pks.im","subject":"Re: [PATCH 9/9] builtin/cat-file: use bitmaps to efficiently filter by object type","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-02-27T11:38:18Z","receivedAt":"2025-02-27T11:38:20Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> While it is now possible to filter objects by type, this mechanism is\n> for now mostly a convenience. Most importantly, we still have to iterate\n> through the whole packfile to find all objects of a specific type. This\n> can be prohibitively expensive depending on the size of the packfiles.\n>\n> It isn't really possible to do better than this when only considering a\n> packfile itself, as the order of objects is not fixed. But when we have\n> a packfile with a corresponding bitmap, either because the packfile\n> itself has one or because the multi-pack index has a bitmap for it, then\n> we can use these bitmaps to improve the runtime.\n>\n> While bitmaps are typically used to compute reachability of objects,\n> they also contain one bitmap per object type encodes which object has\n\nperhaps s/type encodes/type that encodes/\n\n> what type. So instead of reading through the whole packfile(s), we can\n> use the bitmaps and iterate through the type-specific bitmap. Typically,\n> only a subset of packfiles will have a bitmap. But this isn't really\n> much of a problem: we can use bitmaps when available, and then use the\n> non-bitmap walk for every packfile that isn't covered by one.\n>\n> Overall, this leads to quite a significant speedup depending on how many\n> objects of a certain type exist. The following benchmarks have been\n> executed in the Chromium repository, which has a 50GB packfile with\n> almost 25 million objects:\n>\n>     Benchmark 1: git cat-file --batch-check --batch-all-objects --unordered --buffer --no-objects-filter\n>       Time (mean ± σ):     82.806 s ±  6.363 s    [User: 30.956 s, System: 8.264 s]\n>       Range (min … max):   73.936 s … 89.690 s    10 runs\n>\n>     Benchmark 2: git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=object:type=tag\n>       Time (mean ± σ):      20.8 ms ±   1.3 ms    [User: 6.1 ms, System: 14.5 ms]\n>       Range (min … max):    18.2 ms …  23.6 ms    127 runs\n>\n>     Benchmark 3: git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=object:type=commit\n>       Time (mean ± σ):      1.551 s ±  0.008 s    [User: 1.401 s, System: 0.147 s]\n>       Range (min … max):    1.541 s …  1.566 s    10 runs\n>\n>     Benchmark 4: git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=object:type=tree\n>       Time (mean ± σ):     11.169 s ±  0.046 s    [User: 10.076 s, System: 1.063 s]\n>       Range (min … max):   11.114 s … 11.245 s    10 runs\n>\n>     Benchmark 5: git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=object:type=blob\n>       Time (mean ± σ):     67.342 s ±  3.368 s    [User: 20.318 s, System: 7.787 s]\n>       Range (min … max):   62.836 s … 73.618 s    10 runs\n>\n>     Benchmark 6: git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=blob:none\n>       Time (mean ± σ):     13.032 s ±  0.072 s    [User: 11.638 s, System: 1.368 s]\n>       Range (min … max):   12.960 s … 13.199 s    10 runs\n>\n>     Summary\n>       git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=object:type=tag\n>        74.75 ± 4.61 times faster than git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=object:type=commit\n>       538.17 ± 33.17 times faster than git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=object:type=tree\n>       627.98 ± 38.77 times faster than git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=blob:none\n>      3244.93 ± 257.23 times faster than git cat-file --batch-check --batch-all-objects --unordered --buffer --objects-filter=object:type=blob\n>      3990.07 ± 392.72 times faster than git cat-file --batch-check --batch-all-objects --unordered --buffer --no-objects-filter\n>\n> The first benchmark is mostly equivalent in runtime compared to all the\n> others without the bitmap-optimization introduced in this commit. What\n> is noticeable in the benchmarks is that we're I/O-bound, not CPU-bound,\n> as can be seen from the user/system runtimes, which is often way lower\n> than the overall benchmarked runtime.\n>\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  builtin/cat-file.c | 55 +++++++++++++++++++++++++++++++++++++++++++++++++-----\n>  1 file changed, 50 insertions(+), 5 deletions(-)\n>\n> diff --git a/builtin/cat-file.c b/builtin/cat-file.c\n> index 25d5429e391..9021fd52f30 100644\n> --- a/builtin/cat-file.c\n> +++ b/builtin/cat-file.c\n> @@ -21,6 +21,7 @@\n>  #include \"streaming.h\"\n>  #include \"oid-array.h\"\n>  #include \"packfile.h\"\n> +#include \"pack-bitmap.h\"\n>  #include \"object-file.h\"\n>  #include \"object-name.h\"\n>  #include \"object-store-ll.h\"\n> @@ -805,7 +806,20 @@ static int batch_one_object_packed(const struct object_id *oid,\n>  \t\t\t\t payload->payload);\n>  }\n>\n> -static void batch_each_object(for_each_object_fn callback,\n> +static int batch_one_object_bitmapped(const struct object_id *oid,\n> +\t\t\t\t      enum object_type type UNUSED,\n> +\t\t\t\t      int flags UNUSED,\n> +\t\t\t\t      uint32_t hash UNUSED,\n> +\t\t\t\t      struct packed_git *pack,\n> +\t\t\t\t      off_t offset,\n> +\t\t\t\t      void *_payload)\n> +{\n> +\tstruct for_each_object_payload *payload = _payload;\n> +\treturn payload->callback(oid, pack, offset, payload->payload);\n> +}\n> +\n> +static void batch_each_object(struct batch_options *opt,\n> +\t\t\t      for_each_object_fn callback,\n>  \t\t\t      unsigned flags,\n>  \t\t\t      void *_payload)\n>  {\n> @@ -813,9 +827,40 @@ static void batch_each_object(for_each_object_fn callback,\n>  \t\t.callback = callback,\n>  \t\t.payload = _payload,\n>  \t};\n> +\tstruct bitmap_index *bitmap = prepare_bitmap_git(the_repository);\n> +\n>  \tfor_each_loose_object(batch_one_object_loose, &payload, 0);\n> -\tfor_each_packed_object(the_repository, batch_one_object_packed,\n> -\t\t\t       &payload, flags);\n> +\n> +\tif (bitmap &&\n> +\t    (opt->objects_filter.choice == LOFC_OBJECT_TYPE ||\n> +\t     opt->objects_filter.choice == LOFC_BLOB_NONE)) {\n> +\t\tstruct packed_git *pack;\n> +\n> +\t\tif (opt->objects_filter.choice == LOFC_OBJECT_TYPE) {\n> +\t\t\tfor_each_bitmapped_object(bitmap, opt->objects_filter.object_type,\n> +\t\t\t\t\t\t  batch_one_object_bitmapped, &payload);\n> +\t\t} else {\n\nNit: while this can be derived from the if statement above, it would be\nmore readable if this was `if else (opt->objects_filter.choice ==\nLOFC_BLOB_NONE)`\n\n> +\t\t\tfor_each_bitmapped_object(bitmap, OBJ_COMMIT,\n> +\t\t\t\t\t\t  batch_one_object_bitmapped, &payload);\n> +\t\t\tfor_each_bitmapped_object(bitmap, OBJ_TAG,\n> +\t\t\t\t\t\t  batch_one_object_bitmapped, &payload);\n> +\t\t\tfor_each_bitmapped_object(bitmap, OBJ_TREE,\n> +\t\t\t\t\t\t  batch_one_object_bitmapped, &payload);\n> +\t\t}\n> +\n> +\t\tfor (pack = get_all_packs(the_repository); pack; pack = pack->next) {\n> +\t\t\tif (bitmap_index_contains_pack(bitmap, pack) ||\n> +\t\t\t    open_pack_index(pack))\n> +\t\t\t\tcontinue;\n> +\t\t\tfor_each_object_in_pack(pack, batch_one_object_packed,\n> +\t\t\t\t\t\t&payload, flags);\n> +\t\t}\n> +\t} else {\n> +\t\tfor_each_packed_object(the_repository, batch_one_object_packed,\n> +\t\t\t\t       &payload, flags);\n> +\t}\n> +\n> +\tfree_bitmap_index(bitmap);\n>  }\n>\n>  static int batch_objects(struct batch_options *opt)\n> @@ -872,14 +917,14 @@ static int batch_objects(struct batch_options *opt)\n>\n>  \t\t\tcb.seen = &seen;\n>\n> -\t\t\tbatch_each_object(batch_unordered_object,\n> +\t\t\tbatch_each_object(opt, batch_unordered_object,\n>  \t\t\t\t\t  FOR_EACH_OBJECT_PACK_ORDER, &cb);\n>\n>  \t\t\toidset_clear(&seen);\n>  \t\t} else {\n>  \t\t\tstruct oid_array sa = OID_ARRAY_INIT;\n>\n> -\t\t\tbatch_each_object(collect_object, 0, &sa);\n> +\t\t\tbatch_each_object(opt, collect_object, 0, &sa);\n>  \t\t\toid_array_for_each_unique(&sa, batch_object_cb, &cb);\n>\n>  \t\t\toid_array_clear(&sa);\n>\n> --\n> 2.48.1.683.gf705b3209c.dirty\n"},{"id":"513196","messageId":"Z8Dz6EkIpr/g3vuY@nand.local","threadId":"62984","inReplyTo":"xmqqseo35ic8.fsf@gitster.g","subject":"Re: [PATCH 6/9] pack-bitmap: expose function to iterate over bitmapped objects","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-02-27T23:23:20Z","receivedAt":"2025-02-27T23:23:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Feb 24, 2025 at 10:05:27AM -0800, Junio C Hamano wrote:\n> Patrick Steinhardt <ps@pks.im> writes:\n>\n> > Expose a function that allows the caller to iterate over all bitmapped\n> > objects of a specific type. This mechanism allows us to use the object\n> > type-specific bitmaps to enumerate all objects of that type without\n> > having to scan through a complete packfile.\n> >\n> > This functionality will be used in a subsequent commit.\n> >\n> > Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> > ---\n> >  builtin/pack-objects.c |  3 ++-\n> >  builtin/rev-list.c     |  3 ++-\n> >  pack-bitmap.c          | 65 +++++++++++++++++++++++++++++++-------------------\n> >  pack-bitmap.h          | 12 +++++++++-\n> >  reachable.c            |  3 ++-\n> >  5 files changed, 57 insertions(+), 29 deletions(-)\n>\n> After 2189649b (pack-bitmap.c: keep track of each layer's type\n> bitmaps, 2024-11-19) added <type>_all bitmaps to the bitmap_index\n> struct, this step would need some adjustment, I am afraid.\n>\n> Taylor Cc'ed.\n\nThanks, I was going to respond with the same thing.\n\nI was going to suggest leaving that function as-is to prevent future\nbreakage and/or a messy integration into 'seen'. But stepping back I am\nnot sure I understand the purpose of this commit in the first place.\n\nIt looks like the aim here is to introduce a function which executes a\ncallback for each object of some type in a bitmap. That's a thin wrapper\nover the ewah_iterator, but it's not clear why we need a wrapper around\nthat function since it is internal to pack-bitmap.c. Likewise, this is a\nperformance critical area, so I am not sure I'm in favor of adding a\nfunction pointer to a hot path which executes once per object for some\nobject type.\n\nThanks,\nTaylor\n"},{"id":"513197","messageId":"Z8D0p8oKEmMqmzrJ@nand.local","threadId":"62984","inReplyTo":"Z71qQvlqtoOwDabu@pks.im","subject":"Re: [PATCH 6/9] pack-bitmap: expose function to iterate over bitmapped objects","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-02-27T23:26:31Z","receivedAt":"2025-02-27T23:26:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Feb 25, 2025 at 07:59:14AM +0100, Patrick Steinhardt wrote:\n> On Mon, Feb 24, 2025 at 10:05:27AM -0800, Junio C Hamano wrote:\n> > Patrick Steinhardt <ps@pks.im> writes:\n> >\n> > > Expose a function that allows the caller to iterate over all bitmapped\n> > > objects of a specific type. This mechanism allows us to use the object\n> > > type-specific bitmaps to enumerate all objects of that type without\n> > > having to scan through a complete packfile.\n> > >\n> > > This functionality will be used in a subsequent commit.\n> > >\n> > > Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> > > ---\n> > >  builtin/pack-objects.c |  3 ++-\n> > >  builtin/rev-list.c     |  3 ++-\n> > >  pack-bitmap.c          | 65 +++++++++++++++++++++++++++++++-------------------\n> > >  pack-bitmap.h          | 12 +++++++++-\n> > >  reachable.c            |  3 ++-\n> > >  5 files changed, 57 insertions(+), 29 deletions(-)\n> >\n> > After 2189649b (pack-bitmap.c: keep track of each layer's type\n> > bitmaps, 2024-11-19) added <type>_all bitmaps to the bitmap_index\n> > struct, this step would need some adjustment, I am afraid.\n>\n> Hm, does it? I understand that this commit only makes the bitmaps\n> accessible individually per bitmapped packfile, but the bitmap indices\n> part of `struct bitmap_index` would continue to be the union of all of\n> those bitmaps. Oh, but that changes in the subsequent commits indeed,\n> where we start to use an `ewah_or_iterator`.\n\nThat's right; the ewah_or_iterator is the mechanism by which we can\ncombine multiple \"layers\" of the bitmaps into a single iterator.\n\n(As an aside, that was not the first approach I pursued. Initially the\ncaller was supposed to chase the 'next' pointer of each bitmap and\nenumerate through whatever type iterator they're interested in at each\nlayer. But that was too error-prone, since you have to remember and\nupdate the offset into the pseudo-pack order across multiple layers.)\n\n> I see that Taylor's series has been sitting in an unreviewed state for a\n> couple months already. I can review it with the hope of moving it\n> forward and can then pull it in as a dependency of this series. But I'll\n> wait for him to chime in first to see whether anything changed about its\n> current state.\n\nIt would be great to get some review from you on that series. I know\nthat it has been on Peff's (CC'd) radar for a while, but that he has\nlikewise had a few off-list things to deal with lately as well.\n\nI am still not sold on introducing a callback here, though, and would\nmuch rather see callers interact with the iterator directly.\n\nThanks,\nTaylor\n"},{"id":"513198","messageId":"xmqqo6yn55gv.fsf@gitster.g","threadId":"62984","inReplyTo":"Z8Dz6EkIpr/g3vuY@nand.local","subject":"Re: [PATCH 6/9] pack-bitmap: expose function to iterate over bitmapped objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-02-27T23:32:32Z","receivedAt":"2025-02-27T23:32:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> It looks like the aim here is to introduce a function which executes a\n> callback for each object of some type in a bitmap. That's a thin wrapper\n> over the ewah_iterator, but it's not clear why we need a wrapper around\n> that function since it is internal to pack-bitmap.c. Likewise, this is a\n> performance critical area, so I am not sure I'm in favor of adding a\n> function pointer to a hot path which executes once per object for some\n> object type.\n\nIt internally introduced ewah_for_type(), giving the \"struct\nbitmap_index\" object an abstraction that callers can ask for the\nbitmap for any type the caller wants.  Before the <type>_all bitmaps\nwere introduced, there were one ewah-bitmap per type, so it made\nsense for a caller to ask \"Now, for this bitmap_index, give me the\newah-bitmap for commits\", but with \"commits_all\" added to the\nbitmap_index object, it is no longer clear to me what the answer to\nthat question should be.\n\n"},{"id":"513199","messageId":"Z8D2WUZIu041l0WZ@nand.local","threadId":"62984","inReplyTo":"20250221-pks-cat-file-object-type-filter-v1-7-0852530888e2@pks.im","subject":"Re: [PATCH 7/9] pack-bitmap: introduce function to check whether a pack is bitmapped","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-02-27T23:33:45Z","receivedAt":"2025-02-27T23:33:47Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Feb 21, 2025 at 08:47:32AM +0100, Patrick Steinhardt wrote:\n> Introduce a function that allows us to verify whether a pack is\n> bitmapped or not. This functionality will be used in a subsequent\n> commit.\n>\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  pack-bitmap.c | 15 +++++++++++++++\n>  pack-bitmap.h |  7 +++++++\n>  2 files changed, 22 insertions(+)\n>\n> diff --git a/pack-bitmap.c b/pack-bitmap.c\n> index fc92e0aae65..3cbe5bfe909 100644\n> --- a/pack-bitmap.c\n> +++ b/pack-bitmap.c\n> @@ -658,6 +658,21 @@ struct bitmap_index *prepare_midx_bitmap_git(struct multi_pack_index *midx)\n>  \treturn NULL;\n>  }\n>\n> +int bitmap_index_contains_pack(struct bitmap_index *bitmap, struct packed_git *pack)\n> +{\n> +\tif (bitmap->pack)\n> +\t\treturn bitmap->pack == pack;\n\nThe bitmap_is_midx() function should be useful here. I don't think what\nyou wrote is wrong per-se, but that function is supposed to \"hide\" what\nexactly constitutes a pack versus multi-pack bitmap.\n\n> +\tif (!bitmap->midx->chunk_bitmapped_packs)\n> +\t\treturn 0;\n\nWhat is the purpose of this check? The BTMP chunk was a relatively\nrecent addition, but it came long after multi-pack bitmaps were first\nintroduced. The BTMP chunk is necessary for multi-pack reuse, since it\nindicates what sections of the bitmap's object order correspond to what\npacks.\n\nWith or without a BTMP chunk in the MIDX, a multi-pack bitmap is assumed\nto cover all of the packs in that MIDx. So I think the above check is at\nbest not helpful, and at worst will return incorrect results for\npre-BTMP MIDXs.\n\n> +\tfor (size_t i = 0; i < bitmap->midx->num_packs; i++)\n> +\t\tif (bitmap->midx->packs[i] == pack)\n> +\t\t\treturn 1;\n\nThis part looks good to me. If you end up pulling in the incremental\nMIDX bitmaps series in as a dependency of this one, this will have to be\nrewritten something like:\n\n    for (; bitmap; bitmap = bitmap->base) {\n      if (bitmap_is_midx(bitmap)) {\n        for (size_t i = 0; i < bitmap->midx->num_packs; i++) {\n          if (bitmap->midx->packs[i] == pack)\n            return 1;\n        }\n      } else if (bitmap->pack == pack) {\n        return 1;\n      }\n    }\n    return 0;\n\nWithout pulling in that series as a dependency of this one, I think the\nfunction would just contain the body of the above 'for' loop, but not\nthe loop itself.\n\nThanks,\nTaylor\n"},{"id":"513200","messageId":"Z8D3vsnwuKO05KaK@nand.local","threadId":"62984","inReplyTo":"xmqqo6yn55gv.fsf@gitster.g","subject":"Re: [PATCH 6/9] pack-bitmap: expose function to iterate over bitmapped objects","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-02-27T23:39:42Z","receivedAt":"2025-02-27T23:39:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Feb 27, 2025 at 03:32:32PM -0800, Junio C Hamano wrote:\n> Taylor Blau <me@ttaylorr.com> writes:\n>\n> > It looks like the aim here is to introduce a function which executes a\n> > callback for each object of some type in a bitmap. That's a thin wrapper\n> > over the ewah_iterator, but it's not clear why we need a wrapper around\n> > that function since it is internal to pack-bitmap.c. Likewise, this is a\n> > performance critical area, so I am not sure I'm in favor of adding a\n> > function pointer to a hot path which executes once per object for some\n> > object type.\n>\n> It internally introduced ewah_for_type(), giving the \"struct\n> bitmap_index\" object an abstraction that callers can ask for the\n> bitmap for any type the caller wants.  Before the <type>_all bitmaps\n> were introduced, there were one ewah-bitmap per type, so it made\n> sense for a caller to ask \"Now, for this bitmap_index, give me the\n> ewah-bitmap for commits\", but with \"commits_all\" added to the\n> bitmap_index object, it is no longer clear to me what the answer to\n> that question should be.\n\nI think these are orthogonal. (FWIW, I think the correct answer would be\n\"commits_all\" in that world, but that is definitely out of scope for\nPatrick's immediate concern). In any event, I see that later on in the\nseries it is important for callers to enumerate bitmapped objects of a\ncertain type. So having a callback to do that makes sense.\n\nThanks,\nTaylor\n"},{"id":"513201","messageId":"Z8D5xiAVmWOUJJ1v@nand.local","threadId":"62984","inReplyTo":"20250221-pks-cat-file-object-type-filter-v1-9-0852530888e2@pks.im","subject":"Re: [PATCH 9/9] builtin/cat-file: use bitmaps to efficiently filter by object type","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-02-27T23:48:22Z","receivedAt":"2025-02-27T23:48:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Feb 21, 2025 at 08:47:34AM +0100, Patrick Steinhardt wrote:\n> @@ -813,9 +827,40 @@ static void batch_each_object(for_each_object_fn callback,\n>  \t\t.callback = callback,\n>  \t\t.payload = _payload,\n>  \t};\n> +\tstruct bitmap_index *bitmap = prepare_bitmap_git(the_repository);\n> +\n>  \tfor_each_loose_object(batch_one_object_loose, &payload, 0);\n> -\tfor_each_packed_object(the_repository, batch_one_object_packed,\n> -\t\t\t       &payload, flags);\n> +\n> +\tif (bitmap &&\n> +\t    (opt->objects_filter.choice == LOFC_OBJECT_TYPE ||\n> +\t     opt->objects_filter.choice == LOFC_BLOB_NONE)) {\n\nMakes sense. I think there is one more case here that we could handle,\nwhich is\n\n    opt->objects_filter.choice == LOFC_TREE_DEPTH && opt->objects_filter.depth == 0\n\nwhere we'd just want to show commits.\n\nI am scratching my head on if there is a convenient way to unify this\nlogic with pack-bitmap.c::filter_bitmap(). I think there is, but there\nare a couple of wrinkles:\n\n  - filter_bitmap() is really designed to work with a whole 'struct\n    bitmap', and doesn't know how to deal with an ewah_iterator.\n\n  - traverse_bitmap_commit_list() is designed to provide a way for\n    callers to iterate over the set of objects reachable for some\n    rev-list query.\n\n    There we *do* have good facilities for iterating over an\n    ewah_iterator, which is what you'd want. But that function really\n    wants to have performed a bitmap walk first (see the\n    \"assert(bitmap_git->result)\" call at the beginning of that\n    function).\n\nThe new pieces of batch_each_object() introduced in this patch are\ntantalizingly close to much of the existing logic in pack-bitmap.c. I\nthink there is a way to unify them by introducing a way to traverse over\nthe bitmap as a whole as if bitmap_git->result were the all-1s bitmap.\n\nThanks,\nTaylor\n"},{"id":"513271","messageId":"Z8GVKLee75GDG2S4@pks.im","threadId":"62984","inReplyTo":"874j0graug.fsf@iotcl.com","subject":"Re: [PATCH 2/9] builtin/cat-file: wire up an option to filter objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T10:51:20Z","receivedAt":"2025-02-28T10:51:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Feb 26, 2025 at 04:20:55PM +0100, Toon Claes wrote:\n> Patrick Steinhardt <ps@pks.im> writes:\n> \n> > In batch mode, git-cat-file(1) enumerates all objects and prints them\n> > by iterating through both loose and packed objects. This works without\n> > considering their reachability at all, and consequently most options to\n> > filter objects as they exist in e.g. git-rev-list(1) are not applicable.\n> > In some situations it may still be useful though to filter objects based\n> > on properties that are inherent to them. This includes the object size\n> > as well as its type.\n> >\n> > Such a filter already exists in git-rev-list(1) with the `--filter=`\n> > command line option. While this option supports a couple of filters that\n> > are not applicable to our usecase, some of them are quite a neat fit.\n> >\n> > Wire up the filter as an option for git-cat-file(1). This allows us to\n> > reuse the same syntax as in git-rev-list(1) so that we don't have to\n> > reinvent the wheel. For now, we die when any of the filter options has\n> > been passed by the user, but they will be wired up in subsequent\n> > commits.\n> >\n> > Note that we don't use the same `--filter=` name fo the option as we use\n> > in git-rev-list(1). We already have `--filters`, and having both\n> > `--filter=` and `--filters` would be quite confusing. Instead, the new\n> > option is called `--objects-filter`.\n> \n> I'm not sure I agree. I would rather have consistency in various\n> commands. Because `--filters` doesn't accept an argument, so I would say\n> having both `--filters` and `--filter=` is fine. I see in various places\n> we already use `OPT_PARSE_LIST_OBJECTS_FILTER` which defines the option\n> as `--filter=`, so it's pretty standard for several commands. I'd\n> prefer git-cat-file(1) to follow that as well. But that's my 2 cents.\n\nI'll wait for a third party to chime in as a tie breaker here :) I'm not\nfeeling overly strong about it, but still think that it's just too easy\nto get wrong when those options are so extremely similarly named.\n\nPatrick\n"},{"id":"513272","messageId":"Z8GVLrlHY-hwOJeI@pks.im","threadId":"62984","inReplyTo":"871pvkraqn.fsf@iotcl.com","subject":"Re: [PATCH 5/9] builtin/cat-file: support \"object:type=\" objects filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T10:51:26Z","receivedAt":"2025-02-28T10:51:29Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Feb 26, 2025 at 04:23:12PM +0100, Toon Claes wrote:\n> Patrick Steinhardt <ps@pks.im> writes:\n> > diff --git a/builtin/cat-file.c b/builtin/cat-file.c\n> > index f57bf65cb03..b374c2bb104 100644\n> > --- a/builtin/cat-file.c\n> > +++ b/builtin/cat-file.c\n> > @@ -474,7 +474,8 @@ static void batch_object_write(const char *obj_name,\n> >  \n> >  \t\tif (use_mailmap ||\n> >  \t\t    opt->objects_filter.choice == LOFC_BLOB_NONE ||\n> > -\t\t    opt->objects_filter.choice == LOFC_BLOB_LIMIT)\n> > +\t\t    opt->objects_filter.choice == LOFC_BLOB_LIMIT ||\n> > +\t\t    opt->objects_filter.choice == LOFC_OBJECT_TYPE)\n> >  \t\t\tdata->info.typep = &data->type;\n> >  \t\tif (opt->objects_filter.choice == LOFC_BLOB_LIMIT)\n> >  \t\t\tdata->info.sizep = &data->size;\n> > @@ -505,6 +506,10 @@ static void batch_object_write(const char *obj_name,\n> >  \t\t\t    data->size >= opt->objects_filter.blob_limit_value)\n> >  \t\t\t\treturn;\n> >  \t\t\tbreak;\n> > +\t\tcase LOFC_OBJECT_TYPE:\n> > +\t\t\tif (data->type != opt->objects_filter.object_type)\n> > +\t\t\t\treturn;\n> > +\t\t\tbreak;\n> >  \t\tdefault:\n> >  \t\t\tBUG(\"unsupported objects filter\");\n> \n> I see we don't support LOFC_COMBINE, so we won't be supporting repeating\n> the --filter= option, is this intentional? Should we support that too? I\n> feel it would make sense from the start, unless there are good reasons\n> not to?\n\nI think the usefulness of LOFC_COMBINE is quite restricted in our case\nbecause we only support a subset of filters in the first place. There is\nonly a single combination that does make sense: `blob:limit` plus\n`object:type=blob`. All the other combinations are useless as they only\nfilter based on the object type, and thus they would yield the empty\nset.\n\nSo given that this isn't that useful and given that it does add quite a\nbit of complexity to support I decided to not support it for now, also\nbecause I don't have any usecase for it.\n\nPatrick\n"},{"id":"513273","messageId":"Z8GV2k64arkWGpFA@pks.im","threadId":"62984","inReplyTo":"Z8D0p8oKEmMqmzrJ@nand.local","subject":"Re: [PATCH 6/9] pack-bitmap: expose function to iterate over bitmapped objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T10:54:18Z","receivedAt":"2025-02-28T10:54:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Feb 27, 2025 at 06:26:31PM -0500, Taylor Blau wrote:\n> On Tue, Feb 25, 2025 at 07:59:14AM +0100, Patrick Steinhardt wrote:\n> > I see that Taylor's series has been sitting in an unreviewed state for a\n> > couple months already. I can review it with the hope of moving it\n> > forward and can then pull it in as a dependency of this series. But I'll\n> > wait for him to chime in first to see whether anything changed about its\n> > current state.\n> \n> It would be great to get some review from you on that series. I know\n> that it has been on Peff's (CC'd) radar for a while, but that he has\n> likewise had a few off-list things to deal with lately as well.\n\nI've done a first review today. I'll delay my patch series a bit until\nyour series looks like it is close to landing.\n\nThanks!\n\nPatrick\n"},{"id":"513291","messageId":"xmqqikou2ccn.fsf@gitster.g","threadId":"62984","inReplyTo":"Z8GVKLee75GDG2S4@pks.im","subject":"Re: [PATCH 2/9] builtin/cat-file: wire up an option to filter objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-02-28T17:44:24Z","receivedAt":"2025-02-28T17:44:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n>> > Note that we don't use the same `--filter=` name fo the option as we use\n>> > in git-rev-list(1). We already have `--filters`, and having both\n>> > `--filter=` and `--filters` would be quite confusing. Instead, the new\n>> > option is called `--objects-filter`.\n>> \n>> I'm not sure I agree. I would rather have consistency in various\n>> commands. Because `--filters` doesn't accept an argument, so I would say\n>> having both `--filters` and `--filter=` is fine. I see in various places\n>> we already use `OPT_PARSE_LIST_OBJECTS_FILTER` which defines the option\n>> as `--filter=`, so it's pretty standard for several commands. I'd\n>> prefer git-cat-file(1) to follow that as well. But that's my 2 cents.\n>\n> I'll wait for a third party to chime in as a tie breaker here :) I'm not\n> feeling overly strong about it, but still think that it's just too easy\n> to get wrong when those options are so extremely similarly named.\n\n$ git grep '^[^a-z]*--filter' Documentation/\nDocumentation/config/gc.adoc:\t`--filter=<filter-spec>` option of linkgit:git-repack[1].\nDocumentation/git-cat-file.adoc:--filters::\nDocumentation/git-cat-file.adoc:\t`--filters`.\nDocumentation/git-clone.adoc:\t  [--filter=<filter-spec>] [--also-filter-submodules]] [--] <repository>\nDocumentation/git-clone.adoc:`--filter=<filter-spec>`::\nDocumentation/git-clone.adoc:\t`--filter=blob:limit=<size>` will filter out all blobs of size\nDocumentation/git-pack-objects.adoc:--filter=<filter-spec>::\nDocumentation/git-repack.adoc:--filter=<filter-spec>::\nDocumentation/git-repack.adoc:--filter-to=<dir>::\nDocumentation/rev-list-options.adoc:--filter=<filter-spec>::\nDocumentation/rev-list-options.adoc:--filter-provided-objects::\nDocumentation/rev-list-options.adoc:--filter-print-omitted::\n\nThe above does makes it look that whoever called their invention\n\"--filters\" when they added it to \"cat-file\" wasn't paying attention\nto make things consistent, but that is not the case.  The word\n\"filter\" in the context of existing feature set of \"cat-file\" has\nALWAYS refered to the act of applying the \"smudge\" filter chain to\nexternalize an internal \"clean\" blob object contents for the working\ntree representation.  We should thank that somebody for not using\nand squatting on a shorter and sweeter \"--filter\" ;-)\n\n\"cat-file\" should call the feature \"--filter=<filter-spec>\" like\neverybody else does, or the feature should not be added to\n\"cat-file\" at all.  Unless we are willing to rename \"--filter=\"\noptions for _all_ existing commands to \"--object-filter=\", that is.\n\nIn retrospect, such a longer and more explicit name may have been\nnicer.  But given that all users of the \"--filter=<filter-spec>\" are\nabout object transfer, it is understandable that we didn't invoke\ndeliberate redundancy when naming the option.  Historically,\n\"cat-file\" has always been \"give me the contents of the object I\nname\", and never about \"I may ask about many objects but do not\nanswer requests for objects chosen by these criteria\", so it also is\nunderstandable that we didn't redundantly say \"--contents-filter\",\ntoo.\n\nAm I third-party enough?\n"},{"id":"513396","messageId":"Z8WHMV_5rxKjlkhE@pks.im","threadId":"62984","inReplyTo":"xmqqikou2ccn.fsf@gitster.g","subject":"Re: [PATCH 2/9] builtin/cat-file: wire up an option to filter objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-03T10:40:49Z","receivedAt":"2025-03-03T10:40:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Feb 28, 2025 at 09:44:24AM -0800, Junio C Hamano wrote:\n> Patrick Steinhardt <ps@pks.im> writes:\n> \n> >> > Note that we don't use the same `--filter=` name fo the option as we use\n> >> > in git-rev-list(1). We already have `--filters`, and having both\n> >> > `--filter=` and `--filters` would be quite confusing. Instead, the new\n> >> > option is called `--objects-filter`.\n> >> \n> >> I'm not sure I agree. I would rather have consistency in various\n> >> commands. Because `--filters` doesn't accept an argument, so I would say\n> >> having both `--filters` and `--filter=` is fine. I see in various places\n> >> we already use `OPT_PARSE_LIST_OBJECTS_FILTER` which defines the option\n> >> as `--filter=`, so it's pretty standard for several commands. I'd\n> >> prefer git-cat-file(1) to follow that as well. But that's my 2 cents.\n> >\n> > I'll wait for a third party to chime in as a tie breaker here :) I'm not\n> > feeling overly strong about it, but still think that it's just too easy\n> > to get wrong when those options are so extremely similarly named.\n> \n> $ git grep '^[^a-z]*--filter' Documentation/\n> Documentation/config/gc.adoc:\t`--filter=<filter-spec>` option of linkgit:git-repack[1].\n> Documentation/git-cat-file.adoc:--filters::\n> Documentation/git-cat-file.adoc:\t`--filters`.\n> Documentation/git-clone.adoc:\t  [--filter=<filter-spec>] [--also-filter-submodules]] [--] <repository>\n> Documentation/git-clone.adoc:`--filter=<filter-spec>`::\n> Documentation/git-clone.adoc:\t`--filter=blob:limit=<size>` will filter out all blobs of size\n> Documentation/git-pack-objects.adoc:--filter=<filter-spec>::\n> Documentation/git-repack.adoc:--filter=<filter-spec>::\n> Documentation/git-repack.adoc:--filter-to=<dir>::\n> Documentation/rev-list-options.adoc:--filter=<filter-spec>::\n> Documentation/rev-list-options.adoc:--filter-provided-objects::\n> Documentation/rev-list-options.adoc:--filter-print-omitted::\n> \n> The above does makes it look that whoever called their invention\n> \"--filters\" when they added it to \"cat-file\" wasn't paying attention\n> to make things consistent, but that is not the case.  The word\n> \"filter\" in the context of existing feature set of \"cat-file\" has\n> ALWAYS refered to the act of applying the \"smudge\" filter chain to\n> externalize an internal \"clean\" blob object contents for the working\n> tree representation.  We should thank that somebody for not using\n> and squatting on a shorter and sweeter \"--filter\" ;-)\n> \n> \"cat-file\" should call the feature \"--filter=<filter-spec>\" like\n> everybody else does, or the feature should not be added to\n> \"cat-file\" at all.  Unless we are willing to rename \"--filter=\"\n> options for _all_ existing commands to \"--object-filter=\", that is.\n> \n> In retrospect, such a longer and more explicit name may have been\n> nicer.  But given that all users of the \"--filter=<filter-spec>\" are\n> about object transfer, it is understandable that we didn't invoke\n> deliberate redundancy when naming the option.  Historically,\n> \"cat-file\" has always been \"give me the contents of the object I\n> name\", and never about \"I may ask about many objects but do not\n> answer requests for objects chosen by these criteria\", so it also is\n> understandable that we didn't redundantly say \"--contents-filter\",\n> too.\n> \n> Am I third-party enough?\n\nYup :) I'll adapt then, thanks for chiming in!\n\nPatrick\n"},{"id":"515502","messageId":"20250402-pks-cat-file-object-type-filter-v3-0-4da6bb21871c@pks.im","threadId":"62984","inReplyTo":"20250221-pks-cat-file-object-type-filter-v1-0-0852530888e2@pks.im","subject":"[PATCH v3 00/11] builtin/cat-file: allow filtering objects in batch mode","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-02T11:13:35Z","receivedAt":"2025-04-02T11:13:41Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nat GitLab, we sometimes have the need to list all objects regardless of\ntheir reachability. We use git-cat-file(1) with `--batch-all-objects` to\ndo this, and typically this is quite a good fit. In some cases though,\nwe only want to list objects of a specific type, where we then basically\nhave the following pipeline:\n\n    git cat-file --batch-all-objects --batch-check='%(objecttype) %(objectname)' |\n    grep '^commit ' |\n    cut -d' ' -f2 |\n    git cat-file --batch\n\nThis works okayish in medium-sized repositories, but once you reach a\ncertain size this isn't really an option anymore. In the Chromium\nrepository for example [1] simply listing all objects in the first\ninvocation of git-cat-file(1) takes around 80 to 100 seconds. The\nworkload is completely I/O-bottlenecked: my machine reads at ~500MB/s,\nand the packfile is 50GB in size, which matches the 100 seconds that I\nobserve.\n\nThis series addresses the issue by introducing object filters into\ngit-cat-file(1). These object filters use the exact same syntax as the\nfilters we have in git-rev-list(1), but only a subset of them is\nsupported because not all filters can be computed by git-cat-file(1).\nSupported are \"blob:none\", \"blob:limit=\" as well as \"object:type=\".\n\nThe filters alone don't really help though: we still have to scan\nthrough the whole packfile in order to compute the packfiles. While we\nare able to shed a bit of CPU time because we can stop emitting some of\nthe objects, we're still I/O-bottlenecked.\n\nThe second part of the series thus expands the filters so that they can\nmake use of bitmap indices for some of the filters, if available. This\nallows us to efficiently answer the question where to find all objects\nof a specific type, and thus we can avoid scanning through the packfile\nand instead directly look up relevant objects, leading to a significant\nspeedup:\n\n    Benchmark 1: cat-file with filter=object:type=commit (revision = HEAD~)\n      Time (mean ± σ):     86.444 s ±  4.081 s    [User: 36.830 s, System: 11.312 s]\n      Range (min … max):   80.305 s … 93.104 s    10 runs\n\n    Benchmark 2: cat-file with filter=object:type=commit (revision = HEAD)\n      Time (mean ± σ):      2.089 s ±  0.015 s    [User: 1.872 s, System: 0.207 s]\n      Range (min … max):    2.073 s …  2.119 s    10 runs\n\n    Summary\n      cat-file with filter=object:type=commit (revision = HEAD) ran\n       41.38 ± 1.98 times faster than cat-file with filter=object:type=commit (revision = HEAD~)\n\nWe now directly scale with the number of objects of a specific type\ncontained in the packfile instead of scaling with the overall number of\nobjects. It's quite fun to see how the math plays out: if you sum up the\ntimes for each of the types you arrive at the time for the unfiltered\ncase.\n\nChanges in v2:\n  - The series is now built on top of \"master\" at 683c54c999c (Git 2.49,\n    2025-03-14) with \"tb/incremental-midx-part-2\" at 27afc272c49 (midx:\n    implement writing incremental MIDX bitmaps, 2025-03-20) merged into\n    it.\n  - Rename the filter options to \"--filter=\" to match\n    git-pack-objects(1).\n  - The bitmap-filtering is now reusing existing mechanisms that we\n    already have in \"pack-bitmap.c\", as proposed by Taylor.\n  - Link to v1: https://lore.kernel.org/r/20250221-pks-cat-file-object-type-filter-v1-0-0852530888e2@pks.im\n\nChanges in v3:\n  - Wrap some overly long lines.\n  - Better describe how filters interact with the different batch modes.\n  - Adapt the format with `--batch` and `--batch-check` so that we tell\n    the user that the object has been excluded.\n  - Add a test for \"--no-filter\".\n  - Use `OPT_PARSE_LIST_OBJECTS_FILTER()`.\n  - Link to v2: https://lore.kernel.org/r/20250327-pks-cat-file-object-type-filter-v2-0-4bbc7085d7c5@pks.im\n\nThanks!\n\nPatrick\n\n[1]: https://github.com/chromium/chromium.git\n\n---\nPatrick Steinhardt (11):\n      builtin/cat-file: rename variable that tracks usage\n      builtin/cat-file: introduce function to report object status\n      builtin/cat-file: wire up an option to filter objects\n      builtin/cat-file: support \"blob:none\" objects filter\n      builtin/cat-file: support \"blob:limit=\" objects filter\n      builtin/cat-file: support \"object:type=\" objects filter\n      pack-bitmap: allow passing payloads to `show_reachable_fn()`\n      pack-bitmap: add function to iterate over filtered bitmapped objects\n      pack-bitmap: introduce function to check whether a pack is bitmapped\n      builtin/cat-file: deduplicate logic to iterate over all objects\n      builtin/cat-file: use bitmaps to efficiently filter by object type\n\n Documentation/git-cat-file.adoc |  26 ++++\n builtin/cat-file.c              | 256 +++++++++++++++++++++++++++++-----------\n builtin/pack-objects.c          |   3 +-\n builtin/rev-list.c              |   3 +-\n pack-bitmap.c                   |  81 +++++++++++--\n pack-bitmap.h                   |  22 +++-\n reachable.c                     |   3 +-\n t/t1006-cat-file.sh             |  99 ++++++++++++++++\n 8 files changed, 411 insertions(+), 82 deletions(-)\n\nRange-diff versus v2:\n\n 1:  a75888e0bf4 !  1:  b0642b6c495 builtin/cat-file: rename variable that tracks usage\n    @@ builtin/cat-file.c: int cmd_cat_file(int argc,\n      \t\t;\n      \telse if (batch.follow_symlinks)\n     -\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), usage, options,\n    -+\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage, options,\n    - \t\t\t       \"--follow-symlinks\");\n    +-\t\t\t       \"--follow-symlinks\");\n    ++\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage,\n    ++\t\t\t       options, \"--follow-symlinks\");\n      \telse if (batch.buffer_output >= 0)\n     -\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), usage, options,\n    -+\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage, options,\n    - \t\t\t       \"--buffer\");\n    +-\t\t\t       \"--buffer\");\n    ++\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage,\n    ++\t\t\t       options, \"--buffer\");\n      \telse if (batch.all_objects)\n     -\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), usage, options,\n    -+\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage, options,\n    - \t\t\t       \"--batch-all-objects\");\n    +-\t\t\t       \"--batch-all-objects\");\n    ++\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage,\n    ++\t\t\t       options, \"--batch-all-objects\");\n      \telse if (input_nul_terminated)\n     -\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), usage, options,\n    -+\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage, options,\n    - \t\t\t       \"-z\");\n    +-\t\t\t       \"-z\");\n    ++\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage,\n    ++\t\t\t       options, \"-z\");\n      \telse if (nul_terminated)\n     -\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), usage, options,\n    -+\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage, options,\n    - \t\t\t       \"-Z\");\n    +-\t\t\t       \"-Z\");\n    ++\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage,\n    ++\t\t\t       options, \"-Z\");\n      \n      \tbatch.input_delim = batch.output_delim = '\\n';\n    + \tif (input_nul_terminated)\n     @@ builtin/cat-file.c: int cmd_cat_file(int argc,\n      \t\t\tbatch.transform_mode = opt;\n      \t\telse if (opt && opt != 'b')\n    @@ builtin/cat-file.c: int cmd_cat_file(int argc,\n     +\t\t\t\t       builtin_catfile_usage, options, opt);\n      \t\telse if (argc)\n     -\t\t\tusage_msg_opt(_(\"batch modes take no arguments\"), usage,\n    -+\t\t\tusage_msg_opt(_(\"batch modes take no arguments\"), builtin_catfile_usage,\n    - \t\t\t\t      options);\n    +-\t\t\t\t      options);\n    ++\t\t\tusage_msg_opt(_(\"batch modes take no arguments\"),\n    ++\t\t\t\t      builtin_catfile_usage, options);\n      \n      \t\treturn batch_objects(&batch);\n    + \t}\n     @@ builtin/cat-file.c: int cmd_cat_file(int argc,\n      \tif (opt) {\n      \t\tif (!argc && opt == 'c')\n      \t\t\tusage_msg_optf(_(\"<rev> required with '%s'\"),\n     -\t\t\t\t       usage, options, \"--textconv\");\n    -+\t\t\t\t       builtin_catfile_usage, options, \"--textconv\");\n    ++\t\t\t\t       builtin_catfile_usage, options,\n    ++\t\t\t\t       \"--textconv\");\n      \t\telse if (!argc && opt == 'w')\n      \t\t\tusage_msg_optf(_(\"<rev> required with '%s'\"),\n     -\t\t\t\t       usage, options, \"--filters\");\n    -+\t\t\t\t       builtin_catfile_usage, options, \"--filters\");\n    ++\t\t\t\t       builtin_catfile_usage, options,\n    ++\t\t\t\t       \"--filters\");\n      \t\telse if (!argc && opt_epts)\n      \t\t\tusage_msg_optf(_(\"<object> required with '-%c'\"),\n     -\t\t\t\t       usage, options, opt);\n    @@ builtin/cat-file.c: int cmd_cat_file(int argc,\n      \t\t\tobj_name = argv[0];\n      \t\telse\n     -\t\t\tusage_msg_opt(_(\"too many arguments\"), usage, options);\n    -+\t\t\tusage_msg_opt(_(\"too many arguments\"), builtin_catfile_usage, options);\n    ++\t\t\tusage_msg_opt(_(\"too many arguments\"), builtin_catfile_usage,\n    ++\t\t\t\t      options);\n      \t} else if (!argc) {\n     -\t\tusage_with_options(usage, options);\n     +\t\tusage_with_options(builtin_catfile_usage, options);\n -:  ----------- >  2:  18353ba706d builtin/cat-file: introduce function to report object status\n 2:  bee9407c1a9 !  3:  1e46af5d07b builtin/cat-file: wire up an option to filter objects\n    @@ Documentation/git-cat-file.adoc: OPTIONS\n     +--filter=<filter-spec>::\n     +--no-filter::\n     +\tOmit objects from the list of printed objects. This can only be used in\n    -+\tcombination with one of the batched modes. The '<filter-spec>' may be\n    -+\tone of the following:\n    ++\tcombination with one of the batched modes. Excluded objects that have\n    ++\tbeen explicitly requested via any of the batch modes that read objects\n    ++\tvia standard input (`--batch`, `--batch-check`) will be reported as\n    ++\t\"filtered\". Excluded objects in `--batch-all-objects` mode will not be\n    ++\tprinted at all. No filters are supported yet.\n     +\n      --path=<path>::\n      \tFor use with `--textconv` or `--filters`, to allow specifying an object\n      \tname and a path separately, e.g. when it is difficult to figure out\n    +@@ Documentation/git-cat-file.adoc: the repository, then `cat-file` will ignore any custom format and print:\n    + <object> SP missing LF\n    + ------------\n    + \n    ++If a name is specified on stdin that is filtered out via `--filter=`,\n    ++then `cat-file` will ignore any custom format and print:\n    ++\n    ++------------\n    ++<object> SP excluded LF\n    ++------------\n    ++\n    + If a name is specified that might refer to more than one object (an ambiguous short sha), then `cat-file` will ignore any custom format and print:\n    + \n    + ------------\n     \n      ## builtin/cat-file.c ##\n     @@\n    @@ builtin/cat-file.c: int cmd_cat_file(int argc,\n      \t\t\t    N_(\"run filters on object's content\"), 'w'),\n      \t\tOPT_STRING(0, \"path\", &force_path, N_(\"blob|tree\"),\n      \t\t\t   N_(\"use a <path> for (--textconv | --filters); Not with 'batch'\")),\n    -+\t\tOPT_CALLBACK(0, \"filter\", &batch.objects_filter, N_(\"args\"),\n    -+\t\t\t     N_(\"object filtering\"), opt_parse_list_objects_filter),\n    ++\t\tOPT_PARSE_LIST_OBJECTS_FILTER(&batch.objects_filter),\n      \t\tOPT_END()\n      \t};\n      \n    @@ builtin/cat-file.c: int cmd_cat_file(int argc,\n      \tif (opt == 'b')\n      \t\tbatch.all_objects = 1;\n     @@ builtin/cat-file.c: int cmd_cat_file(int argc,\n    - \t\t\tusage_msg_opt(_(\"batch modes take no arguments\"), builtin_catfile_usage,\n    - \t\t\t\t      options);\n    + \t\t\tusage_msg_opt(_(\"batch modes take no arguments\"),\n    + \t\t\t\t      builtin_catfile_usage, options);\n      \n     -\t\treturn batch_objects(&batch);\n     +\t\tret = batch_objects(&batch);\n    @@ t/t1006-cat-file.sh: test_expect_success PERL '--batch-command info is unbuffere\n     +\t\ttest_cmp expect err\n     +\t'\n     +done\n    ++\n    ++test_expect_success 'objects filter: disabled' '\n    ++\tgit -C repo cat-file --batch-check=\"%(objectname)\" --batch-all-objects --no-filter >actual &&\n    ++\tsort actual >actual.sorted &&\n    ++\tgit -C repo rev-list --objects --no-object-names --all >expect &&\n    ++\tsort expect >expect.sorted &&\n    ++\ttest_cmp expect.sorted actual.sorted\n    ++'\n     +\n      test_done\n 3:  ec1d0c63de6 !  4:  878ae8e2a76 builtin/cat-file: support \"blob:none\" objects filter\n    @@ Commit message\n         Implement support for the \"blob:none\" filter in git-cat-file(1), which\n         causes us to omit all blobs.\n     \n    +    Note that this new filter requires us to read the object type via\n    +    `oid_object_info_extended()` in `batch_object_write()`. But as we try to\n    +    optimize away reading objects from the database the `data->info.typep`\n    +    pointer may not be set. We thus have to adapt the logic to conditionally\n    +    set the pointer in cases where the filter is given.\n    +\n         Signed-off-by: Patrick Steinhardt <ps@pks.im>\n     \n      ## Documentation/git-cat-file.adoc ##\n     @@ Documentation/git-cat-file.adoc: OPTIONS\n    - \tOmit objects from the list of printed objects. This can only be used in\n    - \tcombination with one of the batched modes. The '<filter-spec>' may be\n    - \tone of the following:\n    + \tbeen explicitly requested via any of the batch modes that read objects\n    + \tvia standard input (`--batch`, `--batch-check`) will be reported as\n    + \t\"filtered\". Excluded objects in `--batch-all-objects` mode will not be\n    +-\tprinted at all. No filters are supported yet.\n    ++\tprinted at all. The '<filter-spec>' may be one of the following:\n     ++\n     +The form '--filter=blob:none' omits all blobs.\n      \n    @@ builtin/cat-file.c: static void batch_object_write(const char *obj_name,\n      \t\tcase LOFC_DISABLED:\n      \t\t\tbreak;\n     +\t\tcase LOFC_BLOB_NONE:\n    -+\t\t\tif (data->type == OBJ_BLOB)\n    ++\t\t\tif (data->type == OBJ_BLOB) {\n    ++\t\t\t\tif (!opt->all_objects)\n    ++\t\t\t\t\treport_object_status(opt, obj_name,\n    ++\t\t\t\t\t\t\t     &data->oid, \"excluded\");\n     +\t\t\t\treturn;\n    ++\t\t\t}\n     +\t\t\tbreak;\n      \t\tdefault:\n      \t\t\tBUG(\"unsupported objects filter\");\n    @@ t/t1006-cat-file.sh: test_expect_success 'objects filter with unknown option' '\n      do\n      \ttest_expect_success \"objects filter with unsupported option $option\" '\n      \t\tcase \"$option\" in\n    -@@ t/t1006-cat-file.sh: do\n    - \t'\n    - done\n    +@@ t/t1006-cat-file.sh: test_expect_success 'objects filter: disabled' '\n    + \ttest_cmp expect.sorted actual.sorted\n    + '\n      \n     +test_objects_filter () {\n     +\tfilter=\"$1\"\n    @@ t/t1006-cat-file.sh: do\n     +\t\tsort expect >expect.sorted &&\n     +\t\ttest_cmp expect.sorted actual.sorted\n     +\t'\n    ++\n    ++\ttest_expect_success \"objects filter prints excluded objects: $filter\" '\n    ++\t\t# Find all objects that would be excluded by the current filter.\n    ++\t\tgit -C repo rev-list --objects --no-object-names --all >all &&\n    ++\t\tgit -C repo rev-list --objects --no-object-names --all --filter=\"$filter\" --filter-provided-objects >filtered &&\n    ++\t\tsort all >all.sorted &&\n    ++\t\tsort filtered >filtered.sorted &&\n    ++\t\tcomm -23 all.sorted filtered.sorted >expected.excluded &&\n    ++\t\ttest_line_count -gt 0 expected.excluded &&\n    ++\n    ++\t\tgit -C repo cat-file --batch-check=\"%(objectname)\" --filter=\"$filter\" <expected.excluded >actual &&\n    ++\t\tawk \"/excluded/{ print \\$1 }\" actual | sort >actual.excluded &&\n    ++\t\ttest_cmp expected.excluded actual.excluded\n    ++\t'\n     +}\n     +\n     +test_objects_filter \"blob:none\"\n 4:  a3ed054994d !  5:  a88d5d4b60a builtin/cat-file: support \"blob:limit=\" objects filter\n    @@ Commit message\n     \n      ## Documentation/git-cat-file.adoc ##\n     @@ Documentation/git-cat-file.adoc: OPTIONS\n    - \tone of the following:\n    + \tprinted at all. The '<filter-spec>' may be one of the following:\n      +\n      The form '--filter=blob:none' omits all blobs.\n     ++\n    @@ builtin/cat-file.c: static void batch_object_write(const char *obj_name,\n      \t\tif (pack)\n      \t\t\tret = packed_object_info(the_repository, pack, offset,\n     @@ builtin/cat-file.c: static void batch_object_write(const char *obj_name,\n    - \t\t\tif (data->type == OBJ_BLOB)\n      \t\t\t\treturn;\n    + \t\t\t}\n      \t\t\tbreak;\n     +\t\tcase LOFC_BLOB_LIMIT:\n     +\t\t\tif (data->type == OBJ_BLOB &&\n    -+\t\t\t    data->size >= opt->objects_filter.blob_limit_value)\n    ++\t\t\t    data->size >= opt->objects_filter.blob_limit_value) {\n    ++\t\t\t\tif (!opt->all_objects)\n    ++\t\t\t\t\treport_object_status(opt, obj_name,\n    ++\t\t\t\t\t\t\t     &data->oid, \"excluded\");\n     +\t\t\t\treturn;\n    ++\t\t\t}\n     +\t\t\tbreak;\n      \t\tdefault:\n      \t\t\tBUG(\"unsupported objects filter\");\n    @@ t/t1006-cat-file.sh: test_objects_filter () {\n     +test_objects_filter \"blob:limit=1\"\n     +test_objects_filter \"blob:limit=500\"\n     +test_objects_filter \"blob:limit=1000\"\n    -+test_objects_filter \"blob:limit=1g\"\n    ++test_objects_filter \"blob:limit=1k\"\n      \n      test_done\n 5:  8e39cd218c2 !  6:  13be54300c9 builtin/cat-file: support \"object:type=\" objects filter\n    @@ builtin/cat-file.c: static void batch_object_write(const char *obj_name,\n      \t\tif (opt->objects_filter.choice == LOFC_BLOB_LIMIT)\n      \t\t\tdata->info.sizep = &data->size;\n     @@ builtin/cat-file.c: static void batch_object_write(const char *obj_name,\n    - \t\t\t    data->size >= opt->objects_filter.blob_limit_value)\n      \t\t\t\treturn;\n    + \t\t\t}\n      \t\t\tbreak;\n     +\t\tcase LOFC_OBJECT_TYPE:\n    -+\t\t\tif (data->type != opt->objects_filter.object_type)\n    ++\t\t\tif (data->type != opt->objects_filter.object_type) {\n    ++\t\t\t\tif (!opt->all_objects)\n    ++\t\t\t\t\treport_object_status(opt, obj_name,\n    ++\t\t\t\t\t\t\t     &data->oid, \"excluded\");\n     +\t\t\t\treturn;\n    ++\t\t\t}\n     +\t\t\tbreak;\n      \t\tdefault:\n      \t\t\tBUG(\"unsupported objects filter\");\n    @@ t/t1006-cat-file.sh: test_expect_success 'objects filter with unknown option' '\n     @@ t/t1006-cat-file.sh: test_objects_filter \"blob:limit=1\"\n      test_objects_filter \"blob:limit=500\"\n      test_objects_filter \"blob:limit=1000\"\n    - test_objects_filter \"blob:limit=1g\"\n    + test_objects_filter \"blob:limit=1k\"\n     +test_objects_filter \"object:type=blob\"\n     +test_objects_filter \"object:type=commit\"\n     +test_objects_filter \"object:type=tag\"\n 6:  a0655de3ace =  7:  d525a5bc2ef pack-bitmap: allow passing payloads to `show_reachable_fn()`\n 7:  e1e44303dac =  8:  e3cc1ae3a87 pack-bitmap: add function to iterate over filtered bitmapped objects\n 8:  23bc040bb15 =  9:  c0fc0e4ce0c pack-bitmap: introduce function to check whether a pack is bitmapped\n 9:  4eba2a70619 = 10:  28ef93dceec builtin/cat-file: deduplicate logic to iterate over all objects\n10:  d40f1924ef5 = 11:  842a6002c50 builtin/cat-file: use bitmaps to efficiently filter by object type\n\n---\nbase-commit: 003c5f45b8447877015b2a23ceab2297638fe1f1\nchange-id: 20250220-pks-cat-file-object-type-filter-9140c0ed5ee1\n\n"},{"id":"515503","messageId":"20250402-pks-cat-file-object-type-filter-v3-1-4da6bb21871c@pks.im","threadId":"62984","inReplyTo":"20250402-pks-cat-file-object-type-filter-v3-0-4da6bb21871c@pks.im","subject":"[PATCH v3 01/11] builtin/cat-file: rename variable that tracks usage","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-02T11:13:36Z","receivedAt":"2025-04-02T11:13:43Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The usage strings for git-cat-file(1) that we pass to `parse_options()`\nand `usage_msg_optf()` are stored in a variable called `usage`. This\nvariable shadows the declaration of `usage()`, which we'll want to use\nin a subsequent commit.\n\nRename the variable to `builtin_catfile_usage`, which is in line with\nhow the variable is typically called in other builtins.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/cat-file.c | 47 +++++++++++++++++++++++++----------------------\n 1 file changed, 25 insertions(+), 22 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex b13561cf73b..b158b3acef9 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -941,7 +941,7 @@ int cmd_cat_file(int argc,\n \tint input_nul_terminated = 0;\n \tint nul_terminated = 0;\n \n-\tconst char * const usage[] = {\n+\tconst char * const builtin_catfile_usage[] = {\n \t\tN_(\"git cat-file <type> <object>\"),\n \t\tN_(\"git cat-file (-e | -p) <object>\"),\n \t\tN_(\"git cat-file (-t | -s) [--allow-unknown-type] <object>\"),\n@@ -1007,7 +1007,7 @@ int cmd_cat_file(int argc,\n \n \tbatch.buffer_output = -1;\n \n-\targc = parse_options(argc, argv, prefix, options, usage, 0);\n+\targc = parse_options(argc, argv, prefix, options, builtin_catfile_usage, 0);\n \topt_cw = (opt == 'c' || opt == 'w');\n \topt_epts = (opt == 'e' || opt == 'p' || opt == 't' || opt == 's');\n \n@@ -1021,7 +1021,7 @@ int cmd_cat_file(int argc,\n \t/* Option compatibility */\n \tif (force_path && !opt_cw)\n \t\tusage_msg_optf(_(\"'%s=<%s>' needs '%s' or '%s'\"),\n-\t\t\t       usage, options,\n+\t\t\t       builtin_catfile_usage, options,\n \t\t\t       \"--path\", _(\"path|tree-ish\"), \"--filters\",\n \t\t\t       \"--textconv\");\n \n@@ -1029,20 +1029,20 @@ int cmd_cat_file(int argc,\n \tif (batch.enabled)\n \t\t;\n \telse if (batch.follow_symlinks)\n-\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), usage, options,\n-\t\t\t       \"--follow-symlinks\");\n+\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage,\n+\t\t\t       options, \"--follow-symlinks\");\n \telse if (batch.buffer_output >= 0)\n-\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), usage, options,\n-\t\t\t       \"--buffer\");\n+\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage,\n+\t\t\t       options, \"--buffer\");\n \telse if (batch.all_objects)\n-\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), usage, options,\n-\t\t\t       \"--batch-all-objects\");\n+\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage,\n+\t\t\t       options, \"--batch-all-objects\");\n \telse if (input_nul_terminated)\n-\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), usage, options,\n-\t\t\t       \"-z\");\n+\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage,\n+\t\t\t       options, \"-z\");\n \telse if (nul_terminated)\n-\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), usage, options,\n-\t\t\t       \"-Z\");\n+\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage,\n+\t\t\t       options, \"-Z\");\n \n \tbatch.input_delim = batch.output_delim = '\\n';\n \tif (input_nul_terminated)\n@@ -1063,10 +1063,10 @@ int cmd_cat_file(int argc,\n \t\t\tbatch.transform_mode = opt;\n \t\telse if (opt && opt != 'b')\n \t\t\tusage_msg_optf(_(\"'-%c' is incompatible with batch mode\"),\n-\t\t\t\t       usage, options, opt);\n+\t\t\t\t       builtin_catfile_usage, options, opt);\n \t\telse if (argc)\n-\t\t\tusage_msg_opt(_(\"batch modes take no arguments\"), usage,\n-\t\t\t\t      options);\n+\t\t\tusage_msg_opt(_(\"batch modes take no arguments\"),\n+\t\t\t\t      builtin_catfile_usage, options);\n \n \t\treturn batch_objects(&batch);\n \t}\n@@ -1074,22 +1074,25 @@ int cmd_cat_file(int argc,\n \tif (opt) {\n \t\tif (!argc && opt == 'c')\n \t\t\tusage_msg_optf(_(\"<rev> required with '%s'\"),\n-\t\t\t\t       usage, options, \"--textconv\");\n+\t\t\t\t       builtin_catfile_usage, options,\n+\t\t\t\t       \"--textconv\");\n \t\telse if (!argc && opt == 'w')\n \t\t\tusage_msg_optf(_(\"<rev> required with '%s'\"),\n-\t\t\t\t       usage, options, \"--filters\");\n+\t\t\t\t       builtin_catfile_usage, options,\n+\t\t\t\t       \"--filters\");\n \t\telse if (!argc && opt_epts)\n \t\t\tusage_msg_optf(_(\"<object> required with '-%c'\"),\n-\t\t\t\t       usage, options, opt);\n+\t\t\t\t       builtin_catfile_usage, options, opt);\n \t\telse if (argc == 1)\n \t\t\tobj_name = argv[0];\n \t\telse\n-\t\t\tusage_msg_opt(_(\"too many arguments\"), usage, options);\n+\t\t\tusage_msg_opt(_(\"too many arguments\"), builtin_catfile_usage,\n+\t\t\t\t      options);\n \t} else if (!argc) {\n-\t\tusage_with_options(usage, options);\n+\t\tusage_with_options(builtin_catfile_usage, options);\n \t} else if (argc != 2) {\n \t\tusage_msg_optf(_(\"only two arguments allowed in <type> <object> mode, not %d\"),\n-\t\t\t      usage, options, argc);\n+\t\t\t      builtin_catfile_usage, options, argc);\n \t} else if (argc) {\n \t\texp_type = argv[0];\n \t\tobj_name = argv[1];\n\n-- \n2.49.0.604.gff1f9ca942.dirty\n\n"},{"id":"515504","messageId":"20250402-pks-cat-file-object-type-filter-v3-2-4da6bb21871c@pks.im","threadId":"62984","inReplyTo":"20250402-pks-cat-file-object-type-filter-v3-0-4da6bb21871c@pks.im","subject":"[PATCH v3 02/11] builtin/cat-file: introduce function to report object status","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-02T11:13:37Z","receivedAt":"2025-04-02T11:13:44Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have multiple callsites that report the status of an object, for\nexample when the objec tis missing or its name is ambiguous. We're about\nto add a couple more such callsites to report on \"excluded\" objects.\n\nPrepare for this by introducing a new function `report_object_status()`\nthat encapsulates the functionality.\n\nNote that this function also flushes stdout, which is a requirement so\nthat request-response style batched modes can learn about the status\nbefore proceeding to the next object. We already flush correctly at all\nexisting callsites, even though the flush in `batch_one_object()` only\ncomes after the switch statement. That flush is now redundant, and we\ncould in theory deduplicate it by moving it into all branches that don't\nuse `report_object_status()`. But that doesn't quite feel sensible:\n\n  - The duplicate flush should ultimately just be a no-op for us and\n    thus shouldn't impact performance significantly.\n\n  - By keeping the flush in `report_object_status()` we ensure that all\n    future callers get semantics correct.\n\nSo let's just be pragmatic and live with the duplicated flush.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/cat-file.c | 18 +++++++++++++-----\n 1 file changed, 13 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex b158b3acef9..1261a3ce352 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -455,6 +455,16 @@ static void print_default_format(struct strbuf *scratch, struct expand_data *dat\n \t\t    (uintmax_t)data->size, opt->output_delim);\n }\n \n+static void report_object_status(struct batch_options *opt,\n+\t\t\t\t const char *obj_name,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t const char *status)\n+{\n+\tprintf(\"%s %s%c\", obj_name ? obj_name : oid_to_hex(oid),\n+\t       status, opt->output_delim);\n+\tfflush(stdout);\n+}\n+\n /*\n  * If \"pack\" is non-NULL, then \"offset\" is the byte offset within the pack from\n  * which the object may be accessed (though note that we may also rely on\n@@ -481,9 +491,7 @@ static void batch_object_write(const char *obj_name,\n \t\t\t\t\t\t       &data->oid, &data->info,\n \t\t\t\t\t\t       OBJECT_INFO_LOOKUP_REPLACE);\n \t\tif (ret < 0) {\n-\t\t\tprintf(\"%s missing%c\",\n-\t\t\t       obj_name ? obj_name : oid_to_hex(&data->oid), opt->output_delim);\n-\t\t\tfflush(stdout);\n+\t\t\treport_object_status(opt, obj_name, &data->oid, \"missing\");\n \t\t\treturn;\n \t\t}\n \n@@ -535,10 +543,10 @@ static void batch_one_object(const char *obj_name,\n \tif (result != FOUND) {\n \t\tswitch (result) {\n \t\tcase MISSING_OBJECT:\n-\t\t\tprintf(\"%s missing%c\", obj_name, opt->output_delim);\n+\t\t\treport_object_status(opt, obj_name, &data->oid, \"missing\");\n \t\t\tbreak;\n \t\tcase SHORT_NAME_AMBIGUOUS:\n-\t\t\tprintf(\"%s ambiguous%c\", obj_name, opt->output_delim);\n+\t\t\treport_object_status(opt, obj_name, &data->oid, \"ambiguous\");\n \t\t\tbreak;\n \t\tcase DANGLING_SYMLINK:\n \t\t\tprintf(\"dangling %\"PRIuMAX\"%c%s%c\",\n\n-- \n2.49.0.604.gff1f9ca942.dirty\n\n"},{"id":"515505","messageId":"20250402-pks-cat-file-object-type-filter-v3-3-4da6bb21871c@pks.im","threadId":"62984","inReplyTo":"20250402-pks-cat-file-object-type-filter-v3-0-4da6bb21871c@pks.im","subject":"[PATCH v3 03/11] builtin/cat-file: wire up an option to filter objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-02T11:13:38Z","receivedAt":"2025-04-02T11:13:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In batch mode, git-cat-file(1) enumerates all objects and prints them\nby iterating through both loose and packed objects. This works without\nconsidering their reachability at all, and consequently most options to\nfilter objects as they exist in e.g. git-rev-list(1) are not applicable.\nIn some situations it may still be useful though to filter objects based\non properties that are inherent to them. This includes the object size\nas well as its type.\n\nSuch a filter already exists in git-rev-list(1) with the `--filter=`\ncommand line option. While this option supports a couple of filters that\nare not applicable to our usecase, some of them are quite a neat fit.\n\nWire up the filter as an option for git-cat-file(1). This allows us to\nreuse the same syntax as in git-rev-list(1) so that we don't have to\nreinvent the wheel. For now, we die when any of the filter options has\nbeen passed by the user, but they will be wired up in subsequent\ncommits.\n\nFurther note that the filters that we are about to introduce don't\nsignificantly speed up the runtime of git-cat-file(1). While we can skip\nemitting a lot of objects in case they are uninteresting to us, the\nmajority of time is spent reading the packfile, which is bottlenecked by\nI/O and not the processor. This will change though once we start to make\nuse of bitmaps, which will allow us to skip reading the whole packfile.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n Documentation/git-cat-file.adoc | 16 ++++++++++++++++\n builtin/cat-file.c              | 36 ++++++++++++++++++++++++++++++++----\n t/t1006-cat-file.sh             | 40 ++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 88 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-cat-file.adoc b/Documentation/git-cat-file.adoc\nindex d5890ae3686..da92eed1170 100644\n--- a/Documentation/git-cat-file.adoc\n+++ b/Documentation/git-cat-file.adoc\n@@ -81,6 +81,15 @@ OPTIONS\n \tend-of-line conversion, etc). In this case, `<object>` has to be of\n \tthe form `<tree-ish>:<path>`, or `:<path>`.\n \n+--filter=<filter-spec>::\n+--no-filter::\n+\tOmit objects from the list of printed objects. This can only be used in\n+\tcombination with one of the batched modes. Excluded objects that have\n+\tbeen explicitly requested via any of the batch modes that read objects\n+\tvia standard input (`--batch`, `--batch-check`) will be reported as\n+\t\"filtered\". Excluded objects in `--batch-all-objects` mode will not be\n+\tprinted at all. No filters are supported yet.\n+\n --path=<path>::\n \tFor use with `--textconv` or `--filters`, to allow specifying an object\n \tname and a path separately, e.g. when it is difficult to figure out\n@@ -340,6 +349,13 @@ the repository, then `cat-file` will ignore any custom format and print:\n <object> SP missing LF\n ------------\n \n+If a name is specified on stdin that is filtered out via `--filter=`,\n+then `cat-file` will ignore any custom format and print:\n+\n+------------\n+<object> SP excluded LF\n+------------\n+\n If a name is specified that might refer to more than one object (an ambiguous short sha), then `cat-file` will ignore any custom format and print:\n \n ------------\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 1261a3ce352..0e2176c4491 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -15,6 +15,7 @@\n #include \"gettext.h\"\n #include \"hex.h\"\n #include \"ident.h\"\n+#include \"list-objects-filter-options.h\"\n #include \"parse-options.h\"\n #include \"userdiff.h\"\n #include \"streaming.h\"\n@@ -35,6 +36,7 @@ enum batch_mode {\n };\n \n struct batch_options {\n+\tstruct list_objects_filter_options objects_filter;\n \tint enabled;\n \tint follow_symlinks;\n \tenum batch_mode batch_mode;\n@@ -495,6 +497,13 @@ static void batch_object_write(const char *obj_name,\n \t\t\treturn;\n \t\t}\n \n+\t\tswitch (opt->objects_filter.choice) {\n+\t\tcase LOFC_DISABLED:\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tBUG(\"unsupported objects filter\");\n+\t\t}\n+\n \t\tif (use_mailmap && (data->type == OBJ_COMMIT || data->type == OBJ_TAG)) {\n \t\t\tsize_t s = data->size;\n \t\t\tchar *buf = NULL;\n@@ -820,7 +829,8 @@ static int batch_objects(struct batch_options *opt)\n \t\tstruct object_cb_data cb;\n \t\tstruct object_info empty = OBJECT_INFO_INIT;\n \n-\t\tif (!memcmp(&data.info, &empty, sizeof(empty)))\n+\t\tif (!memcmp(&data.info, &empty, sizeof(empty)) &&\n+\t\t    opt->objects_filter.choice == LOFC_DISABLED)\n \t\t\tdata.skip_object_info = 1;\n \n \t\tif (repo_has_promisor_remote(the_repository))\n@@ -944,10 +954,13 @@ int cmd_cat_file(int argc,\n \tint opt_cw = 0;\n \tint opt_epts = 0;\n \tconst char *exp_type = NULL, *obj_name = NULL;\n-\tstruct batch_options batch = {0};\n+\tstruct batch_options batch = {\n+\t\t.objects_filter = LIST_OBJECTS_FILTER_INIT,\n+\t};\n \tint unknown_type = 0;\n \tint input_nul_terminated = 0;\n \tint nul_terminated = 0;\n+\tint ret;\n \n \tconst char * const builtin_catfile_usage[] = {\n \t\tN_(\"git cat-file <type> <object>\"),\n@@ -1008,6 +1021,7 @@ int cmd_cat_file(int argc,\n \t\t\t    N_(\"run filters on object's content\"), 'w'),\n \t\tOPT_STRING(0, \"path\", &force_path, N_(\"blob|tree\"),\n \t\t\t   N_(\"use a <path> for (--textconv | --filters); Not with 'batch'\")),\n+\t\tOPT_PARSE_LIST_OBJECTS_FILTER(&batch.objects_filter),\n \t\tOPT_END()\n \t};\n \n@@ -1022,6 +1036,14 @@ int cmd_cat_file(int argc,\n \tif (use_mailmap)\n \t\tread_mailmap(&mailmap);\n \n+\tswitch (batch.objects_filter.choice) {\n+\tcase LOFC_DISABLED:\n+\t\tbreak;\n+\tdefault:\n+\t\tusagef(_(\"objects filter not supported: '%s'\"),\n+\t\t       list_object_filter_config_name(batch.objects_filter.choice));\n+\t}\n+\n \t/* --batch-all-objects? */\n \tif (opt == 'b')\n \t\tbatch.all_objects = 1;\n@@ -1076,7 +1098,8 @@ int cmd_cat_file(int argc,\n \t\t\tusage_msg_opt(_(\"batch modes take no arguments\"),\n \t\t\t\t      builtin_catfile_usage, options);\n \n-\t\treturn batch_objects(&batch);\n+\t\tret = batch_objects(&batch);\n+\t\tgoto out;\n \t}\n \n \tif (opt) {\n@@ -1108,5 +1131,10 @@ int cmd_cat_file(int argc,\n \n \tif (unknown_type && opt != 't' && opt != 's')\n \t\tdie(\"git cat-file --allow-unknown-type: use with -s or -t\");\n-\treturn cat_one_file(opt, exp_type, obj_name, unknown_type);\n+\n+\tret = cat_one_file(opt, exp_type, obj_name, unknown_type);\n+\n+out:\n+\tlist_objects_filter_release(&batch.objects_filter);\n+\treturn ret;\n }\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 398865d6ebe..9ce4eda6e68 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -1353,4 +1353,44 @@ test_expect_success PERL '--batch-command info is unbuffered by default' '\n \tperl -e \"$script\" -- --batch-command $hello_oid \"$expect\" \"info \"\n '\n \n+test_expect_success 'setup for objects filter' '\n+\tgit init repo\n+'\n+\n+test_expect_success 'objects filter with unknown option' '\n+\tcat >expect <<-EOF &&\n+\tfatal: invalid filter-spec ${SQ}unknown${SQ}\n+\tEOF\n+\ttest_must_fail git -C repo cat-file --filter=unknown 2>err &&\n+\ttest_cmp expect err\n+'\n+\n+for option in blob:none blob:limit=1 object:type=tag sparse:oid=1234 tree:1 sparse:path=x\n+do\n+\ttest_expect_success \"objects filter with unsupported option $option\" '\n+\t\tcase \"$option\" in\n+\t\ttree:1)\n+\t\t\techo \"usage: objects filter not supported: ${SQ}tree${SQ}\" >expect\n+\t\t\t;;\n+\t\tsparse:path=x)\n+\t\t\techo \"fatal: sparse:path filters support has been dropped\" >expect\n+\t\t\t;;\n+\t\t*)\n+\t\t\toption_name=$(echo \"$option\" | cut -d= -f1) &&\n+\t\t\tprintf \"usage: objects filter not supported: ${SQ}%s${SQ}\\n\" \"$option_name\" >expect\n+\t\t\t;;\n+\t\tesac &&\n+\t\ttest_must_fail git -C repo cat-file --filter=$option 2>err &&\n+\t\ttest_cmp expect err\n+\t'\n+done\n+\n+test_expect_success 'objects filter: disabled' '\n+\tgit -C repo cat-file --batch-check=\"%(objectname)\" --batch-all-objects --no-filter >actual &&\n+\tsort actual >actual.sorted &&\n+\tgit -C repo rev-list --objects --no-object-names --all >expect &&\n+\tsort expect >expect.sorted &&\n+\ttest_cmp expect.sorted actual.sorted\n+'\n+\n test_done\n\n-- \n2.49.0.604.gff1f9ca942.dirty\n\n"},{"id":"515506","messageId":"20250402-pks-cat-file-object-type-filter-v3-4-4da6bb21871c@pks.im","threadId":"62984","inReplyTo":"20250402-pks-cat-file-object-type-filter-v3-0-4da6bb21871c@pks.im","subject":"[PATCH v3 04/11] builtin/cat-file: support \"blob:none\" objects filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-02T11:13:39Z","receivedAt":"2025-04-02T11:13:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement support for the \"blob:none\" filter in git-cat-file(1), which\ncauses us to omit all blobs.\n\nNote that this new filter requires us to read the object type via\n`oid_object_info_extended()` in `batch_object_write()`. But as we try to\noptimize away reading objects from the database the `data->info.typep`\npointer may not be set. We thus have to adapt the logic to conditionally\nset the pointer in cases where the filter is given.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n Documentation/git-cat-file.adoc |  4 +++-\n builtin/cat-file.c              | 15 ++++++++++++-\n t/t1006-cat-file.sh             | 47 +++++++++++++++++++++++++++++++++++++++--\n 3 files changed, 62 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-cat-file.adoc b/Documentation/git-cat-file.adoc\nindex da92eed1170..afcdb0a4738 100644\n--- a/Documentation/git-cat-file.adoc\n+++ b/Documentation/git-cat-file.adoc\n@@ -88,7 +88,9 @@ OPTIONS\n \tbeen explicitly requested via any of the batch modes that read objects\n \tvia standard input (`--batch`, `--batch-check`) will be reported as\n \t\"filtered\". Excluded objects in `--batch-all-objects` mode will not be\n-\tprinted at all. No filters are supported yet.\n+\tprinted at all. The '<filter-spec>' may be one of the following:\n++\n+The form '--filter=blob:none' omits all blobs.\n \n --path=<path>::\n \tFor use with `--textconv` or `--filters`, to allow specifying an object\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 0e2176c4491..bcceb646f85 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -482,7 +482,8 @@ static void batch_object_write(const char *obj_name,\n \tif (!data->skip_object_info) {\n \t\tint ret;\n \n-\t\tif (use_mailmap)\n+\t\tif (use_mailmap ||\n+\t\t    opt->objects_filter.choice == LOFC_BLOB_NONE)\n \t\t\tdata->info.typep = &data->type;\n \n \t\tif (pack)\n@@ -500,6 +501,14 @@ static void batch_object_write(const char *obj_name,\n \t\tswitch (opt->objects_filter.choice) {\n \t\tcase LOFC_DISABLED:\n \t\t\tbreak;\n+\t\tcase LOFC_BLOB_NONE:\n+\t\t\tif (data->type == OBJ_BLOB) {\n+\t\t\t\tif (!opt->all_objects)\n+\t\t\t\t\treport_object_status(opt, obj_name,\n+\t\t\t\t\t\t\t     &data->oid, \"excluded\");\n+\t\t\t\treturn;\n+\t\t\t}\n+\t\t\tbreak;\n \t\tdefault:\n \t\t\tBUG(\"unsupported objects filter\");\n \t\t}\n@@ -1039,6 +1048,10 @@ int cmd_cat_file(int argc,\n \tswitch (batch.objects_filter.choice) {\n \tcase LOFC_DISABLED:\n \t\tbreak;\n+\tcase LOFC_BLOB_NONE:\n+\t\tif (!batch.enabled)\n+\t\t\tusage(_(\"objects filter only supported in batch mode\"));\n+\t\tbreak;\n \tdefault:\n \t\tusagef(_(\"objects filter not supported: '%s'\"),\n \t\t       list_object_filter_config_name(batch.objects_filter.choice));\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 9ce4eda6e68..7404c135b1e 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -1354,7 +1354,22 @@ test_expect_success PERL '--batch-command info is unbuffered by default' '\n '\n \n test_expect_success 'setup for objects filter' '\n-\tgit init repo\n+\tgit init repo &&\n+\t(\n+\t\t# Seed the repository with three different sets of objects:\n+\t\t#\n+\t\t#   - The first set is fully packed and has a bitmap.\n+\t\t#   - The second set is packed, but has no bitmap.\n+\t\t#   - The third set is loose.\n+\t\t#\n+\t\t# This ensures that we cover all these types as expected.\n+\t\tcd repo &&\n+\t\ttest_commit first &&\n+\t\tgit repack -Adb &&\n+\t\ttest_commit second &&\n+\t\tgit repack -d &&\n+\t\ttest_commit third\n+\t)\n '\n \n test_expect_success 'objects filter with unknown option' '\n@@ -1365,7 +1380,7 @@ test_expect_success 'objects filter with unknown option' '\n \ttest_cmp expect err\n '\n \n-for option in blob:none blob:limit=1 object:type=tag sparse:oid=1234 tree:1 sparse:path=x\n+for option in blob:limit=1 object:type=tag sparse:oid=1234 tree:1 sparse:path=x\n do\n \ttest_expect_success \"objects filter with unsupported option $option\" '\n \t\tcase \"$option\" in\n@@ -1393,4 +1408,32 @@ test_expect_success 'objects filter: disabled' '\n \ttest_cmp expect.sorted actual.sorted\n '\n \n+test_objects_filter () {\n+\tfilter=\"$1\"\n+\n+\ttest_expect_success \"objects filter: $filter\" '\n+\t\tgit -C repo cat-file --batch-check=\"%(objectname)\" --batch-all-objects --filter=\"$filter\" >actual &&\n+\t\tsort actual >actual.sorted &&\n+\t\tgit -C repo rev-list --objects --no-object-names --all --filter=\"$filter\" --filter-provided-objects >expect &&\n+\t\tsort expect >expect.sorted &&\n+\t\ttest_cmp expect.sorted actual.sorted\n+\t'\n+\n+\ttest_expect_success \"objects filter prints excluded objects: $filter\" '\n+\t\t# Find all objects that would be excluded by the current filter.\n+\t\tgit -C repo rev-list --objects --no-object-names --all >all &&\n+\t\tgit -C repo rev-list --objects --no-object-names --all --filter=\"$filter\" --filter-provided-objects >filtered &&\n+\t\tsort all >all.sorted &&\n+\t\tsort filtered >filtered.sorted &&\n+\t\tcomm -23 all.sorted filtered.sorted >expected.excluded &&\n+\t\ttest_line_count -gt 0 expected.excluded &&\n+\n+\t\tgit -C repo cat-file --batch-check=\"%(objectname)\" --filter=\"$filter\" <expected.excluded >actual &&\n+\t\tawk \"/excluded/{ print \\$1 }\" actual | sort >actual.excluded &&\n+\t\ttest_cmp expected.excluded actual.excluded\n+\t'\n+}\n+\n+test_objects_filter \"blob:none\"\n+\n test_done\n\n-- \n2.49.0.604.gff1f9ca942.dirty\n\n"},{"id":"515507","messageId":"20250402-pks-cat-file-object-type-filter-v3-5-4da6bb21871c@pks.im","threadId":"62984","inReplyTo":"20250402-pks-cat-file-object-type-filter-v3-0-4da6bb21871c@pks.im","subject":"[PATCH v3 05/11] builtin/cat-file: support \"blob:limit=\" objects filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-02T11:13:40Z","receivedAt":"2025-04-02T11:13:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement support for the \"blob:limit=\" filter in git-cat-file(1), which\ncauses us to omit all blobs that are bigger than a certain size.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n Documentation/git-cat-file.adoc |  5 +++++\n builtin/cat-file.c              | 15 ++++++++++++++-\n t/t1006-cat-file.sh             | 18 +++++++++++++++---\n 3 files changed, 34 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-cat-file.adoc b/Documentation/git-cat-file.adoc\nindex afcdb0a4738..48e05e1af52 100644\n--- a/Documentation/git-cat-file.adoc\n+++ b/Documentation/git-cat-file.adoc\n@@ -91,6 +91,11 @@ OPTIONS\n \tprinted at all. The '<filter-spec>' may be one of the following:\n +\n The form '--filter=blob:none' omits all blobs.\n++\n+The form '--filter=blob:limit=<n>[kmg]' omits blobs of size at least n\n+bytes or units.  n may be zero.  The suffixes k, m, and g can be used to name\n+units in KiB, MiB, or GiB.  For example, 'blob:limit=1k' is the same as\n+'blob:limit=1024'.\n \n --path=<path>::\n \tFor use with `--textconv` or `--filters`, to allow specifying an object\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex bcceb646f85..629c6cddcb2 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -483,8 +483,11 @@ static void batch_object_write(const char *obj_name,\n \t\tint ret;\n \n \t\tif (use_mailmap ||\n-\t\t    opt->objects_filter.choice == LOFC_BLOB_NONE)\n+\t\t    opt->objects_filter.choice == LOFC_BLOB_NONE ||\n+\t\t    opt->objects_filter.choice == LOFC_BLOB_LIMIT)\n \t\t\tdata->info.typep = &data->type;\n+\t\tif (opt->objects_filter.choice == LOFC_BLOB_LIMIT)\n+\t\t\tdata->info.sizep = &data->size;\n \n \t\tif (pack)\n \t\t\tret = packed_object_info(the_repository, pack, offset,\n@@ -509,6 +512,15 @@ static void batch_object_write(const char *obj_name,\n \t\t\t\treturn;\n \t\t\t}\n \t\t\tbreak;\n+\t\tcase LOFC_BLOB_LIMIT:\n+\t\t\tif (data->type == OBJ_BLOB &&\n+\t\t\t    data->size >= opt->objects_filter.blob_limit_value) {\n+\t\t\t\tif (!opt->all_objects)\n+\t\t\t\t\treport_object_status(opt, obj_name,\n+\t\t\t\t\t\t\t     &data->oid, \"excluded\");\n+\t\t\t\treturn;\n+\t\t\t}\n+\t\t\tbreak;\n \t\tdefault:\n \t\t\tBUG(\"unsupported objects filter\");\n \t\t}\n@@ -1049,6 +1061,7 @@ int cmd_cat_file(int argc,\n \tcase LOFC_DISABLED:\n \t\tbreak;\n \tcase LOFC_BLOB_NONE:\n+\tcase LOFC_BLOB_LIMIT:\n \t\tif (!batch.enabled)\n \t\t\tusage(_(\"objects filter only supported in batch mode\"));\n \t\tbreak;\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 7404c135b1e..4f14840b71a 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -1356,11 +1356,12 @@ test_expect_success PERL '--batch-command info is unbuffered by default' '\n test_expect_success 'setup for objects filter' '\n \tgit init repo &&\n \t(\n-\t\t# Seed the repository with three different sets of objects:\n+\t\t# Seed the repository with four different sets of objects:\n \t\t#\n \t\t#   - The first set is fully packed and has a bitmap.\n \t\t#   - The second set is packed, but has no bitmap.\n \t\t#   - The third set is loose.\n+\t\t#   - The fourth set is loose and contains big objects.\n \t\t#\n \t\t# This ensures that we cover all these types as expected.\n \t\tcd repo &&\n@@ -1368,7 +1369,14 @@ test_expect_success 'setup for objects filter' '\n \t\tgit repack -Adb &&\n \t\ttest_commit second &&\n \t\tgit repack -d &&\n-\t\ttest_commit third\n+\t\ttest_commit third &&\n+\n+\t\tfor n in 1000 10000\n+\t\tdo\n+\t\t\tprintf \"%\"$n\"s\" X >large.$n || return 1\n+\t\tdone &&\n+\t\tgit add large.* &&\n+\t\tgit commit -m fourth\n \t)\n '\n \n@@ -1380,7 +1388,7 @@ test_expect_success 'objects filter with unknown option' '\n \ttest_cmp expect err\n '\n \n-for option in blob:limit=1 object:type=tag sparse:oid=1234 tree:1 sparse:path=x\n+for option in object:type=tag sparse:oid=1234 tree:1 sparse:path=x\n do\n \ttest_expect_success \"objects filter with unsupported option $option\" '\n \t\tcase \"$option\" in\n@@ -1435,5 +1443,9 @@ test_objects_filter () {\n }\n \n test_objects_filter \"blob:none\"\n+test_objects_filter \"blob:limit=1\"\n+test_objects_filter \"blob:limit=500\"\n+test_objects_filter \"blob:limit=1000\"\n+test_objects_filter \"blob:limit=1k\"\n \n test_done\n\n-- \n2.49.0.604.gff1f9ca942.dirty\n\n"},{"id":"515508","messageId":"20250402-pks-cat-file-object-type-filter-v3-6-4da6bb21871c@pks.im","threadId":"62984","inReplyTo":"20250402-pks-cat-file-object-type-filter-v3-0-4da6bb21871c@pks.im","subject":"[PATCH v3 06/11] builtin/cat-file: support \"object:type=\" objects filter","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-02T11:13:41Z","receivedAt":"2025-04-02T11:13:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement support for the \"object:type=\" filter in git-cat-file(1),\nwhich causes us to omit all objects that don't match the provided object\ntype.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n Documentation/git-cat-file.adoc |  3 +++\n builtin/cat-file.c              | 12 +++++++++++-\n t/t1006-cat-file.sh             |  6 +++++-\n 3 files changed, 19 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-cat-file.adoc b/Documentation/git-cat-file.adoc\nindex 48e05e1af52..74d71c3282e 100644\n--- a/Documentation/git-cat-file.adoc\n+++ b/Documentation/git-cat-file.adoc\n@@ -96,6 +96,9 @@ The form '--filter=blob:limit=<n>[kmg]' omits blobs of size at least n\n bytes or units.  n may be zero.  The suffixes k, m, and g can be used to name\n units in KiB, MiB, or GiB.  For example, 'blob:limit=1k' is the same as\n 'blob:limit=1024'.\n++\n+The form '--filter=object:type=(tag|commit|tree|blob)' omits all objects which\n+are not of the requested type.\n \n --path=<path>::\n \tFor use with `--textconv` or `--filters`, to allow specifying an object\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 629c6cddcb2..0f17175a549 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -484,7 +484,8 @@ static void batch_object_write(const char *obj_name,\n \n \t\tif (use_mailmap ||\n \t\t    opt->objects_filter.choice == LOFC_BLOB_NONE ||\n-\t\t    opt->objects_filter.choice == LOFC_BLOB_LIMIT)\n+\t\t    opt->objects_filter.choice == LOFC_BLOB_LIMIT ||\n+\t\t    opt->objects_filter.choice == LOFC_OBJECT_TYPE)\n \t\t\tdata->info.typep = &data->type;\n \t\tif (opt->objects_filter.choice == LOFC_BLOB_LIMIT)\n \t\t\tdata->info.sizep = &data->size;\n@@ -521,6 +522,14 @@ static void batch_object_write(const char *obj_name,\n \t\t\t\treturn;\n \t\t\t}\n \t\t\tbreak;\n+\t\tcase LOFC_OBJECT_TYPE:\n+\t\t\tif (data->type != opt->objects_filter.object_type) {\n+\t\t\t\tif (!opt->all_objects)\n+\t\t\t\t\treport_object_status(opt, obj_name,\n+\t\t\t\t\t\t\t     &data->oid, \"excluded\");\n+\t\t\t\treturn;\n+\t\t\t}\n+\t\t\tbreak;\n \t\tdefault:\n \t\t\tBUG(\"unsupported objects filter\");\n \t\t}\n@@ -1062,6 +1071,7 @@ int cmd_cat_file(int argc,\n \t\tbreak;\n \tcase LOFC_BLOB_NONE:\n \tcase LOFC_BLOB_LIMIT:\n+\tcase LOFC_OBJECT_TYPE:\n \t\tif (!batch.enabled)\n \t\t\tusage(_(\"objects filter only supported in batch mode\"));\n \t\tbreak;\ndiff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh\nindex 4f14840b71a..98638fa2b9c 100755\n--- a/t/t1006-cat-file.sh\n+++ b/t/t1006-cat-file.sh\n@@ -1388,7 +1388,7 @@ test_expect_success 'objects filter with unknown option' '\n \ttest_cmp expect err\n '\n \n-for option in object:type=tag sparse:oid=1234 tree:1 sparse:path=x\n+for option in sparse:oid=1234 tree:1 sparse:path=x\n do\n \ttest_expect_success \"objects filter with unsupported option $option\" '\n \t\tcase \"$option\" in\n@@ -1447,5 +1447,9 @@ test_objects_filter \"blob:limit=1\"\n test_objects_filter \"blob:limit=500\"\n test_objects_filter \"blob:limit=1000\"\n test_objects_filter \"blob:limit=1k\"\n+test_objects_filter \"object:type=blob\"\n+test_objects_filter \"object:type=commit\"\n+test_objects_filter \"object:type=tag\"\n+test_objects_filter \"object:type=tree\"\n \n test_done\n\n-- \n2.49.0.604.gff1f9ca942.dirty\n\n"},{"id":"515509","messageId":"20250402-pks-cat-file-object-type-filter-v3-7-4da6bb21871c@pks.im","threadId":"62984","inReplyTo":"20250402-pks-cat-file-object-type-filter-v3-0-4da6bb21871c@pks.im","subject":"[PATCH v3 07/11] pack-bitmap: allow passing payloads to `show_reachable_fn()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-02T11:13:42Z","receivedAt":"2025-04-02T11:13:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `show_reachable_fn` callback is used by a couple of functions to\npresent reachable objects to the caller. The function does not provide a\nway for the caller to pass a payload though, which is functionality that\nwe'll require in a subsequent commit.\n\nChange the callback type to accept a payload and adapt all callsites\naccordingly.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/pack-objects.c |  3 ++-\n builtin/rev-list.c     |  3 ++-\n pack-bitmap.c          | 15 ++++++++-------\n pack-bitmap.h          |  3 ++-\n reachable.c            |  3 ++-\n 5 files changed, 16 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex a7e4bb79049..38784613fc0 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1736,7 +1736,8 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n static int add_object_entry_from_bitmap(const struct object_id *oid,\n \t\t\t\t\tenum object_type type,\n \t\t\t\t\tint flags UNUSED, uint32_t name_hash,\n-\t\t\t\t\tstruct packed_git *pack, off_t offset)\n+\t\t\t\t\tstruct packed_git *pack, off_t offset,\n+\t\t\t\t\tvoid *payload UNUSED)\n {\n \tdisplay_progress(progress_state, ++nr_seen);\n \ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex bb26bee0d45..1100dd2abe7 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -429,7 +429,8 @@ static int show_object_fast(\n \tint exclude UNUSED,\n \tuint32_t name_hash UNUSED,\n \tstruct packed_git *found_pack UNUSED,\n-\toff_t found_offset UNUSED)\n+\toff_t found_offset UNUSED,\n+\tvoid *payload UNUSED)\n {\n \tfprintf(stdout, \"%s\\n\", oid_to_hex(oid));\n \treturn 1;\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 6f7fd94c36f..d192fb87da9 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1625,7 +1625,7 @@ static void show_extended_objects(struct bitmap_index *bitmap_git,\n \t\t    (obj->type == OBJ_TAG && !revs->tag_objects))\n \t\t\tcontinue;\n \n-\t\tshow_reach(&obj->oid, obj->type, 0, eindex->hashes[i], NULL, 0);\n+\t\tshow_reach(&obj->oid, obj->type, 0, eindex->hashes[i], NULL, 0, NULL);\n \t}\n }\n \n@@ -1663,7 +1663,8 @@ static void init_type_iterator(struct ewah_or_iterator *it,\n static void show_objects_for_type(\n \tstruct bitmap_index *bitmap_git,\n \tenum object_type object_type,\n-\tshow_reachable_fn show_reach)\n+\tshow_reachable_fn show_reach,\n+\tvoid *payload)\n {\n \tsize_t i = 0;\n \tuint32_t offset;\n@@ -1715,7 +1716,7 @@ static void show_objects_for_type(\n \t\t\tif (bitmap_git->hashes)\n \t\t\t\thash = get_be32(bitmap_git->hashes + index_pos);\n \n-\t\t\tshow_reach(&oid, object_type, 0, hash, pack, ofs);\n+\t\t\tshow_reach(&oid, object_type, 0, hash, pack, ofs, payload);\n \t\t}\n \t}\n \n@@ -2518,13 +2519,13 @@ void traverse_bitmap_commit_list(struct bitmap_index *bitmap_git,\n {\n \tassert(bitmap_git->result);\n \n-\tshow_objects_for_type(bitmap_git, OBJ_COMMIT, show_reachable);\n+\tshow_objects_for_type(bitmap_git, OBJ_COMMIT, show_reachable, NULL);\n \tif (revs->tree_objects)\n-\t\tshow_objects_for_type(bitmap_git, OBJ_TREE, show_reachable);\n+\t\tshow_objects_for_type(bitmap_git, OBJ_TREE, show_reachable, NULL);\n \tif (revs->blob_objects)\n-\t\tshow_objects_for_type(bitmap_git, OBJ_BLOB, show_reachable);\n+\t\tshow_objects_for_type(bitmap_git, OBJ_BLOB, show_reachable, NULL);\n \tif (revs->tag_objects)\n-\t\tshow_objects_for_type(bitmap_git, OBJ_TAG, show_reachable);\n+\t\tshow_objects_for_type(bitmap_git, OBJ_TAG, show_reachable, NULL);\n \n \tshow_extended_objects(bitmap_git, revs, show_reachable);\n }\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex dd0951088f6..de6bf534fef 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -50,7 +50,8 @@ typedef int (*show_reachable_fn)(\n \tint flags,\n \tuint32_t hash,\n \tstruct packed_git *found_pack,\n-\toff_t found_offset);\n+\toff_t found_offset,\n+\tvoid *payload);\n \n struct bitmap_index;\n \ndiff --git a/reachable.c b/reachable.c\nindex 9ee04c89ec6..421d354d3b5 100644\n--- a/reachable.c\n+++ b/reachable.c\n@@ -341,7 +341,8 @@ static int mark_object_seen(const struct object_id *oid,\n \t\t\t     int exclude UNUSED,\n \t\t\t     uint32_t name_hash UNUSED,\n \t\t\t     struct packed_git *found_pack UNUSED,\n-\t\t\t     off_t found_offset UNUSED)\n+\t\t\t     off_t found_offset UNUSED,\n+\t\t\t     void *payload UNUSED)\n {\n \tstruct object *obj = lookup_object_by_type(the_repository, oid, type);\n \tif (!obj)\n\n-- \n2.49.0.604.gff1f9ca942.dirty\n\n"},{"id":"515510","messageId":"20250402-pks-cat-file-object-type-filter-v3-8-4da6bb21871c@pks.im","threadId":"62984","inReplyTo":"20250402-pks-cat-file-object-type-filter-v3-0-4da6bb21871c@pks.im","subject":"[PATCH v3 08/11] pack-bitmap: add function to iterate over filtered bitmapped objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-02T11:13:43Z","receivedAt":"2025-04-02T11:13:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a function that allows the caller to iterate over all\nbitmapped objects that match a given filter. This mechanism will be used\nin a subsequent commit to optimize object filters in git-cat-file(1).\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n pack-bitmap.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++++------\n pack-bitmap.h | 12 ++++++++++++\n 2 files changed, 65 insertions(+), 6 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex d192fb87da9..6adb8aaa1c2 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -1662,6 +1662,7 @@ static void init_type_iterator(struct ewah_or_iterator *it,\n \n static void show_objects_for_type(\n \tstruct bitmap_index *bitmap_git,\n+\tstruct bitmap *objects,\n \tenum object_type object_type,\n \tshow_reachable_fn show_reach,\n \tvoid *payload)\n@@ -1672,8 +1673,6 @@ static void show_objects_for_type(\n \tstruct ewah_or_iterator it;\n \teword_t filter;\n \n-\tstruct bitmap *objects = bitmap_git->result;\n-\n \tinit_type_iterator(&it, bitmap_git, object_type);\n \n \tfor (i = 0; i < objects->word_alloc &&\n@@ -2025,6 +2024,50 @@ static void filter_packed_objects_from_bitmap(struct bitmap_index *bitmap_git,\n \t}\n }\n \n+int for_each_bitmapped_object(struct bitmap_index *bitmap_git,\n+\t\t\t      struct list_objects_filter_options *filter,\n+\t\t\t      show_reachable_fn show_reach,\n+\t\t\t      void *payload)\n+{\n+\tstruct bitmap *filtered_bitmap = NULL;\n+\tuint32_t objects_nr;\n+\tsize_t full_word_count;\n+\tint ret;\n+\n+\tif (!can_filter_bitmap(filter)) {\n+\t\tret = -1;\n+\t\tgoto out;\n+\t}\n+\n+\tobjects_nr = bitmap_num_objects(bitmap_git);\n+\tfull_word_count = objects_nr / BITS_IN_EWORD;\n+\n+\t/* We start from the all-1 bitmap and then filter down from there. */\n+\tfiltered_bitmap = bitmap_word_alloc(full_word_count + !!(objects_nr % BITS_IN_EWORD));\n+\tmemset(filtered_bitmap->words, 0xff, full_word_count * sizeof(*filtered_bitmap->words));\n+\tfor (size_t i = full_word_count * BITS_IN_EWORD; i < objects_nr; i++)\n+\t\tbitmap_set(filtered_bitmap, i);\n+\n+\tif (filter_bitmap(bitmap_git, NULL, filtered_bitmap, filter) < 0) {\n+\t\tret = -1;\n+\t\tgoto out;\n+\t}\n+\n+\tshow_objects_for_type(bitmap_git, filtered_bitmap,\n+\t\t\t      OBJ_COMMIT, show_reach, payload);\n+\tshow_objects_for_type(bitmap_git, filtered_bitmap,\n+\t\t\t      OBJ_TREE, show_reach, payload);\n+\tshow_objects_for_type(bitmap_git, filtered_bitmap,\n+\t\t\t      OBJ_BLOB, show_reach, payload);\n+\tshow_objects_for_type(bitmap_git, filtered_bitmap,\n+\t\t\t      OBJ_TAG, show_reach, payload);\n+\n+\tret = 0;\n+out:\n+\tbitmap_free(filtered_bitmap);\n+\treturn ret;\n+}\n+\n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t\t\t\t\t int filter_provided_objects)\n {\n@@ -2519,13 +2562,17 @@ void traverse_bitmap_commit_list(struct bitmap_index *bitmap_git,\n {\n \tassert(bitmap_git->result);\n \n-\tshow_objects_for_type(bitmap_git, OBJ_COMMIT, show_reachable, NULL);\n+\tshow_objects_for_type(bitmap_git, bitmap_git->result,\n+\t\t\t      OBJ_COMMIT, show_reachable, NULL);\n \tif (revs->tree_objects)\n-\t\tshow_objects_for_type(bitmap_git, OBJ_TREE, show_reachable, NULL);\n+\t\tshow_objects_for_type(bitmap_git, bitmap_git->result,\n+\t\t\t\t      OBJ_TREE, show_reachable, NULL);\n \tif (revs->blob_objects)\n-\t\tshow_objects_for_type(bitmap_git, OBJ_BLOB, show_reachable, NULL);\n+\t\tshow_objects_for_type(bitmap_git, bitmap_git->result,\n+\t\t\t\t      OBJ_BLOB, show_reachable, NULL);\n \tif (revs->tag_objects)\n-\t\tshow_objects_for_type(bitmap_git, OBJ_TAG, show_reachable, NULL);\n+\t\tshow_objects_for_type(bitmap_git, bitmap_git->result,\n+\t\t\t\t      OBJ_TAG, show_reachable, NULL);\n \n \tshow_extended_objects(bitmap_git, revs, show_reachable);\n }\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex de6bf534fef..079bae32466 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -79,6 +79,18 @@ int test_bitmap_pseudo_merges(struct repository *r);\n int test_bitmap_pseudo_merge_commits(struct repository *r, uint32_t n);\n int test_bitmap_pseudo_merge_objects(struct repository *r, uint32_t n);\n \n+struct list_objects_filter_options;\n+\n+/*\n+ * Filter bitmapped objects and iterate through all resulting objects,\n+ * executing `show_reach` for each of them. Returns `-1` in case the filter is\n+ * not supported, `0` otherwise.\n+ */\n+int for_each_bitmapped_object(struct bitmap_index *bitmap_git,\n+\t\t\t      struct list_objects_filter_options *filter,\n+\t\t\t      show_reachable_fn show_reach,\n+\t\t\t      void *payload);\n+\n #define GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL \\\n \t\"GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL\"\n \n\n-- \n2.49.0.604.gff1f9ca942.dirty\n\n"},{"id":"515511","messageId":"20250402-pks-cat-file-object-type-filter-v3-9-4da6bb21871c@pks.im","threadId":"62984","inReplyTo":"20250402-pks-cat-file-object-type-filter-v3-0-4da6bb21871c@pks.im","subject":"[PATCH v3 09/11] pack-bitmap: introduce function to check whether a pack is bitmapped","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-02T11:13:44Z","receivedAt":"2025-04-02T11:13:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a function that allows us to verify whether a pack is\nbitmapped or not. This functionality will be used in a subsequent\ncommit.\n\nHelped-by: Taylor Blau <me@ttaylorr.com>\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n pack-bitmap.c | 15 +++++++++++++++\n pack-bitmap.h |  7 +++++++\n 2 files changed, 22 insertions(+)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 6adb8aaa1c2..edc8f42122d 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -745,6 +745,21 @@ struct bitmap_index *prepare_midx_bitmap_git(struct multi_pack_index *midx)\n \treturn NULL;\n }\n \n+int bitmap_index_contains_pack(struct bitmap_index *bitmap, struct packed_git *pack)\n+{\n+\tfor (; bitmap; bitmap = bitmap->base) {\n+\t\tif (bitmap_is_midx(bitmap)) {\n+\t\t\tfor (size_t i = 0; i < bitmap->midx->num_packs; i++)\n+\t\t\t\tif (bitmap->midx->packs[i] == pack)\n+\t\t\t\t\treturn 1;\n+\t\t} else if (bitmap->pack == pack) {\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\treturn 0;\n+}\n+\n struct include_data {\n \tstruct bitmap_index *bitmap_git;\n \tstruct bitmap *base;\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 079bae32466..55df1b3af5a 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -67,6 +67,13 @@ struct bitmapped_pack {\n \n struct bitmap_index *prepare_bitmap_git(struct repository *r);\n struct bitmap_index *prepare_midx_bitmap_git(struct multi_pack_index *midx);\n+\n+/*\n+ * Given a bitmap index, determine whether it contains the pack either directly\n+ * or via the multi-pack-index.\n+ */\n+int bitmap_index_contains_pack(struct bitmap_index *bitmap, struct packed_git *pack);\n+\n void count_bitmap_commit_list(struct bitmap_index *, uint32_t *commits,\n \t\t\t      uint32_t *trees, uint32_t *blobs, uint32_t *tags);\n void traverse_bitmap_commit_list(struct bitmap_index *,\n\n-- \n2.49.0.604.gff1f9ca942.dirty\n\n"},{"id":"515512","messageId":"20250402-pks-cat-file-object-type-filter-v3-10-4da6bb21871c@pks.im","threadId":"62984","inReplyTo":"20250402-pks-cat-file-object-type-filter-v3-0-4da6bb21871c@pks.im","subject":"[PATCH v3 10/11] builtin/cat-file: deduplicate logic to iterate over all objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-02T11:13:45Z","receivedAt":"2025-04-02T11:13:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Pull out a common function that allows us to iterate over all objects in\na repository. Right now the logic is trivial and would only require two\nfunction calls, making this refactoring a bit pointless. But in the next\ncommit we will iterate on this logic to make use of bitmaps, so this is\nabout to become a bit more complex.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/cat-file.c | 85 ++++++++++++++++++++++++++++++------------------------\n 1 file changed, 48 insertions(+), 37 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 0f17175a549..b0c758eca02 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -642,25 +642,18 @@ static int batch_object_cb(const struct object_id *oid, void *vdata)\n \treturn 0;\n }\n \n-static int collect_loose_object(const struct object_id *oid,\n-\t\t\t\tconst char *path UNUSED,\n-\t\t\t\tvoid *data)\n-{\n-\toid_array_append(data, oid);\n-\treturn 0;\n-}\n-\n-static int collect_packed_object(const struct object_id *oid,\n-\t\t\t\t struct packed_git *pack UNUSED,\n-\t\t\t\t uint32_t pos UNUSED,\n-\t\t\t\t void *data)\n+static int collect_object(const struct object_id *oid,\n+\t\t\t  struct packed_git *pack UNUSED,\n+\t\t\t  off_t offset UNUSED,\n+\t\t\t  void *data)\n {\n \toid_array_append(data, oid);\n \treturn 0;\n }\n \n static int batch_unordered_object(const struct object_id *oid,\n-\t\t\t\t  struct packed_git *pack, off_t offset,\n+\t\t\t\t  struct packed_git *pack,\n+\t\t\t\t  off_t offset,\n \t\t\t\t  void *vdata)\n {\n \tstruct object_cb_data *data = vdata;\n@@ -674,23 +667,6 @@ static int batch_unordered_object(const struct object_id *oid,\n \treturn 0;\n }\n \n-static int batch_unordered_loose(const struct object_id *oid,\n-\t\t\t\t const char *path UNUSED,\n-\t\t\t\t void *data)\n-{\n-\treturn batch_unordered_object(oid, NULL, 0, data);\n-}\n-\n-static int batch_unordered_packed(const struct object_id *oid,\n-\t\t\t\t  struct packed_git *pack,\n-\t\t\t\t  uint32_t pos,\n-\t\t\t\t  void *data)\n-{\n-\treturn batch_unordered_object(oid, pack,\n-\t\t\t\t      nth_packed_object_offset(pack, pos),\n-\t\t\t\t      data);\n-}\n-\n typedef void (*parse_cmd_fn_t)(struct batch_options *, const char *,\n \t\t\t       struct strbuf *, struct expand_data *);\n \n@@ -823,6 +799,45 @@ static void batch_objects_command(struct batch_options *opt,\n \n #define DEFAULT_FORMAT \"%(objectname) %(objecttype) %(objectsize)\"\n \n+typedef int (*for_each_object_fn)(const struct object_id *oid, struct packed_git *pack,\n+\t\t\t\t  off_t offset, void *data);\n+\n+struct for_each_object_payload {\n+\tfor_each_object_fn callback;\n+\tvoid *payload;\n+};\n+\n+static int batch_one_object_loose(const struct object_id *oid,\n+\t\t\t\t  const char *path UNUSED,\n+\t\t\t\t  void *_payload)\n+{\n+\tstruct for_each_object_payload *payload = _payload;\n+\treturn payload->callback(oid, NULL, 0, payload->payload);\n+}\n+\n+static int batch_one_object_packed(const struct object_id *oid,\n+\t\t\t\t   struct packed_git *pack,\n+\t\t\t\t   uint32_t pos,\n+\t\t\t\t   void *_payload)\n+{\n+\tstruct for_each_object_payload *payload = _payload;\n+\treturn payload->callback(oid, pack, nth_packed_object_offset(pack, pos),\n+\t\t\t\t payload->payload);\n+}\n+\n+static void batch_each_object(for_each_object_fn callback,\n+\t\t\t      unsigned flags,\n+\t\t\t      void *_payload)\n+{\n+\tstruct for_each_object_payload payload = {\n+\t\t.callback = callback,\n+\t\t.payload = _payload,\n+\t};\n+\tfor_each_loose_object(batch_one_object_loose, &payload, 0);\n+\tfor_each_packed_object(the_repository, batch_one_object_packed,\n+\t\t\t       &payload, flags);\n+}\n+\n static int batch_objects(struct batch_options *opt)\n {\n \tstruct strbuf input = STRBUF_INIT;\n@@ -877,18 +892,14 @@ static int batch_objects(struct batch_options *opt)\n \n \t\t\tcb.seen = &seen;\n \n-\t\t\tfor_each_loose_object(batch_unordered_loose, &cb, 0);\n-\t\t\tfor_each_packed_object(the_repository, batch_unordered_packed,\n-\t\t\t\t\t       &cb, FOR_EACH_OBJECT_PACK_ORDER);\n+\t\t\tbatch_each_object(batch_unordered_object,\n+\t\t\t\t\t  FOR_EACH_OBJECT_PACK_ORDER, &cb);\n \n \t\t\toidset_clear(&seen);\n \t\t} else {\n \t\t\tstruct oid_array sa = OID_ARRAY_INIT;\n \n-\t\t\tfor_each_loose_object(collect_loose_object, &sa, 0);\n-\t\t\tfor_each_packed_object(the_repository, collect_packed_object,\n-\t\t\t\t\t       &sa, 0);\n-\n+\t\t\tbatch_each_object(collect_object, 0, &sa);\n \t\t\toid_array_for_each_unique(&sa, batch_object_cb, &cb);\n \n \t\t\toid_array_clear(&sa);\n\n-- \n2.49.0.604.gff1f9ca942.dirty\n\n"},{"id":"515513","messageId":"20250402-pks-cat-file-object-type-filter-v3-11-4da6bb21871c@pks.im","threadId":"62984","inReplyTo":"20250402-pks-cat-file-object-type-filter-v3-0-4da6bb21871c@pks.im","subject":"[PATCH v3 11/11] builtin/cat-file: use bitmaps to efficiently filter by object type","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-02T11:13:46Z","receivedAt":"2025-04-02T11:13:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"While it is now possible to filter objects by type, this mechanism is\nfor now mostly a convenience. Most importantly, we still have to iterate\nthrough the whole packfile to find all objects of a specific type. This\ncan be prohibitively expensive depending on the size of the packfiles.\n\nIt isn't really possible to do better than this when only considering a\npackfile itself, as the order of objects is not fixed. But when we have\na packfile with a corresponding bitmap, either because the packfile\nitself has one or because the multi-pack index has a bitmap for it, then\nwe can use these bitmaps to improve the runtime.\n\nWhile bitmaps are typically used to compute reachability of objects,\nthey also contain one bitmap per object type that encodes which object\nhas what type. So instead of reading through the whole packfile(s), we\ncan use the bitmaps and iterate through the type-specific bitmap.\nTypically, only a subset of packfiles will have a bitmap. But this isn't\nreally much of a problem: we can use bitmaps when available, and then\nuse the non-bitmap walk for every packfile that isn't covered by one.\n\nOverall, this leads to quite a significant speedup depending on how many\nobjects of a certain type exist. The following benchmarks have been\nexecuted in the Chromium repository, which has a 50GB packfile with\nalmost 25 million objects. As expected, there isn't really much of a\nchange in performance without an object filter:\n\n    Benchmark 1: cat-file with no-filter (revision = HEAD~)\n      Time (mean ± σ):     89.675 s ±  4.527 s    [User: 40.807 s, System: 10.782 s]\n      Range (min … max):   83.052 s … 96.084 s    10 runs\n\n    Benchmark 2: cat-file with no-filter (revision = HEAD)\n      Time (mean ± σ):     88.991 s ±  2.488 s    [User: 42.278 s, System: 10.305 s]\n      Range (min … max):   82.843 s … 91.271 s    10 runs\n\n    Summary\n      cat-file with no-filter (revision = HEAD) ran\n        1.01 ± 0.06 times faster than cat-file with no-filter (revision = HEAD~)\n\nWe still have to scan through all objects as we yield all of them, so\nusing the bitmap in this case doesn't really buy us anything. What is\nnoticeable in this benchmark is that we're I/O-bound, not CPU-bound, as\ncan be seen from the user/system runtimes, which combined are way lower\nthan the overall benchmarked runtime.\n\nBut when we do use a filter we can see a significant improvement:\n\n    Benchmark 1: cat-file with filter=object:type=commit (revision = HEAD~)\n      Time (mean ± σ):     86.444 s ±  4.081 s    [User: 36.830 s, System: 11.312 s]\n      Range (min … max):   80.305 s … 93.104 s    10 runs\n\n    Benchmark 2: cat-file with filter=object:type=commit (revision = HEAD)\n      Time (mean ± σ):      2.089 s ±  0.015 s    [User: 1.872 s, System: 0.207 s]\n      Range (min … max):    2.073 s …  2.119 s    10 runs\n\n    Summary\n      cat-file with filter=object:type=commit (revision = HEAD) ran\n       41.38 ± 1.98 times faster than cat-file with filter=object:type=commit (revision = HEAD~)\n\nThis is because we don't have to scan through all packfiles anymore, but\ncan instead directly look up relevant objects.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/cat-file.c | 42 +++++++++++++++++++++++++++++++++++++-----\n 1 file changed, 37 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex b0c758eca02..ead7554a57a 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -21,6 +21,7 @@\n #include \"streaming.h\"\n #include \"oid-array.h\"\n #include \"packfile.h\"\n+#include \"pack-bitmap.h\"\n #include \"object-file.h\"\n #include \"object-name.h\"\n #include \"object-store-ll.h\"\n@@ -825,7 +826,20 @@ static int batch_one_object_packed(const struct object_id *oid,\n \t\t\t\t payload->payload);\n }\n \n-static void batch_each_object(for_each_object_fn callback,\n+static int batch_one_object_bitmapped(const struct object_id *oid,\n+\t\t\t\t      enum object_type type UNUSED,\n+\t\t\t\t      int flags UNUSED,\n+\t\t\t\t      uint32_t hash UNUSED,\n+\t\t\t\t      struct packed_git *pack,\n+\t\t\t\t      off_t offset,\n+\t\t\t\t      void *_payload)\n+{\n+\tstruct for_each_object_payload *payload = _payload;\n+\treturn payload->callback(oid, pack, offset, payload->payload);\n+}\n+\n+static void batch_each_object(struct batch_options *opt,\n+\t\t\t      for_each_object_fn callback,\n \t\t\t      unsigned flags,\n \t\t\t      void *_payload)\n {\n@@ -833,9 +847,27 @@ static void batch_each_object(for_each_object_fn callback,\n \t\t.callback = callback,\n \t\t.payload = _payload,\n \t};\n+\tstruct bitmap_index *bitmap = prepare_bitmap_git(the_repository);\n+\n \tfor_each_loose_object(batch_one_object_loose, &payload, 0);\n-\tfor_each_packed_object(the_repository, batch_one_object_packed,\n-\t\t\t       &payload, flags);\n+\n+\tif (bitmap && !for_each_bitmapped_object(bitmap, &opt->objects_filter,\n+\t\t\t\t\t\t batch_one_object_bitmapped, &payload)) {\n+\t\tstruct packed_git *pack;\n+\n+\t\tfor (pack = get_all_packs(the_repository); pack; pack = pack->next) {\n+\t\t\tif (bitmap_index_contains_pack(bitmap, pack) ||\n+\t\t\t    open_pack_index(pack))\n+\t\t\t\tcontinue;\n+\t\t\tfor_each_object_in_pack(pack, batch_one_object_packed,\n+\t\t\t\t\t\t&payload, flags);\n+\t\t}\n+\t} else {\n+\t\tfor_each_packed_object(the_repository, batch_one_object_packed,\n+\t\t\t\t       &payload, flags);\n+\t}\n+\n+\tfree_bitmap_index(bitmap);\n }\n \n static int batch_objects(struct batch_options *opt)\n@@ -892,14 +924,14 @@ static int batch_objects(struct batch_options *opt)\n \n \t\t\tcb.seen = &seen;\n \n-\t\t\tbatch_each_object(batch_unordered_object,\n+\t\t\tbatch_each_object(opt, batch_unordered_object,\n \t\t\t\t\t  FOR_EACH_OBJECT_PACK_ORDER, &cb);\n \n \t\t\toidset_clear(&seen);\n \t\t} else {\n \t\t\tstruct oid_array sa = OID_ARRAY_INIT;\n \n-\t\t\tbatch_each_object(collect_object, 0, &sa);\n+\t\t\tbatch_each_object(opt, collect_object, 0, &sa);\n \t\t\toid_array_for_each_unique(&sa, batch_object_cb, &cb);\n \n \t\t\toid_array_clear(&sa);\n\n-- \n2.49.0.604.gff1f9ca942.dirty\n\n"},{"id":"515567","messageId":"CAOLa=ZTY4AE4ONHoZX+VrHRC9461utdV6g7FFbWtnGuay55G9g@mail.gmail.com","threadId":"62984","inReplyTo":"20250402-pks-cat-file-object-type-filter-v3-0-4da6bb21871c@pks.im","subject":"Re: [PATCH v3 00/11] builtin/cat-file: allow filtering objects in batch mode","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-04-03T08:17:38Z","receivedAt":"2025-04-03T08:17:41Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> Hi,\n>\n> at GitLab, we sometimes have the need to list all objects regardless of\n> their reachability. We use git-cat-file(1) with `--batch-all-objects` to\n> do this, and typically this is quite a good fit. In some cases though,\n> we only want to list objects of a specific type, where we then basically\n> have the following pipeline:\n>\n>     git cat-file --batch-all-objects --batch-check='%(objecttype) %(objectname)' |\n>     grep '^commit ' |\n>     cut -d' ' -f2 |\n>     git cat-file --batch\n>\n> This works okayish in medium-sized repositories, but once you reach a\n> certain size this isn't really an option anymore. In the Chromium\n> repository for example [1] simply listing all objects in the first\n> invocation of git-cat-file(1) takes around 80 to 100 seconds. The\n> workload is completely I/O-bottlenecked: my machine reads at ~500MB/s,\n> and the packfile is 50GB in size, which matches the 100 seconds that I\n> observe.\n>\n> This series addresses the issue by introducing object filters into\n> git-cat-file(1). These object filters use the exact same syntax as the\n> filters we have in git-rev-list(1), but only a subset of them is\n> supported because not all filters can be computed by git-cat-file(1).\n> Supported are \"blob:none\", \"blob:limit=\" as well as \"object:type=\".\n>\n> The filters alone don't really help though: we still have to scan\n> through the whole packfile in order to compute the packfiles. While we\n> are able to shed a bit of CPU time because we can stop emitting some of\n> the objects, we're still I/O-bottlenecked.\n>\n> The second part of the series thus expands the filters so that they can\n> make use of bitmap indices for some of the filters, if available. This\n> allows us to efficiently answer the question where to find all objects\n> of a specific type, and thus we can avoid scanning through the packfile\n> and instead directly look up relevant objects, leading to a significant\n> speedup:\n>\n>     Benchmark 1: cat-file with filter=object:type=commit (revision = HEAD~)\n>       Time (mean ± σ):     86.444 s ±  4.081 s    [User: 36.830 s, System: 11.312 s]\n>       Range (min … max):   80.305 s … 93.104 s    10 runs\n>\n>     Benchmark 2: cat-file with filter=object:type=commit (revision = HEAD)\n>       Time (mean ± σ):      2.089 s ±  0.015 s    [User: 1.872 s, System: 0.207 s]\n>       Range (min … max):    2.073 s …  2.119 s    10 runs\n>\n>     Summary\n>       cat-file with filter=object:type=commit (revision = HEAD) ran\n>        41.38 ± 1.98 times faster than cat-file with filter=object:type=commit (revision = HEAD~)\n>\n> We now directly scale with the number of objects of a specific type\n> contained in the packfile instead of scaling with the overall number of\n> objects. It's quite fun to see how the math plays out: if you sum up the\n> times for each of the types you arrive at the time for the unfiltered\n> case.\n>\n> Changes in v2:\n>   - The series is now built on top of \"master\" at 683c54c999c (Git 2.49,\n>     2025-03-14) with \"tb/incremental-midx-part-2\" at 27afc272c49 (midx:\n>     implement writing incremental MIDX bitmaps, 2025-03-20) merged into\n>     it.\n>   - Rename the filter options to \"--filter=\" to match\n>     git-pack-objects(1).\n>   - The bitmap-filtering is now reusing existing mechanisms that we\n>     already have in \"pack-bitmap.c\", as proposed by Taylor.\n>   - Link to v1: https://lore.kernel.org/r/20250221-pks-cat-file-object-type-filter-v1-0-0852530888e2@pks.im\n>\n> Changes in v3:\n>   - Wrap some overly long lines.\n>   - Better describe how filters interact with the different batch modes.\n>   - Adapt the format with `--batch` and `--batch-check` so that we tell\n>     the user that the object has been excluded.\n>   - Add a test for \"--no-filter\".\n>   - Use `OPT_PARSE_LIST_OBJECTS_FILTER()`.\n>   - Link to v2: https://lore.kernel.org/r/20250327-pks-cat-file-object-type-filter-v2-0-4bbc7085d7c5@pks.im\n>\n> Thanks!\n>\n> Patrick\n>\n> [1]: https://github.com/chromium/chromium.git\n>\n> ---\n> Patrick Steinhardt (11):\n>       builtin/cat-file: rename variable that tracks usage\n>       builtin/cat-file: introduce function to report object status\n>       builtin/cat-file: wire up an option to filter objects\n>       builtin/cat-file: support \"blob:none\" objects filter\n>       builtin/cat-file: support \"blob:limit=\" objects filter\n>       builtin/cat-file: support \"object:type=\" objects filter\n>       pack-bitmap: allow passing payloads to `show_reachable_fn()`\n>       pack-bitmap: add function to iterate over filtered bitmapped objects\n>       pack-bitmap: introduce function to check whether a pack is bitmapped\n>       builtin/cat-file: deduplicate logic to iterate over all objects\n>       builtin/cat-file: use bitmaps to efficiently filter by object type\n>\n>  Documentation/git-cat-file.adoc |  26 ++++\n>  builtin/cat-file.c              | 256 +++++++++++++++++++++++++++++-----------\n>  builtin/pack-objects.c          |   3 +-\n>  builtin/rev-list.c              |   3 +-\n>  pack-bitmap.c                   |  81 +++++++++++--\n>  pack-bitmap.h                   |  22 +++-\n>  reachable.c                     |   3 +-\n>  t/t1006-cat-file.sh             |  99 ++++++++++++++++\n>  8 files changed, 411 insertions(+), 82 deletions(-)\n>\n> Range-diff versus v2:\n>\n>  1:  a75888e0bf4 !  1:  b0642b6c495 builtin/cat-file: rename variable that tracks usage\n>     @@ builtin/cat-file.c: int cmd_cat_file(int argc,\n>       \t\t;\n>       \telse if (batch.follow_symlinks)\n>      -\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), usage, options,\n>     -+\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage, options,\n>     - \t\t\t       \"--follow-symlinks\");\n>     +-\t\t\t       \"--follow-symlinks\");\n>     ++\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage,\n>     ++\t\t\t       options, \"--follow-symlinks\");\n>       \telse if (batch.buffer_output >= 0)\n>      -\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), usage, options,\n>     -+\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage, options,\n>     - \t\t\t       \"--buffer\");\n>     +-\t\t\t       \"--buffer\");\n>     ++\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage,\n>     ++\t\t\t       options, \"--buffer\");\n>       \telse if (batch.all_objects)\n>      -\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), usage, options,\n>     -+\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage, options,\n>     - \t\t\t       \"--batch-all-objects\");\n>     +-\t\t\t       \"--batch-all-objects\");\n>     ++\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage,\n>     ++\t\t\t       options, \"--batch-all-objects\");\n>       \telse if (input_nul_terminated)\n>      -\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), usage, options,\n>     -+\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage, options,\n>     - \t\t\t       \"-z\");\n>     +-\t\t\t       \"-z\");\n>     ++\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage,\n>     ++\t\t\t       options, \"-z\");\n>       \telse if (nul_terminated)\n>      -\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), usage, options,\n>     -+\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage, options,\n>     - \t\t\t       \"-Z\");\n>     +-\t\t\t       \"-Z\");\n>     ++\t\tusage_msg_optf(_(\"'%s' requires a batch mode\"), builtin_catfile_usage,\n>     ++\t\t\t       options, \"-Z\");\n>\n>       \tbatch.input_delim = batch.output_delim = '\\n';\n>     + \tif (input_nul_terminated)\n>      @@ builtin/cat-file.c: int cmd_cat_file(int argc,\n>       \t\t\tbatch.transform_mode = opt;\n>       \t\telse if (opt && opt != 'b')\n>     @@ builtin/cat-file.c: int cmd_cat_file(int argc,\n>      +\t\t\t\t       builtin_catfile_usage, options, opt);\n>       \t\telse if (argc)\n>      -\t\t\tusage_msg_opt(_(\"batch modes take no arguments\"), usage,\n>     -+\t\t\tusage_msg_opt(_(\"batch modes take no arguments\"), builtin_catfile_usage,\n>     - \t\t\t\t      options);\n>     +-\t\t\t\t      options);\n>     ++\t\t\tusage_msg_opt(_(\"batch modes take no arguments\"),\n>     ++\t\t\t\t      builtin_catfile_usage, options);\n>\n>       \t\treturn batch_objects(&batch);\n>     + \t}\n>      @@ builtin/cat-file.c: int cmd_cat_file(int argc,\n>       \tif (opt) {\n>       \t\tif (!argc && opt == 'c')\n>       \t\t\tusage_msg_optf(_(\"<rev> required with '%s'\"),\n>      -\t\t\t\t       usage, options, \"--textconv\");\n>     -+\t\t\t\t       builtin_catfile_usage, options, \"--textconv\");\n>     ++\t\t\t\t       builtin_catfile_usage, options,\n>     ++\t\t\t\t       \"--textconv\");\n>       \t\telse if (!argc && opt == 'w')\n>       \t\t\tusage_msg_optf(_(\"<rev> required with '%s'\"),\n>      -\t\t\t\t       usage, options, \"--filters\");\n>     -+\t\t\t\t       builtin_catfile_usage, options, \"--filters\");\n>     ++\t\t\t\t       builtin_catfile_usage, options,\n>     ++\t\t\t\t       \"--filters\");\n>       \t\telse if (!argc && opt_epts)\n>       \t\t\tusage_msg_optf(_(\"<object> required with '-%c'\"),\n>      -\t\t\t\t       usage, options, opt);\n>     @@ builtin/cat-file.c: int cmd_cat_file(int argc,\n>       \t\t\tobj_name = argv[0];\n>       \t\telse\n>      -\t\t\tusage_msg_opt(_(\"too many arguments\"), usage, options);\n>     -+\t\t\tusage_msg_opt(_(\"too many arguments\"), builtin_catfile_usage, options);\n>     ++\t\t\tusage_msg_opt(_(\"too many arguments\"), builtin_catfile_usage,\n>     ++\t\t\t\t      options);\n>       \t} else if (!argc) {\n>      -\t\tusage_with_options(usage, options);\n>      +\t\tusage_with_options(builtin_catfile_usage, options);\n>  -:  ----------- >  2:  18353ba706d builtin/cat-file: introduce function to report object status\n>  2:  bee9407c1a9 !  3:  1e46af5d07b builtin/cat-file: wire up an option to filter objects\n>     @@ Documentation/git-cat-file.adoc: OPTIONS\n>      +--filter=<filter-spec>::\n>      +--no-filter::\n>      +\tOmit objects from the list of printed objects. This can only be used in\n>     -+\tcombination with one of the batched modes. The '<filter-spec>' may be\n>     -+\tone of the following:\n>     ++\tcombination with one of the batched modes. Excluded objects that have\n>     ++\tbeen explicitly requested via any of the batch modes that read objects\n>     ++\tvia standard input (`--batch`, `--batch-check`) will be reported as\n>     ++\t\"filtered\". Excluded objects in `--batch-all-objects` mode will not be\n>     ++\tprinted at all. No filters are supported yet.\n>      +\n>       --path=<path>::\n>       \tFor use with `--textconv` or `--filters`, to allow specifying an object\n>       \tname and a path separately, e.g. when it is difficult to figure out\n>     +@@ Documentation/git-cat-file.adoc: the repository, then `cat-file` will ignore any custom format and print:\n>     + <object> SP missing LF\n>     + ------------\n>     +\n>     ++If a name is specified on stdin that is filtered out via `--filter=`,\n>     ++then `cat-file` will ignore any custom format and print:\n>     ++\n>     ++------------\n>     ++<object> SP excluded LF\n>     ++------------\n>     ++\n>     + If a name is specified that might refer to more than one object (an ambiguous short sha), then `cat-file` will ignore any custom format and print:\n>     +\n>     + ------------\n>\n>       ## builtin/cat-file.c ##\n>      @@\n>     @@ builtin/cat-file.c: int cmd_cat_file(int argc,\n>       \t\t\t    N_(\"run filters on object's content\"), 'w'),\n>       \t\tOPT_STRING(0, \"path\", &force_path, N_(\"blob|tree\"),\n>       \t\t\t   N_(\"use a <path> for (--textconv | --filters); Not with 'batch'\")),\n>     -+\t\tOPT_CALLBACK(0, \"filter\", &batch.objects_filter, N_(\"args\"),\n>     -+\t\t\t     N_(\"object filtering\"), opt_parse_list_objects_filter),\n>     ++\t\tOPT_PARSE_LIST_OBJECTS_FILTER(&batch.objects_filter),\n>       \t\tOPT_END()\n>       \t};\n>\n>     @@ builtin/cat-file.c: int cmd_cat_file(int argc,\n>       \tif (opt == 'b')\n>       \t\tbatch.all_objects = 1;\n>      @@ builtin/cat-file.c: int cmd_cat_file(int argc,\n>     - \t\t\tusage_msg_opt(_(\"batch modes take no arguments\"), builtin_catfile_usage,\n>     - \t\t\t\t      options);\n>     + \t\t\tusage_msg_opt(_(\"batch modes take no arguments\"),\n>     + \t\t\t\t      builtin_catfile_usage, options);\n>\n>      -\t\treturn batch_objects(&batch);\n>      +\t\tret = batch_objects(&batch);\n>     @@ t/t1006-cat-file.sh: test_expect_success PERL '--batch-command info is unbuffere\n>      +\t\ttest_cmp expect err\n>      +\t'\n>      +done\n>     ++\n>     ++test_expect_success 'objects filter: disabled' '\n>     ++\tgit -C repo cat-file --batch-check=\"%(objectname)\" --batch-all-objects --no-filter >actual &&\n>     ++\tsort actual >actual.sorted &&\n>     ++\tgit -C repo rev-list --objects --no-object-names --all >expect &&\n>     ++\tsort expect >expect.sorted &&\n>     ++\ttest_cmp expect.sorted actual.sorted\n>     ++'\n>      +\n>       test_done\n>  3:  ec1d0c63de6 !  4:  878ae8e2a76 builtin/cat-file: support \"blob:none\" objects filter\n>     @@ Commit message\n>          Implement support for the \"blob:none\" filter in git-cat-file(1), which\n>          causes us to omit all blobs.\n>\n>     +    Note that this new filter requires us to read the object type via\n>     +    `oid_object_info_extended()` in `batch_object_write()`. But as we try to\n>     +    optimize away reading objects from the database the `data->info.typep`\n>     +    pointer may not be set. We thus have to adapt the logic to conditionally\n>     +    set the pointer in cases where the filter is given.\n>     +\n>          Signed-off-by: Patrick Steinhardt <ps@pks.im>\n>\n>       ## Documentation/git-cat-file.adoc ##\n>      @@ Documentation/git-cat-file.adoc: OPTIONS\n>     - \tOmit objects from the list of printed objects. This can only be used in\n>     - \tcombination with one of the batched modes. The '<filter-spec>' may be\n>     - \tone of the following:\n>     + \tbeen explicitly requested via any of the batch modes that read objects\n>     + \tvia standard input (`--batch`, `--batch-check`) will be reported as\n>     + \t\"filtered\". Excluded objects in `--batch-all-objects` mode will not be\n>     +-\tprinted at all. No filters are supported yet.\n>     ++\tprinted at all. The '<filter-spec>' may be one of the following:\n>      ++\n>      +The form '--filter=blob:none' omits all blobs.\n>\n>     @@ builtin/cat-file.c: static void batch_object_write(const char *obj_name,\n>       \t\tcase LOFC_DISABLED:\n>       \t\t\tbreak;\n>      +\t\tcase LOFC_BLOB_NONE:\n>     -+\t\t\tif (data->type == OBJ_BLOB)\n>     ++\t\t\tif (data->type == OBJ_BLOB) {\n>     ++\t\t\t\tif (!opt->all_objects)\n>     ++\t\t\t\t\treport_object_status(opt, obj_name,\n>     ++\t\t\t\t\t\t\t     &data->oid, \"excluded\");\n>      +\t\t\t\treturn;\n>     ++\t\t\t}\n>      +\t\t\tbreak;\n>       \t\tdefault:\n>       \t\t\tBUG(\"unsupported objects filter\");\n>     @@ t/t1006-cat-file.sh: test_expect_success 'objects filter with unknown option' '\n>       do\n>       \ttest_expect_success \"objects filter with unsupported option $option\" '\n>       \t\tcase \"$option\" in\n>     -@@ t/t1006-cat-file.sh: do\n>     - \t'\n>     - done\n>     +@@ t/t1006-cat-file.sh: test_expect_success 'objects filter: disabled' '\n>     + \ttest_cmp expect.sorted actual.sorted\n>     + '\n>\n>      +test_objects_filter () {\n>      +\tfilter=\"$1\"\n>     @@ t/t1006-cat-file.sh: do\n>      +\t\tsort expect >expect.sorted &&\n>      +\t\ttest_cmp expect.sorted actual.sorted\n>      +\t'\n>     ++\n>     ++\ttest_expect_success \"objects filter prints excluded objects: $filter\" '\n>     ++\t\t# Find all objects that would be excluded by the current filter.\n>     ++\t\tgit -C repo rev-list --objects --no-object-names --all >all &&\n>     ++\t\tgit -C repo rev-list --objects --no-object-names --all --filter=\"$filter\" --filter-provided-objects >filtered &&\n>     ++\t\tsort all >all.sorted &&\n>     ++\t\tsort filtered >filtered.sorted &&\n>     ++\t\tcomm -23 all.sorted filtered.sorted >expected.excluded &&\n>     ++\t\ttest_line_count -gt 0 expected.excluded &&\n>     ++\n>     ++\t\tgit -C repo cat-file --batch-check=\"%(objectname)\" --filter=\"$filter\" <expected.excluded >actual &&\n>     ++\t\tawk \"/excluded/{ print \\$1 }\" actual | sort >actual.excluded &&\n>     ++\t\ttest_cmp expected.excluded actual.excluded\n>     ++\t'\n>      +}\n>      +\n>      +test_objects_filter \"blob:none\"\n>  4:  a3ed054994d !  5:  a88d5d4b60a builtin/cat-file: support \"blob:limit=\" objects filter\n>     @@ Commit message\n>\n>       ## Documentation/git-cat-file.adoc ##\n>      @@ Documentation/git-cat-file.adoc: OPTIONS\n>     - \tone of the following:\n>     + \tprinted at all. The '<filter-spec>' may be one of the following:\n>       +\n>       The form '--filter=blob:none' omits all blobs.\n>      ++\n>     @@ builtin/cat-file.c: static void batch_object_write(const char *obj_name,\n>       \t\tif (pack)\n>       \t\t\tret = packed_object_info(the_repository, pack, offset,\n>      @@ builtin/cat-file.c: static void batch_object_write(const char *obj_name,\n>     - \t\t\tif (data->type == OBJ_BLOB)\n>       \t\t\t\treturn;\n>     + \t\t\t}\n>       \t\t\tbreak;\n>      +\t\tcase LOFC_BLOB_LIMIT:\n>      +\t\t\tif (data->type == OBJ_BLOB &&\n>     -+\t\t\t    data->size >= opt->objects_filter.blob_limit_value)\n>     ++\t\t\t    data->size >= opt->objects_filter.blob_limit_value) {\n>     ++\t\t\t\tif (!opt->all_objects)\n>     ++\t\t\t\t\treport_object_status(opt, obj_name,\n>     ++\t\t\t\t\t\t\t     &data->oid, \"excluded\");\n>      +\t\t\t\treturn;\n>     ++\t\t\t}\n>      +\t\t\tbreak;\n>       \t\tdefault:\n>       \t\t\tBUG(\"unsupported objects filter\");\n>     @@ t/t1006-cat-file.sh: test_objects_filter () {\n>      +test_objects_filter \"blob:limit=1\"\n>      +test_objects_filter \"blob:limit=500\"\n>      +test_objects_filter \"blob:limit=1000\"\n>     -+test_objects_filter \"blob:limit=1g\"\n>     ++test_objects_filter \"blob:limit=1k\"\n>\n>       test_done\n>  5:  8e39cd218c2 !  6:  13be54300c9 builtin/cat-file: support \"object:type=\" objects filter\n>     @@ builtin/cat-file.c: static void batch_object_write(const char *obj_name,\n>       \t\tif (opt->objects_filter.choice == LOFC_BLOB_LIMIT)\n>       \t\t\tdata->info.sizep = &data->size;\n>      @@ builtin/cat-file.c: static void batch_object_write(const char *obj_name,\n>     - \t\t\t    data->size >= opt->objects_filter.blob_limit_value)\n>       \t\t\t\treturn;\n>     + \t\t\t}\n>       \t\t\tbreak;\n>      +\t\tcase LOFC_OBJECT_TYPE:\n>     -+\t\t\tif (data->type != opt->objects_filter.object_type)\n>     ++\t\t\tif (data->type != opt->objects_filter.object_type) {\n>     ++\t\t\t\tif (!opt->all_objects)\n>     ++\t\t\t\t\treport_object_status(opt, obj_name,\n>     ++\t\t\t\t\t\t\t     &data->oid, \"excluded\");\n>      +\t\t\t\treturn;\n>     ++\t\t\t}\n>      +\t\t\tbreak;\n>       \t\tdefault:\n>       \t\t\tBUG(\"unsupported objects filter\");\n>     @@ t/t1006-cat-file.sh: test_expect_success 'objects filter with unknown option' '\n>      @@ t/t1006-cat-file.sh: test_objects_filter \"blob:limit=1\"\n>       test_objects_filter \"blob:limit=500\"\n>       test_objects_filter \"blob:limit=1000\"\n>     - test_objects_filter \"blob:limit=1g\"\n>     + test_objects_filter \"blob:limit=1k\"\n>      +test_objects_filter \"object:type=blob\"\n>      +test_objects_filter \"object:type=commit\"\n>      +test_objects_filter \"object:type=tag\"\n>  6:  a0655de3ace =  7:  d525a5bc2ef pack-bitmap: allow passing payloads to `show_reachable_fn()`\n>  7:  e1e44303dac =  8:  e3cc1ae3a87 pack-bitmap: add function to iterate over filtered bitmapped objects\n>  8:  23bc040bb15 =  9:  c0fc0e4ce0c pack-bitmap: introduce function to check whether a pack is bitmapped\n>  9:  4eba2a70619 = 10:  28ef93dceec builtin/cat-file: deduplicate logic to iterate over all objects\n> 10:  d40f1924ef5 = 11:  842a6002c50 builtin/cat-file: use bitmaps to efficiently filter by object type\n>\n\nThanks for the new version, the range-diff looks good. Good that you\nalso added a test for \"excluded\" message too.\n\n> ---\n> base-commit: 003c5f45b8447877015b2a23ceab2297638fe1f1\n> change-id: 20250220-pks-cat-file-object-type-filter-9140c0ed5ee1\n"},{"id":"515812","messageId":"xmqqcydn1oj7.fsf@gitster.g","threadId":"62984","inReplyTo":"CAOLa=ZTY4AE4ONHoZX+VrHRC9461utdV6g7FFbWtnGuay55G9g@mail.gmail.com","subject":"Re: [PATCH v3 00/11] builtin/cat-file: allow filtering objects in batch mode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-04-08T00:32:12Z","receivedAt":"2025-04-08T00:32:15Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak <karthik.188@gmail.com> writes:\n\n> Thanks for the new version, the range-diff looks good. Good that you\n> also added a test for \"excluded\" message too.\n\nThanks, both of you.  Will replace and queue.\nLet me mark the topic for 'next'.\n"}]}