git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2 7/8] odb: introduce object filters to `odb_for_each_object()`

From
Patrick Steinhardt <ps@pks.im>
Date
Jul 10, 2026, 08:48 UTC
Message-ID
<20260710-pks-odb-for-each-object-filter-v2-7-3710a9cc165a@pks.im>
In-Reply-To
<20260710-pks-odb-for-each-object-filter-v2-0-3710a9cc165a@pks.im>

The function `for_each_bitmapped_object()` can be used to iterate through all objects covered by a bitmap. The benefit of this function is that it allows the caller to efficiently handle some object filters. For example, this can be used to filter out objects of a specific type with some simple bitmap operations. But callers are currently required to manually wire up the use of bitmaps though, and to do so they have to reach into internals of a given object database source.

Introduce a new `struct odb_for_each_object_options::filter` field so that the interface becomes generic. When set, then a backend may optionally use the filter to skip some objects that it would have otherwise yielded.

Note that the respective backends are free to ignore this field if they cannot meaningfully optimize for a given filter, and consequently callers need to verify whether they actually want the returned objects. While annoying, we cannot easily lift this restriction anyway as the object filter infrastructure supports some filters that cannot be answered by the object database alone.

An alternative might be to limit the filters to only those that _can_ be answered by backends. But ultimately, the filters that can be answered efficiently by the "packed" backend are completely disjunct from those that can be answered by the "loose" backend, and consequently the set of filters supported by all backends would be empty. Furthermore, it would require us to make assumptions about capabilities of future backends, which may be able to efficiently handle more filters than current ones. So in the end, this alternative would only limit us artificially.

Implement the logic for the "packed" source. Note that we use the new function `prepare_source_bitmap_git()` to open the bitmap: as the backend operates on a single object source, we must only use bitmaps that belong to that specific source. Otherwise we might yield objects that are not part of the source at all, and with multiple sources we would enumerate the same bitmap once per source.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb.h               | 12 +++++++++++
 odb/source-packed.c | 62 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 pack-bitmap.c       |  3 +--
 pack-bitmap.h       |  3 +++
 4 files changed, 78 insertions(+), 2 deletions(-)
diff --git a/odb.h b/odb.h
index a1e222f605..67d0b34942 100644
--- a/odb.h
+++ b/odb.h
@@ -8,6 +8,7 @@
 #include "thread-utils.h"
 
 struct cached_object_entry;
+struct list_objects_filter_options;
 struct odb_source_inmemory;
 struct packed_git;
 struct repository;
@@ -490,6 +491,17 @@ struct odb_for_each_object_options {
 	 */
 	const struct object_id *prefix;
 	size_t prefix_hex_len;
+
+	/*
+	 * Optional object filter that allows backends to skip yielding
+	 * objects that are excluded by the filter as an optimization. The
+	 * filter is a best-effort hint: backends may use it to skip
+	 * excluded objects (e.g. by consulting a reachability bitmap), but
+	 * are also free to ignore it entirely and yield every object. As a
+	 * consequence, callers must re-apply the filter on yielded objects
+	 * if they require strict filtering semantics.
+	 */
+	const struct list_objects_filter_options *filter;
 };
 
 /*
diff --git a/odb/source-packed.c b/odb/source-packed.c
index 9cfa02b7a2..4777395053 100644
--- a/odb/source-packed.c
+++ b/odb/source-packed.c
@@ -3,11 +3,13 @@
 #include "chdir-notify.h"
 #include "dir.h"
 #include "git-zlib.h"
+#include "list-objects-filter-options.h"
 #include "mergesort.h"
 #include "midx.h"
 #include "odb/source-packed.h"
 #include "odb/streaming.h"
 #include "packfile.h"
+#include "pack-bitmap.h"
 
 static int find_pack_entry(struct odb_source_packed *store,
 			   const struct object_id *oid,
@@ -315,6 +317,37 @@ static int odb_source_packed_for_each_prefixed_object(
 	return ret;
 }
 
+struct bitmapped_for_each_object_data {
+	struct odb_source_packed *packed;
+	const struct object_info *request;
+	const struct odb_for_each_object_options *opts;
+	odb_for_each_object_cb cb;
+	void *cb_data;
+};
+
+static int bitmapped_for_each_object(const struct object_id *oid,
+				     enum object_type type UNUSED,
+				     int flags UNUSED,
+				     uint32_t hash UNUSED,
+				     struct packed_git *pack,
+				     off_t offset,
+				     void *cb_data)
+{
+	struct bitmapped_for_each_object_data *data = cb_data;
+
+	if (should_exclude_pack(pack, data->opts->flags))
+		return 0;
+
+	if (data->request) {
+		struct object_info oi = *data->request;
+		if (packed_object_info(data->packed, pack, offset, &oi) < 0)
+			return -1;
+		return data->cb(oid, &oi, data->cb_data);
+	}
+
+	return data->cb(oid, NULL, data->cb_data);
+}
+
 static int odb_source_packed_for_each_object(struct odb_source *source,
 					     const struct object_info *request,
 					     odb_for_each_object_cb cb,
@@ -328,12 +361,33 @@ static int odb_source_packed_for_each_object(struct odb_source *source,
 		.cb = cb,
 		.cb_data = cb_data,
 	};
+	struct bitmap_index *bitmap = NULL;
 	struct packfile_list_entry *e;
 	int pack_errors = 0, ret;
 
 	if (opts->prefix)
 		return odb_source_packed_for_each_prefixed_object(packed, opts, &data);
 
+	if (opts->filter &&
+	    opts->filter->choice != LOFC_DISABLED &&
+	    can_filter_bitmap(opts->filter))
+		bitmap = prepare_bitmap_git_for_source(packed);
+	if (bitmap) {
+		struct bitmapped_for_each_object_data bitmap_data = {
+			.packed = packed,
+			.request = request,
+			.opts = opts,
+			.cb = cb,
+			.cb_data = cb_data,
+		};
+
+		ret = for_each_bitmapped_object(bitmap, opts->filter,
+						bitmapped_for_each_object,
+						&bitmap_data);
+		if (ret)
+			goto out;
+	}
+
 	packed->skip_mru_updates = true;
 
 	for (e = packfile_store_get_packs(packed); e; e = e->next) {
@@ -342,6 +396,13 @@ static int odb_source_packed_for_each_object(struct odb_source *source,
 		if (should_exclude_pack(p, opts->flags))
 			continue;
 
+		/*
+		 * Objects covered by the bitmap have already been yielded
+		 * above; skip them here to avoid duplicates.
+		 */
+		if (bitmap && bitmap_index_contains_pack(bitmap, p))
+			continue;
+
 		if (open_pack_index(p)) {
 			pack_errors = 1;
 			continue;
@@ -357,6 +418,7 @@ static int odb_source_packed_for_each_object(struct odb_source *source,
 
 out:
 	packed->skip_mru_updates = false;
+	free_bitmap_index(bitmap);
 
 	if (!ret && pack_errors)
 		ret = -1;
diff --git a/pack-bitmap.c b/pack-bitmap.c
index 09ba15d26b..f55a0859ea 100644
--- a/pack-bitmap.c
+++ b/pack-bitmap.c
@@ -2039,12 +2039,11 @@ static int filter_bitmap(struct bitmap_index *bitmap_git,
 	return -1;
 }
 
-static int can_filter_bitmap(const struct list_objects_filter_options *filter)
+bool can_filter_bitmap(const struct list_objects_filter_options *filter)
 {
 	return !filter_bitmap(NULL, NULL, NULL, filter);
 }
 
-
 static void filter_packed_objects_from_bitmap(struct bitmap_index *bitmap_git,
 					      struct bitmap *result)
 {
diff --git a/pack-bitmap.h b/pack-bitmap.h
index 9f20fb6e56..1385027c1f 100644
--- a/pack-bitmap.h
+++ b/pack-bitmap.h
@@ -92,6 +92,9 @@ int test_bitmap_pseudo_merge_objects(struct repository *r, uint32_t n);
 
 struct list_objects_filter_options;
 
+/* Check whether the filter can be computed via the bitmap. */
+bool can_filter_bitmap(const struct list_objects_filter_options *filter);
+
 /*
  * Filter bitmapped objects and iterate through all resulting objects,
  * executing `show_reach` for each of them. Returns `-1` in case the filter is
-- 
2.55.0.229.g6434b31f56.dirty
Previous: Patrick SteinhardtNext: Taylor Blau
Message 40 of 75 in “odb: introduce object filters to `odb_for_each_object()`”
  1. 0/7 odb: introduce object filters to `odb_for_each_object()`Patrick Steinhardt, Jul 9, 2026
  2. 1/7 odb/source-packed: improve lookup when enumerating objectsPatrick Steinhardt, Jul 9, 2026
  3. Justin ToblerJul 9, 2026
  4. Patrick SteinhardtJul 10, 2026
  5. 2/7 pack-bitmap: mark object filter as `const`Patrick Steinhardt, Jul 9, 2026
  6. 3/7 pack-bitmap: allow aborting iteration of bitmapped objectsPatrick Steinhardt, Jul 9, 2026
  7. Justin ToblerJul 9, 2026
  8. Patrick SteinhardtJul 10, 2026
  9. Jeff KingJul 11, 2026
  10. 4/7 pack-bitmap: iterate object sources when opening bitmapsPatrick Steinhardt, Jul 9, 2026
  11. Justin ToblerJul 9, 2026
  12. Patrick SteinhardtJul 10, 2026
  13. 5/7 pack-bitmap: introduce function to open bitmap for a single sourcePatrick Steinhardt, Jul 9, 2026
  14. 6/7 odb: introduce object filters to `odb_for_each_object()`Patrick Steinhardt, Jul 9, 2026
  15. Justin ToblerJul 9, 2026
  16. Patrick SteinhardtJul 10, 2026
  17. Toon ClaesJul 15, 2026
  18. 7/7 builtin/cat-file: filter objects via object databasePatrick Steinhardt, Jul 9, 2026
  19. Junio C HamanoJul 9, 2026
  20. Patrick SteinhardtJul 10, 2026
  21. 0/8 odb: introduce object filters to `odb_for_each_object()`Patrick Steinhardt, Jul 10, 2026
  22. 1/8 odb/source-packed: improve lookup when enumerating objectsPatrick Steinhardt, Jul 10, 2026
  23. Taylor BlauJul 10, 2026
  24. Patrick SteinhardtJul 13, 2026
  25. Taylor BlauJul 14, 2026
  26. Patrick SteinhardtJul 14, 2026
  27. 2/8 pack-bitmap: mark object filter as `const`Patrick Steinhardt, Jul 10, 2026
  28. Taylor BlauJul 10, 2026
  29. 3/8 pack-bitmap: allow aborting iteration of bitmapped objectsPatrick Steinhardt, Jul 10, 2026
  30. Taylor BlauJul 10, 2026
  31. Jeff KingJul 11, 2026
  32. Patrick SteinhardtJul 13, 2026
  33. Taylor BlauJul 14, 2026
  34. Jeff KingJul 14, 2026
  35. 4/8 pack-bitmap: iterate object sources when opening bitmapsPatrick Steinhardt, Jul 10, 2026
  36. Taylor BlauJul 10, 2026
  37. 5/8 pack-bitmap: drop `_1` suffix from functions that open bitmapsPatrick Steinhardt, Jul 10, 2026
  38. Taylor BlauJul 10, 2026
  39. 6/8 pack-bitmap: introduce function to open bitmap for a single sourcePatrick Steinhardt, Jul 10, 2026
  40. 7/8 odb: introduce object filters to `odb_for_each_object()`Patrick Steinhardt, Jul 10, 2026
  41. Taylor BlauJul 10, 2026
  42. 8/8 builtin/cat-file: filter objects via object databasePatrick Steinhardt, Jul 10, 2026
  43. 9/8 pack-objects: drop unused return value from add_object_entry()Jeff King, Jul 11, 2026
  44. Junio C HamanoJul 11, 2026
  45. 0/9 odb: introduce object filters to `odb_for_each_object()`Patrick Steinhardt, Jul 13, 2026
  46. 1/9 odb/source-packed: improve lookup when enumerating objectsPatrick Steinhardt, Jul 13, 2026
  47. 2/9 pack-bitmap: mark object filter as `const`Patrick Steinhardt, Jul 13, 2026
  48. 3/9 pack-objects: drop unused return value from add_object_entry()Patrick Steinhardt, Jul 13, 2026
  49. 4/9 pack-bitmap: allow aborting iteration of bitmapped objectsPatrick Steinhardt, Jul 13, 2026
  50. 5/9 pack-bitmap: iterate object sources when opening bitmapsPatrick Steinhardt, Jul 13, 2026
  51. 6/9 pack-bitmap: drop `_1` suffix from functions that open bitmapsPatrick Steinhardt, Jul 13, 2026
  52. 7/9 pack-bitmap: introduce function to open bitmap for a single sourcePatrick Steinhardt, Jul 13, 2026
  53. Toon ClaesJul 15, 2026
  54. Patrick SteinhardtJul 15, 2026
  55. 8/9 odb: introduce object filters to `odb_for_each_object()`Patrick Steinhardt, Jul 13, 2026
  56. Toon ClaesJul 15, 2026
  57. 9/9 builtin/cat-file: filter objects via object databasePatrick Steinhardt, Jul 13, 2026
  58. Toon ClaesJul 15, 2026
  59. Taylor BlauJul 14, 2026
  60. Patrick SteinhardtJul 14, 2026
  61. Toon ClaesJul 15, 2026
  62. Patrick SteinhardtJul 15, 2026
  63. Jeff KingJul 14, 2026
  64. 0/9 odb: introduce object filters to `odb_for_each_object()`Patrick Steinhardt, Jul 15, 2026
  65. 1/9 odb/source-packed: improve lookup when enumerating objectsPatrick Steinhardt, Jul 15, 2026
  66. 2/9 pack-bitmap: mark object filter as `const`Patrick Steinhardt, Jul 15, 2026
  67. 3/9 pack-objects: drop unused return value from add_object_entry()Patrick Steinhardt, Jul 15, 2026
  68. 4/9 pack-bitmap: allow aborting iteration of bitmapped objectsPatrick Steinhardt, Jul 15, 2026
  69. 5/9 pack-bitmap: iterate object sources when opening bitmapsPatrick Steinhardt, Jul 15, 2026
  70. 6/9 pack-bitmap: drop `_1` suffix from functions that open bitmapsPatrick Steinhardt, Jul 15, 2026
  71. 7/9 pack-bitmap: introduce function to open bitmap for a single sourcePatrick Steinhardt, Jul 15, 2026
  72. 8/9 odb: introduce object filters to `odb_for_each_object()`Patrick Steinhardt, Jul 15, 2026
  73. 9/9 builtin/cat-file: filter objects via object databasePatrick Steinhardt, Jul 15, 2026
  74. Toon ClaesJul 15, 2026
  75. Junio C HamanoJul 15, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.