git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2 0/8] odb: introduce object filters to `odb_for_each_object()`

From
Patrick Steinhardt <ps@pks.im>
Date
Jul 10, 2026, 08:48 UTC
Message-ID
<20260710-pks-odb-for-each-object-filter-v2-0-3710a9cc165a@pks.im>
In-Reply-To
<20260709-pks-odb-for-each-object-filter-v1-0-82fe014b12b3@pks.im>
Hi,

this patch series introduces object filters to `odb_for_each_object()`. The intent of this is to make `git cat-file --batch-all-objects` work with pluggable object databases. Right now it doesn't because it reaches into internals of the "packed" backend to efficiently handle bitmapped objects.

The series is built on top of f85a7e6620 (Start Git 2.56 cycle, 2026-07-06) with ps/odb-drop-whence at 8a7ad23e11 (odb: document object info fields, 2026-07-02) merged into it.

Changes in v2:
  - Add another patch to drop the `_1()` prefixes that aren't required
    anymore.
  - Change the approach in `open_bitmap_for_source()` to also use a
    `found` boolean instead of a confusing integer.
  - Add some more explanations to commit messages.
  - Link to v1: https://patch.msgid.link/20260709-pks-odb-for-each-object-filter-v1-0-82fe014b12b3@pks.im
Thanks!
Patrick
---
Patrick Steinhardt (8):
      odb/source-packed: improve lookup when enumerating objects
      pack-bitmap: mark object filter as `const`
      pack-bitmap: allow aborting iteration of bitmapped objects
      pack-bitmap: iterate object sources when opening bitmaps
      pack-bitmap: drop `_1` suffix from functions that open bitmaps
      pack-bitmap: introduce function to open bitmap for a single source
      odb: introduce object filters to `odb_for_each_object()`
      builtin/cat-file: filter objects via object database
 builtin/cat-file.c     |  76 +++--------------------------
 builtin/pack-objects.c |   2 +-
 builtin/rev-list.c     |   2 +-
 odb.h                  |  12 +++++
 odb/source-packed.c    |  77 ++++++++++++++++++++++++++---
 pack-bitmap.c          | 129 +++++++++++++++++++++++++++----------------------
 pack-bitmap.h          |  10 +++-
 7 files changed, 171 insertions(+), 137 deletions(-)
Range-diff versus v1:
1:  7a1a92acbe ! 1:  b675967b78 odb/source-packed: improve lookup when enumerating objects
    @@ Metadata
      ## Commit message ##
         odb/source-packed: improve lookup when enumerating objects
     
    -    When iterating through packed objects via `odb_for_each_object()` we
    -    do so via two different mechanisms:
    +    When iterating through objects of a packed source that have a specific
    +    prefix we do so via two different methods:
     
           - When a multi-pack index is available we use that one to efficiently
             loop through all objects.
    @@ Commit message
     
           - It's subtly wrong, as it may now happen that a specific object will
             be looked up via a different pack in case it exists multiple times.
    +        This is unlikely to have any real-world consequences, but it's still
    +        the wrong thing to do.
     
         Fix the issue by using `packed_object_info()` directly. While at it,
         rename the `store` variable to `source`.
2:  a6c8bd7a61 = 2:  d3f9b2f781 pack-bitmap: mark object filter as `const`
3:  c38b06636b = 3:  825920205a pack-bitmap: allow aborting iteration of bitmapped objects
4:  450cdd13b7 ! 4:  a33ca8fa3b pack-bitmap: iterate object sources when opening bitmaps
    @@ pack-bitmap.c: static int load_bitmap(struct repository *r, struct bitmap_index
     +				  struct bitmap_index *bitmap_git)
      {
     -	struct packed_git *p;
    +-	int ret = -1;
     +	struct multi_pack_index *midx = get_multi_pack_index(source);
     +	struct packfile_list_entry *e;
    - 	int ret = -1;
    ++	bool found = false;
      
     -	repo_for_each_pack(r, p) {
     -		if (open_pack_bitmap_1(bitmap_git, p) == 0) {
    @@ pack-bitmap.c: static int load_bitmap(struct repository *r, struct bitmap_index
     -				break;
     -		}
     +	if (midx && !open_midx_bitmap_1(bitmap_git, midx))
    -+		ret = 0;
    ++		found = true;
     +
     +	for (e = packfile_store_get_packs(source); e; e = e->next) {
     +		/*
     +		 * When tracing is enabled we want to keep looking to report
     +		 * duplicates even if we have already found a bitmap.
     +		 */
    -+		if (!ret && !trace2_is_enabled())
    ++		if (found && !trace2_is_enabled())
     +			break;
     +
    -+		if (open_pack_bitmap_1(bitmap_git, e->pack))
    -+			continue;
    -+		ret = 0;
    ++		if (!open_pack_bitmap_1(bitmap_git, e->pack))
    ++			found = true;
      	}
      
    - 	return ret;
    +-	return ret;
    ++	return found ? 0 : -1;
      }
      
     -static int open_midx_bitmap(struct repository *r,
    @@ pack-bitmap.c: static int load_bitmap(struct repository *r, struct bitmap_index
      {
      	struct odb_source *source;
     -	int ret = -1;
    -+	int found = 0;
    ++	bool found = false;
      
      	assert(!bitmap_git->map);
      
    @@ pack-bitmap.c: static int load_bitmap(struct repository *r, struct bitmap_index
     -	int found;
      
     -	assert(!bitmap_git->map);
    -+		found |= !open_bitmap_for_source(files->packed, bitmap_git);
    ++		if (!open_bitmap_for_source(files->packed, bitmap_git))
    ++			found = true;
      
     -	found = !open_midx_bitmap(r, bitmap_git);
     -
-:  ---------- > 5:  b890ed7163 pack-bitmap: drop `_1` suffix from functions that open bitmaps
5:  26b1957f8b = 6:  f7e466217b pack-bitmap: introduce function to open bitmap for a single source
6:  722727c76d ! 7:  27ecc0802f odb: introduce object filters to `odb_for_each_object()`
    @@ Commit message
         object filter infrastructure supports some filters that cannot be
         answered by the object database alone.
     
    +    An alternative might be to limit the filters to only those that _can_ be
    +    answered by backends. But ultimately, the filters that can be answered
    +    efficiently by the "packed" backend are completely disjunct from those
    +    that can be answered by the "loose" backend, and consequently the set of
    +    filters supported by all backends would be empty. Furthermore, it would
    +    require us to make assumptions about capabilities of future backends,
    +    which may be able to efficiently handle more filters than current ones.
    +    So in the end, this alternative would only limit us artificially.
    +
         Implement the logic for the "packed" source. Note that we use the new
         function `prepare_source_bitmap_git()` to open the bitmap: as the
         backend operates on a single object source, we must only use bitmaps
7:  90be28e904 ! 8:  e4c6aeab0a builtin/cat-file: filter objects via object database
    @@ Metadata
      ## Commit message ##
         builtin/cat-file: filter objects via object database
     
    +    When batching all objects, git-cat-file(1) reaches into the internals of
    +    the object database and manually manages bitmaps to apply object
    +    filters. This creates coupling between the command and the internals of
    +    the respective backend.
    +
         Refactor git-cat-file(1) to use the new object filter option when
         batching all objects. This significantly simplifies the logic and
         ensures that we don't have to reach into internals of the "files" source

--- base-commit: 3c8e2790f2ce15e8b5d4b4e6ced711b12649f32a change-id: 20260708-pks-odb-for-each-object-filter-13286fa3523d

Previous: Patrick SteinhardtNext: Patrick Steinhardt
Message 21 of 75 in “odb: introduce object filters to `odb_for_each_object()`”
  1. 0/7 odb: introduce object filters to `odb_for_each_object()`Patrick Steinhardt, Jul 9, 2026
  2. 1/7 odb/source-packed: improve lookup when enumerating objectsPatrick Steinhardt, Jul 9, 2026
  3. Justin ToblerJul 9, 2026
  4. Patrick SteinhardtJul 10, 2026
  5. 2/7 pack-bitmap: mark object filter as `const`Patrick Steinhardt, Jul 9, 2026
  6. 3/7 pack-bitmap: allow aborting iteration of bitmapped objectsPatrick Steinhardt, Jul 9, 2026
  7. Justin ToblerJul 9, 2026
  8. Patrick SteinhardtJul 10, 2026
  9. Jeff KingJul 11, 2026
  10. 4/7 pack-bitmap: iterate object sources when opening bitmapsPatrick Steinhardt, Jul 9, 2026
  11. Justin ToblerJul 9, 2026
  12. Patrick SteinhardtJul 10, 2026
  13. 5/7 pack-bitmap: introduce function to open bitmap for a single sourcePatrick Steinhardt, Jul 9, 2026
  14. 6/7 odb: introduce object filters to `odb_for_each_object()`Patrick Steinhardt, Jul 9, 2026
  15. Justin ToblerJul 9, 2026
  16. Patrick SteinhardtJul 10, 2026
  17. Toon ClaesJul 15, 2026
  18. 7/7 builtin/cat-file: filter objects via object databasePatrick Steinhardt, Jul 9, 2026
  19. Junio C HamanoJul 9, 2026
  20. Patrick SteinhardtJul 10, 2026
  21. 0/8 odb: introduce object filters to `odb_for_each_object()`Patrick Steinhardt, Jul 10, 2026
  22. 1/8 odb/source-packed: improve lookup when enumerating objectsPatrick Steinhardt, Jul 10, 2026
  23. Taylor BlauJul 10, 2026
  24. Patrick SteinhardtJul 13, 2026
  25. Taylor BlauJul 14, 2026
  26. Patrick SteinhardtJul 14, 2026
  27. 2/8 pack-bitmap: mark object filter as `const`Patrick Steinhardt, Jul 10, 2026
  28. Taylor BlauJul 10, 2026
  29. 3/8 pack-bitmap: allow aborting iteration of bitmapped objectsPatrick Steinhardt, Jul 10, 2026
  30. Taylor BlauJul 10, 2026
  31. Jeff KingJul 11, 2026
  32. Patrick SteinhardtJul 13, 2026
  33. Taylor BlauJul 14, 2026
  34. Jeff KingJul 14, 2026
  35. 4/8 pack-bitmap: iterate object sources when opening bitmapsPatrick Steinhardt, Jul 10, 2026
  36. Taylor BlauJul 10, 2026
  37. 5/8 pack-bitmap: drop `_1` suffix from functions that open bitmapsPatrick Steinhardt, Jul 10, 2026
  38. Taylor BlauJul 10, 2026
  39. 6/8 pack-bitmap: introduce function to open bitmap for a single sourcePatrick Steinhardt, Jul 10, 2026
  40. 7/8 odb: introduce object filters to `odb_for_each_object()`Patrick Steinhardt, Jul 10, 2026
  41. Taylor BlauJul 10, 2026
  42. 8/8 builtin/cat-file: filter objects via object databasePatrick Steinhardt, Jul 10, 2026
  43. 9/8 pack-objects: drop unused return value from add_object_entry()Jeff King, Jul 11, 2026
  44. Junio C HamanoJul 11, 2026
  45. 0/9 odb: introduce object filters to `odb_for_each_object()`Patrick Steinhardt, Jul 13, 2026
  46. 1/9 odb/source-packed: improve lookup when enumerating objectsPatrick Steinhardt, Jul 13, 2026
  47. 2/9 pack-bitmap: mark object filter as `const`Patrick Steinhardt, Jul 13, 2026
  48. 3/9 pack-objects: drop unused return value from add_object_entry()Patrick Steinhardt, Jul 13, 2026
  49. 4/9 pack-bitmap: allow aborting iteration of bitmapped objectsPatrick Steinhardt, Jul 13, 2026
  50. 5/9 pack-bitmap: iterate object sources when opening bitmapsPatrick Steinhardt, Jul 13, 2026
  51. 6/9 pack-bitmap: drop `_1` suffix from functions that open bitmapsPatrick Steinhardt, Jul 13, 2026
  52. 7/9 pack-bitmap: introduce function to open bitmap for a single sourcePatrick Steinhardt, Jul 13, 2026
  53. Toon ClaesJul 15, 2026
  54. Patrick SteinhardtJul 15, 2026
  55. 8/9 odb: introduce object filters to `odb_for_each_object()`Patrick Steinhardt, Jul 13, 2026
  56. Toon ClaesJul 15, 2026
  57. 9/9 builtin/cat-file: filter objects via object databasePatrick Steinhardt, Jul 13, 2026
  58. Toon ClaesJul 15, 2026
  59. Taylor BlauJul 14, 2026
  60. Patrick SteinhardtJul 14, 2026
  61. Toon ClaesJul 15, 2026
  62. Patrick SteinhardtJul 15, 2026
  63. Jeff KingJul 14, 2026
  64. 0/9 odb: introduce object filters to `odb_for_each_object()`Patrick Steinhardt, Jul 15, 2026
  65. 1/9 odb/source-packed: improve lookup when enumerating objectsPatrick Steinhardt, Jul 15, 2026
  66. 2/9 pack-bitmap: mark object filter as `const`Patrick Steinhardt, Jul 15, 2026
  67. 3/9 pack-objects: drop unused return value from add_object_entry()Patrick Steinhardt, Jul 15, 2026
  68. 4/9 pack-bitmap: allow aborting iteration of bitmapped objectsPatrick Steinhardt, Jul 15, 2026
  69. 5/9 pack-bitmap: iterate object sources when opening bitmapsPatrick Steinhardt, Jul 15, 2026
  70. 6/9 pack-bitmap: drop `_1` suffix from functions that open bitmapsPatrick Steinhardt, Jul 15, 2026
  71. 7/9 pack-bitmap: introduce function to open bitmap for a single sourcePatrick Steinhardt, Jul 15, 2026
  72. 8/9 odb: introduce object filters to `odb_for_each_object()`Patrick Steinhardt, Jul 15, 2026
  73. 9/9 builtin/cat-file: filter objects via object databasePatrick Steinhardt, Jul 15, 2026
  74. Toon ClaesJul 15, 2026
  75. Junio C HamanoJul 15, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.