[PATCH v2 00/19] Refactor object read streams to work via object sources
- From
Patrick Steinhardt <ps@pks.im>
- Date
- Nov 21, 2025, 07:40 UTC
- Message-ID
- <20251121-b4-pks-odb-read-stream-v2-0-ca8534963150@pks.im>
- In-Reply-To
- <20251119-b4-pks-odb-read-stream-v1-0-adacf03c2ccf@pks.im>
Hi,
the `git_istream` data structure can be used to read objects from the object database in a streaming fashion. This is used for example to read large files that one doesn't want to load into memory in full.
In the current architecture, all the logic to handle these streams is fully self-contained in "streaming.c". It contains the logic to set up streams for loose, packed, in-memory and filtered objects. This doesn't really play all that well with pluggable object databases, as it should be the responsibility of the object database source itself to handle the logic.
This patch series thus revamps our object read streams: instead of being entirely contained in "streaming.c", the format-specific streams are now created by the ODB sources. This allows each source itself to decide whether and, if so, how to make objects streamable.
This overall requires quite a bit of refactoring, but I think that the end result is an easier-to-understand infrastructure that is an improvement even without pluggable object databases.
This series is built on top of v2.52.0 with ps/object-source-loose at 3e5e360888 (object-file: refactor writing objects via a stream, 2025-11-03) merged into it.
Changes in v2:
- Some commit message improvements.
- Drop the `type` and `size` out pointers in
`odb_read_object_stream()` in an additional commit.
- Improve a "hidden" variable declaration by moving it onto its own
line.
- Link to v1: https://lore.kernel.org/r/20251119-b4-pks-odb-read-stream-v1-0-adacf03c2ccf@pks.imThanks!
Patrick
---
Patrick Steinhardt (19):
streaming: rename `git_istream` into `odb_read_stream`
streaming: drop the `open()` callback function
streaming: propagate final object type via the stream
streaming: explicitly pass packfile info when streaming a packed object
streaming: allocate stream inside the backend-specific logic
streaming: create structure for in-core object streams
streaming: create structure for loose object streams
streaming: create structure for packed object streams
streaming: create structure for filtered object streams
streaming: move zlib stream into backends
packfile: introduce function to read object info from a store
streaming: rely on object sources to create object stream
streaming: get rid of `the_repository`
streaming: make the `odb_read_stream` definition public
streaming: move logic to read loose objects streams into backend
streaming: move logic to read packed objects streams into backend
streaming: refactor interface to be object-database-centric
streaming: move into object database subsystem
streaming: drop redundant type and size pointersMakefile | 2 +- archive-tar.c | 12 +- archive-zip.c | 17 +- builtin/cat-file.c | 4 +- builtin/fsck.c | 5 +- builtin/index-pack.c | 15 +- builtin/log.c | 6 +- builtin/pack-objects.c | 24 ++- entry.c | 4 +- meson.build | 2 +- object-file.c | 183 ++++++++++++++-- object-file.h | 42 +--- odb.c | 29 +-- odb/streaming.c | 294 ++++++++++++++++++++++++++ odb/streaming.h | 67 ++++++ packfile.c | 199 ++++++++++++++++-- packfile.h | 17 +- parallel-checkout.c | 5 +- streaming.c | 561 ------------------------------------------------- streaming.h | 21 -- 20 files changed, 780 insertions(+), 729 deletions(-)
Range-diff versus v1:
1: 89ec27ae18 ! 1: a6534585dd streaming: rename `git_istream` into `odb_read_stream`
@@ streaming.h
-int close_istream(struct git_istream *);
-ssize_t read_istream(struct git_istream *, void *, size_t);
+struct odb_read_stream *open_istream(struct repository *, const struct object_id *,
-+ enum object_type *, unsigned long *,
-+ struct stream_filter *);
++ enum object_type *, unsigned long *,
++ struct stream_filter *);
+int close_istream(struct odb_read_stream *);
+ssize_t read_istream(struct odb_read_stream *, void *, size_t);
2: b4d37fd4f2 ! 2: 23a8704740 streaming: drop the `open()` callback function
@@ Commit message
structure in the source itself.
We could of course work around that and have the loose object source
- create the structure and populate it's `open()` callback, only. But
+ create the structure and populate its `open()` callback, only. But
this doesn't really buy us anything due to the second bullet point
above.
3: b8bae59f58 = 3: badcc5d72b streaming: propagate final object type via the stream
4: 583ed2c4f3 = 4: 09f9d2e3f2 streaming: explicitly pass packfile info when streaming a packed object
5: af1a5a312a ! 5: 40728b509c streaming: allocate stream inside the backend-specific logic
@@ Commit message
This works, but it's awkward in the context of pluggable object
databases. Each backend will need its own member in that union, and as
the structure itself is completely opaque (it's only defined in
- "streamgin.c") it also has the consequence that we must have the logic
+ "streaming.c") it also has the consequence that we must have the logic
that is specific to backends in "streaming.c".
Ideally though, the infrastructure would be reversed: we have a generic
6: 5c5c291bba ! 6: 7d74c31e3d streaming: create structure for in-core object streams
@@ streaming.c: static int open_istream_incore(struct odb_read_stream **out,
- struct odb_read_stream stream = {
- .close = close_istream_incore,
- .read = read_istream_incore,
-- };
+ struct odb_incore_read_stream stream = {
+ .base.close = close_istream_incore,
+ .base.read = read_istream_incore,
-+ }, *st;
+ };
++ struct odb_incore_read_stream *st;
int ret;
- oi.typep = &stream.type;
7: 58d214e576 = 7: dd3440bff2 streaming: create structure for loose object streams
8: 7b3d095e06 = 8: 6de8cc7c9f streaming: create structure for packed object streams
9: 3bca3dfab5 = 9: e00aa2b198 streaming: create structure for filtered object streams
10: 329549b6c7 = 10: f37441494d streaming: move zlib stream into backends
11: 9d47d12cbf = 11: 8c62cfac57 packfile: introduce function to read object info from a store
12: 3a5ad53484 ! 12: 82f186e8b4 streaming: rely on object sources to create object stream
@@ Commit message
the different source backends anymore, but eventually it'll only have to
call the source's callback function.
- Note that at the current poin in time we aren't full there yet:
+ Note that at the current point in time we aren't fully there yet:
- The packfile store still sits on the object database level and is
thus agnostic of the sources.
13: 2fa2f53ac0 = 13: a5c1b3c717 streaming: get rid of `the_repository`
14: 49e6fb06e8 ! 14: 5fdd600a0c streaming: make the `odb_read_stream` definition public
@@ streaming.h
+};
+
struct odb_read_stream *open_istream(struct repository *, const struct object_id *,
- enum object_type *, unsigned long *,
- struct stream_filter *);
+ enum object_type *, unsigned long *,
+ struct stream_filter *);
15: 3a944f3a31 = 15: 460cab31c9 streaming: move logic to read loose objects streams into backend
16: 60b08e3dc5 = 16: 293578ab35 streaming: move logic to read packed objects streams into backend
17: 68ef7721b0 ! 17: e6a242f1b8 streaming: refactor interface to be object-database-centric
@@ streaming.h: struct odb_read_stream {
};
-struct odb_read_stream *open_istream(struct repository *, const struct object_id *,
-- enum object_type *, unsigned long *,
-- struct stream_filter *);
+- enum object_type *, unsigned long *,
+- struct stream_filter *);
-int close_istream(struct odb_read_stream *);
-ssize_t read_istream(struct odb_read_stream *, void *, size_t);
+/*
18: 8afda7d038 = 18: 95e7c2aa9b streaming: move into object database subsystem
-: ---------- > 19: c8b2112d00 streaming: drop redundant type and size pointers--- base-commit: 899e578b5b7c020aec806bd694adf2563f62843c change-id: 20251107-b4-pks-odb-read-stream-7ea7f0e0a8f4