From: Patrick Steinhardt Date: Wed, 19 Nov 2025 07:47:00 GMT Subject: [PATCH 00/18] Refactor object read streams to work via object sources Message-ID: <20251119-b4-pks-odb-read-stream-v1-0-adacf03c2ccf@pks.im> Hi, the `git_istream` data structure can be used to read objects from the object database in a streaming fashion. This is used for example to read large files that one doesn't want to load into memory in full. In the current architecture, all the logic to handle these streams is fully self-contained in "streaming.c". It contains the logic to set up streams for loose, packed, in-memory and filtered objects. This doesn't really play all that well with pluggable object databases, as it should be the responsibility of the object database source itself to handle the logic. This patch series thus revamps our object read streams: instead of being entirely contained in "streaming.c", the format-specific streams are now created by the ODB sources. This allows each source itself to decide whether and, if so, how to make objects streamable. This overall requires quite a bit of refactoring, but I think that the end result is an easier-to-understand infrastructure that is an improvement even without pluggable object databases. This series is built on top of v2.52.0 with ps/object-source-loose at 3e5e360888 (object-file: refactor writing objects via a stream, 2025-11-03) merged into it. Thanks! Patrick --- Patrick Steinhardt (18): streaming: rename `git_istream` into `odb_read_stream` streaming: drop the `open()` callback function streaming: propagate final object type via the stream streaming: explicitly pass packfile info when streaming a packed object streaming: allocate stream inside the backend-specific logic streaming: create structure for in-core object streams streaming: create structure for loose object streams streaming: create structure for packed object streams streaming: create structure for filtered object streams streaming: move zlib stream into backends packfile: introduce function to read object info from a store streaming: rely on object sources to create object stream streaming: get rid of `the_repository` streaming: make the `odb_read_stream` definition public streaming: move logic to read loose objects streams into backend streaming: move logic to read packed objects streams into backend streaming: refactor interface to be object-database-centric streaming: move into object database subsystem Makefile | 2 +- archive-tar.c | 10 +- archive-zip.c | 16 +- builtin/cat-file.c | 4 +- builtin/fsck.c | 5 +- builtin/index-pack.c | 12 +- builtin/log.c | 6 +- builtin/pack-objects.c | 20 +- entry.c | 4 +- meson.build | 2 +- object-file.c | 179 ++++++++++++++-- object-file.h | 42 +--- odb.c | 29 +-- odb/streaming.c | 299 ++++++++++++++++++++++++++ odb/streaming.h | 70 ++++++ packfile.c | 199 ++++++++++++++++-- packfile.h | 17 +- parallel-checkout.c | 5 +- streaming.c | 561 ------------------------------------------------- streaming.h | 21 -- 20 files changed, 784 insertions(+), 719 deletions(-) --- base-commit: 899e578b5b7c020aec806bd694adf2563f62843c change-id: 20251107-b4-pks-odb-read-stream-7ea7f0e0a8f4