git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2 0/8] odb: unify read and write streams

From
Patrick Steinhardt <ps@pks.im>
Date
Aug 5, 2026, 07:44 UTC
Message-ID
<20260805-pks-odb-stream-unification-v2-0-b8c369564641@pks.im>
In-Reply-To
<20260804-pks-odb-stream-unification-v1-0-86d70e82345e@pks.im>
Hi,

we have two different kind of object database streams in our code base: `odb_write_stream` and `odb_read_stream`. While those are used for different use cases, the provided functionality is ultimately the exact same.

This patch series thus refactors these streams so that we have a single `odb_stream`, only. This allows us to reuse the streams for different kinds of purposes and makes them more generally useful overall. For example, it's trivially possible now to create an object stream for any given object and then write that stream into a different source.

The series is built on top of 5b2471720c (The 10th batch, 2026-08-03).
Changes in v2:
  - Use the correct object type when hashing in-memory objects.
  - Remove a stale comment.
  - Adapt a commit message to mention that renames will follow in
    subsequent commits.
  - Add another commit to rename `struct input_zstream_data`.
  - Link to v1: https://patch.msgid.link/20260804-pks-odb-stream-unification-v1-0-86d70e82345e@pks.im
Thanks!
Patrick
---
Patrick Steinhardt (8):
      odb/streaming: track write stream size in the structure
      odb/streaming: drop `is_finished` field
      odb/streaming: support streaming arbitrary object types
      odb/streaming: rename `struct odb_read_stream`
      odb/streaming: consolidate read and write streams
      odb/streaming: rename `struct read_object_fd_data`
      odb/streaming: rename `struct input_zstream_data`
      odb/streaming: unify function names to create new streams
 archive-tar.c                 |   8 ++--
 archive-zip.c                 |  12 ++---
 builtin/index-pack.c          |   8 ++--
 builtin/pack-objects.c        |  18 ++++----
 builtin/unpack-objects.c      |  44 ++++++++++--------
 object-file.c                 |  76 +++++++++++++++---------------
 object-file.h                 |   2 +-
 object.c                      |   6 +--
 odb.c                         |   4 +-
 odb.h                         |   4 +-
 odb/source-files.c            |   7 ++-
 odb/source-inmemory.c         |  35 ++++++++------
 odb/source-loose.c            |  33 ++++++++------
 odb/source-packed.c           |   5 +-
 odb/source.h                  |  13 +++---
 odb/streaming.c               | 104 ++++++++++++++++++++----------------------
 odb/streaming.h               |  69 ++++++++++------------------
 odb/transaction.c             |   6 +--
 odb/transaction.h             |   8 ++--
 pack-check.c                  |   4 +-
 packfile.c                    |   8 ++--
 packfile.h                    |   4 +-
 t/unit-tests/u-odb-inmemory.c |  37 ++++++++-------
 23 files changed, 251 insertions(+), 264 deletions(-)
Range-diff versus v1:
1:  0085df877f = 1:  1966710c12 odb/streaming: track write stream size in the structure
2:  5fbbfd9010 = 2:  87c7981a6c odb/streaming: drop `is_finished` field
3:  52e5b87761 ! 3:  9aede44fba odb/streaming: support streaming arbitrary object types
    @@ object-file.c: int index_fd(struct index_state *istate, struct object_id *oid,
     
      ## odb/source-inmemory.c ##
     @@ odb/source-inmemory.c: static int odb_source_inmemory_write_object_stream(struct odb_source *source,
    - 	hash_object_file(source->odb->repo->hash_algo, data, total_read, OBJ_BLOB, oid);
    + 		goto out;
    + 	}
    + 
    +-	hash_object_file(source->odb->repo->hash_algo, data, total_read, OBJ_BLOB, oid);
    ++	hash_object_file(source->odb->repo->hash_algo, data, total_read,
    ++			 stream->type, oid);
      
      	ret = odb_source_inmemory_write_object(source, data, stream->size,
     -					       OBJ_BLOB, oid, NULL, NULL, 0);
    @@ odb/streaming.h: int odb_stream_blob_to_fd(struct object_database *odb,
      
      #endif /* STREAMING_H */
     
    + ## odb/transaction.h ##
    +@@ odb/transaction.h: struct odb_transaction {
    + 
    + 	/*
    + 	 * This callback is expected to write the given object stream into
    +-	 * the ODB transaction. Note that for now, only blobs support streaming.
    ++	 * the ODB transaction.
    + 	 *
    + 	 * The resulting object ID shall be written into the out pointer. The
    + 	 * callback is expected to return 0 on success, a negative error code
    +
      ## t/unit-tests/u-odb-inmemory.c ##
     @@ t/unit-tests/u-odb-inmemory.c: void test_odb_inmemory__write_object_stream(void)
      	struct odb_source_inmemory *source = odb_source_inmemory_new(odb);
4:  f178d441f0 = 4:  ca84a2b645 odb/streaming: rename `struct odb_read_stream`
5:  0d72d27078 ! 5:  838394bffc odb/streaming: consolidate read and write streams
    @@ Commit message
         new `struct odb_stream` base. Other than that though, the changes are
         rather straight forward.
     
    +    Some of the structures and functions are now somewhat misnamed. These
    +    will be fixed in subsequent commits.
    +
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
      ## builtin/unpack-objects.c ##
6:  ced59bdc85 = 6:  850b7e081d odb/streaming: rename `struct read_object_fd_data`
-:  ---------- > 7:  c3fe9f8b0c odb/streaming: rename `struct input_zstream_data`
7:  f76f4350ef = 8:  4df81651ba odb/streaming: unify function names to create new streams

--- base-commit: 5b2471720c93ee30e5764a19f3d3b3ae9ec9712a change-id: 20260724-pks-odb-stream-unification-334dc2a75888

Previous: Justin ToblerNext: Patrick Steinhardt
Message 19 of 33 in “odb: unify read and write streams”
  1. 0/7 odb: unify read and write streamsPatrick Steinhardt, Aug 4, 2026
  2. 1/7 odb/streaming: track write stream size in the structurePatrick Steinhardt, Aug 4, 2026
  3. Justin ToblerAug 4, 2026
  4. 2/7 odb/streaming: drop `is_finished` fieldPatrick Steinhardt, Aug 4, 2026
  5. Justin ToblerAug 4, 2026
  6. 3/7 odb/streaming: support streaming arbitrary object typesPatrick Steinhardt, Aug 4, 2026
  7. Justin ToblerAug 4, 2026
  8. Patrick SteinhardtAug 5, 2026
  9. Junio C HamanoAug 4, 2026
  10. Patrick SteinhardtAug 5, 2026
  11. 4/7 odb/streaming: rename `struct odb_read_stream`Patrick Steinhardt, Aug 4, 2026
  12. 5/7 odb/streaming: consolidate read and write streamsPatrick Steinhardt, Aug 4, 2026
  13. Justin ToblerAug 4, 2026
  14. Patrick SteinhardtAug 5, 2026
  15. 6/7 odb/streaming: rename `struct read_object_fd_data`Patrick Steinhardt, Aug 4, 2026
  16. Justin ToblerAug 4, 2026
  17. 7/7 odb/streaming: unify function names to create new streamsPatrick Steinhardt, Aug 4, 2026
  18. Justin ToblerAug 4, 2026
  19. 0/8 odb: unify read and write streamsPatrick Steinhardt, Aug 5, 2026
  20. 1/8 odb/streaming: track write stream size in the structurePatrick Steinhardt, Aug 5, 2026
  21. Karthik NayakAug 11, 2026
  22. 2/8 odb/streaming: drop `is_finished` fieldPatrick Steinhardt, Aug 5, 2026
  23. Karthik NayakAug 11, 2026
  24. 3/8 odb/streaming: support streaming arbitrary object typesPatrick Steinhardt, Aug 5, 2026
  25. 4/8 odb/streaming: rename `struct odb_read_stream`Patrick Steinhardt, Aug 5, 2026
  26. 5/8 odb/streaming: consolidate read and write streamsPatrick Steinhardt, Aug 5, 2026
  27. Karthik NayakAug 11, 2026
  28. 6/8 odb/streaming: rename `struct read_object_fd_data`Patrick Steinhardt, Aug 5, 2026
  29. 7/8 odb/streaming: rename `struct input_zstream_data`Patrick Steinhardt, Aug 5, 2026
  30. 8/8 odb/streaming: unify function names to create new streamsPatrick Steinhardt, Aug 5, 2026
  31. Karthik NayakAug 11, 2026
  32. Patrick SteinhardtAug 11, 2026
  33. Justin ToblerAug 11, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.