git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 5/8] odb/streaming: consolidate read and write streams

From
Karthik Nayak <karthik.188@gmail.com>
Date
Aug 11, 2026, 10:12 UTC
Message-ID
<CAOLa=ZTi8tL896_F2ONQck0z+H8NYhzcbTorb80NOdiqvnpjNg@mail.gmail.com>
In-Reply-To
<20260805-pks-odb-stream-unification-v2-5-b8c369564641@pks.im>
Patrick Steinhardt <ps@pks.im> writes:
Show 26 quoted lines
> The `struct odb_read_stream` and `struct odb_write_stream` both provide
> the same functionality: they allow a caller to read object data from an
> arbitrary source. Historically, the only difference was that the read
> stream was used to read data out of the object database, whereas the
> write stream was used to write data into the object database, but the
> interfaces were mostly the same.
>
> Over the preceding commits we have refactored the write stream to have
> almost exactly the same interface as the read stream. With these
> refactorings we can now easily merge those two streams into a single
> interface that's used for both use cases.
>
> While most of the changes are mechanical, there are two sites that need
> special mention:
>
>   - "builtin/unpack-objects.c" creates a write stream from compressed
>     object data.
>
>   - "odb/streaming.c" creates a write stream from a file descriptor.
>
> Adapting these sites to yield the new stream type requires a couple more
> changes. Most importantly, instead of embedding the pointer to the data
> in `struct odb_write_stream`, we now allocate a structure that wraps the
> new `struct odb_stream` base. Other than that though, the changes are
> rather straight forward.
>

So instead of `odb_write_stream.data` which was pointing to the data, we simply wrap the stream with the data's structure, this allows us to get the parent struct if we have the `odb_stream`. Alright!

Show 48 quoted lines
> Some of the structures and functions are now somewhat misnamed. These
> will be fixed in subsequent commits.
>
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  builtin/unpack-objects.c      | 31 ++++++++++++++++---------------
>  object-file.c                 | 25 ++++++++++++-------------
>  odb.c                         |  2 +-
>  odb.h                         |  4 ++--
>  odb/source-files.c            |  2 +-
>  odb/source-inmemory.c         |  4 ++--
>  odb/source-loose.c            |  6 +++---
>  odb/source-packed.c           |  2 +-
>  odb/source.h                  |  4 ++--
>  odb/streaming.c               | 35 ++++++++++++++++-------------------
>  odb/streaming.h               | 31 +++----------------------------
>  odb/transaction.c             |  2 +-
>  odb/transaction.h             |  4 ++--
>  t/unit-tests/u-odb-inmemory.c |  6 +++---
>  14 files changed, 65 insertions(+), 93 deletions(-)
>
> diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c
> index 7439ec53be..05a2d48011 100644
> --- a/builtin/unpack-objects.c
> +++ b/builtin/unpack-objects.c
> @@ -359,20 +359,21 @@ static void unpack_non_delta_entry(enum object_type type, unsigned long size,
>  }
>
>  struct input_zstream_data {
> +	struct odb_stream base;
>  	git_zstream *zstream;
>  	int status;
>  };
>
> -static ssize_t feed_input_zstream(struct odb_write_stream *in_stream,
> -				  unsigned char *buf, size_t buf_len)
> +static ssize_t feed_input_zstream(struct odb_stream *in_stream,
> +				  char *buf, size_t buf_len)
>  {
> -	struct input_zstream_data *data = in_stream->data;
> +	struct input_zstream_data *data = container_of(in_stream, struct input_zstream_data, base);
>  	git_zstream *zstream = data->zstream;
>
>  	if (data->status != Z_OK)
>  		return 0;
>
> -	zstream->next_out = buf;
> +	zstream->next_out = (unsigned char *) buf;

But we do loose information here since we now use 'char *' for both read/write. But we gain flexibility.

[snip]
The rest looks good.
Previous: Patrick SteinhardtNext: Patrick Steinhardt
Message 27 of 33 in “odb: unify read and write streams”
  1. 0/7 odb: unify read and write streamsPatrick Steinhardt, Aug 4, 2026
  2. 1/7 odb/streaming: track write stream size in the structurePatrick Steinhardt, Aug 4, 2026
  3. Justin ToblerAug 4, 2026
  4. 2/7 odb/streaming: drop `is_finished` fieldPatrick Steinhardt, Aug 4, 2026
  5. Justin ToblerAug 4, 2026
  6. 3/7 odb/streaming: support streaming arbitrary object typesPatrick Steinhardt, Aug 4, 2026
  7. Justin ToblerAug 4, 2026
  8. Patrick SteinhardtAug 5, 2026
  9. Junio C HamanoAug 4, 2026
  10. Patrick SteinhardtAug 5, 2026
  11. 4/7 odb/streaming: rename `struct odb_read_stream`Patrick Steinhardt, Aug 4, 2026
  12. 5/7 odb/streaming: consolidate read and write streamsPatrick Steinhardt, Aug 4, 2026
  13. Justin ToblerAug 4, 2026
  14. Patrick SteinhardtAug 5, 2026
  15. 6/7 odb/streaming: rename `struct read_object_fd_data`Patrick Steinhardt, Aug 4, 2026
  16. Justin ToblerAug 4, 2026
  17. 7/7 odb/streaming: unify function names to create new streamsPatrick Steinhardt, Aug 4, 2026
  18. Justin ToblerAug 4, 2026
  19. 0/8 odb: unify read and write streamsPatrick Steinhardt, Aug 5, 2026
  20. 1/8 odb/streaming: track write stream size in the structurePatrick Steinhardt, Aug 5, 2026
  21. Karthik NayakAug 11, 2026
  22. 2/8 odb/streaming: drop `is_finished` fieldPatrick Steinhardt, Aug 5, 2026
  23. Karthik NayakAug 11, 2026
  24. 3/8 odb/streaming: support streaming arbitrary object typesPatrick Steinhardt, Aug 5, 2026
  25. 4/8 odb/streaming: rename `struct odb_read_stream`Patrick Steinhardt, Aug 5, 2026
  26. 5/8 odb/streaming: consolidate read and write streamsPatrick Steinhardt, Aug 5, 2026
  27. Karthik NayakAug 11, 2026
  28. 6/8 odb/streaming: rename `struct read_object_fd_data`Patrick Steinhardt, Aug 5, 2026
  29. 7/8 odb/streaming: rename `struct input_zstream_data`Patrick Steinhardt, Aug 5, 2026
  30. 8/8 odb/streaming: unify function names to create new streamsPatrick Steinhardt, Aug 5, 2026
  31. Karthik NayakAug 11, 2026
  32. Patrick SteinhardtAug 11, 2026
  33. Justin ToblerAug 11, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.