git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 3/7] odb/streaming: support streaming arbitrary object types

From
Justin Tobler <jltobler@gmail.com>
Date
Aug 4, 2026, 18:03 UTC
Message-ID
<anInniMjCtU9Qae7@denethor>
In-Reply-To
<20260804-pks-odb-stream-unification-v1-3-86d70e82345e@pks.im>
On 26/08/04 09:25AM, Patrick Steinhardt wrote:
Show 16 quoted lines
> The object database supports the ability to write object streams into
> it. This functionality is used when we encounter a blob that is larger
> than "core.bigFileThreshold" so that we don't have to soak large files
> into memory.
> 
> As we only ever write large files, the infrastructure doesn't support
> specifying any other object type than "blob". This limitation is quite
> artificial though: there is no reason why we shouldn't support writing
> arbitrary large objects with a stream. While it's very unlikely that we
> encounter a huge object other than a blob, users are known to be
> creative and sometimes like to inflict pain on themselves by creating
> commits or trees that are huge.
> 
> Extend the infrastructure to support streaming arbitrary object types.
> For now we don't use this functionality anywhere, but it brings us a bit
> closer to unify `struct odb_read_stream` and `struct odb_write_stream`.
Very happy to see this change. :)
Show 10 quoted lines
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  builtin/unpack-objects.c      |  1 +
>  object-file.c                 | 31 +++++++++++++++----------------
>  odb/source-inmemory.c         |  2 +-
>  odb/source-loose.c            |  2 +-
>  odb/streaming.c               |  3 ++-
>  odb/streaming.h               |  3 ++-
>  t/unit-tests/u-odb-inmemory.c |  7 +++++--
>  7 files changed, 27 insertions(+), 22 deletions(-)

Just FYI, there is also a comment in "odb/transaction.h" for the `write_object_stream` callback that is also now outdated due to this change. We may want to update that too.

[snip]
Show 6 quoted lines
> @@ -953,7 +953,7 @@ int index_fd(struct index_state *istate, struct object_id *oid,
>  				 type, path, flags);
>  	} else {
>  		struct odb_write_stream stream;
> -		odb_write_stream_from_fd(&stream, fd, xsize_t(st->st_size));
> +		odb_write_stream_from_fd(&stream, fd, xsize_t(st->st_size), OBJ_BLOB);

We still only target large blobs for streaming here, but the underlying infrastructure is now generic which is nice.

[snip]
Show 9 quoted lines
> diff --git a/odb/streaming.h b/odb/streaming.h
> index 5e8e6e532e..3c8ed55129 100644
> --- a/odb/streaming.h
> +++ b/odb/streaming.h
> @@ -56,6 +56,7 @@ struct odb_write_stream {
>  	ssize_t (*read)(struct odb_write_stream *, unsigned char *, size_t);
>  	void *data;
>  	size_t size;
> +	enum object_type type;

We now store the object type in the stream itself. Similar to size information, the type information is always known in advance when creating the object stream.

The rest of this patch is just updating call sites accordingly. Looks good.

-Justin
Previous: Patrick SteinhardtNext: Patrick Steinhardt
Message 7 of 33 in “odb: unify read and write streams”
  1. 0/7 odb: unify read and write streamsPatrick Steinhardt, Aug 4, 2026
  2. 1/7 odb/streaming: track write stream size in the structurePatrick Steinhardt, Aug 4, 2026
  3. Justin ToblerAug 4, 2026
  4. 2/7 odb/streaming: drop `is_finished` fieldPatrick Steinhardt, Aug 4, 2026
  5. Justin ToblerAug 4, 2026
  6. 3/7 odb/streaming: support streaming arbitrary object typesPatrick Steinhardt, Aug 4, 2026
  7. Justin ToblerAug 4, 2026
  8. Patrick SteinhardtAug 5, 2026
  9. Junio C HamanoAug 4, 2026
  10. Patrick SteinhardtAug 5, 2026
  11. 4/7 odb/streaming: rename `struct odb_read_stream`Patrick Steinhardt, Aug 4, 2026
  12. 5/7 odb/streaming: consolidate read and write streamsPatrick Steinhardt, Aug 4, 2026
  13. Justin ToblerAug 4, 2026
  14. Patrick SteinhardtAug 5, 2026
  15. 6/7 odb/streaming: rename `struct read_object_fd_data`Patrick Steinhardt, Aug 4, 2026
  16. Justin ToblerAug 4, 2026
  17. 7/7 odb/streaming: unify function names to create new streamsPatrick Steinhardt, Aug 4, 2026
  18. Justin ToblerAug 4, 2026
  19. 0/8 odb: unify read and write streamsPatrick Steinhardt, Aug 5, 2026
  20. 1/8 odb/streaming: track write stream size in the structurePatrick Steinhardt, Aug 5, 2026
  21. Karthik NayakAug 11, 2026
  22. 2/8 odb/streaming: drop `is_finished` fieldPatrick Steinhardt, Aug 5, 2026
  23. Karthik NayakAug 11, 2026
  24. 3/8 odb/streaming: support streaming arbitrary object typesPatrick Steinhardt, Aug 5, 2026
  25. 4/8 odb/streaming: rename `struct odb_read_stream`Patrick Steinhardt, Aug 5, 2026
  26. 5/8 odb/streaming: consolidate read and write streamsPatrick Steinhardt, Aug 5, 2026
  27. Karthik NayakAug 11, 2026
  28. 6/8 odb/streaming: rename `struct read_object_fd_data`Patrick Steinhardt, Aug 5, 2026
  29. 7/8 odb/streaming: rename `struct input_zstream_data`Patrick Steinhardt, Aug 5, 2026
  30. 8/8 odb/streaming: unify function names to create new streamsPatrick Steinhardt, Aug 5, 2026
  31. Karthik NayakAug 11, 2026
  32. Patrick SteinhardtAug 11, 2026
  33. Justin ToblerAug 11, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.