git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 3/7] odb/streaming: support streaming arbitrary object types

From
Patrick Steinhardt <ps@pks.im>
Date
Aug 5, 2026, 06:06 UTC
Message-ID
<anLS5z22CpF82cd7@pks.im>
In-Reply-To
<xmqqpkzxvhqm.fsf@gitster.g>
On Tue, Aug 04, 2026 at 11:54:41AM -0700, Junio C Hamano wrote:
Show 12 quoted lines
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > The object database supports the ability to write object streams into
> > it. This functionality is used when we encounter a blob that is larger
> > than "core.bigFileThreshold" so that we don't have to soak large files
> > into memory.
> 
> I am still not sold the benefit of using a single "stream" type both
> for reading and writing yet at this point in my reading (I am not
> yet done 50% of the series yet at step 3/7), but I agree that it
> would be a good thing to be able to stream objects that are not
> blobs.

The reason why I want to unify these two streams is mostly that despite their name, they basically do the exact same thing: both stream types allow the user to read data from them in a streaming fashion. The only thing that's different about the "write" stream is that it doesn't encode its information as part of the stream itself, whereas the "read" stream does. So having two types is quite pointless in the first place.

Show 19 quoted lines
> > diff --git a/odb/source-inmemory.c b/odb/source-inmemory.c
> > index 01bb81c63c..4f76db5496 100644
> > --- a/odb/source-inmemory.c
> > +++ b/odb/source-inmemory.c
> > @@ -293,7 +293,7 @@ static int odb_source_inmemory_write_object_stream(struct odb_source *source,
> >  	hash_object_file(source->odb->repo->hash_algo, data, total_read, OBJ_BLOB, oid);
> >  
> >  	ret = odb_source_inmemory_write_object(source, data, stream->size,
> > -					       OBJ_BLOB, oid, NULL, NULL, 0);
> > +					       stream->type, oid, NULL, NULL, 0);
> 
> It is a bit annoying that we treat 'inmemory' as if it were a valid
> single word both in the filename and in the function name, but more
> importantly, hash_object_file() (used to compute the object name of
> the object we are writing into the variable 'oid') still hashes
> assuming that the object is a blob.  What is the implication of
> feeding the data to odb_source_in_memory_write_object() as
> stream->type (which is not necessarily OBJ_BLOB) with that 'oid'
> whose object name was computed as OBJ_BLOB?

Oh, that's an oversight on my part. We'd use the wrong object header, thus arrive at a wrong hash and then ultimately store the object under the wrong hash in the in-memory source. Which doesn't really matter after this patch series as we still only write blobs via streams, but it's a bug waiting to happen. Fixed now.

Patrick
Previous: Junio C HamanoNext: Patrick Steinhardt
Message 10 of 33 in “odb: unify read and write streams”
  1. 0/7 odb: unify read and write streamsPatrick Steinhardt, Aug 4, 2026
  2. 1/7 odb/streaming: track write stream size in the structurePatrick Steinhardt, Aug 4, 2026
  3. Justin ToblerAug 4, 2026
  4. 2/7 odb/streaming: drop `is_finished` fieldPatrick Steinhardt, Aug 4, 2026
  5. Justin ToblerAug 4, 2026
  6. 3/7 odb/streaming: support streaming arbitrary object typesPatrick Steinhardt, Aug 4, 2026
  7. Justin ToblerAug 4, 2026
  8. Patrick SteinhardtAug 5, 2026
  9. Junio C HamanoAug 4, 2026
  10. Patrick SteinhardtAug 5, 2026
  11. 4/7 odb/streaming: rename `struct odb_read_stream`Patrick Steinhardt, Aug 4, 2026
  12. 5/7 odb/streaming: consolidate read and write streamsPatrick Steinhardt, Aug 4, 2026
  13. Justin ToblerAug 4, 2026
  14. Patrick SteinhardtAug 5, 2026
  15. 6/7 odb/streaming: rename `struct read_object_fd_data`Patrick Steinhardt, Aug 4, 2026
  16. Justin ToblerAug 4, 2026
  17. 7/7 odb/streaming: unify function names to create new streamsPatrick Steinhardt, Aug 4, 2026
  18. Justin ToblerAug 4, 2026
  19. 0/8 odb: unify read and write streamsPatrick Steinhardt, Aug 5, 2026
  20. 1/8 odb/streaming: track write stream size in the structurePatrick Steinhardt, Aug 5, 2026
  21. Karthik NayakAug 11, 2026
  22. 2/8 odb/streaming: drop `is_finished` fieldPatrick Steinhardt, Aug 5, 2026
  23. Karthik NayakAug 11, 2026
  24. 3/8 odb/streaming: support streaming arbitrary object typesPatrick Steinhardt, Aug 5, 2026
  25. 4/8 odb/streaming: rename `struct odb_read_stream`Patrick Steinhardt, Aug 5, 2026
  26. 5/8 odb/streaming: consolidate read and write streamsPatrick Steinhardt, Aug 5, 2026
  27. Karthik NayakAug 11, 2026
  28. 6/8 odb/streaming: rename `struct read_object_fd_data`Patrick Steinhardt, Aug 5, 2026
  29. 7/8 odb/streaming: rename `struct input_zstream_data`Patrick Steinhardt, Aug 5, 2026
  30. 8/8 odb/streaming: unify function names to create new streamsPatrick Steinhardt, Aug 5, 2026
  31. Karthik NayakAug 11, 2026
  32. Patrick SteinhardtAug 11, 2026
  33. Justin ToblerAug 11, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.