{"thread":{"id":"66111","subject":"[PATCH 0/7] odb: unify read and write streams","startedAt":"2026-08-04T07:25:44Z","lastAt":"2026-08-11T20:10:02Z","messageCount":33,"participants":["Patrick Steinhardt","Justin Tobler","Junio C Hamano","Karthik Nayak"],"isPatch":true,"patchVersion":1,"patchTotal":7},"messages":[{"id":"549527","messageId":"20260804-pks-odb-stream-unification-v1-0-86d70e82345e@pks.im","threadId":"66111","inReplyTo":null,"subject":"[PATCH 0/7] odb: unify read and write streams","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-04T07:25:28Z","receivedAt":"2026-08-04T07:25:44Z","isPatch":true,"body":"Hi,\n\nwe have two different kind of object database streams in our code base:\n`odb_write_stream` and `odb_read_stream`. While those are used for\ndifferent use cases, the provided functionality is ultimately the exact\nsame.\n\nThis patch series thus refactors these streams so that we have a single\n`odb_stream`, only. This allows us to reuse the streams for different\nkinds of purposes and makes them more generally useful overall. For\nexample, it's trivially possible now to create an object stream for any\ngiven object and then write that stream into a different source.\n\nThe series is built on top of 5b2471720c (The 10th batch, 2026-08-03).\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (7):\n odb/streaming: track write stream size in the structure\n odb/streaming: drop `is_finished` field\n odb/streaming: support streaming arbitrary object types\n odb/streaming: rename `struct odb_read_stream`\n odb/streaming: consolidate read and write streams\n odb/streaming: rename `struct read_object_fd_data`\n odb/streaming: unify function names to create new streams\n\n archive-tar.c | 8 ++--\n archive-zip.c | 12 ++---\n builtin/index-pack.c | 8 ++--\n builtin/pack-objects.c | 18 ++++----\n builtin/unpack-objects.c | 42 +++++++++--------\n object-file.c | 76 +++++++++++++++---------------\n object-file.h | 2 +-\n object.c | 6 +--\n odb.c | 4 +-\n odb.h | 4 +-\n odb/source-files.c | 7 ++-\n odb/source-inmemory.c | 32 +++++++------\n odb/source-loose.c | 33 ++++++++------\n odb/source-packed.c | 5 +-\n odb/source.h | 13 +++---\n odb/streaming.c | 104 ++++++++++++++++++++----------------------\n odb/streaming.h | 69 ++++++++++------------------\n odb/transaction.c | 6 +--\n odb/transaction.h | 6 +--\n pack-check.c | 4 +-\n packfile.c | 8 ++--\n packfile.h | 4 +-\n t/unit-tests/u-odb-inmemory.c | 37 ++++++++-------\n 23 files changed, 247 insertions(+), 261 deletions(-)\n\n\n---\nbase-commit: 5b2471720c93ee30e5764a19f3d3b3ae9ec9712a\nchange-id: 20260724-pks-odb-stream-unification-334dc2a75888\n\n"},{"id":"549528","messageId":"20260804-pks-odb-stream-unification-v1-1-86d70e82345e@pks.im","threadId":"66111","inReplyTo":"20260804-pks-odb-stream-unification-v1-0-86d70e82345e@pks.im","subject":"[PATCH 1/7] odb/streaming: track write stream size in the structure","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-04T07:25:29Z","receivedAt":"2026-08-04T07:25:46Z","isPatch":true,"body":"When passing around a `struct odb_write_stream` we typically also have\nto pass the number of bytes that the stream will yield. This is required\nbecause the object header itself contains that size, and consequently we\ncannot write the header without that information.\n\nMove this information into the stream itself so that it becomes self-\ndescribing. In addition to that, this also brings the `struct\nodb_write_stream` a bit closer to the `struct odb_read_stream` so that\nwe can eventually merge both stream types.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/unpack-objects.c | 3 ++-\n object-file.c | 25 +++++++++++--------------\n odb.c | 4 ++--\n odb.h | 2 +-\n odb/source-files.c | 3 +--\n odb/source-inmemory.c | 11 +++++------\n odb/source-loose.c | 7 +++----\n odb/source-packed.c | 1 -\n odb/source.h | 5 ++---\n odb/streaming.c | 1 +\n odb/streaming.h | 1 +\n odb/transaction.c | 4 ++--\n odb/transaction.h | 4 ++--\n t/unit-tests/u-odb-inmemory.c | 11 +++++------\n 14 files changed, 38 insertions(+), 44 deletions(-)\n\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex 4263edfbec..f3e0b504f4 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -392,13 +392,14 @@ static void stream_blob(unsigned long size, unsigned nr)\n \tstruct odb_write_stream in_stream = {\n \t\t.read = feed_input_zstream,\n \t\t.data = &data,\n+\t\t.size = size,\n \t};\n \tstruct obj_info *info = &obj_list[nr];\n \n \tdata.zstream = &zstream;\n \tgit_inflate_init(&zstream);\n \n-\tif (odb_write_object_stream(the_repository->objects, &in_stream, size, &info->oid))\n+\tif (odb_write_object_stream(the_repository->objects, &in_stream, &info->oid))\n \t\tdie(_(\"failed to write object in stream\"));\n \n \tif (data.status != Z_STREAM_END)\ndiff --git a/object-file.c b/object-file.c\nindex ec35c318bc..b196abb596 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -704,7 +704,7 @@ static void prepare_packfile_transaction(struct odb_transaction_files *transacti\n \n static int hash_blob_stream(struct odb_write_stream *stream,\n \t\t\t const struct git_hash_algo *hash_algo,\n-\t\t\t struct object_id *result_oid, size_t size)\n+\t\t\t struct object_id *result_oid)\n {\n \tunsigned char buf[16384];\n \tstruct git_hash_ctx ctx;\n@@ -712,7 +712,7 @@ static int hash_blob_stream(struct odb_write_stream *stream,\n \tsize_t bytes_hashed = 0;\n \n \theader_len = format_object_header((char *)buf, sizeof(buf),\n-\t\t\t\t\t OBJ_BLOB, size);\n+\t\t\t\t\t OBJ_BLOB, stream->size);\n \tgit_hash_init(&ctx, hash_algo);\n \tgit_hash_update(&ctx, buf, header_len);\n \n@@ -727,7 +727,7 @@ static int hash_blob_stream(struct odb_write_stream *stream,\n \t\tbytes_hashed += read_result;\n \t}\n \n-\tif (bytes_hashed != size)\n+\tif (bytes_hashed != stream->size)\n \t\treturn -1;\n \n \tgit_hash_final_oid(result_oid, &ctx);\n@@ -740,7 +740,7 @@ static int hash_blob_stream(struct odb_write_stream *stream,\n * packfile in state while updating the hash in ctx.\n */\n static void stream_blob_to_pack(struct transaction_packfile *state,\n-\t\t\t\tstruct git_hash_ctx *ctx, size_t size,\n+\t\t\t\tstruct git_hash_ctx *ctx,\n \t\t\t\tstruct odb_write_stream *stream)\n {\n \tgit_zstream s;\n@@ -753,7 +753,7 @@ static void stream_blob_to_pack(struct transaction_packfile *state,\n \n \tgit_deflate_init(&s, cfg->pack_compression_level);\n \n-\thdrlen = encode_in_pack_object_header(obuf, sizeof(obuf), OBJ_BLOB, size);\n+\thdrlen = encode_in_pack_object_header(obuf, sizeof(obuf), OBJ_BLOB, stream->size);\n \ts.next_out = obuf + hdrlen;\n \ts.avail_out = sizeof(obuf) - hdrlen;\n \n@@ -793,9 +793,9 @@ static void stream_blob_to_pack(struct transaction_packfile *state,\n \t\t}\n \t}\n \n-\tif (bytes_read != size)\n+\tif (bytes_read != stream->size)\n \t\tdie(\"read %\" PRIuMAX \" bytes of blob data, but expected %\" PRIuMAX \" bytes\",\n-\t\t (uintmax_t)bytes_read, (uintmax_t)size);\n+\t\t (uintmax_t)bytes_read, (uintmax_t)stream->size);\n \n \tgit_deflate_end(&s);\n }\n@@ -870,7 +870,6 @@ static void flush_packfile_transaction(struct odb_transaction_files *transaction\n */\n static int odb_transaction_files_write_object_stream(struct odb_transaction *base,\n \t\t\t\t\t\t struct odb_write_stream *stream,\n-\t\t\t\t\t\t size_t size,\n \t\t\t\t\t\t struct object_id *result_oid)\n {\n \tstruct odb_transaction_files *transaction = container_of(base,\n@@ -884,7 +883,7 @@ static int odb_transaction_files_write_object_stream(struct odb_transaction *bas\n \tstruct pack_idx_entry *idx;\n \n \theader_len = format_object_header((char *)obuf, sizeof(obuf),\n-\t\t\t\t\t OBJ_BLOB, size);\n+\t\t\t\t\t OBJ_BLOB, stream->size);\n \tgit_hash_init(&ctx, transaction->base.source->odb->repo->hash_algo);\n \tgit_hash_update(&ctx, obuf, header_len);\n \n@@ -899,7 +898,7 @@ static int odb_transaction_files_write_object_stream(struct odb_transaction *bas\n \t * to zlib compression and is sufficient for this check.\n \t */\n \tif (state->nr_written && pack_size_limit_cfg &&\n-\t pack_size_limit_cfg < state->offset + size)\n+\t pack_size_limit_cfg < state->offset + stream->size)\n \t\tflush_packfile_transaction(transaction);\n \n \tCALLOC_ARRAY(idx, 1);\n@@ -909,7 +908,7 @@ static int odb_transaction_files_write_object_stream(struct odb_transaction *bas\n \thashfile_checkpoint(state->f, &checkpoint);\n \tidx->offset = state->offset;\n \tcrc32_begin(state->f);\n-\tstream_blob_to_pack(state, &ctx, size, stream);\n+\tstream_blob_to_pack(state, &ctx, stream);\n \tgit_hash_final_oid(result_oid, &ctx);\n \n \tidx->crc32 = crc32_end(state->f);\n@@ -962,14 +961,12 @@ int index_fd(struct index_state *istate, struct object_id *oid,\n \t\t\t\todb_transaction_begin_or_die(odb, &transaction, 0);\n \t\t\tret = odb_transaction_write_object_stream(transaction,\n \t\t\t\t\t\t\t\t &stream,\n-\t\t\t\t\t\t\t\t xsize_t(st->st_size),\n \t\t\t\t\t\t\t\t oid);\n \t\t\tif (!inflight)\n \t\t\t\todb_transaction_commit(transaction);\n \t\t} else {\n \t\t\tret = hash_blob_stream(&stream,\n-\t\t\t\t\t the_repository->hash_algo, oid,\n-\t\t\t\t\t xsize_t(st->st_size));\n+\t\t\t\t\t the_repository->hash_algo, oid);\n \t\t}\n \n \t\todb_write_stream_release(&stream);\ndiff --git a/odb.c b/odb.c\nindex dabd481f57..585b2b2965 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1028,10 +1028,10 @@ int odb_write_object_ext(struct object_database *odb,\n }\n \n int odb_write_object_stream(struct object_database *odb,\n-\t\t\t struct odb_write_stream *stream, size_t len,\n+\t\t\t struct odb_write_stream *stream,\n \t\t\t struct object_id *oid)\n {\n-\treturn odb_source_write_object_stream(odb->sources, stream, len, oid);\n+\treturn odb_source_write_object_stream(odb->sources, stream, oid);\n }\n \n struct object_database *odb_new(struct repository *repo,\ndiff --git a/odb.h b/odb.h\nindex cbc2f9ced4..019d3af3e8 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -629,7 +629,7 @@ static inline int odb_write_object(struct object_database *odb,\n struct odb_write_stream;\n \n int odb_write_object_stream(struct object_database *odb,\n-\t\t\t struct odb_write_stream *stream, size_t len,\n+\t\t\t struct odb_write_stream *stream,\n \t\t\t struct object_id *oid);\n \n void parse_alternates(const char *string,\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 5e086d266f..f51960bd71 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -175,11 +175,10 @@ static int odb_source_files_write_object(struct odb_source *source,\n \n static int odb_source_files_write_object_stream(struct odb_source *source,\n \t\t\t\t\t\tstruct odb_write_stream *stream,\n-\t\t\t\t\t\tsize_t len,\n \t\t\t\t\t\tstruct object_id *oid)\n {\n \tstruct odb_source_files *files = odb_source_files_downcast(source);\n-\treturn odb_source_write_object_stream(&files->loose->base, stream, len, oid);\n+\treturn odb_source_write_object_stream(&files->loose->base, stream, oid);\n }\n \n static int odb_source_files_begin_transaction(struct odb_source *source,\ndiff --git a/odb/source-inmemory.c b/odb/source-inmemory.c\nindex 3e71611b8e..398131e194 100644\n--- a/odb/source-inmemory.c\n+++ b/odb/source-inmemory.c\n@@ -257,7 +257,6 @@ static int odb_source_inmemory_write_object(struct odb_source *source,\n \n static int odb_source_inmemory_write_object_stream(struct odb_source *source,\n \t\t\t\t\t\t struct odb_write_stream *stream,\n-\t\t\t\t\t\t size_t len,\n \t\t\t\t\t\t struct object_id *oid)\n {\n \tchar buf[16384];\n@@ -265,12 +264,12 @@ static int odb_source_inmemory_write_object_stream(struct odb_source *source,\n \tchar *data;\n \tint ret;\n \n-\tCALLOC_ARRAY(data, len);\n+\tCALLOC_ARRAY(data, stream->size);\n \twhile (!stream->is_finished) {\n \t\tssize_t bytes_read;\n \n \t\tbytes_read = odb_write_stream_read(stream, buf, sizeof(buf));\n-\t\tif (total_read + bytes_read > len) {\n+\t\tif (total_read + bytes_read > stream->size) {\n \t\t\tret = error(\"object stream yielded more bytes than expected\");\n \t\t\tgoto out;\n \t\t}\n@@ -279,15 +278,15 @@ static int odb_source_inmemory_write_object_stream(struct odb_source *source,\n \t\ttotal_read += bytes_read;\n \t}\n \n-\tif (total_read != len) {\n+\tif (total_read != stream->size) {\n \t\tret = error(\"object stream yielded less bytes than expected\");\n \t\tgoto out;\n \t}\n \n \thash_object_file(source->odb->repo->hash_algo, data, total_read, OBJ_BLOB, oid);\n \n-\tret = odb_source_inmemory_write_object(source, data, len, OBJ_BLOB, oid,\n-\t\t\t\t\t NULL, NULL, 0);\n+\tret = odb_source_inmemory_write_object(source, data, stream->size,\n+\t\t\t\t\t OBJ_BLOB, oid, NULL, NULL, 0);\n \tif (ret < 0)\n \t\tgoto out;\n \ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex ef0e919277..77a2adb52a 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -846,7 +846,6 @@ static int odb_source_loose_write_object(struct odb_source *source,\n \n static int odb_source_loose_write_object_stream(struct odb_source *source,\n \t\t\t\t\t\tstruct odb_write_stream *in_stream,\n-\t\t\t\t\t\tsize_t len,\n \t\t\t\t\t\tstruct object_id *oid)\n {\n \tstruct odb_source_loose *loose = odb_source_loose_downcast(source);\n@@ -868,7 +867,7 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,\n \n \t/* Since oid is not determined, save tmp file to odb path. */\n \tstrbuf_addf(&filename, \"%s/\", loose->base.path);\n-\thdrlen = format_object_header(hdr, sizeof(hdr), OBJ_BLOB, len);\n+\thdrlen = format_object_header(hdr, sizeof(hdr), OBJ_BLOB, in_stream->size);\n \n \t/*\n \t * Common steps for write_loose_object and stream_loose_object to\n@@ -916,9 +915,9 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,\n \t\t */\n \t} while (ret == Z_OK || ret == Z_BUF_ERROR);\n \n-\tif (stream.total_in != len + hdrlen)\n+\tif (stream.total_in != in_stream->size + hdrlen)\n \t\tdie(_(\"write stream object %\"PRIuMAX\" != %\"PRIuMAX), (uintmax_t)stream.total_in,\n-\t\t (uintmax_t)len + hdrlen);\n+\t\t (uintmax_t)in_stream->size + hdrlen);\n \n \t/*\n \t * Common steps for write_loose_object and stream_loose_object to\ndiff --git a/odb/source-packed.c b/odb/source-packed.c\nindex 0890704e76..e6ff74833b 100644\n--- a/odb/source-packed.c\n+++ b/odb/source-packed.c\n@@ -610,7 +610,6 @@ static int odb_source_packed_write_object(struct odb_source *source UNUSED,\n \n static int odb_source_packed_write_object_stream(struct odb_source *source UNUSED,\n \t\t\t\t\t\t struct odb_write_stream *stream UNUSED,\n-\t\t\t\t\t\t size_t len UNUSED,\n \t\t\t\t\t\t struct object_id *oid UNUSED)\n {\n \treturn error(\"packed backend cannot write object streams\");\ndiff --git a/odb/source.h b/odb/source.h\nindex fc04dd5cda..0080148ba7 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -221,7 +221,7 @@ struct odb_source {\n \t * otherwise.\n \t */\n \tint (*write_object_stream)(struct odb_source *source,\n-\t\t\t\t struct odb_write_stream *stream, size_t len,\n+\t\t\t\t struct odb_write_stream *stream,\n \t\t\t\t struct object_id *oid);\n \n \t/*\n@@ -437,10 +437,9 @@ static inline int odb_source_write_object(struct odb_source *source,\n */\n static inline int odb_source_write_object_stream(struct odb_source *source,\n \t\t\t\t\t\t struct odb_write_stream *stream,\n-\t\t\t\t\t\t size_t len,\n \t\t\t\t\t\t struct object_id *oid)\n {\n-\treturn source->write_object_stream(source, stream, len, oid);\n+\treturn source->write_object_stream(source, stream, oid);\n }\n \n /*\ndiff --git a/odb/streaming.c b/odb/streaming.c\nindex 20531e864c..38c2f6687c 100644\n--- a/odb/streaming.c\n+++ b/odb/streaming.c\n@@ -336,5 +336,6 @@ void odb_write_stream_from_fd(struct odb_write_stream *stream, int fd,\n \n \tstream->data = data;\n \tstream->read = read_object_fd;\n+\tstream->size = size;\n \tstream->is_finished = 0;\n }\ndiff --git a/odb/streaming.h b/odb/streaming.h\nindex c023671780..4d7d31b5aa 100644\n--- a/odb/streaming.h\n+++ b/odb/streaming.h\n@@ -55,6 +55,7 @@ ssize_t odb_read_stream_read(struct odb_read_stream *stream, void *buf, size_t l\n struct odb_write_stream {\n \tssize_t (*read)(struct odb_write_stream *, unsigned char *, size_t);\n \tvoid *data;\n+\tsize_t size;\n \tint is_finished;\n };\n \ndiff --git a/odb/transaction.c b/odb/transaction.c\nindex dab7da6a9a..6aaf133812 100644\n--- a/odb/transaction.c\n+++ b/odb/transaction.c\n@@ -40,9 +40,9 @@ int odb_transaction_commit(struct odb_transaction *transaction)\n \n int odb_transaction_write_object_stream(struct odb_transaction *transaction,\n \t\t\t\t\tstruct odb_write_stream *stream,\n-\t\t\t\t\tsize_t len, struct object_id *oid)\n+\t\t\t\t\tstruct object_id *oid)\n {\n-\treturn transaction->write_object_stream(transaction, stream, len, oid);\n+\treturn transaction->write_object_stream(transaction, stream, oid);\n }\n \n int odb_transaction_env(struct odb_transaction *transaction, struct strvec *env)\ndiff --git a/odb/transaction.h b/odb/transaction.h\nindex 4cb2eafcbf..ffb279314c 100644\n--- a/odb/transaction.h\n+++ b/odb/transaction.h\n@@ -31,7 +31,7 @@ struct odb_transaction {\n \t * otherwise.\n \t */\n \tint (*write_object_stream)(struct odb_transaction *transaction,\n-\t\t\t\t struct odb_write_stream *stream, size_t len,\n+\t\t\t\t struct odb_write_stream *stream,\n \t\t\t\t struct object_id *oid);\n \n \t/*\n@@ -82,7 +82,7 @@ int odb_transaction_commit(struct odb_transaction *transaction);\n */\n int odb_transaction_write_object_stream(struct odb_transaction *transaction,\n \t\t\t\t\tstruct odb_write_stream *stream,\n-\t\t\t\t\tsize_t len, struct object_id *oid);\n+\t\t\t\t\tstruct object_id *oid);\n \n /*\n * Populates the provided strvec with the environment variables that a child\ndiff --git a/t/unit-tests/u-odb-inmemory.c b/t/unit-tests/u-odb-inmemory.c\nindex ddf2db5c81..5ccc52dccc 100644\n--- a/t/unit-tests/u-odb-inmemory.c\n+++ b/t/unit-tests/u-odb-inmemory.c\n@@ -269,7 +269,6 @@ struct membuf_write_stream {\n \tstruct odb_write_stream base;\n \tconst char *buf;\n \tsize_t offset;\n-\tsize_t size;\n };\n \n static ssize_t membuf_write_stream_read(struct odb_write_stream *stream,\n@@ -280,13 +279,13 @@ static ssize_t membuf_write_stream_read(struct odb_write_stream *stream,\n \n \tif (chunk_size > len)\n \t\tchunk_size = len;\n-\tif (chunk_size > s->size - s->offset)\n-\t\tchunk_size = s->size - s->offset;\n+\tif (chunk_size > s->base.size - s->offset)\n+\t\tchunk_size = s->base.size - s->offset;\n \n \tmemcpy(buf, s->buf + s->offset, chunk_size);\n \n \ts->offset += chunk_size;\n-\tif (s->offset == s->size)\n+\tif (s->offset == s->base.size)\n \t\ts->base.is_finished = 1;\n \n \treturn chunk_size;\n@@ -298,13 +297,13 @@ void test_odb_inmemory__write_object_stream(void)\n \tconst char data[] = \"foobar\";\n \tstruct membuf_write_stream stream = {\n \t\t.base.read = membuf_write_stream_read,\n+\t\t.base.size = strlen(data),\n \t\t.buf = data,\n-\t\t.size = strlen(data),\n \t};\n \tstruct object_id written_oid;\n \n \tcl_must_pass(odb_source_write_object_stream(&source->base, &stream.base,\n-\t\t\t\t\t\t strlen(data), &written_oid));\n+\t\t\t\t\t\t &written_oid));\n \tcl_assert_equal_s(oid_to_hex(&written_oid), FOOBAR_OID);\n \tcl_assert_object_info(source, &written_oid, OBJ_BLOB, \"foobar\");\n \n\n-- \n2.55.0.679.g6767b8d81c.dirty\n\n"},{"id":"549529","messageId":"20260804-pks-odb-stream-unification-v1-2-86d70e82345e@pks.im","threadId":"66111","inReplyTo":"20260804-pks-odb-stream-unification-v1-0-86d70e82345e@pks.im","subject":"[PATCH 2/7] odb/streaming: drop `is_finished` field","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-04T07:25:30Z","receivedAt":"2026-08-04T07:25:50Z","isPatch":true,"body":"The `is_finished` field is used to track whether a write stream is done\nwriting all of its data. Tracking this field as part of the stream\nitself shouldn't be required though: callers will already know when the\nstream is done when the stream's read function returns zero bytes, same\nas when reading from a file descriptor.\n\nThere is one exception where it gets a bit more complicated: when\nconsuming data in \"builtin/unpack-objects.c\" it may happen that we don't\nyield any new bytes after reading from the pipe. This is addressed by\nlooping until we have produced at least a single byte of output.\n\nDrop the field from `struct odb_write_stream`. Again, same as in the\npreceding commit, this brings the structure a bit closer to its sibling\n`struct odb_read_stream`.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/unpack-objects.c | 15 ++++++++-------\n object-file.c | 13 ++++++++-----\n odb/source-inmemory.c | 9 ++++++++-\n odb/source-loose.c | 12 ++++++++----\n odb/streaming.c | 5 +----\n odb/streaming.h | 1 -\n t/unit-tests/u-odb-inmemory.c | 5 +++--\n 7 files changed, 36 insertions(+), 24 deletions(-)\n\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex f3e0b504f4..b7c486ea94 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -368,20 +368,20 @@ static ssize_t feed_input_zstream(struct odb_write_stream *in_stream,\n {\n \tstruct input_zstream_data *data = in_stream->data;\n \tgit_zstream *zstream = data->zstream;\n-\tvoid *in = fill(1);\n \n-\tif (in_stream->is_finished)\n+\tif (data->status != Z_OK)\n \t\treturn 0;\n \n \tzstream->next_out = buf;\n \tzstream->avail_out = buf_len;\n-\tzstream->next_in = in;\n-\tzstream->avail_in = len;\n \n-\tdata->status = git_inflate(zstream, 0);\n+\twhile (data->status == Z_OK && zstream->avail_out == buf_len) {\n+\t\tzstream->next_in = fill(1);\n+\t\tzstream->avail_in = len;\n+\t\tdata->status = git_inflate(zstream, 0);\n+\t\tuse(len - zstream->avail_in);\n+\t}\n \n-\tin_stream->is_finished = data->status != Z_OK;\n-\tuse(len - zstream->avail_in);\n \treturn buf_len - zstream->avail_out;\n }\n \n@@ -397,6 +397,7 @@ static void stream_blob(unsigned long size, unsigned nr)\n \tstruct obj_info *info = &obj_list[nr];\n \n \tdata.zstream = &zstream;\n+\tdata.status = Z_OK;\n \tgit_inflate_init(&zstream);\n \n \tif (odb_write_object_stream(the_repository->objects, &in_stream, &info->oid))\ndiff --git a/object-file.c b/object-file.c\nindex b196abb596..317c09dff8 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -716,12 +716,13 @@ static int hash_blob_stream(struct odb_write_stream *stream,\n \tgit_hash_init(&ctx, hash_algo);\n \tgit_hash_update(&ctx, buf, header_len);\n \n-\twhile (!stream->is_finished) {\n+\twhile (1) {\n \t\tssize_t read_result = odb_write_stream_read(stream, buf,\n \t\t\t\t\t\t\t sizeof(buf));\n-\n \t\tif (read_result < 0)\n \t\t\treturn -1;\n+\t\tif (!read_result)\n+\t\t\tbreak;\n \n \t\tgit_hash_update(&ctx, buf, read_result);\n \t\tbytes_hashed += read_result;\n@@ -749,6 +750,7 @@ static void stream_blob_to_pack(struct transaction_packfile *state,\n \tunsigned hdrlen;\n \tint status = Z_OK;\n \tstruct repo_config_values *cfg = repo_config_values(the_repository);\n+\tbool is_finished = false;\n \tsize_t bytes_read = 0;\n \n \tgit_deflate_init(&s, cfg->pack_compression_level);\n@@ -758,12 +760,13 @@ static void stream_blob_to_pack(struct transaction_packfile *state,\n \ts.avail_out = sizeof(obuf) - hdrlen;\n \n \twhile (status != Z_STREAM_END) {\n-\t\tif (!stream->is_finished && !s.avail_in) {\n+\t\tif (!is_finished && !s.avail_in) {\n \t\t\tssize_t rsize = odb_write_stream_read(stream, ibuf,\n \t\t\t\t\t\t\t sizeof(ibuf));\n-\n \t\t\tif (rsize < 0)\n \t\t\t\tdie(\"failed to read blob data\");\n+\t\t\tif (!rsize)\n+\t\t\t\tis_finished = true;\n \n \t\t\tgit_hash_update(ctx, ibuf, rsize);\n \n@@ -772,7 +775,7 @@ static void stream_blob_to_pack(struct transaction_packfile *state,\n \t\t\tbytes_read += rsize;\n \t\t}\n \n-\t\tstatus = git_deflate(&s, stream->is_finished ? Z_FINISH : 0);\n+\t\tstatus = git_deflate(&s, is_finished ? Z_FINISH : 0);\n \n \t\tif (!s.avail_out || status == Z_STREAM_END) {\n \t\t\tsize_t written = s.next_out - obuf;\ndiff --git a/odb/source-inmemory.c b/odb/source-inmemory.c\nindex 398131e194..01bb81c63c 100644\n--- a/odb/source-inmemory.c\n+++ b/odb/source-inmemory.c\n@@ -265,10 +265,17 @@ static int odb_source_inmemory_write_object_stream(struct odb_source *source,\n \tint ret;\n \n \tCALLOC_ARRAY(data, stream->size);\n-\twhile (!stream->is_finished) {\n+\twhile (1) {\n \t\tssize_t bytes_read;\n \n \t\tbytes_read = odb_write_stream_read(stream, buf, sizeof(buf));\n+\t\tif (bytes_read < 0) {\n+\t\t\tret = error(\"failed to read object stream\");\n+\t\t\tgoto out;\n+\t\t}\n+\t\tif (!bytes_read)\n+\t\t\tbreak;\n+\n \t\tif (total_read + bytes_read > stream->size) {\n \t\t\tret = error(\"object stream yielded more bytes than expected\");\n \t\t\tgoto out;\ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex 77a2adb52a..361b4e2a2a 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -859,6 +859,7 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,\n \tstruct strbuf filename = STRBUF_INIT;\n \tunsigned char buf[8192];\n \tint dirlen;\n+\tbool is_finished = false;\n \tchar hdr[MAX_HEADER_LEN];\n \tint hdrlen;\n \n@@ -889,7 +890,7 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,\n \tdo {\n \t\tunsigned char *in0 = stream.next_in;\n \n-\t\tif (!stream.avail_in && !in_stream->is_finished) {\n+\t\tif (!stream.avail_in && !is_finished) {\n \t\t\tssize_t read_len = odb_write_stream_read(in_stream, buf,\n \t\t\t\t\t\t\t\t sizeof(buf));\n \t\t\tif (read_len < 0) {\n@@ -898,12 +899,15 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,\n \t\t\t\tgoto cleanup;\n \t\t\t}\n \n+\t\t\t/* All data has been read. */\n+\t\t\tif (!read_len) {\n+\t\t\t\tis_finished = true;\n+\t\t\t\tflush = 1;\n+\t\t\t}\n+\n \t\t\tstream.avail_in = read_len;\n \t\t\tstream.next_in = buf;\n \t\t\tin0 = buf;\n-\t\t\t/* All data has been read. */\n-\t\t\tif (in_stream->is_finished)\n-\t\t\t\tflush = 1;\n \t\t}\n \t\tret = write_loose_object_common(loose, &c, &compat_c, &stream, flush, in0, fd,\n \t\t\t\t\t\tcompressed, sizeof(compressed));\ndiff --git a/odb/streaming.c b/odb/streaming.c\nindex 38c2f6687c..912e75e682 100644\n--- a/odb/streaming.c\n+++ b/odb/streaming.c\n@@ -310,7 +310,7 @@ static ssize_t read_object_fd(struct odb_write_stream *stream,\n \tssize_t read_result;\n \tsize_t count;\n \n-\tif (stream->is_finished)\n+\tif (!data->remaining)\n \t\treturn 0;\n \n \tcount = data->remaining < len ? data->remaining : len;\n@@ -319,8 +319,6 @@ static ssize_t read_object_fd(struct odb_write_stream *stream,\n \t\treturn -1;\n \n \tdata->remaining -= count;\n-\tif (!data->remaining)\n-\t\tstream->is_finished = 1;\n \n \treturn read_result;\n }\n@@ -337,5 +335,4 @@ void odb_write_stream_from_fd(struct odb_write_stream *stream, int fd,\n \tstream->data = data;\n \tstream->read = read_object_fd;\n \tstream->size = size;\n-\tstream->is_finished = 0;\n }\ndiff --git a/odb/streaming.h b/odb/streaming.h\nindex 4d7d31b5aa..5e8e6e532e 100644\n--- a/odb/streaming.h\n+++ b/odb/streaming.h\n@@ -56,7 +56,6 @@ struct odb_write_stream {\n \tssize_t (*read)(struct odb_write_stream *, unsigned char *, size_t);\n \tvoid *data;\n \tsize_t size;\n-\tint is_finished;\n };\n \n /*\ndiff --git a/t/unit-tests/u-odb-inmemory.c b/t/unit-tests/u-odb-inmemory.c\nindex 5ccc52dccc..4437140ed0 100644\n--- a/t/unit-tests/u-odb-inmemory.c\n+++ b/t/unit-tests/u-odb-inmemory.c\n@@ -277,6 +277,9 @@ static ssize_t membuf_write_stream_read(struct odb_write_stream *stream,\n \tstruct membuf_write_stream *s = container_of(stream, struct membuf_write_stream, base);\n \tsize_t chunk_size = 2;\n \n+\tif (s->offset == s->base.size)\n+\t\treturn 0;\n+\n \tif (chunk_size > len)\n \t\tchunk_size = len;\n \tif (chunk_size > s->base.size - s->offset)\n@@ -285,8 +288,6 @@ static ssize_t membuf_write_stream_read(struct odb_write_stream *stream,\n \tmemcpy(buf, s->buf + s->offset, chunk_size);\n \n \ts->offset += chunk_size;\n-\tif (s->offset == s->base.size)\n-\t\ts->base.is_finished = 1;\n \n \treturn chunk_size;\n }\n\n-- \n2.55.0.679.g6767b8d81c.dirty\n\n"},{"id":"549530","messageId":"20260804-pks-odb-stream-unification-v1-3-86d70e82345e@pks.im","threadId":"66111","inReplyTo":"20260804-pks-odb-stream-unification-v1-0-86d70e82345e@pks.im","subject":"[PATCH 3/7] odb/streaming: support streaming arbitrary object types","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-04T07:25:31Z","receivedAt":"2026-08-04T07:25:53Z","isPatch":true,"body":"The object database supports the ability to write object streams into\nit. This functionality is used when we encounter a blob that is larger\nthan \"core.bigFileThreshold\" so that we don't have to soak large files\ninto memory.\n\nAs we only ever write large files, the infrastructure doesn't support\nspecifying any other object type than \"blob\". This limitation is quite\nartificial though: there is no reason why we shouldn't support writing\narbitrary large objects with a stream. While it's very unlikely that we\nencounter a huge object other than a blob, users are known to be\ncreative and sometimes like to inflict pain on themselves by creating\ncommits or trees that are huge.\n\nExtend the infrastructure to support streaming arbitrary object types.\nFor now we don't use this functionality anywhere, but it brings us a bit\ncloser to unify `struct odb_read_stream` and `struct odb_write_stream`.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/unpack-objects.c | 1 +\n object-file.c | 31 +++++++++++++++----------------\n odb/source-inmemory.c | 2 +-\n odb/source-loose.c | 2 +-\n odb/streaming.c | 3 ++-\n odb/streaming.h | 3 ++-\n t/unit-tests/u-odb-inmemory.c | 7 +++++--\n 7 files changed, 27 insertions(+), 22 deletions(-)\n\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex b7c486ea94..7439ec53be 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -393,6 +393,7 @@ static void stream_blob(unsigned long size, unsigned nr)\n \t\t.read = feed_input_zstream,\n \t\t.data = &data,\n \t\t.size = size,\n+\t\t.type = OBJ_BLOB,\n \t};\n \tstruct obj_info *info = &obj_list[nr];\n \ndiff --git a/object-file.c b/object-file.c\nindex 317c09dff8..699a6a008c 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -702,9 +702,9 @@ static void prepare_packfile_transaction(struct odb_transaction_files *transacti\n \t\tdie_errno(\"unable to write pack header\");\n }\n \n-static int hash_blob_stream(struct odb_write_stream *stream,\n-\t\t\t const struct git_hash_algo *hash_algo,\n-\t\t\t struct object_id *result_oid)\n+static int hash_stream(struct odb_write_stream *stream,\n+\t\t const struct git_hash_algo *hash_algo,\n+\t\t struct object_id *result_oid)\n {\n \tunsigned char buf[16384];\n \tstruct git_hash_ctx ctx;\n@@ -712,7 +712,7 @@ static int hash_blob_stream(struct odb_write_stream *stream,\n \tsize_t bytes_hashed = 0;\n \n \theader_len = format_object_header((char *)buf, sizeof(buf),\n-\t\t\t\t\t OBJ_BLOB, stream->size);\n+\t\t\t\t\t stream->type, stream->size);\n \tgit_hash_init(&ctx, hash_algo);\n \tgit_hash_update(&ctx, buf, header_len);\n \n@@ -740,9 +740,9 @@ static int hash_blob_stream(struct odb_write_stream *stream,\n * Read the contents from the stream provided, streaming it to the\n * packfile in state while updating the hash in ctx.\n */\n-static void stream_blob_to_pack(struct transaction_packfile *state,\n-\t\t\t\tstruct git_hash_ctx *ctx,\n-\t\t\t\tstruct odb_write_stream *stream)\n+static void stream_to_pack(struct transaction_packfile *state,\n+\t\t\t struct git_hash_ctx *ctx,\n+\t\t\t struct odb_write_stream *stream)\n {\n \tgit_zstream s;\n \tunsigned char ibuf[16384];\n@@ -755,7 +755,7 @@ static void stream_blob_to_pack(struct transaction_packfile *state,\n \n \tgit_deflate_init(&s, cfg->pack_compression_level);\n \n-\thdrlen = encode_in_pack_object_header(obuf, sizeof(obuf), OBJ_BLOB, stream->size);\n+\thdrlen = encode_in_pack_object_header(obuf, sizeof(obuf), stream->type, stream->size);\n \ts.next_out = obuf + hdrlen;\n \ts.avail_out = sizeof(obuf) - hdrlen;\n \n@@ -764,7 +764,7 @@ static void stream_blob_to_pack(struct transaction_packfile *state,\n \t\t\tssize_t rsize = odb_write_stream_read(stream, ibuf,\n \t\t\t\t\t\t\t sizeof(ibuf));\n \t\t\tif (rsize < 0)\n-\t\t\t\tdie(\"failed to read blob data\");\n+\t\t\t\tdie(\"failed to read object data\");\n \t\t\tif (!rsize)\n \t\t\t\tis_finished = true;\n \n@@ -797,7 +797,7 @@ static void stream_blob_to_pack(struct transaction_packfile *state,\n \t}\n \n \tif (bytes_read != stream->size)\n-\t\tdie(\"read %\" PRIuMAX \" bytes of blob data, but expected %\" PRIuMAX \" bytes\",\n+\t\tdie(\"read %\" PRIuMAX \" bytes of object data, but expected %\" PRIuMAX \" bytes\",\n \t\t (uintmax_t)bytes_read, (uintmax_t)stream->size);\n \n \tgit_deflate_end(&s);\n@@ -868,7 +868,7 @@ static void flush_packfile_transaction(struct odb_transaction_files *transaction\n * result, which we need to know beforehand when writing a git object.\n * Since the primary motivation for trying to stream from the working\n * tree file and to avoid mmaping it in core is to deal with large\n- * binary blobs, they generally do not want to get any conversion, and\n+ * objects, they generally do not want to get any conversion, and\n * callers should avoid this code path when filters are requested.\n */\n static int odb_transaction_files_write_object_stream(struct odb_transaction *base,\n@@ -886,7 +886,7 @@ static int odb_transaction_files_write_object_stream(struct odb_transaction *bas\n \tstruct pack_idx_entry *idx;\n \n \theader_len = format_object_header((char *)obuf, sizeof(obuf),\n-\t\t\t\t\t OBJ_BLOB, stream->size);\n+\t\t\t\t\t stream->type, stream->size);\n \tgit_hash_init(&ctx, transaction->base.source->odb->repo->hash_algo);\n \tgit_hash_update(&ctx, obuf, header_len);\n \n@@ -911,7 +911,7 @@ static int odb_transaction_files_write_object_stream(struct odb_transaction *bas\n \thashfile_checkpoint(state->f, &checkpoint);\n \tidx->offset = state->offset;\n \tcrc32_begin(state->f);\n-\tstream_blob_to_pack(state, &ctx, stream);\n+\tstream_to_pack(state, &ctx, stream);\n \tgit_hash_final_oid(result_oid, &ctx);\n \n \tidx->crc32 = crc32_end(state->f);\n@@ -953,7 +953,7 @@ int index_fd(struct index_state *istate, struct object_id *oid,\n \t\t\t\t type, path, flags);\n \t} else {\n \t\tstruct odb_write_stream stream;\n-\t\todb_write_stream_from_fd(&stream, fd, xsize_t(st->st_size));\n+\t\todb_write_stream_from_fd(&stream, fd, xsize_t(st->st_size), OBJ_BLOB);\n \n \t\tif (flags & INDEX_WRITE_OBJECT) {\n \t\t\tstruct object_database *odb = the_repository->objects;\n@@ -968,8 +968,7 @@ int index_fd(struct index_state *istate, struct object_id *oid,\n \t\t\tif (!inflight)\n \t\t\t\todb_transaction_commit(transaction);\n \t\t} else {\n-\t\t\tret = hash_blob_stream(&stream,\n-\t\t\t\t\t the_repository->hash_algo, oid);\n+\t\t\tret = hash_stream(&stream, the_repository->hash_algo, oid);\n \t\t}\n \n \t\todb_write_stream_release(&stream);\ndiff --git a/odb/source-inmemory.c b/odb/source-inmemory.c\nindex 01bb81c63c..4f76db5496 100644\n--- a/odb/source-inmemory.c\n+++ b/odb/source-inmemory.c\n@@ -293,7 +293,7 @@ static int odb_source_inmemory_write_object_stream(struct odb_source *source,\n \thash_object_file(source->odb->repo->hash_algo, data, total_read, OBJ_BLOB, oid);\n \n \tret = odb_source_inmemory_write_object(source, data, stream->size,\n-\t\t\t\t\t OBJ_BLOB, oid, NULL, NULL, 0);\n+\t\t\t\t\t stream->type, oid, NULL, NULL, 0);\n \tif (ret < 0)\n \t\tgoto out;\n \ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex 361b4e2a2a..5681a38f03 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -868,7 +868,7 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,\n \n \t/* Since oid is not determined, save tmp file to odb path. */\n \tstrbuf_addf(&filename, \"%s/\", loose->base.path);\n-\thdrlen = format_object_header(hdr, sizeof(hdr), OBJ_BLOB, in_stream->size);\n+\thdrlen = format_object_header(hdr, sizeof(hdr), in_stream->type, in_stream->size);\n \n \t/*\n \t * Common steps for write_loose_object and stream_loose_object to\ndiff --git a/odb/streaming.c b/odb/streaming.c\nindex 912e75e682..0918cad426 100644\n--- a/odb/streaming.c\n+++ b/odb/streaming.c\n@@ -324,7 +324,7 @@ static ssize_t read_object_fd(struct odb_write_stream *stream,\n }\n \n void odb_write_stream_from_fd(struct odb_write_stream *stream, int fd,\n-\t\t\t size_t size)\n+\t\t\t size_t size, enum object_type type)\n {\n \tstruct read_object_fd_data *data;\n \n@@ -335,4 +335,5 @@ void odb_write_stream_from_fd(struct odb_write_stream *stream, int fd,\n \tstream->data = data;\n \tstream->read = read_object_fd;\n \tstream->size = size;\n+\tstream->type = type;\n }\ndiff --git a/odb/streaming.h b/odb/streaming.h\nindex 5e8e6e532e..3c8ed55129 100644\n--- a/odb/streaming.h\n+++ b/odb/streaming.h\n@@ -56,6 +56,7 @@ struct odb_write_stream {\n \tssize_t (*read)(struct odb_write_stream *, unsigned char *, size_t);\n \tvoid *data;\n \tsize_t size;\n+\tenum object_type type;\n };\n \n /*\n@@ -92,6 +93,6 @@ int odb_stream_blob_to_fd(struct object_database *odb,\n * Sets up an ODB write stream that reads from an fd.\n */\n void odb_write_stream_from_fd(struct odb_write_stream *stream, int fd,\n-\t\t\t size_t size);\n+\t\t\t size_t size, enum object_type type);\n \n #endif /* STREAMING_H */\ndiff --git a/t/unit-tests/u-odb-inmemory.c b/t/unit-tests/u-odb-inmemory.c\nindex 4437140ed0..1ab07af6d6 100644\n--- a/t/unit-tests/u-odb-inmemory.c\n+++ b/t/unit-tests/u-odb-inmemory.c\n@@ -297,8 +297,11 @@ void test_odb_inmemory__write_object_stream(void)\n \tstruct odb_source_inmemory *source = odb_source_inmemory_new(odb);\n \tconst char data[] = \"foobar\";\n \tstruct membuf_write_stream stream = {\n-\t\t.base.read = membuf_write_stream_read,\n-\t\t.base.size = strlen(data),\n+\t\t.base = {\n+\t\t\t.read = membuf_write_stream_read,\n+\t\t\t.size = strlen(data),\n+\t\t\t.type = OBJ_BLOB,\n+\t\t},\n \t\t.buf = data,\n \t};\n \tstruct object_id written_oid;\n\n-- \n2.55.0.679.g6767b8d81c.dirty\n\n"},{"id":"549531","messageId":"20260804-pks-odb-stream-unification-v1-4-86d70e82345e@pks.im","threadId":"66111","inReplyTo":"20260804-pks-odb-stream-unification-v1-0-86d70e82345e@pks.im","subject":"[PATCH 4/7] odb/streaming: rename `struct odb_read_stream`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-04T07:25:32Z","receivedAt":"2026-08-04T07:25:57Z","isPatch":true,"body":"Rename `struct odb_read_stream` to just `struct odb_stream`. This\nprepares for unification of the two different types of streams, as these\nprovide the same functionality with the preceding refactorings.\n\nSigned-off-by: Patrick Steinhardt \n---\n archive-tar.c | 6 +++---\n archive-zip.c | 10 ++++-----\n builtin/index-pack.c | 6 +++---\n builtin/pack-objects.c | 14 ++++++-------\n object-file.c | 4 ++--\n object-file.h | 2 +-\n object.c | 6 +++---\n odb/source-files.c | 2 +-\n odb/source-inmemory.c | 8 ++++----\n odb/source-loose.c | 8 ++++----\n odb/source-packed.c | 2 +-\n odb/source.h | 6 +++---\n odb/streaming.c | 48 +++++++++++++++++++++----------------------\n odb/streaming.h | 24 +++++++++++-----------\n pack-check.c | 4 ++--\n packfile.c | 8 ++++----\n packfile.h | 4 ++--\n t/unit-tests/u-odb-inmemory.c | 12 +++++------\n 18 files changed, 87 insertions(+), 87 deletions(-)\n\ndiff --git a/archive-tar.c b/archive-tar.c\nindex 0fc70d13a8..df2d7fb8e9 100644\n--- a/archive-tar.c\n+++ b/archive-tar.c\n@@ -129,7 +129,7 @@ static void write_trailer(void)\n */\n static int stream_blocked(struct repository *r, const struct object_id *oid)\n {\n-\tstruct odb_read_stream *st;\n+\tstruct odb_stream *st;\n \tchar buf[BLOCKSIZE];\n \tssize_t readlen;\n \n@@ -137,12 +137,12 @@ static int stream_blocked(struct repository *r, const struct object_id *oid)\n \tif (!st)\n \t\treturn error(_(\"cannot stream blob %s\"), oid_to_hex(oid));\n \tfor (;;) {\n-\t\treadlen = odb_read_stream_read(st, buf, sizeof(buf));\n+\t\treadlen = odb_stream_read(st, buf, sizeof(buf));\n \t\tif (readlen <= 0)\n \t\t\tbreak;\n \t\tdo_write_blocked(buf, readlen);\n \t}\n-\todb_read_stream_close(st);\n+\todb_stream_close(st);\n \tif (!readlen)\n \t\tfinish_record();\n \treturn readlen;\ndiff --git a/archive-zip.c b/archive-zip.c\nindex 97ea8d60d6..8095fd04d5 100644\n--- a/archive-zip.c\n+++ b/archive-zip.c\n@@ -309,7 +309,7 @@ static int write_zip_entry(struct archiver_args *args,\n \tenum zip_method method;\n \tunsigned char *out;\n \tvoid *deflated = NULL;\n-\tstruct odb_read_stream *stream = NULL;\n+\tstruct odb_stream *stream = NULL;\n \tunsigned long flags = 0;\n \tint is_binary = -1;\n \tconst char *path_without_prefix = path + args->baselen;\n@@ -428,7 +428,7 @@ static int write_zip_entry(struct archiver_args *args,\n \t\tssize_t readlen;\n \n \t\tfor (;;) {\n-\t\t\treadlen = odb_read_stream_read(stream, buf, sizeof(buf));\n+\t\t\treadlen = odb_stream_read(stream, buf, sizeof(buf));\n \t\t\tif (readlen <= 0)\n \t\t\t\tbreak;\n \t\t\tcrc = crc32(crc, buf, readlen);\n@@ -438,7 +438,7 @@ static int write_zip_entry(struct archiver_args *args,\n \t\t\t\t\t\t\t buf, readlen);\n \t\t\twrite_or_die(1, buf, readlen);\n \t\t}\n-\t\todb_read_stream_close(stream);\n+\t\todb_stream_close(stream);\n \t\tif (readlen)\n \t\t\treturn readlen;\n \n@@ -461,7 +461,7 @@ static int write_zip_entry(struct archiver_args *args,\n \t\tzstream.avail_out = sizeof(compressed);\n \n \t\tfor (;;) {\n-\t\t\treadlen = odb_read_stream_read(stream, buf, sizeof(buf));\n+\t\t\treadlen = odb_stream_read(stream, buf, sizeof(buf));\n \t\t\tif (readlen <= 0)\n \t\t\t\tbreak;\n \t\t\tcrc = crc32(crc, buf, readlen);\n@@ -485,7 +485,7 @@ static int write_zip_entry(struct archiver_args *args,\n \t\t\t}\n \n \t\t}\n-\t\todb_read_stream_close(stream);\n+\t\todb_stream_close(stream);\n \t\tif (readlen)\n \t\t\treturn readlen;\n \ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex bc86925ad0..7226da3e65 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -763,7 +763,7 @@ static void find_ref_delta_children(const struct object_id *oid,\n \n struct compare_data {\n \tstruct object_entry *entry;\n-\tstruct odb_read_stream *st;\n+\tstruct odb_stream *st;\n \tunsigned char *buf;\n \tunsigned long buf_size;\n };\n@@ -780,7 +780,7 @@ static int compare_objects(const unsigned char *buf, unsigned long size,\n \t}\n \n \twhile (size) {\n-\t\tssize_t len = odb_read_stream_read(data->st, data->buf, size);\n+\t\tssize_t len = odb_stream_read(data->st, data->buf, size);\n \t\tif (len == 0)\n \t\t\tdie(_(\"SHA1 COLLISION FOUND WITH %s !\"),\n \t\t\t oid_to_hex(&data->entry->idx.oid));\n@@ -813,7 +813,7 @@ static int check_collison(struct object_entry *entry)\n \t\tdie(_(\"SHA1 COLLISION FOUND WITH %s !\"),\n \t\t oid_to_hex(&entry->idx.oid));\n \tunpack_data(entry, compare_objects, &data);\n-\todb_read_stream_close(data.st);\n+\todb_stream_close(data.st);\n \tfree(data.buf);\n \treturn 0;\n }\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 1ec5b6f206..683160c6bb 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -411,7 +411,7 @@ static unsigned long do_compress(void **pptr, unsigned long size)\n \treturn stream.total_out;\n }\n \n-static unsigned long write_large_blob_data(struct odb_read_stream *st, struct hashfile *f,\n+static unsigned long write_large_blob_data(struct odb_stream *st, struct hashfile *f,\n \t\t\t\t\t const struct object_id *oid)\n {\n \tgit_zstream stream;\n@@ -425,7 +425,7 @@ static unsigned long write_large_blob_data(struct odb_read_stream *st, struct ha\n \tfor (;;) {\n \t\tssize_t readlen;\n \t\tint zret = Z_OK;\n-\t\treadlen = odb_read_stream_read(st, ibuf, sizeof(ibuf));\n+\t\treadlen = odb_stream_read(st, ibuf, sizeof(ibuf));\n \t\tif (readlen == -1)\n \t\t\tdie(_(\"unable to read %s\"), oid_to_hex(oid));\n \n@@ -521,7 +521,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \tunsigned hdrlen;\n \tenum object_type type;\n \tvoid *buf;\n-\tstruct odb_read_stream *st = NULL;\n+\tstruct odb_stream *st = NULL;\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n \n \tif (!usable_delta) {\n@@ -589,7 +589,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\t\tdheader[--pos] = 128 | (--ofs & 127);\n \t\tif (limit && hdrlen + sizeof(dheader) - pos + datalen + hashsz >= limit) {\n \t\t\tif (st)\n-\t\t\t\todb_read_stream_close(st);\n+\t\t\t\todb_stream_close(st);\n \t\t\tfree(buf);\n \t\t\treturn 0;\n \t\t}\n@@ -603,7 +603,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\t */\n \t\tif (limit && hdrlen + hashsz + datalen + hashsz >= limit) {\n \t\t\tif (st)\n-\t\t\t\todb_read_stream_close(st);\n+\t\t\t\todb_stream_close(st);\n \t\t\tfree(buf);\n \t\t\treturn 0;\n \t\t}\n@@ -613,7 +613,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t} else {\n \t\tif (limit && hdrlen + datalen + hashsz >= limit) {\n \t\t\tif (st)\n-\t\t\t\todb_read_stream_close(st);\n+\t\t\t\todb_stream_close(st);\n \t\t\tfree(buf);\n \t\t\treturn 0;\n \t\t}\n@@ -621,7 +621,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t}\n \tif (st) {\n \t\tdatalen = write_large_blob_data(st, f, &entry->idx.oid);\n-\t\todb_read_stream_close(st);\n+\t\todb_stream_close(st);\n \t} else {\n \t\thashwrite(f, buf, datalen);\n \t\tfree(buf);\ndiff --git a/object-file.c b/object-file.c\nindex 699a6a008c..5f6d584c35 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -122,7 +122,7 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n }\n \n int stream_object_signature(struct repository *r,\n-\t\t\t struct odb_read_stream *st,\n+\t\t\t struct odb_stream *st,\n \t\t\t const struct object_id *oid)\n {\n \tstruct object_id real_oid;\n@@ -138,7 +138,7 @@ int stream_object_signature(struct repository *r,\n \tgit_hash_update(&c, hdr, hdrlen);\n \tfor (;;) {\n \t\tchar buf[1024 * 16];\n-\t\tssize_t readlen = odb_read_stream_read(st, buf, sizeof(buf));\n+\t\tssize_t readlen = odb_stream_read(st, buf, sizeof(buf));\n \t\tif (readlen < 0)\n \t\t\treturn -1;\n \t\tif (!readlen)\ndiff --git a/object-file.h b/object-file.h\nindex 805f2cfa28..f44758c4f8 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -101,7 +101,7 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n * the streaming interface and rehash it to do the same.\n */\n int stream_object_signature(struct repository *r,\n-\t\t\t struct odb_read_stream *stream,\n+\t\t\t struct odb_stream *stream,\n \t\t\t const struct object_id *oid);\n \n enum finalize_object_file_flags {\ndiff --git a/object.c b/object.c\nindex 23b84aa7e2..37e6efee47 100644\n--- a/object.c\n+++ b/object.c\n@@ -345,7 +345,7 @@ struct object *parse_object_with_flags(struct repository *r,\n \tif ((!obj || obj->type == OBJ_NONE || obj->type == OBJ_BLOB) &&\n \t odb_read_object_info(r->objects, oid, NULL) == OBJ_BLOB) {\n \t\tif (!skip_hash) {\n-\t\t\tstruct odb_read_stream *stream = odb_read_stream_open(r->objects, oid, NULL);\n+\t\t\tstruct odb_stream *stream = odb_read_stream_open(r->objects, oid, NULL);\n \n \t\t\tif (!stream) {\n \t\t\t\terror(_(\"unable to open object stream for %s\"), oid_to_hex(oid));\n@@ -354,11 +354,11 @@ struct object *parse_object_with_flags(struct repository *r,\n \n \t\t\tif (stream_object_signature(r, stream, repl) < 0) {\n \t\t\t\terror(_(\"hash mismatch %s\"), oid_to_hex(oid));\n-\t\t\t\todb_read_stream_close(stream);\n+\t\t\t\todb_stream_close(stream);\n \t\t\t\treturn NULL;\n \t\t\t}\n \n-\t\t\todb_read_stream_close(stream);\n+\t\t\todb_stream_close(stream);\n \t\t}\n \t\tparse_blob_buffer(lookup_blob(r, oid));\n \t\treturn lookup_object(r, oid);\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex f51960bd71..f7b8c76549 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -63,7 +63,7 @@ static int odb_source_files_read_object_info(struct odb_source *source,\n \treturn -1;\n }\n \n-static int odb_source_files_read_object_stream(struct odb_read_stream **out,\n+static int odb_source_files_read_object_stream(struct odb_stream **out,\n \t\t\t\t\t struct odb_source *source,\n \t\t\t\t\t const struct object_id *oid)\n {\ndiff --git a/odb/source-inmemory.c b/odb/source-inmemory.c\nindex 4f76db5496..4bee3ae699 100644\n--- a/odb/source-inmemory.c\n+++ b/odb/source-inmemory.c\n@@ -73,12 +73,12 @@ static int odb_source_inmemory_read_object_info(struct odb_source *source,\n }\n \n struct odb_read_stream_inmemory {\n-\tstruct odb_read_stream base;\n+\tstruct odb_stream base;\n \tconst unsigned char *buf;\n \tsize_t offset;\n };\n \n-static ssize_t odb_read_stream_inmemory_read(struct odb_read_stream *stream,\n+static ssize_t odb_read_stream_inmemory_read(struct odb_stream *stream,\n \t\t\t\t\t char *buf, size_t buf_len)\n {\n \tstruct odb_read_stream_inmemory *inmemory =\n@@ -94,12 +94,12 @@ static ssize_t odb_read_stream_inmemory_read(struct odb_read_stream *stream,\n \treturn bytes;\n }\n \n-static int odb_read_stream_inmemory_close(struct odb_read_stream *stream UNUSED)\n+static int odb_read_stream_inmemory_close(struct odb_stream *stream UNUSED)\n {\n \treturn 0;\n }\n \n-static int odb_source_inmemory_read_object_stream(struct odb_read_stream **out,\n+static int odb_source_inmemory_read_object_stream(struct odb_stream **out,\n \t\t\t\t\t\t struct odb_source *source,\n \t\t\t\t\t\t const struct object_id *oid)\n {\ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex 5681a38f03..038defd905 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -278,7 +278,7 @@ static void *odb_source_loose_map_object(struct odb_source_loose *loose,\n }\n \n struct odb_loose_read_stream {\n-\tstruct odb_read_stream base;\n+\tstruct odb_stream base;\n \tgit_zstream z;\n \tenum {\n \t\tODB_LOOSE_READ_STREAM_INUSE,\n@@ -292,7 +292,7 @@ struct odb_loose_read_stream {\n \tint hdr_used;\n };\n \n-static ssize_t read_istream_loose(struct odb_read_stream *_st, char *buf, size_t sz)\n+static ssize_t read_istream_loose(struct odb_stream *_st, char *buf, size_t sz)\n {\n \tstruct odb_loose_read_stream *st =\n \t\tcontainer_of(_st, struct odb_loose_read_stream, base);\n@@ -339,7 +339,7 @@ static ssize_t read_istream_loose(struct odb_read_stream *_st, char *buf, size_t\n \treturn total_read;\n }\n \n-static int close_istream_loose(struct odb_read_stream *_st)\n+static int close_istream_loose(struct odb_stream *_st)\n {\n \tstruct odb_loose_read_stream *st =\n \t\tcontainer_of(_st, struct odb_loose_read_stream, base);\n@@ -350,7 +350,7 @@ static int close_istream_loose(struct odb_read_stream *_st)\n \treturn 0;\n }\n \n-static int odb_source_loose_read_object_stream(struct odb_read_stream **out,\n+static int odb_source_loose_read_object_stream(struct odb_stream **out,\n \t\t\t\t\t struct odb_source *source,\n \t\t\t\t\t const struct object_id *oid)\n {\ndiff --git a/odb/source-packed.c b/odb/source-packed.c\nindex e6ff74833b..b3186ca593 100644\n--- a/odb/source-packed.c\n+++ b/odb/source-packed.c\n@@ -70,7 +70,7 @@ static int odb_source_packed_read_object_info(struct odb_source *source,\n \treturn 0;\n }\n \n-static int odb_source_packed_read_object_stream(struct odb_read_stream **out,\n+static int odb_source_packed_read_object_stream(struct odb_stream **out,\n \t\t\t\t\t\tstruct odb_source *source,\n \t\t\t\t\t\tconst struct object_id *oid)\n {\ndiff --git a/odb/source.h b/odb/source.h\nindex 0080148ba7..89b0c39682 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -26,7 +26,7 @@ enum odb_source_type {\n };\n \n struct object_id;\n-struct odb_read_stream;\n+struct odb_stream;\n struct strvec;\n \n /*\n@@ -125,7 +125,7 @@ struct odb_source {\n \t * The callback is expected to return a negative error code in case\n \t * creating the object stream has failed, 0 otherwise.\n \t */\n-\tint (*read_object_stream)(struct odb_read_stream **out,\n+\tint (*read_object_stream)(struct odb_stream **out,\n \t\t\t\t struct odb_source *source,\n \t\t\t\t const struct object_id *oid);\n \n@@ -339,7 +339,7 @@ static inline int odb_source_read_object_info(struct odb_source *source,\n * Create a new read stream for the given object ID. Returns 0 on success, a\n * negative error code otherwise.\n */\n-static inline int odb_source_read_object_stream(struct odb_read_stream **out,\n+static inline int odb_source_read_object_stream(struct odb_stream **out,\n \t\t\t\t\t\tstruct odb_source *source,\n \t\t\t\t\t\tconst struct object_id *oid)\n {\ndiff --git a/odb/streaming.c b/odb/streaming.c\nindex 0918cad426..98e2152e36 100644\n--- a/odb/streaming.c\n+++ b/odb/streaming.c\n@@ -20,8 +20,8 @@\n *****************************************************************/\n \n struct odb_filtered_read_stream {\n-\tstruct odb_read_stream base;\n-\tstruct odb_read_stream *upstream;\n+\tstruct odb_stream base;\n+\tstruct odb_stream *upstream;\n \tstruct stream_filter *filter;\n \tchar ibuf[FILTER_BUFFER];\n \tchar obuf[FILTER_BUFFER];\n@@ -30,14 +30,14 @@ struct odb_filtered_read_stream {\n \tint input_finished;\n };\n \n-static int close_istream_filtered(struct odb_read_stream *_fs)\n+static int close_istream_filtered(struct odb_stream *_fs)\n {\n \tstruct odb_filtered_read_stream *fs = (struct odb_filtered_read_stream *)_fs;\n \tfree_stream_filter(fs->filter);\n-\treturn odb_read_stream_close(fs->upstream);\n+\treturn odb_stream_close(fs->upstream);\n }\n \n-static ssize_t read_istream_filtered(struct odb_read_stream *_fs, char *buf,\n+static ssize_t read_istream_filtered(struct odb_stream *_fs, char *buf,\n \t\t\t\t size_t sz)\n {\n \tstruct odb_filtered_read_stream *fs = (struct odb_filtered_read_stream *)_fs;\n@@ -86,7 +86,7 @@ static ssize_t read_istream_filtered(struct odb_read_stream *_fs, char *buf,\n \n \t\t/* refill the input from the upstream */\n \t\tif (!fs->input_finished) {\n-\t\t\tfs->i_end = odb_read_stream_read(fs->upstream, fs->ibuf, FILTER_BUFFER);\n+\t\t\tfs->i_end = odb_stream_read(fs->upstream, fs->ibuf, FILTER_BUFFER);\n \t\t\tif (fs->i_end < 0)\n \t\t\t\treturn -1;\n \t\t\tif (fs->i_end)\n@@ -97,8 +97,8 @@ static ssize_t read_istream_filtered(struct odb_read_stream *_fs, char *buf,\n \treturn filled;\n }\n \n-static struct odb_read_stream *attach_stream_filter(struct odb_read_stream *st,\n-\t\t\t\t\t\t struct stream_filter *filter)\n+static struct odb_stream *attach_stream_filter(struct odb_stream *st,\n+\t\t\t\t\t struct stream_filter *filter)\n {\n \tstruct odb_filtered_read_stream *fs;\n \n@@ -120,19 +120,19 @@ static struct odb_read_stream *attach_stream_filter(struct odb_read_stream *st,\n *****************************************************************/\n \n struct odb_incore_read_stream {\n-\tstruct odb_read_stream base;\n+\tstruct odb_stream base;\n \tchar *buf; /* from odb_read_object_info_extended() */\n \tunsigned long read_ptr;\n };\n \n-static int close_istream_incore(struct odb_read_stream *_st)\n+static int close_istream_incore(struct odb_stream *_st)\n {\n \tstruct odb_incore_read_stream *st = (struct odb_incore_read_stream *)_st;\n \tfree(st->buf);\n \treturn 0;\n }\n \n-static ssize_t read_istream_incore(struct odb_read_stream *_st, char *buf, size_t sz)\n+static ssize_t read_istream_incore(struct odb_stream *_st, char *buf, size_t sz)\n {\n \tstruct odb_incore_read_stream *st = (struct odb_incore_read_stream *)_st;\n \tsize_t read_size = sz;\n@@ -147,7 +147,7 @@ static ssize_t read_istream_incore(struct odb_read_stream *_st, char *buf, size_\n \treturn read_size;\n }\n \n-static int open_istream_incore(struct odb_read_stream **out,\n+static int open_istream_incore(struct odb_stream **out,\n \t\t\t struct object_database *odb,\n \t\t\t const struct object_id *oid)\n {\n@@ -178,7 +178,7 @@ static int open_istream_incore(struct odb_read_stream **out,\n * static helpers variables and functions for users of streaming interface\n *****************************************************************************/\n \n-static int istream_source(struct odb_read_stream **out,\n+static int istream_source(struct odb_stream **out,\n \t\t\t struct object_database *odb,\n \t\t\t const struct object_id *oid)\n {\n@@ -196,23 +196,23 @@ static int istream_source(struct odb_read_stream **out,\n * Users of streaming interface\n ****************************************************************/\n \n-int odb_read_stream_close(struct odb_read_stream *st)\n+int odb_stream_close(struct odb_stream *st)\n {\n \tint r = st->close(st);\n \tfree(st);\n \treturn r;\n }\n \n-ssize_t odb_read_stream_read(struct odb_read_stream *st, void *buf, size_t sz)\n+ssize_t odb_stream_read(struct odb_stream *st, void *buf, size_t sz)\n {\n \treturn st->read(st, buf, sz);\n }\n \n-struct odb_read_stream *odb_read_stream_open(struct object_database *odb,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t struct stream_filter *filter)\n+struct odb_stream *odb_read_stream_open(struct object_database *odb,\n+\t\t\t\t\tconst struct object_id *oid,\n+\t\t\t\t\tstruct stream_filter *filter)\n {\n-\tstruct odb_read_stream *st;\n+\tstruct odb_stream *st;\n \tconst struct object_id *real = lookup_replace_object(odb->repo, oid);\n \tint ret = istream_source(&st, odb, real);\n \n@@ -221,9 +221,9 @@ struct odb_read_stream *odb_read_stream_open(struct object_database *odb,\n \n \tif (filter) {\n \t\t/* Add \"&& !is_null_stream_filter(filter)\" for performance */\n-\t\tstruct odb_read_stream *nst = attach_stream_filter(st, filter);\n+\t\tstruct odb_stream *nst = attach_stream_filter(st, filter);\n \t\tif (!nst) {\n-\t\t\todb_read_stream_close(st);\n+\t\t\todb_stream_close(st);\n \t\t\treturn NULL;\n \t\t}\n \t\tst = nst;\n@@ -248,7 +248,7 @@ int odb_stream_blob_to_fd(struct object_database *odb,\n \t\t\t struct stream_filter *filter,\n \t\t\t int can_seek)\n {\n-\tstruct odb_read_stream *st;\n+\tstruct odb_stream *st;\n \tssize_t kept = 0;\n \tint result = -1;\n \n@@ -263,7 +263,7 @@ int odb_stream_blob_to_fd(struct object_database *odb,\n \tfor (;;) {\n \t\tchar buf[1024 * 16];\n \t\tssize_t wrote, holeto;\n-\t\tssize_t readlen = odb_read_stream_read(st, buf, sizeof(buf));\n+\t\tssize_t readlen = odb_stream_read(st, buf, sizeof(buf));\n \n \t\tif (readlen < 0)\n \t\t\tgoto close_and_exit;\n@@ -294,7 +294,7 @@ int odb_stream_blob_to_fd(struct object_database *odb,\n \tresult = 0;\n \n close_and_exit:\n-\todb_read_stream_close(st);\n+\todb_stream_close(st);\n \treturn result;\n }\n \ndiff --git a/odb/streaming.h b/odb/streaming.h\nindex 3c8ed55129..037954c231 100644\n--- a/odb/streaming.h\n+++ b/odb/streaming.h\n@@ -8,19 +8,19 @@\n #include \"odb.h\"\n \n struct object_database;\n-struct odb_read_stream;\n+struct odb_stream;\n struct stream_filter;\n \n-typedef int (*odb_read_stream_close_fn)(struct odb_read_stream *);\n-typedef ssize_t (*odb_read_stream_read_fn)(struct odb_read_stream *, char *, size_t);\n+typedef int (*odb_stream_close_fn)(struct odb_stream *);\n+typedef ssize_t (*odb_stream_read_fn)(struct odb_stream *, char *, size_t);\n \n /*\n * A stream that can be used to read an object from the object database without\n * loading all of it into memory.\n */\n-struct odb_read_stream {\n-\todb_read_stream_close_fn close;\n-\todb_read_stream_read_fn read;\n+struct odb_stream {\n+\todb_stream_close_fn close;\n+\todb_stream_read_fn read;\n \tenum object_type type;\n \tsize_t size; /* inflated size of full object */\n };\n@@ -31,22 +31,22 @@ struct odb_read_stream {\n *\n * Returns the stream on success, a `NULL` pointer otherwise.\n */\n-struct odb_read_stream *odb_read_stream_open(struct object_database *odb,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t struct stream_filter *filter);\n+struct odb_stream *odb_read_stream_open(struct object_database *odb,\n+\t\t\t\t\tconst struct object_id *oid,\n+\t\t\t\t\tstruct stream_filter *filter);\n \n /*\n- * Close the given read stream and release all resources associated with it.\n+ * Close the given object stream and release all resources associated with it.\n * Returns 0 on success, a negative error code otherwise.\n */\n-int odb_read_stream_close(struct odb_read_stream *stream);\n+int odb_stream_close(struct odb_stream *stream);\n \n /*\n * Read data from the stream into the buffer. Returns 0 on EOF and the number\n * of bytes read on success. Returns a negative error code in case reading from\n * the stream fails.\n */\n-ssize_t odb_read_stream_read(struct odb_read_stream *stream, void *buf, size_t len);\n+ssize_t odb_stream_read(struct odb_stream *stream, void *buf, size_t len);\n \n /*\n * A stream that provides an object to be written to the object database without\ndiff --git a/pack-check.c b/pack-check.c\nindex c3b8db7c5c..1b5e26847d 100644\n--- a/pack-check.c\n+++ b/pack-check.c\n@@ -106,7 +106,7 @@ static int verify_packfile(struct repository *r,\n \tQSORT(entries, nr_objects, compare_entries);\n \n \tfor (i = 0; i < nr_objects; i++) {\n-\t\tstruct odb_read_stream *stream = NULL;\n+\t\tstruct odb_stream *stream = NULL;\n \t\tvoid *data;\n \t\tstruct object_id oid;\n \t\tenum object_type type;\n@@ -171,7 +171,7 @@ static int verify_packfile(struct repository *r,\n \t\t\tdisplay_progress(progress, base_count + i);\n \n \t\tif (stream)\n-\t\t\todb_read_stream_close(stream);\n+\t\t\todb_stream_close(stream);\n \t\tfree(data);\n \t}\n \ndiff --git a/packfile.c b/packfile.c\nindex 0eee45055f..70254573a3 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2115,7 +2115,7 @@ int parse_pack_header_option(const char *in, unsigned char *out, unsigned int *l\n }\n \n struct odb_packed_read_stream {\n-\tstruct odb_read_stream base;\n+\tstruct odb_stream base;\n \tstruct packed_git *pack;\n \tgit_zstream z;\n \tenum {\n@@ -2127,7 +2127,7 @@ struct odb_packed_read_stream {\n \toff_t pos;\n };\n \n-static ssize_t read_istream_pack_non_delta(struct odb_read_stream *_st, char *buf,\n+static ssize_t read_istream_pack_non_delta(struct odb_stream *_st, char *buf,\n \t\t\t\t\t size_t sz)\n {\n \tstruct odb_packed_read_stream *st = (struct odb_packed_read_stream *)_st;\n@@ -2187,7 +2187,7 @@ static ssize_t read_istream_pack_non_delta(struct odb_read_stream *_st, char *bu\n \treturn total_read;\n }\n \n-static int close_istream_pack_non_delta(struct odb_read_stream *_st)\n+static int close_istream_pack_non_delta(struct odb_stream *_st)\n {\n \tstruct odb_packed_read_stream *st = (struct odb_packed_read_stream *)_st;\n \tif (st->z_state == ODB_PACKED_READ_STREAM_INUSE)\n@@ -2195,7 +2195,7 @@ static int close_istream_pack_non_delta(struct odb_read_stream *_st)\n \treturn 0;\n }\n \n-int packfile_read_object_stream(struct odb_read_stream **out,\n+int packfile_read_object_stream(struct odb_stream **out,\n \t\t\t\tconst struct object_id *oid,\n \t\t\t\tstruct packed_git *pack,\n \t\t\t\toff_t offset)\ndiff --git a/packfile.h b/packfile.h\nindex e1f77152b5..f913cb3d0c 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -12,7 +12,7 @@\n \n /* in odb.h */\n struct object_info;\n-struct odb_read_stream;\n+struct odb_stream;\n \n struct packed_git {\n \tstruct pack_window *windows;\n@@ -306,7 +306,7 @@ off_t get_delta_base(struct packed_git *p, struct pack_window **w_curs,\n \t\t off_t *curpos, enum object_type type,\n \t\t off_t delta_obj_offset);\n \n-int packfile_read_object_stream(struct odb_read_stream **out,\n+int packfile_read_object_stream(struct odb_stream **out,\n \t\t\t\tconst struct object_id *oid,\n \t\t\t\tstruct packed_git *pack,\n \t\t\t\toff_t offset);\ndiff --git a/t/unit-tests/u-odb-inmemory.c b/t/unit-tests/u-odb-inmemory.c\nindex 1ab07af6d6..839a0fd3b7 100644\n--- a/t/unit-tests/u-odb-inmemory.c\n+++ b/t/unit-tests/u-odb-inmemory.c\n@@ -100,7 +100,7 @@ void test_odb_inmemory__read_written_object(void)\n void test_odb_inmemory__read_stream_object(void)\n {\n \tstruct odb_source_inmemory *source = odb_source_inmemory_new(odb);\n-\tstruct odb_read_stream *stream;\n+\tstruct odb_stream *stream;\n \tstruct object_id written_oid;\n \tconst char data[] = \"foobar\";\n \tchar buf[3] = { 0 };\n@@ -112,15 +112,15 @@ void test_odb_inmemory__read_stream_object(void)\n \tcl_assert_equal_i(stream->type, OBJ_BLOB);\n \tcl_assert_equal_u(stream->size, 6);\n \n-\tcl_assert_equal_i(odb_read_stream_read(stream, buf, 2), 2);\n+\tcl_assert_equal_i(odb_stream_read(stream, buf, 2), 2);\n \tcl_assert_equal_s(buf, \"fo\");\n-\tcl_assert_equal_i(odb_read_stream_read(stream, buf, 2), 2);\n+\tcl_assert_equal_i(odb_stream_read(stream, buf, 2), 2);\n \tcl_assert_equal_s(buf, \"ob\");\n-\tcl_assert_equal_i(odb_read_stream_read(stream, buf, 2), 2);\n+\tcl_assert_equal_i(odb_stream_read(stream, buf, 2), 2);\n \tcl_assert_equal_s(buf, \"ar\");\n-\tcl_assert_equal_i(odb_read_stream_read(stream, buf, 2), 0);\n+\tcl_assert_equal_i(odb_stream_read(stream, buf, 2), 0);\n \n-\todb_read_stream_close(stream);\n+\todb_stream_close(stream);\n \todb_source_free(&source->base);\n }\n \n\n-- \n2.55.0.679.g6767b8d81c.dirty\n\n"},{"id":"549532","messageId":"20260804-pks-odb-stream-unification-v1-5-86d70e82345e@pks.im","threadId":"66111","inReplyTo":"20260804-pks-odb-stream-unification-v1-0-86d70e82345e@pks.im","subject":"[PATCH 5/7] odb/streaming: consolidate read and write streams","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-04T07:25:33Z","receivedAt":"2026-08-04T07:26:00Z","isPatch":true,"body":"The `struct odb_read_stream` and `struct odb_write_stream` both provide\nthe same functionality: they allow a caller to read object data from an\narbitrary source. Historically, the only difference was that the read\nstream was used to read data out of the object database, whereas the\nwrite stream was used to write data into the object database, but the\ninterfaces were mostly the same.\n\nOver the preceding commits we have refactored the write stream to have\nalmost exactly the same interface as the read stream. With these\nrefactorings we can now easily merge those two streams into a single\ninterface that's used for both use cases.\n\nWhile most of the changes are mechanical, there are two sites that need\nspecial mention:\n\n - \"builtin/unpack-objects.c\" creates a write stream from compressed\n object data.\n\n - \"odb/streaming.c\" creates a write stream from a file descriptor.\n\nAdapting these sites to yield the new stream type requires a couple more\nchanges. Most importantly, instead of embedding the pointer to the data\nin `struct odb_write_stream`, we now allocate a structure that wraps the\nnew `struct odb_stream` base. Other than that though, the changes are\nrather straight forward.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/unpack-objects.c | 31 ++++++++++++++++---------------\n object-file.c | 25 ++++++++++++-------------\n odb.c | 2 +-\n odb.h | 4 ++--\n odb/source-files.c | 2 +-\n odb/source-inmemory.c | 4 ++--\n odb/source-loose.c | 6 +++---\n odb/source-packed.c | 2 +-\n odb/source.h | 4 ++--\n odb/streaming.c | 35 ++++++++++++++++-------------------\n odb/streaming.h | 31 +++----------------------------\n odb/transaction.c | 2 +-\n odb/transaction.h | 4 ++--\n t/unit-tests/u-odb-inmemory.c | 6 +++---\n 14 files changed, 65 insertions(+), 93 deletions(-)\n\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex 7439ec53be..05a2d48011 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -359,20 +359,21 @@ static void unpack_non_delta_entry(enum object_type type, unsigned long size,\n }\n \n struct input_zstream_data {\n+\tstruct odb_stream base;\n \tgit_zstream *zstream;\n \tint status;\n };\n \n-static ssize_t feed_input_zstream(struct odb_write_stream *in_stream,\n-\t\t\t\t unsigned char *buf, size_t buf_len)\n+static ssize_t feed_input_zstream(struct odb_stream *in_stream,\n+\t\t\t\t char *buf, size_t buf_len)\n {\n-\tstruct input_zstream_data *data = in_stream->data;\n+\tstruct input_zstream_data *data = container_of(in_stream, struct input_zstream_data, base);\n \tgit_zstream *zstream = data->zstream;\n \n \tif (data->status != Z_OK)\n \t\treturn 0;\n \n-\tzstream->next_out = buf;\n+\tzstream->next_out = (unsigned char *) buf;\n \tzstream->avail_out = buf_len;\n \n \twhile (data->status == Z_OK && zstream->avail_out == buf_len) {\n@@ -388,24 +389,24 @@ static ssize_t feed_input_zstream(struct odb_write_stream *in_stream,\n static void stream_blob(unsigned long size, unsigned nr)\n {\n \tgit_zstream zstream = { 0 };\n-\tstruct input_zstream_data data = { 0 };\n-\tstruct odb_write_stream in_stream = {\n-\t\t.read = feed_input_zstream,\n-\t\t.data = &data,\n-\t\t.size = size,\n-\t\t.type = OBJ_BLOB,\n+\tstruct input_zstream_data in_stream = {\n+\t\t.base = {\n+\t\t\t.read = feed_input_zstream,\n+\t\t\t.size = size,\n+\t\t\t.type = OBJ_BLOB,\n+\t\t},\n+\t\t.zstream = &zstream,\n+\t\t.status = Z_OK,\n \t};\n \tstruct obj_info *info = &obj_list[nr];\n \n-\tdata.zstream = &zstream;\n-\tdata.status = Z_OK;\n \tgit_inflate_init(&zstream);\n \n-\tif (odb_write_object_stream(the_repository->objects, &in_stream, &info->oid))\n+\tif (odb_write_object_stream(the_repository->objects, &in_stream.base, &info->oid))\n \t\tdie(_(\"failed to write object in stream\"));\n \n-\tif (data.status != Z_STREAM_END)\n-\t\tdie(_(\"inflate returned (%d)\"), data.status);\n+\tif (in_stream.status != Z_STREAM_END)\n+\t\tdie(_(\"inflate returned (%d)\"), in_stream.status);\n \tgit_inflate_end(&zstream);\n \n \tif (strict) {\ndiff --git a/object-file.c b/object-file.c\nindex 5f6d584c35..068c6e5672 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -702,7 +702,7 @@ static void prepare_packfile_transaction(struct odb_transaction_files *transacti\n \t\tdie_errno(\"unable to write pack header\");\n }\n \n-static int hash_stream(struct odb_write_stream *stream,\n+static int hash_stream(struct odb_stream *stream,\n \t\t const struct git_hash_algo *hash_algo,\n \t\t struct object_id *result_oid)\n {\n@@ -717,8 +717,8 @@ static int hash_stream(struct odb_write_stream *stream,\n \tgit_hash_update(&ctx, buf, header_len);\n \n \twhile (1) {\n-\t\tssize_t read_result = odb_write_stream_read(stream, buf,\n-\t\t\t\t\t\t\t sizeof(buf));\n+\t\tssize_t read_result = odb_stream_read(stream, buf,\n+\t\t\t\t\t\t sizeof(buf));\n \t\tif (read_result < 0)\n \t\t\treturn -1;\n \t\tif (!read_result)\n@@ -742,7 +742,7 @@ static int hash_stream(struct odb_write_stream *stream,\n */\n static void stream_to_pack(struct transaction_packfile *state,\n \t\t\t struct git_hash_ctx *ctx,\n-\t\t\t struct odb_write_stream *stream)\n+\t\t\t struct odb_stream *stream)\n {\n \tgit_zstream s;\n \tunsigned char ibuf[16384];\n@@ -761,8 +761,8 @@ static void stream_to_pack(struct transaction_packfile *state,\n \n \twhile (status != Z_STREAM_END) {\n \t\tif (!is_finished && !s.avail_in) {\n-\t\t\tssize_t rsize = odb_write_stream_read(stream, ibuf,\n-\t\t\t\t\t\t\t sizeof(ibuf));\n+\t\t\tssize_t rsize = odb_stream_read(stream, ibuf,\n+\t\t\t\t\t\t\tsizeof(ibuf));\n \t\t\tif (rsize < 0)\n \t\t\t\tdie(\"failed to read object data\");\n \t\t\tif (!rsize)\n@@ -872,7 +872,7 @@ static void flush_packfile_transaction(struct odb_transaction_files *transaction\n * callers should avoid this code path when filters are requested.\n */\n static int odb_transaction_files_write_object_stream(struct odb_transaction *base,\n-\t\t\t\t\t\t struct odb_write_stream *stream,\n+\t\t\t\t\t\t struct odb_stream *stream,\n \t\t\t\t\t\t struct object_id *result_oid)\n {\n \tstruct odb_transaction_files *transaction = container_of(base,\n@@ -952,8 +952,8 @@ int index_fd(struct index_state *istate, struct object_id *oid,\n \t\tret = index_core(istate, oid, fd, xsize_t(st->st_size),\n \t\t\t\t type, path, flags);\n \t} else {\n-\t\tstruct odb_write_stream stream;\n-\t\todb_write_stream_from_fd(&stream, fd, xsize_t(st->st_size), OBJ_BLOB);\n+\t\tstruct odb_stream *stream = odb_write_stream_from_fd(fd, xsize_t(st->st_size),\n+\t\t\t\t\t\t\t\t OBJ_BLOB);\n \n \t\tif (flags & INDEX_WRITE_OBJECT) {\n \t\t\tstruct object_database *odb = the_repository->objects;\n@@ -963,15 +963,14 @@ int index_fd(struct index_state *istate, struct object_id *oid,\n \t\t\tif (!inflight)\n \t\t\t\todb_transaction_begin_or_die(odb, &transaction, 0);\n \t\t\tret = odb_transaction_write_object_stream(transaction,\n-\t\t\t\t\t\t\t\t &stream,\n-\t\t\t\t\t\t\t\t oid);\n+\t\t\t\t\t\t\t\t stream, oid);\n \t\t\tif (!inflight)\n \t\t\t\todb_transaction_commit(transaction);\n \t\t} else {\n-\t\t\tret = hash_stream(&stream, the_repository->hash_algo, oid);\n+\t\t\tret = hash_stream(stream, the_repository->hash_algo, oid);\n \t\t}\n \n-\t\todb_write_stream_release(&stream);\n+\t\todb_stream_close(stream);\n \t}\n \n \tclose(fd);\ndiff --git a/odb.c b/odb.c\nindex 585b2b2965..eec4cc5302 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1028,7 +1028,7 @@ int odb_write_object_ext(struct object_database *odb,\n }\n \n int odb_write_object_stream(struct object_database *odb,\n-\t\t\t struct odb_write_stream *stream,\n+\t\t\t struct odb_stream *stream,\n \t\t\t struct object_id *oid)\n {\n \treturn odb_source_write_object_stream(odb->sources, stream, oid);\ndiff --git a/odb.h b/odb.h\nindex 019d3af3e8..fbe75c5a81 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -626,10 +626,10 @@ static inline int odb_write_object(struct object_database *odb,\n \treturn odb_write_object_ext(odb, buf, len, type, oid, NULL, 0);\n }\n \n-struct odb_write_stream;\n+struct odb_stream;\n \n int odb_write_object_stream(struct object_database *odb,\n-\t\t\t struct odb_write_stream *stream,\n+\t\t\t struct odb_stream *stream,\n \t\t\t struct object_id *oid);\n \n void parse_alternates(const char *string,\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex f7b8c76549..6defe5ac4f 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -174,7 +174,7 @@ static int odb_source_files_write_object(struct odb_source *source,\n }\n \n static int odb_source_files_write_object_stream(struct odb_source *source,\n-\t\t\t\t\t\tstruct odb_write_stream *stream,\n+\t\t\t\t\t\tstruct odb_stream *stream,\n \t\t\t\t\t\tstruct object_id *oid)\n {\n \tstruct odb_source_files *files = odb_source_files_downcast(source);\ndiff --git a/odb/source-inmemory.c b/odb/source-inmemory.c\nindex 4bee3ae699..bb63cdce86 100644\n--- a/odb/source-inmemory.c\n+++ b/odb/source-inmemory.c\n@@ -256,7 +256,7 @@ static int odb_source_inmemory_write_object(struct odb_source *source,\n }\n \n static int odb_source_inmemory_write_object_stream(struct odb_source *source,\n-\t\t\t\t\t\t struct odb_write_stream *stream,\n+\t\t\t\t\t\t struct odb_stream *stream,\n \t\t\t\t\t\t struct object_id *oid)\n {\n \tchar buf[16384];\n@@ -268,7 +268,7 @@ static int odb_source_inmemory_write_object_stream(struct odb_source *source,\n \twhile (1) {\n \t\tssize_t bytes_read;\n \n-\t\tbytes_read = odb_write_stream_read(stream, buf, sizeof(buf));\n+\t\tbytes_read = odb_stream_read(stream, buf, sizeof(buf));\n \t\tif (bytes_read < 0) {\n \t\t\tret = error(\"failed to read object stream\");\n \t\t\tgoto out;\ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex 038defd905..ff1bede7fe 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -845,7 +845,7 @@ static int odb_source_loose_write_object(struct odb_source *source,\n }\n \n static int odb_source_loose_write_object_stream(struct odb_source *source,\n-\t\t\t\t\t\tstruct odb_write_stream *in_stream,\n+\t\t\t\t\t\tstruct odb_stream *in_stream,\n \t\t\t\t\t\tstruct object_id *oid)\n {\n \tstruct odb_source_loose *loose = odb_source_loose_downcast(source);\n@@ -891,8 +891,8 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,\n \t\tunsigned char *in0 = stream.next_in;\n \n \t\tif (!stream.avail_in && !is_finished) {\n-\t\t\tssize_t read_len = odb_write_stream_read(in_stream, buf,\n-\t\t\t\t\t\t\t\t sizeof(buf));\n+\t\t\tssize_t read_len = odb_stream_read(in_stream, buf,\n+\t\t\t\t\t\t\t sizeof(buf));\n \t\t\tif (read_len < 0) {\n \t\t\t\tclose(fd);\n \t\t\t\terr = -1;\ndiff --git a/odb/source-packed.c b/odb/source-packed.c\nindex b3186ca593..630d955585 100644\n--- a/odb/source-packed.c\n+++ b/odb/source-packed.c\n@@ -609,7 +609,7 @@ static int odb_source_packed_write_object(struct odb_source *source UNUSED,\n }\n \n static int odb_source_packed_write_object_stream(struct odb_source *source UNUSED,\n-\t\t\t\t\t\t struct odb_write_stream *stream UNUSED,\n+\t\t\t\t\t\t struct odb_stream *stream UNUSED,\n \t\t\t\t\t\t struct object_id *oid UNUSED)\n {\n \treturn error(\"packed backend cannot write object streams\");\ndiff --git a/odb/source.h b/odb/source.h\nindex 89b0c39682..0b99c698b5 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -221,7 +221,7 @@ struct odb_source {\n \t * otherwise.\n \t */\n \tint (*write_object_stream)(struct odb_source *source,\n-\t\t\t\t struct odb_write_stream *stream,\n+\t\t\t\t struct odb_stream *stream,\n \t\t\t\t struct object_id *oid);\n \n \t/*\n@@ -436,7 +436,7 @@ static inline int odb_source_write_object(struct odb_source *source,\n * out pointer for the object ID.\n */\n static inline int odb_source_write_object_stream(struct odb_source *source,\n-\t\t\t\t\t\t struct odb_write_stream *stream,\n+\t\t\t\t\t\t struct odb_stream *stream,\n \t\t\t\t\t\t struct object_id *oid)\n {\n \treturn source->write_object_stream(source, stream, oid);\ndiff --git a/odb/streaming.c b/odb/streaming.c\nindex 98e2152e36..1a267e6b90 100644\n--- a/odb/streaming.c\n+++ b/odb/streaming.c\n@@ -232,16 +232,6 @@ struct odb_stream *odb_read_stream_open(struct object_database *odb,\n \treturn st;\n }\n \n-ssize_t odb_write_stream_read(struct odb_write_stream *st, void *buf, size_t sz)\n-{\n-\treturn st->read(st, buf, sz);\n-}\n-\n-void odb_write_stream_release(struct odb_write_stream *st)\n-{\n-\tfree(st->data);\n-}\n-\n int odb_stream_blob_to_fd(struct object_database *odb,\n \t\t\t int fd,\n \t\t\t const struct object_id *oid,\n@@ -299,14 +289,15 @@ int odb_stream_blob_to_fd(struct object_database *odb,\n }\n \n struct read_object_fd_data {\n+\tstruct odb_stream base;\n \tint fd;\n \tsize_t remaining;\n };\n \n-static ssize_t read_object_fd(struct odb_write_stream *stream,\n-\t\t\t unsigned char *buf, size_t len)\n+static ssize_t read_object_fd(struct odb_stream *stream,\n+\t\t\t char *buf, size_t len)\n {\n-\tstruct read_object_fd_data *data = stream->data;\n+\tstruct read_object_fd_data *data = container_of(stream, struct read_object_fd_data, base);\n \tssize_t read_result;\n \tsize_t count;\n \n@@ -323,17 +314,23 @@ static ssize_t read_object_fd(struct odb_write_stream *stream,\n \treturn read_result;\n }\n \n-void odb_write_stream_from_fd(struct odb_write_stream *stream, int fd,\n-\t\t\t size_t size, enum object_type type)\n+static int close_object_fd(struct odb_stream *stream UNUSED)\n+{\n+\t/* The file descriptor is owned by the caller for now. */\n+\treturn 0;\n+}\n+\n+struct odb_stream *odb_write_stream_from_fd(int fd, size_t size, enum object_type type)\n {\n \tstruct read_object_fd_data *data;\n \n \tCALLOC_ARRAY(data, 1);\n+\tdata->base.read = read_object_fd;\n+\tdata->base.close = close_object_fd;\n+\tdata->base.size = size;\n+\tdata->base.type = type;\n \tdata->fd = fd;\n \tdata->remaining = size;\n \n-\tstream->data = data;\n-\tstream->read = read_object_fd;\n-\tstream->size = size;\n-\tstream->type = type;\n+\treturn &data->base;\n }\ndiff --git a/odb/streaming.h b/odb/streaming.h\nindex 037954c231..60b9803190 100644\n--- a/odb/streaming.h\n+++ b/odb/streaming.h\n@@ -15,8 +15,8 @@ typedef int (*odb_stream_close_fn)(struct odb_stream *);\n typedef ssize_t (*odb_stream_read_fn)(struct odb_stream *, char *, size_t);\n \n /*\n- * A stream that can be used to read an object from the object database without\n- * loading all of it into memory.\n+ * A stream that can be used to read an object from or write an object into the\n+ * object database without loading all of it into memory.\n */\n struct odb_stream {\n \todb_stream_close_fn close;\n@@ -48,30 +48,6 @@ int odb_stream_close(struct odb_stream *stream);\n */\n ssize_t odb_stream_read(struct odb_stream *stream, void *buf, size_t len);\n \n-/*\n- * A stream that provides an object to be written to the object database without\n- * loading all of it into memory.\n- */\n-struct odb_write_stream {\n-\tssize_t (*read)(struct odb_write_stream *, unsigned char *, size_t);\n-\tvoid *data;\n-\tsize_t size;\n-\tenum object_type type;\n-};\n-\n-/*\n- * Read data from the stream into the buffer. Returns 0 when finished and the\n- * number of bytes read on success. Returns a negative error code in case\n- * reading from the stream fails.\n- */\n-ssize_t odb_write_stream_read(struct odb_write_stream *stream, void *buf,\n-\t\t\t size_t len);\n-\n-/*\n- * Releases memory allocated for underlying stream data.\n- */\n-void odb_write_stream_release(struct odb_write_stream *stream);\n-\n /*\n * Look up the object by its ID and write the full contents to the file\n * descriptor. The object must be a blob, or the function will fail. When\n@@ -92,7 +68,6 @@ int odb_stream_blob_to_fd(struct object_database *odb,\n /*\n * Sets up an ODB write stream that reads from an fd.\n */\n-void odb_write_stream_from_fd(struct odb_write_stream *stream, int fd,\n-\t\t\t size_t size, enum object_type type);\n+struct odb_stream *odb_write_stream_from_fd(int fd, size_t size, enum object_type type);\n \n #endif /* STREAMING_H */\ndiff --git a/odb/transaction.c b/odb/transaction.c\nindex 6aaf133812..69d71b9e97 100644\n--- a/odb/transaction.c\n+++ b/odb/transaction.c\n@@ -39,7 +39,7 @@ int odb_transaction_commit(struct odb_transaction *transaction)\n }\n \n int odb_transaction_write_object_stream(struct odb_transaction *transaction,\n-\t\t\t\t\tstruct odb_write_stream *stream,\n+\t\t\t\t\tstruct odb_stream *stream,\n \t\t\t\t\tstruct object_id *oid)\n {\n \treturn transaction->write_object_stream(transaction, stream, oid);\ndiff --git a/odb/transaction.h b/odb/transaction.h\nindex ffb279314c..b83c77c80a 100644\n--- a/odb/transaction.h\n+++ b/odb/transaction.h\n@@ -31,7 +31,7 @@ struct odb_transaction {\n \t * otherwise.\n \t */\n \tint (*write_object_stream)(struct odb_transaction *transaction,\n-\t\t\t\t struct odb_write_stream *stream,\n+\t\t\t\t struct odb_stream *stream,\n \t\t\t\t struct object_id *oid);\n \n \t/*\n@@ -81,7 +81,7 @@ int odb_transaction_commit(struct odb_transaction *transaction);\n * error code otherwise.\n */\n int odb_transaction_write_object_stream(struct odb_transaction *transaction,\n-\t\t\t\t\tstruct odb_write_stream *stream,\n+\t\t\t\t\tstruct odb_stream *stream,\n \t\t\t\t\tstruct object_id *oid);\n \n /*\ndiff --git a/t/unit-tests/u-odb-inmemory.c b/t/unit-tests/u-odb-inmemory.c\nindex 839a0fd3b7..b8b331b37d 100644\n--- a/t/unit-tests/u-odb-inmemory.c\n+++ b/t/unit-tests/u-odb-inmemory.c\n@@ -266,13 +266,13 @@ void test_odb_inmemory__freshen_object(void)\n }\n \n struct membuf_write_stream {\n-\tstruct odb_write_stream base;\n+\tstruct odb_stream base;\n \tconst char *buf;\n \tsize_t offset;\n };\n \n-static ssize_t membuf_write_stream_read(struct odb_write_stream *stream,\n-\t\t\t\t\tunsigned char *buf, size_t len)\n+static ssize_t membuf_write_stream_read(struct odb_stream *stream,\n+\t\t\t\t\tchar *buf, size_t len)\n {\n \tstruct membuf_write_stream *s = container_of(stream, struct membuf_write_stream, base);\n \tsize_t chunk_size = 2;\n\n-- \n2.55.0.679.g6767b8d81c.dirty\n\n"},{"id":"549533","messageId":"20260804-pks-odb-stream-unification-v1-6-86d70e82345e@pks.im","threadId":"66111","inReplyTo":"20260804-pks-odb-stream-unification-v1-0-86d70e82345e@pks.im","subject":"[PATCH 6/7] odb/streaming: rename `struct read_object_fd_data`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-04T07:25:34Z","receivedAt":"2026-08-04T07:26:04Z","isPatch":true,"body":"With the preceding refactorings the `struct read_object_fd_data` is now\nsomewhat misnamed, as it doesn't only contain the data anymore, but also\nthe stream itself. Rename the structure to `struct fd_stream` to better\nmatch the new structure.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb/streaming.c | 34 +++++++++++++++++-----------------\n 1 file changed, 17 insertions(+), 17 deletions(-)\n\ndiff --git a/odb/streaming.c b/odb/streaming.c\nindex 1a267e6b90..c436b18d39 100644\n--- a/odb/streaming.c\n+++ b/odb/streaming.c\n@@ -288,33 +288,33 @@ int odb_stream_blob_to_fd(struct object_database *odb,\n \treturn result;\n }\n \n-struct read_object_fd_data {\n+struct fd_stream {\n \tstruct odb_stream base;\n \tint fd;\n \tsize_t remaining;\n };\n \n-static ssize_t read_object_fd(struct odb_stream *stream,\n+static ssize_t fd_stream_read(struct odb_stream *stream,\n \t\t\t char *buf, size_t len)\n {\n-\tstruct read_object_fd_data *data = container_of(stream, struct read_object_fd_data, base);\n+\tstruct fd_stream *fds = container_of(stream, struct fd_stream, base);\n \tssize_t read_result;\n \tsize_t count;\n \n-\tif (!data->remaining)\n+\tif (!fds->remaining)\n \t\treturn 0;\n \n-\tcount = data->remaining < len ? data->remaining : len;\n-\tread_result = read_in_full(data->fd, buf, count);\n+\tcount = fds->remaining < len ? fds->remaining : len;\n+\tread_result = read_in_full(fds->fd, buf, count);\n \tif (read_result < 0 || (size_t)read_result != count)\n \t\treturn -1;\n \n-\tdata->remaining -= count;\n+\tfds->remaining -= count;\n \n \treturn read_result;\n }\n \n-static int close_object_fd(struct odb_stream *stream UNUSED)\n+static int fd_stream_close(struct odb_stream *stream UNUSED)\n {\n \t/* The file descriptor is owned by the caller for now. */\n \treturn 0;\n@@ -322,15 +322,15 @@ static int close_object_fd(struct odb_stream *stream UNUSED)\n \n struct odb_stream *odb_write_stream_from_fd(int fd, size_t size, enum object_type type)\n {\n-\tstruct read_object_fd_data *data;\n+\tstruct fd_stream *fds;\n \n-\tCALLOC_ARRAY(data, 1);\n-\tdata->base.read = read_object_fd;\n-\tdata->base.close = close_object_fd;\n-\tdata->base.size = size;\n-\tdata->base.type = type;\n-\tdata->fd = fd;\n-\tdata->remaining = size;\n+\tCALLOC_ARRAY(fds, 1);\n+\tfds->base.read = fd_stream_read;\n+\tfds->base.close = fd_stream_close;\n+\tfds->base.size = size;\n+\tfds->base.type = type;\n+\tfds->fd = fd;\n+\tfds->remaining = size;\n \n-\treturn &data->base;\n+\treturn &fds->base;\n }\n\n-- \n2.55.0.679.g6767b8d81c.dirty\n\n"},{"id":"549534","messageId":"20260804-pks-odb-stream-unification-v1-7-86d70e82345e@pks.im","threadId":"66111","inReplyTo":"20260804-pks-odb-stream-unification-v1-0-86d70e82345e@pks.im","subject":"[PATCH 7/7] odb/streaming: unify function names to create new streams","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-04T07:25:35Z","receivedAt":"2026-08-04T07:26:08Z","isPatch":true,"body":"Unify the function names to create new streams from different sources so\nthat they follow a common schema. While at it, document the ownership of\nthe file descriptor passed to `odb_stream_from_fd()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n archive-tar.c | 2 +-\n archive-zip.c | 2 +-\n builtin/index-pack.c | 2 +-\n builtin/pack-objects.c | 4 ++--\n object-file.c | 4 ++--\n object.c | 2 +-\n odb/streaming.c | 10 +++++-----\n odb/streaming.h | 23 +++++++++++++----------\n 8 files changed, 26 insertions(+), 23 deletions(-)\n\ndiff --git a/archive-tar.c b/archive-tar.c\nindex df2d7fb8e9..a1c66024d4 100644\n--- a/archive-tar.c\n+++ b/archive-tar.c\n@@ -133,7 +133,7 @@ static int stream_blocked(struct repository *r, const struct object_id *oid)\n \tchar buf[BLOCKSIZE];\n \tssize_t readlen;\n \n-\tst = odb_read_stream_open(r->objects, oid, NULL);\n+\tst = odb_stream_from_object(r->objects, oid, NULL);\n \tif (!st)\n \t\treturn error(_(\"cannot stream blob %s\"), oid_to_hex(oid));\n \tfor (;;) {\ndiff --git a/archive-zip.c b/archive-zip.c\nindex 8095fd04d5..1a948c2f83 100644\n--- a/archive-zip.c\n+++ b/archive-zip.c\n@@ -347,7 +347,7 @@ static int write_zip_entry(struct archiver_args *args,\n \t\t\tmethod = ZIP_METHOD_DEFLATE;\n \n \t\tif (!buffer) {\n-\t\t\tstream = odb_read_stream_open(args->repo->objects, oid, NULL);\n+\t\t\tstream = odb_stream_from_object(args->repo->objects, oid, NULL);\n \t\t\tif (!stream)\n \t\t\t\treturn error(_(\"cannot stream blob %s\"),\n \t\t\t\t\t oid_to_hex(oid));\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 7226da3e65..d1761282db 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -806,7 +806,7 @@ static int check_collison(struct object_entry *entry)\n \n \tmemset(&data, 0, sizeof(data));\n \tdata.entry = entry;\n-\tdata.st = odb_read_stream_open(the_repository->objects, &entry->idx.oid, NULL);\n+\tdata.st = odb_stream_from_object(the_repository->objects, &entry->idx.oid, NULL);\n \tif (!data.st)\n \t\treturn -1;\n \tif (data.st->size != entry->size || data.st->type != entry->type)\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 683160c6bb..10d00ca792 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -528,8 +528,8 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\tif (oe_type(entry) == OBJ_BLOB &&\n \t\t oe_size_greater_than(&to_pack, entry,\n \t\t\t\t\t repo_settings_get_big_file_threshold(the_repository)) &&\n-\t\t (st = odb_read_stream_open(the_repository->objects, &entry->idx.oid,\n-\t\t\t\t\t NULL)) != NULL) {\n+\t\t (st = odb_stream_from_object(the_repository->objects, &entry->idx.oid,\n+\t\t\t\t\t\t NULL)) != NULL) {\n \t\t\tbuf = NULL;\n \t\t\ttype = st->type;\n \t\t\tsize = st->size;\ndiff --git a/object-file.c b/object-file.c\nindex 068c6e5672..11d1af342e 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -952,8 +952,8 @@ int index_fd(struct index_state *istate, struct object_id *oid,\n \t\tret = index_core(istate, oid, fd, xsize_t(st->st_size),\n \t\t\t\t type, path, flags);\n \t} else {\n-\t\tstruct odb_stream *stream = odb_write_stream_from_fd(fd, xsize_t(st->st_size),\n-\t\t\t\t\t\t\t\t OBJ_BLOB);\n+\t\tstruct odb_stream *stream = odb_stream_from_fd(fd, xsize_t(st->st_size),\n+\t\t\t\t\t\t\t OBJ_BLOB);\n \n \t\tif (flags & INDEX_WRITE_OBJECT) {\n \t\t\tstruct object_database *odb = the_repository->objects;\ndiff --git a/object.c b/object.c\nindex 37e6efee47..97f7fc0e87 100644\n--- a/object.c\n+++ b/object.c\n@@ -345,7 +345,7 @@ struct object *parse_object_with_flags(struct repository *r,\n \tif ((!obj || obj->type == OBJ_NONE || obj->type == OBJ_BLOB) &&\n \t odb_read_object_info(r->objects, oid, NULL) == OBJ_BLOB) {\n \t\tif (!skip_hash) {\n-\t\t\tstruct odb_stream *stream = odb_read_stream_open(r->objects, oid, NULL);\n+\t\t\tstruct odb_stream *stream = odb_stream_from_object(r->objects, oid, NULL);\n \n \t\t\tif (!stream) {\n \t\t\t\terror(_(\"unable to open object stream for %s\"), oid_to_hex(oid));\ndiff --git a/odb/streaming.c b/odb/streaming.c\nindex c436b18d39..9c85ec54f5 100644\n--- a/odb/streaming.c\n+++ b/odb/streaming.c\n@@ -208,9 +208,9 @@ ssize_t odb_stream_read(struct odb_stream *st, void *buf, size_t sz)\n \treturn st->read(st, buf, sz);\n }\n \n-struct odb_stream *odb_read_stream_open(struct object_database *odb,\n-\t\t\t\t\tconst struct object_id *oid,\n-\t\t\t\t\tstruct stream_filter *filter)\n+struct odb_stream *odb_stream_from_object(struct object_database *odb,\n+\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t struct stream_filter *filter)\n {\n \tstruct odb_stream *st;\n \tconst struct object_id *real = lookup_replace_object(odb->repo, oid);\n@@ -242,7 +242,7 @@ int odb_stream_blob_to_fd(struct object_database *odb,\n \tssize_t kept = 0;\n \tint result = -1;\n \n-\tst = odb_read_stream_open(odb, oid, filter);\n+\tst = odb_stream_from_object(odb, oid, filter);\n \tif (!st) {\n \t\tif (filter)\n \t\t\tfree_stream_filter(filter);\n@@ -320,7 +320,7 @@ static int fd_stream_close(struct odb_stream *stream UNUSED)\n \treturn 0;\n }\n \n-struct odb_stream *odb_write_stream_from_fd(int fd, size_t size, enum object_type type)\n+struct odb_stream *odb_stream_from_fd(int fd, size_t size, enum object_type type)\n {\n \tstruct fd_stream *fds;\n \ndiff --git a/odb/streaming.h b/odb/streaming.h\nindex 60b9803190..b522ff513f 100644\n--- a/odb/streaming.h\n+++ b/odb/streaming.h\n@@ -26,14 +26,22 @@ struct odb_stream {\n };\n \n /*\n- * Create a new object stream for the given object database. An optional filter\n- * can be used to transform the object's content.\n+ * Create a new object stream for the given object. An optional filter can be\n+ * used to transform the object's content.\n *\n * Returns the stream on success, a `NULL` pointer otherwise.\n */\n-struct odb_stream *odb_read_stream_open(struct object_database *odb,\n-\t\t\t\t\tconst struct object_id *oid,\n-\t\t\t\t\tstruct stream_filter *filter);\n+struct odb_stream *odb_stream_from_object(struct object_database *odb,\n+\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t struct stream_filter *filter);\n+\n+/*\n+ * Create a new object stream for the given file descriptor. This can be used\n+ * to, for example, stream an object into the object database. This function\n+ * does _not_ take ownership of the file descriptor. It's the responsibility of\n+ * the caller to close it after the stream has been closed.\n+ */\n+struct odb_stream *odb_stream_from_fd(int fd, size_t size, enum object_type type);\n \n /*\n * Close the given object stream and release all resources associated with it.\n@@ -65,9 +73,4 @@ int odb_stream_blob_to_fd(struct object_database *odb,\n \t\t\t struct stream_filter *filter,\n \t\t\t int can_seek);\n \n-/*\n- * Sets up an ODB write stream that reads from an fd.\n- */\n-struct odb_stream *odb_write_stream_from_fd(int fd, size_t size, enum object_type type);\n-\n #endif /* STREAMING_H */\n\n-- \n2.55.0.679.g6767b8d81c.dirty\n\n"},{"id":"549578","messageId":"anIWUKV8iBFkT7g9@denethor","threadId":"66111","inReplyTo":"20260804-pks-odb-stream-unification-v1-1-86d70e82345e@pks.im","subject":"Re: [PATCH 1/7] odb/streaming: track write stream size in the structure","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-08-04T16:47:48Z","receivedAt":"2026-08-04T16:47:59Z","isPatch":true,"body":"On 26/08/04 09:25AM, Patrick Steinhardt wrote:\n> When passing around a `struct odb_write_stream` we typically also have\n> to pass the number of bytes that the stream will yield. This is required\n> because the object header itself contains that size, and consequently we\n> cannot write the header without that information.\n> \n> Move this information into the stream itself so that it becomes self-\n> describing. In addition to that, this also brings the `struct\n> odb_write_stream` a bit closer to the `struct odb_read_stream` so that\n> we can eventually merge both stream types.\n\nStoring the object size in the stream directly makes complete sense.\n\n> Signed-off-by: Patrick Steinhardt \n> ---\n> builtin/unpack-objects.c | 3 ++-\n> object-file.c | 25 +++++++++++--------------\n> odb.c | 4 ++--\n> odb.h | 2 +-\n> odb/source-files.c | 3 +--\n> odb/source-inmemory.c | 11 +++++------\n> odb/source-loose.c | 7 +++----\n> odb/source-packed.c | 1 -\n> odb/source.h | 5 ++---\n> odb/streaming.c | 1 +\n> odb/streaming.h | 1 +\n> odb/transaction.c | 4 ++--\n> odb/transaction.h | 4 ++--\n> t/unit-tests/u-odb-inmemory.c | 11 +++++------\n> 14 files changed, 38 insertions(+), 44 deletions(-)\n> \n[snip]\n> diff --git a/odb/streaming.h b/odb/streaming.h\n> index c023671780..4d7d31b5aa 100644\n> --- a/odb/streaming.h\n> +++ b/odb/streaming.h\n> @@ -55,6 +55,7 @@ ssize_t odb_read_stream_read(struct odb_read_stream *stream, void *buf, size_t l\n> struct odb_write_stream {\n> \tssize_t (*read)(struct odb_write_stream *, unsigned char *, size_t);\n> \tvoid *data;\n> +\tsize_t size;\n> \tint is_finished;\n> };\n\nThe size is now stored directly in the stream, the rest of this patch is\nadjusting callers to use the embedded size information instead of\npassing it. Looks good.\n\n-Justin\n"},{"id":"549581","messageId":"anIXut41fFzRcyOI@denethor","threadId":"66111","inReplyTo":"20260804-pks-odb-stream-unification-v1-2-86d70e82345e@pks.im","subject":"Re: [PATCH 2/7] odb/streaming: drop `is_finished` field","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-08-04T17:46:56Z","receivedAt":"2026-08-04T17:47:01Z","isPatch":true,"body":"On 26/08/04 09:25AM, Patrick Steinhardt wrote:\n> The `is_finished` field is used to track whether a write stream is done\n> writing all of its data. Tracking this field as part of the stream\n> itself shouldn't be required though: callers will already know when the\n> stream is done when the stream's read function returns zero bytes, same\n> as when reading from a file descriptor.\n> \n> There is one exception where it gets a bit more complicated: when\n> consuming data in \"builtin/unpack-objects.c\" it may happen that we don't\n> yield any new bytes after reading from the pipe. This is addressed by\n> looping until we have produced at least a single byte of output.\n\nAddressing this one outlier sounds reasonable.\n\n> Drop the field from `struct odb_write_stream`. Again, same as in the\n> preceding commit, this brings the structure a bit closer to its sibling\n> `struct odb_read_stream`.\n\nThis also makes the overal interface a bit simpler. Callers can trust\nthat when `odb_write_stream_read()` returns zero, it is actually\nfinished without having to inspect further.\n\n> Signed-off-by: Patrick Steinhardt \n> ---\n> builtin/unpack-objects.c | 15 ++++++++-------\n> object-file.c | 13 ++++++++-----\n> odb/source-inmemory.c | 9 ++++++++-\n> odb/source-loose.c | 12 ++++++++----\n> odb/streaming.c | 5 +----\n> odb/streaming.h | 1 -\n> t/unit-tests/u-odb-inmemory.c | 5 +++--\n> 7 files changed, 36 insertions(+), 24 deletions(-)\n> \n> diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\n> index f3e0b504f4..b7c486ea94 100644\n> --- a/builtin/unpack-objects.c\n> +++ b/builtin/unpack-objects.c\n> @@ -368,20 +368,20 @@ static ssize_t feed_input_zstream(struct odb_write_stream *in_stream,\n> {\n> \tstruct input_zstream_data *data = in_stream->data;\n> \tgit_zstream *zstream = data->zstream;\n> -\tvoid *in = fill(1);\n> \n> -\tif (in_stream->is_finished)\n> +\tif (data->status != Z_OK)\n> \t\treturn 0;\n> \n> \tzstream->next_out = buf;\n> \tzstream->avail_out = buf_len;\n> -\tzstream->next_in = in;\n> -\tzstream->avail_in = len;\n> \n> -\tdata->status = git_inflate(zstream, 0);\n> +\twhile (data->status == Z_OK && zstream->avail_out == buf_len) {\n> +\t\tzstream->next_in = fill(1);\n> +\t\tzstream->avail_in = len;\n> +\t\tdata->status = git_inflate(zstream, 0);\n> +\t\tuse(len - zstream->avail_in);\n> +\t}\n\nOk, now we call `git_inflate()` in a loop until there is an error or we\nget some data back. This makes it so we can trust that returning zero\ndoes mean that the stream is finished. Previously, it was the callers\nresponsibility to check the `is_finished` stream field to be certain.\n\nI was curious if we needed to update any code documentation with this\nchange, but it looks like the comments for `odb_write_stream_read()`\nalready made it sound like this was the current behavior.\n\n[snip]\n> diff --git a/odb/streaming.h b/odb/streaming.h\n> index 4d7d31b5aa..5e8e6e532e 100644\n> --- a/odb/streaming.h\n> +++ b/odb/streaming.h\n> @@ -56,7 +56,6 @@ struct odb_write_stream {\n> \tssize_t (*read)(struct odb_write_stream *, unsigned char *, size_t);\n> \tvoid *data;\n> \tsize_t size;\n> -\tint is_finished;\n\nThe field is dropped. Nice.\n\nThe rest of this patch looks good.\n\n-Justin\n"},{"id":"549582","messageId":"anInniMjCtU9Qae7@denethor","threadId":"66111","inReplyTo":"20260804-pks-odb-stream-unification-v1-3-86d70e82345e@pks.im","subject":"Re: [PATCH 3/7] odb/streaming: support streaming arbitrary object types","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-08-04T18:03:41Z","receivedAt":"2026-08-04T18:03:46Z","isPatch":true,"body":"On 26/08/04 09:25AM, Patrick Steinhardt wrote:\n> The object database supports the ability to write object streams into\n> it. This functionality is used when we encounter a blob that is larger\n> than \"core.bigFileThreshold\" so that we don't have to soak large files\n> into memory.\n> \n> As we only ever write large files, the infrastructure doesn't support\n> specifying any other object type than \"blob\". This limitation is quite\n> artificial though: there is no reason why we shouldn't support writing\n> arbitrary large objects with a stream. While it's very unlikely that we\n> encounter a huge object other than a blob, users are known to be\n> creative and sometimes like to inflict pain on themselves by creating\n> commits or trees that are huge.\n> \n> Extend the infrastructure to support streaming arbitrary object types.\n> For now we don't use this functionality anywhere, but it brings us a bit\n> closer to unify `struct odb_read_stream` and `struct odb_write_stream`.\n\nVery happy to see this change. :)\n\n> Signed-off-by: Patrick Steinhardt \n> ---\n> builtin/unpack-objects.c | 1 +\n> object-file.c | 31 +++++++++++++++----------------\n> odb/source-inmemory.c | 2 +-\n> odb/source-loose.c | 2 +-\n> odb/streaming.c | 3 ++-\n> odb/streaming.h | 3 ++-\n> t/unit-tests/u-odb-inmemory.c | 7 +++++--\n> 7 files changed, 27 insertions(+), 22 deletions(-)\n\nJust FYI, there is also a comment in \"odb/transaction.h\" for the\n`write_object_stream` callback that is also now outdated due to this\nchange. We may want to update that too.\n\n[snip]\n> @@ -953,7 +953,7 @@ int index_fd(struct index_state *istate, struct object_id *oid,\n> \t\t\t\t type, path, flags);\n> \t} else {\n> \t\tstruct odb_write_stream stream;\n> -\t\todb_write_stream_from_fd(&stream, fd, xsize_t(st->st_size));\n> +\t\todb_write_stream_from_fd(&stream, fd, xsize_t(st->st_size), OBJ_BLOB);\n\nWe still only target large blobs for streaming here, but the underlying\ninfrastructure is now generic which is nice.\n\n[snip]\n> diff --git a/odb/streaming.h b/odb/streaming.h\n> index 5e8e6e532e..3c8ed55129 100644\n> --- a/odb/streaming.h\n> +++ b/odb/streaming.h\n> @@ -56,6 +56,7 @@ struct odb_write_stream {\n> \tssize_t (*read)(struct odb_write_stream *, unsigned char *, size_t);\n> \tvoid *data;\n> \tsize_t size;\n> +\tenum object_type type;\n\nWe now store the object type in the stream itself. Similar to size\ninformation, the type information is always known in advance when\ncreating the object stream.\n\nThe rest of this patch is just updating call sites accordingly. Looks\ngood.\n\n-Justin\n"},{"id":"549588","messageId":"anIrtigj0L7PU2hl@denethor","threadId":"66111","inReplyTo":"20260804-pks-odb-stream-unification-v1-5-86d70e82345e@pks.im","subject":"Re: [PATCH 5/7] odb/streaming: consolidate read and write streams","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-08-04T18:23:42Z","receivedAt":"2026-08-04T18:23:46Z","isPatch":true,"body":"On 26/08/04 09:25AM, Patrick Steinhardt wrote:\n> The `struct odb_read_stream` and `struct odb_write_stream` both provide\n> the same functionality: they allow a caller to read object data from an\n> arbitrary source. Historically, the only difference was that the read\n> stream was used to read data out of the object database, whereas the\n> write stream was used to write data into the object database, but the\n> interfaces were mostly the same.\n\nOk.\n\n> Over the preceding commits we have refactored the write stream to have\n> almost exactly the same interface as the read stream. With these\n> refactorings we can now easily merge those two streams into a single\n> interface that's used for both use cases.\n\nNice.\n\n> While most of the changes are mechanical, there are two sites that need\n> special mention:\n> \n> - \"builtin/unpack-objects.c\" creates a write stream from compressed\n> object data.\n> \n> - \"odb/streaming.c\" creates a write stream from a file descriptor.\n> \n> Adapting these sites to yield the new stream type requires a couple more\n> changes. Most importantly, instead of embedding the pointer to the data\n> in `struct odb_write_stream`, we now allocate a structure that wraps the\n> new `struct odb_stream` base. Other than that though, the changes are\n> rather straight forward.\n\nOk, creating wrapper stream types for these sounds reasonable.\n\n> \n> Signed-off-by: Patrick Steinhardt \n> ---\n> builtin/unpack-objects.c | 31 ++++++++++++++++---------------\n> object-file.c | 25 ++++++++++++-------------\n> odb.c | 2 +-\n> odb.h | 4 ++--\n> odb/source-files.c | 2 +-\n> odb/source-inmemory.c | 4 ++--\n> odb/source-loose.c | 6 +++---\n> odb/source-packed.c | 2 +-\n> odb/source.h | 4 ++--\n> odb/streaming.c | 35 ++++++++++++++++-------------------\n> odb/streaming.h | 31 +++----------------------------\n> odb/transaction.c | 2 +-\n> odb/transaction.h | 4 ++--\n> t/unit-tests/u-odb-inmemory.c | 6 +++---\n> 14 files changed, 65 insertions(+), 93 deletions(-)\n> \n> diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\n> index 7439ec53be..05a2d48011 100644\n> --- a/builtin/unpack-objects.c\n> +++ b/builtin/unpack-objects.c\n> @@ -359,20 +359,21 @@ static void unpack_non_delta_entry(enum object_type type, unsigned long size,\n> }\n> \n> struct input_zstream_data {\n> +\tstruct odb_stream base;\n> \tgit_zstream *zstream;\n> \tint status;\n> };\n\nOk, as mentioned in the commit message, we now embed the stream instead\nstoring a pointer to the extra data. Should we also update the struct\nname here now that `input_zstream_data` is really itself a stream?\n\n> -static ssize_t feed_input_zstream(struct odb_write_stream *in_stream,\n> -\t\t\t\t unsigned char *buf, size_t buf_len)\n> +static ssize_t feed_input_zstream(struct odb_stream *in_stream,\n> +\t\t\t\t char *buf, size_t buf_len)\n> {\n> -\tstruct input_zstream_data *data = in_stream->data;\n> +\tstruct input_zstream_data *data = container_of(in_stream, struct input_zstream_data, base);\n\nCallback is updated to fetch data from the base stream.\n\n> \tgit_zstream *zstream = data->zstream;\n> \n> \tif (data->status != Z_OK)\n> \t\treturn 0;\n> \n> -\tzstream->next_out = buf;\n> +\tzstream->next_out = (unsigned char *) buf;\n> \tzstream->avail_out = buf_len;\n> \n> \twhile (data->status == Z_OK && zstream->avail_out == buf_len) {\n> @@ -388,24 +389,24 @@ static ssize_t feed_input_zstream(struct odb_write_stream *in_stream,\n> static void stream_blob(unsigned long size, unsigned nr)\n> {\n> \tgit_zstream zstream = { 0 };\n> -\tstruct input_zstream_data data = { 0 };\n> -\tstruct odb_write_stream in_stream = {\n> -\t\t.read = feed_input_zstream,\n> -\t\t.data = &data,\n> -\t\t.size = size,\n> -\t\t.type = OBJ_BLOB,\n> +\tstruct input_zstream_data in_stream = {\n> +\t\t.base = {\n> +\t\t\t.read = feed_input_zstream,\n> +\t\t\t.size = size,\n> +\t\t\t.type = OBJ_BLOB,\n> +\t\t},\n> +\t\t.zstream = &zstream,\n> +\t\t.status = Z_OK,\n> \t};\n> \tstruct obj_info *info = &obj_list[nr];\n> \n> -\tdata.zstream = &zstream;\n> -\tdata.status = Z_OK;\n> \tgit_inflate_init(&zstream);\n> \n> -\tif (odb_write_object_stream(the_repository->objects, &in_stream, &info->oid))\n> +\tif (odb_write_object_stream(the_repository->objects, &in_stream.base, &info->oid))\n> \t\tdie(_(\"failed to write object in stream\"));\n> \n> -\tif (data.status != Z_STREAM_END)\n> -\t\tdie(_(\"inflate returned (%d)\"), data.status);\n> +\tif (in_stream.status != Z_STREAM_END)\n> +\t\tdie(_(\"inflate returned (%d)\"), in_stream.status);\n> \tgit_inflate_end(&zstream);\n> \n> \tif (strict) {\n\nStream set up is now updated to use the wrapper stream. Looks good.\n\n[snip]\n> @@ -299,14 +289,15 @@ int odb_stream_blob_to_fd(struct object_database *odb,\n> }\n> \n> struct read_object_fd_data {\n> +\tstruct odb_stream base;\n> \tint fd;\n> \tsize_t remaining;\n> };\n\n`read_object_fd_data` is also now set up as a wrapper stream. Should we\nalso rename it accordingly?\n\n> -static ssize_t read_object_fd(struct odb_write_stream *stream,\n> -\t\t\t unsigned char *buf, size_t len)\n> +static ssize_t read_object_fd(struct odb_stream *stream,\n> +\t\t\t char *buf, size_t len)\n> {\n> -\tstruct read_object_fd_data *data = stream->data;\n> +\tstruct read_object_fd_data *data = container_of(stream, struct read_object_fd_data, base);\n> \tssize_t read_result;\n> \tsize_t count;\n> \n> @@ -323,17 +314,23 @@ static ssize_t read_object_fd(struct odb_write_stream *stream,\n> \treturn read_result;\n> }\n> \n> -void odb_write_stream_from_fd(struct odb_write_stream *stream, int fd,\n> -\t\t\t size_t size, enum object_type type)\n> +static int close_object_fd(struct odb_stream *stream UNUSED)\n> +{\n> +\t/* The file descriptor is owned by the caller for now. */\n> +\treturn 0;\n> +}\n> +\n> +struct odb_stream *odb_write_stream_from_fd(int fd, size_t size, enum object_type type)\n\nShould we also update the name of this function?\n\nThe rest of this patch is just renames and call site updates to\nconsolidate the two stream types. Looks good.\n\n-Justin\n"},{"id":"549589","messageId":"anIuT6GFJ8st-cF4@denethor","threadId":"66111","inReplyTo":"20260804-pks-odb-stream-unification-v1-6-86d70e82345e@pks.im","subject":"Re: [PATCH 6/7] odb/streaming: rename `struct read_object_fd_data`","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-08-04T18:25:21Z","receivedAt":"2026-08-04T18:25:24Z","isPatch":true,"body":"On 26/08/04 09:25AM, Patrick Steinhardt wrote:\n> With the preceding refactorings the `struct read_object_fd_data` is now\n> somewhat misnamed, as it doesn't only contain the data anymore, but also\n> the stream itself. Rename the structure to `struct fd_stream` to better\n> match the new structure.\n\nAh ok, this addresses one of my comments in the last patch. The renames\nhere all look good.\n\n-Justin\n"},{"id":"549590","messageId":"anIu8xOTtZdhDNRD@denethor","threadId":"66111","inReplyTo":"20260804-pks-odb-stream-unification-v1-7-86d70e82345e@pks.im","subject":"Re: [PATCH 7/7] odb/streaming: unify function names to create new streams","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-08-04T18:30:06Z","receivedAt":"2026-08-04T18:30:08Z","isPatch":true,"body":"On 26/08/04 09:25AM, Patrick Steinhardt wrote:\n> Unify the function names to create new streams from different sources so\n> that they follow a common schema. While at it, document the ownership of\n> the file descriptor passed to `odb_stream_from_fd()`.\n> \n> Signed-off-by: Patrick Steinhardt \n> ---\n[snip]\n> +/*\n> + * Create a new object stream for the given file descriptor. This can be used\n> + * to, for example, stream an object into the object database. This function\n> + * does _not_ take ownership of the file descriptor. It's the responsibility of\n> + * the caller to close it after the stream has been closed.\n> + */\n> +struct odb_stream *odb_stream_from_fd(int fd, size_t size, enum object_type type);\n\nAh ok, here we rename `odb_write_stream_from_fd()` to\n`odb_stream_from_fd()`. This also addresses one of my comments from a\nprevious patch.\n\nThe renames in this patch all look sensible to me. Thanks.\n\n-Justin\n"},{"id":"549603","messageId":"xmqqpkzxvhqm.fsf@gitster.g","threadId":"66111","inReplyTo":"20260804-pks-odb-stream-unification-v1-3-86d70e82345e@pks.im","subject":"Re: [PATCH 3/7] odb/streaming: support streaming arbitrary object types","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-08-04T18:54:41Z","receivedAt":"2026-08-04T18:54:44Z","isPatch":true,"body":"Patrick Steinhardt writes:\n\n> The object database supports the ability to write object streams into\n> it. This functionality is used when we encounter a blob that is larger\n> than \"core.bigFileThreshold\" so that we don't have to soak large files\n> into memory.\n\nI am still not sold the benefit of using a single \"stream\" type both\nfor reading and writing yet at this point in my reading (I am not\nyet done 50% of the series yet at step 3/7), but I agree that it\nwould be a good thing to be able to stream objects that are not\nblobs.\n\n> diff --git a/odb/source-inmemory.c b/odb/source-inmemory.c\n> index 01bb81c63c..4f76db5496 100644\n> --- a/odb/source-inmemory.c\n> +++ b/odb/source-inmemory.c\n> @@ -293,7 +293,7 @@ static int odb_source_inmemory_write_object_stream(struct odb_source *source,\n> \thash_object_file(source->odb->repo->hash_algo, data, total_read, OBJ_BLOB, oid);\n> \n> \tret = odb_source_inmemory_write_object(source, data, stream->size,\n> -\t\t\t\t\t OBJ_BLOB, oid, NULL, NULL, 0);\n> +\t\t\t\t\t stream->type, oid, NULL, NULL, 0);\n\nIt is a bit annoying that we treat 'inmemory' as if it were a valid\nsingle word both in the filename and in the function name, but more\nimportantly, hash_object_file() (used to compute the object name of\nthe object we are writing into the variable 'oid') still hashes\nassuming that the object is a blob. What is the implication of\nfeeding the data to odb_source_in_memory_write_object() as\nstream->type (which is not necessarily OBJ_BLOB) with that 'oid'\nwhose object name was computed as OBJ_BLOB?\n"},{"id":"549639","messageId":"anLS27cZglL-tK5s@pks.im","threadId":"66111","inReplyTo":"anIrtigj0L7PU2hl@denethor","subject":"Re: [PATCH 5/7] odb/streaming: consolidate read and write streams","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-05T06:06:19Z","receivedAt":"2026-08-05T06:06:26Z","isPatch":true,"body":"On Tue, Aug 04, 2026 at 01:23:42PM -0500, Justin Tobler wrote:\n> On 26/08/04 09:25AM, Patrick Steinhardt wrote:\n> > diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\n> > index 7439ec53be..05a2d48011 100644\n> > --- a/builtin/unpack-objects.c\n> > +++ b/builtin/unpack-objects.c\n> > @@ -359,20 +359,21 @@ static void unpack_non_delta_entry(enum object_type type, unsigned long size,\n> > }\n> > \n> > struct input_zstream_data {\n> > +\tstruct odb_stream base;\n> > \tgit_zstream *zstream;\n> > \tint status;\n> > };\n> \n> Ok, as mentioned in the commit message, we now embed the stream instead\n> storing a pointer to the extra data. Should we also update the struct\n> name here now that `input_zstream_data` is really itself a stream?\n\nI intentionally didn't rename anything in this commit here to keep churn\nminimal, and deferred the renames into subsequent commits. I should've\nnoted that in the commit message though.\n\nThis one structure I didn't rename though. I'll add a commit.\n\nPatrick\n"},{"id":"549640","messageId":"anLS4CNVCQBm-2JQ@pks.im","threadId":"66111","inReplyTo":"anInniMjCtU9Qae7@denethor","subject":"Re: [PATCH 3/7] odb/streaming: support streaming arbitrary object types","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-05T06:06:24Z","receivedAt":"2026-08-05T06:06:30Z","isPatch":true,"body":"On Tue, Aug 04, 2026 at 01:03:41PM -0500, Justin Tobler wrote:\n> On 26/08/04 09:25AM, Patrick Steinhardt wrote:\n> > builtin/unpack-objects.c | 1 +\n> > object-file.c | 31 +++++++++++++++----------------\n> > odb/source-inmemory.c | 2 +-\n> > odb/source-loose.c | 2 +-\n> > odb/streaming.c | 3 ++-\n> > odb/streaming.h | 3 ++-\n> > t/unit-tests/u-odb-inmemory.c | 7 +++++--\n> > 7 files changed, 27 insertions(+), 22 deletions(-)\n> \n> Just FYI, there is also a comment in \"odb/transaction.h\" for the\n> `write_object_stream` callback that is also now outdated due to this\n> change. We may want to update that too.\n\nGood catch, fixed now.\n\nPatrick\n"},{"id":"549641","messageId":"anLS5z22CpF82cd7@pks.im","threadId":"66111","inReplyTo":"xmqqpkzxvhqm.fsf@gitster.g","subject":"Re: [PATCH 3/7] odb/streaming: support streaming arbitrary object types","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-05T06:06:31Z","receivedAt":"2026-08-05T06:06:35Z","isPatch":true,"body":"On Tue, Aug 04, 2026 at 11:54:41AM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > The object database supports the ability to write object streams into\n> > it. This functionality is used when we encounter a blob that is larger\n> > than \"core.bigFileThreshold\" so that we don't have to soak large files\n> > into memory.\n> \n> I am still not sold the benefit of using a single \"stream\" type both\n> for reading and writing yet at this point in my reading (I am not\n> yet done 50% of the series yet at step 3/7), but I agree that it\n> would be a good thing to be able to stream objects that are not\n> blobs.\n\nThe reason why I want to unify these two streams is mostly that despite\ntheir name, they basically do the exact same thing: both stream types\nallow the user to read data from them in a streaming fashion. The only\nthing that's different about the \"write\" stream is that it doesn't\nencode its information as part of the stream itself, whereas the \"read\"\nstream does. So having two types is quite pointless in the first place.\n\n> > diff --git a/odb/source-inmemory.c b/odb/source-inmemory.c\n> > index 01bb81c63c..4f76db5496 100644\n> > --- a/odb/source-inmemory.c\n> > +++ b/odb/source-inmemory.c\n> > @@ -293,7 +293,7 @@ static int odb_source_inmemory_write_object_stream(struct odb_source *source,\n> > \thash_object_file(source->odb->repo->hash_algo, data, total_read, OBJ_BLOB, oid);\n> > \n> > \tret = odb_source_inmemory_write_object(source, data, stream->size,\n> > -\t\t\t\t\t OBJ_BLOB, oid, NULL, NULL, 0);\n> > +\t\t\t\t\t stream->type, oid, NULL, NULL, 0);\n> \n> It is a bit annoying that we treat 'inmemory' as if it were a valid\n> single word both in the filename and in the function name, but more\n> importantly, hash_object_file() (used to compute the object name of\n> the object we are writing into the variable 'oid') still hashes\n> assuming that the object is a blob. What is the implication of\n> feeding the data to odb_source_in_memory_write_object() as\n> stream->type (which is not necessarily OBJ_BLOB) with that 'oid'\n> whose object name was computed as OBJ_BLOB?\n\nOh, that's an oversight on my part. We'd use the wrong object header,\nthus arrive at a wrong hash and then ultimately store the object under\nthe wrong hash in the in-memory source. Which doesn't really matter\nafter this patch series as we still only write blobs via streams, but\nit's a bug waiting to happen. Fixed now.\n\nPatrick\n"},{"id":"549648","messageId":"20260805-pks-odb-stream-unification-v2-0-b8c369564641@pks.im","threadId":"66111","inReplyTo":"20260804-pks-odb-stream-unification-v1-0-86d70e82345e@pks.im","subject":"[PATCH v2 0/8] odb: unify read and write streams","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-05T07:44:44Z","receivedAt":"2026-08-05T07:44:57Z","isPatch":true,"body":"Hi,\n\nwe have two different kind of object database streams in our code base:\n`odb_write_stream` and `odb_read_stream`. While those are used for\ndifferent use cases, the provided functionality is ultimately the exact\nsame.\n\nThis patch series thus refactors these streams so that we have a single\n`odb_stream`, only. This allows us to reuse the streams for different\nkinds of purposes and makes them more generally useful overall. For\nexample, it's trivially possible now to create an object stream for any\ngiven object and then write that stream into a different source.\n\nThe series is built on top of 5b2471720c (The 10th batch, 2026-08-03).\n\nChanges in v2:\n - Use the correct object type when hashing in-memory objects.\n - Remove a stale comment.\n - Adapt a commit message to mention that renames will follow in\n subsequent commits.\n - Add another commit to rename `struct input_zstream_data`.\n - Link to v1: https://patch.msgid.link/20260804-pks-odb-stream-unification-v1-0-86d70e82345e@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (8):\n odb/streaming: track write stream size in the structure\n odb/streaming: drop `is_finished` field\n odb/streaming: support streaming arbitrary object types\n odb/streaming: rename `struct odb_read_stream`\n odb/streaming: consolidate read and write streams\n odb/streaming: rename `struct read_object_fd_data`\n odb/streaming: rename `struct input_zstream_data`\n odb/streaming: unify function names to create new streams\n\n archive-tar.c | 8 ++--\n archive-zip.c | 12 ++---\n builtin/index-pack.c | 8 ++--\n builtin/pack-objects.c | 18 ++++----\n builtin/unpack-objects.c | 44 ++++++++++--------\n object-file.c | 76 +++++++++++++++---------------\n object-file.h | 2 +-\n object.c | 6 +--\n odb.c | 4 +-\n odb.h | 4 +-\n odb/source-files.c | 7 ++-\n odb/source-inmemory.c | 35 ++++++++------\n odb/source-loose.c | 33 ++++++++------\n odb/source-packed.c | 5 +-\n odb/source.h | 13 +++---\n odb/streaming.c | 104 ++++++++++++++++++++----------------------\n odb/streaming.h | 69 ++++++++++------------------\n odb/transaction.c | 6 +--\n odb/transaction.h | 8 ++--\n pack-check.c | 4 +-\n packfile.c | 8 ++--\n packfile.h | 4 +-\n t/unit-tests/u-odb-inmemory.c | 37 ++++++++-------\n 23 files changed, 251 insertions(+), 264 deletions(-)\n\nRange-diff versus v1:\n\n1: 0085df877f = 1: 1966710c12 odb/streaming: track write stream size in the structure\n2: 5fbbfd9010 = 2: 87c7981a6c odb/streaming: drop `is_finished` field\n3: 52e5b87761 ! 3: 9aede44fba odb/streaming: support streaming arbitrary object types\n @@ object-file.c: int index_fd(struct index_state *istate, struct object_id *oid,\n \n ## odb/source-inmemory.c ##\n @@ odb/source-inmemory.c: static int odb_source_inmemory_write_object_stream(struct odb_source *source,\n - \thash_object_file(source->odb->repo->hash_algo, data, total_read, OBJ_BLOB, oid);\n + \t\tgoto out;\n + \t}\n + \n +-\thash_object_file(source->odb->repo->hash_algo, data, total_read, OBJ_BLOB, oid);\n ++\thash_object_file(source->odb->repo->hash_algo, data, total_read,\n ++\t\t\t stream->type, oid);\n \n \tret = odb_source_inmemory_write_object(source, data, stream->size,\n -\t\t\t\t\t OBJ_BLOB, oid, NULL, NULL, 0);\n @@ odb/streaming.h: int odb_stream_blob_to_fd(struct object_database *odb,\n \n #endif /* STREAMING_H */\n \n + ## odb/transaction.h ##\n +@@ odb/transaction.h: struct odb_transaction {\n + \n + \t/*\n + \t * This callback is expected to write the given object stream into\n +-\t * the ODB transaction. Note that for now, only blobs support streaming.\n ++\t * the ODB transaction.\n + \t *\n + \t * The resulting object ID shall be written into the out pointer. The\n + \t * callback is expected to return 0 on success, a negative error code\n +\n ## t/unit-tests/u-odb-inmemory.c ##\n @@ t/unit-tests/u-odb-inmemory.c: void test_odb_inmemory__write_object_stream(void)\n \tstruct odb_source_inmemory *source = odb_source_inmemory_new(odb);\n4: f178d441f0 = 4: ca84a2b645 odb/streaming: rename `struct odb_read_stream`\n5: 0d72d27078 ! 5: 838394bffc odb/streaming: consolidate read and write streams\n @@ Commit message\n new `struct odb_stream` base. Other than that though, the changes are\n rather straight forward.\n \n + Some of the structures and functions are now somewhat misnamed. These\n + will be fixed in subsequent commits.\n +\n Signed-off-by: Patrick Steinhardt \n \n ## builtin/unpack-objects.c ##\n6: ced59bdc85 = 6: 850b7e081d odb/streaming: rename `struct read_object_fd_data`\n-: ---------- > 7: c3fe9f8b0c odb/streaming: rename `struct input_zstream_data`\n7: f76f4350ef = 8: 4df81651ba odb/streaming: unify function names to create new streams\n\n---\nbase-commit: 5b2471720c93ee30e5764a19f3d3b3ae9ec9712a\nchange-id: 20260724-pks-odb-stream-unification-334dc2a75888\n\n"},{"id":"549649","messageId":"20260805-pks-odb-stream-unification-v2-1-b8c369564641@pks.im","threadId":"66111","inReplyTo":"20260805-pks-odb-stream-unification-v2-0-b8c369564641@pks.im","subject":"[PATCH v2 1/8] odb/streaming: track write stream size in the structure","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-05T07:44:45Z","receivedAt":"2026-08-05T07:44:58Z","isPatch":true,"body":"When passing around a `struct odb_write_stream` we typically also have\nto pass the number of bytes that the stream will yield. This is required\nbecause the object header itself contains that size, and consequently we\ncannot write the header without that information.\n\nMove this information into the stream itself so that it becomes self-\ndescribing. In addition to that, this also brings the `struct\nodb_write_stream` a bit closer to the `struct odb_read_stream` so that\nwe can eventually merge both stream types.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/unpack-objects.c | 3 ++-\n object-file.c | 25 +++++++++++--------------\n odb.c | 4 ++--\n odb.h | 2 +-\n odb/source-files.c | 3 +--\n odb/source-inmemory.c | 11 +++++------\n odb/source-loose.c | 7 +++----\n odb/source-packed.c | 1 -\n odb/source.h | 5 ++---\n odb/streaming.c | 1 +\n odb/streaming.h | 1 +\n odb/transaction.c | 4 ++--\n odb/transaction.h | 4 ++--\n t/unit-tests/u-odb-inmemory.c | 11 +++++------\n 14 files changed, 38 insertions(+), 44 deletions(-)\n\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex 4263edfbec..f3e0b504f4 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -392,13 +392,14 @@ static void stream_blob(unsigned long size, unsigned nr)\n \tstruct odb_write_stream in_stream = {\n \t\t.read = feed_input_zstream,\n \t\t.data = &data,\n+\t\t.size = size,\n \t};\n \tstruct obj_info *info = &obj_list[nr];\n \n \tdata.zstream = &zstream;\n \tgit_inflate_init(&zstream);\n \n-\tif (odb_write_object_stream(the_repository->objects, &in_stream, size, &info->oid))\n+\tif (odb_write_object_stream(the_repository->objects, &in_stream, &info->oid))\n \t\tdie(_(\"failed to write object in stream\"));\n \n \tif (data.status != Z_STREAM_END)\ndiff --git a/object-file.c b/object-file.c\nindex ec35c318bc..b196abb596 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -704,7 +704,7 @@ static void prepare_packfile_transaction(struct odb_transaction_files *transacti\n \n static int hash_blob_stream(struct odb_write_stream *stream,\n \t\t\t const struct git_hash_algo *hash_algo,\n-\t\t\t struct object_id *result_oid, size_t size)\n+\t\t\t struct object_id *result_oid)\n {\n \tunsigned char buf[16384];\n \tstruct git_hash_ctx ctx;\n@@ -712,7 +712,7 @@ static int hash_blob_stream(struct odb_write_stream *stream,\n \tsize_t bytes_hashed = 0;\n \n \theader_len = format_object_header((char *)buf, sizeof(buf),\n-\t\t\t\t\t OBJ_BLOB, size);\n+\t\t\t\t\t OBJ_BLOB, stream->size);\n \tgit_hash_init(&ctx, hash_algo);\n \tgit_hash_update(&ctx, buf, header_len);\n \n@@ -727,7 +727,7 @@ static int hash_blob_stream(struct odb_write_stream *stream,\n \t\tbytes_hashed += read_result;\n \t}\n \n-\tif (bytes_hashed != size)\n+\tif (bytes_hashed != stream->size)\n \t\treturn -1;\n \n \tgit_hash_final_oid(result_oid, &ctx);\n@@ -740,7 +740,7 @@ static int hash_blob_stream(struct odb_write_stream *stream,\n * packfile in state while updating the hash in ctx.\n */\n static void stream_blob_to_pack(struct transaction_packfile *state,\n-\t\t\t\tstruct git_hash_ctx *ctx, size_t size,\n+\t\t\t\tstruct git_hash_ctx *ctx,\n \t\t\t\tstruct odb_write_stream *stream)\n {\n \tgit_zstream s;\n@@ -753,7 +753,7 @@ static void stream_blob_to_pack(struct transaction_packfile *state,\n \n \tgit_deflate_init(&s, cfg->pack_compression_level);\n \n-\thdrlen = encode_in_pack_object_header(obuf, sizeof(obuf), OBJ_BLOB, size);\n+\thdrlen = encode_in_pack_object_header(obuf, sizeof(obuf), OBJ_BLOB, stream->size);\n \ts.next_out = obuf + hdrlen;\n \ts.avail_out = sizeof(obuf) - hdrlen;\n \n@@ -793,9 +793,9 @@ static void stream_blob_to_pack(struct transaction_packfile *state,\n \t\t}\n \t}\n \n-\tif (bytes_read != size)\n+\tif (bytes_read != stream->size)\n \t\tdie(\"read %\" PRIuMAX \" bytes of blob data, but expected %\" PRIuMAX \" bytes\",\n-\t\t (uintmax_t)bytes_read, (uintmax_t)size);\n+\t\t (uintmax_t)bytes_read, (uintmax_t)stream->size);\n \n \tgit_deflate_end(&s);\n }\n@@ -870,7 +870,6 @@ static void flush_packfile_transaction(struct odb_transaction_files *transaction\n */\n static int odb_transaction_files_write_object_stream(struct odb_transaction *base,\n \t\t\t\t\t\t struct odb_write_stream *stream,\n-\t\t\t\t\t\t size_t size,\n \t\t\t\t\t\t struct object_id *result_oid)\n {\n \tstruct odb_transaction_files *transaction = container_of(base,\n@@ -884,7 +883,7 @@ static int odb_transaction_files_write_object_stream(struct odb_transaction *bas\n \tstruct pack_idx_entry *idx;\n \n \theader_len = format_object_header((char *)obuf, sizeof(obuf),\n-\t\t\t\t\t OBJ_BLOB, size);\n+\t\t\t\t\t OBJ_BLOB, stream->size);\n \tgit_hash_init(&ctx, transaction->base.source->odb->repo->hash_algo);\n \tgit_hash_update(&ctx, obuf, header_len);\n \n@@ -899,7 +898,7 @@ static int odb_transaction_files_write_object_stream(struct odb_transaction *bas\n \t * to zlib compression and is sufficient for this check.\n \t */\n \tif (state->nr_written && pack_size_limit_cfg &&\n-\t pack_size_limit_cfg < state->offset + size)\n+\t pack_size_limit_cfg < state->offset + stream->size)\n \t\tflush_packfile_transaction(transaction);\n \n \tCALLOC_ARRAY(idx, 1);\n@@ -909,7 +908,7 @@ static int odb_transaction_files_write_object_stream(struct odb_transaction *bas\n \thashfile_checkpoint(state->f, &checkpoint);\n \tidx->offset = state->offset;\n \tcrc32_begin(state->f);\n-\tstream_blob_to_pack(state, &ctx, size, stream);\n+\tstream_blob_to_pack(state, &ctx, stream);\n \tgit_hash_final_oid(result_oid, &ctx);\n \n \tidx->crc32 = crc32_end(state->f);\n@@ -962,14 +961,12 @@ int index_fd(struct index_state *istate, struct object_id *oid,\n \t\t\t\todb_transaction_begin_or_die(odb, &transaction, 0);\n \t\t\tret = odb_transaction_write_object_stream(transaction,\n \t\t\t\t\t\t\t\t &stream,\n-\t\t\t\t\t\t\t\t xsize_t(st->st_size),\n \t\t\t\t\t\t\t\t oid);\n \t\t\tif (!inflight)\n \t\t\t\todb_transaction_commit(transaction);\n \t\t} else {\n \t\t\tret = hash_blob_stream(&stream,\n-\t\t\t\t\t the_repository->hash_algo, oid,\n-\t\t\t\t\t xsize_t(st->st_size));\n+\t\t\t\t\t the_repository->hash_algo, oid);\n \t\t}\n \n \t\todb_write_stream_release(&stream);\ndiff --git a/odb.c b/odb.c\nindex dabd481f57..585b2b2965 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1028,10 +1028,10 @@ int odb_write_object_ext(struct object_database *odb,\n }\n \n int odb_write_object_stream(struct object_database *odb,\n-\t\t\t struct odb_write_stream *stream, size_t len,\n+\t\t\t struct odb_write_stream *stream,\n \t\t\t struct object_id *oid)\n {\n-\treturn odb_source_write_object_stream(odb->sources, stream, len, oid);\n+\treturn odb_source_write_object_stream(odb->sources, stream, oid);\n }\n \n struct object_database *odb_new(struct repository *repo,\ndiff --git a/odb.h b/odb.h\nindex cbc2f9ced4..019d3af3e8 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -629,7 +629,7 @@ static inline int odb_write_object(struct object_database *odb,\n struct odb_write_stream;\n \n int odb_write_object_stream(struct object_database *odb,\n-\t\t\t struct odb_write_stream *stream, size_t len,\n+\t\t\t struct odb_write_stream *stream,\n \t\t\t struct object_id *oid);\n \n void parse_alternates(const char *string,\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 5e086d266f..f51960bd71 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -175,11 +175,10 @@ static int odb_source_files_write_object(struct odb_source *source,\n \n static int odb_source_files_write_object_stream(struct odb_source *source,\n \t\t\t\t\t\tstruct odb_write_stream *stream,\n-\t\t\t\t\t\tsize_t len,\n \t\t\t\t\t\tstruct object_id *oid)\n {\n \tstruct odb_source_files *files = odb_source_files_downcast(source);\n-\treturn odb_source_write_object_stream(&files->loose->base, stream, len, oid);\n+\treturn odb_source_write_object_stream(&files->loose->base, stream, oid);\n }\n \n static int odb_source_files_begin_transaction(struct odb_source *source,\ndiff --git a/odb/source-inmemory.c b/odb/source-inmemory.c\nindex 3e71611b8e..398131e194 100644\n--- a/odb/source-inmemory.c\n+++ b/odb/source-inmemory.c\n@@ -257,7 +257,6 @@ static int odb_source_inmemory_write_object(struct odb_source *source,\n \n static int odb_source_inmemory_write_object_stream(struct odb_source *source,\n \t\t\t\t\t\t struct odb_write_stream *stream,\n-\t\t\t\t\t\t size_t len,\n \t\t\t\t\t\t struct object_id *oid)\n {\n \tchar buf[16384];\n@@ -265,12 +264,12 @@ static int odb_source_inmemory_write_object_stream(struct odb_source *source,\n \tchar *data;\n \tint ret;\n \n-\tCALLOC_ARRAY(data, len);\n+\tCALLOC_ARRAY(data, stream->size);\n \twhile (!stream->is_finished) {\n \t\tssize_t bytes_read;\n \n \t\tbytes_read = odb_write_stream_read(stream, buf, sizeof(buf));\n-\t\tif (total_read + bytes_read > len) {\n+\t\tif (total_read + bytes_read > stream->size) {\n \t\t\tret = error(\"object stream yielded more bytes than expected\");\n \t\t\tgoto out;\n \t\t}\n@@ -279,15 +278,15 @@ static int odb_source_inmemory_write_object_stream(struct odb_source *source,\n \t\ttotal_read += bytes_read;\n \t}\n \n-\tif (total_read != len) {\n+\tif (total_read != stream->size) {\n \t\tret = error(\"object stream yielded less bytes than expected\");\n \t\tgoto out;\n \t}\n \n \thash_object_file(source->odb->repo->hash_algo, data, total_read, OBJ_BLOB, oid);\n \n-\tret = odb_source_inmemory_write_object(source, data, len, OBJ_BLOB, oid,\n-\t\t\t\t\t NULL, NULL, 0);\n+\tret = odb_source_inmemory_write_object(source, data, stream->size,\n+\t\t\t\t\t OBJ_BLOB, oid, NULL, NULL, 0);\n \tif (ret < 0)\n \t\tgoto out;\n \ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex ef0e919277..77a2adb52a 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -846,7 +846,6 @@ static int odb_source_loose_write_object(struct odb_source *source,\n \n static int odb_source_loose_write_object_stream(struct odb_source *source,\n \t\t\t\t\t\tstruct odb_write_stream *in_stream,\n-\t\t\t\t\t\tsize_t len,\n \t\t\t\t\t\tstruct object_id *oid)\n {\n \tstruct odb_source_loose *loose = odb_source_loose_downcast(source);\n@@ -868,7 +867,7 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,\n \n \t/* Since oid is not determined, save tmp file to odb path. */\n \tstrbuf_addf(&filename, \"%s/\", loose->base.path);\n-\thdrlen = format_object_header(hdr, sizeof(hdr), OBJ_BLOB, len);\n+\thdrlen = format_object_header(hdr, sizeof(hdr), OBJ_BLOB, in_stream->size);\n \n \t/*\n \t * Common steps for write_loose_object and stream_loose_object to\n@@ -916,9 +915,9 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,\n \t\t */\n \t} while (ret == Z_OK || ret == Z_BUF_ERROR);\n \n-\tif (stream.total_in != len + hdrlen)\n+\tif (stream.total_in != in_stream->size + hdrlen)\n \t\tdie(_(\"write stream object %\"PRIuMAX\" != %\"PRIuMAX), (uintmax_t)stream.total_in,\n-\t\t (uintmax_t)len + hdrlen);\n+\t\t (uintmax_t)in_stream->size + hdrlen);\n \n \t/*\n \t * Common steps for write_loose_object and stream_loose_object to\ndiff --git a/odb/source-packed.c b/odb/source-packed.c\nindex 0890704e76..e6ff74833b 100644\n--- a/odb/source-packed.c\n+++ b/odb/source-packed.c\n@@ -610,7 +610,6 @@ static int odb_source_packed_write_object(struct odb_source *source UNUSED,\n \n static int odb_source_packed_write_object_stream(struct odb_source *source UNUSED,\n \t\t\t\t\t\t struct odb_write_stream *stream UNUSED,\n-\t\t\t\t\t\t size_t len UNUSED,\n \t\t\t\t\t\t struct object_id *oid UNUSED)\n {\n \treturn error(\"packed backend cannot write object streams\");\ndiff --git a/odb/source.h b/odb/source.h\nindex fc04dd5cda..0080148ba7 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -221,7 +221,7 @@ struct odb_source {\n \t * otherwise.\n \t */\n \tint (*write_object_stream)(struct odb_source *source,\n-\t\t\t\t struct odb_write_stream *stream, size_t len,\n+\t\t\t\t struct odb_write_stream *stream,\n \t\t\t\t struct object_id *oid);\n \n \t/*\n@@ -437,10 +437,9 @@ static inline int odb_source_write_object(struct odb_source *source,\n */\n static inline int odb_source_write_object_stream(struct odb_source *source,\n \t\t\t\t\t\t struct odb_write_stream *stream,\n-\t\t\t\t\t\t size_t len,\n \t\t\t\t\t\t struct object_id *oid)\n {\n-\treturn source->write_object_stream(source, stream, len, oid);\n+\treturn source->write_object_stream(source, stream, oid);\n }\n \n /*\ndiff --git a/odb/streaming.c b/odb/streaming.c\nindex 20531e864c..38c2f6687c 100644\n--- a/odb/streaming.c\n+++ b/odb/streaming.c\n@@ -336,5 +336,6 @@ void odb_write_stream_from_fd(struct odb_write_stream *stream, int fd,\n \n \tstream->data = data;\n \tstream->read = read_object_fd;\n+\tstream->size = size;\n \tstream->is_finished = 0;\n }\ndiff --git a/odb/streaming.h b/odb/streaming.h\nindex c023671780..4d7d31b5aa 100644\n--- a/odb/streaming.h\n+++ b/odb/streaming.h\n@@ -55,6 +55,7 @@ ssize_t odb_read_stream_read(struct odb_read_stream *stream, void *buf, size_t l\n struct odb_write_stream {\n \tssize_t (*read)(struct odb_write_stream *, unsigned char *, size_t);\n \tvoid *data;\n+\tsize_t size;\n \tint is_finished;\n };\n \ndiff --git a/odb/transaction.c b/odb/transaction.c\nindex dab7da6a9a..6aaf133812 100644\n--- a/odb/transaction.c\n+++ b/odb/transaction.c\n@@ -40,9 +40,9 @@ int odb_transaction_commit(struct odb_transaction *transaction)\n \n int odb_transaction_write_object_stream(struct odb_transaction *transaction,\n \t\t\t\t\tstruct odb_write_stream *stream,\n-\t\t\t\t\tsize_t len, struct object_id *oid)\n+\t\t\t\t\tstruct object_id *oid)\n {\n-\treturn transaction->write_object_stream(transaction, stream, len, oid);\n+\treturn transaction->write_object_stream(transaction, stream, oid);\n }\n \n int odb_transaction_env(struct odb_transaction *transaction, struct strvec *env)\ndiff --git a/odb/transaction.h b/odb/transaction.h\nindex 4cb2eafcbf..ffb279314c 100644\n--- a/odb/transaction.h\n+++ b/odb/transaction.h\n@@ -31,7 +31,7 @@ struct odb_transaction {\n \t * otherwise.\n \t */\n \tint (*write_object_stream)(struct odb_transaction *transaction,\n-\t\t\t\t struct odb_write_stream *stream, size_t len,\n+\t\t\t\t struct odb_write_stream *stream,\n \t\t\t\t struct object_id *oid);\n \n \t/*\n@@ -82,7 +82,7 @@ int odb_transaction_commit(struct odb_transaction *transaction);\n */\n int odb_transaction_write_object_stream(struct odb_transaction *transaction,\n \t\t\t\t\tstruct odb_write_stream *stream,\n-\t\t\t\t\tsize_t len, struct object_id *oid);\n+\t\t\t\t\tstruct object_id *oid);\n \n /*\n * Populates the provided strvec with the environment variables that a child\ndiff --git a/t/unit-tests/u-odb-inmemory.c b/t/unit-tests/u-odb-inmemory.c\nindex ddf2db5c81..5ccc52dccc 100644\n--- a/t/unit-tests/u-odb-inmemory.c\n+++ b/t/unit-tests/u-odb-inmemory.c\n@@ -269,7 +269,6 @@ struct membuf_write_stream {\n \tstruct odb_write_stream base;\n \tconst char *buf;\n \tsize_t offset;\n-\tsize_t size;\n };\n \n static ssize_t membuf_write_stream_read(struct odb_write_stream *stream,\n@@ -280,13 +279,13 @@ static ssize_t membuf_write_stream_read(struct odb_write_stream *stream,\n \n \tif (chunk_size > len)\n \t\tchunk_size = len;\n-\tif (chunk_size > s->size - s->offset)\n-\t\tchunk_size = s->size - s->offset;\n+\tif (chunk_size > s->base.size - s->offset)\n+\t\tchunk_size = s->base.size - s->offset;\n \n \tmemcpy(buf, s->buf + s->offset, chunk_size);\n \n \ts->offset += chunk_size;\n-\tif (s->offset == s->size)\n+\tif (s->offset == s->base.size)\n \t\ts->base.is_finished = 1;\n \n \treturn chunk_size;\n@@ -298,13 +297,13 @@ void test_odb_inmemory__write_object_stream(void)\n \tconst char data[] = \"foobar\";\n \tstruct membuf_write_stream stream = {\n \t\t.base.read = membuf_write_stream_read,\n+\t\t.base.size = strlen(data),\n \t\t.buf = data,\n-\t\t.size = strlen(data),\n \t};\n \tstruct object_id written_oid;\n \n \tcl_must_pass(odb_source_write_object_stream(&source->base, &stream.base,\n-\t\t\t\t\t\t strlen(data), &written_oid));\n+\t\t\t\t\t\t &written_oid));\n \tcl_assert_equal_s(oid_to_hex(&written_oid), FOOBAR_OID);\n \tcl_assert_object_info(source, &written_oid, OBJ_BLOB, \"foobar\");\n \n\n-- \n2.55.0.679.g6767b8d81c.dirty\n\n"},{"id":"549650","messageId":"20260805-pks-odb-stream-unification-v2-2-b8c369564641@pks.im","threadId":"66111","inReplyTo":"20260805-pks-odb-stream-unification-v2-0-b8c369564641@pks.im","subject":"[PATCH v2 2/8] odb/streaming: drop `is_finished` field","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-05T07:44:46Z","receivedAt":"2026-08-05T07:45:01Z","isPatch":true,"body":"The `is_finished` field is used to track whether a write stream is done\nwriting all of its data. Tracking this field as part of the stream\nitself shouldn't be required though: callers will already know when the\nstream is done when the stream's read function returns zero bytes, same\nas when reading from a file descriptor.\n\nThere is one exception where it gets a bit more complicated: when\nconsuming data in \"builtin/unpack-objects.c\" it may happen that we don't\nyield any new bytes after reading from the pipe. This is addressed by\nlooping until we have produced at least a single byte of output.\n\nDrop the field from `struct odb_write_stream`. Again, same as in the\npreceding commit, this brings the structure a bit closer to its sibling\n`struct odb_read_stream`.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/unpack-objects.c | 15 ++++++++-------\n object-file.c | 13 ++++++++-----\n odb/source-inmemory.c | 9 ++++++++-\n odb/source-loose.c | 12 ++++++++----\n odb/streaming.c | 5 +----\n odb/streaming.h | 1 -\n t/unit-tests/u-odb-inmemory.c | 5 +++--\n 7 files changed, 36 insertions(+), 24 deletions(-)\n\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex f3e0b504f4..b7c486ea94 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -368,20 +368,20 @@ static ssize_t feed_input_zstream(struct odb_write_stream *in_stream,\n {\n \tstruct input_zstream_data *data = in_stream->data;\n \tgit_zstream *zstream = data->zstream;\n-\tvoid *in = fill(1);\n \n-\tif (in_stream->is_finished)\n+\tif (data->status != Z_OK)\n \t\treturn 0;\n \n \tzstream->next_out = buf;\n \tzstream->avail_out = buf_len;\n-\tzstream->next_in = in;\n-\tzstream->avail_in = len;\n \n-\tdata->status = git_inflate(zstream, 0);\n+\twhile (data->status == Z_OK && zstream->avail_out == buf_len) {\n+\t\tzstream->next_in = fill(1);\n+\t\tzstream->avail_in = len;\n+\t\tdata->status = git_inflate(zstream, 0);\n+\t\tuse(len - zstream->avail_in);\n+\t}\n \n-\tin_stream->is_finished = data->status != Z_OK;\n-\tuse(len - zstream->avail_in);\n \treturn buf_len - zstream->avail_out;\n }\n \n@@ -397,6 +397,7 @@ static void stream_blob(unsigned long size, unsigned nr)\n \tstruct obj_info *info = &obj_list[nr];\n \n \tdata.zstream = &zstream;\n+\tdata.status = Z_OK;\n \tgit_inflate_init(&zstream);\n \n \tif (odb_write_object_stream(the_repository->objects, &in_stream, &info->oid))\ndiff --git a/object-file.c b/object-file.c\nindex b196abb596..317c09dff8 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -716,12 +716,13 @@ static int hash_blob_stream(struct odb_write_stream *stream,\n \tgit_hash_init(&ctx, hash_algo);\n \tgit_hash_update(&ctx, buf, header_len);\n \n-\twhile (!stream->is_finished) {\n+\twhile (1) {\n \t\tssize_t read_result = odb_write_stream_read(stream, buf,\n \t\t\t\t\t\t\t sizeof(buf));\n-\n \t\tif (read_result < 0)\n \t\t\treturn -1;\n+\t\tif (!read_result)\n+\t\t\tbreak;\n \n \t\tgit_hash_update(&ctx, buf, read_result);\n \t\tbytes_hashed += read_result;\n@@ -749,6 +750,7 @@ static void stream_blob_to_pack(struct transaction_packfile *state,\n \tunsigned hdrlen;\n \tint status = Z_OK;\n \tstruct repo_config_values *cfg = repo_config_values(the_repository);\n+\tbool is_finished = false;\n \tsize_t bytes_read = 0;\n \n \tgit_deflate_init(&s, cfg->pack_compression_level);\n@@ -758,12 +760,13 @@ static void stream_blob_to_pack(struct transaction_packfile *state,\n \ts.avail_out = sizeof(obuf) - hdrlen;\n \n \twhile (status != Z_STREAM_END) {\n-\t\tif (!stream->is_finished && !s.avail_in) {\n+\t\tif (!is_finished && !s.avail_in) {\n \t\t\tssize_t rsize = odb_write_stream_read(stream, ibuf,\n \t\t\t\t\t\t\t sizeof(ibuf));\n-\n \t\t\tif (rsize < 0)\n \t\t\t\tdie(\"failed to read blob data\");\n+\t\t\tif (!rsize)\n+\t\t\t\tis_finished = true;\n \n \t\t\tgit_hash_update(ctx, ibuf, rsize);\n \n@@ -772,7 +775,7 @@ static void stream_blob_to_pack(struct transaction_packfile *state,\n \t\t\tbytes_read += rsize;\n \t\t}\n \n-\t\tstatus = git_deflate(&s, stream->is_finished ? Z_FINISH : 0);\n+\t\tstatus = git_deflate(&s, is_finished ? Z_FINISH : 0);\n \n \t\tif (!s.avail_out || status == Z_STREAM_END) {\n \t\t\tsize_t written = s.next_out - obuf;\ndiff --git a/odb/source-inmemory.c b/odb/source-inmemory.c\nindex 398131e194..01bb81c63c 100644\n--- a/odb/source-inmemory.c\n+++ b/odb/source-inmemory.c\n@@ -265,10 +265,17 @@ static int odb_source_inmemory_write_object_stream(struct odb_source *source,\n \tint ret;\n \n \tCALLOC_ARRAY(data, stream->size);\n-\twhile (!stream->is_finished) {\n+\twhile (1) {\n \t\tssize_t bytes_read;\n \n \t\tbytes_read = odb_write_stream_read(stream, buf, sizeof(buf));\n+\t\tif (bytes_read < 0) {\n+\t\t\tret = error(\"failed to read object stream\");\n+\t\t\tgoto out;\n+\t\t}\n+\t\tif (!bytes_read)\n+\t\t\tbreak;\n+\n \t\tif (total_read + bytes_read > stream->size) {\n \t\t\tret = error(\"object stream yielded more bytes than expected\");\n \t\t\tgoto out;\ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex 77a2adb52a..361b4e2a2a 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -859,6 +859,7 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,\n \tstruct strbuf filename = STRBUF_INIT;\n \tunsigned char buf[8192];\n \tint dirlen;\n+\tbool is_finished = false;\n \tchar hdr[MAX_HEADER_LEN];\n \tint hdrlen;\n \n@@ -889,7 +890,7 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,\n \tdo {\n \t\tunsigned char *in0 = stream.next_in;\n \n-\t\tif (!stream.avail_in && !in_stream->is_finished) {\n+\t\tif (!stream.avail_in && !is_finished) {\n \t\t\tssize_t read_len = odb_write_stream_read(in_stream, buf,\n \t\t\t\t\t\t\t\t sizeof(buf));\n \t\t\tif (read_len < 0) {\n@@ -898,12 +899,15 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,\n \t\t\t\tgoto cleanup;\n \t\t\t}\n \n+\t\t\t/* All data has been read. */\n+\t\t\tif (!read_len) {\n+\t\t\t\tis_finished = true;\n+\t\t\t\tflush = 1;\n+\t\t\t}\n+\n \t\t\tstream.avail_in = read_len;\n \t\t\tstream.next_in = buf;\n \t\t\tin0 = buf;\n-\t\t\t/* All data has been read. */\n-\t\t\tif (in_stream->is_finished)\n-\t\t\t\tflush = 1;\n \t\t}\n \t\tret = write_loose_object_common(loose, &c, &compat_c, &stream, flush, in0, fd,\n \t\t\t\t\t\tcompressed, sizeof(compressed));\ndiff --git a/odb/streaming.c b/odb/streaming.c\nindex 38c2f6687c..912e75e682 100644\n--- a/odb/streaming.c\n+++ b/odb/streaming.c\n@@ -310,7 +310,7 @@ static ssize_t read_object_fd(struct odb_write_stream *stream,\n \tssize_t read_result;\n \tsize_t count;\n \n-\tif (stream->is_finished)\n+\tif (!data->remaining)\n \t\treturn 0;\n \n \tcount = data->remaining < len ? data->remaining : len;\n@@ -319,8 +319,6 @@ static ssize_t read_object_fd(struct odb_write_stream *stream,\n \t\treturn -1;\n \n \tdata->remaining -= count;\n-\tif (!data->remaining)\n-\t\tstream->is_finished = 1;\n \n \treturn read_result;\n }\n@@ -337,5 +335,4 @@ void odb_write_stream_from_fd(struct odb_write_stream *stream, int fd,\n \tstream->data = data;\n \tstream->read = read_object_fd;\n \tstream->size = size;\n-\tstream->is_finished = 0;\n }\ndiff --git a/odb/streaming.h b/odb/streaming.h\nindex 4d7d31b5aa..5e8e6e532e 100644\n--- a/odb/streaming.h\n+++ b/odb/streaming.h\n@@ -56,7 +56,6 @@ struct odb_write_stream {\n \tssize_t (*read)(struct odb_write_stream *, unsigned char *, size_t);\n \tvoid *data;\n \tsize_t size;\n-\tint is_finished;\n };\n \n /*\ndiff --git a/t/unit-tests/u-odb-inmemory.c b/t/unit-tests/u-odb-inmemory.c\nindex 5ccc52dccc..4437140ed0 100644\n--- a/t/unit-tests/u-odb-inmemory.c\n+++ b/t/unit-tests/u-odb-inmemory.c\n@@ -277,6 +277,9 @@ static ssize_t membuf_write_stream_read(struct odb_write_stream *stream,\n \tstruct membuf_write_stream *s = container_of(stream, struct membuf_write_stream, base);\n \tsize_t chunk_size = 2;\n \n+\tif (s->offset == s->base.size)\n+\t\treturn 0;\n+\n \tif (chunk_size > len)\n \t\tchunk_size = len;\n \tif (chunk_size > s->base.size - s->offset)\n@@ -285,8 +288,6 @@ static ssize_t membuf_write_stream_read(struct odb_write_stream *stream,\n \tmemcpy(buf, s->buf + s->offset, chunk_size);\n \n \ts->offset += chunk_size;\n-\tif (s->offset == s->base.size)\n-\t\ts->base.is_finished = 1;\n \n \treturn chunk_size;\n }\n\n-- \n2.55.0.679.g6767b8d81c.dirty\n\n"},{"id":"549651","messageId":"20260805-pks-odb-stream-unification-v2-3-b8c369564641@pks.im","threadId":"66111","inReplyTo":"20260805-pks-odb-stream-unification-v2-0-b8c369564641@pks.im","subject":"[PATCH v2 3/8] odb/streaming: support streaming arbitrary object types","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-05T07:44:47Z","receivedAt":"2026-08-05T07:45:04Z","isPatch":true,"body":"The object database supports the ability to write object streams into\nit. This functionality is used when we encounter a blob that is larger\nthan \"core.bigFileThreshold\" so that we don't have to soak large files\ninto memory.\n\nAs we only ever write large files, the infrastructure doesn't support\nspecifying any other object type than \"blob\". This limitation is quite\nartificial though: there is no reason why we shouldn't support writing\narbitrary large objects with a stream. While it's very unlikely that we\nencounter a huge object other than a blob, users are known to be\ncreative and sometimes like to inflict pain on themselves by creating\ncommits or trees that are huge.\n\nExtend the infrastructure to support streaming arbitrary object types.\nFor now we don't use this functionality anywhere, but it brings us a bit\ncloser to unify `struct odb_read_stream` and `struct odb_write_stream`.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/unpack-objects.c | 1 +\n object-file.c | 31 +++++++++++++++----------------\n odb/source-inmemory.c | 5 +++--\n odb/source-loose.c | 2 +-\n odb/streaming.c | 3 ++-\n odb/streaming.h | 3 ++-\n odb/transaction.h | 2 +-\n t/unit-tests/u-odb-inmemory.c | 7 +++++--\n 8 files changed, 30 insertions(+), 24 deletions(-)\n\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex b7c486ea94..7439ec53be 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -393,6 +393,7 @@ static void stream_blob(unsigned long size, unsigned nr)\n \t\t.read = feed_input_zstream,\n \t\t.data = &data,\n \t\t.size = size,\n+\t\t.type = OBJ_BLOB,\n \t};\n \tstruct obj_info *info = &obj_list[nr];\n \ndiff --git a/object-file.c b/object-file.c\nindex 317c09dff8..699a6a008c 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -702,9 +702,9 @@ static void prepare_packfile_transaction(struct odb_transaction_files *transacti\n \t\tdie_errno(\"unable to write pack header\");\n }\n \n-static int hash_blob_stream(struct odb_write_stream *stream,\n-\t\t\t const struct git_hash_algo *hash_algo,\n-\t\t\t struct object_id *result_oid)\n+static int hash_stream(struct odb_write_stream *stream,\n+\t\t const struct git_hash_algo *hash_algo,\n+\t\t struct object_id *result_oid)\n {\n \tunsigned char buf[16384];\n \tstruct git_hash_ctx ctx;\n@@ -712,7 +712,7 @@ static int hash_blob_stream(struct odb_write_stream *stream,\n \tsize_t bytes_hashed = 0;\n \n \theader_len = format_object_header((char *)buf, sizeof(buf),\n-\t\t\t\t\t OBJ_BLOB, stream->size);\n+\t\t\t\t\t stream->type, stream->size);\n \tgit_hash_init(&ctx, hash_algo);\n \tgit_hash_update(&ctx, buf, header_len);\n \n@@ -740,9 +740,9 @@ static int hash_blob_stream(struct odb_write_stream *stream,\n * Read the contents from the stream provided, streaming it to the\n * packfile in state while updating the hash in ctx.\n */\n-static void stream_blob_to_pack(struct transaction_packfile *state,\n-\t\t\t\tstruct git_hash_ctx *ctx,\n-\t\t\t\tstruct odb_write_stream *stream)\n+static void stream_to_pack(struct transaction_packfile *state,\n+\t\t\t struct git_hash_ctx *ctx,\n+\t\t\t struct odb_write_stream *stream)\n {\n \tgit_zstream s;\n \tunsigned char ibuf[16384];\n@@ -755,7 +755,7 @@ static void stream_blob_to_pack(struct transaction_packfile *state,\n \n \tgit_deflate_init(&s, cfg->pack_compression_level);\n \n-\thdrlen = encode_in_pack_object_header(obuf, sizeof(obuf), OBJ_BLOB, stream->size);\n+\thdrlen = encode_in_pack_object_header(obuf, sizeof(obuf), stream->type, stream->size);\n \ts.next_out = obuf + hdrlen;\n \ts.avail_out = sizeof(obuf) - hdrlen;\n \n@@ -764,7 +764,7 @@ static void stream_blob_to_pack(struct transaction_packfile *state,\n \t\t\tssize_t rsize = odb_write_stream_read(stream, ibuf,\n \t\t\t\t\t\t\t sizeof(ibuf));\n \t\t\tif (rsize < 0)\n-\t\t\t\tdie(\"failed to read blob data\");\n+\t\t\t\tdie(\"failed to read object data\");\n \t\t\tif (!rsize)\n \t\t\t\tis_finished = true;\n \n@@ -797,7 +797,7 @@ static void stream_blob_to_pack(struct transaction_packfile *state,\n \t}\n \n \tif (bytes_read != stream->size)\n-\t\tdie(\"read %\" PRIuMAX \" bytes of blob data, but expected %\" PRIuMAX \" bytes\",\n+\t\tdie(\"read %\" PRIuMAX \" bytes of object data, but expected %\" PRIuMAX \" bytes\",\n \t\t (uintmax_t)bytes_read, (uintmax_t)stream->size);\n \n \tgit_deflate_end(&s);\n@@ -868,7 +868,7 @@ static void flush_packfile_transaction(struct odb_transaction_files *transaction\n * result, which we need to know beforehand when writing a git object.\n * Since the primary motivation for trying to stream from the working\n * tree file and to avoid mmaping it in core is to deal with large\n- * binary blobs, they generally do not want to get any conversion, and\n+ * objects, they generally do not want to get any conversion, and\n * callers should avoid this code path when filters are requested.\n */\n static int odb_transaction_files_write_object_stream(struct odb_transaction *base,\n@@ -886,7 +886,7 @@ static int odb_transaction_files_write_object_stream(struct odb_transaction *bas\n \tstruct pack_idx_entry *idx;\n \n \theader_len = format_object_header((char *)obuf, sizeof(obuf),\n-\t\t\t\t\t OBJ_BLOB, stream->size);\n+\t\t\t\t\t stream->type, stream->size);\n \tgit_hash_init(&ctx, transaction->base.source->odb->repo->hash_algo);\n \tgit_hash_update(&ctx, obuf, header_len);\n \n@@ -911,7 +911,7 @@ static int odb_transaction_files_write_object_stream(struct odb_transaction *bas\n \thashfile_checkpoint(state->f, &checkpoint);\n \tidx->offset = state->offset;\n \tcrc32_begin(state->f);\n-\tstream_blob_to_pack(state, &ctx, stream);\n+\tstream_to_pack(state, &ctx, stream);\n \tgit_hash_final_oid(result_oid, &ctx);\n \n \tidx->crc32 = crc32_end(state->f);\n@@ -953,7 +953,7 @@ int index_fd(struct index_state *istate, struct object_id *oid,\n \t\t\t\t type, path, flags);\n \t} else {\n \t\tstruct odb_write_stream stream;\n-\t\todb_write_stream_from_fd(&stream, fd, xsize_t(st->st_size));\n+\t\todb_write_stream_from_fd(&stream, fd, xsize_t(st->st_size), OBJ_BLOB);\n \n \t\tif (flags & INDEX_WRITE_OBJECT) {\n \t\t\tstruct object_database *odb = the_repository->objects;\n@@ -968,8 +968,7 @@ int index_fd(struct index_state *istate, struct object_id *oid,\n \t\t\tif (!inflight)\n \t\t\t\todb_transaction_commit(transaction);\n \t\t} else {\n-\t\t\tret = hash_blob_stream(&stream,\n-\t\t\t\t\t the_repository->hash_algo, oid);\n+\t\t\tret = hash_stream(&stream, the_repository->hash_algo, oid);\n \t\t}\n \n \t\todb_write_stream_release(&stream);\ndiff --git a/odb/source-inmemory.c b/odb/source-inmemory.c\nindex 01bb81c63c..139618024a 100644\n--- a/odb/source-inmemory.c\n+++ b/odb/source-inmemory.c\n@@ -290,10 +290,11 @@ static int odb_source_inmemory_write_object_stream(struct odb_source *source,\n \t\tgoto out;\n \t}\n \n-\thash_object_file(source->odb->repo->hash_algo, data, total_read, OBJ_BLOB, oid);\n+\thash_object_file(source->odb->repo->hash_algo, data, total_read,\n+\t\t\t stream->type, oid);\n \n \tret = odb_source_inmemory_write_object(source, data, stream->size,\n-\t\t\t\t\t OBJ_BLOB, oid, NULL, NULL, 0);\n+\t\t\t\t\t stream->type, oid, NULL, NULL, 0);\n \tif (ret < 0)\n \t\tgoto out;\n \ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex 361b4e2a2a..5681a38f03 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -868,7 +868,7 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,\n \n \t/* Since oid is not determined, save tmp file to odb path. */\n \tstrbuf_addf(&filename, \"%s/\", loose->base.path);\n-\thdrlen = format_object_header(hdr, sizeof(hdr), OBJ_BLOB, in_stream->size);\n+\thdrlen = format_object_header(hdr, sizeof(hdr), in_stream->type, in_stream->size);\n \n \t/*\n \t * Common steps for write_loose_object and stream_loose_object to\ndiff --git a/odb/streaming.c b/odb/streaming.c\nindex 912e75e682..0918cad426 100644\n--- a/odb/streaming.c\n+++ b/odb/streaming.c\n@@ -324,7 +324,7 @@ static ssize_t read_object_fd(struct odb_write_stream *stream,\n }\n \n void odb_write_stream_from_fd(struct odb_write_stream *stream, int fd,\n-\t\t\t size_t size)\n+\t\t\t size_t size, enum object_type type)\n {\n \tstruct read_object_fd_data *data;\n \n@@ -335,4 +335,5 @@ void odb_write_stream_from_fd(struct odb_write_stream *stream, int fd,\n \tstream->data = data;\n \tstream->read = read_object_fd;\n \tstream->size = size;\n+\tstream->type = type;\n }\ndiff --git a/odb/streaming.h b/odb/streaming.h\nindex 5e8e6e532e..3c8ed55129 100644\n--- a/odb/streaming.h\n+++ b/odb/streaming.h\n@@ -56,6 +56,7 @@ struct odb_write_stream {\n \tssize_t (*read)(struct odb_write_stream *, unsigned char *, size_t);\n \tvoid *data;\n \tsize_t size;\n+\tenum object_type type;\n };\n \n /*\n@@ -92,6 +93,6 @@ int odb_stream_blob_to_fd(struct object_database *odb,\n * Sets up an ODB write stream that reads from an fd.\n */\n void odb_write_stream_from_fd(struct odb_write_stream *stream, int fd,\n-\t\t\t size_t size);\n+\t\t\t size_t size, enum object_type type);\n \n #endif /* STREAMING_H */\ndiff --git a/odb/transaction.h b/odb/transaction.h\nindex ffb279314c..1eb74664c6 100644\n--- a/odb/transaction.h\n+++ b/odb/transaction.h\n@@ -24,7 +24,7 @@ struct odb_transaction {\n \n \t/*\n \t * This callback is expected to write the given object stream into\n-\t * the ODB transaction. Note that for now, only blobs support streaming.\n+\t * the ODB transaction.\n \t *\n \t * The resulting object ID shall be written into the out pointer. The\n \t * callback is expected to return 0 on success, a negative error code\ndiff --git a/t/unit-tests/u-odb-inmemory.c b/t/unit-tests/u-odb-inmemory.c\nindex 4437140ed0..1ab07af6d6 100644\n--- a/t/unit-tests/u-odb-inmemory.c\n+++ b/t/unit-tests/u-odb-inmemory.c\n@@ -297,8 +297,11 @@ void test_odb_inmemory__write_object_stream(void)\n \tstruct odb_source_inmemory *source = odb_source_inmemory_new(odb);\n \tconst char data[] = \"foobar\";\n \tstruct membuf_write_stream stream = {\n-\t\t.base.read = membuf_write_stream_read,\n-\t\t.base.size = strlen(data),\n+\t\t.base = {\n+\t\t\t.read = membuf_write_stream_read,\n+\t\t\t.size = strlen(data),\n+\t\t\t.type = OBJ_BLOB,\n+\t\t},\n \t\t.buf = data,\n \t};\n \tstruct object_id written_oid;\n\n-- \n2.55.0.679.g6767b8d81c.dirty\n\n"},{"id":"549652","messageId":"20260805-pks-odb-stream-unification-v2-4-b8c369564641@pks.im","threadId":"66111","inReplyTo":"20260805-pks-odb-stream-unification-v2-0-b8c369564641@pks.im","subject":"[PATCH v2 4/8] odb/streaming: rename `struct odb_read_stream`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-05T07:44:48Z","receivedAt":"2026-08-05T07:45:07Z","isPatch":true,"body":"Rename `struct odb_read_stream` to just `struct odb_stream`. This\nprepares for unification of the two different types of streams, as these\nprovide the same functionality with the preceding refactorings.\n\nSigned-off-by: Patrick Steinhardt \n---\n archive-tar.c | 6 +++---\n archive-zip.c | 10 ++++-----\n builtin/index-pack.c | 6 +++---\n builtin/pack-objects.c | 14 ++++++-------\n object-file.c | 4 ++--\n object-file.h | 2 +-\n object.c | 6 +++---\n odb/source-files.c | 2 +-\n odb/source-inmemory.c | 8 ++++----\n odb/source-loose.c | 8 ++++----\n odb/source-packed.c | 2 +-\n odb/source.h | 6 +++---\n odb/streaming.c | 48 +++++++++++++++++++++----------------------\n odb/streaming.h | 24 +++++++++++-----------\n pack-check.c | 4 ++--\n packfile.c | 8 ++++----\n packfile.h | 4 ++--\n t/unit-tests/u-odb-inmemory.c | 12 +++++------\n 18 files changed, 87 insertions(+), 87 deletions(-)\n\ndiff --git a/archive-tar.c b/archive-tar.c\nindex 0fc70d13a8..df2d7fb8e9 100644\n--- a/archive-tar.c\n+++ b/archive-tar.c\n@@ -129,7 +129,7 @@ static void write_trailer(void)\n */\n static int stream_blocked(struct repository *r, const struct object_id *oid)\n {\n-\tstruct odb_read_stream *st;\n+\tstruct odb_stream *st;\n \tchar buf[BLOCKSIZE];\n \tssize_t readlen;\n \n@@ -137,12 +137,12 @@ static int stream_blocked(struct repository *r, const struct object_id *oid)\n \tif (!st)\n \t\treturn error(_(\"cannot stream blob %s\"), oid_to_hex(oid));\n \tfor (;;) {\n-\t\treadlen = odb_read_stream_read(st, buf, sizeof(buf));\n+\t\treadlen = odb_stream_read(st, buf, sizeof(buf));\n \t\tif (readlen <= 0)\n \t\t\tbreak;\n \t\tdo_write_blocked(buf, readlen);\n \t}\n-\todb_read_stream_close(st);\n+\todb_stream_close(st);\n \tif (!readlen)\n \t\tfinish_record();\n \treturn readlen;\ndiff --git a/archive-zip.c b/archive-zip.c\nindex 97ea8d60d6..8095fd04d5 100644\n--- a/archive-zip.c\n+++ b/archive-zip.c\n@@ -309,7 +309,7 @@ static int write_zip_entry(struct archiver_args *args,\n \tenum zip_method method;\n \tunsigned char *out;\n \tvoid *deflated = NULL;\n-\tstruct odb_read_stream *stream = NULL;\n+\tstruct odb_stream *stream = NULL;\n \tunsigned long flags = 0;\n \tint is_binary = -1;\n \tconst char *path_without_prefix = path + args->baselen;\n@@ -428,7 +428,7 @@ static int write_zip_entry(struct archiver_args *args,\n \t\tssize_t readlen;\n \n \t\tfor (;;) {\n-\t\t\treadlen = odb_read_stream_read(stream, buf, sizeof(buf));\n+\t\t\treadlen = odb_stream_read(stream, buf, sizeof(buf));\n \t\t\tif (readlen <= 0)\n \t\t\t\tbreak;\n \t\t\tcrc = crc32(crc, buf, readlen);\n@@ -438,7 +438,7 @@ static int write_zip_entry(struct archiver_args *args,\n \t\t\t\t\t\t\t buf, readlen);\n \t\t\twrite_or_die(1, buf, readlen);\n \t\t}\n-\t\todb_read_stream_close(stream);\n+\t\todb_stream_close(stream);\n \t\tif (readlen)\n \t\t\treturn readlen;\n \n@@ -461,7 +461,7 @@ static int write_zip_entry(struct archiver_args *args,\n \t\tzstream.avail_out = sizeof(compressed);\n \n \t\tfor (;;) {\n-\t\t\treadlen = odb_read_stream_read(stream, buf, sizeof(buf));\n+\t\t\treadlen = odb_stream_read(stream, buf, sizeof(buf));\n \t\t\tif (readlen <= 0)\n \t\t\t\tbreak;\n \t\t\tcrc = crc32(crc, buf, readlen);\n@@ -485,7 +485,7 @@ static int write_zip_entry(struct archiver_args *args,\n \t\t\t}\n \n \t\t}\n-\t\todb_read_stream_close(stream);\n+\t\todb_stream_close(stream);\n \t\tif (readlen)\n \t\t\treturn readlen;\n \ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex bc86925ad0..7226da3e65 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -763,7 +763,7 @@ static void find_ref_delta_children(const struct object_id *oid,\n \n struct compare_data {\n \tstruct object_entry *entry;\n-\tstruct odb_read_stream *st;\n+\tstruct odb_stream *st;\n \tunsigned char *buf;\n \tunsigned long buf_size;\n };\n@@ -780,7 +780,7 @@ static int compare_objects(const unsigned char *buf, unsigned long size,\n \t}\n \n \twhile (size) {\n-\t\tssize_t len = odb_read_stream_read(data->st, data->buf, size);\n+\t\tssize_t len = odb_stream_read(data->st, data->buf, size);\n \t\tif (len == 0)\n \t\t\tdie(_(\"SHA1 COLLISION FOUND WITH %s !\"),\n \t\t\t oid_to_hex(&data->entry->idx.oid));\n@@ -813,7 +813,7 @@ static int check_collison(struct object_entry *entry)\n \t\tdie(_(\"SHA1 COLLISION FOUND WITH %s !\"),\n \t\t oid_to_hex(&entry->idx.oid));\n \tunpack_data(entry, compare_objects, &data);\n-\todb_read_stream_close(data.st);\n+\todb_stream_close(data.st);\n \tfree(data.buf);\n \treturn 0;\n }\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 1ec5b6f206..683160c6bb 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -411,7 +411,7 @@ static unsigned long do_compress(void **pptr, unsigned long size)\n \treturn stream.total_out;\n }\n \n-static unsigned long write_large_blob_data(struct odb_read_stream *st, struct hashfile *f,\n+static unsigned long write_large_blob_data(struct odb_stream *st, struct hashfile *f,\n \t\t\t\t\t const struct object_id *oid)\n {\n \tgit_zstream stream;\n@@ -425,7 +425,7 @@ static unsigned long write_large_blob_data(struct odb_read_stream *st, struct ha\n \tfor (;;) {\n \t\tssize_t readlen;\n \t\tint zret = Z_OK;\n-\t\treadlen = odb_read_stream_read(st, ibuf, sizeof(ibuf));\n+\t\treadlen = odb_stream_read(st, ibuf, sizeof(ibuf));\n \t\tif (readlen == -1)\n \t\t\tdie(_(\"unable to read %s\"), oid_to_hex(oid));\n \n@@ -521,7 +521,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \tunsigned hdrlen;\n \tenum object_type type;\n \tvoid *buf;\n-\tstruct odb_read_stream *st = NULL;\n+\tstruct odb_stream *st = NULL;\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n \n \tif (!usable_delta) {\n@@ -589,7 +589,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\t\tdheader[--pos] = 128 | (--ofs & 127);\n \t\tif (limit && hdrlen + sizeof(dheader) - pos + datalen + hashsz >= limit) {\n \t\t\tif (st)\n-\t\t\t\todb_read_stream_close(st);\n+\t\t\t\todb_stream_close(st);\n \t\t\tfree(buf);\n \t\t\treturn 0;\n \t\t}\n@@ -603,7 +603,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\t */\n \t\tif (limit && hdrlen + hashsz + datalen + hashsz >= limit) {\n \t\t\tif (st)\n-\t\t\t\todb_read_stream_close(st);\n+\t\t\t\todb_stream_close(st);\n \t\t\tfree(buf);\n \t\t\treturn 0;\n \t\t}\n@@ -613,7 +613,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t} else {\n \t\tif (limit && hdrlen + datalen + hashsz >= limit) {\n \t\t\tif (st)\n-\t\t\t\todb_read_stream_close(st);\n+\t\t\t\todb_stream_close(st);\n \t\t\tfree(buf);\n \t\t\treturn 0;\n \t\t}\n@@ -621,7 +621,7 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t}\n \tif (st) {\n \t\tdatalen = write_large_blob_data(st, f, &entry->idx.oid);\n-\t\todb_read_stream_close(st);\n+\t\todb_stream_close(st);\n \t} else {\n \t\thashwrite(f, buf, datalen);\n \t\tfree(buf);\ndiff --git a/object-file.c b/object-file.c\nindex 699a6a008c..5f6d584c35 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -122,7 +122,7 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n }\n \n int stream_object_signature(struct repository *r,\n-\t\t\t struct odb_read_stream *st,\n+\t\t\t struct odb_stream *st,\n \t\t\t const struct object_id *oid)\n {\n \tstruct object_id real_oid;\n@@ -138,7 +138,7 @@ int stream_object_signature(struct repository *r,\n \tgit_hash_update(&c, hdr, hdrlen);\n \tfor (;;) {\n \t\tchar buf[1024 * 16];\n-\t\tssize_t readlen = odb_read_stream_read(st, buf, sizeof(buf));\n+\t\tssize_t readlen = odb_stream_read(st, buf, sizeof(buf));\n \t\tif (readlen < 0)\n \t\t\treturn -1;\n \t\tif (!readlen)\ndiff --git a/object-file.h b/object-file.h\nindex 805f2cfa28..f44758c4f8 100644\n--- a/object-file.h\n+++ b/object-file.h\n@@ -101,7 +101,7 @@ int check_object_signature(struct repository *r, const struct object_id *oid,\n * the streaming interface and rehash it to do the same.\n */\n int stream_object_signature(struct repository *r,\n-\t\t\t struct odb_read_stream *stream,\n+\t\t\t struct odb_stream *stream,\n \t\t\t const struct object_id *oid);\n \n enum finalize_object_file_flags {\ndiff --git a/object.c b/object.c\nindex 23b84aa7e2..37e6efee47 100644\n--- a/object.c\n+++ b/object.c\n@@ -345,7 +345,7 @@ struct object *parse_object_with_flags(struct repository *r,\n \tif ((!obj || obj->type == OBJ_NONE || obj->type == OBJ_BLOB) &&\n \t odb_read_object_info(r->objects, oid, NULL) == OBJ_BLOB) {\n \t\tif (!skip_hash) {\n-\t\t\tstruct odb_read_stream *stream = odb_read_stream_open(r->objects, oid, NULL);\n+\t\t\tstruct odb_stream *stream = odb_read_stream_open(r->objects, oid, NULL);\n \n \t\t\tif (!stream) {\n \t\t\t\terror(_(\"unable to open object stream for %s\"), oid_to_hex(oid));\n@@ -354,11 +354,11 @@ struct object *parse_object_with_flags(struct repository *r,\n \n \t\t\tif (stream_object_signature(r, stream, repl) < 0) {\n \t\t\t\terror(_(\"hash mismatch %s\"), oid_to_hex(oid));\n-\t\t\t\todb_read_stream_close(stream);\n+\t\t\t\todb_stream_close(stream);\n \t\t\t\treturn NULL;\n \t\t\t}\n \n-\t\t\todb_read_stream_close(stream);\n+\t\t\todb_stream_close(stream);\n \t\t}\n \t\tparse_blob_buffer(lookup_blob(r, oid));\n \t\treturn lookup_object(r, oid);\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex f51960bd71..f7b8c76549 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -63,7 +63,7 @@ static int odb_source_files_read_object_info(struct odb_source *source,\n \treturn -1;\n }\n \n-static int odb_source_files_read_object_stream(struct odb_read_stream **out,\n+static int odb_source_files_read_object_stream(struct odb_stream **out,\n \t\t\t\t\t struct odb_source *source,\n \t\t\t\t\t const struct object_id *oid)\n {\ndiff --git a/odb/source-inmemory.c b/odb/source-inmemory.c\nindex 139618024a..485d587036 100644\n--- a/odb/source-inmemory.c\n+++ b/odb/source-inmemory.c\n@@ -73,12 +73,12 @@ static int odb_source_inmemory_read_object_info(struct odb_source *source,\n }\n \n struct odb_read_stream_inmemory {\n-\tstruct odb_read_stream base;\n+\tstruct odb_stream base;\n \tconst unsigned char *buf;\n \tsize_t offset;\n };\n \n-static ssize_t odb_read_stream_inmemory_read(struct odb_read_stream *stream,\n+static ssize_t odb_read_stream_inmemory_read(struct odb_stream *stream,\n \t\t\t\t\t char *buf, size_t buf_len)\n {\n \tstruct odb_read_stream_inmemory *inmemory =\n@@ -94,12 +94,12 @@ static ssize_t odb_read_stream_inmemory_read(struct odb_read_stream *stream,\n \treturn bytes;\n }\n \n-static int odb_read_stream_inmemory_close(struct odb_read_stream *stream UNUSED)\n+static int odb_read_stream_inmemory_close(struct odb_stream *stream UNUSED)\n {\n \treturn 0;\n }\n \n-static int odb_source_inmemory_read_object_stream(struct odb_read_stream **out,\n+static int odb_source_inmemory_read_object_stream(struct odb_stream **out,\n \t\t\t\t\t\t struct odb_source *source,\n \t\t\t\t\t\t const struct object_id *oid)\n {\ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex 5681a38f03..038defd905 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -278,7 +278,7 @@ static void *odb_source_loose_map_object(struct odb_source_loose *loose,\n }\n \n struct odb_loose_read_stream {\n-\tstruct odb_read_stream base;\n+\tstruct odb_stream base;\n \tgit_zstream z;\n \tenum {\n \t\tODB_LOOSE_READ_STREAM_INUSE,\n@@ -292,7 +292,7 @@ struct odb_loose_read_stream {\n \tint hdr_used;\n };\n \n-static ssize_t read_istream_loose(struct odb_read_stream *_st, char *buf, size_t sz)\n+static ssize_t read_istream_loose(struct odb_stream *_st, char *buf, size_t sz)\n {\n \tstruct odb_loose_read_stream *st =\n \t\tcontainer_of(_st, struct odb_loose_read_stream, base);\n@@ -339,7 +339,7 @@ static ssize_t read_istream_loose(struct odb_read_stream *_st, char *buf, size_t\n \treturn total_read;\n }\n \n-static int close_istream_loose(struct odb_read_stream *_st)\n+static int close_istream_loose(struct odb_stream *_st)\n {\n \tstruct odb_loose_read_stream *st =\n \t\tcontainer_of(_st, struct odb_loose_read_stream, base);\n@@ -350,7 +350,7 @@ static int close_istream_loose(struct odb_read_stream *_st)\n \treturn 0;\n }\n \n-static int odb_source_loose_read_object_stream(struct odb_read_stream **out,\n+static int odb_source_loose_read_object_stream(struct odb_stream **out,\n \t\t\t\t\t struct odb_source *source,\n \t\t\t\t\t const struct object_id *oid)\n {\ndiff --git a/odb/source-packed.c b/odb/source-packed.c\nindex e6ff74833b..b3186ca593 100644\n--- a/odb/source-packed.c\n+++ b/odb/source-packed.c\n@@ -70,7 +70,7 @@ static int odb_source_packed_read_object_info(struct odb_source *source,\n \treturn 0;\n }\n \n-static int odb_source_packed_read_object_stream(struct odb_read_stream **out,\n+static int odb_source_packed_read_object_stream(struct odb_stream **out,\n \t\t\t\t\t\tstruct odb_source *source,\n \t\t\t\t\t\tconst struct object_id *oid)\n {\ndiff --git a/odb/source.h b/odb/source.h\nindex 0080148ba7..89b0c39682 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -26,7 +26,7 @@ enum odb_source_type {\n };\n \n struct object_id;\n-struct odb_read_stream;\n+struct odb_stream;\n struct strvec;\n \n /*\n@@ -125,7 +125,7 @@ struct odb_source {\n \t * The callback is expected to return a negative error code in case\n \t * creating the object stream has failed, 0 otherwise.\n \t */\n-\tint (*read_object_stream)(struct odb_read_stream **out,\n+\tint (*read_object_stream)(struct odb_stream **out,\n \t\t\t\t struct odb_source *source,\n \t\t\t\t const struct object_id *oid);\n \n@@ -339,7 +339,7 @@ static inline int odb_source_read_object_info(struct odb_source *source,\n * Create a new read stream for the given object ID. Returns 0 on success, a\n * negative error code otherwise.\n */\n-static inline int odb_source_read_object_stream(struct odb_read_stream **out,\n+static inline int odb_source_read_object_stream(struct odb_stream **out,\n \t\t\t\t\t\tstruct odb_source *source,\n \t\t\t\t\t\tconst struct object_id *oid)\n {\ndiff --git a/odb/streaming.c b/odb/streaming.c\nindex 0918cad426..98e2152e36 100644\n--- a/odb/streaming.c\n+++ b/odb/streaming.c\n@@ -20,8 +20,8 @@\n *****************************************************************/\n \n struct odb_filtered_read_stream {\n-\tstruct odb_read_stream base;\n-\tstruct odb_read_stream *upstream;\n+\tstruct odb_stream base;\n+\tstruct odb_stream *upstream;\n \tstruct stream_filter *filter;\n \tchar ibuf[FILTER_BUFFER];\n \tchar obuf[FILTER_BUFFER];\n@@ -30,14 +30,14 @@ struct odb_filtered_read_stream {\n \tint input_finished;\n };\n \n-static int close_istream_filtered(struct odb_read_stream *_fs)\n+static int close_istream_filtered(struct odb_stream *_fs)\n {\n \tstruct odb_filtered_read_stream *fs = (struct odb_filtered_read_stream *)_fs;\n \tfree_stream_filter(fs->filter);\n-\treturn odb_read_stream_close(fs->upstream);\n+\treturn odb_stream_close(fs->upstream);\n }\n \n-static ssize_t read_istream_filtered(struct odb_read_stream *_fs, char *buf,\n+static ssize_t read_istream_filtered(struct odb_stream *_fs, char *buf,\n \t\t\t\t size_t sz)\n {\n \tstruct odb_filtered_read_stream *fs = (struct odb_filtered_read_stream *)_fs;\n@@ -86,7 +86,7 @@ static ssize_t read_istream_filtered(struct odb_read_stream *_fs, char *buf,\n \n \t\t/* refill the input from the upstream */\n \t\tif (!fs->input_finished) {\n-\t\t\tfs->i_end = odb_read_stream_read(fs->upstream, fs->ibuf, FILTER_BUFFER);\n+\t\t\tfs->i_end = odb_stream_read(fs->upstream, fs->ibuf, FILTER_BUFFER);\n \t\t\tif (fs->i_end < 0)\n \t\t\t\treturn -1;\n \t\t\tif (fs->i_end)\n@@ -97,8 +97,8 @@ static ssize_t read_istream_filtered(struct odb_read_stream *_fs, char *buf,\n \treturn filled;\n }\n \n-static struct odb_read_stream *attach_stream_filter(struct odb_read_stream *st,\n-\t\t\t\t\t\t struct stream_filter *filter)\n+static struct odb_stream *attach_stream_filter(struct odb_stream *st,\n+\t\t\t\t\t struct stream_filter *filter)\n {\n \tstruct odb_filtered_read_stream *fs;\n \n@@ -120,19 +120,19 @@ static struct odb_read_stream *attach_stream_filter(struct odb_read_stream *st,\n *****************************************************************/\n \n struct odb_incore_read_stream {\n-\tstruct odb_read_stream base;\n+\tstruct odb_stream base;\n \tchar *buf; /* from odb_read_object_info_extended() */\n \tunsigned long read_ptr;\n };\n \n-static int close_istream_incore(struct odb_read_stream *_st)\n+static int close_istream_incore(struct odb_stream *_st)\n {\n \tstruct odb_incore_read_stream *st = (struct odb_incore_read_stream *)_st;\n \tfree(st->buf);\n \treturn 0;\n }\n \n-static ssize_t read_istream_incore(struct odb_read_stream *_st, char *buf, size_t sz)\n+static ssize_t read_istream_incore(struct odb_stream *_st, char *buf, size_t sz)\n {\n \tstruct odb_incore_read_stream *st = (struct odb_incore_read_stream *)_st;\n \tsize_t read_size = sz;\n@@ -147,7 +147,7 @@ static ssize_t read_istream_incore(struct odb_read_stream *_st, char *buf, size_\n \treturn read_size;\n }\n \n-static int open_istream_incore(struct odb_read_stream **out,\n+static int open_istream_incore(struct odb_stream **out,\n \t\t\t struct object_database *odb,\n \t\t\t const struct object_id *oid)\n {\n@@ -178,7 +178,7 @@ static int open_istream_incore(struct odb_read_stream **out,\n * static helpers variables and functions for users of streaming interface\n *****************************************************************************/\n \n-static int istream_source(struct odb_read_stream **out,\n+static int istream_source(struct odb_stream **out,\n \t\t\t struct object_database *odb,\n \t\t\t const struct object_id *oid)\n {\n@@ -196,23 +196,23 @@ static int istream_source(struct odb_read_stream **out,\n * Users of streaming interface\n ****************************************************************/\n \n-int odb_read_stream_close(struct odb_read_stream *st)\n+int odb_stream_close(struct odb_stream *st)\n {\n \tint r = st->close(st);\n \tfree(st);\n \treturn r;\n }\n \n-ssize_t odb_read_stream_read(struct odb_read_stream *st, void *buf, size_t sz)\n+ssize_t odb_stream_read(struct odb_stream *st, void *buf, size_t sz)\n {\n \treturn st->read(st, buf, sz);\n }\n \n-struct odb_read_stream *odb_read_stream_open(struct object_database *odb,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t struct stream_filter *filter)\n+struct odb_stream *odb_read_stream_open(struct object_database *odb,\n+\t\t\t\t\tconst struct object_id *oid,\n+\t\t\t\t\tstruct stream_filter *filter)\n {\n-\tstruct odb_read_stream *st;\n+\tstruct odb_stream *st;\n \tconst struct object_id *real = lookup_replace_object(odb->repo, oid);\n \tint ret = istream_source(&st, odb, real);\n \n@@ -221,9 +221,9 @@ struct odb_read_stream *odb_read_stream_open(struct object_database *odb,\n \n \tif (filter) {\n \t\t/* Add \"&& !is_null_stream_filter(filter)\" for performance */\n-\t\tstruct odb_read_stream *nst = attach_stream_filter(st, filter);\n+\t\tstruct odb_stream *nst = attach_stream_filter(st, filter);\n \t\tif (!nst) {\n-\t\t\todb_read_stream_close(st);\n+\t\t\todb_stream_close(st);\n \t\t\treturn NULL;\n \t\t}\n \t\tst = nst;\n@@ -248,7 +248,7 @@ int odb_stream_blob_to_fd(struct object_database *odb,\n \t\t\t struct stream_filter *filter,\n \t\t\t int can_seek)\n {\n-\tstruct odb_read_stream *st;\n+\tstruct odb_stream *st;\n \tssize_t kept = 0;\n \tint result = -1;\n \n@@ -263,7 +263,7 @@ int odb_stream_blob_to_fd(struct object_database *odb,\n \tfor (;;) {\n \t\tchar buf[1024 * 16];\n \t\tssize_t wrote, holeto;\n-\t\tssize_t readlen = odb_read_stream_read(st, buf, sizeof(buf));\n+\t\tssize_t readlen = odb_stream_read(st, buf, sizeof(buf));\n \n \t\tif (readlen < 0)\n \t\t\tgoto close_and_exit;\n@@ -294,7 +294,7 @@ int odb_stream_blob_to_fd(struct object_database *odb,\n \tresult = 0;\n \n close_and_exit:\n-\todb_read_stream_close(st);\n+\todb_stream_close(st);\n \treturn result;\n }\n \ndiff --git a/odb/streaming.h b/odb/streaming.h\nindex 3c8ed55129..037954c231 100644\n--- a/odb/streaming.h\n+++ b/odb/streaming.h\n@@ -8,19 +8,19 @@\n #include \"odb.h\"\n \n struct object_database;\n-struct odb_read_stream;\n+struct odb_stream;\n struct stream_filter;\n \n-typedef int (*odb_read_stream_close_fn)(struct odb_read_stream *);\n-typedef ssize_t (*odb_read_stream_read_fn)(struct odb_read_stream *, char *, size_t);\n+typedef int (*odb_stream_close_fn)(struct odb_stream *);\n+typedef ssize_t (*odb_stream_read_fn)(struct odb_stream *, char *, size_t);\n \n /*\n * A stream that can be used to read an object from the object database without\n * loading all of it into memory.\n */\n-struct odb_read_stream {\n-\todb_read_stream_close_fn close;\n-\todb_read_stream_read_fn read;\n+struct odb_stream {\n+\todb_stream_close_fn close;\n+\todb_stream_read_fn read;\n \tenum object_type type;\n \tsize_t size; /* inflated size of full object */\n };\n@@ -31,22 +31,22 @@ struct odb_read_stream {\n *\n * Returns the stream on success, a `NULL` pointer otherwise.\n */\n-struct odb_read_stream *odb_read_stream_open(struct object_database *odb,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t struct stream_filter *filter);\n+struct odb_stream *odb_read_stream_open(struct object_database *odb,\n+\t\t\t\t\tconst struct object_id *oid,\n+\t\t\t\t\tstruct stream_filter *filter);\n \n /*\n- * Close the given read stream and release all resources associated with it.\n+ * Close the given object stream and release all resources associated with it.\n * Returns 0 on success, a negative error code otherwise.\n */\n-int odb_read_stream_close(struct odb_read_stream *stream);\n+int odb_stream_close(struct odb_stream *stream);\n \n /*\n * Read data from the stream into the buffer. Returns 0 on EOF and the number\n * of bytes read on success. Returns a negative error code in case reading from\n * the stream fails.\n */\n-ssize_t odb_read_stream_read(struct odb_read_stream *stream, void *buf, size_t len);\n+ssize_t odb_stream_read(struct odb_stream *stream, void *buf, size_t len);\n \n /*\n * A stream that provides an object to be written to the object database without\ndiff --git a/pack-check.c b/pack-check.c\nindex c3b8db7c5c..1b5e26847d 100644\n--- a/pack-check.c\n+++ b/pack-check.c\n@@ -106,7 +106,7 @@ static int verify_packfile(struct repository *r,\n \tQSORT(entries, nr_objects, compare_entries);\n \n \tfor (i = 0; i < nr_objects; i++) {\n-\t\tstruct odb_read_stream *stream = NULL;\n+\t\tstruct odb_stream *stream = NULL;\n \t\tvoid *data;\n \t\tstruct object_id oid;\n \t\tenum object_type type;\n@@ -171,7 +171,7 @@ static int verify_packfile(struct repository *r,\n \t\t\tdisplay_progress(progress, base_count + i);\n \n \t\tif (stream)\n-\t\t\todb_read_stream_close(stream);\n+\t\t\todb_stream_close(stream);\n \t\tfree(data);\n \t}\n \ndiff --git a/packfile.c b/packfile.c\nindex 0eee45055f..70254573a3 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -2115,7 +2115,7 @@ int parse_pack_header_option(const char *in, unsigned char *out, unsigned int *l\n }\n \n struct odb_packed_read_stream {\n-\tstruct odb_read_stream base;\n+\tstruct odb_stream base;\n \tstruct packed_git *pack;\n \tgit_zstream z;\n \tenum {\n@@ -2127,7 +2127,7 @@ struct odb_packed_read_stream {\n \toff_t pos;\n };\n \n-static ssize_t read_istream_pack_non_delta(struct odb_read_stream *_st, char *buf,\n+static ssize_t read_istream_pack_non_delta(struct odb_stream *_st, char *buf,\n \t\t\t\t\t size_t sz)\n {\n \tstruct odb_packed_read_stream *st = (struct odb_packed_read_stream *)_st;\n@@ -2187,7 +2187,7 @@ static ssize_t read_istream_pack_non_delta(struct odb_read_stream *_st, char *bu\n \treturn total_read;\n }\n \n-static int close_istream_pack_non_delta(struct odb_read_stream *_st)\n+static int close_istream_pack_non_delta(struct odb_stream *_st)\n {\n \tstruct odb_packed_read_stream *st = (struct odb_packed_read_stream *)_st;\n \tif (st->z_state == ODB_PACKED_READ_STREAM_INUSE)\n@@ -2195,7 +2195,7 @@ static int close_istream_pack_non_delta(struct odb_read_stream *_st)\n \treturn 0;\n }\n \n-int packfile_read_object_stream(struct odb_read_stream **out,\n+int packfile_read_object_stream(struct odb_stream **out,\n \t\t\t\tconst struct object_id *oid,\n \t\t\t\tstruct packed_git *pack,\n \t\t\t\toff_t offset)\ndiff --git a/packfile.h b/packfile.h\nindex e1f77152b5..f913cb3d0c 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -12,7 +12,7 @@\n \n /* in odb.h */\n struct object_info;\n-struct odb_read_stream;\n+struct odb_stream;\n \n struct packed_git {\n \tstruct pack_window *windows;\n@@ -306,7 +306,7 @@ off_t get_delta_base(struct packed_git *p, struct pack_window **w_curs,\n \t\t off_t *curpos, enum object_type type,\n \t\t off_t delta_obj_offset);\n \n-int packfile_read_object_stream(struct odb_read_stream **out,\n+int packfile_read_object_stream(struct odb_stream **out,\n \t\t\t\tconst struct object_id *oid,\n \t\t\t\tstruct packed_git *pack,\n \t\t\t\toff_t offset);\ndiff --git a/t/unit-tests/u-odb-inmemory.c b/t/unit-tests/u-odb-inmemory.c\nindex 1ab07af6d6..839a0fd3b7 100644\n--- a/t/unit-tests/u-odb-inmemory.c\n+++ b/t/unit-tests/u-odb-inmemory.c\n@@ -100,7 +100,7 @@ void test_odb_inmemory__read_written_object(void)\n void test_odb_inmemory__read_stream_object(void)\n {\n \tstruct odb_source_inmemory *source = odb_source_inmemory_new(odb);\n-\tstruct odb_read_stream *stream;\n+\tstruct odb_stream *stream;\n \tstruct object_id written_oid;\n \tconst char data[] = \"foobar\";\n \tchar buf[3] = { 0 };\n@@ -112,15 +112,15 @@ void test_odb_inmemory__read_stream_object(void)\n \tcl_assert_equal_i(stream->type, OBJ_BLOB);\n \tcl_assert_equal_u(stream->size, 6);\n \n-\tcl_assert_equal_i(odb_read_stream_read(stream, buf, 2), 2);\n+\tcl_assert_equal_i(odb_stream_read(stream, buf, 2), 2);\n \tcl_assert_equal_s(buf, \"fo\");\n-\tcl_assert_equal_i(odb_read_stream_read(stream, buf, 2), 2);\n+\tcl_assert_equal_i(odb_stream_read(stream, buf, 2), 2);\n \tcl_assert_equal_s(buf, \"ob\");\n-\tcl_assert_equal_i(odb_read_stream_read(stream, buf, 2), 2);\n+\tcl_assert_equal_i(odb_stream_read(stream, buf, 2), 2);\n \tcl_assert_equal_s(buf, \"ar\");\n-\tcl_assert_equal_i(odb_read_stream_read(stream, buf, 2), 0);\n+\tcl_assert_equal_i(odb_stream_read(stream, buf, 2), 0);\n \n-\todb_read_stream_close(stream);\n+\todb_stream_close(stream);\n \todb_source_free(&source->base);\n }\n \n\n-- \n2.55.0.679.g6767b8d81c.dirty\n\n"},{"id":"549653","messageId":"20260805-pks-odb-stream-unification-v2-5-b8c369564641@pks.im","threadId":"66111","inReplyTo":"20260805-pks-odb-stream-unification-v2-0-b8c369564641@pks.im","subject":"[PATCH v2 5/8] odb/streaming: consolidate read and write streams","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-05T07:44:49Z","receivedAt":"2026-08-05T07:45:10Z","isPatch":true,"body":"The `struct odb_read_stream` and `struct odb_write_stream` both provide\nthe same functionality: they allow a caller to read object data from an\narbitrary source. Historically, the only difference was that the read\nstream was used to read data out of the object database, whereas the\nwrite stream was used to write data into the object database, but the\ninterfaces were mostly the same.\n\nOver the preceding commits we have refactored the write stream to have\nalmost exactly the same interface as the read stream. With these\nrefactorings we can now easily merge those two streams into a single\ninterface that's used for both use cases.\n\nWhile most of the changes are mechanical, there are two sites that need\nspecial mention:\n\n - \"builtin/unpack-objects.c\" creates a write stream from compressed\n object data.\n\n - \"odb/streaming.c\" creates a write stream from a file descriptor.\n\nAdapting these sites to yield the new stream type requires a couple more\nchanges. Most importantly, instead of embedding the pointer to the data\nin `struct odb_write_stream`, we now allocate a structure that wraps the\nnew `struct odb_stream` base. Other than that though, the changes are\nrather straight forward.\n\nSome of the structures and functions are now somewhat misnamed. These\nwill be fixed in subsequent commits.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/unpack-objects.c | 31 ++++++++++++++++---------------\n object-file.c | 25 ++++++++++++-------------\n odb.c | 2 +-\n odb.h | 4 ++--\n odb/source-files.c | 2 +-\n odb/source-inmemory.c | 4 ++--\n odb/source-loose.c | 6 +++---\n odb/source-packed.c | 2 +-\n odb/source.h | 4 ++--\n odb/streaming.c | 35 ++++++++++++++++-------------------\n odb/streaming.h | 31 +++----------------------------\n odb/transaction.c | 2 +-\n odb/transaction.h | 4 ++--\n t/unit-tests/u-odb-inmemory.c | 6 +++---\n 14 files changed, 65 insertions(+), 93 deletions(-)\n\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex 7439ec53be..05a2d48011 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -359,20 +359,21 @@ static void unpack_non_delta_entry(enum object_type type, unsigned long size,\n }\n \n struct input_zstream_data {\n+\tstruct odb_stream base;\n \tgit_zstream *zstream;\n \tint status;\n };\n \n-static ssize_t feed_input_zstream(struct odb_write_stream *in_stream,\n-\t\t\t\t unsigned char *buf, size_t buf_len)\n+static ssize_t feed_input_zstream(struct odb_stream *in_stream,\n+\t\t\t\t char *buf, size_t buf_len)\n {\n-\tstruct input_zstream_data *data = in_stream->data;\n+\tstruct input_zstream_data *data = container_of(in_stream, struct input_zstream_data, base);\n \tgit_zstream *zstream = data->zstream;\n \n \tif (data->status != Z_OK)\n \t\treturn 0;\n \n-\tzstream->next_out = buf;\n+\tzstream->next_out = (unsigned char *) buf;\n \tzstream->avail_out = buf_len;\n \n \twhile (data->status == Z_OK && zstream->avail_out == buf_len) {\n@@ -388,24 +389,24 @@ static ssize_t feed_input_zstream(struct odb_write_stream *in_stream,\n static void stream_blob(unsigned long size, unsigned nr)\n {\n \tgit_zstream zstream = { 0 };\n-\tstruct input_zstream_data data = { 0 };\n-\tstruct odb_write_stream in_stream = {\n-\t\t.read = feed_input_zstream,\n-\t\t.data = &data,\n-\t\t.size = size,\n-\t\t.type = OBJ_BLOB,\n+\tstruct input_zstream_data in_stream = {\n+\t\t.base = {\n+\t\t\t.read = feed_input_zstream,\n+\t\t\t.size = size,\n+\t\t\t.type = OBJ_BLOB,\n+\t\t},\n+\t\t.zstream = &zstream,\n+\t\t.status = Z_OK,\n \t};\n \tstruct obj_info *info = &obj_list[nr];\n \n-\tdata.zstream = &zstream;\n-\tdata.status = Z_OK;\n \tgit_inflate_init(&zstream);\n \n-\tif (odb_write_object_stream(the_repository->objects, &in_stream, &info->oid))\n+\tif (odb_write_object_stream(the_repository->objects, &in_stream.base, &info->oid))\n \t\tdie(_(\"failed to write object in stream\"));\n \n-\tif (data.status != Z_STREAM_END)\n-\t\tdie(_(\"inflate returned (%d)\"), data.status);\n+\tif (in_stream.status != Z_STREAM_END)\n+\t\tdie(_(\"inflate returned (%d)\"), in_stream.status);\n \tgit_inflate_end(&zstream);\n \n \tif (strict) {\ndiff --git a/object-file.c b/object-file.c\nindex 5f6d584c35..068c6e5672 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -702,7 +702,7 @@ static void prepare_packfile_transaction(struct odb_transaction_files *transacti\n \t\tdie_errno(\"unable to write pack header\");\n }\n \n-static int hash_stream(struct odb_write_stream *stream,\n+static int hash_stream(struct odb_stream *stream,\n \t\t const struct git_hash_algo *hash_algo,\n \t\t struct object_id *result_oid)\n {\n@@ -717,8 +717,8 @@ static int hash_stream(struct odb_write_stream *stream,\n \tgit_hash_update(&ctx, buf, header_len);\n \n \twhile (1) {\n-\t\tssize_t read_result = odb_write_stream_read(stream, buf,\n-\t\t\t\t\t\t\t sizeof(buf));\n+\t\tssize_t read_result = odb_stream_read(stream, buf,\n+\t\t\t\t\t\t sizeof(buf));\n \t\tif (read_result < 0)\n \t\t\treturn -1;\n \t\tif (!read_result)\n@@ -742,7 +742,7 @@ static int hash_stream(struct odb_write_stream *stream,\n */\n static void stream_to_pack(struct transaction_packfile *state,\n \t\t\t struct git_hash_ctx *ctx,\n-\t\t\t struct odb_write_stream *stream)\n+\t\t\t struct odb_stream *stream)\n {\n \tgit_zstream s;\n \tunsigned char ibuf[16384];\n@@ -761,8 +761,8 @@ static void stream_to_pack(struct transaction_packfile *state,\n \n \twhile (status != Z_STREAM_END) {\n \t\tif (!is_finished && !s.avail_in) {\n-\t\t\tssize_t rsize = odb_write_stream_read(stream, ibuf,\n-\t\t\t\t\t\t\t sizeof(ibuf));\n+\t\t\tssize_t rsize = odb_stream_read(stream, ibuf,\n+\t\t\t\t\t\t\tsizeof(ibuf));\n \t\t\tif (rsize < 0)\n \t\t\t\tdie(\"failed to read object data\");\n \t\t\tif (!rsize)\n@@ -872,7 +872,7 @@ static void flush_packfile_transaction(struct odb_transaction_files *transaction\n * callers should avoid this code path when filters are requested.\n */\n static int odb_transaction_files_write_object_stream(struct odb_transaction *base,\n-\t\t\t\t\t\t struct odb_write_stream *stream,\n+\t\t\t\t\t\t struct odb_stream *stream,\n \t\t\t\t\t\t struct object_id *result_oid)\n {\n \tstruct odb_transaction_files *transaction = container_of(base,\n@@ -952,8 +952,8 @@ int index_fd(struct index_state *istate, struct object_id *oid,\n \t\tret = index_core(istate, oid, fd, xsize_t(st->st_size),\n \t\t\t\t type, path, flags);\n \t} else {\n-\t\tstruct odb_write_stream stream;\n-\t\todb_write_stream_from_fd(&stream, fd, xsize_t(st->st_size), OBJ_BLOB);\n+\t\tstruct odb_stream *stream = odb_write_stream_from_fd(fd, xsize_t(st->st_size),\n+\t\t\t\t\t\t\t\t OBJ_BLOB);\n \n \t\tif (flags & INDEX_WRITE_OBJECT) {\n \t\t\tstruct object_database *odb = the_repository->objects;\n@@ -963,15 +963,14 @@ int index_fd(struct index_state *istate, struct object_id *oid,\n \t\t\tif (!inflight)\n \t\t\t\todb_transaction_begin_or_die(odb, &transaction, 0);\n \t\t\tret = odb_transaction_write_object_stream(transaction,\n-\t\t\t\t\t\t\t\t &stream,\n-\t\t\t\t\t\t\t\t oid);\n+\t\t\t\t\t\t\t\t stream, oid);\n \t\t\tif (!inflight)\n \t\t\t\todb_transaction_commit(transaction);\n \t\t} else {\n-\t\t\tret = hash_stream(&stream, the_repository->hash_algo, oid);\n+\t\t\tret = hash_stream(stream, the_repository->hash_algo, oid);\n \t\t}\n \n-\t\todb_write_stream_release(&stream);\n+\t\todb_stream_close(stream);\n \t}\n \n \tclose(fd);\ndiff --git a/odb.c b/odb.c\nindex 585b2b2965..eec4cc5302 100644\n--- a/odb.c\n+++ b/odb.c\n@@ -1028,7 +1028,7 @@ int odb_write_object_ext(struct object_database *odb,\n }\n \n int odb_write_object_stream(struct object_database *odb,\n-\t\t\t struct odb_write_stream *stream,\n+\t\t\t struct odb_stream *stream,\n \t\t\t struct object_id *oid)\n {\n \treturn odb_source_write_object_stream(odb->sources, stream, oid);\ndiff --git a/odb.h b/odb.h\nindex 019d3af3e8..fbe75c5a81 100644\n--- a/odb.h\n+++ b/odb.h\n@@ -626,10 +626,10 @@ static inline int odb_write_object(struct object_database *odb,\n \treturn odb_write_object_ext(odb, buf, len, type, oid, NULL, 0);\n }\n \n-struct odb_write_stream;\n+struct odb_stream;\n \n int odb_write_object_stream(struct object_database *odb,\n-\t\t\t struct odb_write_stream *stream,\n+\t\t\t struct odb_stream *stream,\n \t\t\t struct object_id *oid);\n \n void parse_alternates(const char *string,\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex f7b8c76549..6defe5ac4f 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -174,7 +174,7 @@ static int odb_source_files_write_object(struct odb_source *source,\n }\n \n static int odb_source_files_write_object_stream(struct odb_source *source,\n-\t\t\t\t\t\tstruct odb_write_stream *stream,\n+\t\t\t\t\t\tstruct odb_stream *stream,\n \t\t\t\t\t\tstruct object_id *oid)\n {\n \tstruct odb_source_files *files = odb_source_files_downcast(source);\ndiff --git a/odb/source-inmemory.c b/odb/source-inmemory.c\nindex 485d587036..795672adf2 100644\n--- a/odb/source-inmemory.c\n+++ b/odb/source-inmemory.c\n@@ -256,7 +256,7 @@ static int odb_source_inmemory_write_object(struct odb_source *source,\n }\n \n static int odb_source_inmemory_write_object_stream(struct odb_source *source,\n-\t\t\t\t\t\t struct odb_write_stream *stream,\n+\t\t\t\t\t\t struct odb_stream *stream,\n \t\t\t\t\t\t struct object_id *oid)\n {\n \tchar buf[16384];\n@@ -268,7 +268,7 @@ static int odb_source_inmemory_write_object_stream(struct odb_source *source,\n \twhile (1) {\n \t\tssize_t bytes_read;\n \n-\t\tbytes_read = odb_write_stream_read(stream, buf, sizeof(buf));\n+\t\tbytes_read = odb_stream_read(stream, buf, sizeof(buf));\n \t\tif (bytes_read < 0) {\n \t\t\tret = error(\"failed to read object stream\");\n \t\t\tgoto out;\ndiff --git a/odb/source-loose.c b/odb/source-loose.c\nindex 038defd905..ff1bede7fe 100644\n--- a/odb/source-loose.c\n+++ b/odb/source-loose.c\n@@ -845,7 +845,7 @@ static int odb_source_loose_write_object(struct odb_source *source,\n }\n \n static int odb_source_loose_write_object_stream(struct odb_source *source,\n-\t\t\t\t\t\tstruct odb_write_stream *in_stream,\n+\t\t\t\t\t\tstruct odb_stream *in_stream,\n \t\t\t\t\t\tstruct object_id *oid)\n {\n \tstruct odb_source_loose *loose = odb_source_loose_downcast(source);\n@@ -891,8 +891,8 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,\n \t\tunsigned char *in0 = stream.next_in;\n \n \t\tif (!stream.avail_in && !is_finished) {\n-\t\t\tssize_t read_len = odb_write_stream_read(in_stream, buf,\n-\t\t\t\t\t\t\t\t sizeof(buf));\n+\t\t\tssize_t read_len = odb_stream_read(in_stream, buf,\n+\t\t\t\t\t\t\t sizeof(buf));\n \t\t\tif (read_len < 0) {\n \t\t\t\tclose(fd);\n \t\t\t\terr = -1;\ndiff --git a/odb/source-packed.c b/odb/source-packed.c\nindex b3186ca593..630d955585 100644\n--- a/odb/source-packed.c\n+++ b/odb/source-packed.c\n@@ -609,7 +609,7 @@ static int odb_source_packed_write_object(struct odb_source *source UNUSED,\n }\n \n static int odb_source_packed_write_object_stream(struct odb_source *source UNUSED,\n-\t\t\t\t\t\t struct odb_write_stream *stream UNUSED,\n+\t\t\t\t\t\t struct odb_stream *stream UNUSED,\n \t\t\t\t\t\t struct object_id *oid UNUSED)\n {\n \treturn error(\"packed backend cannot write object streams\");\ndiff --git a/odb/source.h b/odb/source.h\nindex 89b0c39682..0b99c698b5 100644\n--- a/odb/source.h\n+++ b/odb/source.h\n@@ -221,7 +221,7 @@ struct odb_source {\n \t * otherwise.\n \t */\n \tint (*write_object_stream)(struct odb_source *source,\n-\t\t\t\t struct odb_write_stream *stream,\n+\t\t\t\t struct odb_stream *stream,\n \t\t\t\t struct object_id *oid);\n \n \t/*\n@@ -436,7 +436,7 @@ static inline int odb_source_write_object(struct odb_source *source,\n * out pointer for the object ID.\n */\n static inline int odb_source_write_object_stream(struct odb_source *source,\n-\t\t\t\t\t\t struct odb_write_stream *stream,\n+\t\t\t\t\t\t struct odb_stream *stream,\n \t\t\t\t\t\t struct object_id *oid)\n {\n \treturn source->write_object_stream(source, stream, oid);\ndiff --git a/odb/streaming.c b/odb/streaming.c\nindex 98e2152e36..1a267e6b90 100644\n--- a/odb/streaming.c\n+++ b/odb/streaming.c\n@@ -232,16 +232,6 @@ struct odb_stream *odb_read_stream_open(struct object_database *odb,\n \treturn st;\n }\n \n-ssize_t odb_write_stream_read(struct odb_write_stream *st, void *buf, size_t sz)\n-{\n-\treturn st->read(st, buf, sz);\n-}\n-\n-void odb_write_stream_release(struct odb_write_stream *st)\n-{\n-\tfree(st->data);\n-}\n-\n int odb_stream_blob_to_fd(struct object_database *odb,\n \t\t\t int fd,\n \t\t\t const struct object_id *oid,\n@@ -299,14 +289,15 @@ int odb_stream_blob_to_fd(struct object_database *odb,\n }\n \n struct read_object_fd_data {\n+\tstruct odb_stream base;\n \tint fd;\n \tsize_t remaining;\n };\n \n-static ssize_t read_object_fd(struct odb_write_stream *stream,\n-\t\t\t unsigned char *buf, size_t len)\n+static ssize_t read_object_fd(struct odb_stream *stream,\n+\t\t\t char *buf, size_t len)\n {\n-\tstruct read_object_fd_data *data = stream->data;\n+\tstruct read_object_fd_data *data = container_of(stream, struct read_object_fd_data, base);\n \tssize_t read_result;\n \tsize_t count;\n \n@@ -323,17 +314,23 @@ static ssize_t read_object_fd(struct odb_write_stream *stream,\n \treturn read_result;\n }\n \n-void odb_write_stream_from_fd(struct odb_write_stream *stream, int fd,\n-\t\t\t size_t size, enum object_type type)\n+static int close_object_fd(struct odb_stream *stream UNUSED)\n+{\n+\t/* The file descriptor is owned by the caller for now. */\n+\treturn 0;\n+}\n+\n+struct odb_stream *odb_write_stream_from_fd(int fd, size_t size, enum object_type type)\n {\n \tstruct read_object_fd_data *data;\n \n \tCALLOC_ARRAY(data, 1);\n+\tdata->base.read = read_object_fd;\n+\tdata->base.close = close_object_fd;\n+\tdata->base.size = size;\n+\tdata->base.type = type;\n \tdata->fd = fd;\n \tdata->remaining = size;\n \n-\tstream->data = data;\n-\tstream->read = read_object_fd;\n-\tstream->size = size;\n-\tstream->type = type;\n+\treturn &data->base;\n }\ndiff --git a/odb/streaming.h b/odb/streaming.h\nindex 037954c231..60b9803190 100644\n--- a/odb/streaming.h\n+++ b/odb/streaming.h\n@@ -15,8 +15,8 @@ typedef int (*odb_stream_close_fn)(struct odb_stream *);\n typedef ssize_t (*odb_stream_read_fn)(struct odb_stream *, char *, size_t);\n \n /*\n- * A stream that can be used to read an object from the object database without\n- * loading all of it into memory.\n+ * A stream that can be used to read an object from or write an object into the\n+ * object database without loading all of it into memory.\n */\n struct odb_stream {\n \todb_stream_close_fn close;\n@@ -48,30 +48,6 @@ int odb_stream_close(struct odb_stream *stream);\n */\n ssize_t odb_stream_read(struct odb_stream *stream, void *buf, size_t len);\n \n-/*\n- * A stream that provides an object to be written to the object database without\n- * loading all of it into memory.\n- */\n-struct odb_write_stream {\n-\tssize_t (*read)(struct odb_write_stream *, unsigned char *, size_t);\n-\tvoid *data;\n-\tsize_t size;\n-\tenum object_type type;\n-};\n-\n-/*\n- * Read data from the stream into the buffer. Returns 0 when finished and the\n- * number of bytes read on success. Returns a negative error code in case\n- * reading from the stream fails.\n- */\n-ssize_t odb_write_stream_read(struct odb_write_stream *stream, void *buf,\n-\t\t\t size_t len);\n-\n-/*\n- * Releases memory allocated for underlying stream data.\n- */\n-void odb_write_stream_release(struct odb_write_stream *stream);\n-\n /*\n * Look up the object by its ID and write the full contents to the file\n * descriptor. The object must be a blob, or the function will fail. When\n@@ -92,7 +68,6 @@ int odb_stream_blob_to_fd(struct object_database *odb,\n /*\n * Sets up an ODB write stream that reads from an fd.\n */\n-void odb_write_stream_from_fd(struct odb_write_stream *stream, int fd,\n-\t\t\t size_t size, enum object_type type);\n+struct odb_stream *odb_write_stream_from_fd(int fd, size_t size, enum object_type type);\n \n #endif /* STREAMING_H */\ndiff --git a/odb/transaction.c b/odb/transaction.c\nindex 6aaf133812..69d71b9e97 100644\n--- a/odb/transaction.c\n+++ b/odb/transaction.c\n@@ -39,7 +39,7 @@ int odb_transaction_commit(struct odb_transaction *transaction)\n }\n \n int odb_transaction_write_object_stream(struct odb_transaction *transaction,\n-\t\t\t\t\tstruct odb_write_stream *stream,\n+\t\t\t\t\tstruct odb_stream *stream,\n \t\t\t\t\tstruct object_id *oid)\n {\n \treturn transaction->write_object_stream(transaction, stream, oid);\ndiff --git a/odb/transaction.h b/odb/transaction.h\nindex 1eb74664c6..65248a409c 100644\n--- a/odb/transaction.h\n+++ b/odb/transaction.h\n@@ -31,7 +31,7 @@ struct odb_transaction {\n \t * otherwise.\n \t */\n \tint (*write_object_stream)(struct odb_transaction *transaction,\n-\t\t\t\t struct odb_write_stream *stream,\n+\t\t\t\t struct odb_stream *stream,\n \t\t\t\t struct object_id *oid);\n \n \t/*\n@@ -81,7 +81,7 @@ int odb_transaction_commit(struct odb_transaction *transaction);\n * error code otherwise.\n */\n int odb_transaction_write_object_stream(struct odb_transaction *transaction,\n-\t\t\t\t\tstruct odb_write_stream *stream,\n+\t\t\t\t\tstruct odb_stream *stream,\n \t\t\t\t\tstruct object_id *oid);\n \n /*\ndiff --git a/t/unit-tests/u-odb-inmemory.c b/t/unit-tests/u-odb-inmemory.c\nindex 839a0fd3b7..b8b331b37d 100644\n--- a/t/unit-tests/u-odb-inmemory.c\n+++ b/t/unit-tests/u-odb-inmemory.c\n@@ -266,13 +266,13 @@ void test_odb_inmemory__freshen_object(void)\n }\n \n struct membuf_write_stream {\n-\tstruct odb_write_stream base;\n+\tstruct odb_stream base;\n \tconst char *buf;\n \tsize_t offset;\n };\n \n-static ssize_t membuf_write_stream_read(struct odb_write_stream *stream,\n-\t\t\t\t\tunsigned char *buf, size_t len)\n+static ssize_t membuf_write_stream_read(struct odb_stream *stream,\n+\t\t\t\t\tchar *buf, size_t len)\n {\n \tstruct membuf_write_stream *s = container_of(stream, struct membuf_write_stream, base);\n \tsize_t chunk_size = 2;\n\n-- \n2.55.0.679.g6767b8d81c.dirty\n\n"},{"id":"549654","messageId":"20260805-pks-odb-stream-unification-v2-6-b8c369564641@pks.im","threadId":"66111","inReplyTo":"20260805-pks-odb-stream-unification-v2-0-b8c369564641@pks.im","subject":"[PATCH v2 6/8] odb/streaming: rename `struct read_object_fd_data`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-05T07:44:50Z","receivedAt":"2026-08-05T07:45:13Z","isPatch":true,"body":"With the preceding refactorings the `struct read_object_fd_data` is now\nsomewhat misnamed, as it doesn't only contain the data anymore, but also\nthe stream itself. Rename the structure to `struct fd_stream` to better\nmatch the new structure.\n\nSigned-off-by: Patrick Steinhardt \n---\n odb/streaming.c | 34 +++++++++++++++++-----------------\n 1 file changed, 17 insertions(+), 17 deletions(-)\n\ndiff --git a/odb/streaming.c b/odb/streaming.c\nindex 1a267e6b90..c436b18d39 100644\n--- a/odb/streaming.c\n+++ b/odb/streaming.c\n@@ -288,33 +288,33 @@ int odb_stream_blob_to_fd(struct object_database *odb,\n \treturn result;\n }\n \n-struct read_object_fd_data {\n+struct fd_stream {\n \tstruct odb_stream base;\n \tint fd;\n \tsize_t remaining;\n };\n \n-static ssize_t read_object_fd(struct odb_stream *stream,\n+static ssize_t fd_stream_read(struct odb_stream *stream,\n \t\t\t char *buf, size_t len)\n {\n-\tstruct read_object_fd_data *data = container_of(stream, struct read_object_fd_data, base);\n+\tstruct fd_stream *fds = container_of(stream, struct fd_stream, base);\n \tssize_t read_result;\n \tsize_t count;\n \n-\tif (!data->remaining)\n+\tif (!fds->remaining)\n \t\treturn 0;\n \n-\tcount = data->remaining < len ? data->remaining : len;\n-\tread_result = read_in_full(data->fd, buf, count);\n+\tcount = fds->remaining < len ? fds->remaining : len;\n+\tread_result = read_in_full(fds->fd, buf, count);\n \tif (read_result < 0 || (size_t)read_result != count)\n \t\treturn -1;\n \n-\tdata->remaining -= count;\n+\tfds->remaining -= count;\n \n \treturn read_result;\n }\n \n-static int close_object_fd(struct odb_stream *stream UNUSED)\n+static int fd_stream_close(struct odb_stream *stream UNUSED)\n {\n \t/* The file descriptor is owned by the caller for now. */\n \treturn 0;\n@@ -322,15 +322,15 @@ static int close_object_fd(struct odb_stream *stream UNUSED)\n \n struct odb_stream *odb_write_stream_from_fd(int fd, size_t size, enum object_type type)\n {\n-\tstruct read_object_fd_data *data;\n+\tstruct fd_stream *fds;\n \n-\tCALLOC_ARRAY(data, 1);\n-\tdata->base.read = read_object_fd;\n-\tdata->base.close = close_object_fd;\n-\tdata->base.size = size;\n-\tdata->base.type = type;\n-\tdata->fd = fd;\n-\tdata->remaining = size;\n+\tCALLOC_ARRAY(fds, 1);\n+\tfds->base.read = fd_stream_read;\n+\tfds->base.close = fd_stream_close;\n+\tfds->base.size = size;\n+\tfds->base.type = type;\n+\tfds->fd = fd;\n+\tfds->remaining = size;\n \n-\treturn &data->base;\n+\treturn &fds->base;\n }\n\n-- \n2.55.0.679.g6767b8d81c.dirty\n\n"},{"id":"549655","messageId":"20260805-pks-odb-stream-unification-v2-7-b8c369564641@pks.im","threadId":"66111","inReplyTo":"20260805-pks-odb-stream-unification-v2-0-b8c369564641@pks.im","subject":"[PATCH v2 7/8] odb/streaming: rename `struct input_zstream_data`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-05T07:44:51Z","receivedAt":"2026-08-05T07:45:16Z","isPatch":true,"body":"With the preceding refactorings the `struct input_zstream_data` is now\nsomewhat misnamed, as it doesn't only contain the data anymore, but also\nthe stream itself. Rename the structure to `struct zlib_stream` to\nbetter match the new structure.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/unpack-objects.c | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex 05a2d48011..3392a3b87d 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -358,16 +358,16 @@ static void unpack_non_delta_entry(enum object_type type, unsigned long size,\n \t\twrite_object(nr, type, buf, size);\n }\n \n-struct input_zstream_data {\n+struct zlib_stream {\n \tstruct odb_stream base;\n \tgit_zstream *zstream;\n \tint status;\n };\n \n-static ssize_t feed_input_zstream(struct odb_stream *in_stream,\n-\t\t\t\t char *buf, size_t buf_len)\n+static ssize_t zlib_stream_read(struct odb_stream *in_stream,\n+\t\t\t\tchar *buf, size_t buf_len)\n {\n-\tstruct input_zstream_data *data = container_of(in_stream, struct input_zstream_data, base);\n+\tstruct zlib_stream *data = container_of(in_stream, struct zlib_stream, base);\n \tgit_zstream *zstream = data->zstream;\n \n \tif (data->status != Z_OK)\n@@ -389,9 +389,9 @@ static ssize_t feed_input_zstream(struct odb_stream *in_stream,\n static void stream_blob(unsigned long size, unsigned nr)\n {\n \tgit_zstream zstream = { 0 };\n-\tstruct input_zstream_data in_stream = {\n+\tstruct zlib_stream in_stream = {\n \t\t.base = {\n-\t\t\t.read = feed_input_zstream,\n+\t\t\t.read = zlib_stream_read,\n \t\t\t.size = size,\n \t\t\t.type = OBJ_BLOB,\n \t\t},\n\n-- \n2.55.0.679.g6767b8d81c.dirty\n\n"},{"id":"549656","messageId":"20260805-pks-odb-stream-unification-v2-8-b8c369564641@pks.im","threadId":"66111","inReplyTo":"20260805-pks-odb-stream-unification-v2-0-b8c369564641@pks.im","subject":"[PATCH v2 8/8] odb/streaming: unify function names to create new streams","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-05T07:44:52Z","receivedAt":"2026-08-05T07:45:19Z","isPatch":true,"body":"Unify the function names to create new streams from different sources so\nthat they follow a common schema. While at it, document the ownership of\nthe file descriptor passed to `odb_stream_from_fd()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n archive-tar.c | 2 +-\n archive-zip.c | 2 +-\n builtin/index-pack.c | 2 +-\n builtin/pack-objects.c | 4 ++--\n object-file.c | 4 ++--\n object.c | 2 +-\n odb/streaming.c | 10 +++++-----\n odb/streaming.h | 23 +++++++++++++----------\n 8 files changed, 26 insertions(+), 23 deletions(-)\n\ndiff --git a/archive-tar.c b/archive-tar.c\nindex df2d7fb8e9..a1c66024d4 100644\n--- a/archive-tar.c\n+++ b/archive-tar.c\n@@ -133,7 +133,7 @@ static int stream_blocked(struct repository *r, const struct object_id *oid)\n \tchar buf[BLOCKSIZE];\n \tssize_t readlen;\n \n-\tst = odb_read_stream_open(r->objects, oid, NULL);\n+\tst = odb_stream_from_object(r->objects, oid, NULL);\n \tif (!st)\n \t\treturn error(_(\"cannot stream blob %s\"), oid_to_hex(oid));\n \tfor (;;) {\ndiff --git a/archive-zip.c b/archive-zip.c\nindex 8095fd04d5..1a948c2f83 100644\n--- a/archive-zip.c\n+++ b/archive-zip.c\n@@ -347,7 +347,7 @@ static int write_zip_entry(struct archiver_args *args,\n \t\t\tmethod = ZIP_METHOD_DEFLATE;\n \n \t\tif (!buffer) {\n-\t\t\tstream = odb_read_stream_open(args->repo->objects, oid, NULL);\n+\t\t\tstream = odb_stream_from_object(args->repo->objects, oid, NULL);\n \t\t\tif (!stream)\n \t\t\t\treturn error(_(\"cannot stream blob %s\"),\n \t\t\t\t\t oid_to_hex(oid));\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 7226da3e65..d1761282db 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -806,7 +806,7 @@ static int check_collison(struct object_entry *entry)\n \n \tmemset(&data, 0, sizeof(data));\n \tdata.entry = entry;\n-\tdata.st = odb_read_stream_open(the_repository->objects, &entry->idx.oid, NULL);\n+\tdata.st = odb_stream_from_object(the_repository->objects, &entry->idx.oid, NULL);\n \tif (!data.st)\n \t\treturn -1;\n \tif (data.st->size != entry->size || data.st->type != entry->type)\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 683160c6bb..10d00ca792 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -528,8 +528,8 @@ static unsigned long write_no_reuse_object(struct hashfile *f, struct object_ent\n \t\tif (oe_type(entry) == OBJ_BLOB &&\n \t\t oe_size_greater_than(&to_pack, entry,\n \t\t\t\t\t repo_settings_get_big_file_threshold(the_repository)) &&\n-\t\t (st = odb_read_stream_open(the_repository->objects, &entry->idx.oid,\n-\t\t\t\t\t NULL)) != NULL) {\n+\t\t (st = odb_stream_from_object(the_repository->objects, &entry->idx.oid,\n+\t\t\t\t\t\t NULL)) != NULL) {\n \t\t\tbuf = NULL;\n \t\t\ttype = st->type;\n \t\t\tsize = st->size;\ndiff --git a/object-file.c b/object-file.c\nindex 068c6e5672..11d1af342e 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -952,8 +952,8 @@ int index_fd(struct index_state *istate, struct object_id *oid,\n \t\tret = index_core(istate, oid, fd, xsize_t(st->st_size),\n \t\t\t\t type, path, flags);\n \t} else {\n-\t\tstruct odb_stream *stream = odb_write_stream_from_fd(fd, xsize_t(st->st_size),\n-\t\t\t\t\t\t\t\t OBJ_BLOB);\n+\t\tstruct odb_stream *stream = odb_stream_from_fd(fd, xsize_t(st->st_size),\n+\t\t\t\t\t\t\t OBJ_BLOB);\n \n \t\tif (flags & INDEX_WRITE_OBJECT) {\n \t\t\tstruct object_database *odb = the_repository->objects;\ndiff --git a/object.c b/object.c\nindex 37e6efee47..97f7fc0e87 100644\n--- a/object.c\n+++ b/object.c\n@@ -345,7 +345,7 @@ struct object *parse_object_with_flags(struct repository *r,\n \tif ((!obj || obj->type == OBJ_NONE || obj->type == OBJ_BLOB) &&\n \t odb_read_object_info(r->objects, oid, NULL) == OBJ_BLOB) {\n \t\tif (!skip_hash) {\n-\t\t\tstruct odb_stream *stream = odb_read_stream_open(r->objects, oid, NULL);\n+\t\t\tstruct odb_stream *stream = odb_stream_from_object(r->objects, oid, NULL);\n \n \t\t\tif (!stream) {\n \t\t\t\terror(_(\"unable to open object stream for %s\"), oid_to_hex(oid));\ndiff --git a/odb/streaming.c b/odb/streaming.c\nindex c436b18d39..9c85ec54f5 100644\n--- a/odb/streaming.c\n+++ b/odb/streaming.c\n@@ -208,9 +208,9 @@ ssize_t odb_stream_read(struct odb_stream *st, void *buf, size_t sz)\n \treturn st->read(st, buf, sz);\n }\n \n-struct odb_stream *odb_read_stream_open(struct object_database *odb,\n-\t\t\t\t\tconst struct object_id *oid,\n-\t\t\t\t\tstruct stream_filter *filter)\n+struct odb_stream *odb_stream_from_object(struct object_database *odb,\n+\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t struct stream_filter *filter)\n {\n \tstruct odb_stream *st;\n \tconst struct object_id *real = lookup_replace_object(odb->repo, oid);\n@@ -242,7 +242,7 @@ int odb_stream_blob_to_fd(struct object_database *odb,\n \tssize_t kept = 0;\n \tint result = -1;\n \n-\tst = odb_read_stream_open(odb, oid, filter);\n+\tst = odb_stream_from_object(odb, oid, filter);\n \tif (!st) {\n \t\tif (filter)\n \t\t\tfree_stream_filter(filter);\n@@ -320,7 +320,7 @@ static int fd_stream_close(struct odb_stream *stream UNUSED)\n \treturn 0;\n }\n \n-struct odb_stream *odb_write_stream_from_fd(int fd, size_t size, enum object_type type)\n+struct odb_stream *odb_stream_from_fd(int fd, size_t size, enum object_type type)\n {\n \tstruct fd_stream *fds;\n \ndiff --git a/odb/streaming.h b/odb/streaming.h\nindex 60b9803190..b522ff513f 100644\n--- a/odb/streaming.h\n+++ b/odb/streaming.h\n@@ -26,14 +26,22 @@ struct odb_stream {\n };\n \n /*\n- * Create a new object stream for the given object database. An optional filter\n- * can be used to transform the object's content.\n+ * Create a new object stream for the given object. An optional filter can be\n+ * used to transform the object's content.\n *\n * Returns the stream on success, a `NULL` pointer otherwise.\n */\n-struct odb_stream *odb_read_stream_open(struct object_database *odb,\n-\t\t\t\t\tconst struct object_id *oid,\n-\t\t\t\t\tstruct stream_filter *filter);\n+struct odb_stream *odb_stream_from_object(struct object_database *odb,\n+\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t struct stream_filter *filter);\n+\n+/*\n+ * Create a new object stream for the given file descriptor. This can be used\n+ * to, for example, stream an object into the object database. This function\n+ * does _not_ take ownership of the file descriptor. It's the responsibility of\n+ * the caller to close it after the stream has been closed.\n+ */\n+struct odb_stream *odb_stream_from_fd(int fd, size_t size, enum object_type type);\n \n /*\n * Close the given object stream and release all resources associated with it.\n@@ -65,9 +73,4 @@ int odb_stream_blob_to_fd(struct object_database *odb,\n \t\t\t struct stream_filter *filter,\n \t\t\t int can_seek);\n \n-/*\n- * Sets up an ODB write stream that reads from an fd.\n- */\n-struct odb_stream *odb_write_stream_from_fd(int fd, size_t size, enum object_type type);\n-\n #endif /* STREAMING_H */\n\n-- \n2.55.0.679.g6767b8d81c.dirty\n\n"},{"id":"550274","messageId":"CAOLa=ZTHaiARd2F7BL+uwN8ANNb6=ovfZ5v4=dMkgCY=N6qa7Q@mail.gmail.com","threadId":"66111","inReplyTo":"20260805-pks-odb-stream-unification-v2-1-b8c369564641@pks.im","subject":"Re: [PATCH v2 1/8] odb/streaming: track write stream size in the structure","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-08-11T09:51:38Z","receivedAt":"2026-08-11T09:51:40Z","isPatch":true,"body":"Patrick Steinhardt writes:\n\n> When passing around a `struct odb_write_stream` we typically also have\n> to pass the number of bytes that the stream will yield. This is required\n> because the object header itself contains that size, and consequently we\n> cannot write the header without that information.\n>\n> Move this information into the stream itself so that it becomes self-\n> describing. In addition to that, this also brings the `struct\n> odb_write_stream` a bit closer to the `struct odb_read_stream` so that\n> we can eventually merge both stream types.\n>\n\nOkay, so this will be similar to `odb_read_stream.size`. Makes sense.\n\n[snip]\n\n> diff --git a/odb/streaming.c b/odb/streaming.c\n> index 20531e864c..38c2f6687c 100644\n> --- a/odb/streaming.c\n> +++ b/odb/streaming.c\n> @@ -336,5 +336,6 @@ void odb_write_stream_from_fd(struct odb_write_stream *stream, int fd,\n>\n> \tstream->data = data;\n> \tstream->read = read_object_fd;\n> +\tstream->size = size;\n> \tstream->is_finished = 0;\n> }\n> diff --git a/odb/streaming.h b/odb/streaming.h\n> index c023671780..4d7d31b5aa 100644\n> --- a/odb/streaming.h\n> +++ b/odb/streaming.h\n> @@ -55,6 +55,7 @@ ssize_t odb_read_stream_read(struct odb_read_stream *stream, void *buf, size_t l\n> struct odb_write_stream {\n> \tssize_t (*read)(struct odb_write_stream *, unsigned char *, size_t);\n> \tvoid *data;\n> +\tsize_t size;\n> \tint is_finished;\n> };\n>\n\nOkay so this is the main change. Looks good.\n\n[snip]\n"},{"id":"550276","messageId":"CAOLa=ZQtdUKeuhNbxLC3kBTT9JbxgM8wJUGCzPNKJmEEscA4TA@mail.gmail.com","threadId":"66111","inReplyTo":"20260805-pks-odb-stream-unification-v2-2-b8c369564641@pks.im","subject":"Re: [PATCH v2 2/8] odb/streaming: drop `is_finished` field","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-08-11T10:02:25Z","receivedAt":"2026-08-11T10:02:27Z","isPatch":true,"body":"Patrick Steinhardt writes:\n\n[snip]\n\n> diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\n> index f3e0b504f4..b7c486ea94 100644\n> --- a/builtin/unpack-objects.c\n> +++ b/builtin/unpack-objects.c\n> @@ -368,20 +368,20 @@ static ssize_t feed_input_zstream(struct odb_write_stream *in_stream,\n> {\n> \tstruct input_zstream_data *data = in_stream->data;\n> \tgit_zstream *zstream = data->zstream;\n> -\tvoid *in = fill(1);\n>\n> -\tif (in_stream->is_finished)\n> +\tif (data->status != Z_OK)\n> \t\treturn 0;\n>\n> \tzstream->next_out = buf;\n> \tzstream->avail_out = buf_len;\n> -\tzstream->next_in = in;\n> -\tzstream->avail_in = len;\n>\n> -\tdata->status = git_inflate(zstream, 0);\n> +\twhile (data->status == Z_OK && zstream->avail_out == buf_len) {\n> +\t\tzstream->next_in = fill(1);\n> +\t\tzstream->avail_in = len;\n> +\t\tdata->status = git_inflate(zstream, 0);\n> +\t\tuse(len - zstream->avail_in);\n> +\t}\n>\n> -\tin_stream->is_finished = data->status != Z_OK;\n> -\tuse(len - zstream->avail_in);\n> \treturn buf_len - zstream->avail_out;\n> }\n>\n\nSo we have a bunch of global variables used for zstream parsing. The\nloop ensures that we keep trying until we get some data. The use()\nfunction manipulate `len` accordingly for the next iteration..\n\n[snip]\n"},{"id":"550277","messageId":"CAOLa=ZTi8tL896_F2ONQck0z+H8NYhzcbTorb80NOdiqvnpjNg@mail.gmail.com","threadId":"66111","inReplyTo":"20260805-pks-odb-stream-unification-v2-5-b8c369564641@pks.im","subject":"Re: [PATCH v2 5/8] odb/streaming: consolidate read and write streams","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-08-11T10:12:05Z","receivedAt":"2026-08-11T10:12:12Z","isPatch":true,"body":"Patrick Steinhardt writes:\n\n> The `struct odb_read_stream` and `struct odb_write_stream` both provide\n> the same functionality: they allow a caller to read object data from an\n> arbitrary source. Historically, the only difference was that the read\n> stream was used to read data out of the object database, whereas the\n> write stream was used to write data into the object database, but the\n> interfaces were mostly the same.\n>\n> Over the preceding commits we have refactored the write stream to have\n> almost exactly the same interface as the read stream. With these\n> refactorings we can now easily merge those two streams into a single\n> interface that's used for both use cases.\n>\n> While most of the changes are mechanical, there are two sites that need\n> special mention:\n>\n> - \"builtin/unpack-objects.c\" creates a write stream from compressed\n> object data.\n>\n> - \"odb/streaming.c\" creates a write stream from a file descriptor.\n>\n> Adapting these sites to yield the new stream type requires a couple more\n> changes. Most importantly, instead of embedding the pointer to the data\n> in `struct odb_write_stream`, we now allocate a structure that wraps the\n> new `struct odb_stream` base. Other than that though, the changes are\n> rather straight forward.\n>\n\nSo instead of `odb_write_stream.data` which was pointing to the data, we\nsimply wrap the stream with the data's structure, this allows us to get\nthe parent struct if we have the `odb_stream`. Alright!\n\n> Some of the structures and functions are now somewhat misnamed. These\n> will be fixed in subsequent commits.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> builtin/unpack-objects.c | 31 ++++++++++++++++---------------\n> object-file.c | 25 ++++++++++++-------------\n> odb.c | 2 +-\n> odb.h | 4 ++--\n> odb/source-files.c | 2 +-\n> odb/source-inmemory.c | 4 ++--\n> odb/source-loose.c | 6 +++---\n> odb/source-packed.c | 2 +-\n> odb/source.h | 4 ++--\n> odb/streaming.c | 35 ++++++++++++++++-------------------\n> odb/streaming.h | 31 +++----------------------------\n> odb/transaction.c | 2 +-\n> odb/transaction.h | 4 ++--\n> t/unit-tests/u-odb-inmemory.c | 6 +++---\n> 14 files changed, 65 insertions(+), 93 deletions(-)\n>\n> diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\n> index 7439ec53be..05a2d48011 100644\n> --- a/builtin/unpack-objects.c\n> +++ b/builtin/unpack-objects.c\n> @@ -359,20 +359,21 @@ static void unpack_non_delta_entry(enum object_type type, unsigned long size,\n> }\n>\n> struct input_zstream_data {\n> +\tstruct odb_stream base;\n> \tgit_zstream *zstream;\n> \tint status;\n> };\n>\n> -static ssize_t feed_input_zstream(struct odb_write_stream *in_stream,\n> -\t\t\t\t unsigned char *buf, size_t buf_len)\n> +static ssize_t feed_input_zstream(struct odb_stream *in_stream,\n> +\t\t\t\t char *buf, size_t buf_len)\n> {\n> -\tstruct input_zstream_data *data = in_stream->data;\n> +\tstruct input_zstream_data *data = container_of(in_stream, struct input_zstream_data, base);\n> \tgit_zstream *zstream = data->zstream;\n>\n> \tif (data->status != Z_OK)\n> \t\treturn 0;\n>\n> -\tzstream->next_out = buf;\n> +\tzstream->next_out = (unsigned char *) buf;\n\nBut we do loose information here since we now use 'char *' for both\nread/write. But we gain flexibility.\n\n[snip]\n\nThe rest looks good.\n"},{"id":"550278","messageId":"CAOLa=ZTtn4kpQq6H8gJpEnC9RRbb=eFgKjxGEQyeJGYr5CcW2Q@mail.gmail.com","threadId":"66111","inReplyTo":"20260805-pks-odb-stream-unification-v2-0-b8c369564641@pks.im","subject":"Re: [PATCH v2 0/8] odb: unify read and write streams","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-08-11T10:13:05Z","receivedAt":"2026-08-11T10:13:07Z","isPatch":true,"body":"Patrick Steinhardt writes:\n\n> Hi,\n>\n> we have two different kind of object database streams in our code base:\n> `odb_write_stream` and `odb_read_stream`. While those are used for\n> different use cases, the provided functionality is ultimately the exact\n> same.\n>\n> This patch series thus refactors these streams so that we have a single\n> `odb_stream`, only. This allows us to reuse the streams for different\n> kinds of purposes and makes them more generally useful overall. For\n> example, it's trivially possible now to create an object stream for any\n> given object and then write that stream into a different source.\n>\n> The series is built on top of 5b2471720c (The 10th batch, 2026-08-03).\n>\n> Changes in v2:\n> - Use the correct object type when hashing in-memory objects.\n> - Remove a stale comment.\n> - Adapt a commit message to mention that renames will follow in\n> subsequent commits.\n> - Add another commit to rename `struct input_zstream_data`.\n> - Link to v1: https://patch.msgid.link/20260804-pks-odb-stream-unification-v1-0-86d70e82345e@pks.im\n>\n\nI went through v2 and I think its already in a good state!\n"},{"id":"550279","messageId":"ansPTZ5oV9JiFx2h@pks.im","threadId":"66111","inReplyTo":"CAOLa=ZTtn4kpQq6H8gJpEnC9RRbb=eFgKjxGEQyeJGYr5CcW2Q@mail.gmail.com","subject":"Re: [PATCH v2 0/8] odb: unify read and write streams","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-11T12:02:21Z","receivedAt":"2026-08-11T12:02:30Z","isPatch":true,"body":"On Tue, Aug 11, 2026 at 06:13:05AM -0400, Karthik Nayak wrote:\n> Patrick Steinhardt writes:\n> \n> > Hi,\n> >\n> > we have two different kind of object database streams in our code base:\n> > `odb_write_stream` and `odb_read_stream`. While those are used for\n> > different use cases, the provided functionality is ultimately the exact\n> > same.\n> >\n> > This patch series thus refactors these streams so that we have a single\n> > `odb_stream`, only. This allows us to reuse the streams for different\n> > kinds of purposes and makes them more generally useful overall. For\n> > example, it's trivially possible now to create an object stream for any\n> > given object and then write that stream into a different source.\n> >\n> > The series is built on top of 5b2471720c (The 10th batch, 2026-08-03).\n> >\n> > Changes in v2:\n> > - Use the correct object type when hashing in-memory objects.\n> > - Remove a stale comment.\n> > - Adapt a commit message to mention that renames will follow in\n> > subsequent commits.\n> > - Add another commit to rename `struct input_zstream_data`.\n> > - Link to v1: https://patch.msgid.link/20260804-pks-odb-stream-unification-v1-0-86d70e82345e@pks.im\n> >\n> \n> I went through v2 and I think its already in a good state!\n\nThanks for your review!\n\nPatrick\n"},{"id":"550325","messageId":"anuBdm29ye_qV_Rq@denethor","threadId":"66111","inReplyTo":"20260805-pks-odb-stream-unification-v2-0-b8c369564641@pks.im","subject":"Re: [PATCH v2 0/8] odb: unify read and write streams","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-08-11T20:09:57Z","receivedAt":"2026-08-11T20:10:02Z","isPatch":true,"body":"On 26/08/05 09:44AM, Patrick Steinhardt wrote:\n> Changes in v2:\n> - Use the correct object type when hashing in-memory objects.\n> - Remove a stale comment.\n> - Adapt a commit message to mention that renames will follow in\n> subsequent commits.\n> - Add another commit to rename `struct input_zstream_data`.\n> - Link to v1: https://patch.msgid.link/20260804-pks-odb-stream-unification-v1-0-86d70e82345e@pks.im\n\nThis version of the series addresses all my previous feedback and looks\ngood to me. Thanks.\n\n-Justin\n"}]}