From: Justin Tobler Date: Tue, 31 Mar 2026 14:14:02 GMT Subject: Re: [PATCH 4/6] object-file: avoid fd seekback by checking object size upfront Message-ID: In-Reply-To: On 26/03/31 09:48AM, Patrick Steinhardt wrote: > On Mon, Mar 30, 2026 at 10:38:33PM -0500, Justin Tobler wrote: > > In certain scenarios, Git handles writing blobs that exceed > > "core.bigFilesThreshold" differently by streaming the object directly > > into a packfile. When there is an active ODB transaction, these blobs > > are streamed to the same packfile instead of using a separate packfile > > for each. If "pack.packSizeLimit" is configured and streaming another > > object causes the packfile to exceed the configured limit, the packfile > > is truncated back to the previous object and the object write is > > restarted in a new packfile. > > > > This works fine, but requires the fd being read from to save a > > checkpoint so it becomes possible to rewind the input source via seeking > > back to a known offset at the beginning. In a subsequent commit, blob > > streaming is converted to use `struct odb_write_stream` as a more > > generic input source instead of an fd which doesn't provide a mechanism > > for rewinding. > > > > For this use case though, rewinding the fd is not strictly necessary > > because the inflated size of the object is known and can be used to > > approximate whether writing the object would cause the packfile to > > exceed the configured limit prior to writing anything. These blobs > > written to the packfile are never deltafied thus the size difference > > s/deltafied/deltified/ Will fix. > > diff --git a/object-file.c b/object-file.c > > index 493173eaf4..1de2244ac5 100644 > > --- a/object-file.c > > +++ b/object-file.c > > @@ -1473,15 +1461,10 @@ static int stream_blob_to_pack(struct transaction_packfile *state, > > if ((size_t)read_result != rsize) > > die("failed to read %u bytes from '%s'", > > (unsigned)rsize, path); > > - offset += rsize; > > - if (*already_hashed_to < offset) { > > - size_t hsize = offset - *already_hashed_to; > > - if (rsize < hsize) > > - hsize = rsize; > > - if (hsize) > > - git_hash_update(ctx, ibuf, hsize); > > - *already_hashed_to = offset; > > - } > > + > > + if (rsize) > > + git_hash_update(ctx, ibuf, rsize); > > Is this guard really needed? I wouldn't expect that we ever try to read > zero bytes into `ibuf`, and we bail in case we didn't receive the > expected number of bytes. > > And even if we did, `git_hash_update()` works just fine with no data. Ya you are right, this guard is not needed. Will remove in the next version. > > header_len = format_object_header((char *)obuf, sizeof(obuf), > > OBJ_BLOB, size); > > transaction->base.source->odb->repo->hash_algo->init_fn(&ctx); > > git_hash_update(&ctx, obuf, header_len); > > > > + /* > > + * If writing another object to the packfile could result in it > > + * exceeding the configured size limit, flush the current packfile > > + * transaction. > > + */ > > Do we want to document that this intentionally works on the inflated > size, not the deflated one, with the arguments mentioned in the commit > message? Good suggestion. Will update. Thanks, -Justin