Re: [PATCH 4/6] object-file: avoid fd seekback by checking object size upfront
- From
Justin Tobler <jltobler@gmail.com>
- Date
- Mar 31, 2026, 14:14 UTC
- Message-ID
- <acvV0_7DqGy_q9GY@denethor>
- In-Reply-To
- <act8W1BEg6iyUpHB@pks.im>
On 26/03/31 09:48AM, Patrick Steinhardt wrote:
Show 24 quoted lines
> On Mon, Mar 30, 2026 at 10:38:33PM -0500, Justin Tobler wrote: > > In certain scenarios, Git handles writing blobs that exceed > > "core.bigFilesThreshold" differently by streaming the object directly > > into a packfile. When there is an active ODB transaction, these blobs > > are streamed to the same packfile instead of using a separate packfile > > for each. If "pack.packSizeLimit" is configured and streaming another > > object causes the packfile to exceed the configured limit, the packfile > > is truncated back to the previous object and the object write is > > restarted in a new packfile. > > > > This works fine, but requires the fd being read from to save a > > checkpoint so it becomes possible to rewind the input source via seeking > > back to a known offset at the beginning. In a subsequent commit, blob > > streaming is converted to use `struct odb_write_stream` as a more > > generic input source instead of an fd which doesn't provide a mechanism > > for rewinding. > > > > For this use case though, rewinding the fd is not strictly necessary > > because the inflated size of the object is known and can be used to > > approximate whether writing the object would cause the packfile to > > exceed the configured limit prior to writing anything. These blobs > > written to the packfile are never deltafied thus the size difference > > s/deltafied/deltified/
Will fix.
Show 26 quoted lines
> > diff --git a/object-file.c b/object-file.c
> > index 493173eaf4..1de2244ac5 100644
> > --- a/object-file.c
> > +++ b/object-file.c
> > @@ -1473,15 +1461,10 @@ static int stream_blob_to_pack(struct transaction_packfile *state,
> > if ((size_t)read_result != rsize)
> > die("failed to read %u bytes from '%s'",
> > (unsigned)rsize, path);
> > - offset += rsize;
> > - if (*already_hashed_to < offset) {
> > - size_t hsize = offset - *already_hashed_to;
> > - if (rsize < hsize)
> > - hsize = rsize;
> > - if (hsize)
> > - git_hash_update(ctx, ibuf, hsize);
> > - *already_hashed_to = offset;
> > - }
> > +
> > + if (rsize)
> > + git_hash_update(ctx, ibuf, rsize);
>
> Is this guard really needed? I wouldn't expect that we ever try to read
> zero bytes into `ibuf`, and we bail in case we didn't receive the
> expected number of bytes.
>
> And even if we did, `git_hash_update()` works just fine with no data.Ya you are right, this guard is not needed. Will remove in the next version.
Show 14 quoted lines
> > header_len = format_object_header((char *)obuf, sizeof(obuf), > > OBJ_BLOB, size); > > transaction->base.source->odb->repo->hash_algo->init_fn(&ctx); > > git_hash_update(&ctx, obuf, header_len); > > > > + /* > > + * If writing another object to the packfile could result in it > > + * exceeding the configured size limit, flush the current packfile > > + * transaction. > > + */ > > Do we want to document that this intentionally works on the inflated > size, not the deflated one, with the arguments mentioned in the commit > message?
Good suggestion. Will update.
Thanks, -Justin