Re: [PATCH 4/6] object-file: avoid fd seekback by checking object size upfront
- From
Patrick Steinhardt <ps@pks.im>
- Date
- Mar 31, 2026, 07:48 UTC
- Message-ID
- <act8W1BEg6iyUpHB@pks.im>
- In-Reply-To
- <20260331033835.2863514-5-jltobler@gmail.com>
On Mon, Mar 30, 2026 at 10:38:33PM -0500, Justin Tobler wrote:
Show 21 quoted lines
> In certain scenarios, Git handles writing blobs that exceed > "core.bigFilesThreshold" differently by streaming the object directly > into a packfile. When there is an active ODB transaction, these blobs > are streamed to the same packfile instead of using a separate packfile > for each. If "pack.packSizeLimit" is configured and streaming another > object causes the packfile to exceed the configured limit, the packfile > is truncated back to the previous object and the object write is > restarted in a new packfile. > > This works fine, but requires the fd being read from to save a > checkpoint so it becomes possible to rewind the input source via seeking > back to a known offset at the beginning. In a subsequent commit, blob > streaming is converted to use `struct odb_write_stream` as a more > generic input source instead of an fd which doesn't provide a mechanism > for rewinding. > > For this use case though, rewinding the fd is not strictly necessary > because the inflated size of the object is known and can be used to > approximate whether writing the object would cause the packfile to > exceed the configured limit prior to writing anything. These blobs > written to the packfile are never deltafied thus the size difference
s/deltafied/deltified/
Show 7 quoted lines
> between what is written versus the inflated size is due to zlib > compression. While this does prevent packfiles from being filled to the > potential maximum is some cases, it should be good enough and still > prevents the packfile from exceeding any configured limit. > > Use the inflated blob size to determine whether writing an object to a > packfile will exceed the configured "pack.packSizeLimit".
I agree that this is a reasonable tradeoff:
- For small objects it's probably not going to make a huge difference,
as we'd at most waste a couple kilobytes. - For large objects we can expect that we wouldn't use deltification
anyway due to "core.bigFileThreshold". - We can expect that in many cases large files will be not compress
well, either, as it's more likely than not that a file >512MB (our
default limit for "core.bigFileThreshold") is going to be a binary
file.We also nicely document "pack.packSizeLimit" as "rarely useful, and may result in a larger total on-disk size" in git-config(1), so I think it's fair to not bend ourselves over backwards just to make a rarely-useful feature work exactly the same.
Show 20 quoted lines
> diff --git a/object-file.c b/object-file.c
> index 493173eaf4..1de2244ac5 100644
> --- a/object-file.c
> +++ b/object-file.c
> @@ -1473,15 +1461,10 @@ static int stream_blob_to_pack(struct transaction_packfile *state,
> if ((size_t)read_result != rsize)
> die("failed to read %u bytes from '%s'",
> (unsigned)rsize, path);
> - offset += rsize;
> - if (*already_hashed_to < offset) {
> - size_t hsize = offset - *already_hashed_to;
> - if (rsize < hsize)
> - hsize = rsize;
> - if (hsize)
> - git_hash_update(ctx, ibuf, hsize);
> - *already_hashed_to = offset;
> - }
> +
> + if (rsize)
> + git_hash_update(ctx, ibuf, rsize);Is this guard really needed? I wouldn't expect that we ever try to read zero bytes into `ibuf`, and we bail in case we didn't receive the expected number of bytes.
And even if we did, `git_hash_update()` works just fine with no data.
Show 14 quoted lines
> @@ -1592,48 +1566,34 @@ static int index_blob_packfile_transaction(struct odb_transaction_files *transac
> size_t size, const char *path)
> {
> struct transaction_packfile *state = &transaction->packfile;
> - off_t seekback, already_hashed_to;
> struct git_hash_ctx ctx;
> unsigned char obuf[16384];
> unsigned header_len;
> struct hashfile_checkpoint checkpoint;
> struct pack_idx_entry *idx;
>
> - seekback = lseek(fd, 0, SEEK_CUR);
> - if (seekback == (off_t)-1)
> - return error("cannot find the current offset");Okay, no seeking necessary because we don't restart the write anymore.
Show 10 quoted lines
> header_len = format_object_header((char *)obuf, sizeof(obuf), > OBJ_BLOB, size); > transaction->base.source->odb->repo->hash_algo->init_fn(&ctx); > git_hash_update(&ctx, obuf, header_len); > > + /* > + * If writing another object to the packfile could result in it > + * exceeding the configured size limit, flush the current packfile > + * transaction. > + */
Do we want to document that this intentionally works on the inflated size, not the deflated one, with the arguments mentioned in the commit message?
> + if (state->nr_written && pack_size_limit_cfg && > + pack_size_limit_cfg < state->offset + size) > + flush_packfile_transaction(transaction);
And we now flush the packfile before writing any object that may cause us to bust the size limit. Makes sense.
Show 26 quoted lines
> CALLOC_ARRAY(idx, 1);
> prepare_packfile_transaction(transaction);
> hashfile_checkpoint_init(state->f, &checkpoint);
>
> - already_hashed_to = 0;
> -
> - while (1) {
> - prepare_packfile_transaction(transaction);
> - hashfile_checkpoint(state->f, &checkpoint);
> - idx->offset = state->offset;
> - crc32_begin(state->f);
> -
> - if (!stream_blob_to_pack(state, &ctx, &already_hashed_to,
> - fd, size, path))
> - break;
> - /*
> - * Writing this object to the current pack will make
> - * it too big; we need to truncate it, start a new
> - * pack, and write into it.
> - */
> - hashfile_truncate(state->f, &checkpoint);
> - state->offset = checkpoint.offset;
> - flush_packfile_transaction(transaction);
> - if (lseek(fd, seekback, SEEK_SET) == (off_t)-1)
> - return error("cannot seek back");
> - }Hm. I was briefly wondering whether we'd loop indefinitely in case the object alone is bigger than the packsize. But we have an escape hatch in `stream_blob_to_pack()` that special-cases when we haven't written any data yet, so the answer is "no".
Show 5 quoted lines
> + hashfile_checkpoint(state->f, &checkpoint); > + idx->offset = state->offset; > + crc32_begin(state->f); > + stream_blob_to_pack(state, &ctx, fd, size, path); > git_hash_final_oid(result_oid, &ctx);
Thanks!
Patrick