git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 4/6] object-file: avoid fd seekback by checking object size upfront

From
Patrick Steinhardt <ps@pks.im>
Date
Mar 31, 2026, 07:48 UTC
Message-ID
<act8W1BEg6iyUpHB@pks.im>
In-Reply-To
<20260331033835.2863514-5-jltobler@gmail.com>
On Mon, Mar 30, 2026 at 10:38:33PM -0500, Justin Tobler wrote:
Show 21 quoted lines
> In certain scenarios, Git handles writing blobs that exceed
> "core.bigFilesThreshold" differently by streaming the object directly
> into a packfile. When there is an active ODB transaction, these blobs
> are streamed to the same packfile instead of using a separate packfile
> for each. If "pack.packSizeLimit" is configured and streaming another
> object causes the packfile to exceed the configured limit, the packfile
> is truncated back to the previous object and the object write is
> restarted in a new packfile.
> 
> This works fine, but requires the fd being read from to save a
> checkpoint so it becomes possible to rewind the input source via seeking
> back to a known offset at the beginning. In a subsequent commit, blob
> streaming is converted to use `struct odb_write_stream` as a more
> generic input source instead of an fd which doesn't provide a mechanism
> for rewinding.
> 
> For this use case though, rewinding the fd is not strictly necessary
> because the inflated size of the object is known and can be used to
> approximate whether writing the object would cause the packfile to
> exceed the configured limit prior to writing anything. These blobs
> written to the packfile are never deltafied thus the size difference
s/deltafied/deltified/
Show 7 quoted lines
> between what is written versus the inflated size is due to zlib
> compression. While this does prevent packfiles from being filled to the
> potential maximum is some cases, it should be good enough and still
> prevents the packfile from exceeding any configured limit.
> 
> Use the inflated blob size to determine whether writing an object to a
> packfile will exceed the configured "pack.packSizeLimit".
I agree that this is a reasonable tradeoff:
  - For small objects it's probably not going to make a huge difference,
    as we'd at most waste a couple kilobytes.
  - For large objects we can expect that we wouldn't use deltification
    anyway due to "core.bigFileThreshold".
  - We can expect that in many cases large files will be not compress
    well, either, as it's more likely than not that a file >512MB (our
    default limit for "core.bigFileThreshold") is going to be a binary
    file.

We also nicely document "pack.packSizeLimit" as "rarely useful, and may result in a larger total on-disk size" in git-config(1), so I think it's fair to not bend ourselves over backwards just to make a rarely-useful feature work exactly the same.

Show 20 quoted lines
> diff --git a/object-file.c b/object-file.c
> index 493173eaf4..1de2244ac5 100644
> --- a/object-file.c
> +++ b/object-file.c
> @@ -1473,15 +1461,10 @@ static int stream_blob_to_pack(struct transaction_packfile *state,
>  			if ((size_t)read_result != rsize)
>  				die("failed to read %u bytes from '%s'",
>  				    (unsigned)rsize, path);
> -			offset += rsize;
> -			if (*already_hashed_to < offset) {
> -				size_t hsize = offset - *already_hashed_to;
> -				if (rsize < hsize)
> -					hsize = rsize;
> -				if (hsize)
> -					git_hash_update(ctx, ibuf, hsize);
> -				*already_hashed_to = offset;
> -			}
> +
> +			if (rsize)
> +				git_hash_update(ctx, ibuf, rsize);

Is this guard really needed? I wouldn't expect that we ever try to read zero bytes into `ibuf`, and we bail in case we didn't receive the expected number of bytes.

And even if we did, `git_hash_update()` works just fine with no data.
Show 14 quoted lines
> @@ -1592,48 +1566,34 @@ static int index_blob_packfile_transaction(struct odb_transaction_files *transac
>  					   size_t size, const char *path)
>  {
>  	struct transaction_packfile *state = &transaction->packfile;
> -	off_t seekback, already_hashed_to;
>  	struct git_hash_ctx ctx;
>  	unsigned char obuf[16384];
>  	unsigned header_len;
>  	struct hashfile_checkpoint checkpoint;
>  	struct pack_idx_entry *idx;
>  
> -	seekback = lseek(fd, 0, SEEK_CUR);
> -	if (seekback == (off_t)-1)
> -		return error("cannot find the current offset");
Okay, no seeking necessary because we don't restart the write anymore.
Show 10 quoted lines
>  	header_len = format_object_header((char *)obuf, sizeof(obuf),
>  					  OBJ_BLOB, size);
>  	transaction->base.source->odb->repo->hash_algo->init_fn(&ctx);
>  	git_hash_update(&ctx, obuf, header_len);
>  
> +	/*
> +	 * If writing another object to the packfile could result in it
> +	 * exceeding the configured size limit, flush the current packfile
> +	 * transaction.
> +	 */

Do we want to document that this intentionally works on the inflated size, not the deflated one, with the arguments mentioned in the commit message?

> +	if (state->nr_written && pack_size_limit_cfg &&
> +	    pack_size_limit_cfg < state->offset + size)
> +		flush_packfile_transaction(transaction);

And we now flush the packfile before writing any object that may cause us to bust the size limit. Makes sense.

Show 26 quoted lines
>  	CALLOC_ARRAY(idx, 1);
>  	prepare_packfile_transaction(transaction);
>  	hashfile_checkpoint_init(state->f, &checkpoint);
>  
> -	already_hashed_to = 0;
> -
> -	while (1) {
> -		prepare_packfile_transaction(transaction);
> -		hashfile_checkpoint(state->f, &checkpoint);
> -		idx->offset = state->offset;
> -		crc32_begin(state->f);
> -
> -		if (!stream_blob_to_pack(state, &ctx, &already_hashed_to,
> -					 fd, size, path))
> -			break;
> -		/*
> -		 * Writing this object to the current pack will make
> -		 * it too big; we need to truncate it, start a new
> -		 * pack, and write into it.
> -		 */
> -		hashfile_truncate(state->f, &checkpoint);
> -		state->offset = checkpoint.offset;
> -		flush_packfile_transaction(transaction);
> -		if (lseek(fd, seekback, SEEK_SET) == (off_t)-1)
> -			return error("cannot seek back");
> -	}

Hm. I was briefly wondering whether we'd loop indefinitely in case the object alone is bigger than the packsize. But we have an escape hatch in `stream_blob_to_pack()` that special-cases when we haven't written any data yet, so the answer is "no".

Show 5 quoted lines
> +	hashfile_checkpoint(state->f, &checkpoint);
> +	idx->offset = state->offset;
> +	crc32_begin(state->f);
> +	stream_blob_to_pack(state, &ctx, fd, size, path);
>  	git_hash_final_oid(result_oid, &ctx);
Thanks!
Patrick
Previous: Justin ToblerNext: Justin Tobler
Message 13 of 57 in “odb: add write operation to ODB transaction interface”
  1. 0/6 odb: add write operation to ODB transaction interfaceJustin Tobler, Mar 31, 2026
  2. 1/6 odb: split `struct odb_transaction` into separate headerJustin Tobler, Mar 31, 2026
  3. Patrick SteinhardtMar 31, 2026
  4. Justin ToblerMar 31, 2026
  5. Junio C HamanoMar 31, 2026
  6. Justin ToblerMar 31, 2026
  7. 2/6 odb/transaction: use pluggable `begin_transaction()`Justin Tobler, Mar 31, 2026
  8. Patrick SteinhardtMar 31, 2026
  9. 3/6 object-file: remove flags from transaction packfile writesJustin Tobler, Mar 31, 2026
  10. Patrick SteinhardtMar 31, 2026
  11. Justin ToblerMar 31, 2026
  12. 4/6 object-file: avoid fd seekback by checking object size upfrontJustin Tobler, Mar 31, 2026
  13. Patrick SteinhardtMar 31, 2026
  14. Justin ToblerMar 31, 2026
  15. 5/6 object-file: generalize packfile writes to use odb_write_streamJustin Tobler, Mar 31, 2026
  16. Patrick SteinhardtMar 31, 2026
  17. Justin ToblerMar 31, 2026
  18. Patrick SteinhardtMar 31, 2026
  19. Justin ToblerMar 31, 2026
  20. Patrick SteinhardtMar 31, 2026
  21. 6/6 odb/transaction: make `write_object_stream()` pluggableJustin Tobler, Mar 31, 2026
  22. Patrick SteinhardtMar 31, 2026
  23. Justin ToblerMar 31, 2026
  24. 0/7 odb: add write operation to ODB transaction interfaceJustin Tobler, Apr 1, 2026
  25. 1/7 odb: split `struct odb_transaction` into separate headerJustin Tobler, Apr 1, 2026
  26. 3/7 odb: update `struct odb_write_stream` read() callbackJustin Tobler, Apr 1, 2026
  27. Patrick SteinhardtApr 1, 2026
  28. 2/7 odb/transaction: use pluggable `begin_transaction()`Justin Tobler, Apr 1, 2026
  29. 4/7 object-file: remove flags from transaction packfile writesJustin Tobler, Apr 1, 2026
  30. Patrick SteinhardtApr 1, 2026
  31. Justin ToblerApr 1, 2026
  32. 5/7 object-file: avoid fd seekback by checking object size upfrontJustin Tobler, Apr 1, 2026
  33. 6/7 object-file: generalize packfile writes to use odb_write_streamJustin Tobler, Apr 1, 2026
  34. 7/7 odb/transaction: make `write_object_stream()` pluggableJustin Tobler, Apr 1, 2026
  35. Patrick SteinhardtApr 1, 2026
  36. 0/7 odb: add write operation to ODB transaction interfaceJustin Tobler, Apr 2, 2026
  37. 1/7 odb: split `struct odb_transaction` into separate headerJustin Tobler, Apr 2, 2026
  38. 2/7 odb/transaction: use pluggable `begin_transaction()`Justin Tobler, Apr 2, 2026
  39. 3/7 odb: update `struct odb_write_stream` read() callbackJustin Tobler, Apr 2, 2026
  40. Jeff KingMay 11, 2026
  41. Justin ToblerMay 12, 2026
  42. 4/7 object-file: remove flags from transaction packfile writesJustin Tobler, Apr 2, 2026
  43. Jeff KingApr 6, 2026
  44. Jeff KingApr 6, 2026
  45. 5/7 object-file: avoid fd seekback by checking object size upfrontJustin Tobler, Apr 2, 2026
  46. 6/7 object-file: generalize packfile writes to use odb_write_streamJustin Tobler, Apr 2, 2026
  47. 7/7 odb/transaction: make `write_object_stream()` pluggableJustin Tobler, Apr 2, 2026
  48. Patrick SteinhardtApr 8, 2026
  49. 0/7 odb: add write operation to ODB transaction interfaceJustin Tobler, May 14, 2026
  50. 1/7 odb: split `struct odb_transaction` into separate headerJustin Tobler, May 14, 2026
  51. 2/7 odb/transaction: use pluggable `begin_transaction()`Justin Tobler, May 14, 2026
  52. 3/7 odb: update `struct odb_write_stream` read() callbackJustin Tobler, May 14, 2026
  53. 4/7 object-file: remove flags from transaction packfile writesJustin Tobler, May 14, 2026
  54. 5/7 object-file: avoid fd seekback by checking object size upfrontJustin Tobler, May 14, 2026
  55. 6/7 object-file: generalize packfile writes to use odb_write_streamJustin Tobler, May 14, 2026
  56. 7/7 odb/transaction: make `write_object_stream()` pluggableJustin Tobler, May 14, 2026
  57. Jeff KingMay 15, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.