git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 3/3] object-file: write ".keep" lockfiles during transaction commit

From
Justin Tobler <jltobler@gmail.com>
Date
Oct 16, 2026, 19:22 UTC
Message-ID
<atJxHsn_OxJ_HtWW@denethor>
In-Reply-To
<atHLFIp8MI729ZkP@pks.im>
On 26/10/16 08:58AM, Patrick Steinhardt wrote:
Show 27 quoted lines
> On Thu, Oct 15, 2026 at 09:29:01PM -0500, Justin Tobler wrote:
> > The "files" ODB backend writes a ".keep" lockfile into the transaction
> > quarantine directory via `odb_transaction_files_write_pack()`. When the
> > ODB transaction is committed, the ".keep" lockfile is migrated to the
> > primary ODB alongside its packfile. Writing the temporary lockfile in
> > this manner has a couple of downsides:
> > 
> >   - Failing to migrate the lockfile during transaction commit due to a
> >     concurrent process already acquiring the same lock still allows the
> >     rest of the transaction to be migrated into the primary ODB. This
> >     dirties the primary ODB with new objects even though no references
> >     were updated and the operation failed.
> 
> I'm not fully sure I understand this downside. Are you saying that we
> fail to migrate the lockfile, then continue migrating everything else
> only to then fail the whole transaction?
> 
> ... skipping ahead a bit...
> 
> Oh, I get it now. You propose to first acquire all lockfiles before
> migrating anything so that we cannot conflict with a concurrent process
> that aims to claim the same locks.
> 
> That will of course mean that we will fail the transaction for any
> concurrent process that happens to write the same packfiles. But based
> on what you claim that was already the case anyway, but now we notice
> early and thus won't commit parts of the transaction. Makes sense.

That is correct. Previously when migrating the quarantined objects, if a matching ".keep" file was already present in the primary ODB, `check_collision()` would detect the preexisting ".keep" lockfile and bubble up the error to `tmp_objdir_migrate()`, but still let the migration continue. This ultimately causes `odb_transaction_commit()` to also return an error.

For git-receive-pack(1), an error during ODB transaction commit already aborts the operation. The main change here is that by manually creating the lockfiles during transaction commit, we can avoid the migration step entirely if the lockfiles are not able to be created first.

I'll try rewording the commit message here to be a bit more clear about this.

[snip]
Show 15 quoted lines
> > diff --git a/object-file.c b/object-file.c
> > index ebe46d35de..39e6299be1 100644
> > --- a/object-file.c
> > +++ b/object-file.c
> > @@ -1321,6 +1357,9 @@ static int odb_transaction_files_commit(struct odb_transaction *base)
> >  		delete_tempfile(&temp);
> >  		strbuf_release(&temp_path);
> >  
> > +		if (write_pack_locks(transaction))
> > +			return error(_("unable to lock packfiles"));
> 
> Okay, we'll now write lockfiles before migrating any of the data. So if
> any of the files already happens to exist, we'll bail out here. It would
> have been great to maybe also provide a proper failure message via errno
> for example, but okay.

Ya, good suggestion. I think I'll update the `write_pack_lockfile()` helper in patch 2 to provide this additional context. There I can also print the specific lockfile causing the issue.

[snip]
Show 29 quoted lines
> > @@ -101,4 +100,28 @@ test_expect_success '.keep file is removed after push' '
> >  	test_path_is_missing "$keep"
> >  '
> >  
> > +test_expect_success '.keep file created outside of process is not removed' '
> > +	test_when_finished rm -rf keep.git &&
> > +	git init --bare keep.git &&
> > +
> > +	git -C keep.git config set receive.unpackLimit 0 &&
> > +
> > +	# Create a ".keep" lockfile matching the packfile to simulate a
> > +	# concurrent git-receive-pack(1) that was already written its lockfile.
> > +	test_hook -C keep.git pre-receive <<-\EOF &&
> > +	pack="$(ls "$GIT_QUARANTINE_PATH"/pack/pack-*.pack)" &&
> > +	pack="${pack##*/}" &&
> > +	echo "baz" >"objects/pack/${pack%.pack}.keep"
> > +	EOF
> > +
> > +	test_commit bar &&
> > +	test_must_fail git push keep.git HEAD &&
> > +
> > +	# The ".keep" lockfile of the concurrent git-receive-pack(1) should be
> > +	# left in its place.
> > +	keep="$(ls keep.git/objects/pack/pack-*.keep)" &&
> > +	test_path_is_file "$keep" &&
> > +	test_grep "baz" "$keep"
> 
> Also, there should be no corresponding ".pack" file, right? Do we want
> to assert that, too?

That is correct. I'll add an assertion in the next version to make sure the packfile is not migrated too.

Thanks for the review, -Justin

Previous: Patrick SteinhardtNext: Justin Tobler
Message 10 of 40 in “object-file: fix ".keep" lockfile ownership in ODB transactions”
  1. 0/3 object-file: fix ".keep" lockfile ownership in ODB transactionsJustin Tobler, Oct 16, 2026
  2. 1/3 pack-write: lift and expose index-pack report parsingJustin Tobler, Oct 16, 2026
  3. Patrick SteinhardtOct 16, 2026
  4. Justin ToblerOct 16, 2026
  5. 2/3 object-file: explicitly write ".keep" lockfileJustin Tobler, Oct 16, 2026
  6. Patrick SteinhardtOct 16, 2026
  7. Justin ToblerOct 16, 2026
  8. 3/3 object-file: write ".keep" lockfiles during transaction commitJustin Tobler, Oct 16, 2026
  9. Patrick SteinhardtOct 16, 2026
  10. Justin ToblerOct 16, 2026
  11. 0/4 object-file: fix ".keep" lockfile ownership in ODB transactionsJustin Tobler, Oct 16, 2026
  12. 1/4 pack-write: make index-pack output comment hash agnosticJustin Tobler, Oct 16, 2026
  13. 2/4 pack-write: lift and expose index-pack report parsingJustin Tobler, Oct 16, 2026
  14. 3/4 object-file: explicitly write ".keep" lockfileJustin Tobler, Oct 16, 2026
  15. Patrick SteinhardtOct 19, 2026
  16. Justin ToblerOct 19, 2026
  17. 4/4 object-file: write ".keep" lockfiles during transaction commitJustin Tobler, Oct 16, 2026
  18. Patrick SteinhardtOct 19, 2026
  19. 0/4 object-file: fix ".keep" lockfile ownership in ODB transactionsJustin Tobler, Oct 20, 2026
  20. 1/4 pack-write: make index-pack output comment hash agnosticJustin Tobler, Oct 20, 2026
  21. 2/4 pack-write: lift and expose index-pack report parsingJustin Tobler, Oct 20, 2026
  22. 3/4 object-file: explicitly write ".keep" lockfileJustin Tobler, Oct 20, 2026
  23. Patrick SteinhardtOct 20, 2026
  24. Justin ToblerOct 20, 2026
  25. Junio C HamanoOct 21, 2026
  26. 4/4 object-file: write ".keep" lockfiles during transaction commitJustin Tobler, Oct 20, 2026
  27. Patrick SteinhardtOct 20, 2026
  28. 0/4 object-file: fix ".keep" lockfile ownership in ODB transactionsJustin Tobler, Oct 21, 2026
  29. 1/4 pack-write: make index-pack output comment hash agnosticJustin Tobler, Oct 21, 2026
  30. Karthik NayakOct 26, 2026
  31. Justin ToblerOct 26, 2026
  32. 2/4 pack-write: lift and expose index-pack report parsingJustin Tobler, Oct 21, 2026
  33. 3/4 object-file: explicitly write ".keep" lockfileJustin Tobler, Oct 21, 2026
  34. Karthik NayakOct 26, 2026
  35. Justin ToblerOct 27, 2026
  36. Justin ToblerOct 27, 2026
  37. Karthik NayakOct 26, 2026
  38. 4/4 object-file: write ".keep" lockfiles during transaction commitJustin Tobler, Oct 21, 2026
  39. Karthik NayakOct 26, 2026
  40. Patrick SteinhardtOct 22, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.