git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 7/7] odb/transaction: add transaction interface to write packfiles

From
Justin Tobler <jltobler@gmail.com>
Date
Aug 10, 2026, 19:54 UTC
Message-ID
<anomr5jpSGlrTX2m@denethor>
In-Reply-To
<xmqqwlty7hdz.fsf@gitster.g>
On 26/08/09 09:02PM, Junio C Hamano wrote:
Show 10 quoted lines
> Justin Tobler <jltobler@gmail.com> writes:
> 
> > .... Note that a packfile
> > written via git-index-pack(1) is kept in place by a ".keep" lockfile
> > that must be retained until references are updated. To faciliate this in
> > an ODB backend agnostic manner, the "files" transaction backend takes
> > ownership of these lockfiles and removes them post-commit through its
> > release callback.
> 
> The above is confusing and I am lost.  Care to explain a bit more?

It should say "finalize callback" instead of "release callback". The idea here though is that ".keep" files are likely an implementation detail of the "files" backend, but we still need a way to clean them up after a transaction is committed and reference updates have been performed via the generic ODB transaction interface.

So after `odb_transaction_commit()`, the ".keep" files are tracked by the "files" transaction and only removed once `odb_transaction_finalize()` is invoked. It would be up to callers to ensure that reference updates are performed as required prior to finalize being invoked.

I'll try to explain this better in the next version.
Show 46 quoted lines
> > +		status = start_command(&child);
> > +		if (status) {
> > +			strbuf_addstr(err_msg, "index-pack fork failed");
> > +			return -1;
> > +		}
> > +
> > +		lockfile = index_pack_lockfile(repo, child.out, NULL);
> > +		if (lockfile) {
> > +			ALLOC_GROW(transaction->pack_lockfiles,
> > +				   transaction->pack_lockfiles_nr + 1,
> > +				   transaction->pack_lockfiles_alloc);
> > +			transaction->pack_lockfiles[transaction->pack_lockfiles_nr++] =
> > +				register_tempfile(lockfile);
> > +			free(lockfile);
> > +		}
> 
> Here we add the .keep file to the list of lockfiles.  We have
> finalization step laer in odb_transaction_files_finalize() that
> deletes the tempfiles when we are done, which comes after
> the transaction is committed.
> 
> But isn't the odb_transaction_files_commit() where the migration of
> tmp_objdir_migrate() happens?  Everything in the quarantine directory
> including these .keep files are "migrated" (either link-to-the-new
> followed by unlink-of-the-old, or rename-old-to-new) there.
> 
> And then ...
> 
> > +static int odb_transaction_files_finalize(struct odb_transaction *base)
> > +{
> > +	struct odb_transaction_files *transaction =
> > +		container_of(base, struct odb_transaction_files, base);
> > +	int ret = 0;
> > +
> > +	for (size_t i = 0; i < transaction->pack_lockfiles_nr; i++)
> > +		ret |= delete_tempfile(&transaction->pack_lockfiles[i]);
> > +
> > +	free(transaction->pack_lockfiles);
> > +
> > +	return ret;
> > +}
> 
> ... we do the deletion of tempfile but has anybody migrated the path
> to these files recorded in the lockfile structure?  How are we
> removing the .keep files that were "migrated" when the transaction
> was committed?

The filepath recorded by the ".keep" tempfile structure is _supposed_ to be the final path of the ".keep" file post-commit that way it knows its location after its been migrated and can remove it. The path is generated by `index_pack_lockfile()` and is supposed to use the real ODB source path and not the transaction ODB source path... but this is not happening anymore now that we are using ODB transactions in git-receive-pack(1) which reorders the ODB source list order when the transaction is applied...

This is a bug and needs to be fixed. The fix itself should be fairly straightforward, we just need to tell `index_pack_lockfile()` the correct ODB source it should be using. I'll send a patch later today correct this.

Thanks, -Justin

Previous: Junio C HamanoNext: Justin Tobler
Message 36 of 80 in “builtin/receive-pack: support pluggable packfile writes”
  1. 0/6 builtin/receive-pack: support pluggable packfile writesJustin Tobler, Aug 6, 2026
  2. 1/6 odb/transaction: add transaction release interfaceJustin Tobler, Aug 6, 2026
  3. Patrick SteinhardtAug 7, 2026
  4. Justin ToblerAug 7, 2026
  5. 2/6 builtin/receive-pack: pass shallow file explicitlyJustin Tobler, Aug 6, 2026
  6. Patrick SteinhardtAug 7, 2026
  7. 4/6 builtin/receive-pack: report unpack errors via strbufJustin Tobler, Aug 6, 2026
  8. Patrick SteinhardtAug 7, 2026
  9. Justin ToblerAug 7, 2026
  10. Justin ToblerAug 9, 2026
  11. Patrick SteinhardtAug 10, 2026
  12. 3/6 builtin/receive-pack: lift global state out of unpack()Justin Tobler, Aug 6, 2026
  13. Patrick SteinhardtAug 7, 2026
  14. Justin ToblerAug 7, 2026
  15. 5/6 builtin/receive-pack: explicitly pass packfile fdJustin Tobler, Aug 6, 2026
  16. 6/6 odb/transaction: add transaction interface to write packfilesJustin Tobler, Aug 6, 2026
  17. Patrick SteinhardtAug 7, 2026
  18. Justin ToblerAug 7, 2026
  19. 0/7 builtin/receive-pack: support pluggable packfile writesJustin Tobler, Aug 9, 2026
  20. 1/7 odb/transaction: add transaction finalize interfaceJustin Tobler, Aug 9, 2026
  21. Junio C HamanoAug 10, 2026
  22. Justin ToblerAug 10, 2026
  23. 2/7 builtin/receive-pack: pass shallow file explicitlyJustin Tobler, Aug 9, 2026
  24. 3/7 builtin/receive-pack: read unpack limit config lazilyJustin Tobler, Aug 9, 2026
  25. Patrick SteinhardtAug 10, 2026
  26. Justin ToblerAug 10, 2026
  27. Junio C HamanoAug 10, 2026
  28. Justin ToblerAug 10, 2026
  29. 4/7 builtin/receive-pack: lift global state out of unpack()Justin Tobler, Aug 9, 2026
  30. 5/7 builtin/receive-pack: report unpack errors via strbufJustin Tobler, Aug 9, 2026
  31. 6/7 builtin/receive-pack: explicitly pass packfile fdJustin Tobler, Aug 9, 2026
  32. 7/7 odb/transaction: add transaction interface to write packfilesJustin Tobler, Aug 9, 2026
  33. Junio C HamanoAug 10, 2026
  34. Justin ToblerAug 10, 2026
  35. Junio C HamanoAug 10, 2026
  36. Justin ToblerAug 10, 2026
  37. 0/9 builtin/receive-pack: support pluggable packfile writesJustin Tobler, Aug 11, 2026
  38. 1/9 builtin/receive-pack: properly clean up keep filesJustin Tobler, Aug 11, 2026
  39. Patrick SteinhardtAug 12, 2026
  40. Justin ToblerAug 13, 2026
  41. Patrick SteinhardtAug 14, 2026
  42. 2/9 odb/transaction: add transaction finalize interfaceJustin Tobler, Aug 11, 2026
  43. Patrick SteinhardtAug 12, 2026
  44. 3/9 builtin/receive-pack: pass shallow file explicitlyJustin Tobler, Aug 11, 2026
  45. 4/9 builtin/receive-pack: read unpack limit config lazilyJustin Tobler, Aug 11, 2026
  46. 5/9 builtin/receive-pack: lift global state out of unpack()Justin Tobler, Aug 11, 2026
  47. 6/9 builtin/receive-pack: report unpack errors via strbufJustin Tobler, Aug 11, 2026
  48. 8/9 odb: return temporary ODB source when setJustin Tobler, Aug 11, 2026
  49. Patrick SteinhardtAug 12, 2026
  50. 7/9 builtin/receive-pack: explicitly pass packfile fdJustin Tobler, Aug 11, 2026
  51. 9/9 odb/transaction: add transaction interface to write packfilesJustin Tobler, Aug 11, 2026
  52. Patrick SteinhardtAug 14, 2026
  53. Justin ToblerAug 14, 2026
  54. Patrick SteinhardtAug 17, 2026
  55. 0/9 builtin/receive-pack: support pluggable packfile writesJustin Tobler, Aug 19, 2026
  56. 1/9 builtin/receive-pack: properly clean up keep filesJustin Tobler, Aug 19, 2026
  57. Patrick SteinhardtAug 20, 2026
  58. Justin ToblerAug 20, 2026
  59. 2/9 odb/transaction: add transaction finalize interfaceJustin Tobler, Aug 19, 2026
  60. Patrick SteinhardtAug 20, 2026
  61. 3/9 builtin/receive-pack: pass shallow file explicitlyJustin Tobler, Aug 19, 2026
  62. 4/9 builtin/receive-pack: read unpack limit config lazilyJustin Tobler, Aug 19, 2026
  63. 5/9 builtin/receive-pack: lift global state out of unpack()Justin Tobler, Aug 19, 2026
  64. 6/9 builtin/receive-pack: report unpack errors via strbufJustin Tobler, Aug 19, 2026
  65. 7/9 builtin/receive-pack: explicitly pass packfile fdJustin Tobler, Aug 19, 2026
  66. 8/9 odb: return temporary ODB source when setJustin Tobler, Aug 19, 2026
  67. 9/9 odb/transaction: add transaction interface to write packfilesJustin Tobler, Aug 19, 2026
  68. Patrick SteinhardtAug 20, 2026
  69. 0/9 builtin/receive-pack: support pluggable packfile writesJustin Tobler, Aug 20, 2026
  70. 1/9 builtin/receive-pack: properly clean up keep filesJustin Tobler, Aug 20, 2026
  71. 2/9 odb/transaction: add transaction finalize interfaceJustin Tobler, Aug 20, 2026
  72. 3/9 builtin/receive-pack: pass shallow file explicitlyJustin Tobler, Aug 20, 2026
  73. 4/9 builtin/receive-pack: read unpack limit config lazilyJustin Tobler, Aug 20, 2026
  74. 5/9 builtin/receive-pack: lift global state out of unpack()Justin Tobler, Aug 20, 2026
  75. 6/9 builtin/receive-pack: report unpack errors via strbufJustin Tobler, Aug 20, 2026
  76. 8/9 odb: return temporary ODB source when setJustin Tobler, Aug 20, 2026
  77. 9/9 odb/transaction: add transaction interface to write packfilesJustin Tobler, Aug 20, 2026
  78. Junio C HamanoAug 21, 2026
  79. 7/9 builtin/receive-pack: explicitly pass packfile fdJustin Tobler, Aug 20, 2026
  80. Patrick SteinhardtAug 21, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.