git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v3 1/9] builtin/receive-pack: properly clean up keep files

From
Justin Tobler <jltobler@gmail.com>
Date
Aug 13, 2026, 21:45 UTC
Message-ID
<an41gSCa7EFGkB1r@denethor>
In-Reply-To
<anwNonpw5SZuHADv@pks.im>
On 26/08/12 08:07AM, Patrick Steinhardt wrote:
Show 23 quoted lines
> On Tue, Aug 11, 2026 at 12:54:07PM -0500, Justin Tobler wrote:
> > When git-receive-pack(1) stores an incoming packfile with
> > git-index-pack(1), a ".keep" file is written alongside it to hold the
> > pack in place until the references have been updated, and is removed
> > afterwards. The path used to remove it is derived via
> > `index_pack_lockfile()` from the repository's primary object directory.
> > 
> > In bdee7b3013 (builtin/receive-pack: stage incoming objects via ODB
> > transactions, 2026-07-10), git-receive-pack(1) started using the ODB
> > transaction interfaces instead of managing a temporary directory
> > directly. When starting an ODB transaction, the sources list is
> > reordered to insert the newly created transaction source first as the
> > primary to ensure writes are routed to it accordingly.
> > 
> > Prior to using ODB transactions, git-receive-pack(1) would only set the
> > temporary directory as the primary source for the child
> > git-index-pack(1) and git-unpack-objects(1) processes it spawned and the
> > parent process would set the temporary directory set as an alternate
> > only. By using ODB transactions, the ODB source list is also reordered
> > for the parent process which results in `index_pack_lockfile()` deriving
> > the ".keep" path relative to the temporary directory instead the actual
> 
> Nit: s/instead/& of/
Will fix.
Show 6 quoted lines
> > main ODB source path. Consequently, this prevents the ".keep" file from
> > being properly removed after being migrated into the main ODB source
> > post-commit.
> 
> Hm. Are the temporary packs written into the transaction-managed tempdir
> now, or do they still end up in the main object directory?

The packfile and associated ".keep" lockfiles are both initially written into the temporary directory managed by the ODB transaction. On transaction commit, they are then both migrated to the main ODB.

When registering the keep tempfile, we need to record the future post-commit location of the keep file that way it can be removed when `odb_transaction_finalize()` is invoked. This matches the original behavior prior to ODB transaction being introduced in git-receive-pack(1).

Show 23 quoted lines
> > diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
> > index 86933d8d7e..d74b787148 100644
> > --- a/builtin/receive-pack.c
> > +++ b/builtin/receive-pack.c
> > @@ -2412,7 +2412,13 @@ static const char *unpack(int err_fd, struct shallow_info *si,
> >  		if (status)
> >  			return "index-pack fork failed";
> >  
> > -		lockfile = index_pack_lockfile(the_repository, child.out, NULL);
> > +		/*
> > +		 * The lockfile filepath is expected to be the final location of
> > +		 * the ".keep" file after being migrated to the main ODB source.
> > +		 * This ensures the lockfile can be found and removed later
> > +		 * after the ODB transaction has been committed.
> > +		 */
> > +		lockfile = index_pack_lockfile(transaction->source, child.out, NULL);
> >  		if (lockfile) {
> >  			pack_lockfile = register_tempfile(lockfile);
> >  			free(lockfile);
> 
> Okay. So previously, we wrote the ".keep" file into the main repository,
> whereas now we write it into the temporary object directory? Is the
> packfile itself also written in there?

Not quite, both the packfile and keep file were written to the temporary directory and continue to do so.

Prior to bdee7b3013 (builtin/receive-pack: stage incoming objects via ODB transactions, 2026-07-10), the ".keep" files were also being written to the quarantine directory and migrated alongside the packfiles. The main git-receive-pack(1) process always kept the primary ODB as the first entry in the source list though ensuring that the "filename" registered for keep tempfile was the final location. With ODB transactions though, the source list order _does_ get changed and resulted in the keep tempfile not knowing about its final location. Consequently, it is no longer cleaned up.

Show 7 quoted lines
> What I'm wondering is why we even need a ".keep" file at all anymore if
> we're not storing it in the main object directory. It wouldn't help us
> to avoid the race, because after committing the transaction the ".keep"
> file would remain in the temporary directory, whereas the packfile would
> have been migrated to the main object directory. So it doesn't have a
> ".keep" file at that point, and neither have references been updated to
> point to the new objects yet.

The ".keep" file does end up in the main ODB alongside the packfile when the transaction is committed. The main problem here is that it is not being cleaned up because the post-migration path does not match what the registered tempfile tracks.

Show 15 quoted lines
> So I wonder whether instead, we'd have to:
> 
>   1. Start the transaction, creating the temporary object directory.
>   
>   2. Write the packfile into the temporary object directory, but don't
>      create a ".keep" file.
> 
>   3. At commit time, first write a ".keep" file in the main object
>      directory and then migrate the packfile over.
> 
>   4. At finalization time, prune the ".keep" file from the main object
>      directory.
> 
> That would retain the current properties of the system, but as far as I
> can see this is not what we're doing here.

With this patch, this is effectly what we are doing already. The main difference is that we are creating the ".keep" file alongside the packfile via git-index-pack(1) and migrating both when `odb_transaction_commit()` is invoked.

We could stop relying on git-index-pack(1) to generate the ".keep" file and instead generate it ourselves during the commit phase as you suggested, but I'm not sure that would really buy us anything right now. For now, I think it would be fine to keep the changes more minimal.

I'll try to clarify the commit message a bit in the next version to better explain what is happening.

Show 14 quoted lines
> > diff --git a/pack.h b/pack.h
> > index 1cde92082b..68dcf08cf3 100644
> > --- a/pack.h
> > +++ b/pack.h
> > @@ -3,6 +3,7 @@
> >  
> >  #include "object.h"
> >  #include "csum-file.h"
> > +#include "odb/source.h"
> >  
> >  struct packed_git;
> >  struct pack_window;
> 
> Let's add a forward declaration instead of including this header.
Will do.
Show 25 quoted lines
> > diff --git a/t/t5547-push-quarantine.sh b/t/t5547-push-quarantine.sh
> > index 0798ddab02..400a597606 100755
> > --- a/t/t5547-push-quarantine.sh
> > +++ b/t/t5547-push-quarantine.sh
> > @@ -70,4 +70,18 @@ test_expect_success 'updating a ref from quarantine is forbidden' '
> >  	git -C update.git fsck
> >  '
> >  
> > +test_expect_success '.keep file is removed after push' '
> > +	test_when_finished rm -rf keep.git &&
> > +	git init --bare keep.git &&
> > +
> > +	git -C keep.git config set receive.unpackLimit 0 &&
> > +	test_commit foo &&
> > +	git push keep.git HEAD &&
> > +	pack="$(ls keep.git/objects/pack/pack-*.pack)" &&
> > +	keep="${pack%.pack}.keep" &&
> > +
> > +	test_path_is_file "$pack" &&
> > +	test_path_is_missing "$keep"
> > +'
> 
> This would feel a bit safer if we had a hook that verifies that we
> indeed have the ".keep" file in the right spot before committing
> everything.
I'll try to set something up in the next version. Thanks.
-Justin
Previous: Patrick SteinhardtNext: Patrick Steinhardt
Message 40 of 80 in “builtin/receive-pack: support pluggable packfile writes”
  1. 0/6 builtin/receive-pack: support pluggable packfile writesJustin Tobler, Aug 6, 2026
  2. 1/6 odb/transaction: add transaction release interfaceJustin Tobler, Aug 6, 2026
  3. Patrick SteinhardtAug 7, 2026
  4. Justin ToblerAug 7, 2026
  5. 2/6 builtin/receive-pack: pass shallow file explicitlyJustin Tobler, Aug 6, 2026
  6. Patrick SteinhardtAug 7, 2026
  7. 4/6 builtin/receive-pack: report unpack errors via strbufJustin Tobler, Aug 6, 2026
  8. Patrick SteinhardtAug 7, 2026
  9. Justin ToblerAug 7, 2026
  10. Justin ToblerAug 9, 2026
  11. Patrick SteinhardtAug 10, 2026
  12. 3/6 builtin/receive-pack: lift global state out of unpack()Justin Tobler, Aug 6, 2026
  13. Patrick SteinhardtAug 7, 2026
  14. Justin ToblerAug 7, 2026
  15. 5/6 builtin/receive-pack: explicitly pass packfile fdJustin Tobler, Aug 6, 2026
  16. 6/6 odb/transaction: add transaction interface to write packfilesJustin Tobler, Aug 6, 2026
  17. Patrick SteinhardtAug 7, 2026
  18. Justin ToblerAug 7, 2026
  19. 0/7 builtin/receive-pack: support pluggable packfile writesJustin Tobler, Aug 9, 2026
  20. 1/7 odb/transaction: add transaction finalize interfaceJustin Tobler, Aug 9, 2026
  21. Junio C HamanoAug 10, 2026
  22. Justin ToblerAug 10, 2026
  23. 2/7 builtin/receive-pack: pass shallow file explicitlyJustin Tobler, Aug 9, 2026
  24. 3/7 builtin/receive-pack: read unpack limit config lazilyJustin Tobler, Aug 9, 2026
  25. Patrick SteinhardtAug 10, 2026
  26. Justin ToblerAug 10, 2026
  27. Junio C HamanoAug 10, 2026
  28. Justin ToblerAug 10, 2026
  29. 4/7 builtin/receive-pack: lift global state out of unpack()Justin Tobler, Aug 9, 2026
  30. 5/7 builtin/receive-pack: report unpack errors via strbufJustin Tobler, Aug 9, 2026
  31. 6/7 builtin/receive-pack: explicitly pass packfile fdJustin Tobler, Aug 9, 2026
  32. 7/7 odb/transaction: add transaction interface to write packfilesJustin Tobler, Aug 9, 2026
  33. Junio C HamanoAug 10, 2026
  34. Justin ToblerAug 10, 2026
  35. Junio C HamanoAug 10, 2026
  36. Justin ToblerAug 10, 2026
  37. 0/9 builtin/receive-pack: support pluggable packfile writesJustin Tobler, Aug 11, 2026
  38. 1/9 builtin/receive-pack: properly clean up keep filesJustin Tobler, Aug 11, 2026
  39. Patrick SteinhardtAug 12, 2026
  40. Justin ToblerAug 13, 2026
  41. Patrick SteinhardtAug 14, 2026
  42. 2/9 odb/transaction: add transaction finalize interfaceJustin Tobler, Aug 11, 2026
  43. Patrick SteinhardtAug 12, 2026
  44. 3/9 builtin/receive-pack: pass shallow file explicitlyJustin Tobler, Aug 11, 2026
  45. 4/9 builtin/receive-pack: read unpack limit config lazilyJustin Tobler, Aug 11, 2026
  46. 5/9 builtin/receive-pack: lift global state out of unpack()Justin Tobler, Aug 11, 2026
  47. 6/9 builtin/receive-pack: report unpack errors via strbufJustin Tobler, Aug 11, 2026
  48. 8/9 odb: return temporary ODB source when setJustin Tobler, Aug 11, 2026
  49. Patrick SteinhardtAug 12, 2026
  50. 7/9 builtin/receive-pack: explicitly pass packfile fdJustin Tobler, Aug 11, 2026
  51. 9/9 odb/transaction: add transaction interface to write packfilesJustin Tobler, Aug 11, 2026
  52. Patrick SteinhardtAug 14, 2026
  53. Justin ToblerAug 14, 2026
  54. Patrick SteinhardtAug 17, 2026
  55. 0/9 builtin/receive-pack: support pluggable packfile writesJustin Tobler, Aug 19, 2026
  56. 1/9 builtin/receive-pack: properly clean up keep filesJustin Tobler, Aug 19, 2026
  57. Patrick SteinhardtAug 20, 2026
  58. Justin ToblerAug 20, 2026
  59. 2/9 odb/transaction: add transaction finalize interfaceJustin Tobler, Aug 19, 2026
  60. Patrick SteinhardtAug 20, 2026
  61. 3/9 builtin/receive-pack: pass shallow file explicitlyJustin Tobler, Aug 19, 2026
  62. 4/9 builtin/receive-pack: read unpack limit config lazilyJustin Tobler, Aug 19, 2026
  63. 5/9 builtin/receive-pack: lift global state out of unpack()Justin Tobler, Aug 19, 2026
  64. 6/9 builtin/receive-pack: report unpack errors via strbufJustin Tobler, Aug 19, 2026
  65. 7/9 builtin/receive-pack: explicitly pass packfile fdJustin Tobler, Aug 19, 2026
  66. 8/9 odb: return temporary ODB source when setJustin Tobler, Aug 19, 2026
  67. 9/9 odb/transaction: add transaction interface to write packfilesJustin Tobler, Aug 19, 2026
  68. Patrick SteinhardtAug 20, 2026
  69. 0/9 builtin/receive-pack: support pluggable packfile writesJustin Tobler, Aug 20, 2026
  70. 1/9 builtin/receive-pack: properly clean up keep filesJustin Tobler, Aug 20, 2026
  71. 2/9 odb/transaction: add transaction finalize interfaceJustin Tobler, Aug 20, 2026
  72. 3/9 builtin/receive-pack: pass shallow file explicitlyJustin Tobler, Aug 20, 2026
  73. 4/9 builtin/receive-pack: read unpack limit config lazilyJustin Tobler, Aug 20, 2026
  74. 5/9 builtin/receive-pack: lift global state out of unpack()Justin Tobler, Aug 20, 2026
  75. 6/9 builtin/receive-pack: report unpack errors via strbufJustin Tobler, Aug 20, 2026
  76. 8/9 odb: return temporary ODB source when setJustin Tobler, Aug 20, 2026
  77. 9/9 odb/transaction: add transaction interface to write packfilesJustin Tobler, Aug 20, 2026
  78. Junio C HamanoAug 21, 2026
  79. 7/9 builtin/receive-pack: explicitly pass packfile fdJustin Tobler, Aug 20, 2026
  80. Patrick SteinhardtAug 21, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.