git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH] index-pack: teach --promisor to require --stdin

From
JTJonathan Tan <jonathantanmy@google.com>
Date
Nov 18, 2024, 19:02 UTC
Message-ID
<20241118190210.772105-1-jonathantanmy@google.com>
In-Reply-To
<20241116032352.GA1782794@coredump.intra.peff.net>
Currently,
 - Running "index-pack --promisor" outside a repo segfaults.
 - It may be confusing to a user that running "index-pack --promisor"
   within a repo may make changes to the repo's object DB, especially
   since the packs indexed by the index-pack invocation may not even be
   related to the repo.

As discussed in [1], teaching --promisor to require --stdin and forbid a packfile name solves both these problems. This combination of arguments requires a repo (since we are writing the resulting .pack and .idx to it) and it is clear that the files are related to the repo.

Currently, Git uses "index-pack --promisor" only when fetching into a repo, so it could be argued that we should teach "index-pack" a new argument (say, "--fetching-mode") instead of tying --promisor to a generic argument like "--stdin". However, this --promisor feature could conceivably be used whenever we have a packfile that is known to come from the promisor remote (whether obtained through Git's fetch protocol or through other means) so it seems reasonable to use --stdin here - one could envision a user-made script obtaining a packfile and then running "index-pack --promisor --stdin", for example. In fact, it might be possible to relax the restriction further (say, by also allowing --promisor when indexing a packfile that is in the object DB), but relaxing the restriction is backwards-compatible so we can revisit that later.

One thing to watch out for is the possibility of a future Git feature that indexes a pack in the context of a repo, but does not necessarily write the resulting pack to it (and does not necessarily desire to make any changes to the object DB). One such feature would be fetch quarantine, which might need the repo context in order to detect hash collisions, but would also need to ensure that the object DB is undisturbed in case the fetch fails for whatever reason, even if the reason occurs only after the indexing is complete. It may not be obvious to the implementer of such a feature that "index-pack" could sometimes write packs other than the indexed pack to the object DB, but there are already other ways that "fetch" could write to the object DB (in particular, packfile URIs and bundle URIs), so hopefully the implementation of this future feature would already include a test that the object DB be undisturbed.

This change requires the change to t5300 by 1f52cdfacb (index-pack: document and test the --promisor option, 2022-03-09) to be undone. (--promisor is already tested indirectly, so we don't need the explicit test here any more.)

[1] https://lore.kernel.org/git/20241114005652.GC1140565@coredump.intra.peff.net/
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
---
This is on jt/repack-local-promisor.

Looking into it further, I think that we also need to require no packfile name to be given (so that we are writing the file to the repository). Therefore, I've added that requirement both in the code and in the documentation.

I've tried to summarize our conversation in the commit message - if you
notice anything missing or incorrect, feel free to let me know.
---
 Documentation/git-index-pack.txt | 2 ++
 builtin/index-pack.c             | 4 ++++
 t/t5300-pack-object.sh           | 4 +---
 3 files changed, 7 insertions(+), 3 deletions(-)
diff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt
index 4be09e58e7..ac96935d73 100644
--- a/Documentation/git-index-pack.txt
+++ b/Documentation/git-index-pack.txt
@@ -144,6 +144,8 @@ Also, if there are objects in the given pack that references non-promisor
 objects (in the repo), repacks those non-promisor objects into a promisor
 pack. This avoids a situation in which a repo has non-promisor objects that are
 accessible through promisor objects.
++
+Requires --stdin, and requires <pack-file> to not be specified.
 
 NOTES
 -----
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index 08b340552f..c46b6e4061 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -1970,6 +1970,10 @@ int cmd_index_pack(int argc,
 		usage(index_pack_usage);
 	if (fix_thin_pack && !from_stdin)
 		die(_("the option '%s' requires '%s'"), "--fix-thin", "--stdin");
+	if (promisor_msg && !from_stdin)
+		die(_("the option '%s' requires '%s'"), "--promisor", "--stdin");
+	if (promisor_msg && pack_name)
+		die(_("--promisor cannot be used with a pack name"));
 	if (from_stdin && !startup_info->have_repository)
 		die(_("--stdin requires a git repository"));
 	if (from_stdin && hash_algo)
diff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh
index aff164ddf8..c53f355e48 100755
--- a/t/t5300-pack-object.sh
+++ b/t/t5300-pack-object.sh
@@ -332,10 +332,8 @@ test_expect_success 'build pack index for an existing pack' '
 	git index-pack -o tmp.idx test-3.pack &&
 	cmp tmp.idx test-1-${packname_1}.idx &&
 
-	git index-pack --promisor=message test-3.pack &&
+	git index-pack test-3.pack &&
 	cmp test-3.idx test-1-${packname_1}.idx &&
-	echo message >expect &&
-	test_cmp expect test-3.promisor &&
 
 	cat test-2-${packname_2}.pack >test-3.pack &&
 	git index-pack -o tmp.idx test-2-${packname_2}.pack &&
-- 
2.47.0.338.g60cca15819-goog
Previous: Jeff KingNext: Junio C Hamano
Message 28 of 37 in “When fetching from a promisor remote, repack local objects referenced”
  1. 0/5 When fetching from a promisor remote, repack local objects referencedJonathan Tan, Oct 24, 2024
  2. 1/5 pack-objects: make variable non-staticJonathan Tan, Oct 24, 2024
  3. Taylor BlauOct 28, 2024
  4. Jonathan TanOct 28, 2024
  5. Taylor BlauOct 28, 2024
  6. Jonathan TanOct 28, 2024
  7. 2/5 t0410: make test description clearerJonathan Tan, Oct 24, 2024
  8. 3/5 t0410: use from-scratch serverJonathan Tan, Oct 24, 2024
  9. 4/5 t5300: move --window clamp test next to unclampedJonathan Tan, Oct 24, 2024
  10. 5/5 index-pack: repack local links into promisor packsJonathan Tan, Oct 24, 2024
  11. Josh SteadmonOct 30, 2024
  12. Jonathan TanNov 1, 2024
  13. Han YoungOct 25, 2024
  14. Taylor BlauOct 25, 2024
  15. Junio C HamanoNov 2, 2024
  16. Taylor BlauOct 25, 2024
  17. 0/4 When fetching from a promisor remote, repack local objects referencedJonathan Tan, Nov 1, 2024
  18. 1/4 t0410: make test description clearerJonathan Tan, Nov 1, 2024
  19. 2/4 t0410: use from-scratch serverJonathan Tan, Nov 1, 2024
  20. 3/4 t5300: move --window clamp test next to unclampedJonathan Tan, Nov 1, 2024
  21. Jeff KingNov 13, 2024
  22. Jonathan TanNov 13, 2024
  23. Jeff KingNov 14, 2024
  24. Junio C HamanoNov 14, 2024
  25. Jeff KingNov 15, 2024
  26. Jonathan TanNov 15, 2024
  27. Jeff KingNov 16, 2024
  28. index-pack: teach --promisor to require --stdinJonathan Tan, Nov 18, 2024
  29. Junio C HamanoNov 19, 2024
  30. Jeff KingNov 19, 2024
  31. Junio C HamanoNov 20, 2024
  32. index-pack: teach --promisor to forbid pack nameJonathan Tan, Nov 19, 2024
  33. Jeff KingNov 20, 2024
  34. Jeff KingNov 14, 2024
  35. 4/4 index-pack: repack local links into promisor packsJonathan Tan, Nov 1, 2024
  36. Junio C HamanoNov 4, 2024
  37. Junio C HamanoNov 4, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.