{"thread":{"id":"63286","subject":"[RFC PATCH 0/8] repack: avoid MIDX'ing cruft pack(s) where possible","startedAt":"2025-04-11T23:26:19Z","lastAt":"2025-06-24T16:10:43Z","messageCount":105,"participants":["Taylor Blau","Junio C Hamano","Elijah Newren","Carlo Marcelo Arenas Belón","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":8},"messages":[{"id":"516016","messageId":"cover.1744413969.git.me@ttaylorr.com","threadId":"63286","inReplyTo":null,"subject":"[RFC PATCH 0/8] repack: avoid MIDX'ing cruft pack(s) where possible","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-11T23:26:16Z","receivedAt":"2025-04-11T23:26:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This is a short-ish series I wrote today while thinking through an idea\nthat Peff and I were talking about yesterday that allows us to avoid\nMIDX'ing any cruft pack(s) in a repository when repacking.\n\nThe core of the idea is to introduce a variant of the '--stdin-packs'\noption in 'pack-objects'. The existing behavior is to create a pack\nwhose contents is the set difference between the specified included and\nexclude packs. The new mode (which I'm calling --stdin-packs=follow)\ntweaks the namehash traversal we do at the end of --stdin-packs to also\npick up and pack objects which were reachable from commits in the above\nset difference, but don't appear in the included or excluded pack.\n\nIf you repack consistently using this strategy, you can guarantee that\nthe union of geometrically-repacked packs are closed under reachability\nwithout having to keep track of any cruft pack(s) in the MIDX.\n\nI'm pretty sure that this is all sound, having played with it for the\nbetter part of the day and not being able to come up with any\ncounter-examples. I'm sending this as an RFC because I'm not sure if\nthere's an obvious case that I am missing that makes this whole idea\nbogus.\n\nCode-review is welcome, but I think at this stage it may be more useful\nto center the discussion around whether or not the idea makes sense\nfirst.\n\nThanks in advance :-).\n\nTaylor Blau (8):\n pack-objects: use standard option incompatibility functions\n pack-objects: limit scope in 'add_object_entry_from_pack()'\n pack-objects: factor out handling '--stdin-packs'\n pack-objects: declare 'rev_info' for '--stdin-packs' earlier\n pack-objects: perform name-hash traversal for unpacked objects\n pack-objects: introduce '--stdin-packs=follow'\n repack: keep track of existing MIDX'd packs\n repack: exclude cruft pack(s) from the MIDX where possible\n\n Documentation/git-pack-objects.adoc | 8 +-\n builtin/pack-objects.c | 193 +++++++++++++++++-----------\n builtin/repack.c | 97 +++++++++++---\n t/t5331-pack-objects-stdin.sh | 103 ++++++++++++++-\n t/t7704-repack-cruft.sh | 70 ++++++++++\n 5 files changed, 376 insertions(+), 95 deletions(-)\n\n\nbase-commit: 485f5f863615e670fd97ae40af744e14072cfe18\n-- \n2.49.0.229.g19b69c1246\n"},{"id":"516017","messageId":"63fb4dab30a6bf1dee4318341078fe9eb4716537.1744413969.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744413969.git.me@ttaylorr.com","subject":"[RFC PATCH 1/8] pack-objects: use standard option incompatibility functions","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-11T23:26:20Z","receivedAt":"2025-04-11T23:26:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"pack-objects has a handful of explicit checks for pairs of command-line\noptions which are mutually incompatible. Many of these pre-date\na699367bb8 (i18n: factorize more 'incompatible options' messages,\n2022-01-31).\n\nConvert the explicit checks into die_for_incompatible_opt2() calls,\nwhich simplifies the implementation and standardizes pack-objects'\noutput when given incompatible options (e.g., --stdin-packs with\n--filter gives different output than --keep-unreachable with\n--unpack-unreachable).\n\nThere is one minor piece of test fallout in t5331 that expects the old\nformat, which has been corrected.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 19 ++++++++++---------\n t/t5331-pack-objects-stdin.sh | 2 +-\n 2 files changed, 11 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 6b06d159d2..aaea968ed2 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -4651,9 +4651,10 @@ int cmd_pack_objects(int argc,\n \t\tstrvec_push(&rp, \"--unpacked\");\n \t}\n \n-\tif (exclude_promisor_objects && exclude_promisor_objects_best_effort)\n-\t\tdie(_(\"options '%s' and '%s' cannot be used together\"),\n-\t\t \"--exclude-promisor-objects\", \"--exclude-promisor-objects-best-effort\");\n+\tdie_for_incompatible_opt2(exclude_promisor_objects,\n+\t\t\t\t \"--exclude-promisor-objects\",\n+\t\t\t\t exclude_promisor_objects_best_effort,\n+\t\t\t\t \"--exclude-promisor-objects-best-effort\");\n \tif (exclude_promisor_objects) {\n \t\tuse_internal_rev_list = 1;\n \t\tfetch_if_missing = 0;\n@@ -4691,13 +4692,13 @@ int cmd_pack_objects(int argc,\n \tif (!pack_to_stdout && thin)\n \t\tdie(_(\"--thin cannot be used to build an indexable pack\"));\n \n-\tif (keep_unreachable && unpack_unreachable)\n-\t\tdie(_(\"options '%s' and '%s' cannot be used together\"), \"--keep-unreachable\", \"--unpack-unreachable\");\n+\tdie_for_incompatible_opt2(keep_unreachable, \"--keep-unreachable\",\n+\t\t\t\t unpack_unreachable, \"--unpack-unreachable\");\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n-\tif (stdin_packs && filter_options.choice)\n-\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n+\tdie_for_incompatible_opt2(filter_options.choice, \"--filter\",\n+\t\t\t\t stdin_packs, \"--stdin-packs\");\n \n \tif (stdin_packs && use_internal_rev_list)\n \t\tdie(_(\"cannot use internal rev list with --stdin-packs\"));\n@@ -4705,8 +4706,8 @@ int cmd_pack_objects(int argc,\n \tif (cruft) {\n \t\tif (use_internal_rev_list)\n \t\t\tdie(_(\"cannot use internal rev list with --cruft\"));\n-\t\tif (stdin_packs)\n-\t\t\tdie(_(\"cannot use --stdin-packs with --cruft\"));\n+\t\tdie_for_incompatible_opt2(stdin_packs, \"--stdin-packs\",\n+\t\t\t\t\t cruft, \"--cruft\");\n \t}\n \n \t/*\ndiff --git a/t/t5331-pack-objects-stdin.sh b/t/t5331-pack-objects-stdin.sh\nindex b48c0cbe8f..4f5e2733a2 100755\n--- a/t/t5331-pack-objects-stdin.sh\n+++ b/t/t5331-pack-objects-stdin.sh\n@@ -64,7 +64,7 @@ test_expect_success '--stdin-packs is incompatible with --filter' '\n \t\tcd stdin-packs &&\n \t\ttest_must_fail git pack-objects --stdin-packs --stdout \\\n \t\t\t--filter=blob:none err &&\n-\t\ttest_grep \"cannot use --filter with --stdin-packs\" err\n+\t\ttest_grep \"options .--filter. and .--stdin-packs. cannot be used together\" err\n \t)\n '\n \n-- \n2.49.0.229.g19b69c1246\n\n"},{"id":"516018","messageId":"6357633f6d144f24ac561ccba908aea3aed7aaaf.1744413969.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744413969.git.me@ttaylorr.com","subject":"[RFC PATCH 2/8] pack-objects: limit scope in 'add_object_entry_from_pack()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-11T23:26:23Z","receivedAt":"2025-04-11T23:26:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"add_object_entry_from_pack() handles objects from identified packs by\nchecking their type, before adding commit objects as pending in the\nsubsequent traversal used by `--stdin-packs`.\n\nThere are a couple of quality-of-life refactorings that I noticed while\nworking in this area:\n\n - We declare 'revs' (given to us through the miscellaneous context\n argument) earlier in the \"if (p)\" conditional than is necessary.\n\n - The 'struct object_info' can use a designated initializer to fill in\n the structures type pointer, since that is the only field that we\n care about.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 6 ++----\n 1 file changed, 2 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex aaea968ed2..540e5eba9e 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3490,14 +3490,12 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n \t\treturn 0;\n \n \tif (p) {\n-\t\tstruct rev_info *revs = _data;\n-\t\tstruct object_info oi = OBJECT_INFO_INIT;\n-\n-\t\toi.typep = &type;\n+\t\tstruct object_info oi = { .typep = &type };\n \t\tif (packed_object_info(the_repository, p, ofs, &oi) < 0) {\n \t\t\tdie(_(\"could not get type of object %s in pack %s\"),\n \t\t\t oid_to_hex(oid), p->pack_name);\n \t\t} else if (type == OBJ_COMMIT) {\n+\t\t\tstruct rev_info *revs = _data;\n \t\t\t/*\n \t\t\t * commits in included packs are used as starting points for the\n \t\t\t * subsequent revision walk\n-- \n2.49.0.229.g19b69c1246\n\n"},{"id":"516019","messageId":"43e889b15762b2aed65d0c5fbaef19bcb65e8f14.1744413969.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744413969.git.me@ttaylorr.com","subject":"[RFC PATCH 3/8] pack-objects: factor out handling '--stdin-packs'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-11T23:26:26Z","receivedAt":"2025-04-11T23:26:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"At the bottom of cmd_pack_objects() we check which mode the command is\nrunning in (e.g., generating a cruft pack, handling '--stdin-packs',\nusing the internal rev-list, etc.) and handle the mode appropriately.\n\nThe '--stdin-packs' case is handled inline (dating back to its\nintroduction in 339bce27f4 (builtin/pack-objects.c: add '--stdin-packs'\noption, 2021-02-22)) since it is relatively short. Extract the body of\n\"if (stdin_packs)\" into its own function to prepare for the\nimplementation to become lengthier in a following commit.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 18 ++++++++++++------\n 1 file changed, 12 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 540e5eba9e..793d245721 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3672,6 +3672,17 @@ static void read_packs_list_from_stdin(void)\n \tstring_list_clear(&exclude_packs, 0);\n }\n \n+static void add_unreachable_loose_objects(void);\n+\n+static void read_stdin_packs(int rev_list_unpacked)\n+{\n+\t/* avoids adding objects in excluded packs */\n+\tignore_packed_keep_in_core = 1;\n+\tread_packs_list_from_stdin();\n+\tif (rev_list_unpacked)\n+\t\tadd_unreachable_loose_objects();\n+}\n+\n static void add_cruft_object_entry(const struct object_id *oid, enum object_type type,\n \t\t\t\t struct packed_git *pack, off_t offset,\n \t\t\t\t const char *name, uint32_t mtime)\n@@ -3767,7 +3778,6 @@ static void mark_pack_kept_in_core(struct string_list *packs, unsigned keep)\n \t}\n }\n \n-static void add_unreachable_loose_objects(void);\n static void add_objects_in_unpacked_packs(void);\n \n static void enumerate_cruft_objects(void)\n@@ -4773,11 +4783,7 @@ int cmd_pack_objects(int argc,\n \t\tprogress_state = start_progress(the_repository,\n \t\t\t\t\t\t_(\"Enumerating objects\"), 0);\n \tif (stdin_packs) {\n-\t\t/* avoids adding objects in excluded packs */\n-\t\tignore_packed_keep_in_core = 1;\n-\t\tread_packs_list_from_stdin();\n-\t\tif (rev_list_unpacked)\n-\t\t\tadd_unreachable_loose_objects();\n+\t\tread_stdin_packs(rev_list_unpacked);\n \t} else if (cruft) {\n \t\tread_cruft_objects();\n \t} else if (!use_internal_rev_list) {\n-- \n2.49.0.229.g19b69c1246\n\n"},{"id":"516020","messageId":"07a91be3ec07183d440e5ba6d579995d63c6ef85.1744413969.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744413969.git.me@ttaylorr.com","subject":"[RFC PATCH 4/8] pack-objects: declare 'rev_info' for '--stdin-packs' earlier","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-11T23:26:29Z","receivedAt":"2025-04-11T23:26:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Once 'read_packs_list_from_stdin()' has called for_each_object_in_pack()\non each of the input packs, we do a reachability traversal to discover\nnames for any objects we picked up so we can generate name hash values\nand hopefully get higher quality deltas as a result.\n\nA future commit will change the purpose of this reachability traversal\nto find and pack objects which are reachable from commits in the input\npacks, but are packed in an unknown (not included nor excluded) pack.\n\nExtract the code which initializes and performs the reachability\ntraversal to take place in the caller, not the callee, which prepares us\nto share this code for the '--unpacked' case (see the function\nadd_unreachable_loose_objects() for more details).\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 71 +++++++++++++++++++++---------------------\n 1 file changed, 36 insertions(+), 35 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 793d245721..1689cddd3a 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3556,7 +3556,7 @@ static int pack_mtime_cmp(const void *_a, const void *_b)\n \t\treturn 0;\n }\n \n-static void read_packs_list_from_stdin(void)\n+static void read_packs_list_from_stdin(struct rev_info *revs)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct string_list include_packs = STRING_LIST_INIT_DUP;\n@@ -3564,24 +3564,6 @@ static void read_packs_list_from_stdin(void)\n \tstruct string_list_item *item = NULL;\n \n \tstruct packed_git *p;\n-\tstruct rev_info revs;\n-\n-\trepo_init_revisions(the_repository, &revs, NULL);\n-\t/*\n-\t * Use a revision walk to fill in the namehash of objects in the include\n-\t * packs. To save time, we'll avoid traversing through objects that are\n-\t * in excluded packs.\n-\t *\n-\t * That may cause us to avoid populating all of the namehash fields of\n-\t * all included objects, but our goal is best-effort, since this is only\n-\t * an optimization during delta selection.\n-\t */\n-\trevs.no_kept_objects = 1;\n-\trevs.keep_pack_cache_flags |= IN_CORE_KEEP_PACKS;\n-\trevs.blob_objects = 1;\n-\trevs.tree_objects = 1;\n-\trevs.tag_objects = 1;\n-\trevs.ignore_missing_links = 1;\n \n \twhile (strbuf_getline(&buf, stdin) != EOF) {\n \t\tif (!buf.len)\n@@ -3651,10 +3633,44 @@ static void read_packs_list_from_stdin(void)\n \t\tstruct packed_git *p = item->util;\n \t\tfor_each_object_in_pack(p,\n \t\t\t\t\tadd_object_entry_from_pack,\n-\t\t\t\t\t&revs,\n+\t\t\t\t\trevs,\n \t\t\t\t\tFOR_EACH_OBJECT_PACK_ORDER);\n \t}\n \n+\tstrbuf_release(&buf);\n+\tstring_list_clear(&include_packs, 0);\n+\tstring_list_clear(&exclude_packs, 0);\n+}\n+\n+static void add_unreachable_loose_objects(void);\n+\n+static void read_stdin_packs(int rev_list_unpacked)\n+{\n+\tstruct rev_info revs;\n+\n+\trepo_init_revisions(the_repository, &revs, NULL);\n+\t/*\n+\t * Use a revision walk to fill in the namehash of objects in the include\n+\t * packs. To save time, we'll avoid traversing through objects that are\n+\t * in excluded packs.\n+\t *\n+\t * That may cause us to avoid populating all of the namehash fields of\n+\t * all included objects, but our goal is best-effort, since this is only\n+\t * an optimization during delta selection.\n+\t */\n+\trevs.no_kept_objects = 1;\n+\trevs.keep_pack_cache_flags |= IN_CORE_KEEP_PACKS;\n+\trevs.blob_objects = 1;\n+\trevs.tree_objects = 1;\n+\trevs.tag_objects = 1;\n+\trevs.ignore_missing_links = 1;\n+\n+\t/* avoids adding objects in excluded packs */\n+\tignore_packed_keep_in_core = 1;\n+\tread_packs_list_from_stdin(&revs);\n+\tif (rev_list_unpacked)\n+\t\tadd_unreachable_loose_objects();\n+\n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n \ttraverse_commit_list(&revs,\n@@ -3666,21 +3682,6 @@ static void read_packs_list_from_stdin(void)\n \t\t\t stdin_packs_found_nr);\n \ttrace2_data_intmax(\"pack-objects\", the_repository, \"stdin_packs_hints\",\n \t\t\t stdin_packs_hints_nr);\n-\n-\tstrbuf_release(&buf);\n-\tstring_list_clear(&include_packs, 0);\n-\tstring_list_clear(&exclude_packs, 0);\n-}\n-\n-static void add_unreachable_loose_objects(void);\n-\n-static void read_stdin_packs(int rev_list_unpacked)\n-{\n-\t/* avoids adding objects in excluded packs */\n-\tignore_packed_keep_in_core = 1;\n-\tread_packs_list_from_stdin();\n-\tif (rev_list_unpacked)\n-\t\tadd_unreachable_loose_objects();\n }\n \n static void add_cruft_object_entry(const struct object_id *oid, enum object_type type,\n-- \n2.49.0.229.g19b69c1246\n\n"},{"id":"516021","messageId":"241f7c87e52c4123d0e00d75f5af201bd37fcfd4.1744413969.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744413969.git.me@ttaylorr.com","subject":"[RFC PATCH 5/8] pack-objects: perform name-hash traversal for unpacked objects","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-11T23:26:32Z","receivedAt":"2025-04-11T23:26:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"With '--unpacked', pack-objects adds loose objects (which don't appear\nin any of the excluded packs from '--stdin-packs') to the output pack\nwithout considering them as reachability tips for the name-hash\ntraversal.\n\nThis was an oversight in the original implementation of '--stdin-packs',\nsince the code which enumerates and adds loose objects to the output\npack (`add_unreachable_loose_objects()`) did not have access to the\n'rev_info' struct found in `read_packs_list_from_stdin()`.\n\nExcluding unpacked objects from that traversal doesn't effect the\ncorrectness of the resulting pack, but it does make it harder to\ndiscover good deltas for loose objects.\n\nNow that the 'rev_info' struct is declared outside of\n`read_packs_list_from_stdin()`, we can pass it to\n`add_objects_in_unpacked_packs()` and add any loose objects as tips to\nthe above-mentioned traversal, in theory producing slightly tighter\npacks as a result.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 20 ++++++++++++--------\n 1 file changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 1689cddd3a..2aa12da4af 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3642,7 +3642,7 @@ static void read_packs_list_from_stdin(struct rev_info *revs)\n \tstring_list_clear(&exclude_packs, 0);\n }\n \n-static void add_unreachable_loose_objects(void);\n+static void add_unreachable_loose_objects(struct rev_info *revs);\n \n static void read_stdin_packs(int rev_list_unpacked)\n {\n@@ -3669,7 +3669,7 @@ static void read_stdin_packs(int rev_list_unpacked)\n \tignore_packed_keep_in_core = 1;\n \tread_packs_list_from_stdin(&revs);\n \tif (rev_list_unpacked)\n-\t\tadd_unreachable_loose_objects();\n+\t\tadd_unreachable_loose_objects(&revs);\n \n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n@@ -3788,7 +3788,7 @@ static void enumerate_cruft_objects(void)\n \t\t\t\t\t\t_(\"Enumerating cruft objects\"), 0);\n \n \tadd_objects_in_unpacked_packs();\n-\tadd_unreachable_loose_objects();\n+\tadd_unreachable_loose_objects(NULL);\n \n \tstop_progress(&progress_state);\n }\n@@ -4066,8 +4066,9 @@ static void add_objects_in_unpacked_packs(void)\n }\n \n static int add_loose_object(const struct object_id *oid, const char *path,\n-\t\t\t void *data UNUSED)\n+\t\t\t void *data)\n {\n+\tstruct rev_info *revs = data;\n \tenum object_type type = oid_object_info(the_repository, oid, NULL);\n \n \tif (type < 0) {\n@@ -4088,6 +4089,10 @@ static int add_loose_object(const struct object_id *oid, const char *path,\n \t} else {\n \t\tadd_object_entry(oid, type, \"\", 0);\n \t}\n+\n+\tif (revs && type == OBJ_COMMIT)\n+\t\tadd_pending_oid(revs, NULL, oid, 0);\n+\n \treturn 0;\n }\n \n@@ -4096,11 +4101,10 @@ static int add_loose_object(const struct object_id *oid, const char *path,\n * add_object_entry will weed out duplicates, so we just add every\n * loose object we find.\n */\n-static void add_unreachable_loose_objects(void)\n+static void add_unreachable_loose_objects(struct rev_info *revs)\n {\n \tfor_each_loose_file_in_objdir(repo_get_object_directory(the_repository),\n-\t\t\t\t add_loose_object,\n-\t\t\t\t NULL, NULL, NULL);\n+\t\t\t\t add_loose_object, NULL, NULL, revs);\n }\n \n static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)\n@@ -4356,7 +4360,7 @@ static void get_object_list(struct rev_info *revs, int ac, const char **av)\n \tif (keep_unreachable)\n \t\tadd_objects_in_unpacked_packs();\n \tif (pack_loose_unreachable)\n-\t\tadd_unreachable_loose_objects();\n+\t\tadd_unreachable_loose_objects(NULL);\n \tif (unpack_unreachable)\n \t\tloosen_unused_packed_objects();\n \n-- \n2.49.0.229.g19b69c1246\n\n"},{"id":"516022","messageId":"a0318321ece4c7f7bcffe85c1e4d5cc85a673bdd.1744413969.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744413969.git.me@ttaylorr.com","subject":"[RFC PATCH 6/8] pack-objects: introduce '--stdin-packs=follow'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-11T23:26:35Z","receivedAt":"2025-04-11T23:26:37Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When invoked with '--stdin-packs', pack-objects will generate a pack\nwhich contains the objects found in the \"included\" packs, less any\nobjects from \"excluded\" packs.\n\nPacks that exist in the repository but weren't specified as either\nincluded or excluded are in practice treated like the latter, at least\nin the sense that pack-objects won't include objects from those packs.\nThis behavior forces us to include any cruft pack(s) in a repository's\nmulti-pack index for the reasons described in ddee3703b3\n(builtin/repack.c: add cruft packs to MIDX during geometric repack,\n2022-05-20).\n\nThe full details are in ddee3703b3, but the gist is if you\nhave a once-unreachable object in a cruft pack which later becomes\nreachable via one or more commits in a pack generated with\n'--stdin-packs', you *have* to include that object in the MIDX via the\ncopy in the cruft pack, otherwise we cannot generate reachability\nbitmaps for any commits which reach that object.\n\nThis prepares us for new repacking behavior which will \"resurrect\"\nobjects found in cruft or otherwise unspecified packs when generating\nnew packs. In the context of geometric repacking, this may be used to\nmaintain a sequence of geometrically-repacked packs, the union of which\nis closed under reachability, even in the case described earlier.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-pack-objects.adoc | 8 ++-\n builtin/pack-objects.c | 89 +++++++++++++++++-------\n t/t5331-pack-objects-stdin.sh | 101 ++++++++++++++++++++++++++++\n 3 files changed, 171 insertions(+), 27 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.adoc b/Documentation/git-pack-objects.adoc\nindex 7f69ae4855..c894582799 100644\n--- a/Documentation/git-pack-objects.adoc\n+++ b/Documentation/git-pack-objects.adoc\n@@ -87,13 +87,19 @@ base-name::\n \treference was included in the resulting packfile. This\n \tcan be useful to send new tags to native Git clients.\n \n---stdin-packs::\n+--stdin-packs[=]::\n \tRead the basenames of packfiles (e.g., `pack-1234abcd.pack`)\n \tfrom the standard input, instead of object names or revision\n \targuments. The resulting pack contains all objects listed in the\n \tincluded packs (those not beginning with `^`), excluding any\n \tobjects listed in the excluded packs (beginning with `^`).\n +\n+When `mode` is \"follow\", pack objects which are reachable from objects\n+in the included packs, but appear in packs that are not listed.\n+Reachable objects which appear in excluded packs are not packed. Useful\n+for resurrecting once-cruft objects to generate packs which are closed\n+under reachability up to the excluded packs.\n++\n Incompatible with `--revs`, or options that imply `--revs` (such as\n `--all`), with the exception of `--unpacked`, which is compatible.\n \ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 2aa12da4af..6406f4a5b1 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -272,6 +272,12 @@ static struct oidmap configured_exclusions;\n static struct oidset excluded_by_config;\n static int name_hash_version = -1;\n \n+enum stdin_packs_mode {\n+\tSTDIN_PACKS_MODE_NONE,\n+\tSTDIN_PACKS_MODE_STANDARD,\n+\tSTDIN_PACKS_MODE_FOLLOW,\n+};\n+\n /**\n * Check whether the name_hash_version chosen by user input is appropriate,\n * and also validate whether it is compatible with other features.\n@@ -3511,32 +3517,43 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n \treturn 0;\n }\n \n-static void show_commit_pack_hint(struct commit *commit UNUSED,\n-\t\t\t\t void *data UNUSED)\n-{\n-\t/* nothing to do; commits don't have a namehash */\n-}\n-\n static void show_object_pack_hint(struct object *object, const char *name,\n-\t\t\t\t void *data UNUSED)\n+\t\t\t\t void *data)\n {\n-\tstruct object_entry *oe = packlist_find(&to_pack, &object->oid);\n-\tif (!oe)\n+\tenum stdin_packs_mode mode = *(enum stdin_packs_mode *)data;\n+\tif (mode == STDIN_PACKS_MODE_FOLLOW) {\n+\t\tadd_object_entry(&object->oid, object->type, name, 0);\n+\t} else {\n+\t\tstruct object_entry *oe = packlist_find(&to_pack, &object->oid);\n+\t\tif (!oe)\n+\t\t\treturn;\n+\n+\t\t/*\n+\t\t * Our 'to_pack' list was constructed by iterating all\n+\t\t * objects packed in included packs, and so doesn't\n+\t\t * have a non-zero hash field that you would typically\n+\t\t * pick up during a reachability traversal.\n+\t\t *\n+\t\t * Make a best-effort attempt to fill in the ->hash\n+\t\t * and ->no_try_delta here using a now in order to\n+\t\t * perhaps improve the delta selection process.\n+\t\t */\n+\t\toe->hash = pack_name_hash_fn(name);\n+\t\toe->no_try_delta = name && no_try_delta(name);\n+\n+\t\tstdin_packs_hints_nr++;\n+\t}\n+}\n+\n+static void show_commit_pack_hint(struct commit *commit, void *data)\n+{\n+\tenum stdin_packs_mode mode = *(enum stdin_packs_mode *)data;\n+\tif (mode == STDIN_PACKS_MODE_FOLLOW) {\n+\t\tshow_object_pack_hint((struct object *)commit, \"\", data);\n \t\treturn;\n+\t}\n+\t/* nothing to do; commits don't have a namehash */\n \n-\t/*\n-\t * Our 'to_pack' list was constructed by iterating all objects packed in\n-\t * included packs, and so doesn't have a non-zero hash field that you\n-\t * would typically pick up during a reachability traversal.\n-\t *\n-\t * Make a best-effort attempt to fill in the ->hash and ->no_try_delta\n-\t * here using a now in order to perhaps improve the delta selection\n-\t * process.\n-\t */\n-\toe->hash = pack_name_hash_fn(name);\n-\toe->no_try_delta = name && no_try_delta(name);\n-\n-\tstdin_packs_hints_nr++;\n }\n \n static int pack_mtime_cmp(const void *_a, const void *_b)\n@@ -3644,7 +3661,7 @@ static void read_packs_list_from_stdin(struct rev_info *revs)\n \n static void add_unreachable_loose_objects(struct rev_info *revs);\n \n-static void read_stdin_packs(int rev_list_unpacked)\n+static void read_stdin_packs(enum stdin_packs_mode mode, int rev_list_unpacked)\n {\n \tstruct rev_info revs;\n \n@@ -3676,7 +3693,7 @@ static void read_stdin_packs(int rev_list_unpacked)\n \ttraverse_commit_list(&revs,\n \t\t\t show_commit_pack_hint,\n \t\t\t show_object_pack_hint,\n-\t\t\t NULL);\n+\t\t\t &mode);\n \n \ttrace2_data_intmax(\"pack-objects\", the_repository, \"stdin_packs_found\",\n \t\t\t stdin_packs_found_nr);\n@@ -4467,6 +4484,23 @@ static int is_not_in_promisor_pack(struct commit *commit, void *data) {\n \treturn is_not_in_promisor_pack_obj((struct object *) commit, data);\n }\n \n+static int parse_stdin_packs_mode(const struct option *opt, const char *arg,\n+\t\t\t\t int unset)\n+{\n+\tenum stdin_packs_mode *mode = opt->value;\n+\n+\tif (unset)\n+\t\t*mode = STDIN_PACKS_MODE_NONE;\n+\telse if (!arg || !*arg)\n+\t\t*mode = STDIN_PACKS_MODE_STANDARD;\n+\telse if (!strcmp(arg, \"follow\"))\n+\t\t*mode = STDIN_PACKS_MODE_FOLLOW;\n+\telse\n+\t\tdie(_(\"invalid value for '%s': '%s'\"), opt->long_name, arg);\n+\n+\treturn 0;\n+}\n+\n int cmd_pack_objects(int argc,\n \t\t const char **argv,\n \t\t const char *prefix,\n@@ -4478,7 +4512,7 @@ int cmd_pack_objects(int argc,\n \tstruct strvec rp = STRVEC_INIT;\n \tint rev_list_unpacked = 0, rev_list_all = 0, rev_list_reflog = 0;\n \tint rev_list_index = 0;\n-\tint stdin_packs = 0;\n+\tenum stdin_packs_mode stdin_packs = STDIN_PACKS_MODE_NONE;\n \tstruct string_list keep_pack_list = STRING_LIST_INIT_NODUP;\n \tstruct list_objects_filter_options filter_options =\n \t\tLIST_OBJECTS_FILTER_INIT;\n@@ -4533,6 +4567,9 @@ int cmd_pack_objects(int argc,\n \t\tOPT_SET_INT_F(0, \"indexed-objects\", &rev_list_index,\n \t\t\t N_(\"include objects referred to by the index\"),\n \t\t\t 1, PARSE_OPT_NONEG),\n+\t\tOPT_CALLBACK_F(0, \"stdin-packs\", &stdin_packs, N_(\"mode\"),\n+\t\t\t N_(\"read packs from stdin\"),\n+\t\t\t PARSE_OPT_OPTARG, parse_stdin_packs_mode),\n \t\tOPT_BOOL(0, \"stdin-packs\", &stdin_packs,\n \t\t\t N_(\"read packs from stdin\")),\n \t\tOPT_BOOL(0, \"stdout\", &pack_to_stdout,\n@@ -4788,7 +4825,7 @@ int cmd_pack_objects(int argc,\n \t\tprogress_state = start_progress(the_repository,\n \t\t\t\t\t\t_(\"Enumerating objects\"), 0);\n \tif (stdin_packs) {\n-\t\tread_stdin_packs(rev_list_unpacked);\n+\t\tread_stdin_packs(stdin_packs, rev_list_unpacked);\n \t} else if (cruft) {\n \t\tread_cruft_objects();\n \t} else if (!use_internal_rev_list) {\ndiff --git a/t/t5331-pack-objects-stdin.sh b/t/t5331-pack-objects-stdin.sh\nindex 4f5e2733a2..f97d2d1b71 100755\n--- a/t/t5331-pack-objects-stdin.sh\n+++ b/t/t5331-pack-objects-stdin.sh\n@@ -236,4 +236,105 @@ test_expect_success 'pack-objects --stdin with packfiles from main and alternate\n \ttest_cmp expected-objects actual-objects\n '\n \n+packdir=.git/objects/pack\n+\n+objects_in_packs () {\n+\tfor p in \"$@\"\n+\tdo\n+\t\tgit show-index <\"$packdir/pack-$p.idx\" || return 1\n+\tdone >objects.raw &&\n+\n+\tcut -d' ' -f2 objects.raw | sort &&\n+\trm -f objects.raw\n+}\n+\n+test_expect_success 'setup for --stdin-packs=follow' '\n+\tgit init stdin-packs--follow &&\n+\t(\n+\t\tcd stdin-packs--follow &&\n+\n+\t\tfor c in A B C D\n+\t\tdo\n+\t\t\ttest_commit \"$c\" || return 1\n+\t\tdone &&\n+\n+\t\tA=\"$(echo A | git pack-objects --revs $packdir/pack)\" &&\n+\t\tB=\"$(echo A..B | git pack-objects --revs $packdir/pack)\" &&\n+\t\tC=\"$(echo B..C | git pack-objects --revs $packdir/pack)\" &&\n+\t\tD=\"$(echo C..D | git pack-objects --revs $packdir/pack)\" &&\n+\n+\t\tgit prune-packed\n+\t)\n+'\n+\n+test_expect_success '--stdin-packs=follow walks into unknown packs' '\n+\ttest_when_finished \"rm -fr repo\" &&\n+\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\tfor c in A B C D\n+\t\tdo\n+\t\t\ttest_commit \"$c\" || return 1\n+\t\tdone &&\n+\n+\t\tA=\"$(echo A | git pack-objects --revs $packdir/pack)\" &&\n+\t\tB=\"$(echo A..B | git pack-objects --revs $packdir/pack)\" &&\n+\t\tC=\"$(echo B..C | git pack-objects --revs $packdir/pack)\" &&\n+\t\tD=\"$(echo C..D | git pack-objects --revs $packdir/pack)\" &&\n+\n+\t\tgit prune-packed &&\n+\n+\t\tcat >in <<-EOF &&\n+\t\tpack-$B.pack\n+\t\t^pack-$C.pack\n+\t\tpack-$D.pack\n+\t\tEOF\n+\n+\t\t# With just --stdin-packs, pack \"A\" is unknown to us, so\n+\t\t# only objects from packs \"B\" and \"D\" are included in\n+\t\t# the output pack.\n+\t\tP=$(git pack-objects --stdin-packs $packdir/pack expect &&\n+\t\tobjects_in_packs $P >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\t# But with --stdin-packs=follow, objects from both\n+\t\t# included packs reach objects from the unknown pack, so\n+\t\t# objects from pack \"A\" is included in the output pack\n+\t\t# in addition to the above.\n+\t\tP=$(git pack-objects --stdin-packs=follow $packdir/pack expect &&\n+\t\tobjects_in_packs $P >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\ttest_commit E &&\n+\t\t# And with --unpacked, we will pick up objects from unknown\n+\t\t# packs that are reachable from loose objects. Loose object E\n+\t\t# reaches objects in pack A, but there are three excluded packs\n+\t\t# in between.\n+\t\t#\n+\t\t# The resulting pack should include objects reachable from E\n+\t\t# that are not present in packs B, C, or D, along with those\n+\t\t# present in pack A.\n+\t\tcat >in <<-EOF &&\n+\t\t^pack-$B.pack\n+\t\t^pack-$C.pack\n+\t\t^pack-$D.pack\n+\t\tEOF\n+\n+\t\tP=$(git pack-objects --stdin-packs=follow --unpacked \\\n+\t\t\t$packdir/pack expect.raw &&\n+\t\tsort expect.raw >expect &&\n+\t\tobjects_in_packs $P >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \n2.49.0.229.g19b69c1246\n\n"},{"id":"516023","messageId":"ef0bc38cf0bfa056421e2288641f585a40a44ab0.1744413969.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744413969.git.me@ttaylorr.com","subject":"[RFC PATCH 7/8] repack: keep track of existing MIDX'd packs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-11T23:26:38Z","receivedAt":"2025-04-11T23:26:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The following commit will want to condition whether or not it generates\na pack during geometric repacking with '--stdin-packs=follow' based on\nwhether or not the existing MIDX has a cruft pack in it.\n\nKeep track of that in the 'existing_packs' struct by adding an\nadditional flag bit to denote which packs appear in a MIDX.\n\nSigned-off-by: Taylor Blau \n---\n builtin/repack.c | 19 ++++++++++++++++---\n 1 file changed, 16 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex f3330ade7b..bc47bede7b 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -31,6 +31,7 @@\n \n #define DELETE_PACK 1\n #define RETAIN_PACK 2\n+#define PACK_IN_MIDX 4\n \n static int pack_everything;\n static int delta_base_offset = 1;\n@@ -161,6 +162,11 @@ static int pack_is_retained(struct string_list_item *item)\n \treturn (uintptr_t)item->util & RETAIN_PACK;\n }\n \n+static void pack_mark_in_midx(struct string_list_item *item)\n+{\n+\titem->util = (void*)((uintptr_t)item->util | PACK_IN_MIDX);\n+}\n+\n static void mark_packs_for_deletion_1(struct string_list *names,\n \t\t\t\t struct string_list *list)\n {\n@@ -264,6 +270,7 @@ static void collect_pack_filenames(struct existing_packs *existing,\n \tfor (p = get_all_packs(the_repository); p; p = p->next) {\n \t\tint i;\n \t\tconst char *base;\n+\t\tstruct string_list_item *item;\n \n \t\tif (!p->pack_local)\n \t\t\tcontinue;\n@@ -279,11 +286,17 @@ static void collect_pack_filenames(struct existing_packs *existing,\n \t\tstrbuf_strip_suffix(&buf, \".pack\");\n \n \t\tif ((extra_keep->nr > 0 && i < extra_keep->nr) || p->pack_keep)\n-\t\t\tstring_list_append(&existing->kept_packs, buf.buf);\n+\t\t\titem = string_list_append(&existing->kept_packs,\n+\t\t\t\t\t\t buf.buf);\n \t\telse if (p->is_cruft)\n-\t\t\tstring_list_append(&existing->cruft_packs, buf.buf);\n+\t\t\titem = string_list_append(&existing->cruft_packs,\n+\t\t\t\t\t\t buf.buf);\n \t\telse\n-\t\t\tstring_list_append(&existing->non_kept_packs, buf.buf);\n+\t\t\titem = string_list_append(&existing->non_kept_packs,\n+\t\t\t\t\t\t buf.buf);\n+\n+\t\tif (p->multi_pack_index)\n+\t\t\tpack_mark_in_midx(item);\n \t}\n \n \tstring_list_sort(&existing->kept_packs);\n-- \n2.49.0.229.g19b69c1246\n\n"},{"id":"516024","messageId":"19b69c124696c31c04a09f56ef03bf8ca9c61c4f.1744413969.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744413969.git.me@ttaylorr.com","subject":"[RFC PATCH 8/8] repack: exclude cruft pack(s) from the MIDX where possible","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-11T23:26:41Z","receivedAt":"2025-04-11T23:26:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In ddee3703b3 (builtin/repack.c: add cruft packs to MIDX during\ngeometric repack, 2022-05-20), repack began adding cruft pack(s) to the\nMIDX with '--write-midx' to ensure that the resulting MIDX was always\nclosed under reachability in order to generate reachability bitmaps.\n\nSuppose you have a once-unreachable object packed in a cruft pack, which\nlater on becomes reachable from one or more objects in a geometrically\nrepacked pack. That once-unreachable object *won't* appear in the new\npack, since the cruft pack was specified as neither included nor\nexcluded to 'pack-objects --stdin-packs'. If the bitmap selection\nprocess picks one or more commits which reach the once-unreachable\nobjects, commit ddee3703b3 ensures that the MIDX will be closed under\nreachability. Without it, we would fail to generate a MIDX bitmap.\n\nddee3703b3 alludes to the fact that this is sub-optimal by saying\n\n [...] it's desirable to avoid including cruft packs in the MIDX\n because it causes the MIDX to store a bunch of objects which are\n likely to get thrown away.\n\n, which is true, but hides an even larger problem. If repositories\nrarely prune their unreachable objects and/or have many of them, the\nMIDX must keep track of a large number of objects which bloats the MIDX\nand slows down object lookup.\n\nThis is doubly unfortunate because the vast majority of objects in cruft\npack(s) are unlikely to be read, but object reads that go through the\nMIDX have to search through them anyway.\n\nThis patch causes geometrically-repacked packs to contain a copy of any\nonce-unreachable object(s) with 'git pack-objects --stdin-packs=follow',\nallowing us to avoid including any cruft packs in the MIDX. This is\nbecause a sequence of geometrically-repacked packs that were all\ngenerated with '--stdin-packs=follow' are guaranteed to have their union\nbe closed under reachability.\n\nNote that you cannot guarantee that a collection of packs is closed\nunder reachability if not all of them were generated with following as\nabove. One tell-tale sign that not all geometrically-repacked packs in\nthe MIDX were generated with following is to see if there is a cruft\npack already in the MIDX.\n\nIf there is, then starting to generate packs with following during\ngeometric repacking won't work, since it's open to the same race as\ndescribed above.\n\nBut if you're starting from scratch (e.g., building the first MIDX after\nan all-into-one '--cruft' repack), then you can guarantee that the union\nof subsequently generated packs from geometric repacking *is* closed\nunder reachability.\n\nDetect when this is the case and avoid including cruft packs in the MIDX\nwhere possible.\n\nSigned-off-by: Taylor Blau \n---\n builtin/repack.c | 78 ++++++++++++++++++++++++++++++++---------\n t/t7704-repack-cruft.sh | 70 ++++++++++++++++++++++++++++++++++++\n 2 files changed, 131 insertions(+), 17 deletions(-)\n\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex bc47bede7b..d0b88f12f6 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -167,6 +167,21 @@ static void pack_mark_in_midx(struct string_list_item *item)\n \titem->util = (void*)((uintptr_t)item->util | PACK_IN_MIDX);\n }\n \n+static int pack_is_in_midx(struct string_list_item *item)\n+{\n+\treturn (uintptr_t)item->util & PACK_IN_MIDX;\n+}\n+\n+static int existing_has_cruft_in_midx(struct existing_packs *existing)\n+{\n+\tstruct string_list_item *item;\n+\tfor_each_string_list_item(item, &existing->cruft_packs) {\n+\t\tif (pack_is_in_midx(item))\n+\t\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n static void mark_packs_for_deletion_1(struct string_list *names,\n \t\t\t\t struct string_list *list)\n {\n@@ -821,26 +836,52 @@ static void midx_included_packs(struct string_list *include,\n \t\t}\n \t}\n \n-\tfor_each_string_list_item(item, &existing->cruft_packs) {\n+\tif (existing_has_cruft_in_midx(existing)) {\n \t\t/*\n-\t\t * When doing a --geometric repack, there is no need to check\n-\t\t * for deleted packs, since we're by definition not doing an\n-\t\t * ALL_INTO_ONE repack (hence no packs will be deleted).\n-\t\t * Otherwise we must check for and exclude any packs which are\n-\t\t * enqueued for deletion.\n+\t\t * If we had one or more cruft pack(s) present in the\n+\t\t * MIDX before the repack, keep them as they may be\n+\t\t * required to form a reachability closure if the MIDX\n+\t\t * is bitmapped.\n \t\t *\n-\t\t * So we could omit the conditional below in the --geometric\n-\t\t * case, but doing so is unnecessary since no packs are marked\n-\t\t * as pending deletion (since we only call\n-\t\t * `mark_packs_for_deletion()` when doing an all-into-one\n-\t\t * repack).\n+\t\t * A cruft pack can be required to form a reachability\n+\t\t * closure if the MIDX is bitmapped and one or more of\n+\t\t * its selected commits reaches a once-cruft object that\n+\t\t * was later made reachable.\n \t\t */\n-\t\tif (pack_is_marked_for_deletion(item))\n-\t\t\tcontinue;\n+\t\tfor_each_string_list_item(item, &existing->cruft_packs) {\n+\t\t\t/*\n+\t\t\t * When doing a --geometric repack, there is no\n+\t\t\t * need to check for deleted packs, since we're\n+\t\t\t * by definition not doing an ALL_INTO_ONE\n+\t\t\t * repack (hence no packs will be deleted).\n+\t\t\t * Otherwise we must check for and exclude any\n+\t\t\t * packs which are enqueued for deletion.\n+\t\t\t *\n+\t\t\t * So we could omit the conditional below in the\n+\t\t\t * --geometric case, but doing so is unnecessary\n+\t\t\t * since no packs are marked as pending\n+\t\t\t * deletion (since we only call\n+\t\t\t * `mark_packs_for_deletion()` when doing an\n+\t\t\t * all-into-one repack).\n+\t\t\t */\n+\t\t\tif (pack_is_marked_for_deletion(item))\n+\t\t\t\tcontinue;\n \n-\t\tstrbuf_reset(&buf);\n-\t\tstrbuf_addf(&buf, \"%s.idx\", item->string);\n-\t\tstring_list_insert(include, buf.buf);\n+\t\t\tstrbuf_reset(&buf);\n+\t\t\tstrbuf_addf(&buf, \"%s.idx\", item->string);\n+\t\t\tstring_list_insert(include, buf.buf);\n+\t\t}\n+\t} else {\n+\t\t/*\n+\t\t * Modern versions of Git will write new copies of\n+\t\t * once-cruft objects when doing a --geometric repack.\n+\t\t *\n+\t\t * If the MIDX has no cruft pack, new packs written\n+\t\t * during a --geometric repack will not rely on the\n+\t\t * cruft pack to form a reachability closure, so we can\n+\t\t * avoid including them in the MIDX in that case.\n+\t\t */\n+\t\t;\n \t}\n \n \tstrbuf_release(&buf);\n@@ -1369,7 +1410,10 @@ int cmd_repack(int argc,\n \t\t !(pack_everything & PACK_CRUFT))\n \t\t\tstrvec_push(&cmd.args, \"--pack-loose-unreachable\");\n \t} else if (geometry.split_factor) {\n-\t\tstrvec_push(&cmd.args, \"--stdin-packs\");\n+\t\tif (existing_has_cruft_in_midx(&existing))\n+\t\t\tstrvec_push(&cmd.args, \"--stdin-packs\");\n+\t\telse\n+\t\t\tstrvec_push(&cmd.args, \"--stdin-packs=follow\");\n \t\tstrvec_push(&cmd.args, \"--unpacked\");\n \t} else {\n \t\tstrvec_push(&cmd.args, \"--unpacked\");\ndiff --git a/t/t7704-repack-cruft.sh b/t/t7704-repack-cruft.sh\nindex 8aebfb45f5..33ac58a3a5 100755\n--- a/t/t7704-repack-cruft.sh\n+++ b/t/t7704-repack-cruft.sh\n@@ -724,4 +724,74 @@ test_expect_success 'cruft repack respects --quiet' '\n \t)\n '\n \n+test_expect_success 'repack --write-midx excludes cruft where possible' '\n+\tgit init exclude-cruft-when-possible &&\n+\t(\n+\t\tcd exclude-cruft-when-possible &&\n+\n+\t\ttest_commit one &&\n+\n+\t\ttest_commit --no-tag two &&\n+\t\ttwo=\"$(git rev-parse HEAD)\" &&\n+\t\ttest_commit --no-tag three &&\n+\t\tthree=\"$(git rev-parse HEAD)\" &&\n+\t\tgit reset --hard one &&\n+\n+\t\tgit reflog expire --all --expire=all &&\n+\n+\t\tgit repack --cruft -d &&\n+\t\tls $packdir/pack-*.idx | sort >packs.before &&\n+\n+\t\tgit merge $two &&\n+\t\ttest_commit four &&\n+\t\tgit repack -d --geometric=2 --write-midx --write-bitmap-index &&\n+\t\tls $packdir/pack-*.idx | sort >packs.after &&\n+\n+\t\tcomm -13 packs.before packs.after >packs.new &&\n+\t\ttest_line_count = 1 packs.new &&\n+\n+\t\tgit rev-list --objects --no-object-names one..four >expect.raw &&\n+\t\tsort expect.raw >expect &&\n+\n+\t\tgit show-index <$(cat packs.new) >actual.raw &&\n+\t\tcut -d\" \" -f2 actual.raw | sort >actual &&\n+\n+\t\ttest_cmp expect actual &&\n+\n+\t\ttest-tool read-midx --show-objects $objdir >actual.raw &&\n+\t\tgrep \"\\.pack$\" actual.raw | cut -d\" \" -f1 | sort >actual.objects &&\n+\t\tgit rev-list --objects --no-object-names HEAD >expect.raw &&\n+\t\tsort expect.raw >expect.objects &&\n+\n+\t\ttest_cmp expect.objects actual.objects &&\n+\n+\t\tcruft=\"$(basename $(ls $packdir/*.mtimes))\" &&\n+\t\tgrep \"^pack-\" actual.raw >actual.packs &&\n+\t\t! test_grep \"${cruft%.mtimes}.idx\" actual.packs\n+\t)\n+'\n+\n+test_expect_success 'repack --write-midx includes cruft when necessary' '\n+\t(\n+\t\tcd exclude-cruft-when-possible &&\n+\n+\t\tls $packdir/pack-*.idx | sort >packs.all &&\n+\t\tgrep -o \"pack-.*\\.idx$\" packs.all >in &&\n+\n+\t\tgit multi-pack-index write --stdin-packs --bitmap actual.raw &&\n+\t\tgrep \"\\.pack$\" actual.raw | cut -d\" \" -f1 | sort >actual.objects &&\n+\t\tgit cat-file --batch-all-objects --batch-check=\"%(objectname)\" \\\n+\t\t\t>expect.objects &&\n+\t\ttest_cmp expect.objects actual.objects &&\n+\n+\t\tgrep \"^pack-\" actual.raw >actual.packs &&\n+\t\ttest_line_count = \"$(($(wc -l 2: 920c91eb1e object-store-ll.h: add note about designated initializers\n2: 6357633f6d = 3: f8ac36b110 pack-objects: limit scope in 'add_object_entry_from_pack()'\n3: 43e889b157 = 4: 5e03b482ba pack-objects: factor out handling '--stdin-packs'\n4: 07a91be3ec = 5: bccbac2ec5 pack-objects: declare 'rev_info' for '--stdin-packs' earlier\n5: 241f7c87e5 = 6: 0bc2183dc3 pack-objects: perform name-hash traversal for unpacked objects\n6: a0318321ec = 7: 697a337cb1 pack-objects: introduce '--stdin-packs=follow'\n7: ef0bc38cf0 < -: ---------- repack: keep track of existing MIDX'd packs\n8: 19b69c1246 ! 8: a2ec1b826c repack: exclude cruft pack(s) from the MIDX where possible\n @@ Commit message\n Note that you cannot guarantee that a collection of packs is closed\n under reachability if not all of them were generated with following as\n above. One tell-tale sign that not all geometrically-repacked packs in\n - the MIDX were generated with following is to see if there is a cruft\n - pack already in the MIDX.\n + the MIDX were generated with following is to see if there is a pack in\n + the existing MIDX that is not going to be somehow represented (either\n + verbatim or as part of a geometric rollup) in the new MIDX.\n \n If there is, then starting to generate packs with following during\n geometric repacking won't work, since it's open to the same race as\n @@ Commit message\n under reachability.\n \n Detect when this is the case and avoid including cruft packs in the MIDX\n - where possible.\n + where possible. The existing behavior remains the default, and the new\n + behavior is available with the config 'repack.midxMustIncludeCruft' set\n + to 'false'.\n \n Signed-off-by: Taylor Blau \n \n + ## Documentation/config/repack.adoc ##\n +@@ Documentation/config/repack.adoc: repack.cruftThreads::\n + \ta cruft pack and the respective parameters are not given over\n + \tthe command line. See similarly named `pack.*` configuration\n + \tvariables for defaults and meaning.\n ++\n ++repack.midxMustContainCruft::\n ++\tWhen set to true, linkgit:git-repack[1] will unconditionally include\n ++\tcruft pack(s), if any, in the multi-pack index when invoked with\n ++\t`--write-midx`. When false, cruft packs are only included in the MIDX\n ++\twhen necessary (e.g., because they might be required to form a\n ++\treachability closure with MIDX bitmaps). Defaults to true.\n +\n ## builtin/repack.c ##\n -@@ builtin/repack.c: static void pack_mark_in_midx(struct string_list_item *item)\n - \titem->util = (void*)((uintptr_t)item->util | PACK_IN_MIDX);\n +@@ builtin/repack.c: static int write_bitmaps = -1;\n + static int use_delta_islands;\n + static int run_update_server_info = 1;\n + static char *packdir, *packtmp_name, *packtmp;\n ++static int midx_must_contain_cruft = 1;\n + \n + static const char *const git_repack_usage[] = {\n + \tN_(\"git repack [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\\n\"\n +@@ builtin/repack.c: static int repack_config(const char *var, const char *value,\n + \t\tfree(cruft_po_args->threads);\n + \t\treturn git_config_string(&cruft_po_args->threads, var, value);\n + \t}\n ++\tif (!strcmp(var, \"repack.midxmustcontaincruft\")) {\n ++\t\tmidx_must_contain_cruft = git_config_bool(var, value);\n ++\t\treturn 0;\n ++\t}\n + \treturn git_default_config(var, value, ctx, cb);\n + }\n + \n +@@ builtin/repack.c: static void free_pack_geometry(struct pack_geometry *geometry)\n + \tfree(geometry->pack);\n }\n \n -+static int pack_is_in_midx(struct string_list_item *item)\n ++static int midx_has_unknown_packs(char **midx_pack_names,\n ++\t\t\t\t size_t midx_pack_names_nr,\n ++\t\t\t\t struct string_list *include,\n ++\t\t\t\t struct pack_geometry *geometry,\n ++\t\t\t\t struct existing_packs *existing)\n +{\n -+\treturn (uintptr_t)item->util & PACK_IN_MIDX;\n -+}\n ++\tsize_t i;\n +\n -+static int existing_has_cruft_in_midx(struct existing_packs *existing)\n -+{\n -+\tstruct string_list_item *item;\n -+\tfor_each_string_list_item(item, &existing->cruft_packs) {\n -+\t\tif (pack_is_in_midx(item))\n -+\t\t\treturn 1;\n ++\tstring_list_sort(include);\n ++\n ++\tfor (i = 0; i < midx_pack_names_nr; i++) {\n ++\t\tconst char *pack_name = midx_pack_names[i];\n ++\n ++\t\t/*\n ++\t\t * Determine whether or not each MIDX'd pack from the existing\n ++\t\t * MIDX (if any) is represented in the new MIDX. For each pack\n ++\t\t * in the MIDX, it must either be:\n ++\t\t *\n ++\t\t * - In the \"include\" list of packs to be included in the new\n ++\t\t * MIDX. Note this function is called before the include\n ++\t\t * list is populated with any cruft pack(s).\n ++\t\t *\n ++\t\t * - Below the geometric split line (if using pack geometry),\n ++\t\t * indicating that the pack won't be included in the new\n ++\t\t * MIDX, but its contents were rolled up as part of the\n ++\t\t * geometric repack.\n ++\t\t *\n ++\t\t * - In the existing non-kept packs list (if not using pack\n ++\t\t * geometry), and marked as non-deleted.\n ++\t\t */\n ++\t\tif (string_list_has_string(include, pack_name)) {\n ++\t\t\tcontinue;\n ++\t\t} else if (geometry) {\n ++\t\t\tstruct strbuf buf = STRBUF_INIT;\n ++\t\t\tuint32_t j;\n ++\n ++\t\t\tfor (j = 0; j < geometry->split; j++) {\n ++\t\t\t\tstrbuf_reset(&buf);\n ++\t\t\t\tstrbuf_addstr(&buf, pack_basename(geometry->pack[j]));\n ++\t\t\t\tstrbuf_strip_suffix(&buf, \".pack\");\n ++\t\t\t\tstrbuf_addstr(&buf, \".idx\");\n ++\n ++\t\t\t\tif (!strcmp(pack_name, buf.buf)) {\n ++\t\t\t\t\tstrbuf_release(&buf);\n ++\t\t\t\t\tbreak;\n ++\t\t\t\t}\n ++\t\t\t}\n ++\n ++\t\t\tstrbuf_release(&buf);\n ++\n ++\t\t\tif (j < geometry->split)\n ++\t\t\t\tcontinue;\n ++\t\t} else {\n ++\t\t\tstruct string_list_item *item;\n ++\n ++\t\t\titem = string_list_lookup(&existing->non_kept_packs,\n ++\t\t\t\t\t\t pack_name);\n ++\t\t\tif (item && !pack_is_marked_for_deletion(item))\n ++\t\t\t\tcontinue;\n ++\t\t}\n ++\n ++\t\t/*\n ++\t\t * If we got to this point, the MIDX includes some pack that we\n ++\t\t * don't know about.\n ++\t\t */\n ++\t\treturn 1;\n +\t}\n ++\n +\treturn 0;\n +}\n +\n - static void mark_packs_for_deletion_1(struct string_list *names,\n - \t\t\t\t struct string_list *list)\n + struct midx_snapshot_ref_data {\n + \tstruct tempfile *f;\n + \tstruct oidset seen;\n +@@ builtin/repack.c: static void midx_snapshot_refs(struct tempfile *f)\n + \n + static void midx_included_packs(struct string_list *include,\n + \t\t\t\tstruct existing_packs *existing,\n ++\t\t\t\tchar **midx_pack_names,\n ++\t\t\t\tsize_t midx_pack_names_nr,\n + \t\t\t\tstruct string_list *names,\n + \t\t\t\tstruct pack_geometry *geometry)\n {\n @@ builtin/repack.c: static void midx_included_packs(struct string_list *include,\n \t\t}\n \t}\n \n -\tfor_each_string_list_item(item, &existing->cruft_packs) {\n -+\tif (existing_has_cruft_in_midx(existing)) {\n ++\tif (midx_must_contain_cruft ||\n ++\t midx_has_unknown_packs(midx_pack_names, midx_pack_names_nr,\n ++\t\t\t\t include, geometry, existing)) {\n \t\t/*\n -\t\t * When doing a --geometric repack, there is no need to check\n -\t\t * for deleted packs, since we're by definition not doing an\n -\t\t * ALL_INTO_ONE repack (hence no packs will be deleted).\n -\t\t * Otherwise we must check for and exclude any packs which are\n -\t\t * enqueued for deletion.\n -+\t\t * If we had one or more cruft pack(s) present in the\n -+\t\t * MIDX before the repack, keep them as they may be\n -+\t\t * required to form a reachability closure if the MIDX\n -+\t\t * is bitmapped.\n ++\t\t * If there are one or more unknown pack(s) present (see\n ++\t\t * midx_has_unknown_packs() for what makes a pack\n ++\t\t * \"unknown\") in the MIDX before the repack, keep them\n ++\t\t * as they may be required to form a reachability\n ++\t\t * closure if the MIDX is bitmapped.\n \t\t *\n -\t\t * So we could omit the conditional below in the --geometric\n -\t\t * case, but doing so is unnecessary since no packs are marked\n -\t\t * as pending deletion (since we only call\n -\t\t * `mark_packs_for_deletion()` when doing an all-into-one\n -\t\t * repack).\n -+\t\t * A cruft pack can be required to form a reachability\n -+\t\t * closure if the MIDX is bitmapped and one or more of\n -+\t\t * its selected commits reaches a once-cruft object that\n -+\t\t * was later made reachable.\n ++\t\t * For example, a cruft pack can be required to form a\n ++\t\t * reachability closure if the MIDX is bitmapped and one\n ++\t\t * or more of its selected commits reaches a once-cruft\n ++\t\t * object that was later made reachable.\n \t\t */\n -\t\tif (pack_is_marked_for_deletion(item))\n -\t\t\tcontinue;\n @@ builtin/repack.c: static void midx_included_packs(struct string_list *include,\n \t}\n \n \tstrbuf_release(&buf);\n +@@ builtin/repack.c: int cmd_repack(int argc,\n + \tstruct tempfile *refs_snapshot = NULL;\n + \tint i, ext, ret;\n + \tint show_progress;\n ++\tchar **midx_pack_names = NULL;\n ++\tsize_t midx_pack_names_nr = 0;\n + \n + \t/* variables to be filled by option parsing */\n + \tint delete_redundant = 0;\n @@ builtin/repack.c: int cmd_repack(int argc,\n \t\t !(pack_everything & PACK_CRUFT))\n \t\t\tstrvec_push(&cmd.args, \"--pack-loose-unreachable\");\n \t} else if (geometry.split_factor) {\n -\t\tstrvec_push(&cmd.args, \"--stdin-packs\");\n -+\t\tif (existing_has_cruft_in_midx(&existing))\n ++\t\tif (midx_must_contain_cruft)\n +\t\t\tstrvec_push(&cmd.args, \"--stdin-packs\");\n +\t\telse\n +\t\t\tstrvec_push(&cmd.args, \"--stdin-packs=follow\");\n \t\tstrvec_push(&cmd.args, \"--unpacked\");\n \t} else {\n \t\tstrvec_push(&cmd.args, \"--unpacked\");\n +@@ builtin/repack.c: int cmd_repack(int argc,\n + \n + \tstring_list_sort(&names);\n + \n ++\tif (get_local_multi_pack_index(the_repository)) {\n ++\t\tuint32_t i;\n ++\t\tstruct multi_pack_index *m =\n ++\t\t\tget_local_multi_pack_index(the_repository);\n ++\n ++\t\tALLOC_ARRAY(midx_pack_names, m->num_packs);\n ++\t\tfor (i = 0; i < m->num_packs; i++)\n ++\t\t\tmidx_pack_names[midx_pack_names_nr++] = xstrdup(m->pack_names[i]);\n ++\t}\n ++\n + \tclose_object_store(the_repository->objects);\n + \n + \t/*\n +@@ builtin/repack.c: int cmd_repack(int argc,\n + \n + \tif (write_midx) {\n + \t\tstruct string_list include = STRING_LIST_INIT_DUP;\n +-\t\tmidx_included_packs(&include, &existing, &names, &geometry);\n ++\t\tmidx_included_packs(&include, &existing, midx_pack_names,\n ++\t\t\t\t midx_pack_names_nr, &names, &geometry);\n + \n + \t\tret = write_midx_included_packs(&include, &geometry, &names,\n + \t\t\t\t\t\trefs_snapshot ? get_tempfile_path(refs_snapshot) : NULL,\n +@@ builtin/repack.c: int cmd_repack(int argc,\n + \tstring_list_clear(&names, 1);\n + \texisting_packs_release(&existing);\n + \tfree_pack_geometry(&geometry);\n ++\tfor (size_t i = 0; i < midx_pack_names_nr; i++)\n ++\t\tfree(midx_pack_names[i]);\n ++\tfree(midx_pack_names);\n + \tpack_objects_args_release(&po_args);\n + \tpack_objects_args_release(&cruft_po_args);\n + \n \n ## t/t7704-repack-cruft.sh ##\n @@ t/t7704-repack-cruft.sh: test_expect_success 'cruft repack respects --quiet' '\n \t)\n '\n \n -+test_expect_success 'repack --write-midx excludes cruft where possible' '\n -+\tgit init exclude-cruft-when-possible &&\n ++setup_cruft_exclude_tests() {\n ++\tgit init \"$1\" &&\n +\t(\n -+\t\tcd exclude-cruft-when-possible &&\n ++\t\tcd \"$1\" &&\n ++\n ++\t\tgit config repack.midxMustContainCruft false &&\n +\n +\t\ttest_commit one &&\n +\n @@ t/t7704-repack-cruft.sh: test_expect_success 'cruft repack respects --quiet' '\n +\t\ttest_commit --no-tag three &&\n +\t\tthree=\"$(git rev-parse HEAD)\" &&\n +\t\tgit reset --hard one &&\n -+\n +\t\tgit reflog expire --all --expire=all &&\n +\n -+\t\tgit repack --cruft -d &&\n -+\t\tls $packdir/pack-*.idx | sort >packs.before &&\n ++\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack --cruft -d &&\n +\n +\t\tgit merge $two &&\n -+\t\ttest_commit four &&\n ++\t\ttest_commit four\n ++\t)\n ++}\n ++\n ++test_expect_success 'repack --write-midx excludes cruft where possible' '\n ++\tsetup_cruft_exclude_tests exclude-cruft-when-possible &&\n ++\t(\n ++\t\tcd exclude-cruft-when-possible &&\n ++\n ++\t\tGIT_TEST_MULTI_PACK_INDEX=0 \\\n +\t\tgit repack -d --geometric=2 --write-midx --write-bitmap-index &&\n -+\t\tls $packdir/pack-*.idx | sort >packs.after &&\n +\n -+\t\tcomm -13 packs.before packs.after >packs.new &&\n -+\t\ttest_line_count = 1 packs.new &&\n ++\t\ttest-tool read-midx --show-objects $objdir >midx &&\n ++\t\tcruft=\"$(ls $packdir/*.mtimes)\" &&\n ++\t\ttest_grep ! \"$(basename \"$cruft\" .mtimes).idx\" midx &&\n +\n -+\t\tgit rev-list --objects --no-object-names one..four >expect.raw &&\n -+\t\tsort expect.raw >expect &&\n ++\t\tgit rev-list --all --objects --no-object-names >reachable.raw &&\n ++\t\tsort reachable.raw >reachable.objects &&\n ++\t\tawk \"/\\.pack$/ { print \\$1 }\" midx.objects &&\n +\n -+\t\tgit show-index <$(cat packs.new) >actual.raw &&\n -+\t\tcut -d\" \" -f2 actual.raw | sort >actual &&\n ++\t\ttest_cmp reachable.objects midx.objects\n ++\t)\n ++'\n +\n -+\t\ttest_cmp expect actual &&\n ++test_expect_success 'repack --write-midx includes cruft when instructed' '\n ++\tsetup_cruft_exclude_tests exclude-cruft-when-instructed &&\n ++\t(\n ++\t\tcd exclude-cruft-when-instructed &&\n +\n -+\t\ttest-tool read-midx --show-objects $objdir >actual.raw &&\n -+\t\tgrep \"\\.pack$\" actual.raw | cut -d\" \" -f1 | sort >actual.objects &&\n -+\t\tgit rev-list --objects --no-object-names HEAD >expect.raw &&\n -+\t\tsort expect.raw >expect.objects &&\n ++\t\tGIT_TEST_MULTI_PACK_INDEX=0 \\\n ++\t\tgit -c repack.midxMustContainCruft=true repack \\\n ++\t\t\t-d --geometric=2 --write-midx --write-bitmap-index &&\n +\n -+\t\ttest_cmp expect.objects actual.objects &&\n ++\t\ttest-tool read-midx --show-objects $objdir >midx &&\n ++\t\tcruft=\"$(ls $packdir/*.mtimes)\" &&\n ++\t\ttest_grep \"$(basename \"$cruft\" .mtimes).idx\" midx &&\n +\n -+\t\tcruft=\"$(basename $(ls $packdir/*.mtimes))\" &&\n -+\t\tgrep \"^pack-\" actual.raw >actual.packs &&\n -+\t\t! test_grep \"${cruft%.mtimes}.idx\" actual.packs\n ++\t\tgit cat-file --batch-check=\"%(objectname)\" --batch-all-objects \\\n ++\t\t\t>all.objects &&\n ++\t\tawk \"/\\.pack$/ { print \\$1 }\" midx.objects &&\n ++\n ++\t\ttest_cmp all.objects midx.objects\n +\t)\n +'\n +\n +test_expect_success 'repack --write-midx includes cruft when necessary' '\n ++\tsetup_cruft_exclude_tests exclude-cruft-when-necessary &&\n +\t(\n -+\t\tcd exclude-cruft-when-possible &&\n ++\t\tcd exclude-cruft-when-necessary &&\n +\n ++\t\ttest_path_is_file $(ls $packdir/pack-*.mtimes) &&\n +\t\tls $packdir/pack-*.idx | sort >packs.all &&\n +\t\tgrep -o \"pack-.*\\.idx$\" packs.all >in &&\n +\n +\t\tgit multi-pack-index write --stdin-packs --bitmap actual.raw &&\n -+\t\tgrep \"\\.pack$\" actual.raw | cut -d\" \" -f1 | sort >actual.objects &&\n ++\t\ttest-tool read-midx --show-objects $objdir >midx &&\n ++\t\tawk \"/\\.pack$/ { print \\$1 }\" midx.objects &&\n +\t\tgit cat-file --batch-all-objects --batch-check=\"%(objectname)\" \\\n +\t\t\t>expect.objects &&\n -+\t\ttest_cmp expect.objects actual.objects &&\n ++\t\ttest_cmp expect.objects midx.objects &&\n +\n -+\t\tgrep \"^pack-\" actual.raw >actual.packs &&\n -+\t\ttest_line_count = \"$(($(wc -l midx.packs &&\n ++\t\ttest_line_count = \"$(($(wc -l \n---\n builtin/pack-objects.c | 19 ++++++++++---------\n t/t5331-pack-objects-stdin.sh | 2 +-\n 2 files changed, 11 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 6b06d159d2..aaea968ed2 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -4651,9 +4651,10 @@ int cmd_pack_objects(int argc,\n \t\tstrvec_push(&rp, \"--unpacked\");\n \t}\n \n-\tif (exclude_promisor_objects && exclude_promisor_objects_best_effort)\n-\t\tdie(_(\"options '%s' and '%s' cannot be used together\"),\n-\t\t \"--exclude-promisor-objects\", \"--exclude-promisor-objects-best-effort\");\n+\tdie_for_incompatible_opt2(exclude_promisor_objects,\n+\t\t\t\t \"--exclude-promisor-objects\",\n+\t\t\t\t exclude_promisor_objects_best_effort,\n+\t\t\t\t \"--exclude-promisor-objects-best-effort\");\n \tif (exclude_promisor_objects) {\n \t\tuse_internal_rev_list = 1;\n \t\tfetch_if_missing = 0;\n@@ -4691,13 +4692,13 @@ int cmd_pack_objects(int argc,\n \tif (!pack_to_stdout && thin)\n \t\tdie(_(\"--thin cannot be used to build an indexable pack\"));\n \n-\tif (keep_unreachable && unpack_unreachable)\n-\t\tdie(_(\"options '%s' and '%s' cannot be used together\"), \"--keep-unreachable\", \"--unpack-unreachable\");\n+\tdie_for_incompatible_opt2(keep_unreachable, \"--keep-unreachable\",\n+\t\t\t\t unpack_unreachable, \"--unpack-unreachable\");\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n-\tif (stdin_packs && filter_options.choice)\n-\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n+\tdie_for_incompatible_opt2(filter_options.choice, \"--filter\",\n+\t\t\t\t stdin_packs, \"--stdin-packs\");\n \n \tif (stdin_packs && use_internal_rev_list)\n \t\tdie(_(\"cannot use internal rev list with --stdin-packs\"));\n@@ -4705,8 +4706,8 @@ int cmd_pack_objects(int argc,\n \tif (cruft) {\n \t\tif (use_internal_rev_list)\n \t\t\tdie(_(\"cannot use internal rev list with --cruft\"));\n-\t\tif (stdin_packs)\n-\t\t\tdie(_(\"cannot use --stdin-packs with --cruft\"));\n+\t\tdie_for_incompatible_opt2(stdin_packs, \"--stdin-packs\",\n+\t\t\t\t\t cruft, \"--cruft\");\n \t}\n \n \t/*\ndiff --git a/t/t5331-pack-objects-stdin.sh b/t/t5331-pack-objects-stdin.sh\nindex b48c0cbe8f..4f5e2733a2 100755\n--- a/t/t5331-pack-objects-stdin.sh\n+++ b/t/t5331-pack-objects-stdin.sh\n@@ -64,7 +64,7 @@ test_expect_success '--stdin-packs is incompatible with --filter' '\n \t\tcd stdin-packs &&\n \t\ttest_must_fail git pack-objects --stdin-packs --stdout \\\n \t\t\t--filter=blob:none err &&\n-\t\ttest_grep \"cannot use --filter with --stdin-packs\" err\n+\t\ttest_grep \"options .--filter. and .--stdin-packs. cannot be used together\" err\n \t)\n '\n \n-- \n2.49.0.229.gc267761125.dirty\n\n"},{"id":"516131","messageId":"920c91eb1e5a1b6d5faa54240dd9c85f72968edc.1744661167.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744661167.git.me@ttaylorr.com","subject":"[PATCH v2 2/8] object-store-ll.h: add note about designated initializers","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-14T20:06:14Z","receivedAt":"2025-04-14T20:06:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The following commit will use a designated initializer to initialize a\n'struct object_info'. This obviously depends on having the rest of the\nfields having a default value of zero, since unspecified fields in a\ndesignated initializer are zero'd out.\n\nBefore writing that designated initializer, I wondered if there were\nother spots that also use designated initializers to set up object_info\nstructs, and there are a handful.\n\nTo prevent potential breakage against future object_info changes that\nwould introduce/change a field to have a non-zero default value, note\nthis dependency in a comment near the OBJECT_INFO_INIT macro.\n\nSigned-off-by: Taylor Blau \n---\n object-store-ll.h | 8 ++++++++\n 1 file changed, 8 insertions(+)\n\ndiff --git a/object-store-ll.h b/object-store-ll.h\nindex cd3bd5bd99..7ff180d7f2 100644\n--- a/object-store-ll.h\n+++ b/object-store-ll.h\n@@ -337,6 +337,14 @@ struct object_info {\n /*\n * Initializer for a \"struct object_info\" that wants no items. You may\n * also memset() the memory to all-zeroes.\n+ *\n+ * NOTE: callers expect the initial value of an object_info struct to\n+ * be zero'd out. Designated initializers like\n+ *\n+ * struct object_info oi = { .sizep = &sz };\n+ *\n+ * depend on this behavior, so consider strongly before adding new\n+ * fields that have a non-zero default value.\n */\n #define OBJECT_INFO_INIT { 0 }\n \n-- \n2.49.0.229.gc267761125.dirty\n\n"},{"id":"516132","messageId":"f8ac36b110b91ff3723f5049cfc6bf9f624521bd.1744661167.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744661167.git.me@ttaylorr.com","subject":"[PATCH v2 3/8] pack-objects: limit scope in 'add_object_entry_from_pack()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-14T20:06:17Z","receivedAt":"2025-04-14T20:06:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"add_object_entry_from_pack() handles objects from identified packs by\nchecking their type, before adding commit objects as pending in the\nsubsequent traversal used by `--stdin-packs`.\n\nThere are a couple of quality-of-life refactorings that I noticed while\nworking in this area:\n\n - We declare 'revs' (given to us through the miscellaneous context\n argument) earlier in the \"if (p)\" conditional than is necessary.\n\n - The 'struct object_info' can use a designated initializer to fill in\n the structures type pointer, since that is the only field that we\n care about.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 6 ++----\n 1 file changed, 2 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex aaea968ed2..540e5eba9e 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3490,14 +3490,12 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n \t\treturn 0;\n \n \tif (p) {\n-\t\tstruct rev_info *revs = _data;\n-\t\tstruct object_info oi = OBJECT_INFO_INIT;\n-\n-\t\toi.typep = &type;\n+\t\tstruct object_info oi = { .typep = &type };\n \t\tif (packed_object_info(the_repository, p, ofs, &oi) < 0) {\n \t\t\tdie(_(\"could not get type of object %s in pack %s\"),\n \t\t\t oid_to_hex(oid), p->pack_name);\n \t\t} else if (type == OBJ_COMMIT) {\n+\t\t\tstruct rev_info *revs = _data;\n \t\t\t/*\n \t\t\t * commits in included packs are used as starting points for the\n \t\t\t * subsequent revision walk\n-- \n2.49.0.229.gc267761125.dirty\n\n"},{"id":"516133","messageId":"5e03b482bad307e6638a85733a36d3e045abdea3.1744661167.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744661167.git.me@ttaylorr.com","subject":"[PATCH v2 4/8] pack-objects: factor out handling '--stdin-packs'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-14T20:06:21Z","receivedAt":"2025-04-14T20:06:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"At the bottom of cmd_pack_objects() we check which mode the command is\nrunning in (e.g., generating a cruft pack, handling '--stdin-packs',\nusing the internal rev-list, etc.) and handle the mode appropriately.\n\nThe '--stdin-packs' case is handled inline (dating back to its\nintroduction in 339bce27f4 (builtin/pack-objects.c: add '--stdin-packs'\noption, 2021-02-22)) since it is relatively short. Extract the body of\n\"if (stdin_packs)\" into its own function to prepare for the\nimplementation to become lengthier in a following commit.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 18 ++++++++++++------\n 1 file changed, 12 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 540e5eba9e..793d245721 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3672,6 +3672,17 @@ static void read_packs_list_from_stdin(void)\n \tstring_list_clear(&exclude_packs, 0);\n }\n \n+static void add_unreachable_loose_objects(void);\n+\n+static void read_stdin_packs(int rev_list_unpacked)\n+{\n+\t/* avoids adding objects in excluded packs */\n+\tignore_packed_keep_in_core = 1;\n+\tread_packs_list_from_stdin();\n+\tif (rev_list_unpacked)\n+\t\tadd_unreachable_loose_objects();\n+}\n+\n static void add_cruft_object_entry(const struct object_id *oid, enum object_type type,\n \t\t\t\t struct packed_git *pack, off_t offset,\n \t\t\t\t const char *name, uint32_t mtime)\n@@ -3767,7 +3778,6 @@ static void mark_pack_kept_in_core(struct string_list *packs, unsigned keep)\n \t}\n }\n \n-static void add_unreachable_loose_objects(void);\n static void add_objects_in_unpacked_packs(void);\n \n static void enumerate_cruft_objects(void)\n@@ -4773,11 +4783,7 @@ int cmd_pack_objects(int argc,\n \t\tprogress_state = start_progress(the_repository,\n \t\t\t\t\t\t_(\"Enumerating objects\"), 0);\n \tif (stdin_packs) {\n-\t\t/* avoids adding objects in excluded packs */\n-\t\tignore_packed_keep_in_core = 1;\n-\t\tread_packs_list_from_stdin();\n-\t\tif (rev_list_unpacked)\n-\t\t\tadd_unreachable_loose_objects();\n+\t\tread_stdin_packs(rev_list_unpacked);\n \t} else if (cruft) {\n \t\tread_cruft_objects();\n \t} else if (!use_internal_rev_list) {\n-- \n2.49.0.229.gc267761125.dirty\n\n"},{"id":"516134","messageId":"bccbac2ec5a14cc61a3a111e87c4db651d8dae5a.1744661167.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744661167.git.me@ttaylorr.com","subject":"[PATCH v2 5/8] pack-objects: declare 'rev_info' for '--stdin-packs' earlier","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-14T20:06:24Z","receivedAt":"2025-04-14T20:06:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Once 'read_packs_list_from_stdin()' has called for_each_object_in_pack()\non each of the input packs, we do a reachability traversal to discover\nnames for any objects we picked up so we can generate name hash values\nand hopefully get higher quality deltas as a result.\n\nA future commit will change the purpose of this reachability traversal\nto find and pack objects which are reachable from commits in the input\npacks, but are packed in an unknown (not included nor excluded) pack.\n\nExtract the code which initializes and performs the reachability\ntraversal to take place in the caller, not the callee, which prepares us\nto share this code for the '--unpacked' case (see the function\nadd_unreachable_loose_objects() for more details).\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 71 +++++++++++++++++++++---------------------\n 1 file changed, 36 insertions(+), 35 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 793d245721..1689cddd3a 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3556,7 +3556,7 @@ static int pack_mtime_cmp(const void *_a, const void *_b)\n \t\treturn 0;\n }\n \n-static void read_packs_list_from_stdin(void)\n+static void read_packs_list_from_stdin(struct rev_info *revs)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct string_list include_packs = STRING_LIST_INIT_DUP;\n@@ -3564,24 +3564,6 @@ static void read_packs_list_from_stdin(void)\n \tstruct string_list_item *item = NULL;\n \n \tstruct packed_git *p;\n-\tstruct rev_info revs;\n-\n-\trepo_init_revisions(the_repository, &revs, NULL);\n-\t/*\n-\t * Use a revision walk to fill in the namehash of objects in the include\n-\t * packs. To save time, we'll avoid traversing through objects that are\n-\t * in excluded packs.\n-\t *\n-\t * That may cause us to avoid populating all of the namehash fields of\n-\t * all included objects, but our goal is best-effort, since this is only\n-\t * an optimization during delta selection.\n-\t */\n-\trevs.no_kept_objects = 1;\n-\trevs.keep_pack_cache_flags |= IN_CORE_KEEP_PACKS;\n-\trevs.blob_objects = 1;\n-\trevs.tree_objects = 1;\n-\trevs.tag_objects = 1;\n-\trevs.ignore_missing_links = 1;\n \n \twhile (strbuf_getline(&buf, stdin) != EOF) {\n \t\tif (!buf.len)\n@@ -3651,10 +3633,44 @@ static void read_packs_list_from_stdin(void)\n \t\tstruct packed_git *p = item->util;\n \t\tfor_each_object_in_pack(p,\n \t\t\t\t\tadd_object_entry_from_pack,\n-\t\t\t\t\t&revs,\n+\t\t\t\t\trevs,\n \t\t\t\t\tFOR_EACH_OBJECT_PACK_ORDER);\n \t}\n \n+\tstrbuf_release(&buf);\n+\tstring_list_clear(&include_packs, 0);\n+\tstring_list_clear(&exclude_packs, 0);\n+}\n+\n+static void add_unreachable_loose_objects(void);\n+\n+static void read_stdin_packs(int rev_list_unpacked)\n+{\n+\tstruct rev_info revs;\n+\n+\trepo_init_revisions(the_repository, &revs, NULL);\n+\t/*\n+\t * Use a revision walk to fill in the namehash of objects in the include\n+\t * packs. To save time, we'll avoid traversing through objects that are\n+\t * in excluded packs.\n+\t *\n+\t * That may cause us to avoid populating all of the namehash fields of\n+\t * all included objects, but our goal is best-effort, since this is only\n+\t * an optimization during delta selection.\n+\t */\n+\trevs.no_kept_objects = 1;\n+\trevs.keep_pack_cache_flags |= IN_CORE_KEEP_PACKS;\n+\trevs.blob_objects = 1;\n+\trevs.tree_objects = 1;\n+\trevs.tag_objects = 1;\n+\trevs.ignore_missing_links = 1;\n+\n+\t/* avoids adding objects in excluded packs */\n+\tignore_packed_keep_in_core = 1;\n+\tread_packs_list_from_stdin(&revs);\n+\tif (rev_list_unpacked)\n+\t\tadd_unreachable_loose_objects();\n+\n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n \ttraverse_commit_list(&revs,\n@@ -3666,21 +3682,6 @@ static void read_packs_list_from_stdin(void)\n \t\t\t stdin_packs_found_nr);\n \ttrace2_data_intmax(\"pack-objects\", the_repository, \"stdin_packs_hints\",\n \t\t\t stdin_packs_hints_nr);\n-\n-\tstrbuf_release(&buf);\n-\tstring_list_clear(&include_packs, 0);\n-\tstring_list_clear(&exclude_packs, 0);\n-}\n-\n-static void add_unreachable_loose_objects(void);\n-\n-static void read_stdin_packs(int rev_list_unpacked)\n-{\n-\t/* avoids adding objects in excluded packs */\n-\tignore_packed_keep_in_core = 1;\n-\tread_packs_list_from_stdin();\n-\tif (rev_list_unpacked)\n-\t\tadd_unreachable_loose_objects();\n }\n \n static void add_cruft_object_entry(const struct object_id *oid, enum object_type type,\n-- \n2.49.0.229.gc267761125.dirty\n\n"},{"id":"516135","messageId":"0bc2183dc39a949d7eef08d282994231543d4c26.1744661167.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744661167.git.me@ttaylorr.com","subject":"[PATCH v2 6/8] pack-objects: perform name-hash traversal for unpacked objects","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-14T20:06:27Z","receivedAt":"2025-04-14T20:06:29Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"With '--unpacked', pack-objects adds loose objects (which don't appear\nin any of the excluded packs from '--stdin-packs') to the output pack\nwithout considering them as reachability tips for the name-hash\ntraversal.\n\nThis was an oversight in the original implementation of '--stdin-packs',\nsince the code which enumerates and adds loose objects to the output\npack (`add_unreachable_loose_objects()`) did not have access to the\n'rev_info' struct found in `read_packs_list_from_stdin()`.\n\nExcluding unpacked objects from that traversal doesn't effect the\ncorrectness of the resulting pack, but it does make it harder to\ndiscover good deltas for loose objects.\n\nNow that the 'rev_info' struct is declared outside of\n`read_packs_list_from_stdin()`, we can pass it to\n`add_objects_in_unpacked_packs()` and add any loose objects as tips to\nthe above-mentioned traversal, in theory producing slightly tighter\npacks as a result.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 20 ++++++++++++--------\n 1 file changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 1689cddd3a..2aa12da4af 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3642,7 +3642,7 @@ static void read_packs_list_from_stdin(struct rev_info *revs)\n \tstring_list_clear(&exclude_packs, 0);\n }\n \n-static void add_unreachable_loose_objects(void);\n+static void add_unreachable_loose_objects(struct rev_info *revs);\n \n static void read_stdin_packs(int rev_list_unpacked)\n {\n@@ -3669,7 +3669,7 @@ static void read_stdin_packs(int rev_list_unpacked)\n \tignore_packed_keep_in_core = 1;\n \tread_packs_list_from_stdin(&revs);\n \tif (rev_list_unpacked)\n-\t\tadd_unreachable_loose_objects();\n+\t\tadd_unreachable_loose_objects(&revs);\n \n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n@@ -3788,7 +3788,7 @@ static void enumerate_cruft_objects(void)\n \t\t\t\t\t\t_(\"Enumerating cruft objects\"), 0);\n \n \tadd_objects_in_unpacked_packs();\n-\tadd_unreachable_loose_objects();\n+\tadd_unreachable_loose_objects(NULL);\n \n \tstop_progress(&progress_state);\n }\n@@ -4066,8 +4066,9 @@ static void add_objects_in_unpacked_packs(void)\n }\n \n static int add_loose_object(const struct object_id *oid, const char *path,\n-\t\t\t void *data UNUSED)\n+\t\t\t void *data)\n {\n+\tstruct rev_info *revs = data;\n \tenum object_type type = oid_object_info(the_repository, oid, NULL);\n \n \tif (type < 0) {\n@@ -4088,6 +4089,10 @@ static int add_loose_object(const struct object_id *oid, const char *path,\n \t} else {\n \t\tadd_object_entry(oid, type, \"\", 0);\n \t}\n+\n+\tif (revs && type == OBJ_COMMIT)\n+\t\tadd_pending_oid(revs, NULL, oid, 0);\n+\n \treturn 0;\n }\n \n@@ -4096,11 +4101,10 @@ static int add_loose_object(const struct object_id *oid, const char *path,\n * add_object_entry will weed out duplicates, so we just add every\n * loose object we find.\n */\n-static void add_unreachable_loose_objects(void)\n+static void add_unreachable_loose_objects(struct rev_info *revs)\n {\n \tfor_each_loose_file_in_objdir(repo_get_object_directory(the_repository),\n-\t\t\t\t add_loose_object,\n-\t\t\t\t NULL, NULL, NULL);\n+\t\t\t\t add_loose_object, NULL, NULL, revs);\n }\n \n static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)\n@@ -4356,7 +4360,7 @@ static void get_object_list(struct rev_info *revs, int ac, const char **av)\n \tif (keep_unreachable)\n \t\tadd_objects_in_unpacked_packs();\n \tif (pack_loose_unreachable)\n-\t\tadd_unreachable_loose_objects();\n+\t\tadd_unreachable_loose_objects(NULL);\n \tif (unpack_unreachable)\n \t\tloosen_unused_packed_objects();\n \n-- \n2.49.0.229.gc267761125.dirty\n\n"},{"id":"516136","messageId":"697a337cb147ed638884cf9b8605fef8b572e1c6.1744661167.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744661167.git.me@ttaylorr.com","subject":"[PATCH v2 7/8] pack-objects: introduce '--stdin-packs=follow'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-14T20:06:30Z","receivedAt":"2025-04-14T20:06:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When invoked with '--stdin-packs', pack-objects will generate a pack\nwhich contains the objects found in the \"included\" packs, less any\nobjects from \"excluded\" packs.\n\nPacks that exist in the repository but weren't specified as either\nincluded or excluded are in practice treated like the latter, at least\nin the sense that pack-objects won't include objects from those packs.\nThis behavior forces us to include any cruft pack(s) in a repository's\nmulti-pack index for the reasons described in ddee3703b3\n(builtin/repack.c: add cruft packs to MIDX during geometric repack,\n2022-05-20).\n\nThe full details are in ddee3703b3, but the gist is if you\nhave a once-unreachable object in a cruft pack which later becomes\nreachable via one or more commits in a pack generated with\n'--stdin-packs', you *have* to include that object in the MIDX via the\ncopy in the cruft pack, otherwise we cannot generate reachability\nbitmaps for any commits which reach that object.\n\nThis prepares us for new repacking behavior which will \"resurrect\"\nobjects found in cruft or otherwise unspecified packs when generating\nnew packs. In the context of geometric repacking, this may be used to\nmaintain a sequence of geometrically-repacked packs, the union of which\nis closed under reachability, even in the case described earlier.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-pack-objects.adoc | 8 ++-\n builtin/pack-objects.c | 89 +++++++++++++++++-------\n t/t5331-pack-objects-stdin.sh | 101 ++++++++++++++++++++++++++++\n 3 files changed, 171 insertions(+), 27 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.adoc b/Documentation/git-pack-objects.adoc\nindex 7f69ae4855..c894582799 100644\n--- a/Documentation/git-pack-objects.adoc\n+++ b/Documentation/git-pack-objects.adoc\n@@ -87,13 +87,19 @@ base-name::\n \treference was included in the resulting packfile. This\n \tcan be useful to send new tags to native Git clients.\n \n---stdin-packs::\n+--stdin-packs[=]::\n \tRead the basenames of packfiles (e.g., `pack-1234abcd.pack`)\n \tfrom the standard input, instead of object names or revision\n \targuments. The resulting pack contains all objects listed in the\n \tincluded packs (those not beginning with `^`), excluding any\n \tobjects listed in the excluded packs (beginning with `^`).\n +\n+When `mode` is \"follow\", pack objects which are reachable from objects\n+in the included packs, but appear in packs that are not listed.\n+Reachable objects which appear in excluded packs are not packed. Useful\n+for resurrecting once-cruft objects to generate packs which are closed\n+under reachability up to the excluded packs.\n++\n Incompatible with `--revs`, or options that imply `--revs` (such as\n `--all`), with the exception of `--unpacked`, which is compatible.\n \ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 2aa12da4af..6406f4a5b1 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -272,6 +272,12 @@ static struct oidmap configured_exclusions;\n static struct oidset excluded_by_config;\n static int name_hash_version = -1;\n \n+enum stdin_packs_mode {\n+\tSTDIN_PACKS_MODE_NONE,\n+\tSTDIN_PACKS_MODE_STANDARD,\n+\tSTDIN_PACKS_MODE_FOLLOW,\n+};\n+\n /**\n * Check whether the name_hash_version chosen by user input is appropriate,\n * and also validate whether it is compatible with other features.\n@@ -3511,32 +3517,43 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n \treturn 0;\n }\n \n-static void show_commit_pack_hint(struct commit *commit UNUSED,\n-\t\t\t\t void *data UNUSED)\n-{\n-\t/* nothing to do; commits don't have a namehash */\n-}\n-\n static void show_object_pack_hint(struct object *object, const char *name,\n-\t\t\t\t void *data UNUSED)\n+\t\t\t\t void *data)\n {\n-\tstruct object_entry *oe = packlist_find(&to_pack, &object->oid);\n-\tif (!oe)\n+\tenum stdin_packs_mode mode = *(enum stdin_packs_mode *)data;\n+\tif (mode == STDIN_PACKS_MODE_FOLLOW) {\n+\t\tadd_object_entry(&object->oid, object->type, name, 0);\n+\t} else {\n+\t\tstruct object_entry *oe = packlist_find(&to_pack, &object->oid);\n+\t\tif (!oe)\n+\t\t\treturn;\n+\n+\t\t/*\n+\t\t * Our 'to_pack' list was constructed by iterating all\n+\t\t * objects packed in included packs, and so doesn't\n+\t\t * have a non-zero hash field that you would typically\n+\t\t * pick up during a reachability traversal.\n+\t\t *\n+\t\t * Make a best-effort attempt to fill in the ->hash\n+\t\t * and ->no_try_delta here using a now in order to\n+\t\t * perhaps improve the delta selection process.\n+\t\t */\n+\t\toe->hash = pack_name_hash_fn(name);\n+\t\toe->no_try_delta = name && no_try_delta(name);\n+\n+\t\tstdin_packs_hints_nr++;\n+\t}\n+}\n+\n+static void show_commit_pack_hint(struct commit *commit, void *data)\n+{\n+\tenum stdin_packs_mode mode = *(enum stdin_packs_mode *)data;\n+\tif (mode == STDIN_PACKS_MODE_FOLLOW) {\n+\t\tshow_object_pack_hint((struct object *)commit, \"\", data);\n \t\treturn;\n+\t}\n+\t/* nothing to do; commits don't have a namehash */\n \n-\t/*\n-\t * Our 'to_pack' list was constructed by iterating all objects packed in\n-\t * included packs, and so doesn't have a non-zero hash field that you\n-\t * would typically pick up during a reachability traversal.\n-\t *\n-\t * Make a best-effort attempt to fill in the ->hash and ->no_try_delta\n-\t * here using a now in order to perhaps improve the delta selection\n-\t * process.\n-\t */\n-\toe->hash = pack_name_hash_fn(name);\n-\toe->no_try_delta = name && no_try_delta(name);\n-\n-\tstdin_packs_hints_nr++;\n }\n \n static int pack_mtime_cmp(const void *_a, const void *_b)\n@@ -3644,7 +3661,7 @@ static void read_packs_list_from_stdin(struct rev_info *revs)\n \n static void add_unreachable_loose_objects(struct rev_info *revs);\n \n-static void read_stdin_packs(int rev_list_unpacked)\n+static void read_stdin_packs(enum stdin_packs_mode mode, int rev_list_unpacked)\n {\n \tstruct rev_info revs;\n \n@@ -3676,7 +3693,7 @@ static void read_stdin_packs(int rev_list_unpacked)\n \ttraverse_commit_list(&revs,\n \t\t\t show_commit_pack_hint,\n \t\t\t show_object_pack_hint,\n-\t\t\t NULL);\n+\t\t\t &mode);\n \n \ttrace2_data_intmax(\"pack-objects\", the_repository, \"stdin_packs_found\",\n \t\t\t stdin_packs_found_nr);\n@@ -4467,6 +4484,23 @@ static int is_not_in_promisor_pack(struct commit *commit, void *data) {\n \treturn is_not_in_promisor_pack_obj((struct object *) commit, data);\n }\n \n+static int parse_stdin_packs_mode(const struct option *opt, const char *arg,\n+\t\t\t\t int unset)\n+{\n+\tenum stdin_packs_mode *mode = opt->value;\n+\n+\tif (unset)\n+\t\t*mode = STDIN_PACKS_MODE_NONE;\n+\telse if (!arg || !*arg)\n+\t\t*mode = STDIN_PACKS_MODE_STANDARD;\n+\telse if (!strcmp(arg, \"follow\"))\n+\t\t*mode = STDIN_PACKS_MODE_FOLLOW;\n+\telse\n+\t\tdie(_(\"invalid value for '%s': '%s'\"), opt->long_name, arg);\n+\n+\treturn 0;\n+}\n+\n int cmd_pack_objects(int argc,\n \t\t const char **argv,\n \t\t const char *prefix,\n@@ -4478,7 +4512,7 @@ int cmd_pack_objects(int argc,\n \tstruct strvec rp = STRVEC_INIT;\n \tint rev_list_unpacked = 0, rev_list_all = 0, rev_list_reflog = 0;\n \tint rev_list_index = 0;\n-\tint stdin_packs = 0;\n+\tenum stdin_packs_mode stdin_packs = STDIN_PACKS_MODE_NONE;\n \tstruct string_list keep_pack_list = STRING_LIST_INIT_NODUP;\n \tstruct list_objects_filter_options filter_options =\n \t\tLIST_OBJECTS_FILTER_INIT;\n@@ -4533,6 +4567,9 @@ int cmd_pack_objects(int argc,\n \t\tOPT_SET_INT_F(0, \"indexed-objects\", &rev_list_index,\n \t\t\t N_(\"include objects referred to by the index\"),\n \t\t\t 1, PARSE_OPT_NONEG),\n+\t\tOPT_CALLBACK_F(0, \"stdin-packs\", &stdin_packs, N_(\"mode\"),\n+\t\t\t N_(\"read packs from stdin\"),\n+\t\t\t PARSE_OPT_OPTARG, parse_stdin_packs_mode),\n \t\tOPT_BOOL(0, \"stdin-packs\", &stdin_packs,\n \t\t\t N_(\"read packs from stdin\")),\n \t\tOPT_BOOL(0, \"stdout\", &pack_to_stdout,\n@@ -4788,7 +4825,7 @@ int cmd_pack_objects(int argc,\n \t\tprogress_state = start_progress(the_repository,\n \t\t\t\t\t\t_(\"Enumerating objects\"), 0);\n \tif (stdin_packs) {\n-\t\tread_stdin_packs(rev_list_unpacked);\n+\t\tread_stdin_packs(stdin_packs, rev_list_unpacked);\n \t} else if (cruft) {\n \t\tread_cruft_objects();\n \t} else if (!use_internal_rev_list) {\ndiff --git a/t/t5331-pack-objects-stdin.sh b/t/t5331-pack-objects-stdin.sh\nindex 4f5e2733a2..f97d2d1b71 100755\n--- a/t/t5331-pack-objects-stdin.sh\n+++ b/t/t5331-pack-objects-stdin.sh\n@@ -236,4 +236,105 @@ test_expect_success 'pack-objects --stdin with packfiles from main and alternate\n \ttest_cmp expected-objects actual-objects\n '\n \n+packdir=.git/objects/pack\n+\n+objects_in_packs () {\n+\tfor p in \"$@\"\n+\tdo\n+\t\tgit show-index <\"$packdir/pack-$p.idx\" || return 1\n+\tdone >objects.raw &&\n+\n+\tcut -d' ' -f2 objects.raw | sort &&\n+\trm -f objects.raw\n+}\n+\n+test_expect_success 'setup for --stdin-packs=follow' '\n+\tgit init stdin-packs--follow &&\n+\t(\n+\t\tcd stdin-packs--follow &&\n+\n+\t\tfor c in A B C D\n+\t\tdo\n+\t\t\ttest_commit \"$c\" || return 1\n+\t\tdone &&\n+\n+\t\tA=\"$(echo A | git pack-objects --revs $packdir/pack)\" &&\n+\t\tB=\"$(echo A..B | git pack-objects --revs $packdir/pack)\" &&\n+\t\tC=\"$(echo B..C | git pack-objects --revs $packdir/pack)\" &&\n+\t\tD=\"$(echo C..D | git pack-objects --revs $packdir/pack)\" &&\n+\n+\t\tgit prune-packed\n+\t)\n+'\n+\n+test_expect_success '--stdin-packs=follow walks into unknown packs' '\n+\ttest_when_finished \"rm -fr repo\" &&\n+\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\tfor c in A B C D\n+\t\tdo\n+\t\t\ttest_commit \"$c\" || return 1\n+\t\tdone &&\n+\n+\t\tA=\"$(echo A | git pack-objects --revs $packdir/pack)\" &&\n+\t\tB=\"$(echo A..B | git pack-objects --revs $packdir/pack)\" &&\n+\t\tC=\"$(echo B..C | git pack-objects --revs $packdir/pack)\" &&\n+\t\tD=\"$(echo C..D | git pack-objects --revs $packdir/pack)\" &&\n+\n+\t\tgit prune-packed &&\n+\n+\t\tcat >in <<-EOF &&\n+\t\tpack-$B.pack\n+\t\t^pack-$C.pack\n+\t\tpack-$D.pack\n+\t\tEOF\n+\n+\t\t# With just --stdin-packs, pack \"A\" is unknown to us, so\n+\t\t# only objects from packs \"B\" and \"D\" are included in\n+\t\t# the output pack.\n+\t\tP=$(git pack-objects --stdin-packs $packdir/pack expect &&\n+\t\tobjects_in_packs $P >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\t# But with --stdin-packs=follow, objects from both\n+\t\t# included packs reach objects from the unknown pack, so\n+\t\t# objects from pack \"A\" is included in the output pack\n+\t\t# in addition to the above.\n+\t\tP=$(git pack-objects --stdin-packs=follow $packdir/pack expect &&\n+\t\tobjects_in_packs $P >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\ttest_commit E &&\n+\t\t# And with --unpacked, we will pick up objects from unknown\n+\t\t# packs that are reachable from loose objects. Loose object E\n+\t\t# reaches objects in pack A, but there are three excluded packs\n+\t\t# in between.\n+\t\t#\n+\t\t# The resulting pack should include objects reachable from E\n+\t\t# that are not present in packs B, C, or D, along with those\n+\t\t# present in pack A.\n+\t\tcat >in <<-EOF &&\n+\t\t^pack-$B.pack\n+\t\t^pack-$C.pack\n+\t\t^pack-$D.pack\n+\t\tEOF\n+\n+\t\tP=$(git pack-objects --stdin-packs=follow --unpacked \\\n+\t\t\t$packdir/pack expect.raw &&\n+\t\tsort expect.raw >expect &&\n+\t\tobjects_in_packs $P >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \n2.49.0.229.gc267761125.dirty\n\n"},{"id":"516137","messageId":"a2ec1b826ceb88c33e2b91c6beef49c21ce36632.1744661167.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744661167.git.me@ttaylorr.com","subject":"[PATCH v2 8/8] repack: exclude cruft pack(s) from the MIDX where possible","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-14T20:06:33Z","receivedAt":"2025-04-14T20:06:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In ddee3703b3 (builtin/repack.c: add cruft packs to MIDX during\ngeometric repack, 2022-05-20), repack began adding cruft pack(s) to the\nMIDX with '--write-midx' to ensure that the resulting MIDX was always\nclosed under reachability in order to generate reachability bitmaps.\n\nSuppose you have a once-unreachable object packed in a cruft pack, which\nlater on becomes reachable from one or more objects in a geometrically\nrepacked pack. That once-unreachable object *won't* appear in the new\npack, since the cruft pack was specified as neither included nor\nexcluded to 'pack-objects --stdin-packs'. If the bitmap selection\nprocess picks one or more commits which reach the once-unreachable\nobjects, commit ddee3703b3 ensures that the MIDX will be closed under\nreachability. Without it, we would fail to generate a MIDX bitmap.\n\nddee3703b3 alludes to the fact that this is sub-optimal by saying\n\n [...] it's desirable to avoid including cruft packs in the MIDX\n because it causes the MIDX to store a bunch of objects which are\n likely to get thrown away.\n\n, which is true, but hides an even larger problem. If repositories\nrarely prune their unreachable objects and/or have many of them, the\nMIDX must keep track of a large number of objects which bloats the MIDX\nand slows down object lookup.\n\nThis is doubly unfortunate because the vast majority of objects in cruft\npack(s) are unlikely to be read, but object reads that go through the\nMIDX have to search through them anyway.\n\nThis patch causes geometrically-repacked packs to contain a copy of any\nonce-unreachable object(s) with 'git pack-objects --stdin-packs=follow',\nallowing us to avoid including any cruft packs in the MIDX. This is\nbecause a sequence of geometrically-repacked packs that were all\ngenerated with '--stdin-packs=follow' are guaranteed to have their union\nbe closed under reachability.\n\nNote that you cannot guarantee that a collection of packs is closed\nunder reachability if not all of them were generated with following as\nabove. One tell-tale sign that not all geometrically-repacked packs in\nthe MIDX were generated with following is to see if there is a pack in\nthe existing MIDX that is not going to be somehow represented (either\nverbatim or as part of a geometric rollup) in the new MIDX.\n\nIf there is, then starting to generate packs with following during\ngeometric repacking won't work, since it's open to the same race as\ndescribed above.\n\nBut if you're starting from scratch (e.g., building the first MIDX after\nan all-into-one '--cruft' repack), then you can guarantee that the union\nof subsequently generated packs from geometric repacking *is* closed\nunder reachability.\n\nDetect when this is the case and avoid including cruft packs in the MIDX\nwhere possible. The existing behavior remains the default, and the new\nbehavior is available with the config 'repack.midxMustIncludeCruft' set\nto 'false'.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/config/repack.adoc | 7 ++\n builtin/repack.c | 162 +++++++++++++++++++++++++++----\n t/t7704-repack-cruft.sh | 90 +++++++++++++++++\n 3 files changed, 241 insertions(+), 18 deletions(-)\n\ndiff --git a/Documentation/config/repack.adoc b/Documentation/config/repack.adoc\nindex c79af6d7b8..e9e78dcb19 100644\n--- a/Documentation/config/repack.adoc\n+++ b/Documentation/config/repack.adoc\n@@ -39,3 +39,10 @@ repack.cruftThreads::\n \ta cruft pack and the respective parameters are not given over\n \tthe command line. See similarly named `pack.*` configuration\n \tvariables for defaults and meaning.\n+\n+repack.midxMustContainCruft::\n+\tWhen set to true, linkgit:git-repack[1] will unconditionally include\n+\tcruft pack(s), if any, in the multi-pack index when invoked with\n+\t`--write-midx`. When false, cruft packs are only included in the MIDX\n+\twhen necessary (e.g., because they might be required to form a\n+\treachability closure with MIDX bitmaps). Defaults to true.\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex f3330ade7b..ee43a4f4c1 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -39,6 +39,7 @@ static int write_bitmaps = -1;\n static int use_delta_islands;\n static int run_update_server_info = 1;\n static char *packdir, *packtmp_name, *packtmp;\n+static int midx_must_contain_cruft = 1;\n \n static const char *const git_repack_usage[] = {\n \tN_(\"git repack [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\\n\"\n@@ -107,6 +108,10 @@ static int repack_config(const char *var, const char *value,\n \t\tfree(cruft_po_args->threads);\n \t\treturn git_config_string(&cruft_po_args->threads, var, value);\n \t}\n+\tif (!strcmp(var, \"repack.midxmustcontaincruft\")) {\n+\t\tmidx_must_contain_cruft = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n \treturn git_default_config(var, value, ctx, cb);\n }\n \n@@ -687,6 +692,77 @@ static void free_pack_geometry(struct pack_geometry *geometry)\n \tfree(geometry->pack);\n }\n \n+static int midx_has_unknown_packs(char **midx_pack_names,\n+\t\t\t\t size_t midx_pack_names_nr,\n+\t\t\t\t struct string_list *include,\n+\t\t\t\t struct pack_geometry *geometry,\n+\t\t\t\t struct existing_packs *existing)\n+{\n+\tsize_t i;\n+\n+\tstring_list_sort(include);\n+\n+\tfor (i = 0; i < midx_pack_names_nr; i++) {\n+\t\tconst char *pack_name = midx_pack_names[i];\n+\n+\t\t/*\n+\t\t * Determine whether or not each MIDX'd pack from the existing\n+\t\t * MIDX (if any) is represented in the new MIDX. For each pack\n+\t\t * in the MIDX, it must either be:\n+\t\t *\n+\t\t * - In the \"include\" list of packs to be included in the new\n+\t\t * MIDX. Note this function is called before the include\n+\t\t * list is populated with any cruft pack(s).\n+\t\t *\n+\t\t * - Below the geometric split line (if using pack geometry),\n+\t\t * indicating that the pack won't be included in the new\n+\t\t * MIDX, but its contents were rolled up as part of the\n+\t\t * geometric repack.\n+\t\t *\n+\t\t * - In the existing non-kept packs list (if not using pack\n+\t\t * geometry), and marked as non-deleted.\n+\t\t */\n+\t\tif (string_list_has_string(include, pack_name)) {\n+\t\t\tcontinue;\n+\t\t} else if (geometry) {\n+\t\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\t\tuint32_t j;\n+\n+\t\t\tfor (j = 0; j < geometry->split; j++) {\n+\t\t\t\tstrbuf_reset(&buf);\n+\t\t\t\tstrbuf_addstr(&buf, pack_basename(geometry->pack[j]));\n+\t\t\t\tstrbuf_strip_suffix(&buf, \".pack\");\n+\t\t\t\tstrbuf_addstr(&buf, \".idx\");\n+\n+\t\t\t\tif (!strcmp(pack_name, buf.buf)) {\n+\t\t\t\t\tstrbuf_release(&buf);\n+\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\tstrbuf_release(&buf);\n+\n+\t\t\tif (j < geometry->split)\n+\t\t\t\tcontinue;\n+\t\t} else {\n+\t\t\tstruct string_list_item *item;\n+\n+\t\t\titem = string_list_lookup(&existing->non_kept_packs,\n+\t\t\t\t\t\t pack_name);\n+\t\t\tif (item && !pack_is_marked_for_deletion(item))\n+\t\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/*\n+\t\t * If we got to this point, the MIDX includes some pack that we\n+\t\t * don't know about.\n+\t\t */\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n struct midx_snapshot_ref_data {\n \tstruct tempfile *f;\n \tstruct oidset seen;\n@@ -755,6 +831,8 @@ static void midx_snapshot_refs(struct tempfile *f)\n \n static void midx_included_packs(struct string_list *include,\n \t\t\t\tstruct existing_packs *existing,\n+\t\t\t\tchar **midx_pack_names,\n+\t\t\t\tsize_t midx_pack_names_nr,\n \t\t\t\tstruct string_list *names,\n \t\t\t\tstruct pack_geometry *geometry)\n {\n@@ -808,26 +886,55 @@ static void midx_included_packs(struct string_list *include,\n \t\t}\n \t}\n \n-\tfor_each_string_list_item(item, &existing->cruft_packs) {\n+\tif (midx_must_contain_cruft ||\n+\t midx_has_unknown_packs(midx_pack_names, midx_pack_names_nr,\n+\t\t\t\t include, geometry, existing)) {\n \t\t/*\n-\t\t * When doing a --geometric repack, there is no need to check\n-\t\t * for deleted packs, since we're by definition not doing an\n-\t\t * ALL_INTO_ONE repack (hence no packs will be deleted).\n-\t\t * Otherwise we must check for and exclude any packs which are\n-\t\t * enqueued for deletion.\n+\t\t * If there are one or more unknown pack(s) present (see\n+\t\t * midx_has_unknown_packs() for what makes a pack\n+\t\t * \"unknown\") in the MIDX before the repack, keep them\n+\t\t * as they may be required to form a reachability\n+\t\t * closure if the MIDX is bitmapped.\n \t\t *\n-\t\t * So we could omit the conditional below in the --geometric\n-\t\t * case, but doing so is unnecessary since no packs are marked\n-\t\t * as pending deletion (since we only call\n-\t\t * `mark_packs_for_deletion()` when doing an all-into-one\n-\t\t * repack).\n+\t\t * For example, a cruft pack can be required to form a\n+\t\t * reachability closure if the MIDX is bitmapped and one\n+\t\t * or more of its selected commits reaches a once-cruft\n+\t\t * object that was later made reachable.\n \t\t */\n-\t\tif (pack_is_marked_for_deletion(item))\n-\t\t\tcontinue;\n+\t\tfor_each_string_list_item(item, &existing->cruft_packs) {\n+\t\t\t/*\n+\t\t\t * When doing a --geometric repack, there is no\n+\t\t\t * need to check for deleted packs, since we're\n+\t\t\t * by definition not doing an ALL_INTO_ONE\n+\t\t\t * repack (hence no packs will be deleted).\n+\t\t\t * Otherwise we must check for and exclude any\n+\t\t\t * packs which are enqueued for deletion.\n+\t\t\t *\n+\t\t\t * So we could omit the conditional below in the\n+\t\t\t * --geometric case, but doing so is unnecessary\n+\t\t\t * since no packs are marked as pending\n+\t\t\t * deletion (since we only call\n+\t\t\t * `mark_packs_for_deletion()` when doing an\n+\t\t\t * all-into-one repack).\n+\t\t\t */\n+\t\t\tif (pack_is_marked_for_deletion(item))\n+\t\t\t\tcontinue;\n \n-\t\tstrbuf_reset(&buf);\n-\t\tstrbuf_addf(&buf, \"%s.idx\", item->string);\n-\t\tstring_list_insert(include, buf.buf);\n+\t\t\tstrbuf_reset(&buf);\n+\t\t\tstrbuf_addf(&buf, \"%s.idx\", item->string);\n+\t\t\tstring_list_insert(include, buf.buf);\n+\t\t}\n+\t} else {\n+\t\t/*\n+\t\t * Modern versions of Git will write new copies of\n+\t\t * once-cruft objects when doing a --geometric repack.\n+\t\t *\n+\t\t * If the MIDX has no cruft pack, new packs written\n+\t\t * during a --geometric repack will not rely on the\n+\t\t * cruft pack to form a reachability closure, so we can\n+\t\t * avoid including them in the MIDX in that case.\n+\t\t */\n+\t\t;\n \t}\n \n \tstrbuf_release(&buf);\n@@ -1142,6 +1249,8 @@ int cmd_repack(int argc,\n \tstruct tempfile *refs_snapshot = NULL;\n \tint i, ext, ret;\n \tint show_progress;\n+\tchar **midx_pack_names = NULL;\n+\tsize_t midx_pack_names_nr = 0;\n \n \t/* variables to be filled by option parsing */\n \tint delete_redundant = 0;\n@@ -1356,7 +1465,10 @@ int cmd_repack(int argc,\n \t\t !(pack_everything & PACK_CRUFT))\n \t\t\tstrvec_push(&cmd.args, \"--pack-loose-unreachable\");\n \t} else if (geometry.split_factor) {\n-\t\tstrvec_push(&cmd.args, \"--stdin-packs\");\n+\t\tif (midx_must_contain_cruft)\n+\t\t\tstrvec_push(&cmd.args, \"--stdin-packs\");\n+\t\telse\n+\t\t\tstrvec_push(&cmd.args, \"--stdin-packs=follow\");\n \t\tstrvec_push(&cmd.args, \"--unpacked\");\n \t} else {\n \t\tstrvec_push(&cmd.args, \"--unpacked\");\n@@ -1478,6 +1590,16 @@ int cmd_repack(int argc,\n \n \tstring_list_sort(&names);\n \n+\tif (get_local_multi_pack_index(the_repository)) {\n+\t\tuint32_t i;\n+\t\tstruct multi_pack_index *m =\n+\t\t\tget_local_multi_pack_index(the_repository);\n+\n+\t\tALLOC_ARRAY(midx_pack_names, m->num_packs);\n+\t\tfor (i = 0; i < m->num_packs; i++)\n+\t\t\tmidx_pack_names[midx_pack_names_nr++] = xstrdup(m->pack_names[i]);\n+\t}\n+\n \tclose_object_store(the_repository->objects);\n \n \t/*\n@@ -1519,7 +1641,8 @@ int cmd_repack(int argc,\n \n \tif (write_midx) {\n \t\tstruct string_list include = STRING_LIST_INIT_DUP;\n-\t\tmidx_included_packs(&include, &existing, &names, &geometry);\n+\t\tmidx_included_packs(&include, &existing, midx_pack_names,\n+\t\t\t\t midx_pack_names_nr, &names, &geometry);\n \n \t\tret = write_midx_included_packs(&include, &geometry, &names,\n \t\t\t\t\t\trefs_snapshot ? get_tempfile_path(refs_snapshot) : NULL,\n@@ -1570,6 +1693,9 @@ int cmd_repack(int argc,\n \tstring_list_clear(&names, 1);\n \texisting_packs_release(&existing);\n \tfree_pack_geometry(&geometry);\n+\tfor (size_t i = 0; i < midx_pack_names_nr; i++)\n+\t\tfree(midx_pack_names[i]);\n+\tfree(midx_pack_names);\n \tpack_objects_args_release(&po_args);\n \tpack_objects_args_release(&cruft_po_args);\n \ndiff --git a/t/t7704-repack-cruft.sh b/t/t7704-repack-cruft.sh\nindex 8aebfb45f5..2b0a55f8fd 100755\n--- a/t/t7704-repack-cruft.sh\n+++ b/t/t7704-repack-cruft.sh\n@@ -724,4 +724,94 @@ test_expect_success 'cruft repack respects --quiet' '\n \t)\n '\n \n+setup_cruft_exclude_tests() {\n+\tgit init \"$1\" &&\n+\t(\n+\t\tcd \"$1\" &&\n+\n+\t\tgit config repack.midxMustContainCruft false &&\n+\n+\t\ttest_commit one &&\n+\n+\t\ttest_commit --no-tag two &&\n+\t\ttwo=\"$(git rev-parse HEAD)\" &&\n+\t\ttest_commit --no-tag three &&\n+\t\tthree=\"$(git rev-parse HEAD)\" &&\n+\t\tgit reset --hard one &&\n+\t\tgit reflog expire --all --expire=all &&\n+\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack --cruft -d &&\n+\n+\t\tgit merge $two &&\n+\t\ttest_commit four\n+\t)\n+}\n+\n+test_expect_success 'repack --write-midx excludes cruft where possible' '\n+\tsetup_cruft_exclude_tests exclude-cruft-when-possible &&\n+\t(\n+\t\tcd exclude-cruft-when-possible &&\n+\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 \\\n+\t\tgit repack -d --geometric=2 --write-midx --write-bitmap-index &&\n+\n+\t\ttest-tool read-midx --show-objects $objdir >midx &&\n+\t\tcruft=\"$(ls $packdir/*.mtimes)\" &&\n+\t\ttest_grep ! \"$(basename \"$cruft\" .mtimes).idx\" midx &&\n+\n+\t\tgit rev-list --all --objects --no-object-names >reachable.raw &&\n+\t\tsort reachable.raw >reachable.objects &&\n+\t\tawk \"/\\.pack$/ { print \\$1 }\" midx.objects &&\n+\n+\t\ttest_cmp reachable.objects midx.objects\n+\t)\n+'\n+\n+test_expect_success 'repack --write-midx includes cruft when instructed' '\n+\tsetup_cruft_exclude_tests exclude-cruft-when-instructed &&\n+\t(\n+\t\tcd exclude-cruft-when-instructed &&\n+\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 \\\n+\t\tgit -c repack.midxMustContainCruft=true repack \\\n+\t\t\t-d --geometric=2 --write-midx --write-bitmap-index &&\n+\n+\t\ttest-tool read-midx --show-objects $objdir >midx &&\n+\t\tcruft=\"$(ls $packdir/*.mtimes)\" &&\n+\t\ttest_grep \"$(basename \"$cruft\" .mtimes).idx\" midx &&\n+\n+\t\tgit cat-file --batch-check=\"%(objectname)\" --batch-all-objects \\\n+\t\t\t>all.objects &&\n+\t\tawk \"/\\.pack$/ { print \\$1 }\" midx.objects &&\n+\n+\t\ttest_cmp all.objects midx.objects\n+\t)\n+'\n+\n+test_expect_success 'repack --write-midx includes cruft when necessary' '\n+\tsetup_cruft_exclude_tests exclude-cruft-when-necessary &&\n+\t(\n+\t\tcd exclude-cruft-when-necessary &&\n+\n+\t\ttest_path_is_file $(ls $packdir/pack-*.mtimes) &&\n+\t\tls $packdir/pack-*.idx | sort >packs.all &&\n+\t\tgrep -o \"pack-.*\\.idx$\" packs.all >in &&\n+\n+\t\tgit multi-pack-index write --stdin-packs --bitmap midx &&\n+\t\tawk \"/\\.pack$/ { print \\$1 }\" midx.objects &&\n+\t\tgit cat-file --batch-all-objects --batch-check=\"%(objectname)\" \\\n+\t\t\t>expect.objects &&\n+\t\ttest_cmp expect.objects midx.objects &&\n+\n+\t\tgrep \"^pack-\" midx >midx.packs &&\n+\t\ttest_line_count = \"$(($(wc -l writes:\n\n> pack-objects has a handful of explicit checks for pairs of command-line\n> options which are mutually incompatible. Many of these pre-date\n> a699367bb8 (i18n: factorize more 'incompatible options' messages,\n> 2022-01-31).\n>\n> Convert the explicit checks into die_for_incompatible_opt2() calls,\n> which simplifies the implementation and standardizes pack-objects'\n> output when given incompatible options (e.g., --stdin-packs with\n> --filter gives different output than --keep-unreachable with\n> --unpack-unreachable).\n\nMakes sense.\n\n> -\tif (stdin_packs && filter_options.choice)\n> -\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n> +\tdie_for_incompatible_opt2(filter_options.choice, \"--filter\",\n> +\t\t\t\t stdin_packs, \"--stdin-packs\");\n\nThe order of check is now reversed (which does not make any\ndifference to correctness or performance), but this way, we list the\noptions in the same order in the message as before, which is nice.\n\n> \t\ttest_must_fail git pack-objects --stdin-packs --stdout \\\n> \t\t\t--filter=blob:none err &&\n> -\t\ttest_grep \"cannot use --filter with --stdin-packs\" err\n> +\t\ttest_grep \"options .--filter. and .--stdin-packs. cannot be used together\" err\n> \t)\n> '\n"},{"id":"516142","messageId":"xmqqsemapi40.fsf@gitster.g","threadId":"63286","inReplyTo":"920c91eb1e5a1b6d5faa54240dd9c85f72968edc.1744661167.git.me@ttaylorr.com","subject":"Re: [PATCH v2 2/8] object-store-ll.h: add note about designated initializers","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-04-14T21:07:27Z","receivedAt":"2025-04-14T21:07:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> @@ -337,6 +337,14 @@ struct object_info {\n> /*\n> * Initializer for a \"struct object_info\" that wants no items. You may\n> * also memset() the memory to all-zeroes.\n> + *\n> + * NOTE: callers expect the initial value of an object_info struct to\n> + * be zero'd out. Designated initializers like\n> + *\n> + * struct object_info oi = { .sizep = &sz };\n> + *\n> + * depend on this behavior, so consider strongly before adding new\n> + * fields that have a non-zero default value.\n> */\n> #define OBJECT_INFO_INIT { 0 }\n\nHmph, after thinking hard enough, if a developer cannot come up with\na way to avoid non-zero default value, the callers could just work\nif they instead did\n\n\tstruct object_info oi = OBJECT_INFO_INIT;\n oi.sizep = &sz;\n\nand the member of non-zero default value can be delat with by\nupdating the default initializer, perhaps like\n\n\t#define OBJECT_INFO_INIT { .enabled = 1 }\n\nSo I am not sure how the advice in the new comment really helps the\nintended audiences. Shouldn't the advice be more like\n\n NOTE: when a structure foo has FOO_INIT macro to initialize,\n *never* use your own initialization like so:\n\n\tstruct foo foo_instance = { .member_i_care_about = 13 };\n\n Instead, use the _INIT macro and then assign to the member you\n care about, like so:\n\n\tstruct foo foo_instance = FOO_INIT;\n\tfoo_instance.member_i_care_about = 13;\n\n This is because there may be members of \"struct foo\" whose\n default value is not zero, or there will later be added such\n members to the structure.\n\nperhaps?\n\nI can buy a counter-proposal that does not forbid a custom\ndesignated initializers that depends on \"all zero\" default IF it\ngives a piece of advice to the readers that is more usable than\n\"consider strongly before adding\", as \"consider strongly before\nadding\" there smells like just an euphemism for \"never add\", though.\n\nThanks.\n\n\n\n\n"},{"id":"516154","messageId":"CABPp-BH7U4Vh8b6L9_FNUsBqKB+4hNT_Twn4S7LTocLvbw1LjA@mail.gmail.com","threadId":"63286","inReplyTo":"cover.1744661167.git.me@ttaylorr.com","subject":"Re: [PATCH v2 0/8] repack: avoid MIDX'ing cruft pack(s) where possible","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-04-15T02:57:52Z","receivedAt":"2025-04-15T02:58:04Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Apr 14, 2025 at 1:06 PM Taylor Blau wrote:\n>\n> Here is a non-RFC version of my series to explore creating MIDXs while\n> repacking that don't include the cruft pack.\n>\n> The core idea behind this approach is to ensure that packs generated via\n> geometric repacking traverse through objects that appear in packs which\n> are neither included nor excluded.\n\nThis phrasing feels confusing -- what does it mean for packs to be\nneither included nor excluded? Maybe:\n\n\"The core idea behind this approach is to allow some (most) of the\nobjects in a pack to be excluded, while still including some subset of\nobjects from that pack as part of the repack. In particular, we\ninclude the objects in that pack which are reachable from the other\nobjects we repack. This is different from our current handling which\neither entirely includes or entirely excludes all objects from a given\npack.\"\n\n> Then if some commit (for example) in\n> a pack reaches some once-unreachable object stored in a cruft pack, the\n> pack generated via geometric repacking will pick up and write a copy of\n> that object during its traversal.\n>\n> If you repack consistently using this strategy, you can guarantee that\n> the union of geometrically-repacked packs are closed under reachability\n> without having to keep track of any cruft pack(s) in the MIDX.\n\nAlso, if you do a single non-geometric repack with this strategy, you\nare also closed under reachability, right? Is that the suggested\ntransition plan for those that want to use this...first do a\nnon-geometric repack, and then ensure that subsequent geometric\nrepacks are done with this strategy?\n"},{"id":"516155","messageId":"CABPp-BGH=GRAo2Vy8sia75p=2Uoza0kwTzoy9FKKxE5jDFfGXg@mail.gmail.com","threadId":"63286","inReplyTo":"920c91eb1e5a1b6d5faa54240dd9c85f72968edc.1744661167.git.me@ttaylorr.com","subject":"Re: [PATCH v2 2/8] object-store-ll.h: add note about designated initializers","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-04-15T02:57:59Z","receivedAt":"2025-04-15T02:58:11Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Apr 14, 2025 at 1:06 PM Taylor Blau wrote:\n>\n> The following commit will use a designated initializer to initialize a\n> 'struct object_info'. This obviously depends on having the rest of the\n> fields having a default value of zero, since unspecified fields in a\n> designated initializer are zero'd out.\n>\n> Before writing that designated initializer, I wondered if there were\n> other spots that also use designated initializers to set up object_info\n> structs, and there are a handful.\n>\n> To prevent potential breakage against future object_info changes that\n> would introduce/change a field to have a non-zero default value, note\n> this dependency in a comment near the OBJECT_INFO_INIT macro.\n>\n> Signed-off-by: Taylor Blau \n> ---\n> object-store-ll.h | 8 ++++++++\n> 1 file changed, 8 insertions(+)\n>\n> diff --git a/object-store-ll.h b/object-store-ll.h\n> index cd3bd5bd99..7ff180d7f2 100644\n> --- a/object-store-ll.h\n> +++ b/object-store-ll.h\n> @@ -337,6 +337,14 @@ struct object_info {\n> /*\n> * Initializer for a \"struct object_info\" that wants no items. You may\n> * also memset() the memory to all-zeroes.\n> + *\n> + * NOTE: callers expect the initial value of an object_info struct to\n> + * be zero'd out. Designated initializers like\n> + *\n> + * struct object_info oi = { .sizep = &sz };\n> + *\n> + * depend on this behavior, so consider strongly before adding new\n> + * fields that have a non-zero default value.\n> */\n> #define OBJECT_INFO_INIT { 0 }\n\nThere are 46 #define'd designated initializers in the code base, from\nDIR_INIT to OIDMAP_INIT and everything in-between. The logic used in\nyour comment to suggest not using an all-zeroes initializer doesn't\nseem to depend in any way on something specific to object_info, yet\nnone of those other 46 cases in my quick scanning have such a warning.\nAnd 29 of the 46 define some kind of initial value for some fields\ninstead of using all zeroes. That would suggest that one of the\nfollowing is true: (a) those 29 cases are buggy and shouldn't be doing\nthat, (b) those 29 are all special cases someone has thought through\ncarefully but perhaps someone should add the same warning you have\nhere to those 29 other cases to avoid uncarefully thought cases from\nbeing added, (c) there is something specific about object_info that\nyou didn't call out here, or (d) this warning you add is unnecessary.\n"},{"id":"516156","messageId":"CABPp-BGDgMBE4zJqaeL=m=MwQQ4_1r=kdC69Dg7nErSuz3N7qw@mail.gmail.com","threadId":"63286","inReplyTo":"f8ac36b110b91ff3723f5049cfc6bf9f624521bd.1744661167.git.me@ttaylorr.com","subject":"Re: [PATCH v2 3/8] pack-objects: limit scope in 'add_object_entry_from_pack()'","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-04-15T03:10:37Z","receivedAt":"2025-04-15T03:10:48Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Apr 14, 2025 at 1:06 PM Taylor Blau wrote:\n>\n> add_object_entry_from_pack() handles objects from identified packs by\n> checking their type, before adding commit objects as pending in the\n> subsequent traversal used by `--stdin-packs`.\n>\n> There are a couple of quality-of-life refactorings that I noticed while\n> working in this area:\n>\n> - We declare 'revs' (given to us through the miscellaneous context\n> argument) earlier in the \"if (p)\" conditional than is necessary.\n\nFair enough.\n\n> - The 'struct object_info' can use a designated initializer to fill in\n> the structures type pointer, since that is the only field that we\n> care about.\n\nI prefer the original; it's more future-proof against someone making\nOBJECT_INFO_INIT be something other than an all-zero initializer.\n\n>\n> Signed-off-by: Taylor Blau \n> ---\n> builtin/pack-objects.c | 6 ++----\n> 1 file changed, 2 insertions(+), 4 deletions(-)\n>\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index aaea968ed2..540e5eba9e 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -3490,14 +3490,12 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n> return 0;\n>\n> if (p) {\n> - struct rev_info *revs = _data;\n> - struct object_info oi = OBJECT_INFO_INIT;\n> -\n> - oi.typep = &type;\n> + struct object_info oi = { .typep = &type };\n> if (packed_object_info(the_repository, p, ofs, &oi) < 0) {\n> die(_(\"could not get type of object %s in pack %s\"),\n> oid_to_hex(oid), p->pack_name);\n> } else if (type == OBJ_COMMIT) {\n> + struct rev_info *revs = _data;\n> /*\n> * commits in included packs are used as starting points for the\n> * subsequent revision walk\n> --\n> 2.49.0.229.gc267761125.dirty\n"},{"id":"516157","messageId":"CABPp-BG-XeOCtKd-LvUNGsBGyk6rLecm=BYbUDn-6rGwi=ROzg@mail.gmail.com","threadId":"63286","inReplyTo":"0bc2183dc39a949d7eef08d282994231543d4c26.1744661167.git.me@ttaylorr.com","subject":"Re: [PATCH v2 6/8] pack-objects: perform name-hash traversal for unpacked objects","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-04-15T03:10:51Z","receivedAt":"2025-04-15T03:11:03Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Apr 14, 2025 at 1:06 PM Taylor Blau wrote:\n>\n> With '--unpacked', pack-objects adds loose objects (which don't appear\n> in any of the excluded packs from '--stdin-packs') to the output pack\n> without considering them as reachability tips for the name-hash\n> traversal.\n>\n> This was an oversight in the original implementation of '--stdin-packs',\n> since the code which enumerates and adds loose objects to the output\n> pack (`add_unreachable_loose_objects()`) did not have access to the\n> 'rev_info' struct found in `read_packs_list_from_stdin()`.\n>\n> Excluding unpacked objects from that traversal doesn't effect the\n\ns/effect/affect/ ?\n\n> correctness of the resulting pack, but it does make it harder to\n> discover good deltas for loose objects.\n>\n> Now that the 'rev_info' struct is declared outside of\n> `read_packs_list_from_stdin()`, we can pass it to\n> `add_objects_in_unpacked_packs()` and add any loose objects as tips to\n> the above-mentioned traversal, in theory producing slightly tighter\n> packs as a result.\n>\n> Signed-off-by: Taylor Blau \n> ---\n> builtin/pack-objects.c | 20 ++++++++++++--------\n> 1 file changed, 12 insertions(+), 8 deletions(-)\n>\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 1689cddd3a..2aa12da4af 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -3642,7 +3642,7 @@ static void read_packs_list_from_stdin(struct rev_info *revs)\n> string_list_clear(&exclude_packs, 0);\n> }\n>\n> -static void add_unreachable_loose_objects(void);\n> +static void add_unreachable_loose_objects(struct rev_info *revs);\n>\n> static void read_stdin_packs(int rev_list_unpacked)\n> {\n> @@ -3669,7 +3669,7 @@ static void read_stdin_packs(int rev_list_unpacked)\n> ignore_packed_keep_in_core = 1;\n> read_packs_list_from_stdin(&revs);\n> if (rev_list_unpacked)\n> - add_unreachable_loose_objects();\n> + add_unreachable_loose_objects(&revs);\n>\n> if (prepare_revision_walk(&revs))\n> die(_(\"revision walk setup failed\"));\n> @@ -3788,7 +3788,7 @@ static void enumerate_cruft_objects(void)\n> _(\"Enumerating cruft objects\"), 0);\n>\n> add_objects_in_unpacked_packs();\n> - add_unreachable_loose_objects();\n> + add_unreachable_loose_objects(NULL);\n>\n> stop_progress(&progress_state);\n> }\n> @@ -4066,8 +4066,9 @@ static void add_objects_in_unpacked_packs(void)\n> }\n>\n> static int add_loose_object(const struct object_id *oid, const char *path,\n> - void *data UNUSED)\n> + void *data)\n> {\n> + struct rev_info *revs = data;\n> enum object_type type = oid_object_info(the_repository, oid, NULL);\n>\n> if (type < 0) {\n> @@ -4088,6 +4089,10 @@ static int add_loose_object(const struct object_id *oid, const char *path,\n> } else {\n> add_object_entry(oid, type, \"\", 0);\n> }\n> +\n> + if (revs && type == OBJ_COMMIT)\n> + add_pending_oid(revs, NULL, oid, 0);\n> +\n> return 0;\n> }\n>\n> @@ -4096,11 +4101,10 @@ static int add_loose_object(const struct object_id *oid, const char *path,\n> * add_object_entry will weed out duplicates, so we just add every\n> * loose object we find.\n> */\n> -static void add_unreachable_loose_objects(void)\n> +static void add_unreachable_loose_objects(struct rev_info *revs)\n> {\n> for_each_loose_file_in_objdir(repo_get_object_directory(the_repository),\n> - add_loose_object,\n> - NULL, NULL, NULL);\n> + add_loose_object, NULL, NULL, revs);\n> }\n>\n> static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)\n> @@ -4356,7 +4360,7 @@ static void get_object_list(struct rev_info *revs, int ac, const char **av)\n> if (keep_unreachable)\n> add_objects_in_unpacked_packs();\n> if (pack_loose_unreachable)\n> - add_unreachable_loose_objects();\n> + add_unreachable_loose_objects(NULL);\n> if (unpack_unreachable)\n> loosen_unused_packed_objects();\n>\n> --\n> 2.49.0.229.gc267761125.dirty\n\nShould this patch have some tests demonstrating the difference in\nwhich objects are included?\n"},{"id":"516158","messageId":"CABPp-BFBJP15g=4M90161=KCDei-hEFdnGs7_oY8ERtqgn9s-g@mail.gmail.com","threadId":"63286","inReplyTo":"697a337cb147ed638884cf9b8605fef8b572e1c6.1744661167.git.me@ttaylorr.com","subject":"Re: [PATCH v2 7/8] pack-objects: introduce '--stdin-packs=follow'","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-04-15T03:11:08Z","receivedAt":"2025-04-15T03:11:20Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Apr 14, 2025 at 1:06 PM Taylor Blau wrote:\n>\n> When invoked with '--stdin-packs', pack-objects will generate a pack\n> which contains the objects found in the \"included\" packs, less any\n> objects from \"excluded\" packs.\n>\n> Packs that exist in the repository but weren't specified as either\n> included or excluded are in practice treated like the latter, at least\n> in the sense that pack-objects won't include objects from those packs.\n> This behavior forces us to include any cruft pack(s) in a repository's\n> multi-pack index for the reasons described in ddee3703b3\n> (builtin/repack.c: add cruft packs to MIDX during geometric repack,\n> 2022-05-20).\n>\n> The full details are in ddee3703b3, but the gist is if you\n> have a once-unreachable object in a cruft pack which later becomes\n> reachable via one or more commits in a pack generated with\n> '--stdin-packs', you *have* to include that object in the MIDX via the\n> copy in the cruft pack, otherwise we cannot generate reachability\n> bitmaps for any commits which reach that object.\n>\n> This prepares us for new repacking behavior which will \"resurrect\"\n> objects found in cruft or otherwise unspecified packs when generating\n> new packs. In the context of geometric repacking, this may be used to\n> maintain a sequence of geometrically-repacked packs, the union of which\n> is closed under reachability, even in the case described earlier.\n>\n> Signed-off-by: Taylor Blau \n> ---\n> Documentation/git-pack-objects.adoc | 8 ++-\n> builtin/pack-objects.c | 89 +++++++++++++++++-------\n> t/t5331-pack-objects-stdin.sh | 101 ++++++++++++++++++++++++++++\n> 3 files changed, 171 insertions(+), 27 deletions(-)\n>\n> diff --git a/Documentation/git-pack-objects.adoc b/Documentation/git-pack-objects.adoc\n> index 7f69ae4855..c894582799 100644\n> --- a/Documentation/git-pack-objects.adoc\n> +++ b/Documentation/git-pack-objects.adoc\n> @@ -87,13 +87,19 @@ base-name::\n> reference was included in the resulting packfile. This\n> can be useful to send new tags to native Git clients.\n>\n> ---stdin-packs::\n> +--stdin-packs[=]::\n> Read the basenames of packfiles (e.g., `pack-1234abcd.pack`)\n> from the standard input, instead of object names or revision\n> arguments. The resulting pack contains all objects listed in the\n> included packs (those not beginning with `^`), excluding any\n> objects listed in the excluded packs (beginning with `^`).\n> +\n> +When `mode` is \"follow\", pack objects which are reachable from objects\n> +in the included packs, but appear in packs that are not listed.\n> +Reachable objects which appear in excluded packs are not packed. Useful\n> +for resurrecting once-cruft objects to generate packs which are closed\n> +under reachability up to the excluded packs.\n\nMaybe:\n\nWhen `mode` is \"follow\", objects from packs not listed on stdin\nreceive special treatment. Objects within unlisted packs will be\nincluded if those objects (1) are reachable from the included packs,\nand (2) are not also found in any of the excluded packs. This mode is\nuseful for resurrecting once-cruft objects to generate packs which are\nclosed under reachability up to the boundary set by the excluded\npacks.\n\n> ++\n> Incompatible with `--revs`, or options that imply `--revs` (such as\n> `--all`), with the exception of `--unpacked`, which is compatible.\n>\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 2aa12da4af..6406f4a5b1 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -272,6 +272,12 @@ static struct oidmap configured_exclusions;\n> static struct oidset excluded_by_config;\n> static int name_hash_version = -1;\n>\n> +enum stdin_packs_mode {\n> + STDIN_PACKS_MODE_NONE,\n> + STDIN_PACKS_MODE_STANDARD,\n> + STDIN_PACKS_MODE_FOLLOW,\n> +};\n> +\n> /**\n> * Check whether the name_hash_version chosen by user input is appropriate,\n> * and also validate whether it is compatible with other features.\n> @@ -3511,32 +3517,43 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n> return 0;\n> }\n>\n> -static void show_commit_pack_hint(struct commit *commit UNUSED,\n> - void *data UNUSED)\n> -{\n> - /* nothing to do; commits don't have a namehash */\n> -}\n> -\n> static void show_object_pack_hint(struct object *object, const char *name,\n> - void *data UNUSED)\n> + void *data)\n> {\n> - struct object_entry *oe = packlist_find(&to_pack, &object->oid);\n> - if (!oe)\n> + enum stdin_packs_mode mode = *(enum stdin_packs_mode *)data;\n> + if (mode == STDIN_PACKS_MODE_FOLLOW) {\n> + add_object_entry(&object->oid, object->type, name, 0);\n> + } else {\n> + struct object_entry *oe = packlist_find(&to_pack, &object->oid);\n> + if (!oe)\n> + return;\n> +\n> + /*\n> + * Our 'to_pack' list was constructed by iterating all\n> + * objects packed in included packs, and so doesn't\n> + * have a non-zero hash field that you would typically\n> + * pick up during a reachability traversal.\n> + *\n> + * Make a best-effort attempt to fill in the ->hash\n> + * and ->no_try_delta here using a now in order to\n> + * perhaps improve the delta selection process.\n> + */\n\nI know you just moved this paragraph from below...but it doesn't parse\nfor me. \"using a now in order to perhaps\"? What does that mean?\n\n> + oe->hash = pack_name_hash_fn(name);\n> + oe->no_try_delta = name && no_try_delta(name);\n> +\n> + stdin_packs_hints_nr++;\n> + }\n> +}\n> +\n> +static void show_commit_pack_hint(struct commit *commit, void *data)\n> +{\n> + enum stdin_packs_mode mode = *(enum stdin_packs_mode *)data;\n> + if (mode == STDIN_PACKS_MODE_FOLLOW) {\n> + show_object_pack_hint((struct object *)commit, \"\", data);\n> return;\n> + }\n> + /* nothing to do; commits don't have a namehash */\n>\n> - /*\n> - * Our 'to_pack' list was constructed by iterating all objects packed in\n> - * included packs, and so doesn't have a non-zero hash field that you\n> - * would typically pick up during a reachability traversal.\n> - *\n> - * Make a best-effort attempt to fill in the ->hash and ->no_try_delta\n> - * here using a now in order to perhaps improve the delta selection\n> - * process.\n> - */\n> - oe->hash = pack_name_hash_fn(name);\n> - oe->no_try_delta = name && no_try_delta(name);\n> -\n> - stdin_packs_hints_nr++;\n> }\n\nIt might be worth swapping the order of functions as a preparatory\npatch (both here and when you've done it elsewhere in this series),\njust because it'll make the diff so much easier to read when we can\nsee the changes to the function without have to also deal with the\norder swapping (since order swapping looks like a large deletion and\nlarge addition of one of the two functions).\n\n> static int pack_mtime_cmp(const void *_a, const void *_b)\n> @@ -3644,7 +3661,7 @@ static void read_packs_list_from_stdin(struct rev_info *revs)\n>\n> static void add_unreachable_loose_objects(struct rev_info *revs);\n>\n> -static void read_stdin_packs(int rev_list_unpacked)\n> +static void read_stdin_packs(enum stdin_packs_mode mode, int rev_list_unpacked)\n> {\n> struct rev_info revs;\n>\n> @@ -3676,7 +3693,7 @@ static void read_stdin_packs(int rev_list_unpacked)\n> traverse_commit_list(&revs,\n> show_commit_pack_hint,\n> show_object_pack_hint,\n> - NULL);\n> + &mode);\n>\n> trace2_data_intmax(\"pack-objects\", the_repository, \"stdin_packs_found\",\n> stdin_packs_found_nr);\n> @@ -4467,6 +4484,23 @@ static int is_not_in_promisor_pack(struct commit *commit, void *data) {\n> return is_not_in_promisor_pack_obj((struct object *) commit, data);\n> }\n>\n> +static int parse_stdin_packs_mode(const struct option *opt, const char *arg,\n> + int unset)\n> +{\n> + enum stdin_packs_mode *mode = opt->value;\n> +\n> + if (unset)\n> + *mode = STDIN_PACKS_MODE_NONE;\n> + else if (!arg || !*arg)\n> + *mode = STDIN_PACKS_MODE_STANDARD;\n\nI don't understand why you have both a None mode and a Standard mode,\nespecially since the implementation seems to only care about whether\nor not the Follow mode has been set. Shouldn't these both be setting\nmode to the same value?\n\n> + else if (!strcmp(arg, \"follow\"))\n> + *mode = STDIN_PACKS_MODE_FOLLOW;\n> + else\n> + die(_(\"invalid value for '%s': '%s'\"), opt->long_name, arg);\n> +\n> + return 0;\n> +}\n> +\n> int cmd_pack_objects(int argc,\n> const char **argv,\n> const char *prefix,\n> @@ -4478,7 +4512,7 @@ int cmd_pack_objects(int argc,\n> struct strvec rp = STRVEC_INIT;\n> int rev_list_unpacked = 0, rev_list_all = 0, rev_list_reflog = 0;\n> int rev_list_index = 0;\n> - int stdin_packs = 0;\n> + enum stdin_packs_mode stdin_packs = STDIN_PACKS_MODE_NONE;\n> struct string_list keep_pack_list = STRING_LIST_INIT_NODUP;\n> struct list_objects_filter_options filter_options =\n> LIST_OBJECTS_FILTER_INIT;\n> @@ -4533,6 +4567,9 @@ int cmd_pack_objects(int argc,\n> OPT_SET_INT_F(0, \"indexed-objects\", &rev_list_index,\n> N_(\"include objects referred to by the index\"),\n> 1, PARSE_OPT_NONEG),\n> + OPT_CALLBACK_F(0, \"stdin-packs\", &stdin_packs, N_(\"mode\"),\n> + N_(\"read packs from stdin\"),\n> + PARSE_OPT_OPTARG, parse_stdin_packs_mode),\n> OPT_BOOL(0, \"stdin-packs\", &stdin_packs,\n> N_(\"read packs from stdin\")),\n> OPT_BOOL(0, \"stdout\", &pack_to_stdout,\n> @@ -4788,7 +4825,7 @@ int cmd_pack_objects(int argc,\n> progress_state = start_progress(the_repository,\n> _(\"Enumerating objects\"), 0);\n> if (stdin_packs) {\n> - read_stdin_packs(rev_list_unpacked);\n> + read_stdin_packs(stdin_packs, rev_list_unpacked);\n> } else if (cruft) {\n> read_cruft_objects();\n> } else if (!use_internal_rev_list) {\n> diff --git a/t/t5331-pack-objects-stdin.sh b/t/t5331-pack-objects-stdin.sh\n> index 4f5e2733a2..f97d2d1b71 100755\n> --- a/t/t5331-pack-objects-stdin.sh\n> +++ b/t/t5331-pack-objects-stdin.sh\n> @@ -236,4 +236,105 @@ test_expect_success 'pack-objects --stdin with packfiles from main and alternate\n> test_cmp expected-objects actual-objects\n> '\n>\n> +packdir=.git/objects/pack\n> +\n> +objects_in_packs () {\n> + for p in \"$@\"\n> + do\n> + git show-index <\"$packdir/pack-$p.idx\" || return 1\n> + done >objects.raw &&\n> +\n> + cut -d' ' -f2 objects.raw | sort &&\n> + rm -f objects.raw\n> +}\n> +\n> +test_expect_success 'setup for --stdin-packs=follow' '\n> + git init stdin-packs--follow &&\n> + (\n> + cd stdin-packs--follow &&\n> +\n> + for c in A B C D\n> + do\n> + test_commit \"$c\" || return 1\n> + done &&\n> +\n> + A=\"$(echo A | git pack-objects --revs $packdir/pack)\" &&\n> + B=\"$(echo A..B | git pack-objects --revs $packdir/pack)\" &&\n> + C=\"$(echo B..C | git pack-objects --revs $packdir/pack)\" &&\n> + D=\"$(echo C..D | git pack-objects --revs $packdir/pack)\" &&\n> +\n> + git prune-packed\n> + )\n> +'\n> +\n> +test_expect_success '--stdin-packs=follow walks into unknown packs' '\n> + test_when_finished \"rm -fr repo\" &&\n> +\n> + git init repo &&\n> + (\n> + cd repo &&\n> +\n> + for c in A B C D\n> + do\n> + test_commit \"$c\" || return 1\n> + done &&\n> +\n> + A=\"$(echo A | git pack-objects --revs $packdir/pack)\" &&\n> + B=\"$(echo A..B | git pack-objects --revs $packdir/pack)\" &&\n> + C=\"$(echo B..C | git pack-objects --revs $packdir/pack)\" &&\n> + D=\"$(echo C..D | git pack-objects --revs $packdir/pack)\" &&\n> +\n> + git prune-packed &&\n> +\n> + cat >in <<-EOF &&\n> + pack-$B.pack\n> + ^pack-$C.pack\n> + pack-$D.pack\n> + EOF\n> +\n> + # With just --stdin-packs, pack \"A\" is unknown to us, so\n> + # only objects from packs \"B\" and \"D\" are included in\n> + # the output pack.\n> + P=$(git pack-objects --stdin-packs $packdir/pack + objects_in_packs $B $D >expect &&\n> + objects_in_packs $P >actual &&\n> + test_cmp expect actual &&\n> +\n> + # But with --stdin-packs=follow, objects from both\n> + # included packs reach objects from the unknown pack, so\n> + # objects from pack \"A\" is included in the output pack\n> + # in addition to the above.\n> + P=$(git pack-objects --stdin-packs=follow $packdir/pack + objects_in_packs $A $B $D >expect &&\n> + objects_in_packs $P >actual &&\n> + test_cmp expect actual &&\n> +\n> + test_commit E &&\n> + # And with --unpacked, we will pick up objects from unknown\n> + # packs that are reachable from loose objects. Loose object E\n> + # reaches objects in pack A, but there are three excluded packs\n> + # in between.\n> + #\n> + # The resulting pack should include objects reachable from E\n> + # that are not present in packs B, C, or D, along with those\n> + # present in pack A.\n> + cat >in <<-EOF &&\n> + ^pack-$B.pack\n> + ^pack-$C.pack\n> + ^pack-$D.pack\n> + EOF\n> +\n> + P=$(git pack-objects --stdin-packs=follow --unpacked \\\n> + $packdir/pack +\n> + {\n> + objects_in_packs $A &&\n> + git rev-list --objects --no-object-names D..E\n> + }>expect.raw &&\n> + sort expect.raw >expect &&\n> + objects_in_packs $P >actual &&\n> + test_cmp expect actual\n> + )\n> +'\n> +\n> test_done\n> --\n> 2.49.0.229.gc267761125.dirty\n\nI like the tests -- normal --stdin-packs, then --stdin-packs=follow,\nthen --stdin-packs=follow + --unpacked.\n\nHowever, would it be worthwhile to create commit E immediately after\ncreating the packs?\n\nCurrently, the third test shows us that unpacked objects are included\nwhen --unpacked is passed. But the tests don't let us know whether\nthat flag is necessary, i.e. whether unpacked objects will just be\nincluded anyway. If you move the creation of commit E as a loose\nobject immediately after the pack creation and before all the tests,\nthen these same tests demonstrate this additional bit of information.\n"},{"id":"516159","messageId":"CABPp-BHwewWMp1CY4jAr=sWwxE9of2Q73nyXQ5wpOELJZQ1OPg@mail.gmail.com","threadId":"63286","inReplyTo":"a2ec1b826ceb88c33e2b91c6beef49c21ce36632.1744661167.git.me@ttaylorr.com","subject":"Re: [PATCH v2 8/8] repack: exclude cruft pack(s) from the MIDX where possible","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-04-15T03:11:22Z","receivedAt":"2025-04-15T03:11:34Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Apr 14, 2025 at 1:06 PM Taylor Blau wrote:\n>\n> In ddee3703b3 (builtin/repack.c: add cruft packs to MIDX during\n> geometric repack, 2022-05-20), repack began adding cruft pack(s) to the\n> MIDX with '--write-midx' to ensure that the resulting MIDX was always\n> closed under reachability in order to generate reachability bitmaps.\n>\n> Suppose you have a once-unreachable object packed in a cruft pack, which\n> later on becomes reachable from one or more objects in a geometrically\n> repacked pack. That once-unreachable object *won't* appear in the new\n> pack, since the cruft pack was specified as neither included nor\n> excluded to 'pack-objects --stdin-packs'.\n\nI believe you are talking about the state before your series (i.e.,\nthis is carrying on from the previous paragraph), but it reads as\nthough you are talking about the state after the first seven patches\nof this series. Some kind of connection wording to clarify would\nreally help here.\n\n> If the bitmap selection\n> process picks one or more commits which reach the once-unreachable\n> objects, commit ddee3703b3 ensures that the MIDX will be closed under\n> reachability. Without it, we would fail to generate a MIDX bitmap.\n\nAfter reading this part, I had to go back and re-read and figure out\nwhat point in time everything was referring to.\n\n> ddee3703b3 alludes to the fact that this is sub-optimal by saying\n>\n> [...] it's desirable to avoid including cruft packs in the MIDX\n> because it causes the MIDX to store a bunch of objects which are\n> likely to get thrown away.\n>\n> , which is true, but hides an even larger problem. If repositories\n> rarely prune their unreachable objects and/or have many of them, the\n> MIDX must keep track of a large number of objects which bloats the MIDX\n> and slows down object lookup.\n>\n> This is doubly unfortunate because the vast majority of objects in cruft\n> pack(s) are unlikely to be read, but object reads that go through the\n> MIDX have to search through them anyway.\n\n\"have to search through them\"? That could be read to suggest those\nindividual objects are read, rather than just traversed over. Maybe\n\"...unlikely to be read, so the enlarged MIDX is for mostly tracking\nknown-to-likely-be-irrelevant objects\", or something like that?\n\n> This patch causes geometrically-repacked packs to contain a copy of any\n> once-unreachable object(s) with 'git pack-objects --stdin-packs=follow',\n> allowing us to avoid including any cruft packs in the MIDX. This is\n> because a sequence of geometrically-repacked packs that were all\n> generated with '--stdin-packs=follow' are guaranteed to have their union\n> be closed under reachability.\n>\n> Note that you cannot guarantee that a collection of packs is closed\n> under reachability if not all of them were generated with following as\n\nmaybe: ...with \"follow\" as above. \"follow\" or \"following\" feels like\nit needs quotes so the reader understands its meant as the name of a\nmode, rather than a verb in the sentence.\n\n> above. One tell-tale sign that not all geometrically-repacked packs in\n> the MIDX were generated with following is to see if there is a pack in\n\nsame here with \"following\"...and below.\n\n> the existing MIDX that is not going to be somehow represented (either\n> verbatim or as part of a geometric rollup) in the new MIDX.\n>\n> If there is, then starting to generate packs with following during\n> geometric repacking won't work, since it's open to the same race as\n> described above.\n>\n> But if you're starting from scratch (e.g., building the first MIDX after\n> an all-into-one '--cruft' repack), then you can guarantee that the union\n> of subsequently generated packs from geometric repacking *is* closed\n> under reachability.\n>\n> Detect when this is the case and avoid including cruft packs in the MIDX\n> where possible. The existing behavior remains the default, and the new\n> behavior is available with the config 'repack.midxMustIncludeCruft' set\n> to 'false'.\n>\n> Signed-off-by: Taylor Blau \n> ---\n> Documentation/config/repack.adoc | 7 ++\n> builtin/repack.c | 162 +++++++++++++++++++++++++++----\n> t/t7704-repack-cruft.sh | 90 +++++++++++++++++\n> 3 files changed, 241 insertions(+), 18 deletions(-)\n>\n> diff --git a/Documentation/config/repack.adoc b/Documentation/config/repack.adoc\n> index c79af6d7b8..e9e78dcb19 100644\n> --- a/Documentation/config/repack.adoc\n> +++ b/Documentation/config/repack.adoc\n> @@ -39,3 +39,10 @@ repack.cruftThreads::\n> a cruft pack and the respective parameters are not given over\n> the command line. See similarly named `pack.*` configuration\n> variables for defaults and meaning.\n> +\n> +repack.midxMustContainCruft::\n> + When set to true, linkgit:git-repack[1] will unconditionally include\n> + cruft pack(s), if any, in the multi-pack index when invoked with\n> + `--write-midx`. When false, cruft packs are only included in the MIDX\n> + when necessary (e.g., because they might be required to form a\n> + reachability closure with MIDX bitmaps). Defaults to true.\n> diff --git a/builtin/repack.c b/builtin/repack.c\n> index f3330ade7b..ee43a4f4c1 100644\n> --- a/builtin/repack.c\n> +++ b/builtin/repack.c\n> @@ -39,6 +39,7 @@ static int write_bitmaps = -1;\n> static int use_delta_islands;\n> static int run_update_server_info = 1;\n> static char *packdir, *packtmp_name, *packtmp;\n> +static int midx_must_contain_cruft = 1;\n>\n> static const char *const git_repack_usage[] = {\n> N_(\"git repack [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\\n\"\n> @@ -107,6 +108,10 @@ static int repack_config(const char *var, const char *value,\n> free(cruft_po_args->threads);\n> return git_config_string(&cruft_po_args->threads, var, value);\n> }\n> + if (!strcmp(var, \"repack.midxmustcontaincruft\")) {\n> + midx_must_contain_cruft = git_config_bool(var, value);\n> + return 0;\n> + }\n> return git_default_config(var, value, ctx, cb);\n> }\n>\n> @@ -687,6 +692,77 @@ static void free_pack_geometry(struct pack_geometry *geometry)\n> free(geometry->pack);\n> }\n>\n> +static int midx_has_unknown_packs(char **midx_pack_names,\n> + size_t midx_pack_names_nr,\n> + struct string_list *include,\n> + struct pack_geometry *geometry,\n> + struct existing_packs *existing)\n> +{\n> + size_t i;\n> +\n> + string_list_sort(include);\n> +\n> + for (i = 0; i < midx_pack_names_nr; i++) {\n> + const char *pack_name = midx_pack_names[i];\n> +\n> + /*\n> + * Determine whether or not each MIDX'd pack from the existing\n> + * MIDX (if any) is represented in the new MIDX. For each pack\n> + * in the MIDX, it must either be:\n> + *\n> + * - In the \"include\" list of packs to be included in the new\n> + * MIDX. Note this function is called before the include\n> + * list is populated with any cruft pack(s).\n> + *\n> + * - Below the geometric split line (if using pack geometry),\n> + * indicating that the pack won't be included in the new\n> + * MIDX, but its contents were rolled up as part of the\n> + * geometric repack.\n> + *\n> + * - In the existing non-kept packs list (if not using pack\n> + * geometry), and marked as non-deleted.\n> + */\n> + if (string_list_has_string(include, pack_name)) {\n> + continue;\n> + } else if (geometry) {\n> + struct strbuf buf = STRBUF_INIT;\n> + uint32_t j;\n> +\n> + for (j = 0; j < geometry->split; j++) {\n> + strbuf_reset(&buf);\n> + strbuf_addstr(&buf, pack_basename(geometry->pack[j]));\n> + strbuf_strip_suffix(&buf, \".pack\");\n> + strbuf_addstr(&buf, \".idx\");\n> +\n> + if (!strcmp(pack_name, buf.buf)) {\n> + strbuf_release(&buf);\n> + break;\n> + }\n> + }\n> +\n> + strbuf_release(&buf);\n> +\n> + if (j < geometry->split)\n> + continue;\n> + } else {\n> + struct string_list_item *item;\n> +\n> + item = string_list_lookup(&existing->non_kept_packs,\n> + pack_name);\n> + if (item && !pack_is_marked_for_deletion(item))\n> + continue;\n> + }\n> +\n> + /*\n> + * If we got to this point, the MIDX includes some pack that we\n> + * don't know about.\n> + */\n> + return 1;\n> + }\n> +\n> + return 0;\n> +}\n> +\n> struct midx_snapshot_ref_data {\n> struct tempfile *f;\n> struct oidset seen;\n> @@ -755,6 +831,8 @@ static void midx_snapshot_refs(struct tempfile *f)\n>\n> static void midx_included_packs(struct string_list *include,\n> struct existing_packs *existing,\n> + char **midx_pack_names,\n> + size_t midx_pack_names_nr,\n> struct string_list *names,\n> struct pack_geometry *geometry)\n> {\n> @@ -808,26 +886,55 @@ static void midx_included_packs(struct string_list *include,\n> }\n> }\n>\n> - for_each_string_list_item(item, &existing->cruft_packs) {\n> + if (midx_must_contain_cruft ||\n> + midx_has_unknown_packs(midx_pack_names, midx_pack_names_nr,\n> + include, geometry, existing)) {\n> /*\n> - * When doing a --geometric repack, there is no need to check\n> - * for deleted packs, since we're by definition not doing an\n> - * ALL_INTO_ONE repack (hence no packs will be deleted).\n> - * Otherwise we must check for and exclude any packs which are\n> - * enqueued for deletion.\n> + * If there are one or more unknown pack(s) present (see\n> + * midx_has_unknown_packs() for what makes a pack\n> + * \"unknown\") in the MIDX before the repack, keep them\n> + * as they may be required to form a reachability\n> + * closure if the MIDX is bitmapped.\n> *\n> - * So we could omit the conditional below in the --geometric\n> - * case, but doing so is unnecessary since no packs are marked\n> - * as pending deletion (since we only call\n> - * `mark_packs_for_deletion()` when doing an all-into-one\n> - * repack).\n> + * For example, a cruft pack can be required to form a\n> + * reachability closure if the MIDX is bitmapped and one\n> + * or more of its selected commits reaches a once-cruft\n> + * object that was later made reachable.\n\nThe antecedent of \"its\" is unclear here; just spell it out to reduce\nhow much thinking the reader needs to do?\n\n> */\n> - if (pack_is_marked_for_deletion(item))\n> - continue;\n> + for_each_string_list_item(item, &existing->cruft_packs) {\n> + /*\n> + * When doing a --geometric repack, there is no\n> + * need to check for deleted packs, since we're\n> + * by definition not doing an ALL_INTO_ONE\n> + * repack (hence no packs will be deleted).\n> + * Otherwise we must check for and exclude any\n> + * packs which are enqueued for deletion.\n> + *\n> + * So we could omit the conditional below in the\n> + * --geometric case, but doing so is unnecessary\n> + * since no packs are marked as pending\n> + * deletion (since we only call\n> + * `mark_packs_for_deletion()` when doing an\n> + * all-into-one repack).\n> + */\n> + if (pack_is_marked_for_deletion(item))\n> + continue;\n>\n> - strbuf_reset(&buf);\n> - strbuf_addf(&buf, \"%s.idx\", item->string);\n> - string_list_insert(include, buf.buf);\n> + strbuf_reset(&buf);\n> + strbuf_addf(&buf, \"%s.idx\", item->string);\n> + string_list_insert(include, buf.buf);\n> + }\n> + } else {\n> + /*\n> + * Modern versions of Git will write new copies of\n> + * once-cruft objects when doing a --geometric repack.\n\n\"Modern versions of Git\" -> \"Modern versions of Git with the\nappropriate config setting\" ?\n\n\n> + *\n> + * If the MIDX has no cruft pack, new packs written\n> + * during a --geometric repack will not rely on the\n> + * cruft pack to form a reachability closure, so we can\n> + * avoid including them in the MIDX in that case.\n> + */\n> + ;\n> }\n>\n> strbuf_release(&buf);\n> @@ -1142,6 +1249,8 @@ int cmd_repack(int argc,\n> struct tempfile *refs_snapshot = NULL;\n> int i, ext, ret;\n> int show_progress;\n> + char **midx_pack_names = NULL;\n> + size_t midx_pack_names_nr = 0;\n>\n> /* variables to be filled by option parsing */\n> int delete_redundant = 0;\n> @@ -1356,7 +1465,10 @@ int cmd_repack(int argc,\n> !(pack_everything & PACK_CRUFT))\n> strvec_push(&cmd.args, \"--pack-loose-unreachable\");\n> } else if (geometry.split_factor) {\n> - strvec_push(&cmd.args, \"--stdin-packs\");\n> + if (midx_must_contain_cruft)\n> + strvec_push(&cmd.args, \"--stdin-packs\");\n> + else\n> + strvec_push(&cmd.args, \"--stdin-packs=follow\");\n> strvec_push(&cmd.args, \"--unpacked\");\n> } else {\n> strvec_push(&cmd.args, \"--unpacked\");\n> @@ -1478,6 +1590,16 @@ int cmd_repack(int argc,\n>\n> string_list_sort(&names);\n>\n> + if (get_local_multi_pack_index(the_repository)) {\n> + uint32_t i;\n> + struct multi_pack_index *m =\n> + get_local_multi_pack_index(the_repository);\n> +\n> + ALLOC_ARRAY(midx_pack_names, m->num_packs);\n> + for (i = 0; i < m->num_packs; i++)\n> + midx_pack_names[midx_pack_names_nr++] = xstrdup(m->pack_names[i]);\n> + }\n> +\n> close_object_store(the_repository->objects);\n>\n> /*\n> @@ -1519,7 +1641,8 @@ int cmd_repack(int argc,\n>\n> if (write_midx) {\n> struct string_list include = STRING_LIST_INIT_DUP;\n> - midx_included_packs(&include, &existing, &names, &geometry);\n> + midx_included_packs(&include, &existing, midx_pack_names,\n> + midx_pack_names_nr, &names, &geometry);\n>\n> ret = write_midx_included_packs(&include, &geometry, &names,\n> refs_snapshot ? get_tempfile_path(refs_snapshot) : NULL,\n> @@ -1570,6 +1693,9 @@ int cmd_repack(int argc,\n> string_list_clear(&names, 1);\n> existing_packs_release(&existing);\n> free_pack_geometry(&geometry);\n> + for (size_t i = 0; i < midx_pack_names_nr; i++)\n> + free(midx_pack_names[i]);\n> + free(midx_pack_names);\n> pack_objects_args_release(&po_args);\n> pack_objects_args_release(&cruft_po_args);\n>\n> diff --git a/t/t7704-repack-cruft.sh b/t/t7704-repack-cruft.sh\n> index 8aebfb45f5..2b0a55f8fd 100755\n> --- a/t/t7704-repack-cruft.sh\n> +++ b/t/t7704-repack-cruft.sh\n> @@ -724,4 +724,94 @@ test_expect_success 'cruft repack respects --quiet' '\n> )\n> '\n>\n> +setup_cruft_exclude_tests() {\n> + git init \"$1\" &&\n> + (\n> + cd \"$1\" &&\n> +\n> + git config repack.midxMustContainCruft false &&\n> +\n> + test_commit one &&\n> +\n> + test_commit --no-tag two &&\n> + two=\"$(git rev-parse HEAD)\" &&\n> + test_commit --no-tag three &&\n> + three=\"$(git rev-parse HEAD)\" &&\n> + git reset --hard one &&\n> + git reflog expire --all --expire=all &&\n> +\n> + GIT_TEST_MULTI_PACK_INDEX=0 git repack --cruft -d &&\n> +\n> + git merge $two &&\n> + test_commit four\n> + )\n> +}\n> +\n> +test_expect_success 'repack --write-midx excludes cruft where possible' '\n> + setup_cruft_exclude_tests exclude-cruft-when-possible &&\n> + (\n> + cd exclude-cruft-when-possible &&\n> +\n> + GIT_TEST_MULTI_PACK_INDEX=0 \\\n> + git repack -d --geometric=2 --write-midx --write-bitmap-index &&\n> +\n> + test-tool read-midx --show-objects $objdir >midx &&\n> + cruft=\"$(ls $packdir/*.mtimes)\" &&\n> + test_grep ! \"$(basename \"$cruft\" .mtimes).idx\" midx &&\n> +\n> + git rev-list --all --objects --no-object-names >reachable.raw &&\n> + sort reachable.raw >reachable.objects &&\n> + awk \"/\\.pack$/ { print \\$1 }\" midx.objects &&\n> +\n> + test_cmp reachable.objects midx.objects\n> + )\n> +'\n> +\n> +test_expect_success 'repack --write-midx includes cruft when instructed' '\n> + setup_cruft_exclude_tests exclude-cruft-when-instructed &&\n> + (\n> + cd exclude-cruft-when-instructed &&\n> +\n> + GIT_TEST_MULTI_PACK_INDEX=0 \\\n> + git -c repack.midxMustContainCruft=true repack \\\n> + -d --geometric=2 --write-midx --write-bitmap-index &&\n> +\n> + test-tool read-midx --show-objects $objdir >midx &&\n> + cruft=\"$(ls $packdir/*.mtimes)\" &&\n> + test_grep \"$(basename \"$cruft\" .mtimes).idx\" midx &&\n> +\n> + git cat-file --batch-check=\"%(objectname)\" --batch-all-objects \\\n> + >all.objects &&\n> + awk \"/\\.pack$/ { print \\$1 }\" midx.objects &&\n> +\n> + test_cmp all.objects midx.objects\n> + )\n> +'\n> +\n> +test_expect_success 'repack --write-midx includes cruft when necessary' '\n> + setup_cruft_exclude_tests exclude-cruft-when-necessary &&\n> + (\n> + cd exclude-cruft-when-necessary &&\n> +\n> + test_path_is_file $(ls $packdir/pack-*.mtimes) &&\n> + ls $packdir/pack-*.idx | sort >packs.all &&\n> + grep -o \"pack-.*\\.idx$\" packs.all >in &&\n> +\n> + git multi-pack-index write --stdin-packs --bitmap +\n> + test_commit five &&\n> + GIT_TEST_MULTI_PACK_INDEX=0 \\\n> + git repack -d --geometric=2 --write-midx --write-bitmap-index &&\n> +\n> + test-tool read-midx --show-objects $objdir >midx &&\n> + awk \"/\\.pack$/ { print \\$1 }\" midx.objects &&\n> + git cat-file --batch-all-objects --batch-check=\"%(objectname)\" \\\n> + >expect.objects &&\n> + test_cmp expect.objects midx.objects &&\n> +\n> + grep \"^pack-\" midx >midx.packs &&\n> + test_line_count = \"$(($(wc -l + )\n> +'\n> +\n> test_done\n> --\n> 2.49.0.229.gc267761125.dirty\n"},{"id":"516216","messageId":"Z/60anmKVxke0AVX@nand.local","threadId":"63286","inReplyTo":"xmqqwmbmpjbc.fsf@gitster.g","subject":"Re: [PATCH v2 1/8] pack-objects: use standard option incompatibility functions","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-15T19:32:58Z","receivedAt":"2025-04-15T19:33:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Apr 14, 2025 at 01:41:27PM -0700, Junio C Hamano wrote:\n> > -\tif (stdin_packs && filter_options.choice)\n> > -\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n> > +\tdie_for_incompatible_opt2(filter_options.choice, \"--filter\",\n> > +\t\t\t\t stdin_packs, \"--stdin-packs\");\n>\n> The order of check is now reversed (which does not make any\n> difference to correctness or performance), but this way, we list the\n> options in the same order in the message as before, which is nice.\n\nNow I can't un-see it ;-). Even though it's not a correctness issue as\nyou note, the whole thing leaves a bad taste in my mouth. I'll swap the\nordering to match the original in the next round.\n\nThanks,\nTaylor\n"},{"id":"516217","messageId":"Z/637cMz7AGVXokX@nand.local","threadId":"63286","inReplyTo":"CABPp-BGH=GRAo2Vy8sia75p=2Uoza0kwTzoy9FKKxE5jDFfGXg@mail.gmail.com","subject":"Re: [PATCH v2 2/8] object-store-ll.h: add note about designated initializers","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-15T19:47:57Z","receivedAt":"2025-04-15T19:47:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Apr 14, 2025 at 07:57:59PM -0700, Elijah Newren wrote:\n> There are 46 #define'd designated initializers in the code base, from\n> DIR_INIT to OIDMAP_INIT and everything in-between. The logic used in\n> your comment to suggest not using an all-zeroes initializer doesn't\n> seem to depend in any way on something specific to object_info, yet\n> none of those other 46 cases in my quick scanning have such a warning.\n> And 29 of the 46 define some kind of initial value for some fields\n> instead of using all zeroes. That would suggest that one of the\n> following is true: (a) those 29 cases are buggy and shouldn't be doing\n> that, (b) those 29 are all special cases someone has thought through\n> carefully but perhaps someone should add the same warning you have\n> here to those 29 other cases to avoid uncarefully thought cases from\n> being added, (c) there is something specific about object_info that\n> you didn't call out here, or (d) this warning you add is unnecessary.\n\nFor (a), I wouldn't say that the _INIT macros are the potentially-buggy\ncomponent, but rather the use of a designated initializer in the\npresence of an _INIT macro that initializes the struct to something\nother than all zeros.\n\nI don't think there is anything specific to the object_info structure\nhere, which suggests (d), but I don't think that the warning is\nunnecessary, it's just overly-specific. I think we should have a\nconvention in CodingGuidelines that forbids designated initializers in\nstructures with non-zero _INIT macros that don't otherwise have a\ncomment about their correctness.\n\nBut I think there is some discussion to be had there, and I want to\ndisentangle that from this series. I'm going to drop this and the\nfollowing patch from this series and split them out so we can discuss\nthem more thoroughly while letting this series move ahead.\n\nThanks,\nTaylor\n"},{"id":"516218","messageId":"xmqqo6wxkxy2.fsf@gitster.g","threadId":"63286","inReplyTo":"Z/60anmKVxke0AVX@nand.local","subject":"Re: [PATCH v2 1/8] pack-objects: use standard option incompatibility functions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-04-15T19:48:53Z","receivedAt":"2025-04-15T19:48:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> Now I can't un-see it ;-). Even though it's not a correctness issue as\n> you note, the whole thing leaves a bad taste in my mouth. I'll swap the\n> ordering to match the original in the next round.\n\nI do not think we can be completely faithful to the original in this\nrewrite, simply because the original is not consistent with what\ndie_for_incompat() thing produces and you'd need to adjust the test\nanyway. So unless there are other things you need to reroll, I\nwouldn't worry about it too much.\n\nThanks.\n\n"},{"id":"516219","messageId":"Z/64prrBW77QSznZ@nand.local","threadId":"63286","inReplyTo":"xmqqsemapi40.fsf@gitster.g","subject":"Re: [PATCH v2 2/8] object-store-ll.h: add note about designated initializers","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-15T19:51:02Z","receivedAt":"2025-04-15T19:51:05Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Apr 14, 2025 at 02:07:27PM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > @@ -337,6 +337,14 @@ struct object_info {\n> > /*\n> > * Initializer for a \"struct object_info\" that wants no items. You may\n> > * also memset() the memory to all-zeroes.\n> > + *\n> > + * NOTE: callers expect the initial value of an object_info struct to\n> > + * be zero'd out. Designated initializers like\n> > + *\n> > + * struct object_info oi = { .sizep = &sz };\n> > + *\n> > + * depend on this behavior, so consider strongly before adding new\n> > + * fields that have a non-zero default value.\n> > */\n> > #define OBJECT_INFO_INIT { 0 }\n>\n> Hmph, after thinking hard enough, if a developer cannot come up with\n> a way to avoid non-zero default value, the callers could just work\n> if they instead did\n>\n> \tstruct object_info oi = OBJECT_INFO_INIT;\n> oi.sizep = &sz;\n>\n> and the member of non-zero default value can be delat with by\n> updating the default initializer, perhaps like\n>\n> \t#define OBJECT_INFO_INIT { .enabled = 1 }\n\nYeah... that's what I was trying to get at with this patch. Basically,\n\"if you have an _INIT macro with non-zero defaults, don't use a custom\ndesignated initializer and instead assign fields after using the _INIT\nmacro (which itself is a designated initializer)\".\n\nBut like I wrote to Elijah in the same sub-thread, I think that there is\nprobably more to discuss here, so I ejected this patch from my copy of\nthe series and will re-submit it as its own series later on.\n\nThanks,\nTaylor\n"},{"id":"516220","messageId":"Z/66RtP7eRvxuVmD@nand.local","threadId":"63286","inReplyTo":"CABPp-BG-XeOCtKd-LvUNGsBGyk6rLecm=BYbUDn-6rGwi=ROzg@mail.gmail.com","subject":"Re: [PATCH v2 6/8] pack-objects: perform name-hash traversal for unpacked objects","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-15T19:57:58Z","receivedAt":"2025-04-15T19:58:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Apr 14, 2025 at 08:10:51PM -0700, Elijah Newren wrote:\n> On Mon, Apr 14, 2025 at 1:06 PM Taylor Blau wrote:\n> >\n> > With '--unpacked', pack-objects adds loose objects (which don't appear\n> > in any of the excluded packs from '--stdin-packs') to the output pack\n> > without considering them as reachability tips for the name-hash\n> > traversal.\n> >\n> > This was an oversight in the original implementation of '--stdin-packs',\n> > since the code which enumerates and adds loose objects to the output\n> > pack (`add_unreachable_loose_objects()`) did not have access to the\n> > 'rev_info' struct found in `read_packs_list_from_stdin()`.\n> >\n> > Excluding unpacked objects from that traversal doesn't effect the\n>\n> s/effect/affect/ ?\n\nOops, yes.\n\n> Should this patch have some tests demonstrating the difference in\n> which objects are included?\n\nNo; this patch doesn't actually change the set of objects we include\nwith '--stdin-packs' in conjunction with '--unpacked', it just alters\ntheir name-hash values in an attempt to produce better deltas.\n\nI don't think we have any tests that check this traversal in the packed\nor unpacked case, though we could probably add some. It's not obvious\nhow we'd test that the traversal actually produced better/different\ndeltas, but we could at least check that it happened with the trace2\nidentifier \"pack-objects/stdin_packs_hints\".\n\nI think it's probably worth doing at some point, though I don't think I\nsee it as especially urgent, unless you feel strongly otherwise.\n\nThanks,\nTaylor\n"},{"id":"516221","messageId":"Z/7FdgYID9I1qR7K@nand.local","threadId":"63286","inReplyTo":"CABPp-BFBJP15g=4M90161=KCDei-hEFdnGs7_oY8ERtqgn9s-g@mail.gmail.com","subject":"Re: [PATCH v2 7/8] pack-objects: introduce '--stdin-packs=follow'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-15T20:45:42Z","receivedAt":"2025-04-15T20:45:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Apr 14, 2025 at 08:11:08PM -0700, Elijah Newren wrote:\n> > diff --git a/Documentation/git-pack-objects.adoc b/Documentation/git-pack-objects.adoc\n> > index 7f69ae4855..c894582799 100644\n> > --- a/Documentation/git-pack-objects.adoc\n> > +++ b/Documentation/git-pack-objects.adoc\n> > @@ -87,13 +87,19 @@ base-name::\n> > reference was included in the resulting packfile. This\n> > can be useful to send new tags to native Git clients.\n> >\n> > ---stdin-packs::\n> > +--stdin-packs[=]::\n> > Read the basenames of packfiles (e.g., `pack-1234abcd.pack`)\n> > from the standard input, instead of object names or revision\n> > arguments. The resulting pack contains all objects listed in the\n> > included packs (those not beginning with `^`), excluding any\n> > objects listed in the excluded packs (beginning with `^`).\n> > +\n> > +When `mode` is \"follow\", pack objects which are reachable from objects\n> > +in the included packs, but appear in packs that are not listed.\n> > +Reachable objects which appear in excluded packs are not packed. Useful\n> > +for resurrecting once-cruft objects to generate packs which are closed\n> > +under reachability up to the excluded packs.\n>\n> Maybe:\n>\n> When `mode` is \"follow\", objects from packs not listed on stdin\n> receive special treatment. Objects within unlisted packs will be\n> included if those objects (1) are reachable from the included packs,\n> and (2) are not also found in any of the excluded packs. This mode is\n> useful for resurrecting once-cruft objects to generate packs which are\n> closed under reachability up to the boundary set by the excluded\n> packs.\n\nI like it. I went with your version with some minor rewording and tweaks\non top.\n\n> > + /*\n> > + * Our 'to_pack' list was constructed by iterating all\n> > + * objects packed in included packs, and so doesn't\n> > + * have a non-zero hash field that you would typically\n> > + * pick up during a reachability traversal.\n> > + *\n> > + * Make a best-effort attempt to fill in the ->hash\n> > + * and ->no_try_delta here using a now in order to\n> > + * perhaps improve the delta selection process.\n> > + */\n>\n> I know you just moved this paragraph from below...but it doesn't parse\n> for me. \"using a now in order to perhaps\"? What does that mean?\n\nYeah, this is just bogus, and was so before this patch. The rewording is\nminor enough (just dropping \"using a now\") that I think we can just\nsquash it in with the movement in this patch.\n\n> > + oe->hash = pack_name_hash_fn(name);\n> > + oe->no_try_delta = name && no_try_delta(name);\n> > +\n> > + stdin_packs_hints_nr++;\n> > + }\n> > +}\n> > +\n> > +static void show_commit_pack_hint(struct commit *commit, void *data)\n> > +{\n> > + enum stdin_packs_mode mode = *(enum stdin_packs_mode *)data;\n> > + if (mode == STDIN_PACKS_MODE_FOLLOW) {\n> > + show_object_pack_hint((struct object *)commit, \"\", data);\n> > return;\n> > + }\n> > + /* nothing to do; commits don't have a namehash */\n> >\n> > - /*\n> > - * Our 'to_pack' list was constructed by iterating all objects packed in\n> > - * included packs, and so doesn't have a non-zero hash field that you\n> > - * would typically pick up during a reachability traversal.\n> > - *\n> > - * Make a best-effort attempt to fill in the ->hash and ->no_try_delta\n> > - * here using a now in order to perhaps improve the delta selection\n> > - * process.\n> > - */\n> > - oe->hash = pack_name_hash_fn(name);\n> > - oe->no_try_delta = name && no_try_delta(name);\n> > -\n> > - stdin_packs_hints_nr++;\n> > }\n>\n> It might be worth swapping the order of functions as a preparatory\n> patch (both here and when you've done it elsewhere in this series),\n> just because it'll make the diff so much easier to read when we can\n> see the changes to the function without have to also deal with the\n> order swapping (since order swapping looks like a large deletion and\n> large addition of one of the two functions).\n\nFair enough.\n\n> > @@ -4467,6 +4484,23 @@ static int is_not_in_promisor_pack(struct commit *commit, void *data) {\n> > return is_not_in_promisor_pack_obj((struct object *) commit, data);\n> > }\n> >\n> > +static int parse_stdin_packs_mode(const struct option *opt, const char *arg,\n> > + int unset)\n> > +{\n> > + enum stdin_packs_mode *mode = opt->value;\n> > +\n> > + if (unset)\n> > + *mode = STDIN_PACKS_MODE_NONE;\n> > + else if (!arg || !*arg)\n> > + *mode = STDIN_PACKS_MODE_STANDARD;\n>\n> I don't understand why you have both a None mode and a Standard mode,\n> especially since the implementation seems to only care about whether\n> or not the Follow mode has been set. Shouldn't these both be setting\n> mode to the same value?\n\nI'm not sure I follow your question... stdin_packs is a tri-state. It\ncan be off, on in standard/legacy mode, or on in follow mode.\n\n> > +test_expect_success 'setup for --stdin-packs=follow' '\n> > + git init stdin-packs--follow &&\n> > + (\n> > + cd stdin-packs--follow &&\n> > +\n> > + for c in A B C D\n> > + do\n> > + test_commit \"$c\" || return 1\n> > + done &&\n> > +\n> > + A=\"$(echo A | git pack-objects --revs $packdir/pack)\" &&\n> > + B=\"$(echo A..B | git pack-objects --revs $packdir/pack)\" &&\n> > + C=\"$(echo B..C | git pack-objects --revs $packdir/pack)\" &&\n> > + D=\"$(echo C..D | git pack-objects --revs $packdir/pack)\" &&\n> > +\n> > + git prune-packed\n> > + )\n> > +'\n\nHuh, I have no idea how this snuck in. This \"setup\" test does nothing\nand creates a repository that isn't used later on in the script.\nProbably leftover from writing these tests in the first place, but I've\nremoved it.\n\n> I like the tests -- normal --stdin-packs, then --stdin-packs=follow,\n> then --stdin-packs=follow + --unpacked.\n\nI think the normal tests are accidental since we use pack-objects to\nwrite packs A, B, C, and D. But the --stdin-packs vs.\n--stdin-packs=follow and --stdin-packs=follow + --unpacked was\ndefinitely intentional.\n\n> However, would it be worthwhile to create commit E immediately after\n> creating the packs?\n\nYeah, I think that is a good suggestion. We already have tests that\nexercise --stdin-packs with --unpacked earlier in the same script, but\nobviously not with --stdin-packs=follow. Moving the creation of commit E\nearlier up makes a lot of sense to me, thanks!\n\nThanks,\nTaylor\n"},{"id":"516222","messageId":"Z/7G73FmI/bK1Jti@nand.local","threadId":"63286","inReplyTo":"CABPp-BHwewWMp1CY4jAr=sWwxE9of2Q73nyXQ5wpOELJZQ1OPg@mail.gmail.com","subject":"Re: [PATCH v2 8/8] repack: exclude cruft pack(s) from the MIDX where possible","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-15T20:51:59Z","receivedAt":"2025-04-15T20:52:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Apr 14, 2025 at 08:11:22PM -0700, Elijah Newren wrote:\n> On Mon, Apr 14, 2025 at 1:06 PM Taylor Blau wrote:\n> >\n> > In ddee3703b3 (builtin/repack.c: add cruft packs to MIDX during\n> > geometric repack, 2022-05-20), repack began adding cruft pack(s) to the\n> > MIDX with '--write-midx' to ensure that the resulting MIDX was always\n> > closed under reachability in order to generate reachability bitmaps.\n> >\n> > Suppose you have a once-unreachable object packed in a cruft pack, which\n> > later on becomes reachable from one or more objects in a geometrically\n> > repacked pack. That once-unreachable object *won't* appear in the new\n> > pack, since the cruft pack was specified as neither included nor\n> > excluded to 'pack-objects --stdin-packs'.\n>\n> I believe you are talking about the state before your series (i.e.,\n> this is carrying on from the previous paragraph), but it reads as\n> though you are talking about the state after the first seven patches\n> of this series. Some kind of connection wording to clarify would\n> really help here.\n\nSure.\n\n> > If the bitmap selection\n> > process picks one or more commits which reach the once-unreachable\n> > objects, commit ddee3703b3 ensures that the MIDX will be closed under\n> > reachability. Without it, we would fail to generate a MIDX bitmap.\n>\n> After reading this part, I had to go back and re-read and figure out\n> what point in time everything was referring to.\n\nYeah, this is confusing to me too after reading it back. I made some\ntweaks that I think clarify things.\n\n> > ddee3703b3 alludes to the fact that this is sub-optimal by saying\n> >\n> > [...] it's desirable to avoid including cruft packs in the MIDX\n> > because it causes the MIDX to store a bunch of objects which are\n> > likely to get thrown away.\n> >\n> > , which is true, but hides an even larger problem. If repositories\n> > rarely prune their unreachable objects and/or have many of them, the\n> > MIDX must keep track of a large number of objects which bloats the MIDX\n> > and slows down object lookup.\n> >\n> > This is doubly unfortunate because the vast majority of objects in cruft\n> > pack(s) are unlikely to be read, but object reads that go through the\n> > MIDX have to search through them anyway.\n>\n> \"have to search through them\"? That could be read to suggest those\n> individual objects are read, rather than just traversed over. Maybe\n> \"...unlikely to be read, so the enlarged MIDX is for mostly tracking\n> known-to-likely-be-irrelevant objects\", or something like that?\n\nThanks for pointing out... I clarified this one as well.\n\n> > This patch causes geometrically-repacked packs to contain a copy of any\n> > once-unreachable object(s) with 'git pack-objects --stdin-packs=follow',\n> > allowing us to avoid including any cruft packs in the MIDX. This is\n> > because a sequence of geometrically-repacked packs that were all\n> > generated with '--stdin-packs=follow' are guaranteed to have their union\n> > be closed under reachability.\n> >\n> > Note that you cannot guarantee that a collection of packs is closed\n> > under reachability if not all of them were generated with following as\n>\n> maybe: ...with \"follow\" as above. \"follow\" or \"following\" feels like\n> it needs quotes so the reader understands its meant as the name of a\n> mode, rather than a verb in the sentence.\n>\n> > above. One tell-tale sign that not all geometrically-repacked packs in\n> > the MIDX were generated with following is to see if there is a pack in\n>\n> same here with \"following\"...and below.\n\nGreat calls on both, thanks.\n\n> > @@ -808,26 +886,55 @@ static void midx_included_packs(struct string_list *include,\n> > }\n> > }\n> >\n> > - for_each_string_list_item(item, &existing->cruft_packs) {\n> > + if (midx_must_contain_cruft ||\n> > + midx_has_unknown_packs(midx_pack_names, midx_pack_names_nr,\n> > + include, geometry, existing)) {\n> > /*\n> > - * When doing a --geometric repack, there is no need to check\n> > - * for deleted packs, since we're by definition not doing an\n> > - * ALL_INTO_ONE repack (hence no packs will be deleted).\n> > - * Otherwise we must check for and exclude any packs which are\n> > - * enqueued for deletion.\n> > + * If there are one or more unknown pack(s) present (see\n> > + * midx_has_unknown_packs() for what makes a pack\n> > + * \"unknown\") in the MIDX before the repack, keep them\n> > + * as they may be required to form a reachability\n> > + * closure if the MIDX is bitmapped.\n> > *\n> > - * So we could omit the conditional below in the --geometric\n> > - * case, but doing so is unnecessary since no packs are marked\n> > - * as pending deletion (since we only call\n> > - * `mark_packs_for_deletion()` when doing an all-into-one\n> > - * repack).\n> > + * For example, a cruft pack can be required to form a\n> > + * reachability closure if the MIDX is bitmapped and one\n> > + * or more of its selected commits reaches a once-cruft\n> > + * object that was later made reachable.\n>\n> The antecedent of \"its\" is unclear here; just spell it out to reduce\n> how much thinking the reader needs to do?\n\nEek, good suggestion again. Thanks, I fixed it up and made the\nantecedent explicit.\n\n> > */\n> > - if (pack_is_marked_for_deletion(item))\n> > - continue;\n> > + for_each_string_list_item(item, &existing->cruft_packs) {\n> > + /*\n> > + * When doing a --geometric repack, there is no\n> > + * need to check for deleted packs, since we're\n> > + * by definition not doing an ALL_INTO_ONE\n> > + * repack (hence no packs will be deleted).\n> > + * Otherwise we must check for and exclude any\n> > + * packs which are enqueued for deletion.\n> > + *\n> > + * So we could omit the conditional below in the\n> > + * --geometric case, but doing so is unnecessary\n> > + * since no packs are marked as pending\n> > + * deletion (since we only call\n> > + * `mark_packs_for_deletion()` when doing an\n> > + * all-into-one repack).\n> > + */\n> > + if (pack_is_marked_for_deletion(item))\n> > + continue;\n> >\n> > - strbuf_reset(&buf);\n> > - strbuf_addf(&buf, \"%s.idx\", item->string);\n> > - string_list_insert(include, buf.buf);\n> > + strbuf_reset(&buf);\n> > + strbuf_addf(&buf, \"%s.idx\", item->string);\n> > + string_list_insert(include, buf.buf);\n> > + }\n> > + } else {\n> > + /*\n> > + * Modern versions of Git will write new copies of\n> > + * once-cruft objects when doing a --geometric repack.\n>\n> \"Modern versions of Git\" -> \"Modern versions of Git with the\n> appropriate config setting\" ?\n\nHeh. Great catch. Can you tell this part was written before I added the\nconfiguration option? ;-)\n\nThanks,\nTaylor\n"},{"id":"516225","messageId":"Z/7YOh+bDN17jye+@nand.local","threadId":"63286","inReplyTo":"CABPp-BH7U4Vh8b6L9_FNUsBqKB+4hNT_Twn4S7LTocLvbw1LjA@mail.gmail.com","subject":"Re: [PATCH v2 0/8] repack: avoid MIDX'ing cruft pack(s) where possible","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-15T22:05:46Z","receivedAt":"2025-04-15T22:05:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Apr 14, 2025 at 07:57:52PM -0700, Elijah Newren wrote:\n> On Mon, Apr 14, 2025 at 1:06 PM Taylor Blau wrote:\n> >\n> > Here is a non-RFC version of my series to explore creating MIDXs while\n> > repacking that don't include the cruft pack.\n> >\n> > The core idea behind this approach is to ensure that packs generated via\n> > geometric repacking traverse through objects that appear in packs which\n> > are neither included nor excluded.\n>\n> This phrasing feels confusing -- what does it mean for packs to be\n> neither included nor excluded? Maybe:\n>\n> \"The core idea behind this approach is to allow some (most) of the\n> objects in a pack to be excluded, while still including some subset of\n> objects from that pack as part of the repack. In particular, we\n> include the objects in that pack which are reachable from the other\n> objects we repack. This is different from our current handling which\n> either entirely includes or entirely excludes all objects from a given\n> pack.\"\n\nI am admittedly having a little bit of a hard time parsing your version\nof this, but I think this part:\n\n [...] In particular, we include the objects in that pack which are\n reachable from the other objects we repack.\n\nisn't quite right. It's not that the output pack contains objects\nreachable from the other objects we repack, but rather it contains the\nreachable objects from the other objects we repack *if* those objects\ndon't appear in an excluded pack given as part of the input.\n\n> > Then if some commit (for example) in\n> > a pack reaches some once-unreachable object stored in a cruft pack, the\n> > pack generated via geometric repacking will pick up and write a copy of\n> > that object during its traversal.\n> >\n> > If you repack consistently using this strategy, you can guarantee that\n> > the union of geometrically-repacked packs are closed under reachability\n> > without having to keep track of any cruft pack(s) in the MIDX.\n>\n> Also, if you do a single non-geometric repack with this strategy, you\n> are also closed under reachability, right? Is that the suggested\n> transition plan for those that want to use this...first do a\n> non-geometric repack, and then ensure that subsequent geometric\n> repacks are done with this strategy?\n\nYeah, the last commit gets at this a bit. The property you have to\nmaintain is that the union of geometrically-repacked packs (which form\nthe MIDX) are and stay closed under reachability. I am pretty sure that\nthe way this is constructed, adding new geometrically-repacked packs to\nthe chain does not violate this property[^1].\n\nBut you can't guarantee it part of the way through a sequence of\ngeometric repacks, which is what midx_has_unknown_packs() is checking\nfor.\n\nIf you do an all-into-one cruft repack first, then there is no MIDX to\nbegin with, so there aren't any unknown packs to worry about (since\nthere are no packs in a MIDX to begin with). When that property is met,\nthen we can use the new behavior.\n\nThanks,\nTaylor\n\n[^1]: So long as you don't drop part of the geometric progression, e.g.,\n if you have some pack that was in the existing MIDX, but wasn't\n repacked or included in the new MIDX.\n"},{"id":"516226","messageId":"Z/7dVa+RnEqayEMf@nand.local","threadId":"63286","inReplyTo":"xmqqo6wxkxy2.fsf@gitster.g","subject":"Re: [PATCH v2 1/8] pack-objects: use standard option incompatibility functions","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-15T22:27:33Z","receivedAt":"2025-04-15T22:27:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Apr 15, 2025 at 12:48:53PM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > Now I can't un-see it ;-). Even though it's not a correctness issue as\n> > you note, the whole thing leaves a bad taste in my mouth. I'll swap the\n> > ordering to match the original in the next round.\n>\n> I do not think we can be completely faithful to the original in this\n> rewrite, simply because the original is not consistent with what\n> die_for_incompat() thing produces and you'd need to adjust the test\n> anyway. So unless there are other things you need to reroll, I\n> wouldn't worry about it too much.\n\nYeah, we need to adjust the test either way. I just disliked reading the\npatch and seeing:\n\n if (stdin_packs && filter_options.choice)\n die(_(\"--stdin-packs and --filter can't be used together\"));\n\nturn into\n\n die_for_incompatible_opt2(filter_options.choice, \"--filter\",\n stdin_packs, \"--stdin-packs\");\n\nsince the check is \"stdin_packs then filter_options.choice\" in the\noriginal, but \"filter_options.choice then stdin_packs\" in this patch.\n\nFunny enough, the test that breaks expects output that mentions\n\"--filter\" before \"--stdin-packs\" here, so preserving the order of the\ncheck in the code reverses the order in which the incompatible arguments\nappear in the die() message.\n\n> Thanks.\n\nThanks,\nTaylor\n"},{"id":"516229","messageId":"cover.1744757204.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744413969.git.me@ttaylorr.com","subject":"[PATCH v3 0/9] repack: avoid MIDX'ing cruft pack(s) where possible","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-15T22:46:45Z","receivedAt":"2025-04-15T22:46:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here is a small-ish reroll of my series to explore creating MIDXs while\nrepacking that don't include the cruft pack.\n\nThe bulk of the series is unchanged, save for a few minor points that\nI'll call out here (as usual, a complete range-diff is below for\nconvenience):\n\n * Dropped the patch adding a warning about using designated\n initializers. I think that we should resurrect this patch soon and\n update the CodingGuidelines, but I'd rather disentangle that from\n this series.\n\n * Dropped the designated initializer component of the \"limit scope\"\n patch.\n\n * Swapped ordering on one of the die_for_incompatible_opt2() checks.\n\n * Various wording tweaks.\n\n * Split out some code movement changes into their own patches to make\n substantive patches easier to read/review.\n\n * Test updates and cleanup.\n\nThese changes are all thanks to helpful review from Junio and Elijah.\nThanks, both!\n\nOtherwise the series is unchanged. I still need to deploy it to GitHub's\ninfrastructure and try it out on some internal repos, but I should be\nable to do that tomorrow and report back on my findings a few days after\nthat.\n\nThanks in advance for any review :-).\n\nTaylor Blau (9):\n pack-objects: use standard option incompatibility functions\n pack-objects: limit scope in 'add_object_entry_from_pack()'\n pack-objects: factor out handling '--stdin-packs'\n pack-objects: declare 'rev_info' for '--stdin-packs' earlier\n pack-objects: perform name-hash traversal for unpacked objects\n pack-objects: fix typo in 'show_object_pack_hint()'\n pack-objects: swap 'show_{object,commit}_pack_hint'\n pack-objects: introduce '--stdin-packs=follow'\n repack: exclude cruft pack(s) from the MIDX where possible\n\n Documentation/config/repack.adoc | 7 +\n Documentation/git-pack-objects.adoc | 10 +-\n builtin/pack-objects.c | 192 ++++++++++++++++++----------\n builtin/repack.c | 163 ++++++++++++++++++++---\n t/t5331-pack-objects-stdin.sh | 84 +++++++++++-\n t/t7704-repack-cruft.sh | 90 +++++++++++++\n 6 files changed, 456 insertions(+), 90 deletions(-)\n\nRange-diff against v2:\n 1: 65bc7e4630 ! 1: f8b31c6a8d pack-objects: use standard option incompatibility functions\n @@ builtin/pack-objects.c: int cmd_pack_objects(int argc,\n \n -\tif (stdin_packs && filter_options.choice)\n -\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n -+\tdie_for_incompatible_opt2(filter_options.choice, \"--filter\",\n -+\t\t\t\t stdin_packs, \"--stdin-packs\");\n ++\tdie_for_incompatible_opt2(stdin_packs, \"--stdin-packs\",\n ++\t\t\t\t filter_options.choice, \"--filter\");\n ++\n \n \tif (stdin_packs && use_internal_rev_list)\n \t\tdie(_(\"cannot use internal rev list with --stdin-packs\"));\n @@ t/t5331-pack-objects-stdin.sh: test_expect_success '--stdin-packs is incompatibl\n \t\ttest_must_fail git pack-objects --stdin-packs --stdout \\\n \t\t\t--filter=blob:none err &&\n -\t\ttest_grep \"cannot use --filter with --stdin-packs\" err\n -+\t\ttest_grep \"options .--filter. and .--stdin-packs. cannot be used together\" err\n ++\t\ttest_grep \"options .--stdin-packs. and .--filter. cannot be used together\" err\n \t)\n '\n \n 2: 920c91eb1e < -: ---------- object-store-ll.h: add note about designated initializers\n 3: f8ac36b110 ! 2: 986bef29b5 pack-objects: limit scope in 'add_object_entry_from_pack()'\n @@ Metadata\n ## Commit message ##\n pack-objects: limit scope in 'add_object_entry_from_pack()'\n \n - add_object_entry_from_pack() handles objects from identified packs by\n - checking their type, before adding commit objects as pending in the\n - subsequent traversal used by `--stdin-packs`.\n -\n - There are a couple of quality-of-life refactorings that I noticed while\n - working in this area:\n -\n - - We declare 'revs' (given to us through the miscellaneous context\n - argument) earlier in the \"if (p)\" conditional than is necessary.\n -\n - - The 'struct object_info' can use a designated initializer to fill in\n - the structures type pointer, since that is the only field that we\n - care about.\n + In add_object_entry_from_pack() we declare 'revs' (given to us through\n + the miscellaneous context argument) earlier in the \"if (p)\" conditional\n + than is necessary. Move it down as far as it can go to reduce its\n + scope.\n \n Signed-off-by: Taylor Blau \n \n @@ builtin/pack-objects.c: static int add_object_entry_from_pack(const struct objec\n \n \tif (p) {\n -\t\tstruct rev_info *revs = _data;\n --\t\tstruct object_info oi = OBJECT_INFO_INIT;\n + \t\tstruct object_info oi = OBJECT_INFO_INIT;\n -\n --\t\toi.typep = &type;\n -+\t\tstruct object_info oi = { .typep = &type };\n + \t\toi.typep = &type;\n ++\n \t\tif (packed_object_info(the_repository, p, ofs, &oi) < 0) {\n \t\t\tdie(_(\"could not get type of object %s in pack %s\"),\n \t\t\t oid_to_hex(oid), p->pack_name);\n 4: 5e03b482ba = 3: 6f8fe8a4e1 pack-objects: factor out handling '--stdin-packs'\n 5: bccbac2ec5 = 4: 2a235461a6 pack-objects: declare 'rev_info' for '--stdin-packs' earlier\n 6: 0bc2183dc3 ! 5: 240e90b68d pack-objects: perform name-hash traversal for unpacked objects\n @@ Commit message\n pack (`add_unreachable_loose_objects()`) did not have access to the\n 'rev_info' struct found in `read_packs_list_from_stdin()`.\n \n - Excluding unpacked objects from that traversal doesn't effect the\n + Excluding unpacked objects from that traversal doesn't affect the\n correctness of the resulting pack, but it does make it harder to\n discover good deltas for loose objects.\n \n -: ---------- > 6: 9a18fa2e52 pack-objects: fix typo in 'show_object_pack_hint()'\n -: ---------- > 7: 6c997853f1 pack-objects: swap 'show_{object,commit}_pack_hint'\n 7: 697a337cb1 ! 8: 0ff699f056 pack-objects: introduce '--stdin-packs=follow'\n @@ Documentation/git-pack-objects.adoc: base-name::\n \tincluded packs (those not beginning with `^`), excluding any\n \tobjects listed in the excluded packs (beginning with `^`).\n +\n -+When `mode` is \"follow\", pack objects which are reachable from objects\n -+in the included packs, but appear in packs that are not listed.\n -+Reachable objects which appear in excluded packs are not packed. Useful\n -+for resurrecting once-cruft objects to generate packs which are closed\n -+under reachability up to the excluded packs.\n ++When `mode` is \"follow\", objects from packs not listed on stdin receive\n ++special treatment. Objects within unlisted packs will be included if\n ++those objects are (1) reachable from the included packs, and (2) not\n ++found in any excluded packs. This mode is useful, for example, to\n ++resurrect once-unreachable objects found in cruft packs to generate\n ++packs which are closed under reachability up to the boundary set by the\n ++excluded packs.\n ++\n Incompatible with `--revs`, or options that imply `--revs` (such as\n `--all`), with the exception of `--unpacked`, which is compatible.\n @@ builtin/pack-objects.c: static struct oidmap configured_exclusions;\n * Check whether the name_hash_version chosen by user input is appropriate,\n * and also validate whether it is compatible with other features.\n @@ builtin/pack-objects.c: static int add_object_entry_from_pack(const struct object_id *oid,\n - \treturn 0;\n }\n \n --static void show_commit_pack_hint(struct commit *commit UNUSED,\n --\t\t\t\t void *data UNUSED)\n --{\n --\t/* nothing to do; commits don't have a namehash */\n --}\n --\n static void show_object_pack_hint(struct object *object, const char *name,\n -\t\t\t\t void *data UNUSED)\n +\t\t\t\t void *data)\n {\n -\tstruct object_entry *oe = packlist_find(&to_pack, &object->oid);\n -\tif (!oe)\n +-\t\treturn;\n +\tenum stdin_packs_mode mode = *(enum stdin_packs_mode *)data;\n +\tif (mode == STDIN_PACKS_MODE_FOLLOW) {\n +\t\tadd_object_entry(&object->oid, object->type, name, 0);\n @@ builtin/pack-objects.c: static int add_object_entry_from_pack(const struct objec\n +\t\tstruct object_entry *oe = packlist_find(&to_pack, &object->oid);\n +\t\tif (!oe)\n +\t\t\treturn;\n -+\n -+\t\t/*\n -+\t\t * Our 'to_pack' list was constructed by iterating all\n -+\t\t * objects packed in included packs, and so doesn't\n -+\t\t * have a non-zero hash field that you would typically\n -+\t\t * pick up during a reachability traversal.\n -+\t\t *\n -+\t\t * Make a best-effort attempt to fill in the ->hash\n -+\t\t * and ->no_try_delta here using a now in order to\n -+\t\t * perhaps improve the delta selection process.\n -+\t\t */\n -+\t\toe->hash = pack_name_hash_fn(name);\n -+\t\toe->no_try_delta = name && no_try_delta(name);\n -+\n -+\t\tstdin_packs_hints_nr++;\n -+\t}\n -+}\n -+\n -+static void show_commit_pack_hint(struct commit *commit, void *data)\n -+{\n -+\tenum stdin_packs_mode mode = *(enum stdin_packs_mode *)data;\n -+\tif (mode == STDIN_PACKS_MODE_FOLLOW) {\n -+\t\tshow_object_pack_hint((struct object *)commit, \"\", data);\n - \t\treturn;\n -+\t}\n -+\t/* nothing to do; commits don't have a namehash */\n \n -\t/*\n -\t * Our 'to_pack' list was constructed by iterating all objects packed in\n @@ builtin/pack-objects.c: static int add_object_entry_from_pack(const struct objec\n -\t * would typically pick up during a reachability traversal.\n -\t *\n -\t * Make a best-effort attempt to fill in the ->hash and ->no_try_delta\n --\t * here using a now in order to perhaps improve the delta selection\n +-\t * fields here in order to perhaps improve the delta selection\n -\t * process.\n -\t */\n -\toe->hash = pack_name_hash_fn(name);\n -\toe->no_try_delta = name && no_try_delta(name);\n --\n ++\t\t/*\n ++\t\t * Our 'to_pack' list was constructed by iterating all\n ++\t\t * objects packed in included packs, and so doesn't have\n ++\t\t * a non-zero hash field that you would typically pick\n ++\t\t * up during a reachability traversal.\n ++\t\t *\n ++\t\t * Make a best-effort attempt to fill in the ->hash and\n ++\t\t * ->no_try_delta fields here in order to perhaps\n ++\t\t * improve the delta selection process.\n ++\t\t */\n ++\t\toe->hash = pack_name_hash_fn(name);\n ++\t\toe->no_try_delta = name && no_try_delta(name);\n + \n -\tstdin_packs_hints_nr++;\n ++\t\tstdin_packs_hints_nr++;\n ++\t}\n + }\n + \n +-static void show_commit_pack_hint(struct commit *commit UNUSED,\n +-\t\t\t\t void *data UNUSED)\n ++static void show_commit_pack_hint(struct commit *commit, void *data)\n + {\n ++\tenum stdin_packs_mode mode = *(enum stdin_packs_mode *)data;\n ++\n ++\tif (mode == STDIN_PACKS_MODE_FOLLOW) {\n ++\t\tshow_object_pack_hint((struct object *)commit, \"\", data);\n ++\t\treturn;\n ++\t}\n ++\n + \t/* nothing to do; commits don't have a namehash */\n ++\n }\n \n static int pack_mtime_cmp(const void *_a, const void *_b)\n @@ t/t5331-pack-objects-stdin.sh: test_expect_success 'pack-objects --stdin with pa\n +\trm -f objects.raw\n +}\n +\n -+test_expect_success 'setup for --stdin-packs=follow' '\n -+\tgit init stdin-packs--follow &&\n -+\t(\n -+\t\tcd stdin-packs--follow &&\n -+\n -+\t\tfor c in A B C D\n -+\t\tdo\n -+\t\t\ttest_commit \"$c\" || return 1\n -+\t\tdone &&\n -+\n -+\t\tA=\"$(echo A | git pack-objects --revs $packdir/pack)\" &&\n -+\t\tB=\"$(echo A..B | git pack-objects --revs $packdir/pack)\" &&\n -+\t\tC=\"$(echo B..C | git pack-objects --revs $packdir/pack)\" &&\n -+\t\tD=\"$(echo C..D | git pack-objects --revs $packdir/pack)\" &&\n -+\n -+\t\tgit prune-packed\n -+\t)\n -+'\n -+\n +test_expect_success '--stdin-packs=follow walks into unknown packs' '\n +\ttest_when_finished \"rm -fr repo\" &&\n +\n @@ t/t5331-pack-objects-stdin.sh: test_expect_success 'pack-objects --stdin with pa\n +\t\tB=\"$(echo A..B | git pack-objects --revs $packdir/pack)\" &&\n +\t\tC=\"$(echo B..C | git pack-objects --revs $packdir/pack)\" &&\n +\t\tD=\"$(echo C..D | git pack-objects --revs $packdir/pack)\" &&\n ++\t\ttest_commit E &&\n +\n +\t\tgit prune-packed &&\n +\n @@ t/t5331-pack-objects-stdin.sh: test_expect_success 'pack-objects --stdin with pa\n +\t\tobjects_in_packs $P >actual &&\n +\t\ttest_cmp expect actual &&\n +\n -+\t\ttest_commit E &&\n +\t\t# And with --unpacked, we will pick up objects from unknown\n +\t\t# packs that are reachable from loose objects. Loose object E\n +\t\t# reaches objects in pack A, but there are three excluded packs\n 8: a2ec1b826c ! 9: 58891101f3 repack: exclude cruft pack(s) from the MIDX where possible\n @@ Commit message\n MIDX with '--write-midx' to ensure that the resulting MIDX was always\n closed under reachability in order to generate reachability bitmaps.\n \n - Suppose you have a once-unreachable object packed in a cruft pack, which\n - later on becomes reachable from one or more objects in a geometrically\n - repacked pack. That once-unreachable object *won't* appear in the new\n - pack, since the cruft pack was specified as neither included nor\n - excluded to 'pack-objects --stdin-packs'. If the bitmap selection\n - process picks one or more commits which reach the once-unreachable\n - objects, commit ddee3703b3 ensures that the MIDX will be closed under\n - reachability. Without it, we would fail to generate a MIDX bitmap.\n + Suppose (prior to this patch) you have a once-unreachable object packed\n + in a cruft pack, which later on becomes reachable from one or more\n + objects in a geometrically repacked pack. That once-unreachable object\n + *won't* appear in the new pack, since the cruft pack was specified as\n + neither included nor excluded to 'pack-objects --stdin-packs'. If the\n + new pack is included in a MIDX without the cruft pack, then trying to\n + generate bitmaps for that MIDX may fail. This happens when the bitmap\n + selection process picks one or more commits which reach the\n + once-unreachable objects, commit ddee3703b3 ensures that the MIDX will\n + be closed under reachability. Without it, we would fail to generate a\n + MIDX bitmap.\n \n ddee3703b3 alludes to the fact that this is sub-optimal by saying\n \n @@ Commit message\n and slows down object lookup.\n \n This is doubly unfortunate because the vast majority of objects in cruft\n - pack(s) are unlikely to be read, but object reads that go through the\n - MIDX have to search through them anyway.\n + pack(s) are unlikely to be read. But any object lookups that go through\n + the MIDX must binary search over them anyway, slowing down object\n + lookups using the MIDX.\n \n This patch causes geometrically-repacked packs to contain a copy of any\n once-unreachable object(s) with 'git pack-objects --stdin-packs=follow',\n @@ Commit message\n be closed under reachability.\n \n Note that you cannot guarantee that a collection of packs is closed\n - under reachability if not all of them were generated with following as\n + under reachability if not all of them were generated with \"following\" as\n above. One tell-tale sign that not all geometrically-repacked packs in\n - the MIDX were generated with following is to see if there is a pack in\n + the MIDX were generated with \"following\" is to see if there is a pack in\n the existing MIDX that is not going to be somehow represented (either\n verbatim or as part of a geometric rollup) in the new MIDX.\n \n - If there is, then starting to generate packs with following during\n + If there is, then starting to generate packs with \"following\" during\n geometric repacking won't work, since it's open to the same race as\n described above.\n \n @@ builtin/repack.c: static void midx_included_packs(struct string_list *include,\n -\t\t * repack).\n +\t\t * For example, a cruft pack can be required to form a\n +\t\t * reachability closure if the MIDX is bitmapped and one\n -+\t\t * or more of its selected commits reaches a once-cruft\n -+\t\t * object that was later made reachable.\n ++\t\t * or more of the bitmap's selected commits reaches a\n ++\t\t * once-cruft object that was later made reachable.\n \t\t */\n -\t\tif (pack_is_marked_for_deletion(item))\n -\t\t\tcontinue;\n @@ builtin/repack.c: static void midx_included_packs(struct string_list *include,\n +\t\t}\n +\t} else {\n +\t\t/*\n -+\t\t * Modern versions of Git will write new copies of\n ++\t\t * Modern versions of Git (with the appropriate\n ++\t\t * configuration setting) will write new copies of\n +\t\t * once-cruft objects when doing a --geometric repack.\n +\t\t *\n +\t\t * If the MIDX has no cruft pack, new packs written\n\nbase-commit: 485f5f863615e670fd97ae40af744e14072cfe18\n-- \n2.49.0.230.ga662d77f78\n"},{"id":"516230","messageId":"f8b31c6a8d788fa53f7b39fb9ae4b61ebfc882b9.1744757204.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744757204.git.me@ttaylorr.com","subject":"[PATCH v3 1/9] pack-objects: use standard option incompatibility functions","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-15T22:46:48Z","receivedAt":"2025-04-15T22:46:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"pack-objects has a handful of explicit checks for pairs of command-line\noptions which are mutually incompatible. Many of these pre-date\na699367bb8 (i18n: factorize more 'incompatible options' messages,\n2022-01-31).\n\nConvert the explicit checks into die_for_incompatible_opt2() calls,\nwhich simplifies the implementation and standardizes pack-objects'\noutput when given incompatible options (e.g., --stdin-packs with\n--filter gives different output than --keep-unreachable with\n--unpack-unreachable).\n\nThere is one minor piece of test fallout in t5331 that expects the old\nformat, which has been corrected.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 20 +++++++++++---------\n t/t5331-pack-objects-stdin.sh | 2 +-\n 2 files changed, 12 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 6b06d159d2..20dd870bbf 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -4651,9 +4651,10 @@ int cmd_pack_objects(int argc,\n \t\tstrvec_push(&rp, \"--unpacked\");\n \t}\n \n-\tif (exclude_promisor_objects && exclude_promisor_objects_best_effort)\n-\t\tdie(_(\"options '%s' and '%s' cannot be used together\"),\n-\t\t \"--exclude-promisor-objects\", \"--exclude-promisor-objects-best-effort\");\n+\tdie_for_incompatible_opt2(exclude_promisor_objects,\n+\t\t\t\t \"--exclude-promisor-objects\",\n+\t\t\t\t exclude_promisor_objects_best_effort,\n+\t\t\t\t \"--exclude-promisor-objects-best-effort\");\n \tif (exclude_promisor_objects) {\n \t\tuse_internal_rev_list = 1;\n \t\tfetch_if_missing = 0;\n@@ -4691,13 +4692,14 @@ int cmd_pack_objects(int argc,\n \tif (!pack_to_stdout && thin)\n \t\tdie(_(\"--thin cannot be used to build an indexable pack\"));\n \n-\tif (keep_unreachable && unpack_unreachable)\n-\t\tdie(_(\"options '%s' and '%s' cannot be used together\"), \"--keep-unreachable\", \"--unpack-unreachable\");\n+\tdie_for_incompatible_opt2(keep_unreachable, \"--keep-unreachable\",\n+\t\t\t\t unpack_unreachable, \"--unpack-unreachable\");\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n-\tif (stdin_packs && filter_options.choice)\n-\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n+\tdie_for_incompatible_opt2(stdin_packs, \"--stdin-packs\",\n+\t\t\t\t filter_options.choice, \"--filter\");\n+\n \n \tif (stdin_packs && use_internal_rev_list)\n \t\tdie(_(\"cannot use internal rev list with --stdin-packs\"));\n@@ -4705,8 +4707,8 @@ int cmd_pack_objects(int argc,\n \tif (cruft) {\n \t\tif (use_internal_rev_list)\n \t\t\tdie(_(\"cannot use internal rev list with --cruft\"));\n-\t\tif (stdin_packs)\n-\t\t\tdie(_(\"cannot use --stdin-packs with --cruft\"));\n+\t\tdie_for_incompatible_opt2(stdin_packs, \"--stdin-packs\",\n+\t\t\t\t\t cruft, \"--cruft\");\n \t}\n \n \t/*\ndiff --git a/t/t5331-pack-objects-stdin.sh b/t/t5331-pack-objects-stdin.sh\nindex b48c0cbe8f..8fd07deb8d 100755\n--- a/t/t5331-pack-objects-stdin.sh\n+++ b/t/t5331-pack-objects-stdin.sh\n@@ -64,7 +64,7 @@ test_expect_success '--stdin-packs is incompatible with --filter' '\n \t\tcd stdin-packs &&\n \t\ttest_must_fail git pack-objects --stdin-packs --stdout \\\n \t\t\t--filter=blob:none err &&\n-\t\ttest_grep \"cannot use --filter with --stdin-packs\" err\n+\t\ttest_grep \"options .--stdin-packs. and .--filter. cannot be used together\" err\n \t)\n '\n \n-- \n2.49.0.230.ga662d77f78\n\n"},{"id":"516231","messageId":"986bef29b5f33d32fd366aa9370d439175a9b605.1744757204.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744757204.git.me@ttaylorr.com","subject":"[PATCH v3 2/9] pack-objects: limit scope in 'add_object_entry_from_pack()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-15T22:46:51Z","receivedAt":"2025-04-15T22:46:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In add_object_entry_from_pack() we declare 'revs' (given to us through\nthe miscellaneous context argument) earlier in the \"if (p)\" conditional\nthan is necessary. Move it down as far as it can go to reduce its\nscope.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 20dd870bbf..4ab695a3aa 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3490,14 +3490,14 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n \t\treturn 0;\n \n \tif (p) {\n-\t\tstruct rev_info *revs = _data;\n \t\tstruct object_info oi = OBJECT_INFO_INIT;\n-\n \t\toi.typep = &type;\n+\n \t\tif (packed_object_info(the_repository, p, ofs, &oi) < 0) {\n \t\t\tdie(_(\"could not get type of object %s in pack %s\"),\n \t\t\t oid_to_hex(oid), p->pack_name);\n \t\t} else if (type == OBJ_COMMIT) {\n+\t\t\tstruct rev_info *revs = _data;\n \t\t\t/*\n \t\t\t * commits in included packs are used as starting points for the\n \t\t\t * subsequent revision walk\n-- \n2.49.0.230.ga662d77f78\n\n"},{"id":"516232","messageId":"6f8fe8a4e10198b0339337376279cff4ac654879.1744757204.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744757204.git.me@ttaylorr.com","subject":"[PATCH v3 3/9] pack-objects: factor out handling '--stdin-packs'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-15T22:46:54Z","receivedAt":"2025-04-15T22:46:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"At the bottom of cmd_pack_objects() we check which mode the command is\nrunning in (e.g., generating a cruft pack, handling '--stdin-packs',\nusing the internal rev-list, etc.) and handle the mode appropriately.\n\nThe '--stdin-packs' case is handled inline (dating back to its\nintroduction in 339bce27f4 (builtin/pack-objects.c: add '--stdin-packs'\noption, 2021-02-22)) since it is relatively short. Extract the body of\n\"if (stdin_packs)\" into its own function to prepare for the\nimplementation to become lengthier in a following commit.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 18 ++++++++++++------\n 1 file changed, 12 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 4ab695a3aa..a293267074 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3674,6 +3674,17 @@ static void read_packs_list_from_stdin(void)\n \tstring_list_clear(&exclude_packs, 0);\n }\n \n+static void add_unreachable_loose_objects(void);\n+\n+static void read_stdin_packs(int rev_list_unpacked)\n+{\n+\t/* avoids adding objects in excluded packs */\n+\tignore_packed_keep_in_core = 1;\n+\tread_packs_list_from_stdin();\n+\tif (rev_list_unpacked)\n+\t\tadd_unreachable_loose_objects();\n+}\n+\n static void add_cruft_object_entry(const struct object_id *oid, enum object_type type,\n \t\t\t\t struct packed_git *pack, off_t offset,\n \t\t\t\t const char *name, uint32_t mtime)\n@@ -3769,7 +3780,6 @@ static void mark_pack_kept_in_core(struct string_list *packs, unsigned keep)\n \t}\n }\n \n-static void add_unreachable_loose_objects(void);\n static void add_objects_in_unpacked_packs(void);\n \n static void enumerate_cruft_objects(void)\n@@ -4776,11 +4786,7 @@ int cmd_pack_objects(int argc,\n \t\tprogress_state = start_progress(the_repository,\n \t\t\t\t\t\t_(\"Enumerating objects\"), 0);\n \tif (stdin_packs) {\n-\t\t/* avoids adding objects in excluded packs */\n-\t\tignore_packed_keep_in_core = 1;\n-\t\tread_packs_list_from_stdin();\n-\t\tif (rev_list_unpacked)\n-\t\t\tadd_unreachable_loose_objects();\n+\t\tread_stdin_packs(rev_list_unpacked);\n \t} else if (cruft) {\n \t\tread_cruft_objects();\n \t} else if (!use_internal_rev_list) {\n-- \n2.49.0.230.ga662d77f78\n\n"},{"id":"516233","messageId":"2a235461a611d7abd90311c51174e2ed85eafa1b.1744757204.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744757204.git.me@ttaylorr.com","subject":"[PATCH v3 4/9] pack-objects: declare 'rev_info' for '--stdin-packs' earlier","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-15T22:46:58Z","receivedAt":"2025-04-15T22:47:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Once 'read_packs_list_from_stdin()' has called for_each_object_in_pack()\non each of the input packs, we do a reachability traversal to discover\nnames for any objects we picked up so we can generate name hash values\nand hopefully get higher quality deltas as a result.\n\nA future commit will change the purpose of this reachability traversal\nto find and pack objects which are reachable from commits in the input\npacks, but are packed in an unknown (not included nor excluded) pack.\n\nExtract the code which initializes and performs the reachability\ntraversal to take place in the caller, not the callee, which prepares us\nto share this code for the '--unpacked' case (see the function\nadd_unreachable_loose_objects() for more details).\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 71 +++++++++++++++++++++---------------------\n 1 file changed, 36 insertions(+), 35 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex a293267074..d60cb042c9 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3558,7 +3558,7 @@ static int pack_mtime_cmp(const void *_a, const void *_b)\n \t\treturn 0;\n }\n \n-static void read_packs_list_from_stdin(void)\n+static void read_packs_list_from_stdin(struct rev_info *revs)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct string_list include_packs = STRING_LIST_INIT_DUP;\n@@ -3566,24 +3566,6 @@ static void read_packs_list_from_stdin(void)\n \tstruct string_list_item *item = NULL;\n \n \tstruct packed_git *p;\n-\tstruct rev_info revs;\n-\n-\trepo_init_revisions(the_repository, &revs, NULL);\n-\t/*\n-\t * Use a revision walk to fill in the namehash of objects in the include\n-\t * packs. To save time, we'll avoid traversing through objects that are\n-\t * in excluded packs.\n-\t *\n-\t * That may cause us to avoid populating all of the namehash fields of\n-\t * all included objects, but our goal is best-effort, since this is only\n-\t * an optimization during delta selection.\n-\t */\n-\trevs.no_kept_objects = 1;\n-\trevs.keep_pack_cache_flags |= IN_CORE_KEEP_PACKS;\n-\trevs.blob_objects = 1;\n-\trevs.tree_objects = 1;\n-\trevs.tag_objects = 1;\n-\trevs.ignore_missing_links = 1;\n \n \twhile (strbuf_getline(&buf, stdin) != EOF) {\n \t\tif (!buf.len)\n@@ -3653,10 +3635,44 @@ static void read_packs_list_from_stdin(void)\n \t\tstruct packed_git *p = item->util;\n \t\tfor_each_object_in_pack(p,\n \t\t\t\t\tadd_object_entry_from_pack,\n-\t\t\t\t\t&revs,\n+\t\t\t\t\trevs,\n \t\t\t\t\tFOR_EACH_OBJECT_PACK_ORDER);\n \t}\n \n+\tstrbuf_release(&buf);\n+\tstring_list_clear(&include_packs, 0);\n+\tstring_list_clear(&exclude_packs, 0);\n+}\n+\n+static void add_unreachable_loose_objects(void);\n+\n+static void read_stdin_packs(int rev_list_unpacked)\n+{\n+\tstruct rev_info revs;\n+\n+\trepo_init_revisions(the_repository, &revs, NULL);\n+\t/*\n+\t * Use a revision walk to fill in the namehash of objects in the include\n+\t * packs. To save time, we'll avoid traversing through objects that are\n+\t * in excluded packs.\n+\t *\n+\t * That may cause us to avoid populating all of the namehash fields of\n+\t * all included objects, but our goal is best-effort, since this is only\n+\t * an optimization during delta selection.\n+\t */\n+\trevs.no_kept_objects = 1;\n+\trevs.keep_pack_cache_flags |= IN_CORE_KEEP_PACKS;\n+\trevs.blob_objects = 1;\n+\trevs.tree_objects = 1;\n+\trevs.tag_objects = 1;\n+\trevs.ignore_missing_links = 1;\n+\n+\t/* avoids adding objects in excluded packs */\n+\tignore_packed_keep_in_core = 1;\n+\tread_packs_list_from_stdin(&revs);\n+\tif (rev_list_unpacked)\n+\t\tadd_unreachable_loose_objects();\n+\n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n \ttraverse_commit_list(&revs,\n@@ -3668,21 +3684,6 @@ static void read_packs_list_from_stdin(void)\n \t\t\t stdin_packs_found_nr);\n \ttrace2_data_intmax(\"pack-objects\", the_repository, \"stdin_packs_hints\",\n \t\t\t stdin_packs_hints_nr);\n-\n-\tstrbuf_release(&buf);\n-\tstring_list_clear(&include_packs, 0);\n-\tstring_list_clear(&exclude_packs, 0);\n-}\n-\n-static void add_unreachable_loose_objects(void);\n-\n-static void read_stdin_packs(int rev_list_unpacked)\n-{\n-\t/* avoids adding objects in excluded packs */\n-\tignore_packed_keep_in_core = 1;\n-\tread_packs_list_from_stdin();\n-\tif (rev_list_unpacked)\n-\t\tadd_unreachable_loose_objects();\n }\n \n static void add_cruft_object_entry(const struct object_id *oid, enum object_type type,\n-- \n2.49.0.230.ga662d77f78\n\n"},{"id":"516234","messageId":"240e90b68d18b3231826d2a68e4e251c893e645a.1744757204.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744757204.git.me@ttaylorr.com","subject":"[PATCH v3 5/9] pack-objects: perform name-hash traversal for unpacked objects","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-15T22:47:01Z","receivedAt":"2025-04-15T22:47:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"With '--unpacked', pack-objects adds loose objects (which don't appear\nin any of the excluded packs from '--stdin-packs') to the output pack\nwithout considering them as reachability tips for the name-hash\ntraversal.\n\nThis was an oversight in the original implementation of '--stdin-packs',\nsince the code which enumerates and adds loose objects to the output\npack (`add_unreachable_loose_objects()`) did not have access to the\n'rev_info' struct found in `read_packs_list_from_stdin()`.\n\nExcluding unpacked objects from that traversal doesn't affect the\ncorrectness of the resulting pack, but it does make it harder to\ndiscover good deltas for loose objects.\n\nNow that the 'rev_info' struct is declared outside of\n`read_packs_list_from_stdin()`, we can pass it to\n`add_objects_in_unpacked_packs()` and add any loose objects as tips to\nthe above-mentioned traversal, in theory producing slightly tighter\npacks as a result.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 20 ++++++++++++--------\n 1 file changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex d60cb042c9..eb2a4099cc 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3644,7 +3644,7 @@ static void read_packs_list_from_stdin(struct rev_info *revs)\n \tstring_list_clear(&exclude_packs, 0);\n }\n \n-static void add_unreachable_loose_objects(void);\n+static void add_unreachable_loose_objects(struct rev_info *revs);\n \n static void read_stdin_packs(int rev_list_unpacked)\n {\n@@ -3671,7 +3671,7 @@ static void read_stdin_packs(int rev_list_unpacked)\n \tignore_packed_keep_in_core = 1;\n \tread_packs_list_from_stdin(&revs);\n \tif (rev_list_unpacked)\n-\t\tadd_unreachable_loose_objects();\n+\t\tadd_unreachable_loose_objects(&revs);\n \n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n@@ -3790,7 +3790,7 @@ static void enumerate_cruft_objects(void)\n \t\t\t\t\t\t_(\"Enumerating cruft objects\"), 0);\n \n \tadd_objects_in_unpacked_packs();\n-\tadd_unreachable_loose_objects();\n+\tadd_unreachable_loose_objects(NULL);\n \n \tstop_progress(&progress_state);\n }\n@@ -4068,8 +4068,9 @@ static void add_objects_in_unpacked_packs(void)\n }\n \n static int add_loose_object(const struct object_id *oid, const char *path,\n-\t\t\t void *data UNUSED)\n+\t\t\t void *data)\n {\n+\tstruct rev_info *revs = data;\n \tenum object_type type = oid_object_info(the_repository, oid, NULL);\n \n \tif (type < 0) {\n@@ -4090,6 +4091,10 @@ static int add_loose_object(const struct object_id *oid, const char *path,\n \t} else {\n \t\tadd_object_entry(oid, type, \"\", 0);\n \t}\n+\n+\tif (revs && type == OBJ_COMMIT)\n+\t\tadd_pending_oid(revs, NULL, oid, 0);\n+\n \treturn 0;\n }\n \n@@ -4098,11 +4103,10 @@ static int add_loose_object(const struct object_id *oid, const char *path,\n * add_object_entry will weed out duplicates, so we just add every\n * loose object we find.\n */\n-static void add_unreachable_loose_objects(void)\n+static void add_unreachable_loose_objects(struct rev_info *revs)\n {\n \tfor_each_loose_file_in_objdir(repo_get_object_directory(the_repository),\n-\t\t\t\t add_loose_object,\n-\t\t\t\t NULL, NULL, NULL);\n+\t\t\t\t add_loose_object, NULL, NULL, revs);\n }\n \n static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)\n@@ -4358,7 +4362,7 @@ static void get_object_list(struct rev_info *revs, int ac, const char **av)\n \tif (keep_unreachable)\n \t\tadd_objects_in_unpacked_packs();\n \tif (pack_loose_unreachable)\n-\t\tadd_unreachable_loose_objects();\n+\t\tadd_unreachable_loose_objects(NULL);\n \tif (unpack_unreachable)\n \t\tloosen_unused_packed_objects();\n \n-- \n2.49.0.230.ga662d77f78\n\n"},{"id":"516235","messageId":"9a18fa2e52bfe1bd98ea2d50b8e91509dcf67102.1744757204.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744757204.git.me@ttaylorr.com","subject":"[PATCH v3 6/9] pack-objects: fix typo in 'show_object_pack_hint()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-15T22:47:05Z","receivedAt":"2025-04-15T22:47:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Noticed-by: Elijah Newren \nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex eb2a4099cc..f06b359150 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3532,7 +3532,7 @@ static void show_object_pack_hint(struct object *object, const char *name,\n \t * would typically pick up during a reachability traversal.\n \t *\n \t * Make a best-effort attempt to fill in the ->hash and ->no_try_delta\n-\t * here using a now in order to perhaps improve the delta selection\n+\t * fields here in order to perhaps improve the delta selection\n \t * process.\n \t */\n \toe->hash = pack_name_hash_fn(name);\n-- \n2.49.0.230.ga662d77f78\n\n"},{"id":"516236","messageId":"6c997853f15deb03a7b55577316720c40573be9b.1744757204.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744757204.git.me@ttaylorr.com","subject":"[PATCH v3 7/9] pack-objects: swap 'show_{object,commit}_pack_hint'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-15T22:47:09Z","receivedAt":"2025-04-15T22:47:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"show_commit_pack_hint() has heretofore been a noop, so its position\nwithin its compilation unit only needs to appear before its first use.\n\nBut the following commit will sometimes have `show_commit_pack_hint()`\ncall `show_object_pack_hint()`, so reorder the former to appear after\nthe latter to minimize the code movement in that patch.\n\nSuggested-by: Elijah Newren \nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex f06b359150..f4009cd391 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3513,12 +3513,6 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n \treturn 0;\n }\n \n-static void show_commit_pack_hint(struct commit *commit UNUSED,\n-\t\t\t\t void *data UNUSED)\n-{\n-\t/* nothing to do; commits don't have a namehash */\n-}\n-\n static void show_object_pack_hint(struct object *object, const char *name,\n \t\t\t\t void *data UNUSED)\n {\n@@ -3541,6 +3535,12 @@ static void show_object_pack_hint(struct object *object, const char *name,\n \tstdin_packs_hints_nr++;\n }\n \n+static void show_commit_pack_hint(struct commit *commit UNUSED,\n+\t\t\t\t void *data UNUSED)\n+{\n+\t/* nothing to do; commits don't have a namehash */\n+}\n+\n static int pack_mtime_cmp(const void *_a, const void *_b)\n {\n \tstruct packed_git *a = ((const struct string_list_item*)_a)->util;\n-- \n2.49.0.230.ga662d77f78\n\n"},{"id":"516237","messageId":"0ff699f05636ec3373bbcd16cd082a402cda4c25.1744757204.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744757204.git.me@ttaylorr.com","subject":"[PATCH v3 8/9] pack-objects: introduce '--stdin-packs=follow'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-15T22:47:12Z","receivedAt":"2025-04-15T22:47:14Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When invoked with '--stdin-packs', pack-objects will generate a pack\nwhich contains the objects found in the \"included\" packs, less any\nobjects from \"excluded\" packs.\n\nPacks that exist in the repository but weren't specified as either\nincluded or excluded are in practice treated like the latter, at least\nin the sense that pack-objects won't include objects from those packs.\nThis behavior forces us to include any cruft pack(s) in a repository's\nmulti-pack index for the reasons described in ddee3703b3\n(builtin/repack.c: add cruft packs to MIDX during geometric repack,\n2022-05-20).\n\nThe full details are in ddee3703b3, but the gist is if you\nhave a once-unreachable object in a cruft pack which later becomes\nreachable via one or more commits in a pack generated with\n'--stdin-packs', you *have* to include that object in the MIDX via the\ncopy in the cruft pack, otherwise we cannot generate reachability\nbitmaps for any commits which reach that object.\n\nThis prepares us for new repacking behavior which will \"resurrect\"\nobjects found in cruft or otherwise unspecified packs when generating\nnew packs. In the context of geometric repacking, this may be used to\nmaintain a sequence of geometrically-repacked packs, the union of which\nis closed under reachability, even in the case described earlier.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-pack-objects.adoc | 10 +++-\n builtin/pack-objects.c | 83 +++++++++++++++++++++--------\n t/t5331-pack-objects-stdin.sh | 82 ++++++++++++++++++++++++++++\n 3 files changed, 152 insertions(+), 23 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.adoc b/Documentation/git-pack-objects.adoc\nindex 7f69ae4855..8f0cecaec9 100644\n--- a/Documentation/git-pack-objects.adoc\n+++ b/Documentation/git-pack-objects.adoc\n@@ -87,13 +87,21 @@ base-name::\n \treference was included in the resulting packfile. This\n \tcan be useful to send new tags to native Git clients.\n \n---stdin-packs::\n+--stdin-packs[=]::\n \tRead the basenames of packfiles (e.g., `pack-1234abcd.pack`)\n \tfrom the standard input, instead of object names or revision\n \targuments. The resulting pack contains all objects listed in the\n \tincluded packs (those not beginning with `^`), excluding any\n \tobjects listed in the excluded packs (beginning with `^`).\n +\n+When `mode` is \"follow\", objects from packs not listed on stdin receive\n+special treatment. Objects within unlisted packs will be included if\n+those objects are (1) reachable from the included packs, and (2) not\n+found in any excluded packs. This mode is useful, for example, to\n+resurrect once-unreachable objects found in cruft packs to generate\n+packs which are closed under reachability up to the boundary set by the\n+excluded packs.\n++\n Incompatible with `--revs`, or options that imply `--revs` (such as\n `--all`), with the exception of `--unpacked`, which is compatible.\n \ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex f4009cd391..67a22b2dc4 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -272,6 +272,12 @@ static struct oidmap configured_exclusions;\n static struct oidset excluded_by_config;\n static int name_hash_version = -1;\n \n+enum stdin_packs_mode {\n+\tSTDIN_PACKS_MODE_NONE,\n+\tSTDIN_PACKS_MODE_STANDARD,\n+\tSTDIN_PACKS_MODE_FOLLOW,\n+};\n+\n /**\n * Check whether the name_hash_version chosen by user input is appropriate,\n * and also validate whether it is compatible with other features.\n@@ -3514,31 +3520,44 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n }\n \n static void show_object_pack_hint(struct object *object, const char *name,\n-\t\t\t\t void *data UNUSED)\n+\t\t\t\t void *data)\n {\n-\tstruct object_entry *oe = packlist_find(&to_pack, &object->oid);\n-\tif (!oe)\n-\t\treturn;\n+\tenum stdin_packs_mode mode = *(enum stdin_packs_mode *)data;\n+\tif (mode == STDIN_PACKS_MODE_FOLLOW) {\n+\t\tadd_object_entry(&object->oid, object->type, name, 0);\n+\t} else {\n+\t\tstruct object_entry *oe = packlist_find(&to_pack, &object->oid);\n+\t\tif (!oe)\n+\t\t\treturn;\n \n-\t/*\n-\t * Our 'to_pack' list was constructed by iterating all objects packed in\n-\t * included packs, and so doesn't have a non-zero hash field that you\n-\t * would typically pick up during a reachability traversal.\n-\t *\n-\t * Make a best-effort attempt to fill in the ->hash and ->no_try_delta\n-\t * fields here in order to perhaps improve the delta selection\n-\t * process.\n-\t */\n-\toe->hash = pack_name_hash_fn(name);\n-\toe->no_try_delta = name && no_try_delta(name);\n+\t\t/*\n+\t\t * Our 'to_pack' list was constructed by iterating all\n+\t\t * objects packed in included packs, and so doesn't have\n+\t\t * a non-zero hash field that you would typically pick\n+\t\t * up during a reachability traversal.\n+\t\t *\n+\t\t * Make a best-effort attempt to fill in the ->hash and\n+\t\t * ->no_try_delta fields here in order to perhaps\n+\t\t * improve the delta selection process.\n+\t\t */\n+\t\toe->hash = pack_name_hash_fn(name);\n+\t\toe->no_try_delta = name && no_try_delta(name);\n \n-\tstdin_packs_hints_nr++;\n+\t\tstdin_packs_hints_nr++;\n+\t}\n }\n \n-static void show_commit_pack_hint(struct commit *commit UNUSED,\n-\t\t\t\t void *data UNUSED)\n+static void show_commit_pack_hint(struct commit *commit, void *data)\n {\n+\tenum stdin_packs_mode mode = *(enum stdin_packs_mode *)data;\n+\n+\tif (mode == STDIN_PACKS_MODE_FOLLOW) {\n+\t\tshow_object_pack_hint((struct object *)commit, \"\", data);\n+\t\treturn;\n+\t}\n+\n \t/* nothing to do; commits don't have a namehash */\n+\n }\n \n static int pack_mtime_cmp(const void *_a, const void *_b)\n@@ -3646,7 +3665,7 @@ static void read_packs_list_from_stdin(struct rev_info *revs)\n \n static void add_unreachable_loose_objects(struct rev_info *revs);\n \n-static void read_stdin_packs(int rev_list_unpacked)\n+static void read_stdin_packs(enum stdin_packs_mode mode, int rev_list_unpacked)\n {\n \tstruct rev_info revs;\n \n@@ -3678,7 +3697,7 @@ static void read_stdin_packs(int rev_list_unpacked)\n \ttraverse_commit_list(&revs,\n \t\t\t show_commit_pack_hint,\n \t\t\t show_object_pack_hint,\n-\t\t\t NULL);\n+\t\t\t &mode);\n \n \ttrace2_data_intmax(\"pack-objects\", the_repository, \"stdin_packs_found\",\n \t\t\t stdin_packs_found_nr);\n@@ -4469,6 +4488,23 @@ static int is_not_in_promisor_pack(struct commit *commit, void *data) {\n \treturn is_not_in_promisor_pack_obj((struct object *) commit, data);\n }\n \n+static int parse_stdin_packs_mode(const struct option *opt, const char *arg,\n+\t\t\t\t int unset)\n+{\n+\tenum stdin_packs_mode *mode = opt->value;\n+\n+\tif (unset)\n+\t\t*mode = STDIN_PACKS_MODE_NONE;\n+\telse if (!arg || !*arg)\n+\t\t*mode = STDIN_PACKS_MODE_STANDARD;\n+\telse if (!strcmp(arg, \"follow\"))\n+\t\t*mode = STDIN_PACKS_MODE_FOLLOW;\n+\telse\n+\t\tdie(_(\"invalid value for '%s': '%s'\"), opt->long_name, arg);\n+\n+\treturn 0;\n+}\n+\n int cmd_pack_objects(int argc,\n \t\t const char **argv,\n \t\t const char *prefix,\n@@ -4480,7 +4516,7 @@ int cmd_pack_objects(int argc,\n \tstruct strvec rp = STRVEC_INIT;\n \tint rev_list_unpacked = 0, rev_list_all = 0, rev_list_reflog = 0;\n \tint rev_list_index = 0;\n-\tint stdin_packs = 0;\n+\tenum stdin_packs_mode stdin_packs = STDIN_PACKS_MODE_NONE;\n \tstruct string_list keep_pack_list = STRING_LIST_INIT_NODUP;\n \tstruct list_objects_filter_options filter_options =\n \t\tLIST_OBJECTS_FILTER_INIT;\n@@ -4535,6 +4571,9 @@ int cmd_pack_objects(int argc,\n \t\tOPT_SET_INT_F(0, \"indexed-objects\", &rev_list_index,\n \t\t\t N_(\"include objects referred to by the index\"),\n \t\t\t 1, PARSE_OPT_NONEG),\n+\t\tOPT_CALLBACK_F(0, \"stdin-packs\", &stdin_packs, N_(\"mode\"),\n+\t\t\t N_(\"read packs from stdin\"),\n+\t\t\t PARSE_OPT_OPTARG, parse_stdin_packs_mode),\n \t\tOPT_BOOL(0, \"stdin-packs\", &stdin_packs,\n \t\t\t N_(\"read packs from stdin\")),\n \t\tOPT_BOOL(0, \"stdout\", &pack_to_stdout,\n@@ -4791,7 +4830,7 @@ int cmd_pack_objects(int argc,\n \t\tprogress_state = start_progress(the_repository,\n \t\t\t\t\t\t_(\"Enumerating objects\"), 0);\n \tif (stdin_packs) {\n-\t\tread_stdin_packs(rev_list_unpacked);\n+\t\tread_stdin_packs(stdin_packs, rev_list_unpacked);\n \t} else if (cruft) {\n \t\tread_cruft_objects();\n \t} else if (!use_internal_rev_list) {\ndiff --git a/t/t5331-pack-objects-stdin.sh b/t/t5331-pack-objects-stdin.sh\nindex 8fd07deb8d..60a2b4bc07 100755\n--- a/t/t5331-pack-objects-stdin.sh\n+++ b/t/t5331-pack-objects-stdin.sh\n@@ -236,4 +236,86 @@ test_expect_success 'pack-objects --stdin with packfiles from main and alternate\n \ttest_cmp expected-objects actual-objects\n '\n \n+packdir=.git/objects/pack\n+\n+objects_in_packs () {\n+\tfor p in \"$@\"\n+\tdo\n+\t\tgit show-index <\"$packdir/pack-$p.idx\" || return 1\n+\tdone >objects.raw &&\n+\n+\tcut -d' ' -f2 objects.raw | sort &&\n+\trm -f objects.raw\n+}\n+\n+test_expect_success '--stdin-packs=follow walks into unknown packs' '\n+\ttest_when_finished \"rm -fr repo\" &&\n+\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\tfor c in A B C D\n+\t\tdo\n+\t\t\ttest_commit \"$c\" || return 1\n+\t\tdone &&\n+\n+\t\tA=\"$(echo A | git pack-objects --revs $packdir/pack)\" &&\n+\t\tB=\"$(echo A..B | git pack-objects --revs $packdir/pack)\" &&\n+\t\tC=\"$(echo B..C | git pack-objects --revs $packdir/pack)\" &&\n+\t\tD=\"$(echo C..D | git pack-objects --revs $packdir/pack)\" &&\n+\t\ttest_commit E &&\n+\n+\t\tgit prune-packed &&\n+\n+\t\tcat >in <<-EOF &&\n+\t\tpack-$B.pack\n+\t\t^pack-$C.pack\n+\t\tpack-$D.pack\n+\t\tEOF\n+\n+\t\t# With just --stdin-packs, pack \"A\" is unknown to us, so\n+\t\t# only objects from packs \"B\" and \"D\" are included in\n+\t\t# the output pack.\n+\t\tP=$(git pack-objects --stdin-packs $packdir/pack expect &&\n+\t\tobjects_in_packs $P >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\t# But with --stdin-packs=follow, objects from both\n+\t\t# included packs reach objects from the unknown pack, so\n+\t\t# objects from pack \"A\" is included in the output pack\n+\t\t# in addition to the above.\n+\t\tP=$(git pack-objects --stdin-packs=follow $packdir/pack expect &&\n+\t\tobjects_in_packs $P >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\t# And with --unpacked, we will pick up objects from unknown\n+\t\t# packs that are reachable from loose objects. Loose object E\n+\t\t# reaches objects in pack A, but there are three excluded packs\n+\t\t# in between.\n+\t\t#\n+\t\t# The resulting pack should include objects reachable from E\n+\t\t# that are not present in packs B, C, or D, along with those\n+\t\t# present in pack A.\n+\t\tcat >in <<-EOF &&\n+\t\t^pack-$B.pack\n+\t\t^pack-$C.pack\n+\t\t^pack-$D.pack\n+\t\tEOF\n+\n+\t\tP=$(git pack-objects --stdin-packs=follow --unpacked \\\n+\t\t\t$packdir/pack expect.raw &&\n+\t\tsort expect.raw >expect &&\n+\t\tobjects_in_packs $P >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \n2.49.0.230.ga662d77f78\n\n"},{"id":"516238","messageId":"58891101f377267df120dc4a9edea2997296dbec.1744757204.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744757204.git.me@ttaylorr.com","subject":"[PATCH v3 9/9] repack: exclude cruft pack(s) from the MIDX where possible","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-15T22:47:15Z","receivedAt":"2025-04-15T22:47:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In ddee3703b3 (builtin/repack.c: add cruft packs to MIDX during\ngeometric repack, 2022-05-20), repack began adding cruft pack(s) to the\nMIDX with '--write-midx' to ensure that the resulting MIDX was always\nclosed under reachability in order to generate reachability bitmaps.\n\nSuppose (prior to this patch) you have a once-unreachable object packed\nin a cruft pack, which later on becomes reachable from one or more\nobjects in a geometrically repacked pack. That once-unreachable object\n*won't* appear in the new pack, since the cruft pack was specified as\nneither included nor excluded to 'pack-objects --stdin-packs'. If the\nnew pack is included in a MIDX without the cruft pack, then trying to\ngenerate bitmaps for that MIDX may fail. This happens when the bitmap\nselection process picks one or more commits which reach the\nonce-unreachable objects, commit ddee3703b3 ensures that the MIDX will\nbe closed under reachability. Without it, we would fail to generate a\nMIDX bitmap.\n\nddee3703b3 alludes to the fact that this is sub-optimal by saying\n\n [...] it's desirable to avoid including cruft packs in the MIDX\n because it causes the MIDX to store a bunch of objects which are\n likely to get thrown away.\n\n, which is true, but hides an even larger problem. If repositories\nrarely prune their unreachable objects and/or have many of them, the\nMIDX must keep track of a large number of objects which bloats the MIDX\nand slows down object lookup.\n\nThis is doubly unfortunate because the vast majority of objects in cruft\npack(s) are unlikely to be read. But any object lookups that go through\nthe MIDX must binary search over them anyway, slowing down object\nlookups using the MIDX.\n\nThis patch causes geometrically-repacked packs to contain a copy of any\nonce-unreachable object(s) with 'git pack-objects --stdin-packs=follow',\nallowing us to avoid including any cruft packs in the MIDX. This is\nbecause a sequence of geometrically-repacked packs that were all\ngenerated with '--stdin-packs=follow' are guaranteed to have their union\nbe closed under reachability.\n\nNote that you cannot guarantee that a collection of packs is closed\nunder reachability if not all of them were generated with \"following\" as\nabove. One tell-tale sign that not all geometrically-repacked packs in\nthe MIDX were generated with \"following\" is to see if there is a pack in\nthe existing MIDX that is not going to be somehow represented (either\nverbatim or as part of a geometric rollup) in the new MIDX.\n\nIf there is, then starting to generate packs with \"following\" during\ngeometric repacking won't work, since it's open to the same race as\ndescribed above.\n\nBut if you're starting from scratch (e.g., building the first MIDX after\nan all-into-one '--cruft' repack), then you can guarantee that the union\nof subsequently generated packs from geometric repacking *is* closed\nunder reachability.\n\nDetect when this is the case and avoid including cruft packs in the MIDX\nwhere possible. The existing behavior remains the default, and the new\nbehavior is available with the config 'repack.midxMustIncludeCruft' set\nto 'false'.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/config/repack.adoc | 7 ++\n builtin/repack.c | 163 +++++++++++++++++++++++++++----\n t/t7704-repack-cruft.sh | 90 +++++++++++++++++\n 3 files changed, 242 insertions(+), 18 deletions(-)\n\ndiff --git a/Documentation/config/repack.adoc b/Documentation/config/repack.adoc\nindex c79af6d7b8..e9e78dcb19 100644\n--- a/Documentation/config/repack.adoc\n+++ b/Documentation/config/repack.adoc\n@@ -39,3 +39,10 @@ repack.cruftThreads::\n \ta cruft pack and the respective parameters are not given over\n \tthe command line. See similarly named `pack.*` configuration\n \tvariables for defaults and meaning.\n+\n+repack.midxMustContainCruft::\n+\tWhen set to true, linkgit:git-repack[1] will unconditionally include\n+\tcruft pack(s), if any, in the multi-pack index when invoked with\n+\t`--write-midx`. When false, cruft packs are only included in the MIDX\n+\twhen necessary (e.g., because they might be required to form a\n+\treachability closure with MIDX bitmaps). Defaults to true.\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex f3330ade7b..c9e2e3d04d 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -39,6 +39,7 @@ static int write_bitmaps = -1;\n static int use_delta_islands;\n static int run_update_server_info = 1;\n static char *packdir, *packtmp_name, *packtmp;\n+static int midx_must_contain_cruft = 1;\n \n static const char *const git_repack_usage[] = {\n \tN_(\"git repack [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\\n\"\n@@ -107,6 +108,10 @@ static int repack_config(const char *var, const char *value,\n \t\tfree(cruft_po_args->threads);\n \t\treturn git_config_string(&cruft_po_args->threads, var, value);\n \t}\n+\tif (!strcmp(var, \"repack.midxmustcontaincruft\")) {\n+\t\tmidx_must_contain_cruft = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n \treturn git_default_config(var, value, ctx, cb);\n }\n \n@@ -687,6 +692,77 @@ static void free_pack_geometry(struct pack_geometry *geometry)\n \tfree(geometry->pack);\n }\n \n+static int midx_has_unknown_packs(char **midx_pack_names,\n+\t\t\t\t size_t midx_pack_names_nr,\n+\t\t\t\t struct string_list *include,\n+\t\t\t\t struct pack_geometry *geometry,\n+\t\t\t\t struct existing_packs *existing)\n+{\n+\tsize_t i;\n+\n+\tstring_list_sort(include);\n+\n+\tfor (i = 0; i < midx_pack_names_nr; i++) {\n+\t\tconst char *pack_name = midx_pack_names[i];\n+\n+\t\t/*\n+\t\t * Determine whether or not each MIDX'd pack from the existing\n+\t\t * MIDX (if any) is represented in the new MIDX. For each pack\n+\t\t * in the MIDX, it must either be:\n+\t\t *\n+\t\t * - In the \"include\" list of packs to be included in the new\n+\t\t * MIDX. Note this function is called before the include\n+\t\t * list is populated with any cruft pack(s).\n+\t\t *\n+\t\t * - Below the geometric split line (if using pack geometry),\n+\t\t * indicating that the pack won't be included in the new\n+\t\t * MIDX, but its contents were rolled up as part of the\n+\t\t * geometric repack.\n+\t\t *\n+\t\t * - In the existing non-kept packs list (if not using pack\n+\t\t * geometry), and marked as non-deleted.\n+\t\t */\n+\t\tif (string_list_has_string(include, pack_name)) {\n+\t\t\tcontinue;\n+\t\t} else if (geometry) {\n+\t\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\t\tuint32_t j;\n+\n+\t\t\tfor (j = 0; j < geometry->split; j++) {\n+\t\t\t\tstrbuf_reset(&buf);\n+\t\t\t\tstrbuf_addstr(&buf, pack_basename(geometry->pack[j]));\n+\t\t\t\tstrbuf_strip_suffix(&buf, \".pack\");\n+\t\t\t\tstrbuf_addstr(&buf, \".idx\");\n+\n+\t\t\t\tif (!strcmp(pack_name, buf.buf)) {\n+\t\t\t\t\tstrbuf_release(&buf);\n+\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\tstrbuf_release(&buf);\n+\n+\t\t\tif (j < geometry->split)\n+\t\t\t\tcontinue;\n+\t\t} else {\n+\t\t\tstruct string_list_item *item;\n+\n+\t\t\titem = string_list_lookup(&existing->non_kept_packs,\n+\t\t\t\t\t\t pack_name);\n+\t\t\tif (item && !pack_is_marked_for_deletion(item))\n+\t\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/*\n+\t\t * If we got to this point, the MIDX includes some pack that we\n+\t\t * don't know about.\n+\t\t */\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n struct midx_snapshot_ref_data {\n \tstruct tempfile *f;\n \tstruct oidset seen;\n@@ -755,6 +831,8 @@ static void midx_snapshot_refs(struct tempfile *f)\n \n static void midx_included_packs(struct string_list *include,\n \t\t\t\tstruct existing_packs *existing,\n+\t\t\t\tchar **midx_pack_names,\n+\t\t\t\tsize_t midx_pack_names_nr,\n \t\t\t\tstruct string_list *names,\n \t\t\t\tstruct pack_geometry *geometry)\n {\n@@ -808,26 +886,56 @@ static void midx_included_packs(struct string_list *include,\n \t\t}\n \t}\n \n-\tfor_each_string_list_item(item, &existing->cruft_packs) {\n+\tif (midx_must_contain_cruft ||\n+\t midx_has_unknown_packs(midx_pack_names, midx_pack_names_nr,\n+\t\t\t\t include, geometry, existing)) {\n \t\t/*\n-\t\t * When doing a --geometric repack, there is no need to check\n-\t\t * for deleted packs, since we're by definition not doing an\n-\t\t * ALL_INTO_ONE repack (hence no packs will be deleted).\n-\t\t * Otherwise we must check for and exclude any packs which are\n-\t\t * enqueued for deletion.\n+\t\t * If there are one or more unknown pack(s) present (see\n+\t\t * midx_has_unknown_packs() for what makes a pack\n+\t\t * \"unknown\") in the MIDX before the repack, keep them\n+\t\t * as they may be required to form a reachability\n+\t\t * closure if the MIDX is bitmapped.\n \t\t *\n-\t\t * So we could omit the conditional below in the --geometric\n-\t\t * case, but doing so is unnecessary since no packs are marked\n-\t\t * as pending deletion (since we only call\n-\t\t * `mark_packs_for_deletion()` when doing an all-into-one\n-\t\t * repack).\n+\t\t * For example, a cruft pack can be required to form a\n+\t\t * reachability closure if the MIDX is bitmapped and one\n+\t\t * or more of the bitmap's selected commits reaches a\n+\t\t * once-cruft object that was later made reachable.\n \t\t */\n-\t\tif (pack_is_marked_for_deletion(item))\n-\t\t\tcontinue;\n+\t\tfor_each_string_list_item(item, &existing->cruft_packs) {\n+\t\t\t/*\n+\t\t\t * When doing a --geometric repack, there is no\n+\t\t\t * need to check for deleted packs, since we're\n+\t\t\t * by definition not doing an ALL_INTO_ONE\n+\t\t\t * repack (hence no packs will be deleted).\n+\t\t\t * Otherwise we must check for and exclude any\n+\t\t\t * packs which are enqueued for deletion.\n+\t\t\t *\n+\t\t\t * So we could omit the conditional below in the\n+\t\t\t * --geometric case, but doing so is unnecessary\n+\t\t\t * since no packs are marked as pending\n+\t\t\t * deletion (since we only call\n+\t\t\t * `mark_packs_for_deletion()` when doing an\n+\t\t\t * all-into-one repack).\n+\t\t\t */\n+\t\t\tif (pack_is_marked_for_deletion(item))\n+\t\t\t\tcontinue;\n \n-\t\tstrbuf_reset(&buf);\n-\t\tstrbuf_addf(&buf, \"%s.idx\", item->string);\n-\t\tstring_list_insert(include, buf.buf);\n+\t\t\tstrbuf_reset(&buf);\n+\t\t\tstrbuf_addf(&buf, \"%s.idx\", item->string);\n+\t\t\tstring_list_insert(include, buf.buf);\n+\t\t}\n+\t} else {\n+\t\t/*\n+\t\t * Modern versions of Git (with the appropriate\n+\t\t * configuration setting) will write new copies of\n+\t\t * once-cruft objects when doing a --geometric repack.\n+\t\t *\n+\t\t * If the MIDX has no cruft pack, new packs written\n+\t\t * during a --geometric repack will not rely on the\n+\t\t * cruft pack to form a reachability closure, so we can\n+\t\t * avoid including them in the MIDX in that case.\n+\t\t */\n+\t\t;\n \t}\n \n \tstrbuf_release(&buf);\n@@ -1142,6 +1250,8 @@ int cmd_repack(int argc,\n \tstruct tempfile *refs_snapshot = NULL;\n \tint i, ext, ret;\n \tint show_progress;\n+\tchar **midx_pack_names = NULL;\n+\tsize_t midx_pack_names_nr = 0;\n \n \t/* variables to be filled by option parsing */\n \tint delete_redundant = 0;\n@@ -1356,7 +1466,10 @@ int cmd_repack(int argc,\n \t\t !(pack_everything & PACK_CRUFT))\n \t\t\tstrvec_push(&cmd.args, \"--pack-loose-unreachable\");\n \t} else if (geometry.split_factor) {\n-\t\tstrvec_push(&cmd.args, \"--stdin-packs\");\n+\t\tif (midx_must_contain_cruft)\n+\t\t\tstrvec_push(&cmd.args, \"--stdin-packs\");\n+\t\telse\n+\t\t\tstrvec_push(&cmd.args, \"--stdin-packs=follow\");\n \t\tstrvec_push(&cmd.args, \"--unpacked\");\n \t} else {\n \t\tstrvec_push(&cmd.args, \"--unpacked\");\n@@ -1478,6 +1591,16 @@ int cmd_repack(int argc,\n \n \tstring_list_sort(&names);\n \n+\tif (get_local_multi_pack_index(the_repository)) {\n+\t\tuint32_t i;\n+\t\tstruct multi_pack_index *m =\n+\t\t\tget_local_multi_pack_index(the_repository);\n+\n+\t\tALLOC_ARRAY(midx_pack_names, m->num_packs);\n+\t\tfor (i = 0; i < m->num_packs; i++)\n+\t\t\tmidx_pack_names[midx_pack_names_nr++] = xstrdup(m->pack_names[i]);\n+\t}\n+\n \tclose_object_store(the_repository->objects);\n \n \t/*\n@@ -1519,7 +1642,8 @@ int cmd_repack(int argc,\n \n \tif (write_midx) {\n \t\tstruct string_list include = STRING_LIST_INIT_DUP;\n-\t\tmidx_included_packs(&include, &existing, &names, &geometry);\n+\t\tmidx_included_packs(&include, &existing, midx_pack_names,\n+\t\t\t\t midx_pack_names_nr, &names, &geometry);\n \n \t\tret = write_midx_included_packs(&include, &geometry, &names,\n \t\t\t\t\t\trefs_snapshot ? get_tempfile_path(refs_snapshot) : NULL,\n@@ -1570,6 +1694,9 @@ int cmd_repack(int argc,\n \tstring_list_clear(&names, 1);\n \texisting_packs_release(&existing);\n \tfree_pack_geometry(&geometry);\n+\tfor (size_t i = 0; i < midx_pack_names_nr; i++)\n+\t\tfree(midx_pack_names[i]);\n+\tfree(midx_pack_names);\n \tpack_objects_args_release(&po_args);\n \tpack_objects_args_release(&cruft_po_args);\n \ndiff --git a/t/t7704-repack-cruft.sh b/t/t7704-repack-cruft.sh\nindex 8aebfb45f5..2b0a55f8fd 100755\n--- a/t/t7704-repack-cruft.sh\n+++ b/t/t7704-repack-cruft.sh\n@@ -724,4 +724,94 @@ test_expect_success 'cruft repack respects --quiet' '\n \t)\n '\n \n+setup_cruft_exclude_tests() {\n+\tgit init \"$1\" &&\n+\t(\n+\t\tcd \"$1\" &&\n+\n+\t\tgit config repack.midxMustContainCruft false &&\n+\n+\t\ttest_commit one &&\n+\n+\t\ttest_commit --no-tag two &&\n+\t\ttwo=\"$(git rev-parse HEAD)\" &&\n+\t\ttest_commit --no-tag three &&\n+\t\tthree=\"$(git rev-parse HEAD)\" &&\n+\t\tgit reset --hard one &&\n+\t\tgit reflog expire --all --expire=all &&\n+\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack --cruft -d &&\n+\n+\t\tgit merge $two &&\n+\t\ttest_commit four\n+\t)\n+}\n+\n+test_expect_success 'repack --write-midx excludes cruft where possible' '\n+\tsetup_cruft_exclude_tests exclude-cruft-when-possible &&\n+\t(\n+\t\tcd exclude-cruft-when-possible &&\n+\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 \\\n+\t\tgit repack -d --geometric=2 --write-midx --write-bitmap-index &&\n+\n+\t\ttest-tool read-midx --show-objects $objdir >midx &&\n+\t\tcruft=\"$(ls $packdir/*.mtimes)\" &&\n+\t\ttest_grep ! \"$(basename \"$cruft\" .mtimes).idx\" midx &&\n+\n+\t\tgit rev-list --all --objects --no-object-names >reachable.raw &&\n+\t\tsort reachable.raw >reachable.objects &&\n+\t\tawk \"/\\.pack$/ { print \\$1 }\" midx.objects &&\n+\n+\t\ttest_cmp reachable.objects midx.objects\n+\t)\n+'\n+\n+test_expect_success 'repack --write-midx includes cruft when instructed' '\n+\tsetup_cruft_exclude_tests exclude-cruft-when-instructed &&\n+\t(\n+\t\tcd exclude-cruft-when-instructed &&\n+\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 \\\n+\t\tgit -c repack.midxMustContainCruft=true repack \\\n+\t\t\t-d --geometric=2 --write-midx --write-bitmap-index &&\n+\n+\t\ttest-tool read-midx --show-objects $objdir >midx &&\n+\t\tcruft=\"$(ls $packdir/*.mtimes)\" &&\n+\t\ttest_grep \"$(basename \"$cruft\" .mtimes).idx\" midx &&\n+\n+\t\tgit cat-file --batch-check=\"%(objectname)\" --batch-all-objects \\\n+\t\t\t>all.objects &&\n+\t\tawk \"/\\.pack$/ { print \\$1 }\" midx.objects &&\n+\n+\t\ttest_cmp all.objects midx.objects\n+\t)\n+'\n+\n+test_expect_success 'repack --write-midx includes cruft when necessary' '\n+\tsetup_cruft_exclude_tests exclude-cruft-when-necessary &&\n+\t(\n+\t\tcd exclude-cruft-when-necessary &&\n+\n+\t\ttest_path_is_file $(ls $packdir/pack-*.mtimes) &&\n+\t\tls $packdir/pack-*.idx | sort >packs.all &&\n+\t\tgrep -o \"pack-.*\\.idx$\" packs.all >in &&\n+\n+\t\tgit multi-pack-index write --stdin-packs --bitmap midx &&\n+\t\tawk \"/\\.pack$/ { print \\$1 }\" midx.objects &&\n+\t\tgit cat-file --batch-all-objects --batch-check=\"%(objectname)\" \\\n+\t\t\t>expect.objects &&\n+\t\ttest_cmp expect.objects midx.objects &&\n+\n+\t\tgrep \"^pack-\" midx >midx.packs &&\n+\t\ttest_line_count = \"$(($(wc -l writes:\n\n> In add_object_entry_from_pack() we declare 'revs' (given to us through\n> the miscellaneous context argument) earlier in the \"if (p)\" conditional\n> than is necessary. Move it down as far as it can go to reduce its\n> scope.\n\nThat makes sense, but ...\n\n> Signed-off-by: Taylor Blau \n> ---\n> builtin/pack-objects.c | 4 ++--\n> 1 file changed, 2 insertions(+), 2 deletions(-)\n>\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 20dd870bbf..4ab695a3aa 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -3490,14 +3490,14 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n> \t\treturn 0;\n> \n> \tif (p) {\n> -\t\tstruct rev_info *revs = _data;\n> \t\tstruct object_info oi = OBJECT_INFO_INIT;\n> -\n> \t\toi.typep = &type;\n> +\n\nIsn't this change about spacing around oi's decl and the first\nstatement in the block strictly worsening the code? At least it is\nan unrelated change.\n\n> \t\tif (packed_object_info(the_repository, p, ofs, &oi) < 0) {\n> \t\t\tdie(_(\"could not get type of object %s in pack %s\"),\n> \t\t\t oid_to_hex(oid), p->pack_name);\n> \t\t} else if (type == OBJ_COMMIT) {\n> +\t\t\tstruct rev_info *revs = _data;\n> \t\t\t/*\n> \t\t\t * commits in included packs are used as starting points for the\n> \t\t\t * subsequent revision walk\n"},{"id":"516245","messageId":"xmqqplhckjju.fsf@gitster.g","threadId":"63286","inReplyTo":"6f8fe8a4e10198b0339337376279cff4ac654879.1744757204.git.me@ttaylorr.com","subject":"Re: [PATCH v3 3/9] pack-objects: factor out handling '--stdin-packs'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-04-16T00:59:49Z","receivedAt":"2025-04-16T00:59:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> At the bottom of cmd_pack_objects() we check which mode the command is\n> running in (e.g., generating a cruft pack, handling '--stdin-packs',\n> using the internal rev-list, etc.) and handle the mode appropriately.\n>\n> The '--stdin-packs' case is handled inline (dating back to its\n> introduction in 339bce27f4 (builtin/pack-objects.c: add '--stdin-packs'\n> option, 2021-02-22)) since it is relatively short. Extract the body of\n> \"if (stdin_packs)\" into its own function to prepare for the\n> implementation to become lengthier in a following commit.\n>\n> Signed-off-by: Taylor Blau \n> ---\n> builtin/pack-objects.c | 18 ++++++++++++------\n> 1 file changed, 12 insertions(+), 6 deletions(-)\n\nMakes sense.\n\n>\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 4ab695a3aa..a293267074 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -3674,6 +3674,17 @@ static void read_packs_list_from_stdin(void)\n> \tstring_list_clear(&exclude_packs, 0);\n> }\n> \n> +static void add_unreachable_loose_objects(void);\n> +\n> +static void read_stdin_packs(int rev_list_unpacked)\n> +{\n> +\t/* avoids adding objects in excluded packs */\n> +\tignore_packed_keep_in_core = 1;\n> +\tread_packs_list_from_stdin();\n> +\tif (rev_list_unpacked)\n> +\t\tadd_unreachable_loose_objects();\n> +}\n> +\n> static void add_cruft_object_entry(const struct object_id *oid, enum object_type type,\n> \t\t\t\t struct packed_git *pack, off_t offset,\n> \t\t\t\t const char *name, uint32_t mtime)\n> @@ -3769,7 +3780,6 @@ static void mark_pack_kept_in_core(struct string_list *packs, unsigned keep)\n> \t}\n> }\n> \n> -static void add_unreachable_loose_objects(void);\n> static void add_objects_in_unpacked_packs(void);\n> \n> static void enumerate_cruft_objects(void)\n> @@ -4776,11 +4786,7 @@ int cmd_pack_objects(int argc,\n> \t\tprogress_state = start_progress(the_repository,\n> \t\t\t\t\t\t_(\"Enumerating objects\"), 0);\n> \tif (stdin_packs) {\n> -\t\t/* avoids adding objects in excluded packs */\n> -\t\tignore_packed_keep_in_core = 1;\n> -\t\tread_packs_list_from_stdin();\n> -\t\tif (rev_list_unpacked)\n> -\t\t\tadd_unreachable_loose_objects();\n> +\t\tread_stdin_packs(rev_list_unpacked);\n> \t} else if (cruft) {\n> \t\tread_cruft_objects();\n> \t} else if (!use_internal_rev_list) {\n"},{"id":"516248","messageId":"CABPp-BGbScKfZsADuwkHNU5L7FMnXTwv8aUSwV-Cor-VU3hsiw@mail.gmail.com","threadId":"63286","inReplyTo":"Z/7FdgYID9I1qR7K@nand.local","subject":"Re: [PATCH v2 7/8] pack-objects: introduce '--stdin-packs=follow'","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-04-16T05:26:33Z","receivedAt":"2025-04-16T05:26:45Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Apr 15, 2025 at 1:45 PM Taylor Blau wrote:\n\n> > > @@ -4467,6 +4484,23 @@ static int is_not_in_promisor_pack(struct commit *commit, void *data) {\n> > > return is_not_in_promisor_pack_obj((struct object *) commit, data);\n> > > }\n> > >\n> > > +static int parse_stdin_packs_mode(const struct option *opt, const char *arg,\n> > > + int unset)\n> > > +{\n> > > + enum stdin_packs_mode *mode = opt->value;\n> > > +\n> > > + if (unset)\n> > > + *mode = STDIN_PACKS_MODE_NONE;\n> > > + else if (!arg || !*arg)\n> > > + *mode = STDIN_PACKS_MODE_STANDARD;\n> >\n> > I don't understand why you have both a None mode and a Standard mode,\n> > especially since the implementation seems to only care about whether\n> > or not the Follow mode has been set. Shouldn't these both be setting\n> > mode to the same value?\n>\n> I'm not sure I follow your question... stdin_packs is a tri-state. It\n> can be off, on in standard/legacy mode, or on in follow mode.\n\nI was just confused. I looked in the code for\nSTDIN_PACKS_MODE_{NONE,STANDARD,FOLLOW}, and other than initial setup,\nonly the _FOLLOW variant was used anywhere. I overlooked the \"if\n(stdin_packs\" usage, which is what further distinguishes between _NONE\nand _STANDARD. Sorry.\n"},{"id":"516250","messageId":"CABPp-BG-=6uRtHXrzSOMw=J2Shg_vfXhLzwWDREy7OAQg_Ak6w@mail.gmail.com","threadId":"63286","inReplyTo":"986bef29b5f33d32fd366aa9370d439175a9b605.1744757204.git.me@ttaylorr.com","subject":"Re: [PATCH v3 2/9] pack-objects: limit scope in 'add_object_entry_from_pack()'","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-04-16T05:31:26Z","receivedAt":"2025-04-16T05:31:38Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Apr 15, 2025 at 3:46 PM Taylor Blau wrote:\n>\n> In add_object_entry_from_pack() we declare 'revs' (given to us through\n> the miscellaneous context argument) earlier in the \"if (p)\" conditional\n> than is necessary. Move it down as far as it can go to reduce its\n> scope.\n>\n> Signed-off-by: Taylor Blau \n> ---\n> builtin/pack-objects.c | 4 ++--\n> 1 file changed, 2 insertions(+), 2 deletions(-)\n>\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 20dd870bbf..4ab695a3aa 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -3490,14 +3490,14 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n> return 0;\n>\n> if (p) {\n> - struct rev_info *revs = _data;\n\nThis change is half of what you mention in your commit message.\n\n> struct object_info oi = OBJECT_INFO_INIT;\n> -\n> oi.typep = &type;\n> +\n\nThis is an unrelated, distracting change that I think was accidental\nand came from trying to back out the other change that was part of\nv1/v2 but not quite backing it out completely.\n\n> if (packed_object_info(the_repository, p, ofs, &oi) < 0) {\n> die(_(\"could not get type of object %s in pack %s\"),\n> oid_to_hex(oid), p->pack_name);\n> } else if (type == OBJ_COMMIT) {\n> + struct rev_info *revs = _data;\n\nThis is the other half of what you mention in the commit message.\n\n> /*\n> * commits in included packs are used as starting points for the\n> * subsequent revision walk\n> --\n> 2.49.0.230.ga662d77f78\n"},{"id":"516251","messageId":"CABPp-BF0hLbMtsiqmQTO9nWvjkk9W513d4TSzj7BQoBC9Baw7A@mail.gmail.com","threadId":"63286","inReplyTo":"9a18fa2e52bfe1bd98ea2d50b8e91509dcf67102.1744757204.git.me@ttaylorr.com","subject":"Re: [PATCH v3 6/9] pack-objects: fix typo in 'show_object_pack_hint()'","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-04-16T05:36:02Z","receivedAt":"2025-04-16T05:36:14Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Apr 15, 2025 at 3:47 PM Taylor Blau wrote:\n>\n> Noticed-by: Elijah Newren \n> Signed-off-by: Taylor Blau \n> ---\n> builtin/pack-objects.c | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index eb2a4099cc..f06b359150 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -3532,7 +3532,7 @@ static void show_object_pack_hint(struct object *object, const char *name,\n> * would typically pick up during a reachability traversal.\n> *\n> * Make a best-effort attempt to fill in the ->hash and ->no_try_delta\n> - * here using a now in order to perhaps improve the delta selection\n> + * fields here in order to perhaps improve the delta selection\n> * process.\n\nThanks; much improved.\n\n> */\n> oe->hash = pack_name_hash_fn(name);\n> --\n> 2.49.0.230.ga662d77f78\n>\n"},{"id":"516252","messageId":"CABPp-BHOSiOSFeqTutAqznTCO6Fh-kM7d2B-jF+LU1iTQzOTOw@mail.gmail.com","threadId":"63286","inReplyTo":"58891101f377267df120dc4a9edea2997296dbec.1744757204.git.me@ttaylorr.com","subject":"Re: [PATCH v3 9/9] repack: exclude cruft pack(s) from the MIDX where possible","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-04-16T05:56:59Z","receivedAt":"2025-04-16T05:57:11Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Apr 15, 2025 at 3:47 PM Taylor Blau wrote:\n>\n> In ddee3703b3 (builtin/repack.c: add cruft packs to MIDX during\n> geometric repack, 2022-05-20), repack began adding cruft pack(s) to the\n> MIDX with '--write-midx' to ensure that the resulting MIDX was always\n> closed under reachability in order to generate reachability bitmaps.\n>\n> Suppose (prior to this patch) you have a once-unreachable object packed\n> in a cruft pack, which later on becomes reachable from one or more\n> objects in a geometrically repacked pack. That once-unreachable object\n> *won't* appear in the new pack, since the cruft pack was specified as\n> neither included nor excluded to 'pack-objects --stdin-packs'.\n\nBut immediately prior to this patch you implemented\n--stdin-packs=follow, so the once-unreachable object would actually\nappear in the pack if that new option was used. The \"(prior to this\npatch)\" addition was meant to help clarify here, but to me it doesn't\nsucceed. (If it had been \"(prior to this series)\" it would have\nclarified that we aren't yet using the new feature from the previous\npatch.) Perhaps you meant that geometric repacking doesn't use\n--stdin-packs=follow currently, and therefore the once-unreachable\nobject won't be in the new pack, but if so I think it would be helpful\nto call that out explicitly so the reader can more easily follow which\nhypothetical state you are discussing.\n\n> If the\n> new pack is included in a MIDX without the cruft pack, then trying to\n> generate bitmaps for that MIDX may fail. This happens when the bitmap\n> selection process picks one or more commits which reach the\n> once-unreachable objects, commit ddee3703b3 ensures that the MIDX will\n> be closed under reachability. Without it, we would fail to generate a\n> MIDX bitmap.\n\nThe comma between objects and commit seems insufficient. To me, that\nfeels like a contrasting thought that should start a new sentence.\nPerhaps the last three lines could read something like:\n\n\"\"\"\nonce-unreachable objects. Commit ddee3703b3 ensures that the MIDX will\nbe closed under reachability by including cruft pack(s); without them,\nwe would fail to generate a\nMIDX bitmap.\n\"\"\"\n\n[...]\n> ---\n> Documentation/config/repack.adoc | 7 ++\n> builtin/repack.c | 163 +++++++++++++++++++++++++++----\n> t/t7704-repack-cruft.sh | 90 +++++++++++++++++\n> 3 files changed, 242 insertions(+), 18 deletions(-)\n\nYou addressed the rest of my feedback with this patch, other than the\ntwo items I highlighted above. I'm excited to see how this works out.\n"},{"id":"516263","messageId":"xmqqecxsjwc9.fsf@gitster.g","threadId":"63286","inReplyTo":"240e90b68d18b3231826d2a68e4e251c893e645a.1744757204.git.me@ttaylorr.com","subject":"Re: [PATCH v3 5/9] pack-objects: perform name-hash traversal for unpacked objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-04-16T09:21:10Z","receivedAt":"2025-04-16T09:21:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> With '--unpacked', pack-objects adds loose objects (which don't appear\n> in any of the excluded packs from '--stdin-packs') to the output pack\n> without considering them as reachability tips for the name-hash\n> traversal.\n>\n> This was an oversight in the original implementation of '--stdin-packs',\n> since the code which enumerates and adds loose objects to the output\n> pack (`add_unreachable_loose_objects()`) did not have access to the\n> 'rev_info' struct found in `read_packs_list_from_stdin()`.\n>\n> Excluding unpacked objects from that traversal doesn't affect the\n> correctness of the resulting pack, but it does make it harder to\n> discover good deltas for loose objects.\n>\n> Now that the 'rev_info' struct is declared outside of\n> `read_packs_list_from_stdin()`, we can pass it to\n> `add_objects_in_unpacked_packs()` and add any loose objects as tips to\n> the above-mentioned traversal, in theory producing slightly tighter\n> packs as a result.\n>\n> Signed-off-by: Taylor Blau \n> ---\n\n\nClever. And the necessary changes are surprisingly small. I like\nit.\n\n\n> builtin/pack-objects.c | 20 ++++++++++++--------\n> 1 file changed, 12 insertions(+), 8 deletions(-)\n>\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index d60cb042c9..eb2a4099cc 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -3644,7 +3644,7 @@ static void read_packs_list_from_stdin(struct rev_info *revs)\n> \tstring_list_clear(&exclude_packs, 0);\n> }\n> \n> -static void add_unreachable_loose_objects(void);\n> +static void add_unreachable_loose_objects(struct rev_info *revs);\n> \n> static void read_stdin_packs(int rev_list_unpacked)\n> {\n> @@ -3671,7 +3671,7 @@ static void read_stdin_packs(int rev_list_unpacked)\n> \tignore_packed_keep_in_core = 1;\n> \tread_packs_list_from_stdin(&revs);\n> \tif (rev_list_unpacked)\n> -\t\tadd_unreachable_loose_objects();\n> +\t\tadd_unreachable_loose_objects(&revs);\n> \n> \tif (prepare_revision_walk(&revs))\n> \t\tdie(_(\"revision walk setup failed\"));\n> @@ -3790,7 +3790,7 @@ static void enumerate_cruft_objects(void)\n> \t\t\t\t\t\t_(\"Enumerating cruft objects\"), 0);\n> \n> \tadd_objects_in_unpacked_packs();\n> -\tadd_unreachable_loose_objects();\n> +\tadd_unreachable_loose_objects(NULL);\n> \n> \tstop_progress(&progress_state);\n> }\n> @@ -4068,8 +4068,9 @@ static void add_objects_in_unpacked_packs(void)\n> }\n> \n> static int add_loose_object(const struct object_id *oid, const char *path,\n> -\t\t\t void *data UNUSED)\n> +\t\t\t void *data)\n> {\n> +\tstruct rev_info *revs = data;\n> \tenum object_type type = oid_object_info(the_repository, oid, NULL);\n> \n> \tif (type < 0) {\n> @@ -4090,6 +4091,10 @@ static int add_loose_object(const struct object_id *oid, const char *path,\n> \t} else {\n> \t\tadd_object_entry(oid, type, \"\", 0);\n> \t}\n> +\n> +\tif (revs && type == OBJ_COMMIT)\n> +\t\tadd_pending_oid(revs, NULL, oid, 0);\n> +\n> \treturn 0;\n> }\n> \n> @@ -4098,11 +4103,10 @@ static int add_loose_object(const struct object_id *oid, const char *path,\n> * add_object_entry will weed out duplicates, so we just add every\n> * loose object we find.\n> */\n> -static void add_unreachable_loose_objects(void)\n> +static void add_unreachable_loose_objects(struct rev_info *revs)\n> {\n> \tfor_each_loose_file_in_objdir(repo_get_object_directory(the_repository),\n> -\t\t\t\t add_loose_object,\n> -\t\t\t\t NULL, NULL, NULL);\n> +\t\t\t\t add_loose_object, NULL, NULL, revs);\n> }\n> \n> static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)\n> @@ -4358,7 +4362,7 @@ static void get_object_list(struct rev_info *revs, int ac, const char **av)\n> \tif (keep_unreachable)\n> \t\tadd_objects_in_unpacked_packs();\n> \tif (pack_loose_unreachable)\n> -\t\tadd_unreachable_loose_objects();\n> +\t\tadd_unreachable_loose_objects(NULL);\n> \tif (unpack_unreachable)\n> \t\tloosen_unused_packed_objects();\n"},{"id":"516309","messageId":"aAAqE2NK3eQ8c6r1@nand.local","threadId":"63286","inReplyTo":"xmqqtt6okjm8.fsf@gitster.g","subject":"Re: [PATCH v3 2/9] pack-objects: limit scope in 'add_object_entry_from_pack()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-16T22:07:15Z","receivedAt":"2025-04-16T22:07:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Apr 15, 2025 at 05:58:23PM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > In add_object_entry_from_pack() we declare 'revs' (given to us through\n> > the miscellaneous context argument) earlier in the \"if (p)\" conditional\n> > than is necessary. Move it down as far as it can go to reduce its\n> > scope.\n>\n> That makes sense, but ...\n>\n> > Signed-off-by: Taylor Blau \n> > ---\n> > builtin/pack-objects.c | 4 ++--\n> > 1 file changed, 2 insertions(+), 2 deletions(-)\n> >\n> > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> > index 20dd870bbf..4ab695a3aa 100644\n> > --- a/builtin/pack-objects.c\n> > +++ b/builtin/pack-objects.c\n> > @@ -3490,14 +3490,14 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n> > \t\treturn 0;\n> >\n> > \tif (p) {\n> > -\t\tstruct rev_info *revs = _data;\n> > \t\tstruct object_info oi = OBJECT_INFO_INIT;\n> > -\n> > \t\toi.typep = &type;\n> > +\n>\n> Isn't this change about spacing around oi's decl and the first\n> statement in the block strictly worsening the code? At least it is\n> an unrelated change.\n\nYeah, this is cruft that I thought I had expunged while rebasing. Here's\na better version of the patch, but I'm happy to send a new round of the\nseries if it would be more convenient for you:\n\n--- 8< ---\n\nSubject: [PATCH] pack-objects: limit scope in 'add_object_entry_from_pack()'\n\nIn add_object_entry_from_pack() we declare 'revs' (given to us through\nthe miscellaneous context argument) earlier in the \"if (p)\" conditional\nthan is necessary. Move it down as far as it can go to reduce its\nscope.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 20dd870bbf..682e80be40 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3490,7 +3490,6 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n \t\treturn 0;\n\n \tif (p) {\n-\t\tstruct rev_info *revs = _data;\n \t\tstruct object_info oi = OBJECT_INFO_INIT;\n\n \t\toi.typep = &type;\n@@ -3498,6 +3497,7 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n \t\t\tdie(_(\"could not get type of object %s in pack %s\"),\n \t\t\t oid_to_hex(oid), p->pack_name);\n \t\t} else if (type == OBJ_COMMIT) {\n+\t\t\tstruct rev_info *revs = _data;\n \t\t\t/*\n \t\t\t * commits in included packs are used as starting points for the\n \t\t\t * subsequent revision walk\n--\n2.49.0.230.ga662d77f78\n\nThanks,\nTaylor\n"},{"id":"516310","messageId":"aAAqLWx7dJIgFcr9@nand.local","threadId":"63286","inReplyTo":"CABPp-BG-=6uRtHXrzSOMw=J2Shg_vfXhLzwWDREy7OAQg_Ak6w@mail.gmail.com","subject":"Re: [PATCH v3 2/9] pack-objects: limit scope in 'add_object_entry_from_pack()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-16T22:07:41Z","receivedAt":"2025-04-16T22:07:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Apr 15, 2025 at 10:31:26PM -0700, Elijah Newren wrote:\n> On Tue, Apr 15, 2025 at 3:46 PM Taylor Blau wrote:\n> >\n> > In add_object_entry_from_pack() we declare 'revs' (given to us through\n> > the miscellaneous context argument) earlier in the \"if (p)\" conditional\n> > than is necessary. Move it down as far as it can go to reduce its\n> > scope.\n> >\n> > Signed-off-by: Taylor Blau \n> > ---\n> > builtin/pack-objects.c | 4 ++--\n> > 1 file changed, 2 insertions(+), 2 deletions(-)\n> >\n> > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> > index 20dd870bbf..4ab695a3aa 100644\n> > --- a/builtin/pack-objects.c\n> > +++ b/builtin/pack-objects.c\n> > @@ -3490,14 +3490,14 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n> > return 0;\n> >\n> > if (p) {\n> > - struct rev_info *revs = _data;\n>\n> This change is half of what you mention in your commit message.\n\nYeah, Junio noted the same in his review as well. See my response there\nfor a cleaner version of this patch.\n\nThanks,\nTaylor\n"},{"id":"516311","messageId":"aAAsIUqkqCqjoFHf@nand.local","threadId":"63286","inReplyTo":"CABPp-BHOSiOSFeqTutAqznTCO6Fh-kM7d2B-jF+LU1iTQzOTOw@mail.gmail.com","subject":"Re: [PATCH v3 9/9] repack: exclude cruft pack(s) from the MIDX where possible","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-04-16T22:16:01Z","receivedAt":"2025-04-16T22:16:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Apr 15, 2025 at 10:56:59PM -0700, Elijah Newren wrote:\n> On Tue, Apr 15, 2025 at 3:47 PM Taylor Blau wrote:\n> >\n> > In ddee3703b3 (builtin/repack.c: add cruft packs to MIDX during\n> > geometric repack, 2022-05-20), repack began adding cruft pack(s) to the\n> > MIDX with '--write-midx' to ensure that the resulting MIDX was always\n> > closed under reachability in order to generate reachability bitmaps.\n> >\n> > Suppose (prior to this patch) you have a once-unreachable object packed\n> > in a cruft pack, which later on becomes reachable from one or more\n> > objects in a geometrically repacked pack. That once-unreachable object\n> > *won't* appear in the new pack, since the cruft pack was specified as\n> > neither included nor excluded to 'pack-objects --stdin-packs'.\n>\n> But immediately prior to this patch you implemented\n> --stdin-packs=follow, so the once-unreachable object would actually\n> appear in the pack if that new option was used. The \"(prior to this\n> patch)\" addition was meant to help clarify here, but to me it doesn't\n> succeed. (If it had been \"(prior to this series)\" it would have\n> clarified that we aren't yet using the new feature from the previous\n> patch.) Perhaps you meant that geometric repacking doesn't use\n> --stdin-packs=follow currently, and therefore the once-unreachable\n> object won't be in the new pack, but if so I think it would be helpful\n> to call that out explicitly so the reader can more easily follow which\n> hypothetical state you are discussing.\n\nYeah, I am referring to the state of the world from repack's perspective\nhere. It is the case that prior to this patch (9/9) we don't use\n'--stdin-packs=follow' from the repack code when invoking pack-objects,\nbut I can sympathize with the confusion that this creates since the\ndistinction between the new mode existing and having real-life callers\nfrom other builtins is subtle.\n\n> > If the\n> > new pack is included in a MIDX without the cruft pack, then trying to\n> > generate bitmaps for that MIDX may fail. This happens when the bitmap\n> > selection process picks one or more commits which reach the\n> > once-unreachable objects, commit ddee3703b3 ensures that the MIDX will\n> > be closed under reachability. Without it, we would fail to generate a\n> > MIDX bitmap.\n>\n> The comma between objects and commit seems insufficient. To me, that\n> feels like a contrasting thought that should start a new sentence.\n> Perhaps the last three lines could read something like:\n>\n> \"\"\"\n> once-unreachable objects. Commit ddee3703b3 ensures that the MIDX will\n> be closed under reachability by including cruft pack(s); without them,\n> we would fail to generate a\n> MIDX bitmap.\n> \"\"\"\n\nI think swapping the comma for a semi-colon would have worked as well.\nI'm going to leave it as-is unless you feel strongly about it, if that's\nalright with you.\n\n> > ---\n> > Documentation/config/repack.adoc | 7 ++\n> > builtin/repack.c | 163 +++++++++++++++++++++++++++----\n> > t/t7704-repack-cruft.sh | 90 +++++++++++++++++\n> > 3 files changed, 242 insertions(+), 18 deletions(-)\n>\n> You addressed the rest of my feedback with this patch, other than the\n> two items I highlighted above. I'm excited to see how this works out.\n\nThanks, and thanks for the review! I'm curious to see how it turns out\nmyself, too ;-).\n\nThanks,\nTaylor\n"},{"id":"517925","messageId":"CABPp-BE-tSPPJp2mpAKL8J9Ye2AhejG+x-XP9AF-mvCVNZ2+WA@mail.gmail.com","threadId":"63286","inReplyTo":"aAAsIUqkqCqjoFHf@nand.local","subject":"Re: [PATCH v3 9/9] repack: exclude cruft pack(s) from the MIDX where possible","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-05-13T03:34:14Z","receivedAt":"2025-05-13T03:34:27Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Apr 16, 2025 at 3:16 PM Taylor Blau wrote:\n>\n> On Tue, Apr 15, 2025 at 10:56:59PM -0700, Elijah Newren wrote:\n> > On Tue, Apr 15, 2025 at 3:47 PM Taylor Blau wrote:\n> > >\n> > > In ddee3703b3 (builtin/repack.c: add cruft packs to MIDX during\n> > > geometric repack, 2022-05-20), repack began adding cruft pack(s) to the\n> > > MIDX with '--write-midx' to ensure that the resulting MIDX was always\n> > > closed under reachability in order to generate reachability bitmaps.\n> > >\n> > > Suppose (prior to this patch) you have a once-unreachable object packed\n> > > in a cruft pack, which later on becomes reachable from one or more\n> > > objects in a geometrically repacked pack. That once-unreachable object\n> > > *won't* appear in the new pack, since the cruft pack was specified as\n> > > neither included nor excluded to 'pack-objects --stdin-packs'.\n> >\n> > But immediately prior to this patch you implemented\n> > --stdin-packs=follow, so the once-unreachable object would actually\n> > appear in the pack if that new option was used. The \"(prior to this\n> > patch)\" addition was meant to help clarify here, but to me it doesn't\n> > succeed. (If it had been \"(prior to this series)\" it would have\n> > clarified that we aren't yet using the new feature from the previous\n> > patch.) Perhaps you meant that geometric repacking doesn't use\n> > --stdin-packs=follow currently, and therefore the once-unreachable\n> > object won't be in the new pack, but if so I think it would be helpful\n> > to call that out explicitly so the reader can more easily follow which\n> > hypothetical state you are discussing.\n>\n> Yeah, I am referring to the state of the world from repack's perspective\n> here. It is the case that prior to this patch (9/9) we don't use\n> '--stdin-packs=follow' from the repack code when invoking pack-objects,\n> but I can sympathize with the confusion that this creates since the\n> distinction between the new mode existing and having real-life callers\n> from other builtins is subtle.\n\nOkay, but can we simply call out that --stdin-packs=follow is not yet\nused by default in the commit message to make clear that we're talking\nabout the state without it?\n\nAnd while we're at it, fix up the end of the paragraph as I mentioned before?\n\nSo, something like:\n\nWhile the previous patch added the --stdin-packs=follow option to\npack-objects, it is not on by default. Imagine you have a\nonce-unreachable object packed in a cruft pack, which later on becomes\nreachable from one or more objects in a geometrically repacked\npack. That once-unreachable object *won't* appear in the new pack, since\nthe cruft pack was specified as neither included nor excluded to\n'pack-objects --stdin-packs' (and `--stdin-packs=follow` is not on). If\nthe new pack is included in a MIDX without the cruft pack, then trying\nto generate bitmaps for that MIDX may fail. This happens when the bitmap\nselection process picks one or more commits which reach the\nonce-unreachable objects. Commit ddee3703b3 ensures that the MIDX will\nbe closed under reachability by including cruft pack(s); without them,\nwe would fail to generate a MIDX bitmap.\n\n> >\n> > The comma between objects and commit seems insufficient. To me, that\n> > feels like a contrasting thought that should start a new sentence.\n> > Perhaps the last three lines could read something like:\n> >\n> > \"\"\"\n> > once-unreachable objects. Commit ddee3703b3 ensures that the MIDX will\n> > be closed under reachability by including cruft pack(s); without them,\n> > we would fail to generate a\n> > MIDX bitmap.\n> > \"\"\"\n>\n> I think swapping the comma for a semi-colon would have worked as well.\n\nI didn't feel as strongly about this second suggestion, but\nincorporated it into the alternate wording I provided above for the\nfull paragraph since I think it's still an improvement.\n\nAs a sidenote, though, it's not just a comma and semicolon swap --\nthere's also the addition of a full stop and some extra clarifying\nwords.\n\n> I'm going to leave it as-is unless you feel strongly about it, if that's\n> alright with you.\n\nI originally read this thinking that you were only turning down fixing\nthe second issue I raised, not the first. Sorry for misunderstanding.\nI would like to see a fix for the first issue I raised, at least.\n\nBesides, since Junio didn't take your fixup 2/9 patch, I think you\nshould re-roll at a minimum to get a correct 2/9 out there (seen still\nhas the broken one). And, if you're re-rolling the series, it'd be\nreally nice to incorporate the suggestions above in 9/9. If you do\nthat, I'll put my stamp on the series and encourage Junio to merge it.\n:-)\n"},{"id":"519115","messageId":"cover.1748473889.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744413969.git.me@ttaylorr.com","subject":"[PATCH v4 0/9] repack: avoid MIDX'ing cruft pack(s) where possible","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-05-28T23:20:07Z","receivedAt":"2025-05-28T23:20:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here is a tiny reroll of my series to explore creating MIDXs while\nrepacking that don't include the cruft pack.\n\nThe only changes since last time are as follows:\n\n - removed a stray whitespace change in patch 2/9 caught by Junio and\n Elijah\n\n - reworded the final commit message based on helpful feedback from\n Elijah.\n\nThe rest of the series is unchanged, and a range-diff is included below\nas usual for convenience.\n\nA meta-note on something new since last time: I have deployed a\ncherry-picked version of this series to GitHub's infrastructure a few\nweeks ago. The testing repository (whose maintenance failure many years\nago precipitated commit ddee3703b3 (builtin/repack.c: add cruft packs to\nMIDX during geometric repack, 2022-05-20)) has been happily running\nmaintenance tasks without any issues since then, and the cruft pack has\nbeen excluded from the MIDX.\n\nThanks in advance for any review :-).\n\nTaylor Blau (9):\n pack-objects: use standard option incompatibility functions\n pack-objects: limit scope in 'add_object_entry_from_pack()'\n pack-objects: factor out handling '--stdin-packs'\n pack-objects: declare 'rev_info' for '--stdin-packs' earlier\n pack-objects: perform name-hash traversal for unpacked objects\n pack-objects: fix typo in 'show_object_pack_hint()'\n pack-objects: swap 'show_{object,commit}_pack_hint'\n pack-objects: introduce '--stdin-packs=follow'\n repack: exclude cruft pack(s) from the MIDX where possible\n\n Documentation/config/repack.adoc | 7 +\n Documentation/git-pack-objects.adoc | 10 +-\n builtin/pack-objects.c | 190 ++++++++++++++++++----------\n builtin/repack.c | 163 +++++++++++++++++++++---\n t/t5331-pack-objects-stdin.sh | 84 +++++++++++-\n t/t7704-repack-cruft.sh | 90 +++++++++++++\n 6 files changed, 455 insertions(+), 89 deletions(-)\n\nRange-diff against v3:\n 1: 986bef29b5 ! 1: f8b31c6a8d pack-objects: limit scope in 'add_object_entry_from_pack()'\n @@ Metadata\n Author: Taylor Blau \n\n ## Commit message ##\n - pack-objects: limit scope in 'add_object_entry_from_pack()'\n + pack-objects: use standard option incompatibility functions\n\n - In add_object_entry_from_pack() we declare 'revs' (given to us through\n - the miscellaneous context argument) earlier in the \"if (p)\" conditional\n - than is necessary. Move it down as far as it can go to reduce its\n - scope.\n + pack-objects has a handful of explicit checks for pairs of command-line\n + options which are mutually incompatible. Many of these pre-date\n + a699367bb8 (i18n: factorize more 'incompatible options' messages,\n + 2022-01-31).\n +\n + Convert the explicit checks into die_for_incompatible_opt2() calls,\n + which simplifies the implementation and standardizes pack-objects'\n + output when given incompatible options (e.g., --stdin-packs with\n + --filter gives different output than --keep-unreachable with\n + --unpack-unreachable).\n +\n + There is one minor piece of test fallout in t5331 that expects the old\n + format, which has been corrected.\n\n Signed-off-by: Taylor Blau \n\n ## builtin/pack-objects.c ##\n -@@ builtin/pack-objects.c: static int add_object_entry_from_pack(const struct object_id *oid,\n - \t\treturn 0;\n +@@ builtin/pack-objects.c: int cmd_pack_objects(int argc,\n + \t\tstrvec_push(&rp, \"--unpacked\");\n + \t}\n\n - \tif (p) {\n --\t\tstruct rev_info *revs = _data;\n - \t\tstruct object_info oi = OBJECT_INFO_INIT;\n --\n - \t\toi.typep = &type;\n +-\tif (exclude_promisor_objects && exclude_promisor_objects_best_effort)\n +-\t\tdie(_(\"options '%s' and '%s' cannot be used together\"),\n +-\t\t \"--exclude-promisor-objects\", \"--exclude-promisor-objects-best-effort\");\n ++\tdie_for_incompatible_opt2(exclude_promisor_objects,\n ++\t\t\t\t \"--exclude-promisor-objects\",\n ++\t\t\t\t exclude_promisor_objects_best_effort,\n ++\t\t\t\t \"--exclude-promisor-objects-best-effort\");\n + \tif (exclude_promisor_objects) {\n + \t\tuse_internal_rev_list = 1;\n + \t\tfetch_if_missing = 0;\n +@@ builtin/pack-objects.c: int cmd_pack_objects(int argc,\n + \tif (!pack_to_stdout && thin)\n + \t\tdie(_(\"--thin cannot be used to build an indexable pack\"));\n +\n +-\tif (keep_unreachable && unpack_unreachable)\n +-\t\tdie(_(\"options '%s' and '%s' cannot be used together\"), \"--keep-unreachable\", \"--unpack-unreachable\");\n ++\tdie_for_incompatible_opt2(keep_unreachable, \"--keep-unreachable\",\n ++\t\t\t\t unpack_unreachable, \"--unpack-unreachable\");\n + \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n + \t\tunpack_unreachable_expiration = 0;\n +\n +-\tif (stdin_packs && filter_options.choice)\n +-\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n ++\tdie_for_incompatible_opt2(stdin_packs, \"--stdin-packs\",\n ++\t\t\t\t filter_options.choice, \"--filter\");\n +\n - \t\tif (packed_object_info(the_repository, p, ofs, &oi) < 0) {\n - \t\t\tdie(_(\"could not get type of object %s in pack %s\"),\n - \t\t\t oid_to_hex(oid), p->pack_name);\n - \t\t} else if (type == OBJ_COMMIT) {\n -+\t\t\tstruct rev_info *revs = _data;\n - \t\t\t/*\n - \t\t\t * commits in included packs are used as starting points for the\n - \t\t\t * subsequent revision walk\n +\n + \tif (stdin_packs && use_internal_rev_list)\n + \t\tdie(_(\"cannot use internal rev list with --stdin-packs\"));\n +@@ builtin/pack-objects.c: int cmd_pack_objects(int argc,\n + \tif (cruft) {\n + \t\tif (use_internal_rev_list)\n + \t\t\tdie(_(\"cannot use internal rev list with --cruft\"));\n +-\t\tif (stdin_packs)\n +-\t\t\tdie(_(\"cannot use --stdin-packs with --cruft\"));\n ++\t\tdie_for_incompatible_opt2(stdin_packs, \"--stdin-packs\",\n ++\t\t\t\t\t cruft, \"--cruft\");\n + \t}\n +\n + \t/*\n +\n + ## t/t5331-pack-objects-stdin.sh ##\n +@@ t/t5331-pack-objects-stdin.sh: test_expect_success '--stdin-packs is incompatible with --filter' '\n + \t\tcd stdin-packs &&\n + \t\ttest_must_fail git pack-objects --stdin-packs --stdout \\\n + \t\t\t--filter=blob:none err &&\n +-\t\ttest_grep \"cannot use --filter with --stdin-packs\" err\n ++\t\ttest_grep \"options .--stdin-packs. and .--filter. cannot be used together\" err\n + \t)\n + '\n +\n -: ---------- > 2: 2753e29648 pack-objects: limit scope in 'add_object_entry_from_pack()'\n 2: 6f8fe8a4e1 = 3: 32b49d9073 pack-objects: factor out handling '--stdin-packs'\n 3: 2a235461a6 = 4: a797ff3a83 pack-objects: declare 'rev_info' for '--stdin-packs' earlier\n 4: 240e90b68d = 5: 29bf05633a pack-objects: perform name-hash traversal for unpacked objects\n 5: 9a18fa2e52 = 6: 0696fa1736 pack-objects: fix typo in 'show_object_pack_hint()'\n 6: 6c997853f1 = 7: 1cc45b4472 pack-objects: swap 'show_{object,commit}_pack_hint'\n 7: 0ff699f056 = 8: 3e3d929bd0 pack-objects: introduce '--stdin-packs=follow'\n 8: 58891101f3 ! 9: 52a069ef48 repack: exclude cruft pack(s) from the MIDX where possible\n @@ Commit message\n MIDX with '--write-midx' to ensure that the resulting MIDX was always\n closed under reachability in order to generate reachability bitmaps.\n\n - Suppose (prior to this patch) you have a once-unreachable object packed\n - in a cruft pack, which later on becomes reachable from one or more\n - objects in a geometrically repacked pack. That once-unreachable object\n - *won't* appear in the new pack, since the cruft pack was specified as\n - neither included nor excluded to 'pack-objects --stdin-packs'. If the\n + While the previous patch added the '--stdin-packs=follow' option to\n + pack-objects, it is not yet on by default. Given that, suppose you have\n + a once-unreachable object packed in a cruft pack, which later becomes\n + reachable from one or more objects in a geometrically repacked pack.\n + That once-unreachable object *won't* appear in the new pack, since the\n + cruft pack was not specified as included or excluded when the\n + geometrically repacked pack was created with 'pack-objects\n + --stdin-packs' (*not* '--stdin-packs=follow', which is not on). If that\n new pack is included in a MIDX without the cruft pack, then trying to\n generate bitmaps for that MIDX may fail. This happens when the bitmap\n selection process picks one or more commits which reach the\n - once-unreachable objects, commit ddee3703b3 ensures that the MIDX will\n - be closed under reachability. Without it, we would fail to generate a\n - MIDX bitmap.\n + once-unreachable objects.\n\n + To mitigate this failure mode, commit ddee3703b3 ensures that the MIDX\n + will be closed under reachability by including cruft pack(s). If cruft\n + pack(s) were not included, we would fail to generate a MIDX bitmap. But\n ddee3703b3 alludes to the fact that this is sub-optimal by saying\n\n [...] it's desirable to avoid including cruft packs in the MIDX\n\nbase-commit: 485f5f863615e670fd97ae40af744e14072cfe18\n--\n2.49.0.640.ga4de40e6a8\n"},{"id":"519116","messageId":"f8b31c6a8d788fa53f7b39fb9ae4b61ebfc882b9.1748473890.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1748473889.git.me@ttaylorr.com","subject":"[PATCH v4 1/9] pack-objects: use standard option incompatibility functions","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-05-28T23:20:10Z","receivedAt":"2025-05-28T23:20:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"pack-objects has a handful of explicit checks for pairs of command-line\noptions which are mutually incompatible. Many of these pre-date\na699367bb8 (i18n: factorize more 'incompatible options' messages,\n2022-01-31).\n\nConvert the explicit checks into die_for_incompatible_opt2() calls,\nwhich simplifies the implementation and standardizes pack-objects'\noutput when given incompatible options (e.g., --stdin-packs with\n--filter gives different output than --keep-unreachable with\n--unpack-unreachable).\n\nThere is one minor piece of test fallout in t5331 that expects the old\nformat, which has been corrected.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 20 +++++++++++---------\n t/t5331-pack-objects-stdin.sh | 2 +-\n 2 files changed, 12 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 6b06d159d2..20dd870bbf 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -4651,9 +4651,10 @@ int cmd_pack_objects(int argc,\n \t\tstrvec_push(&rp, \"--unpacked\");\n \t}\n \n-\tif (exclude_promisor_objects && exclude_promisor_objects_best_effort)\n-\t\tdie(_(\"options '%s' and '%s' cannot be used together\"),\n-\t\t \"--exclude-promisor-objects\", \"--exclude-promisor-objects-best-effort\");\n+\tdie_for_incompatible_opt2(exclude_promisor_objects,\n+\t\t\t\t \"--exclude-promisor-objects\",\n+\t\t\t\t exclude_promisor_objects_best_effort,\n+\t\t\t\t \"--exclude-promisor-objects-best-effort\");\n \tif (exclude_promisor_objects) {\n \t\tuse_internal_rev_list = 1;\n \t\tfetch_if_missing = 0;\n@@ -4691,13 +4692,14 @@ int cmd_pack_objects(int argc,\n \tif (!pack_to_stdout && thin)\n \t\tdie(_(\"--thin cannot be used to build an indexable pack\"));\n \n-\tif (keep_unreachable && unpack_unreachable)\n-\t\tdie(_(\"options '%s' and '%s' cannot be used together\"), \"--keep-unreachable\", \"--unpack-unreachable\");\n+\tdie_for_incompatible_opt2(keep_unreachable, \"--keep-unreachable\",\n+\t\t\t\t unpack_unreachable, \"--unpack-unreachable\");\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n-\tif (stdin_packs && filter_options.choice)\n-\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n+\tdie_for_incompatible_opt2(stdin_packs, \"--stdin-packs\",\n+\t\t\t\t filter_options.choice, \"--filter\");\n+\n \n \tif (stdin_packs && use_internal_rev_list)\n \t\tdie(_(\"cannot use internal rev list with --stdin-packs\"));\n@@ -4705,8 +4707,8 @@ int cmd_pack_objects(int argc,\n \tif (cruft) {\n \t\tif (use_internal_rev_list)\n \t\t\tdie(_(\"cannot use internal rev list with --cruft\"));\n-\t\tif (stdin_packs)\n-\t\t\tdie(_(\"cannot use --stdin-packs with --cruft\"));\n+\t\tdie_for_incompatible_opt2(stdin_packs, \"--stdin-packs\",\n+\t\t\t\t\t cruft, \"--cruft\");\n \t}\n \n \t/*\ndiff --git a/t/t5331-pack-objects-stdin.sh b/t/t5331-pack-objects-stdin.sh\nindex b48c0cbe8f..8fd07deb8d 100755\n--- a/t/t5331-pack-objects-stdin.sh\n+++ b/t/t5331-pack-objects-stdin.sh\n@@ -64,7 +64,7 @@ test_expect_success '--stdin-packs is incompatible with --filter' '\n \t\tcd stdin-packs &&\n \t\ttest_must_fail git pack-objects --stdin-packs --stdout \\\n \t\t\t--filter=blob:none err &&\n-\t\ttest_grep \"cannot use --filter with --stdin-packs\" err\n+\t\ttest_grep \"options .--stdin-packs. and .--filter. cannot be used together\" err\n \t)\n '\n \n-- \n2.49.0.640.ga4de40e6a8\n\n"},{"id":"519117","messageId":"2753e29648da5034801413451fe2d977b137ac88.1748473890.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1748473889.git.me@ttaylorr.com","subject":"[PATCH v4 2/9] pack-objects: limit scope in 'add_object_entry_from_pack()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-05-28T23:20:13Z","receivedAt":"2025-05-28T23:20:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In add_object_entry_from_pack() we declare 'revs' (given to us through\nthe miscellaneous context argument) earlier in the \"if (p)\" conditional\nthan is necessary. Move it down as far as it can go to reduce its\nscope.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 20dd870bbf..682e80be40 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3490,7 +3490,6 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n \t\treturn 0;\n \n \tif (p) {\n-\t\tstruct rev_info *revs = _data;\n \t\tstruct object_info oi = OBJECT_INFO_INIT;\n \n \t\toi.typep = &type;\n@@ -3498,6 +3497,7 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n \t\t\tdie(_(\"could not get type of object %s in pack %s\"),\n \t\t\t oid_to_hex(oid), p->pack_name);\n \t\t} else if (type == OBJ_COMMIT) {\n+\t\t\tstruct rev_info *revs = _data;\n \t\t\t/*\n \t\t\t * commits in included packs are used as starting points for the\n \t\t\t * subsequent revision walk\n-- \n2.49.0.640.ga4de40e6a8\n\n"},{"id":"519118","messageId":"32b49d90732faecfca685d96566268e6f1f4a789.1748473890.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1748473889.git.me@ttaylorr.com","subject":"[PATCH v4 3/9] pack-objects: factor out handling '--stdin-packs'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-05-28T23:20:16Z","receivedAt":"2025-05-28T23:20:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"At the bottom of cmd_pack_objects() we check which mode the command is\nrunning in (e.g., generating a cruft pack, handling '--stdin-packs',\nusing the internal rev-list, etc.) and handle the mode appropriately.\n\nThe '--stdin-packs' case is handled inline (dating back to its\nintroduction in 339bce27f4 (builtin/pack-objects.c: add '--stdin-packs'\noption, 2021-02-22)) since it is relatively short. Extract the body of\n\"if (stdin_packs)\" into its own function to prepare for the\nimplementation to become lengthier in a following commit.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 18 ++++++++++++------\n 1 file changed, 12 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 682e80be40..3f6a7c62e6 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3674,6 +3674,17 @@ static void read_packs_list_from_stdin(void)\n \tstring_list_clear(&exclude_packs, 0);\n }\n \n+static void add_unreachable_loose_objects(void);\n+\n+static void read_stdin_packs(int rev_list_unpacked)\n+{\n+\t/* avoids adding objects in excluded packs */\n+\tignore_packed_keep_in_core = 1;\n+\tread_packs_list_from_stdin();\n+\tif (rev_list_unpacked)\n+\t\tadd_unreachable_loose_objects();\n+}\n+\n static void add_cruft_object_entry(const struct object_id *oid, enum object_type type,\n \t\t\t\t struct packed_git *pack, off_t offset,\n \t\t\t\t const char *name, uint32_t mtime)\n@@ -3769,7 +3780,6 @@ static void mark_pack_kept_in_core(struct string_list *packs, unsigned keep)\n \t}\n }\n \n-static void add_unreachable_loose_objects(void);\n static void add_objects_in_unpacked_packs(void);\n \n static void enumerate_cruft_objects(void)\n@@ -4776,11 +4786,7 @@ int cmd_pack_objects(int argc,\n \t\tprogress_state = start_progress(the_repository,\n \t\t\t\t\t\t_(\"Enumerating objects\"), 0);\n \tif (stdin_packs) {\n-\t\t/* avoids adding objects in excluded packs */\n-\t\tignore_packed_keep_in_core = 1;\n-\t\tread_packs_list_from_stdin();\n-\t\tif (rev_list_unpacked)\n-\t\t\tadd_unreachable_loose_objects();\n+\t\tread_stdin_packs(rev_list_unpacked);\n \t} else if (cruft) {\n \t\tread_cruft_objects();\n \t} else if (!use_internal_rev_list) {\n-- \n2.49.0.640.ga4de40e6a8\n\n"},{"id":"519119","messageId":"a797ff3a83bf623592384fc7652e1f1ed0d5d49b.1748473890.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1748473889.git.me@ttaylorr.com","subject":"[PATCH v4 4/9] pack-objects: declare 'rev_info' for '--stdin-packs' earlier","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-05-28T23:20:19Z","receivedAt":"2025-05-28T23:20:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Once 'read_packs_list_from_stdin()' has called for_each_object_in_pack()\non each of the input packs, we do a reachability traversal to discover\nnames for any objects we picked up so we can generate name hash values\nand hopefully get higher quality deltas as a result.\n\nA future commit will change the purpose of this reachability traversal\nto find and pack objects which are reachable from commits in the input\npacks, but are packed in an unknown (not included nor excluded) pack.\n\nExtract the code which initializes and performs the reachability\ntraversal to take place in the caller, not the callee, which prepares us\nto share this code for the '--unpacked' case (see the function\nadd_unreachable_loose_objects() for more details).\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 71 +++++++++++++++++++++---------------------\n 1 file changed, 36 insertions(+), 35 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 3f6a7c62e6..88071b4d8c 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3558,7 +3558,7 @@ static int pack_mtime_cmp(const void *_a, const void *_b)\n \t\treturn 0;\n }\n \n-static void read_packs_list_from_stdin(void)\n+static void read_packs_list_from_stdin(struct rev_info *revs)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct string_list include_packs = STRING_LIST_INIT_DUP;\n@@ -3566,24 +3566,6 @@ static void read_packs_list_from_stdin(void)\n \tstruct string_list_item *item = NULL;\n \n \tstruct packed_git *p;\n-\tstruct rev_info revs;\n-\n-\trepo_init_revisions(the_repository, &revs, NULL);\n-\t/*\n-\t * Use a revision walk to fill in the namehash of objects in the include\n-\t * packs. To save time, we'll avoid traversing through objects that are\n-\t * in excluded packs.\n-\t *\n-\t * That may cause us to avoid populating all of the namehash fields of\n-\t * all included objects, but our goal is best-effort, since this is only\n-\t * an optimization during delta selection.\n-\t */\n-\trevs.no_kept_objects = 1;\n-\trevs.keep_pack_cache_flags |= IN_CORE_KEEP_PACKS;\n-\trevs.blob_objects = 1;\n-\trevs.tree_objects = 1;\n-\trevs.tag_objects = 1;\n-\trevs.ignore_missing_links = 1;\n \n \twhile (strbuf_getline(&buf, stdin) != EOF) {\n \t\tif (!buf.len)\n@@ -3653,10 +3635,44 @@ static void read_packs_list_from_stdin(void)\n \t\tstruct packed_git *p = item->util;\n \t\tfor_each_object_in_pack(p,\n \t\t\t\t\tadd_object_entry_from_pack,\n-\t\t\t\t\t&revs,\n+\t\t\t\t\trevs,\n \t\t\t\t\tFOR_EACH_OBJECT_PACK_ORDER);\n \t}\n \n+\tstrbuf_release(&buf);\n+\tstring_list_clear(&include_packs, 0);\n+\tstring_list_clear(&exclude_packs, 0);\n+}\n+\n+static void add_unreachable_loose_objects(void);\n+\n+static void read_stdin_packs(int rev_list_unpacked)\n+{\n+\tstruct rev_info revs;\n+\n+\trepo_init_revisions(the_repository, &revs, NULL);\n+\t/*\n+\t * Use a revision walk to fill in the namehash of objects in the include\n+\t * packs. To save time, we'll avoid traversing through objects that are\n+\t * in excluded packs.\n+\t *\n+\t * That may cause us to avoid populating all of the namehash fields of\n+\t * all included objects, but our goal is best-effort, since this is only\n+\t * an optimization during delta selection.\n+\t */\n+\trevs.no_kept_objects = 1;\n+\trevs.keep_pack_cache_flags |= IN_CORE_KEEP_PACKS;\n+\trevs.blob_objects = 1;\n+\trevs.tree_objects = 1;\n+\trevs.tag_objects = 1;\n+\trevs.ignore_missing_links = 1;\n+\n+\t/* avoids adding objects in excluded packs */\n+\tignore_packed_keep_in_core = 1;\n+\tread_packs_list_from_stdin(&revs);\n+\tif (rev_list_unpacked)\n+\t\tadd_unreachable_loose_objects();\n+\n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n \ttraverse_commit_list(&revs,\n@@ -3668,21 +3684,6 @@ static void read_packs_list_from_stdin(void)\n \t\t\t stdin_packs_found_nr);\n \ttrace2_data_intmax(\"pack-objects\", the_repository, \"stdin_packs_hints\",\n \t\t\t stdin_packs_hints_nr);\n-\n-\tstrbuf_release(&buf);\n-\tstring_list_clear(&include_packs, 0);\n-\tstring_list_clear(&exclude_packs, 0);\n-}\n-\n-static void add_unreachable_loose_objects(void);\n-\n-static void read_stdin_packs(int rev_list_unpacked)\n-{\n-\t/* avoids adding objects in excluded packs */\n-\tignore_packed_keep_in_core = 1;\n-\tread_packs_list_from_stdin();\n-\tif (rev_list_unpacked)\n-\t\tadd_unreachable_loose_objects();\n }\n \n static void add_cruft_object_entry(const struct object_id *oid, enum object_type type,\n-- \n2.49.0.640.ga4de40e6a8\n\n"},{"id":"519120","messageId":"29bf05633a45cf2fe439f540c675a75b5b3cf690.1748473890.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1748473889.git.me@ttaylorr.com","subject":"[PATCH v4 5/9] pack-objects: perform name-hash traversal for unpacked objects","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-05-28T23:20:22Z","receivedAt":"2025-05-28T23:20:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"With '--unpacked', pack-objects adds loose objects (which don't appear\nin any of the excluded packs from '--stdin-packs') to the output pack\nwithout considering them as reachability tips for the name-hash\ntraversal.\n\nThis was an oversight in the original implementation of '--stdin-packs',\nsince the code which enumerates and adds loose objects to the output\npack (`add_unreachable_loose_objects()`) did not have access to the\n'rev_info' struct found in `read_packs_list_from_stdin()`.\n\nExcluding unpacked objects from that traversal doesn't affect the\ncorrectness of the resulting pack, but it does make it harder to\ndiscover good deltas for loose objects.\n\nNow that the 'rev_info' struct is declared outside of\n`read_packs_list_from_stdin()`, we can pass it to\n`add_objects_in_unpacked_packs()` and add any loose objects as tips to\nthe above-mentioned traversal, in theory producing slightly tighter\npacks as a result.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 20 ++++++++++++--------\n 1 file changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 88071b4d8c..359f0c3c30 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3644,7 +3644,7 @@ static void read_packs_list_from_stdin(struct rev_info *revs)\n \tstring_list_clear(&exclude_packs, 0);\n }\n \n-static void add_unreachable_loose_objects(void);\n+static void add_unreachable_loose_objects(struct rev_info *revs);\n \n static void read_stdin_packs(int rev_list_unpacked)\n {\n@@ -3671,7 +3671,7 @@ static void read_stdin_packs(int rev_list_unpacked)\n \tignore_packed_keep_in_core = 1;\n \tread_packs_list_from_stdin(&revs);\n \tif (rev_list_unpacked)\n-\t\tadd_unreachable_loose_objects();\n+\t\tadd_unreachable_loose_objects(&revs);\n \n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n@@ -3790,7 +3790,7 @@ static void enumerate_cruft_objects(void)\n \t\t\t\t\t\t_(\"Enumerating cruft objects\"), 0);\n \n \tadd_objects_in_unpacked_packs();\n-\tadd_unreachable_loose_objects();\n+\tadd_unreachable_loose_objects(NULL);\n \n \tstop_progress(&progress_state);\n }\n@@ -4068,8 +4068,9 @@ static void add_objects_in_unpacked_packs(void)\n }\n \n static int add_loose_object(const struct object_id *oid, const char *path,\n-\t\t\t void *data UNUSED)\n+\t\t\t void *data)\n {\n+\tstruct rev_info *revs = data;\n \tenum object_type type = oid_object_info(the_repository, oid, NULL);\n \n \tif (type < 0) {\n@@ -4090,6 +4091,10 @@ static int add_loose_object(const struct object_id *oid, const char *path,\n \t} else {\n \t\tadd_object_entry(oid, type, \"\", 0);\n \t}\n+\n+\tif (revs && type == OBJ_COMMIT)\n+\t\tadd_pending_oid(revs, NULL, oid, 0);\n+\n \treturn 0;\n }\n \n@@ -4098,11 +4103,10 @@ static int add_loose_object(const struct object_id *oid, const char *path,\n * add_object_entry will weed out duplicates, so we just add every\n * loose object we find.\n */\n-static void add_unreachable_loose_objects(void)\n+static void add_unreachable_loose_objects(struct rev_info *revs)\n {\n \tfor_each_loose_file_in_objdir(repo_get_object_directory(the_repository),\n-\t\t\t\t add_loose_object,\n-\t\t\t\t NULL, NULL, NULL);\n+\t\t\t\t add_loose_object, NULL, NULL, revs);\n }\n \n static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)\n@@ -4358,7 +4362,7 @@ static void get_object_list(struct rev_info *revs, int ac, const char **av)\n \tif (keep_unreachable)\n \t\tadd_objects_in_unpacked_packs();\n \tif (pack_loose_unreachable)\n-\t\tadd_unreachable_loose_objects();\n+\t\tadd_unreachable_loose_objects(NULL);\n \tif (unpack_unreachable)\n \t\tloosen_unused_packed_objects();\n \n-- \n2.49.0.640.ga4de40e6a8\n\n"},{"id":"519121","messageId":"0696fa17369c545cfa1bd92850ec252b217d897b.1748473890.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1748473889.git.me@ttaylorr.com","subject":"[PATCH v4 6/9] pack-objects: fix typo in 'show_object_pack_hint()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-05-28T23:20:25Z","receivedAt":"2025-05-28T23:20:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Noticed-by: Elijah Newren \nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 359f0c3c30..a68451c3d2 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3532,7 +3532,7 @@ static void show_object_pack_hint(struct object *object, const char *name,\n \t * would typically pick up during a reachability traversal.\n \t *\n \t * Make a best-effort attempt to fill in the ->hash and ->no_try_delta\n-\t * here using a now in order to perhaps improve the delta selection\n+\t * fields here in order to perhaps improve the delta selection\n \t * process.\n \t */\n \toe->hash = pack_name_hash_fn(name);\n-- \n2.49.0.640.ga4de40e6a8\n\n"},{"id":"519122","messageId":"1cc45b44721c7981514cb45203f752729865cb23.1748473890.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1748473889.git.me@ttaylorr.com","subject":"[PATCH v4 7/9] pack-objects: swap 'show_{object,commit}_pack_hint'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-05-28T23:20:28Z","receivedAt":"2025-05-28T23:20:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"show_commit_pack_hint() has heretofore been a noop, so its position\nwithin its compilation unit only needs to appear before its first use.\n\nBut the following commit will sometimes have `show_commit_pack_hint()`\ncall `show_object_pack_hint()`, so reorder the former to appear after\nthe latter to minimize the code movement in that patch.\n\nSuggested-by: Elijah Newren \nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex a68451c3d2..d3dfe983c3 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3513,12 +3513,6 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n \treturn 0;\n }\n \n-static void show_commit_pack_hint(struct commit *commit UNUSED,\n-\t\t\t\t void *data UNUSED)\n-{\n-\t/* nothing to do; commits don't have a namehash */\n-}\n-\n static void show_object_pack_hint(struct object *object, const char *name,\n \t\t\t\t void *data UNUSED)\n {\n@@ -3541,6 +3535,12 @@ static void show_object_pack_hint(struct object *object, const char *name,\n \tstdin_packs_hints_nr++;\n }\n \n+static void show_commit_pack_hint(struct commit *commit UNUSED,\n+\t\t\t\t void *data UNUSED)\n+{\n+\t/* nothing to do; commits don't have a namehash */\n+}\n+\n static int pack_mtime_cmp(const void *_a, const void *_b)\n {\n \tstruct packed_git *a = ((const struct string_list_item*)_a)->util;\n-- \n2.49.0.640.ga4de40e6a8\n\n"},{"id":"519123","messageId":"3e3d929bd023a6909efef0cde47d873e4ef5c7dc.1748473890.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1748473889.git.me@ttaylorr.com","subject":"[PATCH v4 8/9] pack-objects: introduce '--stdin-packs=follow'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-05-28T23:20:31Z","receivedAt":"2025-05-28T23:20:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When invoked with '--stdin-packs', pack-objects will generate a pack\nwhich contains the objects found in the \"included\" packs, less any\nobjects from \"excluded\" packs.\n\nPacks that exist in the repository but weren't specified as either\nincluded or excluded are in practice treated like the latter, at least\nin the sense that pack-objects won't include objects from those packs.\nThis behavior forces us to include any cruft pack(s) in a repository's\nmulti-pack index for the reasons described in ddee3703b3\n(builtin/repack.c: add cruft packs to MIDX during geometric repack,\n2022-05-20).\n\nThe full details are in ddee3703b3, but the gist is if you\nhave a once-unreachable object in a cruft pack which later becomes\nreachable via one or more commits in a pack generated with\n'--stdin-packs', you *have* to include that object in the MIDX via the\ncopy in the cruft pack, otherwise we cannot generate reachability\nbitmaps for any commits which reach that object.\n\nThis prepares us for new repacking behavior which will \"resurrect\"\nobjects found in cruft or otherwise unspecified packs when generating\nnew packs. In the context of geometric repacking, this may be used to\nmaintain a sequence of geometrically-repacked packs, the union of which\nis closed under reachability, even in the case described earlier.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-pack-objects.adoc | 10 +++-\n builtin/pack-objects.c | 83 +++++++++++++++++++++--------\n t/t5331-pack-objects-stdin.sh | 82 ++++++++++++++++++++++++++++\n 3 files changed, 152 insertions(+), 23 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.adoc b/Documentation/git-pack-objects.adoc\nindex 7f69ae4855..8f0cecaec9 100644\n--- a/Documentation/git-pack-objects.adoc\n+++ b/Documentation/git-pack-objects.adoc\n@@ -87,13 +87,21 @@ base-name::\n \treference was included in the resulting packfile. This\n \tcan be useful to send new tags to native Git clients.\n \n---stdin-packs::\n+--stdin-packs[=]::\n \tRead the basenames of packfiles (e.g., `pack-1234abcd.pack`)\n \tfrom the standard input, instead of object names or revision\n \targuments. The resulting pack contains all objects listed in the\n \tincluded packs (those not beginning with `^`), excluding any\n \tobjects listed in the excluded packs (beginning with `^`).\n +\n+When `mode` is \"follow\", objects from packs not listed on stdin receive\n+special treatment. Objects within unlisted packs will be included if\n+those objects are (1) reachable from the included packs, and (2) not\n+found in any excluded packs. This mode is useful, for example, to\n+resurrect once-unreachable objects found in cruft packs to generate\n+packs which are closed under reachability up to the boundary set by the\n+excluded packs.\n++\n Incompatible with `--revs`, or options that imply `--revs` (such as\n `--all`), with the exception of `--unpacked`, which is compatible.\n \ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex d3dfe983c3..c6ec346369 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -272,6 +272,12 @@ static struct oidmap configured_exclusions;\n static struct oidset excluded_by_config;\n static int name_hash_version = -1;\n \n+enum stdin_packs_mode {\n+\tSTDIN_PACKS_MODE_NONE,\n+\tSTDIN_PACKS_MODE_STANDARD,\n+\tSTDIN_PACKS_MODE_FOLLOW,\n+};\n+\n /**\n * Check whether the name_hash_version chosen by user input is appropriate,\n * and also validate whether it is compatible with other features.\n@@ -3514,31 +3520,44 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n }\n \n static void show_object_pack_hint(struct object *object, const char *name,\n-\t\t\t\t void *data UNUSED)\n+\t\t\t\t void *data)\n {\n-\tstruct object_entry *oe = packlist_find(&to_pack, &object->oid);\n-\tif (!oe)\n-\t\treturn;\n+\tenum stdin_packs_mode mode = *(enum stdin_packs_mode *)data;\n+\tif (mode == STDIN_PACKS_MODE_FOLLOW) {\n+\t\tadd_object_entry(&object->oid, object->type, name, 0);\n+\t} else {\n+\t\tstruct object_entry *oe = packlist_find(&to_pack, &object->oid);\n+\t\tif (!oe)\n+\t\t\treturn;\n \n-\t/*\n-\t * Our 'to_pack' list was constructed by iterating all objects packed in\n-\t * included packs, and so doesn't have a non-zero hash field that you\n-\t * would typically pick up during a reachability traversal.\n-\t *\n-\t * Make a best-effort attempt to fill in the ->hash and ->no_try_delta\n-\t * fields here in order to perhaps improve the delta selection\n-\t * process.\n-\t */\n-\toe->hash = pack_name_hash_fn(name);\n-\toe->no_try_delta = name && no_try_delta(name);\n+\t\t/*\n+\t\t * Our 'to_pack' list was constructed by iterating all\n+\t\t * objects packed in included packs, and so doesn't have\n+\t\t * a non-zero hash field that you would typically pick\n+\t\t * up during a reachability traversal.\n+\t\t *\n+\t\t * Make a best-effort attempt to fill in the ->hash and\n+\t\t * ->no_try_delta fields here in order to perhaps\n+\t\t * improve the delta selection process.\n+\t\t */\n+\t\toe->hash = pack_name_hash_fn(name);\n+\t\toe->no_try_delta = name && no_try_delta(name);\n \n-\tstdin_packs_hints_nr++;\n+\t\tstdin_packs_hints_nr++;\n+\t}\n }\n \n-static void show_commit_pack_hint(struct commit *commit UNUSED,\n-\t\t\t\t void *data UNUSED)\n+static void show_commit_pack_hint(struct commit *commit, void *data)\n {\n+\tenum stdin_packs_mode mode = *(enum stdin_packs_mode *)data;\n+\n+\tif (mode == STDIN_PACKS_MODE_FOLLOW) {\n+\t\tshow_object_pack_hint((struct object *)commit, \"\", data);\n+\t\treturn;\n+\t}\n+\n \t/* nothing to do; commits don't have a namehash */\n+\n }\n \n static int pack_mtime_cmp(const void *_a, const void *_b)\n@@ -3646,7 +3665,7 @@ static void read_packs_list_from_stdin(struct rev_info *revs)\n \n static void add_unreachable_loose_objects(struct rev_info *revs);\n \n-static void read_stdin_packs(int rev_list_unpacked)\n+static void read_stdin_packs(enum stdin_packs_mode mode, int rev_list_unpacked)\n {\n \tstruct rev_info revs;\n \n@@ -3678,7 +3697,7 @@ static void read_stdin_packs(int rev_list_unpacked)\n \ttraverse_commit_list(&revs,\n \t\t\t show_commit_pack_hint,\n \t\t\t show_object_pack_hint,\n-\t\t\t NULL);\n+\t\t\t &mode);\n \n \ttrace2_data_intmax(\"pack-objects\", the_repository, \"stdin_packs_found\",\n \t\t\t stdin_packs_found_nr);\n@@ -4469,6 +4488,23 @@ static int is_not_in_promisor_pack(struct commit *commit, void *data) {\n \treturn is_not_in_promisor_pack_obj((struct object *) commit, data);\n }\n \n+static int parse_stdin_packs_mode(const struct option *opt, const char *arg,\n+\t\t\t\t int unset)\n+{\n+\tenum stdin_packs_mode *mode = opt->value;\n+\n+\tif (unset)\n+\t\t*mode = STDIN_PACKS_MODE_NONE;\n+\telse if (!arg || !*arg)\n+\t\t*mode = STDIN_PACKS_MODE_STANDARD;\n+\telse if (!strcmp(arg, \"follow\"))\n+\t\t*mode = STDIN_PACKS_MODE_FOLLOW;\n+\telse\n+\t\tdie(_(\"invalid value for '%s': '%s'\"), opt->long_name, arg);\n+\n+\treturn 0;\n+}\n+\n int cmd_pack_objects(int argc,\n \t\t const char **argv,\n \t\t const char *prefix,\n@@ -4480,7 +4516,7 @@ int cmd_pack_objects(int argc,\n \tstruct strvec rp = STRVEC_INIT;\n \tint rev_list_unpacked = 0, rev_list_all = 0, rev_list_reflog = 0;\n \tint rev_list_index = 0;\n-\tint stdin_packs = 0;\n+\tenum stdin_packs_mode stdin_packs = STDIN_PACKS_MODE_NONE;\n \tstruct string_list keep_pack_list = STRING_LIST_INIT_NODUP;\n \tstruct list_objects_filter_options filter_options =\n \t\tLIST_OBJECTS_FILTER_INIT;\n@@ -4535,6 +4571,9 @@ int cmd_pack_objects(int argc,\n \t\tOPT_SET_INT_F(0, \"indexed-objects\", &rev_list_index,\n \t\t\t N_(\"include objects referred to by the index\"),\n \t\t\t 1, PARSE_OPT_NONEG),\n+\t\tOPT_CALLBACK_F(0, \"stdin-packs\", &stdin_packs, N_(\"mode\"),\n+\t\t\t N_(\"read packs from stdin\"),\n+\t\t\t PARSE_OPT_OPTARG, parse_stdin_packs_mode),\n \t\tOPT_BOOL(0, \"stdin-packs\", &stdin_packs,\n \t\t\t N_(\"read packs from stdin\")),\n \t\tOPT_BOOL(0, \"stdout\", &pack_to_stdout,\n@@ -4791,7 +4830,7 @@ int cmd_pack_objects(int argc,\n \t\tprogress_state = start_progress(the_repository,\n \t\t\t\t\t\t_(\"Enumerating objects\"), 0);\n \tif (stdin_packs) {\n-\t\tread_stdin_packs(rev_list_unpacked);\n+\t\tread_stdin_packs(stdin_packs, rev_list_unpacked);\n \t} else if (cruft) {\n \t\tread_cruft_objects();\n \t} else if (!use_internal_rev_list) {\ndiff --git a/t/t5331-pack-objects-stdin.sh b/t/t5331-pack-objects-stdin.sh\nindex 8fd07deb8d..60a2b4bc07 100755\n--- a/t/t5331-pack-objects-stdin.sh\n+++ b/t/t5331-pack-objects-stdin.sh\n@@ -236,4 +236,86 @@ test_expect_success 'pack-objects --stdin with packfiles from main and alternate\n \ttest_cmp expected-objects actual-objects\n '\n \n+packdir=.git/objects/pack\n+\n+objects_in_packs () {\n+\tfor p in \"$@\"\n+\tdo\n+\t\tgit show-index <\"$packdir/pack-$p.idx\" || return 1\n+\tdone >objects.raw &&\n+\n+\tcut -d' ' -f2 objects.raw | sort &&\n+\trm -f objects.raw\n+}\n+\n+test_expect_success '--stdin-packs=follow walks into unknown packs' '\n+\ttest_when_finished \"rm -fr repo\" &&\n+\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\tfor c in A B C D\n+\t\tdo\n+\t\t\ttest_commit \"$c\" || return 1\n+\t\tdone &&\n+\n+\t\tA=\"$(echo A | git pack-objects --revs $packdir/pack)\" &&\n+\t\tB=\"$(echo A..B | git pack-objects --revs $packdir/pack)\" &&\n+\t\tC=\"$(echo B..C | git pack-objects --revs $packdir/pack)\" &&\n+\t\tD=\"$(echo C..D | git pack-objects --revs $packdir/pack)\" &&\n+\t\ttest_commit E &&\n+\n+\t\tgit prune-packed &&\n+\n+\t\tcat >in <<-EOF &&\n+\t\tpack-$B.pack\n+\t\t^pack-$C.pack\n+\t\tpack-$D.pack\n+\t\tEOF\n+\n+\t\t# With just --stdin-packs, pack \"A\" is unknown to us, so\n+\t\t# only objects from packs \"B\" and \"D\" are included in\n+\t\t# the output pack.\n+\t\tP=$(git pack-objects --stdin-packs $packdir/pack expect &&\n+\t\tobjects_in_packs $P >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\t# But with --stdin-packs=follow, objects from both\n+\t\t# included packs reach objects from the unknown pack, so\n+\t\t# objects from pack \"A\" is included in the output pack\n+\t\t# in addition to the above.\n+\t\tP=$(git pack-objects --stdin-packs=follow $packdir/pack expect &&\n+\t\tobjects_in_packs $P >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\t# And with --unpacked, we will pick up objects from unknown\n+\t\t# packs that are reachable from loose objects. Loose object E\n+\t\t# reaches objects in pack A, but there are three excluded packs\n+\t\t# in between.\n+\t\t#\n+\t\t# The resulting pack should include objects reachable from E\n+\t\t# that are not present in packs B, C, or D, along with those\n+\t\t# present in pack A.\n+\t\tcat >in <<-EOF &&\n+\t\t^pack-$B.pack\n+\t\t^pack-$C.pack\n+\t\t^pack-$D.pack\n+\t\tEOF\n+\n+\t\tP=$(git pack-objects --stdin-packs=follow --unpacked \\\n+\t\t\t$packdir/pack expect.raw &&\n+\t\tsort expect.raw >expect &&\n+\t\tobjects_in_packs $P >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \n2.49.0.640.ga4de40e6a8\n\n"},{"id":"519124","messageId":"52a069ef48f5a573c58a369b69827a40c7712247.1748473890.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1748473889.git.me@ttaylorr.com","subject":"[PATCH v4 9/9] repack: exclude cruft pack(s) from the MIDX where possible","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-05-28T23:20:35Z","receivedAt":"2025-05-28T23:20:37Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In ddee3703b3 (builtin/repack.c: add cruft packs to MIDX during\ngeometric repack, 2022-05-20), repack began adding cruft pack(s) to the\nMIDX with '--write-midx' to ensure that the resulting MIDX was always\nclosed under reachability in order to generate reachability bitmaps.\n\nWhile the previous patch added the '--stdin-packs=follow' option to\npack-objects, it is not yet on by default. Given that, suppose you have\na once-unreachable object packed in a cruft pack, which later becomes\nreachable from one or more objects in a geometrically repacked pack.\nThat once-unreachable object *won't* appear in the new pack, since the\ncruft pack was not specified as included or excluded when the\ngeometrically repacked pack was created with 'pack-objects\n--stdin-packs' (*not* '--stdin-packs=follow', which is not on). If that\nnew pack is included in a MIDX without the cruft pack, then trying to\ngenerate bitmaps for that MIDX may fail. This happens when the bitmap\nselection process picks one or more commits which reach the\nonce-unreachable objects.\n\nTo mitigate this failure mode, commit ddee3703b3 ensures that the MIDX\nwill be closed under reachability by including cruft pack(s). If cruft\npack(s) were not included, we would fail to generate a MIDX bitmap. But\nddee3703b3 alludes to the fact that this is sub-optimal by saying\n\n [...] it's desirable to avoid including cruft packs in the MIDX\n because it causes the MIDX to store a bunch of objects which are\n likely to get thrown away.\n\n, which is true, but hides an even larger problem. If repositories\nrarely prune their unreachable objects and/or have many of them, the\nMIDX must keep track of a large number of objects which bloats the MIDX\nand slows down object lookup.\n\nThis is doubly unfortunate because the vast majority of objects in cruft\npack(s) are unlikely to be read. But any object lookups that go through\nthe MIDX must binary search over them anyway, slowing down object\nlookups using the MIDX.\n\nThis patch causes geometrically-repacked packs to contain a copy of any\nonce-unreachable object(s) with 'git pack-objects --stdin-packs=follow',\nallowing us to avoid including any cruft packs in the MIDX. This is\nbecause a sequence of geometrically-repacked packs that were all\ngenerated with '--stdin-packs=follow' are guaranteed to have their union\nbe closed under reachability.\n\nNote that you cannot guarantee that a collection of packs is closed\nunder reachability if not all of them were generated with \"following\" as\nabove. One tell-tale sign that not all geometrically-repacked packs in\nthe MIDX were generated with \"following\" is to see if there is a pack in\nthe existing MIDX that is not going to be somehow represented (either\nverbatim or as part of a geometric rollup) in the new MIDX.\n\nIf there is, then starting to generate packs with \"following\" during\ngeometric repacking won't work, since it's open to the same race as\ndescribed above.\n\nBut if you're starting from scratch (e.g., building the first MIDX after\nan all-into-one '--cruft' repack), then you can guarantee that the union\nof subsequently generated packs from geometric repacking *is* closed\nunder reachability.\n\nDetect when this is the case and avoid including cruft packs in the MIDX\nwhere possible. The existing behavior remains the default, and the new\nbehavior is available with the config 'repack.midxMustIncludeCruft' set\nto 'false'.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/config/repack.adoc | 7 ++\n builtin/repack.c | 163 +++++++++++++++++++++++++++----\n t/t7704-repack-cruft.sh | 90 +++++++++++++++++\n 3 files changed, 242 insertions(+), 18 deletions(-)\n\ndiff --git a/Documentation/config/repack.adoc b/Documentation/config/repack.adoc\nindex c79af6d7b8..e9e78dcb19 100644\n--- a/Documentation/config/repack.adoc\n+++ b/Documentation/config/repack.adoc\n@@ -39,3 +39,10 @@ repack.cruftThreads::\n \ta cruft pack and the respective parameters are not given over\n \tthe command line. See similarly named `pack.*` configuration\n \tvariables for defaults and meaning.\n+\n+repack.midxMustContainCruft::\n+\tWhen set to true, linkgit:git-repack[1] will unconditionally include\n+\tcruft pack(s), if any, in the multi-pack index when invoked with\n+\t`--write-midx`. When false, cruft packs are only included in the MIDX\n+\twhen necessary (e.g., because they might be required to form a\n+\treachability closure with MIDX bitmaps). Defaults to true.\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex f3330ade7b..c9e2e3d04d 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -39,6 +39,7 @@ static int write_bitmaps = -1;\n static int use_delta_islands;\n static int run_update_server_info = 1;\n static char *packdir, *packtmp_name, *packtmp;\n+static int midx_must_contain_cruft = 1;\n \n static const char *const git_repack_usage[] = {\n \tN_(\"git repack [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\\n\"\n@@ -107,6 +108,10 @@ static int repack_config(const char *var, const char *value,\n \t\tfree(cruft_po_args->threads);\n \t\treturn git_config_string(&cruft_po_args->threads, var, value);\n \t}\n+\tif (!strcmp(var, \"repack.midxmustcontaincruft\")) {\n+\t\tmidx_must_contain_cruft = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n \treturn git_default_config(var, value, ctx, cb);\n }\n \n@@ -687,6 +692,77 @@ static void free_pack_geometry(struct pack_geometry *geometry)\n \tfree(geometry->pack);\n }\n \n+static int midx_has_unknown_packs(char **midx_pack_names,\n+\t\t\t\t size_t midx_pack_names_nr,\n+\t\t\t\t struct string_list *include,\n+\t\t\t\t struct pack_geometry *geometry,\n+\t\t\t\t struct existing_packs *existing)\n+{\n+\tsize_t i;\n+\n+\tstring_list_sort(include);\n+\n+\tfor (i = 0; i < midx_pack_names_nr; i++) {\n+\t\tconst char *pack_name = midx_pack_names[i];\n+\n+\t\t/*\n+\t\t * Determine whether or not each MIDX'd pack from the existing\n+\t\t * MIDX (if any) is represented in the new MIDX. For each pack\n+\t\t * in the MIDX, it must either be:\n+\t\t *\n+\t\t * - In the \"include\" list of packs to be included in the new\n+\t\t * MIDX. Note this function is called before the include\n+\t\t * list is populated with any cruft pack(s).\n+\t\t *\n+\t\t * - Below the geometric split line (if using pack geometry),\n+\t\t * indicating that the pack won't be included in the new\n+\t\t * MIDX, but its contents were rolled up as part of the\n+\t\t * geometric repack.\n+\t\t *\n+\t\t * - In the existing non-kept packs list (if not using pack\n+\t\t * geometry), and marked as non-deleted.\n+\t\t */\n+\t\tif (string_list_has_string(include, pack_name)) {\n+\t\t\tcontinue;\n+\t\t} else if (geometry) {\n+\t\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\t\tuint32_t j;\n+\n+\t\t\tfor (j = 0; j < geometry->split; j++) {\n+\t\t\t\tstrbuf_reset(&buf);\n+\t\t\t\tstrbuf_addstr(&buf, pack_basename(geometry->pack[j]));\n+\t\t\t\tstrbuf_strip_suffix(&buf, \".pack\");\n+\t\t\t\tstrbuf_addstr(&buf, \".idx\");\n+\n+\t\t\t\tif (!strcmp(pack_name, buf.buf)) {\n+\t\t\t\t\tstrbuf_release(&buf);\n+\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\tstrbuf_release(&buf);\n+\n+\t\t\tif (j < geometry->split)\n+\t\t\t\tcontinue;\n+\t\t} else {\n+\t\t\tstruct string_list_item *item;\n+\n+\t\t\titem = string_list_lookup(&existing->non_kept_packs,\n+\t\t\t\t\t\t pack_name);\n+\t\t\tif (item && !pack_is_marked_for_deletion(item))\n+\t\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/*\n+\t\t * If we got to this point, the MIDX includes some pack that we\n+\t\t * don't know about.\n+\t\t */\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n struct midx_snapshot_ref_data {\n \tstruct tempfile *f;\n \tstruct oidset seen;\n@@ -755,6 +831,8 @@ static void midx_snapshot_refs(struct tempfile *f)\n \n static void midx_included_packs(struct string_list *include,\n \t\t\t\tstruct existing_packs *existing,\n+\t\t\t\tchar **midx_pack_names,\n+\t\t\t\tsize_t midx_pack_names_nr,\n \t\t\t\tstruct string_list *names,\n \t\t\t\tstruct pack_geometry *geometry)\n {\n@@ -808,26 +886,56 @@ static void midx_included_packs(struct string_list *include,\n \t\t}\n \t}\n \n-\tfor_each_string_list_item(item, &existing->cruft_packs) {\n+\tif (midx_must_contain_cruft ||\n+\t midx_has_unknown_packs(midx_pack_names, midx_pack_names_nr,\n+\t\t\t\t include, geometry, existing)) {\n \t\t/*\n-\t\t * When doing a --geometric repack, there is no need to check\n-\t\t * for deleted packs, since we're by definition not doing an\n-\t\t * ALL_INTO_ONE repack (hence no packs will be deleted).\n-\t\t * Otherwise we must check for and exclude any packs which are\n-\t\t * enqueued for deletion.\n+\t\t * If there are one or more unknown pack(s) present (see\n+\t\t * midx_has_unknown_packs() for what makes a pack\n+\t\t * \"unknown\") in the MIDX before the repack, keep them\n+\t\t * as they may be required to form a reachability\n+\t\t * closure if the MIDX is bitmapped.\n \t\t *\n-\t\t * So we could omit the conditional below in the --geometric\n-\t\t * case, but doing so is unnecessary since no packs are marked\n-\t\t * as pending deletion (since we only call\n-\t\t * `mark_packs_for_deletion()` when doing an all-into-one\n-\t\t * repack).\n+\t\t * For example, a cruft pack can be required to form a\n+\t\t * reachability closure if the MIDX is bitmapped and one\n+\t\t * or more of the bitmap's selected commits reaches a\n+\t\t * once-cruft object that was later made reachable.\n \t\t */\n-\t\tif (pack_is_marked_for_deletion(item))\n-\t\t\tcontinue;\n+\t\tfor_each_string_list_item(item, &existing->cruft_packs) {\n+\t\t\t/*\n+\t\t\t * When doing a --geometric repack, there is no\n+\t\t\t * need to check for deleted packs, since we're\n+\t\t\t * by definition not doing an ALL_INTO_ONE\n+\t\t\t * repack (hence no packs will be deleted).\n+\t\t\t * Otherwise we must check for and exclude any\n+\t\t\t * packs which are enqueued for deletion.\n+\t\t\t *\n+\t\t\t * So we could omit the conditional below in the\n+\t\t\t * --geometric case, but doing so is unnecessary\n+\t\t\t * since no packs are marked as pending\n+\t\t\t * deletion (since we only call\n+\t\t\t * `mark_packs_for_deletion()` when doing an\n+\t\t\t * all-into-one repack).\n+\t\t\t */\n+\t\t\tif (pack_is_marked_for_deletion(item))\n+\t\t\t\tcontinue;\n \n-\t\tstrbuf_reset(&buf);\n-\t\tstrbuf_addf(&buf, \"%s.idx\", item->string);\n-\t\tstring_list_insert(include, buf.buf);\n+\t\t\tstrbuf_reset(&buf);\n+\t\t\tstrbuf_addf(&buf, \"%s.idx\", item->string);\n+\t\t\tstring_list_insert(include, buf.buf);\n+\t\t}\n+\t} else {\n+\t\t/*\n+\t\t * Modern versions of Git (with the appropriate\n+\t\t * configuration setting) will write new copies of\n+\t\t * once-cruft objects when doing a --geometric repack.\n+\t\t *\n+\t\t * If the MIDX has no cruft pack, new packs written\n+\t\t * during a --geometric repack will not rely on the\n+\t\t * cruft pack to form a reachability closure, so we can\n+\t\t * avoid including them in the MIDX in that case.\n+\t\t */\n+\t\t;\n \t}\n \n \tstrbuf_release(&buf);\n@@ -1142,6 +1250,8 @@ int cmd_repack(int argc,\n \tstruct tempfile *refs_snapshot = NULL;\n \tint i, ext, ret;\n \tint show_progress;\n+\tchar **midx_pack_names = NULL;\n+\tsize_t midx_pack_names_nr = 0;\n \n \t/* variables to be filled by option parsing */\n \tint delete_redundant = 0;\n@@ -1356,7 +1466,10 @@ int cmd_repack(int argc,\n \t\t !(pack_everything & PACK_CRUFT))\n \t\t\tstrvec_push(&cmd.args, \"--pack-loose-unreachable\");\n \t} else if (geometry.split_factor) {\n-\t\tstrvec_push(&cmd.args, \"--stdin-packs\");\n+\t\tif (midx_must_contain_cruft)\n+\t\t\tstrvec_push(&cmd.args, \"--stdin-packs\");\n+\t\telse\n+\t\t\tstrvec_push(&cmd.args, \"--stdin-packs=follow\");\n \t\tstrvec_push(&cmd.args, \"--unpacked\");\n \t} else {\n \t\tstrvec_push(&cmd.args, \"--unpacked\");\n@@ -1478,6 +1591,16 @@ int cmd_repack(int argc,\n \n \tstring_list_sort(&names);\n \n+\tif (get_local_multi_pack_index(the_repository)) {\n+\t\tuint32_t i;\n+\t\tstruct multi_pack_index *m =\n+\t\t\tget_local_multi_pack_index(the_repository);\n+\n+\t\tALLOC_ARRAY(midx_pack_names, m->num_packs);\n+\t\tfor (i = 0; i < m->num_packs; i++)\n+\t\t\tmidx_pack_names[midx_pack_names_nr++] = xstrdup(m->pack_names[i]);\n+\t}\n+\n \tclose_object_store(the_repository->objects);\n \n \t/*\n@@ -1519,7 +1642,8 @@ int cmd_repack(int argc,\n \n \tif (write_midx) {\n \t\tstruct string_list include = STRING_LIST_INIT_DUP;\n-\t\tmidx_included_packs(&include, &existing, &names, &geometry);\n+\t\tmidx_included_packs(&include, &existing, midx_pack_names,\n+\t\t\t\t midx_pack_names_nr, &names, &geometry);\n \n \t\tret = write_midx_included_packs(&include, &geometry, &names,\n \t\t\t\t\t\trefs_snapshot ? get_tempfile_path(refs_snapshot) : NULL,\n@@ -1570,6 +1694,9 @@ int cmd_repack(int argc,\n \tstring_list_clear(&names, 1);\n \texisting_packs_release(&existing);\n \tfree_pack_geometry(&geometry);\n+\tfor (size_t i = 0; i < midx_pack_names_nr; i++)\n+\t\tfree(midx_pack_names[i]);\n+\tfree(midx_pack_names);\n \tpack_objects_args_release(&po_args);\n \tpack_objects_args_release(&cruft_po_args);\n \ndiff --git a/t/t7704-repack-cruft.sh b/t/t7704-repack-cruft.sh\nindex 8aebfb45f5..2b0a55f8fd 100755\n--- a/t/t7704-repack-cruft.sh\n+++ b/t/t7704-repack-cruft.sh\n@@ -724,4 +724,94 @@ test_expect_success 'cruft repack respects --quiet' '\n \t)\n '\n \n+setup_cruft_exclude_tests() {\n+\tgit init \"$1\" &&\n+\t(\n+\t\tcd \"$1\" &&\n+\n+\t\tgit config repack.midxMustContainCruft false &&\n+\n+\t\ttest_commit one &&\n+\n+\t\ttest_commit --no-tag two &&\n+\t\ttwo=\"$(git rev-parse HEAD)\" &&\n+\t\ttest_commit --no-tag three &&\n+\t\tthree=\"$(git rev-parse HEAD)\" &&\n+\t\tgit reset --hard one &&\n+\t\tgit reflog expire --all --expire=all &&\n+\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack --cruft -d &&\n+\n+\t\tgit merge $two &&\n+\t\ttest_commit four\n+\t)\n+}\n+\n+test_expect_success 'repack --write-midx excludes cruft where possible' '\n+\tsetup_cruft_exclude_tests exclude-cruft-when-possible &&\n+\t(\n+\t\tcd exclude-cruft-when-possible &&\n+\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 \\\n+\t\tgit repack -d --geometric=2 --write-midx --write-bitmap-index &&\n+\n+\t\ttest-tool read-midx --show-objects $objdir >midx &&\n+\t\tcruft=\"$(ls $packdir/*.mtimes)\" &&\n+\t\ttest_grep ! \"$(basename \"$cruft\" .mtimes).idx\" midx &&\n+\n+\t\tgit rev-list --all --objects --no-object-names >reachable.raw &&\n+\t\tsort reachable.raw >reachable.objects &&\n+\t\tawk \"/\\.pack$/ { print \\$1 }\" midx.objects &&\n+\n+\t\ttest_cmp reachable.objects midx.objects\n+\t)\n+'\n+\n+test_expect_success 'repack --write-midx includes cruft when instructed' '\n+\tsetup_cruft_exclude_tests exclude-cruft-when-instructed &&\n+\t(\n+\t\tcd exclude-cruft-when-instructed &&\n+\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 \\\n+\t\tgit -c repack.midxMustContainCruft=true repack \\\n+\t\t\t-d --geometric=2 --write-midx --write-bitmap-index &&\n+\n+\t\ttest-tool read-midx --show-objects $objdir >midx &&\n+\t\tcruft=\"$(ls $packdir/*.mtimes)\" &&\n+\t\ttest_grep \"$(basename \"$cruft\" .mtimes).idx\" midx &&\n+\n+\t\tgit cat-file --batch-check=\"%(objectname)\" --batch-all-objects \\\n+\t\t\t>all.objects &&\n+\t\tawk \"/\\.pack$/ { print \\$1 }\" midx.objects &&\n+\n+\t\ttest_cmp all.objects midx.objects\n+\t)\n+'\n+\n+test_expect_success 'repack --write-midx includes cruft when necessary' '\n+\tsetup_cruft_exclude_tests exclude-cruft-when-necessary &&\n+\t(\n+\t\tcd exclude-cruft-when-necessary &&\n+\n+\t\ttest_path_is_file $(ls $packdir/pack-*.mtimes) &&\n+\t\tls $packdir/pack-*.idx | sort >packs.all &&\n+\t\tgrep -o \"pack-.*\\.idx$\" packs.all >in &&\n+\n+\t\tgit multi-pack-index write --stdin-packs --bitmap midx &&\n+\t\tawk \"/\\.pack$/ { print \\$1 }\" midx.objects &&\n+\t\tgit cat-file --batch-all-objects --batch-check=\"%(objectname)\" \\\n+\t\t\t>expect.objects &&\n+\t\ttest_cmp expect.objects midx.objects &&\n+\n+\t\tgrep \"^pack-\" midx >midx.packs &&\n+\t\ttest_line_count = \"$(($(wc -l Range-diff against v3:\n\nHmm. My tool for submitting patches botched the range-diff here. The\ncorrect range-diff is:\n\n1: 986bef29b5 ! 1: 2753e29648 pack-objects: limit scope in 'add_object_entry_from_pack()'\n @@ builtin/pack-objects.c: static int add_object_entry_from_pack(const struct objec\n \tif (p) {\n -\t\tstruct rev_info *revs = _data;\n \t\tstruct object_info oi = OBJECT_INFO_INIT;\n --\n +\n \t\toi.typep = &type;\n -+\n - \t\tif (packed_object_info(the_repository, p, ofs, &oi) < 0) {\n +@@ builtin/pack-objects.c: static int add_object_entry_from_pack(const struct object_id *oid,\n \t\t\tdie(_(\"could not get type of object %s in pack %s\"),\n \t\t\t oid_to_hex(oid), p->pack_name);\n \t\t} else if (type == OBJ_COMMIT) {\n2: 6f8fe8a4e1 = 2: 32b49d9073 pack-objects: factor out handling '--stdin-packs'\n3: 2a235461a6 = 3: a797ff3a83 pack-objects: declare 'rev_info' for '--stdin-packs' earlier\n4: 240e90b68d = 4: 29bf05633a pack-objects: perform name-hash traversal for unpacked objects\n5: 9a18fa2e52 = 5: 0696fa1736 pack-objects: fix typo in 'show_object_pack_hint()'\n6: 6c997853f1 = 6: 1cc45b4472 pack-objects: swap 'show_{object,commit}_pack_hint'\n7: 0ff699f056 = 7: 3e3d929bd0 pack-objects: introduce '--stdin-packs=follow'\n8: 58891101f3 ! 8: 52a069ef48 repack: exclude cruft pack(s) from the MIDX where possible\n @@ Commit message\n MIDX with '--write-midx' to ensure that the resulting MIDX was always\n closed under reachability in order to generate reachability bitmaps.\n\n - Suppose (prior to this patch) you have a once-unreachable object packed\n - in a cruft pack, which later on becomes reachable from one or more\n - objects in a geometrically repacked pack. That once-unreachable object\n - *won't* appear in the new pack, since the cruft pack was specified as\n - neither included nor excluded to 'pack-objects --stdin-packs'. If the\n + While the previous patch added the '--stdin-packs=follow' option to\n + pack-objects, it is not yet on by default. Given that, suppose you have\n + a once-unreachable object packed in a cruft pack, which later becomes\n + reachable from one or more objects in a geometrically repacked pack.\n + That once-unreachable object *won't* appear in the new pack, since the\n + cruft pack was not specified as included or excluded when the\n + geometrically repacked pack was created with 'pack-objects\n + --stdin-packs' (*not* '--stdin-packs=follow', which is not on). If that\n new pack is included in a MIDX without the cruft pack, then trying to\n generate bitmaps for that MIDX may fail. This happens when the bitmap\n selection process picks one or more commits which reach the\n - once-unreachable objects, commit ddee3703b3 ensures that the MIDX will\n - be closed under reachability. Without it, we would fail to generate a\n - MIDX bitmap.\n + once-unreachable objects.\n\n + To mitigate this failure mode, commit ddee3703b3 ensures that the MIDX\n + will be closed under reachability by including cruft pack(s). If cruft\n + pack(s) were not included, we would fail to generate a MIDX bitmap. But\n ddee3703b3 alludes to the fact that this is sub-optimal by saying\n\n [...] it's desirable to avoid including cruft packs in the MIDX\n\nThanks,\nTaylor\n"},{"id":"519128","messageId":"CABPp-BHv938Cn4LcwBDq=jZwrFuOGMJhAenUJiQ6JZNTMe4SUQ@mail.gmail.com","threadId":"63286","inReplyTo":"aDelMNw4AK0L9XHu@nand.local","subject":"Re: [PATCH v4 0/9] repack: avoid MIDX'ing cruft pack(s) where possible","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-05-29T00:15:43Z","receivedAt":"2025-05-29T00:15:54Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, May 28, 2025 at 5:07 PM Taylor Blau wrote:\n>\n> On Wed, May 28, 2025 at 07:20:07PM -0400, Taylor Blau wrote:\n> > Range-diff against v3:\n>\n> Hmm. My tool for submitting patches botched the range-diff here. The\n> correct range-diff is:\n>\n> 1: 986bef29b5 ! 1: 2753e29648 pack-objects: limit scope in 'add_object_entry_from_pack()'\n> @@ builtin/pack-objects.c: static int add_object_entry_from_pack(const struct objec\n> if (p) {\n> - struct rev_info *revs = _data;\n> struct object_info oi = OBJECT_INFO_INIT;\n> --\n> +\n> oi.typep = &type;\n> -+\n> - if (packed_object_info(the_repository, p, ofs, &oi) < 0) {\n> +@@ builtin/pack-objects.c: static int add_object_entry_from_pack(const struct object_id *oid,\n> die(_(\"could not get type of object %s in pack %s\"),\n> oid_to_hex(oid), p->pack_name);\n> } else if (type == OBJ_COMMIT) {\n> 2: 6f8fe8a4e1 = 2: 32b49d9073 pack-objects: factor out handling '--stdin-packs'\n> 3: 2a235461a6 = 3: a797ff3a83 pack-objects: declare 'rev_info' for '--stdin-packs' earlier\n> 4: 240e90b68d = 4: 29bf05633a pack-objects: perform name-hash traversal for unpacked objects\n> 5: 9a18fa2e52 = 5: 0696fa1736 pack-objects: fix typo in 'show_object_pack_hint()'\n> 6: 6c997853f1 = 6: 1cc45b4472 pack-objects: swap 'show_{object,commit}_pack_hint'\n> 7: 0ff699f056 = 7: 3e3d929bd0 pack-objects: introduce '--stdin-packs=follow'\n> 8: 58891101f3 ! 8: 52a069ef48 repack: exclude cruft pack(s) from the MIDX where possible\n> @@ Commit message\n> MIDX with '--write-midx' to ensure that the resulting MIDX was always\n> closed under reachability in order to generate reachability bitmaps.\n>\n> - Suppose (prior to this patch) you have a once-unreachable object packed\n> - in a cruft pack, which later on becomes reachable from one or more\n> - objects in a geometrically repacked pack. That once-unreachable object\n> - *won't* appear in the new pack, since the cruft pack was specified as\n> - neither included nor excluded to 'pack-objects --stdin-packs'. If the\n> + While the previous patch added the '--stdin-packs=follow' option to\n> + pack-objects, it is not yet on by default. Given that, suppose you have\n> + a once-unreachable object packed in a cruft pack, which later becomes\n> + reachable from one or more objects in a geometrically repacked pack.\n> + That once-unreachable object *won't* appear in the new pack, since the\n> + cruft pack was not specified as included or excluded when the\n> + geometrically repacked pack was created with 'pack-objects\n> + --stdin-packs' (*not* '--stdin-packs=follow', which is not on). If that\n> new pack is included in a MIDX without the cruft pack, then trying to\n> generate bitmaps for that MIDX may fail. This happens when the bitmap\n> selection process picks one or more commits which reach the\n> - once-unreachable objects, commit ddee3703b3 ensures that the MIDX will\n> - be closed under reachability. Without it, we would fail to generate a\n> - MIDX bitmap.\n> + once-unreachable objects.\n>\n> + To mitigate this failure mode, commit ddee3703b3 ensures that the MIDX\n> + will be closed under reachability by including cruft pack(s). If cruft\n> + pack(s) were not included, we would fail to generate a MIDX bitmap. But\n> ddee3703b3 alludes to the fact that this is sub-optimal by saying\n>\n> [...] it's desirable to avoid including cruft packs in the MIDX\n>\n> Thanks,\n> Taylor\n\nThanks, this version addresses all the issues I brought up; this round\nlooks good to me.\n"},{"id":"520401","messageId":"lfrgmt2ukanevmcctzsnc422iv2l2nb3qmiddpsj6jnyvz4m4s@5eohhsm6knw3","threadId":"63286","inReplyTo":"52a069ef48f5a573c58a369b69827a40c7712247.1748473890.git.me@ttaylorr.com","subject":"Re: [PATCH v4 9/9] repack: exclude cruft pack(s) from the MIDX where possible","fromName":"Carlo Marcelo Arenas Belón","fromEmail":"carenas@gmail.com","sentAt":"2025-06-19T11:33:02Z","receivedAt":"2025-06-19T11:33:04Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"On Wed, May 28, 2025 at 07:20:35PM -0800, Taylor Blau wrote:\n> diff --git a/t/t7704-repack-cruft.sh b/t/t7704-repack-cruft.sh\n> index 8aebfb45f5..2b0a55f8fd 100755\n> --- a/t/t7704-repack-cruft.sh\n> +++ b/t/t7704-repack-cruft.sh\n> @@ -724,4 +724,94 @@ test_expect_success 'cruft repack respects --quiet' '\n> \t)\n> '\n> \n> +setup_cruft_exclude_tests() {\n> +\tgit init \"$1\" &&\n> +\t(\n> +\t\tcd \"$1\" &&\n> +\n> +\t\tgit config repack.midxMustContainCruft false &&\n> +\n> +\t\ttest_commit one &&\n> +\n> +\t\ttest_commit --no-tag two &&\n> +\t\ttwo=\"$(git rev-parse HEAD)\" &&\n> +\t\ttest_commit --no-tag three &&\n> +\t\tthree=\"$(git rev-parse HEAD)\" &&\n> +\t\tgit reset --hard one &&\n> +\t\tgit reflog expire --all --expire=all &&\n> +\n> +\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack --cruft -d &&\n> +\n> +\t\tgit merge $two &&\n> +\t\ttest_commit four\n> +\t)\n> +}\n> +\n> +test_expect_success 'repack --write-midx excludes cruft where possible' '\n> +\tsetup_cruft_exclude_tests exclude-cruft-when-possible &&\n> +\t(\n> +\t\tcd exclude-cruft-when-possible &&\n> +\n> +\t\tGIT_TEST_MULTI_PACK_INDEX=0 \\\n> +\t\tgit repack -d --geometric=2 --write-midx --write-bitmap-index &&\n> +\n> +\t\ttest-tool read-midx --show-objects $objdir >midx &&\n> +\t\tcruft=\"$(ls $packdir/*.mtimes)\" &&\n> +\t\ttest_grep ! \"$(basename \"$cruft\" .mtimes).idx\" midx &&\n> +\n> +\t\tgit rev-list --all --objects --no-object-names >reachable.raw &&\n> +\t\tsort reachable.raw >reachable.objects &&\n> +\t\tawk \"/\\.pack$/ { print \\$1 }\" midx.objects &&\n> +\n> +\t\ttest_cmp reachable.objects midx.objects\n> +\t)\n> +'\n> +\n> +test_expect_success 'repack --write-midx includes cruft when instructed' '\n> +\tsetup_cruft_exclude_tests exclude-cruft-when-instructed &&\n> +\t(\n> +\t\tcd exclude-cruft-when-instructed &&\n> +\n> +\t\tGIT_TEST_MULTI_PACK_INDEX=0 \\\n> +\t\tgit -c repack.midxMustContainCruft=true repack \\\n> +\t\t\t-d --geometric=2 --write-midx --write-bitmap-index &&\n> +\n> +\t\ttest-tool read-midx --show-objects $objdir >midx &&\n> +\t\tcruft=\"$(ls $packdir/*.mtimes)\" &&\n> +\t\ttest_grep \"$(basename \"$cruft\" .mtimes).idx\" midx &&\n> +\n> +\t\tgit cat-file --batch-check=\"%(objectname)\" --batch-all-objects \\\n> +\t\t\t>all.objects &&\n> +\t\tawk \"/\\.pack$/ { print \\$1 }\" midx.objects &&\n> +\n> +\t\ttest_cmp all.objects midx.objects\n> +\t)\n> +'\n> +\n> +test_expect_success 'repack --write-midx includes cruft when necessary' '\n> +\tsetup_cruft_exclude_tests exclude-cruft-when-necessary &&\n> +\t(\n> +\t\tcd exclude-cruft-when-necessary &&\n> +\n> +\t\ttest_path_is_file $(ls $packdir/pack-*.mtimes) &&\n> +\t\tls $packdir/pack-*.idx | sort >packs.all &&\n> +\t\tgrep -o \"pack-.*\\.idx$\" packs.all >in &&\n\nthis is introducing `grep -o` to our codebase, which is not in POSIX and\ntherefore will not be portable (ex: AIX)\n\nsomething like (untested) :\n\n\tsed -n '/\\(pack-.*\\.idx$\\)/\\1/p' packs.all >in\n\nwill likely work the same and be more portable.\n\nCarlo\n"},{"id":"520405","messageId":"20250619130807.48057-1-carenas@gmail.com","threadId":"63286","inReplyTo":"52a069ef48f5a573c58a369b69827a40c7712247.1748473890.git.me@ttaylorr.com","subject":"[PATCH] fixup! repack: exclude cruft pack(s) from the MIDX where possible","fromName":"Carlo Marcelo Arenas Belón","fromEmail":"carenas@gmail.com","sentAt":"2025-06-19T13:08:07Z","receivedAt":"2025-06-19T13:08:18Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"In a previous commit, `grep -o` was introduced as part of t7704.\n\nPOSIX doesn't have that flag as defined and while it is a popular\none is not available at least in the latest release of AIX.\n\nUse a sed equivalent that ought to be more portable.\n\nSigned-off-by: Carlo Marcelo Arenas Belón \n---\n t/t7704-repack-cruft.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t7704-repack-cruft.sh b/t/t7704-repack-cruft.sh\nindex 2b0a55f8fd..3df6b53cd3 100755\n--- a/t/t7704-repack-cruft.sh\n+++ b/t/t7704-repack-cruft.sh\n@@ -795,7 +795,7 @@ test_expect_success 'repack --write-midx includes cruft when necessary' '\n \n \t\ttest_path_is_file $(ls $packdir/pack-*.mtimes) &&\n \t\tls $packdir/pack-*.idx | sort >packs.all &&\n-\t\tgrep -o \"pack-.*\\.idx$\" packs.all >in &&\n+\t\tsed -n \"s/.*\\(pack-.*\\.idx\\)$/\\1/p\" packs.all >in &&\n \n \t\tgit multi-pack-index write --stdin-packs --bitmap writes:\n\n> In a previous commit, `grep -o` was introduced as part of t7704.\n>\n> POSIX doesn't have that flag as defined and while it is a popular\n> one is not available at least in the latest release of AIX.\n>\n> Use a sed equivalent that ought to be more portable.\n\nOK. The patterns are not exactly the same but as long as we know\n$packdir does *not* contain a substring \"pack-\", it should be OK.\n\nAs the topic is not even in 'next', perhaps a refresh can squash\nthis change in?\n\n> \t\tls $packdir/pack-*.idx | sort >packs.all &&\n> -\t\tgrep -o \"pack-.*\\.idx$\" packs.all >in &&\n> +\t\tsed -n \"s/.*\\(pack-.*\\.idx\\)$/\\1/p\" packs.all >in &&\n> \t\tgit multi-pack-index write --stdin-packs --bitmap Carlo Marcelo Arenas Belón writes:\n>\n> > In a previous commit, `grep -o` was introduced as part of t7704.\n> >\n> > POSIX doesn't have that flag as defined and while it is a popular\n> > one is not available at least in the latest release of AIX.\n> >\n> > Use a sed equivalent that ought to be more portable.\n>\n> OK. The patterns are not exactly the same but as long as we know\n> $packdir does *not* contain a substring \"pack-\", it should be OK.\n\nGood catch; thanks, both.\n\n> As the topic is not even in 'next', perhaps a refresh can squash\n> this change in?\n\nYep, I'll squash this in and adjust the other \"grep -o\" use in this\nseries.\n\n> > \t\tls $packdir/pack-*.idx | sort >packs.all &&\n> > -\t\tgrep -o \"pack-.*\\.idx$\" packs.all >in &&\n> > +\t\tsed -n \"s/.*\\(pack-.*\\.idx\\)$/\\1/p\" packs.all >in &&\n> > \t\tgit multi-pack-index write --stdin-packs --bitmap \n> I do not quite see the need for temporary files or \"grep/sed\" here,\n> though.\n>\n> \t\t(cd \"$packdir\" && ls pack-*.idx) |\n> \t\tsort |\n> \t\tgit multi-pack-index write --stdin-packs --bitmap &&\n\nThe grep/sed is unnecessary (I was just trying to be clever and avoid a\nsub-shell), but the packs.all temporary file is still needed since we\nrun comm against it later on in the test script.\n\n> Tangent to this discussion, but I just noticed that\n>\n> $ git multi-pack-index -h\n>\n> lacks quite a lot of information. Perhaps it needs updating?\n\nDefinitely some good #leftoverbits ;-).\n\nThanks,\nTaylor\n"},{"id":"520436","messageId":"cover.1750375803.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744413969.git.me@ttaylorr.com","subject":"[PATCH v5 0/9] repack: avoid MIDX'ing cruft pack(s) where possible","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-06-19T23:30:05Z","receivedAt":"2025-06-19T23:30:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"(Junio: I rebased this onto the tip of current 'master', which is\nf9aa0eedb3 (Start 2.51 cycle, the first batch, 2025-06-17) at the time\nof writing.)\n\nHere is another small reroll of my series to create MIDXs that do not\ninclude a repository's cruft pack(s).\n\nThe bulk of the series is unchanged, save for a couple of points I'll\ndiscuss below. As usual, a complete-range diff is included below for\nconvenience. There are two tiny but important changes that I included\nhere as a result of rolling this out to GitHub's infrastructure. They\nare:\n\n - Tolerating missing objects in the follow-on traversal. When missing\n an object in a non \"follow\" `--stdin-packs` traversal, we didn't\n notice it because that traversal does not add discovered objects to\n the packing list. That is not the case for `--stdin-packs=follow`\n traversals which do. This has been corrected, and is explained in\n detail in the penultimate commit.\n\n - The behavior when writing a MIDX after a noop repack. There are\n some cases here where we may need to include the cruft pack in the\n MIDX depending on how the existing packs are structured. This is\n corrected and explained in more detail in the final commit.\n\nThere is an additional change that Carlo Marcelo Arenas Belón pointed\nout[1] that drops the non-portable \"grep -o\" invocations added by the\nprevious round.\n\nI promised in the last round that I'd report any interesting findings\nafter deploying this to GitHub's infrastructure. Here they are:\n\n - On unreachable object-heavy repositories, I was able to measure\n about a ~5% performance improvement across many operations.\n\n - GitHub's average time to repack a repository went down by about\n ~20% as a result of rolling this change out everywhere.\n\nThanks in advance for any review :-).\n\n[1]: \n\nTaylor Blau (9):\n pack-objects: use standard option incompatibility functions\n pack-objects: limit scope in 'add_object_entry_from_pack()'\n pack-objects: factor out handling '--stdin-packs'\n pack-objects: declare 'rev_info' for '--stdin-packs' earlier\n pack-objects: perform name-hash traversal for unpacked objects\n pack-objects: fix typo in 'show_object_pack_hint()'\n pack-objects: swap 'show_{object,commit}_pack_hint'\n pack-objects: introduce '--stdin-packs=follow'\n repack: exclude cruft pack(s) from the MIDX where possible\n\n Documentation/config/repack.adoc | 7 +\n Documentation/git-pack-objects.adoc | 10 +-\n builtin/pack-objects.c | 193 ++++++++++++++++++----------\n builtin/repack.c | 184 +++++++++++++++++++++++---\n t/t5331-pack-objects-stdin.sh | 122 +++++++++++++++++-\n t/t7704-repack-cruft.sh | 145 +++++++++++++++++++++\n 6 files changed, 570 insertions(+), 91 deletions(-)\n\nRange-diff against v4:\n 1: f8b31c6a8d = 1: ebaf47262a pack-objects: use standard option incompatibility functions\n 2: 2753e29648 = 2: eaa1f41b25 pack-objects: limit scope in 'add_object_entry_from_pack()'\n 3: 32b49d9073 = 3: 8d0492a80d pack-objects: factor out handling '--stdin-packs'\n 4: a797ff3a83 = 4: 3d5d3b78b2 pack-objects: declare 'rev_info' for '--stdin-packs' earlier\n 5: 29bf05633a = 5: 2a3676cb86 pack-objects: perform name-hash traversal for unpacked objects\n 6: 0696fa1736 = 6: bcbce75695 pack-objects: fix typo in 'show_object_pack_hint()'\n 7: 1cc45b4472 = 7: c8cf316c50 pack-objects: swap 'show_{object,commit}_pack_hint'\n 8: 3e3d929bd0 ! 8: b81b6213e8 pack-objects: introduce '--stdin-packs=follow'\n @@ Commit message\n copy in the cruft pack, otherwise we cannot generate reachability\n bitmaps for any commits which reach that object.\n \n + Note that the traversal here is best-effort, similar to the existing\n + traversal which provides name-hash hints. This means that the object\n + traversal may hand us back a blob that does not actually exist. We\n + *won't* see missing trees/commits with 'ignore_missing_links' because:\n +\n + - missing commit parents are discarded at the commit traversal stage by\n + revision.c::process_parents()\n +\n + - missing tag objects are discarded by revision.c::handle_commit()\n +\n + - missing tree objects are discarded by the list-objects code in\n + list-objects.c::process_tree()\n +\n + But we have to handle potentially-missing blobs specially by making a\n + separate check to ensure they exist in the repository. Failing to do so\n + would mean that we'd add an object to the packing list which doesn't\n + actually exist, rendering us unable to write out the pack.\n +\n This prepares us for new repacking behavior which will \"resurrect\"\n objects found in cruft or otherwise unspecified packs when generating\n new packs. In the context of geometric repacking, this may be used to\n @@ builtin/pack-objects.c: static int add_object_entry_from_pack(const struct objec\n -\t\treturn;\n +\tenum stdin_packs_mode mode = *(enum stdin_packs_mode *)data;\n +\tif (mode == STDIN_PACKS_MODE_FOLLOW) {\n ++\t\tif (object->type == OBJ_BLOB && !has_object(the_repository,\n ++\t\t\t\t\t\t\t &object->oid, 0))\n ++\t\t\treturn;\n +\t\tadd_object_entry(&object->oid, object->type, name, 0);\n +\t} else {\n +\t\tstruct object_entry *oe = packlist_find(&to_pack, &object->oid);\n @@ t/t5331-pack-objects-stdin.sh: test_expect_success 'pack-objects --stdin with pa\n \ttest_cmp expected-objects actual-objects\n '\n \n -+packdir=.git/objects/pack\n ++objdir=.git/objects\n ++packdir=$objdir/pack\n +\n +objects_in_packs () {\n +\tfor p in \"$@\"\n @@ t/t5331-pack-objects-stdin.sh: test_expect_success 'pack-objects --stdin with pa\n +\t\ttest_cmp expect actual\n +\t)\n +'\n ++\n ++stdin_packs__follow_with_only () {\n ++\trm -fr stdin_packs__follow_with_only &&\n ++\tgit init stdin_packs__follow_with_only &&\n ++\t(\n ++\t\tcd stdin_packs__follow_with_only &&\n ++\n ++\t\ttest_commit A &&\n ++\t\ttest_commit B &&\n ++\n ++\t\tgit rev-parse \"$@\" >B.objects &&\n ++\n ++\t\techo A | git pack-objects --revs $packdir/pack &&\n ++\t\tB=\"$(git pack-objects $packdir/pack objs &&\n ++\t\tfor obj in $(cat objs)\n ++\t\tdo\n ++\t\t\trm -f $objdir/$(test_oid_to_path $obj) || return 1\n ++\t\tdone &&\n ++\n ++\t\t( cd $packdir && ls pack-*.pack ) >in &&\n ++\t\tgit pack-objects --stdin-packs=follow --stdout >/dev/null packs.all &&\n -+\t\tgrep -o \"pack-.*\\.idx$\" packs.all >in &&\n -+\n -+\t\tgit multi-pack-index write --stdin-packs --bitmap packs.all &&\n ++\t\tgit multi-pack-index write --stdin-packs --bitmap packs.all &&\n ++\t\tcruft=\"$(ls $packdir/pack-*.mtimes)\" &&\n ++\t\tcruft=\"${cruft%.mtimes}.idx\" &&\n ++\n ++\t\tfor idx in $(grep -v $cruft out &&\n ++\t\t\twc -l sizes.raw &&\n ++\n ++\t\t# Make sure that there are two non-cruft packs, and\n ++\t\t# that one of them contains at least twice as many\n ++\t\t# objects as the other, ensuring that they are already\n ++\t\t# in a geometric progression.\n ++\t\tsort -n sizes.raw >sizes &&\n ++\t\ttest_line_count = 2 sizes &&\n ++\t\ts1=$(head -n 1 sizes) &&\n ++\t\ts2=$(tail -n 1 sizes) &&\n ++\t\ttest \"$s2\" -gt \"$((2 * $s1))\" &&\n ++\n ++\t\tgit -c repack.midxMustContainCruft=false repack --geometric=2 \\\n ++\t\t\t--write-midx --write-bitmap-index\n ++\t)\n ++'\n +\n test_done\n\nbase-commit: f9aa0eedb37eb94d9d3711ef0d565fd7cb3b6148\n-- \n2.50.0.61.gf819b10624.dirty\n"},{"id":"520437","messageId":"ebaf47262a6d18b019d1b23a9b2b2c37a1e60964.1750375803.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1750375803.git.me@ttaylorr.com","subject":"[PATCH v5 1/9] pack-objects: use standard option incompatibility functions","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-06-19T23:30:08Z","receivedAt":"2025-06-19T23:30:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"pack-objects has a handful of explicit checks for pairs of command-line\noptions which are mutually incompatible. Many of these pre-date\na699367bb8 (i18n: factorize more 'incompatible options' messages,\n2022-01-31).\n\nConvert the explicit checks into die_for_incompatible_opt2() calls,\nwhich simplifies the implementation and standardizes pack-objects'\noutput when given incompatible options (e.g., --stdin-packs with\n--filter gives different output than --keep-unreachable with\n--unpack-unreachable).\n\nThere is one minor piece of test fallout in t5331 that expects the old\nformat, which has been corrected.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 20 +++++++++++---------\n t/t5331-pack-objects-stdin.sh | 2 +-\n 2 files changed, 12 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 67941c8a60..e7274e0e00 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -5010,9 +5010,10 @@ int cmd_pack_objects(int argc,\n \t\tstrvec_push(&rp, \"--unpacked\");\n \t}\n \n-\tif (exclude_promisor_objects && exclude_promisor_objects_best_effort)\n-\t\tdie(_(\"options '%s' and '%s' cannot be used together\"),\n-\t\t \"--exclude-promisor-objects\", \"--exclude-promisor-objects-best-effort\");\n+\tdie_for_incompatible_opt2(exclude_promisor_objects,\n+\t\t\t\t \"--exclude-promisor-objects\",\n+\t\t\t\t exclude_promisor_objects_best_effort,\n+\t\t\t\t \"--exclude-promisor-objects-best-effort\");\n \tif (exclude_promisor_objects) {\n \t\tuse_internal_rev_list = 1;\n \t\tfetch_if_missing = 0;\n@@ -5050,13 +5051,14 @@ int cmd_pack_objects(int argc,\n \tif (!pack_to_stdout && thin)\n \t\tdie(_(\"--thin cannot be used to build an indexable pack\"));\n \n-\tif (keep_unreachable && unpack_unreachable)\n-\t\tdie(_(\"options '%s' and '%s' cannot be used together\"), \"--keep-unreachable\", \"--unpack-unreachable\");\n+\tdie_for_incompatible_opt2(keep_unreachable, \"--keep-unreachable\",\n+\t\t\t\t unpack_unreachable, \"--unpack-unreachable\");\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n-\tif (stdin_packs && filter_options.choice)\n-\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n+\tdie_for_incompatible_opt2(stdin_packs, \"--stdin-packs\",\n+\t\t\t\t filter_options.choice, \"--filter\");\n+\n \n \tif (stdin_packs && use_internal_rev_list)\n \t\tdie(_(\"cannot use internal rev list with --stdin-packs\"));\n@@ -5064,8 +5066,8 @@ int cmd_pack_objects(int argc,\n \tif (cruft) {\n \t\tif (use_internal_rev_list)\n \t\t\tdie(_(\"cannot use internal rev list with --cruft\"));\n-\t\tif (stdin_packs)\n-\t\t\tdie(_(\"cannot use --stdin-packs with --cruft\"));\n+\t\tdie_for_incompatible_opt2(stdin_packs, \"--stdin-packs\",\n+\t\t\t\t\t cruft, \"--cruft\");\n \t}\n \n \t/*\ndiff --git a/t/t5331-pack-objects-stdin.sh b/t/t5331-pack-objects-stdin.sh\nindex b48c0cbe8f..8fd07deb8d 100755\n--- a/t/t5331-pack-objects-stdin.sh\n+++ b/t/t5331-pack-objects-stdin.sh\n@@ -64,7 +64,7 @@ test_expect_success '--stdin-packs is incompatible with --filter' '\n \t\tcd stdin-packs &&\n \t\ttest_must_fail git pack-objects --stdin-packs --stdout \\\n \t\t\t--filter=blob:none err &&\n-\t\ttest_grep \"cannot use --filter with --stdin-packs\" err\n+\t\ttest_grep \"options .--stdin-packs. and .--filter. cannot be used together\" err\n \t)\n '\n \n-- \n2.50.0.61.gf819b10624.dirty\n\n"},{"id":"520438","messageId":"eaa1f41b25bbee35cd8329792dee02a54560b8aa.1750375803.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1750375803.git.me@ttaylorr.com","subject":"[PATCH v5 2/9] pack-objects: limit scope in 'add_object_entry_from_pack()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-06-19T23:30:11Z","receivedAt":"2025-06-19T23:30:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In add_object_entry_from_pack() we declare 'revs' (given to us through\nthe miscellaneous context argument) earlier in the \"if (p)\" conditional\nthan is necessary. Move it down as far as it can go to reduce its\nscope.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex e7274e0e00..d04a36a6bf 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3725,7 +3725,6 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n \t\treturn 0;\n \n \tif (p) {\n-\t\tstruct rev_info *revs = _data;\n \t\tstruct object_info oi = OBJECT_INFO_INIT;\n \n \t\toi.typep = &type;\n@@ -3733,6 +3732,7 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n \t\t\tdie(_(\"could not get type of object %s in pack %s\"),\n \t\t\t oid_to_hex(oid), p->pack_name);\n \t\t} else if (type == OBJ_COMMIT) {\n+\t\t\tstruct rev_info *revs = _data;\n \t\t\t/*\n \t\t\t * commits in included packs are used as starting points for the\n \t\t\t * subsequent revision walk\n-- \n2.50.0.61.gf819b10624.dirty\n\n"},{"id":"520439","messageId":"8d0492a80dd6df1f526f319dee3e95c395e53901.1750375803.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1750375803.git.me@ttaylorr.com","subject":"[PATCH v5 3/9] pack-objects: factor out handling '--stdin-packs'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-06-19T23:30:14Z","receivedAt":"2025-06-19T23:30:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"At the bottom of cmd_pack_objects() we check which mode the command is\nrunning in (e.g., generating a cruft pack, handling '--stdin-packs',\nusing the internal rev-list, etc.) and handle the mode appropriately.\n\nThe '--stdin-packs' case is handled inline (dating back to its\nintroduction in 339bce27f4 (builtin/pack-objects.c: add '--stdin-packs'\noption, 2021-02-22)) since it is relatively short. Extract the body of\n\"if (stdin_packs)\" into its own function to prepare for the\nimplementation to become lengthier in a following commit.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 18 ++++++++++++------\n 1 file changed, 12 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex d04a36a6bf..7ce04b71dd 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3909,6 +3909,17 @@ static void read_packs_list_from_stdin(void)\n \tstring_list_clear(&exclude_packs, 0);\n }\n \n+static void add_unreachable_loose_objects(void);\n+\n+static void read_stdin_packs(int rev_list_unpacked)\n+{\n+\t/* avoids adding objects in excluded packs */\n+\tignore_packed_keep_in_core = 1;\n+\tread_packs_list_from_stdin();\n+\tif (rev_list_unpacked)\n+\t\tadd_unreachable_loose_objects();\n+}\n+\n static void add_cruft_object_entry(const struct object_id *oid, enum object_type type,\n \t\t\t\t struct packed_git *pack, off_t offset,\n \t\t\t\t const char *name, uint32_t mtime)\n@@ -4004,7 +4015,6 @@ static void mark_pack_kept_in_core(struct string_list *packs, unsigned keep)\n \t}\n }\n \n-static void add_unreachable_loose_objects(void);\n static void add_objects_in_unpacked_packs(void);\n \n static void enumerate_cruft_objects(void)\n@@ -5135,11 +5145,7 @@ int cmd_pack_objects(int argc,\n \t\tprogress_state = start_progress(the_repository,\n \t\t\t\t\t\t_(\"Enumerating objects\"), 0);\n \tif (stdin_packs) {\n-\t\t/* avoids adding objects in excluded packs */\n-\t\tignore_packed_keep_in_core = 1;\n-\t\tread_packs_list_from_stdin();\n-\t\tif (rev_list_unpacked)\n-\t\t\tadd_unreachable_loose_objects();\n+\t\tread_stdin_packs(rev_list_unpacked);\n \t} else if (cruft) {\n \t\tread_cruft_objects();\n \t} else if (!use_internal_rev_list) {\n-- \n2.50.0.61.gf819b10624.dirty\n\n"},{"id":"520440","messageId":"3d5d3b78b2e56e111a80dc29d247af40f03ce245.1750375803.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1750375803.git.me@ttaylorr.com","subject":"[PATCH v5 4/9] pack-objects: declare 'rev_info' for '--stdin-packs' earlier","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-06-19T23:30:17Z","receivedAt":"2025-06-19T23:30:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Once 'read_packs_list_from_stdin()' has called for_each_object_in_pack()\non each of the input packs, we do a reachability traversal to discover\nnames for any objects we picked up so we can generate name hash values\nand hopefully get higher quality deltas as a result.\n\nA future commit will change the purpose of this reachability traversal\nto find and pack objects which are reachable from commits in the input\npacks, but are packed in an unknown (not included nor excluded) pack.\n\nExtract the code which initializes and performs the reachability\ntraversal to take place in the caller, not the callee, which prepares us\nto share this code for the '--unpacked' case (see the function\nadd_unreachable_loose_objects() for more details).\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 71 +++++++++++++++++++++---------------------\n 1 file changed, 36 insertions(+), 35 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7ce04b71dd..4258ac1792 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3793,7 +3793,7 @@ static int pack_mtime_cmp(const void *_a, const void *_b)\n \t\treturn 0;\n }\n \n-static void read_packs_list_from_stdin(void)\n+static void read_packs_list_from_stdin(struct rev_info *revs)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct string_list include_packs = STRING_LIST_INIT_DUP;\n@@ -3801,24 +3801,6 @@ static void read_packs_list_from_stdin(void)\n \tstruct string_list_item *item = NULL;\n \n \tstruct packed_git *p;\n-\tstruct rev_info revs;\n-\n-\trepo_init_revisions(the_repository, &revs, NULL);\n-\t/*\n-\t * Use a revision walk to fill in the namehash of objects in the include\n-\t * packs. To save time, we'll avoid traversing through objects that are\n-\t * in excluded packs.\n-\t *\n-\t * That may cause us to avoid populating all of the namehash fields of\n-\t * all included objects, but our goal is best-effort, since this is only\n-\t * an optimization during delta selection.\n-\t */\n-\trevs.no_kept_objects = 1;\n-\trevs.keep_pack_cache_flags |= IN_CORE_KEEP_PACKS;\n-\trevs.blob_objects = 1;\n-\trevs.tree_objects = 1;\n-\trevs.tag_objects = 1;\n-\trevs.ignore_missing_links = 1;\n \n \twhile (strbuf_getline(&buf, stdin) != EOF) {\n \t\tif (!buf.len)\n@@ -3888,10 +3870,44 @@ static void read_packs_list_from_stdin(void)\n \t\tstruct packed_git *p = item->util;\n \t\tfor_each_object_in_pack(p,\n \t\t\t\t\tadd_object_entry_from_pack,\n-\t\t\t\t\t&revs,\n+\t\t\t\t\trevs,\n \t\t\t\t\tFOR_EACH_OBJECT_PACK_ORDER);\n \t}\n \n+\tstrbuf_release(&buf);\n+\tstring_list_clear(&include_packs, 0);\n+\tstring_list_clear(&exclude_packs, 0);\n+}\n+\n+static void add_unreachable_loose_objects(void);\n+\n+static void read_stdin_packs(int rev_list_unpacked)\n+{\n+\tstruct rev_info revs;\n+\n+\trepo_init_revisions(the_repository, &revs, NULL);\n+\t/*\n+\t * Use a revision walk to fill in the namehash of objects in the include\n+\t * packs. To save time, we'll avoid traversing through objects that are\n+\t * in excluded packs.\n+\t *\n+\t * That may cause us to avoid populating all of the namehash fields of\n+\t * all included objects, but our goal is best-effort, since this is only\n+\t * an optimization during delta selection.\n+\t */\n+\trevs.no_kept_objects = 1;\n+\trevs.keep_pack_cache_flags |= IN_CORE_KEEP_PACKS;\n+\trevs.blob_objects = 1;\n+\trevs.tree_objects = 1;\n+\trevs.tag_objects = 1;\n+\trevs.ignore_missing_links = 1;\n+\n+\t/* avoids adding objects in excluded packs */\n+\tignore_packed_keep_in_core = 1;\n+\tread_packs_list_from_stdin(&revs);\n+\tif (rev_list_unpacked)\n+\t\tadd_unreachable_loose_objects();\n+\n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n \ttraverse_commit_list(&revs,\n@@ -3903,21 +3919,6 @@ static void read_packs_list_from_stdin(void)\n \t\t\t stdin_packs_found_nr);\n \ttrace2_data_intmax(\"pack-objects\", the_repository, \"stdin_packs_hints\",\n \t\t\t stdin_packs_hints_nr);\n-\n-\tstrbuf_release(&buf);\n-\tstring_list_clear(&include_packs, 0);\n-\tstring_list_clear(&exclude_packs, 0);\n-}\n-\n-static void add_unreachable_loose_objects(void);\n-\n-static void read_stdin_packs(int rev_list_unpacked)\n-{\n-\t/* avoids adding objects in excluded packs */\n-\tignore_packed_keep_in_core = 1;\n-\tread_packs_list_from_stdin();\n-\tif (rev_list_unpacked)\n-\t\tadd_unreachable_loose_objects();\n }\n \n static void add_cruft_object_entry(const struct object_id *oid, enum object_type type,\n-- \n2.50.0.61.gf819b10624.dirty\n\n"},{"id":"520441","messageId":"2a3676cb86a778ced0e24efb8a07dbd8222ea64d.1750375803.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1750375803.git.me@ttaylorr.com","subject":"[PATCH v5 5/9] pack-objects: perform name-hash traversal for unpacked objects","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-06-19T23:30:20Z","receivedAt":"2025-06-19T23:30:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"With '--unpacked', pack-objects adds loose objects (which don't appear\nin any of the excluded packs from '--stdin-packs') to the output pack\nwithout considering them as reachability tips for the name-hash\ntraversal.\n\nThis was an oversight in the original implementation of '--stdin-packs',\nsince the code which enumerates and adds loose objects to the output\npack (`add_unreachable_loose_objects()`) did not have access to the\n'rev_info' struct found in `read_packs_list_from_stdin()`.\n\nExcluding unpacked objects from that traversal doesn't affect the\ncorrectness of the resulting pack, but it does make it harder to\ndiscover good deltas for loose objects.\n\nNow that the 'rev_info' struct is declared outside of\n`read_packs_list_from_stdin()`, we can pass it to\n`add_objects_in_unpacked_packs()` and add any loose objects as tips to\nthe above-mentioned traversal, in theory producing slightly tighter\npacks as a result.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 20 ++++++++++++--------\n 1 file changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 4258ac1792..3437dbd7f1 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3879,7 +3879,7 @@ static void read_packs_list_from_stdin(struct rev_info *revs)\n \tstring_list_clear(&exclude_packs, 0);\n }\n \n-static void add_unreachable_loose_objects(void);\n+static void add_unreachable_loose_objects(struct rev_info *revs);\n \n static void read_stdin_packs(int rev_list_unpacked)\n {\n@@ -3906,7 +3906,7 @@ static void read_stdin_packs(int rev_list_unpacked)\n \tignore_packed_keep_in_core = 1;\n \tread_packs_list_from_stdin(&revs);\n \tif (rev_list_unpacked)\n-\t\tadd_unreachable_loose_objects();\n+\t\tadd_unreachable_loose_objects(&revs);\n \n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n@@ -4025,7 +4025,7 @@ static void enumerate_cruft_objects(void)\n \t\t\t\t\t\t_(\"Enumerating cruft objects\"), 0);\n \n \tadd_objects_in_unpacked_packs();\n-\tadd_unreachable_loose_objects();\n+\tadd_unreachable_loose_objects(NULL);\n \n \tstop_progress(&progress_state);\n }\n@@ -4303,8 +4303,9 @@ static void add_objects_in_unpacked_packs(void)\n }\n \n static int add_loose_object(const struct object_id *oid, const char *path,\n-\t\t\t void *data UNUSED)\n+\t\t\t void *data)\n {\n+\tstruct rev_info *revs = data;\n \tenum object_type type = oid_object_info(the_repository, oid, NULL);\n \n \tif (type < 0) {\n@@ -4325,6 +4326,10 @@ static int add_loose_object(const struct object_id *oid, const char *path,\n \t} else {\n \t\tadd_object_entry(oid, type, \"\", 0);\n \t}\n+\n+\tif (revs && type == OBJ_COMMIT)\n+\t\tadd_pending_oid(revs, NULL, oid, 0);\n+\n \treturn 0;\n }\n \n@@ -4333,11 +4338,10 @@ static int add_loose_object(const struct object_id *oid, const char *path,\n * add_object_entry will weed out duplicates, so we just add every\n * loose object we find.\n */\n-static void add_unreachable_loose_objects(void)\n+static void add_unreachable_loose_objects(struct rev_info *revs)\n {\n \tfor_each_loose_file_in_objdir(repo_get_object_directory(the_repository),\n-\t\t\t\t add_loose_object,\n-\t\t\t\t NULL, NULL, NULL);\n+\t\t\t\t add_loose_object, NULL, NULL, revs);\n }\n \n static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)\n@@ -4684,7 +4688,7 @@ static void get_object_list(struct rev_info *revs, int ac, const char **av)\n \tif (keep_unreachable)\n \t\tadd_objects_in_unpacked_packs();\n \tif (pack_loose_unreachable)\n-\t\tadd_unreachable_loose_objects();\n+\t\tadd_unreachable_loose_objects(NULL);\n \tif (unpack_unreachable)\n \t\tloosen_unused_packed_objects();\n \n-- \n2.50.0.61.gf819b10624.dirty\n\n"},{"id":"520442","messageId":"bcbce75695f45cb64244bb5d5abb40ebfbe2e31d.1750375803.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1750375803.git.me@ttaylorr.com","subject":"[PATCH v5 6/9] pack-objects: fix typo in 'show_object_pack_hint()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-06-19T23:30:23Z","receivedAt":"2025-06-19T23:30:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Noticed-by: Elijah Newren \nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 3437dbd7f1..9580b4ea1a 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3767,7 +3767,7 @@ static void show_object_pack_hint(struct object *object, const char *name,\n \t * would typically pick up during a reachability traversal.\n \t *\n \t * Make a best-effort attempt to fill in the ->hash and ->no_try_delta\n-\t * here using a now in order to perhaps improve the delta selection\n+\t * fields here in order to perhaps improve the delta selection\n \t * process.\n \t */\n \toe->hash = pack_name_hash_fn(name);\n-- \n2.50.0.61.gf819b10624.dirty\n\n"},{"id":"520443","messageId":"c8cf316c5040539772ec66374c2bc74bb11afbad.1750375803.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1750375803.git.me@ttaylorr.com","subject":"[PATCH v5 7/9] pack-objects: swap 'show_{object,commit}_pack_hint'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-06-19T23:30:27Z","receivedAt":"2025-06-19T23:30:29Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"show_commit_pack_hint() has heretofore been a noop, so its position\nwithin its compilation unit only needs to appear before its first use.\n\nBut the following commit will sometimes have `show_commit_pack_hint()`\ncall `show_object_pack_hint()`, so reorder the former to appear after\nthe latter to minimize the code movement in that patch.\n\nSuggested-by: Elijah Newren \nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 9580b4ea1a..f44447a3f9 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3748,12 +3748,6 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n \treturn 0;\n }\n \n-static void show_commit_pack_hint(struct commit *commit UNUSED,\n-\t\t\t\t void *data UNUSED)\n-{\n-\t/* nothing to do; commits don't have a namehash */\n-}\n-\n static void show_object_pack_hint(struct object *object, const char *name,\n \t\t\t\t void *data UNUSED)\n {\n@@ -3776,6 +3770,12 @@ static void show_object_pack_hint(struct object *object, const char *name,\n \tstdin_packs_hints_nr++;\n }\n \n+static void show_commit_pack_hint(struct commit *commit UNUSED,\n+\t\t\t\t void *data UNUSED)\n+{\n+\t/* nothing to do; commits don't have a namehash */\n+}\n+\n static int pack_mtime_cmp(const void *_a, const void *_b)\n {\n \tstruct packed_git *a = ((const struct string_list_item*)_a)->util;\n-- \n2.50.0.61.gf819b10624.dirty\n\n"},{"id":"520444","messageId":"b81b6213e8f98968a98bd147c068090cfadb5f92.1750375803.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1750375803.git.me@ttaylorr.com","subject":"[PATCH v5 8/9] pack-objects: introduce '--stdin-packs=follow'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-06-19T23:30:30Z","receivedAt":"2025-06-19T23:30:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When invoked with '--stdin-packs', pack-objects will generate a pack\nwhich contains the objects found in the \"included\" packs, less any\nobjects from \"excluded\" packs.\n\nPacks that exist in the repository but weren't specified as either\nincluded or excluded are in practice treated like the latter, at least\nin the sense that pack-objects won't include objects from those packs.\nThis behavior forces us to include any cruft pack(s) in a repository's\nmulti-pack index for the reasons described in ddee3703b3\n(builtin/repack.c: add cruft packs to MIDX during geometric repack,\n2022-05-20).\n\nThe full details are in ddee3703b3, but the gist is if you\nhave a once-unreachable object in a cruft pack which later becomes\nreachable via one or more commits in a pack generated with\n'--stdin-packs', you *have* to include that object in the MIDX via the\ncopy in the cruft pack, otherwise we cannot generate reachability\nbitmaps for any commits which reach that object.\n\nNote that the traversal here is best-effort, similar to the existing\ntraversal which provides name-hash hints. This means that the object\ntraversal may hand us back a blob that does not actually exist. We\n*won't* see missing trees/commits with 'ignore_missing_links' because:\n\n - missing commit parents are discarded at the commit traversal stage by\n revision.c::process_parents()\n\n - missing tag objects are discarded by revision.c::handle_commit()\n\n - missing tree objects are discarded by the list-objects code in\n list-objects.c::process_tree()\n\nBut we have to handle potentially-missing blobs specially by making a\nseparate check to ensure they exist in the repository. Failing to do so\nwould mean that we'd add an object to the packing list which doesn't\nactually exist, rendering us unable to write out the pack.\n\nThis prepares us for new repacking behavior which will \"resurrect\"\nobjects found in cruft or otherwise unspecified packs when generating\nnew packs. In the context of geometric repacking, this may be used to\nmaintain a sequence of geometrically-repacked packs, the union of which\nis closed under reachability, even in the case described earlier.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-pack-objects.adoc | 10 ++-\n builtin/pack-objects.c | 86 +++++++++++++++-----\n t/t5331-pack-objects-stdin.sh | 120 ++++++++++++++++++++++++++++\n 3 files changed, 193 insertions(+), 23 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.adoc b/Documentation/git-pack-objects.adoc\nindex b1c5aa27da..eba014c406 100644\n--- a/Documentation/git-pack-objects.adoc\n+++ b/Documentation/git-pack-objects.adoc\n@@ -87,13 +87,21 @@ base-name::\n \treference was included in the resulting packfile. This\n \tcan be useful to send new tags to native Git clients.\n \n---stdin-packs::\n+--stdin-packs[=]::\n \tRead the basenames of packfiles (e.g., `pack-1234abcd.pack`)\n \tfrom the standard input, instead of object names or revision\n \targuments. The resulting pack contains all objects listed in the\n \tincluded packs (those not beginning with `^`), excluding any\n \tobjects listed in the excluded packs (beginning with `^`).\n +\n+When `mode` is \"follow\", objects from packs not listed on stdin receive\n+special treatment. Objects within unlisted packs will be included if\n+those objects are (1) reachable from the included packs, and (2) not\n+found in any excluded packs. This mode is useful, for example, to\n+resurrect once-unreachable objects found in cruft packs to generate\n+packs which are closed under reachability up to the boundary set by the\n+excluded packs.\n++\n Incompatible with `--revs`, or options that imply `--revs` (such as\n `--all`), with the exception of `--unpacked`, which is compatible.\n \ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex f44447a3f9..d51fe9c820 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -284,6 +284,12 @@ static struct oidmap configured_exclusions;\n static struct oidset excluded_by_config;\n static int name_hash_version = -1;\n \n+enum stdin_packs_mode {\n+\tSTDIN_PACKS_MODE_NONE,\n+\tSTDIN_PACKS_MODE_STANDARD,\n+\tSTDIN_PACKS_MODE_FOLLOW,\n+};\n+\n /**\n * Check whether the name_hash_version chosen by user input is appropriate,\n * and also validate whether it is compatible with other features.\n@@ -3749,31 +3755,47 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n }\n \n static void show_object_pack_hint(struct object *object, const char *name,\n-\t\t\t\t void *data UNUSED)\n+\t\t\t\t void *data)\n {\n-\tstruct object_entry *oe = packlist_find(&to_pack, &object->oid);\n-\tif (!oe)\n-\t\treturn;\n+\tenum stdin_packs_mode mode = *(enum stdin_packs_mode *)data;\n+\tif (mode == STDIN_PACKS_MODE_FOLLOW) {\n+\t\tif (object->type == OBJ_BLOB && !has_object(the_repository,\n+\t\t\t\t\t\t\t &object->oid, 0))\n+\t\t\treturn;\n+\t\tadd_object_entry(&object->oid, object->type, name, 0);\n+\t} else {\n+\t\tstruct object_entry *oe = packlist_find(&to_pack, &object->oid);\n+\t\tif (!oe)\n+\t\t\treturn;\n \n-\t/*\n-\t * Our 'to_pack' list was constructed by iterating all objects packed in\n-\t * included packs, and so doesn't have a non-zero hash field that you\n-\t * would typically pick up during a reachability traversal.\n-\t *\n-\t * Make a best-effort attempt to fill in the ->hash and ->no_try_delta\n-\t * fields here in order to perhaps improve the delta selection\n-\t * process.\n-\t */\n-\toe->hash = pack_name_hash_fn(name);\n-\toe->no_try_delta = name && no_try_delta(name);\n+\t\t/*\n+\t\t * Our 'to_pack' list was constructed by iterating all\n+\t\t * objects packed in included packs, and so doesn't have\n+\t\t * a non-zero hash field that you would typically pick\n+\t\t * up during a reachability traversal.\n+\t\t *\n+\t\t * Make a best-effort attempt to fill in the ->hash and\n+\t\t * ->no_try_delta fields here in order to perhaps\n+\t\t * improve the delta selection process.\n+\t\t */\n+\t\toe->hash = pack_name_hash_fn(name);\n+\t\toe->no_try_delta = name && no_try_delta(name);\n \n-\tstdin_packs_hints_nr++;\n+\t\tstdin_packs_hints_nr++;\n+\t}\n }\n \n-static void show_commit_pack_hint(struct commit *commit UNUSED,\n-\t\t\t\t void *data UNUSED)\n+static void show_commit_pack_hint(struct commit *commit, void *data)\n {\n+\tenum stdin_packs_mode mode = *(enum stdin_packs_mode *)data;\n+\n+\tif (mode == STDIN_PACKS_MODE_FOLLOW) {\n+\t\tshow_object_pack_hint((struct object *)commit, \"\", data);\n+\t\treturn;\n+\t}\n+\n \t/* nothing to do; commits don't have a namehash */\n+\n }\n \n static int pack_mtime_cmp(const void *_a, const void *_b)\n@@ -3881,7 +3903,7 @@ static void read_packs_list_from_stdin(struct rev_info *revs)\n \n static void add_unreachable_loose_objects(struct rev_info *revs);\n \n-static void read_stdin_packs(int rev_list_unpacked)\n+static void read_stdin_packs(enum stdin_packs_mode mode, int rev_list_unpacked)\n {\n \tstruct rev_info revs;\n \n@@ -3913,7 +3935,7 @@ static void read_stdin_packs(int rev_list_unpacked)\n \ttraverse_commit_list(&revs,\n \t\t\t show_commit_pack_hint,\n \t\t\t show_object_pack_hint,\n-\t\t\t NULL);\n+\t\t\t &mode);\n \n \ttrace2_data_intmax(\"pack-objects\", the_repository, \"stdin_packs_found\",\n \t\t\t stdin_packs_found_nr);\n@@ -4795,6 +4817,23 @@ static int is_not_in_promisor_pack(struct commit *commit, void *data) {\n \treturn is_not_in_promisor_pack_obj((struct object *) commit, data);\n }\n \n+static int parse_stdin_packs_mode(const struct option *opt, const char *arg,\n+\t\t\t\t int unset)\n+{\n+\tenum stdin_packs_mode *mode = opt->value;\n+\n+\tif (unset)\n+\t\t*mode = STDIN_PACKS_MODE_NONE;\n+\telse if (!arg || !*arg)\n+\t\t*mode = STDIN_PACKS_MODE_STANDARD;\n+\telse if (!strcmp(arg, \"follow\"))\n+\t\t*mode = STDIN_PACKS_MODE_FOLLOW;\n+\telse\n+\t\tdie(_(\"invalid value for '%s': '%s'\"), opt->long_name, arg);\n+\n+\treturn 0;\n+}\n+\n int cmd_pack_objects(int argc,\n \t\t const char **argv,\n \t\t const char *prefix,\n@@ -4805,7 +4844,7 @@ int cmd_pack_objects(int argc,\n \tstruct strvec rp = STRVEC_INIT;\n \tint rev_list_unpacked = 0, rev_list_all = 0, rev_list_reflog = 0;\n \tint rev_list_index = 0;\n-\tint stdin_packs = 0;\n+\tenum stdin_packs_mode stdin_packs = STDIN_PACKS_MODE_NONE;\n \tstruct string_list keep_pack_list = STRING_LIST_INIT_NODUP;\n \tstruct list_objects_filter_options filter_options =\n \t\tLIST_OBJECTS_FILTER_INIT;\n@@ -4860,6 +4899,9 @@ int cmd_pack_objects(int argc,\n \t\tOPT_SET_INT_F(0, \"indexed-objects\", &rev_list_index,\n \t\t\t N_(\"include objects referred to by the index\"),\n \t\t\t 1, PARSE_OPT_NONEG),\n+\t\tOPT_CALLBACK_F(0, \"stdin-packs\", &stdin_packs, N_(\"mode\"),\n+\t\t\t N_(\"read packs from stdin\"),\n+\t\t\t PARSE_OPT_OPTARG, parse_stdin_packs_mode),\n \t\tOPT_BOOL(0, \"stdin-packs\", &stdin_packs,\n \t\t\t N_(\"read packs from stdin\")),\n \t\tOPT_BOOL(0, \"stdout\", &pack_to_stdout,\n@@ -5150,7 +5192,7 @@ int cmd_pack_objects(int argc,\n \t\tprogress_state = start_progress(the_repository,\n \t\t\t\t\t\t_(\"Enumerating objects\"), 0);\n \tif (stdin_packs) {\n-\t\tread_stdin_packs(rev_list_unpacked);\n+\t\tread_stdin_packs(stdin_packs, rev_list_unpacked);\n \t} else if (cruft) {\n \t\tread_cruft_objects();\n \t} else if (!use_internal_rev_list) {\ndiff --git a/t/t5331-pack-objects-stdin.sh b/t/t5331-pack-objects-stdin.sh\nindex 8fd07deb8d..4a8df5a389 100755\n--- a/t/t5331-pack-objects-stdin.sh\n+++ b/t/t5331-pack-objects-stdin.sh\n@@ -236,4 +236,124 @@ test_expect_success 'pack-objects --stdin with packfiles from main and alternate\n \ttest_cmp expected-objects actual-objects\n '\n \n+objdir=.git/objects\n+packdir=$objdir/pack\n+\n+objects_in_packs () {\n+\tfor p in \"$@\"\n+\tdo\n+\t\tgit show-index <\"$packdir/pack-$p.idx\" || return 1\n+\tdone >objects.raw &&\n+\n+\tcut -d' ' -f2 objects.raw | sort &&\n+\trm -f objects.raw\n+}\n+\n+test_expect_success '--stdin-packs=follow walks into unknown packs' '\n+\ttest_when_finished \"rm -fr repo\" &&\n+\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\tfor c in A B C D\n+\t\tdo\n+\t\t\ttest_commit \"$c\" || return 1\n+\t\tdone &&\n+\n+\t\tA=\"$(echo A | git pack-objects --revs $packdir/pack)\" &&\n+\t\tB=\"$(echo A..B | git pack-objects --revs $packdir/pack)\" &&\n+\t\tC=\"$(echo B..C | git pack-objects --revs $packdir/pack)\" &&\n+\t\tD=\"$(echo C..D | git pack-objects --revs $packdir/pack)\" &&\n+\t\ttest_commit E &&\n+\n+\t\tgit prune-packed &&\n+\n+\t\tcat >in <<-EOF &&\n+\t\tpack-$B.pack\n+\t\t^pack-$C.pack\n+\t\tpack-$D.pack\n+\t\tEOF\n+\n+\t\t# With just --stdin-packs, pack \"A\" is unknown to us, so\n+\t\t# only objects from packs \"B\" and \"D\" are included in\n+\t\t# the output pack.\n+\t\tP=$(git pack-objects --stdin-packs $packdir/pack expect &&\n+\t\tobjects_in_packs $P >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\t# But with --stdin-packs=follow, objects from both\n+\t\t# included packs reach objects from the unknown pack, so\n+\t\t# objects from pack \"A\" is included in the output pack\n+\t\t# in addition to the above.\n+\t\tP=$(git pack-objects --stdin-packs=follow $packdir/pack expect &&\n+\t\tobjects_in_packs $P >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\t# And with --unpacked, we will pick up objects from unknown\n+\t\t# packs that are reachable from loose objects. Loose object E\n+\t\t# reaches objects in pack A, but there are three excluded packs\n+\t\t# in between.\n+\t\t#\n+\t\t# The resulting pack should include objects reachable from E\n+\t\t# that are not present in packs B, C, or D, along with those\n+\t\t# present in pack A.\n+\t\tcat >in <<-EOF &&\n+\t\t^pack-$B.pack\n+\t\t^pack-$C.pack\n+\t\t^pack-$D.pack\n+\t\tEOF\n+\n+\t\tP=$(git pack-objects --stdin-packs=follow --unpacked \\\n+\t\t\t$packdir/pack expect.raw &&\n+\t\tsort expect.raw >expect &&\n+\t\tobjects_in_packs $P >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+stdin_packs__follow_with_only () {\n+\trm -fr stdin_packs__follow_with_only &&\n+\tgit init stdin_packs__follow_with_only &&\n+\t(\n+\t\tcd stdin_packs__follow_with_only &&\n+\n+\t\ttest_commit A &&\n+\t\ttest_commit B &&\n+\n+\t\tgit rev-parse \"$@\" >B.objects &&\n+\n+\t\techo A | git pack-objects --revs $packdir/pack &&\n+\t\tB=\"$(git pack-objects $packdir/pack objs &&\n+\t\tfor obj in $(cat objs)\n+\t\tdo\n+\t\t\trm -f $objdir/$(test_oid_to_path $obj) || return 1\n+\t\tdone &&\n+\n+\t\t( cd $packdir && ls pack-*.pack ) >in &&\n+\t\tgit pack-objects --stdin-packs=follow --stdout >/dev/null \n---\n Documentation/config/repack.adoc | 7 ++\n builtin/repack.c | 184 +++++++++++++++++++++++++++----\n t/t7704-repack-cruft.sh | 145 ++++++++++++++++++++++++\n 3 files changed, 316 insertions(+), 20 deletions(-)\n\ndiff --git a/Documentation/config/repack.adoc b/Documentation/config/repack.adoc\nindex c79af6d7b8..e9e78dcb19 100644\n--- a/Documentation/config/repack.adoc\n+++ b/Documentation/config/repack.adoc\n@@ -39,3 +39,10 @@ repack.cruftThreads::\n \ta cruft pack and the respective parameters are not given over\n \tthe command line. See similarly named `pack.*` configuration\n \tvariables for defaults and meaning.\n+\n+repack.midxMustContainCruft::\n+\tWhen set to true, linkgit:git-repack[1] will unconditionally include\n+\tcruft pack(s), if any, in the multi-pack index when invoked with\n+\t`--write-midx`. When false, cruft packs are only included in the MIDX\n+\twhen necessary (e.g., because they might be required to form a\n+\treachability closure with MIDX bitmaps). Defaults to true.\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 5ddc6e7f95..346d44fbcd 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -39,6 +39,7 @@ static int write_bitmaps = -1;\n static int use_delta_islands;\n static int run_update_server_info = 1;\n static char *packdir, *packtmp_name, *packtmp;\n+static int midx_must_contain_cruft = 1;\n \n static const char *const git_repack_usage[] = {\n \tN_(\"git repack [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\\n\"\n@@ -108,6 +109,10 @@ static int repack_config(const char *var, const char *value,\n \t\tfree(cruft_po_args->threads);\n \t\treturn git_config_string(&cruft_po_args->threads, var, value);\n \t}\n+\tif (!strcmp(var, \"repack.midxmustcontaincruft\")) {\n+\t\tmidx_must_contain_cruft = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n \treturn git_default_config(var, value, ctx, cb);\n }\n \n@@ -690,6 +695,77 @@ static void free_pack_geometry(struct pack_geometry *geometry)\n \tfree(geometry->pack);\n }\n \n+static int midx_has_unknown_packs(char **midx_pack_names,\n+\t\t\t\t size_t midx_pack_names_nr,\n+\t\t\t\t struct string_list *include,\n+\t\t\t\t struct pack_geometry *geometry,\n+\t\t\t\t struct existing_packs *existing)\n+{\n+\tsize_t i;\n+\n+\tstring_list_sort(include);\n+\n+\tfor (i = 0; i < midx_pack_names_nr; i++) {\n+\t\tconst char *pack_name = midx_pack_names[i];\n+\n+\t\t/*\n+\t\t * Determine whether or not each MIDX'd pack from the existing\n+\t\t * MIDX (if any) is represented in the new MIDX. For each pack\n+\t\t * in the MIDX, it must either be:\n+\t\t *\n+\t\t * - In the \"include\" list of packs to be included in the new\n+\t\t * MIDX. Note this function is called before the include\n+\t\t * list is populated with any cruft pack(s).\n+\t\t *\n+\t\t * - Below the geometric split line (if using pack geometry),\n+\t\t * indicating that the pack won't be included in the new\n+\t\t * MIDX, but its contents were rolled up as part of the\n+\t\t * geometric repack.\n+\t\t *\n+\t\t * - In the existing non-kept packs list (if not using pack\n+\t\t * geometry), and marked as non-deleted.\n+\t\t */\n+\t\tif (string_list_has_string(include, pack_name)) {\n+\t\t\tcontinue;\n+\t\t} else if (geometry) {\n+\t\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\t\tuint32_t j;\n+\n+\t\t\tfor (j = 0; j < geometry->split; j++) {\n+\t\t\t\tstrbuf_reset(&buf);\n+\t\t\t\tstrbuf_addstr(&buf, pack_basename(geometry->pack[j]));\n+\t\t\t\tstrbuf_strip_suffix(&buf, \".pack\");\n+\t\t\t\tstrbuf_addstr(&buf, \".idx\");\n+\n+\t\t\t\tif (!strcmp(pack_name, buf.buf)) {\n+\t\t\t\t\tstrbuf_release(&buf);\n+\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\tstrbuf_release(&buf);\n+\n+\t\t\tif (j < geometry->split)\n+\t\t\t\tcontinue;\n+\t\t} else {\n+\t\t\tstruct string_list_item *item;\n+\n+\t\t\titem = string_list_lookup(&existing->non_kept_packs,\n+\t\t\t\t\t\t pack_name);\n+\t\t\tif (item && !pack_is_marked_for_deletion(item))\n+\t\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/*\n+\t\t * If we got to this point, the MIDX includes some pack that we\n+\t\t * don't know about.\n+\t\t */\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n struct midx_snapshot_ref_data {\n \tstruct tempfile *f;\n \tstruct oidset seen;\n@@ -758,6 +834,8 @@ static void midx_snapshot_refs(struct tempfile *f)\n \n static void midx_included_packs(struct string_list *include,\n \t\t\t\tstruct existing_packs *existing,\n+\t\t\t\tchar **midx_pack_names,\n+\t\t\t\tsize_t midx_pack_names_nr,\n \t\t\t\tstruct string_list *names,\n \t\t\t\tstruct pack_geometry *geometry)\n {\n@@ -811,26 +889,56 @@ static void midx_included_packs(struct string_list *include,\n \t\t}\n \t}\n \n-\tfor_each_string_list_item(item, &existing->cruft_packs) {\n+\tif (midx_must_contain_cruft ||\n+\t midx_has_unknown_packs(midx_pack_names, midx_pack_names_nr,\n+\t\t\t\t include, geometry, existing)) {\n \t\t/*\n-\t\t * When doing a --geometric repack, there is no need to check\n-\t\t * for deleted packs, since we're by definition not doing an\n-\t\t * ALL_INTO_ONE repack (hence no packs will be deleted).\n-\t\t * Otherwise we must check for and exclude any packs which are\n-\t\t * enqueued for deletion.\n+\t\t * If there are one or more unknown pack(s) present (see\n+\t\t * midx_has_unknown_packs() for what makes a pack\n+\t\t * \"unknown\") in the MIDX before the repack, keep them\n+\t\t * as they may be required to form a reachability\n+\t\t * closure if the MIDX is bitmapped.\n \t\t *\n-\t\t * So we could omit the conditional below in the --geometric\n-\t\t * case, but doing so is unnecessary since no packs are marked\n-\t\t * as pending deletion (since we only call\n-\t\t * `mark_packs_for_deletion()` when doing an all-into-one\n-\t\t * repack).\n+\t\t * For example, a cruft pack can be required to form a\n+\t\t * reachability closure if the MIDX is bitmapped and one\n+\t\t * or more of the bitmap's selected commits reaches a\n+\t\t * once-cruft object that was later made reachable.\n \t\t */\n-\t\tif (pack_is_marked_for_deletion(item))\n-\t\t\tcontinue;\n+\t\tfor_each_string_list_item(item, &existing->cruft_packs) {\n+\t\t\t/*\n+\t\t\t * When doing a --geometric repack, there is no\n+\t\t\t * need to check for deleted packs, since we're\n+\t\t\t * by definition not doing an ALL_INTO_ONE\n+\t\t\t * repack (hence no packs will be deleted).\n+\t\t\t * Otherwise we must check for and exclude any\n+\t\t\t * packs which are enqueued for deletion.\n+\t\t\t *\n+\t\t\t * So we could omit the conditional below in the\n+\t\t\t * --geometric case, but doing so is unnecessary\n+\t\t\t * since no packs are marked as pending\n+\t\t\t * deletion (since we only call\n+\t\t\t * `mark_packs_for_deletion()` when doing an\n+\t\t\t * all-into-one repack).\n+\t\t\t */\n+\t\t\tif (pack_is_marked_for_deletion(item))\n+\t\t\t\tcontinue;\n \n-\t\tstrbuf_reset(&buf);\n-\t\tstrbuf_addf(&buf, \"%s.idx\", item->string);\n-\t\tstring_list_insert(include, buf.buf);\n+\t\t\tstrbuf_reset(&buf);\n+\t\t\tstrbuf_addf(&buf, \"%s.idx\", item->string);\n+\t\t\tstring_list_insert(include, buf.buf);\n+\t\t}\n+\t} else {\n+\t\t/*\n+\t\t * Modern versions of Git (with the appropriate\n+\t\t * configuration setting) will write new copies of\n+\t\t * once-cruft objects when doing a --geometric repack.\n+\t\t *\n+\t\t * If the MIDX has no cruft pack, new packs written\n+\t\t * during a --geometric repack will not rely on the\n+\t\t * cruft pack to form a reachability closure, so we can\n+\t\t * avoid including them in the MIDX in that case.\n+\t\t */\n+\t\t;\n \t}\n \n \tstrbuf_release(&buf);\n@@ -1145,6 +1253,8 @@ int cmd_repack(int argc,\n \tstruct tempfile *refs_snapshot = NULL;\n \tint i, ext, ret;\n \tint show_progress;\n+\tchar **midx_pack_names = NULL;\n+\tsize_t midx_pack_names_nr = 0;\n \n \t/* variables to be filled by option parsing */\n \tint delete_redundant = 0;\n@@ -1361,7 +1471,10 @@ int cmd_repack(int argc,\n \t\t !(pack_everything & PACK_CRUFT))\n \t\t\tstrvec_push(&cmd.args, \"--pack-loose-unreachable\");\n \t} else if (geometry.split_factor) {\n-\t\tstrvec_push(&cmd.args, \"--stdin-packs\");\n+\t\tif (midx_must_contain_cruft)\n+\t\t\tstrvec_push(&cmd.args, \"--stdin-packs\");\n+\t\telse\n+\t\t\tstrvec_push(&cmd.args, \"--stdin-packs=follow\");\n \t\tstrvec_push(&cmd.args, \"--unpacked\");\n \t} else {\n \t\tstrvec_push(&cmd.args, \"--unpacked\");\n@@ -1401,8 +1514,25 @@ int cmd_repack(int argc,\n \tif (ret)\n \t\tgoto cleanup;\n \n-\tif (!names.nr && !po_args.quiet)\n-\t\tprintf_ln(_(\"Nothing new to pack.\"));\n+\tif (!names.nr) {\n+\t\tif (!po_args.quiet)\n+\t\t\tprintf_ln(_(\"Nothing new to pack.\"));\n+\t\t/*\n+\t\t * If we didn't write any new packs, the non-cruft packs\n+\t\t * may refer to once-unreachable objects in the cruft\n+\t\t * pack(s).\n+\t\t *\n+\t\t * If there isn't already a MIDX, the one we write\n+\t\t * must include the cruft pack(s), in case the\n+\t\t * non-cruft pack(s) refer to once-cruft objects.\n+\t\t *\n+\t\t * If there is already a MIDX, we can punt here, since\n+\t\t * midx_has_unknown_packs() will make the decision for\n+\t\t * us.\n+\t\t */\n+\t\tif (!get_local_multi_pack_index(the_repository))\n+\t\t\tmidx_must_contain_cruft = 1;\n+\t}\n \n \tif (pack_everything & PACK_CRUFT) {\n \t\tconst char *pack_prefix = find_pack_prefix(packdir, packtmp);\n@@ -1483,6 +1613,16 @@ int cmd_repack(int argc,\n \n \tstring_list_sort(&names);\n \n+\tif (get_local_multi_pack_index(the_repository)) {\n+\t\tuint32_t i;\n+\t\tstruct multi_pack_index *m =\n+\t\t\tget_local_multi_pack_index(the_repository);\n+\n+\t\tALLOC_ARRAY(midx_pack_names, m->num_packs);\n+\t\tfor (i = 0; i < m->num_packs; i++)\n+\t\t\tmidx_pack_names[midx_pack_names_nr++] = xstrdup(m->pack_names[i]);\n+\t}\n+\n \tclose_object_store(the_repository->objects);\n \n \t/*\n@@ -1524,7 +1664,8 @@ int cmd_repack(int argc,\n \n \tif (write_midx) {\n \t\tstruct string_list include = STRING_LIST_INIT_DUP;\n-\t\tmidx_included_packs(&include, &existing, &names, &geometry);\n+\t\tmidx_included_packs(&include, &existing, midx_pack_names,\n+\t\t\t\t midx_pack_names_nr, &names, &geometry);\n \n \t\tret = write_midx_included_packs(&include, &geometry, &names,\n \t\t\t\t\t\trefs_snapshot ? get_tempfile_path(refs_snapshot) : NULL,\n@@ -1575,6 +1716,9 @@ int cmd_repack(int argc,\n \tstring_list_clear(&names, 1);\n \texisting_packs_release(&existing);\n \tfree_pack_geometry(&geometry);\n+\tfor (size_t i = 0; i < midx_pack_names_nr; i++)\n+\t\tfree(midx_pack_names[i]);\n+\tfree(midx_pack_names);\n \tpack_objects_args_release(&po_args);\n \tpack_objects_args_release(&cruft_po_args);\n \ndiff --git a/t/t7704-repack-cruft.sh b/t/t7704-repack-cruft.sh\nindex 8aebfb45f5..aa2e2e6ad8 100755\n--- a/t/t7704-repack-cruft.sh\n+++ b/t/t7704-repack-cruft.sh\n@@ -724,4 +724,149 @@ test_expect_success 'cruft repack respects --quiet' '\n \t)\n '\n \n+setup_cruft_exclude_tests() {\n+\tgit init \"$1\" &&\n+\t(\n+\t\tcd \"$1\" &&\n+\n+\t\tgit config repack.midxMustContainCruft false &&\n+\n+\t\ttest_commit one &&\n+\n+\t\ttest_commit --no-tag two &&\n+\t\ttwo=\"$(git rev-parse HEAD)\" &&\n+\t\ttest_commit --no-tag three &&\n+\t\tthree=\"$(git rev-parse HEAD)\" &&\n+\t\tgit reset --hard one &&\n+\t\tgit reflog expire --all --expire=all &&\n+\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack --cruft -d &&\n+\n+\t\tgit merge $two &&\n+\t\ttest_commit four\n+\t)\n+}\n+\n+test_expect_success 'repack --write-midx excludes cruft where possible' '\n+\tsetup_cruft_exclude_tests exclude-cruft-when-possible &&\n+\t(\n+\t\tcd exclude-cruft-when-possible &&\n+\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 \\\n+\t\tgit repack -d --geometric=2 --write-midx --write-bitmap-index &&\n+\n+\t\ttest-tool read-midx --show-objects $objdir >midx &&\n+\t\tcruft=\"$(ls $packdir/*.mtimes)\" &&\n+\t\ttest_grep ! \"$(basename \"$cruft\" .mtimes).idx\" midx &&\n+\n+\t\tgit rev-list --all --objects --no-object-names >reachable.raw &&\n+\t\tsort reachable.raw >reachable.objects &&\n+\t\tawk \"/\\.pack$/ { print \\$1 }\" midx.objects &&\n+\n+\t\ttest_cmp reachable.objects midx.objects\n+\t)\n+'\n+\n+test_expect_success 'repack --write-midx includes cruft when instructed' '\n+\tsetup_cruft_exclude_tests exclude-cruft-when-instructed &&\n+\t(\n+\t\tcd exclude-cruft-when-instructed &&\n+\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 \\\n+\t\tgit -c repack.midxMustContainCruft=true repack \\\n+\t\t\t-d --geometric=2 --write-midx --write-bitmap-index &&\n+\n+\t\ttest-tool read-midx --show-objects $objdir >midx &&\n+\t\tcruft=\"$(ls $packdir/*.mtimes)\" &&\n+\t\ttest_grep \"$(basename \"$cruft\" .mtimes).idx\" midx &&\n+\n+\t\tgit cat-file --batch-check=\"%(objectname)\" --batch-all-objects \\\n+\t\t\t>all.objects &&\n+\t\tawk \"/\\.pack$/ { print \\$1 }\" midx.objects &&\n+\n+\t\ttest_cmp all.objects midx.objects\n+\t)\n+'\n+\n+test_expect_success 'repack --write-midx includes cruft when necessary' '\n+\tsetup_cruft_exclude_tests exclude-cruft-when-necessary &&\n+\t(\n+\t\tcd exclude-cruft-when-necessary &&\n+\n+\t\ttest_path_is_file $(ls $packdir/pack-*.mtimes) &&\n+\t\t( cd $packdir && ls pack-*.idx ) | sort >packs.all &&\n+\t\tgit multi-pack-index write --stdin-packs --bitmap midx &&\n+\t\tawk \"/\\.pack$/ { print \\$1 }\" midx.objects &&\n+\t\tgit cat-file --batch-all-objects --batch-check=\"%(objectname)\" \\\n+\t\t\t>expect.objects &&\n+\t\ttest_cmp expect.objects midx.objects &&\n+\n+\t\tgrep \"^pack-\" midx >midx.packs &&\n+\t\ttest_line_count = \"$(($(wc -l packs.all &&\n+\t\tcruft=\"$(ls $packdir/pack-*.mtimes)\" &&\n+\t\tcruft=\"${cruft%.mtimes}.idx\" &&\n+\n+\t\tfor idx in $(grep -v $cruft out &&\n+\t\t\twc -l sizes.raw &&\n+\n+\t\t# Make sure that there are two non-cruft packs, and\n+\t\t# that one of them contains at least twice as many\n+\t\t# objects as the other, ensuring that they are already\n+\t\t# in a geometric progression.\n+\t\tsort -n sizes.raw >sizes &&\n+\t\ttest_line_count = 2 sizes &&\n+\t\ts1=$(head -n 1 sizes) &&\n+\t\ts2=$(tail -n 1 sizes) &&\n+\t\ttest \"$s2\" -gt \"$((2 * $s1))\" &&\n+\n+\t\tgit -c repack.midxMustContainCruft=false repack --geometric=2 \\\n+\t\t\t--write-midx --write-bitmap-index\n+\t)\n+'\n+\n test_done\n-- \n2.50.0.61.gf819b10624.dirty\n"},{"id":"520497","messageId":"xmqq5xgq5t6s.fsf@gitster.g","threadId":"63286","inReplyTo":"b81b6213e8f98968a98bd147c068090cfadb5f92.1750375803.git.me@ttaylorr.com","subject":"Re: [PATCH v5 8/9] pack-objects: introduce '--stdin-packs=follow'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-06-20T15:27:55Z","receivedAt":"2025-06-20T15:27:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> +\tenum stdin_packs_mode mode = *(enum stdin_packs_mode *)data;\n> +\tif (mode == STDIN_PACKS_MODE_FOLLOW) {\n> +\t\tif (object->type == OBJ_BLOB && !has_object(the_repository,\n> +\t\t\t\t\t\t\t &object->oid, 0))\n> +\t\t\treturn;\n\nSorry for making a comment that is not about the contents of the\npatch, but since we were discussing clang-format elsewhere, and this\nhappens to be a case the tool gets it right, the above should read\nmore like:\n\n\t\tif (object->type == OBJ_BLOB &&\n\t\t !has_object(the_repository, &object->oid, 0))\n\t\t\treturn;\n\ncf. Documentation/CodingGuidelines\n\n - When splitting a long logical line, with everything else being\n equal, it is preferable to split after the operator at higher\n level in the parse tree.\n\n\n"},{"id":"520527","messageId":"20250621043551.GA3002138@coredump.intra.peff.net","threadId":"63286","inReplyTo":"6487001f64653d1434890df39b4c4937ea4d0b2c.1750375803.git.me@ttaylorr.com","subject":"Re: [PATCH v5 9/9] repack: exclude cruft pack(s) from the MIDX where possible","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-06-21T04:35:51Z","receivedAt":"2025-06-21T04:36:00Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jun 19, 2025 at 07:30:33PM -0400, Taylor Blau wrote:\n\n> +test_expect_success 'repack --write-midx excludes cruft where possible' '\n> +\tsetup_cruft_exclude_tests exclude-cruft-when-possible &&\n> +\t(\n> +\t\tcd exclude-cruft-when-possible &&\n> +\n> +\t\tGIT_TEST_MULTI_PACK_INDEX=0 \\\n> +\t\tgit repack -d --geometric=2 --write-midx --write-bitmap-index &&\n> +\n> +\t\ttest-tool read-midx --show-objects $objdir >midx &&\n> +\t\tcruft=\"$(ls $packdir/*.mtimes)\" &&\n> +\t\ttest_grep ! \"$(basename \"$cruft\" .mtimes).idx\" midx &&\n> +\n> +\t\tgit rev-list --all --objects --no-object-names >reachable.raw &&\n> +\t\tsort reachable.raw >reachable.objects &&\n> +\t\tawk \"/\\.pack$/ { print \\$1 }\" midx.objects &&\n> +\n> +\t\ttest_cmp reachable.objects midx.objects\n> +\t)\n> +'\n\nThis test (but none of the others) fails when run with:\n\n GIT_TEST_MULTI_PACK_INDEX=1 \\\n GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=1 \\\n ./t7704-repack-cruft.sh\n\nThe culprit is the incremental flag, but you need the first one for the\nsecond to do anything. The issue is that the cruft pack unexpectedly\nappears in the midx:\n\n error: '! grep pack-45dcff625845dc0ad702f91d853d0950f9be0eb9.idx midx' did find a match in:\n header: 4d494458 1 20 6 3\n chunks: pack-names oid-fanout oid-lookup object-offsets\n num_objects: 12\n packs:\n pack-110f8bab659db6e691a75b6462d043214fd1da92.idx\n pack-45dcff625845dc0ad702f91d853d0950f9be0eb9.idx\n pack-91db7b4b856b00c3e675824c5bc5389b6810037a.idx\n object-dir: .git/objects\n 07d4aa2eb79f3a92e1dadaee6ef6b883cdbba641 12\t.git/objects/pack/pack-45dcff625845dc0ad702f91d853d0950f9be0eb9.pack\n 139b20d8e6c5b496de61f033f642d0e3dbff528d 114\t.git/objects/pack/pack-45dcff625845dc0ad702f91d853d0950f9be0eb9.pack\n 25e18d2c3e3563b690593dcce936302010e6aa7e 12\t.git/objects/pack/pack-110f8bab659db6e691a75b6462d043214fd1da92.pack\n 2bdf67abb163a4ffb2d7f3f0880c9fe5068ce782 270\t.git/objects/pack/pack-45dcff625845dc0ad702f91d853d0950f9be0eb9.pack\n 2f00a404aed7e63d867313d504bd0fccea53fd25 285\t.git/objects/pack/pack-45dcff625845dc0ad702f91d853d0950f9be0eb9.pack\n 5626abf0f72e58d7a153368ba57db4c673c0e171 182\t.git/objects/pack/pack-91db7b4b856b00c3e675824c5bc5389b6810037a.pack\n 7c7cd714e262561f73f3079dfca4e8724682ac21 358\t.git/objects/pack/pack-45dcff625845dc0ad702f91d853d0950f9be0eb9.pack\n 8510665149157c2bc901848c3e0b746954e9cbd9 271\t.git/objects/pack/pack-110f8bab659db6e691a75b6462d043214fd1da92.pack\n a7cddf35737959e1438bc929b665619e9e79bfee 138\t.git/objects/pack/pack-91db7b4b856b00c3e675824c5bc5389b6810037a.pack\n d79ce1670bdcb76e6d1da2ae095e890ccb326ae9 12\t.git/objects/pack/pack-91db7b4b856b00c3e675824c5bc5389b6810037a.pack\n db6165b80a148f78daad30f4e29c7b77fe8f04c2 169\t.git/objects/pack/pack-110f8bab659db6e691a75b6462d043214fd1da92.pack\n f719efd430d52bcfc8566a43b2eb655688d38871 517\t.git/objects/pack/pack-45dcff625845dc0ad702f91d853d0950f9be0eb9.pack\n\nI'm not sure if it's just a funky interaction with the hacky GIT_TEST_*\nvariables, or if it's a real bug.\n\n-Peff\n"},{"id":"520582","messageId":"aFmhQZZnYyvxDelO@nand.local","threadId":"63286","inReplyTo":"20250621043551.GA3002138@coredump.intra.peff.net","subject":"Re: [PATCH v5 9/9] repack: exclude cruft pack(s) from the MIDX where possible","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-06-23T18:47:29Z","receivedAt":"2025-06-23T18:47:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sat, Jun 21, 2025 at 12:35:51AM -0400, Jeff King wrote:\n> On Thu, Jun 19, 2025 at 07:30:33PM -0400, Taylor Blau wrote:\n>\n> > +test_expect_success 'repack --write-midx excludes cruft where possible' '\n> > +\tsetup_cruft_exclude_tests exclude-cruft-when-possible &&\n> > +\t(\n> > +\t\tcd exclude-cruft-when-possible &&\n> > +\n> > +\t\tGIT_TEST_MULTI_PACK_INDEX=0 \\\n> > +\t\tgit repack -d --geometric=2 --write-midx --write-bitmap-index &&\n> > +\n> > +\t\ttest-tool read-midx --show-objects $objdir >midx &&\n> > +\t\tcruft=\"$(ls $packdir/*.mtimes)\" &&\n> > +\t\ttest_grep ! \"$(basename \"$cruft\" .mtimes).idx\" midx &&\n> > +\n> > +\t\tgit rev-list --all --objects --no-object-names >reachable.raw &&\n> > +\t\tsort reachable.raw >reachable.objects &&\n> > +\t\tawk \"/\\.pack$/ { print \\$1 }\" midx.objects &&\n> > +\n> > +\t\ttest_cmp reachable.objects midx.objects\n> > +\t)\n> > +'\n>\n> This test (but none of the others) fails when run with:\n>\n> GIT_TEST_MULTI_PACK_INDEX=1 \\\n> GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=1 \\\n> ./t7704-repack-cruft.sh\n>\n> The culprit is the incremental flag, but you need the first one for the\n> second to do anything. The issue is that the cruft pack unexpectedly\n> appears in the midx:\n>\n> [...]\n>\n> I'm not sure if it's just a funky interaction with the hacky GIT_TEST_*\n> variables, or if it's a real bug.\n\nThanks for spotting. This is definitely a real bug. The root cause here\nis that our loop to gather the set of packs we know are in the MIDX does\nnot account for multi-layered / incremental MIDXs.\n\nIn our example, if there's a cruft pack in any other layer of a MIDX\nbesides the tip, the proposed implementation here won't realize it, and\nthus (incorrectly) conclude that the cruft pack is not in the MIDX\nalready, so can thusly be omitted.\n\nIf we do this on top:\n\n--- 8< ---\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 346d44fbcd..8d1540a0fd 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -1614,13 +1614,16 @@ int cmd_repack(int argc,\n \tstring_list_sort(&names);\n\n \tif (get_local_multi_pack_index(the_repository)) {\n-\t\tuint32_t i;\n \t\tstruct multi_pack_index *m =\n \t\t\tget_local_multi_pack_index(the_repository);\n\n-\t\tALLOC_ARRAY(midx_pack_names, m->num_packs);\n-\t\tfor (i = 0; i < m->num_packs; i++)\n-\t\t\tmidx_pack_names[midx_pack_names_nr++] = xstrdup(m->pack_names[i]);\n+\t\tALLOC_ARRAY(midx_pack_names,\n+\t\t\t m->num_packs + m->num_packs_in_base);\n+\n+\t\tfor (; m; m = m->base_midx)\n+\t\t\tfor (uint32_t i = 0; i < m->num_packs; i++)\n+\t\t\t\tmidx_pack_names[midx_pack_names_nr++] =\n+\t\t\t\t\txstrdup(m->pack_names[i]);\n \t}\n\n \tclose_object_store(the_repository->objects);\n--- >8 ---\n\n(Note that this assumes that 'tb/prepare-midx-pack-cleanup' has not\nlanded, this could be a little bit simplified with the\nnth_midx_pack_names() function. I kept these two separate so they could\nproceed independently.)\n\nThings work as expected. I'll send out a new round with this fix\nIncorporated as well as a style issue that Junio noted earlier in this\nthread.\n\nThanks,\nTaylor\n"},{"id":"520591","messageId":"cover.1750717921.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1744413969.git.me@ttaylorr.com","subject":"[PATCH v6 0/9] repack: avoid MIDX'ing cruft pack(s) where possible","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-06-23T22:32:03Z","receivedAt":"2025-06-23T22:32:05Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here is an additional reroll of my series to create MIDXs that do not\ninclude a repository's cruft pack(s).\n\nNearly everything is identical between this version and the previous\n(v5), with two exceptions:\n\n - Adjusted where to split a long line in show_object_pack_hint().\n\n - Fixed a test failure with GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL\n\nThanks for Junio and Peff (respectively) for pointing out each of the\nabove. As usual, a range-diff is attached for convenience.\n\nThanks in advance for any review :-).\n\nTaylor Blau (9):\n pack-objects: use standard option incompatibility functions\n pack-objects: limit scope in 'add_object_entry_from_pack()'\n pack-objects: factor out handling '--stdin-packs'\n pack-objects: declare 'rev_info' for '--stdin-packs' earlier\n pack-objects: perform name-hash traversal for unpacked objects\n pack-objects: fix typo in 'show_object_pack_hint()'\n pack-objects: swap 'show_{object,commit}_pack_hint'\n pack-objects: introduce '--stdin-packs=follow'\n repack: exclude cruft pack(s) from the MIDX where possible\n\n Documentation/config/repack.adoc | 7 +\n Documentation/git-pack-objects.adoc | 10 +-\n builtin/pack-objects.c | 193 ++++++++++++++++++----------\n builtin/repack.c | 187 ++++++++++++++++++++++++---\n t/t5331-pack-objects-stdin.sh | 122 +++++++++++++++++-\n t/t7704-repack-cruft.sh | 145 +++++++++++++++++++++\n 6 files changed, 573 insertions(+), 91 deletions(-)\n\nRange-diff against v5:\n 1: 19fab7a35c = 1: 8e7b2dacc7 pack-objects: use standard option incompatibility functions\n 2: 6f2d3f17a4 = 2: 86fb36d317 pack-objects: limit scope in 'add_object_entry_from_pack()'\n 3: c06f5b264a = 3: 19e8c789e9 pack-objects: factor out handling '--stdin-packs'\n 4: 40d7d87cb1 = 4: c9f874eb94 pack-objects: declare 'rev_info' for '--stdin-packs' earlier\n 5: 5e2599436c = 5: 6b0149a32d pack-objects: perform name-hash traversal for unpacked objects\n 6: 3a5c3f63d8 = 6: f31dd00a98 pack-objects: fix typo in 'show_object_pack_hint()'\n 7: 796e8743f8 = 7: 5d15055985 pack-objects: swap 'show_{object,commit}_pack_hint'\n 8: 8830775beb ! 8: 3699c25337 pack-objects: introduce '--stdin-packs=follow'\n @@ builtin/pack-objects.c: static int add_object_entry_from_pack(const struct objec\n -\t\treturn;\n +\tenum stdin_packs_mode mode = *(enum stdin_packs_mode *)data;\n +\tif (mode == STDIN_PACKS_MODE_FOLLOW) {\n -+\t\tif (object->type == OBJ_BLOB && !has_object(the_repository,\n -+\t\t\t\t\t\t\t &object->oid, 0))\n ++\t\tif (object->type == OBJ_BLOB &&\n ++\t\t !has_object(the_repository, &object->oid, 0))\n +\t\t\treturn;\n +\t\tadd_object_entry(&object->oid, object->type, name, 0);\n +\t} else {\n 9: 8f505179cc ! 9: f519777059 repack: exclude cruft pack(s) from the MIDX where possible\n @@ builtin/repack.c: int cmd_repack(int argc,\n \tstring_list_sort(&names);\n \n +\tif (get_local_multi_pack_index(the_repository)) {\n -+\t\tuint32_t i;\n +\t\tstruct multi_pack_index *m =\n +\t\t\tget_local_multi_pack_index(the_repository);\n +\n -+\t\tALLOC_ARRAY(midx_pack_names, m->num_packs);\n -+\t\tfor (i = 0; i < m->num_packs; i++)\n -+\t\t\tmidx_pack_names[midx_pack_names_nr++] = xstrdup(m->pack_names[i]);\n ++\t\tALLOC_ARRAY(midx_pack_names,\n ++\t\t\t m->num_packs + m->num_packs_in_base);\n ++\n ++\t\tfor (; m; m = m->base_midx)\n ++\t\t\tfor (uint32_t i = 0; i < m->num_packs; i++)\n ++\t\t\t\tmidx_pack_names[midx_pack_names_nr++] =\n ++\t\t\t\t\txstrdup(m->pack_names[i]);\n +\t}\n +\n \tclose_object_store(the_repository->objects);\n\nbase-commit: f9aa0eedb37eb94d9d3711ef0d565fd7cb3b6148\n-- \n2.50.0.61.g1981e40f2d\n"},{"id":"520592","messageId":"8e7b2dacc77623ada5ee938de7a610ae15f49d8e.1750717921.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1750717921.git.me@ttaylorr.com","subject":"[PATCH v6 1/9] pack-objects: use standard option incompatibility functions","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-06-23T22:32:10Z","receivedAt":"2025-06-23T22:32:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"pack-objects has a handful of explicit checks for pairs of command-line\noptions which are mutually incompatible. Many of these pre-date\na699367bb8 (i18n: factorize more 'incompatible options' messages,\n2022-01-31).\n\nConvert the explicit checks into die_for_incompatible_opt2() calls,\nwhich simplifies the implementation and standardizes pack-objects'\noutput when given incompatible options (e.g., --stdin-packs with\n--filter gives different output than --keep-unreachable with\n--unpack-unreachable).\n\nThere is one minor piece of test fallout in t5331 that expects the old\nformat, which has been corrected.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 20 +++++++++++---------\n t/t5331-pack-objects-stdin.sh | 2 +-\n 2 files changed, 12 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 67941c8a60..e7274e0e00 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -5010,9 +5010,10 @@ int cmd_pack_objects(int argc,\n \t\tstrvec_push(&rp, \"--unpacked\");\n \t}\n \n-\tif (exclude_promisor_objects && exclude_promisor_objects_best_effort)\n-\t\tdie(_(\"options '%s' and '%s' cannot be used together\"),\n-\t\t \"--exclude-promisor-objects\", \"--exclude-promisor-objects-best-effort\");\n+\tdie_for_incompatible_opt2(exclude_promisor_objects,\n+\t\t\t\t \"--exclude-promisor-objects\",\n+\t\t\t\t exclude_promisor_objects_best_effort,\n+\t\t\t\t \"--exclude-promisor-objects-best-effort\");\n \tif (exclude_promisor_objects) {\n \t\tuse_internal_rev_list = 1;\n \t\tfetch_if_missing = 0;\n@@ -5050,13 +5051,14 @@ int cmd_pack_objects(int argc,\n \tif (!pack_to_stdout && thin)\n \t\tdie(_(\"--thin cannot be used to build an indexable pack\"));\n \n-\tif (keep_unreachable && unpack_unreachable)\n-\t\tdie(_(\"options '%s' and '%s' cannot be used together\"), \"--keep-unreachable\", \"--unpack-unreachable\");\n+\tdie_for_incompatible_opt2(keep_unreachable, \"--keep-unreachable\",\n+\t\t\t\t unpack_unreachable, \"--unpack-unreachable\");\n \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n \t\tunpack_unreachable_expiration = 0;\n \n-\tif (stdin_packs && filter_options.choice)\n-\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n+\tdie_for_incompatible_opt2(stdin_packs, \"--stdin-packs\",\n+\t\t\t\t filter_options.choice, \"--filter\");\n+\n \n \tif (stdin_packs && use_internal_rev_list)\n \t\tdie(_(\"cannot use internal rev list with --stdin-packs\"));\n@@ -5064,8 +5066,8 @@ int cmd_pack_objects(int argc,\n \tif (cruft) {\n \t\tif (use_internal_rev_list)\n \t\t\tdie(_(\"cannot use internal rev list with --cruft\"));\n-\t\tif (stdin_packs)\n-\t\t\tdie(_(\"cannot use --stdin-packs with --cruft\"));\n+\t\tdie_for_incompatible_opt2(stdin_packs, \"--stdin-packs\",\n+\t\t\t\t\t cruft, \"--cruft\");\n \t}\n \n \t/*\ndiff --git a/t/t5331-pack-objects-stdin.sh b/t/t5331-pack-objects-stdin.sh\nindex b48c0cbe8f..8fd07deb8d 100755\n--- a/t/t5331-pack-objects-stdin.sh\n+++ b/t/t5331-pack-objects-stdin.sh\n@@ -64,7 +64,7 @@ test_expect_success '--stdin-packs is incompatible with --filter' '\n \t\tcd stdin-packs &&\n \t\ttest_must_fail git pack-objects --stdin-packs --stdout \\\n \t\t\t--filter=blob:none err &&\n-\t\ttest_grep \"cannot use --filter with --stdin-packs\" err\n+\t\ttest_grep \"options .--stdin-packs. and .--filter. cannot be used together\" err\n \t)\n '\n \n-- \n2.50.0.61.g1981e40f2d\n\n"},{"id":"520593","messageId":"86fb36d3176198fa350dfaed261e8ae64b49b355.1750717921.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1750717921.git.me@ttaylorr.com","subject":"[PATCH v6 2/9] pack-objects: limit scope in 'add_object_entry_from_pack()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-06-23T22:32:13Z","receivedAt":"2025-06-23T22:32:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In add_object_entry_from_pack() we declare 'revs' (given to us through\nthe miscellaneous context argument) earlier in the \"if (p)\" conditional\nthan is necessary. Move it down as far as it can go to reduce its\nscope.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex e7274e0e00..d04a36a6bf 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3725,7 +3725,6 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n \t\treturn 0;\n \n \tif (p) {\n-\t\tstruct rev_info *revs = _data;\n \t\tstruct object_info oi = OBJECT_INFO_INIT;\n \n \t\toi.typep = &type;\n@@ -3733,6 +3732,7 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n \t\t\tdie(_(\"could not get type of object %s in pack %s\"),\n \t\t\t oid_to_hex(oid), p->pack_name);\n \t\t} else if (type == OBJ_COMMIT) {\n+\t\t\tstruct rev_info *revs = _data;\n \t\t\t/*\n \t\t\t * commits in included packs are used as starting points for the\n \t\t\t * subsequent revision walk\n-- \n2.50.0.61.g1981e40f2d\n\n"},{"id":"520594","messageId":"19e8c789e93a9e87e5a1799bc56d2d5b0ca4ff7e.1750717921.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1750717921.git.me@ttaylorr.com","subject":"[PATCH v6 3/9] pack-objects: factor out handling '--stdin-packs'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-06-23T22:32:15Z","receivedAt":"2025-06-23T22:32:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"At the bottom of cmd_pack_objects() we check which mode the command is\nrunning in (e.g., generating a cruft pack, handling '--stdin-packs',\nusing the internal rev-list, etc.) and handle the mode appropriately.\n\nThe '--stdin-packs' case is handled inline (dating back to its\nintroduction in 339bce27f4 (builtin/pack-objects.c: add '--stdin-packs'\noption, 2021-02-22)) since it is relatively short. Extract the body of\n\"if (stdin_packs)\" into its own function to prepare for the\nimplementation to become lengthier in a following commit.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 18 ++++++++++++------\n 1 file changed, 12 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex d04a36a6bf..7ce04b71dd 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3909,6 +3909,17 @@ static void read_packs_list_from_stdin(void)\n \tstring_list_clear(&exclude_packs, 0);\n }\n \n+static void add_unreachable_loose_objects(void);\n+\n+static void read_stdin_packs(int rev_list_unpacked)\n+{\n+\t/* avoids adding objects in excluded packs */\n+\tignore_packed_keep_in_core = 1;\n+\tread_packs_list_from_stdin();\n+\tif (rev_list_unpacked)\n+\t\tadd_unreachable_loose_objects();\n+}\n+\n static void add_cruft_object_entry(const struct object_id *oid, enum object_type type,\n \t\t\t\t struct packed_git *pack, off_t offset,\n \t\t\t\t const char *name, uint32_t mtime)\n@@ -4004,7 +4015,6 @@ static void mark_pack_kept_in_core(struct string_list *packs, unsigned keep)\n \t}\n }\n \n-static void add_unreachable_loose_objects(void);\n static void add_objects_in_unpacked_packs(void);\n \n static void enumerate_cruft_objects(void)\n@@ -5135,11 +5145,7 @@ int cmd_pack_objects(int argc,\n \t\tprogress_state = start_progress(the_repository,\n \t\t\t\t\t\t_(\"Enumerating objects\"), 0);\n \tif (stdin_packs) {\n-\t\t/* avoids adding objects in excluded packs */\n-\t\tignore_packed_keep_in_core = 1;\n-\t\tread_packs_list_from_stdin();\n-\t\tif (rev_list_unpacked)\n-\t\t\tadd_unreachable_loose_objects();\n+\t\tread_stdin_packs(rev_list_unpacked);\n \t} else if (cruft) {\n \t\tread_cruft_objects();\n \t} else if (!use_internal_rev_list) {\n-- \n2.50.0.61.g1981e40f2d\n\n"},{"id":"520595","messageId":"c9f874eb9470bf2a5d97614b89304e892c30e129.1750717921.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1750717921.git.me@ttaylorr.com","subject":"[PATCH v6 4/9] pack-objects: declare 'rev_info' for '--stdin-packs' earlier","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-06-23T22:32:18Z","receivedAt":"2025-06-23T22:32:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Once 'read_packs_list_from_stdin()' has called for_each_object_in_pack()\non each of the input packs, we do a reachability traversal to discover\nnames for any objects we picked up so we can generate name hash values\nand hopefully get higher quality deltas as a result.\n\nA future commit will change the purpose of this reachability traversal\nto find and pack objects which are reachable from commits in the input\npacks, but are packed in an unknown (not included nor excluded) pack.\n\nExtract the code which initializes and performs the reachability\ntraversal to take place in the caller, not the callee, which prepares us\nto share this code for the '--unpacked' case (see the function\nadd_unreachable_loose_objects() for more details).\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 71 +++++++++++++++++++++---------------------\n 1 file changed, 36 insertions(+), 35 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7ce04b71dd..4258ac1792 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3793,7 +3793,7 @@ static int pack_mtime_cmp(const void *_a, const void *_b)\n \t\treturn 0;\n }\n \n-static void read_packs_list_from_stdin(void)\n+static void read_packs_list_from_stdin(struct rev_info *revs)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct string_list include_packs = STRING_LIST_INIT_DUP;\n@@ -3801,24 +3801,6 @@ static void read_packs_list_from_stdin(void)\n \tstruct string_list_item *item = NULL;\n \n \tstruct packed_git *p;\n-\tstruct rev_info revs;\n-\n-\trepo_init_revisions(the_repository, &revs, NULL);\n-\t/*\n-\t * Use a revision walk to fill in the namehash of objects in the include\n-\t * packs. To save time, we'll avoid traversing through objects that are\n-\t * in excluded packs.\n-\t *\n-\t * That may cause us to avoid populating all of the namehash fields of\n-\t * all included objects, but our goal is best-effort, since this is only\n-\t * an optimization during delta selection.\n-\t */\n-\trevs.no_kept_objects = 1;\n-\trevs.keep_pack_cache_flags |= IN_CORE_KEEP_PACKS;\n-\trevs.blob_objects = 1;\n-\trevs.tree_objects = 1;\n-\trevs.tag_objects = 1;\n-\trevs.ignore_missing_links = 1;\n \n \twhile (strbuf_getline(&buf, stdin) != EOF) {\n \t\tif (!buf.len)\n@@ -3888,10 +3870,44 @@ static void read_packs_list_from_stdin(void)\n \t\tstruct packed_git *p = item->util;\n \t\tfor_each_object_in_pack(p,\n \t\t\t\t\tadd_object_entry_from_pack,\n-\t\t\t\t\t&revs,\n+\t\t\t\t\trevs,\n \t\t\t\t\tFOR_EACH_OBJECT_PACK_ORDER);\n \t}\n \n+\tstrbuf_release(&buf);\n+\tstring_list_clear(&include_packs, 0);\n+\tstring_list_clear(&exclude_packs, 0);\n+}\n+\n+static void add_unreachable_loose_objects(void);\n+\n+static void read_stdin_packs(int rev_list_unpacked)\n+{\n+\tstruct rev_info revs;\n+\n+\trepo_init_revisions(the_repository, &revs, NULL);\n+\t/*\n+\t * Use a revision walk to fill in the namehash of objects in the include\n+\t * packs. To save time, we'll avoid traversing through objects that are\n+\t * in excluded packs.\n+\t *\n+\t * That may cause us to avoid populating all of the namehash fields of\n+\t * all included objects, but our goal is best-effort, since this is only\n+\t * an optimization during delta selection.\n+\t */\n+\trevs.no_kept_objects = 1;\n+\trevs.keep_pack_cache_flags |= IN_CORE_KEEP_PACKS;\n+\trevs.blob_objects = 1;\n+\trevs.tree_objects = 1;\n+\trevs.tag_objects = 1;\n+\trevs.ignore_missing_links = 1;\n+\n+\t/* avoids adding objects in excluded packs */\n+\tignore_packed_keep_in_core = 1;\n+\tread_packs_list_from_stdin(&revs);\n+\tif (rev_list_unpacked)\n+\t\tadd_unreachable_loose_objects();\n+\n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n \ttraverse_commit_list(&revs,\n@@ -3903,21 +3919,6 @@ static void read_packs_list_from_stdin(void)\n \t\t\t stdin_packs_found_nr);\n \ttrace2_data_intmax(\"pack-objects\", the_repository, \"stdin_packs_hints\",\n \t\t\t stdin_packs_hints_nr);\n-\n-\tstrbuf_release(&buf);\n-\tstring_list_clear(&include_packs, 0);\n-\tstring_list_clear(&exclude_packs, 0);\n-}\n-\n-static void add_unreachable_loose_objects(void);\n-\n-static void read_stdin_packs(int rev_list_unpacked)\n-{\n-\t/* avoids adding objects in excluded packs */\n-\tignore_packed_keep_in_core = 1;\n-\tread_packs_list_from_stdin();\n-\tif (rev_list_unpacked)\n-\t\tadd_unreachable_loose_objects();\n }\n \n static void add_cruft_object_entry(const struct object_id *oid, enum object_type type,\n-- \n2.50.0.61.g1981e40f2d\n\n"},{"id":"520596","messageId":"6b0149a32d300268d4ad870c7cb6597a95e0410b.1750717921.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1750717921.git.me@ttaylorr.com","subject":"[PATCH v6 5/9] pack-objects: perform name-hash traversal for unpacked objects","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-06-23T22:32:21Z","receivedAt":"2025-06-23T22:32:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"With '--unpacked', pack-objects adds loose objects (which don't appear\nin any of the excluded packs from '--stdin-packs') to the output pack\nwithout considering them as reachability tips for the name-hash\ntraversal.\n\nThis was an oversight in the original implementation of '--stdin-packs',\nsince the code which enumerates and adds loose objects to the output\npack (`add_unreachable_loose_objects()`) did not have access to the\n'rev_info' struct found in `read_packs_list_from_stdin()`.\n\nExcluding unpacked objects from that traversal doesn't affect the\ncorrectness of the resulting pack, but it does make it harder to\ndiscover good deltas for loose objects.\n\nNow that the 'rev_info' struct is declared outside of\n`read_packs_list_from_stdin()`, we can pass it to\n`add_objects_in_unpacked_packs()` and add any loose objects as tips to\nthe above-mentioned traversal, in theory producing slightly tighter\npacks as a result.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 20 ++++++++++++--------\n 1 file changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 4258ac1792..3437dbd7f1 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3879,7 +3879,7 @@ static void read_packs_list_from_stdin(struct rev_info *revs)\n \tstring_list_clear(&exclude_packs, 0);\n }\n \n-static void add_unreachable_loose_objects(void);\n+static void add_unreachable_loose_objects(struct rev_info *revs);\n \n static void read_stdin_packs(int rev_list_unpacked)\n {\n@@ -3906,7 +3906,7 @@ static void read_stdin_packs(int rev_list_unpacked)\n \tignore_packed_keep_in_core = 1;\n \tread_packs_list_from_stdin(&revs);\n \tif (rev_list_unpacked)\n-\t\tadd_unreachable_loose_objects();\n+\t\tadd_unreachable_loose_objects(&revs);\n \n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n@@ -4025,7 +4025,7 @@ static void enumerate_cruft_objects(void)\n \t\t\t\t\t\t_(\"Enumerating cruft objects\"), 0);\n \n \tadd_objects_in_unpacked_packs();\n-\tadd_unreachable_loose_objects();\n+\tadd_unreachable_loose_objects(NULL);\n \n \tstop_progress(&progress_state);\n }\n@@ -4303,8 +4303,9 @@ static void add_objects_in_unpacked_packs(void)\n }\n \n static int add_loose_object(const struct object_id *oid, const char *path,\n-\t\t\t void *data UNUSED)\n+\t\t\t void *data)\n {\n+\tstruct rev_info *revs = data;\n \tenum object_type type = oid_object_info(the_repository, oid, NULL);\n \n \tif (type < 0) {\n@@ -4325,6 +4326,10 @@ static int add_loose_object(const struct object_id *oid, const char *path,\n \t} else {\n \t\tadd_object_entry(oid, type, \"\", 0);\n \t}\n+\n+\tif (revs && type == OBJ_COMMIT)\n+\t\tadd_pending_oid(revs, NULL, oid, 0);\n+\n \treturn 0;\n }\n \n@@ -4333,11 +4338,10 @@ static int add_loose_object(const struct object_id *oid, const char *path,\n * add_object_entry will weed out duplicates, so we just add every\n * loose object we find.\n */\n-static void add_unreachable_loose_objects(void)\n+static void add_unreachable_loose_objects(struct rev_info *revs)\n {\n \tfor_each_loose_file_in_objdir(repo_get_object_directory(the_repository),\n-\t\t\t\t add_loose_object,\n-\t\t\t\t NULL, NULL, NULL);\n+\t\t\t\t add_loose_object, NULL, NULL, revs);\n }\n \n static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)\n@@ -4684,7 +4688,7 @@ static void get_object_list(struct rev_info *revs, int ac, const char **av)\n \tif (keep_unreachable)\n \t\tadd_objects_in_unpacked_packs();\n \tif (pack_loose_unreachable)\n-\t\tadd_unreachable_loose_objects();\n+\t\tadd_unreachable_loose_objects(NULL);\n \tif (unpack_unreachable)\n \t\tloosen_unused_packed_objects();\n \n-- \n2.50.0.61.g1981e40f2d\n\n"},{"id":"520597","messageId":"f31dd00a980669e746c5e8f1297c0a4b1f9b41b5.1750717921.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1750717921.git.me@ttaylorr.com","subject":"[PATCH v6 6/9] pack-objects: fix typo in 'show_object_pack_hint()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-06-23T22:32:24Z","receivedAt":"2025-06-23T22:32:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Noticed-by: Elijah Newren \nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 3437dbd7f1..9580b4ea1a 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3767,7 +3767,7 @@ static void show_object_pack_hint(struct object *object, const char *name,\n \t * would typically pick up during a reachability traversal.\n \t *\n \t * Make a best-effort attempt to fill in the ->hash and ->no_try_delta\n-\t * here using a now in order to perhaps improve the delta selection\n+\t * fields here in order to perhaps improve the delta selection\n \t * process.\n \t */\n \toe->hash = pack_name_hash_fn(name);\n-- \n2.50.0.61.g1981e40f2d\n\n"},{"id":"520598","messageId":"5d15055985a8d4eb5fd7f7b9026633c74d6c86a9.1750717921.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1750717921.git.me@ttaylorr.com","subject":"[PATCH v6 7/9] pack-objects: swap 'show_{object,commit}_pack_hint'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-06-23T22:32:27Z","receivedAt":"2025-06-23T22:32:29Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"show_commit_pack_hint() has heretofore been a noop, so its position\nwithin its compilation unit only needs to appear before its first use.\n\nBut the following commit will sometimes have `show_commit_pack_hint()`\ncall `show_object_pack_hint()`, so reorder the former to appear after\nthe latter to minimize the code movement in that patch.\n\nSuggested-by: Elijah Newren \nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 9580b4ea1a..f44447a3f9 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3748,12 +3748,6 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n \treturn 0;\n }\n \n-static void show_commit_pack_hint(struct commit *commit UNUSED,\n-\t\t\t\t void *data UNUSED)\n-{\n-\t/* nothing to do; commits don't have a namehash */\n-}\n-\n static void show_object_pack_hint(struct object *object, const char *name,\n \t\t\t\t void *data UNUSED)\n {\n@@ -3776,6 +3770,12 @@ static void show_object_pack_hint(struct object *object, const char *name,\n \tstdin_packs_hints_nr++;\n }\n \n+static void show_commit_pack_hint(struct commit *commit UNUSED,\n+\t\t\t\t void *data UNUSED)\n+{\n+\t/* nothing to do; commits don't have a namehash */\n+}\n+\n static int pack_mtime_cmp(const void *_a, const void *_b)\n {\n \tstruct packed_git *a = ((const struct string_list_item*)_a)->util;\n-- \n2.50.0.61.g1981e40f2d\n\n"},{"id":"520599","messageId":"3699c25337e1c91388bad4c56441b39a9984798b.1750717921.git.me@ttaylorr.com","threadId":"63286","inReplyTo":"cover.1750717921.git.me@ttaylorr.com","subject":"[PATCH v6 8/9] pack-objects: introduce '--stdin-packs=follow'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-06-23T22:32:30Z","receivedAt":"2025-06-23T22:32:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When invoked with '--stdin-packs', pack-objects will generate a pack\nwhich contains the objects found in the \"included\" packs, less any\nobjects from \"excluded\" packs.\n\nPacks that exist in the repository but weren't specified as either\nincluded or excluded are in practice treated like the latter, at least\nin the sense that pack-objects won't include objects from those packs.\nThis behavior forces us to include any cruft pack(s) in a repository's\nmulti-pack index for the reasons described in ddee3703b3\n(builtin/repack.c: add cruft packs to MIDX during geometric repack,\n2022-05-20).\n\nThe full details are in ddee3703b3, but the gist is if you\nhave a once-unreachable object in a cruft pack which later becomes\nreachable via one or more commits in a pack generated with\n'--stdin-packs', you *have* to include that object in the MIDX via the\ncopy in the cruft pack, otherwise we cannot generate reachability\nbitmaps for any commits which reach that object.\n\nNote that the traversal here is best-effort, similar to the existing\ntraversal which provides name-hash hints. This means that the object\ntraversal may hand us back a blob that does not actually exist. We\n*won't* see missing trees/commits with 'ignore_missing_links' because:\n\n - missing commit parents are discarded at the commit traversal stage by\n revision.c::process_parents()\n\n - missing tag objects are discarded by revision.c::handle_commit()\n\n - missing tree objects are discarded by the list-objects code in\n list-objects.c::process_tree()\n\nBut we have to handle potentially-missing blobs specially by making a\nseparate check to ensure they exist in the repository. Failing to do so\nwould mean that we'd add an object to the packing list which doesn't\nactually exist, rendering us unable to write out the pack.\n\nThis prepares us for new repacking behavior which will \"resurrect\"\nobjects found in cruft or otherwise unspecified packs when generating\nnew packs. In the context of geometric repacking, this may be used to\nmaintain a sequence of geometrically-repacked packs, the union of which\nis closed under reachability, even in the case described earlier.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-pack-objects.adoc | 10 ++-\n builtin/pack-objects.c | 86 +++++++++++++++-----\n t/t5331-pack-objects-stdin.sh | 120 ++++++++++++++++++++++++++++\n 3 files changed, 193 insertions(+), 23 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.adoc b/Documentation/git-pack-objects.adoc\nindex b1c5aa27da..eba014c406 100644\n--- a/Documentation/git-pack-objects.adoc\n+++ b/Documentation/git-pack-objects.adoc\n@@ -87,13 +87,21 @@ base-name::\n \treference was included in the resulting packfile. This\n \tcan be useful to send new tags to native Git clients.\n \n---stdin-packs::\n+--stdin-packs[=]::\n \tRead the basenames of packfiles (e.g., `pack-1234abcd.pack`)\n \tfrom the standard input, instead of object names or revision\n \targuments. The resulting pack contains all objects listed in the\n \tincluded packs (those not beginning with `^`), excluding any\n \tobjects listed in the excluded packs (beginning with `^`).\n +\n+When `mode` is \"follow\", objects from packs not listed on stdin receive\n+special treatment. Objects within unlisted packs will be included if\n+those objects are (1) reachable from the included packs, and (2) not\n+found in any excluded packs. This mode is useful, for example, to\n+resurrect once-unreachable objects found in cruft packs to generate\n+packs which are closed under reachability up to the boundary set by the\n+excluded packs.\n++\n Incompatible with `--revs`, or options that imply `--revs` (such as\n `--all`), with the exception of `--unpacked`, which is compatible.\n \ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex f44447a3f9..4ae52c6a29 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -284,6 +284,12 @@ static struct oidmap configured_exclusions;\n static struct oidset excluded_by_config;\n static int name_hash_version = -1;\n \n+enum stdin_packs_mode {\n+\tSTDIN_PACKS_MODE_NONE,\n+\tSTDIN_PACKS_MODE_STANDARD,\n+\tSTDIN_PACKS_MODE_FOLLOW,\n+};\n+\n /**\n * Check whether the name_hash_version chosen by user input is appropriate,\n * and also validate whether it is compatible with other features.\n@@ -3749,31 +3755,47 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n }\n \n static void show_object_pack_hint(struct object *object, const char *name,\n-\t\t\t\t void *data UNUSED)\n+\t\t\t\t void *data)\n {\n-\tstruct object_entry *oe = packlist_find(&to_pack, &object->oid);\n-\tif (!oe)\n-\t\treturn;\n+\tenum stdin_packs_mode mode = *(enum stdin_packs_mode *)data;\n+\tif (mode == STDIN_PACKS_MODE_FOLLOW) {\n+\t\tif (object->type == OBJ_BLOB &&\n+\t\t !has_object(the_repository, &object->oid, 0))\n+\t\t\treturn;\n+\t\tadd_object_entry(&object->oid, object->type, name, 0);\n+\t} else {\n+\t\tstruct object_entry *oe = packlist_find(&to_pack, &object->oid);\n+\t\tif (!oe)\n+\t\t\treturn;\n \n-\t/*\n-\t * Our 'to_pack' list was constructed by iterating all objects packed in\n-\t * included packs, and so doesn't have a non-zero hash field that you\n-\t * would typically pick up during a reachability traversal.\n-\t *\n-\t * Make a best-effort attempt to fill in the ->hash and ->no_try_delta\n-\t * fields here in order to perhaps improve the delta selection\n-\t * process.\n-\t */\n-\toe->hash = pack_name_hash_fn(name);\n-\toe->no_try_delta = name && no_try_delta(name);\n+\t\t/*\n+\t\t * Our 'to_pack' list was constructed by iterating all\n+\t\t * objects packed in included packs, and so doesn't have\n+\t\t * a non-zero hash field that you would typically pick\n+\t\t * up during a reachability traversal.\n+\t\t *\n+\t\t * Make a best-effort attempt to fill in the ->hash and\n+\t\t * ->no_try_delta fields here in order to perhaps\n+\t\t * improve the delta selection process.\n+\t\t */\n+\t\toe->hash = pack_name_hash_fn(name);\n+\t\toe->no_try_delta = name && no_try_delta(name);\n \n-\tstdin_packs_hints_nr++;\n+\t\tstdin_packs_hints_nr++;\n+\t}\n }\n \n-static void show_commit_pack_hint(struct commit *commit UNUSED,\n-\t\t\t\t void *data UNUSED)\n+static void show_commit_pack_hint(struct commit *commit, void *data)\n {\n+\tenum stdin_packs_mode mode = *(enum stdin_packs_mode *)data;\n+\n+\tif (mode == STDIN_PACKS_MODE_FOLLOW) {\n+\t\tshow_object_pack_hint((struct object *)commit, \"\", data);\n+\t\treturn;\n+\t}\n+\n \t/* nothing to do; commits don't have a namehash */\n+\n }\n \n static int pack_mtime_cmp(const void *_a, const void *_b)\n@@ -3881,7 +3903,7 @@ static void read_packs_list_from_stdin(struct rev_info *revs)\n \n static void add_unreachable_loose_objects(struct rev_info *revs);\n \n-static void read_stdin_packs(int rev_list_unpacked)\n+static void read_stdin_packs(enum stdin_packs_mode mode, int rev_list_unpacked)\n {\n \tstruct rev_info revs;\n \n@@ -3913,7 +3935,7 @@ static void read_stdin_packs(int rev_list_unpacked)\n \ttraverse_commit_list(&revs,\n \t\t\t show_commit_pack_hint,\n \t\t\t show_object_pack_hint,\n-\t\t\t NULL);\n+\t\t\t &mode);\n \n \ttrace2_data_intmax(\"pack-objects\", the_repository, \"stdin_packs_found\",\n \t\t\t stdin_packs_found_nr);\n@@ -4795,6 +4817,23 @@ static int is_not_in_promisor_pack(struct commit *commit, void *data) {\n \treturn is_not_in_promisor_pack_obj((struct object *) commit, data);\n }\n \n+static int parse_stdin_packs_mode(const struct option *opt, const char *arg,\n+\t\t\t\t int unset)\n+{\n+\tenum stdin_packs_mode *mode = opt->value;\n+\n+\tif (unset)\n+\t\t*mode = STDIN_PACKS_MODE_NONE;\n+\telse if (!arg || !*arg)\n+\t\t*mode = STDIN_PACKS_MODE_STANDARD;\n+\telse if (!strcmp(arg, \"follow\"))\n+\t\t*mode = STDIN_PACKS_MODE_FOLLOW;\n+\telse\n+\t\tdie(_(\"invalid value for '%s': '%s'\"), opt->long_name, arg);\n+\n+\treturn 0;\n+}\n+\n int cmd_pack_objects(int argc,\n \t\t const char **argv,\n \t\t const char *prefix,\n@@ -4805,7 +4844,7 @@ int cmd_pack_objects(int argc,\n \tstruct strvec rp = STRVEC_INIT;\n \tint rev_list_unpacked = 0, rev_list_all = 0, rev_list_reflog = 0;\n \tint rev_list_index = 0;\n-\tint stdin_packs = 0;\n+\tenum stdin_packs_mode stdin_packs = STDIN_PACKS_MODE_NONE;\n \tstruct string_list keep_pack_list = STRING_LIST_INIT_NODUP;\n \tstruct list_objects_filter_options filter_options =\n \t\tLIST_OBJECTS_FILTER_INIT;\n@@ -4860,6 +4899,9 @@ int cmd_pack_objects(int argc,\n \t\tOPT_SET_INT_F(0, \"indexed-objects\", &rev_list_index,\n \t\t\t N_(\"include objects referred to by the index\"),\n \t\t\t 1, PARSE_OPT_NONEG),\n+\t\tOPT_CALLBACK_F(0, \"stdin-packs\", &stdin_packs, N_(\"mode\"),\n+\t\t\t N_(\"read packs from stdin\"),\n+\t\t\t PARSE_OPT_OPTARG, parse_stdin_packs_mode),\n \t\tOPT_BOOL(0, \"stdin-packs\", &stdin_packs,\n \t\t\t N_(\"read packs from stdin\")),\n \t\tOPT_BOOL(0, \"stdout\", &pack_to_stdout,\n@@ -5150,7 +5192,7 @@ int cmd_pack_objects(int argc,\n \t\tprogress_state = start_progress(the_repository,\n \t\t\t\t\t\t_(\"Enumerating objects\"), 0);\n \tif (stdin_packs) {\n-\t\tread_stdin_packs(rev_list_unpacked);\n+\t\tread_stdin_packs(stdin_packs, rev_list_unpacked);\n \t} else if (cruft) {\n \t\tread_cruft_objects();\n \t} else if (!use_internal_rev_list) {\ndiff --git a/t/t5331-pack-objects-stdin.sh b/t/t5331-pack-objects-stdin.sh\nindex 8fd07deb8d..4a8df5a389 100755\n--- a/t/t5331-pack-objects-stdin.sh\n+++ b/t/t5331-pack-objects-stdin.sh\n@@ -236,4 +236,124 @@ test_expect_success 'pack-objects --stdin with packfiles from main and alternate\n \ttest_cmp expected-objects actual-objects\n '\n \n+objdir=.git/objects\n+packdir=$objdir/pack\n+\n+objects_in_packs () {\n+\tfor p in \"$@\"\n+\tdo\n+\t\tgit show-index <\"$packdir/pack-$p.idx\" || return 1\n+\tdone >objects.raw &&\n+\n+\tcut -d' ' -f2 objects.raw | sort &&\n+\trm -f objects.raw\n+}\n+\n+test_expect_success '--stdin-packs=follow walks into unknown packs' '\n+\ttest_when_finished \"rm -fr repo\" &&\n+\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\tfor c in A B C D\n+\t\tdo\n+\t\t\ttest_commit \"$c\" || return 1\n+\t\tdone &&\n+\n+\t\tA=\"$(echo A | git pack-objects --revs $packdir/pack)\" &&\n+\t\tB=\"$(echo A..B | git pack-objects --revs $packdir/pack)\" &&\n+\t\tC=\"$(echo B..C | git pack-objects --revs $packdir/pack)\" &&\n+\t\tD=\"$(echo C..D | git pack-objects --revs $packdir/pack)\" &&\n+\t\ttest_commit E &&\n+\n+\t\tgit prune-packed &&\n+\n+\t\tcat >in <<-EOF &&\n+\t\tpack-$B.pack\n+\t\t^pack-$C.pack\n+\t\tpack-$D.pack\n+\t\tEOF\n+\n+\t\t# With just --stdin-packs, pack \"A\" is unknown to us, so\n+\t\t# only objects from packs \"B\" and \"D\" are included in\n+\t\t# the output pack.\n+\t\tP=$(git pack-objects --stdin-packs $packdir/pack expect &&\n+\t\tobjects_in_packs $P >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\t# But with --stdin-packs=follow, objects from both\n+\t\t# included packs reach objects from the unknown pack, so\n+\t\t# objects from pack \"A\" is included in the output pack\n+\t\t# in addition to the above.\n+\t\tP=$(git pack-objects --stdin-packs=follow $packdir/pack expect &&\n+\t\tobjects_in_packs $P >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\t# And with --unpacked, we will pick up objects from unknown\n+\t\t# packs that are reachable from loose objects. Loose object E\n+\t\t# reaches objects in pack A, but there are three excluded packs\n+\t\t# in between.\n+\t\t#\n+\t\t# The resulting pack should include objects reachable from E\n+\t\t# that are not present in packs B, C, or D, along with those\n+\t\t# present in pack A.\n+\t\tcat >in <<-EOF &&\n+\t\t^pack-$B.pack\n+\t\t^pack-$C.pack\n+\t\t^pack-$D.pack\n+\t\tEOF\n+\n+\t\tP=$(git pack-objects --stdin-packs=follow --unpacked \\\n+\t\t\t$packdir/pack expect.raw &&\n+\t\tsort expect.raw >expect &&\n+\t\tobjects_in_packs $P >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+stdin_packs__follow_with_only () {\n+\trm -fr stdin_packs__follow_with_only &&\n+\tgit init stdin_packs__follow_with_only &&\n+\t(\n+\t\tcd stdin_packs__follow_with_only &&\n+\n+\t\ttest_commit A &&\n+\t\ttest_commit B &&\n+\n+\t\tgit rev-parse \"$@\" >B.objects &&\n+\n+\t\techo A | git pack-objects --revs $packdir/pack &&\n+\t\tB=\"$(git pack-objects $packdir/pack objs &&\n+\t\tfor obj in $(cat objs)\n+\t\tdo\n+\t\t\trm -f $objdir/$(test_oid_to_path $obj) || return 1\n+\t\tdone &&\n+\n+\t\t( cd $packdir && ls pack-*.pack ) >in &&\n+\t\tgit pack-objects --stdin-packs=follow --stdout >/dev/null \n---\n Documentation/config/repack.adoc | 7 ++\n builtin/repack.c | 187 +++++++++++++++++++++++++++----\n t/t7704-repack-cruft.sh | 145 ++++++++++++++++++++++++\n 3 files changed, 319 insertions(+), 20 deletions(-)\n\ndiff --git a/Documentation/config/repack.adoc b/Documentation/config/repack.adoc\nindex c79af6d7b8..e9e78dcb19 100644\n--- a/Documentation/config/repack.adoc\n+++ b/Documentation/config/repack.adoc\n@@ -39,3 +39,10 @@ repack.cruftThreads::\n \ta cruft pack and the respective parameters are not given over\n \tthe command line. See similarly named `pack.*` configuration\n \tvariables for defaults and meaning.\n+\n+repack.midxMustContainCruft::\n+\tWhen set to true, linkgit:git-repack[1] will unconditionally include\n+\tcruft pack(s), if any, in the multi-pack index when invoked with\n+\t`--write-midx`. When false, cruft packs are only included in the MIDX\n+\twhen necessary (e.g., because they might be required to form a\n+\treachability closure with MIDX bitmaps). Defaults to true.\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 5ddc6e7f95..8d1540a0fd 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -39,6 +39,7 @@ static int write_bitmaps = -1;\n static int use_delta_islands;\n static int run_update_server_info = 1;\n static char *packdir, *packtmp_name, *packtmp;\n+static int midx_must_contain_cruft = 1;\n \n static const char *const git_repack_usage[] = {\n \tN_(\"git repack [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\\n\"\n@@ -108,6 +109,10 @@ static int repack_config(const char *var, const char *value,\n \t\tfree(cruft_po_args->threads);\n \t\treturn git_config_string(&cruft_po_args->threads, var, value);\n \t}\n+\tif (!strcmp(var, \"repack.midxmustcontaincruft\")) {\n+\t\tmidx_must_contain_cruft = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n \treturn git_default_config(var, value, ctx, cb);\n }\n \n@@ -690,6 +695,77 @@ static void free_pack_geometry(struct pack_geometry *geometry)\n \tfree(geometry->pack);\n }\n \n+static int midx_has_unknown_packs(char **midx_pack_names,\n+\t\t\t\t size_t midx_pack_names_nr,\n+\t\t\t\t struct string_list *include,\n+\t\t\t\t struct pack_geometry *geometry,\n+\t\t\t\t struct existing_packs *existing)\n+{\n+\tsize_t i;\n+\n+\tstring_list_sort(include);\n+\n+\tfor (i = 0; i < midx_pack_names_nr; i++) {\n+\t\tconst char *pack_name = midx_pack_names[i];\n+\n+\t\t/*\n+\t\t * Determine whether or not each MIDX'd pack from the existing\n+\t\t * MIDX (if any) is represented in the new MIDX. For each pack\n+\t\t * in the MIDX, it must either be:\n+\t\t *\n+\t\t * - In the \"include\" list of packs to be included in the new\n+\t\t * MIDX. Note this function is called before the include\n+\t\t * list is populated with any cruft pack(s).\n+\t\t *\n+\t\t * - Below the geometric split line (if using pack geometry),\n+\t\t * indicating that the pack won't be included in the new\n+\t\t * MIDX, but its contents were rolled up as part of the\n+\t\t * geometric repack.\n+\t\t *\n+\t\t * - In the existing non-kept packs list (if not using pack\n+\t\t * geometry), and marked as non-deleted.\n+\t\t */\n+\t\tif (string_list_has_string(include, pack_name)) {\n+\t\t\tcontinue;\n+\t\t} else if (geometry) {\n+\t\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\t\tuint32_t j;\n+\n+\t\t\tfor (j = 0; j < geometry->split; j++) {\n+\t\t\t\tstrbuf_reset(&buf);\n+\t\t\t\tstrbuf_addstr(&buf, pack_basename(geometry->pack[j]));\n+\t\t\t\tstrbuf_strip_suffix(&buf, \".pack\");\n+\t\t\t\tstrbuf_addstr(&buf, \".idx\");\n+\n+\t\t\t\tif (!strcmp(pack_name, buf.buf)) {\n+\t\t\t\t\tstrbuf_release(&buf);\n+\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\tstrbuf_release(&buf);\n+\n+\t\t\tif (j < geometry->split)\n+\t\t\t\tcontinue;\n+\t\t} else {\n+\t\t\tstruct string_list_item *item;\n+\n+\t\t\titem = string_list_lookup(&existing->non_kept_packs,\n+\t\t\t\t\t\t pack_name);\n+\t\t\tif (item && !pack_is_marked_for_deletion(item))\n+\t\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/*\n+\t\t * If we got to this point, the MIDX includes some pack that we\n+\t\t * don't know about.\n+\t\t */\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n struct midx_snapshot_ref_data {\n \tstruct tempfile *f;\n \tstruct oidset seen;\n@@ -758,6 +834,8 @@ static void midx_snapshot_refs(struct tempfile *f)\n \n static void midx_included_packs(struct string_list *include,\n \t\t\t\tstruct existing_packs *existing,\n+\t\t\t\tchar **midx_pack_names,\n+\t\t\t\tsize_t midx_pack_names_nr,\n \t\t\t\tstruct string_list *names,\n \t\t\t\tstruct pack_geometry *geometry)\n {\n@@ -811,26 +889,56 @@ static void midx_included_packs(struct string_list *include,\n \t\t}\n \t}\n \n-\tfor_each_string_list_item(item, &existing->cruft_packs) {\n+\tif (midx_must_contain_cruft ||\n+\t midx_has_unknown_packs(midx_pack_names, midx_pack_names_nr,\n+\t\t\t\t include, geometry, existing)) {\n \t\t/*\n-\t\t * When doing a --geometric repack, there is no need to check\n-\t\t * for deleted packs, since we're by definition not doing an\n-\t\t * ALL_INTO_ONE repack (hence no packs will be deleted).\n-\t\t * Otherwise we must check for and exclude any packs which are\n-\t\t * enqueued for deletion.\n+\t\t * If there are one or more unknown pack(s) present (see\n+\t\t * midx_has_unknown_packs() for what makes a pack\n+\t\t * \"unknown\") in the MIDX before the repack, keep them\n+\t\t * as they may be required to form a reachability\n+\t\t * closure if the MIDX is bitmapped.\n \t\t *\n-\t\t * So we could omit the conditional below in the --geometric\n-\t\t * case, but doing so is unnecessary since no packs are marked\n-\t\t * as pending deletion (since we only call\n-\t\t * `mark_packs_for_deletion()` when doing an all-into-one\n-\t\t * repack).\n+\t\t * For example, a cruft pack can be required to form a\n+\t\t * reachability closure if the MIDX is bitmapped and one\n+\t\t * or more of the bitmap's selected commits reaches a\n+\t\t * once-cruft object that was later made reachable.\n \t\t */\n-\t\tif (pack_is_marked_for_deletion(item))\n-\t\t\tcontinue;\n+\t\tfor_each_string_list_item(item, &existing->cruft_packs) {\n+\t\t\t/*\n+\t\t\t * When doing a --geometric repack, there is no\n+\t\t\t * need to check for deleted packs, since we're\n+\t\t\t * by definition not doing an ALL_INTO_ONE\n+\t\t\t * repack (hence no packs will be deleted).\n+\t\t\t * Otherwise we must check for and exclude any\n+\t\t\t * packs which are enqueued for deletion.\n+\t\t\t *\n+\t\t\t * So we could omit the conditional below in the\n+\t\t\t * --geometric case, but doing so is unnecessary\n+\t\t\t * since no packs are marked as pending\n+\t\t\t * deletion (since we only call\n+\t\t\t * `mark_packs_for_deletion()` when doing an\n+\t\t\t * all-into-one repack).\n+\t\t\t */\n+\t\t\tif (pack_is_marked_for_deletion(item))\n+\t\t\t\tcontinue;\n \n-\t\tstrbuf_reset(&buf);\n-\t\tstrbuf_addf(&buf, \"%s.idx\", item->string);\n-\t\tstring_list_insert(include, buf.buf);\n+\t\t\tstrbuf_reset(&buf);\n+\t\t\tstrbuf_addf(&buf, \"%s.idx\", item->string);\n+\t\t\tstring_list_insert(include, buf.buf);\n+\t\t}\n+\t} else {\n+\t\t/*\n+\t\t * Modern versions of Git (with the appropriate\n+\t\t * configuration setting) will write new copies of\n+\t\t * once-cruft objects when doing a --geometric repack.\n+\t\t *\n+\t\t * If the MIDX has no cruft pack, new packs written\n+\t\t * during a --geometric repack will not rely on the\n+\t\t * cruft pack to form a reachability closure, so we can\n+\t\t * avoid including them in the MIDX in that case.\n+\t\t */\n+\t\t;\n \t}\n \n \tstrbuf_release(&buf);\n@@ -1145,6 +1253,8 @@ int cmd_repack(int argc,\n \tstruct tempfile *refs_snapshot = NULL;\n \tint i, ext, ret;\n \tint show_progress;\n+\tchar **midx_pack_names = NULL;\n+\tsize_t midx_pack_names_nr = 0;\n \n \t/* variables to be filled by option parsing */\n \tint delete_redundant = 0;\n@@ -1361,7 +1471,10 @@ int cmd_repack(int argc,\n \t\t !(pack_everything & PACK_CRUFT))\n \t\t\tstrvec_push(&cmd.args, \"--pack-loose-unreachable\");\n \t} else if (geometry.split_factor) {\n-\t\tstrvec_push(&cmd.args, \"--stdin-packs\");\n+\t\tif (midx_must_contain_cruft)\n+\t\t\tstrvec_push(&cmd.args, \"--stdin-packs\");\n+\t\telse\n+\t\t\tstrvec_push(&cmd.args, \"--stdin-packs=follow\");\n \t\tstrvec_push(&cmd.args, \"--unpacked\");\n \t} else {\n \t\tstrvec_push(&cmd.args, \"--unpacked\");\n@@ -1401,8 +1514,25 @@ int cmd_repack(int argc,\n \tif (ret)\n \t\tgoto cleanup;\n \n-\tif (!names.nr && !po_args.quiet)\n-\t\tprintf_ln(_(\"Nothing new to pack.\"));\n+\tif (!names.nr) {\n+\t\tif (!po_args.quiet)\n+\t\t\tprintf_ln(_(\"Nothing new to pack.\"));\n+\t\t/*\n+\t\t * If we didn't write any new packs, the non-cruft packs\n+\t\t * may refer to once-unreachable objects in the cruft\n+\t\t * pack(s).\n+\t\t *\n+\t\t * If there isn't already a MIDX, the one we write\n+\t\t * must include the cruft pack(s), in case the\n+\t\t * non-cruft pack(s) refer to once-cruft objects.\n+\t\t *\n+\t\t * If there is already a MIDX, we can punt here, since\n+\t\t * midx_has_unknown_packs() will make the decision for\n+\t\t * us.\n+\t\t */\n+\t\tif (!get_local_multi_pack_index(the_repository))\n+\t\t\tmidx_must_contain_cruft = 1;\n+\t}\n \n \tif (pack_everything & PACK_CRUFT) {\n \t\tconst char *pack_prefix = find_pack_prefix(packdir, packtmp);\n@@ -1483,6 +1613,19 @@ int cmd_repack(int argc,\n \n \tstring_list_sort(&names);\n \n+\tif (get_local_multi_pack_index(the_repository)) {\n+\t\tstruct multi_pack_index *m =\n+\t\t\tget_local_multi_pack_index(the_repository);\n+\n+\t\tALLOC_ARRAY(midx_pack_names,\n+\t\t\t m->num_packs + m->num_packs_in_base);\n+\n+\t\tfor (; m; m = m->base_midx)\n+\t\t\tfor (uint32_t i = 0; i < m->num_packs; i++)\n+\t\t\t\tmidx_pack_names[midx_pack_names_nr++] =\n+\t\t\t\t\txstrdup(m->pack_names[i]);\n+\t}\n+\n \tclose_object_store(the_repository->objects);\n \n \t/*\n@@ -1524,7 +1667,8 @@ int cmd_repack(int argc,\n \n \tif (write_midx) {\n \t\tstruct string_list include = STRING_LIST_INIT_DUP;\n-\t\tmidx_included_packs(&include, &existing, &names, &geometry);\n+\t\tmidx_included_packs(&include, &existing, midx_pack_names,\n+\t\t\t\t midx_pack_names_nr, &names, &geometry);\n \n \t\tret = write_midx_included_packs(&include, &geometry, &names,\n \t\t\t\t\t\trefs_snapshot ? get_tempfile_path(refs_snapshot) : NULL,\n@@ -1575,6 +1719,9 @@ int cmd_repack(int argc,\n \tstring_list_clear(&names, 1);\n \texisting_packs_release(&existing);\n \tfree_pack_geometry(&geometry);\n+\tfor (size_t i = 0; i < midx_pack_names_nr; i++)\n+\t\tfree(midx_pack_names[i]);\n+\tfree(midx_pack_names);\n \tpack_objects_args_release(&po_args);\n \tpack_objects_args_release(&cruft_po_args);\n \ndiff --git a/t/t7704-repack-cruft.sh b/t/t7704-repack-cruft.sh\nindex 8aebfb45f5..aa2e2e6ad8 100755\n--- a/t/t7704-repack-cruft.sh\n+++ b/t/t7704-repack-cruft.sh\n@@ -724,4 +724,149 @@ test_expect_success 'cruft repack respects --quiet' '\n \t)\n '\n \n+setup_cruft_exclude_tests() {\n+\tgit init \"$1\" &&\n+\t(\n+\t\tcd \"$1\" &&\n+\n+\t\tgit config repack.midxMustContainCruft false &&\n+\n+\t\ttest_commit one &&\n+\n+\t\ttest_commit --no-tag two &&\n+\t\ttwo=\"$(git rev-parse HEAD)\" &&\n+\t\ttest_commit --no-tag three &&\n+\t\tthree=\"$(git rev-parse HEAD)\" &&\n+\t\tgit reset --hard one &&\n+\t\tgit reflog expire --all --expire=all &&\n+\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 git repack --cruft -d &&\n+\n+\t\tgit merge $two &&\n+\t\ttest_commit four\n+\t)\n+}\n+\n+test_expect_success 'repack --write-midx excludes cruft where possible' '\n+\tsetup_cruft_exclude_tests exclude-cruft-when-possible &&\n+\t(\n+\t\tcd exclude-cruft-when-possible &&\n+\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 \\\n+\t\tgit repack -d --geometric=2 --write-midx --write-bitmap-index &&\n+\n+\t\ttest-tool read-midx --show-objects $objdir >midx &&\n+\t\tcruft=\"$(ls $packdir/*.mtimes)\" &&\n+\t\ttest_grep ! \"$(basename \"$cruft\" .mtimes).idx\" midx &&\n+\n+\t\tgit rev-list --all --objects --no-object-names >reachable.raw &&\n+\t\tsort reachable.raw >reachable.objects &&\n+\t\tawk \"/\\.pack$/ { print \\$1 }\" midx.objects &&\n+\n+\t\ttest_cmp reachable.objects midx.objects\n+\t)\n+'\n+\n+test_expect_success 'repack --write-midx includes cruft when instructed' '\n+\tsetup_cruft_exclude_tests exclude-cruft-when-instructed &&\n+\t(\n+\t\tcd exclude-cruft-when-instructed &&\n+\n+\t\tGIT_TEST_MULTI_PACK_INDEX=0 \\\n+\t\tgit -c repack.midxMustContainCruft=true repack \\\n+\t\t\t-d --geometric=2 --write-midx --write-bitmap-index &&\n+\n+\t\ttest-tool read-midx --show-objects $objdir >midx &&\n+\t\tcruft=\"$(ls $packdir/*.mtimes)\" &&\n+\t\ttest_grep \"$(basename \"$cruft\" .mtimes).idx\" midx &&\n+\n+\t\tgit cat-file --batch-check=\"%(objectname)\" --batch-all-objects \\\n+\t\t\t>all.objects &&\n+\t\tawk \"/\\.pack$/ { print \\$1 }\" midx.objects &&\n+\n+\t\ttest_cmp all.objects midx.objects\n+\t)\n+'\n+\n+test_expect_success 'repack --write-midx includes cruft when necessary' '\n+\tsetup_cruft_exclude_tests exclude-cruft-when-necessary &&\n+\t(\n+\t\tcd exclude-cruft-when-necessary &&\n+\n+\t\ttest_path_is_file $(ls $packdir/pack-*.mtimes) &&\n+\t\t( cd $packdir && ls pack-*.idx ) | sort >packs.all &&\n+\t\tgit multi-pack-index write --stdin-packs --bitmap midx &&\n+\t\tawk \"/\\.pack$/ { print \\$1 }\" midx.objects &&\n+\t\tgit cat-file --batch-all-objects --batch-check=\"%(objectname)\" \\\n+\t\t\t>expect.objects &&\n+\t\ttest_cmp expect.objects midx.objects &&\n+\n+\t\tgrep \"^pack-\" midx >midx.packs &&\n+\t\ttest_line_count = \"$(($(wc -l packs.all &&\n+\t\tcruft=\"$(ls $packdir/pack-*.mtimes)\" &&\n+\t\tcruft=\"${cruft%.mtimes}.idx\" &&\n+\n+\t\tfor idx in $(grep -v $cruft out &&\n+\t\t\twc -l sizes.raw &&\n+\n+\t\t# Make sure that there are two non-cruft packs, and\n+\t\t# that one of them contains at least twice as many\n+\t\t# objects as the other, ensuring that they are already\n+\t\t# in a geometric progression.\n+\t\tsort -n sizes.raw >sizes &&\n+\t\ttest_line_count = 2 sizes &&\n+\t\ts1=$(head -n 1 sizes) &&\n+\t\ts2=$(tail -n 1 sizes) &&\n+\t\ttest \"$s2\" -gt \"$((2 * $s1))\" &&\n+\n+\t\tgit -c repack.midxMustContainCruft=false repack --geometric=2 \\\n+\t\t\t--write-midx --write-bitmap-index\n+\t)\n+'\n+\n test_done\n-- \n2.50.0.61.g1981e40f2d\n"},{"id":"520601","messageId":"xmqqtt463wfl.fsf@gitster.g","threadId":"63286","inReplyTo":"86fb36d3176198fa350dfaed261e8ae64b49b355.1750717921.git.me@ttaylorr.com","subject":"Re: [PATCH v6 2/9] pack-objects: limit scope in 'add_object_entry_from_pack()'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-06-23T22:49:50Z","receivedAt":"2025-06-23T22:49:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> In add_object_entry_from_pack() we declare 'revs' (given to us through\n> the miscellaneous context argument) earlier in the \"if (p)\" conditional\n> than is necessary. Move it down as far as it can go to reduce its\n> scope.\n>\n> Signed-off-by: Taylor Blau \n> ---\n> builtin/pack-objects.c | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index e7274e0e00..d04a36a6bf 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -3725,7 +3725,6 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n> \t\treturn 0;\n> \n> \tif (p) {\n> -\t\tstruct rev_info *revs = _data;\n> \t\tstruct object_info oi = OBJECT_INFO_INIT;\n> \n> \t\toi.typep = &type;\n> @@ -3733,6 +3732,7 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n> \t\t\tdie(_(\"could not get type of object %s in pack %s\"),\n> \t\t\t oid_to_hex(oid), p->pack_name);\n> \t\t} else if (type == OBJ_COMMIT) {\n> +\t\t\tstruct rev_info *revs = _data;\n\nNice. This block is the only one that needs this variable. Makes sense.\n\n> \t\t\t/*\n> \t\t\t * commits in included packs are used as starting points for the\n> \t\t\t * subsequent revision walk\n"},{"id":"520602","messageId":"xmqqo6ue3vyy.fsf@gitster.g","threadId":"63286","inReplyTo":"c9f874eb9470bf2a5d97614b89304e892c30e129.1750717921.git.me@ttaylorr.com","subject":"Re: [PATCH v6 4/9] pack-objects: declare 'rev_info' for '--stdin-packs' earlier","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-06-23T22:59:49Z","receivedAt":"2025-06-23T22:59:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> Once 'read_packs_list_from_stdin()' has called for_each_object_in_pack()\n> on each of the input packs, we do a reachability traversal to discover\n> names for any objects we picked up so we can generate name hash values\n> and hopefully get higher quality deltas as a result.\n>\n> A future commit will change the purpose of this reachability traversal\n> to find and pack objects which are reachable from commits in the input\n> packs, but are packed in an unknown (not included nor excluded) pack.\n>\n> Extract the code which initializes and performs the reachability\n> traversal to take place in the caller, not the callee, which prepares us\n> to share this code for the '--unpacked' case (see the function\n> add_unreachable_loose_objects() for more details).\n>\n> Signed-off-by: Taylor Blau \n> ---\n> builtin/pack-objects.c | 71 +++++++++++++++++++++---------------------\n> 1 file changed, 36 insertions(+), 35 deletions(-)\n\nMakes sense. \n\nAnother forward declaration of add_unreachable_loose_objects(),\nafter one was already added in the previous step, confused me a bit,\nbut this step is merely moving that a bit higher, so there is\nnothing funny here. Looking good.\n\n"},{"id":"520604","messageId":"xmqqikkm3vki.fsf@gitster.g","threadId":"63286","inReplyTo":"6b0149a32d300268d4ad870c7cb6597a95e0410b.1750717921.git.me@ttaylorr.com","subject":"Re: [PATCH v6 5/9] pack-objects: perform name-hash traversal for unpacked objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-06-23T23:08:29Z","receivedAt":"2025-06-23T23:08:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> Now that the 'rev_info' struct is declared outside of\n> `read_packs_list_from_stdin()`, we can pass it to\n> `add_objects_in_unpacked_packs()` and add any loose objects as tips to\n> the above-mentioned traversal, in theory producing slightly tighter\n> packs as a result.\n\nSo the idea is to pretend any and all loose commits as if they are\nat the tip of branches? By doing so, we ensure each of the tree and\nblob objects contained in them has a reasonable path-from-the-root?\n\n> @@ -4325,6 +4326,10 @@ static int add_loose_object(const struct object_id *oid, const char *path,\n> \t} else {\n> \t\tadd_object_entry(oid, type, \"\", 0);\n> \t}\n> +\n> +\tif (revs && type == OBJ_COMMIT)\n> +\t\tadd_pending_oid(revs, NULL, oid, 0);\n> +\n> \treturn 0;\n> }\n\nOK.\n"},{"id":"520614","messageId":"xmqq8qli3ub3.fsf@gitster.g","threadId":"63286","inReplyTo":"3699c25337e1c91388bad4c56441b39a9984798b.1750717921.git.me@ttaylorr.com","subject":"Re: [PATCH v6 8/9] pack-objects: introduce '--stdin-packs=follow'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-06-23T23:35:44Z","receivedAt":"2025-06-23T23:35:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> static void show_object_pack_hint(struct object *object, const char *name,\n> -\t\t\t\t void *data UNUSED)\n> +\t\t\t\t void *data)\n> {\n> -\tstruct object_entry *oe = packlist_find(&to_pack, &object->oid);\n> -\tif (!oe)\n> -\t\treturn;\n> +\tenum stdin_packs_mode mode = *(enum stdin_packs_mode *)data;\n> +\tif (mode == STDIN_PACKS_MODE_FOLLOW) {\n> +\t\tif (object->type == OBJ_BLOB &&\n> +\t\t !has_object(the_repository, &object->oid, 0))\n> +\t\t\treturn;\n\nSo, --stdin-packs opened a pack and is feeding the objects contained\nin it to this machinery. show_commit_pack_hint() calls this\nfunction in the `follow` mode. How would such an object be missing?\nAh, lazy clones. OK.\n\n> +\t\tadd_object_entry(&object->oid, object->type, name, 0);\n> +\t} else {\n\nAnd only up to this point is the new code. The \"else\" clause is\njust the original indented one-level deeper.\n\n> +static void show_commit_pack_hint(struct commit *commit, void *data)\n> {\n> +\tenum stdin_packs_mode mode = *(enum stdin_packs_mode *)data;\n> +\n> +\tif (mode == STDIN_PACKS_MODE_FOLLOW) {\n> +\t\tshow_object_pack_hint((struct object *)commit, \"\", data);\n> +\t\treturn;\n> +\t}\n> +\n> \t/* nothing to do; commits don't have a namehash */\n> +\n> }\n\nWhat is this new blank line doing here?\n\n\n"},{"id":"520634","messageId":"20250624105447.GA716990@coredump.intra.peff.net","threadId":"63286","inReplyTo":"aFmhQZZnYyvxDelO@nand.local","subject":"Re: [PATCH v5 9/9] repack: exclude cruft pack(s) from the MIDX where possible","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-06-24T10:54:47Z","receivedAt":"2025-06-24T10:54:48Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jun 23, 2025 at 02:47:29PM -0400, Taylor Blau wrote:\n\n> > This test (but none of the others) fails when run with:\n> >\n> > GIT_TEST_MULTI_PACK_INDEX=1 \\\n> > GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=1 \\\n> > ./t7704-repack-cruft.sh\n> >\n> > The culprit is the incremental flag, but you need the first one for the\n> > second to do anything. The issue is that the cruft pack unexpectedly\n> > appears in the midx:\n> >\n> > [...]\n> >\n> > I'm not sure if it's just a funky interaction with the hacky GIT_TEST_*\n> > variables, or if it's a real bug.\n> \n> Thanks for spotting. This is definitely a real bug. The root cause here\n> is that our loop to gather the set of packs we know are in the MIDX does\n> not account for multi-layered / incremental MIDXs.\n> \n> In our example, if there's a cruft pack in any other layer of a MIDX\n> besides the tip, the proposed implementation here won't realize it, and\n> thus (incorrectly) conclude that the cruft pack is not in the MIDX\n> already, so can thusly be omitted.\n\nAh, right, that makes perfect sense.\n\n> If we do this on top:\n> \n> --- 8< ---\n> diff --git a/builtin/repack.c b/builtin/repack.c\n> index 346d44fbcd..8d1540a0fd 100644\n> --- a/builtin/repack.c\n> +++ b/builtin/repack.c\n> @@ -1614,13 +1614,16 @@ int cmd_repack(int argc,\n> \tstring_list_sort(&names);\n> \n> \tif (get_local_multi_pack_index(the_repository)) {\n> -\t\tuint32_t i;\n> \t\tstruct multi_pack_index *m =\n> \t\t\tget_local_multi_pack_index(the_repository);\n> \n> -\t\tALLOC_ARRAY(midx_pack_names, m->num_packs);\n> -\t\tfor (i = 0; i < m->num_packs; i++)\n> -\t\t\tmidx_pack_names[midx_pack_names_nr++] = xstrdup(m->pack_names[i]);\n> +\t\tALLOC_ARRAY(midx_pack_names,\n> +\t\t\t m->num_packs + m->num_packs_in_base);\n> +\n> +\t\tfor (; m; m = m->base_midx)\n> +\t\t\tfor (uint32_t i = 0; i < m->num_packs; i++)\n> +\t\t\t\tmidx_pack_names[midx_pack_names_nr++] =\n> +\t\t\t\t\txstrdup(m->pack_names[i]);\n> \t}\n> \n> \tclose_object_store(the_repository->objects);\n> --- >8 ---\n\nAnd this fix looks reasonable to me. It is a bit unfortunate that the\nincremental midx concept bleeds all the way out to callers like this,\nbecause it means we might have the same problem in other spots. But that\nis nothing new, and I'm not sure of a good solution. If the\npublic-facing API pretended as if \"struct multi_pack_midx\" contained the\npacks for all of the sub-midx entries of the chain, that would solve it.\nBut then all of the internal parts of the code that look at the\nincremental entries would need a separate representation. And I suspect\nthere's a lot more code in that latter group than the former (most\ncallers won't be this intimate with the midx, and just want to convert\nan oid to a pack/offset pair).\n\nWould we want a test to cover this case? We do catch it in the\nlinux-TEST-vars build, but it might be nice to have coverage in normal\ntest runs. I'm not sure how much of a pain that would be.\n\n-Peff\n"},{"id":"520651","messageId":"xmqqldph16hw.fsf@gitster.g","threadId":"63286","inReplyTo":"8e7b2dacc77623ada5ee938de7a610ae15f49d8e.1750717921.git.me@ttaylorr.com","subject":"Re: [PATCH v6 1/9] pack-objects: use standard option incompatibility functions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-06-24T15:52:59Z","receivedAt":"2025-06-24T15:53:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> @@ -5050,13 +5051,14 @@ int cmd_pack_objects(int argc,\n> \tif (!pack_to_stdout && thin)\n> \t\tdie(_(\"--thin cannot be used to build an indexable pack\"));\n> \n> -\tif (keep_unreachable && unpack_unreachable)\n> -\t\tdie(_(\"options '%s' and '%s' cannot be used together\"), \"--keep-unreachable\", \"--unpack-unreachable\");\n> +\tdie_for_incompatible_opt2(keep_unreachable, \"--keep-unreachable\",\n> +\t\t\t\t unpack_unreachable, \"--unpack-unreachable\");\n> \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n> \t\tunpack_unreachable_expiration = 0;\n> \n> -\tif (stdin_packs && filter_options.choice)\n> -\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n> +\tdie_for_incompatible_opt2(stdin_packs, \"--stdin-packs\",\n> +\t\t\t\t filter_options.choice, \"--filter\");\n> +\n> \n\nWe do not need two blank lines here, do we?\n\n> @@ -5064,8 +5066,8 @@ int cmd_pack_objects(int argc,\n> \tif (cruft) {\n> \t\tif (use_internal_rev_list)\n> \t\t\tdie(_(\"cannot use internal rev list with --cruft\"));\n> -\t\tif (stdin_packs)\n> -\t\t\tdie(_(\"cannot use --stdin-packs with --cruft\"));\n> +\t\tdie_for_incompatible_opt2(stdin_packs, \"--stdin-packs\",\n> +\t\t\t\t\t cruft, \"--cruft\");\n> \t}\n> \n> \t/*\n> diff --git a/t/t5331-pack-objects-stdin.sh b/t/t5331-pack-objects-stdin.sh\n> index b48c0cbe8f..8fd07deb8d 100755\n> --- a/t/t5331-pack-objects-stdin.sh\n> +++ b/t/t5331-pack-objects-stdin.sh\n> @@ -64,7 +64,7 @@ test_expect_success '--stdin-packs is incompatible with --filter' '\n> \t\tcd stdin-packs &&\n> \t\ttest_must_fail git pack-objects --stdin-packs --stdout \\\n> \t\t\t--filter=blob:none err &&\n> -\t\ttest_grep \"cannot use --filter with --stdin-packs\" err\n> +\t\ttest_grep \"options .--stdin-packs. and .--filter. cannot be used together\" err\n\nOK.\n"},{"id":"520653","messageId":"aFrM1ngJfL0Ssc9F@nand.local","threadId":"63286","inReplyTo":"20250624105447.GA716990@coredump.intra.peff.net","subject":"Re: [PATCH v5 9/9] repack: exclude cruft pack(s) from the MIDX where possible","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-06-24T16:05:42Z","receivedAt":"2025-06-24T16:05:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jun 24, 2025 at 06:54:47AM -0400, Jeff King wrote:\n> > If we do this on top:\n> >\n> > --- 8< ---\n> > diff --git a/builtin/repack.c b/builtin/repack.c\n> > index 346d44fbcd..8d1540a0fd 100644\n> > --- a/builtin/repack.c\n> > +++ b/builtin/repack.c\n> > @@ -1614,13 +1614,16 @@ int cmd_repack(int argc,\n> > \tstring_list_sort(&names);\n> >\n> > \tif (get_local_multi_pack_index(the_repository)) {\n> > -\t\tuint32_t i;\n> > \t\tstruct multi_pack_index *m =\n> > \t\t\tget_local_multi_pack_index(the_repository);\n> >\n> > -\t\tALLOC_ARRAY(midx_pack_names, m->num_packs);\n> > -\t\tfor (i = 0; i < m->num_packs; i++)\n> > -\t\t\tmidx_pack_names[midx_pack_names_nr++] = xstrdup(m->pack_names[i]);\n> > +\t\tALLOC_ARRAY(midx_pack_names,\n> > +\t\t\t m->num_packs + m->num_packs_in_base);\n> > +\n> > +\t\tfor (; m; m = m->base_midx)\n> > +\t\t\tfor (uint32_t i = 0; i < m->num_packs; i++)\n> > +\t\t\t\tmidx_pack_names[midx_pack_names_nr++] =\n> > +\t\t\t\t\txstrdup(m->pack_names[i]);\n> > \t}\n> >\n> > \tclose_object_store(the_repository->objects);\n> > --- >8 ---\n>\n> And this fix looks reasonable to me. It is a bit unfortunate that the\n> incremental midx concept bleeds all the way out to callers like this,\n> because it means we might have the same problem in other spots. But that\n> is nothing new, and I'm not sure of a good solution. If the\n> public-facing API pretended as if \"struct multi_pack_midx\" contained the\n> packs for all of the sub-midx entries of the chain, that would solve it.\n> But then all of the internal parts of the code that look at the\n> incremental entries would need a separate representation. And I suspect\n> there's a lot more code in that latter group than the former (most\n> callers won't be this intimate with the midx, and just want to convert\n> an oid to a pack/offset pair).\n>\n> Would we want a test to cover this case? We do catch it in the\n> linux-TEST-vars build, but it might be nice to have coverage in normal\n> test runs. I'm not sure how much of a pain that would be.\n\nI thought quite a bit about this and decided against it. The extra test\nwould really just be this on top:\n\n--- 8< ---\ndiff --git a/t/t7704-repack-cruft.sh b/t/t7704-repack-cruft.sh\nindex aa2e2e6ad8..9b71387325 100755\n--- a/t/t7704-repack-cruft.sh\n+++ b/t/t7704-repack-cruft.sh\n@@ -842,7 +842,9 @@ test_expect_success 'repack --write-midx includes cruft when already geometric'\n \t\t# actually write a new object and subsequently a new\n \t\t# pack to contain it.\n \t\tgit merge --no-ff $C &&\n-\t\tgit repack -d &&\n+\t\tGIT_TEST_MULTI_PACK_INDEX=1 \\\n+\t\tGIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=1 \\\n+\t\t\tgit repack -d &&\n\n \t\tls $packdir/pack-*.idx | sort >packs.all &&\n \t\tcruft=\"$(ls $packdir/pack-*.mtimes)\" &&\n--- >8 ---\n\n, to force us to put the cruft pack in an earlier MIDX layer. But that\nfelt like making this test too-specific to incremental MIDXs when the\noriginal test has very little to do with incremental- vs non-incremental\nMIDXs.\n\nI tried to write a smaller test case that demonstrates the problem but\ncouldn't find a straightforward way to minimize the reproduction. As an\nalternative, we could duplicate and/or parameterize the test entirely,\nbut that felt like overkill.\n\nThanks,\nTaylor\n"},{"id":"520655","messageId":"aFrNG8hmuktduz/4@nand.local","threadId":"63286","inReplyTo":"xmqqldph16hw.fsf@gitster.g","subject":"Re: [PATCH v6 1/9] pack-objects: use standard option incompatibility functions","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-06-24T16:06:51Z","receivedAt":"2025-06-24T16:06:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jun 24, 2025 at 08:52:59AM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > @@ -5050,13 +5051,14 @@ int cmd_pack_objects(int argc,\n> > \tif (!pack_to_stdout && thin)\n> > \t\tdie(_(\"--thin cannot be used to build an indexable pack\"));\n> >\n> > -\tif (keep_unreachable && unpack_unreachable)\n> > -\t\tdie(_(\"options '%s' and '%s' cannot be used together\"), \"--keep-unreachable\", \"--unpack-unreachable\");\n> > +\tdie_for_incompatible_opt2(keep_unreachable, \"--keep-unreachable\",\n> > +\t\t\t\t unpack_unreachable, \"--unpack-unreachable\");\n> > \tif (!rev_list_all || !rev_list_reflog || !rev_list_index)\n> > \t\tunpack_unreachable_expiration = 0;\n> >\n> > -\tif (stdin_packs && filter_options.choice)\n> > -\t\tdie(_(\"cannot use --filter with --stdin-packs\"));\n> > +\tdie_for_incompatible_opt2(stdin_packs, \"--stdin-packs\",\n> > +\t\t\t\t filter_options.choice, \"--filter\");\n> > +\n> >\n>\n> We do not need two blank lines here, do we?\n\nYeah, it looks like an extra one snuck in and I missed it when\nproof-reading.\n\nThanks,\nTaylor\n"},{"id":"520656","messageId":"aFrNauEk9vRq27+t@nand.local","threadId":"63286","inReplyTo":"xmqqikkm3vki.fsf@gitster.g","subject":"Re: [PATCH v6 5/9] pack-objects: perform name-hash traversal for unpacked objects","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-06-24T16:08:10Z","receivedAt":"2025-06-24T16:08:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jun 23, 2025 at 04:08:29PM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > Now that the 'rev_info' struct is declared outside of\n> > `read_packs_list_from_stdin()`, we can pass it to\n> > `add_objects_in_unpacked_packs()` and add any loose objects as tips to\n> > the above-mentioned traversal, in theory producing slightly tighter\n> > packs as a result.\n>\n> So the idea is to pretend any and all loose commits as if they are\n> at the tip of branches? By doing so, we ensure each of the tree and\n> blob objects contained in them has a reasonable path-from-the-root?\n\nThat's right. We had previously only considered commit objects in the\npack(s) being combined as possible traversal tips, but this change\ncauses us to do the same for loose commit objects as well.\n\nI do kind of wonder how much of a difference this makes on delta quality\noverall, and suspect that it is highly workflow-specific and likely very\ndifficult to measure in general.\n\nThanks,\nTaylor\n"},{"id":"520657","messageId":"aFrN+r2CdjAcO3DH@nand.local","threadId":"63286","inReplyTo":"xmqq8qli3ub3.fsf@gitster.g","subject":"Re: [PATCH v6 8/9] pack-objects: introduce '--stdin-packs=follow'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-06-24T16:10:34Z","receivedAt":"2025-06-24T16:10:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jun 23, 2025 at 04:35:44PM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > static void show_object_pack_hint(struct object *object, const char *name,\n> > -\t\t\t\t void *data UNUSED)\n> > +\t\t\t\t void *data)\n> > {\n> > -\tstruct object_entry *oe = packlist_find(&to_pack, &object->oid);\n> > -\tif (!oe)\n> > -\t\treturn;\n> > +\tenum stdin_packs_mode mode = *(enum stdin_packs_mode *)data;\n> > +\tif (mode == STDIN_PACKS_MODE_FOLLOW) {\n> > +\t\tif (object->type == OBJ_BLOB &&\n> > +\t\t !has_object(the_repository, &object->oid, 0))\n> > +\t\t\treturn;\n>\n> So, --stdin-packs opened a pack and is feeding the objects contained\n> in it to this machinery. show_commit_pack_hint() calls this\n> function in the `follow` mode. How would such an object be missing?\n> Ah, lazy clones. OK.\n\nThat is one such place, but another would be that the object is part of\nsome unreachable portion of the repository and points at another\nunreachable object that is missing. We take care to accommodate those\nholes in the unreachable object graph when generating cruft packs, and\nAFAIK in general tolerate broken links and/or missing objects provided\nthey are unreachable.\n\n> > +\t\tadd_object_entry(&object->oid, object->type, name, 0);\n> > +\t} else {\n>\n> And only up to this point is the new code. The \"else\" clause is\n> just the original indented one-level deeper.\n\nRight.\n\n> > +static void show_commit_pack_hint(struct commit *commit, void *data)\n> > {\n> > +\tenum stdin_packs_mode mode = *(enum stdin_packs_mode *)data;\n> > +\n> > +\tif (mode == STDIN_PACKS_MODE_FOLLOW) {\n> > +\t\tshow_object_pack_hint((struct object *)commit, \"\", data);\n> > +\t\treturn;\n> > +\t}\n> > +\n> > \t/* nothing to do; commits don't have a namehash */\n> > +\n> > }\n>\n> What is this new blank line doing here?\n\nWeird, this one evaded my proof-reading as well. Sorry about that -- I\ncan send a new round with this and the other spot fixed up if you want\none.\n\nThanks,\nTaylor\n"}]}