git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[RFC PATCH 09/10] pack-objects: support '--refs-snapshot' with 'follow-reachable'

From
Taylor Blau <me@ttaylorr.com>
Date
Jun 26, 2026, 19:02 UTC
Message-ID
<c9efbad60c773f0def850ecf107ddfaeb2860a16.1782500507.git.me@ttaylorr.com>
In-Reply-To
<cover.1782500507.git.me@ttaylorr.com>

The '--stdin-packs=follow-reachable' mode walks from reference tips to determine which objects in included packs are reachable. Without a snapshot, pack-objects discovers refs by iterating live references, which may change between the time the repack writes the geometric pack and the time it writes the MIDX bitmap.

If a reference is updated during that window, the set of reachable objects seen by pack-objects may differ from the set seen by the MIDX bitmap writer. This can cause reachable objects to end up in the cruft pack (because pack-objects did not see the reference that makes them reachable) rather than the geometric pack. While this does not cause data loss, it has two undesirable consequences:

 - Reachable objects in the cruft pack cannot receive bitmap coverage
   (since the cruft pack may be excluded from the MIDX when
   'repack.midxMustContainCruft' is false).
 - Serving fetches that need those objects requires loading the cruft
   pack, which may contain many unrelated unreachable objects.

To avoid this, teach pack-objects to accept '--refs-snapshot=<path>' when used with '--stdin-packs=follow-reachable'. The snapshot file uses the same format as the MIDX bitmap writer: one hex OID per line, with an optional '+' prefix for preferred bitmap commits.

'pack-objects' happily ignores the '+' prefix for indicating preferred bitmap commits as a convenience, so that the ref-snapshot can be shared between the MIDX generation machinery and 'pack-objects'.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 Documentation/git-pack-objects.adoc |  8 +++++
 builtin/pack-objects.c              | 46 +++++++++++++++++++++++++++--
 2 files changed, 52 insertions(+), 2 deletions(-)
diff --git a/Documentation/git-pack-objects.adoc b/Documentation/git-pack-objects.adoc
index d7b2e39e76c..4ebe407cfaf 100644
--- a/Documentation/git-pack-objects.adoc
+++ b/Documentation/git-pack-objects.adoc
@@ -133,6 +133,14 @@ commits and annotated tag objects.
 Incompatible with `--revs`, or options that imply `--revs` (such as
 `--all`), with the exception of `--unpacked`, which is compatible.
 
+--refs-snapshot=<path>::
+	When used with `--stdin-packs=follow-reachable`, read reference
+	tips from `<path>` instead of iterating live references. The file
+	format is one hex object ID per line, with an optional `+` prefix
+	(for preferred bitmap commits). This ensures a consistent view of
+	references when the same snapshot is shared with other tools (e.g.,
+	the MIDX bitmap writer).
+
 --cruft::
 	Packs unreachable objects into a separate "cruft" pack, denoted
 	by the existence of a `.mtimes` file. Typically used by `git
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 5d96757b645..082ff760abc 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -219,6 +219,7 @@ static int incremental;
 static int ignore_packed_keep_on_disk;
 static int ignore_packed_keep_in_core;
 static int ignore_packed_keep_in_core_open;
+static const char *stdin_packs_refs_snapshot;
 static int ignore_packed_keep_in_core_has_cruft;
 static int allow_ofs_delta;
 static struct pack_idx_option pack_idx_opts;
@@ -4009,6 +4010,38 @@ static int add_ref_to_pending(const struct reference *ref, void *cb_data)
 	return 0;
 }
 
+static void read_refs_snapshot(const char *refs_snapshot,
+			      struct rev_info *revs)
+{
+	struct strbuf buf = STRBUF_INIT;
+	struct object_id oid;
+	FILE *f = xfopen(refs_snapshot, "r");
+
+	while (strbuf_getline(&buf, f) != EOF) {
+		struct object *object;
+		const char *hex = buf.buf;
+		const char *end = NULL;
+
+		if (*hex == '+')
+			hex++;
+
+		if (parse_oid_hex_algop(hex, &oid, &end,
+					the_repository->hash_algo) < 0)
+			die(_("could not parse line: %s"), buf.buf);
+		if (*end)
+			die(_("malformed line: %s"), buf.buf);
+
+		object = parse_object(the_repository, &oid);
+		if (!object)
+			continue;
+
+		add_pending_object(revs, object, "");
+	}
+
+	fclose(f);
+	strbuf_release(&buf);
+}
+
 static void stdin_packs_add_reachable_pack_entries(struct string_list *keys,
 						   struct rev_info *revs,
 						   int rev_list_unpacked)
@@ -4065,8 +4098,11 @@ static void stdin_packs_add_reachable_pack_entries(struct string_list *keys,
 	pre_walk.keep_pack_cache_flags |= KEPT_PACK_IN_CORE;
 	pre_walk.ignore_missing_links = 1;
 
-	refs_for_each_ref(get_main_ref_store(the_repository),
-			  add_ref_to_pending, &pre_walk);
+	if (stdin_packs_refs_snapshot)
+		read_refs_snapshot(stdin_packs_refs_snapshot, &pre_walk);
+	else
+		refs_for_each_ref(get_main_ref_store(the_repository),
+				  add_ref_to_pending, &pre_walk);
 
 	if (prepare_revision_walk(&pre_walk))
 		die(_("revision walk setup failed"));
@@ -5267,6 +5303,8 @@ int cmd_pack_objects(int argc,
 		OPT_CALLBACK_F(0, "stdin-packs", &stdin_packs, N_("mode"),
 			     N_("read packs from stdin"),
 			     PARSE_OPT_OPTARG, parse_stdin_packs_mode),
+		OPT_FILENAME(0, "refs-snapshot", &stdin_packs_refs_snapshot,
+			     N_("refs snapshot for follow-reachable traversal")),
 		OPT_BOOL(0, "stdout", &pack_to_stdout,
 			 N_("output pack to stdout")),
 		OPT_BOOL(0, "include-tag", &include_tag,
@@ -5484,6 +5522,10 @@ int cmd_pack_objects(int argc,
 	if (stdin_packs && use_internal_rev_list)
 		die(_("cannot use internal rev list with --stdin-packs"));
 
+	if (stdin_packs_refs_snapshot &&
+	    stdin_packs != STDIN_PACKS_MODE_FOLLOW_REACHABLE)
+		die(_("--refs-snapshot can only be used with --stdin-packs=follow-reachable"));
+
 	if (cruft) {
 		if (use_internal_rev_list)
 			die(_("cannot use internal rev list with --cruft"));
-- 
2.55.0.rc2.10.g29e31820dce
Previous: Taylor BlauNext: Taylor Blau
Message 16 of 18 in “repack: combine '--geometric' and '--cruft'”
  1. 00/10 repack: combine '--geometric' and '--cruft'Taylor Blau, Jun 26, 2026
  2. 01/10 repack: unconditionally exclude non-kept packsTaylor Blau, Jun 26, 2026
  3. 03/10 repack: mark geometric progression of packs as retainedTaylor Blau, Jun 26, 2026
  4. 04/10 repack: teach MIDX retention about geometric rollupsTaylor Blau, Jun 26, 2026
  5. Junio C HamanoJun 26, 2026
  6. Taylor BlauJun 27, 2026
  7. 05/10 repack: delete geometric packs via existing_packsTaylor Blau, Jun 26, 2026
  8. 06/10 repack-geometry: drop unused redundant-pack removalTaylor Blau, Jun 26, 2026
  9. 02/10 repack: extract `locate_existing_pack()` helperTaylor Blau, Jun 26, 2026
  10. 07/10 pack-objects: extract `stdin_packs_add_all_pack_entries()`Taylor Blau, Jun 26, 2026
  11. 08/10 pack-objects: introduce '--stdin-packs=follow-reachable'Taylor Blau, Jun 26, 2026
  12. Junio C HamanoJun 26, 2026
  13. Taylor BlauJun 27, 2026
  14. Junio C HamanoJun 27, 2026
  15. Taylor BlauJun 27, 2026
  16. 09/10 pack-objects: support '--refs-snapshot' with 'follow-reachable'Taylor Blau, Jun 26, 2026
  17. 10/10 repack: support combining '--geometric' with '--cruft'Taylor Blau, Jun 26, 2026
  18. Junio C HamanoJul 27, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.