git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 0/5] pack-objects: handle excluded-but-open packs via `--stdin-packs=follow`

From
Taylor Blau <me@ttaylorr.com>
Date
Mar 19, 2026, 22:24 UTC
Message-ID
<cover.1773959041.git.me@ttaylorr.com>

This series came from an issue I saw in GitHub's infrastructure where a particular repository was failing to repack with the following in its log output:

    warning: Failed to write bitmap index. Packfile doesn't have full closure (object XYZ is missing)

This was following a geometric repack that generated a MIDX and attempted to generate its corresponding MIDX bitmap. Ordinarily, we should never expect the above when generating MIDX bitmaps, unless:

 - Object XYZ is indeed missing from the repository (though we should
   have seen an earlier failure during the MIDX write itself), or
 - Object XYZ is somehow not included in the MIDX, but is reachable from
   another object which is.

After validating that object XYZ was indeed present at the time the repack failed, I tried to figure out why we would ever generate a MIDX whose set of objects was not closed under reachability.

The precise details are laid out in the fourth patch, but the gist is that:

 1. A pack whose objects are *not* closed under reachability was deemed
    large enough by the geometric repack machinery so as to not need a
    repack.
 2. When generating the new pack with the non-closed pack was marked as
    excluded, some object in an included pack reached an object (XYZ) in
    an unknown pack whose only reachability path involved walking
    through the parents of an object in another excluded pack. That
 3. We wrote a MIDX containing the new pack, along with all of the
    large-enough packs from the previous step
 4. Because XYZ was in an unknown (likely cruft) pack, the resulting
    MIDX does not contain a copy of object XYZ, but does contain a copy
    of an object which reaches it, making it impossible to write
    bitmaps.

This series introduces a special denotation for packs which are excluded but not guaranteed to be closed under reachability. By marking a pack as excluded via '!' (as opposed to the traditional '^'), we will now pick up copies of objects reachable from objects in those pack(s), but exclude objects which appear in excluded packs (open or closed).

In practice this means that the resulting pack contains:
 1. All objects in the set difference between the included packs vs. the
    excluded ones (open or closed).
 2. All objects reachable from at least one object in either an included
    pack or an excluded-open pack which (a) do not appear in an excluded
    pack (of either kind), and (b) has a reachability path that does not
    involve objects in the excluded-closed packs.

The series contains a couple of minor fixups and some refactoring that I did along the way to make the substantive changes easier to read. The first commit is a cleanup, the second is a refactoring, and the remaining patches demonstrate, explain, and fix the bug.

(As a general side-note, I am somewhat unhappy with the growing number of ways to mark a pack as "kept", and tried to untangle this by letting packs hold an arbitrary bitset of flags that is opaque to the object traversal machinery. This ended up being doable but rather complicated, so I ended up punting on it for now.)

Thanks in advance for your review!
Taylor Blau (5):
  pack-objects: plug leak in `read_stdin_packs()`
  pack-objects: refactor `read_packs_list_from_stdin()` to use `strmap`
  t7704: demonstrate failure with once-cruft objects above the geometric
    split
  pack-objects: support excluded-open packs with --stdin-packs
  repack: mark non-MIDX packs above the split as excluded-open
 Documentation/git-pack-objects.adoc |  25 ++-
 builtin/pack-objects.c              | 234 ++++++++++++++++++++--------
 builtin/repack.c                    |  19 ++-
 packfile.c                          |   3 +-
 packfile.h                          |   2 +
 t/t5331-pack-objects-stdin.sh       | 105 +++++++++++++
 t/t7704-repack-cruft.sh             |  22 +++
 7 files changed, 332 insertions(+), 78 deletions(-)
base-commit: 7ff1e8dc1e1680510c96e69965b3fa81372c5037
-- 
2.53.0.614.gc4fd52e751a
Next: Taylor Blau
Message 1 of 41 in “pack-objects: handle excluded-but-open packs via `--stdin-packs=follow`”
  1. 0/5 pack-objects: handle excluded-but-open packs via `--stdin-packs=follow`Taylor Blau, Mar 19, 2026
  2. 1/5 pack-objects: plug leak in `read_stdin_packs()`Taylor Blau, Mar 19, 2026
  3. Patrick SteinhardtMar 24, 2026
  4. Taylor BlauMar 25, 2026
  5. 2/5 pack-objects: refactor `read_packs_list_from_stdin()` to use `strmap`Taylor Blau, Mar 19, 2026
  6. Patrick SteinhardtMar 24, 2026
  7. Taylor BlauMar 25, 2026
  8. 3/5 t7704: demonstrate failure with once-cruft objects above the geometric splitTaylor Blau, Mar 19, 2026
  9. 4/5 pack-objects: support excluded-open packs with --stdin-packsTaylor Blau, Mar 19, 2026
  10. Jeff KingMar 21, 2026
  11. Taylor BlauMar 22, 2026
  12. Taylor BlauMar 25, 2026
  13. 5/5 repack: mark non-MIDX packs above the split as excluded-openTaylor Blau, Mar 19, 2026
  14. 0/5 pack-objects: handle excluded-but-open packs via `--stdin-packs=follow`Taylor Blau, Mar 25, 2026
  15. 1/5 pack-objects: plug leak in `read_stdin_packs()`Taylor Blau, Mar 25, 2026
  16. 2/5 pack-objects: refactor `read_packs_list_from_stdin()` to use `strmap`Taylor Blau, Mar 25, 2026
  17. Derrick StoleeMar 26, 2026
  18. Taylor BlauMar 26, 2026
  19. Junio C HamanoMar 26, 2026
  20. Taylor BlauMar 26, 2026
  21. Derrick StoleeMar 27, 2026
  22. Taylor BlauMar 27, 2026
  23. Derrick StoleeMar 27, 2026
  24. Junio C HamanoMar 27, 2026
  25. Taylor BlauMar 26, 2026
  26. 3/5 t7704: demonstrate failure with once-cruft objects above the geometric splitTaylor Blau, Mar 25, 2026
  27. 4/5 pack-objects: support excluded-open packs with --stdin-packsTaylor Blau, Mar 25, 2026
  28. Derrick StoleeMar 26, 2026
  29. 5/5 repack: mark non-MIDX packs above the split as excluded-openTaylor Blau, Mar 25, 2026
  30. Derrick StoleeMar 26, 2026
  31. Taylor BlauMar 26, 2026
  32. Derrick StoleeMar 26, 2026
  33. Taylor BlauMar 26, 2026
  34. 0/5 pack-objects: handle excluded-but-open packs via `--stdin-packs=follow`Taylor Blau, Mar 27, 2026
  35. 1/5 pack-objects: plug leak in `read_stdin_packs()`Taylor Blau, Mar 27, 2026
  36. 2/5 pack-objects: refactor `read_packs_list_from_stdin()` to use `strmap`Taylor Blau, Mar 27, 2026
  37. 3/5 t7704: demonstrate failure with once-cruft objects above the geometric splitTaylor Blau, Mar 27, 2026
  38. 4/5 pack-objects: support excluded-open packs with --stdin-packsTaylor Blau, Mar 27, 2026
  39. 5/5 repack: mark non-MIDX packs above the split as excluded-openTaylor Blau, Mar 27, 2026
  40. Derrick StoleeMar 27, 2026
  41. Junio C HamanoMar 27, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.