git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 7/8] pack-objects: introduce '--stdin-packs=follow'

From
Taylor Blau <me@ttaylorr.com>
Date
Apr 15, 2025, 20:45 UTC
Message-ID
<Z/7FdgYID9I1qR7K@nand.local>
In-Reply-To
<CABPp-BFBJP15g=4M90161=KCDei-hEFdnGs7_oY8ERtqgn9s-g@mail.gmail.com>
On Mon, Apr 14, 2025 at 08:11:08PM -0700, Elijah Newren wrote:
Show 31 quoted lines
> > diff --git a/Documentation/git-pack-objects.adoc b/Documentation/git-pack-objects.adoc
> > index 7f69ae4855..c894582799 100644
> > --- a/Documentation/git-pack-objects.adoc
> > +++ b/Documentation/git-pack-objects.adoc
> > @@ -87,13 +87,19 @@ base-name::
> >         reference was included in the resulting packfile.  This
> >         can be useful to send new tags to native Git clients.
> >
> > ---stdin-packs::
> > +--stdin-packs[=<mode>]::
> >         Read the basenames of packfiles (e.g., `pack-1234abcd.pack`)
> >         from the standard input, instead of object names or revision
> >         arguments. The resulting pack contains all objects listed in the
> >         included packs (those not beginning with `^`), excluding any
> >         objects listed in the excluded packs (beginning with `^`).
> >  +
> > +When `mode` is "follow", pack objects which are reachable from objects
> > +in the included packs, but appear in packs that are not listed.
> > +Reachable objects which appear in excluded packs are not packed. Useful
> > +for resurrecting once-cruft objects to generate packs which are closed
> > +under reachability up to the excluded packs.
>
> Maybe:
>
> When `mode` is "follow", objects from packs not listed on stdin
> receive special treatment.  Objects within unlisted packs will be
> included if those objects (1) are reachable from the included packs,
> and (2) are not also found in any of the excluded packs.  This mode is
> useful for resurrecting once-cruft objects to generate packs which are
> closed under reachability up to the boundary set by the excluded
> packs.

I like it. I went with your version with some minor rewording and tweaks on top.

Show 13 quoted lines
> > +               /*
> > +                * Our 'to_pack' list was constructed by iterating all
> > +                * objects packed in included packs, and so doesn't
> > +                * have a non-zero hash field that you would typically
> > +                * pick up during a reachability traversal.
> > +                *
> > +                * Make a best-effort attempt to fill in the ->hash
> > +                * and ->no_try_delta here using a now in order to
> > +                * perhaps improve the delta selection process.
> > +                */
>
> I know you just moved this paragraph from below...but it doesn't parse
> for me.  "using a now in order to perhaps"?  What does that mean?

Yeah, this is just bogus, and was so before this patch. The rewording is minor enough (just dropping "using a now") that I think we can just squash it in with the movement in this patch.

Show 37 quoted lines
> > +               oe->hash = pack_name_hash_fn(name);
> > +               oe->no_try_delta = name && no_try_delta(name);
> > +
> > +               stdin_packs_hints_nr++;
> > +       }
> > +}
> > +
> > +static void show_commit_pack_hint(struct commit *commit, void *data)
> > +{
> > +       enum stdin_packs_mode mode = *(enum stdin_packs_mode *)data;
> > +       if (mode == STDIN_PACKS_MODE_FOLLOW) {
> > +               show_object_pack_hint((struct object *)commit, "", data);
> >                 return;
> > +       }
> > +       /* nothing to do; commits don't have a namehash */
> >
> > -       /*
> > -        * Our 'to_pack' list was constructed by iterating all objects packed in
> > -        * included packs, and so doesn't have a non-zero hash field that you
> > -        * would typically pick up during a reachability traversal.
> > -        *
> > -        * Make a best-effort attempt to fill in the ->hash and ->no_try_delta
> > -        * here using a now in order to perhaps improve the delta selection
> > -        * process.
> > -        */
> > -       oe->hash = pack_name_hash_fn(name);
> > -       oe->no_try_delta = name && no_try_delta(name);
> > -
> > -       stdin_packs_hints_nr++;
> >  }
>
> It might be worth swapping the order of functions as a preparatory
> patch (both here and when you've done it elsewhere in this series),
> just because it'll make the diff so much easier to read when we can
> see the changes to the function without have to also deal with the
> order swapping (since order swapping looks like a large deletion and
> large addition of one of the two functions).
Fair enough.
Show 18 quoted lines
> > @@ -4467,6 +4484,23 @@ static int is_not_in_promisor_pack(struct commit *commit, void *data) {
> >         return is_not_in_promisor_pack_obj((struct object *) commit, data);
> >  }
> >
> > +static int parse_stdin_packs_mode(const struct option *opt, const char *arg,
> > +                                 int unset)
> > +{
> > +       enum stdin_packs_mode *mode = opt->value;
> > +
> > +       if (unset)
> > +               *mode = STDIN_PACKS_MODE_NONE;
> > +       else if (!arg || !*arg)
> > +               *mode = STDIN_PACKS_MODE_STANDARD;
>
> I don't understand why you have both a None mode and a Standard mode,
> especially since the implementation seems to only care about whether
> or not the Follow mode has been set.  Shouldn't these both be setting
> mode to the same value?

I'm not sure I follow your question... stdin_packs is a tri-state. It can be off, on in standard/legacy mode, or on in follow mode.

Show 18 quoted lines
> > +test_expect_success 'setup for --stdin-packs=follow' '
> > +       git init stdin-packs--follow &&
> > +       (
> > +               cd stdin-packs--follow &&
> > +
> > +               for c in A B C D
> > +               do
> > +                       test_commit "$c" || return 1
> > +               done &&
> > +
> > +               A="$(echo A | git pack-objects --revs $packdir/pack)" &&
> > +               B="$(echo A..B | git pack-objects --revs $packdir/pack)" &&
> > +               C="$(echo B..C | git pack-objects --revs $packdir/pack)" &&
> > +               D="$(echo C..D | git pack-objects --revs $packdir/pack)" &&
> > +
> > +               git prune-packed
> > +       )
> > +'

Huh, I have no idea how this snuck in. This "setup" test does nothing and creates a repository that isn't used later on in the script. Probably leftover from writing these tests in the first place, but I've removed it.

> I like the tests -- normal --stdin-packs, then --stdin-packs=follow,
> then --stdin-packs=follow + --unpacked.

I think the normal tests are accidental since we use pack-objects to write packs A, B, C, and D. But the --stdin-packs vs. --stdin-packs=follow and --stdin-packs=follow + --unpacked was definitely intentional.

> However, would it be worthwhile to create commit E immediately after
> creating the packs?

Yeah, I think that is a good suggestion. We already have tests that exercise --stdin-packs with --unpacked earlier in the same script, but obviously not with --stdin-packs=follow. Moving the creation of commit E earlier up makes a lot of sense to me, thanks!

Thanks, Taylor

Previous: Elijah NewrenNext: Elijah Newren
Message 30 of 105 in “repack: avoid MIDX'ing cruft pack(s) where possible”
  1. 0/8 repack: avoid MIDX'ing cruft pack(s) where possibleTaylor Blau, Apr 11, 2025
  2. 1/8 pack-objects: use standard option incompatibility functionsTaylor Blau, Apr 11, 2025
  3. 2/8 pack-objects: limit scope in 'add_object_entry_from_pack()'Taylor Blau, Apr 11, 2025
  4. 3/8 pack-objects: factor out handling '--stdin-packs'Taylor Blau, Apr 11, 2025
  5. 4/8 pack-objects: declare 'rev_info' for '--stdin-packs' earlierTaylor Blau, Apr 11, 2025
  6. 5/8 pack-objects: perform name-hash traversal for unpacked objectsTaylor Blau, Apr 11, 2025
  7. 6/8 pack-objects: introduce '--stdin-packs=follow'Taylor Blau, Apr 11, 2025
  8. 7/8 repack: keep track of existing MIDX'd packsTaylor Blau, Apr 11, 2025
  9. 8/8 repack: exclude cruft pack(s) from the MIDX where possibleTaylor Blau, Apr 11, 2025
  10. 0/8 repack: avoid MIDX'ing cruft pack(s) where possibleTaylor Blau, Apr 14, 2025
  11. 1/8 pack-objects: use standard option incompatibility functionsTaylor Blau, Apr 14, 2025
  12. Junio C HamanoApr 14, 2025
  13. Taylor BlauApr 15, 2025
  14. Junio C HamanoApr 15, 2025
  15. Taylor BlauApr 15, 2025
  16. 2/8 object-store-ll.h: add note about designated initializersTaylor Blau, Apr 14, 2025
  17. Junio C HamanoApr 14, 2025
  18. Taylor BlauApr 15, 2025
  19. Elijah NewrenApr 15, 2025
  20. Taylor BlauApr 15, 2025
  21. 3/8 pack-objects: limit scope in 'add_object_entry_from_pack()'Taylor Blau, Apr 14, 2025
  22. Elijah NewrenApr 15, 2025
  23. 4/8 pack-objects: factor out handling '--stdin-packs'Taylor Blau, Apr 14, 2025
  24. 5/8 pack-objects: declare 'rev_info' for '--stdin-packs' earlierTaylor Blau, Apr 14, 2025
  25. 6/8 pack-objects: perform name-hash traversal for unpacked objectsTaylor Blau, Apr 14, 2025
  26. Elijah NewrenApr 15, 2025
  27. Taylor BlauApr 15, 2025
  28. 7/8 pack-objects: introduce '--stdin-packs=follow'Taylor Blau, Apr 14, 2025
  29. Elijah NewrenApr 15, 2025
  30. Taylor BlauApr 15, 2025
  31. Elijah NewrenApr 16, 2025
  32. 8/8 repack: exclude cruft pack(s) from the MIDX where possibleTaylor Blau, Apr 14, 2025
  33. Elijah NewrenApr 15, 2025
  34. Taylor BlauApr 15, 2025
  35. Elijah NewrenApr 15, 2025
  36. Taylor BlauApr 15, 2025
  37. 0/9 repack: avoid MIDX'ing cruft pack(s) where possibleTaylor Blau, Apr 15, 2025
  38. 1/9 pack-objects: use standard option incompatibility functionsTaylor Blau, Apr 15, 2025
  39. 2/9 pack-objects: limit scope in 'add_object_entry_from_pack()'Taylor Blau, Apr 15, 2025
  40. Junio C HamanoApr 16, 2025
  41. Taylor BlauApr 16, 2025
  42. Elijah NewrenApr 16, 2025
  43. Taylor BlauApr 16, 2025
  44. 3/9 pack-objects: factor out handling '--stdin-packs'Taylor Blau, Apr 15, 2025
  45. Junio C HamanoApr 16, 2025
  46. 4/9 pack-objects: declare 'rev_info' for '--stdin-packs' earlierTaylor Blau, Apr 15, 2025
  47. 5/9 pack-objects: perform name-hash traversal for unpacked objectsTaylor Blau, Apr 15, 2025
  48. Junio C HamanoApr 16, 2025
  49. 6/9 pack-objects: fix typo in 'show_object_pack_hint()'Taylor Blau, Apr 15, 2025
  50. Elijah NewrenApr 16, 2025
  51. 7/9 pack-objects: swap 'show_{object,commit}_pack_hint'Taylor Blau, Apr 15, 2025
  52. 8/9 pack-objects: introduce '--stdin-packs=follow'Taylor Blau, Apr 15, 2025
  53. 9/9 repack: exclude cruft pack(s) from the MIDX where possibleTaylor Blau, Apr 15, 2025
  54. Elijah NewrenApr 16, 2025
  55. Taylor BlauApr 16, 2025
  56. Elijah NewrenMay 13, 2025
  57. 0/9 repack: avoid MIDX'ing cruft pack(s) where possibleTaylor Blau, May 28, 2025
  58. 1/9 pack-objects: use standard option incompatibility functionsTaylor Blau, May 28, 2025
  59. 2/9 pack-objects: limit scope in 'add_object_entry_from_pack()'Taylor Blau, May 28, 2025
  60. 3/9 pack-objects: factor out handling '--stdin-packs'Taylor Blau, May 28, 2025
  61. 4/9 pack-objects: declare 'rev_info' for '--stdin-packs' earlierTaylor Blau, May 28, 2025
  62. 5/9 pack-objects: perform name-hash traversal for unpacked objectsTaylor Blau, May 28, 2025
  63. 6/9 pack-objects: fix typo in 'show_object_pack_hint()'Taylor Blau, May 28, 2025
  64. 7/9 pack-objects: swap 'show_{object,commit}_pack_hint'Taylor Blau, May 28, 2025
  65. 8/9 pack-objects: introduce '--stdin-packs=follow'Taylor Blau, May 28, 2025
  66. 9/9 repack: exclude cruft pack(s) from the MIDX where possibleTaylor Blau, May 28, 2025
  67. Carlo Marcelo Arenas BelónJun 19, 2025
  68. fixup! repack: exclude cruft pack(s) from the MIDX where possibleCarlo Marcelo Arenas Belón, Jun 19, 2025
  69. Junio C HamanoJun 19, 2025
  70. Taylor BlauJun 19, 2025
  71. Taylor BlauMay 29, 2025
  72. Elijah NewrenMay 29, 2025
  73. 0/9 repack: avoid MIDX'ing cruft pack(s) where possibleTaylor Blau, Jun 19, 2025
  74. 1/9 pack-objects: use standard option incompatibility functionsTaylor Blau, Jun 19, 2025
  75. 2/9 pack-objects: limit scope in 'add_object_entry_from_pack()'Taylor Blau, Jun 19, 2025
  76. 3/9 pack-objects: factor out handling '--stdin-packs'Taylor Blau, Jun 19, 2025
  77. 4/9 pack-objects: declare 'rev_info' for '--stdin-packs' earlierTaylor Blau, Jun 19, 2025
  78. 5/9 pack-objects: perform name-hash traversal for unpacked objectsTaylor Blau, Jun 19, 2025
  79. 6/9 pack-objects: fix typo in 'show_object_pack_hint()'Taylor Blau, Jun 19, 2025
  80. 7/9 pack-objects: swap 'show_{object,commit}_pack_hint'Taylor Blau, Jun 19, 2025
  81. 8/9 pack-objects: introduce '--stdin-packs=follow'Taylor Blau, Jun 19, 2025
  82. Junio C HamanoJun 20, 2025
  83. 9/9 repack: exclude cruft pack(s) from the MIDX where possibleTaylor Blau, Jun 19, 2025
  84. Jeff KingJun 21, 2025
  85. Taylor BlauJun 23, 2025
  86. Jeff KingJun 24, 2025
  87. Taylor BlauJun 24, 2025
  88. 0/9 repack: avoid MIDX'ing cruft pack(s) where possibleTaylor Blau, Jun 23, 2025
  89. 1/9 pack-objects: use standard option incompatibility functionsTaylor Blau, Jun 23, 2025
  90. Junio C HamanoJun 24, 2025
  91. Taylor BlauJun 24, 2025
  92. 2/9 pack-objects: limit scope in 'add_object_entry_from_pack()'Taylor Blau, Jun 23, 2025
  93. Junio C HamanoJun 23, 2025
  94. 3/9 pack-objects: factor out handling '--stdin-packs'Taylor Blau, Jun 23, 2025
  95. 4/9 pack-objects: declare 'rev_info' for '--stdin-packs' earlierTaylor Blau, Jun 23, 2025
  96. Junio C HamanoJun 23, 2025
  97. 5/9 pack-objects: perform name-hash traversal for unpacked objectsTaylor Blau, Jun 23, 2025
  98. Junio C HamanoJun 23, 2025
  99. Taylor BlauJun 24, 2025
  100. 6/9 pack-objects: fix typo in 'show_object_pack_hint()'Taylor Blau, Jun 23, 2025
  101. 7/9 pack-objects: swap 'show_{object,commit}_pack_hint'Taylor Blau, Jun 23, 2025
  102. 8/9 pack-objects: introduce '--stdin-packs=follow'Taylor Blau, Jun 23, 2025
  103. Junio C HamanoJun 23, 2025
  104. Taylor BlauJun 24, 2025
  105. 9/9 repack: exclude cruft pack(s) from the MIDX where possibleTaylor Blau, Jun 23, 2025

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.