git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 00/24] pack-objects: multi-pack verbatim reuse

From
Patrick Steinhardt <ps@pks.im>
Date
Dec 1, 2023, 08:31 UTC
Message-ID
<ZWmZ0pk6fr-8dmF1@tanuki>
In-Reply-To
<ZWjk/TYzsrdHefgy@nand.local>
On Thu, Nov 30, 2023 at 02:39:41PM -0500, Taylor Blau wrote:
> On Thu, Nov 30, 2023 at 11:18:19AM +0100, Patrick Steinhardt wrote:
[snip]
Show 13 quoted lines
> Then you'd have five patch series, where each series does roughly the
> following:
> 
>   1. Preparatory clean-up.
>   2. Implementing the DISP chunk, as well as --retain-disjoint, without
>      a way to generate such packs.
>   3. Implement a way to generate such packs, but without actually being
>      able to reuse more than one of them.
>   4. Implement multi-pack reuse, but without actually reusing any packs.
>   5. Enable multi-pack reuse (and implement the required scaffolding to
>      do so), and test it.
> 
> That's the most sensible split that I could come up with, at least.
Looks sensible to me.
Show 7 quoted lines
> But
> I still find it relatively unsatisfying for a couple of reasons:
> 
>   - With the exception of the last group of patches, none of the earlier
>     series enable any new, useful behavior on their own. IOW, if we just
>     merged the first three series and then forgot about this topic, we
>     wouldn't have done anything useful ;-).

Well, sometimes I wish we'd buy more into the iterative style of working in the Git project, where it's fine to land patch series that only work into the direction of a specific topic but don't yet do anything interesting by themselves. The benefits would be both that individual pieces land faster while also ensuring that the review load is kept at bay.

But there's of course also downsides to this, especially in an open source project like Git:

  - Contributors may go away in the middle of their endeavour, leaving
    behind dangling pieces that might have complicated some of our
    architecture without actually reaping the intended benefits.
  - Overall, it may take significantly longer to get all pieces into
    Git.
  - Contributors need to do a much better job to explain where they are
    headed when the series by itself doesn't do anything interesting
    yet.
So I understand why we typically don't work this way.

I wonder whether a compromise would be to continue sending complete patch series, but explicitly point out "break points" in your patch series. These break points could be an indicator to the maintainer that it's fine to merge everything up to it while still keeping out the remainder of the patch series.

Show 5 quoted lines
>   - The fourth series (which actually implements multi-pack reuse) still
>     remains the most complicated, and would likely be the most difficult
>     to review. So I think you'd still have one difficult series to
>     review, plus four other series which are slightly less difficult to
>     review ;-).

That's fine in my opinion, there's no surprise here that a complicated topic is demanding for both the author and the reviewer.

Patrick
Previous: Taylor BlauNext: Taylor Blau
Message 68 of 107 in “pack-objects: multi-pack verbatim reuse”
  1. 00/24 pack-objects: multi-pack verbatim reuseTaylor Blau, Nov 28, 2023
  2. 01/24 pack-objects: free packing_data in more placesTaylor Blau, Nov 28, 2023
  3. Patrick SteinhardtNov 30, 2023
  4. Taylor BlauNov 30, 2023
  5. 02/24 pack-bitmap-write: deep-clear the `bb_commit` slabTaylor Blau, Nov 28, 2023
  6. Patrick SteinhardtNov 30, 2023
  7. Taylor BlauNov 30, 2023
  8. Jeff KingDec 12, 2023
  9. Taylor BlauDec 12, 2023
  10. 03/24 pack-bitmap: plug leak in find_objects()Taylor Blau, Nov 28, 2023
  11. Jeff KingDec 12, 2023
  12. 04/24 midx: factor out `fill_pack_info()`Taylor Blau, Nov 28, 2023
  13. Patrick SteinhardtNov 30, 2023
  14. Taylor BlauNov 30, 2023
  15. 05/24 midx: implement `DISP` chunkTaylor Blau, Nov 28, 2023
  16. Patrick SteinhardtNov 30, 2023
  17. Taylor BlauNov 30, 2023
  18. Junio C HamanoDec 3, 2023
  19. Taylor BlauDec 5, 2023
  20. Junio C HamanoDec 9, 2023
  21. Taylor BlauDec 9, 2023
  22. Jeff KingDec 12, 2023
  23. Taylor BlauDec 13, 2023
  24. Junio C HamanoDec 13, 2023
  25. 06/24 midx: implement `midx_locate_pack()`Taylor Blau, Nov 28, 2023
  26. 07/24 midx: implement `--retain-disjoint` modeTaylor Blau, Nov 28, 2023
  27. Patrick SteinhardtNov 30, 2023
  28. Taylor BlauNov 30, 2023
  29. Patrick SteinhardtDec 1, 2023
  30. 08/24 pack-objects: implement `--ignore-disjoint` modeTaylor Blau, Nov 28, 2023
  31. Patrick SteinhardtNov 30, 2023
  32. Taylor BlauNov 30, 2023
  33. Patrick SteinhardtDec 1, 2023
  34. Taylor BlauDec 1, 2023
  35. 09/24 repack: implement `--extend-disjoint` modeTaylor Blau, Nov 28, 2023
  36. Patrick SteinhardtDec 7, 2023
  37. Taylor BlauDec 7, 2023
  38. Patrick SteinhardtDec 8, 2023
  39. Taylor BlauDec 8, 2023
  40. Patrick SteinhardtDec 11, 2023
  41. Taylor BlauDec 11, 2023
  42. 10/24 pack-bitmap: pass `bitmapped_pack` struct to pack-reuse functionsTaylor Blau, Nov 28, 2023
  43. Patrick SteinhardtDec 7, 2023
  44. Taylor BlauDec 7, 2023
  45. Patrick SteinhardtDec 8, 2023
  46. 11/24 pack-bitmap: simplify `reuse_partial_packfile_from_bitmap()` signatureTaylor Blau, Nov 28, 2023
  47. Patrick SteinhardtDec 7, 2023
  48. Taylor BlauDec 7, 2023
  49. 12/24 pack-bitmap: return multiple packs via `reuse_partial_packfile_from_bitmap()`Taylor Blau, Nov 28, 2023
  50. 13/24 pack-objects: parameterize pack-reuse routines over a single packTaylor Blau, Nov 28, 2023
  51. 14/24 pack-objects: keep track of `pack_start` for each reuse packTaylor Blau, Nov 28, 2023
  52. Patrick SteinhardtDec 7, 2023
  53. Taylor BlauDec 7, 2023
  54. 15/24 pack-objects: pass `bitmapped_pack`'s to pack-reuse functionsTaylor Blau, Nov 28, 2023
  55. 16/24 pack-objects: prepare `write_reused_pack()` for multi-pack reuseTaylor Blau, Nov 28, 2023
  56. Patrick SteinhardtDec 7, 2023
  57. Taylor BlauDec 7, 2023
  58. 17/24 pack-objects: prepare `write_reused_pack_verbatim()` for multi-pack reuseTaylor Blau, Nov 28, 2023
  59. 18/24 pack-objects: include number of packs reused in outputTaylor Blau, Nov 28, 2023
  60. 19/24 pack-bitmap: prepare to mark objects from multiple packs for reuseTaylor Blau, Nov 28, 2023
  61. 20/24 pack-objects: add tracing for various packfile metricsTaylor Blau, Nov 28, 2023
  62. 21/24 t/test-lib-functions.sh: implement `test_trace2_data` helperTaylor Blau, Nov 28, 2023
  63. 22/24 pack-objects: allow setting `pack.allowPackReuse` to "single"Taylor Blau, Nov 28, 2023
  64. 23/24 pack-bitmap: reuse objects from all disjoint packsTaylor Blau, Nov 28, 2023
  65. 24/24 t/perf: add performance tests for multi-pack reuseTaylor Blau, Nov 28, 2023
  66. Patrick SteinhardtNov 30, 2023
  67. Taylor BlauNov 30, 2023
  68. Patrick SteinhardtDec 1, 2023
  69. Taylor BlauDec 1, 2023
  70. Patrick SteinhardtDec 4, 2023
  71. Jeff KingDec 12, 2023
  72. Taylor BlauDec 15, 2023
  73. Jeff KingDec 21, 2023
  74. Taylor BlauJan 4, 2024
  75. 00/26 pack-objects: multi-pack verbatim reuseTaylor Blau, Dec 14, 2023
  76. 01/26 pack-objects: free packing_data in more placesTaylor Blau, Dec 14, 2023
  77. 02/26 pack-bitmap-write: deep-clear the `bb_commit` slabTaylor Blau, Dec 14, 2023
  78. 03/26 pack-bitmap: plug leak in find_objects()Taylor Blau, Dec 14, 2023
  79. 04/26 midx: factor out `fill_pack_info()`Taylor Blau, Dec 14, 2023
  80. 05/26 midx: implement `BTMP` chunkTaylor Blau, Dec 14, 2023
  81. 06/26 midx: implement `midx_locate_pack()`Taylor Blau, Dec 14, 2023
  82. 07/26 pack-bitmap: pass `bitmapped_pack` struct to pack-reuse functionsTaylor Blau, Dec 14, 2023
  83. 08/26 ewah: implement `bitmap_is_empty()`Taylor Blau, Dec 14, 2023
  84. 09/26 pack-bitmap: simplify `reuse_partial_packfile_from_bitmap()` signatureTaylor Blau, Dec 14, 2023
  85. 10/26 pack-bitmap: return multiple packs via `reuse_partial_packfile_from_bitmap()`Taylor Blau, Dec 14, 2023
  86. 11/26 pack-objects: parameterize pack-reuse routines over a single packTaylor Blau, Dec 14, 2023
  87. 12/26 pack-objects: keep track of `pack_start` for each reuse packTaylor Blau, Dec 14, 2023
  88. 13/26 pack-objects: pass `bitmapped_pack`'s to pack-reuse functionsTaylor Blau, Dec 14, 2023
  89. 14/26 pack-objects: prepare `write_reused_pack()` for multi-pack reuseTaylor Blau, Dec 14, 2023
  90. 15/26 pack-objects: prepare `write_reused_pack_verbatim()` for multi-pack reuseTaylor Blau, Dec 14, 2023
  91. 16/26 pack-objects: include number of packs reused in outputTaylor Blau, Dec 14, 2023
  92. 17/26 git-compat-util.h: implement checked size_t to uint32_t conversionTaylor Blau, Dec 14, 2023
  93. 18/26 midx: implement `midx_preferred_pack()`Taylor Blau, Dec 14, 2023
  94. 19/26 pack-revindex: factor out `midx_key_to_pack_pos()` helperTaylor Blau, Dec 14, 2023
  95. 20/26 pack-revindex: implement `midx_pair_to_pack_pos()`Taylor Blau, Dec 14, 2023
  96. 21/26 pack-bitmap: prepare to mark objects from multiple packs for reuseTaylor Blau, Dec 14, 2023
  97. 22/26 pack-objects: add tracing for various packfile metricsTaylor Blau, Dec 14, 2023
  98. 23/26 t/test-lib-functions.sh: implement `test_trace2_data` helperTaylor Blau, Dec 14, 2023
  99. 24/26 pack-objects: allow setting `pack.allowPackReuse` to "single"Taylor Blau, Dec 14, 2023
  100. 25/26 pack-bitmap: enable reuse from all bitmapped packsTaylor Blau, Dec 14, 2023
  101. 26/26 t/perf: add performance tests for multi-pack reuseTaylor Blau, Dec 14, 2023
  102. Junio C HamanoDec 15, 2023
  103. Taylor BlauDec 15, 2023
  104. Junio C HamanoDec 15, 2023
  105. Taylor BlauDec 15, 2023
  106. Jeff KingDec 21, 2023
  107. Taylor BlauJan 4, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.