git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] fixup! midx: implement writing incremental MIDX bitmaps

From
Taylor Blau <me@ttaylorr.com>
Date
Aug 29, 2024, 18:57 UTC
Message-ID
<ZtDEhNRfXth63SJs@nand.local>
In-Reply-To
<20240828183356.GA4043247@coredump.intra.peff.net>
On Wed, Aug 28, 2024 at 02:33:56PM -0400, Jeff King wrote:
Show 10 quoted lines
> Is that right, though? It looks like the caller might pass in a
> tempfile name like .../pack/multi-pack-index.d/tmp_midx_XXXXXX,
> if we're in incremental mode. But we'll write directly to
> "multi-pack-index-$hash.bitmap" in the same directory. I'm not sure to
> what degree it matters, since that's the name we want in the long run.
> But would we possibly overwrite an active-in-use file rather than doing
> the atomic rename-into-place if we happened to generate the same midx?
>
> It feels like we should still respect the name the caller is using for
> tempfiles, and then rename it into the correct spot at the end.

In either case, we're going to write to a temporary file initialized by the pack-bitmap machinery and then rename() it into place at the end of bitmap_writer_finish().

On the caller side, in the non-incremental mode, we'll pass $GIT_DIR/objects/pack/multi-pack-index-$hash.bitmap as the location, write its contents into a temporary file, and then rename() it there.

But in the incremental mode this series introduces, I think it would be a bug to pass a tmp_midx_XXXXXX file path there, since nobody would move it from tmp_midx_XXXXX-$HASH.bitmap into its final location.

So I think what's written here with the fixup! patch is right (and should be squashed into 13/13 in the next round), but let me know if I'm missing something.

Thanks, Taylor

Previous: Jeff KingNext: Jeff King
Message 31 of 135 in “midx: incremental multi-pack indexes, part two”
  1. 00/13 midx: incremental multi-pack indexes, part twoTaylor Blau, Aug 15, 2024
  2. 01/13 Documentation: describe incremental MIDX bitmapsTaylor Blau, Aug 15, 2024
  3. 02/13 pack-revindex: prepare for incremental MIDX bitmapsTaylor Blau, Aug 15, 2024
  4. 03/13 pack-bitmap.c: open and store incremental bitmap layersTaylor Blau, Aug 15, 2024
  5. 04/13 pack-bitmap.c: teach `bitmap_for_commit()` about incremental MIDXsTaylor Blau, Aug 15, 2024
  6. 05/13 pack-bitmap.c: teach `show_objects_for_type()` about incremental MIDXsTaylor Blau, Aug 15, 2024
  7. 06/13 pack-bitmap.c: support bitmap pack-reuse with incremental MIDXsTaylor Blau, Aug 15, 2024
  8. 07/13 pack-bitmap.c: teach `rev-list --test-bitmap` about incremental MIDXsTaylor Blau, Aug 15, 2024
  9. 08/13 pack-bitmap.c: compute disk-usage with incremental MIDXsTaylor Blau, Aug 15, 2024
  10. 09/13 pack-bitmap.c: apply pseudo-merge commits with incremental MIDXsTaylor Blau, Aug 15, 2024
  11. 10/13 ewah: implement `struct ewah_or_iterator`Taylor Blau, Aug 15, 2024
  12. 11/13 pack-bitmap.c: keep track of each layer's type bitmapsTaylor Blau, Aug 15, 2024
  13. 12/13 pack-bitmap.c: use `ewah_or_iterator` for type bitmap iteratorsTaylor Blau, Aug 15, 2024
  14. 13/13 midx: implement writing incremental MIDX bitmapsTaylor Blau, Aug 15, 2024
  15. 00/13 midx: incremental multi-pack indexes, part twoTaylor Blau, Aug 15, 2024
  16. 01/13 Documentation: describe incremental MIDX bitmapsTaylor Blau, Aug 15, 2024
  17. 02/13 pack-revindex: prepare for incremental MIDX bitmapsTaylor Blau, Aug 15, 2024
  18. 03/13 pack-bitmap.c: open and store incremental bitmap layersTaylor Blau, Aug 15, 2024
  19. 04/13 pack-bitmap.c: teach `bitmap_for_commit()` about incremental MIDXsTaylor Blau, Aug 15, 2024
  20. 05/13 pack-bitmap.c: teach `show_objects_for_type()` about incremental MIDXsTaylor Blau, Aug 15, 2024
  21. 06/13 pack-bitmap.c: support bitmap pack-reuse with incremental MIDXsTaylor Blau, Aug 15, 2024
  22. 07/13 pack-bitmap.c: teach `rev-list --test-bitmap` about incremental MIDXsTaylor Blau, Aug 15, 2024
  23. 08/13 pack-bitmap.c: compute disk-usage with incremental MIDXsTaylor Blau, Aug 15, 2024
  24. 09/13 pack-bitmap.c: apply pseudo-merge commits with incremental MIDXsTaylor Blau, Aug 15, 2024
  25. 10/13 ewah: implement `struct ewah_or_iterator`Taylor Blau, Aug 15, 2024
  26. 11/13 pack-bitmap.c: keep track of each layer's type bitmapsTaylor Blau, Aug 15, 2024
  27. 12/13 pack-bitmap.c: use `ewah_or_iterator` for type bitmap iteratorsTaylor Blau, Aug 15, 2024
  28. 13/13 midx: implement writing incremental MIDX bitmapsTaylor Blau, Aug 15, 2024
  29. fixup! midx: implement writing incremental MIDX bitmapsJunio C Hamano, Aug 28, 2024
  30. Jeff KingAug 28, 2024
  31. Taylor BlauAug 29, 2024
  32. Jeff KingAug 29, 2024
  33. Taylor BlauNov 19, 2024
  34. 00/13 midx: incremental multi-pack indexes, part twoTaylor Blau, Nov 19, 2024
  35. 01/13 Documentation: describe incremental MIDX bitmapsTaylor Blau, Nov 19, 2024
  36. Patrick SteinhardtFeb 28, 2025
  37. Taylor BlauFeb 28, 2025
  38. Patrick SteinhardtMar 3, 2025
  39. 02/13 pack-revindex: prepare for incremental MIDX bitmapsTaylor Blau, Nov 19, 2024
  40. Patrick SteinhardtFeb 28, 2025
  41. Taylor BlauFeb 28, 2025
  42. 03/13 pack-bitmap.c: open and store incremental bitmap layersTaylor Blau, Nov 19, 2024
  43. Patrick SteinhardtFeb 28, 2025
  44. Taylor BlauFeb 28, 2025
  45. Patrick SteinhardtMar 3, 2025
  46. 04/13 pack-bitmap.c: teach `bitmap_for_commit()` about incremental MIDXsTaylor Blau, Nov 19, 2024
  47. Patrick SteinhardtFeb 28, 2025
  48. Taylor BlauMar 1, 2025
  49. 05/13 pack-bitmap.c: teach `show_objects_for_type()` about incremental MIDXsTaylor Blau, Nov 19, 2024
  50. 06/13 pack-bitmap.c: support bitmap pack-reuse with incremental MIDXsTaylor Blau, Nov 19, 2024
  51. Patrick SteinhardtFeb 28, 2025
  52. Taylor BlauMar 1, 2025
  53. 07/13 pack-bitmap.c: teach `rev-list --test-bitmap` about incremental MIDXsTaylor Blau, Nov 19, 2024
  54. Patrick SteinhardtFeb 28, 2025
  55. Taylor BlauMar 1, 2025
  56. 08/13 pack-bitmap.c: compute disk-usage with incremental MIDXsTaylor Blau, Nov 19, 2024
  57. 09/13 pack-bitmap.c: apply pseudo-merge commits with incremental MIDXsTaylor Blau, Nov 19, 2024
  58. 10/13 ewah: implement `struct ewah_or_iterator`Taylor Blau, Nov 19, 2024
  59. Patrick SteinhardtFeb 28, 2025
  60. Taylor BlauMar 1, 2025
  61. 11/13 pack-bitmap.c: keep track of each layer's type bitmapsTaylor Blau, Nov 19, 2024
  62. Patrick SteinhardtFeb 28, 2025
  63. Taylor BlauMar 1, 2025
  64. 12/13 pack-bitmap.c: use `ewah_or_iterator` for type bitmap iteratorsTaylor Blau, Nov 19, 2024
  65. Patrick SteinhardtFeb 28, 2025
  66. Taylor BlauMar 1, 2025
  67. 13/13 midx: implement writing incremental MIDX bitmapsTaylor Blau, Nov 19, 2024
  68. Patrick SteinhardtFeb 28, 2025
  69. Taylor BlauMar 1, 2025
  70. Junio C HamanoNov 20, 2024
  71. 00/13 midx: incremental multi-pack indexes, part twoTaylor Blau, Mar 14, 2025
  72. 01/13 Documentation: describe incremental MIDX bitmapsTaylor Blau, Mar 14, 2025
  73. Jeff KingMar 18, 2025
  74. Taylor BlauMar 18, 2025
  75. Elijah NewrenMar 18, 2025
  76. Taylor BlauMar 18, 2025
  77. 02/13 pack-revindex: prepare for incremental MIDX bitmapsTaylor Blau, Mar 14, 2025
  78. Jeff KingMar 18, 2025
  79. Taylor BlauMar 19, 2025
  80. Taylor BlauMar 19, 2025
  81. Jeff KingMar 26, 2025
  82. Elijah NewrenMar 18, 2025
  83. Taylor BlauMar 19, 2025
  84. 03/13 pack-bitmap.c: open and store incremental bitmap layersTaylor Blau, Mar 14, 2025
  85. Elijah NewrenMar 18, 2025
  86. Taylor BlauMar 19, 2025
  87. 04/13 pack-bitmap.c: teach `bitmap_for_commit()` about incremental MIDXsTaylor Blau, Mar 14, 2025
  88. Jeff KingMar 18, 2025
  89. Taylor BlauMar 19, 2025
  90. 05/13 pack-bitmap.c: teach `show_objects_for_type()` about incremental MIDXsTaylor Blau, Mar 14, 2025
  91. 06/13 pack-bitmap.c: support bitmap pack-reuse with incremental MIDXsTaylor Blau, Mar 14, 2025
  92. Elijah NewrenMar 18, 2025
  93. Taylor BlauMar 19, 2025
  94. 07/13 pack-bitmap.c: teach `rev-list --test-bitmap` about incremental MIDXsTaylor Blau, Mar 14, 2025
  95. Elijah NewrenMar 18, 2025
  96. Taylor BlauMar 19, 2025
  97. 08/13 pack-bitmap.c: compute disk-usage with incremental MIDXsTaylor Blau, Mar 14, 2025
  98. Jeff KingMar 18, 2025
  99. Taylor BlauMar 19, 2025
  100. 09/13 pack-bitmap.c: apply pseudo-merge commits with incremental MIDXsTaylor Blau, Mar 14, 2025
  101. 10/13 ewah: implement `struct ewah_or_iterator`Taylor Blau, Mar 14, 2025
  102. Jeff KingMar 18, 2025
  103. Taylor BlauMar 19, 2025
  104. 11/13 pack-bitmap.c: keep track of each layer's type bitmapsTaylor Blau, Mar 14, 2025
  105. Jeff KingMar 18, 2025
  106. Taylor BlauMar 19, 2025
  107. Elijah NewrenMar 18, 2025
  108. Taylor BlauMar 19, 2025
  109. 12/13 pack-bitmap.c: use `ewah_or_iterator` for type bitmap iteratorsTaylor Blau, Mar 14, 2025
  110. Jeff KingMar 18, 2025
  111. Taylor BlauMar 19, 2025
  112. 13/13 midx: implement writing incremental MIDX bitmapsTaylor Blau, Mar 14, 2025
  113. Jeff KingMar 18, 2025
  114. Taylor BlauMar 20, 2025
  115. Elijah NewrenMar 18, 2025
  116. Taylor BlauMar 20, 2025
  117. Jeff KingMar 18, 2025
  118. Taylor BlauMar 20, 2025
  119. 00/14 midx: incremental multi-pack indexes, part twoTaylor Blau, Mar 20, 2025
  120. 01/14 Documentation: remove a "future work" item from the MIDX docsTaylor Blau, Mar 20, 2025
  121. 02/14 Documentation: describe incremental MIDX bitmapsTaylor Blau, Mar 20, 2025
  122. 03/14 pack-revindex: prepare for incremental MIDX bitmapsTaylor Blau, Mar 20, 2025
  123. 04/14 pack-bitmap.c: open and store incremental bitmap layersTaylor Blau, Mar 20, 2025
  124. 05/14 pack-bitmap.c: teach `bitmap_for_commit()` about incremental MIDXsTaylor Blau, Mar 20, 2025
  125. 06/14 pack-bitmap.c: teach `show_objects_for_type()` about incremental MIDXsTaylor Blau, Mar 20, 2025
  126. 07/14 pack-bitmap.c: support bitmap pack-reuse with incremental MIDXsTaylor Blau, Mar 20, 2025
  127. 08/14 pack-bitmap.c: teach `rev-list --test-bitmap` about incremental MIDXsTaylor Blau, Mar 20, 2025
  128. 09/14 pack-bitmap.c: compute disk-usage with incremental MIDXsTaylor Blau, Mar 20, 2025
  129. 10/14 pack-bitmap.c: apply pseudo-merge commits with incremental MIDXsTaylor Blau, Mar 20, 2025
  130. 11/14 ewah: implement `struct ewah_or_iterator`Taylor Blau, Mar 20, 2025
  131. 12/14 pack-bitmap.c: keep track of each layer's type bitmapsTaylor Blau, Mar 20, 2025
  132. 13/14 pack-bitmap.c: use `ewah_or_iterator` for type bitmap iteratorsTaylor Blau, Mar 20, 2025
  133. 14/14 midx: implement writing incremental MIDX bitmapsTaylor Blau, Mar 20, 2025
  134. Elijah NewrenMar 20, 2025
  135. Taylor BlauMar 20, 2025

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.