git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 0/8] midx: prevent bitmap corruption when permuting pack order

From
Derrick Stolee <stolee@gmail.com>
Date
Dec 20, 2021, 20:09 UTC
Message-ID
<c729fbc5-9f59-27d3-5f1b-0b339446174e@gmail.com>
In-Reply-To
<YcDe9wl4XwH7PjX1@nand.local>
On 12/20/2021 2:52 PM, Taylor Blau wrote:
> On Mon, Dec 20, 2021 at 01:51:22PM -0500, Derrick Stolee wrote:
Show 20 quoted lines
>> 3. Should the chunk be made mandatory?
>>
>> Unfortunately, the chunk format did not follow the index format's
>> example of making lowercase chunk IDs required. Instead, the chunks
>> that are necessary for v1 are necessary forever and all other chunks
>> are deemed optional. Changing this would require something more
>> drastic like updating the version number or giving some grace period
>> where released versions start treating lowercase chunk IDs as required
>> before creating a new "required" chunk.
>>
>> This does mean that if there is a version incompatibility, the RIDX
>> chunk will just be ignored by the older version of Git.
>>
>> In terms of making this a safe format upgrade, I think Taylor has
>> achieved that.
> 
> Thanks. And yeah, the chunk should (and is) mandatory when writing a
> MIDX bitmap. But if we ran `git multi-pack-index write` without
> `--bitmap`, then we would be free to not write the RIDX chunk (and
> indeed that is what we do).

I just want to be careful of the language here, with respect to how chunks are listed as required or optional in Documentation/technical/pack-format.txt. "Required" means every MIDX needs one or is invalid. "Optional" means that all versions of Git should ignore the chunk if it does not recognize the ID.

So here, the new RIDX chunk will be ignored by older versions and will be written only when we care about bitmaps.

(Nothing I say here is in conflict with what you said, but I anticipate confusion with your use of the word "mandatory".)

Show 12 quoted lines
>> The only thing I can think is that server operators might want to
>> deploy this version with GIT_TEST_MIDX_WRITE_REV=1 for a while, so
>> any need to downgrade would not suffer a performance penalty for a
>> missing .rev file. If that is a planned way to safely deploy this
>> change, then it might be worth adding a test that we safely delete
>> a .rev file after writing both a .rev file and a RIDX chunk. (The
>> RIDX chunk will be preferred, so maybe the previous .rev file hits
>> some logic that would skip its deletion?)
> 
> That logic (that we delete auxiliary files--including the .rev file--not
> matching the checksum of the MIDX we just wrote) is unchanged. So I
> think we should be good there since we have existing coverage.
Sounds good.

Thanks, -Stolee

Previous: Taylor BlauNext: Junio C Hamano
Message 30 of 87 in “midx: prevent bitmap corruption when permuting pack order”
  1. 0/2 midx: prevent bitmap corruption when permuting pack orderTaylor Blau, Dec 8, 2021
  2. 1/2 t5326: demonstrate bitmap corruption after permutationTaylor Blau, Dec 8, 2021
  3. 2/2 midx.c: make changing the preferred pack safeTaylor Blau, Dec 8, 2021
  4. Derrick StoleeDec 8, 2021
  5. Jeff KingDec 8, 2021
  6. Taylor BlauDec 10, 2021
  7. Taylor BlauDec 10, 2021
  8. Taylor BlauDec 11, 2021
  9. Derrick StoleeDec 13, 2021
  10. Taylor BlauDec 13, 2021
  11. 0/8 midx: prevent bitmap corruption when permuting pack orderTaylor Blau, Dec 14, 2021
  12. 1/8 t5326: demonstrate bitmap corruption after permutationTaylor Blau, Dec 14, 2021
  13. 2/8 midx.c: make changing the preferred pack safeTaylor Blau, Dec 14, 2021
  14. 3/8 pack-revindex.c: instrument loading on-disk reverse indexTaylor Blau, Dec 14, 2021
  15. 4/8 t5326: drop unnecessary setupTaylor Blau, Dec 14, 2021
  16. 5/8 t5326: extract `test_rev_exists`Taylor Blau, Dec 14, 2021
  17. Derrick StoleeDec 20, 2021
  18. Taylor BlauJan 4, 2022
  19. 6/8 t5326: move tests to t/lib-bitmap.shTaylor Blau, Dec 14, 2021
  20. 7/8 t/lib-bitmap.sh: parameterize tests over reverse index sourceTaylor Blau, Dec 14, 2021
  21. 8/8 midx: read `RIDX` chunk when presentTaylor Blau, Dec 14, 2021
  22. Derrick StoleeDec 20, 2021
  23. Taylor BlauJan 4, 2022
  24. Junio C HamanoDec 15, 2021
  25. Taylor BlauDec 15, 2021
  26. Junio C HamanoDec 15, 2021
  27. Junio C HamanoDec 15, 2021
  28. Derrick StoleeDec 20, 2021
  29. Taylor BlauDec 20, 2021
  30. Derrick StoleeDec 20, 2021
  31. Junio C HamanoDec 15, 2021
  32. Taylor BlauDec 15, 2021
  33. 0/9 midx: prevent bitmap corruption when permuting pack orderTaylor Blau, Jan 4, 2022
  34. 1/9 t5326: demonstrate bitmap corruption after permutationTaylor Blau, Jan 4, 2022
  35. Jonathan TanJan 20, 2022
  36. Taylor BlauJan 20, 2022
  37. Junio C HamanoJan 20, 2022
  38. Taylor BlauJan 20, 2022
  39. Jonathan TanJan 24, 2022
  40. 2/9 midx.c: make changing the preferred pack safeTaylor Blau, Jan 4, 2022
  41. Junio C HamanoJan 14, 2022
  42. Junio C HamanoJan 14, 2022
  43. Taylor BlauJan 15, 2022
  44. Junio C HamanoJan 15, 2022
  45. Jonathan TanJan 20, 2022
  46. Taylor BlauJan 20, 2022
  47. 3/9 pack-revindex.c: instrument loading on-disk reverse indexTaylor Blau, Jan 4, 2022
  48. Jonathan TanJan 20, 2022
  49. Taylor BlauJan 20, 2022
  50. Jonathan TanJan 24, 2022
  51. 4/9 t5326: drop unnecessary setupTaylor Blau, Jan 4, 2022
  52. 5/9 t5326: extract `test_rev_exists`Taylor Blau, Jan 4, 2022
  53. 6/9 t5326: move tests to t/lib-bitmap.shTaylor Blau, Jan 4, 2022
  54. 7/9 t/lib-bitmap.sh: parameterize tests over reverse index sourceTaylor Blau, Jan 4, 2022
  55. Jonathan TanJan 24, 2022
  56. Taylor BlauJan 25, 2022
  57. Jonathan TanJan 26, 2022
  58. 8/9 midx: read `RIDX` chunk when presentTaylor Blau, Jan 4, 2022
  59. Jonathan TanJan 24, 2022
  60. Taylor BlauJan 25, 2022
  61. Jonathan TanJan 26, 2022
  62. 9/9 pack-bitmap.c: gracefully fallback after opening pack/MIDXTaylor Blau, Jan 4, 2022
  63. Jonathan TanJan 24, 2022
  64. Taylor BlauJan 25, 2022
  65. 0/9 midx: prevent bitmap corruption when permuting pack orderTaylor Blau, Jan 25, 2022
  66. 1/9 t5326: demonstrate bitmap corruption after permutationTaylor Blau, Jan 25, 2022
  67. Ævar Arnfjörð BjarmasonJan 26, 2022
  68. Taylor BlauJan 26, 2022
  69. 2/9 midx.c: make changing the preferred pack safeTaylor Blau, Jan 25, 2022
  70. 3/9 pack-revindex.c: instrument loading on-disk reverse indexTaylor Blau, Jan 25, 2022
  71. Ævar Arnfjörð BjarmasonJan 26, 2022
  72. 4/9 t5326: drop unnecessary setupTaylor Blau, Jan 25, 2022
  73. 5/9 t5326: extract `test_rev_exists`Taylor Blau, Jan 25, 2022
  74. Ævar Arnfjörð BjarmasonJan 26, 2022
  75. Taylor BlauJan 26, 2022
  76. 6/9 t5326: move tests to t/lib-bitmap.shTaylor Blau, Jan 25, 2022
  77. 7/9 t/lib-bitmap.sh: parameterize tests over reverse index sourceTaylor Blau, Jan 25, 2022
  78. 8/9 midx: read `RIDX` chunk when presentTaylor Blau, Jan 25, 2022
  79. Ævar Arnfjörð BjarmasonJan 26, 2022
  80. Taylor BlauJan 26, 2022
  81. 9/9 pack-bitmap.c: gracefully fallback after opening pack/MIDXTaylor Blau, Jan 25, 2022
  82. Ævar Arnfjörð BjarmasonJan 26, 2022
  83. Ævar Arnfjörð BjarmasonJan 26, 2022
  84. Taylor BlauJan 26, 2022
  85. Jonathan TanJan 27, 2022
  86. Taylor BlauFeb 24, 2022
  87. Derrick StoleeJan 27, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.