git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v3 9/9] pack-bitmap.c: gracefully fallback after opening pack/MIDX

From
Taylor Blau <me@ttaylorr.com>
Date
Jan 4, 2022, 18:16 UTC
Message-ID
<9707d5ea4433d9a5c7f8581dbb2d8a05f410efd3.1641320129.git.me@ttaylorr.com>
In-Reply-To
<cover.1641320129.git.me@ttaylorr.com>

When opening a MIDX/pack-bitmap, we call open_midx_bitmap_1() or open_pack_bitmap_1() respectively in a loop over the set of MIDXs/packs. By design, these functions are supposed to be called over every pack and MIDX, since only one of them should have a valid bitmap.

Ordinarily we return '0' from these two functions in order to indicate that we successfully loaded a bitmap To signal that we couldn't load a bitmap corresponding to the MIDX/pack (either because one doesn't exist, or because there was an error with loading it), we can return '-1'. In either case, the callers each enumerate all MIDXs/packs to ensure that at most one bitmap per-kind is present.

But when we fail to load a bitmap that does exist (for example, loading a MIDX bitmap without finding a corresponding reverse index), we'll return -1 but leave the 'midx' field non-NULL. So when we fallback to loading a pack bitmap, we'll complain that the bitmap we're trying to populate already is "opened", even though it isn't.

Rectify this by setting the '->pack' and '->midx' field back to NULL as appropriate. Two tests are added: one to ensure that the MIDX-to-pack bitmap fallback works, and another to ensure we still complain when there are multiple pack bitmaps in a repository.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 pack-bitmap.c                 |  4 ++++
 t/t5310-pack-bitmaps.sh       | 28 ++++++++++++++++++++++++++++
 t/t5326-multi-pack-bitmaps.sh | 19 +++++++++++++++++++
 3 files changed, 51 insertions(+)
diff --git a/pack-bitmap.c b/pack-bitmap.c
index f772d3cb7f..9c666cdb8b 100644
--- a/pack-bitmap.c
+++ b/pack-bitmap.c
@@ -358,7 +358,9 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,
 cleanup:
 	munmap(bitmap_git->map, bitmap_git->map_size);
 	bitmap_git->map_size = 0;
+	bitmap_git->map_pos = 0;
 	bitmap_git->map = NULL;
+	bitmap_git->midx = NULL;
 	return -1;
 }
 
@@ -405,6 +407,8 @@ static int open_pack_bitmap_1(struct bitmap_index *bitmap_git, struct packed_git
 		munmap(bitmap_git->map, bitmap_git->map_size);
 		bitmap_git->map = NULL;
 		bitmap_git->map_size = 0;
+		bitmap_git->map_pos = 0;
+		bitmap_git->pack = NULL;
 		return -1;
 	}
 
diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh
index d05ab716f6..f775fc1ce6 100755
--- a/t/t5310-pack-bitmaps.sh
+++ b/t/t5310-pack-bitmaps.sh
@@ -397,4 +397,32 @@ test_expect_success 'pack.preferBitmapTips' '
 	)
 '
 
+test_expect_success 'complains about multiple pack bitmaps' '
+	rm -fr repo &&
+	git init repo &&
+	test_when_finished "rm -fr repo" &&
+	(
+		cd repo &&
+
+		test_commit base &&
+
+		git repack -adb &&
+		bitmap="$(ls .git/objects/pack/pack-*.bitmap)" &&
+		mv "$bitmap" "$bitmap.bak" &&
+
+		test_commit other &&
+		git repack -ab &&
+
+		mv "$bitmap.bak" "$bitmap" &&
+
+		find .git/objects/pack -type f -name "*.pack" >packs &&
+		find .git/objects/pack -type f -name "*.bitmap" >bitmaps &&
+		test_line_count = 2 packs &&
+		test_line_count = 2 bitmaps &&
+
+		git rev-list --use-bitmap-index HEAD 2>err &&
+		grep "ignoring extra bitmap file" err
+	)
+'
+
 test_done
diff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh
index c0924074c4..3c1ecc7e25 100755
--- a/t/t5326-multi-pack-bitmaps.sh
+++ b/t/t5326-multi-pack-bitmaps.sh
@@ -266,4 +266,23 @@ test_expect_success 'hash-cache values are propagated from pack bitmaps' '
 	)
 '
 
+test_expect_success 'graceful fallback when missing reverse index' '
+	rm -fr repo &&
+	git init repo &&
+	test_when_finished "rm -fr repo" &&
+	(
+		cd repo &&
+
+		test_commit base &&
+
+		# write a pack and MIDX bitmap containing base
+		git repack -adb &&
+		git multi-pack-index write --bitmap &&
+
+		GIT_TEST_MIDX_READ_RIDX=0 \
+			git rev-list --use-bitmap-index HEAD 2>err &&
+		! grep "ignoring extra bitmap file" err
+	)
+'
+
 test_done
-- 
2.34.1.25.gb3157a20e6
Previous: Jonathan TanNext: Jonathan Tan
Message 62 of 87 in “midx: prevent bitmap corruption when permuting pack order”
  1. 0/2 midx: prevent bitmap corruption when permuting pack orderTaylor Blau, Dec 8, 2021
  2. 1/2 t5326: demonstrate bitmap corruption after permutationTaylor Blau, Dec 8, 2021
  3. 2/2 midx.c: make changing the preferred pack safeTaylor Blau, Dec 8, 2021
  4. Derrick StoleeDec 8, 2021
  5. Jeff KingDec 8, 2021
  6. Taylor BlauDec 10, 2021
  7. Taylor BlauDec 10, 2021
  8. Taylor BlauDec 11, 2021
  9. Derrick StoleeDec 13, 2021
  10. Taylor BlauDec 13, 2021
  11. 0/8 midx: prevent bitmap corruption when permuting pack orderTaylor Blau, Dec 14, 2021
  12. 1/8 t5326: demonstrate bitmap corruption after permutationTaylor Blau, Dec 14, 2021
  13. 2/8 midx.c: make changing the preferred pack safeTaylor Blau, Dec 14, 2021
  14. 3/8 pack-revindex.c: instrument loading on-disk reverse indexTaylor Blau, Dec 14, 2021
  15. 4/8 t5326: drop unnecessary setupTaylor Blau, Dec 14, 2021
  16. 5/8 t5326: extract `test_rev_exists`Taylor Blau, Dec 14, 2021
  17. Derrick StoleeDec 20, 2021
  18. Taylor BlauJan 4, 2022
  19. 6/8 t5326: move tests to t/lib-bitmap.shTaylor Blau, Dec 14, 2021
  20. 7/8 t/lib-bitmap.sh: parameterize tests over reverse index sourceTaylor Blau, Dec 14, 2021
  21. 8/8 midx: read `RIDX` chunk when presentTaylor Blau, Dec 14, 2021
  22. Derrick StoleeDec 20, 2021
  23. Taylor BlauJan 4, 2022
  24. Junio C HamanoDec 15, 2021
  25. Taylor BlauDec 15, 2021
  26. Junio C HamanoDec 15, 2021
  27. Junio C HamanoDec 15, 2021
  28. Derrick StoleeDec 20, 2021
  29. Taylor BlauDec 20, 2021
  30. Derrick StoleeDec 20, 2021
  31. Junio C HamanoDec 15, 2021
  32. Taylor BlauDec 15, 2021
  33. 0/9 midx: prevent bitmap corruption when permuting pack orderTaylor Blau, Jan 4, 2022
  34. 1/9 t5326: demonstrate bitmap corruption after permutationTaylor Blau, Jan 4, 2022
  35. Jonathan TanJan 20, 2022
  36. Taylor BlauJan 20, 2022
  37. Junio C HamanoJan 20, 2022
  38. Taylor BlauJan 20, 2022
  39. Jonathan TanJan 24, 2022
  40. 2/9 midx.c: make changing the preferred pack safeTaylor Blau, Jan 4, 2022
  41. Junio C HamanoJan 14, 2022
  42. Junio C HamanoJan 14, 2022
  43. Taylor BlauJan 15, 2022
  44. Junio C HamanoJan 15, 2022
  45. Jonathan TanJan 20, 2022
  46. Taylor BlauJan 20, 2022
  47. 3/9 pack-revindex.c: instrument loading on-disk reverse indexTaylor Blau, Jan 4, 2022
  48. Jonathan TanJan 20, 2022
  49. Taylor BlauJan 20, 2022
  50. Jonathan TanJan 24, 2022
  51. 4/9 t5326: drop unnecessary setupTaylor Blau, Jan 4, 2022
  52. 5/9 t5326: extract `test_rev_exists`Taylor Blau, Jan 4, 2022
  53. 6/9 t5326: move tests to t/lib-bitmap.shTaylor Blau, Jan 4, 2022
  54. 7/9 t/lib-bitmap.sh: parameterize tests over reverse index sourceTaylor Blau, Jan 4, 2022
  55. Jonathan TanJan 24, 2022
  56. Taylor BlauJan 25, 2022
  57. Jonathan TanJan 26, 2022
  58. 8/9 midx: read `RIDX` chunk when presentTaylor Blau, Jan 4, 2022
  59. Jonathan TanJan 24, 2022
  60. Taylor BlauJan 25, 2022
  61. Jonathan TanJan 26, 2022
  62. 9/9 pack-bitmap.c: gracefully fallback after opening pack/MIDXTaylor Blau, Jan 4, 2022
  63. Jonathan TanJan 24, 2022
  64. Taylor BlauJan 25, 2022
  65. 0/9 midx: prevent bitmap corruption when permuting pack orderTaylor Blau, Jan 25, 2022
  66. 1/9 t5326: demonstrate bitmap corruption after permutationTaylor Blau, Jan 25, 2022
  67. Ævar Arnfjörð BjarmasonJan 26, 2022
  68. Taylor BlauJan 26, 2022
  69. 2/9 midx.c: make changing the preferred pack safeTaylor Blau, Jan 25, 2022
  70. 3/9 pack-revindex.c: instrument loading on-disk reverse indexTaylor Blau, Jan 25, 2022
  71. Ævar Arnfjörð BjarmasonJan 26, 2022
  72. 4/9 t5326: drop unnecessary setupTaylor Blau, Jan 25, 2022
  73. 5/9 t5326: extract `test_rev_exists`Taylor Blau, Jan 25, 2022
  74. Ævar Arnfjörð BjarmasonJan 26, 2022
  75. Taylor BlauJan 26, 2022
  76. 6/9 t5326: move tests to t/lib-bitmap.shTaylor Blau, Jan 25, 2022
  77. 7/9 t/lib-bitmap.sh: parameterize tests over reverse index sourceTaylor Blau, Jan 25, 2022
  78. 8/9 midx: read `RIDX` chunk when presentTaylor Blau, Jan 25, 2022
  79. Ævar Arnfjörð BjarmasonJan 26, 2022
  80. Taylor BlauJan 26, 2022
  81. 9/9 pack-bitmap.c: gracefully fallback after opening pack/MIDXTaylor Blau, Jan 25, 2022
  82. Ævar Arnfjörð BjarmasonJan 26, 2022
  83. Ævar Arnfjörð BjarmasonJan 26, 2022
  84. Taylor BlauJan 26, 2022
  85. Jonathan TanJan 27, 2022
  86. Taylor BlauFeb 24, 2022
  87. Derrick StoleeJan 27, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.