git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v4 16/23] config: create core.multiPackIndex setting

From
Derrick Stolee <stolee@gmail.com>
Date
Jul 13, 2018, 00:50 UTC
Message-ID
<61de4c01-daeb-2fad-491e-aa06facc3db8@gmail.com>
In-Reply-To
<xmqqin5kupu3.fsf@gitster-ct.c.googlers.com>
On 7/12/2018 5:05 PM, Junio C Hamano wrote:
Show 24 quoted lines
> Derrick Stolee <stolee@gmail.com> writes:
>
>> The core.multiPackIndex config setting controls the multi-pack-
>> index (MIDX) feature. If false, the setting will disable all reads
>> from the multi-pack-index file.
>>
>> Read this config setting in the new prepare_multi_pack_index_one()
>> which is called during prepare_packed_git(). This check is run once
>> per repository.
>>
>> Add comparison commands in t5319-multi-pack-index.sh to check
>> typical Git behavior remains the same as the config setting is turned
>> on and off. This currently includes 'git rev-list' and 'git log'
>> commands to trigger several object database reads. Currently, these
>> would only catch an error in the prepare_multi_pack_index_one(), but
>> with later commits will catch errors in object lookups, abbreviations,
>> and approximate object counts.
>>
>> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
>>
>> midx: prepare midxed_git struct
>>
>> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
> What is going on around here?

Sorry. I squashed the commits, and intended to drop this second commit message.

Show 30 quoted lines
>
>> diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh
>> index 4a4fa26f7a..601e28a2f0 100755
>> --- a/t/t5319-multi-pack-index.sh
>> +++ b/t/t5319-multi-pack-index.sh
>> @@ -3,6 +3,8 @@
>>   test_description='multi-pack-indexes'
>>   . ./test-lib.sh
>>   
>> +objdir=.git/objects
>> +
>>   midx_read_expect () {
>>   	NUM_PACKS=$1
>>   	NUM_OBJECTS=$2
>> @@ -76,18 +78,35 @@ test_expect_success 'create objects' '
>>   '
>>   
>>   test_expect_success 'write midx with one v1 pack' '
>> -	pack=$(git pack-objects --index-version=1 pack/test <obj-list) &&
>> -	test_when_finished rm pack/test-$pack.pack pack/test-$pack.idx pack/multi-pack-index &&
>> -	git multi-pack-index --object-dir=. write &&
>> -	midx_read_expect 1 18 4 .
>> +	pack=$(git pack-objects --index-version=1 $objdir/pack/test <obj-list) &&
>> +	test_when_finished rm $objdir/pack/test-$pack.pack \
>> +		$objdir/pack/test-$pack.idx $objdir/pack/multi-pack-index &&
>> +	git multi-pack-index --object-dir=$objdir write &&
>> +	midx_read_expect 1 18 4 $objdir
> Hmph, so we used to run tests as if $cwd were GIT_OBJECT_DIRECTORY
> but now we are running them from the top-level of the working tree,
> just like all the other tests?  Interesting.
This is the first time we _need_ them in the .git/object directory.
Show 14 quoted lines
>>   '
>>   
>> +midx_git_two_modes() {
>> +	git -c core.multiPackIndex=false $1 >expect &&
>> +	git -c core.multiPackIndex=true $1 >actual &&
>> +	test_cmp expect actual
>> +}
>> +
>> +compare_results_with_midx() {
> Style: "compare_results_with_midx () {", just like mdx_read_expect
> near the top of the file, but unlike midx_git_two_modes we see
> nearby.  Please keep "git grep 'funcname () {'" a usable way to
> locate where a shell function is defined without forcing people to
> type an asterisk.
Sorry that I missed these two.
Thanks,
-Stolee
Previous: Junio C HamanoNext: Derrick Stolee
Message 184 of 192 in “Multi-pack-index (MIDX)”
  1. 00/23 Multi-pack-index (MIDX)Derrick Stolee, Jun 7, 2018
  2. 01/23 midx: add design documentDerrick Stolee, Jun 7, 2018
  3. Stefan BellerJun 11, 2018
  4. Derrick StoleeJun 18, 2018
  5. 03/23 midx: add midx builtinDerrick Stolee, Jun 7, 2018
  6. Duy NguyenJun 7, 2018
  7. Derrick StoleeJun 18, 2018
  8. Stefan BellerJun 11, 2018
  9. Derrick StoleeJun 18, 2018
  10. Stefan BellerJun 18, 2018
  11. Derrick StoleeJun 18, 2018
  12. 04/23 midx: add 'write' subcommand and basic wiringDerrick Stolee, Jun 7, 2018
  13. Duy NguyenJun 7, 2018
  14. 09/23 midx: write pack names in chunkDerrick Stolee, Jun 7, 2018
  15. Duy NguyenJun 7, 2018
  16. Derrick StoleeJun 21, 2018
  17. Junio C HamanoJun 21, 2018
  18. Derrick StoleeJun 22, 2018
  19. Junio C HamanoJun 22, 2018
  20. Derrick StoleeJun 22, 2018
  21. 12/23 midx: write object ids in a chunkDerrick Stolee, Jun 7, 2018
  22. Duy NguyenJun 9, 2018
  23. 10/23 midx: write a lookup into the pack names chunkDerrick Stolee, Jun 7, 2018
  24. Duy NguyenJun 9, 2018
  25. Derrick StoleeJun 21, 2018
  26. 16/23 midx: prepare midxed_git structDerrick Stolee, Jun 7, 2018
  27. Duy NguyenJun 9, 2018
  28. 11/23 midx: sort and deduplicate objects from packfilesDerrick Stolee, Jun 7, 2018
  29. Duy NguyenJun 9, 2018
  30. Derrick StoleeJun 21, 2018
  31. 21/23 midx: prevent duplicate packfile loadsDerrick Stolee, Jun 7, 2018
  32. Duy NguyenJun 9, 2018
  33. 23/23 midx: clear midx on repackDerrick Stolee, Jun 7, 2018
  34. Duy NguyenJun 9, 2018
  35. Derrick StoleeJun 22, 2018
  36. 22/23 midx: use midx to find ref-deltasDerrick Stolee, Jun 7, 2018
  37. 13/23 midx: write object id fanout chunkDerrick Stolee, Jun 7, 2018
  38. Duy NguyenJun 9, 2018
  39. Derrick StoleeJun 21, 2018
  40. 14/23 midx: write object offsetsDerrick Stolee, Jun 7, 2018
  41. Duy NguyenJun 9, 2018
  42. 20/23 midx: use midx in approximate_object_countDerrick Stolee, Jun 7, 2018
  43. Duy NguyenJun 9, 2018
  44. Derrick StoleeJun 22, 2018
  45. 19/23 midx: use existing midx when writing new oneDerrick Stolee, Jun 7, 2018
  46. 15/23 midx: create core.midx config settingDerrick Stolee, Jun 7, 2018
  47. 18/23 midx: use midx in abbreviation calculationsDerrick Stolee, Jun 7, 2018
  48. Duy NguyenJun 9, 2018
  49. Derrick StoleeJun 22, 2018
  50. 17/23 midx: read objects from multi-pack-indexDerrick Stolee, Jun 7, 2018
  51. Duy NguyenJun 9, 2018
  52. Derrick StoleeJun 21, 2018
  53. 05/23 midx: write header information to lockfileDerrick Stolee, Jun 7, 2018
  54. Duy NguyenJun 7, 2018
  55. Duy NguyenJun 12, 2018
  56. Derrick StoleeJun 19, 2018
  57. Duy NguyenJun 19, 2018
  58. Derrick StoleeJun 19, 2018
  59. 08/23 midx: read packfiles from pack directoryDerrick Stolee, Jun 7, 2018
  60. Duy NguyenJun 7, 2018
  61. packfile: generalize pack directory listDerrick Stolee, Jun 20, 2018
  62. 07/23 midx: expand test dataDerrick Stolee, Jun 7, 2018
  63. 06/23 midx: struct midxed_git and 'read' subcommandDerrick Stolee, Jun 7, 2018
  64. Duy NguyenJun 7, 2018
  65. Derrick StoleeJun 20, 2018
  66. Duy NguyenJun 7, 2018
  67. Derrick StoleeJun 20, 2018
  68. Duy NguyenJun 20, 2018
  69. Derrick StoleeJun 20, 2018
  70. 02/23 midx: add midx format details to pack-format.txtDerrick Stolee, Jun 7, 2018
  71. Stefan BellerJun 11, 2018
  72. Derrick StoleeJun 18, 2018
  73. Stefan BellerJun 18, 2018
  74. Derrick StoleeJun 7, 2018
  75. Ævar Arnfjörð BjarmasonJun 7, 2018
  76. Derrick StoleeJun 7, 2018
  77. 00/24 Multi-pack-index (MIDX)Derrick Stolee, Jun 25, 2018
  78. 01/24 multi-pack-index: add design documentDerrick Stolee, Jun 25, 2018
  79. 02/24 multi-pack-index: add format detailsDerrick Stolee, Jun 25, 2018
  80. 04/24 multi-pack-index: add 'write' verbDerrick Stolee, Jun 25, 2018
  81. 03/24 multi-pack-index: add builtinDerrick Stolee, Jun 25, 2018
  82. Junio C HamanoJun 25, 2018
  83. 05/24 midx: write header information to lockfileDerrick Stolee, Jun 25, 2018
  84. Junio C HamanoJun 25, 2018
  85. Derrick StoleeJul 5, 2018
  86. 08/24 packfile: generalize pack directory listDerrick Stolee, Jun 25, 2018
  87. Junio C HamanoJun 25, 2018
  88. 09/24 multi-pack-index: read packfile listDerrick Stolee, Jun 25, 2018
  89. 06/24 multi-pack-index: load into memoryDerrick Stolee, Jun 25, 2018
  90. Junio C HamanoJun 25, 2018
  91. Derrick StoleeJul 5, 2018
  92. Eric SunshineJul 5, 2018
  93. Junio C HamanoJul 6, 2018
  94. 11/24 midx: read pack names into arrayDerrick Stolee, Jun 25, 2018
  95. Eric SunshineJun 25, 2018
  96. 12/24 midx: sort and deduplicate objects from packfilesDerrick Stolee, Jun 25, 2018
  97. 13/24 midx: write object ids in a chunkDerrick Stolee, Jun 25, 2018
  98. 14/24 midx: write object id fanout chunkDerrick Stolee, Jun 25, 2018
  99. 15/24 midx: write object offsetsDerrick Stolee, Jun 25, 2018
  100. 16/24 config: create core.multiPackIndex settingDerrick Stolee, Jun 25, 2018
  101. 18/24 midx: read objects from multi-pack-indexDerrick Stolee, Jun 25, 2018
  102. 10/24 multi-pack-index: write pack names in chunkDerrick Stolee, Jun 25, 2018
  103. 19/24 midx: use midx in abbreviation calculationsDerrick Stolee, Jun 25, 2018
  104. 17/24 midx: prepare midxed_git structDerrick Stolee, Jun 25, 2018
  105. 23/24 packfile: skip loading index if in multi-pack-indexDerrick Stolee, Jun 25, 2018
  106. 24/24 midx: clear midx on repackDerrick Stolee, Jun 25, 2018
  107. 20/24 midx: use existing midx when writing new oneDerrick Stolee, Jun 25, 2018
  108. 22/24 midx: prevent duplicate packfile loadsDerrick Stolee, Jun 25, 2018
  109. 21/24 midx: use midx in approximate_object_countDerrick Stolee, Jun 25, 2018
  110. 07/24 multi-pack-index: expand test dataDerrick Stolee, Jun 25, 2018
  111. Junio C HamanoJun 25, 2018
  112. 00/24 Multi-pack-index (MIDX)Derrick Stolee, Jul 6, 2018
  113. 01/24 multi-pack-index: add design documentDerrick Stolee, Jul 6, 2018
  114. 02/24 multi-pack-index: add format detailsDerrick Stolee, Jul 6, 2018
  115. 04/24 multi-pack-index: add 'write' verbDerrick Stolee, Jul 6, 2018
  116. Eric SunshineJul 6, 2018
  117. 03/24 multi-pack-index: add builtinDerrick Stolee, Jul 6, 2018
  118. Eric SunshineJul 6, 2018
  119. 05/24 midx: write header information to lockfileDerrick Stolee, Jul 6, 2018
  120. 08/24 packfile: generalize pack directory listDerrick Stolee, Jul 6, 2018
  121. 06/24 multi-pack-index: load into memoryDerrick Stolee, Jul 6, 2018
  122. Eric SunshineJul 6, 2018
  123. Eric SunshineJul 6, 2018
  124. Junio C HamanoJul 9, 2018
  125. Derrick StoleeJul 12, 2018
  126. 09/24 multi-pack-index: read packfile listDerrick Stolee, Jul 6, 2018
  127. 10/24 multi-pack-index: write pack names in chunkDerrick Stolee, Jul 6, 2018
  128. 13/24 midx: write object ids in a chunkDerrick Stolee, Jul 6, 2018
  129. Eric SunshineJul 6, 2018
  130. 12/24 midx: sort and deduplicate objects from packfilesDerrick Stolee, Jul 6, 2018
  131. 14/24 midx: write object id fanout chunkDerrick Stolee, Jul 6, 2018
  132. 15/24 midx: write object offsetsDerrick Stolee, Jul 6, 2018
  133. Eric SunshineJul 6, 2018
  134. Derrick StoleeJul 12, 2018
  135. 18/24 midx: read objects from multi-pack-indexDerrick Stolee, Jul 6, 2018
  136. 19/24 midx: use midx in abbreviation calculationsDerrick Stolee, Jul 6, 2018
  137. 20/24 midx: use existing midx when writing new oneDerrick Stolee, Jul 6, 2018
  138. 24/24 midx: clear midx on repackDerrick Stolee, Jul 6, 2018
  139. Eric SunshineJul 6, 2018
  140. 21/24 midx: use midx in approximate_object_countDerrick Stolee, Jul 6, 2018
  141. 22/24 midx: prevent duplicate packfile loadsDerrick Stolee, Jul 6, 2018
  142. 23/24 packfile: skip loading index if in multi-pack-indexDerrick Stolee, Jul 6, 2018
  143. 16/24 config: create core.multiPackIndex settingDerrick Stolee, Jul 6, 2018
  144. Eric SunshineJul 6, 2018
  145. Derrick StoleeJul 12, 2018
  146. Derrick StoleeJul 12, 2018
  147. SZEDER GáborJul 11, 2018
  148. Derrick StoleeJul 12, 2018
  149. SZEDER GáborJul 12, 2018
  150. Derrick StoleeJul 12, 2018
  151. Junio C HamanoJul 12, 2018
  152. 17/24 midx: prepare midxed_git structDerrick Stolee, Jul 6, 2018
  153. Eric SunshineJul 6, 2018
  154. 07/24 multi-pack-index: expand test dataDerrick Stolee, Jul 6, 2018
  155. Eric SunshineJul 6, 2018
  156. Eric SunshineJul 6, 2018
  157. Derrick StoleeJul 12, 2018
  158. Eric SunshineJul 12, 2018
  159. Derrick StoleeJul 12, 2018
  160. 11/24 midx: read pack names into arrayDerrick Stolee, Jul 6, 2018
  161. Eric SunshineJul 6, 2018
  162. 00/23 Multi-pack-index (MIDX)Derrick Stolee, Jul 12, 2018
  163. 02/23 multi-pack-index: add format detailsDerrick Stolee, Jul 12, 2018
  164. 01/23 multi-pack-index: add design documentDerrick Stolee, Jul 12, 2018
  165. 03/23 multi-pack-index: add builtinDerrick Stolee, Jul 12, 2018
  166. Junio C HamanoJul 20, 2018
  167. brian m. carlsonJul 20, 2018
  168. Junio C HamanoJul 20, 2018
  169. 04/23 multi-pack-index: add 'write' verbDerrick Stolee, Jul 12, 2018
  170. Eric SunshineJul 12, 2018
  171. 06/23 multi-pack-index: load into memoryDerrick Stolee, Jul 12, 2018
  172. 05/23 midx: write header information to lockfileDerrick Stolee, Jul 12, 2018
  173. 07/23 t5319: expand test dataDerrick Stolee, Jul 12, 2018
  174. 08/23 packfile: generalize pack directory listDerrick Stolee, Jul 12, 2018
  175. 09/23 multi-pack-index: read packfile listDerrick Stolee, Jul 12, 2018
  176. 11/23 midx: read pack names into arrayDerrick Stolee, Jul 12, 2018
  177. 10/23 multi-pack-index: write pack names in chunkDerrick Stolee, Jul 12, 2018
  178. 12/23 midx: sort and deduplicate objects from packfilesDerrick Stolee, Jul 12, 2018
  179. 13/23 midx: write object ids in a chunkDerrick Stolee, Jul 12, 2018
  180. 14/23 midx: write object id fanout chunkDerrick Stolee, Jul 12, 2018
  181. 15/23 midx: write object offsetsDerrick Stolee, Jul 12, 2018
  182. 16/23 config: create core.multiPackIndex settingDerrick Stolee, Jul 12, 2018
  183. Junio C HamanoJul 12, 2018
  184. Derrick StoleeJul 13, 2018
  185. 17/23 midx: read objects from multi-pack-indexDerrick Stolee, Jul 12, 2018
  186. 18/23 midx: use midx in abbreviation calculationsDerrick Stolee, Jul 12, 2018
  187. 19/23 midx: use existing midx when writing new oneDerrick Stolee, Jul 12, 2018
  188. 20/23 midx: use midx in approximate_object_countDerrick Stolee, Jul 12, 2018
  189. 21/23 midx: prevent duplicate packfile loadsDerrick Stolee, Jul 12, 2018
  190. 23/23 midx: clear midx on repackDerrick Stolee, Jul 12, 2018
  191. 22/23 packfile: skip loading index if in multi-pack-indexDerrick Stolee, Jul 12, 2018
  192. Junio C HamanoJul 12, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.