git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 02/30] read-cache: add index.computeHash config option

From
Derrick Stolee <derrickstolee@github.com>
Date
Nov 14, 2022, 16:30 UTC
Message-ID
<ca39e1d3-b08c-6ed3-8ab5-238efbd1f2dc@github.com>
In-Reply-To
<CABPp-BEvF3XF+udzTkEtgrtXqYuYEeYi0R65EY5gCespwZgOeg@mail.gmail.com>
On 11/11/2022 6:31 PM, Elijah Newren wrote:
Show 54 quoted lines
> On Mon, Nov 7, 2022 at 10:48 AM Derrick Stolee via GitGitGadget
> <gitgitgadget@gmail.com> wrote:
>>
>> From: Derrick Stolee <derrickstolee@github.com>
>>
>> The previous change allowed skipping the hashing portion of the
>> hashwrite API, using it instead as a buffered write API. Disabling the
>> hashwrite can be particularly helpful when the write operation is in a
>> critical path.
>>
>> One such critical path is the writing of the index. This operation is so
>> critical that the sparse index was created specifically to reduce the
>> size of the index to make these writes (and reads) faster.
>>
>> Following a similar approach to one used in the microsoft/git fork [1],
>> add a new config option that allows disabling this hashing during the
>> index write. The cost is that we can no longer validate the contents for
>> corruption-at-rest using the trailing hash.
>>
>> [1] https://github.com/microsoft/git/commit/21fed2d91410f45d85279467f21d717a2db45201
>>
>> While older Git versions will not recognize the null hash as a special
>> case, the file format itself is still being met in terms of its
>> structure. Using this null hash will still allow Git operations to
>> function across older versions.
>>
>> The one exception is 'git fsck' which checks the hash of the index file.
>> Here, we disable this check if the trailing hash is all zeroes. We add a
>> warning to the config option that this may cause undesirable behavior
>> with older Git versions.
>>
>> As a quick comparison, I tested 'git update-index --force-write' with
>> and without index.computHash=false on a copy of the Linux kernel
>> repository.
>>
>> Benchmark 1: with hash
>>   Time (mean ± σ):      46.3 ms ±  13.8 ms    [User: 34.3 ms, System: 11.9 ms]
>>   Range (min … max):    34.3 ms …  79.1 ms    82 runs
>>
>> Benchmark 2: without hash
>>   Time (mean ± σ):      26.0 ms ±   7.9 ms    [User: 11.8 ms, System: 14.2 ms]
>>   Range (min … max):    16.3 ms …  42.0 ms    69 runs
>>
>> Summary
>>   'without hash' ran
>>     1.78 ± 0.76 times faster than 'with hash'
>>
>> These performance benefits are substantial enough to allow users the
>> ability to opt-in to this feature, even with the potential confusion
>> with older 'git fsck' versions.
> 
> This is impressive and interesting...but an improvement unrelated to
> this series other than the fact that it builds on some of it.  Perhaps
> pull this patch out?

While patch 1 is required for the packed-refs work, this one is an easy way to take advantage of it. I'll submit these two patches soon on their own as the rest of the RFC is discussed.

> Also, would it make sense to integrate index.computeHash with feature.manyFiles?

It would make sense to include in feature.manyFiles and Scalar's recommended config. I expect that it would be good to have the config available in a Git release before updating those configs to include it. Perhaps that is too conservative, though.

Thanks, -Stolee

Previous: Elijah NewrenNext: Ævar Arnfjörð Bjarmason
Message 5 of 56 in “[RFC] extensions.refFormat and packed-refs v2 file format”
  1. 00/30 [RFC] extensions.refFormat and packed-refs v2 file formatDerrick Stolee via GitGitGadget, Nov 7, 2022
  2. 01/30 hashfile: allow skipping the hash functionDerrick Stolee via GitGitGadget, Nov 7, 2022
  3. 02/30 read-cache: add index.computeHash config optionDerrick Stolee via GitGitGadget, Nov 7, 2022
  4. Elijah NewrenNov 11, 2022
  5. Derrick StoleeNov 14, 2022
  6. Ævar Arnfjörð BjarmasonNov 17, 2022
  7. 03/30 extensions: add refFormat extensionDerrick Stolee via GitGitGadget, Nov 7, 2022
  8. Elijah NewrenNov 11, 2022
  9. Derrick StoleeNov 16, 2022
  10. 06/30 refs: allow loose files without packed-refsDerrick Stolee via GitGitGadget, Nov 7, 2022
  11. 07/30 chunk-format: number of chunks is optionalDerrick Stolee via GitGitGadget, Nov 7, 2022
  12. 04/30 config: fix multi-level bulleted listDerrick Stolee via GitGitGadget, Nov 7, 2022
  13. 05/30 repository: wire ref extensions to ref backendsDerrick Stolee via GitGitGadget, Nov 7, 2022
  14. 08/30 chunk-format: document trailing table of contentsDerrick Stolee via GitGitGadget, Nov 7, 2022
  15. 09/30 chunk-format: store chunk offset during writeDerrick Stolee via GitGitGadget, Nov 7, 2022
  16. 11/30 chunk-format: parse trailing table of contentsDerrick Stolee via GitGitGadget, Nov 7, 2022
  17. 10/30 chunk-format: allow trailing table of contentsDerrick Stolee via GitGitGadget, Nov 7, 2022
  18. 13/30 packed-backend: extract add_write_error()Derrick Stolee via GitGitGadget, Nov 7, 2022
  19. 12/30 refs: extract packfile format to new fileDerrick Stolee via GitGitGadget, Nov 7, 2022
  20. 14/30 packed-backend: extract iterator/updates mergeDerrick Stolee via GitGitGadget, Nov 7, 2022
  21. 16/30 config: add config values for packed-refs v2Derrick Stolee via GitGitGadget, Nov 7, 2022
  22. 15/30 packed-backend: create abstraction for writing refsDerrick Stolee via GitGitGadget, Nov 7, 2022
  23. 17/30 packed-backend: create shell of v2 writesDerrick Stolee via GitGitGadget, Nov 7, 2022
  24. 18/30 packed-refs: write file format version 2Derrick Stolee via GitGitGadget, Nov 7, 2022
  25. 19/30 packed-refs: read file format v2Derrick Stolee via GitGitGadget, Nov 7, 2022
  26. 20/30 packed-refs: read optional prefix chunksDerrick Stolee via GitGitGadget, Nov 7, 2022
  27. 21/30 packed-refs: write prefix chunksDerrick Stolee via GitGitGadget, Nov 7, 2022
  28. 22/30 packed-backend: create GIT_TEST_PACKED_REFS_VERSIONDerrick Stolee via GitGitGadget, Nov 7, 2022
  29. 24/30 t5312: allow packed-refs v2 formatDerrick Stolee via GitGitGadget, Nov 7, 2022
  30. 23/30 t1409: test with packed-refs v2Derrick Stolee via GitGitGadget, Nov 7, 2022
  31. 26/30 t3210: require packed-refs v1 for some testsDerrick Stolee via GitGitGadget, Nov 7, 2022
  32. 25/30 t5502: add PACKED_REFS_V1 prerequisiteDerrick Stolee via GitGitGadget, Nov 7, 2022
  33. 27/30 t*: skip packed-refs v2 over http testsDerrick Stolee via GitGitGadget, Nov 7, 2022
  34. 28/30 ci: run GIT_TEST_PACKED_REFS_VERSION=2 in some buildsDerrick Stolee via GitGitGadget, Nov 7, 2022
  35. 29/30 p1401: create performance test for ref operationsDerrick Stolee via GitGitGadget, Nov 7, 2022
  36. 30/30 refs: skip hashing when writing packed-refs v2Derrick Stolee via GitGitGadget, Nov 7, 2022
  37. Derrick StoleeNov 9, 2022
  38. Elijah NewrenNov 11, 2022
  39. Derrick StoleeNov 14, 2022
  40. Elijah NewrenNov 15, 2022
  41. Derrick StoleeNov 16, 2022
  42. Elijah NewrenNov 17, 2022
  43. Junio C HamanoNov 18, 2022
  44. Elijah NewrenNov 19, 2022
  45. Taylor BlauNov 19, 2022
  46. Derrick StoleeNov 30, 2022
  47. Han-Wen NienhuysNov 28, 2022
  48. Derrick StoleeNov 30, 2022
  49. Phillip WoodNov 30, 2022
  50. Taylor BlauNov 30, 2022
  51. Han-Wen NienhuysNov 30, 2022
  52. Sean AllredNov 30, 2022
  53. Derrick StoleeDec 1, 2022
  54. Han-Wen NienhuysDec 2, 2022
  55. Ævar Arnfjörð BjarmasonDec 2, 2022
  56. Junio C HamanoNov 30, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.