git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 0/8] pack-objects: Create an alternative name hash algorithm (recreated)

From
Derrick Stolee <stolee@gmail.com>
Date
Dec 4, 2024, 04:56 UTC
Message-ID
<b07c6b94-b1cc-4391-83fd-adc5bb5f92e3@gmail.com>
In-Reply-To
<xmqq1pypfo05.fsf@gitster.g>
On 12/2/24 10:23 PM, Junio C Hamano wrote:
Show 33 quoted lines
> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
> 
>> This series creates a mechanism to select alternative name hashes using a
>> new --name-hash-version=<n> option. The versions are:
>>
>>   1. Version 1 is the default name hash that already exists. This option
>>      focuses on the final bytes of the path to maximize locality for
>>      cross-path deltas.
>>
>>   2. Version 2 is the new path-component hash function suggested by Jonathan
>>      Tan in the previous version (with some modifications). This hash
>>      function essentially computes the v1 name hash of each path component
>>      and then overlays those hashes with a shift to make the parent
>>      directories contribute less to the final hash, but enough to break many
>>      collisions that exist in v1.
>>
>>   3. Version 3 is the hash function that I submitted under the
>>      --full-name-hash feature in the previous versions. This uses a
>>      pseudorandom hash procedure to minimize collisions but at the expense of
>>      losing on locality. This version is implemented in the final patch of
>>      the series mostly for comparison purposes, as it is unlikely to be
>>      selected as a valuable hash function over v2. The final patch could be
>>      omitted from the merged version.
>>
>> See the patches themselves for detailed results in the p5313-pack-objects.sh
>> performance test and the p5314-name-hash.sh test that demonstrates how many
>> collisions occur with each hash function.
> 
> These do not sound like versions but more like variants to me,
> especially if one is expected to perform better than another in some
> cases and worse in some other cases.  Is it expected that JTan's hash
> to perform better than the original and current hash in almost all
> cases (I would not be surprised at all if that were the case)?

There are some cases, such as the Linux kernel repo, that have slightly worse compression using JTan's hash. But the naming conventions in that repo are such that the v1 name hash was already pretty effective for that repo. The Git repository has similar issues. See Patch 5 for detailed analysis of these scenarios using the p5313-pack-objects.sh test script.

It may be possible to adapt some of the collision rate analysis from the test helper in Patch 6 to create a tool that recommends or dynamically selects the hash function that works best for the repo. Such a feature should be delayed until this code is exercised in more places.

Thanks, -Stolee

Previous: Junio C HamanoNext: Junio C Hamano
Message 61 of 93 in “pack-objects: Create an alternative name hash algorithm (recreated)”
  1. 0/7 pack-objects: Create an alternative name hash algorithm (recreated)Derrick Stolee via GitGitGadget, Nov 5, 2024
  2. 1/7 pack-objects: add --full-name-hash optionDerrick Stolee via GitGitGadget, Nov 5, 2024
  3. Taylor BlauNov 21, 2024
  4. Taylor BlauNov 21, 2024
  5. Junio C HamanoNov 21, 2024
  6. Derrick StoleeNov 22, 2024
  7. Derrick StoleeNov 22, 2024
  8. Patrick SteinhardtNov 26, 2024
  9. 2/7 repack: add --full-name-hash optionDerrick Stolee via GitGitGadget, Nov 5, 2024
  10. Taylor BlauNov 21, 2024
  11. Derrick StoleeNov 22, 2024
  12. 3/7 pack-objects: add GIT_TEST_FULL_NAME_HASHDerrick Stolee via GitGitGadget, Nov 5, 2024
  13. Taylor BlauNov 21, 2024
  14. Derrick StoleeNov 22, 2024
  15. Jonathan TanNov 22, 2024
  16. Junio C HamanoNov 22, 2024
  17. Jonathan TanNov 22, 2024
  18. Junio C HamanoNov 25, 2024
  19. Jonathan TanNov 25, 2024
  20. Junio C HamanoNov 26, 2024
  21. Patrick SteinhardtNov 26, 2024
  22. 4/7 git-repack: update usage to match docsDerrick Stolee via GitGitGadget, Nov 5, 2024
  23. Taylor BlauNov 21, 2024
  24. Derrick StoleeNov 22, 2024
  25. 5/7 p5313: add size comparison testDerrick Stolee via GitGitGadget, Nov 5, 2024
  26. Taylor BlauNov 21, 2024
  27. Derrick StoleeNov 22, 2024
  28. Patrick SteinhardtNov 26, 2024
  29. 6/7 pack-objects: disable --full-name-hash when shallowDerrick Stolee via GitGitGadget, Nov 5, 2024
  30. Taylor BlauNov 21, 2024
  31. Derrick StoleeNov 22, 2024
  32. 7/7 test-tool: add helper for name-hash valuesDerrick Stolee via GitGitGadget, Nov 5, 2024
  33. Taylor BlauNov 21, 2024
  34. Jonathan TanNov 22, 2024
  35. Jonathan TanNov 21, 2024
  36. Junio C HamanoNov 22, 2024
  37. Junio C HamanoNov 22, 2024
  38. Derrick StoleeNov 22, 2024
  39. Junio C HamanoNov 24, 2024
  40. Jonathan TanNov 22, 2024
  41. 0/8 pack-objects: Create an alternative name hash algorithm (recreated)Derrick Stolee via GitGitGadget, Dec 2, 2024
  42. 1/8 pack-objects: create new name-hash function versionJonathan Tan via GitGitGadget, Dec 2, 2024
  43. karthik nayakDec 4, 2024
  44. Junio C HamanoDec 4, 2024
  45. karthik nayakDec 5, 2024
  46. Jonathan TanDec 9, 2024
  47. Junio C HamanoDec 10, 2024
  48. 2/8 pack-objects: add --name-hash-version optionDerrick Stolee via GitGitGadget, Dec 2, 2024
  49. karthik nayakDec 4, 2024
  50. 3/8 repack: add --name-hash-version optionDerrick Stolee via GitGitGadget, Dec 2, 2024
  51. karthik nayakDec 4, 2024
  52. 4/8 pack-objects: add GIT_TEST_NAME_HASH_VERSIONDerrick Stolee via GitGitGadget, Dec 2, 2024
  53. karthik nayakDec 4, 2024
  54. Jonathan TanDec 9, 2024
  55. Derrick StoleeDec 20, 2024
  56. 5/8 p5313: add size comparison testDerrick Stolee via GitGitGadget, Dec 2, 2024
  57. 6/8 test-tool: add helper for name-hash valuesDerrick Stolee via GitGitGadget, Dec 2, 2024
  58. 7/8 pack-objects: prevent name hash version changeDerrick Stolee via GitGitGadget, Dec 2, 2024
  59. 8/8 pack-objects: add third name hash versionDerrick Stolee via GitGitGadget, Dec 2, 2024
  60. Junio C HamanoDec 3, 2024
  61. Derrick StoleeDec 4, 2024
  62. Junio C HamanoDec 4, 2024
  63. 0/8 pack-objects: Create an alternative name hash algorithm (recreated)Derrick Stolee via GitGitGadget, Dec 20, 2024
  64. 1/8 pack-objects: create new name-hash function versionJonathan Tan via GitGitGadget, Dec 20, 2024
  65. Taylor BlauJan 22, 2025
  66. 2/8 pack-objects: add --name-hash-version optionDerrick Stolee via GitGitGadget, Dec 20, 2024
  67. Taylor BlauJan 22, 2025
  68. Derrick StoleeJan 24, 2025
  69. 3/8 repack: add --name-hash-version optionDerrick Stolee via GitGitGadget, Dec 20, 2024
  70. Taylor BlauJan 22, 2025
  71. 4/8 pack-objects: add GIT_TEST_NAME_HASH_VERSIONDerrick Stolee via GitGitGadget, Dec 20, 2024
  72. Taylor BlauJan 22, 2025
  73. 5/8 p5313: add size comparison testDerrick Stolee via GitGitGadget, Dec 20, 2024
  74. 6/8 test-tool: add helper for name-hash valuesDerrick Stolee via GitGitGadget, Dec 20, 2024
  75. 7/8 pack-objects: prevent name hash version changeDerrick Stolee via GitGitGadget, Dec 20, 2024
  76. Taylor BlauJan 22, 2025
  77. 8/8 pack-objects: add third name hash versionDerrick Stolee via GitGitGadget, Dec 20, 2024
  78. Taylor BlauJan 22, 2025
  79. Derrick StoleeJan 24, 2025
  80. Derrick StoleeJan 21, 2025
  81. Taylor BlauJan 22, 2025
  82. Derrick StoleeJan 24, 2025
  83. 0/7 pack-objects: Create an alternative name hash algorithm (recreated)Derrick Stolee via GitGitGadget, Jan 27, 2025
  84. 1/7 pack-objects: create new name-hash function versionJonathan Tan via GitGitGadget, Jan 27, 2025
  85. 3/7 repack: add --name-hash-version optionDerrick Stolee via GitGitGadget, Jan 27, 2025
  86. 2/7 pack-objects: add --name-hash-version optionDerrick Stolee via GitGitGadget, Jan 27, 2025
  87. Junio C HamanoJan 27, 2025
  88. Derrick StoleeJan 29, 2025
  89. 4/7 pack-objects: add GIT_TEST_NAME_HASH_VERSIONDerrick Stolee via GitGitGadget, Jan 27, 2025
  90. 5/7 p5313: add size comparison testDerrick Stolee via GitGitGadget, Jan 27, 2025
  91. 6/7 test-tool: add helper for name-hash valuesDerrick Stolee via GitGitGadget, Jan 27, 2025
  92. 7/7 pack-objects: prevent name hash version changeDerrick Stolee via GitGitGadget, Jan 27, 2025
  93. Taylor BlauJan 31, 2025

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.