git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 3/8] repack: add --name-hash-version option

From
karthik nayak <karthik.188@gmail.com>
Date
Dec 4, 2024, 21:15 UTC
Message-ID
<CAOLa=ZQcicLN462PFz_2sFeKKB0VvthPZhgwuR3CM0RSM6Y-Ow@mail.gmail.com>
In-Reply-To
<1947d1bf448d71ccd4a44ef25751bab50784a73e.1733181682.git.gitgitgadget@gmail.com>
"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
[snip]
Show 36 quoted lines
>  DESCRIPTION
>  -----------
> @@ -249,6 +251,36 @@ linkgit:git-multi-pack-index[1]).
>  	Write a multi-pack index (see linkgit:git-multi-pack-index[1])
>  	containing the non-redundant packs.
>
> +--name-hash-version=<n>::
> +	While performing delta compression, Git groups objects that may be
> +	similar based on heuristics using the path to that object. While
> +	grouping objects by an exact path match is good for paths with
> +	many versions, there are benefits for finding delta pairs across
> +	different full paths. Git collects objects by type and then by a
> +	"name hash" of the path and then by size, hoping to group objects
> +	that will compress well together.
> ++
> +The default name hash version is `1`, which prioritizes hash locality by
> +considering the final bytes of the path as providing the maximum magnitude
> +to the hash function. This version excels at distinguishing short paths
> +and finding renames across directories. However, the hash function depends
> +primarily on the final 16 bytes of the path. If there are many paths in
> +the repo that have the same final 16 bytes and differ only by parent
> +directory, then this name-hash may lead to too many collisions and cause
> +poor results. At the moment, this version is required when writing
> +reachability bitmap files with `--write-bitmap-index`.
> ++
> +The name hash version `2` has similar locality features as version `1`,
> +except it considers each path component separately and overlays the hashes
> +with a shift. This still prioritizes the final bytes of the path, but also
> +"salts" the lower bits of the hash using the parent directory names. This
> +method allows for some of the locality benefits of version `1` while
> +breaking most of the collisions from a similarly-named file appearing in
> +many different directories. At the moment, this version is not allowed
> +when writing reachability bitmap files with `--write-bitmap-index` and it
> +will be automatically changed to version `1`.
> +
> +
Nit: I wonder if it'd be nicer to simply point to the documentation in
'Documentation/git-pack-objects.txt'. This would ensure we have
consistent documentation and a single source of truth.
Show 18 quoted lines
>
>  CONFIGURATION
>  -------------
>
> diff --git a/builtin/repack.c b/builtin/repack.c
> index 05e13adb87f..5e7ff919c1a 100644
> --- a/builtin/repack.c
> +++ b/builtin/repack.c
> @@ -39,7 +39,9 @@ static int run_update_server_info = 1;
>  static char *packdir, *packtmp_name, *packtmp;
>
>  static const char *const git_repack_usage[] = {
> -	N_("git repack [<options>]"),
> +	N_("git repack [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\n"
> +	   "[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\n"
> +	   "[--write-midx] [--name-hash-version=<n>]"),
>  	NULL
>  };
So this fixes the mismatch in t0450 which is seen below.
Nit: might be worthwhile adding this in the commit message.
[snip]
Previous: Derrick Stolee via GitGitGadgetNext: Derrick Stolee via GitGitGadget
Message 51 of 93 in “pack-objects: Create an alternative name hash algorithm (recreated)”
  1. 0/7 pack-objects: Create an alternative name hash algorithm (recreated)Derrick Stolee via GitGitGadget, Nov 5, 2024
  2. 1/7 pack-objects: add --full-name-hash optionDerrick Stolee via GitGitGadget, Nov 5, 2024
  3. Taylor BlauNov 21, 2024
  4. Taylor BlauNov 21, 2024
  5. Junio C HamanoNov 21, 2024
  6. Derrick StoleeNov 22, 2024
  7. Derrick StoleeNov 22, 2024
  8. Patrick SteinhardtNov 26, 2024
  9. 2/7 repack: add --full-name-hash optionDerrick Stolee via GitGitGadget, Nov 5, 2024
  10. Taylor BlauNov 21, 2024
  11. Derrick StoleeNov 22, 2024
  12. 3/7 pack-objects: add GIT_TEST_FULL_NAME_HASHDerrick Stolee via GitGitGadget, Nov 5, 2024
  13. Taylor BlauNov 21, 2024
  14. Derrick StoleeNov 22, 2024
  15. Jonathan TanNov 22, 2024
  16. Junio C HamanoNov 22, 2024
  17. Jonathan TanNov 22, 2024
  18. Junio C HamanoNov 25, 2024
  19. Jonathan TanNov 25, 2024
  20. Junio C HamanoNov 26, 2024
  21. Patrick SteinhardtNov 26, 2024
  22. 4/7 git-repack: update usage to match docsDerrick Stolee via GitGitGadget, Nov 5, 2024
  23. Taylor BlauNov 21, 2024
  24. Derrick StoleeNov 22, 2024
  25. 5/7 p5313: add size comparison testDerrick Stolee via GitGitGadget, Nov 5, 2024
  26. Taylor BlauNov 21, 2024
  27. Derrick StoleeNov 22, 2024
  28. Patrick SteinhardtNov 26, 2024
  29. 6/7 pack-objects: disable --full-name-hash when shallowDerrick Stolee via GitGitGadget, Nov 5, 2024
  30. Taylor BlauNov 21, 2024
  31. Derrick StoleeNov 22, 2024
  32. 7/7 test-tool: add helper for name-hash valuesDerrick Stolee via GitGitGadget, Nov 5, 2024
  33. Taylor BlauNov 21, 2024
  34. Jonathan TanNov 22, 2024
  35. Jonathan TanNov 21, 2024
  36. Junio C HamanoNov 22, 2024
  37. Junio C HamanoNov 22, 2024
  38. Derrick StoleeNov 22, 2024
  39. Junio C HamanoNov 24, 2024
  40. Jonathan TanNov 22, 2024
  41. 0/8 pack-objects: Create an alternative name hash algorithm (recreated)Derrick Stolee via GitGitGadget, Dec 2, 2024
  42. 1/8 pack-objects: create new name-hash function versionJonathan Tan via GitGitGadget, Dec 2, 2024
  43. karthik nayakDec 4, 2024
  44. Junio C HamanoDec 4, 2024
  45. karthik nayakDec 5, 2024
  46. Jonathan TanDec 9, 2024
  47. Junio C HamanoDec 10, 2024
  48. 2/8 pack-objects: add --name-hash-version optionDerrick Stolee via GitGitGadget, Dec 2, 2024
  49. karthik nayakDec 4, 2024
  50. 3/8 repack: add --name-hash-version optionDerrick Stolee via GitGitGadget, Dec 2, 2024
  51. karthik nayakDec 4, 2024
  52. 4/8 pack-objects: add GIT_TEST_NAME_HASH_VERSIONDerrick Stolee via GitGitGadget, Dec 2, 2024
  53. karthik nayakDec 4, 2024
  54. Jonathan TanDec 9, 2024
  55. Derrick StoleeDec 20, 2024
  56. 5/8 p5313: add size comparison testDerrick Stolee via GitGitGadget, Dec 2, 2024
  57. 6/8 test-tool: add helper for name-hash valuesDerrick Stolee via GitGitGadget, Dec 2, 2024
  58. 7/8 pack-objects: prevent name hash version changeDerrick Stolee via GitGitGadget, Dec 2, 2024
  59. 8/8 pack-objects: add third name hash versionDerrick Stolee via GitGitGadget, Dec 2, 2024
  60. Junio C HamanoDec 3, 2024
  61. Derrick StoleeDec 4, 2024
  62. Junio C HamanoDec 4, 2024
  63. 0/8 pack-objects: Create an alternative name hash algorithm (recreated)Derrick Stolee via GitGitGadget, Dec 20, 2024
  64. 1/8 pack-objects: create new name-hash function versionJonathan Tan via GitGitGadget, Dec 20, 2024
  65. Taylor BlauJan 22, 2025
  66. 2/8 pack-objects: add --name-hash-version optionDerrick Stolee via GitGitGadget, Dec 20, 2024
  67. Taylor BlauJan 22, 2025
  68. Derrick StoleeJan 24, 2025
  69. 3/8 repack: add --name-hash-version optionDerrick Stolee via GitGitGadget, Dec 20, 2024
  70. Taylor BlauJan 22, 2025
  71. 4/8 pack-objects: add GIT_TEST_NAME_HASH_VERSIONDerrick Stolee via GitGitGadget, Dec 20, 2024
  72. Taylor BlauJan 22, 2025
  73. 5/8 p5313: add size comparison testDerrick Stolee via GitGitGadget, Dec 20, 2024
  74. 6/8 test-tool: add helper for name-hash valuesDerrick Stolee via GitGitGadget, Dec 20, 2024
  75. 7/8 pack-objects: prevent name hash version changeDerrick Stolee via GitGitGadget, Dec 20, 2024
  76. Taylor BlauJan 22, 2025
  77. 8/8 pack-objects: add third name hash versionDerrick Stolee via GitGitGadget, Dec 20, 2024
  78. Taylor BlauJan 22, 2025
  79. Derrick StoleeJan 24, 2025
  80. Derrick StoleeJan 21, 2025
  81. Taylor BlauJan 22, 2025
  82. Derrick StoleeJan 24, 2025
  83. 0/7 pack-objects: Create an alternative name hash algorithm (recreated)Derrick Stolee via GitGitGadget, Jan 27, 2025
  84. 1/7 pack-objects: create new name-hash function versionJonathan Tan via GitGitGadget, Jan 27, 2025
  85. 3/7 repack: add --name-hash-version optionDerrick Stolee via GitGitGadget, Jan 27, 2025
  86. 2/7 pack-objects: add --name-hash-version optionDerrick Stolee via GitGitGadget, Jan 27, 2025
  87. Junio C HamanoJan 27, 2025
  88. Derrick StoleeJan 29, 2025
  89. 4/7 pack-objects: add GIT_TEST_NAME_HASH_VERSIONDerrick Stolee via GitGitGadget, Jan 27, 2025
  90. 5/7 p5313: add size comparison testDerrick Stolee via GitGitGadget, Jan 27, 2025
  91. 6/7 test-tool: add helper for name-hash valuesDerrick Stolee via GitGitGadget, Jan 27, 2025
  92. 7/7 pack-objects: prevent name hash version changeDerrick Stolee via GitGitGadget, Jan 27, 2025
  93. Taylor BlauJan 31, 2025

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.