{"thread":{"id":"62447","subject":"[PATCH 0/7] pack-objects: Create an alternative name hash algorithm (recreated)","startedAt":"2024-11-05T03:05:12Z","lastAt":"2025-01-31T21:39:43Z","messageCount":93,"participants":["Derrick Stolee via GitGitGadget","Taylor Blau","Junio C Hamano","Jonathan Tan","Derrick Stolee","Patrick Steinhardt","Jonathan Tan via GitGitGadget","karthik nayak"],"isPatch":true,"patchVersion":1,"patchTotal":7},"messages":[{"id":"506590","messageId":"pull.1823.git.1730775907.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":null,"subject":"[PATCH 0/7] pack-objects: Create an alternative name hash algorithm (recreated)","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-11-05T03:05:00Z","receivedAt":"2024-11-05T03:05:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is a recreation of the topic in [1] that was closed. (I force-pushed my\nbranch and GitHub won't let me reopen the PR for GitGitGadget to create this\nas v3.)\n\n[1]\nhttps://lore.kernel.org/git/pull.1785.v2.git.1726692381.gitgitgadget@gmail.com/\n\nI've been focused recently on understanding and mitigating the growth of a\nfew internal repositories. Some of these are growing much larger than\nexpected for the number of contributors, and there are multiple aspects to\nwhy this growth is so large.\n\nThis is part of the RFC I submitted [2] involving the path-walk API, though\nthis doesn't use the path-walk API directly. In full repack cases, it seems\nthat the --full-name-hash option gets nearly as good compression as the\n--path-walk option introduced in that series. I continue to work on that\nfeature as well, so we can review it after this series is complete.\n\n[2]\nhttps://lore.kernel.org/git/pull.1786.git.1725935335.gitgitgadget@gmail.com/\n\nThe main issue plaguing these repositories is that deltas are not being\ncomputed against objects that appear at the same path. While the size of\nthese files at tip is one aspect of growth that would prevent this issue,\nthe changes to these files are reasonable and should result in good delta\ncompression. However, Git is not discovering the connections across\ndifferent versions of the same file.\n\nOne way to find some improvement in these repositories is to increase the\nwindow size, which was an initial indicator that the delta compression could\nbe improved, but was not a clear indicator. After some digging (and\nprototyping some analysis tools) the main discovery was that the current\nname-hash algorithm only considers the last 16 characters in the path name\nand has some naturally-occurring collisions within that scope.\n\nThis series introduces a new name-hash algorithm, but does not replace the\nexisting one. There are cases, such as packing a single snapshot of a\nrepository, where the existing algorithm outperforms the new one.\n\nHowever, my findings show that when a repository has many versions of files\nat the same path (and especially when there are many name-hash collisions)\nthen there are significant gains to be made using the new algorithm.\n\n(This table is updated in v2 with even more private examples that were found\nwhile communicating findings internally.)\n\n| Repo     | Standard Repack | With --full-name-hash |\n|----------|-----------------|-----------------------|\n| fluentui |         438 MB  |               168 MB  |\n| Repo B   |       6,255 MB  |               829 MB  |\n| Repo C   |      37,737 MB  |             7,125 MB  |\n| Repo D   |     130,049 MB  |             6,190 MB  |\n| Repo E   |     100,957 MB  |            22,979 MB  |\n| Repo F   |       8,308 MB  |               746 MB  |\n| Repo G   |       4,329 MB  |             3,643 MB  |\n\n\nSince there has been some interest in these repacking results since [3] was\npublished, I'll say that \"Repo D\" is the one mentioned in that post. The\n--path-walk repack further improves the results to under 4GB.\n\n[3]\nhttps://www.jonathancreamer.com/how-we-shrunk-our-git-repo-size-by-94-percent/\n\nI include Repo G here as an example where the improvement is less drastic,\nsince this repo does not demonstrate a very high rate of name-hash\ncollisions; the collisions that exist seem to be in paths that are not\nchanged very often. Thus, the standard name-hash algorithm is nearly as\neffective in these full repacks.\n\nThe main change in this series is in patch 1, which adds the algorithm and\nthe option to 'git pack-objects' and 'git repack'. The remaining patches are\nfocused on creating more evidence around the value of the new name-hash\nalgorithm and its effects on the packfiles created with it.\n\nI will also try to make clear that I've been focused on client-side\nperformance and size concerns. Based on discussions in v1, it appears that\nthe following is true:\n\n * This feature is completely orthogonal to delta islands.\n\n * Changing the name-hash function can lead to compatibility issues with\n   .bitmap files, as they store a name-hash value. Without augmenting the\n   data structure to indicate which name-hash value was used at write time,\n   the full-name-hash values should not be stored in the .bitmap files or\n   used when reading .bitmap files and other objects. Thus, the\n   full-name-hash is marked as incompatible with bitmaps for now. (In this\n   version, the 'git pack-objects' process will not fail but will prefer the\n   standard name hash algorithm.)\n\n * The --path-walk option is likely incompatible with the delta-islands\n   feature without significant work, so this suggests that the\n   --full-name-hash is a better long-term solution for servers. This would\n   still require the .bitmap modifications to make it work, but it's a\n   smaller leap to get there.\n\nThanks, -Stolee\n\n\nUPDATES SINCE PREVIOUS VERSION\n==============================\n\nThis is recreated after pursuing the path-walk API with 'git pack-objects\n--path-walk' [4] and also cutting the series into fewer patches and only\nincluding the path-walk API in [5].\n\n[4]\nhttps://lore.kernel.org/git/pull.1813.v2.git.1729431810.gitgitgadget@gmail.com/\n\n[5]\nhttps://lore.kernel.org/git/pull.1818.git.1730356023.gitgitgadget@gmail.com/\n\nThis re-roll has these changes:\n\n * The performance script is updated to include a simulation of a depth-1\n   shallow clone.\n\n * The performance numbers in the commit messages are updated with the\n   latest results and include more example repositories.\n\n * Small style nits, especially around whitespace.\n\n * To prevent the --full-name-hash option from having compatibility issues\n   with .bitmap files, the previous version would cause the 'git\n   pack-objects' process to fail. Now, the process succeeds, but the\n   --full-name-hash option is disabled with a warning.\n\n * These patches are rebased on a recent copy of 'master' and thus there are\n   some new tests that need adjustment to work with\n   GIT_TEST_FULL_NAME_HASH=1. They all care about an exact match with stderr\n   and the warning to disable --full-name-hash causes a mismatch.\n\nHere is the range-diff since the previous version:\n\n1:  9c8f8f35f34 ! 1:  812257e197c pack-objects: add --full-name-hash option\n    @@ Commit message\n         Future changes could include making --full-name-hash implied by a config\n         value or even implied by default during a full repack.\n     \n    +    It is important to point out that the name hash value is stored in the\n    +    .bitmap file format, so we must disable the --full-name-hash option when\n    +    bitmaps are being read or written. Later, the bitmap format could be\n    +    updated to be aware of the name hash version so deltas can be quickly\n    +    computed across the bitmapped/not-bitmapped boundary.\n    +\n         Signed-off-by: Derrick Stolee <stolee@gmail.com>\n     \n      ## Documentation/git-pack-objects.txt ##\n    @@ builtin/pack-objects.c: static void add_cruft_object_entry(const struct object_i\n      \t\t\t\t\t    0, name && no_try_delta(name),\n      \t\t\t\t\t    pack, offset);\n      \t}\n    -@@ builtin/pack-objects.c: int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n    +@@ builtin/pack-objects.c: int cmd_pack_objects(int argc,\n      \t\tOPT_STRING_LIST(0, \"uri-protocol\", &uri_protocols,\n      \t\t\t\tN_(\"protocol\"),\n      \t\t\t\tN_(\"exclude any configured uploadpack.blobpackfileuri with this protocol\")),\n    @@ builtin/pack-objects.c: int cmd_pack_objects(int argc, const char **argv, const\n      \t\tOPT_END(),\n      \t};\n      \n    -@@ builtin/pack-objects.c: int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n    +@@ builtin/pack-objects.c: int cmd_pack_objects(int argc,\n      \tif (pack_to_stdout || !rev_list_all)\n      \t\twrite_bitmap_index = 0;\n      \n    -+\tif (write_bitmap_index && use_full_name_hash)\n    -+\t\tdie(_(\"currently, the --full-name-hash option is incompatible with --write-bitmap-index\"));\n    ++\tif (write_bitmap_index && use_full_name_hash) {\n    ++\t\twarning(_(\"currently, the --full-name-hash option is incompatible with --write-bitmap-index\"));\n    ++\t\tuse_full_name_hash = 0;\n    ++\t}\n     +\n      \tif (use_delta_islands)\n      \t\tstrvec_push(&rp, \"--topo-order\");\n    @@ builtin/repack.c: static void prepare_pack_objects(struct child_process *cmd,\n      \tif (args->local)\n      \t\tstrvec_push(&cmd->args,  \"--local\");\n      \tif (args->quiet)\n    -@@ builtin/repack.c: int cmd_repack(int argc, const char **argv, const char *prefix)\n    +@@ builtin/repack.c: int cmd_repack(int argc,\n      \t\t\t\tN_(\"pass --no-reuse-delta to git-pack-objects\")),\n      \t\tOPT_BOOL('F', NULL, &po_args.no_reuse_object,\n      \t\t\t\tN_(\"pass --no-reuse-object to git-pack-objects\")),\n    @@ t/t5300-pack-object.sh: do\n     +# TODO: Make these compatible in the future and replace this test with the\n     +# expected behavior when both are specified.\n     +test_expect_success '--full-name-hash and --write-bitmap-index are incompatible' '\n    -+\ttest_must_fail git pack-objects base --all \\\n    -+\t\t--full-name-hash --write-bitmap-index 2>err &&\n    -+\tgrep incompatible err &&\n    ++\tgit pack-objects base --all --full-name-hash --write-bitmap-index 2>err &&\n    ++\ttest_grep incompatible err &&\n     +\n     +\t# --stdout option silently removes --write-bitmap-index\n    -+\tgit pack-objects --stdout --all --full-name-hash --write-bitmap-index >out\n    ++\tgit pack-objects --stdout --all --full-name-hash --write-bitmap-index >out 2>err &&\n    ++\t! test_grep incompatible err\n     +'\n     +\n      test_done\n2:  612dbd1951b ! 2:  93395c93347 repack: test --full-name-hash option\n    @@ Metadata\n     Author: Derrick Stolee <stolee@gmail.com>\n     \n      ## Commit message ##\n    -    repack: test --full-name-hash option\n    +    repack: add --full-name-hash option\n     \n         The new '--full-name-hash' option for 'git repack' is a simple\n         pass-through to the underlying 'git pack-objects' subcommand. However,\n    @@ t/test-lib-functions.sh: test_subcommand () {\n      \tfi\n      }\n      \n    -+\n     +# Check that the given subcommand was run with the given set of\n     +# arguments in order (but with possible extra arguments).\n     +#\n3:  e173de67b6a ! 3:  259734e0bce pack-objects: add GIT_TEST_FULL_NAME_HASH\n    @@ Commit message\n         now, do the minimal change to make the test work by disabling the test\n         variable.\n     \n    +    Third, there are some tests that compare the exact output of a 'git\n    +    pack-objects' process when using bitmaps. The warning that disables the\n    +    --full-name-hash option causes these tests to fail. Disable the\n    +    environment variable to get around this issue.\n    +\n         Signed-off-by: Derrick Stolee <stolee@gmail.com>\n     \n      ## builtin/pack-objects.c ##\n    @@ builtin/pack-objects.c: struct configured_exclusion {\n      \n      static inline uint32_t pack_name_hash_fn(const char *name)\n      {\n    -@@ builtin/pack-objects.c: int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n    +@@ builtin/pack-objects.c: int cmd_pack_objects(int argc,\n      \tif (pack_to_stdout || !rev_list_all)\n      \t\twrite_bitmap_index = 0;\n      \n    --\tif (write_bitmap_index && use_full_name_hash)\n    -+\tif (write_bitmap_index && use_full_name_hash > 0)\n    - \t\tdie(_(\"currently, the --full-name-hash option is incompatible with --write-bitmap-index\"));\n    +-\tif (write_bitmap_index && use_full_name_hash) {\n     +\tif (use_full_name_hash < 0)\n     +\t\tuse_full_name_hash = git_env_bool(\"GIT_TEST_FULL_NAME_HASH\", 0);\n    - \n    - \tif (use_delta_islands)\n    - \t\tstrvec_push(&rp, \"--topo-order\");\n    ++\n    ++\tif (write_bitmap_index && use_full_name_hash > 0) {\n    + \t\twarning(_(\"currently, the --full-name-hash option is incompatible with --write-bitmap-index\"));\n    + \t\tuse_full_name_hash = 0;\n    + \t}\n     \n      ## ci/run-build-and-tests.sh ##\n     @@ ci/run-build-and-tests.sh: linux-TEST-vars)\n    @@ t/README: a test and then fails then the whole test run will abort. This can hel\n      ------------\n      \n     \n    + ## t/t5310-pack-bitmaps.sh ##\n    +@@ t/t5310-pack-bitmaps.sh: test_bitmap_cases () {\n    + \t\t\tcat >expect <<-\\EOF &&\n    + \t\t\terror: missing value for '\\''pack.preferbitmaptips'\\''\n    + \t\t\tEOF\n    ++\n    ++\t\t\t# Disable --full-name-hash test due to stderr comparison.\n    ++\t\t\tGIT_TEST_FULL_NAME_HASH=0 \\\n    + \t\t\tgit repack -adb 2>actual &&\n    + \t\t\ttest_cmp expect actual\n    + \t\t)\n    +\n    + ## t/t5333-pseudo-merge-bitmaps.sh ##\n    +@@ t/t5333-pseudo-merge-bitmaps.sh: test_expect_success 'bitmapPseudoMerge.stableThreshold creates stable groups' '\n    + '\n    + \n    + test_expect_success 'out of order thresholds are rejected' '\n    ++\t# Disable this option to avoid stderr message\n    ++\tGIT_TEST_FULL_NAME_HASH=0 &&\n    ++\texport GIT_TEST_FULL_NAME_HASH &&\n    ++\n    + \ttest_must_fail git \\\n    + \t\t-c bitmapPseudoMerge.test.pattern=\"refs/*\" \\\n    + \t\t-c bitmapPseudoMerge.test.threshold=1.month.ago \\\n    +\n      ## t/t5510-fetch.sh ##\n     @@ t/t5510-fetch.sh: test_expect_success 'all boundary commits are excluded' '\n      \ttest_tick &&\n    @@ t/t6020-bundle-misc.sh: test_expect_success 'create bundle with --since option'\n      \t\t--since \"Thu Apr 7 15:27:00 2005 -0700\" \\\n      \t\t--all &&\n      \n    +\n    + ## t/t7406-submodule-update.sh ##\n    +@@ t/t7406-submodule-update.sh: test_expect_success 'submodule update --quiet passes quietness to fetch with a s\n    + \t) &&\n    + \tgit clone super4 super5 &&\n    + \t(cd super5 &&\n    ++\t # This test var can mess with the stderr output checked in this test.\n    ++\t GIT_TEST_FULL_NAME_HASH=0 \\\n    + \t git submodule update --quiet --init --depth=1 submodule3 >out 2>err &&\n    + \t test_must_be_empty out &&\n    + \t test_must_be_empty err\n    +\n    + ## t/t7700-repack.sh ##\n    +@@ t/t7700-repack.sh: test_expect_success 'no bitmaps created if .keep files present' '\n    + \tkeep=${pack%.pack}.keep &&\n    + \ttest_when_finished \"rm -f \\\"\\$keep\\\"\" &&\n    + \t>\"$keep\" &&\n    ++\n    ++\t# Disable --full-name-hash test due to stderr comparison.\n    ++\tGIT_TEST_FULL_NAME_HASH=0 \\\n    + \tgit -C bare.git repack -ad 2>stderr &&\n    + \ttest_must_be_empty stderr &&\n    + \tfind bare.git/objects/pack/ -type f -name \"*.bitmap\" >actual &&\n    +@@ t/t7700-repack.sh: test_expect_success 'auto-bitmaps do not complain if unavailable' '\n    + \tblob=$(test-tool genrandom big $((1024*1024)) |\n    + \t       git -C bare.git hash-object -w --stdin) &&\n    + \tgit -C bare.git update-ref refs/tags/big $blob &&\n    ++\n    ++\t# Disable --full-name-hash test due to stderr comparison.\n    ++\tGIT_TEST_FULL_NAME_HASH=0 \\\n    + \tgit -C bare.git repack -ad 2>stderr &&\n    + \ttest_must_be_empty stderr &&\n    + \tfind bare.git/objects/pack -type f -name \"*.bitmap\" >actual &&\n4:  543382b2702 = 4:  65784f85bce git-repack: update usage to match docs\n5:  4d2381a19c4 ! 5:  c14ef6879e4 p5313: add size comparison test\n    @@ Commit message\n     \n         Checked out at the parent of [2], I see the following statistics:\n     \n    -    Test                                           this tree\n    -    ------------------------------------------------------------------\n    -    5313.2: thin pack                              0.02(0.01+0.01)\n    -    5313.3: thin pack size                                    1.1K\n    -    5313.4: thin pack with --full-name-hash        0.02(0.01+0.00)\n    -    5313.5: thin pack size with --full-name-hash              3.0K\n    -    5313.6: big pack                               1.65(3.35+0.24)\n    -    5313.7: big pack size                                    58.0M\n    -    5313.8: big pack with --full-name-hash         1.53(2.52+0.18)\n    -    5313.9: big pack size with --full-name-hash              57.6M\n    -    5313.10: repack                                176.52(706.60+3.53)\n    -    5313.11: repack size                                    446.7K\n    -    5313.12: repack with --full-name-hash          37.47(134.18+3.06)\n    -    5313.13: repack size with --full-name-hash              183.1K\n    -\n    -    Note that this demonstrates a 3x size _increase_ in the case that\n    -    simulates a small \"git push\". The size change is neutral on the case of\n    -    pushing the difference between HEAD and HEAD~1000.\n    -\n    -    However, the full repack case is both faster and more efficient.\n    +    Test                                               HEAD\n    +    ---------------------------------------------------------------------\n    +    5313.2: thin pack                                  0.37(0.43+0.02)\n    +    5313.3: thin pack size                                        1.2M\n    +    5313.4: thin pack with --full-name-hash            0.06(0.09+0.02)\n    +    5313.5: thin pack size with --full-name-hash                 20.4K\n    +    5313.6: big pack                                   2.01(7.73+0.23)\n    +    5313.7: big pack size                                        20.3M\n    +    5313.8: big pack with --full-name-hash             1.32(2.77+0.27)\n    +    5313.9: big pack size with --full-name-hash                  19.9M\n    +    5313.10: shallow fetch pack                        1.40(3.01+0.08)\n    +    5313.11: shallow pack size                                   34.4M\n    +    5313.12: shallow pack with --full-name-hash        1.08(1.25+0.14)\n    +    5313.13: shallow pack size with --full-name-hash             35.4M\n    +    5313.14: repack                                    90.70(672.88+2.46)\n    +    5313.15: repack size                                        439.6M\n    +    5313.16: repack with --full-name-hash              18.53(123.41+2.53)\n    +    5313.17: repack size with --full-name-hash                  169.7M\n    +\n    +    In this case, we see positive behaviors such as a significant shrink in\n    +    the size of the thin pack and full repack. The big pack is slightly\n    +    smaller with --full-name-hash than without. The shallow pack is slightly\n    +    larger with --full-name-hash.\n    +\n    +    In the case of the Git repository, these numbers show some of the issues\n    +    with this approach:\n    +\n    +    Test                                               HEAD\n    +    --------------------------------------------------------------------\n    +    5313.2: thin pack                                  0.00(0.00+0.00)\n    +    5313.3: thin pack size                                         589\n    +    5313.4: thin pack with --full-name-hash            0.00(0.00+0.00)\n    +    5313.5: thin pack size with --full-name-hash                 14.9K\n    +    5313.6: big pack                                   2.07(3.57+0.17)\n    +    5313.7: big pack size                                        17.6M\n    +    5313.8: big pack with --full-name-hash             2.00(3.07+0.19)\n    +    5313.9: big pack size with --full-name-hash                  17.9M\n    +    5313.10: shallow fetch pack                        1.41(2.23+0.06)\n    +    5313.11: shallow pack size                                   12.1M\n    +    5313.12: shallow pack with --full-name-hash        1.22(1.66+0.04)\n    +    5313.13: shallow pack size with --full-name-hash             12.4M\n    +    5313.14: repack                                    15.75(89.29+1.54)\n    +    5313.15: repack size                                        126.4M\n    +    5313.16: repack with --full-name-hash              15.56(89.78+1.32)\n    +    5313.17: repack size with --full-name-hash                  126.0M\n    +\n    +    The thin pack that simulates a push is much worse with --full-name-hash\n    +    in this case. The name hash values are doing a lot to assist with delta\n    +    bases, it seems. The big pack and shallow clone cases are slightly worse\n    +    with the --full-name-hash option. Only the full repack gains some\n    +    benefits in size.\n    +\n    +    The results are similar with the nodejs/node repo:\n    +\n    +    Test                                               HEAD\n    +    ---------------------------------------------------------------------\n    +    5313.2: thin pack                                  0.01(0.01+0.00)\n    +    5313.3: thin pack size                                        1.6K\n    +    5313.4: thin pack with --full-name-hash            0.01(0.00+0.00)\n    +    5313.5: thin pack size with --full-name-hash                  3.1K\n    +    5313.6: big pack                                   4.26(8.03+0.24)\n    +    5313.7: big pack size                                        56.0M\n    +    5313.8: big pack with --full-name-hash             4.16(6.55+0.22)\n    +    5313.9: big pack size with --full-name-hash                  56.2M\n    +    5313.10: shallow fetch pack                        7.67(11.80+0.29)\n    +    5313.11: shallow pack size                                  104.6M\n    +    5313.12: shallow pack with --full-name-hash        7.52(9.65+0.23)\n    +    5313.13: shallow pack size with --full-name-hash            105.9M\n    +    5313.14: repack                                    71.22(317.61+3.95)\n    +    5313.15: repack size                                        739.9M\n    +    5313.16: repack with --full-name-hash              48.85(267.02+3.72)\n    +    5313.17: repack size with --full-name-hash                  793.5M\n    +\n    +    The Linux kernel repository was the initial target of the default name\n    +    hash value, and its naming conventions are practically build to take the\n    +    most advantage of the default name hash values:\n    +\n    +    Test                                               HEAD\n    +    -------------------------------------------------------------------------\n    +    5313.2: thin pack                                  0.15(0.01+0.03)\n    +    5313.3: thin pack size                                        4.6K\n    +    5313.4: thin pack with --full-name-hash            0.03(0.02+0.01)\n    +    5313.5: thin pack size with --full-name-hash                  6.8K\n    +    5313.6: big pack                                   18.51(33.74+0.95)\n    +    5313.7: big pack size                                       201.1M\n    +    5313.8: big pack with --full-name-hash             16.01(29.81+0.88)\n    +    5313.9: big pack size with --full-name-hash                 202.1M\n    +    5313.10: shallow fetch pack                        11.49(17.61+0.54)\n    +    5313.11: shallow pack size                                  269.2M\n    +    5313.12: shallow pack with --full-name-hash        11.24(15.25+0.56)\n    +    5313.13: shallow pack size with --full-name-hash            269.8M\n    +    5313.14: repack                                    1001.25(2271.06+38.86)\n    +    5313.15: repack size                                          2.5G\n    +    5313.16: repack with --full-name-hash              625.75(1941.96+36.09)\n    +    5313.17: repack size with --full-name-hash                    2.6G\n    +\n    +    Finally, an internal Javascript repo of moderate size shows significant\n    +    gains when repacking with --full-name-hash due to it having many name\n    +    hash collisions. However, it's worth noting that only the full repack\n    +    case has enough improvement to be worth it. But the improvements are\n    +    significant: 6.4 GB to 862 MB.\n    +\n    +    Test                                               HEAD\n    +    --------------------------------------------------------------------------\n    +    5313.2: thin pack                                  0.03(0.02+0.00)\n    +    5313.3: thin pack size                                        1.2K\n    +    5313.4: thin pack with --full-name-hash            0.03(0.03+0.00)\n    +    5313.5: thin pack size with --full-name-hash                  2.6K\n    +    5313.6: big pack                                   2.20(3.23+0.30)\n    +    5313.7: big pack size                                       130.7M\n    +    5313.8: big pack with --full-name-hash             2.33(3.17+0.34)\n    +    5313.9: big pack size with --full-name-hash                 131.0M\n    +    5313.10: shallow fetch pack                        3.56(6.02+0.32)\n    +    5313.11: shallow pack size                                   44.5M\n    +    5313.12: shallow pack with --full-name-hash        2.94(3.94+0.32)\n    +    5313.13: shallow pack size with --full-name-hash             45.3M\n    +    5313.14: repack                                    2435.22(12523.11+23.53)\n    +    5313.15: repack size                                          6.4G\n    +    5313.16: repack with --full-name-hash              473.25(1805.11+17.22)\n    +    5313.17: repack size with --full-name-hash                  861.9M\n    +\n    +    These tests demonstrate that it is important to be careful about which\n    +    cases are best for using the --full-name-hash option.\n     \n         Signed-off-by: Derrick Stolee <stolee@gmail.com>\n     \n    @@ t/perf/p5313-pack-objects.sh (new)\n     +\t^$(git rev-parse HEAD~1)\n     +\tEOF\n     +\n    -+\tcat >in-big <<-EOF\n    ++\tcat >in-big <<-EOF &&\n     +\t$(git rev-parse HEAD)\n     +\t^$(git rev-parse HEAD~1000)\n     +\tEOF\n    ++\n    ++\tcat >in-shallow <<-EOF\n    ++\t$(git rev-parse HEAD)\n    ++\t--shallow $(git rev-parse HEAD)\n    ++\tEOF\n     +'\n     +\n     +test_perf 'thin pack' '\n    @@ t/perf/p5313-pack-objects.sh (new)\n     +\ttest_file_size out\n     +'\n     +\n    ++test_perf 'shallow fetch pack' '\n    ++\tgit pack-objects --stdout --revs --sparse --shallow <in-shallow >out\n    ++'\n    ++\n    ++test_size 'shallow pack size' '\n    ++\ttest_file_size out\n    ++'\n    ++\n    ++test_perf 'shallow pack with --full-name-hash' '\n    ++\tgit pack-objects --stdout --revs --sparse --shallow --full-name-hash <in-shallow >out\n    ++'\n    ++\n    ++test_size 'shallow pack size with --full-name-hash' '\n    ++\ttest_file_size out\n    ++'\n    ++\n     +test_perf 'repack' '\n     +\tgit repack -adf\n     +'\n-:  ----------- > 6:  b8a055cb196 pack-objects: disable --full-name-hash when shallow\n6:  80ba362f256 = 7:  ab341dd0e58 test-tool: add helper for name-hash values\n\n\nDerrick Stolee (7):\n  pack-objects: add --full-name-hash option\n  repack: add --full-name-hash option\n  pack-objects: add GIT_TEST_FULL_NAME_HASH\n  git-repack: update usage to match docs\n  p5313: add size comparison test\n  pack-objects: disable --full-name-hash when shallow\n  test-tool: add helper for name-hash values\n\n Documentation/git-pack-objects.txt |  3 +-\n Documentation/git-repack.txt       |  4 +-\n Makefile                           |  1 +\n builtin/pack-objects.c             | 34 +++++++++--\n builtin/repack.c                   |  9 ++-\n ci/run-build-and-tests.sh          |  1 +\n pack-objects.h                     | 21 +++++++\n t/README                           |  4 ++\n t/helper/test-name-hash.c          | 24 ++++++++\n t/helper/test-tool.c               |  1 +\n t/helper/test-tool.h               |  1 +\n t/perf/p5313-pack-objects.sh       | 95 ++++++++++++++++++++++++++++++\n t/perf/p5314-name-hash.sh          | 41 +++++++++++++\n t/t0450/txt-help-mismatches        |  1 -\n t/t5300-pack-object.sh             | 15 +++++\n t/t5310-pack-bitmaps.sh            | 29 +++++++++\n t/t5333-pseudo-merge-bitmaps.sh    |  4 ++\n t/t5510-fetch.sh                   |  7 ++-\n t/t5616-partial-clone.sh           | 26 +++++++-\n t/t6020-bundle-misc.sh             |  6 +-\n t/t7406-submodule-update.sh        |  2 +\n t/t7700-repack.sh                  | 13 ++++\n t/test-lib-functions.sh            | 26 ++++++++\n 23 files changed, 355 insertions(+), 13 deletions(-)\n create mode 100644 t/helper/test-name-hash.c\n create mode 100755 t/perf/p5313-pack-objects.sh\n create mode 100755 t/perf/p5314-name-hash.sh\n\n\nbase-commit: 8f8d6eee531b3fa1a8ef14f169b0cb5035f7a772\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1823%2Fderrickstolee%2Ffull-name-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1823/derrickstolee/full-name-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1823\n-- \ngitgitgadget\n"},{"id":"506592","messageId":"812257e197cfe30bd0d3c68ea6ec0d062631185f.1730775907.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.git.1730775907.gitgitgadget@gmail.com","subject":"[PATCH 1/7] pack-objects: add --full-name-hash option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-11-05T03:05:01Z","receivedAt":"2024-11-05T03:05:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nThe pack_name_hash() method has not been materially changed since it was\nintroduced in ce0bd64299a (pack-objects: improve path grouping\nheuristics., 2006-06-05). The intention here is to group objects by path\nname, but also attempt to group similar file types together by making\nthe most-significant digits of the hash be focused on the final\ncharacters.\n\nHere's the crux of the implementation:\n\n\t/*\n\t * This effectively just creates a sortable number from the\n\t * last sixteen non-whitespace characters. Last characters\n\t * count \"most\", so things that end in \".c\" sort together.\n\t */\n\twhile ((c = *name++) != 0) {\n\t\tif (isspace(c))\n\t\t\tcontinue;\n\t\thash = (hash >> 2) + (c << 24);\n\t}\n\nAs the comment mentions, this only cares about the last sixteen\nnon-whitespace characters. This cause some filenames to collide more\nthan others. Here are some examples that I've seen while investigating\nrepositories that are growing more than they should be:\n\n * \"/CHANGELOG.json\" is 15 characters, and is created by the beachball\n   [1] tool. Only the final character of the parent directory can\n   differntiate different versions of this file, but also only the two\n   most-significant digits. If that character is a letter, then this is\n   always a collision. Similar issues occur with the similar\n   \"/CHANGELOG.md\" path, though there is more opportunity for\n   differences in the parent directory.\n\n * Localization files frequently have common filenames but differentiate\n   via parent directories. In C#, the name \"/strings.resx.lcl\" is used\n   for these localization files and they will all collide in name-hash.\n\n[1] https://github.com/microsoft/beachball\n\nI've come across many other examples where some internal tool uses a\ncommon name across multiple directories and is causing Git to repack\npoorly due to name-hash collisions.\n\nIt is clear that the existing name-hash algorithm is optimized for\nrepositories with short path names, but also is optimized for packing a\nsingle snapshot of a repository, not a repository with many versions of\nthe same file. In my testing, this has proven out where the name-hash\nalgorithm does a good job of finding peer files as delta bases when\nunable to use a historical version of that exact file.\n\nHowever, for repositories that have many versions of most files and\ndirectories, it is more important that the objects that appear at the\nsame path are grouped together.\n\nCreate a new pack_full_name_hash() method and a new --full-name-hash\noption for 'git pack-objects' to call that method instead. Add a simple\npass-through for 'git repack --full-name-hash' for additional testing in\nthe context of a full repack, where I expect this will be most\neffective.\n\nThe hash algorithm is as simple as possible to be reasonably effective:\nfor each character of the path string, add a multiple of that character\nand a large prime number (chosen arbitrarily, but intended to be large\nrelative to the size of a uint32_t). Then, shift the current hash value\nto the right by 5, with overlap. The addition and shift parameters are\nstandard mechanisms for creating hard-to-predict behaviors in the bits\nof the resulting hash.\n\nThis is not meant to be cryptographic at all, but uniformly distributed\nacross the possible hash values. This creates a hash that appears\npseudorandom. There is no ability to consider similar file types as\nbeing close to each other.\n\nIn a later change, a test-tool will be added so the effectiveness of\nthis hash can be demonstrated directly.\n\nFor now, let's consider how effective this mechanism is when repacking a\nrepository with and without the --full-name-hash option. Specifically,\nlet's use 'git repack -adf [--full-name-hash]' as our test.\n\nOn the Git repository, we do not expect much difference. All path names\nare short. This is backed by our results:\n\n| Stage                 | Pack Size | Repack Time |\n|-----------------------|-----------|-------------|\n| After clone           | 260 MB    | N/A         |\n| Standard Repack       | 127MB     | 106s        |\n| With --full-name-hash | 126 MB    | 99s         |\n\nThis example demonstrates how there is some natural overhead coming from\nthe cloned copy because the server is hosting many forks and has not\noptimized for exactly this set of reachable objects. But the full repack\nhas similar characteristics with and without --full-name-hash.\n\nHowever, we can test this in a repository that uses one of the\nproblematic naming conventions above. The fluentui [2] repo uses\nbeachball to generate CHANGELOG.json and CHANGELOG.md files, and these\nfiles have very poor delta characteristics when comparing against\nversions across parent directories.\n\n| Stage                 | Pack Size | Repack Time |\n|-----------------------|-----------|-------------|\n| After clone           | 694 MB    | N/A         |\n| Standard Repack       | 438 MB    | 728s        |\n| With --full-name-hash | 168 MB    | 142s        |\n\n[2] https://github.com/microsoft/fluentui\n\nIn this example, we see significant gains in the compressed packfile\nsize as well as the time taken to compute the packfile.\n\nUsing a collection of repositories that use the beachball tool, I was\nable to make similar comparisions with dramatic results. While the\nfluentui repo is public, the others are private so cannot be shared for\nreproduction. The results are so significant that I find it important to\nshare here:\n\n| Repo     | Standard Repack | With --full-name-hash |\n|----------|-----------------|-----------------------|\n| fluentui |         438 MB  |               168 MB  |\n| Repo B   |       6,255 MB  |               829 MB  |\n| Repo C   |      37,737 MB  |             7,125 MB  |\n| Repo D   |     130,049 MB  |             6,190 MB  |\n\nFuture changes could include making --full-name-hash implied by a config\nvalue or even implied by default during a full repack.\n\nIt is important to point out that the name hash value is stored in the\n.bitmap file format, so we must disable the --full-name-hash option when\nbitmaps are being read or written. Later, the bitmap format could be\nupdated to be aware of the name hash version so deltas can be quickly\ncomputed across the bitmapped/not-bitmapped boundary.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Documentation/git-pack-objects.txt |  3 ++-\n builtin/pack-objects.c             | 25 ++++++++++++++++++++-----\n builtin/repack.c                   |  5 +++++\n pack-objects.h                     | 21 +++++++++++++++++++++\n t/t5300-pack-object.sh             | 15 +++++++++++++++\n 5 files changed, 63 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex e32404c6aae..93861d9f85b 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -15,7 +15,8 @@ SYNOPSIS\n \t[--revs [--unpacked | --all]] [--keep-pack=<pack-name>]\n \t[--cruft] [--cruft-expiration=<time>]\n \t[--stdout [--filter=<filter-spec>] | <base-name>]\n-\t[--shallow] [--keep-true-parents] [--[no-]sparse] < <object-list>\n+\t[--shallow] [--keep-true-parents] [--[no-]sparse]\n+\t[--full-name-hash] < <object-list>\n \n \n DESCRIPTION\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 08007142671..85595dfcd88 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -266,6 +266,14 @@ struct configured_exclusion {\n static struct oidmap configured_exclusions;\n \n static struct oidset excluded_by_config;\n+static int use_full_name_hash;\n+\n+static inline uint32_t pack_name_hash_fn(const char *name)\n+{\n+\tif (use_full_name_hash)\n+\t\treturn pack_full_name_hash(name);\n+\treturn pack_name_hash(name);\n+}\n \n /*\n  * stats\n@@ -1698,7 +1706,7 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n \t\treturn 0;\n \t}\n \n-\tcreate_object_entry(oid, type, pack_name_hash(name),\n+\tcreate_object_entry(oid, type, pack_name_hash_fn(name),\n \t\t\t    exclude, name && no_try_delta(name),\n \t\t\t    found_pack, found_offset);\n \treturn 1;\n@@ -1912,7 +1920,7 @@ static void add_preferred_base_object(const char *name)\n {\n \tstruct pbase_tree *it;\n \tsize_t cmplen;\n-\tunsigned hash = pack_name_hash(name);\n+\tunsigned hash = pack_name_hash_fn(name);\n \n \tif (!num_preferred_base || check_pbase_path(hash))\n \t\treturn;\n@@ -3422,7 +3430,7 @@ static void show_object_pack_hint(struct object *object, const char *name,\n \t * here using a now in order to perhaps improve the delta selection\n \t * process.\n \t */\n-\toe->hash = pack_name_hash(name);\n+\toe->hash = pack_name_hash_fn(name);\n \toe->no_try_delta = name && no_try_delta(name);\n \n \tstdin_packs_hints_nr++;\n@@ -3572,7 +3580,7 @@ static void add_cruft_object_entry(const struct object_id *oid, enum object_type\n \tentry = packlist_find(&to_pack, oid);\n \tif (entry) {\n \t\tif (name) {\n-\t\t\tentry->hash = pack_name_hash(name);\n+\t\t\tentry->hash = pack_name_hash_fn(name);\n \t\t\tentry->no_try_delta = no_try_delta(name);\n \t\t}\n \t} else {\n@@ -3595,7 +3603,7 @@ static void add_cruft_object_entry(const struct object_id *oid, enum object_type\n \t\t\treturn;\n \t\t}\n \n-\t\tentry = create_object_entry(oid, type, pack_name_hash(name),\n+\t\tentry = create_object_entry(oid, type, pack_name_hash_fn(name),\n \t\t\t\t\t    0, name && no_try_delta(name),\n \t\t\t\t\t    pack, offset);\n \t}\n@@ -4429,6 +4437,8 @@ int cmd_pack_objects(int argc,\n \t\tOPT_STRING_LIST(0, \"uri-protocol\", &uri_protocols,\n \t\t\t\tN_(\"protocol\"),\n \t\t\t\tN_(\"exclude any configured uploadpack.blobpackfileuri with this protocol\")),\n+\t\tOPT_BOOL(0, \"full-name-hash\", &use_full_name_hash,\n+\t\t\t N_(\"optimize delta compression across identical path names over time\")),\n \t\tOPT_END(),\n \t};\n \n@@ -4576,6 +4586,11 @@ int cmd_pack_objects(int argc,\n \tif (pack_to_stdout || !rev_list_all)\n \t\twrite_bitmap_index = 0;\n \n+\tif (write_bitmap_index && use_full_name_hash) {\n+\t\twarning(_(\"currently, the --full-name-hash option is incompatible with --write-bitmap-index\"));\n+\t\tuse_full_name_hash = 0;\n+\t}\n+\n \tif (use_delta_islands)\n \t\tstrvec_push(&rp, \"--topo-order\");\n \ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex d6bb37e84ae..ab2a2e46b20 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -58,6 +58,7 @@ struct pack_objects_args {\n \tint no_reuse_object;\n \tint quiet;\n \tint local;\n+\tint full_name_hash;\n \tstruct list_objects_filter_options filter_options;\n };\n \n@@ -306,6 +307,8 @@ static void prepare_pack_objects(struct child_process *cmd,\n \t\tstrvec_pushf(&cmd->args, \"--no-reuse-delta\");\n \tif (args->no_reuse_object)\n \t\tstrvec_pushf(&cmd->args, \"--no-reuse-object\");\n+\tif (args->full_name_hash)\n+\t\tstrvec_pushf(&cmd->args, \"--full-name-hash\");\n \tif (args->local)\n \t\tstrvec_push(&cmd->args,  \"--local\");\n \tif (args->quiet)\n@@ -1203,6 +1206,8 @@ int cmd_repack(int argc,\n \t\t\t\tN_(\"pass --no-reuse-delta to git-pack-objects\")),\n \t\tOPT_BOOL('F', NULL, &po_args.no_reuse_object,\n \t\t\t\tN_(\"pass --no-reuse-object to git-pack-objects\")),\n+\t\tOPT_BOOL(0, \"full-name-hash\", &po_args.full_name_hash,\n+\t\t\t\tN_(\"pass --full-name-hash to git-pack-objects\")),\n \t\tOPT_NEGBIT('n', NULL, &run_update_server_info,\n \t\t\t\tN_(\"do not run git-update-server-info\"), 1),\n \t\tOPT__QUIET(&po_args.quiet, N_(\"be quiet\")),\ndiff --git a/pack-objects.h b/pack-objects.h\nindex b9898a4e64b..88360aa3e8e 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -207,6 +207,27 @@ static inline uint32_t pack_name_hash(const char *name)\n \treturn hash;\n }\n \n+static inline uint32_t pack_full_name_hash(const char *name)\n+{\n+\tconst uint32_t bigp = 1234572167U;\n+\tuint32_t c, hash = bigp;\n+\n+\tif (!name)\n+\t\treturn 0;\n+\n+\t/*\n+\t * Do the simplest thing that will resemble pseudo-randomness: add\n+\t * random multiples of a large prime number with a binary shift.\n+\t * The goal is not to be cryptographic, but to be generally\n+\t * uniformly distributed.\n+\t */\n+\twhile ((c = *name++) != 0) {\n+\t\thash += c * bigp;\n+\t\thash = (hash >> 5) | (hash << 27);\n+\t}\n+\treturn hash;\n+}\n+\n static inline enum object_type oe_type(const struct object_entry *e)\n {\n \treturn e->type_valid ? e->type_ : OBJ_BAD;\ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 3b9dae331a5..7585cac6595 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -674,4 +674,19 @@ do\n \t'\n done\n \n+# The following test is not necessarily a permanent choice, but since we do not\n+# have a \"name hash version\" bit in the .bitmap file format, we cannot write the\n+# full-name hash values into the .bitmap file without risking breakage later.\n+#\n+# TODO: Make these compatible in the future and replace this test with the\n+# expected behavior when both are specified.\n+test_expect_success '--full-name-hash and --write-bitmap-index are incompatible' '\n+\tgit pack-objects base --all --full-name-hash --write-bitmap-index 2>err &&\n+\ttest_grep incompatible err &&\n+\n+\t# --stdout option silently removes --write-bitmap-index\n+\tgit pack-objects --stdout --all --full-name-hash --write-bitmap-index >out 2>err &&\n+\t! test_grep incompatible err\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"506591","messageId":"93395c93347274d075c3e29b3bd20dcc221b15be.1730775908.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.git.1730775907.gitgitgadget@gmail.com","subject":"[PATCH 2/7] repack: add --full-name-hash option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-11-05T03:05:02Z","receivedAt":"2024-11-05T03:05:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nThe new '--full-name-hash' option for 'git repack' is a simple\npass-through to the underlying 'git pack-objects' subcommand. However,\nthis subcommand may have other options and a temporary filename as part\nof the subcommand execution that may not be predictable or could change\nover time.\n\nThe existing test_subcommand method requires an exact list of arguments\nfor the subcommand. This is too rigid for our needs here, so create a\nnew method, test_subcommand_flex. Use it to check that the\n--full-name-hash option is passing through.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n t/t7700-repack.sh       |  7 +++++++\n t/test-lib-functions.sh | 26 ++++++++++++++++++++++++++\n 2 files changed, 33 insertions(+)\n\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex c4c3d1a15d9..fc2cc9d37be 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -777,6 +777,13 @@ test_expect_success 'repack -ad cleans up old .tmp-* packs' '\n \ttest_must_be_empty tmpfiles\n '\n \n+test_expect_success '--full-name-hash option passes through to pack-objects' '\n+\tGIT_TRACE2_EVENT=\"$(pwd)/full-trace.txt\" \\\n+\t\tgit repack -a --full-name-hash &&\n+\ttest_subcommand_flex git pack-objects --full-name-hash <full-trace.txt\n+'\n+\n+\n test_expect_success 'setup for update-server-info' '\n \tgit init update-server-info &&\n \ttest_commit -C update-server-info message\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 78e054ab503..af47247f25f 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1886,6 +1886,32 @@ test_subcommand () {\n \tfi\n }\n \n+# Check that the given subcommand was run with the given set of\n+# arguments in order (but with possible extra arguments).\n+#\n+#\ttest_subcommand_flex [!] <command> <args>... < <trace>\n+#\n+# If the first parameter passed is !, this instead checks that\n+# the given command was not called.\n+#\n+test_subcommand_flex () {\n+\tlocal negate=\n+\tif test \"$1\" = \"!\"\n+\tthen\n+\t\tnegate=t\n+\t\tshift\n+\tfi\n+\n+\tlocal expr=\"$(printf '\"%s\".*' \"$@\")\"\n+\n+\tif test -n \"$negate\"\n+\tthen\n+\t\t! grep \"\\[$expr\\]\"\n+\telse\n+\t\tgrep \"\\[$expr\\]\"\n+\tfi\n+}\n+\n # Check that the given command was invoked as part of the\n # trace2-format trace on stdin.\n #\n-- \ngitgitgadget\n\n"},{"id":"506593","messageId":"259734e0bcea952c2c09b0fb3a017e139922b975.1730775908.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.git.1730775907.gitgitgadget@gmail.com","subject":"[PATCH 3/7] pack-objects: add GIT_TEST_FULL_NAME_HASH","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-11-05T03:05:03Z","receivedAt":"2024-11-05T03:05:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nAdd a new environment variable to opt-in to the --full-name-hash option\nin 'git pack-objects'. This allows for extra testing of the feature\nwithout repeating all of the test scenarios.\n\nBut this option isn't free. There are a few tests that change behavior\nwith the variable enabled.\n\nFirst, there are a few tests that are very sensitive to certain delta\nbases being picked. These are both involving the generation of thin\nbundles and then counting their objects via 'git index-pack --fix-thin'\nwhich pulls the delta base into the new packfile. For these tests,\ndisable the option as a decent long-term option.\n\nSecond, there are two tests in t5616-partial-clone.sh that I believe are\nactually broken scenarios. While the client is set up to clone the\n'promisor-server' repo via a treeless partial clone filter (tree:0),\nthat filter does not translate to the 'server' repo. Thus, fetching from\nthese repos causes the server to think that the client has all reachable\ntrees and blobs from the commits advertised as 'haves'. This leads the\nserver to providing a thin pack assuming those objects as delta bases.\nChanging the name-hash algorithm presents new delta bases and thus\nbreaks the expectations of these tests. An alternative could be to set\nup 'server' as a promisor server with the correct filter enabled. This\nmay also point out more issues with partial clone being set up as a\nremote-based filtering mechanism and not a repository-wide setting. For\nnow, do the minimal change to make the test work by disabling the test\nvariable.\n\nThird, there are some tests that compare the exact output of a 'git\npack-objects' process when using bitmaps. The warning that disables the\n--full-name-hash option causes these tests to fail. Disable the\nenvironment variable to get around this issue.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n builtin/pack-objects.c          |  7 +++++--\n ci/run-build-and-tests.sh       |  1 +\n t/README                        |  4 ++++\n t/t5310-pack-bitmaps.sh         |  3 +++\n t/t5333-pseudo-merge-bitmaps.sh |  4 ++++\n t/t5510-fetch.sh                |  7 ++++++-\n t/t5616-partial-clone.sh        | 26 ++++++++++++++++++++++++--\n t/t6020-bundle-misc.sh          |  6 +++++-\n t/t7406-submodule-update.sh     |  2 ++\n t/t7700-repack.sh               |  6 ++++++\n 10 files changed, 60 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 85595dfcd88..7cb6f0e0942 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -266,7 +266,7 @@ struct configured_exclusion {\n static struct oidmap configured_exclusions;\n \n static struct oidset excluded_by_config;\n-static int use_full_name_hash;\n+static int use_full_name_hash = -1;\n \n static inline uint32_t pack_name_hash_fn(const char *name)\n {\n@@ -4586,7 +4586,10 @@ int cmd_pack_objects(int argc,\n \tif (pack_to_stdout || !rev_list_all)\n \t\twrite_bitmap_index = 0;\n \n-\tif (write_bitmap_index && use_full_name_hash) {\n+\tif (use_full_name_hash < 0)\n+\t\tuse_full_name_hash = git_env_bool(\"GIT_TEST_FULL_NAME_HASH\", 0);\n+\n+\tif (write_bitmap_index && use_full_name_hash > 0) {\n \t\twarning(_(\"currently, the --full-name-hash option is incompatible with --write-bitmap-index\"));\n \t\tuse_full_name_hash = 0;\n \t}\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 2e28d02b20f..75b40f07bbd 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -30,6 +30,7 @@ linux-TEST-vars)\n \texport GIT_TEST_NO_WRITE_REV_INDEX=1\n \texport GIT_TEST_CHECKOUT_WORKERS=2\n \texport GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL=1\n+\texport GIT_TEST_FULL_NAME_HASH=1\n \t;;\n linux-clang)\n \texport GIT_TEST_DEFAULT_HASH=sha1\ndiff --git a/t/README b/t/README\nindex 8c0319b58e5..fe3f89b5b28 100644\n--- a/t/README\n+++ b/t/README\n@@ -492,6 +492,10 @@ a test and then fails then the whole test run will abort. This can help to make\n sure the expected tests are executed and not silently skipped when their\n dependency breaks or is simply not present in a new environment.\n \n+GIT_TEST_FULL_NAME_HASH=<boolean>, when true, sets the default name-hash\n+function in 'git pack-objects' to be the one used by the --full-name-hash\n+option.\n+\n Naming Tests\n ------------\n \ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 7044c7d7c6d..caa3c125548 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -420,6 +420,9 @@ test_bitmap_cases () {\n \t\t\tcat >expect <<-\\EOF &&\n \t\t\terror: missing value for '\\''pack.preferbitmaptips'\\''\n \t\t\tEOF\n+\n+\t\t\t# Disable --full-name-hash test due to stderr comparison.\n+\t\t\tGIT_TEST_FULL_NAME_HASH=0 \\\n \t\t\tgit repack -adb 2>actual &&\n \t\t\ttest_cmp expect actual\n \t\t)\ndiff --git a/t/t5333-pseudo-merge-bitmaps.sh b/t/t5333-pseudo-merge-bitmaps.sh\nindex eca4a1eb8c6..0c7c3e33986 100755\n--- a/t/t5333-pseudo-merge-bitmaps.sh\n+++ b/t/t5333-pseudo-merge-bitmaps.sh\n@@ -209,6 +209,10 @@ test_expect_success 'bitmapPseudoMerge.stableThreshold creates stable groups' '\n '\n \n test_expect_success 'out of order thresholds are rejected' '\n+\t# Disable this option to avoid stderr message\n+\tGIT_TEST_FULL_NAME_HASH=0 &&\n+\texport GIT_TEST_FULL_NAME_HASH &&\n+\n \ttest_must_fail git \\\n \t\t-c bitmapPseudoMerge.test.pattern=\"refs/*\" \\\n \t\t-c bitmapPseudoMerge.test.threshold=1.month.ago \\\ndiff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\nindex 0890b9f61c5..be79c72495e 100755\n--- a/t/t5510-fetch.sh\n+++ b/t/t5510-fetch.sh\n@@ -1062,7 +1062,12 @@ test_expect_success 'all boundary commits are excluded' '\n \ttest_tick &&\n \tgit merge otherside &&\n \tad=$(git log --no-walk --format=%ad HEAD) &&\n-\tgit bundle create twoside-boundary.bdl main --since=\"$ad\" &&\n+\n+\t# If the --full-name-hash function is used here, then no delta\n+\t# pair is found and the bundle does not expand to three objects\n+\t# when fixing the thin object.\n+\tGIT_TEST_FULL_NAME_HASH=0 \\\n+\t\tgit bundle create twoside-boundary.bdl main --since=\"$ad\" &&\n \ttest_bundle_object_count --thin twoside-boundary.bdl 3\n '\n \ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex c53e93be2f7..425aa8d8789 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -516,7 +516,18 @@ test_expect_success 'fetch lazy-fetches only to resolve deltas' '\n \t# Exercise to make sure it works. Git will not fetch anything from the\n \t# promisor remote other than for the big tree (because it needs to\n \t# resolve the delta).\n-\tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -C client \\\n+\t#\n+\t# TODO: the --full-name-hash option is disabled here, since this test\n+\t# is fundamentally broken! When GIT_TEST_FULL_NAME_HASH=1, the server\n+\t# recognizes delta bases in a different way and then sends a _blob_ to\n+\t# the client with a delta base that the client does not have! This is\n+\t# because the client is cloned from \"promisor-server\" with tree:0 but\n+\t# is now fetching from \"server\" withot any filter. This is violating the\n+\t# promise to the server that all reachable objects exist and could be\n+\t# used as delta bases!\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace\" \\\n+\tGIT_TEST_FULL_NAME_HASH=0 \\\n+\t\tgit -C client \\\n \t\tfetch \"file://$(pwd)/server\" main &&\n \n \t# Verify the assumption that the client needed to fetch the delta base\n@@ -535,7 +546,18 @@ test_expect_success 'fetch lazy-fetches only to resolve deltas, protocol v2' '\n \t# Exercise to make sure it works. Git will not fetch anything from the\n \t# promisor remote other than for the big blob (because it needs to\n \t# resolve the delta).\n-\tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -C client \\\n+\t#\n+\t# TODO: the --full-name-hash option is disabled here, since this test\n+\t# is fundamentally broken! When GIT_TEST_FULL_NAME_HASH=1, the server\n+\t# recognizes delta bases in a different way and then sends a _blob_ to\n+\t# the client with a delta base that the client does not have! This is\n+\t# because the client is cloned from \"promisor-server\" with tree:0 but\n+\t# is now fetching from \"server\" withot any filter. This is violating the\n+\t# promise to the server that all reachable objects exist and could be\n+\t# used as delta bases!\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace\" \\\n+\tGIT_TEST_FULL_NAME_HASH=0 \\\n+\t\tgit -C client \\\n \t\tfetch \"file://$(pwd)/server\" main &&\n \n \t# Verify that protocol version 2 was used.\ndiff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\nindex 34b5cd62c20..553a20d10e9 100755\n--- a/t/t6020-bundle-misc.sh\n+++ b/t/t6020-bundle-misc.sh\n@@ -247,7 +247,11 @@ test_expect_success 'create bundle with --since option' '\n \tEOF\n \ttest_cmp expect actual &&\n \n-\tgit bundle create since.bdl \\\n+\t# If the --full-name-hash option is used, then one fewer\n+\t# delta base is found and this counts a different number\n+\t# of objects after performing --fix-thin.\n+\tGIT_TEST_FULL_NAME_HASH=0 \\\n+\t\tgit bundle create since.bdl \\\n \t\t--since \"Thu Apr 7 15:27:00 2005 -0700\" \\\n \t\t--all &&\n \ndiff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh\nindex 0f0c86f9cb2..03f8c976720 100755\n--- a/t/t7406-submodule-update.sh\n+++ b/t/t7406-submodule-update.sh\n@@ -1094,6 +1094,8 @@ test_expect_success 'submodule update --quiet passes quietness to fetch with a s\n \t) &&\n \tgit clone super4 super5 &&\n \t(cd super5 &&\n+\t # This test var can mess with the stderr output checked in this test.\n+\t GIT_TEST_FULL_NAME_HASH=0 \\\n \t git submodule update --quiet --init --depth=1 submodule3 >out 2>err &&\n \t test_must_be_empty out &&\n \t test_must_be_empty err\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex fc2cc9d37be..e3787bacdad 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -309,6 +309,9 @@ test_expect_success 'no bitmaps created if .keep files present' '\n \tkeep=${pack%.pack}.keep &&\n \ttest_when_finished \"rm -f \\\"\\$keep\\\"\" &&\n \t>\"$keep\" &&\n+\n+\t# Disable --full-name-hash test due to stderr comparison.\n+\tGIT_TEST_FULL_NAME_HASH=0 \\\n \tgit -C bare.git repack -ad 2>stderr &&\n \ttest_must_be_empty stderr &&\n \tfind bare.git/objects/pack/ -type f -name \"*.bitmap\" >actual &&\n@@ -320,6 +323,9 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '\n \tblob=$(test-tool genrandom big $((1024*1024)) |\n \t       git -C bare.git hash-object -w --stdin) &&\n \tgit -C bare.git update-ref refs/tags/big $blob &&\n+\n+\t# Disable --full-name-hash test due to stderr comparison.\n+\tGIT_TEST_FULL_NAME_HASH=0 \\\n \tgit -C bare.git repack -ad 2>stderr &&\n \ttest_must_be_empty stderr &&\n \tfind bare.git/objects/pack -type f -name \"*.bitmap\" >actual &&\n-- \ngitgitgadget\n\n"},{"id":"506594","messageId":"65784f85bce943e6a6bf29d7a57bb106aff8226b.1730775908.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.git.1730775907.gitgitgadget@gmail.com","subject":"[PATCH 4/7] git-repack: update usage to match docs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-11-05T03:05:04Z","receivedAt":"2024-11-05T03:05:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nThis also adds the '--full-name-hash' option introduced in the previous\nchange and adds newlines to the synopsis.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Documentation/git-repack.txt | 4 +++-\n builtin/repack.c             | 4 +++-\n t/t0450/txt-help-mismatches  | 1 -\n 3 files changed, 6 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex c902512a9e8..457a793fa89 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -9,7 +9,9 @@ git-repack - Pack unpacked objects in a repository\n SYNOPSIS\n --------\n [verse]\n-'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m] [--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>] [--write-midx]\n+'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\n+\t[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\n+\t[--write-midx] [--full-name-hash]\n \n DESCRIPTION\n -----------\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex ab2a2e46b20..e5f53a6eac7 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -39,7 +39,9 @@ static int run_update_server_info = 1;\n static char *packdir, *packtmp_name, *packtmp;\n \n static const char *const git_repack_usage[] = {\n-\tN_(\"git repack [<options>]\"),\n+\tN_(\"git repack [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\\n\"\n+\t   \"[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\\n\"\n+\t   \"[--write-midx] [--full-name-hash]\"),\n \tNULL\n };\n \ndiff --git a/t/t0450/txt-help-mismatches b/t/t0450/txt-help-mismatches\nindex 28003f18c92..c4a15fd0cb8 100644\n--- a/t/t0450/txt-help-mismatches\n+++ b/t/t0450/txt-help-mismatches\n@@ -45,7 +45,6 @@ rebase\n remote\n remote-ext\n remote-fd\n-repack\n reset\n restore\n rev-parse\n-- \ngitgitgadget\n\n"},{"id":"506595","messageId":"c14ef6879e451401381ebbdb8f30d33c8f56c25b.1730775908.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.git.1730775907.gitgitgadget@gmail.com","subject":"[PATCH 5/7] p5313: add size comparison test","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-11-05T03:05:05Z","receivedAt":"2024-11-05T03:05:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nAs custom options are added to 'git pack-objects' and 'git repack' to\nadjust how compression is done, use this new performance test script to\ndemonstrate their effectiveness in performance and size.\n\nThe recently-added --full-name-hash option swaps the default name-hash\nalgorithm with one that attempts to uniformly distribute the hashes\nbased on the full path name instead of the last 16 characters.\n\nThis has a dramatic effect on full repacks for repositories with many\nversions of most paths. It can have a negative impact on cases such as\npushing a single change.\n\nThis can be seen by running pt5313 on the open source fluentui\nrepository [1]. Most commits will have this kind of output for the thin\nand big pack cases, though certain commits (such as [2]) will have\nproblematic thin pack size for other reasons.\n\n[1] https://github.com/microsoft/fluentui\n[2] a637a06df05360ce5ff21420803f64608226a875\n\nChecked out at the parent of [2], I see the following statistics:\n\nTest                                               HEAD\n---------------------------------------------------------------------\n5313.2: thin pack                                  0.37(0.43+0.02)\n5313.3: thin pack size                                        1.2M\n5313.4: thin pack with --full-name-hash            0.06(0.09+0.02)\n5313.5: thin pack size with --full-name-hash                 20.4K\n5313.6: big pack                                   2.01(7.73+0.23)\n5313.7: big pack size                                        20.3M\n5313.8: big pack with --full-name-hash             1.32(2.77+0.27)\n5313.9: big pack size with --full-name-hash                  19.9M\n5313.10: shallow fetch pack                        1.40(3.01+0.08)\n5313.11: shallow pack size                                   34.4M\n5313.12: shallow pack with --full-name-hash        1.08(1.25+0.14)\n5313.13: shallow pack size with --full-name-hash             35.4M\n5313.14: repack                                    90.70(672.88+2.46)\n5313.15: repack size                                        439.6M\n5313.16: repack with --full-name-hash              18.53(123.41+2.53)\n5313.17: repack size with --full-name-hash                  169.7M\n\nIn this case, we see positive behaviors such as a significant shrink in\nthe size of the thin pack and full repack. The big pack is slightly\nsmaller with --full-name-hash than without. The shallow pack is slightly\nlarger with --full-name-hash.\n\nIn the case of the Git repository, these numbers show some of the issues\nwith this approach:\n\nTest                                               HEAD\n--------------------------------------------------------------------\n5313.2: thin pack                                  0.00(0.00+0.00)\n5313.3: thin pack size                                         589\n5313.4: thin pack with --full-name-hash            0.00(0.00+0.00)\n5313.5: thin pack size with --full-name-hash                 14.9K\n5313.6: big pack                                   2.07(3.57+0.17)\n5313.7: big pack size                                        17.6M\n5313.8: big pack with --full-name-hash             2.00(3.07+0.19)\n5313.9: big pack size with --full-name-hash                  17.9M\n5313.10: shallow fetch pack                        1.41(2.23+0.06)\n5313.11: shallow pack size                                   12.1M\n5313.12: shallow pack with --full-name-hash        1.22(1.66+0.04)\n5313.13: shallow pack size with --full-name-hash             12.4M\n5313.14: repack                                    15.75(89.29+1.54)\n5313.15: repack size                                        126.4M\n5313.16: repack with --full-name-hash              15.56(89.78+1.32)\n5313.17: repack size with --full-name-hash                  126.0M\n\nThe thin pack that simulates a push is much worse with --full-name-hash\nin this case. The name hash values are doing a lot to assist with delta\nbases, it seems. The big pack and shallow clone cases are slightly worse\nwith the --full-name-hash option. Only the full repack gains some\nbenefits in size.\n\nThe results are similar with the nodejs/node repo:\n\nTest                                               HEAD\n---------------------------------------------------------------------\n5313.2: thin pack                                  0.01(0.01+0.00)\n5313.3: thin pack size                                        1.6K\n5313.4: thin pack with --full-name-hash            0.01(0.00+0.00)\n5313.5: thin pack size with --full-name-hash                  3.1K\n5313.6: big pack                                   4.26(8.03+0.24)\n5313.7: big pack size                                        56.0M\n5313.8: big pack with --full-name-hash             4.16(6.55+0.22)\n5313.9: big pack size with --full-name-hash                  56.2M\n5313.10: shallow fetch pack                        7.67(11.80+0.29)\n5313.11: shallow pack size                                  104.6M\n5313.12: shallow pack with --full-name-hash        7.52(9.65+0.23)\n5313.13: shallow pack size with --full-name-hash            105.9M\n5313.14: repack                                    71.22(317.61+3.95)\n5313.15: repack size                                        739.9M\n5313.16: repack with --full-name-hash              48.85(267.02+3.72)\n5313.17: repack size with --full-name-hash                  793.5M\n\nThe Linux kernel repository was the initial target of the default name\nhash value, and its naming conventions are practically build to take the\nmost advantage of the default name hash values:\n\nTest                                               HEAD\n-------------------------------------------------------------------------\n5313.2: thin pack                                  0.15(0.01+0.03)\n5313.3: thin pack size                                        4.6K\n5313.4: thin pack with --full-name-hash            0.03(0.02+0.01)\n5313.5: thin pack size with --full-name-hash                  6.8K\n5313.6: big pack                                   18.51(33.74+0.95)\n5313.7: big pack size                                       201.1M\n5313.8: big pack with --full-name-hash             16.01(29.81+0.88)\n5313.9: big pack size with --full-name-hash                 202.1M\n5313.10: shallow fetch pack                        11.49(17.61+0.54)\n5313.11: shallow pack size                                  269.2M\n5313.12: shallow pack with --full-name-hash        11.24(15.25+0.56)\n5313.13: shallow pack size with --full-name-hash            269.8M\n5313.14: repack                                    1001.25(2271.06+38.86)\n5313.15: repack size                                          2.5G\n5313.16: repack with --full-name-hash              625.75(1941.96+36.09)\n5313.17: repack size with --full-name-hash                    2.6G\n\nFinally, an internal Javascript repo of moderate size shows significant\ngains when repacking with --full-name-hash due to it having many name\nhash collisions. However, it's worth noting that only the full repack\ncase has enough improvement to be worth it. But the improvements are\nsignificant: 6.4 GB to 862 MB.\n\nTest                                               HEAD\n--------------------------------------------------------------------------\n5313.2: thin pack                                  0.03(0.02+0.00)\n5313.3: thin pack size                                        1.2K\n5313.4: thin pack with --full-name-hash            0.03(0.03+0.00)\n5313.5: thin pack size with --full-name-hash                  2.6K\n5313.6: big pack                                   2.20(3.23+0.30)\n5313.7: big pack size                                       130.7M\n5313.8: big pack with --full-name-hash             2.33(3.17+0.34)\n5313.9: big pack size with --full-name-hash                 131.0M\n5313.10: shallow fetch pack                        3.56(6.02+0.32)\n5313.11: shallow pack size                                   44.5M\n5313.12: shallow pack with --full-name-hash        2.94(3.94+0.32)\n5313.13: shallow pack size with --full-name-hash             45.3M\n5313.14: repack                                    2435.22(12523.11+23.53)\n5313.15: repack size                                          6.4G\n5313.16: repack with --full-name-hash              473.25(1805.11+17.22)\n5313.17: repack size with --full-name-hash                  861.9M\n\nThese tests demonstrate that it is important to be careful about which\ncases are best for using the --full-name-hash option.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n t/perf/p5313-pack-objects.sh | 94 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 94 insertions(+)\n create mode 100755 t/perf/p5313-pack-objects.sh\n\ndiff --git a/t/perf/p5313-pack-objects.sh b/t/perf/p5313-pack-objects.sh\nnew file mode 100755\nindex 00000000000..dfa29695315\n--- /dev/null\n+++ b/t/perf/p5313-pack-objects.sh\n@@ -0,0 +1,94 @@\n+#!/bin/sh\n+\n+test_description='Tests pack performance using bitmaps'\n+. ./perf-lib.sh\n+\n+GIT_TEST_PASSING_SANITIZE_LEAK=0\n+export GIT_TEST_PASSING_SANITIZE_LEAK\n+\n+test_perf_large_repo\n+\n+test_expect_success 'create rev input' '\n+\tcat >in-thin <<-EOF &&\n+\t$(git rev-parse HEAD)\n+\t^$(git rev-parse HEAD~1)\n+\tEOF\n+\n+\tcat >in-big <<-EOF &&\n+\t$(git rev-parse HEAD)\n+\t^$(git rev-parse HEAD~1000)\n+\tEOF\n+\n+\tcat >in-shallow <<-EOF\n+\t$(git rev-parse HEAD)\n+\t--shallow $(git rev-parse HEAD)\n+\tEOF\n+'\n+\n+test_perf 'thin pack' '\n+\tgit pack-objects --thin --stdout --revs --sparse  <in-thin >out\n+'\n+\n+test_size 'thin pack size' '\n+\ttest_file_size out\n+'\n+\n+test_perf 'thin pack with --full-name-hash' '\n+\tgit pack-objects --thin --stdout --revs --sparse --full-name-hash <in-thin >out\n+'\n+\n+test_size 'thin pack size with --full-name-hash' '\n+\ttest_file_size out\n+'\n+\n+test_perf 'big pack' '\n+\tgit pack-objects --stdout --revs --sparse  <in-big >out\n+'\n+\n+test_size 'big pack size' '\n+\ttest_file_size out\n+'\n+\n+test_perf 'big pack with --full-name-hash' '\n+\tgit pack-objects --stdout --revs --sparse --full-name-hash <in-big >out\n+'\n+\n+test_size 'big pack size with --full-name-hash' '\n+\ttest_file_size out\n+'\n+\n+test_perf 'shallow fetch pack' '\n+\tgit pack-objects --stdout --revs --sparse --shallow <in-shallow >out\n+'\n+\n+test_size 'shallow pack size' '\n+\ttest_file_size out\n+'\n+\n+test_perf 'shallow pack with --full-name-hash' '\n+\tgit pack-objects --stdout --revs --sparse --shallow --full-name-hash <in-shallow >out\n+'\n+\n+test_size 'shallow pack size with --full-name-hash' '\n+\ttest_file_size out\n+'\n+\n+test_perf 'repack' '\n+\tgit repack -adf\n+'\n+\n+test_size 'repack size' '\n+\tpack=$(ls .git/objects/pack/pack-*.pack) &&\n+\ttest_file_size \"$pack\"\n+'\n+\n+test_perf 'repack with --full-name-hash' '\n+\tgit repack -adf --full-name-hash\n+'\n+\n+test_size 'repack size with --full-name-hash' '\n+\tpack=$(ls .git/objects/pack/pack-*.pack) &&\n+\ttest_file_size \"$pack\"\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"506596","messageId":"b8a055cb196dd971ac21611c1957be319557b4d3.1730775908.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.git.1730775907.gitgitgadget@gmail.com","subject":"[PATCH 6/7] pack-objects: disable --full-name-hash when shallow","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-11-05T03:05:06Z","receivedAt":"2024-11-05T03:05:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nAs demonstrated in the previous change, the --full-name-hash option of\n'git pack-objects' is less effective in a trunctated history. Thus, even\nwhen the option is selected via a command-line option or config, disable\nthis option when the '--shallow' option is specified. This will help\nperformance in servers that choose to enable the --full-name-hash option\nby default for a repository while not regressing their ability to serve\nshallow clones.\n\nThis will not present a compatibility issue in the future when the full\nname hash values are stored in the reachability bitmaps, since shallow\nclones disable bitmaps.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n builtin/pack-objects.c       | 6 ++++++\n t/perf/p5313-pack-objects.sh | 1 +\n 2 files changed, 7 insertions(+)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7cb6f0e0942..f68fc30c9b9 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -4589,6 +4589,12 @@ int cmd_pack_objects(int argc,\n \tif (use_full_name_hash < 0)\n \t\tuse_full_name_hash = git_env_bool(\"GIT_TEST_FULL_NAME_HASH\", 0);\n \n+\tif (shallow && use_full_name_hash > 0 &&\n+\t    !git_env_bool(\"GIT_TEST_USE_FULL_NAME_HASH_WITH_SHALLOW\", 0)) {\n+\t\tuse_full_name_hash = 0;\n+\t\twarning(\"the --full-name-hash option is disabled with the --shallow option\");\n+\t}\n+\n \tif (write_bitmap_index && use_full_name_hash > 0) {\n \t\twarning(_(\"currently, the --full-name-hash option is incompatible with --write-bitmap-index\"));\n \t\tuse_full_name_hash = 0;\ndiff --git a/t/perf/p5313-pack-objects.sh b/t/perf/p5313-pack-objects.sh\nindex dfa29695315..a7f4e0bf8d8 100755\n--- a/t/perf/p5313-pack-objects.sh\n+++ b/t/perf/p5313-pack-objects.sh\n@@ -66,6 +66,7 @@ test_size 'shallow pack size' '\n '\n \n test_perf 'shallow pack with --full-name-hash' '\n+\tGIT_TEST_USE_FULL_NAME_HASH_WITH_SHALLOW=1 \\\n \tgit pack-objects --stdout --revs --sparse --shallow --full-name-hash <in-shallow >out\n '\n \n-- \ngitgitgadget\n\n"},{"id":"506597","messageId":"ab341dd0e58f77b3c7c6f5765d9e34cb02bef56f.1730775908.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.git.1730775907.gitgitgadget@gmail.com","subject":"[PATCH 7/7] test-tool: add helper for name-hash values","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-11-05T03:05:07Z","receivedAt":"2024-11-05T03:05:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nAdd a new test-tool helper, name-hash, to output the value of the\nname-hash algorithms for the input list of strings, one per line.\n\nSince the name-hash values can be stored in the .bitmap files, it is\nimportant that these hash functions do not change across Git versions.\nAdd a simple test to t5310-pack-bitmaps.sh to provide some testing of\nthe current values. Due to how these functions are implemented, it would\nbe difficult to change them without disturbing these values.\n\nCreate a performance test that uses test_size to demonstrate how\ncollisions occur for these hash algorithms. This test helps inform\nsomeone as to the behavior of the name-hash algorithms for their repo\nbased on the paths at HEAD.\n\nMy copy of the Git repository shows modest statistics around the\ncollisions of the default name-hash algorithm:\n\nTest                                              this tree\n-----------------------------------------------------------------\n5314.1: paths at head                                        4.5K\n5314.2: number of distinct name-hashes                       4.1K\n5314.3: number of distinct full-name-hashes                  4.5K\n5314.4: maximum multiplicity of name-hashes                    13\n5314.5: maximum multiplicity of fullname-hashes                 1\n\nHere, the maximum collision multiplicity is 13, but around 10% of paths\nhave a collision with another path.\n\nIn a more interesting example, the microsoft/fluentui [1] repo had these\nstatistics at time of committing:\n\nTest                                              this tree\n-----------------------------------------------------------------\n5314.1: paths at head                                       19.6K\n5314.2: number of distinct name-hashes                       8.2K\n5314.3: number of distinct full-name-hashes                 19.6K\n5314.4: maximum multiplicity of name-hashes                   279\n5314.5: maximum multiplicity of fullname-hashes                 1\n\n[1] https://github.com/microsoft/fluentui\n\nThat demonstrates that of the nearly twenty thousand path names, they\nare assigned around eight thousand distinct values. 279 paths are\nassigned to a single value, leading the packing algorithm to sort\nobjects from those paths together, by size.\n\nIn this repository, no collisions occur for the full-name-hash\nalgorithm.\n\nIn a more extreme example, an internal monorepo had a much worse\ncollision rate:\n\nTest                                              this tree\n-----------------------------------------------------------------\n5314.1: paths at head                                      221.6K\n5314.2: number of distinct name-hashes                      72.0K\n5314.3: number of distinct full-name-hashes                221.6K\n5314.4: maximum multiplicity of name-hashes                 14.4K\n5314.5: maximum multiplicity of fullname-hashes                 2\n\nEven in this repository with many more paths at HEAD, the collision rate\nwas low and the maximum number of paths being grouped into a single\nbucket by the full-path-name algorithm was two.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Makefile                  |  1 +\n t/helper/test-name-hash.c | 24 +++++++++++++++++++++++\n t/helper/test-tool.c      |  1 +\n t/helper/test-tool.h      |  1 +\n t/perf/p5314-name-hash.sh | 41 +++++++++++++++++++++++++++++++++++++++\n t/t5310-pack-bitmaps.sh   | 26 +++++++++++++++++++++++++\n 6 files changed, 94 insertions(+)\n create mode 100644 t/helper/test-name-hash.c\n create mode 100755 t/perf/p5314-name-hash.sh\n\ndiff --git a/Makefile b/Makefile\nindex 6f5986b66ea..65403f6dd09 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -816,6 +816,7 @@ TEST_BUILTINS_OBJS += test-lazy-init-name-hash.o\n TEST_BUILTINS_OBJS += test-match-trees.o\n TEST_BUILTINS_OBJS += test-mergesort.o\n TEST_BUILTINS_OBJS += test-mktemp.o\n+TEST_BUILTINS_OBJS += test-name-hash.o\n TEST_BUILTINS_OBJS += test-online-cpus.o\n TEST_BUILTINS_OBJS += test-pack-mtimes.o\n TEST_BUILTINS_OBJS += test-parse-options.o\ndiff --git a/t/helper/test-name-hash.c b/t/helper/test-name-hash.c\nnew file mode 100644\nindex 00000000000..e4ecd159b76\n--- /dev/null\n+++ b/t/helper/test-name-hash.c\n@@ -0,0 +1,24 @@\n+/*\n+ * test-name-hash.c: Read a list of paths over stdin and report on their\n+ * name-hash and full name-hash.\n+ */\n+\n+#include \"test-tool.h\"\n+#include \"git-compat-util.h\"\n+#include \"pack-objects.h\"\n+#include \"strbuf.h\"\n+\n+int cmd__name_hash(int argc UNUSED, const char **argv UNUSED)\n+{\n+\tstruct strbuf line = STRBUF_INIT;\n+\n+\twhile (!strbuf_getline(&line, stdin)) {\n+\t\tuint32_t name_hash = pack_name_hash(line.buf);\n+\t\tuint32_t full_hash = pack_full_name_hash(line.buf);\n+\n+\t\tprintf(\"%10\"PRIu32\"\\t%10\"PRIu32\"\\t%s\\n\", name_hash, full_hash, line.buf);\n+\t}\n+\n+\tstrbuf_release(&line);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 1ebb69a5dc4..e794058ab6d 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -44,6 +44,7 @@ static struct test_cmd cmds[] = {\n \t{ \"match-trees\", cmd__match_trees },\n \t{ \"mergesort\", cmd__mergesort },\n \t{ \"mktemp\", cmd__mktemp },\n+\t{ \"name-hash\", cmd__name_hash },\n \t{ \"online-cpus\", cmd__online_cpus },\n \t{ \"pack-mtimes\", cmd__pack_mtimes },\n \t{ \"parse-options\", cmd__parse_options },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 21802ac27da..26ff30a5a9a 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -37,6 +37,7 @@ int cmd__lazy_init_name_hash(int argc, const char **argv);\n int cmd__match_trees(int argc, const char **argv);\n int cmd__mergesort(int argc, const char **argv);\n int cmd__mktemp(int argc, const char **argv);\n+int cmd__name_hash(int argc, const char **argv);\n int cmd__online_cpus(int argc, const char **argv);\n int cmd__pack_mtimes(int argc, const char **argv);\n int cmd__parse_options(int argc, const char **argv);\ndiff --git a/t/perf/p5314-name-hash.sh b/t/perf/p5314-name-hash.sh\nnew file mode 100755\nindex 00000000000..9fe26612fac\n--- /dev/null\n+++ b/t/perf/p5314-name-hash.sh\n@@ -0,0 +1,41 @@\n+#!/bin/sh\n+\n+test_description='Tests pack performance using bitmaps'\n+. ./perf-lib.sh\n+\n+GIT_TEST_PASSING_SANITIZE_LEAK=0\n+export GIT_TEST_PASSING_SANITIZE_LEAK\n+\n+test_perf_large_repo\n+\n+test_size 'paths at head' '\n+\tgit ls-tree -r --name-only HEAD >path-list &&\n+\twc -l <path-list\n+'\n+\n+test_size 'number of distinct name-hashes' '\n+\tcat path-list | test-tool name-hash >name-hashes &&\n+\tcat name-hashes | awk \"{ print \\$1; }\" | sort -n | uniq -c >name-hash-count &&\n+\twc -l <name-hash-count\n+'\n+\n+test_size 'number of distinct full-name-hashes' '\n+\tcat name-hashes | awk \"{ print \\$2; }\" | sort -n | uniq -c >full-name-hash-count &&\n+\twc -l <full-name-hash-count\n+'\n+\n+test_size 'maximum multiplicity of name-hashes' '\n+\tcat name-hash-count | \\\n+\t\tsort -nr | \\\n+\t\thead -n 1 | \\\n+\t\tawk \"{ print \\$1; }\"\n+'\n+\n+test_size 'maximum multiplicity of fullname-hashes' '\n+\tcat full-name-hash-count | \\\n+\t\tsort -nr | \\\n+\t\thead -n 1 | \\\n+\t\tawk \"{ print \\$1; }\"\n+'\n+\n+test_done\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex caa3c125548..965c3abca5f 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -27,6 +27,32 @@ has_any () {\n \tgrep -Ff \"$1\" \"$2\"\n }\n \n+# Since name-hash values are stored in the .bitmap files, add a test\n+# that checks that the name-hash calculations are stable across versions.\n+# Not exhaustive, but these hashing algorithms would be hard to change\n+# without causing deviations here.\n+test_expect_success 'name-hash value stability' '\n+\tcat >names <<-\\EOF &&\n+\tfirst\n+\tsecond\n+\tthird\n+\tone-long-enough-for-collisions\n+\ttwo-long-enough-for-collisions\n+\tEOF\n+\n+\ttest-tool name-hash <names >out &&\n+\n+\tcat >expect <<-\\EOF &&\n+\t2582249472\t3109209818\tfirst\n+\t2289942528\t3781118409\tsecond\n+\t2300837888\t3028707182\tthird\n+\t2544516325\t3241327563\tone-long-enough-for-collisions\n+\t2544516325\t4207880830\ttwo-long-enough-for-collisions\n+\tEOF\n+\n+\ttest_cmp expect out\n+'\n+\n test_bitmap_cases () {\n \twriteLookupTable=false\n \tfor i in \"$@\"\n-- \ngitgitgadget\n"},{"id":"507833","messageId":"Zz+TKS2O/ij6GZ1f@nand.local","threadId":"62447","inReplyTo":"812257e197cfe30bd0d3c68ea6ec0d062631185f.1730775907.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/7] pack-objects: add --full-name-hash option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-21T20:08:09Z","receivedAt":"2024-11-21T20:08:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Nov 05, 2024 at 03:05:01AM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee <stolee@gmail.com>\n>\n> The pack_name_hash() method has not been materially changed since it was\n> introduced in ce0bd64299a (pack-objects: improve path grouping\n> heuristics., 2006-06-05). The intention here is to group objects by path\n> name, but also attempt to group similar file types together by making\n> the most-significant digits of the hash be focused on the final\n> characters.\n>\n> Here's the crux of the implementation:\n>\n> \t/*\n> \t * This effectively just creates a sortable number from the\n> \t * last sixteen non-whitespace characters. Last characters\n> \t * count \"most\", so things that end in \".c\" sort together.\n> \t */\n> \twhile ((c = *name++) != 0) {\n> \t\tif (isspace(c))\n> \t\t\tcontinue;\n> \t\thash = (hash >> 2) + (c << 24);\n> \t}\n\nHah. I like that the existing implementation is small enough to fit (in\nits entirety!) into the commit message!\n\n> As the comment mentions, this only cares about the last sixteen\n> non-whitespace characters. This cause some filenames to collide more\n> than others. Here are some examples that I've seen while investigating\n> repositories that are growing more than they should be:\n>\n>  * \"/CHANGELOG.json\" is 15 characters, and is created by the beachball\n>    [1] tool. Only the final character of the parent directory can\n>    differntiate different versions of this file, but also only the two\n\ns/differntiate/differentiate ;-).\n\n>    most-significant digits. If that character is a letter, then this is\n>    always a collision. Similar issues occur with the similar\n>    \"/CHANGELOG.md\" path, though there is more opportunity for\n>    differences in the parent directory.\n>\n>  * Localization files frequently have common filenames but differentiate\n>    via parent directories. In C#, the name \"/strings.resx.lcl\" is used\n>    for these localization files and they will all collide in name-hash.\n>\n> [1] https://github.com/microsoft/beachball\n>\n> I've come across many other examples where some internal tool uses a\n> common name across multiple directories and is causing Git to repack\n> poorly due to name-hash collisions.\n>\n> It is clear that the existing name-hash algorithm is optimized for\n> repositories with short path names, but also is optimized for packing a\n> single snapshot of a repository, not a repository with many versions of\n> the same file. In my testing, this has proven out where the name-hash\n> algorithm does a good job of finding peer files as delta bases when\n> unable to use a historical version of that exact file.\n\nI'm not sure I entirely agree with the suggestion that the existing hash\nfunction is only about packing repositories with short pathnames. I\nthink an important part of the existing implementation is that tries to\ngroup similar files together, regardless of whether or not they appear\nin the same tree.\n\nAs you have shown, this can be a problem when the fact two files that\nhappen to end in \"CHANGELOG.json\" end up in vastly different trees and\n*aren't* related. I don't think that nailing all of these details in the\ncommit message is necessary, but I do think it's worth adjusting what\nthe original commit message says in terms of what the existing algorithm\nis optimized for.\n\n> However, for repositories that have many versions of most files and\n> directories, it is more important that the objects that appear at the\n> same path are grouped together.\n>\n> Create a new pack_full_name_hash() method and a new --full-name-hash\n> option for 'git pack-objects' to call that method instead. Add a simple\n> pass-through for 'git repack --full-name-hash' for additional testing in\n> the context of a full repack, where I expect this will be most\n> effective.\n>\n> The hash algorithm is as simple as possible to be reasonably effective:\n> for each character of the path string, add a multiple of that character\n> and a large prime number (chosen arbitrarily, but intended to be large\n> relative to the size of a uint32_t). Then, shift the current hash value\n> to the right by 5, with overlap. The addition and shift parameters are\n> standard mechanisms for creating hard-to-predict behaviors in the bits\n> of the resulting hash.\n>\n> This is not meant to be cryptographic at all, but uniformly distributed\n> across the possible hash values. This creates a hash that appears\n> pseudorandom. There is no ability to consider similar file types as\n> being close to each other.\n\nI think you hint at this in the series' cover letter, but I suspect that\nthis pseduorandom behavior hurts in some small number of cases and that\nthe full-name hash idea isn't a pure win, e.g., when we really do want\nto delta two paths that both end in CHAGNELOG.json despite being in\ndifferent parts of the tree.\n\nYou have some tables here below that demonstrate a significant\nimprovement with the full-name hash in use, which I think is good worth\nkeeping in my own opinion. It may be worth updating those to include the\nnew examples you highlighted in your revised cover letter as well.\n\n> In a later change, a test-tool will be added so the effectiveness of\n> this hash can be demonstrated directly.\n>\n> For now, let's consider how effective this mechanism is when repacking a\n> repository with and without the --full-name-hash option. Specifically,\n\nIs this repository publicly available? If so, it may be worth mentioning\nhere.\n\n> let's use 'git repack -adf [--full-name-hash]' as our test.\n>\n> On the Git repository, we do not expect much difference. All path names\n> are short. This is backed by our results:\n>\n> | Stage                 | Pack Size | Repack Time |\n> |-----------------------|-----------|-------------|\n> | After clone           | 260 MB    | N/A         |\n> | Standard Repack       | 127MB     | 106s        |\n> | With --full-name-hash | 126 MB    | 99s         |\n\nAhh. Here's a great example of it helping to a smaller extent. Thanks\nfor including this as part of demonstrating the full picture (both the\nbenefits and drawbacks).\n\n> This example demonstrates how there is some natural overhead coming from\n> the cloned copy because the server is hosting many forks and has not\n> optimized for exactly this set of reachable objects. But the full repack\n> has similar characteristics with and without --full-name-hash.\n\nGood.\n\n> However, we can test this in a repository that uses one of the\n> problematic naming conventions above. The fluentui [2] repo uses\n> beachball to generate CHANGELOG.json and CHANGELOG.md files, and these\n> files have very poor delta characteristics when comparing against\n> versions across parent directories.\n>\n> | Stage                 | Pack Size | Repack Time |\n> |-----------------------|-----------|-------------|\n> | After clone           | 694 MB    | N/A         |\n> | Standard Repack       | 438 MB    | 728s        |\n> | With --full-name-hash | 168 MB    | 142s        |\n>\n> [2] https://github.com/microsoft/fluentui\n>\n> In this example, we see significant gains in the compressed packfile\n> size as well as the time taken to compute the packfile.\n\nAmazing!\n\n> Using a collection of repositories that use the beachball tool, I was\n> able to make similar comparisions with dramatic results. While the\n> fluentui repo is public, the others are private so cannot be shared for\n> reproduction. The results are so significant that I find it important to\n> share here:\n>\n> | Repo     | Standard Repack | With --full-name-hash |\n> |----------|-----------------|-----------------------|\n> | fluentui |         438 MB  |               168 MB  |\n> | Repo B   |       6,255 MB  |               829 MB  |\n> | Repo C   |      37,737 MB  |             7,125 MB  |\n> | Repo D   |     130,049 MB  |             6,190 MB  |\n>\n> Future changes could include making --full-name-hash implied by a config\n> value or even implied by default during a full repack.\n>\n> It is important to point out that the name hash value is stored in the\n> .bitmap file format, so we must disable the --full-name-hash option when\n> bitmaps are being read or written. Later, the bitmap format could be\n> updated to be aware of the name hash version so deltas can be quickly\n> computed across the bitmapped/not-bitmapped boundary.\n\nAgreed.\n\n> Signed-off-by: Derrick Stolee <stolee@gmail.com>\n> ---\n>  Documentation/git-pack-objects.txt |  3 ++-\n>  builtin/pack-objects.c             | 25 ++++++++++++++++++++-----\n>  builtin/repack.c                   |  5 +++++\n>  pack-objects.h                     | 21 +++++++++++++++++++++\n>  t/t5300-pack-object.sh             | 15 +++++++++++++++\n>  5 files changed, 63 insertions(+), 6 deletions(-)\n>\n> diff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\n> index e32404c6aae..93861d9f85b 100644\n> --- a/Documentation/git-pack-objects.txt\n> +++ b/Documentation/git-pack-objects.txt\n> @@ -15,7 +15,8 @@ SYNOPSIS\n>  \t[--revs [--unpacked | --all]] [--keep-pack=<pack-name>]\n>  \t[--cruft] [--cruft-expiration=<time>]\n>  \t[--stdout [--filter=<filter-spec>] | <base-name>]\n> -\t[--shallow] [--keep-true-parents] [--[no-]sparse] < <object-list>\n> +\t[--shallow] [--keep-true-parents] [--[no-]sparse]\n> +\t[--full-name-hash] < <object-list>\n\nOK, I see that --full-name-hash is now listed in the synopsis, but I\ndon't see a corresponding description of what the option does later on\nin this file. I took a look through the remaining patches in this series\nand couldn't find any further changes to git-pack-objects(1) either.\n\n>  DESCRIPTION\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 08007142671..85595dfcd88 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -266,6 +266,14 @@ struct configured_exclusion {\n>  static struct oidmap configured_exclusions;\n>\n>  static struct oidset excluded_by_config;\n> +static int use_full_name_hash;\n> +\n> +static inline uint32_t pack_name_hash_fn(const char *name)\n> +{\n> +\tif (use_full_name_hash)\n> +\t\treturn pack_full_name_hash(name);\n> +\treturn pack_name_hash(name);\n> +}\n>\n>  /*\n>   * stats\n> @@ -1698,7 +1706,7 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n>  \t\treturn 0;\n>  \t}\n>\n> -\tcreate_object_entry(oid, type, pack_name_hash(name),\n> +\tcreate_object_entry(oid, type, pack_name_hash_fn(name),\n>  \t\t\t    exclude, name && no_try_delta(name),\n>  \t\t\t    found_pack, found_offset);\n>  \treturn 1;\n> @@ -1912,7 +1920,7 @@ static void add_preferred_base_object(const char *name)\n>  {\n>  \tstruct pbase_tree *it;\n>  \tsize_t cmplen;\n> -\tunsigned hash = pack_name_hash(name);\n> +\tunsigned hash = pack_name_hash_fn(name);\n>\n>  \tif (!num_preferred_base || check_pbase_path(hash))\n>  \t\treturn;\n> @@ -3422,7 +3430,7 @@ static void show_object_pack_hint(struct object *object, const char *name,\n>  \t * here using a now in order to perhaps improve the delta selection\n>  \t * process.\n>  \t */\n> -\toe->hash = pack_name_hash(name);\n> +\toe->hash = pack_name_hash_fn(name);\n>  \toe->no_try_delta = name && no_try_delta(name);\n>\n>  \tstdin_packs_hints_nr++;\n> @@ -3572,7 +3580,7 @@ static void add_cruft_object_entry(const struct object_id *oid, enum object_type\n>  \tentry = packlist_find(&to_pack, oid);\n>  \tif (entry) {\n>  \t\tif (name) {\n> -\t\t\tentry->hash = pack_name_hash(name);\n> +\t\t\tentry->hash = pack_name_hash_fn(name);\n>  \t\t\tentry->no_try_delta = no_try_delta(name);\n>  \t\t}\n>  \t} else {\n> @@ -3595,7 +3603,7 @@ static void add_cruft_object_entry(const struct object_id *oid, enum object_type\n>  \t\t\treturn;\n>  \t\t}\n>\n> -\t\tentry = create_object_entry(oid, type, pack_name_hash(name),\n> +\t\tentry = create_object_entry(oid, type, pack_name_hash_fn(name),\n>  \t\t\t\t\t    0, name && no_try_delta(name),\n>  \t\t\t\t\t    pack, offset);\n>  \t}\n> @@ -4429,6 +4437,8 @@ int cmd_pack_objects(int argc,\n>  \t\tOPT_STRING_LIST(0, \"uri-protocol\", &uri_protocols,\n>  \t\t\t\tN_(\"protocol\"),\n>  \t\t\t\tN_(\"exclude any configured uploadpack.blobpackfileuri with this protocol\")),\n> +\t\tOPT_BOOL(0, \"full-name-hash\", &use_full_name_hash,\n> +\t\t\t N_(\"optimize delta compression across identical path names over time\")),\n>  \t\tOPT_END(),\n>  \t};\n>\n> @@ -4576,6 +4586,11 @@ int cmd_pack_objects(int argc,\n>  \tif (pack_to_stdout || !rev_list_all)\n>  \t\twrite_bitmap_index = 0;\n>\n> +\tif (write_bitmap_index && use_full_name_hash) {\n> +\t\twarning(_(\"currently, the --full-name-hash option is incompatible with --write-bitmap-index\"));\n> +\t\tuse_full_name_hash = 0;\n> +\t}\n> +\n\nGood, we determine this early on in the command, so we don't risk\ncomputing different hash functions within the same process.\n\nI wonder if it's worth guarding against mixing the hash functions within\nthe pack_name_hash() and pack_full_name_hash() functions themselves. I'm\nthinking something like:\n\n    static inline uint32_t pack_name_hash(const char *name)\n    {\n        if (use_full_name_hash)\n            BUG(\"called pack_name_hash() with --full-name-hash\")\n        /* ... */\n    }\n\nand the inverse in pack_full_name_hash(). I don't think it's strictly\nnecessary, but it would be a nice guard against someone calling, e.g.,\npack_full_name_hash() directly instead of pack_name_hash_fn().\n\nThe other small thought I had here is that we should use the convenience\nfunction die_for_incompatible_opt3() here, since it uses an existing\ntranslation string for pairs of incompatible options.\n\n(As an aside, though that function is actually implemented in the\n_opt4() variant, and it knows how to handle a pair, trio, and quartet of\nmutually incompatible options, there is no die_for_incompatible_opt2()\nfunction. It may be worth adding one here since I'm sure there are other\nspots which would benefit from such a function).\n\n> diff --git a/builtin/repack.c b/builtin/repack.c\n> index d6bb37e84ae..ab2a2e46b20 100644\n> --- a/builtin/repack.c\n> +++ b/builtin/repack.c\n\nI'm surprised to see the new option plumbed into repack in this commit.\nI would have thought that it'd appear in the subsequent commit instead.\nThe implementation below looks good, I just imagined it would be placed\nin the next commit instead of this one.\n\nThe remaining parts of this change look good to me.\n\nThanks,\nTaylor\n"},{"id":"507834","messageId":"Zz+UJHclSsb+Bgfo@nand.local","threadId":"62447","inReplyTo":"93395c93347274d075c3e29b3bd20dcc221b15be.1730775908.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/7] repack: add --full-name-hash option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-21T20:12:20Z","receivedAt":"2024-11-21T20:12:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Nov 05, 2024 at 03:05:02AM +0000, Derrick Stolee via GitGitGadget wrote:\n> ---\n>  t/t7700-repack.sh       |  7 +++++++\n>  t/test-lib-functions.sh | 26 ++++++++++++++++++++++++++\n>  2 files changed, 33 insertions(+)\n\nOK, I stand by my thinking in the previous patch that this one is where\nthe changes to builtin/repack.c belong.\n\n> diff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\n> index c4c3d1a15d9..fc2cc9d37be 100755\n> --- a/t/t7700-repack.sh\n> +++ b/t/t7700-repack.sh\n> @@ -777,6 +777,13 @@ test_expect_success 'repack -ad cleans up old .tmp-* packs' '\n>  \ttest_must_be_empty tmpfiles\n>  '\n>\n> +test_expect_success '--full-name-hash option passes through to pack-objects' '\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/full-trace.txt\" \\\n> +\t\tgit repack -a --full-name-hash &&\n> +\ttest_subcommand_flex git pack-objects --full-name-hash <full-trace.txt\n\nOK. To be honest, I am not sure I would have written the same test to\ntest trivially correct behavior, but I am not opposed to having such a\ntest either.\n\nI do think that test_subcommand_flex may be unnecessary though, since\nyou could instead write this as:\n\n    test_subcommand \"git pack-objects.*--full-name-hash\" <full-trace.txt\n\nand get the same behavior.\n\n> +'\n> +\n> +\n\nNit: extra newline here.\n\nThanks,\nTaylor\n"},{"id":"507835","messageId":"Zz+U8IyHqBNRIn6m@nand.local","threadId":"62447","inReplyTo":"259734e0bcea952c2c09b0fb3a017e139922b975.1730775908.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/7] pack-objects: add GIT_TEST_FULL_NAME_HASH","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-21T20:15:44Z","receivedAt":"2024-11-21T20:15:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Nov 05, 2024 at 03:05:03AM +0000, Derrick Stolee via GitGitGadget wrote:\n> diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\n> index 2e28d02b20f..75b40f07bbd 100755\n> --- a/ci/run-build-and-tests.sh\n> +++ b/ci/run-build-and-tests.sh\n> @@ -30,6 +30,7 @@ linux-TEST-vars)\n>  \texport GIT_TEST_NO_WRITE_REV_INDEX=1\n>  \texport GIT_TEST_CHECKOUT_WORKERS=2\n>  \texport GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL=1\n> +\texport GIT_TEST_FULL_NAME_HASH=1\n>  \t;;\n>  linux-clang)\n>  \texport GIT_TEST_DEFAULT_HASH=sha1\n\nHmm. I appreciate what this new GIT_TEST_ variable is trying to do, but\nI am somewhat saddened to see this list in linux-TEST-vars growing\nrather than shrinking.\n\nI'm most definitely part of the problem here, but I think too often we\nadd new entries to this list and let them languish without ever removing\nthem after they have served their intended purpose.\n\nSo I think the question is: what do we hope to get out of running the\ntest suite in a mode where we use the full-name hash all of the time? I\ncan't imagine any interesting breakage (other than individual tests'\nsensitivity to specific delta/base pairs) that would be caught by merely\nchanging the hash function here.\n\nI dunno. Maybe there is some exotic behavior that this shook out for you\nduring development which I'm not aware of. If that were the case, I\nthink that keeping this variable around makes sense, since the appearance\nof that exotic behavior proves that the variable is useful at shaking\nout bugs.\n\nBut assuming not, I think that I would just as soon avoid this test\nvariable entirely, which I think in this case amounts to dropping this\npatch from the series.\n\nThanks,\nTaylor\n"},{"id":"507836","messageId":"Zz+VRZnWjsDI9bLt@nand.local","threadId":"62447","inReplyTo":"65784f85bce943e6a6bf29d7a57bb106aff8226b.1730775908.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 4/7] git-repack: update usage to match docs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-21T20:17:09Z","receivedAt":"2024-11-21T20:17:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Nov 05, 2024 at 03:05:04AM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee <stolee@gmail.com>\n>\n> This also adds the '--full-name-hash' option introduced in the previous\n> change and adds newlines to the synopsis.\n\nI think \"the previous change\" is not quite accurate here, even if\nyou move the implementation to pass through '--full-name-hash' via\nrepack into the second patch.\n\nIt would be nice to have the option added in 'repack' in the same commit\nas adjusts the documentation instead of splitting them apart.\n\nThanks,\nTaylor\n"},{"id":"507838","messageId":"Zz+YrvL8h0Cxwqfy@nand.local","threadId":"62447","inReplyTo":"c14ef6879e451401381ebbdb8f30d33c8f56c25b.1730775908.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/7] p5313: add size comparison test","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-21T20:31:42Z","receivedAt":"2024-11-21T20:31:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Nov 05, 2024 at 03:05:05AM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee <stolee@gmail.com>\n>\n> As custom options are added to 'git pack-objects' and 'git repack' to\n> adjust how compression is done, use this new performance test script to\n> demonstrate their effectiveness in performance and size.\n\nNicely done, thank you for adding a perf test to allow readers to easily\nverify these changes themselves.\n\n> In the case of the Git repository, these numbers show some of the issues\n> with this approach:\n>\n> [...]\n>\n> The thin pack that simulates a push is much worse with --full-name-hash\n> in this case. The name hash values are doing a lot to assist with delta\n> bases, it seems. The big pack and shallow clone cases are slightly worse\n> with the --full-name-hash option. Only the full repack gains some\n> benefits in size.\n\nNot a problem with your patch, but just thinking aloud: do you think\nthere is an easy/straightforward way to suggest when to use\n--full-name-hash or not?\n\n> ---\n>  t/perf/p5313-pack-objects.sh | 94 ++++++++++++++++++++++++++++++++++++\n>  1 file changed, 94 insertions(+)\n>  create mode 100755 t/perf/p5313-pack-objects.sh\n>\n> diff --git a/t/perf/p5313-pack-objects.sh b/t/perf/p5313-pack-objects.sh\n> new file mode 100755\n> index 00000000000..dfa29695315\n> --- /dev/null\n> +++ b/t/perf/p5313-pack-objects.sh\n> @@ -0,0 +1,94 @@\n> +#!/bin/sh\n> +\n> +test_description='Tests pack performance using bitmaps'\n> +. ./perf-lib.sh\n> +\n> +GIT_TEST_PASSING_SANITIZE_LEAK=0\n> +export GIT_TEST_PASSING_SANITIZE_LEAK\n> +\n> +test_perf_large_repo\n> +\n> +test_expect_success 'create rev input' '\n> +\tcat >in-thin <<-EOF &&\n> +\t$(git rev-parse HEAD)\n> +\t^$(git rev-parse HEAD~1)\n> +\tEOF\n> +\n> +\tcat >in-big <<-EOF &&\n> +\t$(git rev-parse HEAD)\n> +\t^$(git rev-parse HEAD~1000)\n> +\tEOF\n> +\n> +\tcat >in-shallow <<-EOF\n> +\t$(git rev-parse HEAD)\n> +\t--shallow $(git rev-parse HEAD)\n> +\tEOF\n> +'\n\nI was going to comment that these could probably be moved into the\nindividual perf test that cares about reading each of these inputs. But\nhaving them shared here makes sense since we are naturally comparing\ngenerating two packs with the same input (with and without\n--full-name-hash). So the shared setup here makes sense to me.\n\n> +\n> +test_perf 'thin pack' '\n> +\tgit pack-objects --thin --stdout --revs --sparse  <in-thin >out\n> +'\n> +\n> +test_size 'thin pack size' '\n> +\ttest_file_size out\n> +'\n\nNice. I always forget about this and end up writing 'wc -c <out'.\n\n> +test_perf 'thin pack with --full-name-hash' '\n> +\tgit pack-objects --thin --stdout --revs --sparse --full-name-hash <in-thin >out\n> +'\n> +\n> +test_size 'thin pack size with --full-name-hash' '\n> +\ttest_file_size out\n> +'\n> +\n> +test_perf 'big pack' '\n> +\tgit pack-objects --stdout --revs --sparse  <in-big >out\n> +'\n> +\n> +test_size 'big pack size' '\n> +\ttest_file_size out\n> +'\n> +\n> +test_perf 'big pack with --full-name-hash' '\n> +\tgit pack-objects --stdout --revs --sparse --full-name-hash <in-big >out\n> +'\n> +\n> +test_size 'big pack size with --full-name-hash' '\n> +\ttest_file_size out\n> +'\n> +\n> +test_perf 'shallow fetch pack' '\n> +\tgit pack-objects --stdout --revs --sparse --shallow <in-shallow >out\n> +'\n> +\n> +test_size 'shallow pack size' '\n> +\ttest_file_size out\n> +'\n> +\n> +test_perf 'shallow pack with --full-name-hash' '\n> +\tgit pack-objects --stdout --revs --sparse --shallow --full-name-hash <in-shallow >out\n> +'\n> +\n> +test_size 'shallow pack size with --full-name-hash' '\n> +\ttest_file_size out\n> +'\n> +\n> +test_perf 'repack' '\n> +\tgit repack -adf\n> +'\n> +\n> +test_size 'repack size' '\n> +\tpack=$(ls .git/objects/pack/pack-*.pack) &&\n> +\ttest_file_size \"$pack\"\n\nHere and below, I think it's fine to inline this as in:\n\n    test_file_size \"$(ls .git/objects/pack/pack-*.pack)\"\n\n...but I wonder: will using \".git\" break this test in bare repositories?\nShould we write instead:\n\n    pack=\"$(ls $(git rev-parse --git-dir)/objects/pack/pack-*.pack)\" &&\n    test_file_size\n\n?\n\nThanks,\nTaylor\n"},{"id":"507839","messageId":"Zz+ZI7IcfhV3Rw79@nand.local","threadId":"62447","inReplyTo":"b8a055cb196dd971ac21611c1957be319557b4d3.1730775908.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 6/7] pack-objects: disable --full-name-hash when shallow","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-21T20:33:39Z","receivedAt":"2024-11-21T20:33:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Nov 05, 2024 at 03:05:06AM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee <stolee@gmail.com>\n>\n> As demonstrated in the previous change, the --full-name-hash option of\n> 'git pack-objects' is less effective in a trunctated history. Thus, even\n> when the option is selected via a command-line option or config, disable\n> this option when the '--shallow' option is specified. This will help\n> performance in servers that choose to enable the --full-name-hash option\n> by default for a repository while not regressing their ability to serve\n> shallow clones.\n>\n> This will not present a compatibility issue in the future when the full\n> name hash values are stored in the reachability bitmaps, since shallow\n> clones disable bitmaps.\n>\n> Signed-off-by: Derrick Stolee <stolee@gmail.com>\n> ---\n>  builtin/pack-objects.c       | 6 ++++++\n>  t/perf/p5313-pack-objects.sh | 1 +\n>  2 files changed, 7 insertions(+)\n\nI appreciate demonstrating the value of declaring --shallow and\n--full-name-hash incompatible by showing the performance numbers in the\nprevious patch.\n\nBut TBH I think that it would be equally fine or slightly better to say\nup front \"when combined with --shallow, this option produces larger\npacks during testing, so the two are incompatible for now\". You could\ninclude some performance numbers there to illustrate that difference in\nthe commit log too if you wanted.\n\nBut I don't think it's worth introducing the pair as compatible only to\nmark them incompatible later on in the same series.\n\nThanks,\nTaylor\n"},{"id":"507840","messageId":"Zz+bLOi6DZoD5CfI@nand.local","threadId":"62447","inReplyTo":"ab341dd0e58f77b3c7c6f5765d9e34cb02bef56f.1730775908.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 7/7] test-tool: add helper for name-hash values","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-21T20:42:20Z","receivedAt":"2024-11-21T20:42:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Nov 05, 2024 at 03:05:07AM +0000, Derrick Stolee via GitGitGadget wrote:\n> Test                                              this tree\n> -----------------------------------------------------------------\n> 5314.1: paths at head                                        4.5K\n> 5314.2: number of distinct name-hashes                       4.1K\n> 5314.3: number of distinct full-name-hashes                  4.5K\n> 5314.4: maximum multiplicity of name-hashes                    13\n> 5314.5: maximum multiplicity of fullname-hashes                 1\n>\n> Here, the maximum collision multiplicity is 13, but around 10% of paths\n> have a collision with another path.\n\nNeat.\n\n> diff --git a/t/helper/test-name-hash.c b/t/helper/test-name-hash.c\n> new file mode 100644\n> index 00000000000..e4ecd159b76\n> --- /dev/null\n> +++ b/t/helper/test-name-hash.c\n> @@ -0,0 +1,24 @@\n> +/*\n> + * test-name-hash.c: Read a list of paths over stdin and report on their\n> + * name-hash and full name-hash.\n> + */\n> +\n> +#include \"test-tool.h\"\n> +#include \"git-compat-util.h\"\n> +#include \"pack-objects.h\"\n> +#include \"strbuf.h\"\n> +\n> +int cmd__name_hash(int argc UNUSED, const char **argv UNUSED)\n> +{\n> +\tstruct strbuf line = STRBUF_INIT;\n> +\n> +\twhile (!strbuf_getline(&line, stdin)) {\n> +\t\tuint32_t name_hash = pack_name_hash(line.buf);\n> +\t\tuint32_t full_hash = pack_full_name_hash(line.buf);\n> +\n> +\t\tprintf(\"%10\"PRIu32\"\\t%10\"PRIu32\"\\t%s\\n\", name_hash, full_hash, line.buf);\n\nI'm definitely nitpicking, but having a tab to separate these two 32-bit\nvalues feels odd when we know already that they will be at most\n10-characters wide.\n\nI probably would have written:\n\n    printf(\"%10\"PRIu32\" %10\"PRIu32\"\\t%s\\n\", name_hash, full_hash, line.buf);\n\ninstead, but this is obviously not a big deal either way ;-).\n\n> diff --git a/t/perf/p5314-name-hash.sh b/t/perf/p5314-name-hash.sh\n> new file mode 100755\n> index 00000000000..9fe26612fac\n> --- /dev/null\n> +++ b/t/perf/p5314-name-hash.sh\n> @@ -0,0 +1,41 @@\n> +#!/bin/sh\n> +\n> +test_description='Tests pack performance using bitmaps'\n> +. ./perf-lib.sh\n> +\n> +GIT_TEST_PASSING_SANITIZE_LEAK=0\n> +export GIT_TEST_PASSING_SANITIZE_LEAK\n\nDoes this conflict with Patrick's series to remove these leak checking\nannotations? I think it might, which is not unexpected given this series\nwas written before that one (and it's my fault for not reviewing it\nearlier).\n\n> +test_perf_large_repo\n> +\n> +test_size 'paths at head' '\n> +\tgit ls-tree -r --name-only HEAD >path-list &&\n> +\twc -l <path-list\n> +'\n> +\n> +test_size 'number of distinct name-hashes' '\n> +\tcat path-list | test-tool name-hash >name-hashes &&\n> +\tcat name-hashes | awk \"{ print \\$1; }\" | sort -n | uniq -c >name-hash-count &&\n\nIn these two (and a handful of others lower down in this same script)\nthe \"cat ... |\" is unnecessary. I think this one should be written as:\n\n    test-tool name-hash <path-list >name-hashes &&\n    awk \"{ print \\$1; }\" <name-hashes | sort | uniq -c >name-hash-count &&\n\n(sort -n is unnecessary, since we just care about getting the list in\nsorted order so that \"uniq -c\" can count the number of unique values).\n\n> +\twc -l <name-hash-count\n> +'\n> +\n> +test_size 'number of distinct full-name-hashes' '\n> +\tcat name-hashes | awk \"{ print \\$2; }\" | sort -n | uniq -c >full-name-hash-count &&\n> +\twc -l <full-name-hash-count\n> +'\n> +\n> +test_size 'maximum multiplicity of name-hashes' '\n> +\tcat name-hash-count | \\\n> +\t\tsort -nr | \\\n> +\t\thead -n 1 | \\\n> +\t\tawk \"{ print \\$1; }\"\n> +'\n> +\n> +test_size 'maximum multiplicity of fullname-hashes' '\n> +\tcat full-name-hash-count | \\\n> +\t\tsort -nr | \\\n> +\t\thead -n 1 | \\\n> +\t\tawk \"{ print \\$1; }\"\n\nNitpicking again, but you could extract the \"sort | head | awk\" pipeline\ninto a function.\n\nThanks,\nTaylor\n"},{"id":"507847","messageId":"Zz+nk4w+y63vCupK@nand.local","threadId":"62447","inReplyTo":"Zz+TKS2O/ij6GZ1f@nand.local","subject":"Re: [PATCH 1/7] pack-objects: add --full-name-hash option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-21T21:35:19Z","receivedAt":"2024-11-21T21:35:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Nov 21, 2024 at 03:08:09PM -0500, Taylor Blau wrote:\n> The remaining parts of this change look good to me.\n\nOops, one thing I forgot (which reading Peff's message in [1] reminded\nme of) is that I think we need to disable full-name hashing when we're\nreusing existing packfiles as is the case with try_partial_reuse().\n\nThere we're always looking at classic name hash values, so mixing the\ntwo would be a mistake. I think that amounts to:\n\n--- 8< ---\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 762949e4c8..7e370bcfc9 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -4070,6 +4070,8 @@ static int get_object_list_from_bitmap(struct rev_info *revs)\n \tif (!(bitmap_git = prepare_bitmap_walk(revs, 0)))\n \t\treturn -1;\n\n+\tuse_full_name_hash = 0;\n+\n \tif (pack_options_allow_reuse())\n \t\treuse_partial_packfile_from_bitmap(bitmap_git,\n \t\t\t\t\t\t   &reuse_packfiles,\n--- >8 ---\n\nThanks,\nTaylor\n\n[1]: https://lore.kernel.org/git/20241104172533.GA2985568@coredump.intra.peff.net/\n"},{"id":"507866","messageId":"xmqqmshsduyh.fsf@gitster.g","threadId":"62447","inReplyTo":"Zz+nk4w+y63vCupK@nand.local","subject":"Re: [PATCH 1/7] pack-objects: add --full-name-hash option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-21T23:32:22Z","receivedAt":"2024-11-21T23:32:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> On Thu, Nov 21, 2024 at 03:08:09PM -0500, Taylor Blau wrote:\n>> The remaining parts of this change look good to me.\n>\n> Oops, one thing I forgot (which reading Peff's message in [1] reminded\n> me of) is that I think we need to disable full-name hashing when we're\n> reusing existing packfiles as is the case with try_partial_reuse().\n>\n> There we're always looking at classic name hash values, so mixing the\n> two would be a mistake. I think that amounts to:\n>\n> --- 8< ---\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 762949e4c8..7e370bcfc9 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -4070,6 +4070,8 @@ static int get_object_list_from_bitmap(struct rev_info *revs)\n>  \tif (!(bitmap_git = prepare_bitmap_walk(revs, 0)))\n>  \t\treturn -1;\n>\n> +\tuse_full_name_hash = 0;\n\nHmph, is this early enough, or has some other code path already\ncomputed the name hashes for the paths for the files to be packed?\n\n    ... Goes and looks ...\n\nThis is called from get_object_list() which\n\n - is not called under --stdin-packs,\n - is not called in cruft mode,\n - is not called when reading object list from --stdin\n\nso we are looking at the bog-standard \"objects to be packed are\ngiven in the form of rev-list command line options from our command\nline\".  And in the function, we walk the history near the end, which\nmakes show_object calls that adds object-entry with the name-hash.\nSo the call to get_object_list_from_bitmap() happens way before the\nfirst use of the name-hash function, so this is probably safe.\n\nAnd obviously get_object_list_from_bitmap() is the only place we\nselect objects to be packed from an existing pack and a bitmap file,\nso even if we gain new callers in the future, it is very likely that\nthe new callers would benefit from this change.\n\nOK.  Nicely done.\n\n>  \tif (pack_options_allow_reuse())\n>  \t\treuse_partial_packfile_from_bitmap(bitmap_git,\n>  \t\t\t\t\t\t   &reuse_packfiles,\n> --- >8 ---\n>\n> Thanks,\n> Taylor\n>\n> [1]: https://lore.kernel.org/git/20241104172533.GA2985568@coredump.intra.peff.net/\n"},{"id":"507867","messageId":"20241121235014.2554033-1-jonathantanmy@google.com","threadId":"62447","inReplyTo":"pull.1823.git.1730775907.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/7] pack-objects: Create an alternative name hash algorithm (recreated)","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2024-11-21T23:50:14Z","receivedAt":"2024-11-21T23:50:17Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n> This series introduces a new name-hash algorithm, but does not replace the\n> existing one. There are cases, such as packing a single snapshot of a\n> repository, where the existing algorithm outperforms the new one.\n\nI came up with a hash function that both uses information from a lot\nmore of the path (not the full name, though) and preserves the sortable\nproperty (diff at the end of this email). It also contains fixes to the\nexisting algorithm: not wasting the most significant bits of the hash\nif files in the repo mostly end in a lowercase alphabetic character, and\nthe cast from a possibly-signed-possibly-unsigned char to a uint32_t.\n\nThe results look quite good. In summary, the pack sizes are comparable\nto Stolee's results in the case of fluentui, and better than Stolee's\nresults in the case of git.\n\nHere's one run on the fluentui repo (git clone https://\ngithub.com/microsoft/fluentui; cd fluentui; git checkout\na637a06df05360ce5ff21420803f64608226a875^ following the instructions\nin [1]:\n\n(before my change)\n\nTest                                               this tree         \n---------------------------------------------------------------------\n5313.2: thin pack                                  0.03(0.01+0.01)   \n5313.3: thin pack size                                        1.1K   \n5313.4: thin pack with --full-name-hash            0.03(0.00+0.02)   \n5313.5: thin pack size with --full-name-hash                  3.0K   \n5313.6: big pack                                   1.60(2.87+0.32)   \n5313.7: big pack size                                        57.9M   \n5313.8: big pack with --full-name-hash             1.41(1.94+0.37)   \n5313.9: big pack size with --full-name-hash                  57.8M   \n5313.10: shallow fetch pack                        1.69(2.70+0.22)   \n5313.11: shallow pack size                                   33.0M   \n5313.12: shallow pack with --full-name-hash        1.49(1.84+0.34)   \n5313.13: shallow pack size with --full-name-hash             33.6M   \n5313.14: repack                                    75.10(537.66+5.47)\n5313.15: repack size                                        454.2M   \n5313.16: repack with --full-name-hash              18.10(92.50+5.14) \n5313.17: repack size with --full-name-hash                  174.8M                                \n\n(after my change)\n\nTest                                               this tree         \n---------------------------------------------------------------------\n5313.2: thin pack                                  0.03(0.01+0.02)   \n5313.3: thin pack size                                        1.1K   \n5313.4: thin pack with --full-name-hash            0.03(0.01+0.02)   \n5313.5: thin pack size with --full-name-hash                  1.1K   \n5313.6: big pack                                   1.62(2.94+0.28)   \n5313.7: big pack size                                        57.9M   \n5313.8: big pack with --full-name-hash             1.35(2.07+0.37)   \n5313.9: big pack size with --full-name-hash                  57.6M   \n5313.10: shallow fetch pack                        1.63(2.52+0.29)   \n5313.11: shallow pack size                                   33.0M   \n5313.12: shallow pack with --full-name-hash        1.50(2.10+0.23)   \n5313.13: shallow pack size with --full-name-hash             33.1M   \n5313.14: repack                                    74.86(531.39+5.49)\n5313.15: repack size                                        454.7M   \n5313.16: repack with --full-name-hash              19.71(111.39+5.12)\n5313.17: repack size with --full-name-hash                  165.6M  \n\nThe tests were run by:\n\n  GENERATE_COMPILATION_DATABASE=yes make CC=clang && (cd t/perf && env GIT_PERF_LARGE_REPO=~/tmp/fluentui ./run -- p5313*.sh)\n\nThe similarity in sizes looked suspicious, so I replaced the contents\nof the hash function with \"return 0;\" and indeed the sizes significantly\nincreased, so hopefully there is nothing wrong with my setup.\n\nThe git repo was called out in [1] as demonstrating \"some of the issues\nwith this approach\", but here are the results, run by:\n\n  GENERATE_COMPILATION_DATABASE=yes make CC=clang && (cd t/perf && ./run -- p5313*.sh)\n\nTest                                               this tree        \n--------------------------------------------------------------------\n5313.2: thin pack                                  0.03(0.00+0.02)  \n5313.3: thin pack size                                        2.9K  \n5313.4: thin pack with --full-name-hash            0.03(0.00+0.02)   \n5313.5: thin pack size with --full-name-hash                  2.9K                                                                                                                                                  \n5313.6: big pack                                   1.69(2.80+0.28)                                                                                                                                                  \n5313.7: big pack size                                        18.7M  \n5313.8: big pack with --full-name-hash             1.68(2.82+0.31)  \n5313.9: big pack size with --full-name-hash                  18.8M  \n5313.10: shallow fetch pack                        0.96(1.47+0.16)  \n5313.11: shallow pack size                                   12.1M  \n5313.12: shallow pack with --full-name-hash        1.01(1.51+0.14)  \n5313.13: shallow pack size with --full-name-hash             12.1M  \n5313.14: repack                                    17.05(69.99+4.33)\n5313.15: repack size                                        116.5M  \n5313.16: repack with --full-name-hash              15.74(67.03+4.18)\n5313.17: repack size with --full-name-hash                  116.1M  \n\n[1] https://lore.kernel.org/git/c14ef6879e451401381ebbdb8f30d33c8f56c25b.1730775908.git.gitgitgadget@gmail.com/\n\n> | Repo     | Standard Repack | With --full-name-hash |\n> |----------|-----------------|-----------------------|\n> | fluentui |         438 MB  |               168 MB  |\n> | Repo B   |       6,255 MB  |               829 MB  |\n> | Repo C   |      37,737 MB  |             7,125 MB  |\n> | Repo D   |     130,049 MB  |             6,190 MB  |\n> | Repo E   |     100,957 MB  |            22,979 MB  |\n> | Repo F   |       8,308 MB  |               746 MB  |\n> | Repo G   |       4,329 MB  |             3,643 MB  |\n\nIf the results are similar for some of the above repos (I do not have\naccess to them), maybe it's worth considering using my hash function (or\na variation of it).\n\nI'll also take a look at the rest of the patch set.\n\n---\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 88360aa3e8..c4f35eafa0 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -209,23 +209,24 @@ static inline uint32_t pack_name_hash(const char *name)\n \n static inline uint32_t pack_full_name_hash(const char *name)\n {\n-       const uint32_t bigp = 1234572167U;\n-       uint32_t c, hash = bigp;\n+       uint32_t hash = 0, base = 0;\n+       uint8_t c;\n \n        if (!name)\n                return 0;\n \n-       /*\n-        * Do the simplest thing that will resemble pseudo-randomness: add\n-        * random multiples of a large prime number with a binary shift.\n-        * The goal is not to be cryptographic, but to be generally\n-        * uniformly distributed.\n-        */\n-       while ((c = *name++) != 0) {\n-               hash += c * bigp;\n-               hash = (hash >> 5) | (hash << 27);\n+       while ((c = (uint8_t) *name++) != 0) {\n+               if (isspace(c))\n+                       continue;\n+               if (c == '/') {\n+                       base = (base >> 6) ^ hash;\n+                       hash = 0;\n+               } else {\n+                       uint8_t nybble_swapped = (c >> 4) + ((c & 15) << 4);\n+                       hash = (hash >> 2) + (nybble_swapped << 24);\n+               }\n        }\n-       return hash;\n+       return (base >> 6) ^ hash;\n }\n"},{"id":"507871","messageId":"20241122011308.2743517-1-jonathantanmy@google.com","threadId":"62447","inReplyTo":"259734e0bcea952c2c09b0fb3a017e139922b975.1730775908.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/7] pack-objects: add GIT_TEST_FULL_NAME_HASH","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2024-11-22T01:13:08Z","receivedAt":"2024-11-22T01:13:11Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n> Second, there are two tests in t5616-partial-clone.sh that I believe are\n> actually broken scenarios. \n\nI took a look...this is a tricky one.\n\n> While the client is set up to clone the\n> 'promisor-server' repo via a treeless partial clone filter (tree:0),\n> that filter does not translate to the 'server' repo. Thus, fetching from\n> these repos causes the server to think that the client has all reachable\n> trees and blobs from the commits advertised as 'haves'. This leads the\n> server to providing a thin pack assuming those objects as delta bases.\n\nIt is expected that the server sometimes sends deltas based on objects\nthat the client doesn't have. In fact, this test tests the ability of\nGit to lazy-fetch delta bases.\n\n> Changing the name-hash algorithm presents new delta bases and thus\n> breaks the expectations of these tests.\n\nTo be precise, the change resulted in no deltas being sent (before this\nchange, one delta was sent). Here's what is meant to happen. The server has:\n\n commitB - treeB - file1 (\"make the tree big\\nanother line\\n\"), file2...file100\n  |\n commitA - treeA - file1...file100 (\"make the tree big\\n\")\n\nThe client only has commitA. (The client does not have treeA or any\nblob, since it was cloned with --filter=tree:0.)\n\nWhen GIT_TEST_FULL_NAME_HASH=0 (matching the current behavior), the\nserver sends a non-delta commitB, a delta treeB (with base treeA), and\na non-delta blob \"make the tree big\\nanother line\\n\". This triggers a\nlazy fetch of treeA, and thus treeB is inflated successfully. During\nthe subsequent connectivity check (with --exclude-promisor-objects,\nsee connected.c), it is noticed that the \"make the tree big\\n\" blob is\nmissing, but since it is a promisor object (referenced by treeA, which\nwas fetched from the promisor remote), the connectivity check since\npasses.\n\nWhen GIT_TEST_FULL_NAME_HASH=1, the server sends a non-delta commitB,\na non-delta treeB, and a non-delta blob \"make the tree big\\nanother\nline\\n\". No lazy fetch is triggered. During the subsequent connectivity\ncheck, the \"make the tree big\\n\" blob (referenced by treeB) is missing.\nThere is nothing that can vouch for it (the client does not have treeA,\nremember) so the client does not consider it a promisor object, and thus\nthe connectivity check fails.\n\nInvestigating this was made a bit harder due to a missing \"git -C\npromisor-remote config --local uploadpack.allowfilter 1\" in the test.\nThe above behavior is after this is included in the test.\n\nI think the solution is to have an algorithm that preserves the property\nthat treeB is sent as a delta object - if not, we need to find another\nway to test the lazy-fetch of delta bases. My proposal in [1] does do\nthat.\n\n[1] https://lore.kernel.org/git/20241121235014.2554033-1-jonathantanmy@google.com/\n\n"},{"id":"507873","messageId":"20241122012359.2764951-1-jonathantanmy@google.com","threadId":"62447","inReplyTo":"ab341dd0e58f77b3c7c6f5765d9e34cb02bef56f.1730775908.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 7/7] test-tool: add helper for name-hash values","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2024-11-22T01:23:58Z","receivedAt":"2024-11-22T01:24:02Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n> From: Derrick Stolee <stolee@gmail.com>\n> \n> Add a new test-tool helper, name-hash, to output the value of the\n> name-hash algorithms for the input list of strings, one per line.\n\nI've looked at all 7 patches.\n\nI didn't really understand the concern with shallow in patch 6 (in\nparticular, the documentation of \"git pack-objects --shallow\" seems\nto imply that it's for use by a server to a shallow client, but at\nthe point that the server would need such a feature, it probably would\nalready have bitmaps packed with the new hash algorithm). I didn't look\nat it further, though, since I had an algorithm that seemed to also do\nOK in the shallow test. So we might be able to drop patch 6.\n\nOther than that, and other than all my comments and Taylor's comments,\nthis series looks good.\n \n"},{"id":"507878","messageId":"xmqqiksgas54.fsf@gitster.g","threadId":"62447","inReplyTo":"20241121235014.2554033-1-jonathantanmy@google.com","subject":"Re: [PATCH 0/7] pack-objects: Create an alternative name hash algorithm (recreated)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-22T03:01:27Z","receivedAt":"2024-11-22T03:01:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> +       while ((c = (uint8_t) *name++) != 0) {\n> +               if (isspace(c))\n> +                       continue;\n> +               if (c == '/') {\n> +                       base = (base >> 6) ^ hash;\n> +                       hash = 0;\n> +               } else {\n> +                       uint8_t nybble_swapped = (c >> 4) + ((c & 15) << 4);\n> +                       hash = (hash >> 2) + (nybble_swapped << 24);\n> +               }\n>         }\n> +       return (base >> 6) ^ hash;\n>  }\n\nNice.  The diff relative to the --full-name-hash version is a bit\nhard to grok, but compared to the current hash function, there are\ntwo and a half changes that matter:\n\n (0) it is more careful with bytes with the MSB set (i.e. non-ASCII\n     pathnames).\n\n (1) it hashes each path component separetely and rotates the whole\n     thing only at a directory boundary.  I'd imagine that this\n     would make a big difference for languages that force overly\n     long filenames at each level.\n\n (2) it gives more weight to lower bits by swapping nybbles of each\n     byte.\n\nI wonder if we do even better if we reverse all 8 bits instead of\nswapping nybbles (if we were to do so, it might be more efficient to\nshift in from the right instead of left end of the base and hash\naccumulators in the loop and then swap the whole resulting word at\nthe end).\n\nThanks for a fun read.\n"},{"id":"507879","messageId":"xmqqcyioar4r.fsf@gitster.g","threadId":"62447","inReplyTo":"20241122011308.2743517-1-jonathantanmy@google.com","subject":"Re: [PATCH 3/7] pack-objects: add GIT_TEST_FULL_NAME_HASH","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-22T03:23:16Z","receivedAt":"2024-11-22T03:23:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> ... During the subsequent connectivity\n> check, the \"make the tree big\\n\" blob (referenced by treeB) is missing.\n> There is nothing that can vouch for it (the client does not have treeA,\n> remember) so the client does not consider it a promisor object, and thus\n> the connectivity check fails.\n\nIt is sad that it is a (probably unfixable) flaw in the \"promisor\nobject\" concept that the \"promisor object\"-ness of blobA depends on\nthe lazy-fetch status of treeA.  This is not merely a test failure,\nbut it would cause blobA pruned if such a lazy fetch happens in the\nwild and then \"git gc\" triggers, no?  It may not manifest as a\nrepository corruption, since we would lazily fetch it again if the\nuser requests to fully fetch what commitA and treeA need, but it\ndoes feel somewhat suboptimal.\n\nThanks for a detailed explanation on what is going on.\n"},{"id":"507882","messageId":"xmqq1pz3c2y1.fsf@gitster.g","threadId":"62447","inReplyTo":"xmqqiksgas54.fsf@gitster.g","subject":"Re: [PATCH 0/7] pack-objects: Create an alternative name hash algorithm (recreated)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-22T04:22:46Z","receivedAt":"2024-11-22T04:22:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> ... (if we were to do so, it might be more efficient to\n> shift in from the right instead of left end of the base and hash\n> accumulators in the loop and then swap the whole resulting word at\n> the end).\n\nThat is garbage.  We could do the \"shift in from the right and then\nreverse the result\" for the hash accumulator, but not the \"base\"\none.  Sorry for the noise.\n\n"},{"id":"507912","messageId":"ec17fdd8-310e-42b6-bdf6-2620a84c2eb3@gmail.com","threadId":"62447","inReplyTo":"Zz+nk4w+y63vCupK@nand.local","subject":"Re: [PATCH 1/7] pack-objects: add --full-name-hash option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2024-11-22T11:46:04Z","receivedAt":"2024-11-22T11:46:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/21/24 4:35 PM, Taylor Blau wrote:\n> On Thu, Nov 21, 2024 at 03:08:09PM -0500, Taylor Blau wrote:\n>> The remaining parts of this change look good to me.\n> \n> Oops, one thing I forgot (which reading Peff's message in [1] reminded\n> me of) is that I think we need to disable full-name hashing when we're\n> reusing existing packfiles as is the case with try_partial_reuse().\n> \n> There we're always looking at classic name hash values, so mixing the\n> two would be a mistake. I think that amounts to:\n> \n> --- 8< ---\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 762949e4c8..7e370bcfc9 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -4070,6 +4070,8 @@ static int get_object_list_from_bitmap(struct rev_info *revs)\n>   \tif (!(bitmap_git = prepare_bitmap_walk(revs, 0)))\n>   \t\treturn -1;\n> \n> +\tuse_full_name_hash = 0;\n> +\nThanks. I have applied this code change with a comment detailing\nthe context around the bitmap file storing only the default name-hash\n(for now) but that it can change in the future.\n\nThanks,\n-Stolee\n\n"},{"id":"507913","messageId":"fbc0d959-48ea-4e24-aa8d-31f573f579e8@gmail.com","threadId":"62447","inReplyTo":"Zz+TKS2O/ij6GZ1f@nand.local","subject":"Re: [PATCH 1/7] pack-objects: add --full-name-hash option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2024-11-22T11:59:28Z","receivedAt":"2024-11-22T11:59:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/21/24 3:08 PM, Taylor Blau wrote:\n> On Tue, Nov 05, 2024 at 03:05:01AM +0000, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee <stolee@gmail.com>\n\n>> It is clear that the existing name-hash algorithm is optimized for\n>> repositories with short path names, but also is optimized for packing a\n>> single snapshot of a repository, not a repository with many versions of\n>> the same file. In my testing, this has proven out where the name-hash\n>> algorithm does a good job of finding peer files as delta bases when\n>> unable to use a historical version of that exact file.\n> \n> I'm not sure I entirely agree with the suggestion that the existing hash\n> function is only about packing repositories with short pathnames. I\n> think an important part of the existing implementation is that tries to\n> group similar files together, regardless of whether or not they appear\n> in the same tree.\n\nI'll be more explicit about the design for \"hash locality\" earlier in\nthe message, but also pointing out that the locality only makes sense as\na benefit when there are not enough versions of a file in history, since\nit's nearly always better to choose a previous version of the same file\ninstead of a different path with a name-hash collision. Directory renames\nare on place where this is a positive decision, but those are typically\nrare compared to the full history of a large repo.\n\n>> This is not meant to be cryptographic at all, but uniformly distributed\n>> across the possible hash values. This creates a hash that appears\n>> pseudorandom. There is no ability to consider similar file types as\n>> being close to each other.\n> \n> I think you hint at this in the series' cover letter, but I suspect that\n> this pseduorandom behavior hurts in some small number of cases and that\n> the full-name hash idea isn't a pure win, e.g., when we really do want\n> to delta two paths that both end in CHAGNELOG.json despite being in\n> different parts of the tree.\n\nI mention that this doesn't work well in all cases when operating under\na 'git push' or in a shallow clone. Shallow clones are disabled in a later\ncommit and we don't have the necessary implementation to make this hash\nfunction be selected within 'git push'.\n\n> You have some tables here below that demonstrate a significant\n> improvement with the full-name hash in use, which I think is good worth\n> keeping in my own opinion. It may be worth updating those to include the\n> new examples you highlighted in your revised cover letter as well.\n\nI'll try to remember to move the newer examples to the cover letter.\n\n>> In a later change, a test-tool will be added so the effectiveness of\n>> this hash can be demonstrated directly.\n>>\n>> For now, let's consider how effective this mechanism is when repacking a\n>> repository with and without the --full-name-hash option. Specifically,\n> \n> Is this repository publicly available? If so, it may be worth mentioning\n> here.\n\nHere, by \"when repacking a repository\" I mean \"we are going to test\nrepacking a number of example repositories, that will be listed in detail\nin the coming tables\".\n\n>> Using a collection of repositories that use the beachball tool, I was\n>> able to make similar comparisions with dramatic results. While the\n>> fluentui repo is public, the others are private so cannot be shared for\n>> reproduction. The results are so significant that I find it important to\n>> share here:\n>>\n>> | Repo     | Standard Repack | With --full-name-hash |\n>> |----------|-----------------|-----------------------|\n>> | fluentui |         438 MB  |               168 MB  |\n>> | Repo B   |       6,255 MB  |               829 MB  |\n>> | Repo C   |      37,737 MB  |             7,125 MB  |\n>> | Repo D   |     130,049 MB  |             6,190 MB  |\n\nThese repos B, C, and D are _not_ publicly available, though.\n\n>> diff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\n>> index e32404c6aae..93861d9f85b 100644\n>> --- a/Documentation/git-pack-objects.txt\n>> +++ b/Documentation/git-pack-objects.txt\n>> @@ -15,7 +15,8 @@ SYNOPSIS\n>>   \t[--revs [--unpacked | --all]] [--keep-pack=<pack-name>]\n>>   \t[--cruft] [--cruft-expiration=<time>]\n>>   \t[--stdout [--filter=<filter-spec>] | <base-name>]\n>> -\t[--shallow] [--keep-true-parents] [--[no-]sparse] < <object-list>\n>> +\t[--shallow] [--keep-true-parents] [--[no-]sparse]\n>> +\t[--full-name-hash] < <object-list>\n> \n> OK, I see that --full-name-hash is now listed in the synopsis, but I\n> don't see a corresponding description of what the option does later on\n> in this file. I took a look through the remaining patches in this series\n> and couldn't find any further changes to git-pack-objects(1) either.\n\nI'll fix that. Thanks. As well as moving the 'git repack' changes out\nof this patch. I'll adjust the commit message to say \"packing all objects'\ninstead of 'git repack' to be clear that this can be done with a direct\ncall to 'git pack-objects' instead of needing 'git repack'.\n\n>> +\tif (write_bitmap_index && use_full_name_hash) {\n>> +\t\twarning(_(\"currently, the --full-name-hash option is incompatible with --write-bitmap-index\"));\n>> +\t\tuse_full_name_hash = 0;\n>> +\t}\n>> +\n> \n> Good, we determine this early on in the command, so we don't risk\n> computing different hash functions within the same process.\n> \n> I wonder if it's worth guarding against mixing the hash functions within\n> the pack_name_hash() and pack_full_name_hash() functions themselves. I'm\n> thinking something like:\n> \n>      static inline uint32_t pack_name_hash(const char *name)\n>      {\n>          if (use_full_name_hash)\n>              BUG(\"called pack_name_hash() with --full-name-hash\")\n>          /* ... */\n>      }\n> \n> and the inverse in pack_full_name_hash(). I don't think it's strictly\n> necessary, but it would be a nice guard against someone calling, e.g.,\n> pack_full_name_hash() directly instead of pack_name_hash_fn().\n\nI think this is interesting defensive programming for future contributions.\n\nWe essentially want the methods to only be called by pack_name_hash_fn()\nand don't have method privacy. We could extract it to its own header file\nbut then would need to modify the prototype to include the signal for\nwhich hash type to use, but that would cause us to lose our ability to\ncheck for a bug like this.\n\nIt may be even better to store a static value for the value of\nuse_full_name_hash when it first executes, so it can exit if it notices\na different value. (This is becoming large enough for its own patch.)\n\n> The other small thought I had here is that we should use the convenience\n> function die_for_incompatible_opt3() here, since it uses an existing\n> translation string for pairs of incompatible options.\n> \n> (As an aside, though that function is actually implemented in the\n> _opt4() variant, and it knows how to handle a pair, trio, and quartet of\n> mutually incompatible options, there is no die_for_incompatible_opt2()\n> function. It may be worth adding one here since I'm sure there are other\n> spots which would benefit from such a function).\n\nInteresting. I've not considered these functions before.\n\n>> diff --git a/builtin/repack.c b/builtin/repack.c\n>> index d6bb37e84ae..ab2a2e46b20 100644\n>> --- a/builtin/repack.c\n>> +++ b/builtin/repack.c\n> \n> I'm surprised to see the new option plumbed into repack in this commit.\n> I would have thought that it'd appear in the subsequent commit instead.\n> The implementation below looks good, I just imagined it would be placed\n> in the next commit instead of this one.\n\nYes, I should delay that to patch 2.\n\nThanks,\n-Stole\n\n\n"},{"id":"507914","messageId":"5400973a-290f-4fb6-a4cb-28d2effd1d83@gmail.com","threadId":"62447","inReplyTo":"Zz+UJHclSsb+Bgfo@nand.local","subject":"Re: [PATCH 2/7] repack: add --full-name-hash option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2024-11-22T12:07:12Z","receivedAt":"2024-11-22T12:07:14Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/21/24 3:12 PM, Taylor Blau wrote:\n> On Tue, Nov 05, 2024 at 03:05:02AM +0000, Derrick Stolee via GitGitGadget wrote:\n>> ---\n>>   t/t7700-repack.sh       |  7 +++++++\n>>   t/test-lib-functions.sh | 26 ++++++++++++++++++++++++++\n>>   2 files changed, 33 insertions(+)\n> \n> OK, I stand by my thinking in the previous patch that this one is where\n> the changes to builtin/repack.c belong.\n\nYes. I should have done this already.\n\n> I do think that test_subcommand_flex may be unnecessary though, since\n> you could instead write this as:\n> \n>      test_subcommand \"git pack-objects.*--full-name-hash\" <full-trace.txt\n> \n> and get the same behavior.\nThis does require knowing a bit about the internals of test_subcommand\nthat may be too much of a burden for future contributors.\n\nThanks,\n-Stolee\n\n"},{"id":"507915","messageId":"4c314b69-46b4-402e-a590-78e4f4e0200e@gmail.com","threadId":"62447","inReplyTo":"Zz+U8IyHqBNRIn6m@nand.local","subject":"Re: [PATCH 3/7] pack-objects: add GIT_TEST_FULL_NAME_HASH","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2024-11-22T12:09:45Z","receivedAt":"2024-11-22T12:09:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/21/24 3:15 PM, Taylor Blau wrote:\n> On Tue, Nov 05, 2024 at 03:05:03AM +0000, Derrick Stolee via GitGitGadget wrote:\n>> diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\n>> index 2e28d02b20f..75b40f07bbd 100755\n>> --- a/ci/run-build-and-tests.sh\n>> +++ b/ci/run-build-and-tests.sh\n>> @@ -30,6 +30,7 @@ linux-TEST-vars)\n>>   \texport GIT_TEST_NO_WRITE_REV_INDEX=1\n>>   \texport GIT_TEST_CHECKOUT_WORKERS=2\n>>   \texport GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL=1\n>> +\texport GIT_TEST_FULL_NAME_HASH=1\n>>   \t;;\n>>   linux-clang)\n>>   \texport GIT_TEST_DEFAULT_HASH=sha1\n> \n> Hmm. I appreciate what this new GIT_TEST_ variable is trying to do, but\n> I am somewhat saddened to see this list in linux-TEST-vars growing\n> rather than shrinking.\nYou make good points that this does not need to be here.\n\nIt's enough that someone could manually check the test suite\nwith this test variable to make sure that enough of the other\noptions are tested with this feature.\n\nThanks,\n-Stolee\n\n"},{"id":"507930","messageId":"6a2f399c-8d65-4d1b-a541-b139d3e8fbf9@gmail.com","threadId":"62447","inReplyTo":"Zz+VRZnWjsDI9bLt@nand.local","subject":"Re: [PATCH 4/7] git-repack: update usage to match docs","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2024-11-22T15:26:13Z","receivedAt":"2024-11-22T15:26:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/21/24 3:17 PM, Taylor Blau wrote:\n> On Tue, Nov 05, 2024 at 03:05:04AM +0000, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee <stolee@gmail.com>\n>>\n>> This also adds the '--full-name-hash' option introduced in the previous\n>> change and adds newlines to the synopsis.\n> \n> I think \"the previous change\" is not quite accurate here, even if\n> you move the implementation to pass through '--full-name-hash' via\n> repack into the second patch.\n\nAh, I should definitely rearrange the commits.\n\n> It would be nice to have the option added in 'repack' in the same commit\n> as adjusts the documentation instead of splitting them apart.\nPart of the point of the split was that the synopsis in builtin/repack.c\nneeds more than just the addition of the --full-name-hash option in order\nto make it match the Documentation synopsis.\n\nBut you're right, the code change is small enough that these things can\nbe combined.\n\nThanks,\n-Stolee\n\n"},{"id":"507931","messageId":"233d4261-64b3-4149-bb6f-aeba66834c1c@gmail.com","threadId":"62447","inReplyTo":"Zz+YrvL8h0Cxwqfy@nand.local","subject":"Re: [PATCH 5/7] p5313: add size comparison test","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2024-11-22T15:26:19Z","receivedAt":"2024-11-22T15:26:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/21/24 3:31 PM, Taylor Blau wrote:\n> On Tue, Nov 05, 2024 at 03:05:05AM +0000, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee <stolee@gmail.com>\n\n>> The thin pack that simulates a push is much worse with --full-name-hash\n>> in this case. The name hash values are doing a lot to assist with delta\n>> bases, it seems. The big pack and shallow clone cases are slightly worse\n>> with the --full-name-hash option. Only the full repack gains some\n>> benefits in size.\n> \n> Not a problem with your patch, but just thinking aloud: do you think\n> there is an easy/straightforward way to suggest when to use\n> --full-name-hash or not?\n\nThe kinds of heuristics I would use are:\n\n1. Are there enough commits that enough files have enough versions\n    across history that it's very important to keep deltas within a path?\n\n2. Is the repository at least 500MB such that there is actually room for\n    a \"meaningful\" change in size?\n\n3. Are there a lot of name-hash collisions? (The last patch in the series\n    helps do this through a test-helper, but isn't something we can expect\n    end users to check themselves.)\n\n\n>> +\tcat >in-shallow <<-EOF\n>> +\t$(git rev-parse HEAD)\n>> +\t--shallow $(git rev-parse HEAD)\n>> +\tEOF\n>> +'\n> \n> I was going to comment that these could probably be moved into the\n> individual perf test that cares about reading each of these inputs. But\n> having them shared here makes sense since we are naturally comparing\n> generating two packs with the same input (with and without\n> --full-name-hash). So the shared setup here makes sense to me.\n\nI also wanted to avoid having these commands be part of the time\nmeasurement, even if they are extremely small.\n\n>> +\n>> +test_perf 'thin pack' '\n>> +\tgit pack-objects --thin --stdout --revs --sparse  <in-thin >out\n>> +'\n>> +\n>> +test_size 'thin pack size' '\n>> +\ttest_file_size out\n>> +'\n> \n> Nice. I always forget about this and end up writing 'wc -c <out'.\n\nI believe this is a Junio recommendation from an earlier version.\n\n>> +test_size 'repack size' '\n>> +\tpack=$(ls .git/objects/pack/pack-*.pack) &&\n>> +\ttest_file_size \"$pack\"\n> \n> Here and below, I think it's fine to inline this as in:\n> \n>      test_file_size \"$(ls .git/objects/pack/pack-*.pack)\"\n\nGenerally I prefer to split things into stages so the verbose output\nprovides a clear definition of the value when calling the Git command.\n\n> ...but I wonder: will using \".git\" break this test in bare repositories?\n> Should we write instead:\n> \n>      pack=\"$(ls $(git rev-parse --git-dir)/objects/pack/pack-*.pack)\" &&\n>      test_file_size\n> \n> ?\nWhile this would break a bare repo, the perf lib makes a bare repo be\ncopied into a non-bare repo as follows:\n\ntest_perf_copy_repo_contents () {\n\tfor stuff in \"$1\"/*\n\tdo\n\t\tcase \"$stuff\" in\n\t\t*/objects|*/hooks|*/config|*/commondir|*/gitdir|*/worktrees|*/fsmonitor--daemon*)\n\t\t\t;;\n\t\t*)\n\t\t\tcp -R \"$stuff\" \"$repo/.git/\" || exit 1\n\t\t\t;;\n\t\tesac\n\tdone\n}\n\nI'll still add the `git rev-parse` suggestion because it's safest.\n\nThanks,\n-Stolee\n\n"},{"id":"507932","messageId":"0314efa2-606f-4a89-b93e-f7a6684d05c7@gmail.com","threadId":"62447","inReplyTo":"Zz+ZI7IcfhV3Rw79@nand.local","subject":"Re: [PATCH 6/7] pack-objects: disable --full-name-hash when shallow","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2024-11-22T15:27:08Z","receivedAt":"2024-11-22T15:27:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/21/24 3:33 PM, Taylor Blau wrote:\n> On Tue, Nov 05, 2024 at 03:05:06AM +0000, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee <stolee@gmail.com>\n>>\n>> As demonstrated in the previous change, the --full-name-hash option of\n>> 'git pack-objects' is less effective in a trunctated history. Thus, even\n>> when the option is selected via a command-line option or config, disable\n>> this option when the '--shallow' option is specified. This will help\n>> performance in servers that choose to enable the --full-name-hash option\n>> by default for a repository while not regressing their ability to serve\n>> shallow clones.\n>>\n>> This will not present a compatibility issue in the future when the full\n>> name hash values are stored in the reachability bitmaps, since shallow\n>> clones disable bitmaps.\n>>\n>> Signed-off-by: Derrick Stolee <stolee@gmail.com>\n>> ---\n>>   builtin/pack-objects.c       | 6 ++++++\n>>   t/perf/p5313-pack-objects.sh | 1 +\n>>   2 files changed, 7 insertions(+)\n> \n> I appreciate demonstrating the value of declaring --shallow and\n> --full-name-hash incompatible by showing the performance numbers in the\n> previous patch.\n> \n> But TBH I think that it would be equally fine or slightly better to say\n> up front \"when combined with --shallow, this option produces larger\n> packs during testing, so the two are incompatible for now\". You could\n> include some performance numbers there to illustrate that difference in\n> the commit log too if you wanted.\n> \n> But I don't think it's worth introducing the pair as compatible only to\n> mark them incompatible later on in the same series.\nI disagree and here's why: they are not functionally incompatible. This\nperformance-focused change is worth justifying with performance test data\n_and_ isolating from the initial implementation with its own reasoning\nfor future history spelunkers. Having these warning lines blame to this\npatch instead of the initial implementation will make it much easier to\nunderstand the justification of this change.\n\nBut maybe this patch can be removed if we use Jonathan's function. I'll\ncheck the performance tests to see if this continues to be justified.\n\nThanks,\n-Stolee\n\n"},{"id":"507933","messageId":"cd3df4d5-efa0-45cb-ab94-6c5c9f0ac695@gmail.com","threadId":"62447","inReplyTo":"xmqqiksgas54.fsf@gitster.g","subject":"Re: [PATCH 0/7] pack-objects: Create an alternative name hash algorithm (recreated)","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2024-11-22T15:27:19Z","receivedAt":"2024-11-22T15:27:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/21/24 10:01 PM, Junio C Hamano wrote:\n> Jonathan Tan <jonathantanmy@google.com> writes:\n> \n>> +       while ((c = (uint8_t) *name++) != 0) {\n>> +               if (isspace(c))\n>> +                       continue;\n>> +               if (c == '/') {\n>> +                       base = (base >> 6) ^ hash;\n>> +                       hash = 0;\n>> +               } else {\n>> +                       uint8_t nybble_swapped = (c >> 4) + ((c & 15) << 4);\n>> +                       hash = (hash >> 2) + (nybble_swapped << 24);\n>> +               }\n>>          }\n>> +       return (base >> 6) ^ hash;\n>>   }\n> \n> Nice.  The diff relative to the --full-name-hash version is a bit\n> hard to grok, but compared to the current hash function, there are\n> two and a half changes that matter:\n> \n>   (0) it is more careful with bytes with the MSB set (i.e. non-ASCII\n>       pathnames).\n> \n>   (1) it hashes each path component separetely and rotates the whole\n>       thing only at a directory boundary.  I'd imagine that this\n>       would make a big difference for languages that force overly\n>       long filenames at each level.\n\nI was confused by the \"rotates the whole thing only at a directory\nboundary\" statement. I think one way to say what you mean is\n\n   Each path component is hashed similarly to the standard name-hash,\n   and parent path component hashes are contributed via XOR after a\n   down-shift of 6 bits per level.\n\nSo we are getting something like\n\n\t[ name-hash for level 0           ]\n         ......[ name-hash for level 1     ](truncated by 6)\n  \t............[name-hash for level 2](truncated by 12)\n  \t..................[...for level 3 ](truncated by 18)\n  \t........................[ level 4 ](truncated by 24)\n  \t..............................[ 5 ](truncated by 30)\n\nand at each layer we get the \"last 16 bytes matter\" issue, though it\nis balanced quite well. Also, the name-hash in each layer is adjusted\nfor nybble swaps.\n\n(I don't think my explanation is _better_ but just that it matches my\npersonal mental model slightly better.)\n\n>   (2) it gives more weight to lower bits by swapping nybbles of each\n>       byte.\n> \n> I wonder if we do even better if we reverse all 8 bits instead of\n> swapping nybbles (if we were to do so, it might be more efficient to\n> shift in from the right instead of left end of the base and hash\n> accumulators in the loop and then swap the whole resulting word at\n> the end).\nI will give this a try in my private repos as well as with the name-hash\ncollision perf test from patch 7.\n\nThanks,\n-Stolee\n\n"},{"id":"507950","messageId":"20241122180144.523048-1-jonathantanmy@google.com","threadId":"62447","inReplyTo":"xmqqcyioar4r.fsf@gitster.g","subject":"Re: [PATCH 3/7] pack-objects: add GIT_TEST_FULL_NAME_HASH","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2024-11-22T18:01:44Z","receivedAt":"2024-11-22T18:01:46Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Junio C Hamano <gitster@pobox.com> writes:\n> It is sad that it is a (probably unfixable) flaw in the \"promisor\n> object\" concept that the \"promisor object\"-ness of blobA depends on\n> the lazy-fetch status of treeA.  This is not merely a test failure,\n> but it would cause blobA pruned if such a lazy fetch happens in the\n> wild and then \"git gc\" triggers, no?  \n\nRight now, it won't be pruned since we never prune promisor objects\n(we just concatenate all of them into one file). But in the future, we\nmight only keep reachable promisor objects, in which case, yes, blobA\nwill be pruned. In this case, though, I think blobA is like any other\nunreachable object in git. If a user memorizes a commit hash but does\nnot point a ref to it (or point a ref to one of its descendants), that\ncommit is still subject to being lost by GC. I think it's the same case\nhere.\n"},{"id":"507951","messageId":"20241122180530.530499-1-jonathantanmy@google.com","threadId":"62447","inReplyTo":"xmqqiksgas54.fsf@gitster.g","subject":"Re: [PATCH 0/7] pack-objects: Create an alternative name hash algorithm (recreated)","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2024-11-22T18:05:30Z","receivedAt":"2024-11-22T18:05:33Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Junio C Hamano <gitster@pobox.com> writes:\n> I wonder if we do even better if we reverse all 8 bits instead of\n> swapping nybbles (if we were to do so, it might be more efficient to\n> shift in from the right instead of left end of the base and hash\n> accumulators in the loop and then swap the whole resulting word at\n> the end).\n> \n> Thanks for a fun read.\n\nAh, yes, reversing is better than swapping nybbles (the least\nsignificant 2 bits have more entropy than the next-least significant\n2 bits). When writing this, I didn't think of shifting in from the\nright (if I had thought of that, I would have indeed reversed the bits\ninstead).\n"},{"id":"507997","messageId":"xmqqed309oca.fsf@gitster.g","threadId":"62447","inReplyTo":"cd3df4d5-efa0-45cb-ab94-6c5c9f0ac695@gmail.com","subject":"Re: [PATCH 0/7] pack-objects: Create an alternative name hash algorithm (recreated)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-24T23:57:57Z","receivedAt":"2024-11-24T23:58:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee <stolee@gmail.com> writes:\n\n>>   (1) it hashes each path component separetely and rotates the whole\n>>       thing only at a directory boundary.  I'd imagine that this\n>>       would make a big difference for languages that force overly\n>>       long filenames at each level.\n>\n> I was confused by the \"rotates the whole thing only at a directory\n> boundary\" statement.\n\nYeah, I guess it was confusing.  What I meant was that the entire\nresult is shifted down with new material from left to right, but\nunlike the original, the outer thing (i.e. what is given to the\ncaller as the result) is shifted only at the directory boundary, so\nwe are not as aggressive to lose early bits by shifting them down to\nthe right, as we are not shifting as fast as before.\n\n> I think one way to say what you mean is\n>\n>   Each path component is hashed similarly to the standard name-hash,\n>   and parent path component hashes are contributed via XOR after a\n>   down-shift of 6 bits per level.\n\nYes.\n"},{"id":"508001","messageId":"xmqqjzcs87u0.fsf@gitster.g","threadId":"62447","inReplyTo":"20241122180144.523048-1-jonathantanmy@google.com","subject":"Re: [PATCH 3/7] pack-objects: add GIT_TEST_FULL_NAME_HASH","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-25T00:39:51Z","receivedAt":"2024-11-25T00:39:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> Junio C Hamano <gitster@pobox.com> writes:\n>> It is sad that it is a (probably unfixable) flaw in the \"promisor\n>> object\" concept that the \"promisor object\"-ness of blobA depends on\n>> the lazy-fetch status of treeA.  This is not merely a test failure,\n>> but it would cause blobA pruned if such a lazy fetch happens in the\n>> wild and then \"git gc\" triggers, no?  \n>\n> Right now, it won't be pruned since we never prune promisor objects\n> (we just concatenate all of them into one file).\n\nSorry, but I am lost.  In the scenario discussed, you have two\ncommits A and B with their trees and blobs.  You initially only have\ncommit A because the partial clone is done with \"tree:0\".  Then you\nfetch commit B (A's child), tree B in non-delta form, and blob B\ncontained within tree B.  Due to the tweak in the name hash\nfunction, we do not know of tree A (we used to learn about it\nbecause tree B was sent as a delta against it with the old name\nhash).  If blob B was sent as a delta against blob A, lazy fetch\nwould later materialize blob A even if you do not still have tree A,\nno?\n\nI thought the story was that we would not know who refers to blobA\nwhen treeA hasn't been lazily fetched, hence we cannot tell if blobA\nis a \"promisor object\" to begin with, no?\n\nThe blob in such a scenario may be reclaimed by GC and we may still\nbe able to refetch from the promisor, so it may not be the end of\nthe world, but feels suboptimal.\n\n"},{"id":"508110","messageId":"20241125194541.809707-1-jonathantanmy@google.com","threadId":"62447","inReplyTo":"xmqqjzcs87u0.fsf@gitster.g","subject":"Re: [PATCH 3/7] pack-objects: add GIT_TEST_FULL_NAME_HASH","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2024-11-25T19:45:41Z","receivedAt":"2024-11-25T19:45:44Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Junio C Hamano <gitster@pobox.com> writes:\n> Jonathan Tan <jonathantanmy@google.com> writes:\n> \n> > Junio C Hamano <gitster@pobox.com> writes:\n> >> It is sad that it is a (probably unfixable) flaw in the \"promisor\n> >> object\" concept that the \"promisor object\"-ness of blobA depends on\n> >> the lazy-fetch status of treeA.  This is not merely a test failure,\n> >> but it would cause blobA pruned if such a lazy fetch happens in the\n> >> wild and then \"git gc\" triggers, no?  \n> >\n> > Right now, it won't be pruned since we never prune promisor objects\n> > (we just concatenate all of them into one file).\n> \n> Sorry, but I am lost.  In the scenario discussed, you have two\n> commits A and B with their trees and blobs.  You initially only have\n> commit A because the partial clone is done with \"tree:0\".  Then you\n> fetch commit B (A's child), tree B in non-delta form, and blob B\n> contained within tree B.  Due to the tweak in the name hash\n> function, we do not know of tree A (we used to learn about it\n> because tree B was sent as a delta against it with the old name\n> hash).  \n\nYes, that's correct.\n\n> If blob B was sent as a delta against blob A, lazy fetch\n> would later materialize blob A even if you do not still have tree A,\n> no?\n\nJust to be clear, this is not happening right now (blob B is sent whole,\nnot as a delta). But let's suppose that blob B was sent as a delta, then\nyes, the lazy fetch would materialize blob A...\n\n> I thought the story was that we would not know who refers to blobA\n> when treeA hasn't been lazily fetched, hence we cannot tell if blobA\n> is a \"promisor object\" to begin with, no?\n\n...ah, in this case, blob A vouches for itself. Whenever we lazy fetch,\nall objects that are fetched go into promisor packs (packfiles with an\nassociated .promisor file), so we know that they are promisor objects.\n"},{"id":"508122","messageId":"xmqqttbu3hq9.fsf@gitster.g","threadId":"62447","inReplyTo":"20241125194541.809707-1-jonathantanmy@google.com","subject":"Re: [PATCH 3/7] pack-objects: add GIT_TEST_FULL_NAME_HASH","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-26T01:29:34Z","receivedAt":"2024-11-26T01:29:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> ...ah, in this case, blob A vouches for itself. Whenever we lazy fetch,\n> all objects that are fetched go into promisor packs (packfiles with an\n> associated .promisor file), so we know that they are promisor objects.\n\nThanks.\n"},{"id":"508161","messageId":"Z0WGLeI2TA8m6Gpu@pks.im","threadId":"62447","inReplyTo":"812257e197cfe30bd0d3c68ea6ec0d062631185f.1730775907.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/7] pack-objects: add --full-name-hash option","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-26T08:26:30Z","receivedAt":"2024-11-26T08:26:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 05, 2024 at 03:05:01AM +0000, Derrick Stolee via GitGitGadget wrote:\n> It is important to point out that the name hash value is stored in the\n> .bitmap file format, so we must disable the --full-name-hash option when\n> bitmaps are being read or written. Later, the bitmap format could be\n> updated to be aware of the name hash version so deltas can be quickly\n> computed across the bitmapped/not-bitmapped boundary.\n\nI was wondering a bit about this: is there any reason why we cannot have\nboth, that is reap the benefits of \"--full-name-hash\" but end up writing\na bitmap with the old name hash so that we can continue to generate\nbitmaps?\n\nForgive me if this question is naive, I'm more at home in the refs\nsubsystem :)\n\n> diff --git a/pack-objects.h b/pack-objects.h\n> index b9898a4e64b..88360aa3e8e 100644\n> --- a/pack-objects.h\n> +++ b/pack-objects.h\n> @@ -207,6 +207,27 @@ static inline uint32_t pack_name_hash(const char *name)\n>  \treturn hash;\n>  }\n>  \n> +static inline uint32_t pack_full_name_hash(const char *name)\n> +{\n> +\tconst uint32_t bigp = 1234572167U;\n> +\tuint32_t c, hash = bigp;\n\nIt would be nice to have a comment here detailing how you came up with\nthat number, and what its requirements are. You briefly mention it in\nthe comment further down, but I think this could be expanded a bit.\n\nPatrick\n"},{"id":"508162","messageId":"Z0WGP34itCSCdYk6@pks.im","threadId":"62447","inReplyTo":"259734e0bcea952c2c09b0fb3a017e139922b975.1730775908.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/7] pack-objects: add GIT_TEST_FULL_NAME_HASH","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-26T08:26:39Z","receivedAt":"2024-11-26T08:26:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 05, 2024 at 03:05:03AM +0000, Derrick Stolee via GitGitGadget wrote:\n> diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\n> index c53e93be2f7..425aa8d8789 100755\n> --- a/t/t5616-partial-clone.sh\n> +++ b/t/t5616-partial-clone.sh\n> @@ -516,7 +516,18 @@ test_expect_success 'fetch lazy-fetches only to resolve deltas' '\n>  \t# Exercise to make sure it works. Git will not fetch anything from the\n>  \t# promisor remote other than for the big tree (because it needs to\n>  \t# resolve the delta).\n> -\tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -C client \\\n> +\t#\n> +\t# TODO: the --full-name-hash option is disabled here, since this test\n> +\t# is fundamentally broken! When GIT_TEST_FULL_NAME_HASH=1, the server\n> +\t# recognizes delta bases in a different way and then sends a _blob_ to\n> +\t# the client with a delta base that the client does not have! This is\n> +\t# because the client is cloned from \"promisor-server\" with tree:0 but\n> +\t# is now fetching from \"server\" withot any filter. This is violating the\n\ns/withot/without/\n\nAlso present in copies of this comment.\n\n> diff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh\n> index 0f0c86f9cb2..03f8c976720 100755\n> --- a/t/t7406-submodule-update.sh\n> +++ b/t/t7406-submodule-update.sh\n> @@ -1094,6 +1094,8 @@ test_expect_success 'submodule update --quiet passes quietness to fetch with a s\n>  \t) &&\n>  \tgit clone super4 super5 &&\n>  \t(cd super5 &&\n> +\t # This test var can mess with the stderr output checked in this test.\n> +\t GIT_TEST_FULL_NAME_HASH=0 \\\n>  \t git submodule update --quiet --init --depth=1 submodule3 >out 2>err &&\n\nNit: This line should now be indented.\n\n>  \t test_must_be_empty out &&\n>  \t test_must_be_empty err\n> diff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\n> index fc2cc9d37be..e3787bacdad 100755\n> --- a/t/t7700-repack.sh\n> +++ b/t/t7700-repack.sh\n> @@ -309,6 +309,9 @@ test_expect_success 'no bitmaps created if .keep files present' '\n>  \tkeep=${pack%.pack}.keep &&\n>  \ttest_when_finished \"rm -f \\\"\\$keep\\\"\" &&\n>  \t>\"$keep\" &&\n> +\n> +\t# Disable --full-name-hash test due to stderr comparison.\n> +\tGIT_TEST_FULL_NAME_HASH=0 \\\n>  \tgit -C bare.git repack -ad 2>stderr &&\n\nSame here.\n\n>  \ttest_must_be_empty stderr &&\n>  \tfind bare.git/objects/pack/ -type f -name \"*.bitmap\" >actual &&\n> @@ -320,6 +323,9 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '\n>  \tblob=$(test-tool genrandom big $((1024*1024)) |\n>  \t       git -C bare.git hash-object -w --stdin) &&\n>  \tgit -C bare.git update-ref refs/tags/big $blob &&\n> +\n> +\t# Disable --full-name-hash test due to stderr comparison.\n> +\tGIT_TEST_FULL_NAME_HASH=0 \\\n>  \tgit -C bare.git repack -ad 2>stderr &&\n\nAnd here.\n\nPatrick\n"},{"id":"508163","messageId":"Z0WGQw6jSw3uhlh4@pks.im","threadId":"62447","inReplyTo":"c14ef6879e451401381ebbdb8f30d33c8f56c25b.1730775908.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/7] p5313: add size comparison test","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-26T08:26:43Z","receivedAt":"2024-11-26T08:26:58Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 05, 2024 at 03:05:05AM +0000, Derrick Stolee via GitGitGadget wrote:\n> These tests demonstrate that it is important to be careful about which\n> cases are best for using the --full-name-hash option.\n\nIs it possible to give general guidelines in our documentation that\nguides the end user for when to use the option and when not to use it?\nAnd if the answer is yes, is it possible for us to figure out at runtime\nwhat the current scenario is via some heuristics and enable the option\nautomatically in a subset of cases?\n\nPatrick\n"},{"id":"508474","messageId":"454b070d5bb0f64e11cab993b126ef5d37a3615b.1733181682.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.v2.git.1733181682.gitgitgadget@gmail.com","subject":"[PATCH v2 1/8] pack-objects: create new name-hash function version","fromName":"Jonathan Tan via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-02T23:21:15Z","receivedAt":"2024-12-02T23:21:27Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"From: Jonathan Tan <jonathantanmy@google.com>\n\nAs we will explore in later changes, the default name-hash function used\nin 'git pack-objects' has a tendency to cause collisions and cause poor\ndelta selection. This change creates an alternative that avoids some\ncollisions while preserving some amount of hash locality.\n\nThe pack_name_hash() method has not been materially changed since it was\nintroduced in ce0bd64 (pack-objects: improve path grouping\nheuristics., 2006-06-05). The intention here is to group objects by path\nname, but also attempt to group similar file types together by making\nthe most-significant digits of the hash be focused on the final\ncharacters.\n\nHere's the crux of the implementation:\n\n\t/*\n\t * This effectively just creates a sortable number from the\n\t * last sixteen non-whitespace characters. Last characters\n\t * count \"most\", so things that end in \".c\" sort together.\n\t */\n\twhile ((c = *name++) != 0) {\n\t\tif (isspace(c))\n\t\t\tcontinue;\n\t\thash = (hash >> 2) + (c << 24);\n\t}\n\nAs the comment mentions, this only cares about the last sixteen\nnon-whitespace characters. This cause some filenames to collide more than\nothers. This collision is somewhat by design in order to promote hash\nlocality for files that have similar types (.c, .h, .json) or could be the\nsame file across a directory rename (a/foo.txt to b/foo.txt). This leads to\ndecent cross-path deltas in cases like shallow clones or packing a\nrepository with very few historical versions of files that share common data\nwith other similarly-named files.\n\nHowever, when the name-hash instead leads to a large number of name-hash\ncollisions for otherwise unrelated files, this can lead to confusing the\ndelta calculation to prefer cross-path deltas over previous versions of the\nsame file.\n\nThe new pack_name_hash_v2() function attempts to fix this issue by\ntaking more of the directory path into account through its hash\nfunction. Its naming implies that we will later wire up details for\nchoosing a name-hash function by version.\n\nThe first change is to be more careful about paths using non-ASCII\ncharacters. With these characters in mind, reverse the bits in the byte\nas the least-significant bits have the highest entropy and we want to\nmaximize their influence. This is done with some bit manipulation that\nswaps the two halves, then the quarters within those halves, and then\nthe bits within those quarters.\n\nThe second change is to perform hash composition operations at every\nlevel of the path. This is done by storing a 'base' hash value that\ncontains the hash of the parent directory. When reaching a directory\nboundary, we XOR the current level's name-hash value with a downshift of\nthe previous level's hash. This perturbation intends to create low-bit\ndistinctions for paths with the same final 16 bytes but distinct parent\ndirectory structures.\n\nThe collision rate and effectiveness of this hash function will be\nexplored in later changes as the function is integrated with 'git\npack-objects' and 'git repack'.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n pack-objects.h | 28 ++++++++++++++++++++++++++++\n 1 file changed, 28 insertions(+)\n\ndiff --git a/pack-objects.h b/pack-objects.h\nindex b9898a4e64b..15be8368d21 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -207,6 +207,34 @@ static inline uint32_t pack_name_hash(const char *name)\n \treturn hash;\n }\n \n+static inline uint32_t pack_name_hash_v2(const char *name)\n+{\n+\tuint32_t hash = 0, base = 0, c;\n+\n+\tif (!name)\n+\t\treturn 0;\n+\n+\twhile ((c = *name++)) {\n+\t\tif (isspace(c))\n+\t\t\tcontinue;\n+\t\tif (c == '/') {\n+\t\t\tbase = (base >> 6) ^ hash;\n+\t\t\thash = 0;\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * 'c' is only a single byte. Reverse it and move\n+\t\t\t * it to the top of the hash, moving the rest to\n+\t\t\t * less-significant bits.\n+\t\t\t */\n+\t\t\tc = (c & 0xF0) >> 4 | (c & 0x0F) << 4;\n+\t\t\tc = (c & 0xCC) >> 2 | (c & 0x33) << 2;\n+\t\t\tc = (c & 0xAA) >> 1 | (c & 0x55) << 1;\n+\t\t\thash = (hash >> 2) + (c << 24);\n+\t\t}\n+\t}\n+\treturn (base >> 6) ^ hash;\n+}\n+\n static inline enum object_type oe_type(const struct object_entry *e)\n {\n \treturn e->type_valid ? e->type_ : OBJ_BAD;\n-- \ngitgitgadget\n\n"},{"id":"508477","messageId":"pull.1823.v2.git.1733181682.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.git.1730775907.gitgitgadget@gmail.com","subject":"[PATCH v2 0/8] pack-objects: Create an alternative name hash algorithm (recreated)","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-02T23:21:14Z","receivedAt":"2024-12-02T23:21:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is a recreation of the topic in [1] that was closed. (I force-pushed my\nbranch and GitHub won't let me reopen the PR for GitGitGadget to create this\nas v3.)\n\n[1]\nhttps://lore.kernel.org/git/pull.1785.v2.git.1726692381.gitgitgadget@gmail.com/\n\nI've been focused recently on understanding and mitigating the growth of a\nfew internal repositories. Some of these are growing much larger than\nexpected for the number of contributors, and there are multiple aspects to\nwhy this growth is so large.\n\nThis is part of the RFC I submitted [2] involving the path-walk API, though\nthis doesn't use the path-walk API directly. In full repack cases, it seems\nthat the --full-name-hash option gets nearly as good compression as the\n--path-walk option introduced in that series. I continue to work on that\nfeature as well, so we can review it after this series is complete.\n\n[2]\nhttps://lore.kernel.org/git/pull.1786.git.1725935335.gitgitgadget@gmail.com/\n\nThe main issue plaguing these repositories is that deltas are not being\ncomputed against objects that appear at the same path. While the size of\nthese files at tip is one aspect of growth that would prevent this issue,\nthe changes to these files are reasonable and should result in good delta\ncompression. However, Git is not discovering the connections across\ndifferent versions of the same file.\n\nOne way to find some improvement in these repositories is to increase the\nwindow size, which was an initial indicator that the delta compression could\nbe improved, but was not a clear indicator. After some digging (and\nprototyping some analysis tools) the main discovery was that the current\nname-hash algorithm only considers the last 16 characters in the path name\nand has some naturally-occurring collisions within that scope.\n\nThis series creates a mechanism to select alternative name hashes using a\nnew --name-hash-version=<n> option. The versions are:\n\n 1. Version 1 is the default name hash that already exists. This option\n    focuses on the final bytes of the path to maximize locality for\n    cross-path deltas.\n\n 2. Version 2 is the new path-component hash function suggested by Jonathan\n    Tan in the previous version (with some modifications). This hash\n    function essentially computes the v1 name hash of each path component\n    and then overlays those hashes with a shift to make the parent\n    directories contribute less to the final hash, but enough to break many\n    collisions that exist in v1.\n\n 3. Version 3 is the hash function that I submitted under the\n    --full-name-hash feature in the previous versions. This uses a\n    pseudorandom hash procedure to minimize collisions but at the expense of\n    losing on locality. This version is implemented in the final patch of\n    the series mostly for comparison purposes, as it is unlikely to be\n    selected as a valuable hash function over v2. The final patch could be\n    omitted from the merged version.\n\nSee the patches themselves for detailed results in the p5313-pack-objects.sh\nperformance test and the p5314-name-hash.sh test that demonstrates how many\ncollisions occur with each hash function.\n\nIn general, the v2 name hash function gets very close to the compression\nresults of v3 in the full repack case, even in the repositories that feature\nmany name hash collisions. These benefits come as well without downsides to\nother kinds of packfiles, including small pushed packs, larger incremental\nfetch packs, and shallow clones.\n\nI should point out that there is still a significant jump in compression\neffectiveness between these name hash version options and the --path-walk\nfeature I suggested in my RFC [2] and has review underway in [3] (along with\nchanges to git pack-objects and git repack in [4]).\n\n[3]\nhttps://lore.kernel.org/git/pull.1818.v2.git.1731181272.gitgitgadget@gmail.com/\n\n[4] https://github.com/gitgitgadget/git/pull/1819\n\nTo compare these options in a set of Javascript repositories that have\ndifferent levels of name hash collisions, see the following table that lists\nthe size of the packfile after git repack -adf\n[--name-hash-version=<n>|--path-walk]:\n\n| Repo     | V1 Size   | V2 Size | V3 Size | Path Walk Size |\n|----------|-----------|---------|---------|----------------|\n| fluentui |     440 M |   161 M |   170 M |          123 M |\n| Repo B   |   6,248 M |   856 M |   840 M |          782 M |\n| Repo C   |  37,278 M | 6,921 M | 6,755 M |        6,156 M |\n| Repo D   | 131,204 M | 7,463 M | 7,124 M |        4,005 M |\n\n\nAs we can see, v2 nearly reaches the effectiveness of v3 (and outperforms it\nonce!) but there is still a significant change between the\n--name-hash-version feature and the --path-walk feature.\n\nThe main reason we are considering this --name-hash-version feature is that\nit has the least amount of stretch required in order for it to be integrated\nwith reachability bitmaps, required for server environments. In fact, the\nchange in this version to use a numerical version makes it more obvious how\nto connect the version number to a value in the .bitmap file format. Tests\nare added to guarantee that the hash functions preserve their behavior over\ntime, since data files depend on that.\n\nThanks, -Stolee\n\n\nUPDATES SINCE V1\n================\n\n * BIG CHANGE: --full-name-hash is replaced with --name-hash-version=<n>.\n\n * --name-hash-version=2 uses Jonathan Tan's hash function (with some\n   adjustments). See the first patch for this implementation, credited to\n   him.\n\n * --name-hash-version=3 uses the hash function I wrote for the previous\n   version's --full-name-hash. This is left as the final patch so it could\n   be easily removed from the series if not considered worth having since it\n   has some pain points that are resolved from v2 without significant issues\n   to overall repo size.\n\n * Commit messaes are updated with these changes, as well as a better\n   attempt to indicate the benefit of cross-path delta pairs, such as\n   renames or similar content based on file extension.\n\n * Performance numbers are regenerated for the same set of repositories.\n   Size data is somewhat nondeterministic due to concurrent threads\n   competing over delta computations.\n\n * The --name-hash-version option is not added to git repack until its own\n   patch.\n\n * The patch that updates git repack's synopsis match its docs is squashed\n   into the patch that adds the option to git repack.\n\n * Documentation is expanded for git pack-objects and reused for git repack.\n\n * GIT_TEST_FULL_NAME_HASH is now GIT_TEST_NAME_HASH_VERSION with similar\n   caveats required for tests. It is removed from the linux-TEST-vars CI\n   job.\n\n * The performance test p5313-pack-objects.sh is now organized via a loop\n   over the different versions. This separates the scenarios, which makes\n   things harder to compare directly, but makes it trivial to add new\n   versions.\n\n * The patch that disabled --full-name-hash when performing a shallow clone\n   is no longer present, as it is not necessary when using\n   --name-hash-version=2. Perhaps it would be valuable for repo using v3, if\n   that is kept in the series.\n\n * We force name hash version 1 when writing or reading bitmaps.\n\n * A small patch is added to cause a BUG() failure if the name hash version\n   global changes between calls to pack_name_hash_fn(). This is solely\n   defensive programming.\n\n * Several typos, style issues, or suggested comments are resolved.\n\nDerrick Stolee (7):\n  pack-objects: add --name-hash-version option\n  repack: add --name-hash-version option\n  pack-objects: add GIT_TEST_NAME_HASH_VERSION\n  p5313: add size comparison test\n  test-tool: add helper for name-hash values\n  pack-objects: prevent name hash version change\n  pack-objects: add third name hash version\n\nJonathan Tan (1):\n  pack-objects: create new name-hash function version\n\n Documentation/git-pack-objects.txt | 41 ++++++++++++++++-\n Documentation/git-repack.txt       | 43 +++++++++++++++++-\n Makefile                           |  1 +\n builtin/pack-objects.c             | 63 ++++++++++++++++++++++++---\n builtin/repack.c                   |  9 +++-\n pack-objects.h                     | 54 +++++++++++++++++++++++\n t/README                           |  4 ++\n t/helper/test-name-hash.c          | 24 ++++++++++\n t/helper/test-tool.c               |  1 +\n t/helper/test-tool.h               |  1 +\n t/perf/p5313-pack-objects.sh       | 70 ++++++++++++++++++++++++++++++\n t/perf/p5314-name-hash.sh          | 31 +++++++++++++\n t/t0450/txt-help-mismatches        |  1 -\n t/t5300-pack-object.sh             | 34 +++++++++++++++\n t/t5310-pack-bitmaps.sh            | 35 ++++++++++++++-\n t/t5333-pseudo-merge-bitmaps.sh    |  4 ++\n t/t5510-fetch.sh                   |  7 ++-\n t/t5616-partial-clone.sh           | 26 ++++++++++-\n t/t6020-bundle-misc.sh             |  6 ++-\n t/t7406-submodule-update.sh        |  4 +-\n t/t7700-repack.sh                  | 16 ++++++-\n t/test-lib-functions.sh            | 26 +++++++++++\n 22 files changed, 484 insertions(+), 17 deletions(-)\n create mode 100644 t/helper/test-name-hash.c\n create mode 100755 t/perf/p5313-pack-objects.sh\n create mode 100755 t/perf/p5314-name-hash.sh\n\n\nbase-commit: 8f8d6eee531b3fa1a8ef14f169b0cb5035f7a772\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1823%2Fderrickstolee%2Ffull-name-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1823/derrickstolee/full-name-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1823\n\nRange-diff vs v1:\n\n -:  ----------- > 1:  454b070d5bb pack-objects: create new name-hash function version\n 1:  812257e197c ! 2:  fb52ca509da pack-objects: add --full-name-hash option\n     @@ Metadata\n      Author: Derrick Stolee <dstolee@microsoft.com>\n      \n       ## Commit message ##\n     -    pack-objects: add --full-name-hash option\n     +    pack-objects: add --name-hash-version option\n      \n     -    The pack_name_hash() method has not been materially changed since it was\n     -    introduced in ce0bd64299a (pack-objects: improve path grouping\n     -    heuristics., 2006-06-05). The intention here is to group objects by path\n     -    name, but also attempt to group similar file types together by making\n     -    the most-significant digits of the hash be focused on the final\n     -    characters.\n     +    The previous change introduced a new pack_name_hash_v2() function that\n     +    intends to satisfy much of the hash locality features of the existing\n     +    pack_name_hash() function while also distinguishing paths with similar\n     +    final components of their paths.\n      \n     -    Here's the crux of the implementation:\n     +    This change adds a new --name-hash-version option for 'git pack-objects'\n     +    to allow users to select their preferred function version. This use of\n     +    an integer version allows for future expansion and a direct way to later\n     +    store a name hash version in the .bitmap format.\n      \n     -            /*\n     -             * This effectively just creates a sortable number from the\n     -             * last sixteen non-whitespace characters. Last characters\n     -             * count \"most\", so things that end in \".c\" sort together.\n     -             */\n     -            while ((c = *name++) != 0) {\n     -                    if (isspace(c))\n     -                            continue;\n     -                    hash = (hash >> 2) + (c << 24);\n     -            }\n     +    For now, let's consider how effective this mechanism is when repacking a\n     +    repository with different name hash versions. Specifically, we will\n     +    execute 'git pack-objects' the same way a 'git repack -adf' process\n     +    would, except we include --name-hash-version=<n> for testing.\n     +\n     +    On the Git repository, we do not expect much difference. All path names\n     +    are short. This is backed by our results:\n      \n     -    As the comment mentions, this only cares about the last sixteen\n     -    non-whitespace characters. This cause some filenames to collide more\n     -    than others. Here are some examples that I've seen while investigating\n     -    repositories that are growing more than they should be:\n     +    | Stage                 | Pack Size | Repack Time |\n     +    |-----------------------|-----------|-------------|\n     +    | After clone           | 260 MB    | N/A         |\n     +    | --name-hash-version=1 | 127 MB    | 129s        |\n     +    | --name-hash-version=2 | 127 MB    | 112s        |\n     +\n     +    This example demonstrates how there is some natural overhead coming from\n     +    the cloned copy because the server is hosting many forks and has not\n     +    optimized for exactly this set of reachable objects. But the full repack\n     +    has similar characteristics for both versions.\n     +\n     +    Let's consider some repositories that are hitting too many collisions\n     +    with version 1. First, let's explore the kinds of paths that are\n     +    commonly causing these collisions:\n      \n           * \"/CHANGELOG.json\" is 15 characters, and is created by the beachball\n             [1] tool. Only the final character of the parent directory can\n     -       differntiate different versions of this file, but also only the two\n     +       differentiate different versions of this file, but also only the two\n             most-significant digits. If that character is a letter, then this is\n             always a collision. Similar issues occur with the similar\n             \"/CHANGELOG.md\" path, though there is more opportunity for\n     -       differences in the parent directory.\n     +       differences In the parent directory.\n      \n     -     * Localization files frequently have common filenames but differentiate\n     -       via parent directories. In C#, the name \"/strings.resx.lcl\" is used\n     -       for these localization files and they will all collide in name-hash.\n     +     * Localization files frequently have common filenames but\n     +       differentiates via parent directories. In C#, the name\n     +       \"/strings.resx.lcl\" is used for these localization files and they\n     +       will all collide in name-hash.\n      \n          [1] https://github.com/microsoft/beachball\n      \n     @@ Commit message\n          common name across multiple directories and is causing Git to repack\n          poorly due to name-hash collisions.\n      \n     -    It is clear that the existing name-hash algorithm is optimized for\n     -    repositories with short path names, but also is optimized for packing a\n     -    single snapshot of a repository, not a repository with many versions of\n     -    the same file. In my testing, this has proven out where the name-hash\n     -    algorithm does a good job of finding peer files as delta bases when\n     -    unable to use a historical version of that exact file.\n     -\n     -    However, for repositories that have many versions of most files and\n     -    directories, it is more important that the objects that appear at the\n     -    same path are grouped together.\n     -\n     -    Create a new pack_full_name_hash() method and a new --full-name-hash\n     -    option for 'git pack-objects' to call that method instead. Add a simple\n     -    pass-through for 'git repack --full-name-hash' for additional testing in\n     -    the context of a full repack, where I expect this will be most\n     -    effective.\n     -\n     -    The hash algorithm is as simple as possible to be reasonably effective:\n     -    for each character of the path string, add a multiple of that character\n     -    and a large prime number (chosen arbitrarily, but intended to be large\n     -    relative to the size of a uint32_t). Then, shift the current hash value\n     -    to the right by 5, with overlap. The addition and shift parameters are\n     -    standard mechanisms for creating hard-to-predict behaviors in the bits\n     -    of the resulting hash.\n     -\n     -    This is not meant to be cryptographic at all, but uniformly distributed\n     -    across the possible hash values. This creates a hash that appears\n     -    pseudorandom. There is no ability to consider similar file types as\n     -    being close to each other.\n     -\n     -    In a later change, a test-tool will be added so the effectiveness of\n     -    this hash can be demonstrated directly.\n     -\n     -    For now, let's consider how effective this mechanism is when repacking a\n     -    repository with and without the --full-name-hash option. Specifically,\n     -    let's use 'git repack -adf [--full-name-hash]' as our test.\n     -\n     -    On the Git repository, we do not expect much difference. All path names\n     -    are short. This is backed by our results:\n     -\n     -    | Stage                 | Pack Size | Repack Time |\n     -    |-----------------------|-----------|-------------|\n     -    | After clone           | 260 MB    | N/A         |\n     -    | Standard Repack       | 127MB     | 106s        |\n     -    | With --full-name-hash | 126 MB    | 99s         |\n     -\n     -    This example demonstrates how there is some natural overhead coming from\n     -    the cloned copy because the server is hosting many forks and has not\n     -    optimized for exactly this set of reachable objects. But the full repack\n     -    has similar characteristics with and without --full-name-hash.\n     -\n     -    However, we can test this in a repository that uses one of the\n     -    problematic naming conventions above. The fluentui [2] repo uses\n     -    beachball to generate CHANGELOG.json and CHANGELOG.md files, and these\n     -    files have very poor delta characteristics when comparing against\n     -    versions across parent directories.\n     +    One open-source example is the fluentui [2] repo, which  uses beachball\n     +    to generate CHANGELOG.json and CHANGELOG.md files, and these files have\n     +    very poor delta characteristics when comparing against versions across\n     +    parent directories.\n      \n          | Stage                 | Pack Size | Repack Time |\n          |-----------------------|-----------|-------------|\n          | After clone           | 694 MB    | N/A         |\n     -    | Standard Repack       | 438 MB    | 728s        |\n     -    | With --full-name-hash | 168 MB    | 142s        |\n     +    | --name-hash-version=1 | 438 MB    | 728s        |\n     +    | --name-hash-version=2 | 168 MB    | 142s        |\n      \n          [2] https://github.com/microsoft/fluentui\n      \n     @@ Commit message\n          reproduction. The results are so significant that I find it important to\n          share here:\n      \n     -    | Repo     | Standard Repack | With --full-name-hash |\n     -    |----------|-----------------|-----------------------|\n     -    | fluentui |         438 MB  |               168 MB  |\n     -    | Repo B   |       6,255 MB  |               829 MB  |\n     -    | Repo C   |      37,737 MB  |             7,125 MB  |\n     -    | Repo D   |     130,049 MB  |             6,190 MB  |\n     +    | Repo     | --name-hash-version=1 | --name-hash-version=2 |\n     +    |----------|-----------------------|-----------------------|\n     +    | fluentui |               440 MB  |               161 MB  |\n     +    | Repo B   |             6,248 MB  |               856 MB  |\n     +    | Repo C   |            37,278 MB  |             6,755 MB  |\n     +    | Repo D   |           131,204 MB  |             7,463 MB  |\n      \n     -    Future changes could include making --full-name-hash implied by a config\n     +    Future changes could include making --name-hash-version implied by a config\n          value or even implied by default during a full repack.\n      \n          It is important to point out that the name hash value is stored in the\n     -    .bitmap file format, so we must disable the --full-name-hash option when\n     -    bitmaps are being read or written. Later, the bitmap format could be\n     -    updated to be aware of the name hash version so deltas can be quickly\n     -    computed across the bitmapped/not-bitmapped boundary.\n     +    .bitmap file format, so we must force --name-hash-version=1 when bitmaps\n     +    are being read or written. Later, the bitmap format could be updated to\n     +    be aware of the name hash version so deltas can be quickly computed\n     +    across the bitmapped/not-bitmapped boundary.\n      \n          Signed-off-by: Derrick Stolee <stolee@gmail.com>\n      \n     @@ Documentation/git-pack-objects.txt: SYNOPSIS\n       \t[--stdout [--filter=<filter-spec>] | <base-name>]\n      -\t[--shallow] [--keep-true-parents] [--[no-]sparse] < <object-list>\n      +\t[--shallow] [--keep-true-parents] [--[no-]sparse]\n     -+\t[--full-name-hash] < <object-list>\n     ++\t[--name-hash-version=<n>] < <object-list>\n       \n       \n       DESCRIPTION\n     +@@ Documentation/git-pack-objects.txt: raise an error.\n     + \tRestrict delta matches based on \"islands\". See DELTA ISLANDS\n     + \tbelow.\n     + \n     ++--name-hash-version=<n>::\n     ++\tWhile performing delta compression, Git groups objects that may be\n     ++\tsimilar based on heuristics using the path to that object. While\n     ++\tgrouping objects by an exact path match is good for paths with\n     ++\tmany versions, there are benefits for finding delta pairs across\n     ++\tdifferent full paths. Git collects objects by type and then by a\n     ++\t\"name hash\" of the path and then by size, hoping to group objects\n     ++\tthat will compress well together.\n     +++\n     ++The default name hash version is `1`, which prioritizes hash locality by\n     ++considering the final bytes of the path as providing the maximum magnitude\n     ++to the hash function. This version excels at distinguishing short paths\n     ++and finding renames across directories. However, the hash function depends\n     ++primarily on the final 16 bytes of the path. If there are many paths in\n     ++the repo that have the same final 16 bytes and differ only by parent\n     ++directory, then this name-hash may lead to too many collisions and cause\n     ++poor results. At the moment, this version is required when writing\n     ++reachability bitmap files with `--write-bitmap-index`.\n     +++\n     ++The name hash version `2` has similar locality features as version `1`,\n     ++except it considers each path component separately and overlays the hashes\n     ++with a shift. This still prioritizes the final bytes of the path, but also\n     ++\"salts\" the lower bits of the hash using the parent directory names. This\n     ++method allows for some of the locality benefits of version `1` while\n     ++breaking most of the collisions from a similarly-named file appearing in\n     ++many different directories. At the moment, this version is not allowed\n     ++when writing reachability bitmap files with `--write-bitmap-index` and it\n     ++will be automatically changed to version `1`.\n     ++\n     + \n     + DELTA ISLANDS\n     + -------------\n      \n       ## builtin/pack-objects.c ##\n      @@ builtin/pack-objects.c: struct configured_exclusion {\n       static struct oidmap configured_exclusions;\n       \n       static struct oidset excluded_by_config;\n     -+static int use_full_name_hash;\n     ++static int name_hash_version = 1;\n     ++\n     ++static void validate_name_hash_version(void)\n     ++{\n     ++\tif (name_hash_version < 1 || name_hash_version > 2)\n     ++\t\tdie(_(\"invalid --name-hash-version option: %d\"), name_hash_version);\n     ++}\n      +\n      +static inline uint32_t pack_name_hash_fn(const char *name)\n      +{\n     -+\tif (use_full_name_hash)\n     -+\t\treturn pack_full_name_hash(name);\n     -+\treturn pack_name_hash(name);\n     ++\tswitch (name_hash_version)\n     ++\t{\n     ++\tcase 1:\n     ++\t\treturn pack_name_hash(name);\n     ++\n     ++\tcase 2:\n     ++\t\treturn pack_name_hash_v2(name);\n     ++\n     ++\tdefault:\n     ++\t\tBUG(\"invalid name-hash version: %d\", name_hash_version);\n     ++\t}\n      +}\n       \n       /*\n     @@ builtin/pack-objects.c: static void add_cruft_object_entry(const struct object_i\n       \t\t\t\t\t    0, name && no_try_delta(name),\n       \t\t\t\t\t    pack, offset);\n       \t}\n     +@@ builtin/pack-objects.c: static int get_object_list_from_bitmap(struct rev_info *revs)\n     + \tif (!(bitmap_git = prepare_bitmap_walk(revs, 0)))\n     + \t\treturn -1;\n     + \n     ++\t/*\n     ++\t * For now, force the name-hash version to be 1 since that\n     ++\t * is the version implied by the bitmap format. Later, the\n     ++\t * format can include this version explicitly in its format,\n     ++\t * allowing readers to know the version that was used during\n     ++\t * the bitmap write.\n     ++\t */\n     ++\tname_hash_version = 1;\n     ++\n     + \tif (pack_options_allow_reuse())\n     + \t\treuse_partial_packfile_from_bitmap(bitmap_git,\n     + \t\t\t\t\t\t   &reuse_packfiles,\n      @@ builtin/pack-objects.c: int cmd_pack_objects(int argc,\n       \t\tOPT_STRING_LIST(0, \"uri-protocol\", &uri_protocols,\n       \t\t\t\tN_(\"protocol\"),\n       \t\t\t\tN_(\"exclude any configured uploadpack.blobpackfileuri with this protocol\")),\n     -+\t\tOPT_BOOL(0, \"full-name-hash\", &use_full_name_hash,\n     -+\t\t\t N_(\"optimize delta compression across identical path names over time\")),\n     ++\t\tOPT_INTEGER(0, \"name-hash-version\", &name_hash_version,\n     ++\t\t\t N_(\"use the specified name-hash function to group similar objects\")),\n       \t\tOPT_END(),\n       \t};\n       \n     @@ builtin/pack-objects.c: int cmd_pack_objects(int argc,\n       \tif (pack_to_stdout || !rev_list_all)\n       \t\twrite_bitmap_index = 0;\n       \n     -+\tif (write_bitmap_index && use_full_name_hash) {\n     -+\t\twarning(_(\"currently, the --full-name-hash option is incompatible with --write-bitmap-index\"));\n     -+\t\tuse_full_name_hash = 0;\n     ++\tvalidate_name_hash_version();\n     ++\tif (write_bitmap_index && name_hash_version != 1) {\n     ++\t\twarning(_(\"currently, --write-bitmap-index requires --name-hash-version=1\"));\n     ++\t\tname_hash_version = 1;\n      +\t}\n      +\n       \tif (use_delta_islands)\n     @@ builtin/repack.c: struct pack_objects_args {\n       \tstruct list_objects_filter_options filter_options;\n       };\n       \n     -@@ builtin/repack.c: static void prepare_pack_objects(struct child_process *cmd,\n     - \t\tstrvec_pushf(&cmd->args, \"--no-reuse-delta\");\n     - \tif (args->no_reuse_object)\n     - \t\tstrvec_pushf(&cmd->args, \"--no-reuse-object\");\n     -+\tif (args->full_name_hash)\n     -+\t\tstrvec_pushf(&cmd->args, \"--full-name-hash\");\n     - \tif (args->local)\n     - \t\tstrvec_push(&cmd->args,  \"--local\");\n     - \tif (args->quiet)\n     -@@ builtin/repack.c: int cmd_repack(int argc,\n     - \t\t\t\tN_(\"pass --no-reuse-delta to git-pack-objects\")),\n     - \t\tOPT_BOOL('F', NULL, &po_args.no_reuse_object,\n     - \t\t\t\tN_(\"pass --no-reuse-object to git-pack-objects\")),\n     -+\t\tOPT_BOOL(0, \"full-name-hash\", &po_args.full_name_hash,\n     -+\t\t\t\tN_(\"pass --full-name-hash to git-pack-objects\")),\n     - \t\tOPT_NEGBIT('n', NULL, &run_update_server_info,\n     - \t\t\t\tN_(\"do not run git-update-server-info\"), 1),\n     - \t\tOPT__QUIET(&po_args.quiet, N_(\"be quiet\")),\n     -\n     - ## pack-objects.h ##\n     -@@ pack-objects.h: static inline uint32_t pack_name_hash(const char *name)\n     - \treturn hash;\n     - }\n     - \n     -+static inline uint32_t pack_full_name_hash(const char *name)\n     -+{\n     -+\tconst uint32_t bigp = 1234572167U;\n     -+\tuint32_t c, hash = bigp;\n     -+\n     -+\tif (!name)\n     -+\t\treturn 0;\n     -+\n     -+\t/*\n     -+\t * Do the simplest thing that will resemble pseudo-randomness: add\n     -+\t * random multiples of a large prime number with a binary shift.\n     -+\t * The goal is not to be cryptographic, but to be generally\n     -+\t * uniformly distributed.\n     -+\t */\n     -+\twhile ((c = *name++) != 0) {\n     -+\t\thash += c * bigp;\n     -+\t\thash = (hash >> 5) | (hash << 27);\n     -+\t}\n     -+\treturn hash;\n     -+}\n     -+\n     - static inline enum object_type oe_type(const struct object_entry *e)\n     - {\n     - \treturn e->type_valid ? e->type_ : OBJ_BAD;\n      \n       ## t/t5300-pack-object.sh ##\n      @@ t/t5300-pack-object.sh: do\n       \t'\n       done\n       \n     ++test_expect_success 'valid and invalid --name-hash-versions' '\n     ++\t# Valid values are hard to verify other than \"do not fail\".\n     ++\t# Performance tests will be more valuable to validate these versions.\n     ++\tfor value in 1 2\n     ++\tdo\n     ++\t\tgit pack-objects base --all --name-hash-version=$value || return 1\n     ++\tdone &&\n     ++\n     ++\t# Invalid values have clear post-conditions.\n     ++\tfor value in -1 0 3\n     ++\tdo\n     ++\t\ttest_must_fail git pack-objects base --all --name-hash-version=$value 2>err &&\n     ++\t\ttest_grep \"invalid --name-hash-version option\" err || return 1\n     ++\tdone\n     ++'\n     ++\n      +# The following test is not necessarily a permanent choice, but since we do not\n      +# have a \"name hash version\" bit in the .bitmap file format, we cannot write the\n     -+# full-name hash values into the .bitmap file without risking breakage later.\n     ++# hash values into the .bitmap file without risking breakage later.\n      +#\n      +# TODO: Make these compatible in the future and replace this test with the\n      +# expected behavior when both are specified.\n     -+test_expect_success '--full-name-hash and --write-bitmap-index are incompatible' '\n     -+\tgit pack-objects base --all --full-name-hash --write-bitmap-index 2>err &&\n     -+\ttest_grep incompatible err &&\n     ++test_expect_success '--name-hash-version=2 and --write-bitmap-index are incompatible' '\n     ++\tgit pack-objects base --all --name-hash-version=2 --write-bitmap-index 2>err &&\n     ++\ttest_grep \"currently, --write-bitmap-index requires --name-hash-version=1\" err &&\n      +\n      +\t# --stdout option silently removes --write-bitmap-index\n     -+\tgit pack-objects --stdout --all --full-name-hash --write-bitmap-index >out 2>err &&\n     -+\t! test_grep incompatible err\n     ++\tgit pack-objects --stdout --all --name-hash-version=2 --write-bitmap-index >out 2>err &&\n     ++\t! test_grep \"currently, --write-bitmap-index requires --name-hash-version=1\" err\n      +'\n      +\n       test_done\n 2:  93395c93347 ! 3:  1947d1bf448 repack: add --full-name-hash option\n     @@ Metadata\n      Author: Derrick Stolee <dstolee@microsoft.com>\n      \n       ## Commit message ##\n     -    repack: add --full-name-hash option\n     +    repack: add --name-hash-version option\n      \n     -    The new '--full-name-hash' option for 'git repack' is a simple\n     +    The new '--name-hash-version' option for 'git repack' is a simple\n          pass-through to the underlying 'git pack-objects' subcommand. However,\n          this subcommand may have other options and a temporary filename as part\n          of the subcommand execution that may not be predictable or could change\n     @@ Commit message\n          The existing test_subcommand method requires an exact list of arguments\n          for the subcommand. This is too rigid for our needs here, so create a\n          new method, test_subcommand_flex. Use it to check that the\n     -    --full-name-hash option is passing through.\n     +    --name-hash-version option is passing through.\n      \n          Signed-off-by: Derrick Stolee <stolee@gmail.com>\n      \n     + ## Documentation/git-repack.txt ##\n     +@@ Documentation/git-repack.txt: git-repack - Pack unpacked objects in a repository\n     + SYNOPSIS\n     + --------\n     + [verse]\n     +-'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m] [--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>] [--write-midx]\n     ++'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\n     ++\t[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\n     ++\t[--write-midx] [--name-hash-version=<n>]\n     + \n     + DESCRIPTION\n     + -----------\n     +@@ Documentation/git-repack.txt: linkgit:git-multi-pack-index[1]).\n     + \tWrite a multi-pack index (see linkgit:git-multi-pack-index[1])\n     + \tcontaining the non-redundant packs.\n     + \n     ++--name-hash-version=<n>::\n     ++\tWhile performing delta compression, Git groups objects that may be\n     ++\tsimilar based on heuristics using the path to that object. While\n     ++\tgrouping objects by an exact path match is good for paths with\n     ++\tmany versions, there are benefits for finding delta pairs across\n     ++\tdifferent full paths. Git collects objects by type and then by a\n     ++\t\"name hash\" of the path and then by size, hoping to group objects\n     ++\tthat will compress well together.\n     +++\n     ++The default name hash version is `1`, which prioritizes hash locality by\n     ++considering the final bytes of the path as providing the maximum magnitude\n     ++to the hash function. This version excels at distinguishing short paths\n     ++and finding renames across directories. However, the hash function depends\n     ++primarily on the final 16 bytes of the path. If there are many paths in\n     ++the repo that have the same final 16 bytes and differ only by parent\n     ++directory, then this name-hash may lead to too many collisions and cause\n     ++poor results. At the moment, this version is required when writing\n     ++reachability bitmap files with `--write-bitmap-index`.\n     +++\n     ++The name hash version `2` has similar locality features as version `1`,\n     ++except it considers each path component separately and overlays the hashes\n     ++with a shift. This still prioritizes the final bytes of the path, but also\n     ++\"salts\" the lower bits of the hash using the parent directory names. This\n     ++method allows for some of the locality benefits of version `1` while\n     ++breaking most of the collisions from a similarly-named file appearing in\n     ++many different directories. At the moment, this version is not allowed\n     ++when writing reachability bitmap files with `--write-bitmap-index` and it\n     ++will be automatically changed to version `1`.\n     ++\n     ++\n     + CONFIGURATION\n     + -------------\n     + \n     +\n     + ## builtin/repack.c ##\n     +@@ builtin/repack.c: static int run_update_server_info = 1;\n     + static char *packdir, *packtmp_name, *packtmp;\n     + \n     + static const char *const git_repack_usage[] = {\n     +-\tN_(\"git repack [<options>]\"),\n     ++\tN_(\"git repack [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\\n\"\n     ++\t   \"[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\\n\"\n     ++\t   \"[--write-midx] [--name-hash-version=<n>]\"),\n     + \tNULL\n     + };\n     + \n     +@@ builtin/repack.c: struct pack_objects_args {\n     + \tint no_reuse_object;\n     + \tint quiet;\n     + \tint local;\n     +-\tint full_name_hash;\n     ++\tint name_hash_version;\n     + \tstruct list_objects_filter_options filter_options;\n     + };\n     + \n     +@@ builtin/repack.c: static void prepare_pack_objects(struct child_process *cmd,\n     + \t\tstrvec_pushf(&cmd->args, \"--no-reuse-delta\");\n     + \tif (args->no_reuse_object)\n     + \t\tstrvec_pushf(&cmd->args, \"--no-reuse-object\");\n     ++\tif (args->name_hash_version)\n     ++\t\tstrvec_pushf(&cmd->args, \"--name-hash-version=%d\", args->name_hash_version);\n     + \tif (args->local)\n     + \t\tstrvec_push(&cmd->args,  \"--local\");\n     + \tif (args->quiet)\n     +@@ builtin/repack.c: int cmd_repack(int argc,\n     + \t\t\t\tN_(\"pass --no-reuse-delta to git-pack-objects\")),\n     + \t\tOPT_BOOL('F', NULL, &po_args.no_reuse_object,\n     + \t\t\t\tN_(\"pass --no-reuse-object to git-pack-objects\")),\n     ++\t\tOPT_INTEGER(0, \"name-hash-version\", &po_args.name_hash_version,\n     ++\t\t\t\tN_(\"specify the name hash version to use for grouping similar objects by path\")),\n     + \t\tOPT_NEGBIT('n', NULL, &run_update_server_info,\n     + \t\t\t\tN_(\"do not run git-update-server-info\"), 1),\n     + \t\tOPT__QUIET(&po_args.quiet, N_(\"be quiet\")),\n     +\n     + ## t/t0450/txt-help-mismatches ##\n     +@@ t/t0450/txt-help-mismatches: rebase\n     + remote\n     + remote-ext\n     + remote-fd\n     +-repack\n     + reset\n     + restore\n     + rev-parse\n     +\n       ## t/t7700-repack.sh ##\n      @@ t/t7700-repack.sh: test_expect_success 'repack -ad cleans up old .tmp-* packs' '\n       \ttest_must_be_empty tmpfiles\n       '\n       \n     -+test_expect_success '--full-name-hash option passes through to pack-objects' '\n     -+\tGIT_TRACE2_EVENT=\"$(pwd)/full-trace.txt\" \\\n     -+\t\tgit repack -a --full-name-hash &&\n     -+\ttest_subcommand_flex git pack-objects --full-name-hash <full-trace.txt\n     ++test_expect_success '--name-hash-version option passes through to pack-objects' '\n     ++\tGIT_TRACE2_EVENT=\"$(pwd)/hash-trace.txt\" \\\n     ++\t\tgit repack -a --name-hash-version=2 &&\n     ++\ttest_subcommand_flex git pack-objects --name-hash-version=2 <hash-trace.txt\n      +'\n     -+\n      +\n       test_expect_success 'setup for update-server-info' '\n       \tgit init update-server-info &&\n 3:  259734e0bce ! 4:  6a95708bf97 pack-objects: add GIT_TEST_FULL_NAME_HASH\n     @@ Metadata\n      Author: Derrick Stolee <dstolee@microsoft.com>\n      \n       ## Commit message ##\n     -    pack-objects: add GIT_TEST_FULL_NAME_HASH\n     +    pack-objects: add GIT_TEST_NAME_HASH_VERSION\n      \n     -    Add a new environment variable to opt-in to the --full-name-hash option\n     -    in 'git pack-objects'. This allows for extra testing of the feature\n     -    without repeating all of the test scenarios.\n     +    Add a new environment variable to opt-in to differen values of the\n     +    --name-hash-version=<n> option in 'git pack-objects'. This allows for\n     +    extra testing of the feature without repeating all of the test\n     +    scenarios. Unlike many GIT_TEST_* variables, we are choosing to not add\n     +    this to the linux-TEST-vars CI build as that test run is already\n     +    overloaded. The behavior exposed by this test variable is of low risk\n     +    and should be sufficient to allow manual testing when an issue arises.\n      \n          But this option isn't free. There are a few tests that change behavior\n          with the variable enabled.\n     @@ Commit message\n          variable.\n      \n          Third, there are some tests that compare the exact output of a 'git\n     -    pack-objects' process when using bitmaps. The warning that disables the\n     -    --full-name-hash option causes these tests to fail. Disable the\n     -    environment variable to get around this issue.\n     +    pack-objects' process when using bitmaps. The warning that ignores the\n     +    --name-hash-version=2 and forces version 1 causes these tests to fail.\n     +    Disable the environment variable to get around this issue.\n      \n          Signed-off-by: Derrick Stolee <stolee@gmail.com>\n      \n     @@ builtin/pack-objects.c: struct configured_exclusion {\n       static struct oidmap configured_exclusions;\n       \n       static struct oidset excluded_by_config;\n     --static int use_full_name_hash;\n     -+static int use_full_name_hash = -1;\n     +-static int name_hash_version = 1;\n     ++static int name_hash_version = -1;\n       \n     - static inline uint32_t pack_name_hash_fn(const char *name)\n     + static void validate_name_hash_version(void)\n       {\n      @@ builtin/pack-objects.c: int cmd_pack_objects(int argc,\n       \tif (pack_to_stdout || !rev_list_all)\n       \t\twrite_bitmap_index = 0;\n       \n     --\tif (write_bitmap_index && use_full_name_hash) {\n     -+\tif (use_full_name_hash < 0)\n     -+\t\tuse_full_name_hash = git_env_bool(\"GIT_TEST_FULL_NAME_HASH\", 0);\n     ++\tif (name_hash_version < 0)\n     ++\t\tname_hash_version = (int)git_env_ulong(\"GIT_TEST_NAME_HASH_VERSION\", 1);\n      +\n     -+\tif (write_bitmap_index && use_full_name_hash > 0) {\n     - \t\twarning(_(\"currently, the --full-name-hash option is incompatible with --write-bitmap-index\"));\n     - \t\tuse_full_name_hash = 0;\n     - \t}\n     -\n     - ## ci/run-build-and-tests.sh ##\n     -@@ ci/run-build-and-tests.sh: linux-TEST-vars)\n     - \texport GIT_TEST_NO_WRITE_REV_INDEX=1\n     - \texport GIT_TEST_CHECKOUT_WORKERS=2\n     - \texport GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL=1\n     -+\texport GIT_TEST_FULL_NAME_HASH=1\n     - \t;;\n     - linux-clang)\n     - \texport GIT_TEST_DEFAULT_HASH=sha1\n     + \tvalidate_name_hash_version();\n     + \tif (write_bitmap_index && name_hash_version != 1) {\n     + \t\twarning(_(\"currently, --write-bitmap-index requires --name-hash-version=1\"));\n      \n       ## t/README ##\n      @@ t/README: a test and then fails then the whole test run will abort. This can help to make\n       sure the expected tests are executed and not silently skipped when their\n       dependency breaks or is simply not present in a new environment.\n       \n     -+GIT_TEST_FULL_NAME_HASH=<boolean>, when true, sets the default name-hash\n     -+function in 'git pack-objects' to be the one used by the --full-name-hash\n     -+option.\n     ++GIT_TEST_NAME_HASH_VERSION=<int>, when set, causes 'git pack-objects' to\n     ++assume '--name-hash-version=<n>'.\n     ++\n      +\n       Naming Tests\n       ------------\n       \n      \n     + ## t/t5300-pack-object.sh ##\n     +@@ t/t5300-pack-object.sh: do\n     + done\n     + \n     + test_expect_success 'valid and invalid --name-hash-versions' '\n     ++\tsane_unset GIT_TEST_NAME_HASH_VERSION &&\n     ++\n     + \t# Valid values are hard to verify other than \"do not fail\".\n     + \t# Performance tests will be more valuable to validate these versions.\n     +-\tfor value in 1 2\n     ++\t# Negative values are converted to version 1.\n     ++\tfor value in -1 1 2\n     + \tdo\n     + \t\tgit pack-objects base --all --name-hash-version=$value || return 1\n     + \tdone &&\n     + \n     + \t# Invalid values have clear post-conditions.\n     +-\tfor value in -1 0 3\n     ++\tfor value in 0 3\n     + \tdo\n     + \t\ttest_must_fail git pack-objects base --all --name-hash-version=$value 2>err &&\n     + \t\ttest_grep \"invalid --name-hash-version option\" err || return 1\n     +\n       ## t/t5310-pack-bitmaps.sh ##\n      @@ t/t5310-pack-bitmaps.sh: test_bitmap_cases () {\n       \t\t\tcat >expect <<-\\EOF &&\n       \t\t\terror: missing value for '\\''pack.preferbitmaptips'\\''\n       \t\t\tEOF\n     +-\t\t\tgit repack -adb 2>actual &&\n      +\n     -+\t\t\t# Disable --full-name-hash test due to stderr comparison.\n     -+\t\t\tGIT_TEST_FULL_NAME_HASH=0 \\\n     - \t\t\tgit repack -adb 2>actual &&\n     ++\t\t\t# Disable name hash version adjustment due to stderr comparison.\n     ++\t\t\tGIT_TEST_NAME_HASH_VERSION=1 \\\n     ++\t\t\t\tgit repack -adb 2>actual &&\n       \t\t\ttest_cmp expect actual\n       \t\t)\n     + \t'\n      \n       ## t/t5333-pseudo-merge-bitmaps.sh ##\n      @@ t/t5333-pseudo-merge-bitmaps.sh: test_expect_success 'bitmapPseudoMerge.stableThreshold creates stable groups' '\n     @@ t/t5333-pseudo-merge-bitmaps.sh: test_expect_success 'bitmapPseudoMerge.stableTh\n       \n       test_expect_success 'out of order thresholds are rejected' '\n      +\t# Disable this option to avoid stderr message\n     -+\tGIT_TEST_FULL_NAME_HASH=0 &&\n     -+\texport GIT_TEST_FULL_NAME_HASH &&\n     ++\tGIT_TEST_NAME_HASH_VERSION=1 &&\n     ++\texport GIT_TEST_NAME_HASH_VERSION &&\n      +\n       \ttest_must_fail git \\\n       \t\t-c bitmapPseudoMerge.test.pattern=\"refs/*\" \\\n     @@ t/t5510-fetch.sh: test_expect_success 'all boundary commits are excluded' '\n       \tad=$(git log --no-walk --format=%ad HEAD) &&\n      -\tgit bundle create twoside-boundary.bdl main --since=\"$ad\" &&\n      +\n     -+\t# If the --full-name-hash function is used here, then no delta\n     ++\t# If the a different name hash function is used here, then no delta\n      +\t# pair is found and the bundle does not expand to three objects\n      +\t# when fixing the thin object.\n     -+\tGIT_TEST_FULL_NAME_HASH=0 \\\n     ++\tGIT_TEST_NAME_HASH_VERSION=1 \\\n      +\t\tgit bundle create twoside-boundary.bdl main --since=\"$ad\" &&\n       \ttest_bundle_object_count --thin twoside-boundary.bdl 3\n       '\n     @@ t/t5616-partial-clone.sh: test_expect_success 'fetch lazy-fetches only to resolv\n      -\tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -C client \\\n      +\t#\n      +\t# TODO: the --full-name-hash option is disabled here, since this test\n     -+\t# is fundamentally broken! When GIT_TEST_FULL_NAME_HASH=1, the server\n     ++\t# is fundamentally broken! When GIT_TEST_NAME_HASH_VERSION=2, the server\n      +\t# recognizes delta bases in a different way and then sends a _blob_ to\n      +\t# the client with a delta base that the client does not have! This is\n      +\t# because the client is cloned from \"promisor-server\" with tree:0 but\n     -+\t# is now fetching from \"server\" withot any filter. This is violating the\n     ++\t# is now fetching from \"server\" without any filter. This is violating the\n      +\t# promise to the server that all reachable objects exist and could be\n      +\t# used as delta bases!\n      +\tGIT_TRACE_PACKET=\"$(pwd)/trace\" \\\n     -+\tGIT_TEST_FULL_NAME_HASH=0 \\\n     ++\t\tGIT_TEST_NAME_HASH_VERSION=1 \\\n      +\t\tgit -C client \\\n       \t\tfetch \"file://$(pwd)/server\" main &&\n       \n     @@ t/t5616-partial-clone.sh: test_expect_success 'fetch lazy-fetches only to resolv\n      -\tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -C client \\\n      +\t#\n      +\t# TODO: the --full-name-hash option is disabled here, since this test\n     -+\t# is fundamentally broken! When GIT_TEST_FULL_NAME_HASH=1, the server\n     ++\t# is fundamentally broken! When GIT_TEST_NAME_HASH_VERSION=2, the server\n      +\t# recognizes delta bases in a different way and then sends a _blob_ to\n      +\t# the client with a delta base that the client does not have! This is\n      +\t# because the client is cloned from \"promisor-server\" with tree:0 but\n     -+\t# is now fetching from \"server\" withot any filter. This is violating the\n     ++\t# is now fetching from \"server\" without any filter. This is violating the\n      +\t# promise to the server that all reachable objects exist and could be\n      +\t# used as delta bases!\n      +\tGIT_TRACE_PACKET=\"$(pwd)/trace\" \\\n     -+\tGIT_TEST_FULL_NAME_HASH=0 \\\n     ++\t\tGIT_TEST_NAME_HASH_VERSION=1 \\\n      +\t\tgit -C client \\\n       \t\tfetch \"file://$(pwd)/server\" main &&\n       \n     @@ t/t6020-bundle-misc.sh: test_expect_success 'create bundle with --since option'\n       \ttest_cmp expect actual &&\n       \n      -\tgit bundle create since.bdl \\\n     -+\t# If the --full-name-hash option is used, then one fewer\n     ++\t# If a different name hash function is used, then one fewer\n      +\t# delta base is found and this counts a different number\n      +\t# of objects after performing --fix-thin.\n     -+\tGIT_TEST_FULL_NAME_HASH=0 \\\n     ++\tGIT_TEST_NAME_HASH_VERSION=1 \\\n      +\t\tgit bundle create since.bdl \\\n       \t\t--since \"Thu Apr 7 15:27:00 2005 -0700\" \\\n       \t\t--all &&\n     @@ t/t7406-submodule-update.sh: test_expect_success 'submodule update --quiet passe\n       \t) &&\n       \tgit clone super4 super5 &&\n       \t(cd super5 &&\n     +-\t git submodule update --quiet --init --depth=1 submodule3 >out 2>err &&\n      +\t # This test var can mess with the stderr output checked in this test.\n     -+\t GIT_TEST_FULL_NAME_HASH=0 \\\n     - \t git submodule update --quiet --init --depth=1 submodule3 >out 2>err &&\n     ++\t GIT_TEST_NAME_HASH_VERSION=1 \\\n     ++\t\tgit submodule update --quiet --init --depth=1 submodule3 >out 2>err &&\n       \t test_must_be_empty out &&\n       \t test_must_be_empty err\n     + \t) &&\n      \n       ## t/t7700-repack.sh ##\n      @@ t/t7700-repack.sh: test_expect_success 'no bitmaps created if .keep files present' '\n       \tkeep=${pack%.pack}.keep &&\n       \ttest_when_finished \"rm -f \\\"\\$keep\\\"\" &&\n       \t>\"$keep\" &&\n     +-\tgit -C bare.git repack -ad 2>stderr &&\n      +\n     -+\t# Disable --full-name-hash test due to stderr comparison.\n     -+\tGIT_TEST_FULL_NAME_HASH=0 \\\n     - \tgit -C bare.git repack -ad 2>stderr &&\n     ++\t# Disable --name-hash-version test due to stderr comparison.\n     ++\tGIT_TEST_NAME_HASH_VERSION=1 \\\n     ++\t\tgit -C bare.git repack -ad 2>stderr &&\n       \ttest_must_be_empty stderr &&\n       \tfind bare.git/objects/pack/ -type f -name \"*.bitmap\" >actual &&\n     + \ttest_must_be_empty actual\n      @@ t/t7700-repack.sh: test_expect_success 'auto-bitmaps do not complain if unavailable' '\n       \tblob=$(test-tool genrandom big $((1024*1024)) |\n       \t       git -C bare.git hash-object -w --stdin) &&\n       \tgit -C bare.git update-ref refs/tags/big $blob &&\n     +-\tgit -C bare.git repack -ad 2>stderr &&\n      +\n     -+\t# Disable --full-name-hash test due to stderr comparison.\n     -+\tGIT_TEST_FULL_NAME_HASH=0 \\\n     - \tgit -C bare.git repack -ad 2>stderr &&\n     ++\t# Disable --name-hash-version test due to stderr comparison.\n     ++\tGIT_TEST_NAME_HASH_VERSION=1 \\\n     ++\t\tgit -C bare.git repack -ad 2>stderr &&\n       \ttest_must_be_empty stderr &&\n       \tfind bare.git/objects/pack -type f -name \"*.bitmap\" >actual &&\n     + \ttest_must_be_empty actual\n 4:  65784f85bce < -:  ----------- git-repack: update usage to match docs\n 5:  c14ef6879e4 ! 5:  3b5697467c9 p5313: add size comparison test\n     @@ Commit message\n          adjust how compression is done, use this new performance test script to\n          demonstrate their effectiveness in performance and size.\n      \n     -    The recently-added --full-name-hash option swaps the default name-hash\n     -    algorithm with one that attempts to uniformly distribute the hashes\n     -    based on the full path name instead of the last 16 characters.\n     +    The recently-added --name-hash-version option allows for testing\n     +    different name hash functions. Version 2 intends to preserve some of the\n     +    locality of version 1 while more often breaking collisions due to long\n     +    filenames.\n      \n     -    This has a dramatic effect on full repacks for repositories with many\n     -    versions of most paths. It can have a negative impact on cases such as\n     -    pushing a single change.\n     +    Distinguishing objects by more of the path is critical when there are\n     +    many name hash collisions and several versions of the same path in the\n     +    full history, giving a significant boost to the full repack case. The\n     +    locality of the hash function is critical to compressing something like\n     +    a shallow clone or a thin pack representing a push of a single commit.\n      \n          This can be seen by running pt5313 on the open source fluentui\n          repository [1]. Most commits will have this kind of output for the thin\n     @@ Commit message\n      \n          Checked out at the parent of [2], I see the following statistics:\n      \n     -    Test                                               HEAD\n     -    ---------------------------------------------------------------------\n     -    5313.2: thin pack                                  0.37(0.43+0.02)\n     -    5313.3: thin pack size                                        1.2M\n     -    5313.4: thin pack with --full-name-hash            0.06(0.09+0.02)\n     -    5313.5: thin pack size with --full-name-hash                 20.4K\n     -    5313.6: big pack                                   2.01(7.73+0.23)\n     -    5313.7: big pack size                                        20.3M\n     -    5313.8: big pack with --full-name-hash             1.32(2.77+0.27)\n     -    5313.9: big pack size with --full-name-hash                  19.9M\n     -    5313.10: shallow fetch pack                        1.40(3.01+0.08)\n     -    5313.11: shallow pack size                                   34.4M\n     -    5313.12: shallow pack with --full-name-hash        1.08(1.25+0.14)\n     -    5313.13: shallow pack size with --full-name-hash             35.4M\n     -    5313.14: repack                                    90.70(672.88+2.46)\n     -    5313.15: repack size                                        439.6M\n     -    5313.16: repack with --full-name-hash              18.53(123.41+2.53)\n     -    5313.17: repack size with --full-name-hash                  169.7M\n     -\n     -    In this case, we see positive behaviors such as a significant shrink in\n     -    the size of the thin pack and full repack. The big pack is slightly\n     -    smaller with --full-name-hash than without. The shallow pack is slightly\n     -    larger with --full-name-hash.\n     +    Test                                         HEAD\n     +    ---------------------------------------------------------------\n     +    5313.2: thin pack with version 1             0.37(0.44+0.02)\n     +    5313.3: thin pack size with version 1                   1.2M\n     +    5313.4: big pack with version 1              2.04(7.77+0.23)\n     +    5313.5: big pack size with version 1                   20.4M\n     +    5313.6: shallow fetch pack with version 1    1.41(2.94+0.11)\n     +    5313.7: shallow pack size with version 1               34.4M\n     +    5313.8: repack with version 1                95.70(676.41+2.87)\n     +    5313.9: repack size with version 1                    439.3M\n     +    5313.10: thin pack with version 2            0.12(0.12+0.06)\n     +    5313.11: thin pack size with version 2                 22.0K\n     +    5313.12: big pack with version 2             2.80(5.43+0.34)\n     +    5313.13: big pack size with version 2                  25.9M\n     +    5313.14: shallow fetch pack with version 2   1.77(2.80+0.19)\n     +    5313.15: shallow pack size with version 2              33.7M\n     +    5313.16: repack with version 2               33.68(139.52+2.58)\n     +    5313.17: repack size with version 2                   160.5M\n     +\n     +    To make comparisons easier, I will reformat this output into a different\n     +    table style:\n     +\n     +    | Test         | V1 Time | V2 Time | V1 Size | V2 Size |\n     +    |--------------|---------|---------|---------|---------|\n     +    | Thin Pack    |  0.37 s |  0.12 s |   1.2 M |  22.0 K |\n     +    | Big Pack     |  2.04 s |  2.80 s |  20.4 M |  25.9 M |\n     +    | Shallow Pack |  1.41 s |  1.77 s |  34.4 M |  33.7 M |\n     +    | Repack       | 95.70 s | 33.68 s | 439.3 M | 160.5 M |\n     +\n     +    The v2 hash function successfully differentiates the CHANGELOG.md files\n     +    from each other, which leads to significant improvements in the thin\n     +    pack (simulating a push of this commit) and the full repack. There is\n     +    some bloat in the \"big pack\" scenario and essentially the same results\n     +    for the shallow pack.\n      \n          In the case of the Git repository, these numbers show some of the issues\n          with this approach:\n      \n     -    Test                                               HEAD\n     -    --------------------------------------------------------------------\n     -    5313.2: thin pack                                  0.00(0.00+0.00)\n     -    5313.3: thin pack size                                         589\n     -    5313.4: thin pack with --full-name-hash            0.00(0.00+0.00)\n     -    5313.5: thin pack size with --full-name-hash                 14.9K\n     -    5313.6: big pack                                   2.07(3.57+0.17)\n     -    5313.7: big pack size                                        17.6M\n     -    5313.8: big pack with --full-name-hash             2.00(3.07+0.19)\n     -    5313.9: big pack size with --full-name-hash                  17.9M\n     -    5313.10: shallow fetch pack                        1.41(2.23+0.06)\n     -    5313.11: shallow pack size                                   12.1M\n     -    5313.12: shallow pack with --full-name-hash        1.22(1.66+0.04)\n     -    5313.13: shallow pack size with --full-name-hash             12.4M\n     -    5313.14: repack                                    15.75(89.29+1.54)\n     -    5313.15: repack size                                        126.4M\n     -    5313.16: repack with --full-name-hash              15.56(89.78+1.32)\n     -    5313.17: repack size with --full-name-hash                  126.0M\n     -\n     -    The thin pack that simulates a push is much worse with --full-name-hash\n     -    in this case. The name hash values are doing a lot to assist with delta\n     -    bases, it seems. The big pack and shallow clone cases are slightly worse\n     -    with the --full-name-hash option. Only the full repack gains some\n     -    benefits in size.\n     +    | Test         | V1 Time | V2 Time | V1 Size | V2 Size |\n     +    |--------------|---------|---------|---------|---------|\n     +    | Thin Pack    |  0.02 s |  0.02 s |   1.1 K |   1.1 K |\n     +    | Big Pack     |  1.69 s |  1.95 s |  13.5 M |  14.5 M |\n     +    | Shallow Pack |  1.26 s |  1.29 s |  12.0 M |  12.2 M |\n     +    | Repack       | 29.51 s | 29.01 s | 237.7 M | 238.2 M |\n     +\n     +    Here, the attempts to remove conflicts in the v2 function seem to cause\n     +    slight bloat to these sizes. This shows that the Git repository benefits\n     +    a lot from cross-path delta pairs.\n      \n          The results are similar with the nodejs/node repo:\n      \n     -    Test                                               HEAD\n     -    ---------------------------------------------------------------------\n     -    5313.2: thin pack                                  0.01(0.01+0.00)\n     -    5313.3: thin pack size                                        1.6K\n     -    5313.4: thin pack with --full-name-hash            0.01(0.00+0.00)\n     -    5313.5: thin pack size with --full-name-hash                  3.1K\n     -    5313.6: big pack                                   4.26(8.03+0.24)\n     -    5313.7: big pack size                                        56.0M\n     -    5313.8: big pack with --full-name-hash             4.16(6.55+0.22)\n     -    5313.9: big pack size with --full-name-hash                  56.2M\n     -    5313.10: shallow fetch pack                        7.67(11.80+0.29)\n     -    5313.11: shallow pack size                                  104.6M\n     -    5313.12: shallow pack with --full-name-hash        7.52(9.65+0.23)\n     -    5313.13: shallow pack size with --full-name-hash            105.9M\n     -    5313.14: repack                                    71.22(317.61+3.95)\n     -    5313.15: repack size                                        739.9M\n     -    5313.16: repack with --full-name-hash              48.85(267.02+3.72)\n     -    5313.17: repack size with --full-name-hash                  793.5M\n     +    | Test         | V1 Time | V2 Time | V1 Size | V2 Size |\n     +    |--------------|---------|---------|---------|---------|\n     +    | Thin Pack    |  0.02 s |  0.02 s |   1.6 K |   1.6 K |\n     +    | Big Pack     |  4.61 s |  3.26 s |  56.0 M |  52.8 M |\n     +    | Shallow Pack |  7.82 s |  7.51 s | 104.6 M | 107.0 M |\n     +    | Repack       | 88.90 s | 73.75 s | 740.1 M | 764.5 M |\n     +\n     +    Here, the v2 name-hash causes some size bloat more often than it reduces\n     +    the size, but it also universally improves performance time, which is an\n     +    interesting reversal. This must mean that it is helping to short-circuit\n     +    some delta computations even if it is not finding the most efficient\n     +    ones. The performance improvement cannot be explained only due to the\n     +    I/O cost of writing the resulting packfile.\n      \n          The Linux kernel repository was the initial target of the default name\n          hash value, and its naming conventions are practically build to take the\n          most advantage of the default name hash values:\n      \n     -    Test                                               HEAD\n     -    -------------------------------------------------------------------------\n     -    5313.2: thin pack                                  0.15(0.01+0.03)\n     -    5313.3: thin pack size                                        4.6K\n     -    5313.4: thin pack with --full-name-hash            0.03(0.02+0.01)\n     -    5313.5: thin pack size with --full-name-hash                  6.8K\n     -    5313.6: big pack                                   18.51(33.74+0.95)\n     -    5313.7: big pack size                                       201.1M\n     -    5313.8: big pack with --full-name-hash             16.01(29.81+0.88)\n     -    5313.9: big pack size with --full-name-hash                 202.1M\n     -    5313.10: shallow fetch pack                        11.49(17.61+0.54)\n     -    5313.11: shallow pack size                                  269.2M\n     -    5313.12: shallow pack with --full-name-hash        11.24(15.25+0.56)\n     -    5313.13: shallow pack size with --full-name-hash            269.8M\n     -    5313.14: repack                                    1001.25(2271.06+38.86)\n     -    5313.15: repack size                                          2.5G\n     -    5313.16: repack with --full-name-hash              625.75(1941.96+36.09)\n     -    5313.17: repack size with --full-name-hash                    2.6G\n     +    | Test         | V1 Time  | V2 Time  | V1 Size | V2 Size |\n     +    |--------------|----------|----------|---------|---------|\n     +    | Thin Pack    |   0.17 s |   0.07 s |   4.6 K |   4.6 K |\n     +    | Big Pack     |  17.88 s |  12.35 s | 201.1 M | 159.1 M |\n     +    | Shallow Pack |  11.05 s |  22.94 s | 269.2 M | 273.8 M |\n     +    | Repack       | 727.39 s | 566.95 s |   2.5 G |   2.5 G |\n     +\n     +    Here, the thin and big packs gain some performance boosts in time, with\n     +    a modest gain in the size of the big pack. The shallow pack, however, is\n     +    more expensive to compute, likely because similarly-named files across\n     +    different directories are farther apart in the name hash ordering in v2.\n     +    The repack also gains benefits in computation time but no meaningful\n     +    change to the full size.\n      \n          Finally, an internal Javascript repo of moderate size shows significant\n     -    gains when repacking with --full-name-hash due to it having many name\n     +    gains when repacking with --name-hash-version=2 due to it having many name\n          hash collisions. However, it's worth noting that only the full repack\n     -    case has enough improvement to be worth it. But the improvements are\n     -    significant: 6.4 GB to 862 MB.\n     -\n     -    Test                                               HEAD\n     -    --------------------------------------------------------------------------\n     -    5313.2: thin pack                                  0.03(0.02+0.00)\n     -    5313.3: thin pack size                                        1.2K\n     -    5313.4: thin pack with --full-name-hash            0.03(0.03+0.00)\n     -    5313.5: thin pack size with --full-name-hash                  2.6K\n     -    5313.6: big pack                                   2.20(3.23+0.30)\n     -    5313.7: big pack size                                       130.7M\n     -    5313.8: big pack with --full-name-hash             2.33(3.17+0.34)\n     -    5313.9: big pack size with --full-name-hash                 131.0M\n     -    5313.10: shallow fetch pack                        3.56(6.02+0.32)\n     -    5313.11: shallow pack size                                   44.5M\n     -    5313.12: shallow pack with --full-name-hash        2.94(3.94+0.32)\n     -    5313.13: shallow pack size with --full-name-hash             45.3M\n     -    5313.14: repack                                    2435.22(12523.11+23.53)\n     -    5313.15: repack size                                          6.4G\n     -    5313.16: repack with --full-name-hash              473.25(1805.11+17.22)\n     -    5313.17: repack size with --full-name-hash                  861.9M\n     -\n     -    These tests demonstrate that it is important to be careful about which\n     -    cases are best for using the --full-name-hash option.\n     +    case has significant differences from the v1 name hash:\n     +\n     +    | Test      | V1 Time   | V2 Time  | V1 Size | V2 Size |\n     +    |-----------|-----------|----------|---------|---------|\n     +    | Thin Pack |    8.28 s |   7.28 s |  16.8 K |  16.8 K |\n     +    | Big Pack  |   12.81 s |  11.66 s |  29.1 M |  29.1 M |\n     +    | Shallow   |    4.86 s |   4.06 s |  42.5 M |  44.1 M |\n     +    | Repack    | 3126.50 s | 496.33 s |   6.2 G | 855.6 M |\n      \n          Signed-off-by: Derrick Stolee <stolee@gmail.com>\n      \n     @@ t/perf/p5313-pack-objects.sh (new)\n      +\tEOF\n      +'\n      +\n     -+test_perf 'thin pack' '\n     -+\tgit pack-objects --thin --stdout --revs --sparse  <in-thin >out\n     -+'\n     -+\n     -+test_size 'thin pack size' '\n     -+\ttest_file_size out\n     -+'\n     ++for version in 1 2\n     ++do\n     ++\texport version\n      +\n     -+test_perf 'thin pack with --full-name-hash' '\n     -+\tgit pack-objects --thin --stdout --revs --sparse --full-name-hash <in-thin >out\n     -+'\n     ++\ttest_perf \"thin pack with version $version\" '\n     ++\t\tgit pack-objects --thin --stdout --revs --sparse \\\n     ++\t\t\t--name-hash-version=$version <in-thin >out\n     ++\t'\n      +\n     -+test_size 'thin pack size with --full-name-hash' '\n     -+\ttest_file_size out\n     -+'\n     ++\ttest_size \"thin pack size with version $version\" '\n     ++\t\ttest_file_size out\n     ++\t'\n      +\n     -+test_perf 'big pack' '\n     -+\tgit pack-objects --stdout --revs --sparse  <in-big >out\n     -+'\n     ++\ttest_perf \"big pack with version $version\" '\n     ++\t\tgit pack-objects --stdout --revs --sparse \\\n     ++\t\t\t--name-hash-version=$version <in-big >out\n     ++\t'\n      +\n     -+test_size 'big pack size' '\n     -+\ttest_file_size out\n     -+'\n     ++\ttest_size \"big pack size with version $version\" '\n     ++\t\ttest_file_size out\n     ++\t'\n      +\n     -+test_perf 'big pack with --full-name-hash' '\n     -+\tgit pack-objects --stdout --revs --sparse --full-name-hash <in-big >out\n     -+'\n     ++\ttest_perf \"shallow fetch pack with version $version\" '\n     ++\t\tgit pack-objects --stdout --revs --sparse --shallow \\\n     ++\t\t\t--name-hash-version=$version <in-shallow >out\n     ++\t'\n      +\n     -+test_size 'big pack size with --full-name-hash' '\n     -+\ttest_file_size out\n     -+'\n     ++\ttest_size \"shallow pack size with version $version\" '\n     ++\t\ttest_file_size out\n     ++\t'\n      +\n     -+test_perf 'shallow fetch pack' '\n     -+\tgit pack-objects --stdout --revs --sparse --shallow <in-shallow >out\n     -+'\n     ++\ttest_perf \"repack with version $version\" '\n     ++\t\tgit repack -adf --name-hash-version=$version\n     ++\t'\n      +\n     -+test_size 'shallow pack size' '\n     -+\ttest_file_size out\n     -+'\n     -+\n     -+test_perf 'shallow pack with --full-name-hash' '\n     -+\tgit pack-objects --stdout --revs --sparse --shallow --full-name-hash <in-shallow >out\n     -+'\n     -+\n     -+test_size 'shallow pack size with --full-name-hash' '\n     -+\ttest_file_size out\n     -+'\n     -+\n     -+test_perf 'repack' '\n     -+\tgit repack -adf\n     -+'\n     -+\n     -+test_size 'repack size' '\n     -+\tpack=$(ls .git/objects/pack/pack-*.pack) &&\n     -+\ttest_file_size \"$pack\"\n     -+'\n     -+\n     -+test_perf 'repack with --full-name-hash' '\n     -+\tgit repack -adf --full-name-hash\n     -+'\n     -+\n     -+test_size 'repack size with --full-name-hash' '\n     -+\tpack=$(ls .git/objects/pack/pack-*.pack) &&\n     -+\ttest_file_size \"$pack\"\n     -+'\n     ++\ttest_size \"repack size with version $version\" '\n     ++\t\tgitdir=$(git rev-parse --git-dir) &&\n     ++\t\tpack=$(ls $gitdir/objects/pack/pack-*.pack) &&\n     ++\t\ttest_file_size \"$pack\"\n     ++\t'\n     ++done\n      +\n      +test_done\n 6:  b8a055cb196 < -:  ----------- pack-objects: disable --full-name-hash when shallow\n 7:  ab341dd0e58 ! 6:  36f2811e3d9 test-tool: add helper for name-hash values\n     @@ Commit message\n          important that these hash functions do not change across Git versions.\n          Add a simple test to t5310-pack-bitmaps.sh to provide some testing of\n          the current values. Due to how these functions are implemented, it would\n     -    be difficult to change them without disturbing these values.\n     +    be difficult to change them without disturbing these values. The paths\n     +    used for this test are carefully selected to demonstrate some of the\n     +    behavior differences of the two current name hash versions, including\n     +    which conditions will cause them to collide.\n      \n          Create a performance test that uses test_size to demonstrate how\n          collisions occur for these hash algorithms. This test helps inform\n     @@ Commit message\n          My copy of the Git repository shows modest statistics around the\n          collisions of the default name-hash algorithm:\n      \n     -    Test                                              this tree\n     -    -----------------------------------------------------------------\n     -    5314.1: paths at head                                        4.5K\n     -    5314.2: number of distinct name-hashes                       4.1K\n     -    5314.3: number of distinct full-name-hashes                  4.5K\n     -    5314.4: maximum multiplicity of name-hashes                    13\n     -    5314.5: maximum multiplicity of fullname-hashes                 1\n     +    Test                               this tree\n     +    --------------------------------------------------\n     +    5314.1: paths at head                         4.5K\n     +    5314.2: distinct hash value: v1               4.1K\n     +    5314.3: maximum multiplicity: v1                13\n     +    5314.4: distinct hash value: v2               4.2K\n     +    5314.5: maximum multiplicity: v2                 9\n      \n          Here, the maximum collision multiplicity is 13, but around 10% of paths\n          have a collision with another path.\n     @@ Commit message\n          In a more interesting example, the microsoft/fluentui [1] repo had these\n          statistics at time of committing:\n      \n     -    Test                                              this tree\n     -    -----------------------------------------------------------------\n     -    5314.1: paths at head                                       19.6K\n     -    5314.2: number of distinct name-hashes                       8.2K\n     -    5314.3: number of distinct full-name-hashes                 19.6K\n     -    5314.4: maximum multiplicity of name-hashes                   279\n     -    5314.5: maximum multiplicity of fullname-hashes                 1\n     +    Test                               this tree\n     +    --------------------------------------------------\n     +    5314.1: paths at head                        19.5K\n     +    5314.2: distinct hash value: v1               8.2K\n     +    5314.3: maximum multiplicity: v1               279\n     +    5314.4: distinct hash value: v2              17.8K\n     +    5314.5: maximum multiplicity: v2                44\n      \n          [1] https://github.com/microsoft/fluentui\n      \n     @@ Commit message\n          assigned to a single value, leading the packing algorithm to sort\n          objects from those paths together, by size.\n      \n     -    In this repository, no collisions occur for the full-name-hash\n     -    algorithm.\n     +    With the v2 name hash function, the maximum multiplicity lowers to 44,\n     +    leaving some room for further improvement.\n      \n          In a more extreme example, an internal monorepo had a much worse\n          collision rate:\n      \n     -    Test                                              this tree\n     -    -----------------------------------------------------------------\n     -    5314.1: paths at head                                      221.6K\n     -    5314.2: number of distinct name-hashes                      72.0K\n     -    5314.3: number of distinct full-name-hashes                221.6K\n     -    5314.4: maximum multiplicity of name-hashes                 14.4K\n     -    5314.5: maximum multiplicity of fullname-hashes                 2\n     +    Test                               this tree\n     +    --------------------------------------------------\n     +    5314.1: paths at head                       227.3K\n     +    5314.2: distinct hash value: v1              72.3K\n     +    5314.3: maximum multiplicity: v1             14.4K\n     +    5314.4: distinct hash value: v2             166.5K\n     +    5314.5: maximum multiplicity: v2               138\n      \n     -    Even in this repository with many more paths at HEAD, the collision rate\n     -    was low and the maximum number of paths being grouped into a single\n     -    bucket by the full-path-name algorithm was two.\n     +    Here, we can see that the v2 name hash function provides somem\n     +    improvements, but there are still a number of collisions that could lead\n     +    to repacking problems at this scale.\n      \n          Signed-off-by: Derrick Stolee <stolee@gmail.com>\n      \n     @@ t/helper/test-name-hash.c (new)\n      +\tstruct strbuf line = STRBUF_INIT;\n      +\n      +\twhile (!strbuf_getline(&line, stdin)) {\n     -+\t\tuint32_t name_hash = pack_name_hash(line.buf);\n     -+\t\tuint32_t full_hash = pack_full_name_hash(line.buf);\n     -+\n     -+\t\tprintf(\"%10\"PRIu32\"\\t%10\"PRIu32\"\\t%s\\n\", name_hash, full_hash, line.buf);\n     ++\t\tprintf(\"%10u \", pack_name_hash(line.buf));\n     ++\t\tprintf(\"%10u \", pack_name_hash_v2(line.buf));\n     ++\t\tprintf(\"%s\\n\", line.buf);\n      +\t}\n      +\n      +\tstrbuf_release(&line);\n     @@ t/perf/p5314-name-hash.sh (new)\n      +\n      +test_size 'paths at head' '\n      +\tgit ls-tree -r --name-only HEAD >path-list &&\n     -+\twc -l <path-list\n     -+'\n     -+\n     -+test_size 'number of distinct name-hashes' '\n     -+\tcat path-list | test-tool name-hash >name-hashes &&\n     -+\tcat name-hashes | awk \"{ print \\$1; }\" | sort -n | uniq -c >name-hash-count &&\n     -+\twc -l <name-hash-count\n     ++\twc -l <path-list &&\n     ++\ttest-tool name-hash <path-list >name-hashes\n      +'\n      +\n     -+test_size 'number of distinct full-name-hashes' '\n     -+\tcat name-hashes | awk \"{ print \\$2; }\" | sort -n | uniq -c >full-name-hash-count &&\n     -+\twc -l <full-name-hash-count\n     -+'\n     ++for version in 1 2\n     ++do\n     ++\ttest_size \"distinct hash value: v$version\" '\n     ++\t\tawk \"{ print \\$$version; }\" <name-hashes | sort | \\\n     ++\t\t\tuniq -c >name-hash-count &&\n     ++\t\twc -l <name-hash-count\n     ++\t'\n      +\n     -+test_size 'maximum multiplicity of name-hashes' '\n     -+\tcat name-hash-count | \\\n     -+\t\tsort -nr | \\\n     -+\t\thead -n 1 | \\\n     -+\t\tawk \"{ print \\$1; }\"\n     -+'\n     -+\n     -+test_size 'maximum multiplicity of fullname-hashes' '\n     -+\tcat full-name-hash-count | \\\n     -+\t\tsort -nr | \\\n     -+\t\thead -n 1 | \\\n     -+\t\tawk \"{ print \\$1; }\"\n     -+'\n     ++\ttest_size \"maximum multiplicity: v$version\" '\n     ++\t\tsort -nr <name-hash-count | head -n 1 |\t\\\n     ++\t\t\tawk \"{ print \\$1; }\"\n     ++\t'\n     ++done\n      +\n      +test_done\n      \n     @@ t/t5310-pack-bitmaps.sh: has_any () {\n      +\tfirst\n      +\tsecond\n      +\tthird\n     -+\tone-long-enough-for-collisions\n     -+\ttwo-long-enough-for-collisions\n     ++\ta/one-long-enough-for-collisions\n     ++\tb/two-long-enough-for-collisions\n     ++\tmany/parts/to/this/path/enough/to/collide/in/v2\n     ++\tenough/parts/to/this/path/enough/to/collide/in/v2\n      +\tEOF\n      +\n      +\ttest-tool name-hash <names >out &&\n      +\n      +\tcat >expect <<-\\EOF &&\n     -+\t2582249472\t3109209818\tfirst\n     -+\t2289942528\t3781118409\tsecond\n     -+\t2300837888\t3028707182\tthird\n     -+\t2544516325\t3241327563\tone-long-enough-for-collisions\n     -+\t2544516325\t4207880830\ttwo-long-enough-for-collisions\n     ++\t2582249472 1763573760 first\n     ++\t2289942528 1188134912 second\n     ++\t2300837888 1130758144 third\n     ++\t2544516325 3963087891 a/one-long-enough-for-collisions\n     ++\t2544516325 4013419539 b/two-long-enough-for-collisions\n     ++\t1420111091 1709547268 many/parts/to/this/path/enough/to/collide/in/v2\n     ++\t1420111091 1709547268 enough/parts/to/this/path/enough/to/collide/in/v2\n      +\tEOF\n      +\n      +\ttest_cmp expect out\n -:  ----------- > 7:  3885ef8a2f7 pack-objects: prevent name hash version change\n -:  ----------- > 8:  64fd7b3ccad pack-objects: add third name hash version\n\n-- \ngitgitgadget\n"},{"id":"508475","messageId":"fb52ca509da6b7a58d7148e3a15ae222ff209cc6.1733181682.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.v2.git.1733181682.gitgitgadget@gmail.com","subject":"[PATCH v2 2/8] pack-objects: add --name-hash-version option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-02T23:21:16Z","receivedAt":"2024-12-02T23:21:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nThe previous change introduced a new pack_name_hash_v2() function that\nintends to satisfy much of the hash locality features of the existing\npack_name_hash() function while also distinguishing paths with similar\nfinal components of their paths.\n\nThis change adds a new --name-hash-version option for 'git pack-objects'\nto allow users to select their preferred function version. This use of\nan integer version allows for future expansion and a direct way to later\nstore a name hash version in the .bitmap format.\n\nFor now, let's consider how effective this mechanism is when repacking a\nrepository with different name hash versions. Specifically, we will\nexecute 'git pack-objects' the same way a 'git repack -adf' process\nwould, except we include --name-hash-version=<n> for testing.\n\nOn the Git repository, we do not expect much difference. All path names\nare short. This is backed by our results:\n\n| Stage                 | Pack Size | Repack Time |\n|-----------------------|-----------|-------------|\n| After clone           | 260 MB    | N/A         |\n| --name-hash-version=1 | 127 MB    | 129s        |\n| --name-hash-version=2 | 127 MB    | 112s        |\n\nThis example demonstrates how there is some natural overhead coming from\nthe cloned copy because the server is hosting many forks and has not\noptimized for exactly this set of reachable objects. But the full repack\nhas similar characteristics for both versions.\n\nLet's consider some repositories that are hitting too many collisions\nwith version 1. First, let's explore the kinds of paths that are\ncommonly causing these collisions:\n\n * \"/CHANGELOG.json\" is 15 characters, and is created by the beachball\n   [1] tool. Only the final character of the parent directory can\n   differentiate different versions of this file, but also only the two\n   most-significant digits. If that character is a letter, then this is\n   always a collision. Similar issues occur with the similar\n   \"/CHANGELOG.md\" path, though there is more opportunity for\n   differences In the parent directory.\n\n * Localization files frequently have common filenames but\n   differentiates via parent directories. In C#, the name\n   \"/strings.resx.lcl\" is used for these localization files and they\n   will all collide in name-hash.\n\n[1] https://github.com/microsoft/beachball\n\nI've come across many other examples where some internal tool uses a\ncommon name across multiple directories and is causing Git to repack\npoorly due to name-hash collisions.\n\nOne open-source example is the fluentui [2] repo, which  uses beachball\nto generate CHANGELOG.json and CHANGELOG.md files, and these files have\nvery poor delta characteristics when comparing against versions across\nparent directories.\n\n| Stage                 | Pack Size | Repack Time |\n|-----------------------|-----------|-------------|\n| After clone           | 694 MB    | N/A         |\n| --name-hash-version=1 | 438 MB    | 728s        |\n| --name-hash-version=2 | 168 MB    | 142s        |\n\n[2] https://github.com/microsoft/fluentui\n\nIn this example, we see significant gains in the compressed packfile\nsize as well as the time taken to compute the packfile.\n\nUsing a collection of repositories that use the beachball tool, I was\nable to make similar comparisions with dramatic results. While the\nfluentui repo is public, the others are private so cannot be shared for\nreproduction. The results are so significant that I find it important to\nshare here:\n\n| Repo     | --name-hash-version=1 | --name-hash-version=2 |\n|----------|-----------------------|-----------------------|\n| fluentui |               440 MB  |               161 MB  |\n| Repo B   |             6,248 MB  |               856 MB  |\n| Repo C   |            37,278 MB  |             6,755 MB  |\n| Repo D   |           131,204 MB  |             7,463 MB  |\n\nFuture changes could include making --name-hash-version implied by a config\nvalue or even implied by default during a full repack.\n\nIt is important to point out that the name hash value is stored in the\n.bitmap file format, so we must force --name-hash-version=1 when bitmaps\nare being read or written. Later, the bitmap format could be updated to\nbe aware of the name hash version so deltas can be quickly computed\nacross the bitmapped/not-bitmapped boundary.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Documentation/git-pack-objects.txt | 32 ++++++++++++++++++-\n builtin/pack-objects.c             | 49 +++++++++++++++++++++++++++---\n builtin/repack.c                   |  1 +\n t/t5300-pack-object.sh             | 31 +++++++++++++++++++\n 4 files changed, 107 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex e32404c6aae..7f69ae4855f 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -15,7 +15,8 @@ SYNOPSIS\n \t[--revs [--unpacked | --all]] [--keep-pack=<pack-name>]\n \t[--cruft] [--cruft-expiration=<time>]\n \t[--stdout [--filter=<filter-spec>] | <base-name>]\n-\t[--shallow] [--keep-true-parents] [--[no-]sparse] < <object-list>\n+\t[--shallow] [--keep-true-parents] [--[no-]sparse]\n+\t[--name-hash-version=<n>] < <object-list>\n \n \n DESCRIPTION\n@@ -345,6 +346,35 @@ raise an error.\n \tRestrict delta matches based on \"islands\". See DELTA ISLANDS\n \tbelow.\n \n+--name-hash-version=<n>::\n+\tWhile performing delta compression, Git groups objects that may be\n+\tsimilar based on heuristics using the path to that object. While\n+\tgrouping objects by an exact path match is good for paths with\n+\tmany versions, there are benefits for finding delta pairs across\n+\tdifferent full paths. Git collects objects by type and then by a\n+\t\"name hash\" of the path and then by size, hoping to group objects\n+\tthat will compress well together.\n++\n+The default name hash version is `1`, which prioritizes hash locality by\n+considering the final bytes of the path as providing the maximum magnitude\n+to the hash function. This version excels at distinguishing short paths\n+and finding renames across directories. However, the hash function depends\n+primarily on the final 16 bytes of the path. If there are many paths in\n+the repo that have the same final 16 bytes and differ only by parent\n+directory, then this name-hash may lead to too many collisions and cause\n+poor results. At the moment, this version is required when writing\n+reachability bitmap files with `--write-bitmap-index`.\n++\n+The name hash version `2` has similar locality features as version `1`,\n+except it considers each path component separately and overlays the hashes\n+with a shift. This still prioritizes the final bytes of the path, but also\n+\"salts\" the lower bits of the hash using the parent directory names. This\n+method allows for some of the locality benefits of version `1` while\n+breaking most of the collisions from a similarly-named file appearing in\n+many different directories. At the moment, this version is not allowed\n+when writing reachability bitmap files with `--write-bitmap-index` and it\n+will be automatically changed to version `1`.\n+\n \n DELTA ISLANDS\n -------------\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 08007142671..e36a93a0082 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -266,6 +266,28 @@ struct configured_exclusion {\n static struct oidmap configured_exclusions;\n \n static struct oidset excluded_by_config;\n+static int name_hash_version = 1;\n+\n+static void validate_name_hash_version(void)\n+{\n+\tif (name_hash_version < 1 || name_hash_version > 2)\n+\t\tdie(_(\"invalid --name-hash-version option: %d\"), name_hash_version);\n+}\n+\n+static inline uint32_t pack_name_hash_fn(const char *name)\n+{\n+\tswitch (name_hash_version)\n+\t{\n+\tcase 1:\n+\t\treturn pack_name_hash(name);\n+\n+\tcase 2:\n+\t\treturn pack_name_hash_v2(name);\n+\n+\tdefault:\n+\t\tBUG(\"invalid name-hash version: %d\", name_hash_version);\n+\t}\n+}\n \n /*\n  * stats\n@@ -1698,7 +1720,7 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n \t\treturn 0;\n \t}\n \n-\tcreate_object_entry(oid, type, pack_name_hash(name),\n+\tcreate_object_entry(oid, type, pack_name_hash_fn(name),\n \t\t\t    exclude, name && no_try_delta(name),\n \t\t\t    found_pack, found_offset);\n \treturn 1;\n@@ -1912,7 +1934,7 @@ static void add_preferred_base_object(const char *name)\n {\n \tstruct pbase_tree *it;\n \tsize_t cmplen;\n-\tunsigned hash = pack_name_hash(name);\n+\tunsigned hash = pack_name_hash_fn(name);\n \n \tif (!num_preferred_base || check_pbase_path(hash))\n \t\treturn;\n@@ -3422,7 +3444,7 @@ static void show_object_pack_hint(struct object *object, const char *name,\n \t * here using a now in order to perhaps improve the delta selection\n \t * process.\n \t */\n-\toe->hash = pack_name_hash(name);\n+\toe->hash = pack_name_hash_fn(name);\n \toe->no_try_delta = name && no_try_delta(name);\n \n \tstdin_packs_hints_nr++;\n@@ -3572,7 +3594,7 @@ static void add_cruft_object_entry(const struct object_id *oid, enum object_type\n \tentry = packlist_find(&to_pack, oid);\n \tif (entry) {\n \t\tif (name) {\n-\t\t\tentry->hash = pack_name_hash(name);\n+\t\t\tentry->hash = pack_name_hash_fn(name);\n \t\t\tentry->no_try_delta = no_try_delta(name);\n \t\t}\n \t} else {\n@@ -3595,7 +3617,7 @@ static void add_cruft_object_entry(const struct object_id *oid, enum object_type\n \t\t\treturn;\n \t\t}\n \n-\t\tentry = create_object_entry(oid, type, pack_name_hash(name),\n+\t\tentry = create_object_entry(oid, type, pack_name_hash_fn(name),\n \t\t\t\t\t    0, name && no_try_delta(name),\n \t\t\t\t\t    pack, offset);\n \t}\n@@ -4069,6 +4091,15 @@ static int get_object_list_from_bitmap(struct rev_info *revs)\n \tif (!(bitmap_git = prepare_bitmap_walk(revs, 0)))\n \t\treturn -1;\n \n+\t/*\n+\t * For now, force the name-hash version to be 1 since that\n+\t * is the version implied by the bitmap format. Later, the\n+\t * format can include this version explicitly in its format,\n+\t * allowing readers to know the version that was used during\n+\t * the bitmap write.\n+\t */\n+\tname_hash_version = 1;\n+\n \tif (pack_options_allow_reuse())\n \t\treuse_partial_packfile_from_bitmap(bitmap_git,\n \t\t\t\t\t\t   &reuse_packfiles,\n@@ -4429,6 +4460,8 @@ int cmd_pack_objects(int argc,\n \t\tOPT_STRING_LIST(0, \"uri-protocol\", &uri_protocols,\n \t\t\t\tN_(\"protocol\"),\n \t\t\t\tN_(\"exclude any configured uploadpack.blobpackfileuri with this protocol\")),\n+\t\tOPT_INTEGER(0, \"name-hash-version\", &name_hash_version,\n+\t\t\t N_(\"use the specified name-hash function to group similar objects\")),\n \t\tOPT_END(),\n \t};\n \n@@ -4576,6 +4609,12 @@ int cmd_pack_objects(int argc,\n \tif (pack_to_stdout || !rev_list_all)\n \t\twrite_bitmap_index = 0;\n \n+\tvalidate_name_hash_version();\n+\tif (write_bitmap_index && name_hash_version != 1) {\n+\t\twarning(_(\"currently, --write-bitmap-index requires --name-hash-version=1\"));\n+\t\tname_hash_version = 1;\n+\t}\n+\n \tif (use_delta_islands)\n \t\tstrvec_push(&rp, \"--topo-order\");\n \ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex d6bb37e84ae..05e13adb87f 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -58,6 +58,7 @@ struct pack_objects_args {\n \tint no_reuse_object;\n \tint quiet;\n \tint local;\n+\tint full_name_hash;\n \tstruct list_objects_filter_options filter_options;\n };\n \ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 3b9dae331a5..4270eabe8b7 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -674,4 +674,35 @@ do\n \t'\n done\n \n+test_expect_success 'valid and invalid --name-hash-versions' '\n+\t# Valid values are hard to verify other than \"do not fail\".\n+\t# Performance tests will be more valuable to validate these versions.\n+\tfor value in 1 2\n+\tdo\n+\t\tgit pack-objects base --all --name-hash-version=$value || return 1\n+\tdone &&\n+\n+\t# Invalid values have clear post-conditions.\n+\tfor value in -1 0 3\n+\tdo\n+\t\ttest_must_fail git pack-objects base --all --name-hash-version=$value 2>err &&\n+\t\ttest_grep \"invalid --name-hash-version option\" err || return 1\n+\tdone\n+'\n+\n+# The following test is not necessarily a permanent choice, but since we do not\n+# have a \"name hash version\" bit in the .bitmap file format, we cannot write the\n+# hash values into the .bitmap file without risking breakage later.\n+#\n+# TODO: Make these compatible in the future and replace this test with the\n+# expected behavior when both are specified.\n+test_expect_success '--name-hash-version=2 and --write-bitmap-index are incompatible' '\n+\tgit pack-objects base --all --name-hash-version=2 --write-bitmap-index 2>err &&\n+\ttest_grep \"currently, --write-bitmap-index requires --name-hash-version=1\" err &&\n+\n+\t# --stdout option silently removes --write-bitmap-index\n+\tgit pack-objects --stdout --all --name-hash-version=2 --write-bitmap-index >out 2>err &&\n+\t! test_grep \"currently, --write-bitmap-index requires --name-hash-version=1\" err\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"508476","messageId":"1947d1bf448d71ccd4a44ef25751bab50784a73e.1733181682.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.v2.git.1733181682.gitgitgadget@gmail.com","subject":"[PATCH v2 3/8] repack: add --name-hash-version option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-02T23:21:17Z","receivedAt":"2024-12-02T23:21:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nThe new '--name-hash-version' option for 'git repack' is a simple\npass-through to the underlying 'git pack-objects' subcommand. However,\nthis subcommand may have other options and a temporary filename as part\nof the subcommand execution that may not be predictable or could change\nover time.\n\nThe existing test_subcommand method requires an exact list of arguments\nfor the subcommand. This is too rigid for our needs here, so create a\nnew method, test_subcommand_flex. Use it to check that the\n--name-hash-version option is passing through.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Documentation/git-repack.txt | 34 +++++++++++++++++++++++++++++++++-\n builtin/repack.c             | 10 ++++++++--\n t/t0450/txt-help-mismatches  |  1 -\n t/t7700-repack.sh            |  6 ++++++\n t/test-lib-functions.sh      | 26 ++++++++++++++++++++++++++\n 5 files changed, 73 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex c902512a9e8..ea69fbe1891 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -9,7 +9,9 @@ git-repack - Pack unpacked objects in a repository\n SYNOPSIS\n --------\n [verse]\n-'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m] [--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>] [--write-midx]\n+'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\n+\t[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\n+\t[--write-midx] [--name-hash-version=<n>]\n \n DESCRIPTION\n -----------\n@@ -249,6 +251,36 @@ linkgit:git-multi-pack-index[1]).\n \tWrite a multi-pack index (see linkgit:git-multi-pack-index[1])\n \tcontaining the non-redundant packs.\n \n+--name-hash-version=<n>::\n+\tWhile performing delta compression, Git groups objects that may be\n+\tsimilar based on heuristics using the path to that object. While\n+\tgrouping objects by an exact path match is good for paths with\n+\tmany versions, there are benefits for finding delta pairs across\n+\tdifferent full paths. Git collects objects by type and then by a\n+\t\"name hash\" of the path and then by size, hoping to group objects\n+\tthat will compress well together.\n++\n+The default name hash version is `1`, which prioritizes hash locality by\n+considering the final bytes of the path as providing the maximum magnitude\n+to the hash function. This version excels at distinguishing short paths\n+and finding renames across directories. However, the hash function depends\n+primarily on the final 16 bytes of the path. If there are many paths in\n+the repo that have the same final 16 bytes and differ only by parent\n+directory, then this name-hash may lead to too many collisions and cause\n+poor results. At the moment, this version is required when writing\n+reachability bitmap files with `--write-bitmap-index`.\n++\n+The name hash version `2` has similar locality features as version `1`,\n+except it considers each path component separately and overlays the hashes\n+with a shift. This still prioritizes the final bytes of the path, but also\n+\"salts\" the lower bits of the hash using the parent directory names. This\n+method allows for some of the locality benefits of version `1` while\n+breaking most of the collisions from a similarly-named file appearing in\n+many different directories. At the moment, this version is not allowed\n+when writing reachability bitmap files with `--write-bitmap-index` and it\n+will be automatically changed to version `1`.\n+\n+\n CONFIGURATION\n -------------\n \ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 05e13adb87f..5e7ff919c1a 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -39,7 +39,9 @@ static int run_update_server_info = 1;\n static char *packdir, *packtmp_name, *packtmp;\n \n static const char *const git_repack_usage[] = {\n-\tN_(\"git repack [<options>]\"),\n+\tN_(\"git repack [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\\n\"\n+\t   \"[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\\n\"\n+\t   \"[--write-midx] [--name-hash-version=<n>]\"),\n \tNULL\n };\n \n@@ -58,7 +60,7 @@ struct pack_objects_args {\n \tint no_reuse_object;\n \tint quiet;\n \tint local;\n-\tint full_name_hash;\n+\tint name_hash_version;\n \tstruct list_objects_filter_options filter_options;\n };\n \n@@ -307,6 +309,8 @@ static void prepare_pack_objects(struct child_process *cmd,\n \t\tstrvec_pushf(&cmd->args, \"--no-reuse-delta\");\n \tif (args->no_reuse_object)\n \t\tstrvec_pushf(&cmd->args, \"--no-reuse-object\");\n+\tif (args->name_hash_version)\n+\t\tstrvec_pushf(&cmd->args, \"--name-hash-version=%d\", args->name_hash_version);\n \tif (args->local)\n \t\tstrvec_push(&cmd->args,  \"--local\");\n \tif (args->quiet)\n@@ -1204,6 +1208,8 @@ int cmd_repack(int argc,\n \t\t\t\tN_(\"pass --no-reuse-delta to git-pack-objects\")),\n \t\tOPT_BOOL('F', NULL, &po_args.no_reuse_object,\n \t\t\t\tN_(\"pass --no-reuse-object to git-pack-objects\")),\n+\t\tOPT_INTEGER(0, \"name-hash-version\", &po_args.name_hash_version,\n+\t\t\t\tN_(\"specify the name hash version to use for grouping similar objects by path\")),\n \t\tOPT_NEGBIT('n', NULL, &run_update_server_info,\n \t\t\t\tN_(\"do not run git-update-server-info\"), 1),\n \t\tOPT__QUIET(&po_args.quiet, N_(\"be quiet\")),\ndiff --git a/t/t0450/txt-help-mismatches b/t/t0450/txt-help-mismatches\nindex 28003f18c92..c4a15fd0cb8 100644\n--- a/t/t0450/txt-help-mismatches\n+++ b/t/t0450/txt-help-mismatches\n@@ -45,7 +45,6 @@ rebase\n remote\n remote-ext\n remote-fd\n-repack\n reset\n restore\n rev-parse\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex c4c3d1a15d9..b9a5759e01d 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -777,6 +777,12 @@ test_expect_success 'repack -ad cleans up old .tmp-* packs' '\n \ttest_must_be_empty tmpfiles\n '\n \n+test_expect_success '--name-hash-version option passes through to pack-objects' '\n+\tGIT_TRACE2_EVENT=\"$(pwd)/hash-trace.txt\" \\\n+\t\tgit repack -a --name-hash-version=2 &&\n+\ttest_subcommand_flex git pack-objects --name-hash-version=2 <hash-trace.txt\n+'\n+\n test_expect_success 'setup for update-server-info' '\n \tgit init update-server-info &&\n \ttest_commit -C update-server-info message\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 78e054ab503..af47247f25f 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1886,6 +1886,32 @@ test_subcommand () {\n \tfi\n }\n \n+# Check that the given subcommand was run with the given set of\n+# arguments in order (but with possible extra arguments).\n+#\n+#\ttest_subcommand_flex [!] <command> <args>... < <trace>\n+#\n+# If the first parameter passed is !, this instead checks that\n+# the given command was not called.\n+#\n+test_subcommand_flex () {\n+\tlocal negate=\n+\tif test \"$1\" = \"!\"\n+\tthen\n+\t\tnegate=t\n+\t\tshift\n+\tfi\n+\n+\tlocal expr=\"$(printf '\"%s\".*' \"$@\")\"\n+\n+\tif test -n \"$negate\"\n+\tthen\n+\t\t! grep \"\\[$expr\\]\"\n+\telse\n+\t\tgrep \"\\[$expr\\]\"\n+\tfi\n+}\n+\n # Check that the given command was invoked as part of the\n # trace2-format trace on stdin.\n #\n-- \ngitgitgadget\n\n"},{"id":"508478","messageId":"6a95708bf972cb22c8abf1da389350fc9f53c4ca.1733181682.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.v2.git.1733181682.gitgitgadget@gmail.com","subject":"[PATCH v2 4/8] pack-objects: add GIT_TEST_NAME_HASH_VERSION","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-02T23:21:18Z","receivedAt":"2024-12-02T23:21:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nAdd a new environment variable to opt-in to differen values of the\n--name-hash-version=<n> option in 'git pack-objects'. This allows for\nextra testing of the feature without repeating all of the test\nscenarios. Unlike many GIT_TEST_* variables, we are choosing to not add\nthis to the linux-TEST-vars CI build as that test run is already\noverloaded. The behavior exposed by this test variable is of low risk\nand should be sufficient to allow manual testing when an issue arises.\n\nBut this option isn't free. There are a few tests that change behavior\nwith the variable enabled.\n\nFirst, there are a few tests that are very sensitive to certain delta\nbases being picked. These are both involving the generation of thin\nbundles and then counting their objects via 'git index-pack --fix-thin'\nwhich pulls the delta base into the new packfile. For these tests,\ndisable the option as a decent long-term option.\n\nSecond, there are two tests in t5616-partial-clone.sh that I believe are\nactually broken scenarios. While the client is set up to clone the\n'promisor-server' repo via a treeless partial clone filter (tree:0),\nthat filter does not translate to the 'server' repo. Thus, fetching from\nthese repos causes the server to think that the client has all reachable\ntrees and blobs from the commits advertised as 'haves'. This leads the\nserver to providing a thin pack assuming those objects as delta bases.\nChanging the name-hash algorithm presents new delta bases and thus\nbreaks the expectations of these tests. An alternative could be to set\nup 'server' as a promisor server with the correct filter enabled. This\nmay also point out more issues with partial clone being set up as a\nremote-based filtering mechanism and not a repository-wide setting. For\nnow, do the minimal change to make the test work by disabling the test\nvariable.\n\nThird, there are some tests that compare the exact output of a 'git\npack-objects' process when using bitmaps. The warning that ignores the\n--name-hash-version=2 and forces version 1 causes these tests to fail.\nDisable the environment variable to get around this issue.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n builtin/pack-objects.c          |  5 ++++-\n t/README                        |  4 ++++\n t/t5300-pack-object.sh          |  7 +++++--\n t/t5310-pack-bitmaps.sh         |  5 ++++-\n t/t5333-pseudo-merge-bitmaps.sh |  4 ++++\n t/t5510-fetch.sh                |  7 ++++++-\n t/t5616-partial-clone.sh        | 26 ++++++++++++++++++++++++--\n t/t6020-bundle-misc.sh          |  6 +++++-\n t/t7406-submodule-update.sh     |  4 +++-\n t/t7700-repack.sh               | 10 ++++++++--\n 10 files changed, 67 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex e36a93a0082..e2f6431d614 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -266,7 +266,7 @@ struct configured_exclusion {\n static struct oidmap configured_exclusions;\n \n static struct oidset excluded_by_config;\n-static int name_hash_version = 1;\n+static int name_hash_version = -1;\n \n static void validate_name_hash_version(void)\n {\n@@ -4609,6 +4609,9 @@ int cmd_pack_objects(int argc,\n \tif (pack_to_stdout || !rev_list_all)\n \t\twrite_bitmap_index = 0;\n \n+\tif (name_hash_version < 0)\n+\t\tname_hash_version = (int)git_env_ulong(\"GIT_TEST_NAME_HASH_VERSION\", 1);\n+\n \tvalidate_name_hash_version();\n \tif (write_bitmap_index && name_hash_version != 1) {\n \t\twarning(_(\"currently, --write-bitmap-index requires --name-hash-version=1\"));\ndiff --git a/t/README b/t/README\nindex 8c0319b58e5..e63d2360852 100644\n--- a/t/README\n+++ b/t/README\n@@ -492,6 +492,10 @@ a test and then fails then the whole test run will abort. This can help to make\n sure the expected tests are executed and not silently skipped when their\n dependency breaks or is simply not present in a new environment.\n \n+GIT_TEST_NAME_HASH_VERSION=<int>, when set, causes 'git pack-objects' to\n+assume '--name-hash-version=<n>'.\n+\n+\n Naming Tests\n ------------\n \ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 4270eabe8b7..97fe9e561c6 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -675,15 +675,18 @@ do\n done\n \n test_expect_success 'valid and invalid --name-hash-versions' '\n+\tsane_unset GIT_TEST_NAME_HASH_VERSION &&\n+\n \t# Valid values are hard to verify other than \"do not fail\".\n \t# Performance tests will be more valuable to validate these versions.\n-\tfor value in 1 2\n+\t# Negative values are converted to version 1.\n+\tfor value in -1 1 2\n \tdo\n \t\tgit pack-objects base --all --name-hash-version=$value || return 1\n \tdone &&\n \n \t# Invalid values have clear post-conditions.\n-\tfor value in -1 0 3\n+\tfor value in 0 3\n \tdo\n \t\ttest_must_fail git pack-objects base --all --name-hash-version=$value 2>err &&\n \t\ttest_grep \"invalid --name-hash-version option\" err || return 1\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 7044c7d7c6d..c30522b57fd 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -420,7 +420,10 @@ test_bitmap_cases () {\n \t\t\tcat >expect <<-\\EOF &&\n \t\t\terror: missing value for '\\''pack.preferbitmaptips'\\''\n \t\t\tEOF\n-\t\t\tgit repack -adb 2>actual &&\n+\n+\t\t\t# Disable name hash version adjustment due to stderr comparison.\n+\t\t\tGIT_TEST_NAME_HASH_VERSION=1 \\\n+\t\t\t\tgit repack -adb 2>actual &&\n \t\t\ttest_cmp expect actual\n \t\t)\n \t'\ndiff --git a/t/t5333-pseudo-merge-bitmaps.sh b/t/t5333-pseudo-merge-bitmaps.sh\nindex eca4a1eb8c6..b1553cbaf7f 100755\n--- a/t/t5333-pseudo-merge-bitmaps.sh\n+++ b/t/t5333-pseudo-merge-bitmaps.sh\n@@ -209,6 +209,10 @@ test_expect_success 'bitmapPseudoMerge.stableThreshold creates stable groups' '\n '\n \n test_expect_success 'out of order thresholds are rejected' '\n+\t# Disable this option to avoid stderr message\n+\tGIT_TEST_NAME_HASH_VERSION=1 &&\n+\texport GIT_TEST_NAME_HASH_VERSION &&\n+\n \ttest_must_fail git \\\n \t\t-c bitmapPseudoMerge.test.pattern=\"refs/*\" \\\n \t\t-c bitmapPseudoMerge.test.threshold=1.month.ago \\\ndiff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\nindex 0890b9f61c5..1699c3a3bb8 100755\n--- a/t/t5510-fetch.sh\n+++ b/t/t5510-fetch.sh\n@@ -1062,7 +1062,12 @@ test_expect_success 'all boundary commits are excluded' '\n \ttest_tick &&\n \tgit merge otherside &&\n \tad=$(git log --no-walk --format=%ad HEAD) &&\n-\tgit bundle create twoside-boundary.bdl main --since=\"$ad\" &&\n+\n+\t# If the a different name hash function is used here, then no delta\n+\t# pair is found and the bundle does not expand to three objects\n+\t# when fixing the thin object.\n+\tGIT_TEST_NAME_HASH_VERSION=1 \\\n+\t\tgit bundle create twoside-boundary.bdl main --since=\"$ad\" &&\n \ttest_bundle_object_count --thin twoside-boundary.bdl 3\n '\n \ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex c53e93be2f7..9371d2606c6 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -516,7 +516,18 @@ test_expect_success 'fetch lazy-fetches only to resolve deltas' '\n \t# Exercise to make sure it works. Git will not fetch anything from the\n \t# promisor remote other than for the big tree (because it needs to\n \t# resolve the delta).\n-\tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -C client \\\n+\t#\n+\t# TODO: the --full-name-hash option is disabled here, since this test\n+\t# is fundamentally broken! When GIT_TEST_NAME_HASH_VERSION=2, the server\n+\t# recognizes delta bases in a different way and then sends a _blob_ to\n+\t# the client with a delta base that the client does not have! This is\n+\t# because the client is cloned from \"promisor-server\" with tree:0 but\n+\t# is now fetching from \"server\" without any filter. This is violating the\n+\t# promise to the server that all reachable objects exist and could be\n+\t# used as delta bases!\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace\" \\\n+\t\tGIT_TEST_NAME_HASH_VERSION=1 \\\n+\t\tgit -C client \\\n \t\tfetch \"file://$(pwd)/server\" main &&\n \n \t# Verify the assumption that the client needed to fetch the delta base\n@@ -535,7 +546,18 @@ test_expect_success 'fetch lazy-fetches only to resolve deltas, protocol v2' '\n \t# Exercise to make sure it works. Git will not fetch anything from the\n \t# promisor remote other than for the big blob (because it needs to\n \t# resolve the delta).\n-\tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -C client \\\n+\t#\n+\t# TODO: the --full-name-hash option is disabled here, since this test\n+\t# is fundamentally broken! When GIT_TEST_NAME_HASH_VERSION=2, the server\n+\t# recognizes delta bases in a different way and then sends a _blob_ to\n+\t# the client with a delta base that the client does not have! This is\n+\t# because the client is cloned from \"promisor-server\" with tree:0 but\n+\t# is now fetching from \"server\" without any filter. This is violating the\n+\t# promise to the server that all reachable objects exist and could be\n+\t# used as delta bases!\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace\" \\\n+\t\tGIT_TEST_NAME_HASH_VERSION=1 \\\n+\t\tgit -C client \\\n \t\tfetch \"file://$(pwd)/server\" main &&\n \n \t# Verify that protocol version 2 was used.\ndiff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\nindex 34b5cd62c20..a1f18ae71f1 100755\n--- a/t/t6020-bundle-misc.sh\n+++ b/t/t6020-bundle-misc.sh\n@@ -247,7 +247,11 @@ test_expect_success 'create bundle with --since option' '\n \tEOF\n \ttest_cmp expect actual &&\n \n-\tgit bundle create since.bdl \\\n+\t# If a different name hash function is used, then one fewer\n+\t# delta base is found and this counts a different number\n+\t# of objects after performing --fix-thin.\n+\tGIT_TEST_NAME_HASH_VERSION=1 \\\n+\t\tgit bundle create since.bdl \\\n \t\t--since \"Thu Apr 7 15:27:00 2005 -0700\" \\\n \t\t--all &&\n \ndiff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh\nindex 0f0c86f9cb2..ebd9941075a 100755\n--- a/t/t7406-submodule-update.sh\n+++ b/t/t7406-submodule-update.sh\n@@ -1094,7 +1094,9 @@ test_expect_success 'submodule update --quiet passes quietness to fetch with a s\n \t) &&\n \tgit clone super4 super5 &&\n \t(cd super5 &&\n-\t git submodule update --quiet --init --depth=1 submodule3 >out 2>err &&\n+\t # This test var can mess with the stderr output checked in this test.\n+\t GIT_TEST_NAME_HASH_VERSION=1 \\\n+\t\tgit submodule update --quiet --init --depth=1 submodule3 >out 2>err &&\n \t test_must_be_empty out &&\n \t test_must_be_empty err\n \t) &&\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex b9a5759e01d..16861f80c9c 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -309,7 +309,10 @@ test_expect_success 'no bitmaps created if .keep files present' '\n \tkeep=${pack%.pack}.keep &&\n \ttest_when_finished \"rm -f \\\"\\$keep\\\"\" &&\n \t>\"$keep\" &&\n-\tgit -C bare.git repack -ad 2>stderr &&\n+\n+\t# Disable --name-hash-version test due to stderr comparison.\n+\tGIT_TEST_NAME_HASH_VERSION=1 \\\n+\t\tgit -C bare.git repack -ad 2>stderr &&\n \ttest_must_be_empty stderr &&\n \tfind bare.git/objects/pack/ -type f -name \"*.bitmap\" >actual &&\n \ttest_must_be_empty actual\n@@ -320,7 +323,10 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '\n \tblob=$(test-tool genrandom big $((1024*1024)) |\n \t       git -C bare.git hash-object -w --stdin) &&\n \tgit -C bare.git update-ref refs/tags/big $blob &&\n-\tgit -C bare.git repack -ad 2>stderr &&\n+\n+\t# Disable --name-hash-version test due to stderr comparison.\n+\tGIT_TEST_NAME_HASH_VERSION=1 \\\n+\t\tgit -C bare.git repack -ad 2>stderr &&\n \ttest_must_be_empty stderr &&\n \tfind bare.git/objects/pack -type f -name \"*.bitmap\" >actual &&\n \ttest_must_be_empty actual\n-- \ngitgitgadget\n\n"},{"id":"508479","messageId":"3b5697467c997b5d2080fb02debd911d716e8383.1733181682.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.v2.git.1733181682.gitgitgadget@gmail.com","subject":"[PATCH v2 5/8] p5313: add size comparison test","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-02T23:21:19Z","receivedAt":"2024-12-02T23:21:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nAs custom options are added to 'git pack-objects' and 'git repack' to\nadjust how compression is done, use this new performance test script to\ndemonstrate their effectiveness in performance and size.\n\nThe recently-added --name-hash-version option allows for testing\ndifferent name hash functions. Version 2 intends to preserve some of the\nlocality of version 1 while more often breaking collisions due to long\nfilenames.\n\nDistinguishing objects by more of the path is critical when there are\nmany name hash collisions and several versions of the same path in the\nfull history, giving a significant boost to the full repack case. The\nlocality of the hash function is critical to compressing something like\na shallow clone or a thin pack representing a push of a single commit.\n\nThis can be seen by running pt5313 on the open source fluentui\nrepository [1]. Most commits will have this kind of output for the thin\nand big pack cases, though certain commits (such as [2]) will have\nproblematic thin pack size for other reasons.\n\n[1] https://github.com/microsoft/fluentui\n[2] a637a06df05360ce5ff21420803f64608226a875\n\nChecked out at the parent of [2], I see the following statistics:\n\nTest                                         HEAD\n---------------------------------------------------------------\n5313.2: thin pack with version 1             0.37(0.44+0.02)\n5313.3: thin pack size with version 1                   1.2M\n5313.4: big pack with version 1              2.04(7.77+0.23)\n5313.5: big pack size with version 1                   20.4M\n5313.6: shallow fetch pack with version 1    1.41(2.94+0.11)\n5313.7: shallow pack size with version 1               34.4M\n5313.8: repack with version 1                95.70(676.41+2.87)\n5313.9: repack size with version 1                    439.3M\n5313.10: thin pack with version 2            0.12(0.12+0.06)\n5313.11: thin pack size with version 2                 22.0K\n5313.12: big pack with version 2             2.80(5.43+0.34)\n5313.13: big pack size with version 2                  25.9M\n5313.14: shallow fetch pack with version 2   1.77(2.80+0.19)\n5313.15: shallow pack size with version 2              33.7M\n5313.16: repack with version 2               33.68(139.52+2.58)\n5313.17: repack size with version 2                   160.5M\n\nTo make comparisons easier, I will reformat this output into a different\ntable style:\n\n| Test         | V1 Time | V2 Time | V1 Size | V2 Size |\n|--------------|---------|---------|---------|---------|\n| Thin Pack    |  0.37 s |  0.12 s |   1.2 M |  22.0 K |\n| Big Pack     |  2.04 s |  2.80 s |  20.4 M |  25.9 M |\n| Shallow Pack |  1.41 s |  1.77 s |  34.4 M |  33.7 M |\n| Repack       | 95.70 s | 33.68 s | 439.3 M | 160.5 M |\n\nThe v2 hash function successfully differentiates the CHANGELOG.md files\nfrom each other, which leads to significant improvements in the thin\npack (simulating a push of this commit) and the full repack. There is\nsome bloat in the \"big pack\" scenario and essentially the same results\nfor the shallow pack.\n\nIn the case of the Git repository, these numbers show some of the issues\nwith this approach:\n\n| Test         | V1 Time | V2 Time | V1 Size | V2 Size |\n|--------------|---------|---------|---------|---------|\n| Thin Pack    |  0.02 s |  0.02 s |   1.1 K |   1.1 K |\n| Big Pack     |  1.69 s |  1.95 s |  13.5 M |  14.5 M |\n| Shallow Pack |  1.26 s |  1.29 s |  12.0 M |  12.2 M |\n| Repack       | 29.51 s | 29.01 s | 237.7 M | 238.2 M |\n\nHere, the attempts to remove conflicts in the v2 function seem to cause\nslight bloat to these sizes. This shows that the Git repository benefits\na lot from cross-path delta pairs.\n\nThe results are similar with the nodejs/node repo:\n\n| Test         | V1 Time | V2 Time | V1 Size | V2 Size |\n|--------------|---------|---------|---------|---------|\n| Thin Pack    |  0.02 s |  0.02 s |   1.6 K |   1.6 K |\n| Big Pack     |  4.61 s |  3.26 s |  56.0 M |  52.8 M |\n| Shallow Pack |  7.82 s |  7.51 s | 104.6 M | 107.0 M |\n| Repack       | 88.90 s | 73.75 s | 740.1 M | 764.5 M |\n\nHere, the v2 name-hash causes some size bloat more often than it reduces\nthe size, but it also universally improves performance time, which is an\ninteresting reversal. This must mean that it is helping to short-circuit\nsome delta computations even if it is not finding the most efficient\nones. The performance improvement cannot be explained only due to the\nI/O cost of writing the resulting packfile.\n\nThe Linux kernel repository was the initial target of the default name\nhash value, and its naming conventions are practically build to take the\nmost advantage of the default name hash values:\n\n| Test         | V1 Time  | V2 Time  | V1 Size | V2 Size |\n|--------------|----------|----------|---------|---------|\n| Thin Pack    |   0.17 s |   0.07 s |   4.6 K |   4.6 K |\n| Big Pack     |  17.88 s |  12.35 s | 201.1 M | 159.1 M |\n| Shallow Pack |  11.05 s |  22.94 s | 269.2 M | 273.8 M |\n| Repack       | 727.39 s | 566.95 s |   2.5 G |   2.5 G |\n\nHere, the thin and big packs gain some performance boosts in time, with\na modest gain in the size of the big pack. The shallow pack, however, is\nmore expensive to compute, likely because similarly-named files across\ndifferent directories are farther apart in the name hash ordering in v2.\nThe repack also gains benefits in computation time but no meaningful\nchange to the full size.\n\nFinally, an internal Javascript repo of moderate size shows significant\ngains when repacking with --name-hash-version=2 due to it having many name\nhash collisions. However, it's worth noting that only the full repack\ncase has significant differences from the v1 name hash:\n\n| Test      | V1 Time   | V2 Time  | V1 Size | V2 Size |\n|-----------|-----------|----------|---------|---------|\n| Thin Pack |    8.28 s |   7.28 s |  16.8 K |  16.8 K |\n| Big Pack  |   12.81 s |  11.66 s |  29.1 M |  29.1 M |\n| Shallow   |    4.86 s |   4.06 s |  42.5 M |  44.1 M |\n| Repack    | 3126.50 s | 496.33 s |   6.2 G | 855.6 M |\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n t/perf/p5313-pack-objects.sh | 70 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 70 insertions(+)\n create mode 100755 t/perf/p5313-pack-objects.sh\n\ndiff --git a/t/perf/p5313-pack-objects.sh b/t/perf/p5313-pack-objects.sh\nnew file mode 100755\nindex 00000000000..be5229a0ecd\n--- /dev/null\n+++ b/t/perf/p5313-pack-objects.sh\n@@ -0,0 +1,70 @@\n+#!/bin/sh\n+\n+test_description='Tests pack performance using bitmaps'\n+. ./perf-lib.sh\n+\n+GIT_TEST_PASSING_SANITIZE_LEAK=0\n+export GIT_TEST_PASSING_SANITIZE_LEAK\n+\n+test_perf_large_repo\n+\n+test_expect_success 'create rev input' '\n+\tcat >in-thin <<-EOF &&\n+\t$(git rev-parse HEAD)\n+\t^$(git rev-parse HEAD~1)\n+\tEOF\n+\n+\tcat >in-big <<-EOF &&\n+\t$(git rev-parse HEAD)\n+\t^$(git rev-parse HEAD~1000)\n+\tEOF\n+\n+\tcat >in-shallow <<-EOF\n+\t$(git rev-parse HEAD)\n+\t--shallow $(git rev-parse HEAD)\n+\tEOF\n+'\n+\n+for version in 1 2\n+do\n+\texport version\n+\n+\ttest_perf \"thin pack with version $version\" '\n+\t\tgit pack-objects --thin --stdout --revs --sparse \\\n+\t\t\t--name-hash-version=$version <in-thin >out\n+\t'\n+\n+\ttest_size \"thin pack size with version $version\" '\n+\t\ttest_file_size out\n+\t'\n+\n+\ttest_perf \"big pack with version $version\" '\n+\t\tgit pack-objects --stdout --revs --sparse \\\n+\t\t\t--name-hash-version=$version <in-big >out\n+\t'\n+\n+\ttest_size \"big pack size with version $version\" '\n+\t\ttest_file_size out\n+\t'\n+\n+\ttest_perf \"shallow fetch pack with version $version\" '\n+\t\tgit pack-objects --stdout --revs --sparse --shallow \\\n+\t\t\t--name-hash-version=$version <in-shallow >out\n+\t'\n+\n+\ttest_size \"shallow pack size with version $version\" '\n+\t\ttest_file_size out\n+\t'\n+\n+\ttest_perf \"repack with version $version\" '\n+\t\tgit repack -adf --name-hash-version=$version\n+\t'\n+\n+\ttest_size \"repack size with version $version\" '\n+\t\tgitdir=$(git rev-parse --git-dir) &&\n+\t\tpack=$(ls $gitdir/objects/pack/pack-*.pack) &&\n+\t\ttest_file_size \"$pack\"\n+\t'\n+done\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"508480","messageId":"36f2811e3d917405c16433491353fc9adb2bb1f9.1733181682.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.v2.git.1733181682.gitgitgadget@gmail.com","subject":"[PATCH v2 6/8] test-tool: add helper for name-hash values","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-02T23:21:20Z","receivedAt":"2024-12-02T23:21:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nAdd a new test-tool helper, name-hash, to output the value of the\nname-hash algorithms for the input list of strings, one per line.\n\nSince the name-hash values can be stored in the .bitmap files, it is\nimportant that these hash functions do not change across Git versions.\nAdd a simple test to t5310-pack-bitmaps.sh to provide some testing of\nthe current values. Due to how these functions are implemented, it would\nbe difficult to change them without disturbing these values. The paths\nused for this test are carefully selected to demonstrate some of the\nbehavior differences of the two current name hash versions, including\nwhich conditions will cause them to collide.\n\nCreate a performance test that uses test_size to demonstrate how\ncollisions occur for these hash algorithms. This test helps inform\nsomeone as to the behavior of the name-hash algorithms for their repo\nbased on the paths at HEAD.\n\nMy copy of the Git repository shows modest statistics around the\ncollisions of the default name-hash algorithm:\n\nTest                               this tree\n--------------------------------------------------\n5314.1: paths at head                         4.5K\n5314.2: distinct hash value: v1               4.1K\n5314.3: maximum multiplicity: v1                13\n5314.4: distinct hash value: v2               4.2K\n5314.5: maximum multiplicity: v2                 9\n\nHere, the maximum collision multiplicity is 13, but around 10% of paths\nhave a collision with another path.\n\nIn a more interesting example, the microsoft/fluentui [1] repo had these\nstatistics at time of committing:\n\nTest                               this tree\n--------------------------------------------------\n5314.1: paths at head                        19.5K\n5314.2: distinct hash value: v1               8.2K\n5314.3: maximum multiplicity: v1               279\n5314.4: distinct hash value: v2              17.8K\n5314.5: maximum multiplicity: v2                44\n\n[1] https://github.com/microsoft/fluentui\n\nThat demonstrates that of the nearly twenty thousand path names, they\nare assigned around eight thousand distinct values. 279 paths are\nassigned to a single value, leading the packing algorithm to sort\nobjects from those paths together, by size.\n\nWith the v2 name hash function, the maximum multiplicity lowers to 44,\nleaving some room for further improvement.\n\nIn a more extreme example, an internal monorepo had a much worse\ncollision rate:\n\nTest                               this tree\n--------------------------------------------------\n5314.1: paths at head                       227.3K\n5314.2: distinct hash value: v1              72.3K\n5314.3: maximum multiplicity: v1             14.4K\n5314.4: distinct hash value: v2             166.5K\n5314.5: maximum multiplicity: v2               138\n\nHere, we can see that the v2 name hash function provides somem\nimprovements, but there are still a number of collisions that could lead\nto repacking problems at this scale.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Makefile                  |  1 +\n t/helper/test-name-hash.c | 23 +++++++++++++++++++++++\n t/helper/test-tool.c      |  1 +\n t/helper/test-tool.h      |  1 +\n t/perf/p5314-name-hash.sh | 31 +++++++++++++++++++++++++++++++\n t/t5310-pack-bitmaps.sh   | 30 ++++++++++++++++++++++++++++++\n 6 files changed, 87 insertions(+)\n create mode 100644 t/helper/test-name-hash.c\n create mode 100755 t/perf/p5314-name-hash.sh\n\ndiff --git a/Makefile b/Makefile\nindex 6f5986b66ea..65403f6dd09 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -816,6 +816,7 @@ TEST_BUILTINS_OBJS += test-lazy-init-name-hash.o\n TEST_BUILTINS_OBJS += test-match-trees.o\n TEST_BUILTINS_OBJS += test-mergesort.o\n TEST_BUILTINS_OBJS += test-mktemp.o\n+TEST_BUILTINS_OBJS += test-name-hash.o\n TEST_BUILTINS_OBJS += test-online-cpus.o\n TEST_BUILTINS_OBJS += test-pack-mtimes.o\n TEST_BUILTINS_OBJS += test-parse-options.o\ndiff --git a/t/helper/test-name-hash.c b/t/helper/test-name-hash.c\nnew file mode 100644\nindex 00000000000..5b402362020\n--- /dev/null\n+++ b/t/helper/test-name-hash.c\n@@ -0,0 +1,23 @@\n+/*\n+ * test-name-hash.c: Read a list of paths over stdin and report on their\n+ * name-hash and full name-hash.\n+ */\n+\n+#include \"test-tool.h\"\n+#include \"git-compat-util.h\"\n+#include \"pack-objects.h\"\n+#include \"strbuf.h\"\n+\n+int cmd__name_hash(int argc UNUSED, const char **argv UNUSED)\n+{\n+\tstruct strbuf line = STRBUF_INIT;\n+\n+\twhile (!strbuf_getline(&line, stdin)) {\n+\t\tprintf(\"%10u \", pack_name_hash(line.buf));\n+\t\tprintf(\"%10u \", pack_name_hash_v2(line.buf));\n+\t\tprintf(\"%s\\n\", line.buf);\n+\t}\n+\n+\tstrbuf_release(&line);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 1ebb69a5dc4..e794058ab6d 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -44,6 +44,7 @@ static struct test_cmd cmds[] = {\n \t{ \"match-trees\", cmd__match_trees },\n \t{ \"mergesort\", cmd__mergesort },\n \t{ \"mktemp\", cmd__mktemp },\n+\t{ \"name-hash\", cmd__name_hash },\n \t{ \"online-cpus\", cmd__online_cpus },\n \t{ \"pack-mtimes\", cmd__pack_mtimes },\n \t{ \"parse-options\", cmd__parse_options },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 21802ac27da..26ff30a5a9a 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -37,6 +37,7 @@ int cmd__lazy_init_name_hash(int argc, const char **argv);\n int cmd__match_trees(int argc, const char **argv);\n int cmd__mergesort(int argc, const char **argv);\n int cmd__mktemp(int argc, const char **argv);\n+int cmd__name_hash(int argc, const char **argv);\n int cmd__online_cpus(int argc, const char **argv);\n int cmd__pack_mtimes(int argc, const char **argv);\n int cmd__parse_options(int argc, const char **argv);\ndiff --git a/t/perf/p5314-name-hash.sh b/t/perf/p5314-name-hash.sh\nnew file mode 100755\nindex 00000000000..4ef0ba77114\n--- /dev/null\n+++ b/t/perf/p5314-name-hash.sh\n@@ -0,0 +1,31 @@\n+#!/bin/sh\n+\n+test_description='Tests pack performance using bitmaps'\n+. ./perf-lib.sh\n+\n+GIT_TEST_PASSING_SANITIZE_LEAK=0\n+export GIT_TEST_PASSING_SANITIZE_LEAK\n+\n+test_perf_large_repo\n+\n+test_size 'paths at head' '\n+\tgit ls-tree -r --name-only HEAD >path-list &&\n+\twc -l <path-list &&\n+\ttest-tool name-hash <path-list >name-hashes\n+'\n+\n+for version in 1 2\n+do\n+\ttest_size \"distinct hash value: v$version\" '\n+\t\tawk \"{ print \\$$version; }\" <name-hashes | sort | \\\n+\t\t\tuniq -c >name-hash-count &&\n+\t\twc -l <name-hash-count\n+\t'\n+\n+\ttest_size \"maximum multiplicity: v$version\" '\n+\t\tsort -nr <name-hash-count | head -n 1 |\t\\\n+\t\t\tawk \"{ print \\$1; }\"\n+\t'\n+done\n+\n+test_done\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex c30522b57fd..871ce01401a 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -27,6 +27,36 @@ has_any () {\n \tgrep -Ff \"$1\" \"$2\"\n }\n \n+# Since name-hash values are stored in the .bitmap files, add a test\n+# that checks that the name-hash calculations are stable across versions.\n+# Not exhaustive, but these hashing algorithms would be hard to change\n+# without causing deviations here.\n+test_expect_success 'name-hash value stability' '\n+\tcat >names <<-\\EOF &&\n+\tfirst\n+\tsecond\n+\tthird\n+\ta/one-long-enough-for-collisions\n+\tb/two-long-enough-for-collisions\n+\tmany/parts/to/this/path/enough/to/collide/in/v2\n+\tenough/parts/to/this/path/enough/to/collide/in/v2\n+\tEOF\n+\n+\ttest-tool name-hash <names >out &&\n+\n+\tcat >expect <<-\\EOF &&\n+\t2582249472 1763573760 first\n+\t2289942528 1188134912 second\n+\t2300837888 1130758144 third\n+\t2544516325 3963087891 a/one-long-enough-for-collisions\n+\t2544516325 4013419539 b/two-long-enough-for-collisions\n+\t1420111091 1709547268 many/parts/to/this/path/enough/to/collide/in/v2\n+\t1420111091 1709547268 enough/parts/to/this/path/enough/to/collide/in/v2\n+\tEOF\n+\n+\ttest_cmp expect out\n+'\n+\n test_bitmap_cases () {\n \twriteLookupTable=false\n \tfor i in \"$@\"\n-- \ngitgitgadget\n\n"},{"id":"508481","messageId":"3885ef8a2f76968d67b03503455cb4f0761dd89c.1733181682.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.v2.git.1733181682.gitgitgadget@gmail.com","subject":"[PATCH v2 7/8] pack-objects: prevent name hash version change","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-02T23:21:21Z","receivedAt":"2024-12-02T23:21:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nWhen the --name-hash-version option is used in 'git pack-objects', it\ncan change from the initial assignment to when it is used based on\ninteractions with other arguments. Specifically, when writing or reading\nbitmaps, we must force version 1 for now. This could change in the\nfuture when the bitmap format can store a name hash version value,\nindicating which was used during the writing of the packfile.\n\nProtect the 'git pack-objects' process from getting confused by failing\nwith a BUG() statement if the value of the name hash version changes\nbetween calls to pack_name_hash_fn().\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n builtin/pack-objects.c | 8 ++++++++\n 1 file changed, 8 insertions(+)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex e2f6431d614..7f1cb8de2fe 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -276,6 +276,14 @@ static void validate_name_hash_version(void)\n \n static inline uint32_t pack_name_hash_fn(const char *name)\n {\n+\tstatic int seen_version = -1;\n+\n+\tif (seen_version < 0)\n+\t\tseen_version = name_hash_version;\n+\telse if (seen_version != name_hash_version)\n+\t\tBUG(\"name hash version changed from %d to %d mid-process\",\n+\t\t    seen_version, name_hash_version);\n+\n \tswitch (name_hash_version)\n \t{\n \tcase 1:\n-- \ngitgitgadget\n\n"},{"id":"508482","messageId":"64fd7b3ccadabad7a2eb4515e99cae3ec3d9b2b1.1733181682.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.v2.git.1733181682.gitgitgadget@gmail.com","subject":"[PATCH v2 8/8] pack-objects: add third name hash version","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-02T23:21:22Z","receivedAt":"2024-12-02T23:21:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nThe '--name-hash-version=<n>' option in 'git pack-objects' was\nintroduced to allow for specifying an alternative name hash function\nwhen organizing objects for delta compression. The pack_name_hash_v2()\nfunction was designed to break some collisions while also preserving\nsome amount of locality for cross-path deltas.\n\nHowever, in some repositories, that effort to preserve locality results\nin enough collisions that it causes issues with full repacks.\n\nCreate a third name hash function and extend the '--name-hash-version'\noption in 'git pack-objects' and 'git repack' to understand it. This\nhash version abandons all efforts for locality and focuses on creating a\nsomewhat uniformly-distributed hash function to minimize collisions.\n\nWe can observe the effect of this collision avoidance in a large\ninternal monorepo that suffered from collisions in the previous\nversions. The updates to p5314-name-hash.sh show these results:\n\nTest                               this tree\n--------------------------------------------------\n5314.1: paths at head                       227.3K\n5314.2: distinct hash value: v1              72.3K\n5314.3: maximum multiplicity: v1             14.4K\n5314.4: distinct hash value: v2             166.5K\n5314.5: maximum multiplicity: v2               138\n5314.6: distinct hash value: v3             227.3K\n5314.7: maximum multiplicity: v3                 2\n\nThese results demonstrate that of the 227,000+ paths, nearly all of them\nfind distinct hash values. The maximum multiplicity is 2, improved from\n138 in the v2 hash function. The v2 hash function also had only 166K\ndistinct values, so it had a wide spread of collisions.\n\nA more modest improvement is available in the open source fluentui repo\n[1] with these results:\n\nTest                               this tree\n--------------------------------------------------\n5314.1: paths at head                        19.5K\n5314.2: distinct hash value: v1               8.2K\n5314.3: maximum multiplicity: v1               279\n5314.4: distinct hash value: v2              17.8K\n5314.5: maximum multiplicity: v2                44\n5314.6: distinct hash value: v3              19.5K\n5314.7: maximum multiplicity: v3                 1\n\n[1] https://github.com/microsoft/fluentui\n\nHowever, it is important to demonstrate the effectiveness of this\nfunction in the context of compressing a repository. We can use\np5313-pack-objects.sh to measure these changes. I will use a simplified\ntable summarizing the output of that performance test.\n\n | Test      | V1 Time | V2 Time | V3 Time | V1 Size | V2 Size | V3 Size |\n |-----------|---------|---------|---------|---------|---------|---------|\n | Thin Pack |  0.37 s |  0.12 s |  0.07 s |   1.2 M |  22.0 K |  20.4 K |\n | Big Pack  |  2.04 s |  2.80 s |  1.40 s |  20.4 M |  25.9 M |  19.2 M |\n | Shallow   |  1.41 s |  1.77 s |  1.27 s |  34.4 M |  33.7 M |  34.8 M |\n | Repack    | 95.70 s | 33.68 s | 20.88 s | 439.3 M | 160.5 M | 169.1 M |\n\nHere, there are some performance improvements on a time basis, and the\nthin and big packs are somewhat smaller in v3. The shallow and repacked\npacks are somewhat bigger, though, compared to v2.\n\nTwo repositories that have very few collisions in the v1 name hash are\nthe Git and Linux repositories. Here are their stats for p5313:\n\nGit:\n\n | Test      | V1 Time | V2 Time | V3 Time | V1 Size | V2 Size | V3 Size |\n |-----------|---------|---------|---------|---------|---------|---------|\n | Thin Pack |  0.02 s |  0.02 s |  0.02 s |   1.1 K |   1.1 K |  15.3 K |\n | Big Pack  |  1.69 s |  1.95 s |  1.67 s |  13.5 M |  14.5 M |  14.9 M |\n | Shallow   |  1.26 s |  1.29 s |  1.16 s |  12.0 M |  12.2 M |  12.5 M |\n | Repack    | 29.51 s | 29.01 s | 29.08 s | 237.7 M | 238.2 M | 237.7 M |\n\nLinux:\n\n | Test      | V1 Time  | V2 Time  | V3 Time  | V1 Size | V2 Size | V3 Size |\n |-----------|----------|----------|----------|---------|---------|---------|\n | Thin Pack |   0.17 s |   0.07 s |   0.07 s |   4.6 K |   4.6 K |   6.8 K |\n | Big Pack  |  17.88 s |  12.35 s |  12.14 s | 201.1 M | 149.1 M | 160.4 M |\n | Shallow   |  11.05 s |  22.94 s |  22.16 s | 269.2 M | 273.8 M | 271.8 M |\n | Repack    | 727.39 s | 566.95 s | 539.33 s |   2.5 G |   2.5 G |   2.6 G |\n\nThese repositories make good use of the cross-path deltas that come\nabout from the v1 name hash function, so they already had mixed results\nwith the v2 function. The v3 function is generally worse for these\nrepositories.\n\nAn internal Javascript-based repository with name hash collisions\nsimilar to the fluentui repo has these results:\n\n | Test      | V1 Time   | V2 Time  | V3 Time  | V1 Size | V2 Size | V3 Size |\n |-----------|-----------|----------|----------|---------|---------|---------|\n | Thin Pack |    8.28 s |   7.28 s |   0.04 s |  16.8 K |  16.8 K |   3.2 K |\n | Big Pack  |   12.81 s |  11.66 s |   2.52 s |  29.1 M |  29.1 M |  30.6 M |\n | Shallow   |    4.86 s |   4.06 s |   3.77 s |  42.5 M |  44.1 M |  45.7 M |\n | Repack    | 3126.50 s | 496.33 s | 306.86 s |   6.2 G | 855.6 M | 838.2 M |\n\nThis repository is also listed as \"Repo B\" in the repacking size table\nbelow, along with other Javascript repos that have many name hash\ncollisions with the v1 name hash:\n\n | Repo     | V1 Size   | V2 Size | V3 Size |\n |----------|-----------|---------|---------|\n | fluentui |     440 M |   161 M |   170 M |\n | Repo B   |   6,248 M |   856 M |   840 M |\n | Repo C   |  37,278 M | 6,921 M | 6,755 M |\n | Repo D   | 131,204 M | 7,463 M | 7,124 M |\n\nWhile the fluentui repo had an increase in size using the v3 name hash,\nthe others had modest improvements over the v2 name hash. But those\nmodest improvements are dwarfed by the difference from v1 to v2, so it\nis unlikely that the regression seen in the other scenarios (packfiles\nthat are not from full repacks) will be worth using v3 over v2. That is,\nunless there are enough collisions even with v2 that the full repack\nscenario has larger improvements than these.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Documentation/git-pack-objects.txt |  9 +++++++++\n Documentation/git-repack.txt       |  9 +++++++++\n builtin/pack-objects.c             |  5 ++++-\n pack-objects.h                     | 26 ++++++++++++++++++++++++++\n t/helper/test-name-hash.c          |  1 +\n t/perf/p5313-pack-objects.sh       |  2 +-\n t/perf/p5314-name-hash.sh          |  2 +-\n t/t5300-pack-object.sh             |  4 ++--\n t/t5310-pack-bitmaps.sh            | 14 +++++++-------\n 9 files changed, 60 insertions(+), 12 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 7f69ae4855f..9fe25c53415 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -374,6 +374,15 @@ breaking most of the collisions from a similarly-named file appearing in\n many different directories. At the moment, this version is not allowed\n when writing reachability bitmap files with `--write-bitmap-index` and it\n will be automatically changed to version `1`.\n++\n+The name hash version `3` abandons the locality features of versions `1`\n+and `2` in favor of minimizing collisions. The goal here is to separate\n+objects by their full path and abandon hope for cross-path delta\n+compression. For this reason, this option is preferred for repacking large\n+repositories with many versions and many name hash collisions when using\n+the first two versions. At the moment, this version is not allowed when\n+writing reachability bitmap files with `--write-bitmap-index` and it will\n+be automatically changed to version `1`.\n \n \n DELTA ISLANDS\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex ea69fbe1891..bc74d9333f0 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -279,6 +279,15 @@ breaking most of the collisions from a similarly-named file appearing in\n many different directories. At the moment, this version is not allowed\n when writing reachability bitmap files with `--write-bitmap-index` and it\n will be automatically changed to version `1`.\n++\n+The name hash version `3` abandons the locality features of versions `1`\n+and `2` in favor of minimizing collisions. The goal here is to separate\n+objects by their full path and abandon hope for cross-path delta\n+compression. For this reason, this option is preferred for repacking large\n+repositories with many versions and many name hash collisions when using\n+the first two versions. At the moment, this version is not allowed when\n+writing reachability bitmap files with `--write-bitmap-index` and it will\n+be automatically changed to version `1`.\n \n \n CONFIGURATION\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7f1cb8de2fe..66efd43b036 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -270,7 +270,7 @@ static int name_hash_version = -1;\n \n static void validate_name_hash_version(void)\n {\n-\tif (name_hash_version < 1 || name_hash_version > 2)\n+\tif (name_hash_version < 1 || name_hash_version > 3)\n \t\tdie(_(\"invalid --name-hash-version option: %d\"), name_hash_version);\n }\n \n@@ -292,6 +292,9 @@ static inline uint32_t pack_name_hash_fn(const char *name)\n \tcase 2:\n \t\treturn pack_name_hash_v2(name);\n \n+\tcase 3:\n+\t\treturn pack_name_hash_v3(name);\n+\n \tdefault:\n \t\tBUG(\"invalid name-hash version: %d\", name_hash_version);\n \t}\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 15be8368d21..8c0840983e1 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -235,6 +235,32 @@ static inline uint32_t pack_name_hash_v2(const char *name)\n \treturn (base >> 6) ^ hash;\n }\n \n+static inline uint32_t pack_name_hash_v3(const char *name)\n+{\n+\t/*\n+\t * This 'bigp' value is a large prime, at least 25% of the max\n+\t * value of an uint32_t. Multiplying by this value (modulo 2^32)\n+\t * causes the 32 bits to change pseudo-randomly.\n+\t */\n+\tconst uint32_t bigp = 1234572167U;\n+\tuint32_t c, hash = bigp;\n+\n+\tif (!name)\n+\t\treturn 0;\n+\n+\t/*\n+\t * Do the simplest thing that will resemble pseudo-randomness: add\n+\t * random multiples of a large prime number with a binary shift.\n+\t * The goal is not to be cryptographic, but to be generally\n+\t * uniformly distributed.\n+\t */\n+\twhile ((c = *name++) != 0) {\n+\t\thash += c * bigp;\n+\t\thash = (hash >> 5) | (hash << 27);\n+\t}\n+\treturn hash;\n+}\n+\n static inline enum object_type oe_type(const struct object_entry *e)\n {\n \treturn e->type_valid ? e->type_ : OBJ_BAD;\ndiff --git a/t/helper/test-name-hash.c b/t/helper/test-name-hash.c\nindex 5b402362020..1f71a41e2a8 100644\n--- a/t/helper/test-name-hash.c\n+++ b/t/helper/test-name-hash.c\n@@ -15,6 +15,7 @@ int cmd__name_hash(int argc UNUSED, const char **argv UNUSED)\n \twhile (!strbuf_getline(&line, stdin)) {\n \t\tprintf(\"%10u \", pack_name_hash(line.buf));\n \t\tprintf(\"%10u \", pack_name_hash_v2(line.buf));\n+\t\tprintf(\"%10u \", pack_name_hash_v3(line.buf));\n \t\tprintf(\"%s\\n\", line.buf);\n \t}\n \ndiff --git a/t/perf/p5313-pack-objects.sh b/t/perf/p5313-pack-objects.sh\nindex be5229a0ecd..493872e656d 100755\n--- a/t/perf/p5313-pack-objects.sh\n+++ b/t/perf/p5313-pack-objects.sh\n@@ -25,7 +25,7 @@ test_expect_success 'create rev input' '\n \tEOF\n '\n \n-for version in 1 2\n+for version in 1 2 3\n do\n \texport version\n \ndiff --git a/t/perf/p5314-name-hash.sh b/t/perf/p5314-name-hash.sh\nindex 4ef0ba77114..e58a218d1ae 100755\n--- a/t/perf/p5314-name-hash.sh\n+++ b/t/perf/p5314-name-hash.sh\n@@ -14,7 +14,7 @@ test_size 'paths at head' '\n \ttest-tool name-hash <path-list >name-hashes\n '\n \n-for version in 1 2\n+for version in 1 2 3\n do\n \ttest_size \"distinct hash value: v$version\" '\n \t\tawk \"{ print \\$$version; }\" <name-hashes | sort | \\\ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 97fe9e561c6..279a9deca9f 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -680,13 +680,13 @@ test_expect_success 'valid and invalid --name-hash-versions' '\n \t# Valid values are hard to verify other than \"do not fail\".\n \t# Performance tests will be more valuable to validate these versions.\n \t# Negative values are converted to version 1.\n-\tfor value in -1 1 2\n+\tfor value in -1 1 2 3\n \tdo\n \t\tgit pack-objects base --all --name-hash-version=$value || return 1\n \tdone &&\n \n \t# Invalid values have clear post-conditions.\n-\tfor value in 0 3\n+\tfor value in 0 4\n \tdo\n \t\ttest_must_fail git pack-objects base --all --name-hash-version=$value 2>err &&\n \t\ttest_grep \"invalid --name-hash-version option\" err || return 1\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 871ce01401a..2bf75e2a5d0 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -45,13 +45,13 @@ test_expect_success 'name-hash value stability' '\n \ttest-tool name-hash <names >out &&\n \n \tcat >expect <<-\\EOF &&\n-\t2582249472 1763573760 first\n-\t2289942528 1188134912 second\n-\t2300837888 1130758144 third\n-\t2544516325 3963087891 a/one-long-enough-for-collisions\n-\t2544516325 4013419539 b/two-long-enough-for-collisions\n-\t1420111091 1709547268 many/parts/to/this/path/enough/to/collide/in/v2\n-\t1420111091 1709547268 enough/parts/to/this/path/enough/to/collide/in/v2\n+\t2582249472 1763573760 3109209818 first\n+\t2289942528 1188134912 3781118409 second\n+\t2300837888 1130758144 3028707182 third\n+\t2544516325 3963087891 3586976147 a/one-long-enough-for-collisions\n+\t2544516325 4013419539 1701624798 b/two-long-enough-for-collisions\n+\t1420111091 1709547268 2676129939 many/parts/to/this/path/enough/to/collide/in/v2\n+\t1420111091 1709547268 2740459187 enough/parts/to/this/path/enough/to/collide/in/v2\n \tEOF\n \n \ttest_cmp expect out\n-- \ngitgitgadget\n"},{"id":"508505","messageId":"xmqq1pypfo05.fsf@gitster.g","threadId":"62447","inReplyTo":"pull.1823.v2.git.1733181682.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 0/8] pack-objects: Create an alternative name hash algorithm (recreated)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-03T03:23:38Z","receivedAt":"2024-12-03T03:23:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> This series creates a mechanism to select alternative name hashes using a\n> new --name-hash-version=<n> option. The versions are:\n>\n>  1. Version 1 is the default name hash that already exists. This option\n>     focuses on the final bytes of the path to maximize locality for\n>     cross-path deltas.\n>\n>  2. Version 2 is the new path-component hash function suggested by Jonathan\n>     Tan in the previous version (with some modifications). This hash\n>     function essentially computes the v1 name hash of each path component\n>     and then overlays those hashes with a shift to make the parent\n>     directories contribute less to the final hash, but enough to break many\n>     collisions that exist in v1.\n>\n>  3. Version 3 is the hash function that I submitted under the\n>     --full-name-hash feature in the previous versions. This uses a\n>     pseudorandom hash procedure to minimize collisions but at the expense of\n>     losing on locality. This version is implemented in the final patch of\n>     the series mostly for comparison purposes, as it is unlikely to be\n>     selected as a valuable hash function over v2. The final patch could be\n>     omitted from the merged version.\n>\n> See the patches themselves for detailed results in the p5313-pack-objects.sh\n> performance test and the p5314-name-hash.sh test that demonstrates how many\n> collisions occur with each hash function.\n\nThese do not sound like versions but more like variants to me,\nespecially if one is expected to perform better than another in some\ncases and worse in some other cases.  Is it expected that JTan's hash\nto perform better than the original and current hash in almost all\ncases (I would not be surprised at all if that were the case)?\n\n> In general, the v2 name hash function gets very close to the compression\n> results of v3 in the full repack case, even in the repositories that feature\n> many name hash collisions. These benefits come as well without downsides to\n> other kinds of packfiles, including small pushed packs, larger incremental\n> fetch packs, and shallow clones.\n\nNice.\n\n> As we can see, v2 nearly reaches the effectiveness of v3 (and outperforms it\n> once!) but there is still a significant change between the\n> --name-hash-version feature and the --path-walk feature.\n>\n> The main reason we are considering this --name-hash-version feature is that\n> it has the least amount of stretch required in order for it to be integrated\n> with reachability bitmaps, required for server environments. In fact, the\n> change in this version to use a numerical version makes it more obvious how\n> to connect the version number to a value in the .bitmap file format. Tests\n> are added to guarantee that the hash functions preserve their behavior over\n> time, since data files depend on that.\n\nYeah, that aspect certainly is an attractive one.  We should be able\nto teach the bitmap file to say which name-hash function is in use\nand all the pack data reuse logic we have should be reusable.\n\n"},{"id":"508599","messageId":"b07c6b94-b1cc-4391-83fd-adc5bb5f92e3@gmail.com","threadId":"62447","inReplyTo":"xmqq1pypfo05.fsf@gitster.g","subject":"Re: [PATCH v2 0/8] pack-objects: Create an alternative name hash algorithm (recreated)","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2024-12-04T04:56:33Z","receivedAt":"2024-12-04T04:56:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/2/24 10:23 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n> \n>> This series creates a mechanism to select alternative name hashes using a\n>> new --name-hash-version=<n> option. The versions are:\n>>\n>>   1. Version 1 is the default name hash that already exists. This option\n>>      focuses on the final bytes of the path to maximize locality for\n>>      cross-path deltas.\n>>\n>>   2. Version 2 is the new path-component hash function suggested by Jonathan\n>>      Tan in the previous version (with some modifications). This hash\n>>      function essentially computes the v1 name hash of each path component\n>>      and then overlays those hashes with a shift to make the parent\n>>      directories contribute less to the final hash, but enough to break many\n>>      collisions that exist in v1.\n>>\n>>   3. Version 3 is the hash function that I submitted under the\n>>      --full-name-hash feature in the previous versions. This uses a\n>>      pseudorandom hash procedure to minimize collisions but at the expense of\n>>      losing on locality. This version is implemented in the final patch of\n>>      the series mostly for comparison purposes, as it is unlikely to be\n>>      selected as a valuable hash function over v2. The final patch could be\n>>      omitted from the merged version.\n>>\n>> See the patches themselves for detailed results in the p5313-pack-objects.sh\n>> performance test and the p5314-name-hash.sh test that demonstrates how many\n>> collisions occur with each hash function.\n> \n> These do not sound like versions but more like variants to me,\n> especially if one is expected to perform better than another in some\n> cases and worse in some other cases.  Is it expected that JTan's hash\n> to perform better than the original and current hash in almost all\n> cases (I would not be surprised at all if that were the case)?\n\nThere are some cases, such as the Linux kernel repo, that have slightly\nworse compression using JTan's hash. But the naming conventions in that\nrepo are such that the v1 name hash was already pretty effective for\nthat repo. The Git repository has similar issues. See Patch 5 for\ndetailed analysis of these scenarios using the p5313-pack-objects.sh\ntest script.\n\nIt may be possible to adapt some of the collision rate analysis from\nthe test helper in Patch 6 to create a tool that recommends or\ndynamically selects the hash function that works best for the repo.\nSuch a feature should be delayed until this code is exercised in more\nplaces.\n\nThanks,\n-Stolee\n\n\n"},{"id":"508600","messageId":"xmqqldww3usp.fsf@gitster.g","threadId":"62447","inReplyTo":"b07c6b94-b1cc-4391-83fd-adc5bb5f92e3@gmail.com","subject":"Re: [PATCH v2 0/8] pack-objects: Create an alternative name hash algorithm (recreated)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-04T05:02:14Z","receivedAt":"2024-12-04T05:02:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee <stolee@gmail.com> writes:\n\n>> These do not sound like versions but more like variants to me,\n>> especially if one is expected to perform better than another in some\n>> cases and worse in some other cases.  Is it expected that JTan's hash\n>> to perform better than the original and current hash in almost all\n>> cases (I would not be surprised at all if that were the case)?\n>\n> There are some cases, such as the Linux kernel repo, that have slightly\n> worse compression using JTan's hash. But the naming conventions in that\n> repo are such that the v1 name hash was already pretty effective for\n> that repo. The Git repository has similar issues. See Patch 5 for\n> detailed analysis of these scenarios using the p5313-pack-objects.sh\n> test script.\n\nSo it indeed is more \"variant\" than \"version\" where v(N+1) is almost\nalways an implementation of a better idea than vN.\n\n> It may be possible to adapt some of the collision rate analysis from\n> the test helper in Patch 6 to create a tool that recommends or\n> dynamically selects the hash function that works best for the repo.\n> Such a feature should be delayed until this code is exercised in more\n> places.\n\nSurely.  But that is more advanced feature that can wait.  It\ncertainly has to wait until we have a foundation to use more than\none variant safely, which this series lays a good foundation for.\n\nThanks.\n\n"},{"id":"508620","messageId":"CAOLa=ZRTgJafxDTB_LWGJxGZ_YOP4fO3=s14BHNvPaHEqf4Q_A@mail.gmail.com","threadId":"62447","inReplyTo":"454b070d5bb0f64e11cab993b126ef5d37a3615b.1733181682.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 1/8] pack-objects: create new name-hash function version","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-04T20:06:48Z","receivedAt":"2024-12-04T20:06:50Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"\"Jonathan Tan via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n[snip]\n\n> diff --git a/pack-objects.h b/pack-objects.h\n> index b9898a4e64b..15be8368d21 100644\n> --- a/pack-objects.h\n> +++ b/pack-objects.h\n> @@ -207,6 +207,34 @@ static inline uint32_t pack_name_hash(const char *name)\n>  \treturn hash;\n>  }\n>\n> +static inline uint32_t pack_name_hash_v2(const char *name)\n> +{\n> +\tuint32_t hash = 0, base = 0, c;\n> +\n> +\tif (!name)\n> +\t\treturn 0;\n> +\n> +\twhile ((c = *name++)) {\n> +\t\tif (isspace(c))\n> +\t\t\tcontinue;\n> +\t\tif (c == '/') {\n> +\t\t\tbase = (base >> 6) ^ hash;\n\nJust two questions about the implementation for my own knowledge.\n1. Why use '6' here? I couldn't understand the reasoning for the\nspecific value.\n2. Generally in hashing algorithms the XOR is used to ensure that the\noutput distribution is uniform which reduces collisions. Here, as you\nnoted, we're more finding values for sorting rather than hashing in the\ntraditional sense. So why use an XOR?\n\n> +\t\t\thash = 0;\n> +\t\t} else {\n> +\t\t\t/*\n> +\t\t\t * 'c' is only a single byte. Reverse it and move\n> +\t\t\t * it to the top of the hash, moving the rest to\n> +\t\t\t * less-significant bits.\n> +\t\t\t */\n> +\t\t\tc = (c & 0xF0) >> 4 | (c & 0x0F) << 4;\n> +\t\t\tc = (c & 0xCC) >> 2 | (c & 0x33) << 2;\n> +\t\t\tc = (c & 0xAA) >> 1 | (c & 0x55) << 1;\n> +\t\t\thash = (hash >> 2) + (c << 24);\n> +\t\t}\n> +\t}\n> +\treturn (base >> 6) ^ hash;\n> +}\n> +\n>  static inline enum object_type oe_type(const struct object_entry *e)\n>  {\n>  \treturn e->type_valid ? e->type_ : OBJ_BAD;\n> --\n> gitgitgadget\n"},{"id":"508623","messageId":"CAOLa=ZR9QbEZ15aiEKCFcRaewMw7mWs=5xes+NM=9UJ4_CJo7Q@mail.gmail.com","threadId":"62447","inReplyTo":"fb52ca509da6b7a58d7148e3a15ae222ff209cc6.1733181682.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 2/8] pack-objects: add --name-hash-version option","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-04T20:53:49Z","receivedAt":"2024-12-04T20:53:51Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n[snip]\n\n> diff --git a/builtin/repack.c b/builtin/repack.c\n> index d6bb37e84ae..05e13adb87f 100644\n> --- a/builtin/repack.c\n> +++ b/builtin/repack.c\n> @@ -58,6 +58,7 @@ struct pack_objects_args {\n>  \tint no_reuse_object;\n>  \tint quiet;\n>  \tint local;\n> +\tint full_name_hash;\n>  \tstruct list_objects_filter_options filter_options;\n>  };\n>\n\nThis variable doesn't seem to be used anywhere in this commit, and the\nfollowing commit replaces it. I'm guessing it is from the previous\nversion.\n\n[snip]\n"},{"id":"508624","messageId":"xmqqbjxr2m7z.fsf@gitster.g","threadId":"62447","inReplyTo":"CAOLa=ZRTgJafxDTB_LWGJxGZ_YOP4fO3=s14BHNvPaHEqf4Q_A@mail.gmail.com","subject":"Re: [PATCH v2 1/8] pack-objects: create new name-hash function version","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-04T21:05:04Z","receivedAt":"2024-12-04T21:05:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"karthik nayak <karthik.188@gmail.com> writes:\n\n> 2. Generally in hashing algorithms the XOR is used to ensure that the\n> output distribution is uniform which reduces collisions. Here, as you\n> noted, we're more finding values for sorting rather than hashing in the\n> traditional sense. So why use an XOR?\n\nI am not Jonathan, but since the mixing-of-bits is done with XOR in\nthe original that Linus and I wrote, I think the question applies to\nour version as well.  We prefer not to lose entropy from input\nbytes, so we do not want to use OR or AND, which tend to paint\neverything with 1 or 0 as you mix in bits from more bytes.\n\nAnyway the question sounds like \"generally when you take tests you\nwrite with a pencil, but right now you are merely taking notes and\nnot taking any tests. why are you writing with a pencil?\"  There are\nmultiple occasions that a pencil is a great fit as writing equipment.\n"},{"id":"508625","messageId":"CAOLa=ZQcicLN462PFz_2sFeKKB0VvthPZhgwuR3CM0RSM6Y-Ow@mail.gmail.com","threadId":"62447","inReplyTo":"1947d1bf448d71ccd4a44ef25751bab50784a73e.1733181682.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 3/8] repack: add --name-hash-version option","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-04T21:15:01Z","receivedAt":"2024-12-04T21:15:02Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n[snip]\n\n>  DESCRIPTION\n>  -----------\n> @@ -249,6 +251,36 @@ linkgit:git-multi-pack-index[1]).\n>  \tWrite a multi-pack index (see linkgit:git-multi-pack-index[1])\n>  \tcontaining the non-redundant packs.\n>\n> +--name-hash-version=<n>::\n> +\tWhile performing delta compression, Git groups objects that may be\n> +\tsimilar based on heuristics using the path to that object. While\n> +\tgrouping objects by an exact path match is good for paths with\n> +\tmany versions, there are benefits for finding delta pairs across\n> +\tdifferent full paths. Git collects objects by type and then by a\n> +\t\"name hash\" of the path and then by size, hoping to group objects\n> +\tthat will compress well together.\n> ++\n> +The default name hash version is `1`, which prioritizes hash locality by\n> +considering the final bytes of the path as providing the maximum magnitude\n> +to the hash function. This version excels at distinguishing short paths\n> +and finding renames across directories. However, the hash function depends\n> +primarily on the final 16 bytes of the path. If there are many paths in\n> +the repo that have the same final 16 bytes and differ only by parent\n> +directory, then this name-hash may lead to too many collisions and cause\n> +poor results. At the moment, this version is required when writing\n> +reachability bitmap files with `--write-bitmap-index`.\n> ++\n> +The name hash version `2` has similar locality features as version `1`,\n> +except it considers each path component separately and overlays the hashes\n> +with a shift. This still prioritizes the final bytes of the path, but also\n> +\"salts\" the lower bits of the hash using the parent directory names. This\n> +method allows for some of the locality benefits of version `1` while\n> +breaking most of the collisions from a similarly-named file appearing in\n> +many different directories. At the moment, this version is not allowed\n> +when writing reachability bitmap files with `--write-bitmap-index` and it\n> +will be automatically changed to version `1`.\n> +\n> +\n\nNit: I wonder if it'd be nicer to simply point to the documentation in\n'Documentation/git-pack-objects.txt'. This would ensure we have\nconsistent documentation and a single source of truth.\n>\n>  CONFIGURATION\n>  -------------\n>\n> diff --git a/builtin/repack.c b/builtin/repack.c\n> index 05e13adb87f..5e7ff919c1a 100644\n> --- a/builtin/repack.c\n> +++ b/builtin/repack.c\n> @@ -39,7 +39,9 @@ static int run_update_server_info = 1;\n>  static char *packdir, *packtmp_name, *packtmp;\n>\n>  static const char *const git_repack_usage[] = {\n> -\tN_(\"git repack [<options>]\"),\n> +\tN_(\"git repack [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\\n\"\n> +\t   \"[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\\n\"\n> +\t   \"[--write-midx] [--name-hash-version=<n>]\"),\n>  \tNULL\n>  };\n\nSo this fixes the mismatch in t0450 which is seen below.\n\nNit: might be worthwhile adding this in the commit message.\n\n[snip]\n"},{"id":"508626","messageId":"CAOLa=ZTuAKtYvtWoR0cvaORiXPsjBxq-nhXL8NJZpJSmBtEdpg@mail.gmail.com","threadId":"62447","inReplyTo":"6a95708bf972cb22c8abf1da389350fc9f53c4ca.1733181682.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 4/8] pack-objects: add GIT_TEST_NAME_HASH_VERSION","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-04T21:21:22Z","receivedAt":"2024-12-04T21:21:24Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> From: Derrick Stolee <stolee@gmail.com>\n>\n> Add a new environment variable to opt-in to differen values of the\n\ns/differen/different\n\n> --name-hash-version=<n> option in 'git pack-objects'. This allows for\n> extra testing of the feature without repeating all of the test\n> scenarios. Unlike many GIT_TEST_* variables, we are choosing to not add\n> this to the linux-TEST-vars CI build as that test run is already\n> overloaded. The behavior exposed by this test variable is of low risk\n> and should be sufficient to allow manual testing when an issue arises.\n>\n> But this option isn't free. There are a few tests that change behavior\n> with the variable enabled.\n>\n> First, there are a few tests that are very sensitive to certain delta\n> bases being picked. These are both involving the generation of thin\n> bundles and then counting their objects via 'git index-pack --fix-thin'\n> which pulls the delta base into the new packfile. For these tests,\n> disable the option as a decent long-term option.\n>\n> Second, there are two tests in t5616-partial-clone.sh that I believe are\n> actually broken scenarios. While the client is set up to clone the\n> 'promisor-server' repo via a treeless partial clone filter (tree:0),\n> that filter does not translate to the 'server' repo. Thus, fetching from\n> these repos causes the server to think that the client has all reachable\n> trees and blobs from the commits advertised as 'haves'. This leads the\n> server to providing a thin pack assuming those objects as delta bases.\n> Changing the name-hash algorithm presents new delta bases and thus\n> breaks the expectations of these tests. An alternative could be to set\n> up 'server' as a promisor server with the correct filter enabled. This\n> may also point out more issues with partial clone being set up as a\n> remote-based filtering mechanism and not a repository-wide setting. For\n> now, do the minimal change to make the test work by disabling the test\n> variable.\n>\n> Third, there are some tests that compare the exact output of a 'git\n> pack-objects' process when using bitmaps. The warning that ignores the\n> --name-hash-version=2 and forces version 1 causes these tests to fail.\n> Disable the environment variable to get around this issue.\n>\n\nThe patch itself looked good to me.\n\n[snip]\n"},{"id":"508663","messageId":"CAOLa=ZRDkWymubTQq79+XgttVeK5Ti96wXju_QiDKmy__pTyNA@mail.gmail.com","threadId":"62447","inReplyTo":"xmqqbjxr2m7z.fsf@gitster.g","subject":"Re: [PATCH v2 1/8] pack-objects: create new name-hash function version","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-05T09:46:32Z","receivedAt":"2024-12-05T09:46:35Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> karthik nayak <karthik.188@gmail.com> writes:\n>\n>> 2. Generally in hashing algorithms the XOR is used to ensure that the\n>> output distribution is uniform which reduces collisions. Here, as you\n>> noted, we're more finding values for sorting rather than hashing in the\n>> traditional sense. So why use an XOR?\n>\n> I am not Jonathan, but since the mixing-of-bits is done with XOR in\n> the original that Linus and I wrote, I think the question applies to\n> our version as well.  We prefer not to lose entropy from input\n> bytes, so we do not want to use OR or AND, which tend to paint\n> everything with 1 or 0 as you mix in bits from more bytes.\n>\n\nThanks for the answering. My reasoning at the start was more of my\nthoughts on why XOR could be used here and your explanation makes it\nclearer.\n\n> Anyway the question sounds like \"generally when you take tests you\n> write with a pencil, but right now you are merely taking notes and\n> not taking any tests. why are you writing with a pencil?\"  There are\n> multiple occasions that a pencil is a great fit as writing equipment.\n\nI'd say my questions was more of \"I know a pencil is used when taking\ntests because of XYZ reasons. On similar lines, why is a pencil used\nhere?\" :)\n"},{"id":"508890","messageId":"20241209231201.841076-1-jonathantanmy@google.com","threadId":"62447","inReplyTo":"6a95708bf972cb22c8abf1da389350fc9f53c4ca.1733181682.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 4/8] pack-objects: add GIT_TEST_NAME_HASH_VERSION","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2024-12-09T23:12:00Z","receivedAt":"2024-12-09T23:12:03Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n>  t/t5616-partial-clone.sh        | 26 ++++++++++++++++++++++++--\n\nI believe the changes to this file are no longer needed.\n"},{"id":"508891","messageId":"20241209231522.841908-1-jonathantanmy@google.com","threadId":"62447","inReplyTo":"454b070d5bb0f64e11cab993b126ef5d37a3615b.1733181682.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 1/8] pack-objects: create new name-hash function version","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2024-12-09T23:15:22Z","receivedAt":"2024-12-09T23:15:25Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"\"Jonathan Tan via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n> diff --git a/pack-objects.h b/pack-objects.h\n> index b9898a4e64b..15be8368d21 100644\n> --- a/pack-objects.h\n> +++ b/pack-objects.h\n> @@ -207,6 +207,34 @@ static inline uint32_t pack_name_hash(const char *name)\n>  \treturn hash;\n>  }\n>  \n> +static inline uint32_t pack_name_hash_v2(const char *name)\n> +{\n> +\tuint32_t hash = 0, base = 0, c;\n> +\n> +\tif (!name)\n> +\t\treturn 0;\n> +\n> +\twhile ((c = *name++)) {\n> +\t\tif (isspace(c))\n> +\t\t\tcontinue;\n> +\t\tif (c == '/') {\n> +\t\t\tbase = (base >> 6) ^ hash;\n> +\t\t\thash = 0;\n> +\t\t} else {\n> +\t\t\t/*\n> +\t\t\t * 'c' is only a single byte. Reverse it and move\n> +\t\t\t * it to the top of the hash, moving the rest to\n> +\t\t\t * less-significant bits.\n> +\t\t\t */\n> +\t\t\tc = (c & 0xF0) >> 4 | (c & 0x0F) << 4;\n> +\t\t\tc = (c & 0xCC) >> 2 | (c & 0x33) << 2;\n> +\t\t\tc = (c & 0xAA) >> 1 | (c & 0x55) << 1;\n> +\t\t\thash = (hash >> 2) + (c << 24);\n> +\t\t}\n> +\t}\n> +\treturn (base >> 6) ^ hash;\n> +}\n\nThis works because `c` is masked before any arithmetic is performed on\nit, but the cast from potentially signed char to uint32_t still makes\nme nervous - if char is signed, it behaves as if it was first cast to\nint32_t and only then to uint32_t, as you can see from running the code\nbelow:\n\n    #include <stdio.h>\n    int main() {\n        signed char c = 128;\n        unsigned int u = c;\n        printf(\"hello %u\\n\", u);\n        return 0;\n    }\n\nI would declare `c` as uint8_t or unsigned char.\n"},{"id":"508896","messageId":"xmqqjzc8mmns.fsf@gitster.g","threadId":"62447","inReplyTo":"20241209231522.841908-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 1/8] pack-objects: create new name-hash function version","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-10T00:01:11Z","receivedAt":"2024-12-10T00:01:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> \"Jonathan Tan via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n>> diff --git a/pack-objects.h b/pack-objects.h\n>> index b9898a4e64b..15be8368d21 100644\n>> --- a/pack-objects.h\n>> +++ b/pack-objects.h\n>> @@ -207,6 +207,34 @@ static inline uint32_t pack_name_hash(const char *name)\n>>  \treturn hash;\n>>  }\n>>  \n>> +static inline uint32_t pack_name_hash_v2(const char *name)\n>> +{\n>> +\tuint32_t hash = 0, base = 0, c;\n>> + ...\n>> +\twhile ((c = *name++)) {\n>> + ...\n>> +\t\t\t/*\n>> +\t\t\t * 'c' is only a single byte. Reverse it and move\n>> +\t\t\t * it to the top of the hash, moving the rest to\n>> +\t\t\t * less-significant bits.\n>> +\t\t\t */\n>> +\t\t\tc = (c & 0xF0) >> 4 | (c & 0x0F) << 4;\n>> + ...\n> This works because `c` is masked before any arithmetic is performed on\n> it, but the cast from potentially signed char to uint32_t still makes\n> me nervous - if char is signed, it behaves as if it was first cast to\n> int32_t and only then to uint32_t, ...\n> I would declare `c` as uint8_t or unsigned char.\n\nI think you meant the type of \"name\", and your worry is that *name\nmay pick up a negative integer from there when the platform char is\nsigned?  Here we are dealing with a pathname that has often UTF-8\nencoded non-ASCII letters with 8th bit set, and I agree with you\nthat being explicitly unsigned would certainly help reduce the\ncognitive load.\n\nThanks.\n\n\n"},{"id":"509426","messageId":"718f22e5-5ddb-4efc-a46e-33c8d7c4f362@gmail.com","threadId":"62447","inReplyTo":"20241209231201.841076-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 4/8] pack-objects: add GIT_TEST_NAME_HASH_VERSION","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2024-12-20T17:03:10Z","receivedAt":"2024-12-20T17:03:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/9/24 6:12 PM, Jonathan Tan wrote:\n> \"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n>>   t/t5616-partial-clone.sh        | 26 ++++++++++++++++++++++++--\n> \n> I believe the changes to this file are no longer needed.\n\nYou're right. They are only needed in the last patch when the v3\nname-hash is possible.\n\n(Unless maybe you fixed this issue in 'master' and I didn't see it)\n\nThanks,\n-Stolee\n\n"},{"id":"509427","messageId":"68b4127580e2d475bec0d7cd0f6a9ae5e626b3c9.1734715194.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.v3.git.1734715194.gitgitgadget@gmail.com","subject":"[PATCH v3 1/8] pack-objects: create new name-hash function version","fromName":"Jonathan Tan via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-20T17:19:47Z","receivedAt":"2024-12-20T17:19:59Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"From: Jonathan Tan <jonathantanmy@google.com>\n\nAs we will explore in later changes, the default name-hash function used\nin 'git pack-objects' has a tendency to cause collisions and cause poor\ndelta selection. This change creates an alternative that avoids some\ncollisions while preserving some amount of hash locality.\n\nThe pack_name_hash() method has not been materially changed since it was\nintroduced in ce0bd64 (pack-objects: improve path grouping\nheuristics., 2006-06-05). The intention here is to group objects by path\nname, but also attempt to group similar file types together by making\nthe most-significant digits of the hash be focused on the final\ncharacters.\n\nHere's the crux of the implementation:\n\n\t/*\n\t * This effectively just creates a sortable number from the\n\t * last sixteen non-whitespace characters. Last characters\n\t * count \"most\", so things that end in \".c\" sort together.\n\t */\n\twhile ((c = *name++) != 0) {\n\t\tif (isspace(c))\n\t\t\tcontinue;\n\t\thash = (hash >> 2) + (c << 24);\n\t}\n\nAs the comment mentions, this only cares about the last sixteen\nnon-whitespace characters. This cause some filenames to collide more than\nothers. This collision is somewhat by design in order to promote hash\nlocality for files that have similar types (.c, .h, .json) or could be the\nsame file across a directory rename (a/foo.txt to b/foo.txt). This leads to\ndecent cross-path deltas in cases like shallow clones or packing a\nrepository with very few historical versions of files that share common data\nwith other similarly-named files.\n\nHowever, when the name-hash instead leads to a large number of name-hash\ncollisions for otherwise unrelated files, this can lead to confusing the\ndelta calculation to prefer cross-path deltas over previous versions of the\nsame file.\n\nThe new pack_name_hash_v2() function attempts to fix this issue by\ntaking more of the directory path into account through its hash\nfunction. Its naming implies that we will later wire up details for\nchoosing a name-hash function by version.\n\nThe first change is to be more careful about paths using non-ASCII\ncharacters. With these characters in mind, reverse the bits in the byte\nas the least-significant bits have the highest entropy and we want to\nmaximize their influence. This is done with some bit manipulation that\nswaps the two halves, then the quarters within those halves, and then\nthe bits within those quarters.\n\nThe second change is to perform hash composition operations at every\nlevel of the path. This is done by storing a 'base' hash value that\ncontains the hash of the parent directory. When reaching a directory\nboundary, we XOR the current level's name-hash value with a downshift of\nthe previous level's hash. This perturbation intends to create low-bit\ndistinctions for paths with the same final 16 bytes but distinct parent\ndirectory structures.\n\nThe collision rate and effectiveness of this hash function will be\nexplored in later changes as the function is integrated with 'git\npack-objects' and 'git repack'.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n pack-objects.h | 28 ++++++++++++++++++++++++++++\n 1 file changed, 28 insertions(+)\n\ndiff --git a/pack-objects.h b/pack-objects.h\nindex b9898a4e64b..681c1116486 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -207,6 +207,34 @@ static inline uint32_t pack_name_hash(const char *name)\n \treturn hash;\n }\n \n+static inline uint32_t pack_name_hash_v2(const unsigned char *name)\n+{\n+\tuint32_t hash = 0, base = 0, c;\n+\n+\tif (!name)\n+\t\treturn 0;\n+\n+\twhile ((c = *name++)) {\n+\t\tif (isspace(c))\n+\t\t\tcontinue;\n+\t\tif (c == '/') {\n+\t\t\tbase = (base >> 6) ^ hash;\n+\t\t\thash = 0;\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * 'c' is only a single byte. Reverse it and move\n+\t\t\t * it to the top of the hash, moving the rest to\n+\t\t\t * less-significant bits.\n+\t\t\t */\n+\t\t\tc = (c & 0xF0) >> 4 | (c & 0x0F) << 4;\n+\t\t\tc = (c & 0xCC) >> 2 | (c & 0x33) << 2;\n+\t\t\tc = (c & 0xAA) >> 1 | (c & 0x55) << 1;\n+\t\t\thash = (hash >> 2) + (c << 24);\n+\t\t}\n+\t}\n+\treturn (base >> 6) ^ hash;\n+}\n+\n static inline enum object_type oe_type(const struct object_entry *e)\n {\n \treturn e->type_valid ? e->type_ : OBJ_BAD;\n-- \ngitgitgadget\n\n"},{"id":"509428","messageId":"pull.1823.v3.git.1734715194.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.v2.git.1733181682.gitgitgadget@gmail.com","subject":"[PATCH v3 0/8] pack-objects: Create an alternative name hash algorithm (recreated)","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-20T17:19:46Z","receivedAt":"2024-12-20T17:19:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is a recreation of the topic in [1] that was closed. (I force-pushed my\nbranch and GitHub won't let me reopen the PR for GitGitGadget to create this\nas v3.)\n\n[1]\nhttps://lore.kernel.org/git/pull.1785.v2.git.1726692381.gitgitgadget@gmail.com/\n\nI've been focused recently on understanding and mitigating the growth of a\nfew internal repositories. Some of these are growing much larger than\nexpected for the number of contributors, and there are multiple aspects to\nwhy this growth is so large.\n\nThis is part of the RFC I submitted [2] involving the path-walk API, though\nthis doesn't use the path-walk API directly. In full repack cases, it seems\nthat the --full-name-hash option gets nearly as good compression as the\n--path-walk option introduced in that series. I continue to work on that\nfeature as well, so we can review it after this series is complete.\n\n[2]\nhttps://lore.kernel.org/git/pull.1786.git.1725935335.gitgitgadget@gmail.com/\n\nThe main issue plaguing these repositories is that deltas are not being\ncomputed against objects that appear at the same path. While the size of\nthese files at tip is one aspect of growth that would prevent this issue,\nthe changes to these files are reasonable and should result in good delta\ncompression. However, Git is not discovering the connections across\ndifferent versions of the same file.\n\nOne way to find some improvement in these repositories is to increase the\nwindow size, which was an initial indicator that the delta compression could\nbe improved, but was not a clear indicator. After some digging (and\nprototyping some analysis tools) the main discovery was that the current\nname-hash algorithm only considers the last 16 characters in the path name\nand has some naturally-occurring collisions within that scope.\n\nThis series creates a mechanism to select alternative name hashes using a\nnew --name-hash-version=<n> option. The versions are:\n\n 1. Version 1 is the default name hash that already exists. This option\n    focuses on the final bytes of the path to maximize locality for\n    cross-path deltas.\n\n 2. Version 2 is the new path-component hash function suggested by Jonathan\n    Tan in the previous version (with some modifications). This hash\n    function essentially computes the v1 name hash of each path component\n    and then overlays those hashes with a shift to make the parent\n    directories contribute less to the final hash, but enough to break many\n    collisions that exist in v1.\n\n 3. Version 3 is the hash function that I submitted under the\n    --full-name-hash feature in the previous versions. This uses a\n    pseudorandom hash procedure to minimize collisions but at the expense of\n    losing on locality. This version is implemented in the final patch of\n    the series mostly for comparison purposes, as it is unlikely to be\n    selected as a valuable hash function over v2. The final patch could be\n    omitted from the merged version.\n\nSee the patches themselves for detailed results in the p5313-pack-objects.sh\nperformance test and the p5314-name-hash.sh test that demonstrates how many\ncollisions occur with each hash function.\n\nIn general, the v2 name hash function gets very close to the compression\nresults of v3 in the full repack case, even in the repositories that feature\nmany name hash collisions. These benefits come as well without downsides to\nother kinds of packfiles, including small pushed packs, larger incremental\nfetch packs, and shallow clones.\n\nI should point out that there is still a significant jump in compression\neffectiveness between these name hash version options and the --path-walk\nfeature I suggested in my RFC [2] and has review underway in [3] (along with\nchanges to git pack-objects and git repack in [4]).\n\n[3]\nhttps://lore.kernel.org/git/pull.1818.v2.git.1731181272.gitgitgadget@gmail.com/\n\n[4] https://github.com/gitgitgadget/git/pull/1819\n\nTo compare these options in a set of Javascript repositories that have\ndifferent levels of name hash collisions, see the following table that lists\nthe size of the packfile after git repack -adf\n[--name-hash-version=<n>|--path-walk]:\n\n| Repo     | V1 Size   | V2 Size | V3 Size | Path Walk Size |\n|----------|-----------|---------|---------|----------------|\n| fluentui |     440 M |   161 M |   170 M |          123 M |\n| Repo B   |   6,248 M |   856 M |   840 M |          782 M |\n| Repo C   |  37,278 M | 6,921 M | 6,755 M |        6,156 M |\n| Repo D   | 131,204 M | 7,463 M | 7,124 M |        4,005 M |\n\n\nAs we can see, v2 nearly reaches the effectiveness of v3 (and outperforms it\nonce!) but there is still a significant change between the\n--name-hash-version feature and the --path-walk feature.\n\nThe main reason we are considering this --name-hash-version feature is that\nit has the least amount of stretch required in order for it to be integrated\nwith reachability bitmaps, required for server environments. In fact, the\nchange in this version to use a numerical version makes it more obvious how\nto connect the version number to a value in the .bitmap file format. Tests\nare added to guarantee that the hash functions preserve their behavior over\ntime, since data files depend on that.\n\nThanks, -Stolee\n\n\nUPDATES SINCE V1\n================\n\n * BIG CHANGE: --full-name-hash is replaced with --name-hash-version=<n>.\n\n * --name-hash-version=2 uses Jonathan Tan's hash function (with some\n   adjustments). See the first patch for this implementation, credited to\n   him.\n\n * --name-hash-version=3 uses the hash function I wrote for the previous\n   version's --full-name-hash. This is left as the final patch so it could\n   be easily removed from the series if not considered worth having since it\n   has some pain points that are resolved from v2 without significant issues\n   to overall repo size.\n\n * Commit messaes are updated with these changes, as well as a better\n   attempt to indicate the benefit of cross-path delta pairs, such as\n   renames or similar content based on file extension.\n\n * Performance numbers are regenerated for the same set of repositories.\n   Size data is somewhat nondeterministic due to concurrent threads\n   competing over delta computations.\n\n * The --name-hash-version option is not added to git repack until its own\n   patch.\n\n * The patch that updates git repack's synopsis match its docs is squashed\n   into the patch that adds the option to git repack.\n\n * Documentation is expanded for git pack-objects and reused for git repack.\n\n * GIT_TEST_FULL_NAME_HASH is now GIT_TEST_NAME_HASH_VERSION with similar\n   caveats required for tests. It is removed from the linux-TEST-vars CI\n   job.\n\n * The performance test p5313-pack-objects.sh is now organized via a loop\n   over the different versions. This separates the scenarios, which makes\n   things harder to compare directly, but makes it trivial to add new\n   versions.\n\n * The patch that disabled --full-name-hash when performing a shallow clone\n   is no longer present, as it is not necessary when using\n   --name-hash-version=2. Perhaps it would be valuable for repo using v3, if\n   that is kept in the series.\n\n * We force name hash version 1 when writing or reading bitmaps.\n\n * A small patch is added to cause a BUG() failure if the name hash version\n   global changes between calls to pack_name_hash_fn(). This is solely\n   defensive programming.\n\n * Several typos, style issues, or suggested comments are resolved.\n\n\nUPDATES SINCE v2\n================\n\n * For extra safety, the new name-hash algorithm uses unsigned characters.\n\n * A stray 'full_name_hash' variable is removed.\n\n * Commit messages are improved.\n\n * 'git repack' documentation now points to 'git pack-objects' docs for the\n   --name-hash-version option.\n\n * The changes to t5616 are delayed until the introduction of version 3,\n   since the v2 name hash does not demonstrate the behavior.\n\nDerrick Stolee (7):\n  pack-objects: add --name-hash-version option\n  repack: add --name-hash-version option\n  pack-objects: add GIT_TEST_NAME_HASH_VERSION\n  p5313: add size comparison test\n  test-tool: add helper for name-hash values\n  pack-objects: prevent name hash version change\n  pack-objects: add third name hash version\n\nJonathan Tan (1):\n  pack-objects: create new name-hash function version\n\n Documentation/git-pack-objects.txt | 41 ++++++++++++++++-\n Documentation/git-repack.txt       |  9 +++-\n Makefile                           |  1 +\n builtin/pack-objects.c             | 63 ++++++++++++++++++++++++---\n builtin/repack.c                   |  9 +++-\n pack-objects.h                     | 54 +++++++++++++++++++++++\n t/README                           |  4 ++\n t/helper/test-name-hash.c          | 24 ++++++++++\n t/helper/test-tool.c               |  1 +\n t/helper/test-tool.h               |  1 +\n t/perf/p5313-pack-objects.sh       | 70 ++++++++++++++++++++++++++++++\n t/perf/p5314-name-hash.sh          | 31 +++++++++++++\n t/t0450/txt-help-mismatches        |  1 -\n t/t5300-pack-object.sh             | 34 +++++++++++++++\n t/t5310-pack-bitmaps.sh            | 35 ++++++++++++++-\n t/t5333-pseudo-merge-bitmaps.sh    |  4 ++\n t/t5510-fetch.sh                   |  7 ++-\n t/t5616-partial-clone.sh           | 26 ++++++++++-\n t/t6020-bundle-misc.sh             |  6 ++-\n t/t7406-submodule-update.sh        |  4 +-\n t/t7700-repack.sh                  | 16 ++++++-\n t/test-lib-functions.sh            | 26 +++++++++++\n 22 files changed, 450 insertions(+), 17 deletions(-)\n create mode 100644 t/helper/test-name-hash.c\n create mode 100755 t/perf/p5313-pack-objects.sh\n create mode 100755 t/perf/p5314-name-hash.sh\n\n\nbase-commit: 8f8d6eee531b3fa1a8ef14f169b0cb5035f7a772\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1823%2Fderrickstolee%2Ffull-name-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1823/derrickstolee/full-name-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/1823\n\nRange-diff vs v2:\n\n 1:  454b070d5bb ! 1:  68b4127580e pack-objects: create new name-hash function version\n     @@ pack-objects.h: static inline uint32_t pack_name_hash(const char *name)\n       \treturn hash;\n       }\n       \n     -+static inline uint32_t pack_name_hash_v2(const char *name)\n     ++static inline uint32_t pack_name_hash_v2(const unsigned char *name)\n      +{\n      +\tuint32_t hash = 0, base = 0, c;\n      +\n 2:  fb52ca509da ! 2:  d035e3e59f4 pack-objects: add --name-hash-version option\n     @@ builtin/pack-objects.c: struct configured_exclusion {\n      +\t\treturn pack_name_hash(name);\n      +\n      +\tcase 2:\n     -+\t\treturn pack_name_hash_v2(name);\n     ++\t\treturn pack_name_hash_v2((const unsigned char *)name);\n      +\n      +\tdefault:\n      +\t\tBUG(\"invalid name-hash version: %d\", name_hash_version);\n     @@ builtin/pack-objects.c: int cmd_pack_objects(int argc,\n       \t\tstrvec_push(&rp, \"--topo-order\");\n       \n      \n     - ## builtin/repack.c ##\n     -@@ builtin/repack.c: struct pack_objects_args {\n     - \tint no_reuse_object;\n     - \tint quiet;\n     - \tint local;\n     -+\tint full_name_hash;\n     - \tstruct list_objects_filter_options filter_options;\n     - };\n     - \n     -\n       ## t/t5300-pack-object.sh ##\n      @@ t/t5300-pack-object.sh: do\n       \t'\n 3:  1947d1bf448 ! 3:  e2191244f6b repack: add --name-hash-version option\n     @@ Commit message\n          new method, test_subcommand_flex. Use it to check that the\n          --name-hash-version option is passing through.\n      \n     +    Since we are modifying the 'git repack' command, let's bring its usage\n     +    in line with the Documentation's synopsis. This removes it from the\n     +    allow list in t0450 so it will remain in sync in the future.\n     +\n          Signed-off-by: Derrick Stolee <stolee@gmail.com>\n      \n       ## Documentation/git-repack.txt ##\n     @@ Documentation/git-repack.txt: linkgit:git-multi-pack-index[1]).\n       \tcontaining the non-redundant packs.\n       \n      +--name-hash-version=<n>::\n     -+\tWhile performing delta compression, Git groups objects that may be\n     -+\tsimilar based on heuristics using the path to that object. While\n     -+\tgrouping objects by an exact path match is good for paths with\n     -+\tmany versions, there are benefits for finding delta pairs across\n     -+\tdifferent full paths. Git collects objects by type and then by a\n     -+\t\"name hash\" of the path and then by size, hoping to group objects\n     -+\tthat will compress well together.\n     -++\n     -+The default name hash version is `1`, which prioritizes hash locality by\n     -+considering the final bytes of the path as providing the maximum magnitude\n     -+to the hash function. This version excels at distinguishing short paths\n     -+and finding renames across directories. However, the hash function depends\n     -+primarily on the final 16 bytes of the path. If there are many paths in\n     -+the repo that have the same final 16 bytes and differ only by parent\n     -+directory, then this name-hash may lead to too many collisions and cause\n     -+poor results. At the moment, this version is required when writing\n     -+reachability bitmap files with `--write-bitmap-index`.\n     -++\n     -+The name hash version `2` has similar locality features as version `1`,\n     -+except it considers each path component separately and overlays the hashes\n     -+with a shift. This still prioritizes the final bytes of the path, but also\n     -+\"salts\" the lower bits of the hash using the parent directory names. This\n     -+method allows for some of the locality benefits of version `1` while\n     -+breaking most of the collisions from a similarly-named file appearing in\n     -+many different directories. At the moment, this version is not allowed\n     -+when writing reachability bitmap files with `--write-bitmap-index` and it\n     -+will be automatically changed to version `1`.\n     ++\tProvide this argument to the underlying `git pack-objects` process.\n     ++\tSee linkgit:git-pack-objects[1] for full details.\n      +\n      +\n       CONFIGURATION\n     @@ builtin/repack.c: struct pack_objects_args {\n       \tint no_reuse_object;\n       \tint quiet;\n       \tint local;\n     --\tint full_name_hash;\n      +\tint name_hash_version;\n       \tstruct list_objects_filter_options filter_options;\n       };\n 4:  6a95708bf97 ! 4:  86ff0d0a15e pack-objects: add GIT_TEST_NAME_HASH_VERSION\n     @@ Metadata\n       ## Commit message ##\n          pack-objects: add GIT_TEST_NAME_HASH_VERSION\n      \n     -    Add a new environment variable to opt-in to differen values of the\n     +    Add a new environment variable to opt-in to different values of the\n          --name-hash-version=<n> option in 'git pack-objects'. This allows for\n          extra testing of the feature without repeating all of the test\n          scenarios. Unlike many GIT_TEST_* variables, we are choosing to not add\n     @@ Commit message\n          which pulls the delta base into the new packfile. For these tests,\n          disable the option as a decent long-term option.\n      \n     -    Second, there are two tests in t5616-partial-clone.sh that I believe are\n     -    actually broken scenarios. While the client is set up to clone the\n     -    'promisor-server' repo via a treeless partial clone filter (tree:0),\n     -    that filter does not translate to the 'server' repo. Thus, fetching from\n     -    these repos causes the server to think that the client has all reachable\n     -    trees and blobs from the commits advertised as 'haves'. This leads the\n     -    server to providing a thin pack assuming those objects as delta bases.\n     -    Changing the name-hash algorithm presents new delta bases and thus\n     -    breaks the expectations of these tests. An alternative could be to set\n     -    up 'server' as a promisor server with the correct filter enabled. This\n     -    may also point out more issues with partial clone being set up as a\n     -    remote-based filtering mechanism and not a repository-wide setting. For\n     -    now, do the minimal change to make the test work by disabling the test\n     -    variable.\n     -\n     -    Third, there are some tests that compare the exact output of a 'git\n     +    Second, there are some tests that compare the exact output of a 'git\n          pack-objects' process when using bitmaps. The warning that ignores the\n          --name-hash-version=2 and forces version 1 causes these tests to fail.\n          Disable the environment variable to get around this issue.\n     @@ t/t5510-fetch.sh: test_expect_success 'all boundary commits are excluded' '\n       '\n       \n      \n     - ## t/t5616-partial-clone.sh ##\n     -@@ t/t5616-partial-clone.sh: test_expect_success 'fetch lazy-fetches only to resolve deltas' '\n     - \t# Exercise to make sure it works. Git will not fetch anything from the\n     - \t# promisor remote other than for the big tree (because it needs to\n     - \t# resolve the delta).\n     --\tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -C client \\\n     -+\t#\n     -+\t# TODO: the --full-name-hash option is disabled here, since this test\n     -+\t# is fundamentally broken! When GIT_TEST_NAME_HASH_VERSION=2, the server\n     -+\t# recognizes delta bases in a different way and then sends a _blob_ to\n     -+\t# the client with a delta base that the client does not have! This is\n     -+\t# because the client is cloned from \"promisor-server\" with tree:0 but\n     -+\t# is now fetching from \"server\" without any filter. This is violating the\n     -+\t# promise to the server that all reachable objects exist and could be\n     -+\t# used as delta bases!\n     -+\tGIT_TRACE_PACKET=\"$(pwd)/trace\" \\\n     -+\t\tGIT_TEST_NAME_HASH_VERSION=1 \\\n     -+\t\tgit -C client \\\n     - \t\tfetch \"file://$(pwd)/server\" main &&\n     - \n     - \t# Verify the assumption that the client needed to fetch the delta base\n     -@@ t/t5616-partial-clone.sh: test_expect_success 'fetch lazy-fetches only to resolve deltas, protocol v2' '\n     - \t# Exercise to make sure it works. Git will not fetch anything from the\n     - \t# promisor remote other than for the big blob (because it needs to\n     - \t# resolve the delta).\n     --\tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -C client \\\n     -+\t#\n     -+\t# TODO: the --full-name-hash option is disabled here, since this test\n     -+\t# is fundamentally broken! When GIT_TEST_NAME_HASH_VERSION=2, the server\n     -+\t# recognizes delta bases in a different way and then sends a _blob_ to\n     -+\t# the client with a delta base that the client does not have! This is\n     -+\t# because the client is cloned from \"promisor-server\" with tree:0 but\n     -+\t# is now fetching from \"server\" without any filter. This is violating the\n     -+\t# promise to the server that all reachable objects exist and could be\n     -+\t# used as delta bases!\n     -+\tGIT_TRACE_PACKET=\"$(pwd)/trace\" \\\n     -+\t\tGIT_TEST_NAME_HASH_VERSION=1 \\\n     -+\t\tgit -C client \\\n     - \t\tfetch \"file://$(pwd)/server\" main &&\n     - \n     - \t# Verify that protocol version 2 was used.\n     -\n       ## t/t6020-bundle-misc.sh ##\n      @@ t/t6020-bundle-misc.sh: test_expect_success 'create bundle with --since option' '\n       \tEOF\n 5:  3b5697467c9 = 5:  163aaab3e1b p5313: add size comparison test\n 6:  36f2811e3d9 ! 6:  e9ce79fa6e7 test-tool: add helper for name-hash values\n     @@ t/helper/test-name-hash.c (new)\n      +\n      +\twhile (!strbuf_getline(&line, stdin)) {\n      +\t\tprintf(\"%10u \", pack_name_hash(line.buf));\n     -+\t\tprintf(\"%10u \", pack_name_hash_v2(line.buf));\n     ++\t\tprintf(\"%10u \", pack_name_hash_v2((unsigned const char *)line.buf));\n      +\t\tprintf(\"%s\\n\", line.buf);\n      +\t}\n      +\n 7:  3885ef8a2f7 = 7:  18a41f2fe6f pack-objects: prevent name hash version change\n 8:  64fd7b3ccad ! 8:  3d63954f318 pack-objects: add third name hash version\n     @@ Commit message\n          unless there are enough collisions even with v2 that the full repack\n          scenario has larger improvements than these.\n      \n     +    When using GIT_TEST_NAME_HASH_VERSION=3, there are some necessary\n     +    changes to t5616-partial-clone.sh since the server now picks different\n     +    delta bases that the client does not have (and does not then fetch\n     +    dynamically). These changes are a minimal patch and the functionality\n     +    should be fixed in other changes.\n     +\n          Signed-off-by: Derrick Stolee <stolee@gmail.com>\n      \n       ## Documentation/git-pack-objects.txt ##\n     @@ Documentation/git-pack-objects.txt: breaking most of the collisions from a simil\n       \n       DELTA ISLANDS\n      \n     - ## Documentation/git-repack.txt ##\n     -@@ Documentation/git-repack.txt: breaking most of the collisions from a similarly-named file appearing in\n     - many different directories. At the moment, this version is not allowed\n     - when writing reachability bitmap files with `--write-bitmap-index` and it\n     - will be automatically changed to version `1`.\n     -++\n     -+The name hash version `3` abandons the locality features of versions `1`\n     -+and `2` in favor of minimizing collisions. The goal here is to separate\n     -+objects by their full path and abandon hope for cross-path delta\n     -+compression. For this reason, this option is preferred for repacking large\n     -+repositories with many versions and many name hash collisions when using\n     -+the first two versions. At the moment, this version is not allowed when\n     -+writing reachability bitmap files with `--write-bitmap-index` and it will\n     -+be automatically changed to version `1`.\n     - \n     - \n     - CONFIGURATION\n     -\n       ## builtin/pack-objects.c ##\n      @@ builtin/pack-objects.c: static int name_hash_version = -1;\n       \n     @@ builtin/pack-objects.c: static int name_hash_version = -1;\n       \n      @@ builtin/pack-objects.c: static inline uint32_t pack_name_hash_fn(const char *name)\n       \tcase 2:\n     - \t\treturn pack_name_hash_v2(name);\n     + \t\treturn pack_name_hash_v2((const unsigned char *)name);\n       \n      +\tcase 3:\n      +\t\treturn pack_name_hash_v3(name);\n     @@ builtin/pack-objects.c: static inline uint32_t pack_name_hash_fn(const char *nam\n       \t}\n      \n       ## pack-objects.h ##\n     -@@ pack-objects.h: static inline uint32_t pack_name_hash_v2(const char *name)\n     +@@ pack-objects.h: static inline uint32_t pack_name_hash_v2(const unsigned char *name)\n       \treturn (base >> 6) ^ hash;\n       }\n       \n     @@ t/helper/test-name-hash.c\n      @@ t/helper/test-name-hash.c: int cmd__name_hash(int argc UNUSED, const char **argv UNUSED)\n       \twhile (!strbuf_getline(&line, stdin)) {\n       \t\tprintf(\"%10u \", pack_name_hash(line.buf));\n     - \t\tprintf(\"%10u \", pack_name_hash_v2(line.buf));\n     + \t\tprintf(\"%10u \", pack_name_hash_v2((unsigned const char *)line.buf));\n      +\t\tprintf(\"%10u \", pack_name_hash_v3(line.buf));\n       \t\tprintf(\"%s\\n\", line.buf);\n       \t}\n     @@ t/t5310-pack-bitmaps.sh: test_expect_success 'name-hash value stability' '\n       \tEOF\n       \n       \ttest_cmp expect out\n     +\n     + ## t/t5616-partial-clone.sh ##\n     +@@ t/t5616-partial-clone.sh: test_expect_success 'fetch lazy-fetches only to resolve deltas' '\n     + \t# Exercise to make sure it works. Git will not fetch anything from the\n     + \t# promisor remote other than for the big tree (because it needs to\n     + \t# resolve the delta).\n     +-\tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -C client \\\n     ++\t#\n     ++\t# TODO: the --name-hash-version option is disabled here, since this test\n     ++\t# is fundamentally broken! When GIT_TEST_NAME_HASH_VERSION=3, the server\n     ++\t# recognizes delta bases in a different way and then sends a _blob_ to\n     ++\t# the client with a delta base that the client does not have! This is\n     ++\t# because the client is cloned from \"promisor-server\" with tree:0 but\n     ++\t# is now fetching from \"server\" without any filter. This is violating the\n     ++\t# promise to the server that all reachable objects exist and could be\n     ++\t# used as delta bases!\n     ++\tGIT_TRACE_PACKET=\"$(pwd)/trace\" \\\n     ++\t\tGIT_TEST_NAME_HASH_VERSION=1 \\\n     ++\t\tgit -C client \\\n     + \t\tfetch \"file://$(pwd)/server\" main &&\n     + \n     + \t# Verify the assumption that the client needed to fetch the delta base\n     +@@ t/t5616-partial-clone.sh: test_expect_success 'fetch lazy-fetches only to resolve deltas, protocol v2' '\n     + \t# Exercise to make sure it works. Git will not fetch anything from the\n     + \t# promisor remote other than for the big blob (because it needs to\n     + \t# resolve the delta).\n     +-\tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -C client \\\n     ++\t#\n     ++\t# TODO: the --name-hash-verion option is disabled here, since this test\n     ++\t# is fundamentally broken! When GIT_TEST_NAME_HASH_VERSION=3, the server\n     ++\t# recognizes delta bases in a different way and then sends a _blob_ to\n     ++\t# the client with a delta base that the client does not have! This is\n     ++\t# because the client is cloned from \"promisor-server\" with tree:0 but\n     ++\t# is now fetching from \"server\" without any filter. This is violating the\n     ++\t# promise to the server that all reachable objects exist and could be\n     ++\t# used as delta bases!\n     ++\tGIT_TRACE_PACKET=\"$(pwd)/trace\" \\\n     ++\t\tGIT_TEST_NAME_HASH_VERSION=1 \\\n     ++\t\tgit -C client \\\n     + \t\tfetch \"file://$(pwd)/server\" main &&\n     + \n     + \t# Verify that protocol version 2 was used.\n\n-- \ngitgitgadget\n"},{"id":"509429","messageId":"d035e3e59f42c75760e8dd6fe8ed6dff12bc8b9a.1734715194.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.v3.git.1734715194.gitgitgadget@gmail.com","subject":"[PATCH v3 2/8] pack-objects: add --name-hash-version option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-20T17:19:48Z","receivedAt":"2024-12-20T17:20:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nThe previous change introduced a new pack_name_hash_v2() function that\nintends to satisfy much of the hash locality features of the existing\npack_name_hash() function while also distinguishing paths with similar\nfinal components of their paths.\n\nThis change adds a new --name-hash-version option for 'git pack-objects'\nto allow users to select their preferred function version. This use of\nan integer version allows for future expansion and a direct way to later\nstore a name hash version in the .bitmap format.\n\nFor now, let's consider how effective this mechanism is when repacking a\nrepository with different name hash versions. Specifically, we will\nexecute 'git pack-objects' the same way a 'git repack -adf' process\nwould, except we include --name-hash-version=<n> for testing.\n\nOn the Git repository, we do not expect much difference. All path names\nare short. This is backed by our results:\n\n| Stage                 | Pack Size | Repack Time |\n|-----------------------|-----------|-------------|\n| After clone           | 260 MB    | N/A         |\n| --name-hash-version=1 | 127 MB    | 129s        |\n| --name-hash-version=2 | 127 MB    | 112s        |\n\nThis example demonstrates how there is some natural overhead coming from\nthe cloned copy because the server is hosting many forks and has not\noptimized for exactly this set of reachable objects. But the full repack\nhas similar characteristics for both versions.\n\nLet's consider some repositories that are hitting too many collisions\nwith version 1. First, let's explore the kinds of paths that are\ncommonly causing these collisions:\n\n * \"/CHANGELOG.json\" is 15 characters, and is created by the beachball\n   [1] tool. Only the final character of the parent directory can\n   differentiate different versions of this file, but also only the two\n   most-significant digits. If that character is a letter, then this is\n   always a collision. Similar issues occur with the similar\n   \"/CHANGELOG.md\" path, though there is more opportunity for\n   differences In the parent directory.\n\n * Localization files frequently have common filenames but\n   differentiates via parent directories. In C#, the name\n   \"/strings.resx.lcl\" is used for these localization files and they\n   will all collide in name-hash.\n\n[1] https://github.com/microsoft/beachball\n\nI've come across many other examples where some internal tool uses a\ncommon name across multiple directories and is causing Git to repack\npoorly due to name-hash collisions.\n\nOne open-source example is the fluentui [2] repo, which  uses beachball\nto generate CHANGELOG.json and CHANGELOG.md files, and these files have\nvery poor delta characteristics when comparing against versions across\nparent directories.\n\n| Stage                 | Pack Size | Repack Time |\n|-----------------------|-----------|-------------|\n| After clone           | 694 MB    | N/A         |\n| --name-hash-version=1 | 438 MB    | 728s        |\n| --name-hash-version=2 | 168 MB    | 142s        |\n\n[2] https://github.com/microsoft/fluentui\n\nIn this example, we see significant gains in the compressed packfile\nsize as well as the time taken to compute the packfile.\n\nUsing a collection of repositories that use the beachball tool, I was\nable to make similar comparisions with dramatic results. While the\nfluentui repo is public, the others are private so cannot be shared for\nreproduction. The results are so significant that I find it important to\nshare here:\n\n| Repo     | --name-hash-version=1 | --name-hash-version=2 |\n|----------|-----------------------|-----------------------|\n| fluentui |               440 MB  |               161 MB  |\n| Repo B   |             6,248 MB  |               856 MB  |\n| Repo C   |            37,278 MB  |             6,755 MB  |\n| Repo D   |           131,204 MB  |             7,463 MB  |\n\nFuture changes could include making --name-hash-version implied by a config\nvalue or even implied by default during a full repack.\n\nIt is important to point out that the name hash value is stored in the\n.bitmap file format, so we must force --name-hash-version=1 when bitmaps\nare being read or written. Later, the bitmap format could be updated to\nbe aware of the name hash version so deltas can be quickly computed\nacross the bitmapped/not-bitmapped boundary.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Documentation/git-pack-objects.txt | 32 ++++++++++++++++++-\n builtin/pack-objects.c             | 49 +++++++++++++++++++++++++++---\n t/t5300-pack-object.sh             | 31 +++++++++++++++++++\n 3 files changed, 106 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex e32404c6aae..7f69ae4855f 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -15,7 +15,8 @@ SYNOPSIS\n \t[--revs [--unpacked | --all]] [--keep-pack=<pack-name>]\n \t[--cruft] [--cruft-expiration=<time>]\n \t[--stdout [--filter=<filter-spec>] | <base-name>]\n-\t[--shallow] [--keep-true-parents] [--[no-]sparse] < <object-list>\n+\t[--shallow] [--keep-true-parents] [--[no-]sparse]\n+\t[--name-hash-version=<n>] < <object-list>\n \n \n DESCRIPTION\n@@ -345,6 +346,35 @@ raise an error.\n \tRestrict delta matches based on \"islands\". See DELTA ISLANDS\n \tbelow.\n \n+--name-hash-version=<n>::\n+\tWhile performing delta compression, Git groups objects that may be\n+\tsimilar based on heuristics using the path to that object. While\n+\tgrouping objects by an exact path match is good for paths with\n+\tmany versions, there are benefits for finding delta pairs across\n+\tdifferent full paths. Git collects objects by type and then by a\n+\t\"name hash\" of the path and then by size, hoping to group objects\n+\tthat will compress well together.\n++\n+The default name hash version is `1`, which prioritizes hash locality by\n+considering the final bytes of the path as providing the maximum magnitude\n+to the hash function. This version excels at distinguishing short paths\n+and finding renames across directories. However, the hash function depends\n+primarily on the final 16 bytes of the path. If there are many paths in\n+the repo that have the same final 16 bytes and differ only by parent\n+directory, then this name-hash may lead to too many collisions and cause\n+poor results. At the moment, this version is required when writing\n+reachability bitmap files with `--write-bitmap-index`.\n++\n+The name hash version `2` has similar locality features as version `1`,\n+except it considers each path component separately and overlays the hashes\n+with a shift. This still prioritizes the final bytes of the path, but also\n+\"salts\" the lower bits of the hash using the parent directory names. This\n+method allows for some of the locality benefits of version `1` while\n+breaking most of the collisions from a similarly-named file appearing in\n+many different directories. At the moment, this version is not allowed\n+when writing reachability bitmap files with `--write-bitmap-index` and it\n+will be automatically changed to version `1`.\n+\n \n DELTA ISLANDS\n -------------\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 08007142671..90ea19417bc 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -266,6 +266,28 @@ struct configured_exclusion {\n static struct oidmap configured_exclusions;\n \n static struct oidset excluded_by_config;\n+static int name_hash_version = 1;\n+\n+static void validate_name_hash_version(void)\n+{\n+\tif (name_hash_version < 1 || name_hash_version > 2)\n+\t\tdie(_(\"invalid --name-hash-version option: %d\"), name_hash_version);\n+}\n+\n+static inline uint32_t pack_name_hash_fn(const char *name)\n+{\n+\tswitch (name_hash_version)\n+\t{\n+\tcase 1:\n+\t\treturn pack_name_hash(name);\n+\n+\tcase 2:\n+\t\treturn pack_name_hash_v2((const unsigned char *)name);\n+\n+\tdefault:\n+\t\tBUG(\"invalid name-hash version: %d\", name_hash_version);\n+\t}\n+}\n \n /*\n  * stats\n@@ -1698,7 +1720,7 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n \t\treturn 0;\n \t}\n \n-\tcreate_object_entry(oid, type, pack_name_hash(name),\n+\tcreate_object_entry(oid, type, pack_name_hash_fn(name),\n \t\t\t    exclude, name && no_try_delta(name),\n \t\t\t    found_pack, found_offset);\n \treturn 1;\n@@ -1912,7 +1934,7 @@ static void add_preferred_base_object(const char *name)\n {\n \tstruct pbase_tree *it;\n \tsize_t cmplen;\n-\tunsigned hash = pack_name_hash(name);\n+\tunsigned hash = pack_name_hash_fn(name);\n \n \tif (!num_preferred_base || check_pbase_path(hash))\n \t\treturn;\n@@ -3422,7 +3444,7 @@ static void show_object_pack_hint(struct object *object, const char *name,\n \t * here using a now in order to perhaps improve the delta selection\n \t * process.\n \t */\n-\toe->hash = pack_name_hash(name);\n+\toe->hash = pack_name_hash_fn(name);\n \toe->no_try_delta = name && no_try_delta(name);\n \n \tstdin_packs_hints_nr++;\n@@ -3572,7 +3594,7 @@ static void add_cruft_object_entry(const struct object_id *oid, enum object_type\n \tentry = packlist_find(&to_pack, oid);\n \tif (entry) {\n \t\tif (name) {\n-\t\t\tentry->hash = pack_name_hash(name);\n+\t\t\tentry->hash = pack_name_hash_fn(name);\n \t\t\tentry->no_try_delta = no_try_delta(name);\n \t\t}\n \t} else {\n@@ -3595,7 +3617,7 @@ static void add_cruft_object_entry(const struct object_id *oid, enum object_type\n \t\t\treturn;\n \t\t}\n \n-\t\tentry = create_object_entry(oid, type, pack_name_hash(name),\n+\t\tentry = create_object_entry(oid, type, pack_name_hash_fn(name),\n \t\t\t\t\t    0, name && no_try_delta(name),\n \t\t\t\t\t    pack, offset);\n \t}\n@@ -4069,6 +4091,15 @@ static int get_object_list_from_bitmap(struct rev_info *revs)\n \tif (!(bitmap_git = prepare_bitmap_walk(revs, 0)))\n \t\treturn -1;\n \n+\t/*\n+\t * For now, force the name-hash version to be 1 since that\n+\t * is the version implied by the bitmap format. Later, the\n+\t * format can include this version explicitly in its format,\n+\t * allowing readers to know the version that was used during\n+\t * the bitmap write.\n+\t */\n+\tname_hash_version = 1;\n+\n \tif (pack_options_allow_reuse())\n \t\treuse_partial_packfile_from_bitmap(bitmap_git,\n \t\t\t\t\t\t   &reuse_packfiles,\n@@ -4429,6 +4460,8 @@ int cmd_pack_objects(int argc,\n \t\tOPT_STRING_LIST(0, \"uri-protocol\", &uri_protocols,\n \t\t\t\tN_(\"protocol\"),\n \t\t\t\tN_(\"exclude any configured uploadpack.blobpackfileuri with this protocol\")),\n+\t\tOPT_INTEGER(0, \"name-hash-version\", &name_hash_version,\n+\t\t\t N_(\"use the specified name-hash function to group similar objects\")),\n \t\tOPT_END(),\n \t};\n \n@@ -4576,6 +4609,12 @@ int cmd_pack_objects(int argc,\n \tif (pack_to_stdout || !rev_list_all)\n \t\twrite_bitmap_index = 0;\n \n+\tvalidate_name_hash_version();\n+\tif (write_bitmap_index && name_hash_version != 1) {\n+\t\twarning(_(\"currently, --write-bitmap-index requires --name-hash-version=1\"));\n+\t\tname_hash_version = 1;\n+\t}\n+\n \tif (use_delta_islands)\n \t\tstrvec_push(&rp, \"--topo-order\");\n \ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 3b9dae331a5..4270eabe8b7 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -674,4 +674,35 @@ do\n \t'\n done\n \n+test_expect_success 'valid and invalid --name-hash-versions' '\n+\t# Valid values are hard to verify other than \"do not fail\".\n+\t# Performance tests will be more valuable to validate these versions.\n+\tfor value in 1 2\n+\tdo\n+\t\tgit pack-objects base --all --name-hash-version=$value || return 1\n+\tdone &&\n+\n+\t# Invalid values have clear post-conditions.\n+\tfor value in -1 0 3\n+\tdo\n+\t\ttest_must_fail git pack-objects base --all --name-hash-version=$value 2>err &&\n+\t\ttest_grep \"invalid --name-hash-version option\" err || return 1\n+\tdone\n+'\n+\n+# The following test is not necessarily a permanent choice, but since we do not\n+# have a \"name hash version\" bit in the .bitmap file format, we cannot write the\n+# hash values into the .bitmap file without risking breakage later.\n+#\n+# TODO: Make these compatible in the future and replace this test with the\n+# expected behavior when both are specified.\n+test_expect_success '--name-hash-version=2 and --write-bitmap-index are incompatible' '\n+\tgit pack-objects base --all --name-hash-version=2 --write-bitmap-index 2>err &&\n+\ttest_grep \"currently, --write-bitmap-index requires --name-hash-version=1\" err &&\n+\n+\t# --stdout option silently removes --write-bitmap-index\n+\tgit pack-objects --stdout --all --name-hash-version=2 --write-bitmap-index >out 2>err &&\n+\t! test_grep \"currently, --write-bitmap-index requires --name-hash-version=1\" err\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"509430","messageId":"e2191244f6b21792f2551946a89cfc48af3989c5.1734715194.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.v3.git.1734715194.gitgitgadget@gmail.com","subject":"[PATCH v3 3/8] repack: add --name-hash-version option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-20T17:19:49Z","receivedAt":"2024-12-20T17:20:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nThe new '--name-hash-version' option for 'git repack' is a simple\npass-through to the underlying 'git pack-objects' subcommand. However,\nthis subcommand may have other options and a temporary filename as part\nof the subcommand execution that may not be predictable or could change\nover time.\n\nThe existing test_subcommand method requires an exact list of arguments\nfor the subcommand. This is too rigid for our needs here, so create a\nnew method, test_subcommand_flex. Use it to check that the\n--name-hash-version option is passing through.\n\nSince we are modifying the 'git repack' command, let's bring its usage\nin line with the Documentation's synopsis. This removes it from the\nallow list in t0450 so it will remain in sync in the future.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Documentation/git-repack.txt |  9 ++++++++-\n builtin/repack.c             |  9 ++++++++-\n t/t0450/txt-help-mismatches  |  1 -\n t/t7700-repack.sh            |  6 ++++++\n t/test-lib-functions.sh      | 26 ++++++++++++++++++++++++++\n 5 files changed, 48 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex c902512a9e8..5852a5c9736 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -9,7 +9,9 @@ git-repack - Pack unpacked objects in a repository\n SYNOPSIS\n --------\n [verse]\n-'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m] [--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>] [--write-midx]\n+'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\n+\t[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\n+\t[--write-midx] [--name-hash-version=<n>]\n \n DESCRIPTION\n -----------\n@@ -249,6 +251,11 @@ linkgit:git-multi-pack-index[1]).\n \tWrite a multi-pack index (see linkgit:git-multi-pack-index[1])\n \tcontaining the non-redundant packs.\n \n+--name-hash-version=<n>::\n+\tProvide this argument to the underlying `git pack-objects` process.\n+\tSee linkgit:git-pack-objects[1] for full details.\n+\n+\n CONFIGURATION\n -------------\n \ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex d6bb37e84ae..5e7ff919c1a 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -39,7 +39,9 @@ static int run_update_server_info = 1;\n static char *packdir, *packtmp_name, *packtmp;\n \n static const char *const git_repack_usage[] = {\n-\tN_(\"git repack [<options>]\"),\n+\tN_(\"git repack [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\\n\"\n+\t   \"[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\\n\"\n+\t   \"[--write-midx] [--name-hash-version=<n>]\"),\n \tNULL\n };\n \n@@ -58,6 +60,7 @@ struct pack_objects_args {\n \tint no_reuse_object;\n \tint quiet;\n \tint local;\n+\tint name_hash_version;\n \tstruct list_objects_filter_options filter_options;\n };\n \n@@ -306,6 +309,8 @@ static void prepare_pack_objects(struct child_process *cmd,\n \t\tstrvec_pushf(&cmd->args, \"--no-reuse-delta\");\n \tif (args->no_reuse_object)\n \t\tstrvec_pushf(&cmd->args, \"--no-reuse-object\");\n+\tif (args->name_hash_version)\n+\t\tstrvec_pushf(&cmd->args, \"--name-hash-version=%d\", args->name_hash_version);\n \tif (args->local)\n \t\tstrvec_push(&cmd->args,  \"--local\");\n \tif (args->quiet)\n@@ -1203,6 +1208,8 @@ int cmd_repack(int argc,\n \t\t\t\tN_(\"pass --no-reuse-delta to git-pack-objects\")),\n \t\tOPT_BOOL('F', NULL, &po_args.no_reuse_object,\n \t\t\t\tN_(\"pass --no-reuse-object to git-pack-objects\")),\n+\t\tOPT_INTEGER(0, \"name-hash-version\", &po_args.name_hash_version,\n+\t\t\t\tN_(\"specify the name hash version to use for grouping similar objects by path\")),\n \t\tOPT_NEGBIT('n', NULL, &run_update_server_info,\n \t\t\t\tN_(\"do not run git-update-server-info\"), 1),\n \t\tOPT__QUIET(&po_args.quiet, N_(\"be quiet\")),\ndiff --git a/t/t0450/txt-help-mismatches b/t/t0450/txt-help-mismatches\nindex 28003f18c92..c4a15fd0cb8 100644\n--- a/t/t0450/txt-help-mismatches\n+++ b/t/t0450/txt-help-mismatches\n@@ -45,7 +45,6 @@ rebase\n remote\n remote-ext\n remote-fd\n-repack\n reset\n restore\n rev-parse\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex c4c3d1a15d9..b9a5759e01d 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -777,6 +777,12 @@ test_expect_success 'repack -ad cleans up old .tmp-* packs' '\n \ttest_must_be_empty tmpfiles\n '\n \n+test_expect_success '--name-hash-version option passes through to pack-objects' '\n+\tGIT_TRACE2_EVENT=\"$(pwd)/hash-trace.txt\" \\\n+\t\tgit repack -a --name-hash-version=2 &&\n+\ttest_subcommand_flex git pack-objects --name-hash-version=2 <hash-trace.txt\n+'\n+\n test_expect_success 'setup for update-server-info' '\n \tgit init update-server-info &&\n \ttest_commit -C update-server-info message\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 78e054ab503..af47247f25f 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1886,6 +1886,32 @@ test_subcommand () {\n \tfi\n }\n \n+# Check that the given subcommand was run with the given set of\n+# arguments in order (but with possible extra arguments).\n+#\n+#\ttest_subcommand_flex [!] <command> <args>... < <trace>\n+#\n+# If the first parameter passed is !, this instead checks that\n+# the given command was not called.\n+#\n+test_subcommand_flex () {\n+\tlocal negate=\n+\tif test \"$1\" = \"!\"\n+\tthen\n+\t\tnegate=t\n+\t\tshift\n+\tfi\n+\n+\tlocal expr=\"$(printf '\"%s\".*' \"$@\")\"\n+\n+\tif test -n \"$negate\"\n+\tthen\n+\t\t! grep \"\\[$expr\\]\"\n+\telse\n+\t\tgrep \"\\[$expr\\]\"\n+\tfi\n+}\n+\n # Check that the given command was invoked as part of the\n # trace2-format trace on stdin.\n #\n-- \ngitgitgadget\n\n"},{"id":"509431","messageId":"86ff0d0a15e4263ccd541a9b8dcdb99438784a70.1734715194.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.v3.git.1734715194.gitgitgadget@gmail.com","subject":"[PATCH v3 4/8] pack-objects: add GIT_TEST_NAME_HASH_VERSION","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-20T17:19:50Z","receivedAt":"2024-12-20T17:20:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nAdd a new environment variable to opt-in to different values of the\n--name-hash-version=<n> option in 'git pack-objects'. This allows for\nextra testing of the feature without repeating all of the test\nscenarios. Unlike many GIT_TEST_* variables, we are choosing to not add\nthis to the linux-TEST-vars CI build as that test run is already\noverloaded. The behavior exposed by this test variable is of low risk\nand should be sufficient to allow manual testing when an issue arises.\n\nBut this option isn't free. There are a few tests that change behavior\nwith the variable enabled.\n\nFirst, there are a few tests that are very sensitive to certain delta\nbases being picked. These are both involving the generation of thin\nbundles and then counting their objects via 'git index-pack --fix-thin'\nwhich pulls the delta base into the new packfile. For these tests,\ndisable the option as a decent long-term option.\n\nSecond, there are some tests that compare the exact output of a 'git\npack-objects' process when using bitmaps. The warning that ignores the\n--name-hash-version=2 and forces version 1 causes these tests to fail.\nDisable the environment variable to get around this issue.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n builtin/pack-objects.c          |  5 ++++-\n t/README                        |  4 ++++\n t/t5300-pack-object.sh          |  7 +++++--\n t/t5310-pack-bitmaps.sh         |  5 ++++-\n t/t5333-pseudo-merge-bitmaps.sh |  4 ++++\n t/t5510-fetch.sh                |  7 ++++++-\n t/t6020-bundle-misc.sh          |  6 +++++-\n t/t7406-submodule-update.sh     |  4 +++-\n t/t7700-repack.sh               | 10 ++++++++--\n 9 files changed, 43 insertions(+), 9 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 90ea19417bc..b19c3665003 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -266,7 +266,7 @@ struct configured_exclusion {\n static struct oidmap configured_exclusions;\n \n static struct oidset excluded_by_config;\n-static int name_hash_version = 1;\n+static int name_hash_version = -1;\n \n static void validate_name_hash_version(void)\n {\n@@ -4609,6 +4609,9 @@ int cmd_pack_objects(int argc,\n \tif (pack_to_stdout || !rev_list_all)\n \t\twrite_bitmap_index = 0;\n \n+\tif (name_hash_version < 0)\n+\t\tname_hash_version = (int)git_env_ulong(\"GIT_TEST_NAME_HASH_VERSION\", 1);\n+\n \tvalidate_name_hash_version();\n \tif (write_bitmap_index && name_hash_version != 1) {\n \t\twarning(_(\"currently, --write-bitmap-index requires --name-hash-version=1\"));\ndiff --git a/t/README b/t/README\nindex 8c0319b58e5..e63d2360852 100644\n--- a/t/README\n+++ b/t/README\n@@ -492,6 +492,10 @@ a test and then fails then the whole test run will abort. This can help to make\n sure the expected tests are executed and not silently skipped when their\n dependency breaks or is simply not present in a new environment.\n \n+GIT_TEST_NAME_HASH_VERSION=<int>, when set, causes 'git pack-objects' to\n+assume '--name-hash-version=<n>'.\n+\n+\n Naming Tests\n ------------\n \ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 4270eabe8b7..97fe9e561c6 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -675,15 +675,18 @@ do\n done\n \n test_expect_success 'valid and invalid --name-hash-versions' '\n+\tsane_unset GIT_TEST_NAME_HASH_VERSION &&\n+\n \t# Valid values are hard to verify other than \"do not fail\".\n \t# Performance tests will be more valuable to validate these versions.\n-\tfor value in 1 2\n+\t# Negative values are converted to version 1.\n+\tfor value in -1 1 2\n \tdo\n \t\tgit pack-objects base --all --name-hash-version=$value || return 1\n \tdone &&\n \n \t# Invalid values have clear post-conditions.\n-\tfor value in -1 0 3\n+\tfor value in 0 3\n \tdo\n \t\ttest_must_fail git pack-objects base --all --name-hash-version=$value 2>err &&\n \t\ttest_grep \"invalid --name-hash-version option\" err || return 1\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 7044c7d7c6d..c30522b57fd 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -420,7 +420,10 @@ test_bitmap_cases () {\n \t\t\tcat >expect <<-\\EOF &&\n \t\t\terror: missing value for '\\''pack.preferbitmaptips'\\''\n \t\t\tEOF\n-\t\t\tgit repack -adb 2>actual &&\n+\n+\t\t\t# Disable name hash version adjustment due to stderr comparison.\n+\t\t\tGIT_TEST_NAME_HASH_VERSION=1 \\\n+\t\t\t\tgit repack -adb 2>actual &&\n \t\t\ttest_cmp expect actual\n \t\t)\n \t'\ndiff --git a/t/t5333-pseudo-merge-bitmaps.sh b/t/t5333-pseudo-merge-bitmaps.sh\nindex eca4a1eb8c6..b1553cbaf7f 100755\n--- a/t/t5333-pseudo-merge-bitmaps.sh\n+++ b/t/t5333-pseudo-merge-bitmaps.sh\n@@ -209,6 +209,10 @@ test_expect_success 'bitmapPseudoMerge.stableThreshold creates stable groups' '\n '\n \n test_expect_success 'out of order thresholds are rejected' '\n+\t# Disable this option to avoid stderr message\n+\tGIT_TEST_NAME_HASH_VERSION=1 &&\n+\texport GIT_TEST_NAME_HASH_VERSION &&\n+\n \ttest_must_fail git \\\n \t\t-c bitmapPseudoMerge.test.pattern=\"refs/*\" \\\n \t\t-c bitmapPseudoMerge.test.threshold=1.month.ago \\\ndiff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\nindex 0890b9f61c5..1699c3a3bb8 100755\n--- a/t/t5510-fetch.sh\n+++ b/t/t5510-fetch.sh\n@@ -1062,7 +1062,12 @@ test_expect_success 'all boundary commits are excluded' '\n \ttest_tick &&\n \tgit merge otherside &&\n \tad=$(git log --no-walk --format=%ad HEAD) &&\n-\tgit bundle create twoside-boundary.bdl main --since=\"$ad\" &&\n+\n+\t# If the a different name hash function is used here, then no delta\n+\t# pair is found and the bundle does not expand to three objects\n+\t# when fixing the thin object.\n+\tGIT_TEST_NAME_HASH_VERSION=1 \\\n+\t\tgit bundle create twoside-boundary.bdl main --since=\"$ad\" &&\n \ttest_bundle_object_count --thin twoside-boundary.bdl 3\n '\n \ndiff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\nindex 34b5cd62c20..a1f18ae71f1 100755\n--- a/t/t6020-bundle-misc.sh\n+++ b/t/t6020-bundle-misc.sh\n@@ -247,7 +247,11 @@ test_expect_success 'create bundle with --since option' '\n \tEOF\n \ttest_cmp expect actual &&\n \n-\tgit bundle create since.bdl \\\n+\t# If a different name hash function is used, then one fewer\n+\t# delta base is found and this counts a different number\n+\t# of objects after performing --fix-thin.\n+\tGIT_TEST_NAME_HASH_VERSION=1 \\\n+\t\tgit bundle create since.bdl \\\n \t\t--since \"Thu Apr 7 15:27:00 2005 -0700\" \\\n \t\t--all &&\n \ndiff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh\nindex 0f0c86f9cb2..ebd9941075a 100755\n--- a/t/t7406-submodule-update.sh\n+++ b/t/t7406-submodule-update.sh\n@@ -1094,7 +1094,9 @@ test_expect_success 'submodule update --quiet passes quietness to fetch with a s\n \t) &&\n \tgit clone super4 super5 &&\n \t(cd super5 &&\n-\t git submodule update --quiet --init --depth=1 submodule3 >out 2>err &&\n+\t # This test var can mess with the stderr output checked in this test.\n+\t GIT_TEST_NAME_HASH_VERSION=1 \\\n+\t\tgit submodule update --quiet --init --depth=1 submodule3 >out 2>err &&\n \t test_must_be_empty out &&\n \t test_must_be_empty err\n \t) &&\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex b9a5759e01d..16861f80c9c 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -309,7 +309,10 @@ test_expect_success 'no bitmaps created if .keep files present' '\n \tkeep=${pack%.pack}.keep &&\n \ttest_when_finished \"rm -f \\\"\\$keep\\\"\" &&\n \t>\"$keep\" &&\n-\tgit -C bare.git repack -ad 2>stderr &&\n+\n+\t# Disable --name-hash-version test due to stderr comparison.\n+\tGIT_TEST_NAME_HASH_VERSION=1 \\\n+\t\tgit -C bare.git repack -ad 2>stderr &&\n \ttest_must_be_empty stderr &&\n \tfind bare.git/objects/pack/ -type f -name \"*.bitmap\" >actual &&\n \ttest_must_be_empty actual\n@@ -320,7 +323,10 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '\n \tblob=$(test-tool genrandom big $((1024*1024)) |\n \t       git -C bare.git hash-object -w --stdin) &&\n \tgit -C bare.git update-ref refs/tags/big $blob &&\n-\tgit -C bare.git repack -ad 2>stderr &&\n+\n+\t# Disable --name-hash-version test due to stderr comparison.\n+\tGIT_TEST_NAME_HASH_VERSION=1 \\\n+\t\tgit -C bare.git repack -ad 2>stderr &&\n \ttest_must_be_empty stderr &&\n \tfind bare.git/objects/pack -type f -name \"*.bitmap\" >actual &&\n \ttest_must_be_empty actual\n-- \ngitgitgadget\n\n"},{"id":"509432","messageId":"163aaab3e1bec5bf92e4e056df84aa76848b31a4.1734715194.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.v3.git.1734715194.gitgitgadget@gmail.com","subject":"[PATCH v3 5/8] p5313: add size comparison test","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-20T17:19:51Z","receivedAt":"2024-12-20T17:20:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nAs custom options are added to 'git pack-objects' and 'git repack' to\nadjust how compression is done, use this new performance test script to\ndemonstrate their effectiveness in performance and size.\n\nThe recently-added --name-hash-version option allows for testing\ndifferent name hash functions. Version 2 intends to preserve some of the\nlocality of version 1 while more often breaking collisions due to long\nfilenames.\n\nDistinguishing objects by more of the path is critical when there are\nmany name hash collisions and several versions of the same path in the\nfull history, giving a significant boost to the full repack case. The\nlocality of the hash function is critical to compressing something like\na shallow clone or a thin pack representing a push of a single commit.\n\nThis can be seen by running pt5313 on the open source fluentui\nrepository [1]. Most commits will have this kind of output for the thin\nand big pack cases, though certain commits (such as [2]) will have\nproblematic thin pack size for other reasons.\n\n[1] https://github.com/microsoft/fluentui\n[2] a637a06df05360ce5ff21420803f64608226a875\n\nChecked out at the parent of [2], I see the following statistics:\n\nTest                                         HEAD\n---------------------------------------------------------------\n5313.2: thin pack with version 1             0.37(0.44+0.02)\n5313.3: thin pack size with version 1                   1.2M\n5313.4: big pack with version 1              2.04(7.77+0.23)\n5313.5: big pack size with version 1                   20.4M\n5313.6: shallow fetch pack with version 1    1.41(2.94+0.11)\n5313.7: shallow pack size with version 1               34.4M\n5313.8: repack with version 1                95.70(676.41+2.87)\n5313.9: repack size with version 1                    439.3M\n5313.10: thin pack with version 2            0.12(0.12+0.06)\n5313.11: thin pack size with version 2                 22.0K\n5313.12: big pack with version 2             2.80(5.43+0.34)\n5313.13: big pack size with version 2                  25.9M\n5313.14: shallow fetch pack with version 2   1.77(2.80+0.19)\n5313.15: shallow pack size with version 2              33.7M\n5313.16: repack with version 2               33.68(139.52+2.58)\n5313.17: repack size with version 2                   160.5M\n\nTo make comparisons easier, I will reformat this output into a different\ntable style:\n\n| Test         | V1 Time | V2 Time | V1 Size | V2 Size |\n|--------------|---------|---------|---------|---------|\n| Thin Pack    |  0.37 s |  0.12 s |   1.2 M |  22.0 K |\n| Big Pack     |  2.04 s |  2.80 s |  20.4 M |  25.9 M |\n| Shallow Pack |  1.41 s |  1.77 s |  34.4 M |  33.7 M |\n| Repack       | 95.70 s | 33.68 s | 439.3 M | 160.5 M |\n\nThe v2 hash function successfully differentiates the CHANGELOG.md files\nfrom each other, which leads to significant improvements in the thin\npack (simulating a push of this commit) and the full repack. There is\nsome bloat in the \"big pack\" scenario and essentially the same results\nfor the shallow pack.\n\nIn the case of the Git repository, these numbers show some of the issues\nwith this approach:\n\n| Test         | V1 Time | V2 Time | V1 Size | V2 Size |\n|--------------|---------|---------|---------|---------|\n| Thin Pack    |  0.02 s |  0.02 s |   1.1 K |   1.1 K |\n| Big Pack     |  1.69 s |  1.95 s |  13.5 M |  14.5 M |\n| Shallow Pack |  1.26 s |  1.29 s |  12.0 M |  12.2 M |\n| Repack       | 29.51 s | 29.01 s | 237.7 M | 238.2 M |\n\nHere, the attempts to remove conflicts in the v2 function seem to cause\nslight bloat to these sizes. This shows that the Git repository benefits\na lot from cross-path delta pairs.\n\nThe results are similar with the nodejs/node repo:\n\n| Test         | V1 Time | V2 Time | V1 Size | V2 Size |\n|--------------|---------|---------|---------|---------|\n| Thin Pack    |  0.02 s |  0.02 s |   1.6 K |   1.6 K |\n| Big Pack     |  4.61 s |  3.26 s |  56.0 M |  52.8 M |\n| Shallow Pack |  7.82 s |  7.51 s | 104.6 M | 107.0 M |\n| Repack       | 88.90 s | 73.75 s | 740.1 M | 764.5 M |\n\nHere, the v2 name-hash causes some size bloat more often than it reduces\nthe size, but it also universally improves performance time, which is an\ninteresting reversal. This must mean that it is helping to short-circuit\nsome delta computations even if it is not finding the most efficient\nones. The performance improvement cannot be explained only due to the\nI/O cost of writing the resulting packfile.\n\nThe Linux kernel repository was the initial target of the default name\nhash value, and its naming conventions are practically build to take the\nmost advantage of the default name hash values:\n\n| Test         | V1 Time  | V2 Time  | V1 Size | V2 Size |\n|--------------|----------|----------|---------|---------|\n| Thin Pack    |   0.17 s |   0.07 s |   4.6 K |   4.6 K |\n| Big Pack     |  17.88 s |  12.35 s | 201.1 M | 159.1 M |\n| Shallow Pack |  11.05 s |  22.94 s | 269.2 M | 273.8 M |\n| Repack       | 727.39 s | 566.95 s |   2.5 G |   2.5 G |\n\nHere, the thin and big packs gain some performance boosts in time, with\na modest gain in the size of the big pack. The shallow pack, however, is\nmore expensive to compute, likely because similarly-named files across\ndifferent directories are farther apart in the name hash ordering in v2.\nThe repack also gains benefits in computation time but no meaningful\nchange to the full size.\n\nFinally, an internal Javascript repo of moderate size shows significant\ngains when repacking with --name-hash-version=2 due to it having many name\nhash collisions. However, it's worth noting that only the full repack\ncase has significant differences from the v1 name hash:\n\n| Test      | V1 Time   | V2 Time  | V1 Size | V2 Size |\n|-----------|-----------|----------|---------|---------|\n| Thin Pack |    8.28 s |   7.28 s |  16.8 K |  16.8 K |\n| Big Pack  |   12.81 s |  11.66 s |  29.1 M |  29.1 M |\n| Shallow   |    4.86 s |   4.06 s |  42.5 M |  44.1 M |\n| Repack    | 3126.50 s | 496.33 s |   6.2 G | 855.6 M |\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n t/perf/p5313-pack-objects.sh | 70 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 70 insertions(+)\n create mode 100755 t/perf/p5313-pack-objects.sh\n\ndiff --git a/t/perf/p5313-pack-objects.sh b/t/perf/p5313-pack-objects.sh\nnew file mode 100755\nindex 00000000000..be5229a0ecd\n--- /dev/null\n+++ b/t/perf/p5313-pack-objects.sh\n@@ -0,0 +1,70 @@\n+#!/bin/sh\n+\n+test_description='Tests pack performance using bitmaps'\n+. ./perf-lib.sh\n+\n+GIT_TEST_PASSING_SANITIZE_LEAK=0\n+export GIT_TEST_PASSING_SANITIZE_LEAK\n+\n+test_perf_large_repo\n+\n+test_expect_success 'create rev input' '\n+\tcat >in-thin <<-EOF &&\n+\t$(git rev-parse HEAD)\n+\t^$(git rev-parse HEAD~1)\n+\tEOF\n+\n+\tcat >in-big <<-EOF &&\n+\t$(git rev-parse HEAD)\n+\t^$(git rev-parse HEAD~1000)\n+\tEOF\n+\n+\tcat >in-shallow <<-EOF\n+\t$(git rev-parse HEAD)\n+\t--shallow $(git rev-parse HEAD)\n+\tEOF\n+'\n+\n+for version in 1 2\n+do\n+\texport version\n+\n+\ttest_perf \"thin pack with version $version\" '\n+\t\tgit pack-objects --thin --stdout --revs --sparse \\\n+\t\t\t--name-hash-version=$version <in-thin >out\n+\t'\n+\n+\ttest_size \"thin pack size with version $version\" '\n+\t\ttest_file_size out\n+\t'\n+\n+\ttest_perf \"big pack with version $version\" '\n+\t\tgit pack-objects --stdout --revs --sparse \\\n+\t\t\t--name-hash-version=$version <in-big >out\n+\t'\n+\n+\ttest_size \"big pack size with version $version\" '\n+\t\ttest_file_size out\n+\t'\n+\n+\ttest_perf \"shallow fetch pack with version $version\" '\n+\t\tgit pack-objects --stdout --revs --sparse --shallow \\\n+\t\t\t--name-hash-version=$version <in-shallow >out\n+\t'\n+\n+\ttest_size \"shallow pack size with version $version\" '\n+\t\ttest_file_size out\n+\t'\n+\n+\ttest_perf \"repack with version $version\" '\n+\t\tgit repack -adf --name-hash-version=$version\n+\t'\n+\n+\ttest_size \"repack size with version $version\" '\n+\t\tgitdir=$(git rev-parse --git-dir) &&\n+\t\tpack=$(ls $gitdir/objects/pack/pack-*.pack) &&\n+\t\ttest_file_size \"$pack\"\n+\t'\n+done\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"509433","messageId":"e9ce79fa6e770688f3fd14ca1c19bca185a81bbf.1734715194.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.v3.git.1734715194.gitgitgadget@gmail.com","subject":"[PATCH v3 6/8] test-tool: add helper for name-hash values","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-20T17:19:52Z","receivedAt":"2024-12-20T17:20:06Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nAdd a new test-tool helper, name-hash, to output the value of the\nname-hash algorithms for the input list of strings, one per line.\n\nSince the name-hash values can be stored in the .bitmap files, it is\nimportant that these hash functions do not change across Git versions.\nAdd a simple test to t5310-pack-bitmaps.sh to provide some testing of\nthe current values. Due to how these functions are implemented, it would\nbe difficult to change them without disturbing these values. The paths\nused for this test are carefully selected to demonstrate some of the\nbehavior differences of the two current name hash versions, including\nwhich conditions will cause them to collide.\n\nCreate a performance test that uses test_size to demonstrate how\ncollisions occur for these hash algorithms. This test helps inform\nsomeone as to the behavior of the name-hash algorithms for their repo\nbased on the paths at HEAD.\n\nMy copy of the Git repository shows modest statistics around the\ncollisions of the default name-hash algorithm:\n\nTest                               this tree\n--------------------------------------------------\n5314.1: paths at head                         4.5K\n5314.2: distinct hash value: v1               4.1K\n5314.3: maximum multiplicity: v1                13\n5314.4: distinct hash value: v2               4.2K\n5314.5: maximum multiplicity: v2                 9\n\nHere, the maximum collision multiplicity is 13, but around 10% of paths\nhave a collision with another path.\n\nIn a more interesting example, the microsoft/fluentui [1] repo had these\nstatistics at time of committing:\n\nTest                               this tree\n--------------------------------------------------\n5314.1: paths at head                        19.5K\n5314.2: distinct hash value: v1               8.2K\n5314.3: maximum multiplicity: v1               279\n5314.4: distinct hash value: v2              17.8K\n5314.5: maximum multiplicity: v2                44\n\n[1] https://github.com/microsoft/fluentui\n\nThat demonstrates that of the nearly twenty thousand path names, they\nare assigned around eight thousand distinct values. 279 paths are\nassigned to a single value, leading the packing algorithm to sort\nobjects from those paths together, by size.\n\nWith the v2 name hash function, the maximum multiplicity lowers to 44,\nleaving some room for further improvement.\n\nIn a more extreme example, an internal monorepo had a much worse\ncollision rate:\n\nTest                               this tree\n--------------------------------------------------\n5314.1: paths at head                       227.3K\n5314.2: distinct hash value: v1              72.3K\n5314.3: maximum multiplicity: v1             14.4K\n5314.4: distinct hash value: v2             166.5K\n5314.5: maximum multiplicity: v2               138\n\nHere, we can see that the v2 name hash function provides somem\nimprovements, but there are still a number of collisions that could lead\nto repacking problems at this scale.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Makefile                  |  1 +\n t/helper/test-name-hash.c | 23 +++++++++++++++++++++++\n t/helper/test-tool.c      |  1 +\n t/helper/test-tool.h      |  1 +\n t/perf/p5314-name-hash.sh | 31 +++++++++++++++++++++++++++++++\n t/t5310-pack-bitmaps.sh   | 30 ++++++++++++++++++++++++++++++\n 6 files changed, 87 insertions(+)\n create mode 100644 t/helper/test-name-hash.c\n create mode 100755 t/perf/p5314-name-hash.sh\n\ndiff --git a/Makefile b/Makefile\nindex 6f5986b66ea..65403f6dd09 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -816,6 +816,7 @@ TEST_BUILTINS_OBJS += test-lazy-init-name-hash.o\n TEST_BUILTINS_OBJS += test-match-trees.o\n TEST_BUILTINS_OBJS += test-mergesort.o\n TEST_BUILTINS_OBJS += test-mktemp.o\n+TEST_BUILTINS_OBJS += test-name-hash.o\n TEST_BUILTINS_OBJS += test-online-cpus.o\n TEST_BUILTINS_OBJS += test-pack-mtimes.o\n TEST_BUILTINS_OBJS += test-parse-options.o\ndiff --git a/t/helper/test-name-hash.c b/t/helper/test-name-hash.c\nnew file mode 100644\nindex 00000000000..af1d52de101\n--- /dev/null\n+++ b/t/helper/test-name-hash.c\n@@ -0,0 +1,23 @@\n+/*\n+ * test-name-hash.c: Read a list of paths over stdin and report on their\n+ * name-hash and full name-hash.\n+ */\n+\n+#include \"test-tool.h\"\n+#include \"git-compat-util.h\"\n+#include \"pack-objects.h\"\n+#include \"strbuf.h\"\n+\n+int cmd__name_hash(int argc UNUSED, const char **argv UNUSED)\n+{\n+\tstruct strbuf line = STRBUF_INIT;\n+\n+\twhile (!strbuf_getline(&line, stdin)) {\n+\t\tprintf(\"%10u \", pack_name_hash(line.buf));\n+\t\tprintf(\"%10u \", pack_name_hash_v2((unsigned const char *)line.buf));\n+\t\tprintf(\"%s\\n\", line.buf);\n+\t}\n+\n+\tstrbuf_release(&line);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 1ebb69a5dc4..e794058ab6d 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -44,6 +44,7 @@ static struct test_cmd cmds[] = {\n \t{ \"match-trees\", cmd__match_trees },\n \t{ \"mergesort\", cmd__mergesort },\n \t{ \"mktemp\", cmd__mktemp },\n+\t{ \"name-hash\", cmd__name_hash },\n \t{ \"online-cpus\", cmd__online_cpus },\n \t{ \"pack-mtimes\", cmd__pack_mtimes },\n \t{ \"parse-options\", cmd__parse_options },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 21802ac27da..26ff30a5a9a 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -37,6 +37,7 @@ int cmd__lazy_init_name_hash(int argc, const char **argv);\n int cmd__match_trees(int argc, const char **argv);\n int cmd__mergesort(int argc, const char **argv);\n int cmd__mktemp(int argc, const char **argv);\n+int cmd__name_hash(int argc, const char **argv);\n int cmd__online_cpus(int argc, const char **argv);\n int cmd__pack_mtimes(int argc, const char **argv);\n int cmd__parse_options(int argc, const char **argv);\ndiff --git a/t/perf/p5314-name-hash.sh b/t/perf/p5314-name-hash.sh\nnew file mode 100755\nindex 00000000000..4ef0ba77114\n--- /dev/null\n+++ b/t/perf/p5314-name-hash.sh\n@@ -0,0 +1,31 @@\n+#!/bin/sh\n+\n+test_description='Tests pack performance using bitmaps'\n+. ./perf-lib.sh\n+\n+GIT_TEST_PASSING_SANITIZE_LEAK=0\n+export GIT_TEST_PASSING_SANITIZE_LEAK\n+\n+test_perf_large_repo\n+\n+test_size 'paths at head' '\n+\tgit ls-tree -r --name-only HEAD >path-list &&\n+\twc -l <path-list &&\n+\ttest-tool name-hash <path-list >name-hashes\n+'\n+\n+for version in 1 2\n+do\n+\ttest_size \"distinct hash value: v$version\" '\n+\t\tawk \"{ print \\$$version; }\" <name-hashes | sort | \\\n+\t\t\tuniq -c >name-hash-count &&\n+\t\twc -l <name-hash-count\n+\t'\n+\n+\ttest_size \"maximum multiplicity: v$version\" '\n+\t\tsort -nr <name-hash-count | head -n 1 |\t\\\n+\t\t\tawk \"{ print \\$1; }\"\n+\t'\n+done\n+\n+test_done\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex c30522b57fd..871ce01401a 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -27,6 +27,36 @@ has_any () {\n \tgrep -Ff \"$1\" \"$2\"\n }\n \n+# Since name-hash values are stored in the .bitmap files, add a test\n+# that checks that the name-hash calculations are stable across versions.\n+# Not exhaustive, but these hashing algorithms would be hard to change\n+# without causing deviations here.\n+test_expect_success 'name-hash value stability' '\n+\tcat >names <<-\\EOF &&\n+\tfirst\n+\tsecond\n+\tthird\n+\ta/one-long-enough-for-collisions\n+\tb/two-long-enough-for-collisions\n+\tmany/parts/to/this/path/enough/to/collide/in/v2\n+\tenough/parts/to/this/path/enough/to/collide/in/v2\n+\tEOF\n+\n+\ttest-tool name-hash <names >out &&\n+\n+\tcat >expect <<-\\EOF &&\n+\t2582249472 1763573760 first\n+\t2289942528 1188134912 second\n+\t2300837888 1130758144 third\n+\t2544516325 3963087891 a/one-long-enough-for-collisions\n+\t2544516325 4013419539 b/two-long-enough-for-collisions\n+\t1420111091 1709547268 many/parts/to/this/path/enough/to/collide/in/v2\n+\t1420111091 1709547268 enough/parts/to/this/path/enough/to/collide/in/v2\n+\tEOF\n+\n+\ttest_cmp expect out\n+'\n+\n test_bitmap_cases () {\n \twriteLookupTable=false\n \tfor i in \"$@\"\n-- \ngitgitgadget\n\n"},{"id":"509434","messageId":"18a41f2fe6f2219f16f998699394005bd57ac463.1734715194.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.v3.git.1734715194.gitgitgadget@gmail.com","subject":"[PATCH v3 7/8] pack-objects: prevent name hash version change","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-20T17:19:53Z","receivedAt":"2024-12-20T17:20:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nWhen the --name-hash-version option is used in 'git pack-objects', it\ncan change from the initial assignment to when it is used based on\ninteractions with other arguments. Specifically, when writing or reading\nbitmaps, we must force version 1 for now. This could change in the\nfuture when the bitmap format can store a name hash version value,\nindicating which was used during the writing of the packfile.\n\nProtect the 'git pack-objects' process from getting confused by failing\nwith a BUG() statement if the value of the name hash version changes\nbetween calls to pack_name_hash_fn().\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n builtin/pack-objects.c | 8 ++++++++\n 1 file changed, 8 insertions(+)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex b19c3665003..4d10baf7ac9 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -276,6 +276,14 @@ static void validate_name_hash_version(void)\n \n static inline uint32_t pack_name_hash_fn(const char *name)\n {\n+\tstatic int seen_version = -1;\n+\n+\tif (seen_version < 0)\n+\t\tseen_version = name_hash_version;\n+\telse if (seen_version != name_hash_version)\n+\t\tBUG(\"name hash version changed from %d to %d mid-process\",\n+\t\t    seen_version, name_hash_version);\n+\n \tswitch (name_hash_version)\n \t{\n \tcase 1:\n-- \ngitgitgadget\n\n"},{"id":"509435","messageId":"3d63954f318e5133630b1f579a399a123e434cf8.1734715194.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.v3.git.1734715194.gitgitgadget@gmail.com","subject":"[PATCH v3 8/8] pack-objects: add third name hash version","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-20T17:19:54Z","receivedAt":"2024-12-20T17:20:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nThe '--name-hash-version=<n>' option in 'git pack-objects' was\nintroduced to allow for specifying an alternative name hash function\nwhen organizing objects for delta compression. The pack_name_hash_v2()\nfunction was designed to break some collisions while also preserving\nsome amount of locality for cross-path deltas.\n\nHowever, in some repositories, that effort to preserve locality results\nin enough collisions that it causes issues with full repacks.\n\nCreate a third name hash function and extend the '--name-hash-version'\noption in 'git pack-objects' and 'git repack' to understand it. This\nhash version abandons all efforts for locality and focuses on creating a\nsomewhat uniformly-distributed hash function to minimize collisions.\n\nWe can observe the effect of this collision avoidance in a large\ninternal monorepo that suffered from collisions in the previous\nversions. The updates to p5314-name-hash.sh show these results:\n\nTest                               this tree\n--------------------------------------------------\n5314.1: paths at head                       227.3K\n5314.2: distinct hash value: v1              72.3K\n5314.3: maximum multiplicity: v1             14.4K\n5314.4: distinct hash value: v2             166.5K\n5314.5: maximum multiplicity: v2               138\n5314.6: distinct hash value: v3             227.3K\n5314.7: maximum multiplicity: v3                 2\n\nThese results demonstrate that of the 227,000+ paths, nearly all of them\nfind distinct hash values. The maximum multiplicity is 2, improved from\n138 in the v2 hash function. The v2 hash function also had only 166K\ndistinct values, so it had a wide spread of collisions.\n\nA more modest improvement is available in the open source fluentui repo\n[1] with these results:\n\nTest                               this tree\n--------------------------------------------------\n5314.1: paths at head                        19.5K\n5314.2: distinct hash value: v1               8.2K\n5314.3: maximum multiplicity: v1               279\n5314.4: distinct hash value: v2              17.8K\n5314.5: maximum multiplicity: v2                44\n5314.6: distinct hash value: v3              19.5K\n5314.7: maximum multiplicity: v3                 1\n\n[1] https://github.com/microsoft/fluentui\n\nHowever, it is important to demonstrate the effectiveness of this\nfunction in the context of compressing a repository. We can use\np5313-pack-objects.sh to measure these changes. I will use a simplified\ntable summarizing the output of that performance test.\n\n | Test      | V1 Time | V2 Time | V3 Time | V1 Size | V2 Size | V3 Size |\n |-----------|---------|---------|---------|---------|---------|---------|\n | Thin Pack |  0.37 s |  0.12 s |  0.07 s |   1.2 M |  22.0 K |  20.4 K |\n | Big Pack  |  2.04 s |  2.80 s |  1.40 s |  20.4 M |  25.9 M |  19.2 M |\n | Shallow   |  1.41 s |  1.77 s |  1.27 s |  34.4 M |  33.7 M |  34.8 M |\n | Repack    | 95.70 s | 33.68 s | 20.88 s | 439.3 M | 160.5 M | 169.1 M |\n\nHere, there are some performance improvements on a time basis, and the\nthin and big packs are somewhat smaller in v3. The shallow and repacked\npacks are somewhat bigger, though, compared to v2.\n\nTwo repositories that have very few collisions in the v1 name hash are\nthe Git and Linux repositories. Here are their stats for p5313:\n\nGit:\n\n | Test      | V1 Time | V2 Time | V3 Time | V1 Size | V2 Size | V3 Size |\n |-----------|---------|---------|---------|---------|---------|---------|\n | Thin Pack |  0.02 s |  0.02 s |  0.02 s |   1.1 K |   1.1 K |  15.3 K |\n | Big Pack  |  1.69 s |  1.95 s |  1.67 s |  13.5 M |  14.5 M |  14.9 M |\n | Shallow   |  1.26 s |  1.29 s |  1.16 s |  12.0 M |  12.2 M |  12.5 M |\n | Repack    | 29.51 s | 29.01 s | 29.08 s | 237.7 M | 238.2 M | 237.7 M |\n\nLinux:\n\n | Test      | V1 Time  | V2 Time  | V3 Time  | V1 Size | V2 Size | V3 Size |\n |-----------|----------|----------|----------|---------|---------|---------|\n | Thin Pack |   0.17 s |   0.07 s |   0.07 s |   4.6 K |   4.6 K |   6.8 K |\n | Big Pack  |  17.88 s |  12.35 s |  12.14 s | 201.1 M | 149.1 M | 160.4 M |\n | Shallow   |  11.05 s |  22.94 s |  22.16 s | 269.2 M | 273.8 M | 271.8 M |\n | Repack    | 727.39 s | 566.95 s | 539.33 s |   2.5 G |   2.5 G |   2.6 G |\n\nThese repositories make good use of the cross-path deltas that come\nabout from the v1 name hash function, so they already had mixed results\nwith the v2 function. The v3 function is generally worse for these\nrepositories.\n\nAn internal Javascript-based repository with name hash collisions\nsimilar to the fluentui repo has these results:\n\n | Test      | V1 Time   | V2 Time  | V3 Time  | V1 Size | V2 Size | V3 Size |\n |-----------|-----------|----------|----------|---------|---------|---------|\n | Thin Pack |    8.28 s |   7.28 s |   0.04 s |  16.8 K |  16.8 K |   3.2 K |\n | Big Pack  |   12.81 s |  11.66 s |   2.52 s |  29.1 M |  29.1 M |  30.6 M |\n | Shallow   |    4.86 s |   4.06 s |   3.77 s |  42.5 M |  44.1 M |  45.7 M |\n | Repack    | 3126.50 s | 496.33 s | 306.86 s |   6.2 G | 855.6 M | 838.2 M |\n\nThis repository is also listed as \"Repo B\" in the repacking size table\nbelow, along with other Javascript repos that have many name hash\ncollisions with the v1 name hash:\n\n | Repo     | V1 Size   | V2 Size | V3 Size |\n |----------|-----------|---------|---------|\n | fluentui |     440 M |   161 M |   170 M |\n | Repo B   |   6,248 M |   856 M |   840 M |\n | Repo C   |  37,278 M | 6,921 M | 6,755 M |\n | Repo D   | 131,204 M | 7,463 M | 7,124 M |\n\nWhile the fluentui repo had an increase in size using the v3 name hash,\nthe others had modest improvements over the v2 name hash. But those\nmodest improvements are dwarfed by the difference from v1 to v2, so it\nis unlikely that the regression seen in the other scenarios (packfiles\nthat are not from full repacks) will be worth using v3 over v2. That is,\nunless there are enough collisions even with v2 that the full repack\nscenario has larger improvements than these.\n\nWhen using GIT_TEST_NAME_HASH_VERSION=3, there are some necessary\nchanges to t5616-partial-clone.sh since the server now picks different\ndelta bases that the client does not have (and does not then fetch\ndynamically). These changes are a minimal patch and the functionality\nshould be fixed in other changes.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Documentation/git-pack-objects.txt |  9 +++++++++\n builtin/pack-objects.c             |  5 ++++-\n pack-objects.h                     | 26 ++++++++++++++++++++++++++\n t/helper/test-name-hash.c          |  1 +\n t/perf/p5313-pack-objects.sh       |  2 +-\n t/perf/p5314-name-hash.sh          |  2 +-\n t/t5300-pack-object.sh             |  4 ++--\n t/t5310-pack-bitmaps.sh            | 14 +++++++-------\n t/t5616-partial-clone.sh           | 26 ++++++++++++++++++++++++--\n 9 files changed, 75 insertions(+), 14 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 7f69ae4855f..9fe25c53415 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -374,6 +374,15 @@ breaking most of the collisions from a similarly-named file appearing in\n many different directories. At the moment, this version is not allowed\n when writing reachability bitmap files with `--write-bitmap-index` and it\n will be automatically changed to version `1`.\n++\n+The name hash version `3` abandons the locality features of versions `1`\n+and `2` in favor of minimizing collisions. The goal here is to separate\n+objects by their full path and abandon hope for cross-path delta\n+compression. For this reason, this option is preferred for repacking large\n+repositories with many versions and many name hash collisions when using\n+the first two versions. At the moment, this version is not allowed when\n+writing reachability bitmap files with `--write-bitmap-index` and it will\n+be automatically changed to version `1`.\n \n \n DELTA ISLANDS\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 4d10baf7ac9..8297af1a272 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -270,7 +270,7 @@ static int name_hash_version = -1;\n \n static void validate_name_hash_version(void)\n {\n-\tif (name_hash_version < 1 || name_hash_version > 2)\n+\tif (name_hash_version < 1 || name_hash_version > 3)\n \t\tdie(_(\"invalid --name-hash-version option: %d\"), name_hash_version);\n }\n \n@@ -292,6 +292,9 @@ static inline uint32_t pack_name_hash_fn(const char *name)\n \tcase 2:\n \t\treturn pack_name_hash_v2((const unsigned char *)name);\n \n+\tcase 3:\n+\t\treturn pack_name_hash_v3(name);\n+\n \tdefault:\n \t\tBUG(\"invalid name-hash version: %d\", name_hash_version);\n \t}\ndiff --git a/pack-objects.h b/pack-objects.h\nindex 681c1116486..2b20a56fc99 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -235,6 +235,32 @@ static inline uint32_t pack_name_hash_v2(const unsigned char *name)\n \treturn (base >> 6) ^ hash;\n }\n \n+static inline uint32_t pack_name_hash_v3(const char *name)\n+{\n+\t/*\n+\t * This 'bigp' value is a large prime, at least 25% of the max\n+\t * value of an uint32_t. Multiplying by this value (modulo 2^32)\n+\t * causes the 32 bits to change pseudo-randomly.\n+\t */\n+\tconst uint32_t bigp = 1234572167U;\n+\tuint32_t c, hash = bigp;\n+\n+\tif (!name)\n+\t\treturn 0;\n+\n+\t/*\n+\t * Do the simplest thing that will resemble pseudo-randomness: add\n+\t * random multiples of a large prime number with a binary shift.\n+\t * The goal is not to be cryptographic, but to be generally\n+\t * uniformly distributed.\n+\t */\n+\twhile ((c = *name++) != 0) {\n+\t\thash += c * bigp;\n+\t\thash = (hash >> 5) | (hash << 27);\n+\t}\n+\treturn hash;\n+}\n+\n static inline enum object_type oe_type(const struct object_entry *e)\n {\n \treturn e->type_valid ? e->type_ : OBJ_BAD;\ndiff --git a/t/helper/test-name-hash.c b/t/helper/test-name-hash.c\nindex af1d52de101..cc5acd58a65 100644\n--- a/t/helper/test-name-hash.c\n+++ b/t/helper/test-name-hash.c\n@@ -15,6 +15,7 @@ int cmd__name_hash(int argc UNUSED, const char **argv UNUSED)\n \twhile (!strbuf_getline(&line, stdin)) {\n \t\tprintf(\"%10u \", pack_name_hash(line.buf));\n \t\tprintf(\"%10u \", pack_name_hash_v2((unsigned const char *)line.buf));\n+\t\tprintf(\"%10u \", pack_name_hash_v3(line.buf));\n \t\tprintf(\"%s\\n\", line.buf);\n \t}\n \ndiff --git a/t/perf/p5313-pack-objects.sh b/t/perf/p5313-pack-objects.sh\nindex be5229a0ecd..493872e656d 100755\n--- a/t/perf/p5313-pack-objects.sh\n+++ b/t/perf/p5313-pack-objects.sh\n@@ -25,7 +25,7 @@ test_expect_success 'create rev input' '\n \tEOF\n '\n \n-for version in 1 2\n+for version in 1 2 3\n do\n \texport version\n \ndiff --git a/t/perf/p5314-name-hash.sh b/t/perf/p5314-name-hash.sh\nindex 4ef0ba77114..e58a218d1ae 100755\n--- a/t/perf/p5314-name-hash.sh\n+++ b/t/perf/p5314-name-hash.sh\n@@ -14,7 +14,7 @@ test_size 'paths at head' '\n \ttest-tool name-hash <path-list >name-hashes\n '\n \n-for version in 1 2\n+for version in 1 2 3\n do\n \ttest_size \"distinct hash value: v$version\" '\n \t\tawk \"{ print \\$$version; }\" <name-hashes | sort | \\\ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 97fe9e561c6..279a9deca9f 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -680,13 +680,13 @@ test_expect_success 'valid and invalid --name-hash-versions' '\n \t# Valid values are hard to verify other than \"do not fail\".\n \t# Performance tests will be more valuable to validate these versions.\n \t# Negative values are converted to version 1.\n-\tfor value in -1 1 2\n+\tfor value in -1 1 2 3\n \tdo\n \t\tgit pack-objects base --all --name-hash-version=$value || return 1\n \tdone &&\n \n \t# Invalid values have clear post-conditions.\n-\tfor value in 0 3\n+\tfor value in 0 4\n \tdo\n \t\ttest_must_fail git pack-objects base --all --name-hash-version=$value 2>err &&\n \t\ttest_grep \"invalid --name-hash-version option\" err || return 1\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 871ce01401a..2bf75e2a5d0 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -45,13 +45,13 @@ test_expect_success 'name-hash value stability' '\n \ttest-tool name-hash <names >out &&\n \n \tcat >expect <<-\\EOF &&\n-\t2582249472 1763573760 first\n-\t2289942528 1188134912 second\n-\t2300837888 1130758144 third\n-\t2544516325 3963087891 a/one-long-enough-for-collisions\n-\t2544516325 4013419539 b/two-long-enough-for-collisions\n-\t1420111091 1709547268 many/parts/to/this/path/enough/to/collide/in/v2\n-\t1420111091 1709547268 enough/parts/to/this/path/enough/to/collide/in/v2\n+\t2582249472 1763573760 3109209818 first\n+\t2289942528 1188134912 3781118409 second\n+\t2300837888 1130758144 3028707182 third\n+\t2544516325 3963087891 3586976147 a/one-long-enough-for-collisions\n+\t2544516325 4013419539 1701624798 b/two-long-enough-for-collisions\n+\t1420111091 1709547268 2676129939 many/parts/to/this/path/enough/to/collide/in/v2\n+\t1420111091 1709547268 2740459187 enough/parts/to/this/path/enough/to/collide/in/v2\n \tEOF\n \n \ttest_cmp expect out\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex c53e93be2f7..4e7863af9e0 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -516,7 +516,18 @@ test_expect_success 'fetch lazy-fetches only to resolve deltas' '\n \t# Exercise to make sure it works. Git will not fetch anything from the\n \t# promisor remote other than for the big tree (because it needs to\n \t# resolve the delta).\n-\tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -C client \\\n+\t#\n+\t# TODO: the --name-hash-version option is disabled here, since this test\n+\t# is fundamentally broken! When GIT_TEST_NAME_HASH_VERSION=3, the server\n+\t# recognizes delta bases in a different way and then sends a _blob_ to\n+\t# the client with a delta base that the client does not have! This is\n+\t# because the client is cloned from \"promisor-server\" with tree:0 but\n+\t# is now fetching from \"server\" without any filter. This is violating the\n+\t# promise to the server that all reachable objects exist and could be\n+\t# used as delta bases!\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace\" \\\n+\t\tGIT_TEST_NAME_HASH_VERSION=1 \\\n+\t\tgit -C client \\\n \t\tfetch \"file://$(pwd)/server\" main &&\n \n \t# Verify the assumption that the client needed to fetch the delta base\n@@ -535,7 +546,18 @@ test_expect_success 'fetch lazy-fetches only to resolve deltas, protocol v2' '\n \t# Exercise to make sure it works. Git will not fetch anything from the\n \t# promisor remote other than for the big blob (because it needs to\n \t# resolve the delta).\n-\tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -C client \\\n+\t#\n+\t# TODO: the --name-hash-verion option is disabled here, since this test\n+\t# is fundamentally broken! When GIT_TEST_NAME_HASH_VERSION=3, the server\n+\t# recognizes delta bases in a different way and then sends a _blob_ to\n+\t# the client with a delta base that the client does not have! This is\n+\t# because the client is cloned from \"promisor-server\" with tree:0 but\n+\t# is now fetching from \"server\" without any filter. This is violating the\n+\t# promise to the server that all reachable objects exist and could be\n+\t# used as delta bases!\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace\" \\\n+\t\tGIT_TEST_NAME_HASH_VERSION=1 \\\n+\t\tgit -C client \\\n \t\tfetch \"file://$(pwd)/server\" main &&\n \n \t# Verify that protocol version 2 was used.\n-- \ngitgitgadget\n"},{"id":"511016","messageId":"35026c72-f9b4-40a3-b528-1c28b1238972@gmail.com","threadId":"62447","inReplyTo":"pull.1823.v3.git.1734715194.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 0/8] pack-objects: Create an alternative name hash algorithm (recreated)","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2025-01-21T20:21:15Z","receivedAt":"2025-01-21T20:21:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/20/24 12:19 PM, Derrick Stolee via GitGitGadget wrote:\n> This is a recreation of the topic in [1] that was closed. (I force-pushed my\n> branch and GitHub won't let me reopen the PR for GitGitGadget to create this\n> as v3.)\n> \n> [1]\n> https://lore.kernel.org/git/pull.1785.v2.git.1726692381.gitgitgadget@gmail.com/\n> \n> I've been focused recently on understanding and mitigating the growth of a\n> few internal repositories. Some of these are growing much larger than\n> expected for the number of contributors, and there are multiple aspects to\n> why this growth is so large.\n\n> The main issue plaguing these repositories is that deltas are not being\n> computed against objects that appear at the same path. While the size of\n> these files at tip is one aspect of growth that would prevent this issue,\n> the changes to these files are reasonable and should result in good delta\n> compression. However, Git is not discovering the connections across\n> different versions of the same file.\n\n> This series creates a mechanism to select alternative name hashes using a\n> new --name-hash-version=<n> option. The versions are:\n> \n>   1. Version 1 is the default name hash that already exists. This option\n>      focuses on the final bytes of the path to maximize locality for\n>      cross-path deltas.\n> \n>   2. Version 2 is the new path-component hash function suggested by Jonathan\n>      Tan in the previous version (with some modifications). This hash\n>      function essentially computes the v1 name hash of each path component\n>      and then overlays those hashes with a shift to make the parent\n>      directories contribute less to the final hash, but enough to break many\n>      collisions that exist in v1.\n> \n>   3. Version 3 is the hash function that I submitted under the\n>      --full-name-hash feature in the previous versions. This uses a\n>      pseudorandom hash procedure to minimize collisions but at the expense of\n>      losing on locality. This version is implemented in the final patch of\n>      the series mostly for comparison purposes, as it is unlikely to be\n>      selected as a valuable hash function over v2. The final patch could be\n>      omitted from the merged version.\nThis series has been at this version for a while. I'm pretty sure that this\nis the most promising direction we have at the moment for improving delta\ncompression for many users.\n\nThe only decision point I think remains is whether or not to include the last\npatch (--name-hash-version=3) which I would be happy either way.\n\nThanks,\n-Stolee\n"},{"id":"511090","messageId":"Z5FsZ7MK6YcmeYIV@nand.local","threadId":"62447","inReplyTo":"68b4127580e2d475bec0d7cd0f6a9ae5e626b3c9.1734715194.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 1/8] pack-objects: create new name-hash function version","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-22T22:08:39Z","receivedAt":"2025-01-22T22:08:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Dec 20, 2024 at 05:19:47PM +0000, Jonathan Tan via GitGitGadget wrote:\n> The first change is to be more careful about paths using non-ASCII\n> characters. With these characters in mind, reverse the bits in the byte\n> as the least-significant bits have the highest entropy and we want to\n> maximize their influence. This is done with some bit manipulation that\n> swaps the two halves, then the quarters within those halves, and then\n> the bits within those quarters.\n\nMakes sense, and seems quite reasonable.\n\n> The second change is to perform hash composition operations at every\n> level of the path. This is done by storing a 'base' hash value that\n> contains the hash of the parent directory. When reaching a directory\n> boundary, we XOR the current level's name-hash value with a downshift of\n> the previous level's hash. This perturbation intends to create low-bit\n> distinctions for paths with the same final 16 bytes but distinct parent\n> directory structures.\n\nVery clever, I love this idea.\n\nThanks,\nTaylor\n"},{"id":"511092","messageId":"Z5FugEXhdKjhwcnP@nand.local","threadId":"62447","inReplyTo":"d035e3e59f42c75760e8dd6fe8ed6dff12bc8b9a.1734715194.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 2/8] pack-objects: add --name-hash-version option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-22T22:17:36Z","receivedAt":"2025-01-22T22:17:39Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Dec 20, 2024 at 05:19:48PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee <stolee@gmail.com>\n>\n> The previous change introduced a new pack_name_hash_v2() function that\n> intends to satisfy much of the hash locality features of the existing\n> pack_name_hash() function while also distinguishing paths with similar\n> final components of their paths.\n>\n> This change adds a new --name-hash-version option for 'git pack-objects'\n> to allow users to select their preferred function version. This use of\n> an integer version allows for future expansion and a direct way to later\n> store a name hash version in the .bitmap format.\n>\n> For now, let's consider how effective this mechanism is when repacking a\n> repository with different name hash versions. Specifically, we will\n> execute 'git pack-objects' the same way a 'git repack -adf' process\n> would, except we include --name-hash-version=<n> for testing.\n>\n> On the Git repository, we do not expect much difference. All path names\n> are short. This is backed by our results:\n>\n> | Stage                 | Pack Size | Repack Time |\n> |-----------------------|-----------|-------------|\n> | After clone           | 260 MB    | N/A         |\n> | --name-hash-version=1 | 127 MB    | 129s        |\n> | --name-hash-version=2 | 127 MB    | 112s        |\n>\n> This example demonstrates how there is some natural overhead coming from\n> the cloned copy because the server is hosting many forks and has not\n> optimized for exactly this set of reachable objects. But the full repack\n> has similar characteristics for both versions.\n>\n> Let's consider some repositories that are hitting too many collisions\n> with version 1. First, let's explore the kinds of paths that are\n> commonly causing these collisions:\n>\n>  * \"/CHANGELOG.json\" is 15 characters, and is created by the beachball\n>    [1] tool. Only the final character of the parent directory can\n>    differentiate different versions of this file, but also only the two\n>    most-significant digits. If that character is a letter, then this is\n>    always a collision. Similar issues occur with the similar\n>    \"/CHANGELOG.md\" path, though there is more opportunity for\n>    differences In the parent directory.\n>\n>  * Localization files frequently have common filenames but\n>    differentiates via parent directories. In C#, the name\n>    \"/strings.resx.lcl\" is used for these localization files and they\n>    will all collide in name-hash.\n>\n> [1] https://github.com/microsoft/beachball\n>\n> I've come across many other examples where some internal tool uses a\n> common name across multiple directories and is causing Git to repack\n> poorly due to name-hash collisions.\n>\n> One open-source example is the fluentui [2] repo, which  uses beachball\n> to generate CHANGELOG.json and CHANGELOG.md files, and these files have\n> very poor delta characteristics when comparing against versions across\n> parent directories.\n>\n> | Stage                 | Pack Size | Repack Time |\n> |-----------------------|-----------|-------------|\n> | After clone           | 694 MB    | N/A         |\n> | --name-hash-version=1 | 438 MB    | 728s        |\n> | --name-hash-version=2 | 168 MB    | 142s        |\n>\n> [2] https://github.com/microsoft/fluentui\n>\n> In this example, we see significant gains in the compressed packfile\n> size as well as the time taken to compute the packfile.\n>\n> Using a collection of repositories that use the beachball tool, I was\n> able to make similar comparisions with dramatic results. While the\n> fluentui repo is public, the others are private so cannot be shared for\n> reproduction. The results are so significant that I find it important to\n> share here:\n>\n> | Repo     | --name-hash-version=1 | --name-hash-version=2 |\n> |----------|-----------------------|-----------------------|\n> | fluentui |               440 MB  |               161 MB  |\n> | Repo B   |             6,248 MB  |               856 MB  |\n> | Repo C   |            37,278 MB  |             6,755 MB  |\n> | Repo D   |           131,204 MB  |             7,463 MB  |\n>\n> Future changes could include making --name-hash-version implied by a config\n> value or even implied by default during a full repack.\n>\n> It is important to point out that the name hash value is stored in the\n> .bitmap file format, so we must force --name-hash-version=1 when bitmaps\n> are being read or written. Later, the bitmap format could be updated to\n> be aware of the name hash version so deltas can be quickly computed\n> across the bitmapped/not-bitmapped boundary.\n>\n> Signed-off-by: Derrick Stolee <stolee@gmail.com>\n> ---\n>  Documentation/git-pack-objects.txt | 32 ++++++++++++++++++-\n>  builtin/pack-objects.c             | 49 +++++++++++++++++++++++++++---\n>  t/t5300-pack-object.sh             | 31 +++++++++++++++++++\n>  3 files changed, 106 insertions(+), 6 deletions(-)\n>\n> diff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\n> index e32404c6aae..7f69ae4855f 100644\n> --- a/Documentation/git-pack-objects.txt\n> +++ b/Documentation/git-pack-objects.txt\n> @@ -15,7 +15,8 @@ SYNOPSIS\n>  \t[--revs [--unpacked | --all]] [--keep-pack=<pack-name>]\n>  \t[--cruft] [--cruft-expiration=<time>]\n>  \t[--stdout [--filter=<filter-spec>] | <base-name>]\n> -\t[--shallow] [--keep-true-parents] [--[no-]sparse] < <object-list>\n> +\t[--shallow] [--keep-true-parents] [--[no-]sparse]\n> +\t[--name-hash-version=<n>] < <object-list>\n>\n>\n>  DESCRIPTION\n> @@ -345,6 +346,35 @@ raise an error.\n>  \tRestrict delta matches based on \"islands\". See DELTA ISLANDS\n>  \tbelow.\n>\n> +--name-hash-version=<n>::\n> +\tWhile performing delta compression, Git groups objects that may be\n> +\tsimilar based on heuristics using the path to that object. While\n> +\tgrouping objects by an exact path match is good for paths with\n> +\tmany versions, there are benefits for finding delta pairs across\n> +\tdifferent full paths. Git collects objects by type and then by a\n> +\t\"name hash\" of the path and then by size, hoping to group objects\n> +\tthat will compress well together.\n> ++\n> +The default name hash version is `1`, which prioritizes hash locality by\n> +considering the final bytes of the path as providing the maximum magnitude\n> +to the hash function. This version excels at distinguishing short paths\n> +and finding renames across directories. However, the hash function depends\n> +primarily on the final 16 bytes of the path. If there are many paths in\n> +the repo that have the same final 16 bytes and differ only by parent\n> +directory, then this name-hash may lead to too many collisions and cause\n> +poor results. At the moment, this version is required when writing\n> +reachability bitmap files with `--write-bitmap-index`.\n> ++\n> +The name hash version `2` has similar locality features as version `1`,\n> +except it considers each path component separately and overlays the hashes\n> +with a shift. This still prioritizes the final bytes of the path, but also\n> +\"salts\" the lower bits of the hash using the parent directory names. This\n> +method allows for some of the locality benefits of version `1` while\n> +breaking most of the collisions from a similarly-named file appearing in\n> +many different directories. At the moment, this version is not allowed\n> +when writing reachability bitmap files with `--write-bitmap-index` and it\n> +will be automatically changed to version `1`.\n> +\n>\n>  DELTA ISLANDS\n>  -------------\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 08007142671..90ea19417bc 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -266,6 +266,28 @@ struct configured_exclusion {\n>  static struct oidmap configured_exclusions;\n>\n>  static struct oidset excluded_by_config;\n> +static int name_hash_version = 1;\n> +\n> +static void validate_name_hash_version(void)\n> +{\n> +\tif (name_hash_version < 1 || name_hash_version > 2)\n> +\t\tdie(_(\"invalid --name-hash-version option: %d\"), name_hash_version);\n> +}\n> +\n> +static inline uint32_t pack_name_hash_fn(const char *name)\n> +{\n> +\tswitch (name_hash_version)\n> +\t{\n\nThis is definitely a nitpick, but the opening curly brace should appear\non the preceding line. I don't think our CodingGuidelines explicitly say\nthis. But in my head the convention is that only function bodies have\ntheir enclosing braces on their own line, as opposed to:\n\n    static inline uint32_t pack_name_hash_fn(const char *name) {\n\n> +\tcase 1:\n> +\t\treturn pack_name_hash(name);\n> +\n> +\tcase 2:\n> +\t\treturn pack_name_hash_v2((const unsigned char *)name);\n> +\n> +\tdefault:\n> +\t\tBUG(\"invalid name-hash version: %d\", name_hash_version);\n> +\t}\n> +}\n\nOtherwise this function looks very reasonable.\n\n> @@ -4576,6 +4609,12 @@ int cmd_pack_objects(int argc,\n>  \tif (pack_to_stdout || !rev_list_all)\n>  \t\twrite_bitmap_index = 0;\n>\n> +\tvalidate_name_hash_version();\n> +\tif (write_bitmap_index && name_hash_version != 1) {\n> +\t\twarning(_(\"currently, --write-bitmap-index requires --name-hash-version=1\"));\n> +\t\tname_hash_version = 1;\n> +\t}\n> +\n\nHmm. validate_name_hash_version() is its own function, which seems good,\nbut then we do further validation on it here. I wonder if we should\neither move the latter step into validate_name_hash_version(), which\nwould require us to pass a pointer to name_hash_version, or assign it\nthe return value of validate_name_hash_version() (assuming it were\nchanged to return the appropriate type instead of void).\n\nI think that we are probably pretty far into bike-shedding territory,\nbut figured I'd share as it jumped out to me while reviewing.\n\n>  \tif (use_delta_islands)\n>  \t\tstrvec_push(&rp, \"--topo-order\");\n>\n> diff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\n> index 3b9dae331a5..4270eabe8b7 100755\n> --- a/t/t5300-pack-object.sh\n> +++ b/t/t5300-pack-object.sh\n> @@ -674,4 +674,35 @@ do\n>  \t'\n>  done\n>\n> +test_expect_success 'valid and invalid --name-hash-versions' '\n> +\t# Valid values are hard to verify other than \"do not fail\".\n> +\t# Performance tests will be more valuable to validate these versions.\n> +\tfor value in 1 2\n> +\tdo\n> +\t\tgit pack-objects base --all --name-hash-version=$value || return 1\n> +\tdone &&\n> +\n> +\t# Invalid values have clear post-conditions.\n> +\tfor value in -1 0 3\n> +\tdo\n> +\t\ttest_must_fail git pack-objects base --all --name-hash-version=$value 2>err &&\n> +\t\ttest_grep \"invalid --name-hash-version option\" err || return 1\n> +\tdone\n> +'\n\nLooks great, and thanks for handling a few nonsensical values in the\nlatter loop.\n\n> +# The following test is not necessarily a permanent choice, but since we do not\n> +# have a \"name hash version\" bit in the .bitmap file format, we cannot write the\n> +# hash values into the .bitmap file without risking breakage later.\n> +#\n> +# TODO: Make these compatible in the future and replace this test with the\n> +# expected behavior when both are specified.\n> +test_expect_success '--name-hash-version=2 and --write-bitmap-index are incompatible' '\n> +\tgit pack-objects base --all --name-hash-version=2 --write-bitmap-index 2>err &&\n> +\ttest_grep \"currently, --write-bitmap-index requires --name-hash-version=1\" err &&\n> +\n> +\t# --stdout option silently removes --write-bitmap-index\n> +\tgit pack-objects --stdout --all --name-hash-version=2 --write-bitmap-index >out 2>err &&\n> +\t! test_grep \"currently, --write-bitmap-index requires --name-hash-version=1\" err\n> +'\n\nThis is grea, too, I appreciate having the reminder here for the future\n(and it'll feel so satisfying to change/remove this test when the time\ncomes) ;-).\n\nThanks,\nTaylor\n"},{"id":"511093","messageId":"Z5Fu0h6Mb9/hdA6E@nand.local","threadId":"62447","inReplyTo":"e2191244f6b21792f2551946a89cfc48af3989c5.1734715194.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 3/8] repack: add --name-hash-version option","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-22T22:18:58Z","receivedAt":"2025-01-22T22:19:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Dec 20, 2024 at 05:19:49PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee <stolee@gmail.com>\n>\n> The new '--name-hash-version' option for 'git repack' is a simple\n> pass-through to the underlying 'git pack-objects' subcommand. However,\n> this subcommand may have other options and a temporary filename as part\n> of the subcommand execution that may not be predictable or could change\n> over time.\n>\n> The existing test_subcommand method requires an exact list of arguments\n> for the subcommand. This is too rigid for our needs here, so create a\n> new method, test_subcommand_flex. Use it to check that the\n> --name-hash-version option is passing through.\n>\n> Since we are modifying the 'git repack' command, let's bring its usage\n> in line with the Documentation's synopsis. This removes it from the\n> allow list in t0450 so it will remain in sync in the future.\n>\n> Signed-off-by: Derrick Stolee <stolee@gmail.com>\n> ---\n>  Documentation/git-repack.txt |  9 ++++++++-\n>  builtin/repack.c             |  9 ++++++++-\n>  t/t0450/txt-help-mismatches  |  1 -\n>  t/t7700-repack.sh            |  6 ++++++\n>  t/test-lib-functions.sh      | 26 ++++++++++++++++++++++++++\n>  5 files changed, 48 insertions(+), 3 deletions(-)\n>\n> diff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\n> index c902512a9e8..5852a5c9736 100644\n> --- a/Documentation/git-repack.txt\n> +++ b/Documentation/git-repack.txt\n> @@ -9,7 +9,9 @@ git-repack - Pack unpacked objects in a repository\n>  SYNOPSIS\n>  --------\n>  [verse]\n> -'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m] [--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>] [--write-midx]\n> +'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\n> +\t[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\n> +\t[--write-midx] [--name-hash-version=<n>]\n\nI probably would have split this change into two separate patches (one\nto adjust the line wrapping in the synopsis, and another to introduce\nthe new option). But regardless, I am really glad to see this change,\nsince the long synopsis line has always bothered me, but I never got\naround to changing it. Thanks for deciding that enough is enough!\n\n> diff --git a/t/t0450/txt-help-mismatches b/t/t0450/txt-help-mismatches\n> index 28003f18c92..c4a15fd0cb8 100644\n> --- a/t/t0450/txt-help-mismatches\n> +++ b/t/t0450/txt-help-mismatches\n> @@ -45,7 +45,6 @@ rebase\n>  remote\n>  remote-ext\n>  remote-fd\n> -repack\n\n:-).\n\nThanks,\nTaylor\n"},{"id":"511094","messageId":"Z5FvP4KL+POd64hh@nand.local","threadId":"62447","inReplyTo":"86ff0d0a15e4263ccd541a9b8dcdb99438784a70.1734715194.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 4/8] pack-objects: add GIT_TEST_NAME_HASH_VERSION","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-22T22:20:47Z","receivedAt":"2025-01-22T22:20:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Dec 20, 2024 at 05:19:50PM +0000, Derrick Stolee via GitGitGadget wrote:\n> @@ -209,6 +209,10 @@ test_expect_success 'bitmapPseudoMerge.stableThreshold creates stable groups' '\n>  '\n>\n>  test_expect_success 'out of order thresholds are rejected' '\n> +\t# Disable this option to avoid stderr message\n> +\tGIT_TEST_NAME_HASH_VERSION=1 &&\n> +\texport GIT_TEST_NAME_HASH_VERSION &&\n> +\n>  \ttest_must_fail git \\\n>  \t\t-c bitmapPseudoMerge.test.pattern=\"refs/*\" \\\n>  \t\t-c bitmapPseudoMerge.test.threshold=1.month.ago \\\n\nThis is the only one that sets GIT_TEST_NAME_HASH_VERSION via an export.\nI suspect that this is to get around calling the shell function with a\nsingle-shot environment variable. But I think our convention for this is\n\n    test_must_fail env GIT_TEST_NAME_HASH_VERSION=1 git ...\n\nProbably not a big deal, but I figured I'd mention it regardless in case\nyou happen to reroll.\n\nThanks,\nTaylor\n"},{"id":"511095","messageId":"Z5FviP+M7Mi1z1Q9@nand.local","threadId":"62447","inReplyTo":"18a41f2fe6f2219f16f998699394005bd57ac463.1734715194.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 7/8] pack-objects: prevent name hash version change","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-22T22:22:00Z","receivedAt":"2025-01-22T22:22:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Dec 20, 2024 at 05:19:53PM +0000, Derrick Stolee via GitGitGadget wrote:\n> ---\n>  builtin/pack-objects.c | 8 ++++++++\n>  1 file changed, 8 insertions(+)\n\nThis is a very nice guard in my opinion, thanks for adding it!\n\nThanks,\nTaylor\n"},{"id":"511096","messageId":"Z5FzE1XpBlEyhK2T@nand.local","threadId":"62447","inReplyTo":"3d63954f318e5133630b1f579a399a123e434cf8.1734715194.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 8/8] pack-objects: add third name hash version","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-22T22:37:07Z","receivedAt":"2025-01-22T22:37:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Dec 20, 2024 at 05:19:54PM +0000, Derrick Stolee via GitGitGadget wrote:\n> Create a third name hash function and extend the '--name-hash-version'\n> option in 'git pack-objects' and 'git repack' to understand it. This\n> hash version abandons all efforts for locality and focuses on creating a\n> somewhat uniformly-distributed hash function to minimize collisions.\n>\n> We can observe the effect of this collision avoidance in a large\n> internal monorepo that suffered from collisions in the previous\n> versions. The updates to p5314-name-hash.sh show these results:\n>\n> Test                               this tree\n> --------------------------------------------------\n> 5314.1: paths at head                       227.3K\n> 5314.2: distinct hash value: v1              72.3K\n> 5314.3: maximum multiplicity: v1             14.4K\n> 5314.4: distinct hash value: v2             166.5K\n> 5314.5: maximum multiplicity: v2               138\n> 5314.6: distinct hash value: v3             227.3K\n> 5314.7: maximum multiplicity: v3                 2\n>\n> These results demonstrate that of the 227,000+ paths, nearly all of them\n> find distinct hash values. The maximum multiplicity is 2, improved from\n> 138 in the v2 hash function. The v2 hash function also had only 166K\n> distinct values, so it had a wide spread of collisions.\n\nI had a little trouble reading this section of the commit message. I\nthink the framing makes sense (v2 has collisions which can impact pack\ngeneration time and/or size), but this section explains v3 I think one\nlevel too deep.\n\nThis comparison (and the one below it for v3) shows a reduction in\ndistinct hash values and the maximum multiplicity (I'm assuming for\ncolliding hash values, in which case I might suggest renaming it as\n\"maximum collisions\").\n\nBut I imagine that many readers will primarily care about the effect of\nthe new hash function on pack generation time and size. You show that\nbelow, but I think that it should potentially appear earlier in the\ncommit message.\n\nAlternatively, you could consider leaving the time/size table alone\nwhere it is, and devote an extra sentence or two to explaining the\nimpact on repacking time/size that the two metrics above (distinct hash\nvalues, multiplicity/collisions) have on the repacking time/size.\n\n> A more modest improvement is available in the open source fluentui repo\n> [1] with these results:\n>\n> Test                               this tree\n> --------------------------------------------------\n> 5314.1: paths at head                        19.5K\n> 5314.2: distinct hash value: v1               8.2K\n> 5314.3: maximum multiplicity: v1               279\n> 5314.4: distinct hash value: v2              17.8K\n> 5314.5: maximum multiplicity: v2                44\n> 5314.6: distinct hash value: v3              19.5K\n> 5314.7: maximum multiplicity: v3                 1\n>\n> [1] https://github.com/microsoft/fluentui\n>\n> However, it is important to demonstrate the effectiveness of this\n> function in the context of compressing a repository. We can use\n> p5313-pack-objects.sh to measure these changes. I will use a simplified\n> table summarizing the output of that performance test.\n>\n>  | Test      | V1 Time | V2 Time | V3 Time | V1 Size | V2 Size | V3 Size |\n>  |-----------|---------|---------|---------|---------|---------|---------|\n>  | Thin Pack |  0.37 s |  0.12 s |  0.07 s |   1.2 M |  22.0 K |  20.4 K |\n>  | Big Pack  |  2.04 s |  2.80 s |  1.40 s |  20.4 M |  25.9 M |  19.2 M |\n>  | Shallow   |  1.41 s |  1.77 s |  1.27 s |  34.4 M |  33.7 M |  34.8 M |\n>  | Repack    | 95.70 s | 33.68 s | 20.88 s | 439.3 M | 160.5 M | 169.1 M |\n\nOK, now we get to the chart that I demonstrates the effects of each hash\nfunction on the most externally visible effects. Are these measurements\ntaken from the fluentui repo, or somewhere else? In either case, it\nmay be worth mentioning.\n\n> Here, there are some performance improvements on a time basis, and the\n> thin and big packs are somewhat smaller in v3. The shallow and repacked\n> packs are somewhat bigger, though, compared to v2.\n>\n> Two repositories that have very few collisions in the v1 name hash are\n> the Git and Linux repositories. Here are their stats for p5313:\n>\n> Git:\n>\n>  | Test      | V1 Time | V2 Time | V3 Time | V1 Size | V2 Size | V3 Size |\n>  |-----------|---------|---------|---------|---------|---------|---------|\n>  | Thin Pack |  0.02 s |  0.02 s |  0.02 s |   1.1 K |   1.1 K |  15.3 K |\n>  | Big Pack  |  1.69 s |  1.95 s |  1.67 s |  13.5 M |  14.5 M |  14.9 M |\n>  | Shallow   |  1.26 s |  1.29 s |  1.16 s |  12.0 M |  12.2 M |  12.5 M |\n>  | Repack    | 29.51 s | 29.01 s | 29.08 s | 237.7 M | 238.2 M | 237.7 M |\n>\n> Linux:\n>\n>  | Test      | V1 Time  | V2 Time  | V3 Time  | V1 Size | V2 Size | V3 Size |\n>  |-----------|----------|----------|----------|---------|---------|---------|\n>  | Thin Pack |   0.17 s |   0.07 s |   0.07 s |   4.6 K |   4.6 K |   6.8 K |\n>  | Big Pack  |  17.88 s |  12.35 s |  12.14 s | 201.1 M | 149.1 M | 160.4 M |\n>  | Shallow   |  11.05 s |  22.94 s |  22.16 s | 269.2 M | 273.8 M | 271.8 M |\n>  | Repack    | 727.39 s | 566.95 s | 539.33 s |   2.5 G |   2.5 G |   2.6 G |\n>\n> These repositories make good use of the cross-path deltas that come\n> about from the v1 name hash function, so they already had mixed results\n> with the v2 function. The v3 function is generally worse for these\n> repositories.\n\nI appreciate you sharing some counterexamples as well.\n\n> While the fluentui repo had an increase in size using the v3 name hash,\n> the others had modest improvements over the v2 name hash. But those\n> modest improvements are dwarfed by the difference from v1 to v2, so it\n> is unlikely that the regression seen in the other scenarios (packfiles\n> that are not from full repacks) will be worth using v3 over v2. That is,\n> unless there are enough collisions even with v2 that the full repack\n> scenario has larger improvements than these.\n\nThis is the paragraph that I thought most about (both while reading the\nabove sections, and then again after seeing my internal thoughts written\ndown here).\n\nIt seems like the general conclusion is that v2 is a strict improvement\non v1 in almost all cases. v3 appears to be an improvement on v2 in some\ncases, and a regression (as you note) in others. But I think more\nimportantly (again as you note) is that the improvement from v1 to v2 is\nso pronounced that it's unlikely that the regression from v2 to v3 will\nmatter or even be noticeable in most cases.\n\nAre there easy ways to detect when v3 would be an improvement over v2?\nIf so, then I think exposing those detection mechanisms to users (either\nas an automated tool or through documentation, perhaps in\ngit-packing(7), which is perfect for this sort of discussion) would be\nworthwhile. Then users could make an informed decision about which hash\nfunction to use for their repositories.\n\nBut if there isn't such a mechanism, then I wonder what would drive a\nuser to choose v3 over v2. I suspect the answer is that curious users\nwould try repacking both ways, and then stick with whichever one has a\nbigger impact on the metric(s) they care most about.\n\nIf that's the case, I suspect that v2 will be the dominant choice,\nespecially if we consider changing the default from 1 to 2 at some point\nin the future. Given all of that, I share your feeling that it may be\nworth dropping this patch entirely. It is true that some cases will be\nworse off (at least compared to v2) without this part of the series. But\nit gets us out of having to support v3 forever, or go through the\nprocess of deprecating it. I'd like the project to avoid both of those\nif possible, especially if we don't anticipate many users will select v3\nover v2.\n\nThanks,\nTaylor\n"},{"id":"511098","messageId":"Z5F/JdnSAYqUBJ8s@nand.local","threadId":"62447","inReplyTo":"35026c72-f9b4-40a3-b528-1c28b1238972@gmail.com","subject":"Re: [PATCH v3 0/8] pack-objects: Create an alternative name hash algorithm (recreated)","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-22T23:28:37Z","receivedAt":"2025-01-22T23:28:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jan 21, 2025 at 03:21:15PM -0500, Derrick Stolee wrote:\n> This series has been at this version for a while. I'm pretty sure that this\n> is the most promising direction we have at the moment for improving delta\n> compression for many users.\n>\n> The only decision point I think remains is whether or not to include the last\n> patch (--name-hash-version=3) which I would be happy either way.\n\nSorry that I punted on reviewing this for way longer than I should have,\nand thanks for bearing with me.\n\nI took a close look at this latest round of patches, skipping over the\nparts that I remembered from previous rounds. My memory is far from\nperfect, so I may have commented on things that we've already discussed,\nin which case I apologize :-).\n\nI left a handful of comments on the patches themselves, but they are\nmostly cosmetic. My idle thought before having a chance to review this\nseries is that the --name-hash-version option was handing over too much\ncontrol to the user without clear instruction on when to use one version\nover the other.\n\nAfter reviewing, I think the idea of having a versioned name-hash is a\ngood one, and I agree that it'll make the eventual .bitmap changes much\neasier to implement.\n\nSo I think in that sense exposing a `--name-hash-version` is the right\nthing to do. My feeling is that we should probably just add Jonathan's\n\"v2\", since it appears to be a improvement in nearly all cases against\nv1, and more often an improvement than not when compared to v3. In that\nworld, just introducing v2 leaves us with less code to maintain and\nfewer, clearer options presented to users.\n\nIf you feel strongly about keeping v3, I am definitely open to changing\nmy mind here, but my feeling on first blush of this most recent round is\nthat I would probably just include v2.\n\nI'm excited about seeing these patches land, and I am glad that someone\nis working on them!\n\nThanks,\nTaylor\n"},{"id":"511181","messageId":"d30f6373-e98b-4fec-9a73-eb3bb54b9380@gmail.com","threadId":"62447","inReplyTo":"Z5FugEXhdKjhwcnP@nand.local","subject":"Re: [PATCH v3 2/8] pack-objects: add --name-hash-version option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2025-01-24T17:29:03Z","receivedAt":"2025-01-24T17:29:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/22/25 5:17 PM, Taylor Blau wrote:\n> On Fri, Dec 20, 2024 at 05:19:48PM +0000, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee <stolee@gmail.com>\n\n>> +static inline uint32_t pack_name_hash_fn(const char *name)\n>> +{\n>> +\tswitch (name_hash_version)\n>> +\t{\n> \n> This is definitely a nitpick, but the opening curly brace should appear\n> on the preceding line. I don't think our CodingGuidelines explicitly say\n> this. But in my head the convention is that only function bodies have\n> their enclosing braces on their own line, as opposed to:\n\nYou're right, I messed up on the switch statement.\n\n>      static inline uint32_t pack_name_hash_fn(const char *name) {\n\nbut based on my expectations and your earlier comment I think that this\nsuggestion is a copy/paste error and you meant to highlight the switch\nstatement.\n\n>> +\tvalidate_name_hash_version();\n>> +\tif (write_bitmap_index && name_hash_version != 1) {\n>> +\t\twarning(_(\"currently, --write-bitmap-index requires --name-hash-version=1\"));\n>> +\t\tname_hash_version = 1;\n>> +\t}\n>> +\n> \n> Hmm. validate_name_hash_version() is its own function, which seems good,\n> but then we do further validation on it here. I wonder if we should\n> either move the latter step into validate_name_hash_version(), which\n> would require us to pass a pointer to name_hash_version, or assign it\n> the return value of validate_name_hash_version() (assuming it were\n> changed to return the appropriate type instead of void).\n> \n> I think that we are probably pretty far into bike-shedding territory,\n> but figured I'd share as it jumped out to me while reviewing.\n\nSounds good. I'll try to minimize the uses of name_hash_version outside\nof the cluster of methods that implement its details. This may become\nmore complicated later when the timing of these checks is more interesting.\n\nThanks,\n-Stolee\n\n"},{"id":"511182","messageId":"3a9b10f4-95b4-466e-9214-dff54d2e2123@gmail.com","threadId":"62447","inReplyTo":"Z5FzE1XpBlEyhK2T@nand.local","subject":"Re: [PATCH v3 8/8] pack-objects: add third name hash version","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2025-01-24T17:34:47Z","receivedAt":"2025-01-24T17:34:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/22/25 5:37 PM, Taylor Blau wrote:\n> On Fri, Dec 20, 2024 at 05:19:54PM +0000, Derrick Stolee via GitGitGadget wrote:\n>> Create a third name hash function and extend the '--name-hash-version'\n>> option in 'git pack-objects' and 'git repack' to understand it. This\n>> hash version abandons all efforts for locality and focuses on creating a\n>> somewhat uniformly-distributed hash function to minimize collisions.\n>>\n>> We can observe the effect of this collision avoidance in a large\n>> internal monorepo that suffered from collisions in the previous\n>> versions. The updates to p5314-name-hash.sh show these results:\n>>\n>> Test                               this tree\n>> --------------------------------------------------\n>> 5314.1: paths at head                       227.3K\n>> 5314.2: distinct hash value: v1              72.3K\n>> 5314.3: maximum multiplicity: v1             14.4K\n>> 5314.4: distinct hash value: v2             166.5K\n>> 5314.5: maximum multiplicity: v2               138\n>> 5314.6: distinct hash value: v3             227.3K\n>> 5314.7: maximum multiplicity: v3                 2\n>>\n>> These results demonstrate that of the 227,000+ paths, nearly all of them\n>> find distinct hash values. The maximum multiplicity is 2, improved from\n>> 138 in the v2 hash function. The v2 hash function also had only 166K\n>> distinct values, so it had a wide spread of collisions.\n> \n> I had a little trouble reading this section of the commit message. I\n> think the framing makes sense (v2 has collisions which can impact pack\n> generation time and/or size), but this section explains v3 I think one\n> level too deep.\n> \n> This comparison (and the one below it for v3) shows a reduction in\n> distinct hash values and the maximum multiplicity (I'm assuming for\n> colliding hash values, in which case I might suggest renaming it as\n> \"maximum collisions\").\n> \n> But I imagine that many readers will primarily care about the effect of\n> the new hash function on pack generation time and size. You show that\n> below, but I think that it should potentially appear earlier in the\n> commit message.\n> \n> Alternatively, you could consider leaving the time/size table alone\n> where it is, and devote an extra sentence or two to explaining the\n> impact on repacking time/size that the two metrics above (distinct hash\n> values, multiplicity/collisions) have on the repacking time/size.\n> \n>> A more modest improvement is available in the open source fluentui repo\n>> [1] with these results:\n>>\n>> Test                               this tree\n>> --------------------------------------------------\n>> 5314.1: paths at head                        19.5K\n>> 5314.2: distinct hash value: v1               8.2K\n>> 5314.3: maximum multiplicity: v1               279\n>> 5314.4: distinct hash value: v2              17.8K\n>> 5314.5: maximum multiplicity: v2                44\n>> 5314.6: distinct hash value: v3              19.5K\n>> 5314.7: maximum multiplicity: v3                 1\n>>\n>> [1] https://github.com/microsoft/fluentui\n>>\n>> However, it is important to demonstrate the effectiveness of this\n>> function in the context of compressing a repository. We can use\n>> p5313-pack-objects.sh to measure these changes. I will use a simplified\n>> table summarizing the output of that performance test.\n>>\n>>   | Test      | V1 Time | V2 Time | V3 Time | V1 Size | V2 Size | V3 Size |\n>>   |-----------|---------|---------|---------|---------|---------|---------|\n>>   | Thin Pack |  0.37 s |  0.12 s |  0.07 s |   1.2 M |  22.0 K |  20.4 K |\n>>   | Big Pack  |  2.04 s |  2.80 s |  1.40 s |  20.4 M |  25.9 M |  19.2 M |\n>>   | Shallow   |  1.41 s |  1.77 s |  1.27 s |  34.4 M |  33.7 M |  34.8 M |\n>>   | Repack    | 95.70 s | 33.68 s | 20.88 s | 439.3 M | 160.5 M | 169.1 M |\n> \n> OK, now we get to the chart that I demonstrates the effects of each hash\n> function on the most externally visible effects. Are these measurements\n> taken from the fluentui repo, or somewhere else? In either case, it\n> may be worth mentioning.\n> \n>> Here, there are some performance improvements on a time basis, and the\n>> thin and big packs are somewhat smaller in v3. The shallow and repacked\n>> packs are somewhat bigger, though, compared to v2.\n>>\n>> Two repositories that have very few collisions in the v1 name hash are\n>> the Git and Linux repositories. Here are their stats for p5313:\n>>\n>> Git:\n>>\n>>   | Test      | V1 Time | V2 Time | V3 Time | V1 Size | V2 Size | V3 Size |\n>>   |-----------|---------|---------|---------|---------|---------|---------|\n>>   | Thin Pack |  0.02 s |  0.02 s |  0.02 s |   1.1 K |   1.1 K |  15.3 K |\n>>   | Big Pack  |  1.69 s |  1.95 s |  1.67 s |  13.5 M |  14.5 M |  14.9 M |\n>>   | Shallow   |  1.26 s |  1.29 s |  1.16 s |  12.0 M |  12.2 M |  12.5 M |\n>>   | Repack    | 29.51 s | 29.01 s | 29.08 s | 237.7 M | 238.2 M | 237.7 M |\n>>\n>> Linux:\n>>\n>>   | Test      | V1 Time  | V2 Time  | V3 Time  | V1 Size | V2 Size | V3 Size |\n>>   |-----------|----------|----------|----------|---------|---------|---------|\n>>   | Thin Pack |   0.17 s |   0.07 s |   0.07 s |   4.6 K |   4.6 K |   6.8 K |\n>>   | Big Pack  |  17.88 s |  12.35 s |  12.14 s | 201.1 M | 149.1 M | 160.4 M |\n>>   | Shallow   |  11.05 s |  22.94 s |  22.16 s | 269.2 M | 273.8 M | 271.8 M |\n>>   | Repack    | 727.39 s | 566.95 s | 539.33 s |   2.5 G |   2.5 G |   2.6 G |\n>>\n>> These repositories make good use of the cross-path deltas that come\n>> about from the v1 name hash function, so they already had mixed results\n>> with the v2 function. The v3 function is generally worse for these\n>> repositories.\n> \n> I appreciate you sharing some counterexamples as well.\n> \n>> While the fluentui repo had an increase in size using the v3 name hash,\n>> the others had modest improvements over the v2 name hash. But those\n>> modest improvements are dwarfed by the difference from v1 to v2, so it\n>> is unlikely that the regression seen in the other scenarios (packfiles\n>> that are not from full repacks) will be worth using v3 over v2. That is,\n>> unless there are enough collisions even with v2 that the full repack\n>> scenario has larger improvements than these.\n> \n> This is the paragraph that I thought most about (both while reading the\n> above sections, and then again after seeing my internal thoughts written\n> down here).\n> \n> It seems like the general conclusion is that v2 is a strict improvement\n> on v1 in almost all cases. v3 appears to be an improvement on v2 in some\n> cases, and a regression (as you note) in others. But I think more\n> importantly (again as you note) is that the improvement from v1 to v2 is\n> so pronounced that it's unlikely that the regression from v2 to v3 will\n> matter or even be noticeable in most cases.\n> \n> Are there easy ways to detect when v3 would be an improvement over v2?\n> If so, then I think exposing those detection mechanisms to users (either\n> as an automated tool or through documentation, perhaps in\n> git-packing(7), which is perfect for this sort of discussion) would be\n> worthwhile. Then users could make an informed decision about which hash\n> function to use for their repositories.\n> \n> But if there isn't such a mechanism, then I wonder what would drive a\n> user to choose v3 over v2. I suspect the answer is that curious users\n> would try repacking both ways, and then stick with whichever one has a\n> bigger impact on the metric(s) they care most about.\n> \n> If that's the case, I suspect that v2 will be the dominant choice,\n> especially if we consider changing the default from 1 to 2 at some point\n> in the future. Given all of that, I share your feeling that it may be\n> worth dropping this patch entirely. It is true that some cases will be\n> worse off (at least compared to v2) without this part of the series. But\n> it gets us out of having to support v3 forever, or go through the\n> process of deprecating it. I'd like the project to avoid both of those\n> if possible, especially if we don't anticipate many users will select v3\n> over v2.\n\nThank you for these detailed considerations. The most important one, in my\nopinion is this:\n\n > Are there easy ways to detect when v3 would be an improvement over v2?\n > If so, then I think exposing those detection mechanisms to users\n > ...would be worthwhile.\n\nI agree that having those detection mechanisms would be good. The test\nhelpers can provide some of the information that helps make that\ndecision, but doesn't form opinions or recommend thresholds for one\nover another.\n\nI agree with your overall thought that we should eject this patch (for\nnow) and focus on the v2 as something that will help most users. We can\nlearn from that and use that to inform any future iterations built on\nthis framework.\n\nThanks,\n-Stolee\n\n"},{"id":"511184","messageId":"7fe5f33c-4923-42a4-b98e-e7c2116a2782@gmail.com","threadId":"62447","inReplyTo":"Z5F/JdnSAYqUBJ8s@nand.local","subject":"Re: [PATCH v3 0/8] pack-objects: Create an alternative name hash algorithm (recreated)","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2025-01-24T17:45:49Z","receivedAt":"2025-01-24T17:45:51Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/22/25 6:28 PM, Taylor Blau wrote:\n> On Tue, Jan 21, 2025 at 03:21:15PM -0500, Derrick Stolee wrote:\n\n> Sorry that I punted on reviewing this for way longer than I should have,\n> and thanks for bearing with me.\n\nI wanted to give people time to recover from release mechanics before\npoking this series again. Thanks for reviewing so quickly after my\nmessage.\n\n> I left a handful of comments on the patches themselves, but they are\n> mostly cosmetic. \n\nThanks. I have prepped a v4 with those cosmetic updates and will intend\nto send it on Monday, unless there are more comments before then.\n\n> After reviewing, I think the idea of having a versioned name-hash is a\n> good one, and I agree that it'll make the eventual .bitmap changes much\n> easier to implement.\n\nThanks!\n\n[I reordered a paragraph below]\n\n> My idle thought before having a chance to review this\n> series is that the --name-hash-version option was handing over too much\n> control to the user without clear instruction on when to use one version\n> over the other.\n...\n> So I think in that sense exposing a `--name-hash-version` is the right\n> thing to do. My feeling is that we should probably just add Jonathan's\n> \"v2\", since it appears to be a improvement in nearly all cases against\n> v1, and more often an improvement than not when compared to v3. In that\n> world, just introducing v2 leaves us with less code to maintain and\n> fewer, clearer options presented to users.\n\nI think these ideas are related. The thought that really convinced me\nthat v3 isn't worth it right now is that users won't know which version to\nuse without some kind of opinion being voiced by tooling. If users assume\nthat \"newer is better\" then they may accidentally get into a worse\nsituation by defaulting to v3 over v2. It's unsatisfying to say \"try both\nand see which is better\" especially when v3 is rarely better.\n\nI'll drop the last patch in the next version, but I'll keep it in my fork\nfor possible future resurrection.\n\nThanks,\n-Stolee\n\n"},{"id":"511288","messageId":"pull.1823.v4.git.1738004554.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.v3.git.1734715194.gitgitgadget@gmail.com","subject":"[PATCH v4 0/7] pack-objects: Create an alternative name hash algorithm (recreated)","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-01-27T19:02:27Z","receivedAt":"2025-01-27T19:02:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is a recreation of the topic in [1] that was closed. (I force-pushed my\nbranch and GitHub won't let me reopen the PR for GitGitGadget to create this\nas v3.)\n\n[1]\nhttps://lore.kernel.org/git/pull.1785.v2.git.1726692381.gitgitgadget@gmail.com/\n\nI've been focused recently on understanding and mitigating the growth of a\nfew internal repositories. Some of these are growing much larger than\nexpected for the number of contributors, and there are multiple aspects to\nwhy this growth is so large.\n\nThis is part of the RFC I submitted [2] involving the path-walk API, though\nthis doesn't use the path-walk API directly. In full repack cases, it seems\nthat the --full-name-hash option gets nearly as good compression as the\n--path-walk option introduced in that series. I continue to work on that\nfeature as well, so we can review it after this series is complete.\n\n[2]\nhttps://lore.kernel.org/git/pull.1786.git.1725935335.gitgitgadget@gmail.com/\n\nThe main issue plaguing these repositories is that deltas are not being\ncomputed against objects that appear at the same path. While the size of\nthese files at tip is one aspect of growth that would prevent this issue,\nthe changes to these files are reasonable and should result in good delta\ncompression. However, Git is not discovering the connections across\ndifferent versions of the same file.\n\nOne way to find some improvement in these repositories is to increase the\nwindow size, which was an initial indicator that the delta compression could\nbe improved, but was not a clear indicator. After some digging (and\nprototyping some analysis tools) the main discovery was that the current\nname-hash algorithm only considers the last 16 characters in the path name\nand has some naturally-occurring collisions within that scope.\n\nThis series creates a mechanism to select alternative name hashes using a\nnew --name-hash-version=<n> option. The versions are:\n\n 1. Version 1 is the default name hash that already exists. This option\n    focuses on the final bytes of the path to maximize locality for\n    cross-path deltas.\n\n 2. Version 2 is the new path-component hash function suggested by Jonathan\n    Tan in the previous version (with some modifications). This hash\n    function essentially computes the v1 name hash of each path component\n    and then overlays those hashes with a shift to make the parent\n    directories contribute less to the final hash, but enough to break many\n    collisions that exist in v1.\n\n 3. Version 3 is the hash function that I submitted under the\n    --full-name-hash feature in the previous versions. This uses a\n    pseudorandom hash procedure to minimize collisions but at the expense of\n    losing on locality. This version is implemented in the final patch of\n    the series mostly for comparison purposes, as it is unlikely to be\n    selected as a valuable hash function over v2. The final patch could be\n    omitted from the merged version.\n\nSee the patches themselves for detailed results in the p5313-pack-objects.sh\nperformance test and the p5314-name-hash.sh test that demonstrates how many\ncollisions occur with each hash function.\n\nIn general, the v2 name hash function gets very close to the compression\nresults of v3 in the full repack case, even in the repositories that feature\nmany name hash collisions. These benefits come as well without downsides to\nother kinds of packfiles, including small pushed packs, larger incremental\nfetch packs, and shallow clones.\n\nI should point out that there is still a significant jump in compression\neffectiveness between these name hash version options and the --path-walk\nfeature I suggested in my RFC [2] and has review underway in [3] (along with\nchanges to git pack-objects and git repack in [4]).\n\n[3]\nhttps://lore.kernel.org/git/pull.1818.v2.git.1731181272.gitgitgadget@gmail.com/\n\n[4] https://github.com/gitgitgadget/git/pull/1819\n\nTo compare these options in a set of Javascript repositories that have\ndifferent levels of name hash collisions, see the following table that lists\nthe size of the packfile after git repack -adf\n[--name-hash-version=<n>|--path-walk]:\n\n| Repo     | V1 Size   | V2 Size | V3 Size | Path Walk Size |\n|----------|-----------|---------|---------|----------------|\n| fluentui |     440 M |   161 M |   170 M |          123 M |\n| Repo B   |   6,248 M |   856 M |   840 M |          782 M |\n| Repo C   |  37,278 M | 6,921 M | 6,755 M |        6,156 M |\n| Repo D   | 131,204 M | 7,463 M | 7,124 M |        4,005 M |\n\n\nAs we can see, v2 nearly reaches the effectiveness of v3 (and outperforms it\nonce!) but there is still a significant change between the\n--name-hash-version feature and the --path-walk feature.\n\nThe main reason we are considering this --name-hash-version feature is that\nit has the least amount of stretch required in order for it to be integrated\nwith reachability bitmaps, required for server environments. In fact, the\nchange in this version to use a numerical version makes it more obvious how\nto connect the version number to a value in the .bitmap file format. Tests\nare added to guarantee that the hash functions preserve their behavior over\ntime, since data files depend on that.\n\nThanks, -Stolee\n\n\nUPDATES SINCE V1\n================\n\n * BIG CHANGE: --full-name-hash is replaced with --name-hash-version=<n>.\n\n * --name-hash-version=2 uses Jonathan Tan's hash function (with some\n   adjustments). See the first patch for this implementation, credited to\n   him.\n\n * --name-hash-version=3 uses the hash function I wrote for the previous\n   version's --full-name-hash. This is left as the final patch so it could\n   be easily removed from the series if not considered worth having since it\n   has some pain points that are resolved from v2 without significant issues\n   to overall repo size.\n\n * Commit messaes are updated with these changes, as well as a better\n   attempt to indicate the benefit of cross-path delta pairs, such as\n   renames or similar content based on file extension.\n\n * Performance numbers are regenerated for the same set of repositories.\n   Size data is somewhat nondeterministic due to concurrent threads\n   competing over delta computations.\n\n * The --name-hash-version option is not added to git repack until its own\n   patch.\n\n * The patch that updates git repack's synopsis match its docs is squashed\n   into the patch that adds the option to git repack.\n\n * Documentation is expanded for git pack-objects and reused for git repack.\n\n * GIT_TEST_FULL_NAME_HASH is now GIT_TEST_NAME_HASH_VERSION with similar\n   caveats required for tests. It is removed from the linux-TEST-vars CI\n   job.\n\n * The performance test p5313-pack-objects.sh is now organized via a loop\n   over the different versions. This separates the scenarios, which makes\n   things harder to compare directly, but makes it trivial to add new\n   versions.\n\n * The patch that disabled --full-name-hash when performing a shallow clone\n   is no longer present, as it is not necessary when using\n   --name-hash-version=2. Perhaps it would be valuable for repo using v3, if\n   that is kept in the series.\n\n * We force name hash version 1 when writing or reading bitmaps.\n\n * A small patch is added to cause a BUG() failure if the name hash version\n   global changes between calls to pack_name_hash_fn(). This is solely\n   defensive programming.\n\n * Several typos, style issues, or suggested comments are resolved.\n\n\nUPDATES SINCE v2\n================\n\n * For extra safety, the new name-hash algorithm uses unsigned characters.\n\n * A stray 'full_name_hash' variable is removed.\n\n * Commit messages are improved.\n\n * 'git repack' documentation now points to 'git pack-objects' docs for the\n   --name-hash-version option.\n\n * The changes to t5616 are delayed until the introduction of version 3,\n   since the v2 name hash does not demonstrate the behavior.\n\n\nUPDATES SINCE v3\n================\n\n * Style fixes for switch statement and setting a test environment variable.\n\n * validate_name_hash_version() is now responsible for checking\n   compatibility with other options.\n\n * The --name-hash-version=3 patch is removed to avoid user confusion since\n   we don't have a clear way to predict when it would provide (modest)\n   improvements over v2.\n\nDerrick Stolee (6):\n  pack-objects: add --name-hash-version option\n  repack: add --name-hash-version option\n  pack-objects: add GIT_TEST_NAME_HASH_VERSION\n  p5313: add size comparison test\n  test-tool: add helper for name-hash values\n  pack-objects: prevent name hash version change\n\nJonathan Tan (1):\n  pack-objects: create new name-hash function version\n\n Documentation/git-pack-objects.txt | 32 +++++++++++++-\n Documentation/git-repack.txt       |  9 +++-\n Makefile                           |  1 +\n builtin/pack-objects.c             | 63 ++++++++++++++++++++++++---\n builtin/repack.c                   |  9 +++-\n pack-objects.h                     | 28 ++++++++++++\n t/README                           |  4 ++\n t/helper/test-name-hash.c          | 23 ++++++++++\n t/helper/test-tool.c               |  1 +\n t/helper/test-tool.h               |  1 +\n t/perf/p5313-pack-objects.sh       | 70 ++++++++++++++++++++++++++++++\n t/perf/p5314-name-hash.sh          | 31 +++++++++++++\n t/t0450/txt-help-mismatches        |  1 -\n t/t5300-pack-object.sh             | 34 +++++++++++++++\n t/t5310-pack-bitmaps.sh            | 35 ++++++++++++++-\n t/t5333-pseudo-merge-bitmaps.sh    |  3 +-\n t/t5510-fetch.sh                   |  7 ++-\n t/t6020-bundle-misc.sh             |  6 ++-\n t/t7406-submodule-update.sh        |  4 +-\n t/t7700-repack.sh                  | 16 ++++++-\n t/test-lib-functions.sh            | 26 +++++++++++\n 21 files changed, 388 insertions(+), 16 deletions(-)\n create mode 100644 t/helper/test-name-hash.c\n create mode 100755 t/perf/p5313-pack-objects.sh\n create mode 100755 t/perf/p5314-name-hash.sh\n\n\nbase-commit: 8f8d6eee531b3fa1a8ef14f169b0cb5035f7a772\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1823%2Fderrickstolee%2Ffull-name-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1823/derrickstolee/full-name-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/1823\n\nRange-diff vs v3:\n\n 1:  68b4127580e = 1:  68b4127580e pack-objects: create new name-hash function version\n 2:  d035e3e59f4 ! 2:  7ee1845144f pack-objects: add --name-hash-version option\n     @@ Commit message\n          .bitmap file format, so we must force --name-hash-version=1 when bitmaps\n          are being read or written. Later, the bitmap format could be updated to\n          be aware of the name hash version so deltas can be quickly computed\n     -    across the bitmapped/not-bitmapped boundary.\n     +    across the bitmapped/not-bitmapped boundary. To promote the safety of\n     +    this parameter, the validate_name_hash_version() method will die() if\n     +    the given name-hash version is incorrect and will disable newer versions\n     +    if not yet compatible with other features, such as --write-bitmap-index.\n      \n          Signed-off-by: Derrick Stolee <stolee@gmail.com>\n      \n     @@ builtin/pack-objects.c: struct configured_exclusion {\n       static struct oidset excluded_by_config;\n      +static int name_hash_version = 1;\n      +\n     ++/**\n     ++ * Check whether the name_hash_version chosen by user input is apporpriate,\n     ++ * and also validate whether it is appropriate with other features.\n     ++ */\n      +static void validate_name_hash_version(void)\n      +{\n      +\tif (name_hash_version < 1 || name_hash_version > 2)\n      +\t\tdie(_(\"invalid --name-hash-version option: %d\"), name_hash_version);\n     ++\tif (write_bitmap_index && name_hash_version != 1) {\n     ++\t\twarning(_(\"currently, --write-bitmap-index requires --name-hash-version=1\"));\n     ++\t\tname_hash_version = 1;\n     ++\t}\n      +}\n      +\n      +static inline uint32_t pack_name_hash_fn(const char *name)\n      +{\n     -+\tswitch (name_hash_version)\n     -+\t{\n     ++\tswitch (name_hash_version) {\n      +\tcase 1:\n      +\t\treturn pack_name_hash(name);\n      +\n     @@ builtin/pack-objects.c: int cmd_pack_objects(int argc,\n       \t\twrite_bitmap_index = 0;\n       \n      +\tvalidate_name_hash_version();\n     -+\tif (write_bitmap_index && name_hash_version != 1) {\n     -+\t\twarning(_(\"currently, --write-bitmap-index requires --name-hash-version=1\"));\n     -+\t\tname_hash_version = 1;\n     -+\t}\n      +\n       \tif (use_delta_islands)\n       \t\tstrvec_push(&rp, \"--topo-order\");\n 3:  e2191244f6b = 3:  a4f3d127686 repack: add --name-hash-version option\n 4:  86ff0d0a15e ! 4:  a507be2f19b pack-objects: add GIT_TEST_NAME_HASH_VERSION\n     @@ builtin/pack-objects.c: struct configured_exclusion {\n      -static int name_hash_version = 1;\n      +static int name_hash_version = -1;\n       \n     - static void validate_name_hash_version(void)\n     - {\n     + /**\n     +  * Check whether the name_hash_version chosen by user input is apporpriate,\n      @@ builtin/pack-objects.c: int cmd_pack_objects(int argc,\n       \tif (pack_to_stdout || !rev_list_all)\n       \t\twrite_bitmap_index = 0;\n     @@ builtin/pack-objects.c: int cmd_pack_objects(int argc,\n      +\t\tname_hash_version = (int)git_env_ulong(\"GIT_TEST_NAME_HASH_VERSION\", 1);\n      +\n       \tvalidate_name_hash_version();\n     - \tif (write_bitmap_index && name_hash_version != 1) {\n     - \t\twarning(_(\"currently, --write-bitmap-index requires --name-hash-version=1\"));\n     + \n     + \tif (use_delta_islands)\n      \n       ## t/README ##\n      @@ t/README: a test and then fails then the whole test run will abort. This can help to make\n     @@ t/t5333-pseudo-merge-bitmaps.sh: test_expect_success 'bitmapPseudoMerge.stableTh\n       '\n       \n       test_expect_success 'out of order thresholds are rejected' '\n     -+\t# Disable this option to avoid stderr message\n     -+\tGIT_TEST_NAME_HASH_VERSION=1 &&\n     -+\texport GIT_TEST_NAME_HASH_VERSION &&\n     -+\n     - \ttest_must_fail git \\\n     +-\ttest_must_fail git \\\n     ++\t# Disable the test var to remove a stderr message.\n     ++\ttest_must_fail env GIT_TEST_NAME_HASH_VERSION=1 git \\\n       \t\t-c bitmapPseudoMerge.test.pattern=\"refs/*\" \\\n       \t\t-c bitmapPseudoMerge.test.threshold=1.month.ago \\\n     + \t\t-c bitmapPseudoMerge.test.stableThreshold=1.week.ago \\\n      \n       ## t/t5510-fetch.sh ##\n      @@ t/t5510-fetch.sh: test_expect_success 'all boundary commits are excluded' '\n 5:  163aaab3e1b = 5:  a0247a679ac p5313: add size comparison test\n 6:  e9ce79fa6e7 = 6:  06a95063186 test-tool: add helper for name-hash values\n 7:  18a41f2fe6f ! 7:  bab2ac31880 pack-objects: prevent name hash version change\n     @@ builtin/pack-objects.c: static void validate_name_hash_version(void)\n      +\t\tBUG(\"name hash version changed from %d to %d mid-process\",\n      +\t\t    seen_version, name_hash_version);\n      +\n     - \tswitch (name_hash_version)\n     - \t{\n     + \tswitch (name_hash_version) {\n       \tcase 1:\n     + \t\treturn pack_name_hash(name);\n 8:  3d63954f318 < -:  ----------- pack-objects: add third name hash version\n\n-- \ngitgitgadget\n"},{"id":"511289","messageId":"68b4127580e2d475bec0d7cd0f6a9ae5e626b3c9.1738004555.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.v4.git.1738004554.gitgitgadget@gmail.com","subject":"[PATCH v4 1/7] pack-objects: create new name-hash function version","fromName":"Jonathan Tan via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-01-27T19:02:28Z","receivedAt":"2025-01-27T19:02:40Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"From: Jonathan Tan <jonathantanmy@google.com>\n\nAs we will explore in later changes, the default name-hash function used\nin 'git pack-objects' has a tendency to cause collisions and cause poor\ndelta selection. This change creates an alternative that avoids some\ncollisions while preserving some amount of hash locality.\n\nThe pack_name_hash() method has not been materially changed since it was\nintroduced in ce0bd64 (pack-objects: improve path grouping\nheuristics., 2006-06-05). The intention here is to group objects by path\nname, but also attempt to group similar file types together by making\nthe most-significant digits of the hash be focused on the final\ncharacters.\n\nHere's the crux of the implementation:\n\n\t/*\n\t * This effectively just creates a sortable number from the\n\t * last sixteen non-whitespace characters. Last characters\n\t * count \"most\", so things that end in \".c\" sort together.\n\t */\n\twhile ((c = *name++) != 0) {\n\t\tif (isspace(c))\n\t\t\tcontinue;\n\t\thash = (hash >> 2) + (c << 24);\n\t}\n\nAs the comment mentions, this only cares about the last sixteen\nnon-whitespace characters. This cause some filenames to collide more than\nothers. This collision is somewhat by design in order to promote hash\nlocality for files that have similar types (.c, .h, .json) or could be the\nsame file across a directory rename (a/foo.txt to b/foo.txt). This leads to\ndecent cross-path deltas in cases like shallow clones or packing a\nrepository with very few historical versions of files that share common data\nwith other similarly-named files.\n\nHowever, when the name-hash instead leads to a large number of name-hash\ncollisions for otherwise unrelated files, this can lead to confusing the\ndelta calculation to prefer cross-path deltas over previous versions of the\nsame file.\n\nThe new pack_name_hash_v2() function attempts to fix this issue by\ntaking more of the directory path into account through its hash\nfunction. Its naming implies that we will later wire up details for\nchoosing a name-hash function by version.\n\nThe first change is to be more careful about paths using non-ASCII\ncharacters. With these characters in mind, reverse the bits in the byte\nas the least-significant bits have the highest entropy and we want to\nmaximize their influence. This is done with some bit manipulation that\nswaps the two halves, then the quarters within those halves, and then\nthe bits within those quarters.\n\nThe second change is to perform hash composition operations at every\nlevel of the path. This is done by storing a 'base' hash value that\ncontains the hash of the parent directory. When reaching a directory\nboundary, we XOR the current level's name-hash value with a downshift of\nthe previous level's hash. This perturbation intends to create low-bit\ndistinctions for paths with the same final 16 bytes but distinct parent\ndirectory structures.\n\nThe collision rate and effectiveness of this hash function will be\nexplored in later changes as the function is integrated with 'git\npack-objects' and 'git repack'.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n pack-objects.h | 28 ++++++++++++++++++++++++++++\n 1 file changed, 28 insertions(+)\n\ndiff --git a/pack-objects.h b/pack-objects.h\nindex b9898a4e64b..681c1116486 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -207,6 +207,34 @@ static inline uint32_t pack_name_hash(const char *name)\n \treturn hash;\n }\n \n+static inline uint32_t pack_name_hash_v2(const unsigned char *name)\n+{\n+\tuint32_t hash = 0, base = 0, c;\n+\n+\tif (!name)\n+\t\treturn 0;\n+\n+\twhile ((c = *name++)) {\n+\t\tif (isspace(c))\n+\t\t\tcontinue;\n+\t\tif (c == '/') {\n+\t\t\tbase = (base >> 6) ^ hash;\n+\t\t\thash = 0;\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * 'c' is only a single byte. Reverse it and move\n+\t\t\t * it to the top of the hash, moving the rest to\n+\t\t\t * less-significant bits.\n+\t\t\t */\n+\t\t\tc = (c & 0xF0) >> 4 | (c & 0x0F) << 4;\n+\t\t\tc = (c & 0xCC) >> 2 | (c & 0x33) << 2;\n+\t\t\tc = (c & 0xAA) >> 1 | (c & 0x55) << 1;\n+\t\t\thash = (hash >> 2) + (c << 24);\n+\t\t}\n+\t}\n+\treturn (base >> 6) ^ hash;\n+}\n+\n static inline enum object_type oe_type(const struct object_entry *e)\n {\n \treturn e->type_valid ? e->type_ : OBJ_BAD;\n-- \ngitgitgadget\n\n"},{"id":"511290","messageId":"a4f3d1276861ff0b96e1ffe94b8ff6db1df24729.1738004555.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.v4.git.1738004554.gitgitgadget@gmail.com","subject":"[PATCH v4 3/7] repack: add --name-hash-version option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-01-27T19:02:30Z","receivedAt":"2025-01-27T19:02:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nThe new '--name-hash-version' option for 'git repack' is a simple\npass-through to the underlying 'git pack-objects' subcommand. However,\nthis subcommand may have other options and a temporary filename as part\nof the subcommand execution that may not be predictable or could change\nover time.\n\nThe existing test_subcommand method requires an exact list of arguments\nfor the subcommand. This is too rigid for our needs here, so create a\nnew method, test_subcommand_flex. Use it to check that the\n--name-hash-version option is passing through.\n\nSince we are modifying the 'git repack' command, let's bring its usage\nin line with the Documentation's synopsis. This removes it from the\nallow list in t0450 so it will remain in sync in the future.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Documentation/git-repack.txt |  9 ++++++++-\n builtin/repack.c             |  9 ++++++++-\n t/t0450/txt-help-mismatches  |  1 -\n t/t7700-repack.sh            |  6 ++++++\n t/test-lib-functions.sh      | 26 ++++++++++++++++++++++++++\n 5 files changed, 48 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex c902512a9e8..5852a5c9736 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -9,7 +9,9 @@ git-repack - Pack unpacked objects in a repository\n SYNOPSIS\n --------\n [verse]\n-'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m] [--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>] [--write-midx]\n+'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\n+\t[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\n+\t[--write-midx] [--name-hash-version=<n>]\n \n DESCRIPTION\n -----------\n@@ -249,6 +251,11 @@ linkgit:git-multi-pack-index[1]).\n \tWrite a multi-pack index (see linkgit:git-multi-pack-index[1])\n \tcontaining the non-redundant packs.\n \n+--name-hash-version=<n>::\n+\tProvide this argument to the underlying `git pack-objects` process.\n+\tSee linkgit:git-pack-objects[1] for full details.\n+\n+\n CONFIGURATION\n -------------\n \ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex d6bb37e84ae..5e7ff919c1a 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -39,7 +39,9 @@ static int run_update_server_info = 1;\n static char *packdir, *packtmp_name, *packtmp;\n \n static const char *const git_repack_usage[] = {\n-\tN_(\"git repack [<options>]\"),\n+\tN_(\"git repack [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\\n\"\n+\t   \"[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\\n\"\n+\t   \"[--write-midx] [--name-hash-version=<n>]\"),\n \tNULL\n };\n \n@@ -58,6 +60,7 @@ struct pack_objects_args {\n \tint no_reuse_object;\n \tint quiet;\n \tint local;\n+\tint name_hash_version;\n \tstruct list_objects_filter_options filter_options;\n };\n \n@@ -306,6 +309,8 @@ static void prepare_pack_objects(struct child_process *cmd,\n \t\tstrvec_pushf(&cmd->args, \"--no-reuse-delta\");\n \tif (args->no_reuse_object)\n \t\tstrvec_pushf(&cmd->args, \"--no-reuse-object\");\n+\tif (args->name_hash_version)\n+\t\tstrvec_pushf(&cmd->args, \"--name-hash-version=%d\", args->name_hash_version);\n \tif (args->local)\n \t\tstrvec_push(&cmd->args,  \"--local\");\n \tif (args->quiet)\n@@ -1203,6 +1208,8 @@ int cmd_repack(int argc,\n \t\t\t\tN_(\"pass --no-reuse-delta to git-pack-objects\")),\n \t\tOPT_BOOL('F', NULL, &po_args.no_reuse_object,\n \t\t\t\tN_(\"pass --no-reuse-object to git-pack-objects\")),\n+\t\tOPT_INTEGER(0, \"name-hash-version\", &po_args.name_hash_version,\n+\t\t\t\tN_(\"specify the name hash version to use for grouping similar objects by path\")),\n \t\tOPT_NEGBIT('n', NULL, &run_update_server_info,\n \t\t\t\tN_(\"do not run git-update-server-info\"), 1),\n \t\tOPT__QUIET(&po_args.quiet, N_(\"be quiet\")),\ndiff --git a/t/t0450/txt-help-mismatches b/t/t0450/txt-help-mismatches\nindex 28003f18c92..c4a15fd0cb8 100644\n--- a/t/t0450/txt-help-mismatches\n+++ b/t/t0450/txt-help-mismatches\n@@ -45,7 +45,6 @@ rebase\n remote\n remote-ext\n remote-fd\n-repack\n reset\n restore\n rev-parse\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex c4c3d1a15d9..b9a5759e01d 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -777,6 +777,12 @@ test_expect_success 'repack -ad cleans up old .tmp-* packs' '\n \ttest_must_be_empty tmpfiles\n '\n \n+test_expect_success '--name-hash-version option passes through to pack-objects' '\n+\tGIT_TRACE2_EVENT=\"$(pwd)/hash-trace.txt\" \\\n+\t\tgit repack -a --name-hash-version=2 &&\n+\ttest_subcommand_flex git pack-objects --name-hash-version=2 <hash-trace.txt\n+'\n+\n test_expect_success 'setup for update-server-info' '\n \tgit init update-server-info &&\n \ttest_commit -C update-server-info message\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 78e054ab503..af47247f25f 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1886,6 +1886,32 @@ test_subcommand () {\n \tfi\n }\n \n+# Check that the given subcommand was run with the given set of\n+# arguments in order (but with possible extra arguments).\n+#\n+#\ttest_subcommand_flex [!] <command> <args>... < <trace>\n+#\n+# If the first parameter passed is !, this instead checks that\n+# the given command was not called.\n+#\n+test_subcommand_flex () {\n+\tlocal negate=\n+\tif test \"$1\" = \"!\"\n+\tthen\n+\t\tnegate=t\n+\t\tshift\n+\tfi\n+\n+\tlocal expr=\"$(printf '\"%s\".*' \"$@\")\"\n+\n+\tif test -n \"$negate\"\n+\tthen\n+\t\t! grep \"\\[$expr\\]\"\n+\telse\n+\t\tgrep \"\\[$expr\\]\"\n+\tfi\n+}\n+\n # Check that the given command was invoked as part of the\n # trace2-format trace on stdin.\n #\n-- \ngitgitgadget\n\n"},{"id":"511291","messageId":"7ee1845144fda5b8192dfa13eaab3cbd669b39ed.1738004555.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.v4.git.1738004554.gitgitgadget@gmail.com","subject":"[PATCH v4 2/7] pack-objects: add --name-hash-version option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-01-27T19:02:29Z","receivedAt":"2025-01-27T19:02:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nThe previous change introduced a new pack_name_hash_v2() function that\nintends to satisfy much of the hash locality features of the existing\npack_name_hash() function while also distinguishing paths with similar\nfinal components of their paths.\n\nThis change adds a new --name-hash-version option for 'git pack-objects'\nto allow users to select their preferred function version. This use of\nan integer version allows for future expansion and a direct way to later\nstore a name hash version in the .bitmap format.\n\nFor now, let's consider how effective this mechanism is when repacking a\nrepository with different name hash versions. Specifically, we will\nexecute 'git pack-objects' the same way a 'git repack -adf' process\nwould, except we include --name-hash-version=<n> for testing.\n\nOn the Git repository, we do not expect much difference. All path names\nare short. This is backed by our results:\n\n| Stage                 | Pack Size | Repack Time |\n|-----------------------|-----------|-------------|\n| After clone           | 260 MB    | N/A         |\n| --name-hash-version=1 | 127 MB    | 129s        |\n| --name-hash-version=2 | 127 MB    | 112s        |\n\nThis example demonstrates how there is some natural overhead coming from\nthe cloned copy because the server is hosting many forks and has not\noptimized for exactly this set of reachable objects. But the full repack\nhas similar characteristics for both versions.\n\nLet's consider some repositories that are hitting too many collisions\nwith version 1. First, let's explore the kinds of paths that are\ncommonly causing these collisions:\n\n * \"/CHANGELOG.json\" is 15 characters, and is created by the beachball\n   [1] tool. Only the final character of the parent directory can\n   differentiate different versions of this file, but also only the two\n   most-significant digits. If that character is a letter, then this is\n   always a collision. Similar issues occur with the similar\n   \"/CHANGELOG.md\" path, though there is more opportunity for\n   differences In the parent directory.\n\n * Localization files frequently have common filenames but\n   differentiates via parent directories. In C#, the name\n   \"/strings.resx.lcl\" is used for these localization files and they\n   will all collide in name-hash.\n\n[1] https://github.com/microsoft/beachball\n\nI've come across many other examples where some internal tool uses a\ncommon name across multiple directories and is causing Git to repack\npoorly due to name-hash collisions.\n\nOne open-source example is the fluentui [2] repo, which  uses beachball\nto generate CHANGELOG.json and CHANGELOG.md files, and these files have\nvery poor delta characteristics when comparing against versions across\nparent directories.\n\n| Stage                 | Pack Size | Repack Time |\n|-----------------------|-----------|-------------|\n| After clone           | 694 MB    | N/A         |\n| --name-hash-version=1 | 438 MB    | 728s        |\n| --name-hash-version=2 | 168 MB    | 142s        |\n\n[2] https://github.com/microsoft/fluentui\n\nIn this example, we see significant gains in the compressed packfile\nsize as well as the time taken to compute the packfile.\n\nUsing a collection of repositories that use the beachball tool, I was\nable to make similar comparisions with dramatic results. While the\nfluentui repo is public, the others are private so cannot be shared for\nreproduction. The results are so significant that I find it important to\nshare here:\n\n| Repo     | --name-hash-version=1 | --name-hash-version=2 |\n|----------|-----------------------|-----------------------|\n| fluentui |               440 MB  |               161 MB  |\n| Repo B   |             6,248 MB  |               856 MB  |\n| Repo C   |            37,278 MB  |             6,755 MB  |\n| Repo D   |           131,204 MB  |             7,463 MB  |\n\nFuture changes could include making --name-hash-version implied by a config\nvalue or even implied by default during a full repack.\n\nIt is important to point out that the name hash value is stored in the\n.bitmap file format, so we must force --name-hash-version=1 when bitmaps\nare being read or written. Later, the bitmap format could be updated to\nbe aware of the name hash version so deltas can be quickly computed\nacross the bitmapped/not-bitmapped boundary. To promote the safety of\nthis parameter, the validate_name_hash_version() method will die() if\nthe given name-hash version is incorrect and will disable newer versions\nif not yet compatible with other features, such as --write-bitmap-index.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Documentation/git-pack-objects.txt | 32 +++++++++++++++++-\n builtin/pack-objects.c             | 52 +++++++++++++++++++++++++++---\n t/t5300-pack-object.sh             | 31 ++++++++++++++++++\n 3 files changed, 109 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex e32404c6aae..7f69ae4855f 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -15,7 +15,8 @@ SYNOPSIS\n \t[--revs [--unpacked | --all]] [--keep-pack=<pack-name>]\n \t[--cruft] [--cruft-expiration=<time>]\n \t[--stdout [--filter=<filter-spec>] | <base-name>]\n-\t[--shallow] [--keep-true-parents] [--[no-]sparse] < <object-list>\n+\t[--shallow] [--keep-true-parents] [--[no-]sparse]\n+\t[--name-hash-version=<n>] < <object-list>\n \n \n DESCRIPTION\n@@ -345,6 +346,35 @@ raise an error.\n \tRestrict delta matches based on \"islands\". See DELTA ISLANDS\n \tbelow.\n \n+--name-hash-version=<n>::\n+\tWhile performing delta compression, Git groups objects that may be\n+\tsimilar based on heuristics using the path to that object. While\n+\tgrouping objects by an exact path match is good for paths with\n+\tmany versions, there are benefits for finding delta pairs across\n+\tdifferent full paths. Git collects objects by type and then by a\n+\t\"name hash\" of the path and then by size, hoping to group objects\n+\tthat will compress well together.\n++\n+The default name hash version is `1`, which prioritizes hash locality by\n+considering the final bytes of the path as providing the maximum magnitude\n+to the hash function. This version excels at distinguishing short paths\n+and finding renames across directories. However, the hash function depends\n+primarily on the final 16 bytes of the path. If there are many paths in\n+the repo that have the same final 16 bytes and differ only by parent\n+directory, then this name-hash may lead to too many collisions and cause\n+poor results. At the moment, this version is required when writing\n+reachability bitmap files with `--write-bitmap-index`.\n++\n+The name hash version `2` has similar locality features as version `1`,\n+except it considers each path component separately and overlays the hashes\n+with a shift. This still prioritizes the final bytes of the path, but also\n+\"salts\" the lower bits of the hash using the parent directory names. This\n+method allows for some of the locality benefits of version `1` while\n+breaking most of the collisions from a similarly-named file appearing in\n+many different directories. At the moment, this version is not allowed\n+when writing reachability bitmap files with `--write-bitmap-index` and it\n+will be automatically changed to version `1`.\n+\n \n DELTA ISLANDS\n -------------\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 08007142671..e65d4793edf 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -266,6 +266,35 @@ struct configured_exclusion {\n static struct oidmap configured_exclusions;\n \n static struct oidset excluded_by_config;\n+static int name_hash_version = 1;\n+\n+/**\n+ * Check whether the name_hash_version chosen by user input is apporpriate,\n+ * and also validate whether it is appropriate with other features.\n+ */\n+static void validate_name_hash_version(void)\n+{\n+\tif (name_hash_version < 1 || name_hash_version > 2)\n+\t\tdie(_(\"invalid --name-hash-version option: %d\"), name_hash_version);\n+\tif (write_bitmap_index && name_hash_version != 1) {\n+\t\twarning(_(\"currently, --write-bitmap-index requires --name-hash-version=1\"));\n+\t\tname_hash_version = 1;\n+\t}\n+}\n+\n+static inline uint32_t pack_name_hash_fn(const char *name)\n+{\n+\tswitch (name_hash_version) {\n+\tcase 1:\n+\t\treturn pack_name_hash(name);\n+\n+\tcase 2:\n+\t\treturn pack_name_hash_v2((const unsigned char *)name);\n+\n+\tdefault:\n+\t\tBUG(\"invalid name-hash version: %d\", name_hash_version);\n+\t}\n+}\n \n /*\n  * stats\n@@ -1698,7 +1727,7 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,\n \t\treturn 0;\n \t}\n \n-\tcreate_object_entry(oid, type, pack_name_hash(name),\n+\tcreate_object_entry(oid, type, pack_name_hash_fn(name),\n \t\t\t    exclude, name && no_try_delta(name),\n \t\t\t    found_pack, found_offset);\n \treturn 1;\n@@ -1912,7 +1941,7 @@ static void add_preferred_base_object(const char *name)\n {\n \tstruct pbase_tree *it;\n \tsize_t cmplen;\n-\tunsigned hash = pack_name_hash(name);\n+\tunsigned hash = pack_name_hash_fn(name);\n \n \tif (!num_preferred_base || check_pbase_path(hash))\n \t\treturn;\n@@ -3422,7 +3451,7 @@ static void show_object_pack_hint(struct object *object, const char *name,\n \t * here using a now in order to perhaps improve the delta selection\n \t * process.\n \t */\n-\toe->hash = pack_name_hash(name);\n+\toe->hash = pack_name_hash_fn(name);\n \toe->no_try_delta = name && no_try_delta(name);\n \n \tstdin_packs_hints_nr++;\n@@ -3572,7 +3601,7 @@ static void add_cruft_object_entry(const struct object_id *oid, enum object_type\n \tentry = packlist_find(&to_pack, oid);\n \tif (entry) {\n \t\tif (name) {\n-\t\t\tentry->hash = pack_name_hash(name);\n+\t\t\tentry->hash = pack_name_hash_fn(name);\n \t\t\tentry->no_try_delta = no_try_delta(name);\n \t\t}\n \t} else {\n@@ -3595,7 +3624,7 @@ static void add_cruft_object_entry(const struct object_id *oid, enum object_type\n \t\t\treturn;\n \t\t}\n \n-\t\tentry = create_object_entry(oid, type, pack_name_hash(name),\n+\t\tentry = create_object_entry(oid, type, pack_name_hash_fn(name),\n \t\t\t\t\t    0, name && no_try_delta(name),\n \t\t\t\t\t    pack, offset);\n \t}\n@@ -4069,6 +4098,15 @@ static int get_object_list_from_bitmap(struct rev_info *revs)\n \tif (!(bitmap_git = prepare_bitmap_walk(revs, 0)))\n \t\treturn -1;\n \n+\t/*\n+\t * For now, force the name-hash version to be 1 since that\n+\t * is the version implied by the bitmap format. Later, the\n+\t * format can include this version explicitly in its format,\n+\t * allowing readers to know the version that was used during\n+\t * the bitmap write.\n+\t */\n+\tname_hash_version = 1;\n+\n \tif (pack_options_allow_reuse())\n \t\treuse_partial_packfile_from_bitmap(bitmap_git,\n \t\t\t\t\t\t   &reuse_packfiles,\n@@ -4429,6 +4467,8 @@ int cmd_pack_objects(int argc,\n \t\tOPT_STRING_LIST(0, \"uri-protocol\", &uri_protocols,\n \t\t\t\tN_(\"protocol\"),\n \t\t\t\tN_(\"exclude any configured uploadpack.blobpackfileuri with this protocol\")),\n+\t\tOPT_INTEGER(0, \"name-hash-version\", &name_hash_version,\n+\t\t\t N_(\"use the specified name-hash function to group similar objects\")),\n \t\tOPT_END(),\n \t};\n \n@@ -4576,6 +4616,8 @@ int cmd_pack_objects(int argc,\n \tif (pack_to_stdout || !rev_list_all)\n \t\twrite_bitmap_index = 0;\n \n+\tvalidate_name_hash_version();\n+\n \tif (use_delta_islands)\n \t\tstrvec_push(&rp, \"--topo-order\");\n \ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 3b9dae331a5..4270eabe8b7 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -674,4 +674,35 @@ do\n \t'\n done\n \n+test_expect_success 'valid and invalid --name-hash-versions' '\n+\t# Valid values are hard to verify other than \"do not fail\".\n+\t# Performance tests will be more valuable to validate these versions.\n+\tfor value in 1 2\n+\tdo\n+\t\tgit pack-objects base --all --name-hash-version=$value || return 1\n+\tdone &&\n+\n+\t# Invalid values have clear post-conditions.\n+\tfor value in -1 0 3\n+\tdo\n+\t\ttest_must_fail git pack-objects base --all --name-hash-version=$value 2>err &&\n+\t\ttest_grep \"invalid --name-hash-version option\" err || return 1\n+\tdone\n+'\n+\n+# The following test is not necessarily a permanent choice, but since we do not\n+# have a \"name hash version\" bit in the .bitmap file format, we cannot write the\n+# hash values into the .bitmap file without risking breakage later.\n+#\n+# TODO: Make these compatible in the future and replace this test with the\n+# expected behavior when both are specified.\n+test_expect_success '--name-hash-version=2 and --write-bitmap-index are incompatible' '\n+\tgit pack-objects base --all --name-hash-version=2 --write-bitmap-index 2>err &&\n+\ttest_grep \"currently, --write-bitmap-index requires --name-hash-version=1\" err &&\n+\n+\t# --stdout option silently removes --write-bitmap-index\n+\tgit pack-objects --stdout --all --name-hash-version=2 --write-bitmap-index >out 2>err &&\n+\t! test_grep \"currently, --write-bitmap-index requires --name-hash-version=1\" err\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"511292","messageId":"a507be2f19bb3429077fc4b0f87c91d5d797d136.1738004555.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.v4.git.1738004554.gitgitgadget@gmail.com","subject":"[PATCH v4 4/7] pack-objects: add GIT_TEST_NAME_HASH_VERSION","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-01-27T19:02:31Z","receivedAt":"2025-01-27T19:02:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nAdd a new environment variable to opt-in to different values of the\n--name-hash-version=<n> option in 'git pack-objects'. This allows for\nextra testing of the feature without repeating all of the test\nscenarios. Unlike many GIT_TEST_* variables, we are choosing to not add\nthis to the linux-TEST-vars CI build as that test run is already\noverloaded. The behavior exposed by this test variable is of low risk\nand should be sufficient to allow manual testing when an issue arises.\n\nBut this option isn't free. There are a few tests that change behavior\nwith the variable enabled.\n\nFirst, there are a few tests that are very sensitive to certain delta\nbases being picked. These are both involving the generation of thin\nbundles and then counting their objects via 'git index-pack --fix-thin'\nwhich pulls the delta base into the new packfile. For these tests,\ndisable the option as a decent long-term option.\n\nSecond, there are some tests that compare the exact output of a 'git\npack-objects' process when using bitmaps. The warning that ignores the\n--name-hash-version=2 and forces version 1 causes these tests to fail.\nDisable the environment variable to get around this issue.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n builtin/pack-objects.c          |  5 ++++-\n t/README                        |  4 ++++\n t/t5300-pack-object.sh          |  7 +++++--\n t/t5310-pack-bitmaps.sh         |  5 ++++-\n t/t5333-pseudo-merge-bitmaps.sh |  3 ++-\n t/t5510-fetch.sh                |  7 ++++++-\n t/t6020-bundle-misc.sh          |  6 +++++-\n t/t7406-submodule-update.sh     |  4 +++-\n t/t7700-repack.sh               | 10 ++++++++--\n 9 files changed, 41 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex e65d4793edf..57277429900 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -266,7 +266,7 @@ struct configured_exclusion {\n static struct oidmap configured_exclusions;\n \n static struct oidset excluded_by_config;\n-static int name_hash_version = 1;\n+static int name_hash_version = -1;\n \n /**\n  * Check whether the name_hash_version chosen by user input is apporpriate,\n@@ -4616,6 +4616,9 @@ int cmd_pack_objects(int argc,\n \tif (pack_to_stdout || !rev_list_all)\n \t\twrite_bitmap_index = 0;\n \n+\tif (name_hash_version < 0)\n+\t\tname_hash_version = (int)git_env_ulong(\"GIT_TEST_NAME_HASH_VERSION\", 1);\n+\n \tvalidate_name_hash_version();\n \n \tif (use_delta_islands)\ndiff --git a/t/README b/t/README\nindex 8c0319b58e5..e63d2360852 100644\n--- a/t/README\n+++ b/t/README\n@@ -492,6 +492,10 @@ a test and then fails then the whole test run will abort. This can help to make\n sure the expected tests are executed and not silently skipped when their\n dependency breaks or is simply not present in a new environment.\n \n+GIT_TEST_NAME_HASH_VERSION=<int>, when set, causes 'git pack-objects' to\n+assume '--name-hash-version=<n>'.\n+\n+\n Naming Tests\n ------------\n \ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 4270eabe8b7..97fe9e561c6 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -675,15 +675,18 @@ do\n done\n \n test_expect_success 'valid and invalid --name-hash-versions' '\n+\tsane_unset GIT_TEST_NAME_HASH_VERSION &&\n+\n \t# Valid values are hard to verify other than \"do not fail\".\n \t# Performance tests will be more valuable to validate these versions.\n-\tfor value in 1 2\n+\t# Negative values are converted to version 1.\n+\tfor value in -1 1 2\n \tdo\n \t\tgit pack-objects base --all --name-hash-version=$value || return 1\n \tdone &&\n \n \t# Invalid values have clear post-conditions.\n-\tfor value in -1 0 3\n+\tfor value in 0 3\n \tdo\n \t\ttest_must_fail git pack-objects base --all --name-hash-version=$value 2>err &&\n \t\ttest_grep \"invalid --name-hash-version option\" err || return 1\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex 7044c7d7c6d..c30522b57fd 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -420,7 +420,10 @@ test_bitmap_cases () {\n \t\t\tcat >expect <<-\\EOF &&\n \t\t\terror: missing value for '\\''pack.preferbitmaptips'\\''\n \t\t\tEOF\n-\t\t\tgit repack -adb 2>actual &&\n+\n+\t\t\t# Disable name hash version adjustment due to stderr comparison.\n+\t\t\tGIT_TEST_NAME_HASH_VERSION=1 \\\n+\t\t\t\tgit repack -adb 2>actual &&\n \t\t\ttest_cmp expect actual\n \t\t)\n \t'\ndiff --git a/t/t5333-pseudo-merge-bitmaps.sh b/t/t5333-pseudo-merge-bitmaps.sh\nindex eca4a1eb8c6..971f9d2d4ee 100755\n--- a/t/t5333-pseudo-merge-bitmaps.sh\n+++ b/t/t5333-pseudo-merge-bitmaps.sh\n@@ -209,7 +209,8 @@ test_expect_success 'bitmapPseudoMerge.stableThreshold creates stable groups' '\n '\n \n test_expect_success 'out of order thresholds are rejected' '\n-\ttest_must_fail git \\\n+\t# Disable the test var to remove a stderr message.\n+\ttest_must_fail env GIT_TEST_NAME_HASH_VERSION=1 git \\\n \t\t-c bitmapPseudoMerge.test.pattern=\"refs/*\" \\\n \t\t-c bitmapPseudoMerge.test.threshold=1.month.ago \\\n \t\t-c bitmapPseudoMerge.test.stableThreshold=1.week.ago \\\ndiff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\nindex 0890b9f61c5..1699c3a3bb8 100755\n--- a/t/t5510-fetch.sh\n+++ b/t/t5510-fetch.sh\n@@ -1062,7 +1062,12 @@ test_expect_success 'all boundary commits are excluded' '\n \ttest_tick &&\n \tgit merge otherside &&\n \tad=$(git log --no-walk --format=%ad HEAD) &&\n-\tgit bundle create twoside-boundary.bdl main --since=\"$ad\" &&\n+\n+\t# If the a different name hash function is used here, then no delta\n+\t# pair is found and the bundle does not expand to three objects\n+\t# when fixing the thin object.\n+\tGIT_TEST_NAME_HASH_VERSION=1 \\\n+\t\tgit bundle create twoside-boundary.bdl main --since=\"$ad\" &&\n \ttest_bundle_object_count --thin twoside-boundary.bdl 3\n '\n \ndiff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\nindex 34b5cd62c20..a1f18ae71f1 100755\n--- a/t/t6020-bundle-misc.sh\n+++ b/t/t6020-bundle-misc.sh\n@@ -247,7 +247,11 @@ test_expect_success 'create bundle with --since option' '\n \tEOF\n \ttest_cmp expect actual &&\n \n-\tgit bundle create since.bdl \\\n+\t# If a different name hash function is used, then one fewer\n+\t# delta base is found and this counts a different number\n+\t# of objects after performing --fix-thin.\n+\tGIT_TEST_NAME_HASH_VERSION=1 \\\n+\t\tgit bundle create since.bdl \\\n \t\t--since \"Thu Apr 7 15:27:00 2005 -0700\" \\\n \t\t--all &&\n \ndiff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh\nindex 0f0c86f9cb2..ebd9941075a 100755\n--- a/t/t7406-submodule-update.sh\n+++ b/t/t7406-submodule-update.sh\n@@ -1094,7 +1094,9 @@ test_expect_success 'submodule update --quiet passes quietness to fetch with a s\n \t) &&\n \tgit clone super4 super5 &&\n \t(cd super5 &&\n-\t git submodule update --quiet --init --depth=1 submodule3 >out 2>err &&\n+\t # This test var can mess with the stderr output checked in this test.\n+\t GIT_TEST_NAME_HASH_VERSION=1 \\\n+\t\tgit submodule update --quiet --init --depth=1 submodule3 >out 2>err &&\n \t test_must_be_empty out &&\n \t test_must_be_empty err\n \t) &&\ndiff --git a/t/t7700-repack.sh b/t/t7700-repack.sh\nindex b9a5759e01d..16861f80c9c 100755\n--- a/t/t7700-repack.sh\n+++ b/t/t7700-repack.sh\n@@ -309,7 +309,10 @@ test_expect_success 'no bitmaps created if .keep files present' '\n \tkeep=${pack%.pack}.keep &&\n \ttest_when_finished \"rm -f \\\"\\$keep\\\"\" &&\n \t>\"$keep\" &&\n-\tgit -C bare.git repack -ad 2>stderr &&\n+\n+\t# Disable --name-hash-version test due to stderr comparison.\n+\tGIT_TEST_NAME_HASH_VERSION=1 \\\n+\t\tgit -C bare.git repack -ad 2>stderr &&\n \ttest_must_be_empty stderr &&\n \tfind bare.git/objects/pack/ -type f -name \"*.bitmap\" >actual &&\n \ttest_must_be_empty actual\n@@ -320,7 +323,10 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '\n \tblob=$(test-tool genrandom big $((1024*1024)) |\n \t       git -C bare.git hash-object -w --stdin) &&\n \tgit -C bare.git update-ref refs/tags/big $blob &&\n-\tgit -C bare.git repack -ad 2>stderr &&\n+\n+\t# Disable --name-hash-version test due to stderr comparison.\n+\tGIT_TEST_NAME_HASH_VERSION=1 \\\n+\t\tgit -C bare.git repack -ad 2>stderr &&\n \ttest_must_be_empty stderr &&\n \tfind bare.git/objects/pack -type f -name \"*.bitmap\" >actual &&\n \ttest_must_be_empty actual\n-- \ngitgitgadget\n\n"},{"id":"511293","messageId":"a0247a679ac4b81a437b1c1a7151bddd99d6f966.1738004555.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.v4.git.1738004554.gitgitgadget@gmail.com","subject":"[PATCH v4 5/7] p5313: add size comparison test","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-01-27T19:02:32Z","receivedAt":"2025-01-27T19:02:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nAs custom options are added to 'git pack-objects' and 'git repack' to\nadjust how compression is done, use this new performance test script to\ndemonstrate their effectiveness in performance and size.\n\nThe recently-added --name-hash-version option allows for testing\ndifferent name hash functions. Version 2 intends to preserve some of the\nlocality of version 1 while more often breaking collisions due to long\nfilenames.\n\nDistinguishing objects by more of the path is critical when there are\nmany name hash collisions and several versions of the same path in the\nfull history, giving a significant boost to the full repack case. The\nlocality of the hash function is critical to compressing something like\na shallow clone or a thin pack representing a push of a single commit.\n\nThis can be seen by running pt5313 on the open source fluentui\nrepository [1]. Most commits will have this kind of output for the thin\nand big pack cases, though certain commits (such as [2]) will have\nproblematic thin pack size for other reasons.\n\n[1] https://github.com/microsoft/fluentui\n[2] a637a06df05360ce5ff21420803f64608226a875\n\nChecked out at the parent of [2], I see the following statistics:\n\nTest                                         HEAD\n---------------------------------------------------------------\n5313.2: thin pack with version 1             0.37(0.44+0.02)\n5313.3: thin pack size with version 1                   1.2M\n5313.4: big pack with version 1              2.04(7.77+0.23)\n5313.5: big pack size with version 1                   20.4M\n5313.6: shallow fetch pack with version 1    1.41(2.94+0.11)\n5313.7: shallow pack size with version 1               34.4M\n5313.8: repack with version 1                95.70(676.41+2.87)\n5313.9: repack size with version 1                    439.3M\n5313.10: thin pack with version 2            0.12(0.12+0.06)\n5313.11: thin pack size with version 2                 22.0K\n5313.12: big pack with version 2             2.80(5.43+0.34)\n5313.13: big pack size with version 2                  25.9M\n5313.14: shallow fetch pack with version 2   1.77(2.80+0.19)\n5313.15: shallow pack size with version 2              33.7M\n5313.16: repack with version 2               33.68(139.52+2.58)\n5313.17: repack size with version 2                   160.5M\n\nTo make comparisons easier, I will reformat this output into a different\ntable style:\n\n| Test         | V1 Time | V2 Time | V1 Size | V2 Size |\n|--------------|---------|---------|---------|---------|\n| Thin Pack    |  0.37 s |  0.12 s |   1.2 M |  22.0 K |\n| Big Pack     |  2.04 s |  2.80 s |  20.4 M |  25.9 M |\n| Shallow Pack |  1.41 s |  1.77 s |  34.4 M |  33.7 M |\n| Repack       | 95.70 s | 33.68 s | 439.3 M | 160.5 M |\n\nThe v2 hash function successfully differentiates the CHANGELOG.md files\nfrom each other, which leads to significant improvements in the thin\npack (simulating a push of this commit) and the full repack. There is\nsome bloat in the \"big pack\" scenario and essentially the same results\nfor the shallow pack.\n\nIn the case of the Git repository, these numbers show some of the issues\nwith this approach:\n\n| Test         | V1 Time | V2 Time | V1 Size | V2 Size |\n|--------------|---------|---------|---------|---------|\n| Thin Pack    |  0.02 s |  0.02 s |   1.1 K |   1.1 K |\n| Big Pack     |  1.69 s |  1.95 s |  13.5 M |  14.5 M |\n| Shallow Pack |  1.26 s |  1.29 s |  12.0 M |  12.2 M |\n| Repack       | 29.51 s | 29.01 s | 237.7 M | 238.2 M |\n\nHere, the attempts to remove conflicts in the v2 function seem to cause\nslight bloat to these sizes. This shows that the Git repository benefits\na lot from cross-path delta pairs.\n\nThe results are similar with the nodejs/node repo:\n\n| Test         | V1 Time | V2 Time | V1 Size | V2 Size |\n|--------------|---------|---------|---------|---------|\n| Thin Pack    |  0.02 s |  0.02 s |   1.6 K |   1.6 K |\n| Big Pack     |  4.61 s |  3.26 s |  56.0 M |  52.8 M |\n| Shallow Pack |  7.82 s |  7.51 s | 104.6 M | 107.0 M |\n| Repack       | 88.90 s | 73.75 s | 740.1 M | 764.5 M |\n\nHere, the v2 name-hash causes some size bloat more often than it reduces\nthe size, but it also universally improves performance time, which is an\ninteresting reversal. This must mean that it is helping to short-circuit\nsome delta computations even if it is not finding the most efficient\nones. The performance improvement cannot be explained only due to the\nI/O cost of writing the resulting packfile.\n\nThe Linux kernel repository was the initial target of the default name\nhash value, and its naming conventions are practically build to take the\nmost advantage of the default name hash values:\n\n| Test         | V1 Time  | V2 Time  | V1 Size | V2 Size |\n|--------------|----------|----------|---------|---------|\n| Thin Pack    |   0.17 s |   0.07 s |   4.6 K |   4.6 K |\n| Big Pack     |  17.88 s |  12.35 s | 201.1 M | 159.1 M |\n| Shallow Pack |  11.05 s |  22.94 s | 269.2 M | 273.8 M |\n| Repack       | 727.39 s | 566.95 s |   2.5 G |   2.5 G |\n\nHere, the thin and big packs gain some performance boosts in time, with\na modest gain in the size of the big pack. The shallow pack, however, is\nmore expensive to compute, likely because similarly-named files across\ndifferent directories are farther apart in the name hash ordering in v2.\nThe repack also gains benefits in computation time but no meaningful\nchange to the full size.\n\nFinally, an internal Javascript repo of moderate size shows significant\ngains when repacking with --name-hash-version=2 due to it having many name\nhash collisions. However, it's worth noting that only the full repack\ncase has significant differences from the v1 name hash:\n\n| Test      | V1 Time   | V2 Time  | V1 Size | V2 Size |\n|-----------|-----------|----------|---------|---------|\n| Thin Pack |    8.28 s |   7.28 s |  16.8 K |  16.8 K |\n| Big Pack  |   12.81 s |  11.66 s |  29.1 M |  29.1 M |\n| Shallow   |    4.86 s |   4.06 s |  42.5 M |  44.1 M |\n| Repack    | 3126.50 s | 496.33 s |   6.2 G | 855.6 M |\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n t/perf/p5313-pack-objects.sh | 70 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 70 insertions(+)\n create mode 100755 t/perf/p5313-pack-objects.sh\n\ndiff --git a/t/perf/p5313-pack-objects.sh b/t/perf/p5313-pack-objects.sh\nnew file mode 100755\nindex 00000000000..be5229a0ecd\n--- /dev/null\n+++ b/t/perf/p5313-pack-objects.sh\n@@ -0,0 +1,70 @@\n+#!/bin/sh\n+\n+test_description='Tests pack performance using bitmaps'\n+. ./perf-lib.sh\n+\n+GIT_TEST_PASSING_SANITIZE_LEAK=0\n+export GIT_TEST_PASSING_SANITIZE_LEAK\n+\n+test_perf_large_repo\n+\n+test_expect_success 'create rev input' '\n+\tcat >in-thin <<-EOF &&\n+\t$(git rev-parse HEAD)\n+\t^$(git rev-parse HEAD~1)\n+\tEOF\n+\n+\tcat >in-big <<-EOF &&\n+\t$(git rev-parse HEAD)\n+\t^$(git rev-parse HEAD~1000)\n+\tEOF\n+\n+\tcat >in-shallow <<-EOF\n+\t$(git rev-parse HEAD)\n+\t--shallow $(git rev-parse HEAD)\n+\tEOF\n+'\n+\n+for version in 1 2\n+do\n+\texport version\n+\n+\ttest_perf \"thin pack with version $version\" '\n+\t\tgit pack-objects --thin --stdout --revs --sparse \\\n+\t\t\t--name-hash-version=$version <in-thin >out\n+\t'\n+\n+\ttest_size \"thin pack size with version $version\" '\n+\t\ttest_file_size out\n+\t'\n+\n+\ttest_perf \"big pack with version $version\" '\n+\t\tgit pack-objects --stdout --revs --sparse \\\n+\t\t\t--name-hash-version=$version <in-big >out\n+\t'\n+\n+\ttest_size \"big pack size with version $version\" '\n+\t\ttest_file_size out\n+\t'\n+\n+\ttest_perf \"shallow fetch pack with version $version\" '\n+\t\tgit pack-objects --stdout --revs --sparse --shallow \\\n+\t\t\t--name-hash-version=$version <in-shallow >out\n+\t'\n+\n+\ttest_size \"shallow pack size with version $version\" '\n+\t\ttest_file_size out\n+\t'\n+\n+\ttest_perf \"repack with version $version\" '\n+\t\tgit repack -adf --name-hash-version=$version\n+\t'\n+\n+\ttest_size \"repack size with version $version\" '\n+\t\tgitdir=$(git rev-parse --git-dir) &&\n+\t\tpack=$(ls $gitdir/objects/pack/pack-*.pack) &&\n+\t\ttest_file_size \"$pack\"\n+\t'\n+done\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"511295","messageId":"06a95063186598376c1b449e617693c783190feb.1738004555.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.v4.git.1738004554.gitgitgadget@gmail.com","subject":"[PATCH v4 6/7] test-tool: add helper for name-hash values","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-01-27T19:02:33Z","receivedAt":"2025-01-27T19:02:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nAdd a new test-tool helper, name-hash, to output the value of the\nname-hash algorithms for the input list of strings, one per line.\n\nSince the name-hash values can be stored in the .bitmap files, it is\nimportant that these hash functions do not change across Git versions.\nAdd a simple test to t5310-pack-bitmaps.sh to provide some testing of\nthe current values. Due to how these functions are implemented, it would\nbe difficult to change them without disturbing these values. The paths\nused for this test are carefully selected to demonstrate some of the\nbehavior differences of the two current name hash versions, including\nwhich conditions will cause them to collide.\n\nCreate a performance test that uses test_size to demonstrate how\ncollisions occur for these hash algorithms. This test helps inform\nsomeone as to the behavior of the name-hash algorithms for their repo\nbased on the paths at HEAD.\n\nMy copy of the Git repository shows modest statistics around the\ncollisions of the default name-hash algorithm:\n\nTest                               this tree\n--------------------------------------------------\n5314.1: paths at head                         4.5K\n5314.2: distinct hash value: v1               4.1K\n5314.3: maximum multiplicity: v1                13\n5314.4: distinct hash value: v2               4.2K\n5314.5: maximum multiplicity: v2                 9\n\nHere, the maximum collision multiplicity is 13, but around 10% of paths\nhave a collision with another path.\n\nIn a more interesting example, the microsoft/fluentui [1] repo had these\nstatistics at time of committing:\n\nTest                               this tree\n--------------------------------------------------\n5314.1: paths at head                        19.5K\n5314.2: distinct hash value: v1               8.2K\n5314.3: maximum multiplicity: v1               279\n5314.4: distinct hash value: v2              17.8K\n5314.5: maximum multiplicity: v2                44\n\n[1] https://github.com/microsoft/fluentui\n\nThat demonstrates that of the nearly twenty thousand path names, they\nare assigned around eight thousand distinct values. 279 paths are\nassigned to a single value, leading the packing algorithm to sort\nobjects from those paths together, by size.\n\nWith the v2 name hash function, the maximum multiplicity lowers to 44,\nleaving some room for further improvement.\n\nIn a more extreme example, an internal monorepo had a much worse\ncollision rate:\n\nTest                               this tree\n--------------------------------------------------\n5314.1: paths at head                       227.3K\n5314.2: distinct hash value: v1              72.3K\n5314.3: maximum multiplicity: v1             14.4K\n5314.4: distinct hash value: v2             166.5K\n5314.5: maximum multiplicity: v2               138\n\nHere, we can see that the v2 name hash function provides somem\nimprovements, but there are still a number of collisions that could lead\nto repacking problems at this scale.\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n Makefile                  |  1 +\n t/helper/test-name-hash.c | 23 +++++++++++++++++++++++\n t/helper/test-tool.c      |  1 +\n t/helper/test-tool.h      |  1 +\n t/perf/p5314-name-hash.sh | 31 +++++++++++++++++++++++++++++++\n t/t5310-pack-bitmaps.sh   | 30 ++++++++++++++++++++++++++++++\n 6 files changed, 87 insertions(+)\n create mode 100644 t/helper/test-name-hash.c\n create mode 100755 t/perf/p5314-name-hash.sh\n\ndiff --git a/Makefile b/Makefile\nindex 6f5986b66ea..65403f6dd09 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -816,6 +816,7 @@ TEST_BUILTINS_OBJS += test-lazy-init-name-hash.o\n TEST_BUILTINS_OBJS += test-match-trees.o\n TEST_BUILTINS_OBJS += test-mergesort.o\n TEST_BUILTINS_OBJS += test-mktemp.o\n+TEST_BUILTINS_OBJS += test-name-hash.o\n TEST_BUILTINS_OBJS += test-online-cpus.o\n TEST_BUILTINS_OBJS += test-pack-mtimes.o\n TEST_BUILTINS_OBJS += test-parse-options.o\ndiff --git a/t/helper/test-name-hash.c b/t/helper/test-name-hash.c\nnew file mode 100644\nindex 00000000000..af1d52de101\n--- /dev/null\n+++ b/t/helper/test-name-hash.c\n@@ -0,0 +1,23 @@\n+/*\n+ * test-name-hash.c: Read a list of paths over stdin and report on their\n+ * name-hash and full name-hash.\n+ */\n+\n+#include \"test-tool.h\"\n+#include \"git-compat-util.h\"\n+#include \"pack-objects.h\"\n+#include \"strbuf.h\"\n+\n+int cmd__name_hash(int argc UNUSED, const char **argv UNUSED)\n+{\n+\tstruct strbuf line = STRBUF_INIT;\n+\n+\twhile (!strbuf_getline(&line, stdin)) {\n+\t\tprintf(\"%10u \", pack_name_hash(line.buf));\n+\t\tprintf(\"%10u \", pack_name_hash_v2((unsigned const char *)line.buf));\n+\t\tprintf(\"%s\\n\", line.buf);\n+\t}\n+\n+\tstrbuf_release(&line);\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 1ebb69a5dc4..e794058ab6d 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -44,6 +44,7 @@ static struct test_cmd cmds[] = {\n \t{ \"match-trees\", cmd__match_trees },\n \t{ \"mergesort\", cmd__mergesort },\n \t{ \"mktemp\", cmd__mktemp },\n+\t{ \"name-hash\", cmd__name_hash },\n \t{ \"online-cpus\", cmd__online_cpus },\n \t{ \"pack-mtimes\", cmd__pack_mtimes },\n \t{ \"parse-options\", cmd__parse_options },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 21802ac27da..26ff30a5a9a 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -37,6 +37,7 @@ int cmd__lazy_init_name_hash(int argc, const char **argv);\n int cmd__match_trees(int argc, const char **argv);\n int cmd__mergesort(int argc, const char **argv);\n int cmd__mktemp(int argc, const char **argv);\n+int cmd__name_hash(int argc, const char **argv);\n int cmd__online_cpus(int argc, const char **argv);\n int cmd__pack_mtimes(int argc, const char **argv);\n int cmd__parse_options(int argc, const char **argv);\ndiff --git a/t/perf/p5314-name-hash.sh b/t/perf/p5314-name-hash.sh\nnew file mode 100755\nindex 00000000000..4ef0ba77114\n--- /dev/null\n+++ b/t/perf/p5314-name-hash.sh\n@@ -0,0 +1,31 @@\n+#!/bin/sh\n+\n+test_description='Tests pack performance using bitmaps'\n+. ./perf-lib.sh\n+\n+GIT_TEST_PASSING_SANITIZE_LEAK=0\n+export GIT_TEST_PASSING_SANITIZE_LEAK\n+\n+test_perf_large_repo\n+\n+test_size 'paths at head' '\n+\tgit ls-tree -r --name-only HEAD >path-list &&\n+\twc -l <path-list &&\n+\ttest-tool name-hash <path-list >name-hashes\n+'\n+\n+for version in 1 2\n+do\n+\ttest_size \"distinct hash value: v$version\" '\n+\t\tawk \"{ print \\$$version; }\" <name-hashes | sort | \\\n+\t\t\tuniq -c >name-hash-count &&\n+\t\twc -l <name-hash-count\n+\t'\n+\n+\ttest_size \"maximum multiplicity: v$version\" '\n+\t\tsort -nr <name-hash-count | head -n 1 |\t\\\n+\t\t\tawk \"{ print \\$1; }\"\n+\t'\n+done\n+\n+test_done\ndiff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh\nindex c30522b57fd..871ce01401a 100755\n--- a/t/t5310-pack-bitmaps.sh\n+++ b/t/t5310-pack-bitmaps.sh\n@@ -27,6 +27,36 @@ has_any () {\n \tgrep -Ff \"$1\" \"$2\"\n }\n \n+# Since name-hash values are stored in the .bitmap files, add a test\n+# that checks that the name-hash calculations are stable across versions.\n+# Not exhaustive, but these hashing algorithms would be hard to change\n+# without causing deviations here.\n+test_expect_success 'name-hash value stability' '\n+\tcat >names <<-\\EOF &&\n+\tfirst\n+\tsecond\n+\tthird\n+\ta/one-long-enough-for-collisions\n+\tb/two-long-enough-for-collisions\n+\tmany/parts/to/this/path/enough/to/collide/in/v2\n+\tenough/parts/to/this/path/enough/to/collide/in/v2\n+\tEOF\n+\n+\ttest-tool name-hash <names >out &&\n+\n+\tcat >expect <<-\\EOF &&\n+\t2582249472 1763573760 first\n+\t2289942528 1188134912 second\n+\t2300837888 1130758144 third\n+\t2544516325 3963087891 a/one-long-enough-for-collisions\n+\t2544516325 4013419539 b/two-long-enough-for-collisions\n+\t1420111091 1709547268 many/parts/to/this/path/enough/to/collide/in/v2\n+\t1420111091 1709547268 enough/parts/to/this/path/enough/to/collide/in/v2\n+\tEOF\n+\n+\ttest_cmp expect out\n+'\n+\n test_bitmap_cases () {\n \twriteLookupTable=false\n \tfor i in \"$@\"\n-- \ngitgitgadget\n\n"},{"id":"511294","messageId":"bab2ac318802ed9993a5fffccd73966b6309cef6.1738004555.git.gitgitgadget@gmail.com","threadId":"62447","inReplyTo":"pull.1823.v4.git.1738004554.gitgitgadget@gmail.com","subject":"[PATCH v4 7/7] pack-objects: prevent name hash version change","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2025-01-27T19:02:34Z","receivedAt":"2025-01-27T19:02:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <stolee@gmail.com>\n\nWhen the --name-hash-version option is used in 'git pack-objects', it\ncan change from the initial assignment to when it is used based on\ninteractions with other arguments. Specifically, when writing or reading\nbitmaps, we must force version 1 for now. This could change in the\nfuture when the bitmap format can store a name hash version value,\nindicating which was used during the writing of the packfile.\n\nProtect the 'git pack-objects' process from getting confused by failing\nwith a BUG() statement if the value of the name hash version changes\nbetween calls to pack_name_hash_fn().\n\nSigned-off-by: Derrick Stolee <stolee@gmail.com>\n---\n builtin/pack-objects.c | 8 ++++++++\n 1 file changed, 8 insertions(+)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 57277429900..7c488d2d3b0 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -284,6 +284,14 @@ static void validate_name_hash_version(void)\n \n static inline uint32_t pack_name_hash_fn(const char *name)\n {\n+\tstatic int seen_version = -1;\n+\n+\tif (seen_version < 0)\n+\t\tseen_version = name_hash_version;\n+\telse if (seen_version != name_hash_version)\n+\t\tBUG(\"name hash version changed from %d to %d mid-process\",\n+\t\t    seen_version, name_hash_version);\n+\n \tswitch (name_hash_version) {\n \tcase 1:\n \t\treturn pack_name_hash(name);\n-- \ngitgitgadget\n"},{"id":"511303","messageId":"xmqqy0ywoszt.fsf@gitster.g","threadId":"62447","inReplyTo":"7ee1845144fda5b8192dfa13eaab3cbd669b39ed.1738004555.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 2/7] pack-objects: add --name-hash-version option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-01-27T21:18:46Z","receivedAt":"2025-01-27T21:18:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> +/**\n> + * Check whether the name_hash_version chosen by user input is apporpriate,\n\nappropriate.  Will tweak while queueing, so no need to resend only\nto change this.\n\n> + * and also validate whether it is appropriate with other features.\n> + */\n\nThanks.\n"},{"id":"511406","messageId":"001e3382-e851-41a4-acc2-48e44501efca@gmail.com","threadId":"62447","inReplyTo":"xmqqy0ywoszt.fsf@gitster.g","subject":"Re: [PATCH v4 2/7] pack-objects: add --name-hash-version option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2025-01-29T13:38:12Z","receivedAt":"2025-01-29T13:38:14Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/27/25 4:18 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n> \n>> +/**\n>> + * Check whether the name_hash_version chosen by user input is apporpriate,\n> \n> appropriate.  Will tweak while queueing, so no need to resend only\n> to change this.\nThank you for catching and fixing this!\n\n-Stolee\n"},{"id":"511596","messageId":"Z51DHdS8SStRGalX@nand.local","threadId":"62447","inReplyTo":"pull.1823.v4.git.1738004554.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 0/7] pack-objects: Create an alternative name hash algorithm (recreated)","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-31T21:39:41Z","receivedAt":"2025-01-31T21:39:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jan 27, 2025 at 07:02:27PM +0000, Derrick Stolee via GitGitGadget wrote:\n> UPDATES SINCE v3\n> ================\n>\n>  * Style fixes for switch statement and setting a test environment variable.\n>\n>  * validate_name_hash_version() is now responsible for checking\n>    compatibility with other options.\n>\n>  * The --name-hash-version=3 patch is removed to avoid user confusion since\n>    we don't have a clear way to predict when it would provide (modest)\n>    improvements over v2.\n\nThanks, this round looks great to me. I'm excited to see this topic\nmoving forward!\n\nThanks,\nTaylor\n"}]}