git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 5/6] p5313: add size comparison test

From
Junio C Hamano <gitster@pobox.com>
Date
Sep 19, 2024, 21:58 UTC
Message-ID
<xmqqcykz5mrg.fsf@gitster.g>
In-Reply-To
<999b1d094241b0ba8d6924ac6976eafc64c7d4a6.1726692382.git.gitgitgadget@gmail.com>
"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
Show 44 quoted lines
> From: Derrick Stolee <stolee@gmail.com>
>
> As custom options are added to 'git pack-objects' and 'git repack' to
> adjust how compression is done, use this new performance test script to
> demonstrate their effectiveness in performance and size.
>
> The recently-added --full-name-hash option swaps the default name-hash
> algorithm with one that attempts to uniformly distribute the hashes
> based on the full path name instead of the last 16 characters.
>
> This has a dramatic effect on full repacks for repositories with many
> versions of most paths. It can have a negative impact on cases such as
> pushing a single change.
>
> This can be seen by running pt5313 on the open source fluentui
> repository [1]. Most commits will have this kind of output for the thin
> and big pack cases, though certain commits (such as [2]) will have
> problematic thin pack size for other reasons.
>
> [1] https://github.com/microsoft/fluentui
> [2] a637a06df05360ce5ff21420803f64608226a875
>
> Checked out at the parent of [2], I see the following statistics:
>
> Test                                           this tree
> ------------------------------------------------------------------
> 5313.2: thin pack                              0.02(0.01+0.01)
> 5313.3: thin pack size                                    1.1K
> 5313.4: thin pack with --full-name-hash        0.02(0.01+0.00)
> 5313.5: thin pack size with --full-name-hash              3.0K
> 5313.6: big pack                               1.65(3.35+0.24)
> 5313.7: big pack size                                    58.0M
> 5313.8: big pack with --full-name-hash         1.53(2.52+0.18)
> 5313.9: big pack size with --full-name-hash              57.6M
> 5313.10: repack                                176.52(706.60+3.53)
> 5313.11: repack size                                    446.7K
> 5313.12: repack with --full-name-hash          37.47(134.18+3.06)
> 5313.13: repack size with --full-name-hash              183.1K
>
> Note that this demonstrates a 3x size _increase_ in the case that
> simulates a small "git push". The size change is neutral on the case of
> pushing the difference between HEAD and HEAD~1000.
>
> However, the full repack case is both faster and more efficient.
Nice.
Show 5 quoted lines
> Signed-off-by: Derrick Stolee <stolee@gmail.com>
> ---
>  t/perf/p5313-pack-objects.sh | 71 ++++++++++++++++++++++++++++++++++++
>  1 file changed, 71 insertions(+)
>  create mode 100755 t/perf/p5313-pack-objects.sh
"wc -c" -> "test_file_size" or "test-tool path-utils file-size"?
Previous: Derrick Stolee via GitGitGadgetNext: Derrick Stolee
Message 28 of 38 in “pack-objects: create new name-hash algorithm”
  1. 0/4 pack-objects: create new name-hash algorithmDerrick Stolee via GitGitGadget, Sep 9, 2024
  2. 1/4 pack-objects: add --full-name-hash optionDerrick Stolee via GitGitGadget, Sep 9, 2024
  3. Junio C HamanoSep 9, 2024
  4. Derrick StoleeSep 10, 2024
  5. 2/4 git-repack: update usage to match docsDerrick Stolee via GitGitGadget, Sep 9, 2024
  6. Junio C HamanoSep 9, 2024
  7. 3/4 p5313: add size comparison testDerrick Stolee via GitGitGadget, Sep 9, 2024
  8. 4/4 p5314: add a size test for name-hash collisionsDerrick Stolee via GitGitGadget, Sep 9, 2024
  9. Derrick StoleeSep 9, 2024
  10. Junio C HamanoSep 9, 2024
  11. Derrick StoleeSep 10, 2024
  12. Taylor BlauSep 10, 2024
  13. Derrick StoleeSep 10, 2024
  14. Jeff KingSep 11, 2024
  15. Junio C HamanoSep 10, 2024
  16. Junio C HamanoSep 10, 2024
  17. Derrick StoleeSep 10, 2024
  18. 0/6 pack-objects: create new name-hash algorithmDerrick Stolee via GitGitGadget, Sep 18, 2024
  19. 1/6 pack-objects: add --full-name-hash optionDerrick Stolee via GitGitGadget, Sep 18, 2024
  20. Junio C HamanoSep 19, 2024
  21. 2/6 repack: test --full-name-hash optionDerrick Stolee via GitGitGadget, Sep 18, 2024
  22. Junio C HamanoSep 19, 2024
  23. 3/6 pack-objects: add GIT_TEST_FULL_NAME_HASHDerrick Stolee via GitGitGadget, Sep 18, 2024
  24. Junio C HamanoSep 19, 2024
  25. Derrick StoleeSep 23, 2024
  26. 4/6 git-repack: update usage to match docsDerrick Stolee via GitGitGadget, Sep 18, 2024
  27. 5/6 p5313: add size comparison testDerrick Stolee via GitGitGadget, Sep 18, 2024
  28. Junio C HamanoSep 19, 2024
  29. Derrick StoleeSep 23, 2024
  30. Junio C HamanoSep 23, 2024
  31. 6/6 test-tool: add helper for name-hash valuesDerrick Stolee via GitGitGadget, Sep 18, 2024
  32. Patrick SteinhardtSep 24, 2024
  33. Derrick StoleeSep 25, 2024
  34. Derrick StoleeSep 18, 2024
  35. Junio C HamanoOct 4, 2024
  36. Taylor BlauOct 4, 2024
  37. Jeff KingOct 4, 2024
  38. Derrick StoleeOct 8, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.