git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 5/7] p5313: add size comparison test

From
Taylor Blau <me@ttaylorr.com>
Date
Nov 21, 2024, 20:31 UTC
Message-ID
<Zz+YrvL8h0Cxwqfy@nand.local>
In-Reply-To
<c14ef6879e451401381ebbdb8f30d33c8f56c25b.1730775908.git.gitgitgadget@gmail.com>
On Tue, Nov 05, 2024 at 03:05:05AM +0000, Derrick Stolee via GitGitGadget wrote:
Show 5 quoted lines
> From: Derrick Stolee <stolee@gmail.com>
>
> As custom options are added to 'git pack-objects' and 'git repack' to
> adjust how compression is done, use this new performance test script to
> demonstrate their effectiveness in performance and size.

Nicely done, thank you for adding a perf test to allow readers to easily verify these changes themselves.

Show 10 quoted lines
> In the case of the Git repository, these numbers show some of the issues
> with this approach:
>
> [...]
>
> The thin pack that simulates a push is much worse with --full-name-hash
> in this case. The name hash values are doing a lot to assist with delta
> bases, it seems. The big pack and shallow clone cases are slightly worse
> with the --full-name-hash option. Only the full repack gains some
> benefits in size.

Not a problem with your patch, but just thinking aloud: do you think there is an easy/straightforward way to suggest when to use --full-name-hash or not?

Show 37 quoted lines
> ---
>  t/perf/p5313-pack-objects.sh | 94 ++++++++++++++++++++++++++++++++++++
>  1 file changed, 94 insertions(+)
>  create mode 100755 t/perf/p5313-pack-objects.sh
>
> diff --git a/t/perf/p5313-pack-objects.sh b/t/perf/p5313-pack-objects.sh
> new file mode 100755
> index 00000000000..dfa29695315
> --- /dev/null
> +++ b/t/perf/p5313-pack-objects.sh
> @@ -0,0 +1,94 @@
> +#!/bin/sh
> +
> +test_description='Tests pack performance using bitmaps'
> +. ./perf-lib.sh
> +
> +GIT_TEST_PASSING_SANITIZE_LEAK=0
> +export GIT_TEST_PASSING_SANITIZE_LEAK
> +
> +test_perf_large_repo
> +
> +test_expect_success 'create rev input' '
> +	cat >in-thin <<-EOF &&
> +	$(git rev-parse HEAD)
> +	^$(git rev-parse HEAD~1)
> +	EOF
> +
> +	cat >in-big <<-EOF &&
> +	$(git rev-parse HEAD)
> +	^$(git rev-parse HEAD~1000)
> +	EOF
> +
> +	cat >in-shallow <<-EOF
> +	$(git rev-parse HEAD)
> +	--shallow $(git rev-parse HEAD)
> +	EOF
> +'

I was going to comment that these could probably be moved into the individual perf test that cares about reading each of these inputs. But having them shared here makes sense since we are naturally comparing generating two packs with the same input (with and without --full-name-hash). So the shared setup here makes sense to me.

Show 8 quoted lines
> +
> +test_perf 'thin pack' '
> +	git pack-objects --thin --stdout --revs --sparse  <in-thin >out
> +'
> +
> +test_size 'thin pack size' '
> +	test_file_size out
> +'
Nice. I always forget about this and end up writing 'wc -c <out'.
Show 47 quoted lines
> +test_perf 'thin pack with --full-name-hash' '
> +	git pack-objects --thin --stdout --revs --sparse --full-name-hash <in-thin >out
> +'
> +
> +test_size 'thin pack size with --full-name-hash' '
> +	test_file_size out
> +'
> +
> +test_perf 'big pack' '
> +	git pack-objects --stdout --revs --sparse  <in-big >out
> +'
> +
> +test_size 'big pack size' '
> +	test_file_size out
> +'
> +
> +test_perf 'big pack with --full-name-hash' '
> +	git pack-objects --stdout --revs --sparse --full-name-hash <in-big >out
> +'
> +
> +test_size 'big pack size with --full-name-hash' '
> +	test_file_size out
> +'
> +
> +test_perf 'shallow fetch pack' '
> +	git pack-objects --stdout --revs --sparse --shallow <in-shallow >out
> +'
> +
> +test_size 'shallow pack size' '
> +	test_file_size out
> +'
> +
> +test_perf 'shallow pack with --full-name-hash' '
> +	git pack-objects --stdout --revs --sparse --shallow --full-name-hash <in-shallow >out
> +'
> +
> +test_size 'shallow pack size with --full-name-hash' '
> +	test_file_size out
> +'
> +
> +test_perf 'repack' '
> +	git repack -adf
> +'
> +
> +test_size 'repack size' '
> +	pack=$(ls .git/objects/pack/pack-*.pack) &&
> +	test_file_size "$pack"
Here and below, I think it's fine to inline this as in:
    test_file_size "$(ls .git/objects/pack/pack-*.pack)"

...but I wonder: will using ".git" break this test in bare repositories? Should we write instead:

    pack="$(ls $(git rev-parse --git-dir)/objects/pack/pack-*.pack)" &&
    test_file_size
?

Thanks, Taylor

Previous: Derrick Stolee via GitGitGadgetNext: Derrick Stolee
Message 26 of 93 in “pack-objects: Create an alternative name hash algorithm (recreated)”
  1. 0/7 pack-objects: Create an alternative name hash algorithm (recreated)Derrick Stolee via GitGitGadget, Nov 5, 2024
  2. 1/7 pack-objects: add --full-name-hash optionDerrick Stolee via GitGitGadget, Nov 5, 2024
  3. Taylor BlauNov 21, 2024
  4. Taylor BlauNov 21, 2024
  5. Junio C HamanoNov 21, 2024
  6. Derrick StoleeNov 22, 2024
  7. Derrick StoleeNov 22, 2024
  8. Patrick SteinhardtNov 26, 2024
  9. 2/7 repack: add --full-name-hash optionDerrick Stolee via GitGitGadget, Nov 5, 2024
  10. Taylor BlauNov 21, 2024
  11. Derrick StoleeNov 22, 2024
  12. 3/7 pack-objects: add GIT_TEST_FULL_NAME_HASHDerrick Stolee via GitGitGadget, Nov 5, 2024
  13. Taylor BlauNov 21, 2024
  14. Derrick StoleeNov 22, 2024
  15. Jonathan TanNov 22, 2024
  16. Junio C HamanoNov 22, 2024
  17. Jonathan TanNov 22, 2024
  18. Junio C HamanoNov 25, 2024
  19. Jonathan TanNov 25, 2024
  20. Junio C HamanoNov 26, 2024
  21. Patrick SteinhardtNov 26, 2024
  22. 4/7 git-repack: update usage to match docsDerrick Stolee via GitGitGadget, Nov 5, 2024
  23. Taylor BlauNov 21, 2024
  24. Derrick StoleeNov 22, 2024
  25. 5/7 p5313: add size comparison testDerrick Stolee via GitGitGadget, Nov 5, 2024
  26. Taylor BlauNov 21, 2024
  27. Derrick StoleeNov 22, 2024
  28. Patrick SteinhardtNov 26, 2024
  29. 6/7 pack-objects: disable --full-name-hash when shallowDerrick Stolee via GitGitGadget, Nov 5, 2024
  30. Taylor BlauNov 21, 2024
  31. Derrick StoleeNov 22, 2024
  32. 7/7 test-tool: add helper for name-hash valuesDerrick Stolee via GitGitGadget, Nov 5, 2024
  33. Taylor BlauNov 21, 2024
  34. Jonathan TanNov 22, 2024
  35. Jonathan TanNov 21, 2024
  36. Junio C HamanoNov 22, 2024
  37. Junio C HamanoNov 22, 2024
  38. Derrick StoleeNov 22, 2024
  39. Junio C HamanoNov 24, 2024
  40. Jonathan TanNov 22, 2024
  41. 0/8 pack-objects: Create an alternative name hash algorithm (recreated)Derrick Stolee via GitGitGadget, Dec 2, 2024
  42. 1/8 pack-objects: create new name-hash function versionJonathan Tan via GitGitGadget, Dec 2, 2024
  43. karthik nayakDec 4, 2024
  44. Junio C HamanoDec 4, 2024
  45. karthik nayakDec 5, 2024
  46. Jonathan TanDec 9, 2024
  47. Junio C HamanoDec 10, 2024
  48. 2/8 pack-objects: add --name-hash-version optionDerrick Stolee via GitGitGadget, Dec 2, 2024
  49. karthik nayakDec 4, 2024
  50. 3/8 repack: add --name-hash-version optionDerrick Stolee via GitGitGadget, Dec 2, 2024
  51. karthik nayakDec 4, 2024
  52. 4/8 pack-objects: add GIT_TEST_NAME_HASH_VERSIONDerrick Stolee via GitGitGadget, Dec 2, 2024
  53. karthik nayakDec 4, 2024
  54. Jonathan TanDec 9, 2024
  55. Derrick StoleeDec 20, 2024
  56. 5/8 p5313: add size comparison testDerrick Stolee via GitGitGadget, Dec 2, 2024
  57. 6/8 test-tool: add helper for name-hash valuesDerrick Stolee via GitGitGadget, Dec 2, 2024
  58. 7/8 pack-objects: prevent name hash version changeDerrick Stolee via GitGitGadget, Dec 2, 2024
  59. 8/8 pack-objects: add third name hash versionDerrick Stolee via GitGitGadget, Dec 2, 2024
  60. Junio C HamanoDec 3, 2024
  61. Derrick StoleeDec 4, 2024
  62. Junio C HamanoDec 4, 2024
  63. 0/8 pack-objects: Create an alternative name hash algorithm (recreated)Derrick Stolee via GitGitGadget, Dec 20, 2024
  64. 1/8 pack-objects: create new name-hash function versionJonathan Tan via GitGitGadget, Dec 20, 2024
  65. Taylor BlauJan 22, 2025
  66. 2/8 pack-objects: add --name-hash-version optionDerrick Stolee via GitGitGadget, Dec 20, 2024
  67. Taylor BlauJan 22, 2025
  68. Derrick StoleeJan 24, 2025
  69. 3/8 repack: add --name-hash-version optionDerrick Stolee via GitGitGadget, Dec 20, 2024
  70. Taylor BlauJan 22, 2025
  71. 4/8 pack-objects: add GIT_TEST_NAME_HASH_VERSIONDerrick Stolee via GitGitGadget, Dec 20, 2024
  72. Taylor BlauJan 22, 2025
  73. 5/8 p5313: add size comparison testDerrick Stolee via GitGitGadget, Dec 20, 2024
  74. 6/8 test-tool: add helper for name-hash valuesDerrick Stolee via GitGitGadget, Dec 20, 2024
  75. 7/8 pack-objects: prevent name hash version changeDerrick Stolee via GitGitGadget, Dec 20, 2024
  76. Taylor BlauJan 22, 2025
  77. 8/8 pack-objects: add third name hash versionDerrick Stolee via GitGitGadget, Dec 20, 2024
  78. Taylor BlauJan 22, 2025
  79. Derrick StoleeJan 24, 2025
  80. Derrick StoleeJan 21, 2025
  81. Taylor BlauJan 22, 2025
  82. Derrick StoleeJan 24, 2025
  83. 0/7 pack-objects: Create an alternative name hash algorithm (recreated)Derrick Stolee via GitGitGadget, Jan 27, 2025
  84. 1/7 pack-objects: create new name-hash function versionJonathan Tan via GitGitGadget, Jan 27, 2025
  85. 3/7 repack: add --name-hash-version optionDerrick Stolee via GitGitGadget, Jan 27, 2025
  86. 2/7 pack-objects: add --name-hash-version optionDerrick Stolee via GitGitGadget, Jan 27, 2025
  87. Junio C HamanoJan 27, 2025
  88. Derrick StoleeJan 29, 2025
  89. 4/7 pack-objects: add GIT_TEST_NAME_HASH_VERSIONDerrick Stolee via GitGitGadget, Jan 27, 2025
  90. 5/7 p5313: add size comparison testDerrick Stolee via GitGitGadget, Jan 27, 2025
  91. 6/7 test-tool: add helper for name-hash valuesDerrick Stolee via GitGitGadget, Jan 27, 2025
  92. 7/7 pack-objects: prevent name hash version changeDerrick Stolee via GitGitGadget, Jan 27, 2025
  93. Taylor BlauJan 31, 2025

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.