git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2 02/10] t/perf: add pack-objects filter and path-walk benchmark

From
Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com>
Date
May 4, 2026, 20:21 UTC
Message-ID
<9ea81aa7269c6fac61723ff46323549f35f5d06d.1777926079.git.gitgitgadget@gmail.com>
In-Reply-To
<pull.2101.v2.git.1777926079.gitgitgadget@gmail.com>
From: Derrick Stolee <stolee@gmail.com>

Add p5315-pack-objects-filter.sh to measure the performance of 'git pack-objects --revs --all' under different filter and traversal combinations:

 * no filter (baseline)
 * --filter=blob:none (blobless)
 * --filter=sparse:oid=<oid> (cone-mode sparse)

Each filter scenario is tested both with and without --path-walk, producing paired measurements that show the impact of the path-walk traversal for each filter type as we integrate the --path-walk feature with different --filter options. It currently has no integration so falls back to the standard revision walk. Thus, there are no significant differences in the current results other than a full repack (and even then, the --path-walk feature is not incredibly different for the default Git repository):

Test HEAD ----------------------------------------------------- 5315.2: repack (no filter) 27.91 5315.3: repack size (no filter) 250.7M 5315.4: repack (no filter, --path-walk) 34.92 5315.5: repack size (no filter, --path-walk) 220.0M 5315.6: repack (blob:none) 13.63 5315.7: repack size (blob:none) 137.6M 5315.8: repack (blob:none, --path-walk) 13.48 5315.9: repack size (blob:none, --path-walk) 137.7M 5315.10: repack (sparse:oid) 72.67 5315.11: repack size (sparse:oid) 187.4M 5315.12: repack (sparse:oid, --path-walk) 72.47 5315.13: repack size (sparse:oid, --path-walk) 187.4M

The sparse filter definition is built automatically by sampling depth-2 directories from the test repository, making the test work on any repo passed via GIT_PERF_LARGE_REPO. For repos that lack depth-2 directories, a single top-level directory is used; for flat repos, the sparse tests are skipped via prerequisite.

Signed-off-by: Derrick Stolee <stolee@gmail.com>
---
 t/perf/p5315-pack-objects-filter.sh | 129 ++++++++++++++++++++++++++++
 1 file changed, 129 insertions(+)
 create mode 100755 t/perf/p5315-pack-objects-filter.sh
diff --git a/t/perf/p5315-pack-objects-filter.sh b/t/perf/p5315-pack-objects-filter.sh
new file mode 100755
index 0000000000..b009039c89
--- /dev/null
+++ b/t/perf/p5315-pack-objects-filter.sh
@@ -0,0 +1,129 @@
+#!/bin/sh
+
+test_description='Tests pack-objects performance with filters and --path-walk'
+. ./perf-lib.sh
+
+test_perf_large_repo
+
+test_expect_success 'setup filter inputs' '
+	# Sample a few depth-2 directories from the test repo to build
+	# a cone-mode sparse-checkout definition.  The sampling picks
+	# directories at evenly-spaced positions so the choice is stable
+	# and scales to repos of any shape.
+
+	git ls-tree -d --name-only HEAD >top-dirs &&
+	top_nr=$(wc -l <top-dirs) &&
+
+	>depth2-dirs &&
+	while read tdir
+	do
+		git ls-tree -d --name-only "HEAD:$tdir" 2>/dev/null |
+		sed "s|^|$tdir/|" >>depth2-dirs || return 1
+	done <top-dirs &&
+
+	d2_nr=$(wc -l <depth2-dirs) &&
+
+	if test "$d2_nr" -ge 2
+	then
+		# Pick two directories from evenly-spaced positions.
+		first=$(sed -n "1p" depth2-dirs) &&
+		mid=$(sed -n "$((d2_nr / 2 + 1))p" depth2-dirs) &&
+
+		p1=$(dirname "$first") &&
+		p2=$(dirname "$mid") &&
+
+		# Build cone-mode sparse-checkout patterns.
+		{
+			echo "/*" &&
+			echo "!/*/" &&
+			echo "/$p1/" &&
+			echo "!/$p1/*/" &&
+			if test "$p1" != "$p2"
+			then
+				echo "/$p2/" &&
+				echo "!/$p2/*/"
+			fi &&
+			echo "/$first/" &&
+			if test "$first" != "$mid"
+			then
+				echo "/$mid/"
+			fi
+		} >sparse-patterns &&
+
+		git hash-object -w sparse-patterns >sparse-oid &&
+		echo "Sparse cone: $first $mid" &&
+		cat sparse-patterns &&
+		test_set_prereq SPARSE_OID
+	elif test "$top_nr" -ge 1
+	then
+		# Fallback: use a single top-level directory.
+		first=$(sed -n "1p" top-dirs) &&
+		{
+			echo "/*" &&
+			echo "!/*/" &&
+			echo "/$first/"
+		} >sparse-patterns &&
+
+		git hash-object -w sparse-patterns >sparse-oid &&
+		echo "Sparse cone: $first" &&
+		cat sparse-patterns &&
+		test_set_prereq SPARSE_OID
+	fi
+'
+
+test_perf 'repack (no filter)' '
+	git pack-objects --stdout --no-reuse-delta --revs --all </dev/null >pk
+'
+
+test_size 'repack size (no filter)' '
+	test_file_size pk
+'
+
+test_perf 'repack (no filter, --path-walk)' '
+	git pack-objects --stdout --no-reuse-delta --revs --all --path-walk </dev/null >pk
+'
+
+test_size 'repack size (no filter, --path-walk)' '
+	test_file_size pk
+'
+
+test_perf 'repack (blob:none)' '
+	git pack-objects --stdout --no-reuse-delta --revs --all --filter=blob:none </dev/null >pk
+'
+
+test_size 'repack size (blob:none)' '
+	test_file_size pk
+'
+
+test_perf 'repack (blob:none, --path-walk)' '
+	git pack-objects --stdout --no-reuse-delta --revs --all --path-walk \
+		--filter=blob:none </dev/null >pk
+'
+
+test_size 'repack size (blob:none, --path-walk)' '
+	test_file_size pk
+'
+
+test_perf 'repack (sparse:oid)' \
+	--prereq SPARSE_OID '
+	git pack-objects --stdout --no-reuse-delta --revs --all \
+		--filter=sparse:oid=$(cat sparse-oid) </dev/null >pk
+'
+
+test_size 'repack size (sparse:oid)' \
+	--prereq SPARSE_OID '
+	test_file_size pk
+'
+
+test_perf 'repack (sparse:oid, --path-walk)' \
+	--prereq SPARSE_OID '
+	git pack-objects --stdout --no-reuse-delta --revs --all --path-walk \
+		--filter=sparse:oid=$(cat sparse-oid) </dev/null >pk
+'
+
+test_size 'repack size (sparse:oid, --path-walk)' \
+	--prereq SPARSE_OID '
+	test_file_size pk
+'
+
+test_done
-- 
gitgitgadget
Previous: Derrick Stolee via GitGitGadgetNext: Derrick Stolee via GitGitGadget
Message 15 of 96 in “pack-objects: integrate --path-walk and some --filter options”
  1. 0/7 pack-objects: integrate --path-walk and some --filter optionsDerrick Stolee via GitGitGadget, May 2, 2026
  2. 1/7 pack-objects: pass --objects with --path-walkDerrick Stolee via GitGitGadget, May 2, 2026
  3. Junio C HamanoMay 4, 2026
  4. Derrick StoleeMay 4, 2026
  5. 2/7 t/perf: add pack-objects filter and path-walk benchmarkDerrick Stolee via GitGitGadget, May 2, 2026
  6. 3/7 path-walk: support blobless filterDerrick Stolee via GitGitGadget, May 2, 2026
  7. 4/7 backfill: die on incompatible filter optionsDerrick Stolee via GitGitGadget, May 2, 2026
  8. Junio C HamanoMay 3, 2026
  9. Derrick StoleeMay 4, 2026
  10. 5/7 path-walk: support blob size limit filterDerrick Stolee via GitGitGadget, May 2, 2026
  11. 6/7 path-walk: add pl_sparse_trees to control tree pruningDerrick Stolee via GitGitGadget, May 2, 2026
  12. 7/7 pack-objects: support sparse:oid filter with path-walkDerrick Stolee via GitGitGadget, May 2, 2026
  13. 00/10 pack-objects: integrate --path-walk and some --filter optionsDerrick Stolee via GitGitGadget, May 4, 2026
  14. 01/10 pack-objects: pass --objects with --path-walkDerrick Stolee via GitGitGadget, May 4, 2026
  15. 02/10 t/perf: add pack-objects filter and path-walk benchmarkDerrick Stolee via GitGitGadget, May 4, 2026
  16. 03/10 path-walk: support blobless filterDerrick Stolee via GitGitGadget, May 4, 2026
  17. 04/10 backfill: die on incompatible filter optionsDerrick Stolee via GitGitGadget, May 4, 2026
  18. 05/10 path-walk: support blob size limit filterDerrick Stolee via GitGitGadget, May 4, 2026
  19. 06/10 path-walk: add pl_sparse_trees to control tree pruningDerrick Stolee via GitGitGadget, May 4, 2026
  20. 07/10 pack-objects: support sparse:oid filter with path-walkDerrick Stolee via GitGitGadget, May 4, 2026
  21. 08/10 path-walk: support `tree:0` filterTaylor Blau via GitGitGadget, May 4, 2026
  22. 09/10 path-walk: support `object:type` filterTaylor Blau via GitGitGadget, May 4, 2026
  23. 10/10 path-walk: support `combine` filterTaylor Blau via GitGitGadget, May 4, 2026
  24. Derrick StoleeMay 5, 2026
  25. Taylor BlauMay 5, 2026
  26. Derrick StoleeMay 5, 2026
  27. Taylor BlauMay 5, 2026
  28. Derrick StoleeMay 7, 2026
  29. Junio C HamanoMay 11, 2026
  30. Derrick StoleeMay 11, 2026
  31. 00/12 pack-objects: integrate --path-walk and some --filter optionsDerrick Stolee via GitGitGadget, May 11, 2026
  32. 01/12 t5620: make test work with path-walk varDerrick Stolee via GitGitGadget, May 11, 2026
  33. Taylor BlauMay 12, 2026
  34. 02/12 pack-objects: pass --objects with --path-walkDerrick Stolee via GitGitGadget, May 11, 2026
  35. Taylor BlauMay 12, 2026
  36. 03/12 t/perf: add pack-objects filter and path-walk benchmarkDerrick Stolee via GitGitGadget, May 11, 2026
  37. Taylor BlauMay 12, 2026
  38. Derrick StoleeMay 13, 2026
  39. 04/12 path-walk: always emit directly-requested objectsDerrick Stolee via GitGitGadget, May 11, 2026
  40. Taylor BlauMay 12, 2026
  41. Derrick StoleeMay 13, 2026
  42. 05/12 path-walk: support blobless filterDerrick Stolee via GitGitGadget, May 11, 2026
  43. Taylor BlauMay 11, 2026
  44. Derrick StoleeMay 11, 2026
  45. 06/12 backfill: die on incompatible filter optionsDerrick Stolee via GitGitGadget, May 11, 2026
  46. Taylor BlauMay 12, 2026
  47. 07/12 path-walk: support blob size limit filterDerrick Stolee via GitGitGadget, May 11, 2026
  48. Taylor BlauMay 12, 2026
  49. Derrick StoleeMay 13, 2026
  50. 08/12 path-walk: add pl_sparse_trees to control tree pruningDerrick Stolee via GitGitGadget, May 11, 2026
  51. 09/12 pack-objects: support sparse:oid filter with path-walkDerrick Stolee via GitGitGadget, May 11, 2026
  52. 10/12 path-walk: support `tree:0` filterTaylor Blau via GitGitGadget, May 11, 2026
  53. Taylor BlauMay 12, 2026
  54. Derrick StoleeMay 13, 2026
  55. 11/12 path-walk: support `object:type` filterTaylor Blau via GitGitGadget, May 11, 2026
  56. 12/12 path-walk: support `combine` filterTaylor Blau via GitGitGadget, May 11, 2026
  57. Taylor BlauMay 12, 2026
  58. 00/13 pack-objects: integrate --path-walk and some --filter optionsDerrick Stolee via GitGitGadget, May 13, 2026
  59. 01/13 t5620: make test work with path-walk varDerrick Stolee via GitGitGadget, May 13, 2026
  60. 02/13 pack-objects: pass --objects with --path-walkDerrick Stolee via GitGitGadget, May 13, 2026
  61. 03/13 t/perf: add pack-objects filter and path-walk benchmarkDerrick Stolee via GitGitGadget, May 13, 2026
  62. Taylor BlauMay 19, 2026
  63. Derrick StoleeMay 21, 2026
  64. 04/13 path-walk: always emit directly-requested objectsDerrick Stolee via GitGitGadget, May 13, 2026
  65. Taylor BlauMay 19, 2026
  66. Derrick StoleeMay 21, 2026
  67. Taylor BlauMay 22, 2026
  68. 05/13 path-walk: support blobless filterDerrick Stolee via GitGitGadget, May 13, 2026
  69. 06/13 backfill: die on incompatible filter optionsDerrick Stolee via GitGitGadget, May 13, 2026
  70. 07/13 path-walk: support blob size limit filterDerrick Stolee via GitGitGadget, May 13, 2026
  71. 08/13 path-walk: add pl_sparse_trees to control tree pruningDerrick Stolee via GitGitGadget, May 13, 2026
  72. 09/13 pack-objects: support sparse:oid filter with path-walkDerrick Stolee via GitGitGadget, May 13, 2026
  73. 10/13 t6601: tag otherwise-unreachable treesDerrick Stolee via GitGitGadget, May 13, 2026
  74. 11/13 path-walk: support `tree:0` filterTaylor Blau via GitGitGadget, May 13, 2026
  75. 12/13 path-walk: support `object:type` filterTaylor Blau via GitGitGadget, May 13, 2026
  76. 13/13 path-walk: support `combine` filterTaylor Blau via GitGitGadget, May 13, 2026
  77. Taylor BlauMay 19, 2026
  78. Derrick StoleeMay 21, 2026
  79. Taylor BlauMay 22, 2026
  80. Derrick StoleeMay 22, 2026
  81. 00/13 pack-objects: integrate --path-walk and some --filter optionsDerrick Stolee via GitGitGadget, May 22, 2026
  82. 01/13 t5620: make test work with path-walk varDerrick Stolee via GitGitGadget, May 22, 2026
  83. 02/13 pack-objects: pass --objects with --path-walkDerrick Stolee via GitGitGadget, May 22, 2026
  84. 03/13 t/perf: add pack-objects filter and path-walk benchmarkDerrick Stolee via GitGitGadget, May 22, 2026
  85. 04/13 path-walk: always emit directly-requested objectsDerrick Stolee via GitGitGadget, May 22, 2026
  86. 05/13 path-walk: support blobless filterDerrick Stolee via GitGitGadget, May 22, 2026
  87. 06/13 backfill: die on incompatible filter optionsDerrick Stolee via GitGitGadget, May 22, 2026
  88. 07/13 path-walk: support blob size limit filterDerrick Stolee via GitGitGadget, May 22, 2026
  89. 08/13 path-walk: add pl_sparse_trees to control tree pruningDerrick Stolee via GitGitGadget, May 22, 2026
  90. 09/13 pack-objects: support sparse:oid filter with path-walkDerrick Stolee via GitGitGadget, May 22, 2026
  91. 10/13 t6601: tag otherwise-unreachable treesDerrick Stolee via GitGitGadget, May 22, 2026
  92. 11/13 path-walk: support `tree:0` filterTaylor Blau via GitGitGadget, May 22, 2026
  93. 12/13 path-walk: support `object:type` filterTaylor Blau via GitGitGadget, May 22, 2026
  94. 13/13 path-walk: support `combine` filterTaylor Blau via GitGitGadget, May 22, 2026
  95. Taylor BlauMay 22, 2026
  96. Junio C HamanoMay 24, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.