git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 9/9] t/perf: add perf tests for for-each-ref

From
Victoria Dye via GitGitGadget <gitgitgadget@gmail.com>
Date
Nov 7, 2023, 01:26 UTC
Message-ID
<a409d77305766b7e4d391837f393cc22f9adaeca.1699320362.git.gitgitgadget@gmail.com>
In-Reply-To
<pull.1609.git.1699320361.gitgitgadget@gmail.com>
From: Victoria Dye <vdye@github.com>

Add performance tests for 'for-each-ref'. The tests exercise different combinations of filters/formats/options, as well as the overall performance of 'git for-each-ref | git cat-file --batch-check' to demonstrate the performance difference vs. 'git for-each-ref --full-deref'.

All tests are run against a repository with 40k loose refs - 10k commits, each having a unique:

- branch
- custom ref (refs/custom/special_*)
- annotated tag pointing at the commit
- annotated tag pointing at the other annotated tag (i.e., a nested tag)

After those tests are finished, the refs are packed with 'pack-refs --all' and the same tests are rerun.

Signed-off-by: Victoria Dye <vdye@github.com>
---
 t/perf/p6300-for-each-ref.sh | 87 ++++++++++++++++++++++++++++++++++++
 1 file changed, 87 insertions(+)
 create mode 100755 t/perf/p6300-for-each-ref.sh
diff --git a/t/perf/p6300-for-each-ref.sh b/t/perf/p6300-for-each-ref.sh
new file mode 100755
index 00000000000..172fd68a4e9
--- /dev/null
+++ b/t/perf/p6300-for-each-ref.sh
@@ -0,0 +1,87 @@
+#!/bin/sh
+
+test_description='performance of for-each-ref'
+. ./perf-lib.sh
+
+test_perf_fresh_repo
+
+ref_count_per_type=10000
+test_iteration_count=10
+
+test_expect_success "setup" '
+	test_commit_bulk $(( 1 + $ref_count_per_type )) &&
+
+	# Create refs
+	test_seq $ref_count_per_type |
+		sed "s,.*,update refs/heads/branch_& HEAD~&\nupdate refs/custom/special_& HEAD~&," |
+		git update-ref --stdin &&
+
+	# Create annotated tags
+	for i in $(test_seq $ref_count_per_type)
+	do
+		# Base tags
+		echo "tag tag_$i" &&
+		echo "mark :$i" &&
+		echo "from HEAD~$i" &&
+		printf "tagger %s <%s> %s\n" \
+			"$GIT_COMMITTER_NAME" \
+			"$GIT_COMMITTER_EMAIL" \
+			"$GIT_COMMITTER_DATE" &&
+		echo "data <<EOF" &&
+		echo "tag $i" &&
+		echo "EOF" &&
+
+		# Nested tags
+		echo "tag nested_$i" &&
+		echo "from :$i" &&
+		printf "tagger %s <%s> %s\n" \
+			"$GIT_COMMITTER_NAME" \
+			"$GIT_COMMITTER_EMAIL" \
+			"$GIT_COMMITTER_DATE" &&
+		echo "data <<EOF" &&
+		echo "nested tag $i" &&
+		echo "EOF" || return 1
+	done | git fast-import
+'
+
+test_for_each_ref () {
+	title="for-each-ref"
+	if test $# -gt 0; then
+		title="$title ($1)"
+		shift
+	fi
+	args="$@"
+
+	test_perf "$title" "
+		for i in \$(test_seq $test_iteration_count); do
+			git for-each-ref $args >/dev/null
+		done
+	"
+}
+
+run_tests () {
+	test_for_each_ref "$1"
+	test_for_each_ref "$1, no sort" --no-sort
+	test_for_each_ref "$1, tags" refs/tags/
+	test_for_each_ref "$1, tags, no sort" --no-sort refs/tags/
+	test_for_each_ref "$1, tags, shallow deref" '--format="%(refname) %(objectname) %(*objectname)"' refs/tags/
+	test_for_each_ref "$1, tags, shallow deref, no sort" --no-sort '--format="%(refname) %(objectname) %(*objectname)"' refs/tags/
+	test_for_each_ref "$1, tags, full deref" --full-deref '--format="%(refname) %(objectname) %(*objectname)"' refs/tags/
+	test_for_each_ref "$1, tags, full deref, no sort" --no-sort --full-deref '--format="%(refname) %(objectname) %(*objectname)"' refs/tags/
+
+	test_perf "for-each-ref ($1, tags) + cat-file --batch-check (full deref)" "
+		for i in \$(test_seq $test_iteration_count); do
+			git for-each-ref --format='%(objectname)^{} %(refname) %(objectname)' refs/tags/ | \
+				git cat-file --batch-check='%(objectname) %(rest)' >/dev/null
+		done
+	"
+}
+
+run_tests "loose"
+
+test_expect_success 'pack refs' '
+	git pack-refs --all
+'
+run_tests "packed"
+
+test_done
-- 
gitgitgadget
Previous: Junio C HamanoNext: Junio C Hamano
Message 29 of 49 in “for-each-ref optimizations & usability improvements”
  1. 0/9 for-each-ref optimizations & usability improvementsVictoria Dye via GitGitGadget, Nov 7, 2023
  2. 2/9 for-each-ref: clarify interaction of --omit-empty & --countVictoria Dye via GitGitGadget, Nov 7, 2023
  3. Øystein WalleNov 7, 2023
  4. Victoria DyeNov 7, 2023
  5. Øystein WalleNov 8, 2023
  6. Kristoffer HaugsbakkNov 8, 2023
  7. 1/9 ref-filter.c: really don't sort when using --no-sortVictoria Dye via GitGitGadget, Nov 7, 2023
  8. Patrick SteinhardtNov 7, 2023
  9. Victoria DyeNov 7, 2023
  10. 3/9 ref-filter.h: add max_count and omit_empty to ref_formatVictoria Dye via GitGitGadget, Nov 7, 2023
  11. 4/9 ref-filter.h: move contains caches into filterVictoria Dye via GitGitGadget, Nov 7, 2023
  12. Patrick SteinhardtNov 7, 2023
  13. 5/9 ref-filter.h: add functions for filter/format & format-onlyVictoria Dye via GitGitGadget, Nov 7, 2023
  14. 6/9 ref-filter.c: refactor to create common helper functionsVictoria Dye via GitGitGadget, Nov 7, 2023
  15. Patrick SteinhardtNov 7, 2023
  16. Victoria DyeNov 7, 2023
  17. 7/9 ref-filter.c: filter & format refs in the same callbackVictoria Dye via GitGitGadget, Nov 7, 2023
  18. Patrick SteinhardtNov 7, 2023
  19. Victoria DyeNov 7, 2023
  20. 8/9 for-each-ref: add option to fully dereference tagsVictoria Dye via GitGitGadget, Nov 7, 2023
  21. Patrick SteinhardtNov 7, 2023
  22. Victoria DyeNov 8, 2023
  23. Junio C HamanoNov 8, 2023
  24. Patrick SteinhardtNov 8, 2023
  25. Victoria DyeNov 8, 2023
  26. Junio C HamanoNov 9, 2023
  27. Junio C HamanoNov 9, 2023
  28. Junio C HamanoNov 9, 2023
  29. 9/9 t/perf: add perf tests for for-each-refVictoria Dye via GitGitGadget, Nov 7, 2023
  30. Junio C HamanoNov 7, 2023
  31. Victoria DyeNov 7, 2023
  32. Junio C HamanoNov 7, 2023
  33. Patrick SteinhardtNov 7, 2023
  34. Victoria DyeNov 8, 2023
  35. 00/10 for-each-ref optimizations & usability improvementsVictoria Dye via GitGitGadget, Nov 14, 2023
  36. 01/10 ref-filter.c: really don't sort when using --no-sortVictoria Dye via GitGitGadget, Nov 14, 2023
  37. Junio C HamanoNov 16, 2023
  38. 02/10 ref-filter.h: add max_count and omit_empty to ref_formatVictoria Dye via GitGitGadget, Nov 14, 2023
  39. Øystein WalleNov 16, 2023
  40. 03/10 ref-filter.h: move contains caches into filterVictoria Dye via GitGitGadget, Nov 14, 2023
  41. 04/10 ref-filter.h: add functions for filter/format & format-onlyVictoria Dye via GitGitGadget, Nov 14, 2023
  42. Junio C HamanoNov 16, 2023
  43. 05/10 ref-filter.c: rename 'ref_filter_handler()' to 'filter_one()'Victoria Dye via GitGitGadget, Nov 14, 2023
  44. 06/10 ref-filter.c: refactor to create common helper functionsVictoria Dye via GitGitGadget, Nov 14, 2023
  45. 07/10 ref-filter.c: filter & format refs in the same callbackVictoria Dye via GitGitGadget, Nov 14, 2023
  46. 08/10 for-each-ref: clean up documentation of --formatVictoria Dye via GitGitGadget, Nov 14, 2023
  47. 09/10 ref-filter.c: use peeled tag for '*' format fieldsVictoria Dye via GitGitGadget, Nov 14, 2023
  48. Junio C HamanoNov 16, 2023
  49. 10/10 t/perf: add perf tests for for-each-refVictoria Dye via GitGitGadget, Nov 14, 2023

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.