[PATCH v2 00/22] avoid redundant pipelines
As follow-up to [1], here's a series that eliminates redundant pipelines, mostly under t/.
Spawning a process is notoriously slow on Windows, so this will improve test performance.
1/22 to 14/22 remove redundant uses of "cat" where the shell handles input redirection from a file or the command can take a file argument.
15/22 to 20/22 merge redundant uses of "grep" into the following "sed" or "awk" command. Both of these are capable of filtering themselves.
21/22 merges multiple "sed" calls into one and replaces a whole "cat | grep | awk" construct with a single "awk" call and uses "sort -u" instead of "sort | uniq".
22/22 finally is a bit of an outlier in that it replaces a subshell with a shell compound command.
In the tests, I have completely arbitrarily separated the commits into thousands blocks to keep the commits and their reviews manageable.
All tests that I was able to run still pass. I don't have p4 or cvs installed.
[1] https://lore.kernel.org/git/20240216171046.927552-1-dev+git@drbeat.li/
Changes from v1 (see also the range-diff below):
- 09/22: leave a "cat" that concatenates two files as input to "git am"
- 14/22: replace all instances of
cat >expect.unsorted <<... && sort expect.unsorted >expect
with
"sort >expect <<..."
- 19/22: avoid "sed -E" (use extended regex syntax) which may not be
supported on each platform with equivalent basic regexes
Beat Bolli (22):
doc: avoid redundant use of cat
contrib/subtree/t: avoid redundant use of cat
t/lib-cvs.sh: avoid redundant use of cat
t/annotate-tests.sh: avoid redundant use of cat
t/perf: avoid redundant use of cat
t/t0*: avoid redundant uses of cat
t/t1*: avoid redundant uses of cat
t/t3*: avoid redundant uses of cat
t/t4*: avoid redundant uses of cat
t/t5*: avoid redundant uses of cat
t/t6*: avoid redundant uses of cat
t/t7*: avoid redundant use of cat
t/t8*: avoid redundant use of cat
t/t9*: avoid redundant uses of cat
t/t1*: merge a "grep | sed" pipeline
t/t3*: merge a "grep | awk" pipeline
t/t4*: merge a "grep | sed" pipeline
t/t5*: merge a "grep | sed" pipeline
t/t8*: merge "grep | sed" pipelines
t/t9*: merge "grep | sed" pipelines
contrib/coverage-diff: avoid redundant pipelines
git-quiltimport: avoid an unnecessary subshell
Documentation/howto/update-hook-example.txt | 4 +--
contrib/coverage-diff.sh | 9 ++----
contrib/subtree/t/t7900-subtree.sh | 2 +-
git-quiltimport.sh | 2 +-
t/annotate-tests.sh | 2 +-
t/lib-cvs.sh | 4 +--
t/perf/repos/inflate-repo.sh | 2 +-
t/t0002-gitfile.sh | 2 +-
t/t0011-hashmap.sh | 2 +-
t/t0028-working-tree-encoding.sh | 4 +--
t/t0204-gettext-reencode-sanity.sh | 2 +-
t/t1007-hash-object.sh | 6 ++--
t/t1091-sparse-checkout-builtin.sh | 2 +-
t/t1509/prepare-chroot.sh | 2 +-
t/t3200-branch.sh | 2 +-
t/t3321-notes-stripspace.sh | 8 ++---
t/t3920-crlf-messages.sh | 4 +--
t/t4002-diff-basic.sh | 2 +-
t/t4020-diff-external.sh | 2 +-
t/t4205-log-pretty-formats.sh | 2 +-
t/t4301-merge-tree-write-tree.sh | 8 ++---
t/t5100-mailinfo.sh | 2 +-
t/t5317-pack-objects-filter-objects.sh | 2 +-
t/t5401-update-hooks.sh | 2 +-
t/t5534-push-signed.sh | 2 +-
t/t6112-rev-list-filters-objects.sh | 2 +-
t/t6413-merge-crlf.sh | 4 +--
t/t7704-repack-cruft.sh | 2 +-
t/t8010-cat-file-filters.sh | 2 +-
t/t8013-blame-ignore-revs.sh | 28 ++++++++---------
t/t9118-git-svn-funky-branch-names.sh | 2 +-
t/t9300-fast-import.sh | 14 +++------
t/t9350-fast-export.sh | 4 +--
t/t9400-git-cvsserver-server.sh | 35 ++++++++++-----------
t/t9802-git-p4-filetype.sh | 2 +-
t/t9807-git-p4-submit.sh | 2 +-
t/t9824-git-p4-git-lfs.sh | 4 +--
37 files changed, 85 insertions(+), 97 deletions(-)
Range-diff from v1:
1: 270f19b35e36 = 1: 270f19b35e36 doc: avoid redundant use of cat
2: eed1fc58a7e4 = 2: eed1fc58a7e4 contrib/subtree/t: avoid redundant use of cat
3: c131707668ac = 3: c131707668ac t/lib-cvs.sh: avoid redundant use of cat
4: da69f859de63 = 4: da69f859de63 t/annotate-tests.sh: avoid redundant use of cat
5: 07b3bf33577f = 5: 07b3bf33577f t/perf: avoid redundant use of cat
6: 38907650b33e = 6: 38907650b33e t/t0*: avoid redundant uses of cat
7: b8b8b7ae47a6 = 7: b8b8b7ae47a6 t/t1*: avoid redundant uses of cat
8: 7c91c4092e7d = 8: 7c91c4092e7d t/t3*: avoid redundant uses of cat
9: 3c9df39774d8 ! 9: 1a80eaa97169 t/t4*: avoid redundant uses of cat
@@ t/t4020-diff-external.sh: keep_only_cr () {
test_expect_success 'diff --cached' '
- ## t/t4150-am.sh ##
-@@ t/t4150-am.sh: test_expect_success 'am takes patches from a Pine mailbox' '
- rm -fr .git/rebase-apply &&
- git reset --hard &&
- git checkout first &&
-- cat pine patch1 | git am &&
-+ git am pine patch1 &&
- test_path_is_missing .git/rebase-apply &&
- git diff --exit-code main^..HEAD
- '
-
## t/t4205-log-pretty-formats.sh ##
@@ t/t4205-log-pretty-formats.sh: test_expect_success 'NUL termination with --reflog --pretty=oneline' '
for r in $revs
10: a3207b26a80e = 10: 3ac01bbcaafa t/t5*: avoid redundant uses of cat
11: bde50bfd2552 = 11: a4a6aaf4654b t/t6*: avoid redundant uses of cat
12: 84978c31631f = 12: 4b7b9df129a3 t/t7*: avoid redundant use of cat
13: a555a1809e84 = 13: ebf1e5eeadbd t/t8*: avoid redundant use of cat
14: 667c04d6c665 ! 14: 34cf66aeff80 t/t9*: avoid redundant uses of cat
@@ t/t9300-fast-import.sh: test_expect_success 'L: nested tree copy does not corrup
test_cmp expect actual &&
git fsck $(git rev-parse L2)
'
-@@ t/t9300-fast-import.sh: test_expect_success 'Q: verify first notes tree' '
+@@ t/t9300-fast-import.sh: test_expect_success 'Q: verify first notes commit' '
+ '
+
+ test_expect_success 'Q: verify first notes tree' '
+- cat >expect.unsorted <<-EOF &&
++ sort >expect <<-EOF &&
+ 100644 blob $commit1
100644 blob $commit2
100644 blob $commit3
EOF
- cat expect.unsorted | sort >expect &&
-+ sort expect.unsorted >expect &&
git cat-file -p refs/notes/foobar~2^{tree} | sed "s/ [0-9a-f]* / /" >actual &&
test_cmp expect actual
'
-@@ t/t9300-fast-import.sh: test_expect_success 'Q: verify second notes tree' '
+@@ t/t9300-fast-import.sh: test_expect_success 'Q: verify second notes commit' '
+ '
+
+ test_expect_success 'Q: verify second notes tree' '
+- cat >expect.unsorted <<-EOF &&
++ sort >expect <<-EOF &&
+ 100644 blob $commit1
100644 blob $commit2
100644 blob $commit3
EOF
- cat expect.unsorted | sort >expect &&
-+ sort expect.unsorted >expect &&
git cat-file -p refs/notes/foobar^^{tree} | sed "s/ [0-9a-f]* / /" >actual &&
test_cmp expect actual
'
-@@ t/t9300-fast-import.sh: test_expect_success 'Q: verify third notes tree' '
- cat >expect.unsorted <<-EOF &&
+@@ t/t9300-fast-import.sh: test_expect_success 'Q: verify third notes commit' '
+ '
+
+ test_expect_success 'Q: verify third notes tree' '
+- cat >expect.unsorted <<-EOF &&
++ sort >expect <<-EOF &&
100644 blob $commit1
EOF
- cat expect.unsorted | sort >expect &&
-+ sort expect.unsorted >expect &&
git cat-file -p refs/notes/foobar2^{tree} | sed "s/ [0-9a-f]* / /" >actual &&
test_cmp expect actual
'
-@@ t/t9300-fast-import.sh: test_expect_success 'Q: verify fourth notes tree' '
- cat >expect.unsorted <<-EOF &&
+@@ t/t9300-fast-import.sh: test_expect_success 'Q: verify fourth notes commit' '
+ '
+
+ test_expect_success 'Q: verify fourth notes tree' '
+- cat >expect.unsorted <<-EOF &&
++ sort >expect <<-EOF &&
100644 blob $commit2
EOF
- cat expect.unsorted | sort >expect &&
-+ sort expect.unsorted >expect &&
git cat-file -p refs/notes/foobar^{tree} | sed "s/ [0-9a-f]* / /" >actual &&
test_cmp expect actual
'
15: 6dd79d30c42a = 15: be300cb0a65a t/t1*: merge a "grep | sed" pipeline
16: db61133fa92f = 16: d6f5e21942ff t/t3*: merge a "grep | awk" pipeline
17: 5ce7ddeb2fbd = 17: 68a5f3f743d0 t/t4*: merge a "grep | sed" pipeline
18: 6ed68fe96493 = 18: 3a7e2a8fc388 t/t5*: merge a "grep | sed" pipeline
19: 698a5524b5c4 ! 19: e6ec4b3a0451 t/t8*: merge "grep | sed" pipelines
@@ t/t8013-blame-ignore-revs.sh: test_expect_success setup '
git blame --line-porcelain file >blame_raw &&
- grep -E "^[0-9a-f]+ [0-9]+ 1" blame_raw | sed -e "s/ .*//" >actual &&
-+ sed -Ene "/^[0-9a-f]+ [0-9]+ 1/s/ .*//p" blame_raw >actual &&
++ sed -ne "/^[0-9a-f][0-9a-f]* [0-9][0-9]* 1/s/ .*//p" blame_raw >actual &&
git rev-parse X >expect &&
test_cmp expect actual &&
- grep -E "^[0-9a-f]+ [0-9]+ 2" blame_raw | sed -e "s/ .*//" >actual &&
-+ sed -Ene "/^[0-9a-f]+ [0-9]+ 2/s/ .*//p" blame_raw >actual &&
++ sed -ne "/^[0-9a-f][0-9a-f]* [0-9][0-9]* 2/s/ .*//p" blame_raw >actual &&
git rev-parse X >expect &&
test_cmp expect actual
'
@@ t/t8013-blame-ignore-revs.sh: do
git blame --line-porcelain --ignore-rev $I file >blame_raw &&
- grep -E "^[0-9a-f]+ [0-9]+ 1" blame_raw | sed -e "s/ .*//" >actual &&
-+ sed -Ene "/^[0-9a-f]+ [0-9]+ 1/s/ .*//p" blame_raw >actual &&
++ sed -ne "/^[0-9a-f][0-9a-f]* [0-9][0-9]* 1/s/ .*//p" blame_raw >actual &&
git rev-parse A >expect &&
test_cmp expect actual &&
- grep -E "^[0-9a-f]+ [0-9]+ 2" blame_raw | sed -e "s/ .*//" >actual &&
-+ sed -Ene "/^[0-9a-f]+ [0-9]+ 2/s/ .*//p" blame_raw >actual &&
++ sed -ne "/^[0-9a-f][0-9a-f]* [0-9][0-9]* 2/s/ .*//p" blame_raw >actual &&
git rev-parse B >expect &&
test_cmp expect actual
'
@@ t/t8013-blame-ignore-revs.sh: test_expect_success ignore_rev_adding_unblamable_l
git blame --line-porcelain file --ignore-rev Y >blame_raw &&
- grep -E "^[0-9a-f]+ [0-9]+ 3" blame_raw | sed -e "s/ .*//" >actual &&
-+ sed -Ene "/^[0-9a-f]+ [0-9]+ 3/s/ .*//p" blame_raw >actual &&
++ sed -ne "/^[0-9a-f][0-9a-f]* [0-9][0-9]* 3/s/ .*//p" blame_raw >actual &&
test_cmp expect actual &&
- grep -E "^[0-9a-f]+ [0-9]+ 4" blame_raw | sed -e "s/ .*//" >actual &&
-+ sed -Ene "/^[0-9a-f]+ [0-9]+ 4/s/ .*//p" blame_raw >actual &&
++ sed -ne "/^[0-9a-f][0-9a-f]* [0-9][0-9]* 4/s/ .*//p" blame_raw >actual &&
test_cmp expect actual
'
@@ t/t8013-blame-ignore-revs.sh: test_expect_success ignore_revs_from_files '
git blame --line-porcelain file --ignore-revs-file ignore_x --ignore-revs-file ignore_y >blame_raw &&
- grep -E "^[0-9a-f]+ [0-9]+ 1" blame_raw | sed -e "s/ .*//" >actual &&
-+ sed -Ene "/^[0-9a-f]+ [0-9]+ 1/s/ .*//p" blame_raw >actual &&
++ sed -ne "/^[0-9a-f][0-9a-f]* [0-9][0-9]* 1/s/ .*//p" blame_raw >actual &&
git rev-parse A >expect &&
test_cmp expect actual &&
- grep -E "^[0-9a-f]+ [0-9]+ 2" blame_raw | sed -e "s/ .*//" >actual &&
-+ sed -Ene "/^[0-9a-f]+ [0-9]+ 2/s/ .*//p" blame_raw >actual &&
++ sed -ne "/^[0-9a-f][0-9a-f]* [0-9][0-9]* 2/s/ .*//p" blame_raw >actual &&
git rev-parse B >expect &&
test_cmp expect actual
'
@@ t/t8013-blame-ignore-revs.sh: test_expect_success ignore_revs_from_configs_and_f
git blame --line-porcelain file --ignore-revs-file ignore_y >blame_raw &&
- grep -E "^[0-9a-f]+ [0-9]+ 1" blame_raw | sed -e "s/ .*//" >actual &&
-+ sed -Ene "/^[0-9a-f]+ [0-9]+ 1/s/ .*//p" blame_raw >actual &&
++ sed -ne "/^[0-9a-f][0-9a-f]* [0-9][0-9]* 1/s/ .*//p" blame_raw >actual &&
git rev-parse A >expect &&
test_cmp expect actual &&
- grep -E "^[0-9a-f]+ [0-9]+ 2" blame_raw | sed -e "s/ .*//" >actual &&
-+ sed -Ene "/^[0-9a-f]+ [0-9]+ 2/s/ .*//p" blame_raw >actual &&
++ sed -ne "/^[0-9a-f][0-9a-f]* [0-9][0-9]* 2/s/ .*//p" blame_raw >actual &&
git rev-parse B >expect &&
test_cmp expect actual
'
@@ t/t8013-blame-ignore-revs.sh: test_expect_success override_ignore_revs_file '
git rev-parse X >expect &&
- grep -E "^[0-9a-f]+ [0-9]+ 1" blame_raw | sed -e "s/ .*//" >actual &&
-+ sed -Ene "/^[0-9a-f]+ [0-9]+ 1/s/ .*//p" blame_raw >actual &&
++ sed -ne "/^[0-9a-f][0-9a-f]* [0-9][0-9]* 1/s/ .*//p" blame_raw >actual &&
test_cmp expect actual &&
- grep -E "^[0-9a-f]+ [0-9]+ 2" blame_raw | sed -e "s/ .*//" >actual &&
-+ sed -Ene "/^[0-9a-f]+ [0-9]+ 2/s/ .*//p" blame_raw >actual &&
++ sed -ne "/^[0-9a-f][0-9a-f]* [0-9][0-9]* 2/s/ .*//p" blame_raw >actual &&
test_cmp expect actual
'
test_expect_success bad_files_and_revs '
@@ t/t8013-blame-ignore-revs.sh: test_expect_success ignore_merge '
git blame --line-porcelain file --ignore-rev M >blame_raw &&
- grep -E "^[0-9a-f]+ [0-9]+ 1" blame_raw | sed -e "s/ .*//" >actual &&
-+ sed -Ene "/^[0-9a-f]+ [0-9]+ 1/s/ .*//p" blame_raw >actual &&
++ sed -ne "/^[0-9a-f][0-9a-f]* [0-9][0-9]* 1/s/ .*//p" blame_raw >actual &&
git rev-parse B >expect &&
test_cmp expect actual &&
- grep -E "^[0-9a-f]+ [0-9]+ 9" blame_raw | sed -e "s/ .*//" >actual &&
-+ sed -Ene "/^[0-9a-f]+ [0-9]+ 9/s/ .*//p" blame_raw >actual &&
++ sed -ne "/^[0-9a-f][0-9a-f]* [0-9][0-9]* 9/s/ .*//p" blame_raw >actual &&
git rev-parse C >expect &&
test_cmp expect actual
'
20: 6b3345667040 = 20: 4772acb80163 t/t9*: merge "grep | sed" pipelines
21: 1784d5f3fdc1 = 21: a34653f91589 contrib/coverage-diff: avoid redundant pipelines
22: 4858de8ce44f = 22: 6c62dee0daa1 git-quiltimport: avoid an unnecessary subshell--
2.44.0
From: Beat Bolli <dev+git@drbeat.li>
*** BLURB HERE ***
Beat Bolli (22):
doc: avoid redundant use of cat
contrib/subtree/t: avoid redundant use of cat
t/lib-cvs.sh: avoid redundant use of cat
t/annotate-tests.sh: avoid redundant use of cat
t/perf: avoid redundant use of cat
t/t0*: avoid redundant uses of cat
t/t1*: avoid redundant uses of cat
t/t3*: avoid redundant uses of cat
t/t4*: avoid redundant uses of cat
t/t5*: avoid redundant uses of cat
t/t6*: avoid redundant uses of cat
t/t7*: avoid redundant use of cat
t/t8*: avoid redundant use of cat
t/t9*: avoid redundant uses of cat
t/t1*: merge a "grep | sed" pipeline
t/t3*: merge a "grep | awk" pipeline
t/t4*: merge a "grep | sed" pipeline
t/t5*: merge a "grep | sed" pipeline
t/t8*: merge "grep | sed" pipelines
t/t9*: merge "grep | sed" pipelines
contrib/coverage-diff: avoid redundant pipelines
git-quiltimport: avoid an unnecessary subshell
Documentation/howto/update-hook-example.txt | 4 +--
contrib/coverage-diff.sh | 9 ++----
contrib/subtree/t/t7900-subtree.sh | 2 +-
git-quiltimport.sh | 2 +-
t/annotate-tests.sh | 2 +-
t/lib-cvs.sh | 4 +--
t/perf/repos/inflate-repo.sh | 2 +-
t/t0002-gitfile.sh | 2 +-
t/t0011-hashmap.sh | 2 +-
t/t0028-working-tree-encoding.sh | 4 +--
t/t0204-gettext-reencode-sanity.sh | 2 +-
t/t1007-hash-object.sh | 6 ++--
t/t1091-sparse-checkout-builtin.sh | 2 +-
t/t1509/prepare-chroot.sh | 2 +-
t/t3200-branch.sh | 2 +-
t/t3321-notes-stripspace.sh | 8 ++---
t/t3920-crlf-messages.sh | 4 +--
t/t4002-diff-basic.sh | 2 +-
t/t4020-diff-external.sh | 2 +-
t/t4205-log-pretty-formats.sh | 2 +-
t/t4301-merge-tree-write-tree.sh | 8 ++---
t/t5100-mailinfo.sh | 2 +-
t/t5317-pack-objects-filter-objects.sh | 2 +-
t/t5401-update-hooks.sh | 2 +-
t/t5534-push-signed.sh | 2 +-
t/t6112-rev-list-filters-objects.sh | 2 +-
t/t6413-merge-crlf.sh | 4 +--
t/t7704-repack-cruft.sh | 2 +-
t/t8010-cat-file-filters.sh | 2 +-
t/t8013-blame-ignore-revs.sh | 28 ++++++++---------
t/t9118-git-svn-funky-branch-names.sh | 2 +-
t/t9300-fast-import.sh | 14 +++------
t/t9350-fast-export.sh | 4 +--
t/t9400-git-cvsserver-server.sh | 35 ++++++++++-----------
t/t9802-git-p4-filetype.sh | 2 +-
t/t9807-git-p4-submit.sh | 2 +-
t/t9824-git-p4-git-lfs.sh | 4 +--
37 files changed, 85 insertions(+), 97 deletions(-)
--
2.44.0
