git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [RFC PATCH 2/5] t3427: accelerate this test by using fast-export and fast-import

From
Derrick Stolee <stolee@gmail.com>
Date
Aug 27, 2019, 01:25 UTC
Message-ID
<ad7446ac-34cc-b566-302b-053bf5dafc63@gmail.com>
In-Reply-To
<20190826235226.15386-3-newren@gmail.com>
On 8/26/2019 7:52 PM, Elijah Newren wrote:
Show 9 quoted lines
> fast-export and fast-import can easily handle the simple rewrite that
> was being done by filter-branch, and should be significantly faster on
> systems with a slow fork.  Timings from before and after on two laptops
> that I have access to (measured via `time ./t3427-rebase-subtree.sh`,
> i.e. including everything in this test -- not just the filter-branch or
> fast-export/fast-import pair):
> 
>    Linux:  4.305s -> 3.684s (~17% speedup)
>    Mac:   10.128s -> 7.038s (~30% speedup)
Again, impressive speedup!
Show 18 quoted lines
> Signed-off-by: Elijah Newren <newren@gmail.com>
> ---
>  t/t3427-rebase-subtree.sh | 32 ++++++++++++++++++++++++--------
>  1 file changed, 24 insertions(+), 8 deletions(-)
> 
> diff --git a/t/t3427-rebase-subtree.sh b/t/t3427-rebase-subtree.sh
> index d8640522a0..d05fcce5dc 100755
> --- a/t/t3427-rebase-subtree.sh
> +++ b/t/t3427-rebase-subtree.sh
> @@ -42,7 +42,9 @@ test_expect_failure REBASE_P \
>  	'Rebase -Xsubtree --preserve-merges --onto commit 4' '
>  	reset_rebase &&
>  	git checkout -b rebase-preserve-merges-4 master &&
> -	git filter-branch --prune-empty -f --subdirectory-filter files_subtree &&
> +	git fast-export --no-data HEAD -- files_subtree/ \
> +		| sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" \
> +		| git fast-import --force --quiet &&
>  	git commit -m "Empty commit" --allow-empty &&

There's a lot of repetition in these changes. Is this a good time to introduce a helper method? This trio of commands happens to be difficult to parse, so I'd rather do it just once.

Show 80 quoted lines
>  	git rebase -Xsubtree=files_subtree --preserve-merges --onto files-master master &&
>  	verbose test "$(commit_message HEAD~)" = "files_subtree/master4"
> @@ -53,7 +55,9 @@ test_expect_failure REBASE_P \
>  	'Rebase -Xsubtree --preserve-merges --onto commit 5' '
>  	reset_rebase &&
>  	git checkout -b rebase-preserve-merges-5 master &&
> -	git filter-branch --prune-empty -f --subdirectory-filter files_subtree &&
> +	git fast-export --no-data HEAD -- files_subtree/ \
> +		| sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" \
> +		| git fast-import --force --quiet &&
>  	git commit -m "Empty commit" --allow-empty &&
>  	git rebase -Xsubtree=files_subtree --preserve-merges --onto files-master master &&
>  	verbose test "$(commit_message HEAD)" = "files_subtree/master5"
> @@ -64,7 +68,9 @@ test_expect_failure REBASE_P \
>  	'Rebase -Xsubtree --keep-empty --preserve-merges --onto commit 4' '
>  	reset_rebase &&
>  	git checkout -b rebase-keep-empty-4 master &&
> -	git filter-branch --prune-empty -f --subdirectory-filter files_subtree &&
> +	git fast-export --no-data HEAD -- files_subtree/ \
> +		| sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" \
> +		| git fast-import --force --quiet &&
>  	git commit -m "Empty commit" --allow-empty &&
>  	git rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master &&
>  	verbose test "$(commit_message HEAD~2)" = "files_subtree/master4"
> @@ -75,7 +81,9 @@ test_expect_failure REBASE_P \
>  	'Rebase -Xsubtree --keep-empty --preserve-merges --onto commit 5' '
>  	reset_rebase &&
>  	git checkout -b rebase-keep-empty-5 master &&
> -	git filter-branch --prune-empty -f --subdirectory-filter files_subtree &&
> +	git fast-export --no-data HEAD -- files_subtree/ \
> +		| sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" \
> +		| git fast-import --force --quiet &&
>  	git commit -m "Empty commit" --allow-empty &&
>  	git rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master &&
>  	verbose test "$(commit_message HEAD~)" = "files_subtree/master5"
> @@ -86,7 +94,9 @@ test_expect_failure REBASE_P \
>  	'Rebase -Xsubtree --keep-empty --preserve-merges --onto empty commit' '
>  	reset_rebase &&
>  	git checkout -b rebase-keep-empty-empty master &&
> -	git filter-branch --prune-empty -f --subdirectory-filter files_subtree &&
> +	git fast-export --no-data HEAD -- files_subtree/ \
> +		| sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" \
> +		| git fast-import --force --quiet &&
>  	git commit -m "Empty commit" --allow-empty &&
>  	git rebase -Xsubtree=files_subtree --keep-empty --preserve-merges --onto files-master master &&
>  	verbose test "$(commit_message HEAD)" = "Empty commit"
> @@ -96,7 +106,9 @@ test_expect_failure REBASE_P \
>  test_expect_failure 'Rebase -Xsubtree --onto commit 4' '
>  	reset_rebase &&
>  	git checkout -b rebase-onto-4 master &&
> -	git filter-branch --prune-empty -f --subdirectory-filter files_subtree &&
> +	git fast-export --no-data HEAD -- files_subtree/ \
> +		| sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" \
> +		| git fast-import --force --quiet &&
>  	git commit -m "Empty commit" --allow-empty &&
>  	git rebase -Xsubtree=files_subtree --onto files-master master &&
>  	verbose test "$(commit_message HEAD~2)" = "files_subtree/master4"
> @@ -106,7 +118,9 @@ test_expect_failure 'Rebase -Xsubtree --onto commit 4' '
>  test_expect_failure 'Rebase -Xsubtree --onto commit 5' '
>  	reset_rebase &&
>  	git checkout -b rebase-onto-5 master &&
> -	git filter-branch --prune-empty -f --subdirectory-filter files_subtree &&
> +	git fast-export --no-data HEAD -- files_subtree/ \
> +		| sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" \
> +		| git fast-import --force --quiet &&
>  	git commit -m "Empty commit" --allow-empty &&
>  	git rebase -Xsubtree=files_subtree --onto files-master master &&
>  	verbose test "$(commit_message HEAD~)" = "files_subtree/master5"
> @@ -115,7 +129,9 @@ test_expect_failure 'Rebase -Xsubtree --onto commit 5' '
>  test_expect_failure 'Rebase -Xsubtree --onto empty commit' '
>  	reset_rebase &&
>  	git checkout -b rebase-onto-empty master &&
> -	git filter-branch --prune-empty -f --subdirectory-filter files_subtree &&
> +	git fast-export --no-data HEAD -- files_subtree/ \
> +		| sed -e "s%\([0-9a-f]\{40\} \)files_subtree/%\1%" \
> +		| git fast-import --force --quiet &&
>  	git commit -m "Empty commit" --allow-empty &&
>  	git rebase -Xsubtree=files_subtree --onto files-master master &&
>  	verbose test "$(commit_message HEAD)" = "Empty commit"
>
Previous: Elijah NewrenNext: Elijah Newren
Message 9 of 73 in “RFC: Proposing git-filter-repo for inclusion in git.git”
  1. Elijah NewrenAug 22, 2019
  2. Junio C HamanoAug 22, 2019
  3. Elijah NewrenAug 22, 2019
  4. Junio C HamanoAug 22, 2019
  5. 0/5 Remove git-filter-branch from git.git; host it elsewhereElijah Newren, Aug 26, 2019
  6. 1/5 t6006: simplify and optimize empty message testElijah Newren, Aug 26, 2019
  7. Derrick StoleeAug 27, 2019
  8. 2/5 t3427: accelerate this test by using fast-export and fast-importElijah Newren, Aug 26, 2019
  9. Derrick StoleeAug 27, 2019
  10. 3/5 git-sh-i18n: work with external scriptsElijah Newren, Aug 26, 2019
  11. Derrick StoleeAug 27, 2019
  12. 4/5 Recommend git-filter-repo instead of git-filter-branch in documentationElijah Newren, Aug 26, 2019
  13. Derrick StoleeAug 27, 2019
  14. Elijah NewrenAug 27, 2019
  15. 5/5 Remove git-filter-branch, it is now external to git.gitElijah Newren, Aug 26, 2019
  16. Derrick StoleeAug 27, 2019
  17. Elijah NewrenAug 27, 2019
  18. Eric WongAug 27, 2019
  19. Sergey OrganovAug 27, 2019
  20. Elijah NewrenAug 27, 2019
  21. Sergey OrganovAug 28, 2019
  22. Elijah NewrenAug 28, 2019
  23. Sergey OrganovAug 28, 2019
  24. Johannes SchindelinAug 30, 2019
  25. Elijah NewrenAug 30, 2019
  26. Johannes SchindelinSep 2, 2019
  27. Elijah NewrenSep 3, 2019
  28. 0/4 Warn about git-filter-branch usage and avoid itElijah Newren, Aug 28, 2019
  29. 1/4 t6006: simplify and optimize empty message testElijah Newren, Aug 28, 2019
  30. 2/4 t3427: accelerate this test by using fast-export and fast-importElijah Newren, Aug 28, 2019
  31. Eric SunshineAug 28, 2019
  32. 3/4 Recommend git-filter-repo instead of git-filter-branchElijah Newren, Aug 28, 2019
  33. Eric SunshineAug 28, 2019
  34. Elijah NewrenAug 28, 2019
  35. 4/4 Remove git-filter-branch, it is now external to git.gitElijah Newren, Aug 28, 2019
  36. 0/4 Warn about git-filter-branch usage and avoid itElijah Newren, Aug 29, 2019
  37. 1/4 t6006: simplify and optimize empty message testElijah Newren, Aug 29, 2019
  38. 2/4 t3427: accelerate this test by using fast-export and fast-importElijah Newren, Aug 29, 2019
  39. 4/4 t9902: use a non-deprecated command for testingElijah Newren, Aug 29, 2019
  40. 3/4 Recommend git-filter-repo instead of git-filter-branchElijah Newren, Aug 29, 2019
  41. Eric SunshineAug 29, 2019
  42. Elijah NewrenAug 30, 2019
  43. 0/4 Warn about git-filter-branch usage and avoid itElijah Newren, Aug 30, 2019
  44. 1/4 t6006: simplify and optimize empty message testElijah Newren, Aug 30, 2019
  45. Johannes SchindelinSep 2, 2019
  46. 2/4 t3427: accelerate this test by using fast-export and fast-importElijah Newren, Aug 30, 2019
  47. Johannes SchindelinSep 2, 2019
  48. 4/4 t9902: use a non-deprecated command for testingElijah Newren, Aug 30, 2019
  49. 3/4 Recommend git-filter-repo instead of git-filter-branchElijah Newren, Aug 30, 2019
  50. 0/4 Warn about git-filter-branch usage and avoid itElijah Newren, Sep 3, 2019
  51. 2/4 t3427: accelerate this test by using fast-export and fast-importElijah Newren, Sep 3, 2019
  52. Junio C HamanoSep 3, 2019
  53. Junio C HamanoSep 3, 2019
  54. Elijah NewrenSep 4, 2019
  55. 1/4 t6006: simplify and optimize empty message testElijah Newren, Sep 3, 2019
  56. Junio C HamanoSep 3, 2019
  57. Elijah NewrenSep 3, 2019
  58. Junio C HamanoSep 3, 2019
  59. 3/4 Recommend git-filter-repo instead of git-filter-branchElijah Newren, Sep 3, 2019
  60. Junio C HamanoSep 3, 2019
  61. Elijah NewrenSep 4, 2019
  62. 4/4 t9902: use a non-deprecated command for testingElijah Newren, Sep 3, 2019
  63. 0/3 Warn about git-filter-branch usage and avoid itElijah Newren, Sep 4, 2019
  64. 1/3 t6006: simplify, fix, and optimize empty message testElijah Newren, Sep 4, 2019
  65. 2/3 Recommend git-filter-repo instead of git-filter-branchElijah Newren, Sep 4, 2019
  66. 3/3 t9902: use a non-deprecated command for testingElijah Newren, Sep 4, 2019
  67. Eric WongAug 23, 2019
  68. Elijah NewrenAug 23, 2019
  69. Elijah NewrenAug 23, 2019
  70. Johannes SchindelinAug 28, 2019
  71. Junio C HamanoAug 28, 2019
  72. Derrick StoleeAug 23, 2019
  73. Jeff KingAug 26, 2019

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.