git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v5 0/4] Warn about git-filter-branch usage and avoid it

From
Elijah Newren <newren@gmail.com>
Date
Sep 3, 2019, 18:55 UTC
Message-ID
<20190903185524.13467-1-newren@gmail.com>
In-Reply-To
<20190828002210.8862-1-newren@gmail.com>

It's been about 5 days with no further feedback, other than some timings from Dscho for Windows showing that my fixes help there too. So, I did one last re-read, made a couple small wording tweaks, and am resending as ready for inclusion.

Changes since v4:
  * Included the windows timings from Dscho in the commit messages for
    the first two perf patches
  * A few slight wording tweaks to the manpage
Elijah Newren (4):
  t6006: simplify and optimize empty message test
  t3427: accelerate this test by using fast-export and fast-import
  Recommend git-filter-repo instead of git-filter-branch
  t9902: use a non-deprecated command for testing
 Documentation/git-fast-export.txt   |   6 +-
 Documentation/git-filter-branch.txt | 273 +++++++++++++++++++++++++---
 Documentation/git-gc.txt            |  17 +-
 Documentation/git-rebase.txt        |   3 +-
 Documentation/git-replace.txt       |  10 +-
 Documentation/git-svn.txt           |  10 +-
 Documentation/githooks.txt          |  10 +-
 contrib/svn-fe/svn-fe.txt           |   4 +-
 git-filter-branch.sh                |  13 ++
 t/t3427-rebase-subtree.sh           |  24 ++-
 t/t6006-rev-list-format.sh          |   5 +-
 t/t9902-completion.sh               |  12 +-
 12 files changed, 310 insertions(+), 77 deletions(-)
Range-diff:
1:  7ddbeea2ca ! 1:  ccea0e5846 t6006: simplify and optimize empty message test
    @@ Commit message
         Despite only being one piece of the 71st test and there being 73 tests
         overall, this small change to just this one test speeds up the overall
         execution time of t6006 (as measured by the best of 3 runs of `time
    -    ./t6006-rev-list-format.sh`) by about 11% on Linux and by 13% on
    -    Mac.
    +    ./t6006-rev-list-format.sh`) by about 11% on Linux, 13% on Mac, and
    +    about 15% on Windows.
     
         Signed-off-by: Elijah Newren <newren@gmail.com>
     
2:  e1e63189c1 ! 2:  6d73135006 t3427: accelerate this test by using fast-export and fast-import
    @@ Commit message
     
         fast-export and fast-import can easily handle the simple rewrite that
         was being done by filter-branch, and should be significantly faster on
    -    systems with a slow fork.  Timings from before and after on two laptops
    -    that I have access to (measured via `time ./t3427-rebase-subtree.sh`,
    -    i.e. including everything in this test -- not just the filter-branch or
    -    fast-export/fast-import pair):
    +    systems with a slow fork.  Timings from before and after on a few
    +    laptops that I or others measured on (measured via `time
    +    ./t3427-rebase-subtree.sh`, i.e. including everything in this test --
    +    not just the filter-branch or fast-export/fast-import pair):
     
    -       Linux:  4.305s -> 3.684s (~17% speedup)
    -       Mac:   10.128s -> 7.038s (~30% speedup)
    +       Linux:    4.305s -> 3.684s (~17% speedup)
    +       Mac:     10.128s -> 7.038s (~30% speedup)
    +       Windows:  1m 37s -> 1m 17s (~26% speedup)
     
         Signed-off-by: Elijah Newren <newren@gmail.com>
     
3:  ed6505584f ! 3:  2f225c8697 Recommend git-filter-repo instead of git-filter-branch
    @@ Documentation/git-filter-branch.txt: warned.
     +document or provide to a coworker, who then runs them on a different OS
     +where the same commands are not working/tested (some examples in the
     +git-filter-branch manpage are also affected by this).  BSD vs. GNU
    -+userland differences can really bite.  If you're lucky, you get ugly
    -+error messages spewed.  But just as likely, the commands either don't do
    -+the filtering requested, or silently corrupt making some unwanted
    -+change.  The unwanted change may only affect a few commits, so it's not
    -+necessarily obvious either.  (The fact that problems won't necessarily
    -+be obvious means they are likely to go unnoticed until the rewritten
    -+history is in use for quite a while, at which point it's really hard to
    -+justify another flag-day for another rewrite.)
    ++userland differences can really bite.  If lucky, error messages are
    ++spewed.  But just as likely, the commands either don't do the filtering
    ++requested, or silently corrupt by making some unwanted change.  The
    ++unwanted change may only affect a few commits, so it's not necessarily
    ++obvious either.  (The fact that problems won't necessarily be obvious
    ++means they are likely to go unnoticed until the rewritten history is in
    ++use for quite a while, at which point it's really hard to justify
    ++another flag-day for another rewrite.)
     +
     +* Filenames with spaces are often mishandled by shell snippets since
     +they cause problems for shell pipelines.  Not everyone is familiar with
     +find -print0, xargs -0, git-ls-files -z, etc.  Even people who are
    -+familiar with these may assume such needs are not relevant because
    ++familiar with these may assume such flags are not relevant because
     +someone else renamed any such files in their repo back before the person
    -+doing the filtering joined the project.  And, often, even those familiar
    -+with handling arguments with spaces my not do so just because they
    ++doing the filtering joined the project.  And often, even those familiar
    ++with handling arguments with spaces may not do so just because they
     +aren't in the mindset of thinking about everything that could possibly
     +go wrong.
     +
     +* Non-ascii filenames can be silently removed despite being in a desired
    -+directory.  The desire to select paths to keep often use pipelines like
    ++directory.  Keeping only wanted paths is often done using pipelines like
     +`git ls-files | grep -v ^WANTED_DIR/ | xargs git rm`.  ls-files will
    -+only quote filenames if needed so folks may not notice that one of the
    -+files didn't match the regex, again until it's much too late.  Yes,
    -+someone who knows about core.quotePath can avoid this (unless they have
    -+other special characters like \t, \n, or "), and people who use ls-files
    -+-z with something other than grep can avoid this, but that doesn't mean
    -+they will.
    ++only quote filenames if needed, so folks may not notice that one of the
    ++files didn't match the regex (at least not until it's much too late).
    ++Yes, someone who knows about core.quotePath can avoid this (unless they
    ++have other special characters like \t, \n, or "), and people who use
    ++ls-files -z with something other than grep can avoid this, but that
    ++doesn't mean they will.
     +
     +* Similarly, when moving files around, one can find that filenames with
     +non-ascii or special characters end up in a different directory, one
    @@ Documentation/git-filter-branch.txt: warned.
     +the same name, no warning or error is provided; git-filter-branch simply
     +overwrites each tag in some undocumented pre-defined order resulting in
     +only one tag at the end.  (A git-filter-branch regression test requires
    -+this.)
    ++this surprising behavior.)
     +
    -+Also, the poor performance of git-filter-branch often leads to safety issues:
    ++Also, the poor performance of git-filter-branch often leads to safety
    ++issues:
     +
     +* Coming up with the correct shell snippet to do the filtering you want
     +is sometimes difficult unless you're just doing a trivial modification
4:  ca8e124cb3 = 4:  048eba375b t9902: use a non-deprecated command for testing
-- 
2.23.0.39.gf92d9de5c3
Previous: Elijah NewrenNext: Elijah Newren
Message 50 of 73 in “RFC: Proposing git-filter-repo for inclusion in git.git”
  1. Elijah NewrenAug 22, 2019
  2. Junio C HamanoAug 22, 2019
  3. Elijah NewrenAug 22, 2019
  4. Junio C HamanoAug 22, 2019
  5. 0/5 Remove git-filter-branch from git.git; host it elsewhereElijah Newren, Aug 26, 2019
  6. 1/5 t6006: simplify and optimize empty message testElijah Newren, Aug 26, 2019
  7. Derrick StoleeAug 27, 2019
  8. 2/5 t3427: accelerate this test by using fast-export and fast-importElijah Newren, Aug 26, 2019
  9. Derrick StoleeAug 27, 2019
  10. 3/5 git-sh-i18n: work with external scriptsElijah Newren, Aug 26, 2019
  11. Derrick StoleeAug 27, 2019
  12. 4/5 Recommend git-filter-repo instead of git-filter-branch in documentationElijah Newren, Aug 26, 2019
  13. Derrick StoleeAug 27, 2019
  14. Elijah NewrenAug 27, 2019
  15. 5/5 Remove git-filter-branch, it is now external to git.gitElijah Newren, Aug 26, 2019
  16. Derrick StoleeAug 27, 2019
  17. Elijah NewrenAug 27, 2019
  18. Eric WongAug 27, 2019
  19. Sergey OrganovAug 27, 2019
  20. Elijah NewrenAug 27, 2019
  21. Sergey OrganovAug 28, 2019
  22. Elijah NewrenAug 28, 2019
  23. Sergey OrganovAug 28, 2019
  24. Johannes SchindelinAug 30, 2019
  25. Elijah NewrenAug 30, 2019
  26. Johannes SchindelinSep 2, 2019
  27. Elijah NewrenSep 3, 2019
  28. 0/4 Warn about git-filter-branch usage and avoid itElijah Newren, Aug 28, 2019
  29. 1/4 t6006: simplify and optimize empty message testElijah Newren, Aug 28, 2019
  30. 2/4 t3427: accelerate this test by using fast-export and fast-importElijah Newren, Aug 28, 2019
  31. Eric SunshineAug 28, 2019
  32. 3/4 Recommend git-filter-repo instead of git-filter-branchElijah Newren, Aug 28, 2019
  33. Eric SunshineAug 28, 2019
  34. Elijah NewrenAug 28, 2019
  35. 4/4 Remove git-filter-branch, it is now external to git.gitElijah Newren, Aug 28, 2019
  36. 0/4 Warn about git-filter-branch usage and avoid itElijah Newren, Aug 29, 2019
  37. 1/4 t6006: simplify and optimize empty message testElijah Newren, Aug 29, 2019
  38. 2/4 t3427: accelerate this test by using fast-export and fast-importElijah Newren, Aug 29, 2019
  39. 4/4 t9902: use a non-deprecated command for testingElijah Newren, Aug 29, 2019
  40. 3/4 Recommend git-filter-repo instead of git-filter-branchElijah Newren, Aug 29, 2019
  41. Eric SunshineAug 29, 2019
  42. Elijah NewrenAug 30, 2019
  43. 0/4 Warn about git-filter-branch usage and avoid itElijah Newren, Aug 30, 2019
  44. 1/4 t6006: simplify and optimize empty message testElijah Newren, Aug 30, 2019
  45. Johannes SchindelinSep 2, 2019
  46. 2/4 t3427: accelerate this test by using fast-export and fast-importElijah Newren, Aug 30, 2019
  47. Johannes SchindelinSep 2, 2019
  48. 4/4 t9902: use a non-deprecated command for testingElijah Newren, Aug 30, 2019
  49. 3/4 Recommend git-filter-repo instead of git-filter-branchElijah Newren, Aug 30, 2019
  50. 0/4 Warn about git-filter-branch usage and avoid itElijah Newren, Sep 3, 2019
  51. 2/4 t3427: accelerate this test by using fast-export and fast-importElijah Newren, Sep 3, 2019
  52. Junio C HamanoSep 3, 2019
  53. Junio C HamanoSep 3, 2019
  54. Elijah NewrenSep 4, 2019
  55. 1/4 t6006: simplify and optimize empty message testElijah Newren, Sep 3, 2019
  56. Junio C HamanoSep 3, 2019
  57. Elijah NewrenSep 3, 2019
  58. Junio C HamanoSep 3, 2019
  59. 3/4 Recommend git-filter-repo instead of git-filter-branchElijah Newren, Sep 3, 2019
  60. Junio C HamanoSep 3, 2019
  61. Elijah NewrenSep 4, 2019
  62. 4/4 t9902: use a non-deprecated command for testingElijah Newren, Sep 3, 2019
  63. 0/3 Warn about git-filter-branch usage and avoid itElijah Newren, Sep 4, 2019
  64. 1/3 t6006: simplify, fix, and optimize empty message testElijah Newren, Sep 4, 2019
  65. 2/3 Recommend git-filter-repo instead of git-filter-branchElijah Newren, Sep 4, 2019
  66. 3/3 t9902: use a non-deprecated command for testingElijah Newren, Sep 4, 2019
  67. Eric WongAug 23, 2019
  68. Elijah NewrenAug 23, 2019
  69. Elijah NewrenAug 23, 2019
  70. Johannes SchindelinAug 28, 2019
  71. Junio C HamanoAug 28, 2019
  72. Derrick StoleeAug 23, 2019
  73. Jeff KingAug 26, 2019

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.