git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [RFC PATCH 0/5] Remove git-filter-branch from git.git; host it elsewhere

From
Johannes Schindelin <johannes.schindelin@gmx.de>
Date
Sep 2, 2019, 09:29 UTC
Message-ID
<nycvar.QRO.7.76.6.1909021111440.46@tvgsbejvaqbjf.bet>
In-Reply-To
<CABPp-BHMXAQGPaBYyg2dtVeN5h8fW8G4YdhddCeAjY5r74BAzw@mail.gmail.com>
Hi Elijah,
On Fri, 30 Aug 2019, Elijah Newren wrote:
Show 29 quoted lines
> On Fri, Aug 30, 2019 at 1:40 PM Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
>
> > [...]
> > In my most recent instance of this, I wanted to publish the script I
> > used to use for submitting patch series to the Git mailing list,
> > maintaining tags for iterations and generating cover letters from branch
> > descriptions and interdiffs (this script eventually became GitGitGadget,
> > https://github.com/gitgitgadget/gitgitgadget/commits?after=6fb0ede48f86e729292ee1542729bc0f5a30cfa6+0
> > demonstrates this).
> >
> > To do that, I ran a `git filter-branch` in the repository where I track
> > all the scripts I deem unsuitable for public consumption, to remove all
> > files but `mail-patch-series.sh`, then pushed it to
> > https://github.com/dscho/mail-patch-series
> >
> > Please note that most crucially, I wanted to rewrite a newly-created
> > branch, and only that branch.
> >
> > Could I have done the same using `git fast-export`, filtering the output
> > with a Perl script, then passing it to `git fast-import`? Sure, I was
> > really tempted to do that. In the end, it took less of _my_ time to just
> > let `git filter-branch` do its work with a not-too-complicated index
> > filter.
>
> Why a perl script?  Shouldn't
>     git fast-export [--no-data] HEAD -- $PATH | git fast-import --force --quiet
> do the trick?  And it's probably simpler and shorter than the index
> filter you used.

Does that not keep the full `$PATH`? I wanted the resulting branch to have the file in the top-level directory.

Show 13 quoted lines
> That said, yeah it'd be nice to get automatic rewriting of commit
> hashes in commit messages and other niceties from filter-repo (e.g.
> future automatic reattaching of notes to the rewritten commits).  Some
> questions:
>
>   * What's the backup strategy in case you specify the wrong filters
> (e.g. you have a typo in the pathnames)?  filter-repo encourages folks
> to make a clone and then filter the fresh clone, because if anything
> goes awry, you can just delete and restart.  (I am heavily opposed to
> the refs/original/ backup mechanism used by filter-branch, for
> multiple reasons.)  Is your safety stance just "If I mess up it's my
> own fault; do the rewrite?"  Or are you okay with cloning before
> filtering?

Please note that the `refs/original/` refs should not have been written at all anymore, not after reflogs were introduced.

Incidentally, that is my answer to your question: the reflog is my backup.

Show 5 quoted lines
>   * If you're okay with cloning before filtering...then is there an
> issue with rewriting all branches, and just pushing the one you need?
> (Is there an issue with "this branch is small, the others are huge,
> and filter-branch is slow -- so rewriting one branch saves me lots of
> time"?  Or are there other issues at play too?)
I am not okay with cloning before filtering.
First of all, it is wasteful.

Second of all, in my case it would have been *particularly* wasteful because the repository in question also has quite a few quite large blobs (hysterical raisins, don't ask).

Show 5 quoted lines
>   * What if the user has auxiliary information for the branch in other
> refs?  For example, git-notes pointing at any of the commits, or tags
> in the history of the branch that might be relevant, or perhaps even
> replace refs in combination with GIT_NO_REPLACE_OBJECTS=1?  Is this an
> "I don't care, toss that stuff and just rewrite just this branch?"

In my case: there are no notes. The only time when I make heavy use of notes is in GitGitGadget. I don't use that feature otherwise.

Show 5 quoted lines
>   * filter-repo by default creates new replace references so that you
> can refer to new commit IDs using old (unabbreviated) commit IDs.
> Would that be considered helpful for this usecase?  unhelpful?
> irrelevant, since you'll just push the branch you want somewhere and
> nuke the temporary clone?

I definitely did not need that mapping in all of my `git filter-branch` use cases.

Of course, I can see how it can come in handy in other circumstances, just not in the ones I experienced so far.

Show 7 quoted lines
> I'm not by any means ruling out the possibility of documenting --refs
> and adjusting the defaults when it is used so the user can just run
> something like
>    git filter-repo --path $PATH --refs $MYBRANCH
> but I feel like I need to understand answers to questions like the
> above ones so that I can know how to phrase warnings and adjust
> defaults and update the documentation.

In all the scenarios where I used `git filter-branch` (some dozen per year, so not all *that* many), I needed to rewrite one particular branch, typically a freshly-created one. I never, ever ever needed to rewrite all the refs in the repository. Not once ;-)

Show 17 quoted lines
> > In another instance, a long, long time ago, I needed to restart a
> > repository which had included way too many files for its own good, then
> > rename the old repository and start with a fresh `master` that contained
> > but a single commit whose tree was identical to the previous `master`'s
> > tip commit. I simply grafted that commit, ran `git filter-branch` and
> > had precisely what I needed.
>
> filter-repo supports grafts and replace objects, the same as
> filter-branch.  (Although, technically, I didn't have to do a thing to
> support it; fast-export does the special handling of rewriting based
> on grafts and replace objects.)  So, I'd say this is fully supported.
>
> Side question: the git-replace documents suggest that the graft file
> is deprecated.  Are there any timeframes or plans for phasing out
> beyond the git-replace manpage existing?  Should I avoid documenting
> the graft file support in filter-repo?  Should I include examples
> using not just git-replace but also using the graft file?

I had meant to prepare a patch series to remove `grafts` support that Junio could carry in `pu` until the time he considers it appropriate to merge to `master`, but it seems that this task fell under the rag.

The deprecation itself has been introduced in tags/v2.18.0-rc0~54^2~4, i.e. it is official as of Git v2.18.0, which was released in mid-June last year.

My personal gut feeling is that we should let it simmer for another year before removing support for the `grafts` file (and we may want to update the label "grafted" when `git log` shows a shallow commit before we remove that support for `grafts`).

So I'll not work on that patch for now.
Show 18 quoted lines
> > I would be _delighted_ if these kinds of use case (rewriting a branch,
> > or even just a commit range) became more of a first-class citizen with
> > `git filter-repo`.
>
> I've got all the pieces for supporting a single branch or a commit
> range (e.g. 'git filter-repo --path foo --refs ^master~4 ^stable~23
> mybranch'), but the defaults (error out unless in a bare repo, move
> refs/remotes/origin/* to refs/heads/*, disconnect origin remote,
> expire reflogs & repack & prune, create new replace references so
> folks can access new commits using old commit IDs) may be somewhat
> friction-filled for this usecase.  Those defaults other than the new
> replace refs happen to all be turned off with the combination of
> --force and --target, so, assuming turning them off is what you need,
> you could cheat and just specify 'git filter-repo --force --target .
> --refs $MYBRANCH' today and perhaps get what you want, but that's a
> really non-intuitive command line that is way too ugly to recommend.
> And I don't want to tie myself to '--target .' being the magic sauce
> in the future either.

I agree. I would love for my use cases to become more of first-class citizens. Maybe `--branch <branch>` could serve as the knob?

What I also found really helpful in `git filter-branch` is that it was possible to pass one-liner shell scripts directly to the command, giving a lot of freedom about the transformations. I understand that Python makes it hard to write spaghetti-code one-liners, so you cannot really pass the snippet in via the command-line, but I hope there is a way to script things in `git filter-repo`?

Ciao, Dscho

Previous: Elijah NewrenNext: Elijah Newren
Message 26 of 73 in “RFC: Proposing git-filter-repo for inclusion in git.git”
  1. Elijah NewrenAug 22, 2019
  2. Junio C HamanoAug 22, 2019
  3. Elijah NewrenAug 22, 2019
  4. Junio C HamanoAug 22, 2019
  5. 0/5 Remove git-filter-branch from git.git; host it elsewhereElijah Newren, Aug 26, 2019
  6. 1/5 t6006: simplify and optimize empty message testElijah Newren, Aug 26, 2019
  7. Derrick StoleeAug 27, 2019
  8. 2/5 t3427: accelerate this test by using fast-export and fast-importElijah Newren, Aug 26, 2019
  9. Derrick StoleeAug 27, 2019
  10. 3/5 git-sh-i18n: work with external scriptsElijah Newren, Aug 26, 2019
  11. Derrick StoleeAug 27, 2019
  12. 4/5 Recommend git-filter-repo instead of git-filter-branch in documentationElijah Newren, Aug 26, 2019
  13. Derrick StoleeAug 27, 2019
  14. Elijah NewrenAug 27, 2019
  15. 5/5 Remove git-filter-branch, it is now external to git.gitElijah Newren, Aug 26, 2019
  16. Derrick StoleeAug 27, 2019
  17. Elijah NewrenAug 27, 2019
  18. Eric WongAug 27, 2019
  19. Sergey OrganovAug 27, 2019
  20. Elijah NewrenAug 27, 2019
  21. Sergey OrganovAug 28, 2019
  22. Elijah NewrenAug 28, 2019
  23. Sergey OrganovAug 28, 2019
  24. Johannes SchindelinAug 30, 2019
  25. Elijah NewrenAug 30, 2019
  26. Johannes SchindelinSep 2, 2019
  27. Elijah NewrenSep 3, 2019
  28. 0/4 Warn about git-filter-branch usage and avoid itElijah Newren, Aug 28, 2019
  29. 1/4 t6006: simplify and optimize empty message testElijah Newren, Aug 28, 2019
  30. 2/4 t3427: accelerate this test by using fast-export and fast-importElijah Newren, Aug 28, 2019
  31. Eric SunshineAug 28, 2019
  32. 3/4 Recommend git-filter-repo instead of git-filter-branchElijah Newren, Aug 28, 2019
  33. Eric SunshineAug 28, 2019
  34. Elijah NewrenAug 28, 2019
  35. 4/4 Remove git-filter-branch, it is now external to git.gitElijah Newren, Aug 28, 2019
  36. 0/4 Warn about git-filter-branch usage and avoid itElijah Newren, Aug 29, 2019
  37. 1/4 t6006: simplify and optimize empty message testElijah Newren, Aug 29, 2019
  38. 2/4 t3427: accelerate this test by using fast-export and fast-importElijah Newren, Aug 29, 2019
  39. 4/4 t9902: use a non-deprecated command for testingElijah Newren, Aug 29, 2019
  40. 3/4 Recommend git-filter-repo instead of git-filter-branchElijah Newren, Aug 29, 2019
  41. Eric SunshineAug 29, 2019
  42. Elijah NewrenAug 30, 2019
  43. 0/4 Warn about git-filter-branch usage and avoid itElijah Newren, Aug 30, 2019
  44. 1/4 t6006: simplify and optimize empty message testElijah Newren, Aug 30, 2019
  45. Johannes SchindelinSep 2, 2019
  46. 2/4 t3427: accelerate this test by using fast-export and fast-importElijah Newren, Aug 30, 2019
  47. Johannes SchindelinSep 2, 2019
  48. 4/4 t9902: use a non-deprecated command for testingElijah Newren, Aug 30, 2019
  49. 3/4 Recommend git-filter-repo instead of git-filter-branchElijah Newren, Aug 30, 2019
  50. 0/4 Warn about git-filter-branch usage and avoid itElijah Newren, Sep 3, 2019
  51. 2/4 t3427: accelerate this test by using fast-export and fast-importElijah Newren, Sep 3, 2019
  52. Junio C HamanoSep 3, 2019
  53. Junio C HamanoSep 3, 2019
  54. Elijah NewrenSep 4, 2019
  55. 1/4 t6006: simplify and optimize empty message testElijah Newren, Sep 3, 2019
  56. Junio C HamanoSep 3, 2019
  57. Elijah NewrenSep 3, 2019
  58. Junio C HamanoSep 3, 2019
  59. 3/4 Recommend git-filter-repo instead of git-filter-branchElijah Newren, Sep 3, 2019
  60. Junio C HamanoSep 3, 2019
  61. Elijah NewrenSep 4, 2019
  62. 4/4 t9902: use a non-deprecated command for testingElijah Newren, Sep 3, 2019
  63. 0/3 Warn about git-filter-branch usage and avoid itElijah Newren, Sep 4, 2019
  64. 1/3 t6006: simplify, fix, and optimize empty message testElijah Newren, Sep 4, 2019
  65. 2/3 Recommend git-filter-repo instead of git-filter-branchElijah Newren, Sep 4, 2019
  66. 3/3 t9902: use a non-deprecated command for testingElijah Newren, Sep 4, 2019
  67. Eric WongAug 23, 2019
  68. Elijah NewrenAug 23, 2019
  69. Elijah NewrenAug 23, 2019
  70. Johannes SchindelinAug 28, 2019
  71. Junio C HamanoAug 28, 2019
  72. Derrick StoleeAug 23, 2019
  73. Jeff KingAug 26, 2019

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.