Re: New command/tool: git filter-repo

From: Elijah Newren <newren@gmail.com>
Date: Jun 8, 2019, 16:20 UTC
Message-ID: <CABPp-BHqdKvsUDx40Tz-A3z32TN_KWcDENWb8zsaidxxf8SNCg@mail.gmail.com>
In-Reply-To: <875ztukt77.fsf@evledraar.gmail.com>

Hi,

Now that there's a released version of git that has all necessary flags and features[1] to run git filter-repo (https://github.com/newren/git-filter-repo), I thought I'd send an update...

On Fri, Feb 8, 2019 at 10:53 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:

Show 46 quoted lines

> On Thu, Jan 31 2019, Elijah Newren wrote:
>
> > What's the future?  (Core command of git.git?  place it in contrib?  keep it
> > in a separate repo?)  I'm hoping to discuss that at the contributor summit
> > today, but feedback on the list is also welcome.
>
> Some of this I may have mentioned at the summit, but here for the list:
>
> * I think it should be a candidate for a core (not "just contrib")
>   git.git command, given that we have someone willing to maintain it &
>   deal with bugs etc. I'm not worried about that given the author.
>
> * It's unfortunate in terms of API we need to support going forward that
>   this obligates us to support a fairly intricate python API going
>   forward, so it's similar (but more detailed) to Git.pm (which I also
>   tried to get rid of as an external API a while ago).
>
>   However, as you correctly note that's the only way a command like this
>   can be really fast, we already have the "no special API" command with
>   git-filter-branch, and that's horribly slow.
>
>   But perhaps there's ways we can in advance deal with a potential
>   future breaking API change. E.g. some Pythonic way of versioning the
>   API, or just prominently documenting whatever (low?) stability
>   guarantees we're making.
>
>   I imagine if we need to make breaking changes in the future that'll
>   less big of a deal than in other cases, since we'd expect the API use
>   to be one-off migration scripts, although maybe it'll get used for
>   all-the-time exports (e.g. mirroring internal->external repos with
>   filtering).
>
> * The rest of our commands are hooked up to the i18n framework. I don't
>   think this should be a blocker, but it's worth thinking about what the
>   plan for this is.
>
>   Are we going to need the equivalent of Git::I18N for Python (which
>   presumably will be a run-time dependency on something needing the
>   Python API that links to gettext).
>
>   Or perhaps we could do the translated strings in C, by making the
>   program you're invoking be a C command, invoking the Python part as a
>   helper (which would need to re-invoke a helper if it prints its own
>   messages).
>
> Thanks for working on this!

I've implemented these, and several other things too.  Changes since last time:

* Now i18n-ized
* Several disclaimers about API backcompat (this is more of a one-shot
conversion tool) [2]
* Converted to Python3 (Python2 is EOL at EOY)
* Pruning of become-empty and become-degenerate+empty commits has been
fixed up (I mentioned this as a concern last time)
* Testsuite has been fleshed out, including not only multiple small
fixes to filter repo, but more fixes to git itself[3]
* Usage, Examples, Internals, and Limitations documentation now exists
(in README.md format and built-in -h help; no manpage yet)
* Several new filters and abilities have been added

Now that filter-repo is complete and more easily tested, what are folks thoughts on incorporating it in git.git? (And if we do go that route, can we avoid losing its history so I can bisect issues if necessary?)

Thanks, Elijah

[1] Well, except people will get an error if they use --preserve-commit-encoding saying they don't have a new enough git since en/fast-export-encoding is still sitting in next and didn't make it in to git-2.22. But I only know of one repo that even uses special commit encodings, so I suspect few if any will even care about that flag.

[2] The warnings appear in several places to try to make sure people notice them; a not quite complete list: https://github.com/newren/git-filter-repo/blame/master/README.md#L620-L625 https://github.com/newren/git-filter-repo/blame/master/README.md#L727-L730 https://github.com/newren/git-filter-repo/blame/master/README.md#L989-L993 https://github.com/newren/git-filter-repo/blob/master/git-filter-repo#L13-L30 https://github.com/newren/git-filter-repo/blob/master/git-filter-repo#L1523,L1524 https://github.com/newren/git-filter-repo/blob/master/t/t9391/commit_info.py#L4-L6 https://github.com/newren/git-filter-repo/blob/master/t/t9391/create_fast_export_output.py#L4-L6 https://github.com/newren/git-filter-repo/blob/master/t/t9391/file_filter.py#L4-L6 https://github.com/newren/git-filter-repo/blob/master/t/t9391/splice_repos.py#L4-L6 https://github.com/newren/git-filter-repo/blob/master/t/t9391/strip-cvs-keywords.py#L4-L6

[3] https://github.com/newren/git-filter-repo/tree/develop#upstream-improvements