threads / discuss / 58322

Programmatic patches (transform commits)

Subject: Programmatic patches (transform commits)

## tl;dr

3 messages between Aug 18, 2022 and Aug 19, 2022.

replies: 2people: 3as markdown or json

Mark Fulton· Aug 18, 2022, 15:21 UTC · lore

Is there a way to commit a transform script that programmatically applies file changes rather than committing the file changes directly?

e.g. Imagine in a large repository that a contributor wants to replace certain instances of "abc" with "xyz". A transform script might be like the following:

```sh #!/bin/sh

sed -i 's/abc/xyz/g' $(find .) ```

Applying such a "programmatic patch" will potentially edit many files. Doing a code review on such a change is error prone due to authors resolving merge conflicts manually, etc. while reviewing the patch in some circumstances is much easier (especially tools for specifically this type of file transformations are used to make it easy to parse code, traverse abstract syntax trees, make edits, etc.).

Does anything like this exist today? Depending on the implementation I could see there being cross-platform support challenges but maybe there is something that already exists to assist with this which I can learn about.

As an alternative to making this part of Git I can see tools like GitHub Actions being used to look for commits of "programmatic patch" files, pick those up, run them, and commit and push the change but having a solution for this as part of Git itself would make it independent of GitHub and more reusable, etc.

Ævar Arnfjörð Bjarmason· Aug 18, 2022, 15:48 UTC · re: Mark Fulton · lore

Re: Programmatic patches (transform commits)

On Thu, Aug 18 2022, Mark Fulton wrote:

Show 30 quoted lines

> Is there a way to commit a transform script that programmatically
> applies file changes rather than committing the file changes directly?
>
> e.g. Imagine in a large repository that a contributor wants to replace
> certain instances of "abc" with "xyz". A transform script might be
> like the following:
>
> ```sh
> #!/bin/sh
>
> sed -i 's/abc/xyz/g' $(find .)
> ```
>
> Applying such a "programmatic patch" will potentially edit many files.
> Doing a code review on such a change is error prone due to authors
> resolving merge conflicts manually, etc. while reviewing the patch in
> some circumstances is much easier (especially tools for specifically
> this type of file transformations are used to make it easy to parse
> code, traverse abstract syntax trees, make edits, etc.).
>
> Does anything like this exist today? Depending on the implementation I
> could see there being cross-platform support challenges but maybe
> there is something that already exists to assist with this which I can
> learn about.
>
> As an alternative to making this part of Git I can see tools like
> GitHub Actions being used to look for commits of "programmatic patch"
> files, pick those up, run them, and commit and push the change but
> having a solution for this as part of Git itself would make it
> independent of GitHub and more reusable, etc.

Yes, e.g. for C (and used by linux.git and this project): https://www.kernel.org/doc/html/v4.14/dev-tools/coccinelle.html

Generally speaking I've made such ad-hoc commits in the past, where I've e.g. included the needed script (or one-liner) in the commit message, along with manually adding the diff to the commit message that *isn't* the part changed by the script or one-liner.

But I don't think there's any sort of widespread standard for this sort of thing, other than doing that, and the same (sometimes leading to non-atomic commits) of having the manual part of the change applied before or after in a separate commit.

Johannes Schindelin· Aug 19, 2022, 10:15 UTC · re: Mark Fulton · lore

Re: Programmatic patches (transform commits)

Hi Mark,

On Thu, 18 Aug 2022, Mark Fulton wrote:

Show 12 quoted lines

> Is there a way to commit a transform script that programmatically
> applies file changes rather than committing the file changes directly?
>
> e.g. Imagine in a large repository that a contributor wants to replace
> certain instances of "abc" with "xyz". A transform script might be
> like the following:
>
> ```sh
> #!/bin/sh
>
> sed -i 's/abc/xyz/g' $(find .)
> ```

I do have a script to do such a thing, and I am sure other users do, too. It would be a simpler script if it did not have to accommodate for `sed`'s `-i` option behaving differently between GNU and BSD variants. For the GNU variant, I use

	git grep -zl "$regex" | xargs -0r sed -i "s/$regex/$replacement/g"

and for the BSD variant I use `sed -i ''` (i.e. an extra, empty argument) and skip the `-r` option because BSD `xargs` does not understand it.

Show 6 quoted lines

> Applying such a "programmatic patch" will potentially edit many files.
> Doing a code review on such a change is error prone due to authors
> resolving merge conflicts manually, etc. while reviewing the patch in
> some circumstances is much easier (especially tools for specifically
> this type of file transformations are used to make it easy to parse
> code, traverse abstract syntax trees, make edits, etc.).

In the Git project, we do have a track record of mentioning the exact commands for automated transformation in the commit message, see e.g. https://github.com/git/git/commit/8dcf73c5c940.

The idea is that any reviewer could run the same command, verify that the outcome is the same, and give their ACK. Or, in case of merge conflicts, they would re-run the command and commit the result using the original commit message.

Ciao, Dscho