# Coccinelle for automated refactors 5 messages from 2016-06-05 to 2016-06-06. Participants: brian m. carlson, Stefan Beller, Junio C Hamano, Jacob Keller. Thread: https://gitlist.dev/t/42533 ## brian m. carlson, 2016-06-05 20:55 Subject: Coccinelle for automated refactors Message-ID: <20160605205518.GA153578@vauxhall.crustytoothpaste.net> URL: https://gitlist.dev/e/20160605205518.GA153578%40vauxhall.crustytoothpaste.net ``` One thing that I've noticed with the struct object_id conversion is that most of the work is mechanical transformations of a data member from one type into another. Doing this by hand is both boring and error-prone, and it requires a tiresome review of nearly-identical changes. I've noticed that Coccinelle[0], a tool for automated refactors, has been used with great success on LKML, because it understands C well and can perform the transformations precisely and rapidly. It also does nice things like indenting the code it modifies if necessary. An example semantic patch looks like this: @@ expression E1; @@ - is_null_sha1(E1.hash) + is_null_oid(&E1) @@ expression E1; @@ - is_null_sha1(E1->hash) + is_null_oid(E1) This does what you think it does: transforms calls to is_null_sha1 that use the struct object_id hash member into calls to is_null_oid. I'd like to use this for some of the struct object_id work if others think this is a good idea. I feel it's likely to reduce the reviewing overhead and allow people to better reason about the quality and behavior of the sent patches. Of course, I would still review the patches manually for errors and improvements, and would still accept responsibility for the content of the patches. If there's interest, I can send a patch with a set of basic object_id transforms to make it easier for others to make those changes when they're doing work elsewhere in the codebase. [0] http://coccinelle.lip6.fr/ -- brian m. carlson / brian with sandals: Houston, Texas, US +1 832 623 2791 | https://www.crustytoothpaste.net/~bmc | My opinion only OpenPGP: https://keybase.io/bk2204 ``` ## Stefan Beller, 2016-06-06 17:43 Subject: Re: Coccinelle for automated refactors Message-ID: URL: https://gitlist.dev/e/CAGZ79kbDgP6PJ6F6ksA33qDs%3D%3DFHDBe8EwE%3DiBOWe8as_oFLkA%40mail.gmail.com In-Reply-To: <20160605205518.GA153578@vauxhall.crustytoothpaste.net> ``` On Sun, Jun 5, 2016 at 1:55 PM, brian m. carlson wrote: > One thing that I've noticed with the struct object_id conversion is that > most of the work is mechanical transformations of a data member from one > type into another. Doing this by hand is both boring and error-prone, > and it requires a tiresome review of nearly-identical changes. > > I've noticed that Coccinelle[0], a tool for automated refactors, has > been used with great success on LKML, because it understands C well and > can perform the transformations precisely and rapidly. It also does > nice things like indenting the code it modifies if necessary. > > An example semantic patch looks like this: > > @@ > expression E1; > @@ > - is_null_sha1(E1.hash) > + is_null_oid(&E1) > > @@ > expression E1; > @@ > - is_null_sha1(E1->hash) > + is_null_oid(E1) So is this all we have to review then? > > This does what you think it does: transforms calls to is_null_sha1 that > use the struct object_id hash member into calls to is_null_oid. > > I'd like to use this for some of the struct object_id work if others > think this is a good idea. I feel it's likely to reduce the reviewing > overhead and allow people to better reason about the quality and > behavior of the sent patches. Of course, I would still review the > patches manually for errors and improvements, and would still accept > responsibility for the content of the patches. > > If there's interest, I can send a patch with a set of basic object_id > transforms to make it easier for others to make those changes when > they're doing work elsewhere in the codebase. I would be interested in reviewing such a patch. Thanks, Stefan > > [0] http://coccinelle.lip6.fr/ > -- > brian m. carlson / brian with sandals: Houston, Texas, US > +1 832 623 2791 | https://www.crustytoothpaste.net/~bmc | My opinion only > OpenPGP: https://keybase.io/bk2204 ``` ## Junio C Hamano, 2016-06-06 18:55 Subject: Re: Coccinelle for automated refactors Message-ID: URL: https://gitlist.dev/e/xmqqy46iw3ih.fsf%40gitster.mtv.corp.google.com In-Reply-To: <20160605205518.GA153578@vauxhall.crustytoothpaste.net> ``` "brian m. carlson" writes: > An example semantic patch looks like this: > > @@ > expression E1; > @@ > - is_null_sha1(E1.hash) > + is_null_oid(&E1) > > @@ > expression E1; > @@ > - is_null_sha1(E1->hash) > + is_null_oid(E1) > > This does what you think it does: transforms calls to is_null_sha1 that > use the struct object_id hash member into calls to is_null_oid. > > I'd like to use this for some of the struct object_id work if others > think this is a good idea. I feel it's likely to reduce the reviewing > overhead and allow people to better reason about the quality and > behavior of the sent patches. Of course, I would still review the > patches manually for errors and improvements, and would still accept > responsibility for the content of the patches. Is the plan for such a "refactor" patch to compose such a series as two patch series: [1/2] automatic refactor which gives the "semantic patch" in the proposed log message as part of its description, and the automated result (with possible misconversions that may have come from bugs in the automated tools), with a separate [2/2] manual fixups that corrects what was misconverted and what was missed? As long as [2/2] can be kept to the minimum (and an automated tool that is worth using should make it so), I think that is a good way forward. Another possibility would be to send the end-result as a single patch, with description on the manual fixups in the proposed log message, but it would be a lot more work to generate and review such a patch, I would think. > If there's interest, I can send a patch with a set of basic object_id > transforms to make it easier for others to make those changes when > they're doing work elsewhere in the codebase. > > [0] http://coccinelle.lip6.fr/ ``` ## Jacob Keller, 2016-06-06 20:38 Subject: Re: Coccinelle for automated refactors Message-ID: URL: https://gitlist.dev/e/CA%2BP7%2Bxqszc-BRBYo7vaiQBFy8c9MMJwTY3sLrX5gOFZc21Mm8Q%40mail.gmail.com In-Reply-To: ``` On Mon, Jun 6, 2016 at 11:55 AM, Junio C Hamano wrote: > "brian m. carlson" writes: > >> An example semantic patch looks like this: >> >> @@ >> expression E1; >> @@ >> - is_null_sha1(E1.hash) >> + is_null_oid(&E1) >> >> @@ >> expression E1; >> @@ >> - is_null_sha1(E1->hash) >> + is_null_oid(E1) >> >> This does what you think it does: transforms calls to is_null_sha1 that >> use the struct object_id hash member into calls to is_null_oid. >> >> I'd like to use this for some of the struct object_id work if others >> think this is a good idea. I feel it's likely to reduce the reviewing >> overhead and allow people to better reason about the quality and >> behavior of the sent patches. Of course, I would still review the >> patches manually for errors and improvements, and would still accept >> responsibility for the content of the patches. > > Is the plan for such a "refactor" patch to compose such a series as > two patch series: > > [1/2] automatic refactor > > which gives the "semantic patch" in the proposed log message as part > of its description, and the automated result (with possible > misconversions that may have come from bugs in the automated tools), > with a separate > > [2/2] manual fixups > > that corrects what was misconverted and what was missed? > > As long as [2/2] can be kept to the minimum (and an automated tool > that is worth using should make it so), I think that is a good way > forward. Another possibility would be to send the end-result as a > single patch, with description on the manual fixups in the proposed > log message, but it would be a lot more work to generate and review > such a patch, I would think. > >> If there's interest, I can send a patch with a set of basic object_id >> transforms to make it easier for others to make those changes when >> they're doing work elsewhere in the codebase. >> >> [0] http://coccinelle.lip6.fr/ > -- I've used coccinelle before on the Linux Kernel, and I've found that it works quite well without producing any incorrect changes, especially if you keep the semantic patch small and simple. I'd definitely go with the 1/2 2/2 approach but I suspect in most useful cases 2/2 would be very minor if at all. Thanks, Jake ``` ## brian m. carlson, 2016-06-06 23:09 Subject: Re: Coccinelle for automated refactors Message-ID: <20160606230936.GA60445@vauxhall.crustytoothpaste.net> URL: https://gitlist.dev/e/20160606230936.GA60445%40vauxhall.crustytoothpaste.net In-Reply-To: ``` On Mon, Jun 06, 2016 at 11:55:50AM -0700, Junio C Hamano wrote: > Is the plan for such a "refactor" patch to compose such a series as > two patch series: > > [1/2] automatic refactor > > which gives the "semantic patch" in the proposed log message as part > of its description, and the automated result (with possible > misconversions that may have come from bugs in the automated tools), > with a separate > > [2/2] manual fixups > > that corrects what was misconverted and what was missed? > > As long as [2/2] can be kept to the minimum (and an automated tool > that is worth using should make it so), I think that is a good way > forward. Another possibility would be to send the end-result as a > single patch, with description on the manual fixups in the proposed > log message, but it would be a lot more work to generate and review > such a patch, I would think. My hope is that I can make the automated changes such that manual fixups are more along the lines of cleaning up related functions in the module, fixing issues noticed during the refactor, and the like: in other words, things that one might have done incidentally as part of the patch, but could defensibly be done in a second patch anyway. My goal is to make the series as much like a human-edited series as possible, but with less work on all sides. I can send an RFC series that demonstrates this a little later to see if it's an acceptable direction for work. -- brian m. carlson / brian with sandals: Houston, Texas, US +1 832 623 2791 | https://www.crustytoothpaste.net/~bmc | My opinion only OpenPGP: https://keybase.io/bk2204 ```