Re: [PATCH] Makefile: fix bugs in coccicheck and speed it up

From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date: Mar 3, 2021, 11:45 UTC
Message-ID: <87a6rkjwa7.fsf@evledraar.gmail.com>
In-Reply-To: <20210302205103.12230-1-avarab@gmail.com>

On Tue, Mar 02 2021, Ævar Arnfjörð Bjarmason wrote:

A few notes, for a probable re-roll sometime later:

> I've often wondered why "make coccicheck" takes so long. This change
> speeds it up by ~2x and makes it use much less memory. Or a reduction
> of a max of around ~2GB per-process (under the old
> SPATCH_BATCH_SIZE=0) to around ~200MB.

I've also tested this with e.g. "strace -c -f -e trace=openat ", before/after with just contrib/coccinelle/swap.cocci.patch:

    192612
    13106

Grepping e.g. for strbuf.h shows that when running "make coccicheck" we open it 12 times as expected (we have 12 rules)

    strace -f -e trace=openat make -j8 coccicheck 2>&1|grep 'openat.*"strbuf\.h"' | wc -l
    12

(Is there a more direct way to filter this type of thing with strace?)

Before:

    $ strace -f -e trace=openat make -j8 coccicheck 2>&1|grep 'openat.*"strbuf\.h"' | wc -l
    137

> [...]
> I'm also the whole "cat $@.log" introduced in [3]. Since we don't call
> this in a loop anymore (and xargs will early-exit) we can just rely on
> standard V=1 for debugging issues.

Noticed by Denton in the side-thread, should be:

    I'm also removing the whole "cat $@.log" feature introduced in...

> +		xargs \
> +			-n 32 $(SPATCH_XARGS_FLAGS) \

As an aside this relies on xargs not barfing on "xargs -n 1 -n 2", i.e. taking 2 over 1. I tested it on Linux/AIX/OpenBSD/Solaris, works consistently on all of them.