Re: [PATCH v4 3/4] Makefile/coccicheck: allow for setting xargs concurrency

From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date: Mar 25, 2021, 02:29 UTC
Message-ID: <877dlwotjc.fsf@evledraar.gmail.com>
In-Reply-To: <YFuSSqbAjTmaEMCB@coredump.intra.peff.net>

On Wed, Mar 24 2021, Jeff King wrote:

Show 28 quoted lines

> On Mon, Mar 22, 2021 at 01:11:49PM +0100, Ævar Arnfjörð Bjarmason wrote:
>
>> Extend the SPATCH_BATCH_SIZE facility added in
>> 960154b9c17 (coccicheck: optionally batch spatch invocations,
>> 2019-05-06) and bcb4edf7af7 (coccicheck: make batch size of 0 mean
>> "unlimited", 2019-05-08) to allow for both setting
>> SPATCH_BATCH_SIZE=N, and also to have a more advanced SPATCH_XARGS
>> argument.
>> 
>> The reason to replace the "$$limit" is that now you actually see under
>> V=1 what argument your program will get invoked with.
>> 
>> The reason for the "9999" limit is that if you e.g. try to define an
>> "$(XARGS)" which is conditionally an empty string or not depending on
>> this setting then e.g.:
>> 
>>     echo $(FILES) | $(XARGS) $(XARGS_FLAGS) $(SPATCH)
>> 
>> Over multiple lines with "\" will error out. I think just setting it
>> to "xargs -n 9999" as a trivial workaround is the best solution here.
>
> I don't understand this 9999 comment. The original was sometimes setting
> $limit to the empty string, and then doing:
>
>  xargs $limit
>
> How is that any different than setting SPATCH_XARGS to just "xargs" for
> the unlimited case?

The "over multiple lines" is important. But it seems not anymore. I.e. in an earlier version I had:

    $(XARGS) \
        $(XARGS_FLAGS) \
        $(SPATCH)

And it would brek if XARGS_FLAGS was empty. So I set it to -n 9999 as a fallback.

Show 16 quoted lines

>> +# For the 'coccicheck' target; SPATCH_XARGS can be used to manually
>> +# tweak the xargs invocation. By default we invoke "xargs -n 1", and
>> +# "xargs -n 9999" under SPATCH_BATCH_SIZE=0.
>> +#
>> +# Setting SPATCH_XARGS overrides SPATCH_BATCH_SIZE. To get concurrency
>> +# when targeting a single contrib/coccinelle/%.patch use e.g. "-P" if
>> +# your xargs(1) supports it:
>> +#
>> +#    make contrib/coccinelle/strbuf.cocci.patch SPATCH_XARGS="xargs -P 8 -n 8"
>> +#
>> +# Or a combination -jN and "xargs -P":
>> +#
>> +#    make -j4 coccicheck SPATCH_XARGS="xargs -P 2 -n 8"
>
> As I mentioned in the last round, using "-P" is racy. I'm not sure if
> it's something we should be recommending to people.

Yes, this would need to use N tempfiles we'd cat together at the end to be POSIX-reliable.

In practice I've never seen this sort of thing be unreliable for a case like this.

POSIX just guarantees that the output won't be interleaved up to PIPE_BUF, which is typically 4k. We certainly get patches bigger than that from spatch in some cases.

But from the OS's perspective emitting this output happens at a glacial pace. So even if it crosses that boundary it's unlikely to be interleaved.

Even:

    perl -wE 'print "a" for 1..1024*1024*100' >1
    perl -wE 'print "b" for 1..1024*1024*100' >2
    perl -wE 'print "\n" for 1..1024*1024*100' >3
    $ du -shc 1 2 3
    100M    1
    100M    2
    100M    3
    300M    total

Which at least on this computer I can't get to not print:

    $ echo 1 2 3 | xargs -P 3 -n 1 cat|wc -l
    104857600

Suggesting that even for output of that size the \n's aren't mixed up. YMMV.

Anyway. I think I'll just redact this topic. Maybe I'll change my mind on it, or pick it up some other month/year/whatever.

I appreciate everyone's feedback on it, but for something I thought would be a rather trivial bugfix I've sunk way too much time into it. At this point I can't even remember why I was trying to run coccicheck in the first place.

There's e.g. still your outstanding feedback on what's really going on with that underlying spatch race I was trying to fix with this --include-headers-for-types workaround.

I think digging into that would be interesting for someone, but right now that's not me. I really don't want to spend time on digging into spatch's OCaml guts to figure that & any other outstanding stuff out.