Re: [PATCH 0/8] Makefile: make command-list.h 2-5x as fast with -jN

From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date: Oct 22, 2021, 20:50 UTC
Message-ID: <211022.865ytok9ey.gmgdl@evledraar.gmail.com>
In-Reply-To: <YXMDm3ZzgE1UGSwk@coredump.intra.peff.net>

On Fri, Oct 22 2021, Jeff King wrote:

Show 26 quoted lines

> On Fri, Oct 22, 2021 at 12:51:11PM +0200, Ævar Arnfjörð Bjarmason wrote:
>
>> Yes, the same numbers are noted in the 7/8 commit message. I.e. it's
>> slower on -j1, but faster with higher -j<n> numbers.
>> 
>> Aside from any changes I'm proposing here it seems rather pointless to
>> me to optimize the runtime of -j1 runs.
>> 
>> I think we use those in e.g. CI, so of course if they become *really*
>> slow it will matter, but the purpose of this change is to make hacking
>> on git easier, both in terms of runtime and discovering what the
>> Makefile is doing wih V=1.
>> 
>> I think anyone hacking on git is going to be on a system with -j2 at
>> least. So again, separate from these specific changes, if we've got a
>> change that speeds up -jN runs at the cost of a -j1 run that seems like
>> good thing.
>
> It seems weird to me to assume that all of our processors are available
> to build command-list.h. In most cases you are not running "make -j16
> command-list.h", but rather "make -j16", and those other processors are
> doing useful things, like say, building actual C code.
>
> So counting CPU time is the interesting thing, because every cycle you
> save there gets used for other work. And "make -j1" just brings
> wall-clock and CPU time together.

There's been a lot of goalpost moving in the discussion around this series, but if you look at the numbers for the v1 I proposed at <patch-7.8-0c6f9b80d3b-20211020T183533Z-avarab@gmail.com> it also had a significant drop in "user" time.

The only thing that was slower in my tests in either wallclock or user time was the wallclock time on the building from scratch case (the user time was lower). Maybe those measurements were bad or whatever, but that's the context for the above.

Now, since then I've submitted a v2 with just the "make the shellscript faster" parts of this, including with some of your suggestions, making the shellscript version faster.

FWIW there's a rather trivial addition to my version that makes the "make all the things" faster again in wallclock/user time. We don't need to re-parse the command-list.txt at all again to emit the headers if command-list.txt doesn't change, which is the common case. So we can just cache the whole header portion in another *.gen file and "cat" it.

Anyway, I've been running these changes on top of other Makefile changes I've got locally where we re-build almost nothing as HEAD changes. No FORCE targets shellscripting (but the same via native GNU make features), and no *.sh/*.perl script re-generation on HEAD changes (the former being on-list).

So with that the common case really is that everything hangs on the command-list.h generation. Since we have maybe 1-5 *.c files modified, then we need to link everything, and the linking is hanging on help.o, which needs command-list.h.