Re: [PATCH v3] coccicheck: process every source file at once

From: SZEDER Gábor <szeder.dev@gmail.com>
Date: Oct 5, 2018, 18:39 UTC
Message-ID: <20181005183904.GV23446@localhost>
In-Reply-To: <20181005162517.GB11254@sigill.intra.peff.net>

On Fri, Oct 05, 2018 at 12:25:17PM -0400, Jeff King wrote:

Show 19 quoted lines

> On Fri, Oct 05, 2018 at 02:40:48PM +0200, SZEDER Gábor wrote:
> 
> > On Thu, Oct 04, 2018 at 07:17:47PM -0700, Jacob Keller wrote:
> > > Junio, do you want me to update the commit message on my side with the
> > > memory concerns? Or could you update it to mention memory as a noted
> > > trade off.
> > 
> > We have been running 'make -j2 coccicheck' in the static analysis
> > build job on Travis CI, which worked just fine so far.  The Travis CI
> > build environments have 3GB of memory available [1], but, as shown in
> > [2], with this patch the memory consumption jumps up to about
> > 1.3-1.8GB for each of those jobs.  So with two parallel jobs we will
> > very likely bump into this limit.
> > 
> > So this patch should definitely change that build script to run only a
> > single job.
> 
> It should still be a net win, since the total CPU seems to drop by a
> factor of 3-4.

Well, that's true when you have unlimited resources... :) or it's true even then, when I have just enough resources, but not much contention. After all, Coccinelle doesn't have to parse the same header files over and over again. However, on Travis CI, where who knows how many other build jobs are running next to our static analysis, it doesn't seem to be the case.

On current master with an additional 'time' in front:

  time make --jobs=2 coccicheck
  <...>
  695.70user 50.27system 6:27.88elapsed 192%CPU (0avgtext+0avgdata 91448maxresident)k
  5976inputs+2536outputs (42major+18411888minor)pagefaults 0swaps

  https://travis-ci.org/szeder/git/jobs/437733874#L574

With this patch, but without -j2 to fit into 3GB:

  960.50user 22.59system 16:23.74elapsed 99%CPU (0avgtext+0avgdata 1606156maxresident)k
  5976inputs+1320outputs (26major+4548440minor)pagefaults 0swaps

  https://travis-ci.org/szeder/git/jobs/437734003#L575

Note that both the runtime and the CPU time increased. (and RSS, of course)

Show 6 quoted lines

> Are we OK with saying 1.3-1.8GB is necessary to run coccicheck? That
> doesn't feel like an exorbitant request for a developer-only tool these
> days, but I have noticed some people on the list tend to have lousier
> machines than I do. ;)
> 
> -Peff