scripting speedups [was: [Summit topic] Crazy (and not so crazy) ideas]

From: Eric Wong <e@80x24.org>
Date: Oct 26, 2021, 20:14 UTC
Message-ID: <20211026201448.GA29480@dcvr>
In-Reply-To: <nycvar.QRO.7.76.6.2110211144490.56@tvgsbejvaqbjf.bet>

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

Show 7 quoted lines

> * Test suite is slow. Shell scripts and process forking.
> 
>    * What if we had a special shell that interpreted the commands in a
>      single process?
> 
>    * Even Git commands like rev-parse and hash-object, as long as that’s
>      not the command you’re trying to test

This is something I've wanted in a very long time as a scripter. fast-import has been great over the years, as is "cat-file --batch(-check)", but there's gaps should be filled (preferably without fragile linkage of shared libraries into a script process)

Show 6 quoted lines

>    * Dscho wants to slip in a C-based solution
> 
>    * Jonathan tan commented: going back to your custom shell for tests
>      idea, one thing we could do is have a custom command that generates
>      the repo commits that we want (and that saves process spawns and
>      might make the tests simpler too)

Perhaps a not-seriously-proposed patch from 2006 could be modernized for our now-libified internals:

https://yhbt.net/lore/git/Pine.LNX.4.64.0602232229340.3771@g5.osdl.org/

Show 5 quoted lines

>       * We could replace several “setup repo” steps with “git fast-import”
>         instead.
> 
>    * Dscho measured: 0.5 sec - 30 sec in setup steps. Can use fast-import,
>      or can make a new format that helps us set up the test scenario

0.5s - 30s across the whole suite or individual tests?

Having a way to disable fsync globally should further improve things, especially for people on slower storage. libeatmydata is available, but perhaps not widely available/known.

>    * Elijah: test-lib-functions helpers could be built ins