Re: [PATCH v4 3/3] refs/reftable: reload locked stack when preparing transaction

From: Jeff King <peff@peff.net>
Date: Oct 1, 2024, 19:08 UTC
Message-ID: <20241001190823.GA2194387@coredump.intra.peff.net>
In-Reply-To: <joxzwkbi4uqyb6d67e24ivfvb54ev5ciqgvuxgxh4nntonjwqa@az3xgiiqg4x7>

On Tue, Oct 01, 2024 at 11:53:54AM -0700, Josh Steadmon wrote:

Show 20 quoted lines

> > > We're seeing repeated failures from this test case with ASan enabled.
> > > Unfortunately, we've only been able to reproduce this on our
> > > $DAYJOB-specific build system. I haven't been able to get it to fail
> > > using just the upstream Makefile so far. I'll keep trying to find a way
> > > to reproduce this.
> > >
> > > FWIW, we're not getting I/O errors, we see the following:
> > > fatal: update_ref failed for ref 'refs/heads/branch-20': cannot lock references
> > >
> > > We tried increasing the timeout in the test to 2 minutes (up from 10s),
> > > but it didn't fix the failures.
> > 
> > Thanks for a report, and please keep digging ;-).
> > 
> > Is your build, like Peff's, for Windows, or your variant of Linux?
> > 
> > Thanks.
> 
> It's our internal Debian-based variant of Linux. Sorry for not
> specifying earlier.

I just tried doing (on 'next', since the new test is added there):

  make SANITIZE=address,undefined CFLAGS=-O0
  cd t
  ./t0610-reftable-basics.sh --run=1-47 --stress

and got lots of failures like you describe. But in my case bumping the timeout from 10s to 100s made that go away. So I think it was just the sanitizers making things _really_ slow and causing legit timeouts.

Have you tried going beyond 2 minutes? That seems like a crazy amount of time, but I guess it's possible on a really overloaded system. Of course it's also quite possible that there are multiple issues and you are seeing something else entirely. :)

-Peff