git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v4 3/3] refs/reftable: reload locked stack when preparing transaction

From
Patrick Steinhardt <ps@pks.im>
Date
Oct 1, 2024, 04:27 UTC
Message-ID
<Zvt6LxWm8gtJGw9S@pks.im>
In-Reply-To
<c4lz3begoplgde5iimvk4k7cufiyryntccqo46u3fy5qvqauv3@tta5wfg2ik5t>
On Mon, Sep 30, 2024 at 03:19:04PM -0700, Josh Steadmon wrote:
Show 45 quoted lines
> On 2024.09.27 00:07, Jeff King wrote:
> > On Tue, Sep 24, 2024 at 07:33:08AM +0200, Patrick Steinhardt wrote:
> > 
> > > +test_expect_success 'ref transaction: many concurrent writers' '
> > > +	test_when_finished "rm -rf repo" &&
> > > +	git init repo &&
> > > +	(
> > > +		cd repo &&
> > > +		# Set a high timeout such that a busy CI machine will not abort
> > > +		# early. 10 seconds should hopefully be ample of time to make
> > > +		# this non-flaky.
> > > +		git config set reftable.lockTimeout 10000 &&
> > 
> > I saw this test racily fail in the Windows CI build. The failure is as
> > you might imagine, a few of the background update-ref invocations
> > failed:
> > 
> >   fatal: update_ref failed for ref 'refs/heads/branch-21': reftable: transaction failure: I/O error
> > 
> > but of course we don't notice because they're backgrounded. And then the
> > expected output is missing the branch-21 entry (and in my case,
> > branch-64 suffered a similar fate).
> > 
> > At first I thought we probably needed to bump the timeout (and EIO was
> > just our way of passing that up the stack). But looking at the
> > timestamps in the Actions log, the whole loop took less than 10ms to
> > run.
> > 
> > So could this be indicative of a real contention issue specific to
> > Windows? I'm wondering if something like the old "you can't delete a
> > file somebody else has open" restriction is biting us somehow.
> > 
> > -Peff
> 
> We're seeing repeated failures from this test case with ASan enabled.
> Unfortunately, we've only been able to reproduce this on our
> $DAYJOB-specific build system. I haven't been able to get it to fail
> using just the upstream Makefile so far. I'll keep trying to find a way
> to reproduce this.
> 
> FWIW, we're not getting I/O errors, we see the following:
> fatal: update_ref failed for ref 'refs/heads/branch-20': cannot lock references
> 
> We tried increasing the timeout in the test to 2 minutes (up from 10s),
> but it didn't fix the failures.

If this is causing problems for folks I'd say we can do the below change for now. It's of course only a stop-gap solution until I find the time to debug this, which should be later this week or early next week.

Patrick
diff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh
index 2d951c8ceb..ad7bb39b79 100755
--- a/t/t0610-reftable-basics.sh
+++ b/t/t0610-reftable-basics.sh
@@ -450,7 +450,7 @@ test_expect_success 'ref transaction: retry acquiring tables.list lock' '
 	)
 '
 
-test_expect_success 'ref transaction: many concurrent writers' '
+test_expect_success !WINDOWS 'ref transaction: many concurrent writers' '
 	test_when_finished "rm -rf repo" &&
 	git init repo &&
 	(
Previous: Josh SteadmonNext: Jeff King
Message 36 of 42 in “reftable: graceful concurrent writes”
  1. 0/3 reftable: graceful concurrent writesPatrick Steinhardt, Sep 17, 2024
  2. 1/3 refs/reftable: introduce "reftable.lockTimeout"Patrick Steinhardt, Sep 17, 2024
  3. karthik nayakSep 17, 2024
  4. Eric SunshineSep 17, 2024
  5. Patrick SteinhardtSep 18, 2024
  6. Patrick SteinhardtSep 18, 2024
  7. 2/3 reftable/stack: allow locking of outdated stacksPatrick Steinhardt, Sep 17, 2024
  8. 3/3 refs/reftable: reload locked stack when preparing transactionPatrick Steinhardt, Sep 17, 2024
  9. karthik nayakSep 17, 2024
  10. Patrick SteinhardtSep 18, 2024
  11. 0/3 reftable: graceful concurrent writesPatrick Steinhardt, Sep 18, 2024
  12. 1/3 refs/reftable: introduce "reftable.lockTimeout"Patrick Steinhardt, Sep 18, 2024
  13. James LiuSep 18, 2024
  14. Patrick SteinhardtSep 18, 2024
  15. 2/3 reftable/stack: allow locking of outdated stacksPatrick Steinhardt, Sep 18, 2024
  16. James LiuSep 18, 2024
  17. Patrick SteinhardtSep 18, 2024
  18. 3/3 refs/reftable: reload locked stack when preparing transactionPatrick Steinhardt, Sep 18, 2024
  19. James LiuSep 18, 2024
  20. 0/3 reftable: graceful concurrent writesPatrick Steinhardt, Sep 18, 2024
  21. 1/3 refs/reftable: introduce "reftable.lockTimeout"Patrick Steinhardt, Sep 18, 2024
  22. Junio C HamanoSep 19, 2024
  23. 2/3 reftable/stack: allow locking of outdated stacksPatrick Steinhardt, Sep 18, 2024
  24. Junio C HamanoSep 20, 2024
  25. Patrick SteinhardtSep 24, 2024
  26. 3/3 refs/reftable: reload locked stack when preparing transactionPatrick Steinhardt, Sep 18, 2024
  27. James LiuSep 18, 2024
  28. Patrick SteinhardtSep 24, 2024
  29. 0/3 reftable: graceful concurrent writesPatrick Steinhardt, Sep 24, 2024
  30. 1/3 refs/reftable: introduce "reftable.lockTimeout"Patrick Steinhardt, Sep 24, 2024
  31. 2/3 reftable/stack: allow locking of outdated stacksPatrick Steinhardt, Sep 24, 2024
  32. 3/3 refs/reftable: reload locked stack when preparing transactionPatrick Steinhardt, Sep 24, 2024
  33. Jeff KingSep 27, 2024
  34. Patrick SteinhardtSep 30, 2024
  35. Josh SteadmonSep 30, 2024
  36. Patrick SteinhardtOct 1, 2024
  37. Jeff KingOct 1, 2024
  38. Junio C HamanoOct 1, 2024
  39. Patrick SteinhardtOct 2, 2024
  40. Junio C HamanoOct 1, 2024
  41. Josh SteadmonOct 1, 2024
  42. Jeff KingOct 1, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.