git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 08/10] files_ref_store: use a transaction to update packed refs

From
Jeff King <peff@peff.net>
Date
Sep 8, 2017, 12:57 UTC
Message-ID
<20170908125733.afpo66vbummuaalg@sigill.intra.peff.net>
In-Reply-To
<21ec2708-e8e2-8503-fe90-48a70c802cc6@alum.mit.edu>
On Fri, Sep 08, 2017 at 02:44:57PM +0200, Michael Haggerty wrote:
Show 14 quoted lines
> > That means we're holding the packed-refs lock for a slightly longer
> > period. I think this could mean worse lock contention between otherwise
> > unrelated transactions over the packed-refs file. I wonder if the
> > lock-retry timeout might need to be increased to accommodate this. On
> > the other hand, it looks like we take it after getting the individual
> > locks, which I'd think would be the expensive part.
> 
> That was my thinking, yes. While the packed-refs lock is held, the
> references being created/updated have their reflogs written and are
> renamed into place. I don't see how that can be shortened without
> compromising on correctness (in particular, that we want to process
> creates/updates before deletions to try to preserve reachability as much
> as possible during the transaction). As an added optimization, the
> packed-refs lock is not acquired at all if no references are being deleted.

Makes sense. I guess in theory a process could do significant unrelated work between the "prepare" and "finish" steps, while holding the lock. But I don't know why it would, and arguably that itself is a bug.

Show 7 quoted lines
> The packed-refs file and loose references are never locked at the same
> time during pack-refs, so no deadlock is possible.
> 
> But you are right to assume that it *should* be so. The algorithm
> written above is a tiny bit unsafe (and has been for years). It is
> possible, though admittedly very unlikely, for the following to happen
> in the gap between steps 5 and 6:

Thanks for explaining (and I think that your "should" is why I thought it was so; we've discussed this race before).

Show 10 quoted lines
> This would leave "foo" at the obsolete value "B" (i.e., the value
> written to the `packed-refs` file for it by the nested `pack-refs` process.
> 
> I think that fixing this problem would require the `packed-refs` lock to
> be held while `pack-refs` is pruning the loose references. But given how
> unlikely that chain of events seems, and that fixing it would increase
> contention on the `packed-refs` file and allow the deadlock that you
> described, I lean towards leaving it as-is. Though admittedly,
> contention over a loose reference lock could make the race more likely
> to be hit.

Agreed. It's a pretty unlikely sequence of events. And IMHO the real solution is a new storage format that's easier to reason about.

-Peff
Previous: Michael HaggertyNext: Michael Haggerty
Message 20 of 31 in “Implement transactions for the packed ref store”
  1. 00/10 Implement transactions for the packed ref storeMichael Haggerty, Aug 29, 2017
  2. 05/10 files_pack_refs(): use a reference transaction to write packed refsMichael Haggerty, Aug 29, 2017
  3. 04/10 packed_delete_refs(): implement methodMichael Haggerty, Aug 29, 2017
  4. Brandon WilliamsAug 29, 2017
  5. Michael HaggertyAug 30, 2017
  6. 06/10 files_initial_transaction_commit(): use a transaction for packed refsMichael Haggerty, Aug 29, 2017
  7. Jeff KingSep 8, 2017
  8. Michael HaggertySep 8, 2017
  9. 07/10 t1404: demonstrate two problems with reference transactionsMichael Haggerty, Aug 29, 2017
  10. Stefan BellerAug 30, 2017
  11. Michael HaggertyAug 31, 2017
  12. Junio C HamanoSep 8, 2017
  13. Jeff KingSep 8, 2017
  14. Michael HaggertySep 8, 2017
  15. 09/10 packed-backend: rip out some now-unused codeMichael Haggerty, Aug 29, 2017
  16. Brandon WilliamsAug 29, 2017
  17. 08/10 files_ref_store: use a transaction to update packed refsMichael Haggerty, Aug 29, 2017
  18. Jeff KingSep 8, 2017
  19. Michael HaggertySep 8, 2017
  20. Jeff KingSep 8, 2017
  21. 10/10 files_transaction_finish(): delete reflogs before referencesMichael Haggerty, Aug 29, 2017
  22. 01/10 packed-backend: don't adjust the reference count on lock/unlockMichael Haggerty, Aug 29, 2017
  23. Jeff KingSep 8, 2017
  24. Michael HaggertySep 8, 2017
  25. 02/10 struct ref_transaction: add a place for backends to store dataMichael Haggerty, Aug 29, 2017
  26. Jeff KingSep 8, 2017
  27. Michael HaggertySep 8, 2017
  28. Jeff KingSep 8, 2017
  29. 03/10 packed_ref_store: implement reference transactionsMichael Haggerty, Aug 29, 2017
  30. Brandon WilliamsAug 29, 2017
  31. Jeff KingSep 8, 2017

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.