git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 07/10] t1404: demonstrate two problems with reference transactions

From
Michael Haggerty <mhagger@alum.mit.edu>
Date
Aug 31, 2017, 03:42 UTC
Message-ID
<810bbfa2-4486-e3dd-8243-ea6116f21b2c@alum.mit.edu>
In-Reply-To
<CAGZ79ka+uaBD_1xk6sx9J83KuP4A_mUWVqtyniNusbDeQiUZyQ@mail.gmail.com>
On 08/30/2017 07:21 PM, Stefan Beller wrote:
Show 40 quoted lines
> On Tue, Aug 29, 2017 at 1:20 AM, Michael Haggerty <mhagger@alum.mit.edu> wrote:
>> [...]
>> +test_expect_failure 'no bogus intermediate values during delete' '
>> +       prefix=refs/slow-transaction &&
>> +       # Set up a reference with differing loose and packed versions:
>> +       git update-ref $prefix/foo $C &&
>> +       git pack-refs --all &&
>> +       git update-ref $prefix/foo $D &&
>> +       git for-each-ref $prefix >unchanged &&
>> +       # Now try to update the reference, but hold the `packed-refs` lock
>> +       # for a while to see what happens while the process is blocked:
>> +       : >.git/packed-refs.lock &&
>> +       test_when_finished "rm -f .git/packed-refs.lock" &&
>> +       {
>> +               sleep 1 &&
>> +               rm -f .git/packed-refs.lock &
>> +       } &&
>> +       pid1=$! &&
>> +       {
>> +               # Note: the following command is intentionally run in the
>> +               # background. We extend the timeout so that `update-ref`
>> +               # tries to acquire the `packed-refs` lock longer than it
>> +               # takes the background process above to delete it:
>> +               git -c core.packedrefstimeout=2000 update-ref -d $prefix/foo &
>> +       } &&
>> +       pid2=$! &&
>> +       ok=true &&
>> +       while kill -0 $pid2 2>/dev/null
> 
>     If sig is 0, then no signal is sent, but error checking is still
>     performed; this can be used to check for the existence of a
>     process ID or process group ID.
> 
> So the kill -0 is the idiomatic form of "while $pid2 is still alive"?
> ignoring errors due to the dev/null redirection?
> 
> And due to the nature of this test we have to have a busy
> loop, we cannot rate limit the cpu usage inside the loop
> via some shorter sleeps, as ideally we want to observe
> the ref at any time.
Correct on both counts.

I just noticed that there is a stray line `ok=true &&` from an earlier draft. I'll remove that in v2.

>     In an ideal world this test would instruct the kernel to interrupt
>     the executing program (update-ref) at certain events such as
>     touching/writing/deleting files and in each interrupt we could
>     inspect the file system in a read only fashion.

A tool like `strace` could be used for tests like this, but it would be terribly non-portable. (But I often use strace manually to check that the ordering of filesystem events is correct.)

Show 35 quoted lines
>> +       do
>> +               sha1=$(git rev-parse --verify --quiet $prefix/foo || echo undefined) &&
>> +               case "$sha1" in
>> +               $D)
>> +                       # This is OK; it just means that nothing has happened yet.
>> +                       : ;;
>> +               undefined)
>> +                       # This is OK; it means the deletion was successful.
>> +                       : ;;
>> +               $C)
>> +                       # This value should never be seen. Probably the loose
>> +                       # reference has been deleted but the packed reference
>> +                       # is still there:
>> +                       echo "$prefix/foo incorrectly observed to be C" &&
>> +                       break
>> +                       ;;
>> +               *)
>> +                       # WTF?
>> +                       echo "$prefix/foo unexpected value observed: $sha1" &&
>> +                       break
>> +                       ;;
>> +               esac
>> +       done >out &&
>> +       wait $pid1 &&
>> +       wait $pid2 &&
> 
> oh, you use explicit pids here to check each exit code.
> 
>> If anybody has suggestions for better ways to test these things,
>> please speak up :-)
> 
> I don't think I'd have a satisfactory answer to that, as the timing is inherent
> to the things we test. In other software projects that are less low level, I
> would have suggested to use a time/clock mock, which can be stopped
> and then inspection can be performed at defined states.

I just realized that, given that the main goal here is to check the value of the reference while `update-ref` is waiting on the `packed-refs` lock, we can do the test without a busy loop. Instead, we roughly

	: >.git/packed-refs.lock &&
	{
		git -c core.packedrefstimeout=2000 update-ref -d $prefix/foo &
	} &&
	pid2=$! &&
	sleep 1 &&
	# Verify that update-ref is still running:
	kill -0 $pid2 &&
	# ...verify that the reference still has its old value...
	rm -f .git/packed-refs.lock &&
	wait $pid2 &&
	# ...verify that the reference is now gone...

It's true that this version wouldn't discover incorrect transitional values of the reference that happen at other times, but that was very unlikely anyway given the speed disparity between C and shell. I'll make this change in v2.

Michael
Previous: Stefan BellerNext: Junio C Hamano
Message 11 of 31 in “Implement transactions for the packed ref store”
  1. 00/10 Implement transactions for the packed ref storeMichael Haggerty, Aug 29, 2017
  2. 05/10 files_pack_refs(): use a reference transaction to write packed refsMichael Haggerty, Aug 29, 2017
  3. 04/10 packed_delete_refs(): implement methodMichael Haggerty, Aug 29, 2017
  4. Brandon WilliamsAug 29, 2017
  5. Michael HaggertyAug 30, 2017
  6. 06/10 files_initial_transaction_commit(): use a transaction for packed refsMichael Haggerty, Aug 29, 2017
  7. Jeff KingSep 8, 2017
  8. Michael HaggertySep 8, 2017
  9. 07/10 t1404: demonstrate two problems with reference transactionsMichael Haggerty, Aug 29, 2017
  10. Stefan BellerAug 30, 2017
  11. Michael HaggertyAug 31, 2017
  12. Junio C HamanoSep 8, 2017
  13. Jeff KingSep 8, 2017
  14. Michael HaggertySep 8, 2017
  15. 09/10 packed-backend: rip out some now-unused codeMichael Haggerty, Aug 29, 2017
  16. Brandon WilliamsAug 29, 2017
  17. 08/10 files_ref_store: use a transaction to update packed refsMichael Haggerty, Aug 29, 2017
  18. Jeff KingSep 8, 2017
  19. Michael HaggertySep 8, 2017
  20. Jeff KingSep 8, 2017
  21. 10/10 files_transaction_finish(): delete reflogs before referencesMichael Haggerty, Aug 29, 2017
  22. 01/10 packed-backend: don't adjust the reference count on lock/unlockMichael Haggerty, Aug 29, 2017
  23. Jeff KingSep 8, 2017
  24. Michael HaggertySep 8, 2017
  25. 02/10 struct ref_transaction: add a place for backends to store dataMichael Haggerty, Aug 29, 2017
  26. Jeff KingSep 8, 2017
  27. Michael HaggertySep 8, 2017
  28. Jeff KingSep 8, 2017
  29. 03/10 packed_ref_store: implement reference transactionsMichael Haggerty, Aug 29, 2017
  30. Brandon WilliamsAug 29, 2017
  31. Jeff KingSep 8, 2017

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.