git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 0/4] t5552: fix flakiness by introducing proper locking for GIT_TRACE

From
Jeff King <peff@peff.net>
Date
Aug 9, 2018, 19:47 UTC
Message-ID
<20180809194712.GC32376@sigill.intra.peff.net>
In-Reply-To
<pull.17.git.gitgitgadget@gmail.com>
On Thu, Aug 09, 2018 at 10:35:24AM -0700, Johannes Schindelin via GitGitGadget wrote:
> The culprit is that two processes try simultaneously to write to the same
> file specified via GIT_TRACE_PACKET, and it is not well defined how that
> should work, even on Linux.

Are you sure that it's not well-defined? We open the path with O_APPEND, which means every write() will be atomically positioned at the end of file. So we would never lose or overwrite data.

We do our own buffering in a strbuf, writing the result out in a single write() call (modulo the OS returning a short write, but that should not generally happen when writing short strings to a file). So we should get individual trace lines as atomic units.

The order of lines from the two processes is undefined, of course.
Show 5 quoted lines
> This patch series addresses that by locking the trace fd. I chose to use 
> flock() instead of fcntl() because the Win32 API LockFileEx()
> [https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-lockfileex] 
> (which does exactly what I want in this context) has much more similar
> semantics to the former than the latter.

I must admit I'm not excited to see us adding extra locking and complexity when it is not necessarily needed. Is this a feature we actually care about delivering for normal users of GIT_TRACE, or do we just care about solving the test flakiness here?

Because we should be able to do the latter with the patch below.

I also wonder if we should be clearing GIT_TRACE as part of clearing repo-env. It would do what we want automatically in this case, though there are definitely cases where I prefer the current behavior (because I really do want to see the upload-pack side of it). Of course I could always use "--upload-pack 'GIT_TRACE_PACKET=... upload-pack'", so this is really just a default.

diff --git a/t/t5552-skipping-fetch-negotiator.sh b/t/t5552-skipping-fetch-negotiator.sh
index 3b60bd44e0..5ad5bece55 100755
--- a/t/t5552-skipping-fetch-negotiator.sh
+++ b/t/t5552-skipping-fetch-negotiator.sh
@@ -28,6 +28,19 @@ have_not_sent () {
 	done
 }
 
+# trace_fetch <client_dir> <server_dir> [args]
+#
+# Trace the packet output of fetch, but make sure we disable the variable
+# in the child upload-pack, so we don't combine the results in the same file.
+trace_fetch () {
+	client=$1; shift
+	server=$1; shift
+	GIT_TRACE_PACKET="$(pwd)/trace" \
+	git -C "$client" fetch \
+	  --upload-pack 'unset GIT_TRACE_PACKET; git-upload-pack' \
+	  "$server" "$@"
+}
+
 test_expect_success 'commits with no parents are sent regardless of skip distance' '
 	git init server &&
 	test_commit -C server to_fetch &&
@@ -42,7 +55,7 @@ test_expect_success 'commits with no parents are sent regardless of skip distanc
 	# "c1" has no parent, it is still sent as "have" even though it would
 	# normally be skipped.
 	test_config -C client fetch.negotiationalgorithm skipping &&
-	GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch "$(pwd)/server" &&
+	trace_fetch client "$(pwd)/server" &&
 	have_sent c7 c5 c2 c1 &&
 	have_not_sent c6 c4 c3
 '
@@ -88,7 +101,7 @@ test_expect_success 'when two skips collide, favor the larger one' '
 	# the next "have" sent will be "c1" (from "c6" skip 4) and not "c4"
 	# (from "c5side" skip 1).
 	test_config -C client fetch.negotiationalgorithm skipping &&
-	GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch "$(pwd)/server" &&
+	trace_fetch client "$(pwd)/server" &&
 	have_sent c5side c11 c9 c6 c1 &&
 	have_not_sent c10 c8 c7 c5 c4 c3 c2
 '
@@ -114,7 +127,7 @@ test_expect_success 'use ref advertisement to filter out commits' '
 	# not need to send any ancestors of "c3", but we still need to send "c3"
 	# itself.
 	test_config -C client fetch.negotiationalgorithm skipping &&
-	GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch origin to_fetch &&
+	trace_fetch client origin to_fetch &&
 	have_sent c5 c4^ c2side &&
 	have_not_sent c4 c4^^ c4^^^
 '
@@ -144,7 +157,7 @@ test_expect_success 'handle clock skew' '
 	# and sent, because (due to clock skew) its only parent has already been
 	# popped off the priority queue.
 	test_config -C client fetch.negotiationalgorithm skipping &&
-	GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch "$(pwd)/server" &&
+	trace_fetch client "$(pwd)/server" &&
 	have_sent c2 c1 old4 old2 old1 &&
 	have_not_sent old3
 '
@@ -176,7 +189,7 @@ test_expect_success 'do not send "have" with ancestors of commits that server AC
 	test_commit -C server commit-on-b1 &&
 
 	test_config -C client fetch.negotiationalgorithm skipping &&
-	GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch "$(pwd)/server" to_fetch &&
+	trace_fetch client "$(pwd)/server" to_fetch &&
 	grep "  fetch" trace &&
 
 	# fetch-pack sends 2 requests each containing 16 "have" lines before
Previous: Johannes Schindelin via GitGitGadgetNext: Junio C Hamano
Message 8 of 43 in “t5552: fix flakiness by introducing proper locking for GIT_TRACE”
  1. 0/4 t5552: fix flakiness by introducing proper locking for GIT_TRACEJohannes Schindelin via GitGitGadget, Aug 9, 2018
  2. 1/4 Introduce a function to lock/unlock file descriptors when appendingJohannes Schindelin via GitGitGadget, Aug 9, 2018
  3. Junio C HamanoAug 9, 2018
  4. Johannes SchindelinAug 10, 2018
  5. 2/4 mingw: implement lock_or_unlock_fd_for_appending()Johannes Schindelin via GitGitGadget, Aug 9, 2018
  6. 3/4 trace: lock the trace file to avoid racy trace_write() callsJohannes Schindelin via GitGitGadget, Aug 9, 2018
  7. 4/4 trace: verify that locking worksJohannes Schindelin via GitGitGadget, Aug 9, 2018
  8. Jeff KingAug 9, 2018
  9. Junio C HamanoAug 9, 2018
  10. Junio C HamanoAug 9, 2018
  11. Jeff KingAug 9, 2018
  12. Jeff KingAug 10, 2018
  13. Junio C HamanoAug 10, 2018
  14. Junio C HamanoAug 10, 2018
  15. Johannes SchindelinAug 10, 2018
  16. Jeff KingAug 10, 2018
  17. Junio C HamanoAug 10, 2018
  18. Johannes SchindelinAug 10, 2018
  19. Johannes SixtAug 10, 2018
  20. Jeff HostetlerAug 10, 2018
  21. Jeff HostetlerAug 10, 2018
  22. Johannes SixtAug 10, 2018
  23. Junio C HamanoAug 10, 2018
  24. Jeff KingAug 10, 2018
  25. mingw: enable atomic O_APPENDJohannes Sixt, Aug 13, 2018
  26. Junio C HamanoAug 13, 2018
  27. Johannes SixtAug 13, 2018
  28. Ævar Arnfjörð BjarmasonAug 13, 2018
  29. Junio C HamanoAug 13, 2018
  30. Jeff KingAug 13, 2018
  31. Ævar Arnfjörð BjarmasonAug 14, 2018
  32. Jeff KingAug 14, 2018
  33. Johannes SixtAug 14, 2018
  34. Jeff KingAug 14, 2018
  35. Jeff HostetlerAug 14, 2018
  36. Junio C HamanoAug 14, 2018
  37. 0/4 t5552: fix flakiness by introducing proper locking for GIT_TRACEJohannes Schindelin via GitGitGadget, Aug 10, 2018
  38. 1/4 Introduce a function to lock/unlock file descriptors when appendingJohannes Schindelin via GitGitGadget, Aug 10, 2018
  39. Junio C HamanoAug 10, 2018
  40. Johannes SchindelinAug 10, 2018
  41. 2/4 mingw: implement lock_or_unlock_fd_for_appending()Johannes Schindelin via GitGitGadget, Aug 10, 2018
  42. 3/4 trace: lock the trace file to avoid racy trace_write() callsJohannes Schindelin via GitGitGadget, Aug 10, 2018
  43. 4/4 trace: verify that locking worksJohannes Schindelin via GitGitGadget, Aug 10, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.