git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: t5570-git-daemon fails with SIGPIPE on OSX

From
SZEDER Gábor <szeder.dev@gmail.com>
Date
Feb 8, 2019, 12:54 UTC
Message-ID
<20190208125435.GW10587@szeder.dev>
In-Reply-To
<nycvar.QRO.7.76.6.1902080928500.41@tvgsbejvaqbjf.bet>
On Fri, Feb 08, 2019 at 09:32:32AM +0100, Johannes Schindelin wrote:
Show 19 quoted lines
> Team,
> 
> On Mon, 6 Aug 2018, SZEDER Gábor wrote:
> 
> > [Resending with Clemens' last used email address.
> > Clemens, please consider sending a patch to update our .mailmap file.]
> > 
> > 
> > On Mon, Aug 6, 2018 at 5:11 PM SZEDER Gábor <szeder.dev@gmail.com> wrote:
> > >
> > > Travis CI changed its default OSX image to use XCode 9.4 on 2018-07-31
> > > [1].  Since then OSX build jobs fail rather frequently because of a
> > > SIGPIPE in the tests 'fetch notices corrupt pack' or 'fetch notices
> > > corrupt idx' in 't5570-git-daemon.sh' [2].  I think this is a symptom
> > > a real bug in Git affecting other platforms as well, but these tests
> > > are too lax to catch it.
> 
> I am seeing this very frequently now, as it feels like failing in the
> Azure Pipeline about half of the time.

I was wondering whether it's only an issue on Travis CI, and was waiting whether you'll complain about it :) Evidently it is not, but I still would like to see a report about macOS running directly on someone's Mac hardware, i.e. without all the CI/cloud/whatnot magic.

> Any ideas how to fix this test, anyone?

I'm afraid that this is not merely an issue with the test, but a platform issue that we should work around somehow. I would have suggested to follow up on what Peff suggested, but I see that you already did, and it didn't work out...

In the meantime, for lack of a better option, I started to skip the two failure-prone tests in the OSX build jobs in my automated CI builds with:

 -- >8 --
Subject: [PATCH] travis-ci: skip flaky tests in 't5570-git-daemon.sh' in OSX
 build jobs
See: https://public-inbox.org/git/CAM0VKj=MCS+cmOgzf_XyPeb+qZrFmuMH52-PV_NDMZA9X+rRoA@mail.gmail.com/
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
---
 ci/lib-travisci.sh | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/ci/lib-travisci.sh b/ci/lib-travisci.sh
index 69dff4d1ec..1cd2c1db7d 100755
--- a/ci/lib-travisci.sh
+++ b/ci/lib-travisci.sh
@@ -121,7 +121,9 @@ osx-clang|osx-gcc)
 	# t9810 occasionally fails on Travis CI OS X
 	# t9816 occasionally fails with "TAP out of sequence errors" on
 	# Travis CI OS X
-	export GIT_SKIP_TESTS="t9810 t9816"
+	# In 't5570-git-daemon.sh', the tests 'fetch notices corrupt pack'
+	# and 'fetch notices corrupt idx' fail rather frequently.
+	export GIT_SKIP_TESTS="t9810 t9816 t5570.9 t5570.10"
 	;;
 GIT_TEST_GETTEXT_POISON)
 	export GIT_TEST_GETTEXT_POISON=YesPlease

-- 
2.20.1.940.g8404bb2d1a

 -- >8 --


> > > What it boils down to is this sequence:
> > >
> > >   - The test first prepares a repository containing a corrupt pack,
> > >     ready to be server via 'git daemon'.
> > >
> > >   - Then the test runs 'test_must_fail git fetch ....', which connects
> > >     to 'git daemon', which forks 'git upload-pack', which then
> > >     advertises refs (only HEAD) and capabilities.  So far so good.
> > >
> > >   - 'git fetch' eventually calls fetch-pack.c:find_common().  The
> > >     first half of this function assembles a request consisting of a
> > >     want and a flush pkt-line, and sends it via a send_request() call.
> > >
> > >     At this point the scheduling becomes important: let's suppose that
> > >     fetch is slow and upload-pack is fast.
> > >
> > >   - 'git upload-pack' receives the request, parses the want line,
> > >     notices the corrupt pack, responds with an 'ERR upload-pack: not
> > >     our ref' pkt-line, and die()s right away.
> > >
> > >   - 'git fetch' finally approaches the end of the function, where it
> > >     attempts to send a done pkt-line via another send_request() call
> > >     through the now closing TCP socket.
> > >
> > >   - What happens now seems to depend on the platform:
> > >
> > >     - On Linux, both on my machine and on Travis CI, it shows textbook
> > >       example behaviour: write() returns with error and sets errno to
> > >       ECONNRESET.  Since it happens in write_or_die(), 'git fetch'
> > >       die()s with 'fatal: write error: Connection reset by peer', and
> > >       doesn't show the error send by 'git upload-pack'; how could it,
> > >       it doesn't even get as far to receive upload-pack's ERR
> > >       pkt-line.
> > >
> > >       The test only checks that 'git fetch' fails, but it doesn't
> > >       check whether it failed with the right error message, so the
> > >       test still succeeds.  Had it checked the error message as well,
> > >       we most likely had noticed this issue already, it doesn't happen
> > >       all that rarely.
> > >
> > >     - On the new OSX images with XCode 9.4 on Travis CI the write()
> > >       triggers SIGPIPE right away, and 'test_must_fail' notices it and
> > >       fails the test.  I couldn't see any sign of an ECONNRESET or any
> > >       other error that we could act upon to avoid the SIGPIPE.
> > >
> > >     - On OSX with XCode 9.2 on Travis CI there is neither SIGPIPE, nor
> > >       ECONNRESET, but sending the request actually succeeds even
> > >       though there is no process on the other end of the socket
> > >       anymore.  'git fetch' then simply continues execution, reads and
> > >       parses the ERR pkt-line, and then dies()s with 'fatal: remote
> > >       error: upload-pack: not our ref'.  So, on the face of it, it
> > >       shows the desired behaviour, but I have no idea how that write()
> > >       could succeed instead of returning error.
> > >
> > > I don't know what happens on a real Mac as I don't have access to one;
> > > I figured out all the above by enabling packet tracing, adding a
> > > couple of well placed tracing printf() and sleep() calls, running a
> > > bunch of builds on Travis CI, and looking through their logs.  But
> > > without access to a debugger and netstat and what not I can't really
> > > go any further.  So I would now happily pass the baton to those who
> > > have a Mac and know a thing or two about its porting issues to first
> > > check whether OSX on a real Mac shows the same behaviour as it does in
> > > Travis CI's virtualized(?) environment.  And then they can pass the
> > > baton to those who know all the intricacies of the pack protocol and
> > > its implementation to decide what to do with this issue.
> > >
> > > For a mostly reliable reproduction recipe you might want to fetch this
> > > branch:
> > >
> > >   https://github.com/szeder/git t5570-git-daemon-sigpipe
> > >
> > > and then run 'make && cd t && ./t5570-git-daemon.sh -v -x'
> > >
> > >
> > > Have fun! ;)
> > >
> > >
> > > 1 - https://blog.travis-ci.com/2018-07-19-xcode9-4-default-announce
> > >
> > > 2 - On git.git's master:
> > >     https://travis-ci.org/git/git/jobs/411517552#L2717
> > 
> >
Previous: Johannes SchindelinNext: Jeff King
Message 4 of 21 in “t5570-git-daemon fails with SIGPIPE on OSX”
  1. SZEDER GáborAug 6, 2018
  2. SZEDER GáborAug 6, 2018
  3. Johannes SchindelinFeb 8, 2019
  4. SZEDER GáborFeb 8, 2019
  5. Jeff KingAug 14, 2018
  6. Jeff KingAug 14, 2018
  7. Johannes SchindelinFeb 8, 2019
  8. Johannes SchindelinFeb 8, 2019
  9. Jeff KingFeb 8, 2019
  10. Johannes SchindelinMar 1, 2019
  11. Jeff KingMar 1, 2019
  12. Johannes SchindelinMar 2, 2019
  13. Jeff KingMar 3, 2019
  14. 1/2 fetch: avoid calling write_or_die()Jeff King, Mar 3, 2019
  15. Duy NguyenMar 4, 2019
  16. Jeff KingMar 5, 2019
  17. 2/2 fetch: ignore SIGPIPE during network operationJeff King, Mar 3, 2019
  18. Junio C HamanoMar 4, 2019
  19. Jeff KingMar 5, 2019
  20. Junio C HamanoMar 3, 2019
  21. Johannes SchindelinMar 3, 2019

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.