git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2 4/6] pipe_command(): avoid xwrite() for writing to pipe

From
Jeff King <peff@peff.net>
Date
Aug 17, 2022, 06:08 UTC
Message-ID
<YvyFxry9+2x05Di3@coredump.intra.peff.net>
In-Reply-To
<YvyE1RDFVAg4uQo1@coredump.intra.peff.net>

If xwrite() sees an EAGAIN response, it will loop forever until the write succeeds (or encounters a real error). This is due to ef1cf0167a (xwrite: poll on non-blocking FDs, 2016-06-26), with the idea that we won't be surprised by a descriptor unexpectedly set as non-blocking.

But that will make things awkward when we do want a non-blocking descriptor, and a future patch will switch pipe_command() to using one. In that case, looping on EAGAIN is bad, because the process on the other end of the pipe may be waiting on us before doing another read() on the pipe, which would mean we deadlock.

In practice we're not supposed to ever see EAGAIN here, since poll() will have just told us the descriptor is ready for writing. But our Windows emulation of poll() will always return "ready" for writing to a pipe descriptor! This is due to 94f4d01932 (mingw: workaround for hangs when sending STDIN, 2020-02-17).

Our best bet in that case is to keep handling other descriptors, as any read() we do may allow the child command to make forward progress (i.e., its write() finishes, and then it read()s from its stdin, freeing up space in the pipe buffer). This means we might busy-loop between poll() and write() on Windows if the child command is slow to read our input, but it's much better than the alternative of deadlocking.

In practice, this busy-looping should be rare:
  - for small inputs, we'll just write the whole thing in a single
    write() anyway, non-blocking or not
  - for larger inputs where the child reads input and then processes it
    before writing (e.g., gpg verifying a signature), we may make a few
    extra write() calls that get EAGAIN during the initial write, but
    once it has taken in the whole input, we'll correctly block waiting
    to read back the data.
  - for larger inputs where the child process is streaming output back
    (like a diff filter), we'll likewise see some extra EAGAINs, but
    most of them will be followed immediately by a read(), which will
    let the child command make forward progress.

Of course it won't happen at all for now, since we don't yet use a non-blocking pipe. This is just preparation for when we do.

Helped-by: René Scharfe <l.s.r@web.de>
Signed-off-by: Jeff King <peff@peff.net>
---
 run-command.c | 22 +++++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)
diff --git a/run-command.c b/run-command.c
index 14f17830f5..e078c3046f 100644
--- a/run-command.c
+++ b/run-command.c
@@ -1364,12 +1364,24 @@ static int pump_io_round(struct io_pump *slots, int nr, struct pollfd *pfd)
 			continue;
 
 		if (io->type == POLLOUT) {
-			ssize_t len = xwrite(io->fd,
-					     io->u.out.buf, io->u.out.len);
+			ssize_t len;
+
+			/*
+			 * Don't use xwrite() here. It loops forever on EAGAIN,
+			 * and we're in our own poll() loop here.
+			 *
+			 * Note that we lose xwrite()'s handling of MAX_IO_SIZE
+			 * and EINTR, so we have to implement those ourselves.
+			 */
+			len = write(io->fd, io->u.out.buf,
+				    io->u.out.len <= MAX_IO_SIZE ?
+				    io->u.out.len : MAX_IO_SIZE);
 			if (len < 0) {
-				io->error = errno;
-				close(io->fd);
-				io->fd = -1;
+				if (errno != EINTR && errno != EAGAIN) {
+					io->error = errno;
+					close(io->fd);
+					io->fd = -1;
+				}
 			} else {
 				io->u.out.buf += len;
 				io->u.out.len -= len;
-- 
2.37.2.881.gb57357660c
Previous: Jeff KingNext: Jeff King
Message 28 of 44 in “pipe_command(): mark stdin descriptor as non-blocking”
  1. pipe_command(): mark stdin descriptor as non-blockingJeff King, Aug 2, 2022
  2. Junio C HamanoAug 2, 2022
  3. Jeff KingAug 2, 2022
  4. Junio C HamanoAug 2, 2022
  5. pipe_command(): mark stdin descriptor as non-blockingJeff King, Aug 3, 2022
  6. René ScharfeAug 3, 2022
  7. Jeff KingAug 3, 2022
  8. René ScharfeAug 3, 2022
  9. Jeff KingAug 5, 2022
  10. René ScharfeAug 5, 2022
  11. René ScharfeAug 7, 2022
  12. Jeff KingAug 7, 2022
  13. mingw: handle writes to non-blocking pipeRené Scharfe, Aug 10, 2022
  14. Johannes SchindelinAug 10, 2022
  15. Jeff KingAug 10, 2022
  16. René ScharfeAug 10, 2022
  17. Jeff KingAug 11, 2022
  18. René ScharfeAug 11, 2022
  19. Jeff KingAug 11, 2022
  20. René ScharfeAug 14, 2022
  21. Jeff KingAug 17, 2022
  22. 0/6 fix pipe_command() deadlockJeff King, Aug 17, 2022
  23. 1/6 compat: add function to enable nonblocking pipesJeff King, Aug 17, 2022
  24. Junio C HamanoAug 17, 2022
  25. Jeff KingAug 18, 2022
  26. 2/6 nonblock: support WindowsJeff King, Aug 17, 2022
  27. 3/6 git-compat-util: make MAX_IO_SIZE define globally availableJeff King, Aug 17, 2022
  28. 4/6 pipe_command(): avoid xwrite() for writing to pipeJeff King, Aug 17, 2022
  29. 5/6 pipe_command(): handle ENOSPC when writing to a pipeJeff King, Aug 17, 2022
  30. Junio C HamanoAug 17, 2022
  31. Jeff KingAug 18, 2022
  32. 6/6 pipe_command(): mark stdin descriptor as non-blockingJeff King, Aug 17, 2022
  33. Jeff KingAug 17, 2022
  34. René ScharfeAug 19, 2022
  35. Jeff KingAug 20, 2022
  36. René ScharfeAug 10, 2022
  37. Jeff KingAug 10, 2022
  38. René ScharfeAug 10, 2022
  39. Jeff KingAug 11, 2022
  40. René ScharfeAug 7, 2022
  41. Johannes SchindelinAug 8, 2022
  42. Johannes SchindelinAug 8, 2022
  43. Jeff KingAug 9, 2022
  44. Johannes SchindelinAug 9, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.