git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] mingw: handle writes to non-blocking pipe

From
René Scharfe <l.s.r@web.de>
Date
Aug 11, 2022, 17:35 UTC
Message-ID
<976ac297-28ec-0a38-c4e1-eb7b94d0eb8c@web.de>
In-Reply-To
<YvTCIVN2VBir7WEP@coredump.intra.peff.net>
Am 11.08.2022 um 10:47 schrieb Jeff King:
Show 42 quoted lines
> On Thu, Aug 11, 2022 at 12:34:46AM +0200, René Scharfe wrote:
>
>>> OK, so we call GetNamedPipeInfo() to find the size of the pipe buffer.
>>> It's unclear to me from Microsoft's docs if that is the _total_ size, or
>>> if it's the remaining available size. Hopefully the latter, since none
>>> of this works otherwise. ;)
>>>
>>> But two corner cases:
>>>
>>>   - If we fail to get the size, we guess that it's the maximum. Is this
>>>     dangerous? I'm not sure why the call would fail, but if for some
>>>     reason it did fail and we can't make forward progress, would we
>>>     enter an infinite recursion of mingw_write()? Would it be safer to
>>>     bail with EAGAIN in such a case (through granted, that probably just
>>>     puts us into an infinite loop in xwrite())?
>>
>> AFAIU it's the total size, not the available space.  I think I confused
>> it with PIPE_BUF, which we should use instead.
>
> Hmm. If it's giving us the total size, that seems like it may fail in a
> case like this:
>
>   - we write a smaller amount to the pipe, say 7168 bytes. That leaves
>     1024 bytes in the buffer. The reader reads nothing yet.
>
>   - now we try to write another 4096 bytes. That's too big, so we get
>     ENOSPC. But when we ask for the pipe size, it tells us 8192. So we
>     _don't_ try to do a partial write of the remaining size, and return
>     EAGAIN. But now we've made no forward progress (i.e., even though
>     poll() said we could write, we don't, and we end up in the xwrite
>     loop).
>
> So we really do want to try to get a partial write. Even a single byte
> means we are making forward progress.
>
>> Alternatively we could retry with ever smaller sizes, down to one byte,
>> to avoid EAGAIN as much as possible.  Sounds costly, though.
>
> It's definitely not optimal, but it may not be too bad. If we cut the
> size in half each time, then at worst we make log2(N) extra write
> attempts, and we end up with a partial write within 50% of the optimal
> size.

OK, but we can't just split anything up as we like: POSIX wants us to keep writes up to PIPE_BUF atomic. When I read that name I mistakenly thought it was the size of the pipe buffer, but it's a different value. The minimum value according to POSIX is 512 bytes; on Linux it's 4KB.

And Windows doesn't seem to define it. Bash's ulimit -p returns 8, which is in units of 512 bytes, so it's 4KB like on Linux. But that's apparently not respected by Windows' write.

I just realized that we can interprete the situation slightly differently. Windows has effectively PIPE_BUF = 2^32, which means all writes are atomic. I don't see POSIX specifying a maximum allowed value, so this must be allowed. Which means you cannot rely on write performing partial writes. Makes sense?

If we accept that, then we need a special write function for non-blocking pipes that chops the data into small enough chunks.

Another oddity: t3701 with yesterday's patch finishes fine with -i, but hangs without it (not just when running it via prove). O_o

René
Previous: Jeff KingNext: Jeff King
Message 18 of 44 in “pipe_command(): mark stdin descriptor as non-blocking”
  1. pipe_command(): mark stdin descriptor as non-blockingJeff King, Aug 2, 2022
  2. Junio C HamanoAug 2, 2022
  3. Jeff KingAug 2, 2022
  4. Junio C HamanoAug 2, 2022
  5. pipe_command(): mark stdin descriptor as non-blockingJeff King, Aug 3, 2022
  6. René ScharfeAug 3, 2022
  7. Jeff KingAug 3, 2022
  8. René ScharfeAug 3, 2022
  9. Jeff KingAug 5, 2022
  10. René ScharfeAug 5, 2022
  11. René ScharfeAug 7, 2022
  12. Jeff KingAug 7, 2022
  13. mingw: handle writes to non-blocking pipeRené Scharfe, Aug 10, 2022
  14. Johannes SchindelinAug 10, 2022
  15. Jeff KingAug 10, 2022
  16. René ScharfeAug 10, 2022
  17. Jeff KingAug 11, 2022
  18. René ScharfeAug 11, 2022
  19. Jeff KingAug 11, 2022
  20. René ScharfeAug 14, 2022
  21. Jeff KingAug 17, 2022
  22. 0/6 fix pipe_command() deadlockJeff King, Aug 17, 2022
  23. 1/6 compat: add function to enable nonblocking pipesJeff King, Aug 17, 2022
  24. Junio C HamanoAug 17, 2022
  25. Jeff KingAug 18, 2022
  26. 2/6 nonblock: support WindowsJeff King, Aug 17, 2022
  27. 3/6 git-compat-util: make MAX_IO_SIZE define globally availableJeff King, Aug 17, 2022
  28. 4/6 pipe_command(): avoid xwrite() for writing to pipeJeff King, Aug 17, 2022
  29. 5/6 pipe_command(): handle ENOSPC when writing to a pipeJeff King, Aug 17, 2022
  30. Junio C HamanoAug 17, 2022
  31. Jeff KingAug 18, 2022
  32. 6/6 pipe_command(): mark stdin descriptor as non-blockingJeff King, Aug 17, 2022
  33. Jeff KingAug 17, 2022
  34. René ScharfeAug 19, 2022
  35. Jeff KingAug 20, 2022
  36. René ScharfeAug 10, 2022
  37. Jeff KingAug 10, 2022
  38. René ScharfeAug 10, 2022
  39. Jeff KingAug 11, 2022
  40. René ScharfeAug 7, 2022
  41. Johannes SchindelinAug 8, 2022
  42. Johannes SchindelinAug 8, 2022
  43. Jeff KingAug 9, 2022
  44. Johannes SchindelinAug 9, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.