tests broken with curl-8.7.0

I noticed some http-related failures in the test suite on my Debian unstable system, which recently got an upgraded curl package. It looks like it's related to cases where we use the remote-curl helper in "connect" mode (i.e., protocol v2) and the http buffer is small (requiring us to stream the data to curl). Besides just running t5551, an easy reproduction is:

  [this works]
  $ git ls-remote https://git.kernel.org/pub/scm/git/git.git | wc -l
  1867

  [this doesn't]
  $ git -c http.postbuffer=65536 ls-remote https://git.kernel.org/pub/scm/git/git.git
  fatal: expected flush after ref listing

The error message comes from ls-remote itself, which was expecting a FLUSH packet from the remote. Instead it gets the RESPONSE_END from remote-curl (remember that in connect mode, remote-curl is just ferrying bytes back and forth between ls-remote and the server).

It works with older versions of libcurl, but not 8.7.0 (or 8.7.1). Bisecting in libcurl points to 9369c30cd (lib: Curl_read/Curl_write clarifications, 2024-02-15).

Running with GIT_TRACE_CURL=1 shows weirdness on the POST we send to issue the ls-refs command. With older curl, I see this:

  => Send header: POST /pub/scm/git/git.git/git-upload-pack HTTP/1.1
  => Send header: Host: git.kernel.org
  => Send header: User-Agent: git/2.44.0.789.g252ee96bc5.dirty
  => Send header: Accept-Encoding: deflate, gzip
  => Send header: Content-Type: application/x-git-upload-pack-request
  => Send header: Accept: application/x-git-upload-pack-result
  => Send header: Git-Protocol: version=2
  => Send header: Transfer-Encoding: chunked
  => Send header:
  => Send data: 14..0014command=ls-refs...
  => Send data: 2a..002aagent=git/2.44.0.789.g252ee96bc5.dirty..
  [and so on until...]
  == Info: Signaling end of chunked upload via terminating chunk.

But with the broken version, I get:

  => Send header: POST /pub/scm/git/git.git/git-upload-pack HTTP/1.1
  => Send header: Host: git.kernel.org
  => Send header: User-Agent: git/2.44.0.789.g252ee96bc5.dirty
  => Send header: Accept-Encoding: deflate, gzip, br, zstd
  => Send header: Content-Type: application/x-git-upload-pack-request
  => Send header: Accept: application/x-git-upload-pack-result
  => Send header: Git-Protocol: version=2
  => Send header: Transfer-Encoding: chunked
  => Send header:
  => Send data, 0000000014 bytes (0x0000000e)
  => Send data: 4..0014..0....
  == Info: upload completely sent off: 14 bytes

So we only get the first 4 bytes, and then we quit (the double mention of 14 is confusing, but I think it is both the size of the pkt-line ("command=ls-refs\n") but also the length of the 4-byte string when framed with chunked transfer-encoding). Those 4 bytes are the first thing returned by rpc_out(), which we use as our CURLOPT_READFUNCTION.

It's possible that we're doing something wrong with our read/write function callbacks. But I don't see how; we say "here's 4 bytes", but then we never get called again. It's like curl is giving up on trying to read the post input early for some reason.

I'm not sure how to dig further. That commit is pretty big and scary. I did check that the tip of master in curl.git is still affected (I'd hoped maybe the 0-length write fixes in b30d694a027 would be related, but that's not it).

Ideas?

-Peff