Re: curl 8.10.0 regression breaks uploads with HTTP/2 and http.postbuffer

From: Patrick Steinhardt <ps@pks.im>
Date: Sep 13, 2024, 08:20 UTC
Message-ID: <ZuP168QTTMiv_DxH@pks.im>
In-Reply-To: <565691o1-3451-o06o-2594-2750r90nqq6p@unkk.fr>

On Fri, Sep 13, 2024 at 09:49:27AM +0200, Daniel Stenberg wrote:

Show 13 quoted lines

> On Fri, 13 Sep 2024, Patrick Steinhardt wrote:
> 
> > In a nutshell:
> 
> Thanks, this is helpful.
> 
> >  - We then clone a repository from Apache with http.postbuffer=65536,
> >    which makes us use a small buffer when POSTing data via curl. We
> >    typically use 1MB buffers, and when changing it back to 1MB instead
> >    of 65kB the test works just fine.
> 
> Is this a git buffer size or is this a value you tell libcurl in an option
> to set a buffer size?

I'm not all that familiar with the "remote-curl.c" remote helper in Git, so let me try to figure out things as we go.

  - The code that sets up the POST buffer is `stateless_connect()`. The
    buffer is allocated by ourselves.

  - We then execute `post_rpc()` in a loop until we see EOF.

  - `post_rpc()` itself is doing all the work to set up the curl handle,
    mostly via calls to `curl_easy_setopt()`.

  - In there we hit the `large_request` code path. We set up
    CURLOPT_READFUNCTION and CURLOPT_SEEKFUNCTION. The callback that
    uses our buffer is the one set up via CURLOPT_READFUNCTION, which is
    `rpc_out()`.

Whether or not we hit `large_request` depends on out POST buffer size. We first try to read all the data we want to send into the buffer, and if it fits we send it out in a single call to curl by setting up CURLOPT_POSTFIELDS and CURLOPT_POSTFIELDSIZE_LARGE. If it doesn't fit into the buffer, which is the case for in this testcase, we instead use the callbacks to write data via curl.

Show 12 quoted lines

> > I've appended two curl traces, the working one with 1MB buffers and the
> > failing one with 65kB buffers. I hope that helps.
> 
> How are you feeding the data to libcurl? (callback or by setting the
> postfields option?) I noticed that in the working case log, the POST
> requests always have a content-length header while the failing case log
> shows that header lacking in the final POST request.
> 
> Is that on purpose?
> 
> libcurl should still handle it fine, it might just be a clue for me to
> narrow down my search.

I think so. We're using a "chunked" transfer encoding in the `large_request` case and do not yet know how much data we are about to send. We'll only figure that out as we go.

Patrick