git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH/RFC] Allow curl to rewind the RPC read buffer at any time

From
Shawn O. Pearce <spearce@spearce.org>
Date
Dec 1, 2009, 16:14 UTC
Message-ID
<20091201161428.GC21299@spearce.org>
In-Reply-To
<alpine.DEB.2.00.0912011236360.5582@cone.home.martin.st>
Martin Storsj? <martin@martin.st> wrote:
Show 6 quoted lines
> When using multi-pass authentication methods, the curl library may
> need to rewind the read buffers used for providing data to HTTP POST,
> if data has been output before a 401 error is received.
> 
> This solution buffers all data read by the curl library, in order to allow
> it to rewind the reading buffer at any time later.
NAK.

In the case of git-upload-pack requests, we should fit into 1 MiB almost all of the time, and thus not need to grow the http.postBuffer to support a rewind. The state data plus current have list isn't all that large. A 1 MiB request means we have over 20,900 commits in common with the remote and still haven't been able to find a sufficient cut point. Or the remote has 20,000 active, unrelated branches we are trying to fetch. Either way, this is a really sick and twisted situation.

In the case of git-receive-pack requests, we might be uploading an entire project to an empty repository on the remote side. This could be 8 GiB worth of data if the project was something huge like KDE. We can't assume that we should malloc 8 GiB of memory to buffer the payload.

The *correct* way to support an arbitrary rewind is to modify the outgoing channel from remote-curl to its protocol engine (client.in within the rpc_service method) to somehow request the protocol engine (aka git-send-pack or git-fetch-pack) to stop and regenerate the current request.

Another approach would be to modify http-backend (and the protocol) to support an "auth ping" request prior to spooling out the entire payload if its more than an http.postBuffer size. Basically we do what the "Expect: 100-continue" protocol is supposed to do, but in the application layer rather than the HTTP/1.1 layer, so our CGI actually gets invoked.

This unfortunately still relies on the underlying libcurl to not discard the authentication data after that initial "auth ping". But to be honest, I think that is a reasonable expectation. The #@!*@!* library should be able to generate two requests back-to-back to the same URL without needing to rewind the 2nd request.

-- 
Shawn.
Previous: Martin StorsjöNext: Martin Storsjö
Message 15 of 22 in “Add an option for using any HTTP authentication scheme, not only basic”
  1. Add an option for using any HTTP authentication scheme, not only basicMartin Storsjö, Apr 14, 2009
  2. 0/2 http: allow multi-pass authenticationTay Ray Chuan, Nov 27, 2009
  3. 1/2 http: maintain curl sessionsTay Ray Chuan, Nov 27, 2009
  4. 2/2 Add an option for using any HTTP authentication scheme, not only basicTay Ray Chuan, Nov 27, 2009
  5. Martin StorsjöDec 1, 2009
  6. Allow curl to rewind the RPC read bufferMartin Storsjö, Dec 1, 2009
  7. Shawn O. PearceDec 1, 2009
  8. Tay Ray ChuanDec 1, 2009
  9. Shawn O. PearceDec 1, 2009
  10. Martin StorsjöDec 1, 2009
  11. Junio C HamanoDec 1, 2009
  12. Tay Ray ChuanDec 2, 2009
  13. Martin StorsjöDec 2, 2009
  14. Allow curl to rewind the RPC read buffer at any timeMartin Storsjö, Dec 1, 2009
  15. Shawn O. PearceDec 1, 2009
  16. Martin StorsjöDec 1, 2009
  17. Tay Ray ChuanDec 2, 2009
  18. Daniel StenbergDec 1, 2009
  19. Tay Ray ChuanDec 2, 2009
  20. Daniel StenbergDec 2, 2009
  21. Martin StorsjöDec 2, 2009
  22. Daniel StenbergDec 2, 2009

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.