{"thread":{"id":"60263","subject":"Issues with git clone over HTTP/2 and closed connections","startedAt":"2023-09-23T13:06:15Z","lastAt":"2023-09-24T14:48:02Z","messageCount":3,"participants":["David Härdeman","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"482209","messageId":"bb757ebd66b5ac4c81d62b01d5cff2f75250090d@hardeman.nu","threadId":"60263","inReplyTo":null,"subject":"Issues with git clone over HTTP/2 and closed connections","fromName":"David Härdeman","fromEmail":"david@hardeman.nu","sentAt":"2023-09-23T12:58:09Z","receivedAt":"2023-09-23T13:06:15Z","isPatch":false,"sender":{"key":"david@hardeman.nu","avatar":"https://gravatar.com/avatar/9d166a5d9d3ce5c7aea74a0080677a73785727b9b25f0c3e547d5b359bff27d5?d=mp&s=160"},"body":"Hi,\n\nI just tried to clone a repo from a server over HTTPS, which failed with a message like this:\n\n error: (curl_result = 55, http_code = 0, sha1 = \n error: Unable to find under https://example.com/myrepo.git\n Fetching objects: 20790, done.\n Cannot obtain needed tree \n while processing commit \n error: fetch failed.\n\nEvery time I retried cloning, and changed, but the error message was the same.\n\nBy running \"GIT_CURL_VERBOSE=1 git clone https://example.com/myrepo.git\", I noticed that:\n\n a) HTTP/2 was being used; and\n b) just before the error the server returned a GOAWAY [1]:\n \"== Info: received GOAWAY, error=0, last_stream=1999\"\n\nOn the client side I'm using Debian Unstable (libcurl 8.3.0, git 2.40.1), and the server is running Debian Stable (nginx 1.22.1-9).\n\nnginx will, by default, close HTTP/2 connections after \"http2_max_requests\", (default: 1000, i.e. 1999 streams, note that the error message above says last_stream=1999) and it seems that it is using GOAWAY to do so, which seems to confuse git/libcurl.\n\nAnd sure enough, after running \"git config --global http.version HTTP/1.1\" on the client and trying again, the \"git clone\" was successful (I'm guessing I could/should also bump http2_max_requests on the server).\n\nFrom what I understand, git should close the connection, try to open a new one and resume the clone operation before erroring out (because the GOAWAY message could mean anything).\n\nIs this a known bug and is it something that would need to be fixed in libcurl or in git?\n\nCheers,\nDavid\n\nPS. Not subscribed, please CC: me on any replies.\n\n[1] https://www.rfc-editor.org/rfc/rfc7540#section-6.8\n[2] http://nginx.org/en/docs/http/ngx_http_v2_module.html#http2_max_requests\n"},{"id":"482218","messageId":"20230924035022.GA1503477@coredump.intra.peff.net","threadId":"60263","inReplyTo":"bb757ebd66b5ac4c81d62b01d5cff2f75250090d@hardeman.nu","subject":"Re: Issues with git clone over HTTP/2 and closed connections","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-09-24T03:50:22Z","receivedAt":"2023-09-24T04:25:03Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Sep 23, 2023 at 12:58:09PM +0000, David Härdeman wrote:\n\n> By running \"GIT_CURL_VERBOSE=1 git clone https://example.com/myrepo.git\", I noticed that:\n> \n> a) HTTP/2 was being used; and\n> b) just before the error the server returned a GOAWAY [1]:\n> \"== Info: received GOAWAY, error=0, last_stream=1999\"\n> \n> On the client side I'm using Debian Unstable (libcurl 8.3.0, git\n> 2.40.1), and the server is running Debian Stable (nginx 1.22.1-9).\n> \n> nginx will, by default, close HTTP/2 connections after\n> \"http2_max_requests\", (default: 1000, i.e. 1999 streams, note that the\n> error message above says last_stream=1999) and it seems that it is\n> using GOAWAY to do so, which seems to confuse git/libcurl.\n> \n> And sure enough, after running \"git config --global http.version\n> HTTP/1.1\" on the client and trying again, the \"git clone\" was\n> successful (I'm guessing I could/should also bump http2_max_requests\n> on the server).\n\nThanks for a detailed report. Your analysis all makes sense to me.\n\n> From what I understand, git should close the connection, try to open a\n> new one and resume the clone operation before erroring out (because\n> the GOAWAY message could mean anything).\n> \n> Is this a known bug and is it something that would need to be fixed in\n> libcurl or in git?\n\nI don't think we've heard of such a problem before with Git. I don't\nknow enough about GOAWAY to comment on the correct behavior, but this is\nalmost certainly a curl issue, not a Git one. All of the connection\nhandling, reuse, etc, is happening invisibly at the curl layer.\n\nIt's probably worth poking around libcurl's issue tracker. This seems\nlike it might be related:\n\n https://github.com/curl/curl/issues/11859\n\nAnd one final comment: 2000 is a lot of requests for one clone. That\nplus the error you are seeing from Git makes me think you're using the\n\"dumb\" http protocol (i.e., your webserver is not set up to run the\nserver side of Git's smart protocol, so it is just serving files\nblindly).\n\nI don't know if using it is intentional or not. But the smart protocol\nis much more efficient, and in general I would expect it to have fewer\ncorner cases (none of the major forges allow dumb-http at all).\n\nYou can find more details on setting it up in \"git help http-backend\".\n\nIf you do want to keep using the dumb protocol, consider running \"git\ngc\" on the server side repository. 2000 requests implies you have many\nloose objects, which could be served much more efficiently as a single\npack.\n\n-Peff\n"},{"id":"482229","messageId":"4454d8e1dab565118a316409d653844bbbfacfc7@hardeman.nu","threadId":"60263","inReplyTo":"20230924035022.GA1503477@coredump.intra.peff.net","subject":"Re: Issues with git clone over HTTP/2 and closed connections","fromName":"David Härdeman","fromEmail":"david@hardeman.nu","sentAt":"2023-09-24T14:47:56Z","receivedAt":"2023-09-24T14:48:02Z","isPatch":false,"sender":{"key":"david@hardeman.nu","avatar":"https://gravatar.com/avatar/9d166a5d9d3ce5c7aea74a0080677a73785727b9b25f0c3e547d5b359bff27d5?d=mp&s=160"},"body":"September 24, 2023 at 5:50 AM, \"Jeff King\" wrote:\n> On Sat, Sep 23, 2023 at 12:58:09PM +0000, David Härdeman wrote:\n>> From what I understand, git should close the connection, try to open a\n>> new one and resume the clone operation before erroring out (because\n>> the GOAWAY message could mean anything).\n>> \n>> Is this a known bug and is it something that would need to be fixed in\n>> libcurl or in git?\n>> \n> \n> I don't think we've heard of such a problem before with Git. I don't\n> know enough about GOAWAY to comment on the correct behavior, but this is\n> almost certainly a curl issue, not a Git one. All of the connection\n> handling, reuse, etc, is happening invisibly at the curl layer.\n> \n> It's probably worth poking around libcurl's issue tracker. This seems\n> like it might be related:\n> \n> https://github.com/curl/curl/issues/11859\n\nYeah, looks very relevant. I'll keep an eye on that issue instead.\n\nThanks for the prompt feedback.\n\n> And one final comment: 2000 is a lot of requests for one clone. That\n> plus the error you are seeing from Git makes me think you're using the\n> \"dumb\" http protocol (i.e., your webserver is not set up to run the\n> server side of Git's smart protocol, so it is just serving files\n> blindly).\n\nYeah, thanks for the advice...I've already sorted this out so that I'm not\naffected, but I wanted to make sure that I posted the bug report before I\nforgot all the details.\n\nCheers,\nDavid\n"}]}