{"thread":{"id":"34790","subject":"http.postBuffer set at the server side?","startedAt":"2013-08-28T23:08:02Z","lastAt":"2013-08-29T06:31:02Z","messageCount":4,"participants":["Pyeron, Jason J CTR (US)","Jeff King","Jason Pyeron"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"226130","messageId":"871B6C10EBEFE342A772D1159D132085416127CF@umechphj.easf.csd.disa.mil","threadId":"34790","inReplyTo":null,"subject":"http.postBuffer set at the server side?","fromName":"Pyeron, Jason J CTR (US)","fromEmail":"jason.j.pyeron.ctr@mail.mil","sentAt":"2013-08-28T23:08:02Z","receivedAt":"2013-08-28T23:08:02Z","isPatch":false,"sender":{"key":"jason.j.pyeron.ctr@mail.mil","avatar":null},"body":"We have systems hosting git which are behind proxies, and unless the client sets the http.postBuffer to a large size they connections fails.\n\nIs there a way to set this on the server side? If not would a patch be possible to fix this?\n\njason.pyeron@hostname /home/jason.pyeron/desktop/projectname\n$ git push remote --all\nUsername for 'https://server.fqdn':\nPassword for 'https://jpyeron@server.fqdn':\nCounting objects: 1820, done.\nDelta compression using up to 4 threads.\nCompressing objects: 100% (1276/1276), done.\nerror: RPC failed; result=22, HTTP code = 411\nfatal: The remote end hung up unexpectedly\nWriting objects: 100% (1820/1820), 17.72 MiB | 5.50 MiB/s, done.\nTotal 1820 (delta 527), reused 26 (delta 6)\nfatal: The remote end hung up unexpectedly\n\njason.pyeron@hostname /home/jason.pyeron/desktop/projectname\n$ git config http.postBuffer 524288000\n\njason.pyeron@hostname /home/jason.pyeron/desktop/projectname\n$ git push remote --all\nUsername for 'https://server.fqdn':\nPassword for 'https://jpyeron@server.fqdn':\nCounting objects: 1820, done.\nDelta compression using up to 4 threads.\nCompressing objects: 100% (1276/1276), done.\nWriting objects: 100% (1820/1820), 17.72 MiB | 11.31 MiB/s, done.\nTotal 1820 (delta 519), reused 26 (delta 6)\nTo https://server.fqdn/git/netasset-portal/\n * [new branch]      master -> master\n\n"},{"id":"226134","messageId":"20130829035209.GC22788@sigill.intra.peff.net","threadId":"34790","inReplyTo":"871B6C10EBEFE342A772D1159D132085416127CF@umechphj.easf.csd.disa.mil","subject":"Re: http.postBuffer set at the server side?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-08-29T03:52:09Z","receivedAt":"2013-08-29T03:52:09Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Aug 28, 2013 at 11:08:02PM +0000, Pyeron, Jason J CTR (US) wrote:\n\n> We have systems hosting git which are behind proxies, and unless the\n> client sets the http.postBuffer to a large size they connections\n> fails.\n> \n> Is there a way to set this on the server side? If not would a patch be\n> possible to fix this?\n\nWhat would it mean to set it on the server?  It is the size at which the\nclient decides to use a \"chunked\" transfer-encoding rather than\nbuffering the whole output to send at once. So you'd want to figure out\nwhy the server is upset about the chunked encoding.\n\n> jason.pyeron@hostname /home/jason.pyeron/desktop/projectname\n> $ git push remote --all\n> Username for 'https://server.fqdn':\n> Password for 'https://jpyeron@server.fqdn':\n> Counting objects: 1820, done.\n> Delta compression using up to 4 threads.\n> Compressing objects: 100% (1276/1276), done.\n> error: RPC failed; result=22, HTTP code = 411\n> fatal: The remote end hung up unexpectedly\n> Writing objects: 100% (1820/1820), 17.72 MiB | 5.50 MiB/s, done.\n> Total 1820 (delta 527), reused 26 (delta 6)\n> fatal: The remote end hung up unexpectedly\n\nThe server (or the proxy) returns 411, complaining that it didn't get a\nContent-Length header. That's because the git http client doesn't know\nhow big the content is ahead of time (and that's kind of the point of\nchunked encoding; the content is streamed).\n\n> jason.pyeron@hostname /home/jason.pyeron/desktop/projectname\n> $ git config http.postBuffer 524288000\n> \n> jason.pyeron@hostname /home/jason.pyeron/desktop/projectname\n> $ git push remote --all\n> Username for 'https://server.fqdn':\n> Password for 'https://jpyeron@server.fqdn':\n> Counting objects: 1820, done.\n> Delta compression using up to 4 threads.\n> Compressing objects: 100% (1276/1276), done.\n> Writing objects: 100% (1820/1820), 17.72 MiB | 11.31 MiB/s, done.\n> Total 1820 (delta 519), reused 26 (delta 6)\n> To https://server.fqdn/git/netasset-portal/\n>  * [new branch]      master -> master\n\nAnd here you've bumped the buffer to 500MB, so git will potentially\nbuffer that much in memory before sending anything. Which works for your\n17MB packfile, as we buffer the whole thing and then send the exact size\nahead of time, appeasing the proxy.\n\nBut there are two problems I see with just bumping the postBuffer value:\n\n  1. You've just postponed the problem. The first 501MB push will fail\n     again. You can bump it higher, but you may eventually hit a point\n     where your buffer is too big to fit in RAM.\n\n  2. You've lost the pipelining. With a small postBuffer, we are\n     streaming content up to the server as pack-objects generates it.\n     But with a large buffer, we generate all of the content, then start\n     sending the first byte (notice how the progress meter, which is\n     generated by pack-objects, shows twice as fast in the second case.\n     It is not measuring the network at all, but is streaming into\n     git-remote-https's buffer).\n\nIf the server really insists on a content-length header, then we can't\never fix (2). But we could fix (1) by spooling the packfile to disk and\nthen sending from there (under the assumption that you have way more\ntemporary disk space than RAM).\n\nHowever, if you have control of the proxies, the best thing would be to\ntweak its config to stop complaining about a lack of content-length\nheader (at least in cases where you're getting a \"chunked\"\ncontent-transfer-encoding). That would solve both issues (and without\nclients having to change anything).\n\n-Peff\n"},{"id":"226135","messageId":"D7C6383E8CCC40338EABF61A1CDADFC8@porsche","threadId":"34790","inReplyTo":"20130829035209.GC22788@sigill.intra.peff.net","subject":"RE: http.postBuffer set at the server side?","fromName":"Jason Pyeron","fromEmail":"jpyeron@pdinc.us","sentAt":"2013-08-29T04:40:16Z","receivedAt":"2013-08-29T04:40:16Z","isPatch":false,"sender":{"key":"jpyeron@pdinc.us","avatar":"https://gravatar.com/avatar/c2e53452caa53d940768a1ffc9cf76196d851b9b534b7a39cd39852a70a0508f?d=mp&s=160"},"body":"> -----Original Message-----\n> From: Jeff King\n> Sent: Wednesday, August 28, 2013 23:52\n> To: Pyeron, Jason J CTR (US)\n> Cc: git@vger.kernel.org\n> Subject: Re: http.postBuffer set at the server side?\n> \n> On Wed, Aug 28, 2013 at 11:08:02PM +0000, Pyeron, Jason J CTR \n> (US) wrote:\n> \n> > We have systems hosting git which are behind proxies, and \n> unless the \n> > client sets the http.postBuffer to a large size they connections \n> > fails.\n> > \n> > Is there a way to set this on the server side? If not would \n> a patch be \n> > possible to fix this?\n> \n> What would it mean to set it on the server?  It is the size \n> at which the client decides to use a \"chunked\" \n\nTo tell the client...\n\n> transfer-encoding rather than buffering the whole output to \n> send at once. So you'd want to figure out why the server is \n> upset about the chunked encoding.\n> \n\nUnchangable settings in a specific case here.\n\n> > jason.pyeron@hostname /home/jason.pyeron/desktop/projectname\n> > $ git push remote --all\n> > Username for 'https://server.fqdn':\n> > Password for 'https://jpyeron@server.fqdn':\n> > Counting objects: 1820, done.\n> > Delta compression using up to 4 threads.\n> > Compressing objects: 100% (1276/1276), done.\n> > error: RPC failed; result=22, HTTP code = 411\n> > fatal: The remote end hung up unexpectedly Writing objects: 100% \n> > (1820/1820), 17.72 MiB | 5.50 MiB/s, done.\n> > Total 1820 (delta 527), reused 26 (delta 6)\n> > fatal: The remote end hung up unexpectedly\n> \n> The server (or the proxy) returns 411, complaining that it \n> didn't get a Content-Length header. That's because the git \n> http client doesn't know how big the content is ahead of time \n> (and that's kind of the point of chunked encoding; the \n> content is streamed).\n> \n> > jason.pyeron@hostname /home/jason.pyeron/desktop/projectname\n> > $ git config http.postBuffer 524288000\n> > \n> > jason.pyeron@hostname /home/jason.pyeron/desktop/projectname\n> > $ git push remote --all\n> > Username for 'https://server.fqdn':\n> > Password for 'https://jpyeron@server.fqdn':\n> > Counting objects: 1820, done.\n> > Delta compression using up to 4 threads.\n> > Compressing objects: 100% (1276/1276), done.\n> > Writing objects: 100% (1820/1820), 17.72 MiB | 11.31 MiB/s, done.\n> > Total 1820 (delta 519), reused 26 (delta 6) To \n> > https://server.fqdn/git/netasset-portal/\n> >  * [new branch]      master -> master\n> \n> And here you've bumped the buffer to 500MB, so git will \n> potentially buffer that much in memory before sending \n> anything. Which works for your 17MB packfile, as we buffer \n> the whole thing and then send the exact size ahead of time, \n> appeasing the proxy.\n> \n> But there are two problems I see with just bumping the \n> postBuffer value:\n> \n>   1. You've just postponed the problem. The first 501MB push will fail\n>      again. You can bump it higher, but you may eventually hit a point\n>      where your buffer is too big to fit in RAM.\n> \n\nAgreed. By then I hope to get our infrastructure team to address the proxies.\nLiking the spool idea below. The other idea would be restrict the size of any\none transfer. (would that work if a given commit is large than the threshold???)\n\n>   2. You've lost the pipelining. With a small postBuffer, we are\n>      streaming content up to the server as pack-objects generates it.\n>      But with a large buffer, we generate all of the content, \n> then start\n>      sending the first byte (notice how the progress meter, which is\n>      generated by pack-objects, shows twice as fast in the \n> second case.\n>      It is not measuring the network at all, but is streaming into\n>      git-remote-https's buffer).\n> \n> If the server really insists on a content-length header, then \n> we can't ever fix (2). But we could fix (1) by spooling the \n> packfile to disk and then sending from there (under the \n> assumption that you have way more temporary disk space than RAM).\n> \n\nHmmm. So if the server says, hey I have a borked infrastructure, please send me\na content length git could spool then.\n\n> However, if you have control of the proxies, the best thing \n> would be to tweak its config to stop complaining about a lack \n> of content-length header (at least in cases where you're \n> getting a \"chunked\"\n> content-transfer-encoding). That would solve both issues (and \n> without clients having to change anything).\n> \n\nOne of the many is under my control, I will get that one addressed but the\nothers there is no hope.\n\n-Jason\n\n\n--\n-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-\n-                                                               -\n- Jason Pyeron                      PD Inc. http://www.pdinc.us -\n- Principal Consultant              10 West 24th Street #100    -\n- +1 (443) 269-1555 x333            Baltimore, Maryland 21218   -\n-                                                               -\n-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-\nThis message is copyright PD Inc, subject to license 20080407P00.\n\n \n"},{"id":"226137","messageId":"20130829063102.GA28827@sigill.intra.peff.net","threadId":"34790","inReplyTo":"D7C6383E8CCC40338EABF61A1CDADFC8@porsche","subject":"Re: http.postBuffer set at the server side?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-08-29T06:31:02Z","receivedAt":"2013-08-29T06:31:02Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Aug 29, 2013 at 12:40:16AM -0400, Jason Pyeron wrote:\n\n> > What would it mean to set it on the server?  It is the size \n> > at which the client decides to use a \"chunked\" \n> \n> To tell the client...\n\nBut the server in this case cannot tell it a useful size. Only \"whatever\nyou do, do not chunk\".\n\n> Agreed. By then I hope to get our infrastructure team to address the proxies.\n> Liking the spool idea below. The other idea would be restrict the size of any\n> one transfer. (would that work if a given commit is large than the threshold???)\n\nI don't think you can do that elegantly. The packfile needs to come up\nas a whole, because the server is stateless. The best you could do is to\nsplit the packfile up into small updates of history, and then have the\nserver receive each (and update the refs) incrementally. But even that\nisn't foolproof; it could be that a single large blob object is larger\nthan the limit.\n\n> > If the server really insists on a content-length header, then \n> > we can't ever fix (2). But we could fix (1) by spooling the \n> > packfile to disk and then sending from there (under the \n> > assumption that you have way more temporary disk space than RAM).\n> \n> Hmmm. So if the server says, hey I have a borked infrastructure, please send me\n> a content length git could spool then.\n\nI'm not sure how the server would say that. It can return a 411, which\nit is currently doing, but the git client will have already discarded\nthe data by that point. So it would have to come during one of the\nearlier responses (I guess an extra http header? I surely would not want\nto pollute the git protocol with such hackery).\n\nSo let's imagine that we figure that part out, and now we can ship a 1G\npackfile up in a single POST by spooling it to disk. Do the proxies\nactually allow that? Or do they also have a max-size-per-post setting?\n\nIf there is a maximum size, we can't get around that. And if it's small\nenough, there's no point in spooling, as we can just get by with\ntweaking http.postBuffer instead.\n\n-Peff\n"}]}