{"thread":{"id":"49724","subject":"[BUG?] protocol.version=2 sends HTTP \"Expect\" headers","startedAt":"2018-10-31T16:03:57Z","lastAt":"2018-11-01T18:35:16Z","messageCount":3,"participants":["Jeff King","brian m. carlson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"362114","messageId":"20181031160353.GA17267@sigill.intra.peff.net","threadId":"49724","inReplyTo":null,"subject":"[BUG?] protocol.version=2 sends HTTP \"Expect\" headers","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-31T16:03:53Z","receivedAt":"2018-10-31T16:03:57Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"Since 959dfcf42f (smart-http: Really never use Expect: 100-continue,\n2011-03-14), we try to avoid sending \"Expect\" headers, since some\nproxies apparently don't handle them well. There we have to explicitly\ntell curl not to use them.\n\nThe exception is large requests with GSSAPI, as explained in c80d96ca0c\n(remote-curl: fix large pushes with GSSAPI, 2013-10-31).\n\nHowever, Jon Simons noticed that when using protocol.version=2, we've\nstarted sending Expect headers again. That's because rather than going\nthrough post_rpc(), we push the stateless data through a proxy_state\nstruct. And in proxy_state_init(), when we set up the headers, we do not\ndisable curl's Expect handling.\n\nSo a few questions:\n\n  - is this a bug or not? I.e., do we still need to care about proxies\n    that can't handle Expect? The original commit was from 2011. Maybe\n    things are better now. Or maybe that's blind optimism.\n\n    Nobody has complained yet, but that's probably just because v2 isn't\n    widely deployed yet.\n\n  - if it is a bug, how can we handle it like the v0 code? There we\n    enable it only for GSSAPI on large requests. But I'm not sure we can\n    know here whether the request is large, since we're inherently just\n    streaming through chunked data. It looks like post_rpc tries to read\n    a single packet first, and considers anything over 64k to be large.\n\n  - alternatively, should we just leave it on for v2, and provide a\n    config switch to disable it if you have a crappy proxy? I don't know\n    how widespread the problem is, but I can imagine that the issue is\n    subtle enough that most users wouldn't even know.\n\nI think I've convinced myself that we probably do need to do the \"peek\nat the first packet\" thing like post_rpc() does, but I think it might be\ntricky with the way the proxy_state code is structured.\n\nThoughts from people with more HTTP knowledge/experience?\n\n-Peff\n"},{"id":"362137","messageId":"20181101004803.GA731755@genre.crustytoothpaste.net","threadId":"49724","inReplyTo":"20181031160353.GA17267@sigill.intra.peff.net","subject":"Re: [BUG?] protocol.version=2 sends HTTP \"Expect\" headers","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-01T00:48:04Z","receivedAt":"2018-11-01T00:49:14Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Wed, Oct 31, 2018 at 12:03:53PM -0400, Jeff King wrote:\n> Since 959dfcf42f (smart-http: Really never use Expect: 100-continue,\n> 2011-03-14), we try to avoid sending \"Expect\" headers, since some\n> proxies apparently don't handle them well. There we have to explicitly\n> tell curl not to use them.\n> \n> The exception is large requests with GSSAPI, as explained in c80d96ca0c\n> (remote-curl: fix large pushes with GSSAPI, 2013-10-31).\n> \n> However, Jon Simons noticed that when using protocol.version=2, we've\n> started sending Expect headers again. That's because rather than going\n> through post_rpc(), we push the stateless data through a proxy_state\n> struct. And in proxy_state_init(), when we set up the headers, we do not\n> disable curl's Expect handling.\n> \n> So a few questions:\n> \n>   - is this a bug or not? I.e., do we still need to care about proxies\n>     that can't handle Expect? The original commit was from 2011. Maybe\n>     things are better now. Or maybe that's blind optimism.\n> \n>     Nobody has complained yet, but that's probably just because v2 isn't\n>     widely deployed yet.\n\nHTTP/1.1 requires support for 100 Continue on the server side and in\nproxies (it is mandatory to implement).  The original commit disabling\nit (959dfcf42f (\"smart-http: Really never use Expect: 100-continue\",\n2011-03-14)), describes proxies as the reason for disabling it.\n\nIt's my understanding that all major proxies (including, as of version\n3.2, Squid) support HTTP/1.1 at this point.  Moreover, Kerberos is more\nlikely to be used in enterprises, where proxies (especially poorly\nbehaved and outright broken proxies) are more common.  We haven't seen\nany complaints about Kerberos support in a long time.  This leads me to\nbelieve that things generally work[0].\n\nFinally, modern versions of libcurl also have a timeout after which they\nassume that the server is not going to respond and just send the request\nbody anyways.  This makes the issue mostly moot.\n\n>   - alternatively, should we just leave it on for v2, and provide a\n>     config switch to disable it if you have a crappy proxy? I don't know\n>     how widespread the problem is, but I can imagine that the issue is\n>     subtle enough that most users wouldn't even know.\n\nFor the reasons I mentioned above, I'd leave it on for now.  Between\nlibcurl and better proxy support, I think this issue is mostly solved.\nIf we need to fix it in the future, we can, and people can fall back to\nolder protocols via config in the interim.\n\n[0] In some environments, people use SSH because the proxy breaks\neverything that looks like HTTP, but that's beyond the scope of this\ndiscussion.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"362203","messageId":"20181101183511.GA32547@sigill.intra.peff.net","threadId":"49724","inReplyTo":"20181101004803.GA731755@genre.crustytoothpaste.net","subject":"Re: [BUG?] protocol.version=2 sends HTTP \"Expect\" headers","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-11-01T18:35:12Z","receivedAt":"2018-11-01T18:35:16Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Nov 01, 2018 at 12:48:04AM +0000, brian m. carlson wrote:\n\n> > So a few questions:\n> > \n> >   - is this a bug or not? I.e., do we still need to care about proxies\n> >     that can't handle Expect? The original commit was from 2011. Maybe\n> >     things are better now. Or maybe that's blind optimism.\n> > \n> >     Nobody has complained yet, but that's probably just because v2 isn't\n> >     widely deployed yet.\n> \n> HTTP/1.1 requires support for 100 Continue on the server side and in\n> proxies (it is mandatory to implement).  The original commit disabling\n> it (959dfcf42f (\"smart-http: Really never use Expect: 100-continue\",\n> 2011-03-14)), describes proxies as the reason for disabling it.\n> \n> It's my understanding that all major proxies (including, as of version\n> 3.2, Squid) support HTTP/1.1 at this point.  Moreover, Kerberos is more\n> likely to be used in enterprises, where proxies (especially poorly\n> behaved and outright broken proxies) are more common.  We haven't seen\n> any complaints about Kerberos support in a long time.  This leads me to\n> believe that things generally work[0].\n\nRooting around in the archive a bit, I found this discussion:\n\n  https://public-inbox.org/git/CAJo=hJvyorMjFYZnVwz4iZr88ewor6LuqOE-mpt4LsPyoddBqg@mail.gmail.com/\n\nThe original motivation for 959dfcf42f was apparently Google's own\nservers. And they are likely the biggest users of the v2 protocol\n(since my impression is that Google ships a modified client to most of\ntheir devs that flips the v2 switch). So if they're not having problems,\nmaybe that's a sign that this is a non-issue.\n\n> Finally, modern versions of libcurl also have a timeout after which they\n> assume that the server is not going to respond and just send the request\n> body anyways.  This makes the issue mostly moot.\n\nI think this was always the case. It's just that it introduced annoying\nstalls into the protocol.\n\n> >   - alternatively, should we just leave it on for v2, and provide a\n> >     config switch to disable it if you have a crappy proxy? I don't know\n> >     how widespread the problem is, but I can imagine that the issue is\n> >     subtle enough that most users wouldn't even know.\n> \n> For the reasons I mentioned above, I'd leave it on for now.  Between\n> libcurl and better proxy support, I think this issue is mostly solved.\n> If we need to fix it in the future, we can, and people can fall back to\n> older protocols via config in the interim.\n\nOK. If nobody is complaining, this seems like a good way to ease into\nthe migration. I.e., if we dropped the old \"suppress Expect headers\"\ncode, people might complain that things are now broken. But if it\ngradually follows the v2 adoptions, that's a bit more of a gentle curve\n(well, until we hit the cliff where we switch the default to trying v2).\n\n-Peff\n"}]}