{"thread":{"id":"35069","subject":"GSS-Negotiate authentication requires that all data fit into postbuffer","startedAt":"2013-10-06T00:42:36Z","lastAt":"2013-10-07T22:56:07Z","messageCount":8,"participants":["brian m. carlson","Ilari Liusvaara","Daniel Stenberg","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"228575","messageId":"20131006004236.GB3460@vauxhall.crustytoothpaste.net","threadId":"35069","inReplyTo":null,"subject":"GSS-Negotiate authentication requires that all data fit into postbuffer","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2013-10-06T00:42:36Z","receivedAt":"2013-10-06T00:42:36Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"One thing I just noticed is that if git is using GSS-Negotiate\nauthentication, the entire POST contents have to fit into however much\nmemory is specified by http.postbuffer:\n\n  vauxhall ok % git push https://bmc@git.crustytoothpaste.net/git/bmc/test.git development\n  Counting objects: 37994, done.\n  Delta compression using up to 4 threads.\n  Compressing objects: 100% (10683/10683), done.\n  Writing objects: 100% (37994/37994), 9.15 MiB | 4.45 MiB/s, done.\n  Total 37994 (delta 26760), reused 37633 (delta 26467)\n  Unable to rewind rpc post data - try increasing http.postBuffer\n  Password for 'https://bmc@git.crustytoothpaste.net': \n\nGSS-Negotiate authentication always requires a rewind with CURL.\n\nThe remote in question only supports Negotiate authentication, so\nprompting for a password in this case isn't going to help.  I'm probably\ngoing to look into this over the next couple of days, but two things\nneed to be done here: 1) do not prompt for a password if only Negotiate\nauthentication is requested, since it just won't work, and 2) recreate\nthe data as needed if we have to rewind, since otherwise pushing a fresh\ncopy of the Linux kernel repo simply isn't going to work as the buffer\nwill have to be too large.  An alternative is to send a small amount of\ndata, smaller than the postbuffer, in the first chunk and only fail to\nrewind if the second or subsequent chunks need rewinding.\n\n-- \nbrian m. carlson / brian with sandals: Houston, Texas, US\n+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only\nOpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187\n"},{"id":"228579","messageId":"20131006105013.GA24950@LK-Perkele-VII","threadId":"35069","inReplyTo":"20131006004236.GB3460@vauxhall.crustytoothpaste.net","subject":"Re: GSS-Negotiate authentication requires that all data fit into postbuffer","fromName":"Ilari Liusvaara","fromEmail":"ilari.liusvaara@elisanet.fi","sentAt":"2013-10-06T10:50:13Z","receivedAt":"2013-10-06T10:50:13Z","isPatch":false,"sender":{"key":"ilari.liusvaara@elisanet.fi","avatar":null},"body":"On Sun, Oct 06, 2013 at 12:42:36AM +0000, brian m. carlson wrote:\n> \n> GSS-Negotiate authentication always requires a rewind with CURL.\n> \n> The remote in question only supports Negotiate authentication, so\n> prompting for a password in this case isn't going to help.  I'm probably\n> going to look into this over the next couple of days, but two things\n> need to be done here: 1) do not prompt for a password if only Negotiate\n> authentication is requested, since it just won't work, and 2) recreate\n> the data as needed if we have to rewind, since otherwise pushing a fresh\n> copy of the Linux kernel repo simply isn't going to work as the buffer\n> will have to be too large.  An alternative is to send a small amount of\n> data, smaller than the postbuffer, in the first chunk and only fail to\n> rewind if the second or subsequent chunks need rewinding.\n\nIsn't 'Expect: 100-Continue' meant for stuff like this (not that it is\nalways supported properly)?\n\n-Ilari\n"},{"id":"228581","messageId":"alpine.DEB.2.00.1310061658330.6366@tvnag.unkk.fr","threadId":"35069","inReplyTo":"20131006105013.GA24950@LK-Perkele-VII","subject":"Re: GSS-Negotiate authentication requires that all data fit into postbuffer","fromName":"Daniel Stenberg","fromEmail":"daniel@haxx.se","sentAt":"2013-10-06T15:00:02Z","receivedAt":"2013-10-06T15:00:02Z","isPatch":false,"sender":{"key":"daniel@haxx.se","avatar":"https://gravatar.com/avatar/69fdca87edd17cee21ca2e79fc2ff671d644603c3dc27167430f3cd3dbab7ba8?d=mp&s=160"},"body":"On Sun, 6 Oct 2013, Ilari Liusvaara wrote:\n\n>> GSS-Negotiate authentication always requires a rewind with CURL.\n\n> Isn't 'Expect: 100-Continue' meant for stuff like this (not that it is \n> always supported properly)?\n\nYes it is and libcurl uses 100-Continue by default for that purpose. But the \nharsh reality is that lots of (most?) servers just don't care and aren't setup \nto respond properly and instead we end up having to send data multiple times \nin vain.\n\n-- \n\n  / daniel.haxx.se\n"},{"id":"228582","messageId":"20131006152908.GD3460@vauxhall.crustytoothpaste.net","threadId":"35069","inReplyTo":"alpine.DEB.2.00.1310061658330.6366@tvnag.unkk.fr","subject":"Re: GSS-Negotiate authentication requires that all data fit into postbuffer","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2013-10-06T15:29:08Z","receivedAt":"2013-10-06T15:29:08Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Sun, Oct 06, 2013 at 05:00:02PM +0200, Daniel Stenberg wrote:\n> On Sun, 6 Oct 2013, Ilari Liusvaara wrote:\n> \n> >>GSS-Negotiate authentication always requires a rewind with CURL.\n> \n> >Isn't 'Expect: 100-Continue' meant for stuff like this (not that\n> >it is always supported properly)?\n> \n> Yes it is and libcurl uses 100-Continue by default for that purpose.\n> But the harsh reality is that lots of (most?) servers just don't\n> care and aren't setup to respond properly and instead we end up\n> having to send data multiple times in vain.\n\nIf there's a way to make Apache with mod_auth_kerb do that with curl,\nthen it doesn't require a change to git, and I'm happy to make it on my\nend.  But using the curl command line client, I don't see an Expect:\n100-continue anywhere during the connection using Debian's curl\n7.32.0-1.  Do I need to send a certain amount of data to see that\nbehavior?\n\nThe command line I used was\n\n  curl -v -d '0000' -H'Transfer-Encoding: chunked' -H'Content-Type: application/x-git-receive-pack-request' --negotiate -u bmc: https://git.crustytoothpaste.net/git/bmc/test.git/git-receive-pack\n\n-- \nbrian m. carlson / brian with sandals: Houston, Texas, US\n+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only\nOpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187\n"},{"id":"228583","messageId":"alpine.DEB.2.00.1310061737000.6366@tvnag.unkk.fr","threadId":"35069","inReplyTo":"20131006152908.GD3460@vauxhall.crustytoothpaste.net","subject":"Re: GSS-Negotiate authentication requires that all data fit into postbuffer","fromName":"Daniel Stenberg","fromEmail":"daniel@haxx.se","sentAt":"2013-10-06T15:38:24Z","receivedAt":"2013-10-06T15:38:24Z","isPatch":false,"sender":{"key":"daniel@haxx.se","avatar":"https://gravatar.com/avatar/69fdca87edd17cee21ca2e79fc2ff671d644603c3dc27167430f3cd3dbab7ba8?d=mp&s=160"},"body":"On Sun, 6 Oct 2013, brian m. carlson wrote:\n\n> If there's a way to make Apache with mod_auth_kerb do that with curl, then \n> it doesn't require a change to git, and I'm happy to make it on my end. \n> But using the curl command line client, I don't see an Expect: 100-continue \n> anywhere during the connection using Debian's curl 7.32.0-1.  Do I need to \n> send a certain amount of data to see that behavior?\n\nCorrect, curl will enable \"Expect: 100-continue\" if the post size is > 1024 \nbytes.\n\n-- \n\n  / daniel.haxx.se\n"},{"id":"228584","messageId":"20131006174959.GE3460@vauxhall.crustytoothpaste.net","threadId":"35069","inReplyTo":"alpine.DEB.2.00.1310061737000.6366@tvnag.unkk.fr","subject":"Re: GSS-Negotiate authentication requires that all data fit into postbuffer","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2013-10-06T17:50:00Z","receivedAt":"2013-10-06T17:50:00Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Sun, Oct 06, 2013 at 05:38:24PM +0200, Daniel Stenberg wrote:\n> On Sun, 6 Oct 2013, brian m. carlson wrote:\n> \n> >If there's a way to make Apache with mod_auth_kerb do that with\n> >curl, then it doesn't require a change to git, and I'm happy to\n> >make it on my end. But using the curl command line client, I don't\n> >see an Expect: 100-continue anywhere during the connection using\n> >Debian's curl 7.32.0-1.  Do I need to send a certain amount of\n> >data to see that behavior?\n> \n> Correct, curl will enable \"Expect: 100-continue\" if the post size is\n> > 1024 bytes.\n\nI've been able to reproduce this behavior with the curl command line\nclient, but it looks like we disable Expect: 100-continue in git since\nsome proxy servers are too stupid to understand it, according to commit\n959dfcf.\n\n-- \nbrian m. carlson / brian with sandals: Houston, Texas, US\n+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only\nOpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187\n"},{"id":"228598","messageId":"20131007120241.GC5792@sigill.intra.peff.net","threadId":"35069","inReplyTo":"20131006174959.GE3460@vauxhall.crustytoothpaste.net","subject":"Re: GSS-Negotiate authentication requires that all data fit into postbuffer","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-10-07T12:02:41Z","receivedAt":"2013-10-07T12:02:41Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Oct 06, 2013 at 05:50:00PM +0000, brian m. carlson wrote:\n\n> On Sun, Oct 06, 2013 at 05:38:24PM +0200, Daniel Stenberg wrote:\n> > On Sun, 6 Oct 2013, brian m. carlson wrote:\n> > \n> > >If there's a way to make Apache with mod_auth_kerb do that with\n> > >curl, then it doesn't require a change to git, and I'm happy to\n> > >make it on my end. But using the curl command line client, I don't\n> > >see an Expect: 100-continue anywhere during the connection using\n> > >Debian's curl 7.32.0-1.  Do I need to send a certain amount of\n> > >data to see that behavior?\n> > \n> > Correct, curl will enable \"Expect: 100-continue\" if the post size is\n> > > 1024 bytes.\n> \n> I've been able to reproduce this behavior with the curl command line\n> client, but it looks like we disable Expect: 100-continue in git since\n> some proxy servers are too stupid to understand it, according to commit\n> 959dfcf.\n\nYeah, instead we try to make two separate requests, and assume that the\nfirst one clears the path for any further requests. Of course that\ndoesn't work for auth methods that actually negotiate for each request.\n\nWe should probably make the \"Expect\" suppression optional for people who\nknow they have working systems. It would be nice to trigger it\nautomatically when people are using something like GSS, but that\ndecision happens at the curl layer.\n\n-Peff\n"},{"id":"228602","messageId":"20131007225607.GA30156@vauxhall.crustytoothpaste.net","threadId":"35069","inReplyTo":"20131007120241.GC5792@sigill.intra.peff.net","subject":"Re: GSS-Negotiate authentication requires that all data fit into postbuffer","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2013-10-07T22:56:07Z","receivedAt":"2013-10-07T22:56:07Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Mon, Oct 07, 2013 at 08:02:41AM -0400, Jeff King wrote:\n> Yeah, instead we try to make two separate requests, and assume that the\n> first one clears the path for any further requests. Of course that\n> doesn't work for auth methods that actually negotiate for each request.\n> \n> We should probably make the \"Expect\" suppression optional for people who\n> know they have working systems. It would be nice to trigger it\n> automatically when people are using something like GSS, but that\n> decision happens at the curl layer.\n\nYeah, that's what I ultimately determined would be the best choice.\nTechnically, we could respawn the process, but that would be a\nlogistical nightmare and extremely inefficient for large transfers.\n\n-- \nbrian m. carlson / brian with sandals: Houston, Texas, US\n+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only\nOpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187\n"}]}