{"thread":{"id":"50001","subject":"Git clone fails with fatal: the remote end hung up unexpectedly","startedAt":"2018-12-11T00:55:19Z","lastAt":"2018-12-11T10:08:45Z","messageCount":4,"participants":["Owen Ofiesh","Bryan Turner","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"365000","messageId":"CY4PR19MB1526704FC790724C2DC68465ADA60@CY4PR19MB1526.namprd19.prod.outlook.com","threadId":"50001","inReplyTo":null,"subject":"Git clone fails with fatal: the remote end hung up unexpectedly","fromName":"Owen Ofiesh","fromEmail":"oofiesh@maxlinear.com","sentAt":"2018-12-11T00:48:57Z","receivedAt":"2018-12-11T00:55:19Z","isPatch":false,"sender":{"key":"oofiesh@maxlinear.com","avatar":null},"body":"We are seeing an issue where git clone in Linux Ubuntu 14.04.5 LTS fails with the following error using the HTTP protocol.\n\nThe error on the client is:\n fatal: the remote end hung up unexpectedly\n fatal: early EOF\n fatal: index-pack failed\n\nThe client is writing to an NFS volume.\n\nThe HTTP POST error on the server is:\n c.a.s.i.w.filters.StreamGuardFilter The remote client has aborted the connection\n c.a.s.i.w.filters.StreamGuardFilter Failed to flush buffer; the remote client aborted the connection\n\nOur git repositories are managed by Atlassian's bitbucket server v5.5.1.\n\nWe see this with the Linux git client SW v1.9.1 and v2.19.2 (doesn't seem to matter the client SW version).\n\nThe Linux git server is SW v2.9.5.\n\nPer discussions about this we've tried the following:\n1. We increased the GIT_HTTP_MAX_REQUEST_BUFFER setting on the server to 100MBs.\n2. We set the http.postBuffer client value to 100MBs.\nNeither of these helped.\n\nAny thoughts about this please?\n\nNote: \nFor reasons I will not enter into, we cannot use SSH for this.\n\nThank you.\n\nend\n\nRegards,\nOwen Ofiesh, SCM, Software Test and Development Manager\nMaxLinear, Inc.  |  www.maxlinear.com \n5966 La Place Court Suite 100 Carlsbad, CA 92008\nPhone: 760-517-1109\nCell: 858-335-1690\noofiesh@maxlinear.com \n\n\n"},{"id":"365001","messageId":"CAGyf7-HHnwJyEu8G4p4A0RxJELJkEaJGMVj5nNznhWthAgS19g@mail.gmail.com","threadId":"50001","inReplyTo":"CY4PR19MB1526704FC790724C2DC68465ADA60@CY4PR19MB1526.namprd19.prod.outlook.com","subject":"Re: Git clone fails with fatal: the remote end hung up unexpectedly","fromName":"Bryan Turner","fromEmail":"bturner@atlassian.com","sentAt":"2018-12-11T01:15:57Z","receivedAt":"2018-12-11T01:17:24Z","isPatch":false,"sender":{"key":"bturner@atlassian.com","avatar":"https://gravatar.com/avatar/16bcf3167981c1ef7c804e502642366d888a35b0d0b0a4ca01fdc442aa1acb1e?d=mp&s=160"},"body":"On Mon, Dec 10, 2018 at 4:55 PM Owen Ofiesh wrote:\n>\n> We are seeing an issue where git clone in Linux Ubuntu 14.04.5 LTS fails with the following error using the HTTP protocol.\n>\n> The error on the client is:\n> fatal: the remote end hung up unexpectedly\n> fatal: early EOF\n> fatal: index-pack failed\n>\n> The client is writing to an NFS volume.\n\nA further detail on this (Owen correct me if I'm wrong), but the same\nclone performed to a local disk, rather than NFS, succeeds.\n\n>\n> The HTTP POST error on the server is:\n> c.a.s.i.w.filters.StreamGuardFilter The remote client has aborted the connection\n> c.a.s.i.w.filters.StreamGuardFilter Failed to flush buffer; the remote client aborted the connection\n\nBitbucket Server developer here. I just want to clarify one thing\nhere, for the list. The above is logged when the cloning client\ndisconnects unexpectedly in the middle of a hosting operation (push or\npull). So, from the server's perspective, everything was going well\nand then the client left without saying goodbye.\n\n>\n> Our git repositories are managed by Atlassian's bitbucket server v5.5.1.\n>\n> We see this with the Linux git client SW v1.9.1 and v2.19.2 (doesn't seem to matter the client SW version).\n>\n> The Linux git server is SW v2.9.5.\n>\n> Per discussions about this we've tried the following:\n> 1. We increased the GIT_HTTP_MAX_REQUEST_BUFFER setting on the server to 100MBs.\n> 2. We set the http.postBuffer client value to 100MBs.\n> Neither of these helped.\n>\n> Any thoughts about this please?\n>\n> Note:\n> For reasons I will not enter into, we cannot use SSH for this.\n>\n> Thank you.\n>\n> end\n>\n> Regards,\n> Owen Ofiesh, SCM, Software Test and Development Manager\n> MaxLinear, Inc. | www.maxlinear.com\n> 5966 La Place Court Suite 100 Carlsbad, CA 92008\n> Phone: 760-517-1109\n> Cell: 858-335-1690\n> oofiesh@maxlinear.com\n>\n>\n"},{"id":"365013","messageId":"CY4PR19MB152668FA1C3E592CA123E3E5ADA60@CY4PR19MB1526.namprd19.prod.outlook.com","threadId":"50001","inReplyTo":"CAGyf7-HHnwJyEu8G4p4A0RxJELJkEaJGMVj5nNznhWthAgS19g@mail.gmail.com","subject":"RE: Git clone fails with fatal: the remote end hung up unexpectedly","fromName":"Owen Ofiesh","fromEmail":"oofiesh@maxlinear.com","sentAt":"2018-12-11T02:25:41Z","receivedAt":"2018-12-11T02:25:51Z","isPatch":false,"sender":{"key":"oofiesh@maxlinear.com","avatar":null},"body":"Hi Bryan, all,\n\n> From: Bryan Turner [mailto:bturner@atlassian.com] \n> Sent: Monday, December 10, 2018 5:16 PM\n> To: Owen Ofiesh \n> Cc: Git Users \n> Subject: Re: Git clone fails with fatal: the remote end hung up unexpectedly\n> \n> On Mon, Dec 10, 2018 at 4:55 PM Owen Ofiesh wrote:\n> >\n> > We are seeing an issue where git clone in Linux Ubuntu 14.04.5 LTS fails\n> > with the following error using the HTTP protocol.\n> >\n> > The error on the client is:\n> > fatal: the remote end hung up unexpectedly\n> > fatal: early EOF\n> > fatal: index-pack failed\n> >\n> > The client is writing to an NFS volume.\n> \n> A further detail on this (Owen correct me if I'm wrong), but the same\n> clone performed to a local disk, rather than NFS, succeeds.\n\nCorrect. The clone to local disk succeeds.\n\nOne more data point. I tried this with SSH just now, and the behavior\nis somewhat improved in that I am only seeing the failure on the NFS\nvolume so far with 1 in 6 clone attempts (5 successful). Whereas with\nHTTP, we see the failure every time. \n\n> >\n> > The HTTP POST error on the server is:\n> > c.a.s.i.w.filters.StreamGuardFilter The remote client has aborted the connection\n> > c.a.s.i.w.filters.StreamGuardFilter Failed to flush buffer; the remote client aborted the connection\n> \n> Bitbucket Server developer here. I just want to clarify one thing\n> here, for the list. The above is logged when the cloning client\n> disconnects unexpectedly in the middle of a hosting operation (push or\n> pull). So, from the server's perspective, everything was going well\n> and then the client left without saying goodbye.\n\nCorrect in that we see this behavior when cloning. Small updates\n(push/pull) seems to work okay. We tested this previously by cloning to\nlocal disk, then copying to NFS, and then trying very small updates.\n\n> > Our git repositories are managed by Atlassian's bitbucket server v5.5.1.\n> >\n> > We see this with the Linux git client SW v1.9.1 and v2.19.2 (doesn't seem\n> > to matter the client SW version).\n> >\n> > The Linux git server is SW v2.9.5.\n> >\n> > Per discussions about this we've tried the following:\n> > 1. We increased the GIT_HTTP_MAX_REQUEST_BUFFER setting on the server\n> > to 100MBs.\n> > 2. We set the http.postBuffer client value to 100MBs.\n> > Neither of these helped.\n> >\n> > Any thoughts about this please?\n> >\n> > Note:\n> > For reasons I will not enter into, we cannot use SSH for this.\n> >\n> > Thank you.\n> >\n> > end\n> >\n> > Regards,\n> > Owen Ofiesh, SCM, Software Test and Development Manager\n> > MaxLinear, Inc. | http://www.maxlinear.com\n> > 5966 La Place Court Suite 100 Carlsbad, CA 92008\n> > Phone: 760-517-1109\n> > Cell: 858-335-1690\n> > mailto:oofiesh@maxlinear.com\n> >\n> >\n"},{"id":"365051","messageId":"20181211100819.GD31588@sigill.intra.peff.net","threadId":"50001","inReplyTo":"CY4PR19MB152668FA1C3E592CA123E3E5ADA60@CY4PR19MB1526.namprd19.prod.outlook.com","subject":"Re: Git clone fails with fatal: the remote end hung up unexpectedly","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-12-11T10:08:19Z","receivedAt":"2018-12-11T10:08:45Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Dec 11, 2018 at 02:25:41AM +0000, Owen Ofiesh wrote:\n\n> > On Mon, Dec 10, 2018 at 4:55 PM Owen Ofiesh wrote:\n> > >\n> > > We are seeing an issue where git clone in Linux Ubuntu 14.04.5 LTS fails\n> > > with the following error using the HTTP protocol.\n> > >\n> > > The error on the client is:\n> > > fatal: the remote end hung up unexpectedly\n> > > fatal: early EOF\n> > > fatal: index-pack failed\n> > >\n> > > The client is writing to an NFS volume.\n> > \n> > A further detail on this (Owen correct me if I'm wrong), but the same\n> > clone performed to a local disk, rather than NFS, succeeds.\n> \n> Correct. The clone to local disk succeeds.\n\nThat's weird. The messages imply that the server has started sending the\npackfile, at which point we should be pumping data from git-remote-https\nto fetch-pack, and from there into an index-pack process. And the\nmessages imply that the client saw a hangup of the network connection,\nat which point index-pack complained of the truncated packfile (and then\nwe complained that index-pack complained).\n\nBut what's weird is that none of that should really be relevant to the\nchoice of local filesystem. Is it possible that using NFS stresses the\nnetwork to the point that the HTTP connection may be killed?\n\nIt's also possible there's something racy in Git's handling of the data,\nand NFS is slower to write. What version of Git is this?\n\n> One more data point. I tried this with SSH just now, and the behavior\n> is somewhat improved in that I am only seeing the failure on the NFS\n> volume so far with 1 in 6 clone attempts (5 successful). Whereas with\n> HTTP, we see the failure every time.\n\nIf you use ssh's verbose mode, what does a failing case look like? If\nyou have Git v2.10 or later, you can just do:\n\n GIT_SSH_COMMAND='ssh -v' git clone ...\n\nIf it's older, the simplest thing is probably to put:\n\n Host yourserver...\n LogLevel Verbose\n\ninto your ~/.ssh/config.\n\nI know the problem is more common with HTTP, but I suspect ssh's logging\nmay be better than ours. ;) You can also try:\n\n GIT_CURL_VERBOSE=1 git clone ...\n\nor if you have recent enough (Git v2.10 again, I think), you can use:\n\n GIT_TRACE_CURL=1 git clone ...\n\nwhich has a few more details.\n\n-Peff\n"}]}