threads / discuss / 50001

Git clone fails with fatal: the remote end hung up unexpectedly

Subject: Git clone fails with fatal: the remote end hung up unexpectedly

## tl;dr

4 messages between Dec 11, 2018 and Dec 11, 2018.

replies: 3people: 3as markdown or json

Owen Ofiesh· Dec 11, 2018, 00:48 UTC · lore
We are seeing an issue where git clone in Linux Ubuntu 14.04.5 LTS fails with the following error using the HTTP protocol.
The error on the client is:
   fatal: the remote end hung up unexpectedly
   fatal: early EOF
   fatal: index-pack failed
The client is writing to an NFS volume.
The HTTP POST error on the server is:
   c.a.s.i.w.filters.StreamGuardFilter The remote client has aborted the connection
   c.a.s.i.w.filters.StreamGuardFilter Failed to flush buffer; the remote client aborted the connection
Our git repositories are managed by Atlassian's bitbucket server v5.5.1.
We see this with the Linux git client SW v1.9.1 and v2.19.2 (doesn't seem to matter the client SW version).
The Linux git server is SW v2.9.5.
Per discussions about this we've tried the following:
1. We increased the GIT_HTTP_MAX_REQUEST_BUFFER setting on the server to 100MBs.
2. We set the http.postBuffer client value to 100MBs.
Neither of these helped.
Any thoughts about this please?
Note: 
For reasons I will not enter into, we cannot use SSH for this.
Thank you.
end
Regards,
Owen Ofiesh, SCM, Software Test and Development Manager
MaxLinear, Inc.  |  www.maxlinear.com 
5966 La Place Court Suite 100 Carlsbad, CA 92008
Phone: 760-517-1109
Cell: 858-335-1690
oofiesh@maxlinear.com
Bryan Turner· Dec 11, 2018, 01:15 UTC · re: Owen Ofiesh · lore

Re: Git clone fails with fatal: the remote end hung up unexpectedly

On Mon, Dec 10, 2018 at 4:55 PM Owen Ofiesh <oofiesh@maxlinear.com> wrote:
Show 9 quoted lines
>
> We are seeing an issue where git clone in Linux Ubuntu 14.04.5 LTS fails with the following error using the HTTP protocol.
>
> The error on the client is:
>    fatal: the remote end hung up unexpectedly
>    fatal: early EOF
>    fatal: index-pack failed
>
> The client is writing to an NFS volume.

A further detail on this (Owen correct me if I'm wrong), but the same clone performed to a local disk, rather than NFS, succeeds.

>
> The HTTP POST error on the server is:
>    c.a.s.i.w.filters.StreamGuardFilter The remote client has aborted the connection
>    c.a.s.i.w.filters.StreamGuardFilter Failed to flush buffer; the remote client aborted the connection

Bitbucket Server developer here. I just want to clarify one thing here, for the list. The above is logged when the cloning client disconnects unexpectedly in the middle of a hosting operation (push or pull). So, from the server's perspective, everything was going well and then the client left without saying goodbye.

Show 30 quoted lines
>
> Our git repositories are managed by Atlassian's bitbucket server v5.5.1.
>
> We see this with the Linux git client SW v1.9.1 and v2.19.2 (doesn't seem to matter the client SW version).
>
> The Linux git server is SW v2.9.5.
>
> Per discussions about this we've tried the following:
> 1. We increased the GIT_HTTP_MAX_REQUEST_BUFFER setting on the server to 100MBs.
> 2. We set the http.postBuffer client value to 100MBs.
> Neither of these helped.
>
> Any thoughts about this please?
>
> Note:
> For reasons I will not enter into, we cannot use SSH for this.
>
> Thank you.
>
> end
>
> Regards,
> Owen Ofiesh, SCM, Software Test and Development Manager
> MaxLinear, Inc.  |  www.maxlinear.com
> 5966 La Place Court Suite 100 Carlsbad, CA 92008
> Phone: 760-517-1109
> Cell: 858-335-1690
> oofiesh@maxlinear.com
>
>
Owen Ofiesh· Dec 11, 2018, 02:25 UTC · re: Bryan Turner · lore

RE: Git clone fails with fatal: the remote end hung up unexpectedly

Hi Bryan, all,
Show 20 quoted lines
> From: Bryan Turner [mailto:bturner@atlassian.com] 
> Sent: Monday, December 10, 2018 5:16 PM
> To: Owen Ofiesh <oofiesh@maxlinear.com>
> Cc: Git Users <git@vger.kernel.org>
> Subject: Re: Git clone fails with fatal: the remote end hung up unexpectedly
> 
> On Mon, Dec 10, 2018 at 4:55 PM Owen Ofiesh <mailto:oofiesh@maxlinear.com> wrote:
> >
> > We are seeing an issue where git clone in Linux Ubuntu 14.04.5 LTS fails
> > with the following error using the HTTP protocol.
> >
> > The error on the client is:
> > fatal: the remote end hung up unexpectedly
> > fatal: early EOF
> > fatal: index-pack failed
> >
> > The client is writing to an NFS volume.
> 
> A further detail on this (Owen correct me if I'm wrong), but the same
> clone performed to a local disk, rather than NFS, succeeds.
Correct. The clone to local disk succeeds.

One more data point. I tried this with SSH just now, and the behavior is somewhat improved in that I am only seeing the failure on the NFS volume so far with 1 in 6 clone attempts (5 successful). Whereas with HTTP, we see the failure every time.

Show 10 quoted lines
> >
> > The HTTP POST error on the server is:
> > c.a.s.i.w.filters.StreamGuardFilter The remote client has aborted the connection
> > c.a.s.i.w.filters.StreamGuardFilter Failed to flush buffer; the remote client aborted the connection
> 
> Bitbucket Server developer here. I just want to clarify one thing
> here, for the list. The above is logged when the cloning client
> disconnects unexpectedly in the middle of a hosting operation (push or
> pull). So, from the server's perspective, everything was going well
> and then the client left without saying goodbye.

Correct in that we see this behavior when cloning. Small updates (push/pull) seems to work okay. We tested this previously by cloning to local disk, then copying to NFS, and then trying very small updates.

Show 31 quoted lines
> > Our git repositories are managed by Atlassian's bitbucket server v5.5.1.
> >
> > We see this with the Linux git client SW v1.9.1 and v2.19.2 (doesn't seem
> > to matter the client SW version).
> >
> > The Linux git server is SW v2.9.5.
> >
> > Per discussions about this we've tried the following:
> > 1. We increased the GIT_HTTP_MAX_REQUEST_BUFFER setting on the server
> >    to 100MBs.
> > 2. We set the http.postBuffer client value to 100MBs.
> > Neither of these helped.
> >
> > Any thoughts about this please?
> >
> > Note:
> > For reasons I will not enter into, we cannot use SSH for this.
> >
> > Thank you.
> >
> > end
> >
> > Regards,
> > Owen Ofiesh, SCM, Software Test and Development Manager
> > MaxLinear, Inc. | http://www.maxlinear.com
> > 5966 La Place Court Suite 100 Carlsbad, CA 92008
> > Phone: 760-517-1109
> > Cell: 858-335-1690
> > mailto:oofiesh@maxlinear.com
> >
> >
Jeff King· Dec 11, 2018, 10:08 UTC · re: Owen Ofiesh · lore

Re: Git clone fails with fatal: the remote end hung up unexpectedly

On Tue, Dec 11, 2018 at 02:25:41AM +0000, Owen Ofiesh wrote:
Show 16 quoted lines
> > On Mon, Dec 10, 2018 at 4:55 PM Owen Ofiesh <mailto:oofiesh@maxlinear.com> wrote:
> > >
> > > We are seeing an issue where git clone in Linux Ubuntu 14.04.5 LTS fails
> > > with the following error using the HTTP protocol.
> > >
> > > The error on the client is:
> > > fatal: the remote end hung up unexpectedly
> > > fatal: early EOF
> > > fatal: index-pack failed
> > >
> > > The client is writing to an NFS volume.
> > 
> > A further detail on this (Owen correct me if I'm wrong), but the same
> > clone performed to a local disk, rather than NFS, succeeds.
> 
> Correct. The clone to local disk succeeds.

That's weird. The messages imply that the server has started sending the packfile, at which point we should be pumping data from git-remote-https to fetch-pack, and from there into an index-pack process. And the messages imply that the client saw a hangup of the network connection, at which point index-pack complained of the truncated packfile (and then we complained that index-pack complained).

But what's weird is that none of that should really be relevant to the choice of local filesystem. Is it possible that using NFS stresses the network to the point that the HTTP connection may be killed?

It's also possible there's something racy in Git's handling of the data, and NFS is slower to write. What version of Git is this?

> One more data point. I tried this with SSH just now, and the behavior
> is somewhat improved in that I am only seeing the failure on the NFS
> volume so far with 1 in 6 clone attempts (5 successful). Whereas with
> HTTP, we see the failure every time.

If you use ssh's verbose mode, what does a failing case look like? If you have Git v2.10 or later, you can just do:

  GIT_SSH_COMMAND='ssh -v' git clone ...
If it's older, the simplest thing is probably to put:
  Host yourserver...
  LogLevel Verbose
into your ~/.ssh/config.

I know the problem is more common with HTTP, but I suspect ssh's logging may be better than ours. ;) You can also try:

  GIT_CURL_VERBOSE=1 git clone ...
or if you have recent enough (Git v2.10 again, I think), you can use:
  GIT_TRACE_CURL=1 git clone ...
which has a few more details.
-Peff

← back to recent threads