Re: clone hang prevention / timeout?

From: Eric Wong <normalperson@yhbt.net>
Date: Apr 30, 2016, 09:04 UTC
Message-ID: <20160430090429.GA32220@dcvr.yhbt.net>
In-Reply-To: <CALyZvKzzB9yv-aavw2RH7D7fK=vG88-j9W0LAmD3P6=CkSJCuw@mail.gmail.com>

Jason Vas Dias <jason.vas.dias@gmail.com> wrote:

Show 5 quoted lines

> Thanks very much Eric & Jeff for your reply .
> 
> Personally, I would recommend setting the SO_RECVTIMEO for GIT server
> sockets to a fixed default (eg. 5mins) , settable by a
> '--receive-timeout'   argument or configuration parameter .

(apologies for the delay, I thought I replied earlier :x)

SO_RCVTIMEO only triggers EAGAIN, and AFAIK the git read/write wrappers are used to transparently retry on EAGAIN... So it's not so simple as doing a single setsockopt.

Show 7 quoted lines

> The problem I was trying to overcome was cloning all the repositories under
> https://anongit.freedesktop.org/xorg/* .
> 
> About 4 git clones would succeed in succession, but then typically the 5th
> would hang in read() forever - I left one such hung 'git clone' for nearly an
> hour and it had not progressed or timed out . I tried inserting a delay of
> up to 30 seconds between clones, but this did not help.

Are you in contact with any of the admins of that server to help? Is the problematic repo any larger or in any way stranger than the others?

> Maybe freedesktop.org's GIT server is too overloaded and they have
> to resort to disabling 1 out of 5 GIT successive clone operations from
> same connection or something.

Anyways I've been thinking about overloaded git servers, lately. Pack generation on big repos is painful, and having lots of slow clients can tie up server memory. So maybe an HTTP server which can switch between dumb and smart operation depending on load could be useful for the resource-constrained.

> Here is my solution, in case anyone else needs it :

It'd be nice to get an strace to know where in the clone process it hangs to help the admin figure out how far things got.

And please don't top-post, it's a waste of resources.