From: Jason Vas Dias <jason.vas.dias@gmail.com>
Date: Thu, 14 Apr 2016 18:32:24 GMT
Subject: Re: clone hang prevention / timeout?
Message-ID: <CALyZvKzzB9yv-aavw2RH7D7fK=vG88-j9W0LAmD3P6=CkSJCuw@mail.gmail.com>
In-Reply-To: <20160413222900.GA10011@sigill.intra.peff.net>

Thanks very much Eric & Jeff for your reply .

Personally, I would recommend setting the SO_RECVTIMEO for GIT server
sockets to a fixed default (eg. 5mins) , settable by a
'--receive-timeout'   argument or configuration parameter .

The problem I was trying to overcome was cloning all the repositories under
https://anongit.freedesktop.org/xorg/* .

About 4 git clones would succeed in succession, but then typically the 5th
would hang in read() forever - I left one such hung 'git clone' for nearly an
hour and it had not progressed or timed out . I tried inserting a delay of
up to 30 seconds between clones, but this did not help.

Maybe freedesktop.org's GIT server is too overloaded and they have
to resort to disabling 1 out of 5 GIT successive clone operations from
same connection or something.

Here is my solution, in case anyone else needs it :

<quote><pre>
      eips=()
       counts=()
       declare -i failed=0;
       { echo "$BASHPID" >/tmp/git.pid;
         GIT_TRACE=2 exec git clone
${proto}://${user}anongit.freedesktop.org/${repo}$name; }&
       while [ ! -f /tmp/git.pid ]; do sleep 1; done
       git_pid="$(cat /tmp/git.pid)";
       while [ -d /proc/$git_pid ]; do
           IFS=$'\n';
           declare -a kids=($(ps --ppid $git_pid -o 'pid=,eip='));
           unset IFS;
           declare -i n_kids=${#kids[@]} kid_n;
           for ((kid_n=0; kid_n < n_kids; kid_n+=1)); do
             declare -a ke=(${kids[kid_n]});
             kid=${ke[0]}
             eip=${ke[1]}
             if [ ! -v 'eips['$kid']' ]; then
                eips[$kid]="$eip";
             elif [ "${eips[$kid]}" = "$eip" ]; then
                if [ x = x"${counts[$kid]}" ]; then
                   counts[$kid]=1;
                else
                   counts[$kid]=$((${counts[$kid]}+1));
                   if (( ${counts[$kid]} >= 30 )); then
                      echo 'child process '$kid' of git main process
'$git_pid' appears to be stuck - killing it.';
                      kill -TERM $kid;
                      ((failed=1));
                   fi
                fi
             else
                eips[$kid]="$eip";
                counts[$kid]='';
             fi
          done ;
          sleep 1;
       done
       wait
</quote></pre>

This is part of a script that reads a list of the Xorg projects,
sets $repo to top level subdirectory, and $name to the project name,
and initiates the GIT clone .
It deems any GIT _CHILD_ process (eg. git-index-pack) that have not
changed their instruction pointer register (EIP)  for 30 seconds to be
"hung" .
There is logic at the end to retry all the failed clones.
It does work, but is far from pretty .
It sure would be nice if GIT had a timeout mechanism !

Thanks & Regards,
Jason






On 13/04/2016, Jeff King <peff@peff.net> wrote:
> On Mon, Apr 11, 2016 at 10:49:19PM +0100, Jason Vas Dias wrote:
>
>> Is there any option I can specify to get the clone to timeout, or do I
>> manually
>> have to strace the git process and send it a signal after a hang is
>> detected?
>
> Oh, one other thing you might consider, it something like "timeout" from
> GNU coreutils, which puts a hard cap on the length of time a process can
> run.
>
> It's totally unaware of the state of the process, though, so if you
> really do have a clone which takes an hour, it might very well kill it
> at 99% complete. It has no mechanism for "gee, this process looks like
> it hasn't done anything for 5 minutes".
>
> I don't know offhand of a general tool for that.
>
> -Peff
>