{"thread":{"id":"41996","subject":"clone hang prevention / timeout?","startedAt":"2016-04-11T21:49:19Z","lastAt":"2016-04-30T09:04:29Z","messageCount":6,"participants":["Jason Vas Dias","Eric Wong","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"283157","messageId":"CALyZvKwxE4T8-FmEYgPMgAFKLmeAY5f-y-mDL8S9twTb0umRaA@mail.gmail.com","threadId":"41996","inReplyTo":null,"subject":"clone hang prevention / timeout?","fromName":"Jason Vas Dias","fromEmail":"jason.vas.dias@gmail.com","sentAt":"2016-04-11T21:49:19Z","receivedAt":"2016-04-11T21:49:19Z","isPatch":false,"sender":{"key":"jason.vas.dias@gmail.com","avatar":null},"body":"It appears GIT has no way of specifying a timeout for a clone operation -\nif the server decides not to complete a get request, the clone can\nhang forever -\nis this correct ?\nThis appears to be what I am seeing, in a script that is attempting to do many\nsuccessive clone operations, eg. of\ngit://anongit.freedesktop.org/xorg/* , the script\noccasionally hangs in a clone - I can see with netstat + strace that the TCP\nconnection is open and GIT is trying to read .\nIs there any option I can specify to get the clone to timeout, or do I manually\nhave to strace the git process and send it a signal after a hang is detected?\n"},{"id":"283173","messageId":"20160412080149.GA9087@dcvr.yhbt.net","threadId":"41996","inReplyTo":"CALyZvKwxE4T8-FmEYgPMgAFKLmeAY5f-y-mDL8S9twTb0umRaA@mail.gmail.com","subject":"Re: clone hang prevention / timeout?","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2016-04-12T08:01:49Z","receivedAt":"2016-04-12T08:01:49Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Jason Vas Dias wrote:\n> It appears GIT has no way of specifying a timeout for a clone operation -\n> if the server decides not to complete a get request, the clone can\n> hang forever -\n> is this correct ?\n\ngit uses SO_KEEPALIVE for all connections it makes, so whatever\nyour kernel TCP keepalive knobs are set at.\n\nBy default, it's very long (around 2 hours), but you can change them\nusing the tcp_keepalive_* knobs in /proc/sys/net/ipv4/ under Linux.\n\nI suppose we can do shorter timeouts (at least under Linux) via\nsetsockopt(.. TCP_KEEP*) knobs, or we can call poll() ourselves\nto timeout connections. However, git packing operations on the\nserver can take a long time; so it might be bad to timeout\nmanually unless we know the connection is really dead.\n\n> This appears to be what I am seeing, in a script that is attempting to do many\n> successive clone operations, eg. of\n> git://anongit.freedesktop.org/xorg/* , the script\n> occasionally hangs in a clone - I can see with netstat + strace that the TCP\n> connection is open and GIT is trying to read .\n> Is there any option I can specify to get the clone to timeout, or do I manually\n> have to strace the git process and send it a signal after a hang is detected?\n\nI added git:// support for SO_KEEPALIVE in commit e47a8583a202\n(\"enable SO_KEEPALIVE for connected TCP sockets\")\nback in 2011 (v1.7.10),\nand http:// support later in 2013 (v1.8.5) with\ncommit a15d069a1986 (\"http: enable keepalive on TCP sockets\")\n"},{"id":"283423","messageId":"20160413222414.GA9882@sigill.intra.peff.net","threadId":"41996","inReplyTo":"CALyZvKwxE4T8-FmEYgPMgAFKLmeAY5f-y-mDL8S9twTb0umRaA@mail.gmail.com","subject":"Re: clone hang prevention / timeout?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-04-13T22:24:14Z","receivedAt":"2016-04-13T22:24:14Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Apr 11, 2016 at 10:49:19PM +0100, Jason Vas Dias wrote:\n\n> It appears GIT has no way of specifying a timeout for a clone operation -\n> if the server decides not to complete a get request, the clone can\n> hang forever -\n> is this correct ?\n\nYes. Git's protocol has no timeouts, though each side is generally\neither writing or reading at any moment, and so an interrupted\nconnection should cause either EPIPE or EOF, ending the process. The\nexceptions I have seen are:\n\n - protocol / implementation bugs that cause a true deadlock. At this\n we've fixed all known cases, but that doesn't mean there aren't bugs\n lurking.\n\n - the network drops out in such a way that the OS doesn't realize the\n connection is gone, and the reading side is left waiting for input\n forever\n\nI think the TCP keepalive stuff that Eric mentioned should address the\nlatter, though I don't know how well it works in practice. We used to\nsometimes see processes hung for days on GitHub, but it's been a long\ntime. I don't recall if it was pre-v1.8.5 (which introduced\nSO_KEEPALIVE), or if we made some other change (we have a load-balancing\nlayer in front that has more aggressive timeouts).\n\n> This appears to be what I am seeing, in a script that is attempting to do many\n> successive clone operations, eg. of\n> git://anongit.freedesktop.org/xorg/* , the script\n> occasionally hangs in a clone - I can see with netstat + strace that the TCP\n> connection is open and GIT is trying to read .\n> Is there any option I can specify to get the clone to timeout, or do I manually\n> have to strace the git process and send it a signal after a hang is detected?\n\nThere are periods where a git client may have to wait for a while in\nread() while the other side is quiet (e.g., when the other side is badly\npacked and needs to do a lot of up-front CPU work to prepare the\npackfile). Since v1.8.4.2, the server side of a clone should generate\napplication-level keepalive packets, so that the client never sees\nsilence for more than ~5 seconds. The freedesktop servers appear to be\non v2.1.4, so a long read() as you're seeing probably is a real hang.\n\nNote that pushing has a similar problem (the client may wait a long time\nwhile the server chews on the uploaded packfile before reporting\nstatus). There are no keepalives in that direction, though I have a\nseries there that I need to polish and submit.\n\n-Peff\n"},{"id":"283424","messageId":"20160413222900.GA10011@sigill.intra.peff.net","threadId":"41996","inReplyTo":"CALyZvKwxE4T8-FmEYgPMgAFKLmeAY5f-y-mDL8S9twTb0umRaA@mail.gmail.com","subject":"Re: clone hang prevention / timeout?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-04-13T22:29:00Z","receivedAt":"2016-04-13T22:29:00Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Apr 11, 2016 at 10:49:19PM +0100, Jason Vas Dias wrote:\n\n> Is there any option I can specify to get the clone to timeout, or do I manually\n> have to strace the git process and send it a signal after a hang is detected?\n\nOh, one other thing you might consider, it something like \"timeout\" from\nGNU coreutils, which puts a hard cap on the length of time a process can\nrun.\n\nIt's totally unaware of the state of the process, though, so if you\nreally do have a clone which takes an hour, it might very well kill it\nat 99% complete. It has no mechanism for \"gee, this process looks like\nit hasn't done anything for 5 minutes\".\n\nI don't know offhand of a general tool for that.\n\n-Peff\n"},{"id":"283477","messageId":"CALyZvKzzB9yv-aavw2RH7D7fK=vG88-j9W0LAmD3P6=CkSJCuw@mail.gmail.com","threadId":"41996","inReplyTo":"20160413222900.GA10011@sigill.intra.peff.net","subject":"Re: clone hang prevention / timeout?","fromName":"Jason Vas Dias","fromEmail":"jason.vas.dias@gmail.com","sentAt":"2016-04-14T18:32:24Z","receivedAt":"2016-04-14T18:32:24Z","isPatch":false,"sender":{"key":"jason.vas.dias@gmail.com","avatar":null},"body":"Thanks very much Eric & Jeff for your reply .\n\nPersonally, I would recommend setting the SO_RECVTIMEO for GIT server\nsockets to a fixed default (eg. 5mins) , settable by a\n'--receive-timeout' argument or configuration parameter .\n\nThe problem I was trying to overcome was cloning all the repositories under\nhttps://anongit.freedesktop.org/xorg/* .\n\nAbout 4 git clones would succeed in succession, but then typically the 5th\nwould hang in read() forever - I left one such hung 'git clone' for nearly an\nhour and it had not progressed or timed out . I tried inserting a delay of\nup to 30 seconds between clones, but this did not help.\n\nMaybe freedesktop.org's GIT server is too overloaded and they have\nto resort to disabling 1 out of 5 GIT successive clone operations from\nsame connection or something.\n\nHere is my solution, in case anyone else needs it :\n\n
\n      eips=()\n       counts=()\n       declare -i failed=0;\n       { echo \"$BASHPID\" >/tmp/git.pid;\n         GIT_TRACE=2 exec git clone\n${proto}://${user}anongit.freedesktop.org/${repo}$name; }&\n       while [ ! -f /tmp/git.pid ]; do sleep 1; done\n       git_pid=\"$(cat /tmp/git.pid)\";\n       while [ -d /proc/$git_pid ]; do\n           IFS=$'\\n';\n           declare -a kids=($(ps --ppid $git_pid -o 'pid=,eip='));\n           unset IFS;\n           declare -i n_kids=${#kids[@]} kid_n;\n           for ((kid_n=0; kid_n < n_kids; kid_n+=1)); do\n             declare -a ke=(${kids[kid_n]});\n             kid=${ke[0]}\n             eip=${ke[1]}\n             if [ ! -v 'eips['$kid']' ]; then\n                eips[$kid]=\"$eip\";\n             elif [ \"${eips[$kid]}\" = \"$eip\" ]; then\n                if [ x = x\"${counts[$kid]}\" ]; then\n                   counts[$kid]=1;\n                else\n                   counts[$kid]=$((${counts[$kid]}+1));\n                   if (( ${counts[$kid]} >= 30 )); then\n                      echo 'child process '$kid' of git main process\n'$git_pid' appears to be stuck - killing it.';\n                      kill -TERM $kid;\n                      ((failed=1));\n                   fi\n                fi\n             else\n                eips[$kid]=\"$eip\";\n                counts[$kid]='';\n             fi\n          done ;\n          sleep 1;\n       done\n       wait\n
\n\nThis is part of a script that reads a list of the Xorg projects,\nsets $repo to top level subdirectory, and $name to the project name,\nand initiates the GIT clone .\nIt deems any GIT _CHILD_ process (eg. git-index-pack) that have not\nchanged their instruction pointer register (EIP) for 30 seconds to be\n\"hung\" .\nThere is logic at the end to retry all the failed clones.\nIt does work, but is far from pretty .\nIt sure would be nice if GIT had a timeout mechanism !\n\nThanks & Regards,\nJason\n\n\n\n\n\n\nOn 13/04/2016, Jeff King wrote:\n> On Mon, Apr 11, 2016 at 10:49:19PM +0100, Jason Vas Dias wrote:\n>\n>> Is there any option I can specify to get the clone to timeout, or do I\n>> manually\n>> have to strace the git process and send it a signal after a hang is\n>> detected?\n>\n> Oh, one other thing you might consider, it something like \"timeout\" from\n> GNU coreutils, which puts a hard cap on the length of time a process can\n> run.\n>\n> It's totally unaware of the state of the process, though, so if you\n> really do have a clone which takes an hour, it might very well kill it\n> at 99% complete. It has no mechanism for \"gee, this process looks like\n> it hasn't done anything for 5 minutes\".\n>\n> I don't know offhand of a general tool for that.\n>\n> -Peff\n>\n"},{"id":"285044","messageId":"20160430090429.GA32220@dcvr.yhbt.net","threadId":"41996","inReplyTo":"CALyZvKzzB9yv-aavw2RH7D7fK=vG88-j9W0LAmD3P6=CkSJCuw@mail.gmail.com","subject":"Re: clone hang prevention / timeout?","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2016-04-30T09:04:29Z","receivedAt":"2016-04-30T09:04:29Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Jason Vas Dias wrote:\n> Thanks very much Eric & Jeff for your reply .\n> \n> Personally, I would recommend setting the SO_RECVTIMEO for GIT server\n> sockets to a fixed default (eg. 5mins) , settable by a\n> '--receive-timeout' argument or configuration parameter .\n\n(apologies for the delay, I thought I replied earlier :x)\n\nSO_RCVTIMEO only triggers EAGAIN, and AFAIK the git read/write\nwrappers are used to transparently retry on EAGAIN... So it's\nnot so simple as doing a single setsockopt.\n\n> The problem I was trying to overcome was cloning all the repositories under\n> https://anongit.freedesktop.org/xorg/* .\n> \n> About 4 git clones would succeed in succession, but then typically the 5th\n> would hang in read() forever - I left one such hung 'git clone' for nearly an\n> hour and it had not progressed or timed out . I tried inserting a delay of\n> up to 30 seconds between clones, but this did not help.\n\nAre you in contact with any of the admins of that server to\nhelp? Is the problematic repo any larger or in any way\nstranger than the others?\n\n> Maybe freedesktop.org's GIT server is too overloaded and they have\n> to resort to disabling 1 out of 5 GIT successive clone operations from\n> same connection or something.\n\nAnyways I've been thinking about overloaded git servers, lately.\nPack generation on big repos is painful, and having lots of slow\nclients can tie up server memory. So maybe an HTTP server\nwhich can switch between dumb and smart operation depending on\nload could be useful for the resource-constrained.\n\n> Here is my solution, in case anyone else needs it :\n\nIt'd be nice to get an strace to know where in the clone process\nit hangs to help the admin figure out how far things got.\n\nAnd please don't top-post, it's a waste of resources.\n"}]}