{"thread":{"id":"30615","subject":"Git hangs at “Writing objects: 11%”","startedAt":"2012-05-24T21:40:41Z","lastAt":"2012-05-29T20:42:29Z","messageCount":9,"participants":["Timothy Normand Miller","Jeff King","Seth Robertson","Martin Storsjö"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"192127","messageId":"EF4D4C5F-2D6B-46F2-B5A4-9DB1BA55BB6B@att.net","threadId":"30615","inReplyTo":null,"subject":"Git hangs at “Writing objects: 11%”","fromName":"Timothy Normand Miller","fromEmail":"theosib@att.net","sentAt":"2012-05-24T21:40:41Z","receivedAt":"2012-05-24T21:40:41Z","isPatch":false,"sender":{"key":"theosib@att.net","avatar":null},"body":"Hi, everyone,\n\nI've run into a problem that I cannot debug.  I've talked with people at length on IRC, I've made ample use of Google.  I'm getting nowhere.  I'm not even getting any attention on stackexchange (where usually, questions are answered with surprising alacrity).  \n\nThis is not the common scenario of Windows and git-daemon that I have seen answered elsewhere. This is Linux to Linux, using ssh. On both the client and server, git version is 1.7.8.6.\n\nI've tried doing the following:\n\n\t• Create a local empty repo and push to that -- works fine\n\t• Pull from local repo to local repo -- works fine\n\t• Clone the remote repo to the local machine -- works fine\n\t• git fsck and git gc on both server and client repos\n\t• Push to pre-existing remote repo -- hangs writing objects at 11%\n\t• Create empty remote repo and push to that -- hangs writing objects at 2%\n\nThere's an env variable I can set to make git produce debug messages. Nothing useful there.\n\nCan anyone here help me figure out what's going wrong?\n\n\nThe post to stackexchange is at:  http://stackoverflow.com/questions/10740601/cannot-push-to-remote-git-repo-over-ssh-stuck-at-writing-objects-11\n\n\nThanks in advance for any advice you can give!\n"},{"id":"192146","messageId":"20120525005156.GC11300@sigill.intra.peff.net","threadId":"30615","inReplyTo":"EF4D4C5F-2D6B-46F2-B5A4-9DB1BA55BB6B@att.net","subject":"Re: Git hangs at “Writing objects: 11%”","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-05-25T00:51:56Z","receivedAt":"2012-05-25T00:51:56Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 24, 2012 at 05:40:41PM -0400, Timothy Normand Miller wrote:\n\n> I've run into a problem that I cannot debug.  I've talked with people\n> at length on IRC, I've made ample use of Google.  I'm getting nowhere.\n> I'm not even getting any attention on stackexchange (where usually,\n> questions are answered with surprising alacrity).  \n> \n> This is not the common scenario of Windows and git-daemon that I have\n> seen answered elsewhere. This is Linux to Linux, using ssh. On both\n> the client and server, git version is 1.7.8.6.\n\nI'd first start with trying to remove as many variables as possible.\nDoes the problem only happen over ssh, or does it also happen when\npushing across a pipe to a repo on the local machine? If so, does it\nalso happen during a fetch of the same data?\n\nIf you can reproduce it at will (which it sounds like you can), you\ncould also try some older or newer git versions to see if they work any\nbetter. If you can find a working version, it might be worth trying to\nbisect and find the commit that introduces the breakage.\n\nIf the problem still exists in the latest version, then I'd start by\nstracing as much as possible. On the client side, you can use \"strace\n-f\" to see what all of the processes are doing; you'll probably also\nwant to pass:\n\n  --receive-pack='strace -f -o foo.out git-receive-pack'\n\nto git-push to ask the remote side to strace. There's a reasonable\nchance you'll simply see that the client side is waiting on the server\nside for I/O, so you'll want to know what the server side is doing.\n\nI see you posted an strace snippet of a process waiting in select() on\nstack overflow. It's hard to tell what's going on from there, though,\nbecause we can't see which processes are which (we see the pids, but we\ndon't know which programs they're running, or where the commands go). A\nfull strace log would help a lot (if it's long and you need a place to\npost it, try something like https://gist.github.com).\n\nAnd finally, if the repo is something you can make public, I can try to\nreproduce on my machine. That might tell us if the problem is with your\nrepo, or something else about your machines or setup.\n\n-Peff\n"},{"id":"192150","messageId":"033AF49C-4CB3-4412-8845-0246D356358C@att.net","threadId":"30615","inReplyTo":"20120525005156.GC11300@sigill.intra.peff.net","subject":"Re: Git hangs at “Writing objects: 11%”","fromName":"Timothy Normand Miller","fromEmail":"theosib@att.net","sentAt":"2012-05-25T01:41:55Z","receivedAt":"2012-05-25T01:41:55Z","isPatch":false,"sender":{"key":"theosib@att.net","avatar":null},"body":"\nOn May 24, 2012, at 8:51 PM, Jeff King wrote:\n\n> On Thu, May 24, 2012 at 05:40:41PM -0400, Timothy Normand Miller wrote:\n> \n>> I've run into a problem that I cannot debug.  I've talked with people\n>> at length on IRC, I've made ample use of Google.  I'm getting nowhere.\n>> I'm not even getting any attention on stackexchange (where usually,\n>> questions are answered with surprising alacrity).  \n>> \n>> This is not the common scenario of Windows and git-daemon that I have\n>> seen answered elsewhere. This is Linux to Linux, using ssh. On both\n>> the client and server, git version is 1.7.8.6.\n\nThank you for helping!\n\n> \n> I'd first start with trying to remove as many variables as possible.\n> Does the problem only happen over ssh, or does it also happen when\n> pushing across a pipe to a repo on the local machine? If so, does it\n> also happen during a fetch of the same data?\n\nIf the local repo I'm pushing to is given by pathname, it's fine.  If I'm pushing to another account via ssh on the localhost, it has the same hangup.\n\nAll fetches work fine.\n\n> \n> If you can reproduce it at will (which it sounds like you can), you\n> could also try some older or newer git versions to see if they work any\n> better. If you can find a working version, it might be worth trying to\n> bisect and find the commit that introduces the breakage.\n\nThere's this one CentOS 6 machine where I know for a fact that git hasn't been updated since I was last able to push this particular repo.  That machine has git version 1.7.1.  That's where I first encountered the problem.  So I copied the repo to my Mac (version 1.7.8.6), but it freezes there at 11%.  Then I tried copying it to the same machine that hosts the repo (Gentoo Linux with git version 1.7.8.6), and push from there (still using ssh, but locally), and it also freezes at 11%.\n\nSo there's something wonky about the repo that git doesn't like, although it only has the problem over ssh.  I haven't tried git's native protocol.\n\n> \n> If the problem still exists in the latest version, then I'd start by\n> stracing as much as possible. On the client side, you can use \"strace\n> -f\" to see what all of the processes are doing; you'll probably also\n> want to pass:\n> \n>  --receive-pack='strace -f -o foo.out git-receive-pack'\n\nFirst, I ran the push and then killed it, and I got this:\t\nhttp://www.cse.ohio-state.edu/~millerti/foo.out\n\nBut then I realized that you'd want to see where it hung up, so I can it again and then sent the file before killing the push:\nhttp://www.cse.ohio-state.edu/~millerti/foo2.out\n\n\n> \n> to git-push to ask the remote side to strace. There's a reasonable\n> chance you'll simply see that the client side is waiting on the server\n> side for I/O, so you'll want to know what the server side is doing.\n\nI really don't know how to interpret the trace.\n\n> \n> I see you posted an strace snippet of a process waiting in select() on\n> stack overflow. It's hard to tell what's going on from there, though,\n> because we can't see which processes are which (we see the pids, but we\n> don't know which programs they're running, or where the commands go). A\n> full strace log would help a lot (if it's long and you need a place to\n> post it, try something like https://gist.github.com).\n\nTry this:\nhttp://www.cse.ohio-state.edu/~millerti/revue_strace3.txt\n\nThis one was from earlier in the day.  It's from a push from the same machine that hosts the repo.\n\n> \n> And finally, if the repo is something you can make public, I can try to\n> reproduce on my machine. That might tell us if the problem is with your\n> repo, or something else about your machines or setup.\n\nThat would make things a lot easier, but I'm not at liberty to share it.\n\n\nThanks again!\n"},{"id":"192254","messageId":"2DE05B0B-1D7E-451F-9151-B01CDDF4593E@att.net","threadId":"30615","inReplyTo":"033AF49C-4CB3-4412-8845-0246D356358C@att.net","subject":"Re: Git hangs at “Writing objects: 11%”","fromName":"Timothy Normand Miller","fromEmail":"theosib@att.net","sentAt":"2012-05-26T22:54:57Z","receivedAt":"2012-05-26T22:54:57Z","isPatch":false,"sender":{"key":"theosib@att.net","avatar":null},"body":"I did some more research and testing.  This is an ssh problem.  If I change the protocol to git://, then I can push remotely all I want.  Same for file://.  But using ssh, it hangs.  Since I want ssh encryption, I still need to figure out why ssh is going wrong.\n\n\nOn May 24, 2012, at 9:41 PM, Timothy Normand Miller wrote:\n\n> \n> On May 24, 2012, at 8:51 PM, Jeff King wrote:\n> \n>> On Thu, May 24, 2012 at 05:40:41PM -0400, Timothy Normand Miller wrote:\n>> \n>>> I've run into a problem that I cannot debug.  I've talked with people\n>>> at length on IRC, I've made ample use of Google.  I'm getting nowhere.\n>>> I'm not even getting any attention on stackexchange (where usually,\n>>> questions are answered with surprising alacrity).  \n>>> \n>>> This is not the common scenario of Windows and git-daemon that I have\n>>> seen answered elsewhere. This is Linux to Linux, using ssh. On both\n>>> the client and server, git version is 1.7.8.6.\n> \n> Thank you for helping!\n> \n>> \n>> I'd first start with trying to remove as many variables as possible.\n>> Does the problem only happen over ssh, or does it also happen when\n>> pushing across a pipe to a repo on the local machine? If so, does it\n>> also happen during a fetch of the same data?\n> \n> If the local repo I'm pushing to is given by pathname, it's fine.  If I'm pushing to another account via ssh on the localhost, it has the same hangup.\n> \n> All fetches work fine.\n> \n>> \n>> If you can reproduce it at will (which it sounds like you can), you\n>> could also try some older or newer git versions to see if they work any\n>> better. If you can find a working version, it might be worth trying to\n>> bisect and find the commit that introduces the breakage.\n> \n> There's this one CentOS 6 machine where I know for a fact that git hasn't been updated since I was last able to push this particular repo.  That machine has git version 1.7.1.  That's where I first encountered the problem.  So I copied the repo to my Mac (version 1.7.8.6), but it freezes there at 11%.  Then I tried copying it to the same machine that hosts the repo (Gentoo Linux with git version 1.7.8.6), and push from there (still using ssh, but locally), and it also freezes at 11%.\n> \n> So there's something wonky about the repo that git doesn't like, although it only has the problem over ssh.  I haven't tried git's native protocol.\n> \n>> \n>> If the problem still exists in the latest version, then I'd start by\n>> stracing as much as possible. On the client side, you can use \"strace\n>> -f\" to see what all of the processes are doing; you'll probably also\n>> want to pass:\n>> \n>> --receive-pack='strace -f -o foo.out git-receive-pack'\n> \n> First, I ran the push and then killed it, and I got this:\t\n> http://www.cse.ohio-state.edu/~millerti/foo.out\n> \n> But then I realized that you'd want to see where it hung up, so I can it again and then sent the file before killing the push:\n> http://www.cse.ohio-state.edu/~millerti/foo2.out\n> \n> \n>> \n>> to git-push to ask the remote side to strace. There's a reasonable\n>> chance you'll simply see that the client side is waiting on the server\n>> side for I/O, so you'll want to know what the server side is doing.\n> \n> I really don't know how to interpret the trace.\n> \n>> \n>> I see you posted an strace snippet of a process waiting in select() on\n>> stack overflow. It's hard to tell what's going on from there, though,\n>> because we can't see which processes are which (we see the pids, but we\n>> don't know which programs they're running, or where the commands go). A\n>> full strace log would help a lot (if it's long and you need a place to\n>> post it, try something like https://gist.github.com).\n> \n> Try this:\n> http://www.cse.ohio-state.edu/~millerti/revue_strace3.txt\n> \n> This one was from earlier in the day.  It's from a push from the same machine that hosts the repo.\n> \n>> \n>> And finally, if the repo is something you can make public, I can try to\n>> reproduce on my machine. That might tell us if the problem is with your\n>> repo, or something else about your machines or setup.\n> \n> That would make things a lot easier, but I'm not at liberty to share it.\n> \n> \n> Thanks again!\n> \n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n"},{"id":"192255","messageId":"201205270312.q4R3Chef014187@no.baka.org","threadId":"30615","inReplyTo":"2DE05B0B-1D7E-451F-9151-B01CDDF4593E@att.net","subject":"Re: Git hangs at “Writing objects: 11%”","fromName":"Seth Robertson","fromEmail":"in-gitvger@baka.org","sentAt":"2012-05-27T03:12:43Z","receivedAt":"2012-05-27T03:12:43Z","isPatch":false,"sender":{"key":"in-gitvger@baka.org","avatar":null},"body":"\nIn message <2DE05B0B-1D7E-451F-9151-B01CDDF4593E@att.net>, Timothy Normand Miller writes:\n\n    I did some more research and testing.  This is an ssh problem.  If\n    I change the protocol to git://, then I can push remotely all I\n    want.  Same for file://.  But using ssh, it hangs.  Since I want\n    ssh encryption, I still need to figure out why ssh is going wrong.\n\nYou could try smart-http with encryption, either as an experiment or\nas a workaround.\n\nhttp://sethrobertson.github.com/HowToPutGitOnTheWeb/ provides\ninformation on how to put a git repository on the web, with encryption\nand password/pam/ldap/public key authentication.\n\n\t\t\t\t\t-Seth Robertson\n"},{"id":"192385","messageId":"3BBC77C3-B747-4F68-A135-954C3C00ECBA@att.net","threadId":"30615","inReplyTo":"201205270312.q4R3Chef014187@no.baka.org","subject":"Re: Git hangs at “Writing objects: 11%”","fromName":"Timothy Normand Miller","fromEmail":"theosib@att.net","sentAt":"2012-05-28T12:48:32Z","receivedAt":"2012-05-28T12:48:32Z","isPatch":false,"sender":{"key":"theosib@att.net","avatar":null},"body":"I tried uploading a file via sftp, and it stalls out at 160K.  So this is the cause of the problem.  SSH is broken.  Unfortunately, googling for ssh and stall isn't getting me anywhere.\n\nOn May 26, 2012, at 11:12 PM, Seth Robertson wrote:\n\n> \n> In message <2DE05B0B-1D7E-451F-9151-B01CDDF4593E@att.net>, Timothy Normand Miller writes:\n> \n>    I did some more research and testing.  This is an ssh problem.  If\n>    I change the protocol to git://, then I can push remotely all I\n>    want.  Same for file://.  But using ssh, it hangs.  Since I want\n>    ssh encryption, I still need to figure out why ssh is going wrong.\n> \n> You could try smart-http with encryption, either as an experiment or\n> as a workaround.\n> \n> http://sethrobertson.github.com/HowToPutGitOnTheWeb/ provides\n> information on how to put a git repository on the web, with encryption\n> and password/pam/ldap/public key authentication.\n> \n> \t\t\t\t\t-Seth Robertson\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n"},{"id":"192393","messageId":"20120528192316.GA2861@sigill.intra.peff.net","threadId":"30615","inReplyTo":"3BBC77C3-B747-4F68-A135-954C3C00ECBA@att.net","subject":"Re: Git hangs at “Writing objects: 11%”","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-05-28T19:23:16Z","receivedAt":"2012-05-28T19:23:16Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, May 28, 2012 at 08:48:32AM -0400, Timothy Normand Miller wrote:\n\n> I tried uploading a file via sftp, and it stalls out at 160K.  So this\n> is the cause of the problem.  SSH is broken.  Unfortunately, googling\n> for ssh and stall isn't getting me anywhere.\n\nThanks for following up. I hadn't had a chance to look at your strace\noutput until today, and now you have saved me the trouble. :)\n\nSo I think we can rule out a git bug at this point. I would say it is an\nissue with network settings, but it sounds like you reproduced it while\nsshing to the local host (though depending on your setup, you might\nstill be able to trigger firewall rules in that case, which could have\nan impact).\n\nAt this point, I'd try upgrading ssh if possible, and otherwise submit a\nbug report to the ssh vendor (either your distro, or openssh directly if\nyou can reproduce it on pristine source).\n\n-Peff\n"},{"id":"192394","messageId":"alpine.DEB.2.00.1205282225590.31146@cone.martin.st","threadId":"30615","inReplyTo":"3BBC77C3-B747-4F68-A135-954C3C00ECBA@att.net","subject":"Re: Git hangs at “Writing objects: 11%”","fromName":"Martin Storsjö","fromEmail":"martin@martin.st","sentAt":"2012-05-28T19:36:44Z","receivedAt":"2012-05-28T19:36:44Z","isPatch":false,"sender":{"key":"martin@martin.st","avatar":"https://avatars.githubusercontent.com/u/69727?v=4"},"body":"On Mon, 28 May 2012, Timothy Normand Miller wrote:\n\n> I tried uploading a file via sftp, and it stalls out at 160K.  So this \n> is the cause of the problem.  SSH is broken.  Unfortunately, googling \n> for ssh and stall isn't getting me anywhere.\n\nNot sure if this is the same issue you're seeing or not, but I ran into \nsomething similar a while ago - when connected from less reliable \nnetworks, connections (especially rsync or scp) over SSH stalled after a \nshort while - interactive sessions worked fine.\n\nIn my case, the issue was a too pedantic iptables configuration. When \ncommunicating with the server from a less reliable network, packets could \narrive out of order. If the server received packets that \niptables/conntrack didn't think belonged to the session, my config sent \nICMP unreachable packets as response (-j REJECT), basically terminating \nthe connection.\n\nI fixed it by adjusting the iptables config to not send ICMP unreachable \nfor this case. An alternative solution was to set sysctl \n\"net.netfilter.nf_conntrack_tcp_be_liberal = 1\" - this made the out of \norder packets still be counted as part of the session, not triggering the \nbad behaviour of my iptables config.\n\nAgain, not sure if this is related to your issue or not, but it at least a \nsimilar symptom.\n\n// Martin\n"},{"id":"192436","messageId":"995429D1-7204-4B8E-A7CE-94ABB99D707B@att.net","threadId":"30615","inReplyTo":"alpine.DEB.2.00.1205282225590.31146@cone.martin.st","subject":"Re: Git hangs at “Writing objects: 11%”","fromName":"Timothy Normand Miller","fromEmail":"theosib@att.net","sentAt":"2012-05-29T20:42:29Z","receivedAt":"2012-05-29T20:42:29Z","isPatch":false,"sender":{"key":"theosib@att.net","avatar":null},"body":"\nThe fix is to set \"TcpRcvBufPoll no\" in sshd_config.\n\nhttp://forums.gentoo.org/viewtopic-t-925044-start-0-postdays-0-postorder-asc-highlight-.html\nhttp://forums.gentoo.org/viewtopic-p-7036250.html\n\n\nOn May 28, 2012, at 3:36 PM, Martin Storsjö wrote:\n\n> On Mon, 28 May 2012, Timothy Normand Miller wrote:\n> \n>> I tried uploading a file via sftp, and it stalls out at 160K.  So this is the cause of the problem.  SSH is broken.  Unfortunately, googling for ssh and stall isn't getting me anywhere.\n> \n> Not sure if this is the same issue you're seeing or not, but I ran into something similar a while ago - when connected from less reliable networks, connections (especially rsync or scp) over SSH stalled after a short while - interactive sessions worked fine.\n> \n> In my case, the issue was a too pedantic iptables configuration. When communicating with the server from a less reliable network, packets could arrive out of order. If the server received packets that iptables/conntrack didn't think belonged to the session, my config sent ICMP unreachable packets as response (-j REJECT), basically terminating the connection.\n> \n> I fixed it by adjusting the iptables config to not send ICMP unreachable for this case. An alternative solution was to set sysctl \"net.netfilter.nf_conntrack_tcp_be_liberal = 1\" - this made the out of order packets still be counted as part of the session, not triggering the bad behaviour of my iptables config.\n> \n> Again, not sure if this is related to your issue or not, but it at least a similar symptom.\n> \n> // Martin\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n"}]}