threads / discuss / 30615

Git hangs at “Writing objects: 11%”

Subject: Git hangs at “Writing objects: 11%”

## tl;dr

9 messages between May 24, 2012 and May 29, 2012.

replies: 8people: 4as markdown or json

Timothy Normand Miller· May 24, 2012, 21:40 UTC · lore
Hi, everyone,
I've run into a problem that I cannot debug.  I've talked with people at length on IRC, I've made ample use of Google.  I'm getting nowhere.  I'm not even getting any attention on stackexchange (where usually, questions are answered with surprising alacrity).  
This is not the common scenario of Windows and git-daemon that I have seen answered elsewhere. This is Linux to Linux, using ssh. On both the client and server, git version is 1.7.8.6.
I've tried doing the following:
	• Create a local empty repo and push to that -- works fine
	• Pull from local repo to local repo -- works fine
	• Clone the remote repo to the local machine -- works fine
	• git fsck and git gc on both server and client repos
	• Push to pre-existing remote repo -- hangs writing objects at 11%
	• Create empty remote repo and push to that -- hangs writing objects at 2%
There's an env variable I can set to make git produce debug messages. Nothing useful there.
Can anyone here help me figure out what's going wrong?
The post to stackexchange is at:  http://stackoverflow.com/questions/10740601/cannot-push-to-remote-git-repo-over-ssh-stuck-at-writing-objects-11
Thanks in advance for any advice you can give!
Jeff King· May 25, 2012, 00:51 UTC · re: Timothy Normand Miller · lore

Re: Git hangs at “Writing objects: 11%”

On Thu, May 24, 2012 at 05:40:41PM -0400, Timothy Normand Miller wrote:
Show 8 quoted lines
> I've run into a problem that I cannot debug.  I've talked with people
> at length on IRC, I've made ample use of Google.  I'm getting nowhere.
> I'm not even getting any attention on stackexchange (where usually,
> questions are answered with surprising alacrity).  
> 
> This is not the common scenario of Windows and git-daemon that I have
> seen answered elsewhere. This is Linux to Linux, using ssh. On both
> the client and server, git version is 1.7.8.6.

I'd first start with trying to remove as many variables as possible. Does the problem only happen over ssh, or does it also happen when pushing across a pipe to a repo on the local machine? If so, does it also happen during a fetch of the same data?

If you can reproduce it at will (which it sounds like you can), you could also try some older or newer git versions to see if they work any better. If you can find a working version, it might be worth trying to bisect and find the commit that introduces the breakage.

If the problem still exists in the latest version, then I'd start by stracing as much as possible. On the client side, you can use "strace -f" to see what all of the processes are doing; you'll probably also want to pass:

  --receive-pack='strace -f -o foo.out git-receive-pack'

to git-push to ask the remote side to strace. There's a reasonable chance you'll simply see that the client side is waiting on the server side for I/O, so you'll want to know what the server side is doing.

I see you posted an strace snippet of a process waiting in select() on stack overflow. It's hard to tell what's going on from there, though, because we can't see which processes are which (we see the pids, but we don't know which programs they're running, or where the commands go). A full strace log would help a lot (if it's long and you need a place to post it, try something like https://gist.github.com).

And finally, if the repo is something you can make public, I can try to reproduce on my machine. That might tell us if the problem is with your repo, or something else about your machines or setup.

-Peff
Timothy Normand Miller· May 25, 2012, 01:41 UTC · re: Jeff King · lore

Re: Git hangs at “Writing objects: 11%”

On May 24, 2012, at 8:51 PM, Jeff King wrote:
Show 10 quoted lines
> On Thu, May 24, 2012 at 05:40:41PM -0400, Timothy Normand Miller wrote:
> 
>> I've run into a problem that I cannot debug.  I've talked with people
>> at length on IRC, I've made ample use of Google.  I'm getting nowhere.
>> I'm not even getting any attention on stackexchange (where usually,
>> questions are answered with surprising alacrity).  
>> 
>> This is not the common scenario of Windows and git-daemon that I have
>> seen answered elsewhere. This is Linux to Linux, using ssh. On both
>> the client and server, git version is 1.7.8.6.
Thank you for helping!
Show 5 quoted lines
> 
> I'd first start with trying to remove as many variables as possible.
> Does the problem only happen over ssh, or does it also happen when
> pushing across a pipe to a repo on the local machine? If so, does it
> also happen during a fetch of the same data?
If the local repo I'm pushing to is given by pathname, it's fine.  If I'm pushing to another account via ssh on the localhost, it has the same hangup.
All fetches work fine.
Show 5 quoted lines
> 
> If you can reproduce it at will (which it sounds like you can), you
> could also try some older or newer git versions to see if they work any
> better. If you can find a working version, it might be worth trying to
> bisect and find the commit that introduces the breakage.
There's this one CentOS 6 machine where I know for a fact that git hasn't been updated since I was last able to push this particular repo.  That machine has git version 1.7.1.  That's where I first encountered the problem.  So I copied the repo to my Mac (version 1.7.8.6), but it freezes there at 11%.  Then I tried copying it to the same machine that hosts the repo (Gentoo Linux with git version 1.7.8.6), and push from there (still using ssh, but locally), and it also freezes at 11%.
So there's something wonky about the repo that git doesn't like, although it only has the problem over ssh.  I haven't tried git's native protocol.
Show 7 quoted lines
> 
> If the problem still exists in the latest version, then I'd start by
> stracing as much as possible. On the client side, you can use "strace
> -f" to see what all of the processes are doing; you'll probably also
> want to pass:
> 
>  --receive-pack='strace -f -o foo.out git-receive-pack'

First, I ran the push and then killed it, and I got this: http://www.cse.ohio-state.edu/~millerti/foo.out

But then I realized that you'd want to see where it hung up, so I can it again and then sent the file before killing the push: http://www.cse.ohio-state.edu/~millerti/foo2.out

> 
> to git-push to ask the remote side to strace. There's a reasonable
> chance you'll simply see that the client side is waiting on the server
> side for I/O, so you'll want to know what the server side is doing.
I really don't know how to interpret the trace.
Show 7 quoted lines
> 
> I see you posted an strace snippet of a process waiting in select() on
> stack overflow. It's hard to tell what's going on from there, though,
> because we can't see which processes are which (we see the pids, but we
> don't know which programs they're running, or where the commands go). A
> full strace log would help a lot (if it's long and you need a place to
> post it, try something like https://gist.github.com).

Try this: http://www.cse.ohio-state.edu/~millerti/revue_strace3.txt

This one was from earlier in the day.  It's from a push from the same machine that hosts the repo.
> 
> And finally, if the repo is something you can make public, I can try to
> reproduce on my machine. That might tell us if the problem is with your
> repo, or something else about your machines or setup.
That would make things a lot easier, but I'm not at liberty to share it.
Thanks again!
Timothy Normand Miller· May 26, 2012, 22:54 UTC · re: Timothy Normand Miller · lore

Re: Git hangs at “Writing objects: 11%”

I did some more research and testing.  This is an ssh problem.  If I change the protocol to git://, then I can push remotely all I want.  Same for file://.  But using ssh, it hangs.  Since I want ssh encryption, I still need to figure out why ssh is going wrong.
On May 24, 2012, at 9:41 PM, Timothy Normand Miller wrote:
Show 85 quoted lines
> 
> On May 24, 2012, at 8:51 PM, Jeff King wrote:
> 
>> On Thu, May 24, 2012 at 05:40:41PM -0400, Timothy Normand Miller wrote:
>> 
>>> I've run into a problem that I cannot debug.  I've talked with people
>>> at length on IRC, I've made ample use of Google.  I'm getting nowhere.
>>> I'm not even getting any attention on stackexchange (where usually,
>>> questions are answered with surprising alacrity).  
>>> 
>>> This is not the common scenario of Windows and git-daemon that I have
>>> seen answered elsewhere. This is Linux to Linux, using ssh. On both
>>> the client and server, git version is 1.7.8.6.
> 
> Thank you for helping!
> 
>> 
>> I'd first start with trying to remove as many variables as possible.
>> Does the problem only happen over ssh, or does it also happen when
>> pushing across a pipe to a repo on the local machine? If so, does it
>> also happen during a fetch of the same data?
> 
> If the local repo I'm pushing to is given by pathname, it's fine.  If I'm pushing to another account via ssh on the localhost, it has the same hangup.
> 
> All fetches work fine.
> 
>> 
>> If you can reproduce it at will (which it sounds like you can), you
>> could also try some older or newer git versions to see if they work any
>> better. If you can find a working version, it might be worth trying to
>> bisect and find the commit that introduces the breakage.
> 
> There's this one CentOS 6 machine where I know for a fact that git hasn't been updated since I was last able to push this particular repo.  That machine has git version 1.7.1.  That's where I first encountered the problem.  So I copied the repo to my Mac (version 1.7.8.6), but it freezes there at 11%.  Then I tried copying it to the same machine that hosts the repo (Gentoo Linux with git version 1.7.8.6), and push from there (still using ssh, but locally), and it also freezes at 11%.
> 
> So there's something wonky about the repo that git doesn't like, although it only has the problem over ssh.  I haven't tried git's native protocol.
> 
>> 
>> If the problem still exists in the latest version, then I'd start by
>> stracing as much as possible. On the client side, you can use "strace
>> -f" to see what all of the processes are doing; you'll probably also
>> want to pass:
>> 
>> --receive-pack='strace -f -o foo.out git-receive-pack'
> 
> First, I ran the push and then killed it, and I got this:	
> http://www.cse.ohio-state.edu/~millerti/foo.out
> 
> But then I realized that you'd want to see where it hung up, so I can it again and then sent the file before killing the push:
> http://www.cse.ohio-state.edu/~millerti/foo2.out
> 
> 
>> 
>> to git-push to ask the remote side to strace. There's a reasonable
>> chance you'll simply see that the client side is waiting on the server
>> side for I/O, so you'll want to know what the server side is doing.
> 
> I really don't know how to interpret the trace.
> 
>> 
>> I see you posted an strace snippet of a process waiting in select() on
>> stack overflow. It's hard to tell what's going on from there, though,
>> because we can't see which processes are which (we see the pids, but we
>> don't know which programs they're running, or where the commands go). A
>> full strace log would help a lot (if it's long and you need a place to
>> post it, try something like https://gist.github.com).
> 
> Try this:
> http://www.cse.ohio-state.edu/~millerti/revue_strace3.txt
> 
> This one was from earlier in the day.  It's from a push from the same machine that hosts the repo.
> 
>> 
>> And finally, if the repo is something you can make public, I can try to
>> reproduce on my machine. That might tell us if the problem is with your
>> repo, or something else about your machines or setup.
> 
> That would make things a lot easier, but I'm not at liberty to share it.
> 
> 
> Thanks again!
> 
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
Seth Robertson· May 27, 2012, 03:12 UTC · re: Timothy Normand Miller · lore

Re: Git hangs at “Writing objects: 11%”

In message <2DE05B0B-1D7E-451F-9151-B01CDDF4593E@att.net>, Timothy Normand Miller writes:
    I did some more research and testing.  This is an ssh problem.  If
    I change the protocol to git://, then I can push remotely all I
    want.  Same for file://.  But using ssh, it hangs.  Since I want
    ssh encryption, I still need to figure out why ssh is going wrong.

You could try smart-http with encryption, either as an experiment or as a workaround.

http://sethrobertson.github.com/HowToPutGitOnTheWeb/ provides information on how to put a git repository on the web, with encryption and password/pam/ldap/public key authentication.

					-Seth Robertson
Timothy Normand Miller· May 28, 2012, 12:48 UTC · re: Seth Robertson · lore

Re: Git hangs at “Writing objects: 11%”

I tried uploading a file via sftp, and it stalls out at 160K.  So this is the cause of the problem.  SSH is broken.  Unfortunately, googling for ssh and stall isn't getting me anywhere.
On May 26, 2012, at 11:12 PM, Seth Robertson wrote:
Show 20 quoted lines
> 
> In message <2DE05B0B-1D7E-451F-9151-B01CDDF4593E@att.net>, Timothy Normand Miller writes:
> 
>    I did some more research and testing.  This is an ssh problem.  If
>    I change the protocol to git://, then I can push remotely all I
>    want.  Same for file://.  But using ssh, it hangs.  Since I want
>    ssh encryption, I still need to figure out why ssh is going wrong.
> 
> You could try smart-http with encryption, either as an experiment or
> as a workaround.
> 
> http://sethrobertson.github.com/HowToPutGitOnTheWeb/ provides
> information on how to put a git repository on the web, with encryption
> and password/pam/ldap/public key authentication.
> 
> 					-Seth Robertson
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jeff King· May 28, 2012, 19:23 UTC · re: Timothy Normand Miller · lore

Re: Git hangs at “Writing objects: 11%”

On Mon, May 28, 2012 at 08:48:32AM -0400, Timothy Normand Miller wrote:
> I tried uploading a file via sftp, and it stalls out at 160K.  So this
> is the cause of the problem.  SSH is broken.  Unfortunately, googling
> for ssh and stall isn't getting me anywhere.

Thanks for following up. I hadn't had a chance to look at your strace output until today, and now you have saved me the trouble. :)

So I think we can rule out a git bug at this point. I would say it is an issue with network settings, but it sounds like you reproduced it while sshing to the local host (though depending on your setup, you might still be able to trigger firewall rules in that case, which could have an impact).

At this point, I'd try upgrading ssh if possible, and otherwise submit a bug report to the ssh vendor (either your distro, or openssh directly if you can reproduce it on pristine source).

-Peff
Martin Storsjö· May 28, 2012, 19:36 UTC · re: Timothy Normand Miller · lore

Re: Git hangs at “Writing objects: 11%”

On Mon, 28 May 2012, Timothy Normand Miller wrote:
> I tried uploading a file via sftp, and it stalls out at 160K.  So this 
> is the cause of the problem.  SSH is broken.  Unfortunately, googling 
> for ssh and stall isn't getting me anywhere.

Not sure if this is the same issue you're seeing or not, but I ran into something similar a while ago - when connected from less reliable networks, connections (especially rsync or scp) over SSH stalled after a short while - interactive sessions worked fine.

In my case, the issue was a too pedantic iptables configuration. When communicating with the server from a less reliable network, packets could arrive out of order. If the server received packets that iptables/conntrack didn't think belonged to the session, my config sent ICMP unreachable packets as response (-j REJECT), basically terminating the connection.

I fixed it by adjusting the iptables config to not send ICMP unreachable for this case. An alternative solution was to set sysctl "net.netfilter.nf_conntrack_tcp_be_liberal = 1" - this made the out of order packets still be counted as part of the session, not triggering the bad behaviour of my iptables config.

Again, not sure if this is related to your issue or not, but it at least a similar symptom.

// Martin
Timothy Normand Miller· May 29, 2012, 20:42 UTC · re: Martin Storsjö · lore

Re: Git hangs at “Writing objects: 11%”

The fix is to set "TcpRcvBufPoll no" in sshd_config.

http://forums.gentoo.org/viewtopic-t-925044-start-0-postdays-0-postorder-asc-highlight-.html http://forums.gentoo.org/viewtopic-p-7036250.html

On May 28, 2012, at 3:36 PM, Martin Storsjö wrote:
Show 17 quoted lines
> On Mon, 28 May 2012, Timothy Normand Miller wrote:
> 
>> I tried uploading a file via sftp, and it stalls out at 160K.  So this is the cause of the problem.  SSH is broken.  Unfortunately, googling for ssh and stall isn't getting me anywhere.
> 
> Not sure if this is the same issue you're seeing or not, but I ran into something similar a while ago - when connected from less reliable networks, connections (especially rsync or scp) over SSH stalled after a short while - interactive sessions worked fine.
> 
> In my case, the issue was a too pedantic iptables configuration. When communicating with the server from a less reliable network, packets could arrive out of order. If the server received packets that iptables/conntrack didn't think belonged to the session, my config sent ICMP unreachable packets as response (-j REJECT), basically terminating the connection.
> 
> I fixed it by adjusting the iptables config to not send ICMP unreachable for this case. An alternative solution was to set sysctl "net.netfilter.nf_conntrack_tcp_be_liberal = 1" - this made the out of order packets still be counted as part of the session, not triggering the bad behaviour of my iptables config.
> 
> Again, not sure if this is related to your issue or not, but it at least a similar symptom.
> 
> // Martin
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

← back to recent threads