Git fetch/pull stopped working

Hi guys!

git fetch and pull stopped working a few days ago on my server with the following message:

$ git fetch
ERROR: Repository not found.
fatal: The remote end hung up unexpectedly

fetch/push/pull still work just fine with the same (GitHub) repository from my local PC, and they also work fine with another GH repo from another server.

I tried comparing "git remote -v" output for local PC and server, they're the same. "$ GIT_TRACE=1 git fetch" gives somewhat different output on local PC and server: http://pastecode.com/bd3fc1a79f8e9d1eaf30911d9895938051c472f4 but the output is identical for the two servers I have (both are AWS instances, if that matters).

There are two clones of the repository on the server in question, both giving the same error. ssh git@github.com works as it should, so SSH keys are alright.

One quirk I noticed is that "git pull origin st+[Tab]" used to expand to a branch name; now it expands to a name of the file in current directory (and the filename is not the same as branch name).

I guess it's some problem with git install on the server - any ideas how I could try and diagnose this, besides GIT_TRACE?

A discussion on Stackoverflow: http://stackoverflow.com/questions/10438082/git-fetch-pull-have-stopped-working

Thanks in advance!

Egor.

Egor Ryabkov <egor.ryabkov@gmail.com> writes:

Are you sure it is an error on the remote side (presumably on GitHub)?

More interestingly, neither "Repository not found" nor "ERROR:" hits anything relevant to "git fetch" codepath. The closest I can find is in http-backend.c that says "Repository not exported", which is on the server side, but you were talking about ssh connection, so it probably is not what you are seeing.

Oh, wait.

So you are going over http:// after all. But it does not look like github to me, so you may be comparing apples and oranges, which would not help in diagnosing the issue very much.

Puzzled...

On Mon, May 07, 2012 at 04:44:23PM -0700, Junio C Hamano wrote:

I can reproduce this issue. It happens somewhere between my phone jack and rackspace.

I can perform 'git fetch' on other hosts and ssh -T git@github.com works fine. However, trying to fetch from github gives me an error.

Not only this, but trying to browse to github.com or rubygems.org, both of which seem to be hosted on the same datacentre gives me timeouts or takes really long to answer. Routing traffic through my server in the next country over makes it work, as does fetching from that host.

The only difference I've been able to find is that the traffic looks like it goes through different hosts when it goes through the UK on its way to the US.

Yeah... It looks like the Internet is being silly. So much for routing around damage. What I'm not sure is how this would also affect AWS; maybe the Irish one?

   cmn

On Tue, May 08, 2012 at 02:10:18AM +0200, Carlos Martín Nieto wrote:

I can't reproduce this behavior at all. Neither from rackspace's DFW data center, nor from the IAD one (the latter is where github.com is located).

If it was a routing issue, I wonder if it has gone away. Are you still seeing problems?

If there's further discussion, we should probably take it off the git list. Routing issues are definitely not a git-core problem. :)

-Peff

No, I'm pretty sure it's something on my side, as clones on other PCs still work.

No, I'm using SSH connection. Did you imply that it's better to paste logs directly in the email? Here they go:

############# Local results #############

$ GIT_TRACE=1 git fetch setup: git_dir: .git setup: worktree: c:/Projects/(local_path) setup: cwd: c:/Projects/(local_path) setup: prefix: (null) trace: built-in: git 'fetch' trace: run_command: 'ssh' 'git@github.com' 'git-upload-pack '\''(username)/(reponame).git'\''' Enter passphrase for key '(keyname)': trace: run_command: 'rev-list' '--quiet' '--objects' '--stdin' '--not' '--all'

############# AWS results #############

$ GIT_TRACE=1 git fetch
trace: built-in: git 'fetch'
trace: run_command: 'ssh' 'git@github.com' 'git-upload-pack
'\''(username)/(reponame).git'\'''
ERROR: Repository not found.
fatal: The remote end hung up unexpectedly

I tried re-adding remote: SSH version fails with the same error, but HTTP version works. I'd rather have SSH working, though. And ideas about what could've caused it, or how to diagnose it further?

Egor Ryabkov <egor.ryabkov@gmail.com> writes:

No, I did not *imply* it; it just didn't "click" that the above was a URL to a pastebin.

Assuming that these "(REDACTED)" are not hiding some crucial difference between the two destinations, it seems to be the git client installed on the latter one seems to be the one that is at fault.

But as I said, "ERROR: Repository not found." does not seem to match any message of the released version of Git, so I cannot even tell where it is failing.

It could be that the message is coming from a customized version of ssh login handler github uses to shard and map the user accounts to directories (it would have said "fatal: '/var/tmp/xx' does not appear to be a git repository" if they were running the vanilla Git), and if that is the case, perhaps the argument thrown at git-upload-pack from your local box and the problematic box somehow look different to Github, but that is in the "(REDACTED)" part, so I cannot tell from what was pasted in the pastebin.

Incidentally, oogling [github "ERROR: Repository not found."] seems to give quite a many hits.

On Tue, May 08, 2012 at 12:37:18PM -0700, Junio C Hamano wrote:

That is exactly where it comes from.

Agreed. It's hard to tell if there's a real problem or not. If the redacted bits are the same, then it is probably a github issue. In that case, Egor, you should email support@github.com about it and mention the name of the problematic repo.

I'll look into the weird routing thing that Carlos mentioned, too.

-Peff

On Tue, May 08, 2012 at 04:08:42PM -0400, Jeff King wrote:

FWIW, it looks like it got solved. My ISP (or their ISP) seem pretty inept at keeping traffic flowing, so I wouldn't be surprised if it showed up again later. I sent an e-mail to support@ with some traces, if you're interested.

   cmn

Yes, I checked this a few times and tried re-creating remote with the address copied from GH site. HTTP worked, SSH didn't.

Yes, I actually contacted them last Friday, working with Simon Rozet now. I updated him on what we've found and sent him the link to this thread.

Thanks for your help guys :)

Problem solved.

Turns out that was a miscommunication on out side: the guy who setup the keys on that box left, and his access to GH repo has been revoked.

And GitHub returns "repo not found" rather than "you have no access rights for this repository".

On Wed, May 9, 2012 at 1:28 AM, Egor Ryabkov <egor.ryabkov@gmail.com> wrote:

On Sat, May 12, 2012 at 12:17 AM, Egor Ryabkov <egor.ryabkov@gmail.com> wrote:

I have seen a lot of github guys in this list. I hope they will fix this soon. I had this issue a long time back and now I'm understanding what exactly happened then :)

On Sat, May 12, 2012 at 01:25:42AM +0530, jaseem abid wrote:

This is not a bug, but rather the intentional behavior. It is there to prevent an information leak about the names of private repositories that the user has. Git-daemon has the same behavior by default (though it has a "friendly" mode if you are willing to leak that information for your site).

-Peff