{"thread":{"id":"30461","subject":"Git fetch/pull stopped working","startedAt":"2012-05-07T22:45:08Z","lastAt":"2012-05-11T20:28:28Z","messageCount":12,"participants":["Egor Ryabkov","Junio C Hamano","Carlos Martín Nieto","Jeff King","jaseem abid"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"191050","messageId":"CADB4Qb1BhWpm1mAwiHy+edySvo7P+YcvrSiTSc8dVzKQsnNCfQ@mail.gmail.com","threadId":"30461","inReplyTo":null,"subject":"Git fetch/pull stopped working","fromName":"Egor Ryabkov","fromEmail":"egor.ryabkov@gmail.com","sentAt":"2012-05-07T22:45:08Z","receivedAt":"2012-05-07T22:45:08Z","isPatch":false,"sender":{"key":"egor.ryabkov@gmail.com","avatar":"https://gravatar.com/avatar/534a9220a253af01bb94452f4630ace065a252f640cad1363482c7e2a2aade08?d=mp&s=160"},"body":"Hi guys!\n\ngit fetch and pull stopped working a few days ago on my server with\nthe following message:\n\n$ git fetch\nERROR: Repository not found.\nfatal: The remote end hung up unexpectedly\n\n\nfetch/push/pull still work just fine with the same (GitHub) repository\nfrom my local PC, and they also work fine with another GH repo from\nanother server.\n\nI tried comparing \"git remote -v\" output for local PC and server,\nthey're the same.\n\"$ GIT_TRACE=1 git fetch\" gives somewhat different output on local PC\nand server:\nhttp://pastecode.com/bd3fc1a79f8e9d1eaf30911d9895938051c472f4\nbut the output is identical for the two servers I have (both are AWS\ninstances, if that matters).\n\nThere are two clones of the repository on the server in question, both\ngiving the same error. ssh git@github.com works as it should, so SSH\nkeys are alright.\n\n\nOne quirk I noticed is that \"git pull origin st+[Tab]\" used to expand\nto a branch name; now it expands to a name of the file in current\ndirectory (and the filename is not the same as branch name).\n\nI guess it's some problem with git install on the server - any ideas\nhow I could try and diagnose this, besides GIT_TRACE?\n\n\nA discussion on Stackoverflow:\nhttp://stackoverflow.com/questions/10438082/git-fetch-pull-have-stopped-working\n\nThanks in advance!\n\nEgor.\n"},{"id":"191054","messageId":"7vehqv36aw.fsf@alter.siamese.dyndns.org","threadId":"30461","inReplyTo":"CADB4Qb1BhWpm1mAwiHy+edySvo7P+YcvrSiTSc8dVzKQsnNCfQ@mail.gmail.com","subject":"Re: Git fetch/pull stopped working","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-05-07T23:44:23Z","receivedAt":"2012-05-07T23:44:23Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Egor Ryabkov <egor.ryabkov@gmail.com> writes:\n\n> git fetch and pull stopped working a few days ago on my server with\n> the following message:\n>\n> $ git fetch\n> ERROR: Repository not found.\n> fatal: The remote end hung up unexpectedly\n\nAre you sure it is an error on the remote side (presumably on GitHub)?\n\nMore interestingly, neither \"Repository not found\" nor \"ERROR:\" hits\nanything relevant to \"git fetch\" codepath.  The closest I can find is in\nhttp-backend.c that says \"Repository not exported\", which is on the server\nside, but you were talking about ssh connection, so it probably is not\nwhat you are seeing.\n\nOh, wait.\n\n> \"$ GIT_TRACE=1 git fetch\" gives somewhat different output on local PC\n> and server:\n> http://pastecode.com/bd3fc1a79f8e9d1eaf30911d9895938051c472f4\n\nSo you are going over http:// after all.  But it does not look like github\nto me, so you may be comparing apples and oranges, which would not help in\ndiagnosing the issue very much.\n\nPuzzled...\n"},{"id":"191057","messageId":"20120508001018.GB5067@centaur.lab.cmartin.tk","threadId":"30461","inReplyTo":"7vehqv36aw.fsf@alter.siamese.dyndns.org","subject":"Re: Git fetch/pull stopped working","fromName":"Carlos Martín Nieto","fromEmail":"cmn@elego.de","sentAt":"2012-05-08T00:10:18Z","receivedAt":"2012-05-08T00:10:18Z","isPatch":false,"sender":{"key":"cmn@elego.de","avatar":"https://avatars.githubusercontent.com/u/335443?v=4"},"body":"On Mon, May 07, 2012 at 04:44:23PM -0700, Junio C Hamano wrote:\n> Egor Ryabkov <egor.ryabkov@gmail.com> writes:\n> \n> > git fetch and pull stopped working a few days ago on my server with\n> > the following message:\n> >\n> > $ git fetch\n> > ERROR: Repository not found.\n> > fatal: The remote end hung up unexpectedly\n> \n> Are you sure it is an error on the remote side (presumably on GitHub)?\n> \n> More interestingly, neither \"Repository not found\" nor \"ERROR:\" hits\n> anything relevant to \"git fetch\" codepath.  The closest I can find is in\n> http-backend.c that says \"Repository not exported\", which is on the server\n> side, but you were talking about ssh connection, so it probably is not\n> what you are seeing.\n> \n> Oh, wait.\n> \n> > \"$ GIT_TRACE=1 git fetch\" gives somewhat different output on local PC\n> > and server:\n> > http://pastecode.com/bd3fc1a79f8e9d1eaf30911d9895938051c472f4\n> \n> So you are going over http:// after all.  But it does not look like github\n> to me, so you may be comparing apples and oranges, which would not help in\n> diagnosing the issue very much.\n\nI can reproduce this issue. It happens somewhere between my phone jack\nand rackspace.\n\nI can perform 'git fetch' on other hosts and ssh -T git@github.com\nworks fine. However, trying to fetch from github gives me an error.\n\nNot only this, but trying to browse to github.com or rubygems.org,\nboth of which seem to be hosted on the same datacentre gives me\ntimeouts or takes really long to answer. Routing traffic through my\nserver in the next country over makes it work, as does fetching from\nthat host.\n\nThe only difference I've been able to find is that the traffic looks\nlike it goes through different hosts when it goes through the UK on\nits way to the US.\n\n> \n> Puzzled...\n\nYeah... It looks like the Internet is being silly. So much for routing\naround damage. What I'm not sure is how this would also affect AWS;\nmaybe the Irish one?\n\n   cmn\n"},{"id":"191092","messageId":"CADB4Qb35FfTL=XX04iR71+2Rg8p4s2roqAF8b_BdFNP9YN=sfA@mail.gmail.com","threadId":"30461","inReplyTo":"7vehqv36aw.fsf@alter.siamese.dyndns.org","subject":"Re: Git fetch/pull stopped working","fromName":"Egor Ryabkov","fromEmail":"egor.ryabkov@gmail.com","sentAt":"2012-05-08T13:59:05Z","receivedAt":"2012-05-08T13:59:05Z","isPatch":false,"sender":{"key":"egor.ryabkov@gmail.com","avatar":"https://gravatar.com/avatar/534a9220a253af01bb94452f4630ace065a252f640cad1363482c7e2a2aade08?d=mp&s=160"},"body":"> Are you sure it is an error on the remote side (presumably on GitHub)?\n\nNo, I'm pretty sure it's something on my side, as clones on other PCs\nstill work.\n\n\n>> \"$ GIT_TRACE=1 git fetch\" gives somewhat different output on local PC\n>> and server:\n>> http://pastecode.com/bd3fc1a79f8e9d1eaf30911d9895938051c472f4\n>\n> So you are going over http:// after all.  But it does not look like github\n> to me\n\nNo, I'm using SSH connection.\nDid you imply that it's better to paste logs directly in the email?\nHere they go:\n\n############# Local results #############\n\n$ GIT_TRACE=1 git fetch\nsetup: git_dir: .git\nsetup: worktree: c:/Projects/(local_path)\nsetup: cwd: c:/Projects/(local_path)\nsetup: prefix: (null)\ntrace: built-in: git 'fetch'\ntrace: run_command: 'ssh' 'git@github.com' 'git-upload-pack\n'\\''(username)/(reponame).git'\\'''\nEnter passphrase for key '(keyname)':\ntrace: run_command: 'rev-list' '--quiet' '--objects' '--stdin' '--not' '--all'\n\n\n############# AWS results #############\n\n$ GIT_TRACE=1 git fetch\ntrace: built-in: git 'fetch'\ntrace: run_command: 'ssh' 'git@github.com' 'git-upload-pack\n'\\''(username)/(reponame).git'\\'''\nERROR: Repository not found.\nfatal: The remote end hung up unexpectedly\n\n\n\nI tried re-adding remote: SSH version fails with the same error, but\nHTTP version works. I'd rather have SSH working, though.\nAnd ideas about what could've caused it, or how to diagnose it further?\n"},{"id":"191144","messageId":"7v8vh2v501.fsf@alter.siamese.dyndns.org","threadId":"30461","inReplyTo":"CADB4Qb35FfTL=XX04iR71+2Rg8p4s2roqAF8b_BdFNP9YN=sfA@mail.gmail.com","subject":"Re: Git fetch/pull stopped working","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-05-08T19:37:18Z","receivedAt":"2012-05-08T19:37:18Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Egor Ryabkov <egor.ryabkov@gmail.com> writes:\n\n>>> \"$ GIT_TRACE=1 git fetch\" gives somewhat different output on local PC\n>>> and server:\n>>> http://pastecode.com/bd3fc1a79f8e9d1eaf30911d9895938051c472f4\n>> ...\n> Did you imply that it's better to paste logs directly in the email?\n\nNo, I did not *imply* it; it just didn't \"click\" that the above was a\nURL to a pastebin.\n\n> ############# AWS results #############\n>\n> $ GIT_TRACE=1 git fetch\n> trace: built-in: git 'fetch'\n> trace: run_command: 'ssh' 'git@github.com' 'git-upload-pack\n> '\\''(username)/(reponame).git'\\'''\n> ERROR: Repository not found.\n> fatal: The remote end hung up unexpectedly\n\nAssuming that these \"(REDACTED)\" are not hiding some crucial difference\nbetween the two destinations, it seems to be the git client installed on\nthe latter one seems to be the one that is at fault.\n\nBut as I said, \"ERROR: Repository not found.\" does not seem to match any\nmessage of the released version of Git, so I cannot even tell where it is\nfailing.\n\nIt could be that the message is coming from a customized version of ssh\nlogin handler github uses to shard and map the user accounts to\ndirectories (it would have said \"fatal: '/var/tmp/xx' does not appear to\nbe a git repository\" if they were running the vanilla Git), and if that is\nthe case, perhaps the argument thrown at git-upload-pack from your local\nbox and the problematic box somehow look different to Github, but that is\nin the \"(REDACTED)\" part, so I cannot tell from what was pasted in the\npastebin.\n\nIncidentally, oogling [github \"ERROR: Repository not found.\"] seems to\ngive quite a many hits.\n"},{"id":"191145","messageId":"20120508200842.GA14779@sigill.intra.peff.net","threadId":"30461","inReplyTo":"7v8vh2v501.fsf@alter.siamese.dyndns.org","subject":"Re: Git fetch/pull stopped working","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-05-08T20:08:42Z","receivedAt":"2012-05-08T20:08:42Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, May 08, 2012 at 12:37:18PM -0700, Junio C Hamano wrote:\n\n> But as I said, \"ERROR: Repository not found.\" does not seem to match any\n> message of the released version of Git, so I cannot even tell where it is\n> failing.\n> \n> It could be that the message is coming from a customized version of ssh\n> login handler github uses to shard and map the user accounts to\n> directories[...]\n\nThat is exactly where it comes from.\n\n> if they were running the vanilla Git), and if that is the case,\n> perhaps the argument thrown at git-upload-pack from your local box and\n> the problematic box somehow look different to Github, but that is in\n> the \"(REDACTED)\" part, so I cannot tell from what was pasted in the\n> pastebin.\n\nAgreed. It's hard to tell if there's a real problem or not. If the\nredacted bits are the same, then it is probably a github issue. In that\ncase, Egor, you should email support@github.com about it and mention the\nname of the problematic repo.\n\nI'll look into the weird routing thing that Carlos mentioned, too.\n\n-Peff\n"},{"id":"191146","messageId":"20120508201832.GA15287@sigill.intra.peff.net","threadId":"30461","inReplyTo":"20120508001018.GB5067@centaur.lab.cmartin.tk","subject":"Re: Git fetch/pull stopped working","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-05-08T20:18:32Z","receivedAt":"2012-05-08T20:18:32Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, May 08, 2012 at 02:10:18AM +0200, Carlos Martín Nieto wrote:\n\n> I can reproduce this issue. It happens somewhere between my phone jack\n> and rackspace.\n> \n> I can perform 'git fetch' on other hosts and ssh -T git@github.com\n> works fine. However, trying to fetch from github gives me an error.\n> \n> Not only this, but trying to browse to github.com or rubygems.org,\n> both of which seem to be hosted on the same datacentre gives me\n> timeouts or takes really long to answer. Routing traffic through my\n> server in the next country over makes it work, as does fetching from\n> that host.\n\nI can't reproduce this behavior at all. Neither from rackspace's DFW\ndata center, nor from the IAD one (the latter is where github.com is\nlocated).\n\n> Yeah... It looks like the Internet is being silly. So much for routing\n> around damage. What I'm not sure is how this would also affect AWS;\n> maybe the Irish one?\n\nIf it was a routing issue, I wonder if it has gone away. Are you still\nseeing problems?\n\nIf there's further discussion, we should probably take it off the git\nlist. Routing issues are definitely not a git-core problem. :)\n\n-Peff\n"},{"id":"191149","messageId":"20120508202934.GA19037@centaur.lab.cmartin.tk","threadId":"30461","inReplyTo":"20120508200842.GA14779@sigill.intra.peff.net","subject":"Re: Git fetch/pull stopped working","fromName":"Carlos Martín Nieto","fromEmail":"cmn@elego.de","sentAt":"2012-05-08T20:29:34Z","receivedAt":"2012-05-08T20:29:34Z","isPatch":false,"sender":{"key":"cmn@elego.de","avatar":"https://avatars.githubusercontent.com/u/335443?v=4"},"body":"On Tue, May 08, 2012 at 04:08:42PM -0400, Jeff King wrote:\n> \n> I'll look into the weird routing thing that Carlos mentioned, too.\n\nFWIW, it looks like it got solved. My ISP (or their ISP) seem pretty\ninept at keeping traffic flowing, so I wouldn't be surprised if it\nshowed up again later. I sent an e-mail to support@ with some traces,\nif you're interested.\n\n   cmn\n"},{"id":"191153","messageId":"CADB4Qb3sqy859k6QPuqU9u1cdxwz0LSQ7bdUXXFW_gQqaT+P3A@mail.gmail.com","threadId":"30461","inReplyTo":"20120508200842.GA14779@sigill.intra.peff.net","subject":"Re: Git fetch/pull stopped working","fromName":"Egor Ryabkov","fromEmail":"egor.ryabkov@gmail.com","sentAt":"2012-05-08T21:28:08Z","receivedAt":"2012-05-08T21:28:08Z","isPatch":false,"sender":{"key":"egor.ryabkov@gmail.com","avatar":"https://gravatar.com/avatar/534a9220a253af01bb94452f4630ace065a252f640cad1363482c7e2a2aade08?d=mp&s=160"},"body":"> Agreed. It's hard to tell if there's a real problem or not. If the\n> redacted bits are the same,\n\nYes, I checked this a few times and tried re-creating remote with the\naddress copied from GH site.\nHTTP worked, SSH didn't.\n\n\n> then it is probably a github issue. In that\n> case, Egor, you should email support@github.com about it and mention the\n> name of the problematic repo.\n\nYes, I actually contacted them last Friday, working with Simon Rozet now.\nI updated him on what we've found and sent him the link to this thread.\n\n\nThanks for your help guys :)\n"},{"id":"191408","messageId":"CADB4Qb2-VB0LTrP8_i75V9e2FDeSjNLc+Pc4m73UJ0AX9NfX=w@mail.gmail.com","threadId":"30461","inReplyTo":"CADB4Qb3sqy859k6QPuqU9u1cdxwz0LSQ7bdUXXFW_gQqaT+P3A@mail.gmail.com","subject":"Re: Git fetch/pull stopped working","fromName":"Egor Ryabkov","fromEmail":"egor.ryabkov@gmail.com","sentAt":"2012-05-11T18:47:29Z","receivedAt":"2012-05-11T18:47:29Z","isPatch":false,"sender":{"key":"egor.ryabkov@gmail.com","avatar":"https://gravatar.com/avatar/534a9220a253af01bb94452f4630ace065a252f640cad1363482c7e2a2aade08?d=mp&s=160"},"body":"Problem solved.\n\nTurns out that was a miscommunication on out side: the guy who setup\nthe keys on that box left, and his access to GH repo has been revoked.\n\nAnd GitHub returns \"repo not found\" rather than \"you have no access\nrights for this repository\".\n\n\nOn Wed, May 9, 2012 at 1:28 AM, Egor Ryabkov <egor.ryabkov@gmail.com> wrote:\n>\n> > Agreed. It's hard to tell if there's a real problem or not. If the\n> > redacted bits are the same,\n>\n> Yes, I checked this a few times and tried re-creating remote with the\n> address copied from GH site.\n> HTTP worked, SSH didn't.\n>\n>\n> > then it is probably a github issue. In that\n> > case, Egor, you should email support@github.com about it and mention the\n> > name of the problematic repo.\n>\n> Yes, I actually contacted them last Friday, working with Simon Rozet now.\n> I updated him on what we've found and sent him the link to this thread.\n>\n>\n> Thanks for your help guys :)\n"},{"id":"191414","messageId":"CAH-tXsCi+zL5hKAHROA0hVfvOr07o4cknS8jy9qngapwB77QnQ@mail.gmail.com","threadId":"30461","inReplyTo":"CADB4Qb2-VB0LTrP8_i75V9e2FDeSjNLc+Pc4m73UJ0AX9NfX=w@mail.gmail.com","subject":"Re: Git fetch/pull stopped working","fromName":"jaseem abid","fromEmail":"jaseemabid@gmail.com","sentAt":"2012-05-11T19:55:42Z","receivedAt":"2012-05-11T19:55:42Z","isPatch":false,"sender":{"key":"jaseemabid@gmail.com","avatar":"https://gravatar.com/avatar/8b0432c96e4d3c8a9a96c9961ee842df7b7a869744da9c25187a51a992eabd81?d=mp&s=160"},"body":"On Sat, May 12, 2012 at 12:17 AM, Egor Ryabkov <egor.ryabkov@gmail.com> wrote:\n> Problem solved.\n>\n> Turns out that was a miscommunication on out side: the guy who setup\n> the keys on that box left, and his access to GH repo has been revoked.\n>\n> And GitHub returns \"repo not found\" rather than \"you have no access\n> rights for this repository\".\n\nI have seen a lot of github guys in this list. I hope they will fix\nthis soon. I had this issue a long time back and now I'm understanding\nwhat exactly happened then :)\n\n\n-- \nJaseem Abid\nhttp://jaseemabid.github.com\n"},{"id":"191416","messageId":"20120511202828.GA17750@sigill.intra.peff.net","threadId":"30461","inReplyTo":"CAH-tXsCi+zL5hKAHROA0hVfvOr07o4cknS8jy9qngapwB77QnQ@mail.gmail.com","subject":"Re: Git fetch/pull stopped working","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-05-11T20:28:28Z","receivedAt":"2012-05-11T20:28:28Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, May 12, 2012 at 01:25:42AM +0530, jaseem abid wrote:\n\n> On Sat, May 12, 2012 at 12:17 AM, Egor Ryabkov <egor.ryabkov@gmail.com> wrote:\n> > Problem solved.\n> >\n> > Turns out that was a miscommunication on out side: the guy who setup\n> > the keys on that box left, and his access to GH repo has been revoked.\n> >\n> > And GitHub returns \"repo not found\" rather than \"you have no access\n> > rights for this repository\".\n> \n> I have seen a lot of github guys in this list. I hope they will fix\n> this soon. I had this issue a long time back and now I'm understanding\n> what exactly happened then :)\n\nThis is not a bug, but rather the intentional behavior. It is there to\nprevent an information leak about the names of private repositories that\nthe user has. Git-daemon has the same behavior by default (though it has\na \"friendly\" mode if you are willing to leak that information for your\nsite).\n\n-Peff\n"}]}