{"thread":{"id":"14222","subject":"Corruption: empty refs/heads in otherwise filled repo: cannot clone?","startedAt":"2008-06-30T09:49:18Z","lastAt":"2008-06-30T19:46:22Z","messageCount":10,"participants":["Jan Wielemaker","Jakub Narebski","Björn Steinbrink","Daniel Barkalow"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"81734","messageId":"200806301149.18115.J.Wielemaker@uva.nl","threadId":"14222","inReplyTo":null,"subject":"Corruption: empty refs/heads in otherwise filled repo: cannot clone?","fromName":"Jan Wielemaker","fromEmail":"j.wielemaker@uva.nl","sentAt":"2008-06-30T09:49:18Z","receivedAt":"2008-06-30T09:49:18Z","isPatch":false,"sender":{"key":"j.wielemaker@uva.nl","avatar":null},"body":"Hi,\n\nI'm a bit puzzled. I have a bare repository, somehow without any files\nin refs/heads.\n\nIt started using conversion from CVS via SVN using git-1.5.3.4. It is\nconfigured for group access and there are a number of SSH accounts on\nthe machine in the proper group using git-shell for fellow developers. \n\nI never have any problems, but after upgrading to git-git (1.5.6.rc3)\nfellow users get error:\n\nerror: no such remote ref refs/heads/JPL31\nerror: no such remote ref refs/heads/V57X\nerror: no such remote ref refs/heads/V5_4_patches\nerror: no such remote ref refs/heads/XML_UNICODE\nerror: no such remote ref refs/heads/attvar\nerror: no such remote ref refs/heads/gmp\n\nCreating a new account and doing a fresh clone I even got the fatal\nerror there are no remote branches!? Still, using my own user id, I\ncould do anything I wanted. I checked permissions to see whether there\nare differences between group and owner permissions, but couldn't find\nanything suspicious. git fsck --full on the repo gives no errors.\n\nIt turns out the directory refs/heads is empty!?\n\nI made a fresh clone of the repo as myself, creating all references\nnicely in refs/remotes/origin. Then I copied these files to refs/heads\ninto the main bare repository and now all appears to work nicely again.\nStill, this is a bit worrying ... My questions:\n\n * Is my work-around safe and sound?\n * How can this have happened?\n * Why can I clone as myself and not as anyone else!? Stranger:\n after a \"sudo -u /bin/bash\" I could not clone. I can\n copy the entire tree using cp -a, but I still cannot clone the\n copy with this user, while as myself I can clone the copy. \n Somehow `as myself' appears to get the branchheads from somewhere.\n There is only one copy of git installed, so both users use the\n same. What can cause this difference?\n\n\tThanks --- Jan\n"},{"id":"81738","messageId":"m3r6afgrme.fsf@localhost.localdomain","threadId":"14222","inReplyTo":"200806301149.18115.J.Wielemaker@uva.nl","subject":"Re: Corruption: empty refs/heads in otherwise filled repo: cannot clone?","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-06-30T10:10:56Z","receivedAt":"2008-06-30T10:10:56Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Jan Wielemaker writes:\n\n> I'm a bit puzzled. I have a bare repository, somehow without any files\n> in refs/heads.\n\nDo you have .git/packed-refs file? How do you have gc.packrefs set\n(\"git config --get gc.packrefs\")? What are the git version on the\nclients that have trouble accessing repository, and what protocol\n(transport merhod) do they use: \"file\", http(s)/ftp, git, ssh?\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"81740","messageId":"200806301230.38980.J.Wielemaker@uva.nl","threadId":"14222","inReplyTo":"m3r6afgrme.fsf@localhost.localdomain","subject":"Re: Corruption: empty refs/heads in otherwise filled repo: cannot clone?","fromName":"Jan Wielemaker","fromEmail":"j.wielemaker@uva.nl","sentAt":"2008-06-30T10:30:38Z","receivedAt":"2008-06-30T10:30:38Z","isPatch":false,"sender":{"key":"j.wielemaker@uva.nl","avatar":null},"body":"Hi Jakub,\n\nThanks for the quick reply. Info:\n\nOn Monday 30 June 2008 12:10, Jakub Narebski wrote:\n> Jan Wielemaker writes:\n> > I'm a bit puzzled. I have a bare repository, somehow without any files\n> > in refs/heads.\n>\n> Do you have .git/packed-refs file? How do you have gc.packrefs set\n\nIts a bare repo, so I'll forget the .git. Yes, I have that file and\nit contains nice references, I checked a few by hand, and they have\nthe same SHA1 as the files I copied.\n\n> (\"git config --get gc.packrefs\")? What are the git version on the\n\nThis gives no output, so I guess the answer is 'no'. Is that wrong?\nThis is indeed the case (config):\n\n----------------------------------------------------------------\n[core]\n repositoryformatversion = 0\n filemode = true\n bare = true\n sharedrepository = 1\n[gitcvs]\n enabled = 1\n logfile = /home/git/pl.git/cvs.log\n allbinary = 1\n----------------------------------------------------------------\n\n> clients that have trouble accessing repository, and what protocol\n> (transport merhod) do they use: \"file\", http(s)/ftp, git, ssh?\n\nI could reproduce the problem using the same git that maintains the bare\nrepository, which started as git-1.5.3.4 and is now 1.5.6.rc3. It\nreproduces both using local file access and ssh. Other people have a\nvariety of versions. I know of concrete problems using the stable 1.5.5\nover ssh.\n\nThe problem that I can't even clone is indeed after running a git-gc.\n\nSo, now I have all branch head refs double (in packed-refs and in\nrefs/heads). That explains ar least where they come from, but it fails\nto explain why doing exactly the same using the same git on the same\nrepo as two users yields a different result. Of course, besides the\ndesire to understand how his works, I'm mostly interested in how to fix\nthis :-)\n\n\tThanks --- Jan\n"},{"id":"81744","messageId":"200806301326.12140.jnareb@gmail.com","threadId":"14222","inReplyTo":"200806301230.38980.J.Wielemaker@uva.nl","subject":"Re: Corruption: empty refs/heads in otherwise filled repo: cannot clone?","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-06-30T11:26:10Z","receivedAt":"2008-06-30T11:26:10Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Jan Wielemaker wrote:\n> On Monday 30 June 2008 12:10, Jakub Narebski wrote:\n>> Jan Wielemaker writes:\n>>>\n>>> I'm a bit puzzled. I have a bare repository, somehow without any files\n>>> in refs/heads.\n>>\n>> Do you have .git/packed-refs file?\n> \n> Its a bare repo, so I'll forget the .git. Yes, I have that file and\n> it contains nice references, I checked a few by hand, and they have\n> the same SHA1 as the files I copied.\n\nThat is where the refs are instead of being in individual files under\nrefs/ - those are so called \"packed refs\".\n\n When a ref is missing from the traditional $GIT_DIR/refs hierarchy,\n it is looked up in $GIT_DIR/packed-refs and used if found.\n\nI'm not sure why branches are also packed, because git used to pack\nonly tags and refs which were packed already.\n\n>> How do you have gc.packrefs set (\"git config --get gc.packrefs\")?\n> \n> This gives no output, so I guess the answer is 'no'.\n\nIn git-config(1) you can find the following:\n\n gc.packrefs::\n `git gc` does not run `git pack-refs` in a bare repository by\n default so that older dumb-transport clients can still fetch\n from the repository. Setting this to `true` lets `git\n gc` to run `git pack-refs`. Setting this to `false` tells\n `git gc` never to run `git pack-refs`. The default setting is\n `notbare`. Enable it only when you know you do not have to\n support such clients. The default setting will change to `true`\n at some stage, and setting this to `false` will continue to\n prevent `git pack-refs` from being run from `git gc`.\n\nUnless something changed (and git Documentation was not updated) git\nshould not pack refs by default.\n\n>> What are the git version on the\n>> clients that have trouble accessing repository, and what protocol\n>> (transport method) do they use: \"file\", http(s)/ftp, git, ssh?\n> \n> I could reproduce the problem using the same git that maintains the bare\n> repository, which started as git-1.5.3.4 and is now 1.5.6.rc3. It\n> reproduces both using local file access and ssh. Other people have a\n> variety of versions. I know of concrete problems using the stable 1.5.5\n> over ssh.\n>\n> The problem that I can't even clone is indeed after running a git-gc.\n\nHmmm... strange. I know that there can be problems with older clients\n(those which do not understand packed-refs format) accessing\nrepositories with packed refs (without loose refs) via \"dumb\"\nprotocols. Problems with accessing repositories locally and via ssh\nmight point to some troubles with permissions and ownership of\n$GIT_DIR and $GIT_DIR/packed-refs.\n \ngit-gc can pack refs, that is what running it causes problems.\n\n> So, now I have all branch head refs double (in packed-refs and in\n> refs/heads). That explains ar least where they come from, but it fails\n> to explain why doing exactly the same using the same git on the same\n> repo as two users yields a different result. Of course, besides the\n> desire to understand how his works, I'm mostly interested in how to fix\n> this :-)\n\nUnfortunately as it is not caused by the issue I thought about I cannot\nhelp you further...\n\n-- \nJakub Narebski\nPoland\n"},{"id":"81746","messageId":"200806301344.09938.J.Wielemaker@uva.nl","threadId":"14222","inReplyTo":"200806301326.12140.jnareb@gmail.com","subject":"Re: Corruption: empty refs/heads in otherwise filled repo: cannot clone?","fromName":"Jan Wielemaker","fromEmail":"j.wielemaker@uva.nl","sentAt":"2008-06-30T11:44:09Z","receivedAt":"2008-06-30T11:44:09Z","isPatch":false,"sender":{"key":"j.wielemaker@uva.nl","avatar":null},"body":"Hi Jakub,\n\nSummarising, I think the conclusion is that git pack-refs has somehow\nbeen run on this repository, and being a bare one this is not a\nparticulary good idea right now. I have the impression I should `unpack'\nthem by putting the appriate files in heads (done) and tags (now still)\nand (re)move packed-refs.\n\n\tCheers --- Jan\n\nOn Monday 30 June 2008 13:26, Jakub Narebski wrote:\n> Jan Wielemaker wrote:\n> > On Monday 30 June 2008 12:10, Jakub Narebski wrote:\n> >> Jan Wielemaker writes:\n> >>> I'm a bit puzzled. I have a bare repository, somehow without any files\n> >>> in refs/heads.\n> >>\n> >> Do you have .git/packed-refs file?\n> >\n> > Its a bare repo, so I'll forget the .git. Yes, I have that file and\n> > it contains nice references, I checked a few by hand, and they have\n> > the same SHA1 as the files I copied.\n>\n> That is where the refs are instead of being in individual files under\n> refs/ - those are so called \"packed refs\".\n>\n> When a ref is missing from the traditional $GIT_DIR/refs hierarchy,\n> it is looked up in $GIT_DIR/packed-refs and used if found.\n>\n> I'm not sure why branches are also packed, because git used to pack\n> only tags and refs which were packed already.\n>\n> >> How do you have gc.packrefs set (\"git config --get gc.packrefs\")?\n> >\n> > This gives no output, so I guess the answer is 'no'.\n>\n> In git-config(1) you can find the following:\n>\n> gc.packrefs::\n> `git gc` does not run `git pack-refs` in a bare repository by\n> default so that older dumb-transport clients can still fetch\n> from the repository. Setting this to `true` lets `git\n> gc` to run `git pack-refs`. Setting this to `false` tells\n> `git gc` never to run `git pack-refs`. The default setting is\n> `notbare`. Enable it only when you know you do not have to\n> support such clients. The default setting will change to `true`\n> at some stage, and setting this to `false` will continue to\n> prevent `git pack-refs` from being run from `git gc`.\n>\n> Unless something changed (and git Documentation was not updated) git\n> should not pack refs by default.\n>\n> >> What are the git version on the\n> >> clients that have trouble accessing repository, and what protocol\n> >> (transport method) do they use: \"file\", http(s)/ftp, git, ssh?\n> >\n> > I could reproduce the problem using the same git that maintains the bare\n> > repository, which started as git-1.5.3.4 and is now 1.5.6.rc3. It\n> > reproduces both using local file access and ssh. Other people have a\n> > variety of versions. I know of concrete problems using the stable 1.5.5\n> > over ssh.\n> >\n> > The problem that I can't even clone is indeed after running a git-gc.\n>\n> Hmmm... strange. I know that there can be problems with older clients\n> (those which do not understand packed-refs format) accessing\n> repositories with packed refs (without loose refs) via \"dumb\"\n> protocols. Problems with accessing repositories locally and via ssh\n> might point to some troubles with permissions and ownership of\n> $GIT_DIR and $GIT_DIR/packed-refs.\n>\n> git-gc can pack refs, that is what running it causes problems.\n>\n> > So, now I have all branch head refs double (in packed-refs and in\n> > refs/heads). That explains ar least where they come from, but it fails\n> > to explain why doing exactly the same using the same git on the same\n> > repo as two users yields a different result. Of course, besides the\n> > desire to understand how his works, I'm mostly interested in how to fix\n> > this :-)\n>\n> Unfortunately as it is not caused by the issue I thought about I cannot\n> help you further...\n"},{"id":"81747","messageId":"20080630115026.GA1336@atjola.homenet","threadId":"14222","inReplyTo":"200806301326.12140.jnareb@gmail.com","subject":"Re: Corruption: empty refs/heads in otherwise filled repo: cannot clone?","fromName":"Björn Steinbrink","fromEmail":"b.steinbrink@gmx.de","sentAt":"2008-06-30T11:50:26Z","receivedAt":"2008-06-30T11:50:26Z","isPatch":false,"sender":{"key":"b.steinbrink@gmx.de","avatar":"https://avatars.githubusercontent.com/u/230962?v=4"},"body":"On 2008.06.30 13:26:10 +0200, Jakub Narebski wrote:\n> Jan Wielemaker wrote:\n> > On Monday 30 June 2008 12:10, Jakub Narebski wrote:\n> >> Jan Wielemaker writes:\n> >>>\n> >>> I'm a bit puzzled. I have a bare repository, somehow without any files\n> >>> in refs/heads.\n> >>\n> >> Do you have .git/packed-refs file?\n> > \n> > Its a bare repo, so I'll forget the .git. Yes, I have that file and\n> > it contains nice references, I checked a few by hand, and they have\n> > the same SHA1 as the files I copied.\n> \n> That is where the refs are instead of being in individual files under\n> refs/ - those are so called \"packed refs\".\n> \n> When a ref is missing from the traditional $GIT_DIR/refs hierarchy,\n> it is looked up in $GIT_DIR/packed-refs and used if found.\n> \n> I'm not sure why branches are also packed, because git used to pack\n> only tags and refs which were packed already.\n> \n> >> How do you have gc.packrefs set (\"git config --get gc.packrefs\")?\n> > \n> > This gives no output, so I guess the answer is 'no'.\n> \n> In git-config(1) you can find the following:\n> \n> gc.packrefs::\n> `git gc` does not run `git pack-refs` in a bare repository by\n> default so that older dumb-transport clients can still fetch\n> from the repository. Setting this to `true` lets `git\n> gc` to run `git pack-refs`. Setting this to `false` tells\n> `git gc` never to run `git pack-refs`. The default setting is\n> `notbare`. Enable it only when you know you do not have to\n> support such clients. The default setting will change to `true`\n> at some stage, and setting this to `false` will continue to\n> prevent `git pack-refs` from being run from `git gc`.\n> \n> Unless something changed (and git Documentation was not updated) git\n> should not pack refs by default.\n\nI had a quick glance over the log for builtin-gc.c, and that changed\nlast year in:\n\n56752391 Make \"git gc\" pack all refs by default\n\nBjörn - who lacks time to provide a doc update patch :-(\n"},{"id":"81749","messageId":"200806301403.57900.jnareb@gmail.com","threadId":"14222","inReplyTo":"200806301344.09938.J.Wielemaker@uva.nl","subject":"Re: Corruption: empty refs/heads in otherwise filled repo: cannot clone?","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-06-30T12:03:57Z","receivedAt":"2008-06-30T12:03:57Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Jan Wielemaker wrote:\n\n> Summarising, I think the conclusion is that git pack-refs has somehow\n> been run on this repository, and being a bare one this is not a\n> particulary good idea right now. I have the impression I should `unpack'\n> them by putting the appriate files in heads (done) and tags (now still)\n> and (re)move packed-refs.\n\nIf you use new enough git both on server and on clients it should\nnot have problems with packed-refs. I would rather check permissions\nof $GIT_DIR and $GIT_DIR/packed-refs.\n\nIf \"git show-ref\" and \"git for-each-ref\" works, then \n\"git ls-remote \" should work, and git-fetch/git-clone\nshoulw work too...\n\n-- \nJakub Narebski\nPoland\n"},{"id":"81750","messageId":"200806301420.12872.J.Wielemaker@uva.nl","threadId":"14222","inReplyTo":"200806301403.57900.jnareb@gmail.com","subject":"Re: Corruption: empty refs/heads in otherwise filled repo: cannot clone?","fromName":"Jan Wielemaker","fromEmail":"j.wielemaker@uva.nl","sentAt":"2008-06-30T12:20:12Z","receivedAt":"2008-06-30T12:20:12Z","isPatch":false,"sender":{"key":"j.wielemaker@uva.nl","avatar":null},"body":"On Monday 30 June 2008 14:03, Jakub Narebski wrote:\n> Jan Wielemaker wrote:\n> > Summarising, I think the conclusion is that git pack-refs has somehow\n> > been run on this repository, and being a bare one this is not a\n> > particulary good idea right now. I have the impression I should `unpack'\n> > them by putting the appriate files in heads (done) and tags (now still)\n> > and (re)move packed-refs.\n>\n> If you use new enough git both on server and on clients it should\n> not have problems with packed-refs. I would rather check permissions\n> of $GIT_DIR and $GIT_DIR/packed-refs.\n\nThere is no permission problem, as a I proved by doing a recursive copy\nof the whole repo (cp -a, no errors) and the problem prevails on the\ncopy. A serious scan for permission errors was my first step. Almost\nlooks like something in the environment, but I can't find anything weird\nthere either.\n\n> If \"git show-ref\" and \"git for-each-ref\" works, then\n> \"git ls-remote \" should work, and git-fetch/git-clone\n> shoulw work too...\n\nThanks for the pointers. I'll leave it for now (busy ...). I now\nunderstand how it is supposed to work. If I can find time at some point\nI'll run strace in different setups to see whether I can spot the\ndifference.\n\nAnyway, thanks to your help I am fairly confident my repo is in good\nshape again.\n\n\tCheers --- Jan\n"},{"id":"81771","messageId":"alpine.LNX.1.00.0806301528500.19665@iabervon.org","threadId":"14222","inReplyTo":"200806301420.12872.J.Wielemaker@uva.nl","subject":"Re: Corruption: empty refs/heads in otherwise filled repo: cannot clone?","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2008-06-30T19:33:50Z","receivedAt":"2008-06-30T19:33:50Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Mon, 30 Jun 2008, Jan Wielemaker wrote:\n\n> On Monday 30 June 2008 14:03, Jakub Narebski wrote:\n> > Jan Wielemaker wrote:\n> > > Summarising, I think the conclusion is that git pack-refs has somehow\n> > > been run on this repository, and being a bare one this is not a\n> > > particulary good idea right now. I have the impression I should `unpack'\n> > > them by putting the appriate files in heads (done) and tags (now still)\n> > > and (re)move packed-refs.\n> >\n> > If you use new enough git both on server and on clients it should\n> > not have problems with packed-refs. I would rather check permissions\n> > of $GIT_DIR and $GIT_DIR/packed-refs.\n> \n> There is no permission problem, as a I proved by doing a recursive copy\n> of the whole repo (cp -a, no errors) and the problem prevails on the\n> copy. A serious scan for permission errors was my first step. Almost\n> looks like something in the environment, but I can't find anything weird\n> there either.\n\nThat's a \"cp -a\" as somebody else, I assume? (If it were as you or root, \nyou'd generate a copy of the repository with any permission problem \nunchanged, since -a includes -p.)\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"81773","messageId":"200806302146.23120.J.Wielemaker@uva.nl","threadId":"14222","inReplyTo":"alpine.LNX.1.00.0806301528500.19665@iabervon.org","subject":"Re: Corruption: empty refs/heads in otherwise filled repo: cannot clone?","fromName":"Jan Wielemaker","fromEmail":"j.wielemaker@uva.nl","sentAt":"2008-06-30T19:46:22Z","receivedAt":"2008-06-30T19:46:22Z","isPatch":false,"sender":{"key":"j.wielemaker@uva.nl","avatar":null},"body":"On Monday 30 June 2008 21:33:50 Daniel Barkalow wrote:\n> On Mon, 30 Jun 2008, Jan Wielemaker wrote:\n> > On Monday 30 June 2008 14:03, Jakub Narebski wrote:\n> > > Jan Wielemaker wrote:\n> > > > Summarising, I think the conclusion is that git pack-refs has somehow\n> > > > been run on this repository, and being a bare one this is not a\n> > > > particulary good idea right now. I have the impression I should\n> > > > `unpack' them by putting the appriate files in heads (done) and tags\n> > > > (now still) and (re)move packed-refs.\n> > >\n> > > If you use new enough git both on server and on clients it should\n> > > not have problems with packed-refs. I would rather check permissions\n> > > of $GIT_DIR and $GIT_DIR/packed-refs.\n> >\n> > There is no permission problem, as a I proved by doing a recursive copy\n> > of the whole repo (cp -a, no errors) and the problem prevails on the\n> > copy. A serious scan for permission errors was my first step. Almost\n> > looks like something in the environment, but I can't find anything weird\n> > there either.\n>\n> That's a \"cp -a\" as somebody else, I assume? (If it were as you or root,\n> you'd generate a copy of the repository with any permission problem\n> unchanged, since -a includes -p.)\n\nI'm an old time Unix guy :-). The failing user has a git-shell. I did\n(reconstructing from memory):\n\n\t% sudo -u failinguser /bin/bash\n\t\n\t% Use whoami to validate I was this user\n\t% git clone /home/git/pl.git\n\t\n\t% cp -a /home/git/pl.git tmp.git\n\t\n\t% git clone tmp.git\n\t\n\nFunny enough, as myself I can clone tmp.git!? I have only one git \ninstallation\nin /usr/local/bin.\n\n\tCheers --- Jan\n"}]}